🏠 back to Observable

Private notebooks organizing problems

As notebooks count grows, it’s harder to find the notebook you want

It would be good if

  • User would have possibility to include private (not yet shared) notebooks into collections
  • Notebooks in my profile to have pagination with the option to choose how many notebooks we want to see (standard component) - In this case, we could leverage native browser ‘find’ - ctrl+f component

44%20PM

  • Include private notebooks in search, if this search is initiated by the current user

It seems I can’t add a notebook that’s unpublished to a collection. The option isn’t available in the menu that appears when clicking the “…” button on the notebook page. I have a notebook I want to keep private and add to a specific collection for my team. Is there any way to add it to a collection?

Not yet; collections are currently for showcasing public notebooks, but I expect we’ll add support for private collections (or other ways of organizing private notebooks) in the near future.

1 Like

Update: Private (and shared) notebooks are now included in your personalized search results.

3 Likes

Update #2: Observable now has private collections (which are accessible only to you or your team, but may contain private, shared and public notebooks) … as well as the ability to add private and shared notebooks (invisibly to others) to your public collections.

3 Likes

Yes! Finally! Thank you! :partying_face:

  • One remark: It would be great if the Collections dialog in a notebook could stay open when new collections are created on the fly. Right now I have to add to a new collection, then reopen the dialog if I want to create and add to another.
  • Edit: Also, ?tab=collections is sorted by date of creation (not by name like the collections dialog). This makes it difficult to manage a larger number of collections and compare them.
  • Edit 2: The collections tab is still missing a type filter (you’re probably already aware of that).
  • Edit 3: Minor bug: If a collections dialog on a notebook page is reopened after collections have been added in another tab, the collection names might appear in the wrong order (i.e. not sorted alphabetically). Works as designed, assigned collections are listed first.
  • Edit 4: Private collections produce a slug that cannot be changed. If I want to rename a collection and stay consistent I have to create a new collection with the correct title/slug, reassign every notebook and delete the old collection.
1 Like

I’ve hacked together a snippet that adds any collections as linked tags to the current list of notebooks:

The snippet can be run in the dev tools (or turned into bookmarklet). Please be aware though that it performs a fetch request for each single notebook that is listed.

The snippet:

{
  const options = {
    // Assign custom colors by collection prefix.
    colors: {
      '#status-': '#dcfdcb',
      '#type-': '#fbdccd',
      ':Project': '#dff0ff'
    }
  };
  for(let n of document.querySelectorAll('.listing-grid')) attach(n, options);

  async function attach(node, {colors = {}}) {
    purge(node);
    node.appendChild(getStyle(colors));
    const user = await request('user');
    const isOwner = slug => user && isPrefix(`@${user.login}/`, slug);

    findItems(node).map(async ({node: n, slug}) => {
      const data = await request('document/' + slug + (isOwner(slug) ? '@latest' : ''));
      if(!data.collections || !data.collections.length) return;
      const w = injectWrapper(n);
      for(let c of data.collections) w.appendChild(createTag(c));
    })
  }

  // Returns a list of notebook teasers and their slugs.
  function findItems(node) {
    const items = node.querySelectorAll(':scope > .flex-auto');
    return Array.from(items)
      .map(n => {
        const a = n.querySelector('a[aria-label][href]');
        return { node: n, slug: a && getSlug(a.pathname) };
      })
      .filter(o => o.slug);
  }

  // Removes previously attached elements.
  function purge(node) {
    const s = '[data-type="collection-wrapper"], style[data-type="collections"]';
    for(let n of node.querySelectorAll(s)) n.parentNode.removeChild(n);
  }

  // Extracts identifying part from a notebook path.
  function getSlug(pathname) {
    return (pathname.match(/^\/d\/([a-f0-9]{16})$/) || pathname.match(/^\/(@[^\/]+?\/[^\/]+?)$/) || [])[1];
  }

  // Injects collections wrapper into an item.
  function injectWrapper(node) {
    const w = html('<div class="flex flex-wrap ph2 pt2 pb2 f7 b--black-10 bt" data-type="collection-wrapper">');
    return node.querySelector(':scope > .bg-white').appendChild(w);
  }

  function createTag(collection) {
    const a = attr(html('<a data-type=collection class="dib nowrap br2 mb1 bg-light-gray near-black no-underline mr1 ph1 lh-f6">'), {
      title: `${collection.owner.name} / ${collection.title}`,
      style: `--avatar: url(${collection.owner.avatar_url})`,
      'data-count': collection.document_count,
      'data-title': collection.title,
      href: `/collection/@${collection.owner.login}/${collection.slug}`,
    });
    return a.textContent = collection.title, a;
  }

  // Creates style element for custom CSS.
  function getStyle(prefixColors = {}) {
    const s = html('<style data-type=collections>');
    s.textContent = `
      a[data-type="collection"] {border:1px solid rgba(0,0,0,.05); overflow: hidden}
      a[data-type="collection"]:before {
        content: "";
        background: var(--avatar) center/contain no-repeat;
        width: 16px;
        height: 16px;
        border-radius: var(--border-radius-circle);
        margin-right: var(--spacing-extra-small);
        vertical-align: text-top;
        display: inline-block;
      }
      a[data-type="collection"][data-count]:after {
        content: " ("attr(data-count)")";
      }
      ${Object.entries(prefixColors).map(([prefix, color]) => `
        a[data-type="collection"][data-title^="${prefix}"] { background-color: ${color} }
      `).join('')}
    `;
    return s;
  }

  // Tests for string prefix.
  function isPrefix(prefix, str) {
    return str.slice(0, prefix.length) === prefix;
  }

  // Runs authenticated request against api.observablehq.com.
  async function request(path) {
    const opts = {credentials: 'include'};
    const url = 'https://api.observablehq.com/' + path.replace(/^\//, '');
    return (await fetch(url, opts)).json();
  }

  // Sets attributes on an Element.
  function attr(node, attrs) {
    return Object.entries(attrs).map(([k,v]) => node.setAttribute(k,v)), node;
  }

  // Creates an Element from an HTML string.
  function html(markup) {
    return document.createRange().createContextualFragment(markup).firstChild;
  }
}
1 Like

I would add it as an tamperMonkey script, if there weren’t too much network calls …

I am worried about Observable’s server load :confused:

Yep, that’s what kept me from running it at all time. The number of automated requests might violate Observable’s TOS.

But for the moment it helps me a lot to discover notebooks that haven’t been sorted into collections yet.

1 Like

@mootari, Thanks for the considerate feedback!

It should all be sorted alphabetically. Are you still seeing a different sort somewhere?

Fixed, the collections tab now has a type filter.

This, and your extremely nifty hack that displays collections as links on the notebook listing, are things we’re going to have to think about further…

Yes, unfortunately. It seems that collections in the collections tab are sorted by their slug, not the title.

Thanks for the quick fix!

Being able to create private notebooks has already been a huge help - I finally feel like I have a chance to get back control over the mess that has accumulated over the months. At the moment my conventions for collection names are:

  • prefix “:”: a group of notebooks (i.e. a notebook should only have one group assigned at a time), e.g. “:Project | GIF Capturing”, “:Project | Clip Space”
  • prefix “#type-”: type of a notebook (only one assigned), e.g. “#type-sketch”, “#type-project”, “#type-support”, “#type-iteration” (for forks/variants)
  • prefix “#status-”: temporary state of a notebook (multiple assignable), e.g. “#status-wip”, “#status-broken
  • everything else: public collections

I’ve tweaked my script some more. The output currently looks like this:

But I’ve also quickly hit the limit of what can be done reasonably with collections, especially on the collections tab. The large (and frequently changing) thumbnails add a lot of visual clutter that makes it difficult to find a specific collection:

And the size of the collections dialog is heavily restricted both horizontally and vertically:


(Also, I’ve noticed that my name always appears on top - this is probably a Teams feature?)

To fight the problem of both notebook and collection slug changes my recommendation would be to introduce redirects:

  1. When a slug changes, store the old slug and notebook/collection id as slug redirect.
  2. Test if the new slug duplicates an existing slug redirect:
    • if slug exists and has same entity, delete old redirect
    • if slug exists and has different entity, perform usual adjustments to new slug
  3. When a request hits a nonexistant slug, check for redirect slugs. If found, store access timestamp for slug and issue a 301 redirect to current slug
  4. During regular cron intervals remove slug redirects that no longer have an ID associated or haven’t been accessed in N months.

Edit: Something I forgot to mention: Right now there’s no direct navigation path from a collection detail back to the collections tab; instead I have to click on my name, then navigate to the tab.

2 Likes

There’s an oddity that I haven’t figured out yet. Some requests against the /document route return the data with an empty collections set, even though a collection is assigned:

  • https://api.observablehq.com/document/f990a4814916f644 has collection
  • https://api.observablehq.com/document/@mootari/test-input-events doesn’t have collection
  • https://api.observablehq.com/document/@mootari/test-input-events@latest has collection (this is the route used by Observable)

Note: Any @revision routes don’t even include the collections key - this is probably an optimization.

All requests were made with:

(await (await fetch(path, {credentials:"include",cache:"no-cache"})).json()).collections

This behavior causes my snippet to drop some collection tags, so any insight is highly appreciated. :slight_smile:

What you’re seeing is our public caching.

For public notebooks, we use different URLs for requests from the general public than for requests notebook owners; the former contain only public data and are cached, the latter may contain private data and are not.

In this case, it’s a private collection that is assigned to the notebook — right?

1 Like

Yup, that’s right. I guess I’ll have to introduce some logic to determine wether the current user is the owner of the listed notebook (and can can safely append @latest to the API route). Edit: Fixed, updated snippet.

I ran a few more tests: The sorting appears to ignore any non-letter characters. This causes a different order compared to the collections dialog (and also mixes my prefixed and standard collections):

  • “Test A0”
  • “Test A1” / “#Test A1” / “Test#A1”
  • “Test A2”

Edit: Here’s a screenshot from the Visualization collection with added collection tags. I believe it demonstrates nicely how these tags can help discover related collections and notebooks:

2 Likes