Visual Dataflow!

Thanks Jeremy. Speaking only for myself, I would prefer a colored background (or changed font color) to an underline, I just would like the coloring to be very subtle. YMMV.

My convention in many notebooks has been to do

Title = md`
## Some extended section title

...`

See for example Cubic Spline / Jacob Rus | Observable

It would be good if one could drag and drop the mini-map’s cell to reorder his notebook.

6 Likes

Is there an explanation on the minimap somewhere? I just don’t get it.
Left is inputs, right is outputs, and then you have links with the affected nodes, took a while but I’ve figured that out (directed edges would be really nice, or some kind of label above otherwise it’s really not intuitive).

But what I really don’t get is, that cell move around from left to right even if they’re not connected to the selected cell, why, how, what does it mean?

It’s pretty yeah, but my brain hurts.

I wish I could just disable it and get rid of the distraction :frowning:

Edit:
I think one of the reasons why it’s so distracting is also that it feels like a scrollbar that’s stuck in your peripheral vision which is just annoying.

This is exactly what I want. I really like the red spot which helps to immediately identify any cell which threw an exception w/o walking through the whole notebook. I’m also keen to see if this added minimap help me sort out the relations btn cells of a notebook, before it reaches a deadlock (notebook unresponsiveness), given that there is no cyclic dependency warnings, which happens sometimes for large notebook and it is sometimes hard to avoid until I ran into a deadlock. Great work! :slight_smile:

1 Like

I can understand the aesthetic preference, but using subtle colors really is problematic for accessibility - see also my feedback a while ago about the former diffing color scheme.

Subtle background coloring is the bane of my protanomalous existence :wink:

2 Likes

The new minimap is great. I even found myself double-clicking a local variable hoping to jump to the let that defined it. :slight_smile:

My natural inclination was to cmd+click to go to definition rather than alt+double-click, not sure why.

+1 to the minimap being difficult to use with longer notebooks. If you want to see all cells that depend on the chosen cell, for example, or to jump to a dependency that you know exists, it’s often out of view and you need to scroll the page to find it in the minimap.

+1 to the floating headers getting in the way and taking up valuable screen space.

+1 to dragging in the minimap to reorder, and maybe something like an option to eg. topologically sort all dependencies of a cell below the cell (“Tidy dependencies”? though maybe this wants to be a temporary reordering, in case you have the notebook organized in a thoughtful way otherwise).

It might be useful if <style> and <link> cells were annotated — often I want to add a new style rule and have to hunt around for the cell. It’s not possible to search for the “<style>” text since it’s added with CSS.

3 Likes

You’re 100% right that the left/right alignment of cells not connected by wires is probably the least obvious aspect of the minimap at first.

For a cell like this on the right, the alignment is meant to show that, even though the cell does not directly reference the focused cell, it is indirectly dependent on the focused cell (i.e. somewhere up the chain of its dependencies, it is connected to the focused cell). So, re-evaluating the focused cell will result in the re-evaluation of the cell on the right. Similarly, the focused cell is indirectly dependent on the left-aligned cells without wires, so re-evaluating those cells will result in the re-evaluation of the focused cell.

The Introducing Visual Dataflow notebook has some helpful information regarding these minimap details, particularly this cell of the notebook, which illustrates how the minimap will behave when displaying various different cell relationships.

However, I totally agree with you that there’s more we can do on the documentation front to better explain the logic behind the minimap’s behavior and more we can do within the UI to make it intuitive. Thanks again for sharing your thoughts — we really appreciate it :+1:

1 Like

Ok after having some more time with it I changed my mind, I don’t dislike it, I absolutely hate the entire thing.

Don’t get me wrong, I love what you do, and I love Observable, so I really don’t want to hate on this. But this feature is a solution in search of a problem. Worse it doesn’t solve any and only generates new ones, and it makes my notebook take longer to load and more distracting which just isn’t cool if you want to share them with customers.

rant>

Let’s be real, the sublime mini code view this apparently was inspired by is chart junk,
yes it looks pretty, and it feels very csi-miami-hackerish and cool, but it doesn’t work.
Are you really gonna get a better overview of your code by looking at a visualisation of how long each line in your file is? Shouldn’t you split that file up if it’s too big to scroll through? And aren’t you navigating by searching for keywords, or by jumping to definitions anyways?
Every sublime user I know has that features disabled because it adds no value and it costs screen estate.

The mini-map feature has the same problem. What’s the use-case?

Debugging? Once you’re at the point where you need to debug your dependency graph, you want to have a proper visualisation of your dependency graph, because you’re suddenly in “debug mode” with special debugging needs. In that scenario having the tiny mini map just feels cramped, like somebody is suddenly asking you to type with one hand only, and when frustration is at its peak you really don’t want that.
Also during debugging you want to be able to isolate and trace individual effects as they propagate. With the mini-map you either:
A. execute something and watch the little dots blip, but then everything else keeps running at the same time, so there is so much noise that it’s like looking at an ethernet switch in full load, trying to figure out which port has a particular traffic
B. “explore” the tree by repeatedly clicking on nodes, but then you still have to remember all the connections and build the dependency graph your head, so what’s the win here?
So if it’s not as useful for debugging as a plain old dependency graph, maybe there’s a different use case?

Maybe it’s navigation? So that you understand how your program is interconnected?
The cool thing about the literate programming style of observable is that it allows you to write a narrative. Humans are insanely capable of remembering stuff with the help of narratives, all the memoization tricksters use narratives and spatial narratives to remember thousands of objects. So if you’re not capable of keeping the notebook structure in your head via that narrative you’re sure as hell not gonna remember all the “meaningless” connections of the dots to form a cohesive picture of what’s going on.

It’s like trying to figure out what’s going on through a keyhole, not only horizontally but also vertically because of scrolling. Like all good chart junk this looks amazing when you only have 10 cells, but once you actually start to need it, it just breaks down. Not sure about you, but my poor brain won’t make a connection between the dots once I have to scroll a few pagefuls to get to the next node.

So it doesn’t really add anything for navigation, and isn’t as useful as a plain old graph would be for debugging, so what does it cost?
Notebook load times have gone up by 10 seconds for large notebooks (https://observablehq.com/@bultmann/triplemq), for super active notebooks it’s like having a network switch just outside your field of view, which is f*ing ace when you try to concentrate, and once you use it it’s frustrating as hell.

It’s not even THAT aesthetically pleasing. Even Sublime had the wisdom to replace their scrollbars with their code viewer, because these things just don’t mix well. How many UX elements do you know that are positioned just around or below the scroll bars? Even this discussion page with its questionable date/post/navigation slider figured out that it should be as far away from the scrollbars as possible.

Looks like my regular Observable workflow will now incorporate opening the inspector and killing the entire thing outright, but hey I needed that one open anyways for debugging.

<rant

don’t take it personally. Having a strong opinion does not validate it. Maybe in future try to be more objective :slight_smile:

using subtle colors really is problematic for accessibility

From my perspective, visual distractions are also problematic for accessibility.

This is a big reason that most text editors have user-adjustable syntax highlighting. Then if there’s a feature that one person finds essential, another person thinks is occasionally useful but only during specific tasks, and a third person thinks is an actively distracting gimmick, all three can give that feature as much or as little visual emphasis as they like, when they like.

When trying to design a view which must be the same for not only all authors/editors but also all readers, I think it’s better to err on the side of conservatism.

A substantial visual emphasis on cell names can be quite helpful for at least some authors/readers in at least some circumstances (perhaps even most readers in most circumstances, who knows), but just speaking for myself I want to be able to see the highlights if I am looking for them but I don’t want them to call attention to themselves the rest of the time.

2 Likes

I’m sorry you hate it!! I know you discussed some of these issues with Toph, too, but I just wanted to quickly follow up on some of the difficulties you mentioned.

We totally agree that the minimap is frustrating with large notebooks. Making this better is at the very top of our list. It can be particularly frustrating when the wires connected to your focused cell go offscreen and require you to scroll the whole page to get to the end of them, or when the focused cell might not even be visible on the minimap due to our proportional scrolling.

Like Jeremy mentioned, we’ve made a few performance improvements and believe that there shouldn’t be any noticeable overhead from the minimap. We aren’t able to reproduce the loading issues you were encountering, but if you’re still having problems with this, please let us know and we’ll do our best to track down the issue.

One of our biggest motivations for developing this feature is that the reactivity of Observable can often be confusing. The fact that notebooks don’t evaluate linearly and the order of cells is arbitrary is a barrier to entry for a lot of people. Even for those who are by no means new to Observable, it can often be difficult to track down all of the dependencies of cells or to cleanly refactor code.

We wanted to create a tool that provides insight into how cells are connected, reinforces understanding of the notebook’s behavior, helps you discover issues in your code, and allows you to quickly navigate between related cells. We definitely don’t want users to feel like they need to memorize all the wires, dots, and connections between them — when it’s working well, the minimap should keep track of these things so that you don’t have to.

I know I’ve said this already, but we really do appreciate the feedback. We’ve been staring at the feature for so long internally that pushing it out and hearing what the community has to say really is the quickest path to making it better for all of us, so thanks for tolerating our growing pains :slight_smile:

3 Likes

In case it wasn’t clear, I think the feature is a great start! Hopefully you folks aren’t feeling discouraged.

I would be interested to hear at some point how non-Observable-using readers react to it if they just encounter miscellaneous notebooks… either just as informal feedback or even in some kind of formal usability study. I have a feeling it’s going to be at least a bit confusing for a decent number of first-time viewers, but I don’t have any immediate ideas for improvements.

2 Likes

making it optional and off by default would help simplify the notebooks for casual readers

1 Like

it can block access to the LIKE button

4 Likes

I must say, the underline does get cute here though:

2 Likes

Marking cells that do not partake in data-flow with a smaller dot didn’t go unnoticed :wink:
Keep up the good work :D!

2 Likes

Hi again, all! We wanted to give an update on some of the improvements we’ve made based on your feedback over the last 2+ weeks and what’s still on our to-do list.

You’ve likely already noticed one or two of these changes, but here’s what we’ve updated so far:

  • Smaller circles in the minimap de-emphasize cells with no name, inputs or outputs as they’re neither “connected” nor “connectable” to other cells.
  • Notebooks now perform better than they did before the launch of visual dataflow thanks to the deferred rendering of offscreen editors, among a few other improvements. Hopefully you’ll especially notice this change in your larger notebooks.
  • The minimap is now integrated with (1) visual mode to show all of the selected and focused cells, and (2) compare view to indicate which cells have been edited between two notebooks or notebook versions.

We’re still working to improve the scroll behavior of the minimap in long notebooks, as well as add some helpful shortcut keys to toggle back and forth between your previously focused cells.

And as usual, thank you thank you thank you for sharing all of your thoughts on this feature so far — we’re super excited to keep refining the minimap and see how it continues to evolve with all of your help. :blush:

4 Likes

I’m curious what the minimap would be like if focusing on a cell hid all unconnected dots.

The relative order of dots would still hint at notebook structure, and removing the unrelated dots would allow more room to show information specifically related to the dependency structure of the selected cell.

1 Like

we tried this during initial development and it felt disorienting; it messed with the mental map of map to territory, felt unstable, made it harder to be like, “wait, which one is that one?” … but i still like the general idea

2 Likes