First Class Cells

[somethingelseentire]

Hey,

My biggest pain points with the Observable runtime have always been mutable state and the implicitness of reactivity in general. Mutable and viewof feel like hacks that completely circumnavigate the reactive parts, relying on an additional event/notification mechanism that is then reintegrated into the runtime through Generators.observe.

Generators work reasonably well on the other hand and, as long as one is familiar with the underlying Javascript mechanics, provide a relatively clear and explicit picture of what is going on.

However, as of now they don’t compose, which makes them limited to cases where only a single generator is sufficient.

It would be extremely nice if there was a cell keyword, which returned a first class representation of said cell. This reified cell could then provide metadata like the name and comments, and contain methods to control and read said cell. The ability to force reevaluation for example is something which I could really use to implement a BibTex style reference system for Scientific Papers, which currently requires a reload.
Most importantly it could provide a method like nextValue() which returns a promise for the next value in the cell.

This would allow for things like:

changingState = {
  let i = 0;
  while (true) {
    yield Promises.delay(0, i++);
  }
}

dependentState = {
  let i = 0;
  while (true) {
    yield [i++, await cell changingState.nextValue()];
  }
}

Right now, the pair would always be [0, x], as it’s not possible to prevent full reevaluation., since mutable doesn’t support generators (it doesn’t provide the correct semantics anyways because we DO want to yield whenever changingState changes, we just don’t want i to be reset).

Being able to explicitly await cells would make a lot of code a lot easier to understand and straightforward to write.

Cheers!

[Fil]

Using this you can do something like:

dependentState = {
  let i = this ? this[0] : 0;
  while (true) {
    yield [i++, changingState];
  }
}

it probably doesn’t answer your request but it’s another way to solve the issue.

[somethingelseentire]

Yeah, this however suffers from its inversion of control and makes code quite messy and hard to understand, especially when showing it to people new to observable.

Edit: Also your changes introduced a subtle bug, which nicely illustrates the point .
Due to the i++ you will “loose” one increment per full reevaluation, repeating the previously returned one, which is pretty hard to spot.

[mbostock]

Can you provide a more “real world” example to motivate this use case? It’s not clear to me why knowing the number of times that changingState has changed (since the notebook loaded? since the changingState cell was edited?) is useful.

There are a variety of ways to accomplish things like this, but I think the discussion might be productive with a more concrete example. Thanks!

Also, yes, I don’t recommend using this. Probably the closest translation of the example given would be using mutable, but again, it’s a little abstract to say.

mutable dependentState = [-1, undefined]

{
  const [i] = mutable dependentState;
  mutable dependentState = [i + 1, changingState];
}

Edit: I see you mentioned BibTex, which is interesting, but I’m not seeing how this translates to practice.

[mbostock]

Here’s a related technique I sometimes use, if it helps:

Reference Tracking

[somethingelseentire]

Hey Mike,

Thanks for taking the time !

As for a concrete example of the generator use case:
I’m writing the GUI for a NLP parser in observable, including a visualisation for the search graph, and would like to keep a d3.simulation object in one cell with the current nodes and their positions yielded with every tick. I then have several visualisations (different canvases) render these nodes whenever they change with different visuals and interactivity. The cell managing the simulation itself is also dependent on another stateful “search-space” cell which adds new nodes as they are explored.
I’ve experimented with various ways to propagate these changes over the time, and found while(true) ... await ... yield ... to be the most straightforward pattern to explain to “non-observablers”.
Mutable doesn’t really solve this case because it either provides no reactivity at all, or just regular reactivity which has the same “how to keep the boilerplate state” problem which can only be solved with this.

As for the BibTex example:
Your example notebook doesn’t work for me (things don’t get removed when I delete them), but I get the idea.
The issue I’ve had when building a Table Of Contents and a Bibliography, is that repeated reevaluation of cells not only introduces and removes elements, but it messes with the ordering (especially non-ideal for TOCs, so I’m currently including X.Y.Z header labels and parsing them).
Cells might get moved around, or they might get reevaluated in a random order.
Being able to inspect the cells directly would either allow to also get info on their ordering (this doesn’t catch inner cell ordering), or to remember which cells partake in the TOC/Bibliography and clearing the TOC/Bibliography-state and forcefully reevaluate all of them whenever one of them reevaluates.

Hope this makes things a bit clearer

[tomlarkworthy]

This sounds like you are fighting the representation issue between:

a stream of states Vs. a stream of delta updates

If you underlying state is a set of objects. you can either transmit ADD/DELETE operations (stream of updates) OR, you can transmit the whole set each time (stream of states).

Things are usually simpler in the stream of states world, as the representation is fully encapsulated. There is no outside state you need to track in orderer to decode the updates. It’s robust against missing events (important in distributed environments). But I have noticed many people try to go the stream of updates route for either performance arguments or I think its more intuitive because its nearer the event model we have on UI.

Performance-wise you can make stream of states competitive, because if your state uses a functional immutable representation you can usually make the stream of states very close performance wise. Stream of states is generally easier to compute with downstream as there is not “hidden state” to track. Coarsely, the stream of states approach is the functional one, and the stream of updates is the imperative one.

If you trying to streamify a dikjstra search for instance, but you want to highlight the most recent expansions, you might transmit the whole expansion history each tick e.g.

dijstraSnapshot = {
   currentTick<int>
   closed<set>
   open<set>
   expansion_history<int => set>
}

If you worry about performance then you can represents complex objects efficiently with Immutable.js leading to log(n) updates, but I personally would not worry about it.

[somethingelseentire]

@tomlarkworthy Interesting but a bit besides the point .

The contents of the stream (snapshots vs. deltas), are orthogonal to the mechanics of processing them (with stateful operators).

In my concrete example I’m also streaming the entire snapshot of node positions. The issue is that it’s really cumbersome to have “stateful” cells / data-flow operators (in my case they have a DOM element which needs to be retained), because you either use this (easy to get wrong because control flow is very implicit, and unintuitive for newcomers because it overloads this in a non-standard way), or you have to use mutable which brings in the need for side effects and additional message-passing mechanisms outside of control flow e.g. EventTarget and “input” events as done for viewof.

[tomlarkworthy]

For stateful DOM I “reconcile” against “this” Hypertext literal reconciliation with nanomorph / Tom Larkworthy / Observable. I think thats one of the few good uses of ‘this’ as it pushes the state management to a general library component.

[mootari]

Related discussion: Refresh cells on a timer - #9 by mootari

[mbostock]

@somethingelseentire

Your example notebook doesn’t work for me (things don’t get removed when I delete them), but I get the idea.

Ah, whoops, I was generating a Text node rather than an Element, so it should work now…

[edeboursetty]

There is a hacky way to do what you need.
See here: Partial cell update hack / Etienne de Boursetty / Observable

The idea is that while variables ticks, their properties do not.
Then you can add a simple callback mechanism to await for the property to update.

Edit: I’ve packaged it all in an observer object that you can easily use elsewhere. See the notebook.

[mootari]

Yes, you can use a basic PubSub pattern:

PubSub

However you lose the state management offered by the Runtime and have to ensure that handlers get removed on invalidation. Also, no dependency resolution on import.

[edeboursetty]

Yeah…
The version I made doesn’t go through events though, so I don’t think it has the invalidation problem.
The rest is the price you pay for working around the runtime…