"Streaming" Data Chart in D3

TL;DR: I’d like to use only D3 to create a dynamic chart that changes every “tick” of a simulation.

I want it to show all the data in the chart over time, “compressing” the chart, and automatically adjusting the x/y axes for the data to fit correctly.

Whew, that’s a lot! I haven’t found the right phrase for searching, but I did come up with two examples, one using Vega the other Plot.ly.

This one line change to a @tom "Streaming data into Vega " example:

And these plot.ly examples:

The key is this: using “raw” D3.

I realize avoiding use of a charting library may seem silly, but the reason for this is that my project team would like to use D3 for other purposes and thus learn more D3 details. We’d also like to avoid version conflicts with these libraries that often use older versions of D3. I’m being pushed to be the pioneer! : ) I hope it is a reasonable request.

They look like these:

That sounds like a reasonable enough thing you could build. What is stopping you?

my project team would like to use D3 for other purposes and thus learn more D3 details

Best way to learn is to dive in and try to build what you want.

1 Like

Hi, thanks for the response.

What is stopping you?

Well, being new to D3, just knowing the D3 & SVG “patterns” and “vocabulary”.

For example, I don’t know how to add points to a path to incrementally extend a line chart. Generally you add SVG/DOM elements (circles, rects etc) but a line chart I think is a single path of many lines, a poly-line. It might be easier to start with each point being a SVG rect or circle if they are easier to use.

I’d guess that each step would have to modify several transforms and scales, both for the chart and for the axes. Also would recalculate the scale extents.

Oddly enough, one problem is the right search phrase to use! I thought “stream” and “data” would do the trick but there is a “streamgraph” that isn’t what I want. I tried “time” and “series” and that was a bit closer. Basically knowing the D3/SVG/Charting world well enough to come up with the appropriate search has been elusive!

1 Like

start here: http://d3indepth.com/

1 Like

Whoa, that is amazing! Thanks for the pointer!

1 Like

Observable is the perfect environment to learn D3—no accident there!

You can create a cell that holds the streaming points, and have the plot in another cell that draws them. That takes care of the streaming part due to the reactivity between them. Just think about drawing the whole plot every time instead of incrementally extending the line (it’s fast).

In terms of actually drawing the plot, you mainly want to look at these parts of d3:

  • Use d3-scale to map your coordinates to the appropriate pixel on your plot. e.g. scaleX(point.time) to get the x pixel. scaleY(point.value) to get the y pixel.
  • Use d3-array’s extent function to figure out the min/max of a given set of points so you can adjust your scale to fit all points in view.
  • d3-axis will create an SVG axis for you based on the scaleX and scaleY objects you already created.
  • d3-shape lets you create a line function that takes your points and scale objects, and outputs either a SVG line or draws to a canvas.
  • d3-zoom adds a mouse listener to an element to handle panning and zooming of your plot. It hands you a transform object that you use to manipulate your scale objects. e.g. transform.rescaleX(scaleX)
1 Like

Sweet, thanks!

I did start with the Vega demo Tom made, along with the Vega-Lite tutorial on streaming. https://beta.observablehq.com/@backspaces/port-of-vega-lite-streaming-data-tutorial

I’ll use that as a basis for charting an AgentScript Flock model shortly. Then with that, I’ll try to translate that into D3 which should be interesting.

Here’s the Vega chart of the FlockModel:

I don’t have the reactivity right, I think. When I resize the notebook, things go wrong. Sigh. : )