Flickering elements in chart with streaming data

Firstly, it is a special honor to be the first to post on this new board about data applications!

I am building a financial dashboard that uses streaming data and I hit an obstacle I couldn’t find a way around for a while. I would appreciate any help I can get.

The issue: whenever the data updates via the stream, some elements flicker in and out of existence. This likely happens because they are refreshed to incorporate the new data.

The most noticeable case is the crosshair on the main chart of the dashboard:

CleanShot 2024-09-03 at 06.37.10

A more subtle issue is that about 10-15% of clicks on a set of buttons fail (i.e., the associated function does not trigger when the button is clicked). This failure seems related to the flickering issue, as the buttons also flicker each time the data refreshes:

CleanShot 2024-09-03 at 06.43.50

:point_up_2:So, here, I believe some clicks coincide with the refresh window, leading to the failed clicks.

The flicker is also visible when inspecting DOM:
CleanShot 2024-09-03 at 06.46.58

Has anyone encountered something like this before and/or can help me resolve it? The issue might be due to my clumsy implementation of the streaming function or an incompatibility with the Observable Framework. If you’d like to take a look at the app yourself, it is here.

I’m not sure if it is possible to directly inspect framework code once it’s deployed. If it is not, here is the source code for a minimal version I made for this post. Specifically, the code for the streamPrice function is here, and its implementation to the chart is here (price is passed via the priceData parameter).

I would appreciate any help on this issue. Thanks in advance for any time you may spare.

1 Like

I don’t have a direct answer for you, but if sub-second updates aren’t a hard requirement for you then I’d consider debouncing your data stream so that updates are only yielded every few seconds, decreasing the number of interruptions.

(To be honest, I’m actually surprised that the crosshair rerenders without any mouse movement!)

1 Like

The buttons are regenerated each time the charts are updated, because they are all part of the same html code block. If you inline the HTML instead, this problem disappears. (You’ll need to remove all the newlines to make markdown happy.)

The flickering of the crosshair mark is a more complicated problem, because this mark relies on the pointer transform, which is asynchronous. We can make it “resync” a tick earlier with a hack (which I’ll share in a moment), and eliminate the flickering. But this approach will not work with, say, tooltips, as in that case we’d lose the state of the tooltip.

The fundamental issue here is that we are throwing out the whole chart and recreating a new one each time new data is streaming in. Instead, we’d want to update the chart “in place”, which at the moment requires a lot of effort because all the marks depend on the x scale, which in turn depends on the date range of the dataset.

This is something we want to work on for Plot, but we haven’t made much progress yet.

A detail: instead of calling pass(), to avoid the implicit display, you could just add a semicolon to your code block.

1 Like

Here’s a patch with the two changes described above:

Thank you very much @mootari @Fil for your help and for both workarounds!

Inline HTML in Markdown

Ah, an interesting fix! I assume that it works because each HTML element is updated separately unless grouped by an HTML code fence.

Resync

I really appreciate the explanation and the patch. For others who may have a similar problem, I have implemented your patch in the app, with the updated source code (and the initial version) here.

Despite quite a bit of effort to grasp the precise logic of the workaround, I still only partially understand it. For example, I am uncertain about the exact significance of the changing parameter values in the implementation of the resync function. I am also unclear on how the render property operates. I attempted to locate some documentation on it, but was unsuccessful. Could you perhaps point me to it if it exists?

Here is a version of your workaround, adapted to the style of my code base. If you are interested and would spot anything misunderstood (e.g., an inaccurate variable name), I would really appreciate a nod to it:

/**
 * Prevents flickering of the crosshair mark by resyncing the pointer transform.
 * This approach ensures that the crosshair mark is updated a tick earlier.
 * Note: This will not work with elements like tooltips, as they would lose their state.
 *
 * Different indices for the 'resync' function calls are used to track the state
 * independently for each mark type (e.g., crosshair X line, crosshair Y line,
 * crosshair X text, and crosshair Y text). This helps manage their transformations individually.
 *
 * @param {number} i - Index to track the previous value for resync.
 * @param {Array} renderHistory - An array to hold the previous values during the iteration of the function.
 * @returns {function}
 *
 * @author This function is based on a workaround suggested by Fil @ Observable.
 *   See the discussion here: https://talk.observablehq.com/t/flickering-elements-in-chart-with-streaming-data/9733/2
 *
 * Example usage:
 *
 * ```javascript
 * let renderHistory = []
 * // ...
 *
 * // Crosshair X Line
 * Plot.ruleY(
 *     OHLCVData,
 *     Plot.pointerY({
 *         px: "time",
 *         y: "close",
 *         stroke: "orange",
 *         strokeWidth: 1,
 *         strokeDasharray: "4,4",
 *         render: resync(renderHistory, 0)
 *     })
 * ),
 *
 * // Crosshair Y Line
 * Plot.ruleX(
 *     OHLCVData,
 *     Plot.pointerX({
 *         x: "time",
 *         py: "close",
 *         stroke: "orange",
 *         strokeWidth: 1,
 *         strokeDasharray: "4,4",
 *         render: resync(renderHistory, 1)
 *     })
 * ),
 * // ...
 * ```
 */
function resync( renderHistory, i ) {

    const resyncFunction = ( index, scales, values, dimensions, context, next ) => {

        const svgIsConnected = context.ownerSVGElement.isConnected

        // If the SVG is not connected, use the saved previous index value.
        if ( !svgIsConnected ) {
            index = [ renderHistory[i] ]
        }

        // Otherwise, save the current index value for the next call.
        if ( svgIsConnected ) {
            renderHistory[i] = index[0]
        }

        // Proceed with the next function in the render chain.
        return next( index, scales, values, dimensions, context )
    }

    return resyncFunction
}

Crosshair Synchronization Between Charts

Like tooltips, I assume crosshair pinning is also affected due to the workaround, as I can no longer pin the crosshair for the charts for which I applied the workaround:

CleanShot 2024-09-05 at 04.08.30
:point_up_2:(The top chart has the workaround implemented; bottom chart is still the old, flickery crosshair code. I can pin the crosshair in the bottom chart.)

This is an acceptable tradeoff, but I expect it to complicate things a bit. Earlier, I spent a significant amount of time trying to synchronize the crosshair locations between multiple charts, but I was unsuccessful. The best result I achieved was making the first chart broadcast its crosshair position to the second chart, which then used it to draw the crosshair. However, there was a considerable delay between the two charts, and the implementation was problematic, so I had reverted the changes. I expect the current fix to potentially complicate things further when I try again. Could you perhaps be able to provide some guidance on how to achieve crosshair location synchronization between multiple charts in this scenario, given the current workaround?

(If you prefer that I post this in a new thread or change the thread title, since this may be considered a new issue, please let me know.)

Many thanks for reading and for any further help or comments in advance.

Yes the workaround only avoids the flickering; it does not maintain state (the pinning), because it does not address the fundamental issue I described above (we are throwing out the whole chart and recreating a new one each time new data is streaming in. Instead, we’d want to update the chart “in place”).

This will require quite a bit more work in Plot’s architecture. You want to take all the marks that depend on the x scale (including the implicit axisX mark), and update them when new data streams in. (It’s only really visible in the 1m case, of course, since for the other charts the x scale is as good as fixed.)

For now, it might be easier to do this with the usual D3 update patterns than to do it with Plot. (A mix of both is always possible, though, you can leverage what Plot gives you: scales, etc., and only use D3 for the updates.)

Thank you for reiterating and for your suggestion. I look forward to seeing such updates one day in Plot then, even if it may mean waiting for a couple of years or so. Otherwise, I suppose it may be better to convert this project from Plot to D3 one day, at least partially, and maybe start my next project with D3 instead of Plot if it is going to involve steaming data or possible workarounds. Too bad; I find Plot a joy to work with and loathe working with D3 every time I have to, even though I appreciate its capabilities :sweat_smile:. I’ll try to stick with Plot as long as I can for now, and then see. Thanks for all the help again!