This is a tricky situation – one that involves muting some of Observable’s reactivity (if that’s the right word).
@tomlarkworthy was helping me to work through a similar issue for Leaflet layers. He nearly got it (and he gave me the ingredients to keep working through the problem… I just haven’t had too much time since to take it the final step).
In case any of Tom’s work helps you conceptually, here’s a draft notebook:
If you figure this out for deck.gl, could you please share?! It would be a very helpful example!
Thank you, and good luck!