Hi there! Can you suggest the optimal way to cache downloaded files using d3.csv() and DuckDB on the client-side browser? I use the Observable framework and Google Spreadsheet CSV files to create a marketing and analytics trends dashboard. Here is a code example:
Every time I reload the page or open it in a new tab, it takes too much time to download all the CSV data, which can distract users and make the page heavily loaded
It definitely works, thanks!
But it forces me to make the whole Google Document accessible via the internet, and I have some data in there I donât want to share with others .
The second thing is that users still have to wait for the data to load after reloading the page or opening it in new tabs.
I asked ChatGPT, and it suggested using the browserâs storage API (like localStorage) to store data locally, but maybe there is another, more convenient way to achieve the same result using Observable functionality or another API?
Yes you can also use local storage to keep the assets in the browser, maybe revalidate in the background for 0 latency on page load. You can also downlaod the CSV and serve via framework which is better optimized. You can switch from CSV to compressed parquet (GitHub - hyparam/hyparquet: parquet file parser for javascript).
Also are you sure its the file that is the slow part? Coz often I find the 7MB duckdb dependancy is the actually cause of slow starts. Did you measure in the network tab of the browser.
Okay, probably I am doing something wrong, but I set up slow 4G without cache to test weak network conditions in DevTools (Chrome, Linux OS).
Spreadsheet Request: Waiting and downloading from the spreadsheet takes 700 ms and 400 ms accordingly, which is not bad at all.