Data Loader dependent of other Data Loaders

Community Help
1

Hi,

I’m using Observable Framework to build some dashboards that consumes some json api from an external site.

I have a data loader ./src/products/[category]/data.json.ts that essentially pulls the data from the api. A couple of MB.

I don’t need all that data in the client, so I would rather have a data loader ./src/products/[category]/stats.json.ts that would consume the ./data.json of that category.

This would allow other data loaders to exist that could also depend on the ./data.json. So effectively building a processing pipeline using data loaders, leveraging the cache to download MBs of api data only once on build time, and sending just the needed bits to the client by having specific data loaders.

Given the answer at Observable Framework Data Loader with local csv file - #2 by mbostock I assume this is not possible, right?

Without adding an ad-hoc cache of the api, how would you structure a project to avoid requesting multiple times the api?

2

I have not tried it but maybe the child can call the parent dataloader’s endpoint with a http call, so it ends up invoking the parent data loader? i.e. do the chaining in the network layer

3

see Chaining data loaders · Issue #332 · observablehq/framework · GitHub

1 Like
4

@tomlarkworthy idea would only work today in preview mode and does not account for caching/dependency tracking.

@Fil chaining data loaders by Fil · Pull Request #1522 · observablehq/framework · GitHub is exactly what I was hopping for :rocket:. I think is very valuable to be able to shrink/pre-process data for clients when possible. Is there something that could be done to push for that feature?

My alternative is to wrap the http calls and do some local cache to avoid downloads. It’s not as powerful but it will suffice for me.

5

What I’ve ended up doing to avoid repeated API calls is to have a single data loader (in Python) that outputs various files along the way. Then set up GitHub Actions to automatically commit and push the files. (It worked fine on my own computer without that step, but the files weren’t saving when it ran on GitHub.) I have no idea if it’s kosher, but it works. I describe the process in step #4 here, and you can see the data loader itself here.