fetch gzipped csv

Hi,

Does anyone know how I can fetch a gzipped csv?

It does not look like d3 supports this directly

data = d3.csv(url_to_cs_gz)

I don’t think that would work without first unzipping the file. You could fetch the file and unzip it first, and then use d3.csvParse on the unzipped content? This may help:

2 Likes

I used this to load a zip file in observable: New Zealand GeoJSON regional Map / Brett Cooper / Observable

The wasm version might be faster. Big datasets can be slow to work with.

1 Like

This is an example of unzipping from URL and reserving via CDN using serverless cells which is quite nice if you expect the underlying CSV to change over time

since that was made we support streaming so the config

     'transfer-encoding': null, // Its trying to send them in a chunked reponse which is not working ATM" 

is probably not required (and setting the X-requested-by header was a work around for that particular server which is also probably not required).

1 Like

Thanks @tomlarkworthy that notebook worked (see fork). However, it seems to only work when I have published the notebook - otherwise it says

results = Error: Cannot determine notebook name

on several cells. Thanks all for the help :smiley:

1 Like

yeah the runtime has to be able to read a public URL, so you need to publish or link share.

Now we have user <-> user notebook sharing even when private perhaps its possible to remove this annoying aspect by sharing it with a bot.

Ok, for my purposes I was able to use fetch since I’m getting data from GitHub which allows CORS requests. Switching to fetch allows me to use a private notebook apparently.

1 Like