In Streaming Shapefiles, Mike demonstrates a method for progressively reading in information from a somewhat large (13.6 MB) shapefile. Before I go too far trying to reproduce this with a JSON file that is 1,684,583 KB (1.6 GB), I wanted to ask if the operation is possible.
What gives me pause is that QGIS is also able to ‘stream’ in shapefiles—rendering them progressively as it reads through the file data. However it can’t do this with a geojson: I must wait for it to fully load in order to render.
Mike’s notebook uses a utility shapefile@0.6 to open the shapefile, and more specifically uses this command:
shapefile.open(await FileAttachment("UScounties.shp").stream()
He also notes that for this to work, a browser must support streaming fetch.
I’ve referred to the docs on the fetch API and on consuming fetch as a stream, but it’s all very technical and really far over my head. Confusing especially for me is the (apparently simple) example they give for fetch stream method:
// Fetch the original image
fetch('./tortoise.png')
// Retrieve its body as ReadableStream
.then(response => response.body)
To me, this conceptually doesn’t look like it would work in the same way as the shapefile > open > stream sequence from Mikes example (rather, it reads like it would first need to fetch everything, then return it).
Or maybe this is all just ‘automagical’ with today’s browsers? When I try just fetching the hosted URL in Observable, my cell just hangs there for about 10 minutes before finally returning the (complete) fetched data.
It goes without saying that I realize I should just cut down the data in QGIS first so as to focus on my specific problem area. At the same time, there’s something to be said for hosting a complete record, and then using JS to clip out the bits I need and re-export those clips without having to open QGIS each time.
Thanks in advance for your help and guidance!