read data from Google Cloud Storage public bucket

Hi, does anyone know if/how we can load data from a public Google cloud storage bucket into ObservableHQ?

2 Likes

yes. There are many ways depending on what kind of access control you want on it. Obviously public read access is very easy. If it has sensitive info you might want to put it behind a login. What are your requirements?

It’s also relevant if you want write access from observable, or purely read.

If you did not know, Firebase storage is a dedicated clientside wrapper for GCP buckets. So thats the most full featured approach. I demoed firebase storage in a twitch yesterday:

I also have an example of using GCP storage as a BQ cache here:

But it’s not necessary to go through Firebase if you don’t need all those features.

1 Like

Thanks @tomlarkworthy! For now, we only want to read data from a public GCP bucket and we want that to be done on a public Observablehq notebook (so there’s no current login requirement). Preferably, we could talk directly to Google without a firebase intermediate just to keep things simple.

I’ll look over the examples (I knew I should have watched that twitch live :grin:). Do you know if we can use Google’s mode.js client libraries on Observablehq https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-nodejs ?

Node.js clients do not work (gRPC) but you can use the machine generated GAPI clients… thats how I access BigQuery from the notebook Google API Client / Tom Larkworthy / Observable

its only a few steps better than REST though and pretty painful all the way.

However, for public info in a public bucket. You can just fetch(URL) them right? No need for clients, the buckets speak REST. I think you just need to enable a CORS policy for the bucket and the data is public over vanilla HTTP.

1 Like

Thanks @tomlarkworthy, I think the issue is that our bucket is not enabling CORS. I’ll give this a try and confirm it works. This must be how @enjalot is getting data from Google cloud storage here Pigs / Ian Johnson / Observable

@cornhundred Yes I manually enabled CORS via Configuring cross-origin resource sharing (CORS)  |  Cloud Storage
on my bucket to use it in the notebook

1 Like