# read data from Google Cloud Storage public bucket **URL:** \ **Category:** Help\ **Created:** [April 15, 2021, 9:19pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923 "2021-04-15T21:19:38Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![cornhundred](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/cornhundred/32/1705_2.png) [@cornhundred](https://talk.observablehq.com/u/cornhundred)\ **Post date:** [April 15, 2021, 9:19pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/1 "2021-04-15T21:19:38Z")
Hi, does anyone know if/how we can load data from a public Google cloud storage bucket into ObservableHQ? ---
**Author:** ![tomlarkworthy](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/tomlarkworthy/32/5940_2.png) [@tomlarkworthy](https://talk.observablehq.com/u/tomlarkworthy)\ **Post date:** [April 16, 2021, 8:19am UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/2 "2021-04-16T08:19:04Z")
yes. There are many ways depending on what kind of access control you want on it. Obviously public read access is very easy. If it has sensitive info you might want to put it behind a login. What are your requirements? It’s also relevant if you want write access from observable, or purely read. If you did not know, Firebase storage is a dedicated clientside wrapper for GCP buckets. So thats the most full featured approach. I demoed firebase storage in a twitch yesterday: > **[Chat application with Firebase on Observable (Twitch)](https://observablehq.com/@tomlarkworthy/chat-application)** > > Build a chat app Security I also have an example of using GCP storage as a BQ cache here: > **[How to cache BigQuery results in a public Notebook with Firebase Storage](https://observablehq.com/@endpointservices/cache-bigquery)** > > BigQuery is powerful, but expensive if misused, so you never want to grant BigQuery API access to the public. For dataviz, a cache of BigQuery data is preferred. But in the ObservableHQ world, we want to be transparent about how we prepared our... But it’s not necessary to go through Firebase if you don’t need all those features. ---
**Author:** ![cornhundred](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/cornhundred/32/1705_2.png) [@cornhundred](https://talk.observablehq.com/u/cornhundred)\ **Post date:** [April 16, 2021, 11:39am UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/3 "2021-04-16T11:39:54Z")
Thanks @tomlarkworthy! For now, we only want to read data from a public GCP bucket and we want that to be done on a public Observablehq notebook (so there’s no current login requirement). Preferably, we could talk directly to Google without a firebase intermediate just to keep things simple. I’ll look over the examples (I knew I should have watched that twitch live 😁). Do you know if we can use Google’s mode.js client libraries on Observablehq [https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-nodejs](https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-nodejs) ? ---
**Author:** ![tomlarkworthy](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/tomlarkworthy/32/5940_2.png) [@tomlarkworthy](https://talk.observablehq.com/u/tomlarkworthy)\ **Post date:** [April 16, 2021, 1:06pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/4 "2021-04-16T13:06:11Z")
Node.js clients do not work (gRPC) but you can use the machine generated GAPI clients… thats how I access BigQuery from the notebook [Google API Client / Tom Larkworthy / Observable](https://observablehq.com/@tomlarkworthy/gapi) its only a few steps better than REST though and pretty painful all the way. However, for public info in a public bucket. You can just fetch(URL) them right? No need for clients, the buckets speak REST. I think you just need to enable a CORS policy for the bucket and the data is public over vanilla HTTP. > **[Configuring cross-origin resource sharing (CORS)  |  Cloud Storage](https://cloud.google.com/storage/docs/configuring-cors)** ---
**Author:** ![cornhundred](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/cornhundred/32/1705_2.png) [@cornhundred](https://talk.observablehq.com/u/cornhundred)\ **Post date:** [April 16, 2021, 2:47pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/5 "2021-04-16T14:47:08Z")
Thanks @tomlarkworthy, I think the issue is that our bucket is not enabling CORS. I’ll give this a try and confirm it works. This must be how @enjalot is getting data from Google cloud storage here [Pigs / Ian Johnson / Observable](https://observablehq.com/@enjalot/pigs) ---
**Author:** ![enjalot](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/enjalot/32/5248_2.png) [@enjalot](https://talk.observablehq.com/u/enjalot)\ **Post date:** [May 7, 2021, 6:29pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/6 "2021-05-07T18:29:31Z")
@cornhundred Yes I manually enabled CORS via [Configuring cross-origin resource sharing (CORS)  |  Cloud Storage](https://cloud.google.com/storage/docs/configuring-cors) on my bucket to use it in the notebook ---
**Author:** ![cornhundred](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/cornhundred/32/1705_2.png) [@cornhundred](https://talk.observablehq.com/u/cornhundred)\ **Post date:** [October 17, 2024, 5:11pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/7 "2024-10-17T17:11:10Z")
Hi, I have to read from a private bucket that has CORS enabled and can be accessed by providing a token. Are there any example ObservableHQ notebooks that show how to login into a Google account to obtain a token? I’m doing something similar in Python here ```auto from google.colab import auth from google.auth import default import google.auth.transport.requests auth.authenticate_user() creds, _ = default() request = google.auth.transport.requests.Request() creds.refresh(request) token = creds.token ``` and I’m able to use this token in an ObservableHQ notebook to access the data on my private bucket. I’m wondering if I can get this token from within a notebook? ---
**Author:** ![tomlarkworthy](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/tomlarkworthy/32/5940_2.png) [@tomlarkworthy](https://talk.observablehq.com/u/tomlarkworthy)\ **Post date:** [October 17, 2024, 6:09pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/8 "2024-10-17T18:09:53Z")
So if you want third party access its a huge pain and you need somewhere to put secrets. I do have a Google Oauth login example here:- > **[Oauth 2.0 Client Examples](https://observablehq.com/@tomlarkworthy/oauth-examples#google)** > > These are preconfigured clients that can authorize this notebook. You would need to create your own secrets and client\_id and redirect\_uri's if you were to host your own, but here you can experience it working. We only support the Oauth 2.0 \_... However, if you just want to give yourself write access and everybody else public read access (or just yourself read access), the simplest is to use an S3 compatible API key, which you can put in an `Input.password`, bind to local storage [@tomlarkworth y/localStorageView](https://observablehq.com/@tomlarkworthy/local-storage-view) so you only have to do it once per device. These S3 compatible keys don’t expire. > **[How can I create Access/Secret keys for Google Cloud Storage?](https://developer.bitmovin.com/encoding/docs/how-can-i-create-accesssecret-keys-for-google-cloud-storage)** > > Those keys are called "interoperable keys" or "migration keys". As Google Cloud Storage does provide an S3 interface you could use an S3 Client, like "S3 Browser", or CLI tool like "s3cmd" in order to work with Google Cloud Storage buckets as well.... With GCS in S3 compatability mode you can just use any of the zillions of S3-lite clients or even AWSs browser S3 client. I personally have converged to S3 API being the one-true-storage API ---
**Author:** ![cornhundred](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/cornhundred/32/1705_2.png) [@cornhundred](https://talk.observablehq.com/u/cornhundred)\ **Post date:** [October 17, 2024, 6:25pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/9 "2024-10-17T18:25:27Z")
Thanks, I’ll look through the notebook. My use case would be that I would create a notebook pointing to a dataset on a private Google bucket that I and someone else has access to via our Google accounts. The notebook user would then click the Google login link to receive a temporary token for read only access to their data via CORS. Would I need to use secrets in my case? ---
**Author:** ![tomlarkworthy](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/tomlarkworthy/32/5940_2.png) [@tomlarkworthy](https://talk.observablehq.com/u/tomlarkworthy)\ **Post date:** [October 17, 2024, 6:28pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/10 "2024-10-17T18:28:44Z")
yes, the CLIENT\_SECRET in Oauth. There is a device login flow for Oauth that avoids it, which is hinted at here in Google docs [OAuth 2.0 dành cho TV và ứng dụng thiết bị đầu vào hạn chế  |  Authorization  |  Google for Developers](https://developers.google.com/identity/protocols/oauth2/limited-input-device) so maybe this is a better way but I have not tried it. Let me know if that is good enough for GCS access. ---
**Author:** ![cornhundred](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/cornhundred/32/1705_2.png) [@cornhundred](https://talk.observablehq.com/u/cornhundred)\ **Post date:** [October 17, 2024, 7:23pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/11 "2024-10-17T19:23:11Z")
Ok, let me give the Oauth notebook a try and let you know how it goes. ---
**Author:** ![tomlarkworthy](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/tomlarkworthy/32/5940_2.png) [@tomlarkworthy](https://talk.observablehq.com/u/tomlarkworthy)\ **Post date:** [October 17, 2024, 9:31pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/12 "2024-10-17T21:31:29Z")
I think with device Oauth flow you won’t need that oauth notebook. Also consider just sharing an API key with the other person if you know them, you can also create a separate key and revoke it later if needed ---
**Author:** ![cornhundred](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/cornhundred/32/1705_2.png) [@cornhundred](https://talk.observablehq.com/u/cornhundred)\ **Post date:** [October 18, 2024, 5:12pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/13 "2024-10-18T17:12:55Z")
Thanks, I’m a bit lost with all the options 🙂 but I’ll read over again and see if I can work it out. I guess another option might be to set up an enterprise account with login using Framework [Observable Framework for Private Data (login required) - #2 by tophtucker](https://talk.observablehq.com/t/observable-framework-for-private-data-login-required/9162/2) ---
**Author:** ![tomlarkworthy](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/tomlarkworthy/32/5940_2.png) [@tomlarkworthy](https://talk.observablehq.com/u/tomlarkworthy)\ **Post date:** [October 18, 2024, 6:17pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/14 "2024-10-18T18:17:29Z")
Yes I agree its too complicated, for personal stuff I shove API keys in localstorage. I have a Google doc with all my keys, and when I use a new device I copy and paste the key over. If I want to collaborate in a group I trust, I put a shared key in URL params and share the link. I can imagine a more advanced version of this where everybody gets their own key to put in their own personal password manager. I kinda think oauth is overkill for small groups and its only justified for actual products and not really the right level of ROI for ad-hoc collaborations. ---
**Author:** ![cornhundred](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/cornhundred/32/1705_2.png) [@cornhundred](https://talk.observablehq.com/u/cornhundred)\ **Post date:** [October 19, 2024, 11:32am UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/15 "2024-10-19T11:32:25Z")
Thanks for the advice, unfortunately the Google bucket I’m using being created within a Terra.bio workspace and I don’t currently have permission to create a key (assuming I’m understanding correctly). I am able to get a short lived token from a Terra notebook (where the notebook knows the user is authenticated) or by using the above Google Colab approach by logging into my Terra associated account and copying the token over to ObservableHQ. So for now we might just recommend users create the token on Terra and manually copy it over to ObservableHQ - our use case is that Observable will create a visualization of a dataset that they are analyzing on a Jupyter notebook. Or we can use Google Colab to quickly create a token if a user wants to visualize their data on Observable without using Terra. ---
**Author:** ![tomlarkworthy](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/tomlarkworthy/32/5940_2.png) [@tomlarkworthy](https://talk.observablehq.com/u/tomlarkworthy)\ **Post date:** [October 19, 2024, 7:46pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/16 "2024-10-19T19:46:30Z")
who owns the storage bucket? We can do a call if you want. ---
**Author:** ![cornhundred](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/cornhundred/32/1705_2.png) [@cornhundred](https://talk.observablehq.com/u/cornhundred)\ **Post date:** [October 19, 2024, 11:30pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/17 "2024-10-19T23:30:05Z")
The bucket is owned by Terra. Sure a call would be great if you’re available. I’ll message you my email. ---
**Author:** ![tomlarkworthy](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/tomlarkworthy/32/5940_2.png) [@tomlarkworthy](https://talk.observablehq.com/u/tomlarkworthy)\ **Post date:** [February 1, 2025, 6:21pm UTC](https://talk.observablehq.com/t/read-data-from-google-cloud-storage-public-bucket/4923/18 "2025-02-01T18:21:55Z")
for the record we did manage to authenticate with Google using the oauth notebook but then we figured out a simpler solution altogether.