Hi; I see related questions on the topic of sharing database connections in public notebooks, and I understand why that can be complex/risky.
I don’t especially need a public version of a notebook to actually run queries. But I’d like to work on a notebook iteratively using bigquery, modifying my queries as I go, before publishing the results for broader consumption. That is: I’d like the public version to carry the results of the last run.
I can see a simple way to work around this (e.g., save the data from the query, then upload the same data back into the same notebook?), but that requires manual intervention and perhaps a couple of errors on the public page if I want to keep the queries in place for readers.
I’ve been using observable for ~24 hours, and I love it so far. Perhaps I’ve missed something obvious! Is there a no-brainer way of sharing the results of database queries in public notebooks without the errors?
Hi, and welcome to Observable! Amazing that you got to this point in only 24 hours
Your current workflow is exactly right for now… And you are right, it is a bit too manual. One of the ideas we are discussing is how to make it easy to cache these results for notebook consumption (private or public), possibly as file attachments. If you want to suggest a workflow, we would love for you to log a feature request in our feedback repository here and then you can monitor the progress and suggest ideas towards the best solution.
That’s a bit of a journey – I had to poke some things in Firebase – but it definitely gets things a lot of the way there. Thanks!
There’s a different downside with this approach, that the jobs.query endpoint returns separate schema/values arrays in the response, not the array of key:value dictionaries that the DatabaseClientquery call returns. As a one-off I can mangle my data fairly easily, but is there an alternative API endpoint to retrieve results and/or has somebody already written code to parse these cleanly?
That said: if I skip the JS API pieces, and instead keep the DatabaseClient query call then feed the results of query() into firebase/retrieve from firebase into a different variable, I think I wind up somewhere similar. I can feed the data retrieved from firebase into vega-lite plots as before; I imagine there might be consistency issues if I’m not careful, I haven’t properly road-tested yet.
The notebook still throws errors about the DatabaseClient when it’s published (of course) so it’s not totally clean, but it can still read the most recent cached result and build out plots. That’s closer to what I’d like, without as much extra work. (I may also look into whether it’s possible to achieve the same using Google Cloud Storage directly, without the extra Firebase stuff!)
Oh yeah! you can probably achieve the same role based access control just with IAM on cloud buckets without Firebase Storage. In the notebook we are minting access_tokens so you can do any cloud Bearer token thing. It’s very easy to access cloud storage with a token. That would totally work.
So, today I finally got back to looking at this. And I figured I could have the service account for the database side, and a service account for the cloud storage side. I quickly got lost in the maze of google documentation, oauth2 authentication, service keys, API keys, access tokens, and opaque 401s.
Do you have, or know of, a minimum working example to upload a file to cloud storage from JS? I appreciate the firebase stuff, but in terms of simplifying the surface for me (and our support staff ) it’d be neat to keep things as clean as possible!