Deploying notebooks as web services

Show and Tell
1

A few months I go I started experimenting with deploying my observable notebooks as web services.

The idea is to develop in the browser and then click a button to deploy that notebook at a well-known domain. I outlined a bit about my approach in a Twitter thread: https://twitter.com/ajbouh/status/1184720560359530496

Of course the service “compiler” is itself implemented as an observable notebook! :slight_smile:

It can either use jsdom, node-fetch and a headless implementation of canvas to provide an approximation of the browser runtime environment, or use a headless chrome instance to provide a real browser runtime environment.

Is this something that other people care about? Would it be worth me cleaning up for others to try out for themselves on their own notebooks?

4 Likes
2

I would be interested in seeing more details / code. Unfortunately, I’m not sufficiently familiar with the tech you mentioned in the tweets to be able to reconstruct what you did…

3

Interesting. What use case is driving this? Just a more interactive development of code that needs to be a web service? Or are you providing stuff you do anyway in a notebook, as a web service for consumption by non-browsers or non-Observable pages?

I’m finding Observable to be a great environment for writing new code, except for headaches switching to the browser’s debugger. Observable’s inspector is superior and more than makes up for it.

Coincidentally, earlier this evening a suggestion from @fil to look at packd led me to find I created a zeit.com account a year ago and forgotten about it. I’ve been thinking about offloading large dataset reduction to a server. This could all fit together…

4

I really like how fast the dev experience is for Observable. When I needed to run code that ran in a server-side environments, I realized that I wanted to avoid npm and JavaScript build pipelines for as long as possible. My initial need was rendering images to embed on other pages. Once I started experimenting with the approach I got very excited about it and ended up making an end-to-end deployment solution for notebooks.

My feeling is that most other people aren’t there yet on the adoption curve. Hopefully this changes soon! :slight_smile:

3 Likes
5

I’ve recently gone the other route, and built a workflow that efficiently pushes to NPM. In large part, driven by wanting to use Typescript and extensive test cases.

My workflow looks roughly like this:

  1. Experiment a bit on Observable
  2. Copy my experiments over to WebSphere, convert to Typescript
  3. Write unit tests, docs, etc.
  4. Push to GitHub (https://github.com/BobKerns/physics-math)
  5. GitHub Actions builds it in 4 different environments. (Could also use Travis or CircleCI).
  6. Define a release on GitHub (I make this an explicit step rather than automatic)
  7. GitHub Actions publishes the package to NPM and the documentation to a GitHub page
  8. I go to https://observablehq.com/@bobkerns/testing-physics-math
  9. I select my new release, and I’m back to playing with Observable again.
  10. If I need to make experimental changes, I can run a local server (npm run server) and load from there instead of steps 4-7.

It ends up being fairly easy to switch back and forth, depending on whether I’m exploring or engineering, but it took a lot of infrastructure effort to get here. I really benefit from from Typescripts checking and from being able to debug unit tests, but as soon as I want to explore ideas, I’m back in Observable with little overhead.

I make Step 5 (defining a release on GitHub) manual to give my page a source of info about the releases (via their REST API), and allow me to select to compare behavior, etc.

But it’s taken way too much knowledge, time, and experimentation. I hope others can benefit from my efforts; I hope to write them up in a usable recipe form. I’d love to see this sort of two-camps workflow be simpler to set up and manage.

Your direct-to-zeit (now Vercel) approach would be perfect for every server-side bit I’ve wanted. The factors that pushed me toward offline/NPM involve complexity, documentation , testing, reusability.

But my server-side stuff is just “not in the browser” bits. Actually, I wonder if I could eliminate the server-side entirely with workers?

Being set up to load my package from a local server is a similar effort to reduce time out-of-Observable. I wonder if I could set up a page to serve an JS attachment and documents and cut out GitHub/GitHub Pages/NPM?

It probably wouldn’t save me anything at this point, but it would potentially be a lot easier to explain without invoking a dozen other technologies, just to deliver code to ObservableHQ pages!