Serverless Cells

I have been a quiet because I have been completely rethinking the way todo authentication.

I was not happy Firebase Auth insists user emails are stored in US. There are also lots of annoyances like having to preconfigure the redirect URI. Most Oauth infra does not really work so well with Observable because of the changing origins. So switching to Auth0 would not have helped. Another general problem encountered is users identities don’t tend to be portable across hosts. I did try to tackle these issues when I wrote federated login, which kinda worked (except on Safari) but uses 3rd cookies which will break everywhere soon.

Anyway, I was overjoyed to discover the concept of IndieAuth, which uses decentralised identities (identity URLs) and public oauth clients only, so essentially all the annoyances disappear and it makes a lot more sense for Observable use. It has lots of privacy benefits in general. Users are URLs (portable). Client_ids are also URLs and they always have to match the redirect_uri but do not need to be registered in advance. Its fixes everything!

Sadly there are not many implementations of IndieAuth, and those that do exist are Ruby + PhP + SQL + Linux + Redis so kinda of difficult to host.

SO I REBUILT IT ALL IN OBSERVABLE! This was also a test for me to see if we could make a serious appliance in Observable on Serverless Cells / Endpoint Services / Observable. Funny enough, despite serverless-cells cold start latencies, I think we are about performance to the equivalent server on indieauth.com

Anyway, read more on the core notebook:

4 Likes

The cold start latencies were too damn high!

Without a code change latencies have been reduced drastically. us-central1 is our fastest region now and you can expect a cold latency of 1.5s! (for a lightweight function)

The technical details are written up here :-

In other news I released a file storage service (Store Files with Storage / Endpoint Services / Observable) which I hope to be a long term thing to accompany serverless-cells. It uses the IndieAuth server to authenticate, so your activities are somewhat masked against tracking for those that worry about these things, and you don’t need to formally signup to anything to get started!

The storage feature was completely implemented in Observable on top of serverless-cells. I am hoping storage will unlock a lot of interesting use cases. I can’t wait!

2 Likes

Hey Tom, glad to see continued updates! I finally got around to fleshing-out the version of this I had shared earlier, using the model where people would host their individual versions of the backend server, and restricting the notebooks executed to those controlled by said user. Despite the differences, there is much in common between our two approaches and we can likely share some code. I implemented by backend server directly in a notebook here in ObservableHQ, alongside a minimum nodejs app. I link-shared the details here: Heroku - keystroke-observablehq / Bryant Richardson / Observable

Below shows what I have running “outside” the ObservableHQ platform. It’s the minimum code to load a notebook as an ES6 module and then execute. I decided on a pattern where I look for a cell called “main” and call it as a function. This had some interesting implications in how I had to structure my notebook code since I had to orchestrate it all from a single entry-point cell. I doubt this approach is the most elegant or powerful one, but I didn’t have the brain power or appetite to fully explore the runtime APIs that are available for executing a notebook.

import Observable from '@observablehq/runtime';
import notebook from '@keystroke/observablehq';
import express from 'express';
import cors from 'cors';
import puppeteer from 'puppeteer';
import fetch from 'node-fetch';

runNotebook(notebook, { process, express, cors, puppeteer, fetch })
    .catch(console.error.bind(console));

async function runNotebook(notebook, context, cell = 'main') {
    return await (await new Observable.Runtime().module(notebook).value(cell))(context);
}

The linked notebook above has all the code so I won’t repeat myself too much here, but I’m definitely interested to see how you’re going to wire-up streaming and such. Right now one of my “concerns” is around serializing requests / responses from the nodejs process into the headless browser process and back. It feels a bit suboptimal.

2 Likes

@tomlarkworthy I noticed here you navigate the page to the notebook url:

You use this with an embed link for the notebook:

I think if we load the notebook using the https://api.observablehq.com/${notebook}.js?v=3 url (to get the script content) and add to page as a script tag (see my loadNotebook cell), it might be faster as browser won’t need to render or download any html and the code can just execute. I did this my notebook linked above but this is my first time using puppeteer so there might be an event more optimal approach.

One thing I wasn’t certain of was the use of page.waitforfunction and then frame.evaluate or the other options; I put everything in a single waitforfunction, but not sure on perf implications here.

I noticed your recent post was on improving startup speeds and if above is right then maybe some or all of this is another way to save a few ms.

After playing with Service Workers (Offline embedded notebooks / Tom Larkworthy / Observable) and Bluetooth bridge proxies (Dash Robot SDK for Observable / Tom Larkworthy / Observable) I come to the conclusion:

Idiomatic web inter-process communication is ‘postMessage’. Google Comlink makes it a bit nicer. You can use Fetch API to manually create Request and Response objects (Service workers do it). So maybe I made a mistake copying the Express API, as the browser already has a decent reusable Request Response implementations.

You can hook postMessage in puppeteer google chrome - How to listen to postMessage messages in Puppeteer? - Stack Overflow

I very much want to tackle streaming next as a few users are hitting the memory limits when trying to transform modest size datasets (100MB) on the fly. This seems to be a major usecase for serverless-cells ATM.

I will stick to my express handler model in userspace, due to inertia but I think a better API would be identical to how service worker’s hook fetch events. I think the cleanest interface between puppeteer and the runtime is a postMessage one, perhaps based on comlink. Note the service worker fetch API has recipes ready to rock for streaming:- Stream Your Way to Immediate Responses  |  Web  |  Google Developers so ideally that interface would just be exchanging Request and stream Response objects modelled after the (service worker) FetchAPI over postMessage.

Yeah I use the embed API just coz that is a maintained interface for embedding blessed by observable for 3rd party use and definitely CDN backed. If I run into issues I can just see what it looks like in the vanilla embed page and then see if its a general issue or unique to my environment.

I have noticed most performance is lost my transitive dependancy resolution and networks trips to the US. I think the next level of performance gains (after streaming) would be a local proxy to cache resources. Chrome has quite mature configuration for doing this so it can be done at an infra level (Proxy support in Chrome).

Also I looked at the “download code” feature which zips up notebook AND ITS DEPENDANCIES AND FILE ATTACHMENTS into a single tar.gz. Notebook Backup Tool / Tom Larkworthy / Observable . This is quite a neat distribution mechanism and very cachable so I think a http cache over the tars would be pretty performance +++

@tomlarkworthy I am using the tgz endpoint for my nodejs application like this:

{
  "name": "observablehq",
  "version": "1.0.0",
  "description": "Observable Notebook Server",
  "main": "index.mjs",
  "scripts": {
    "start": "node index.mjs",
    "refresh": "npm update && npm start"
  },
  "author": "Bryant Richardson",
  "license": "ISC",
  "dependencies": {
    "@keystroke/observablehq": "https://api.observablehq.com/@keystroke/observablehq.tgz?v=3",
    "@observablehq/runtime": "^4.8.2",
    "cors": "^2.8.5",
    "express": "^4.17.1",
    "node-fetch": "^2.6.1",
    "puppeteer": "^8.0.0"
  }
}

Above is not version-pinned, so every time you run npm update it will fetch the latest version of the notebook and if it has changed, will modify the package-lock.json integrity hash.

If you look back at the embed notebook options, one of them is “Runtime with JavaScript” and it uses the API endpoint for the notebook content that I’m using to add the script tag to the empty page. I am then using the etag header on this endpoint to see when the notebook content has changed, which means I can reuse the page on subsequent requests which should be faster than relying on browser cache and a new page load. I think you can rely on this to the same degree you can rely on the embed endpoint for getting notebook content. Something to consider.

Thanks for the links! I will take a look at the models and see if I can make use of those flows. I was thinking to refactor how I’m running the nodejs notebook, as currently its called via a single function. I was thinking I can hook into the runtime a bit more deeply and “replace” cell values at runtime, sort of like how “import with” works. This would allow me to declare the dependencies like express or puppeteer as their own named-cells, and from nodejs I can set their values with the actual import libraries. This way I wont have to pipe a “context” object all the way through my code acting as a “bag of globals” and instead I can reference the cells directly. This also gives the opportunity to mock the cell values in the notebook as the default values, so it can potentially execute directly in the web editor when working on the code (instead of just defining functions that go uncalled). The main problem is that I don’t want to induce errors when the actual nodejs content isn’t available and references can’t be undeclared magic variables like how the “process” object would be referenced to read env vars. However, this modification isn’t strictly necessary now that the code is already written and I’m not sure the code is complex enough to benefit greatly from such a change. But I am curious to further develop the idea of implementing a “traditional” nodejs application on this platform and how wrapping the notebook to execute and using the topological run order could work.

I think I’ll probably look at the post message interfaces you linked first. Seems a more direct boon to what I’m writing. I still need to refine the design model for how my functions will work and what they’ll have access to in terms of the direct request / response objects. For example if I ever want to support setting cookies I will need a way to let declared functions set them, but I would like to keep the ability to have simple functions that just take an input and return an output without needing to interact with the “lower-level” http layer. Although now that every site is asking me to accept their cookies I’m feeling like I want to do my part and just not use them at all lol :slight_smile:

1 Like

@tomlarkworthy
Hi Tom,
is the deploy service still active? I can’t access a deployed csv file anymore

https://endpointservice.web.app/notebooks/@ericmauviere/stockage-cdn-fichier-de-reference-bdm-idbanks/deploys/insee_bdm_idbanks_150521.csv
=> Deployment ‘insee_bdm_idbanks_150521.csv’ not found, did you remember to publish your notebook, or is your deploy function slow?

It was working fine last week, and i just found some notebooks broken

I use the latency monitor as a status check:-

All looks good.

The problem appears to be that something has become a little slower. The deploy cell is blocked until the zip is converted. The runtime looks for the deploy commands so it this causes a timeout.

You want deploy calls to resolve quicker, so here it a suggestion to send data to “res” outside of the handler. Fingers cross that will get it working again.

FYI, I am on the final touches of getting streaming working, which will make these kinds of data set facades much more efficient if you can support streaming all the way through.

In the end their were problems further up, and errors were not caught in .then() and swallowed silently somewhere in the dataflow graph. If you are doing serverless cells you have to be triple careful about passing your errors forward otherwise you get a very unhelpful “its broken” with no insight in the myriad of places it could break. I strongly suggest getting Logs Manager / Endpoint Services / Observable working

So that website hosting the zip had a fair amount of defenses against hotlinking. Sometimes I think, is this hacking? But I read their terms of service and they definitely want the data used but they seem to have a very picky server configuration so its very hard to read the zip. A lack of origin header broke it. On my fault it used a chunked transfer encoding which we don’t support (yet! thats streaming! it’s nearly there).

The latest version of fetchp allows overriding response headers and so we can kinda fix it and get data. Though I realize there is a fair amount of esoteric knowledge there.

results = fetchp(insee_csv_zipped, {
  headers: {
    'x-requested-with': "observablehq.com" // Upstream blocks us without this, silly
  },
  responseHeaders: {
    'transfer-encoding': null, // Its trying to send them in a chunked reponse which is not working ATM
    'Cache-control': `public, max-age=${24 * 7 * 3600}` // We can cache at the proxy level now too!
  }
})

If you get fetchp working, when a serverless cell calls fetchp it skips actually proxying it. Only when its in a browser does it proxy it to serverless cell. So for difficult data problems its quite nice to get it working with fetchp as that transfers better to serverless-cells do.

Serverless-cells do not send an origin header, which is a good thing coz its not CORS, but also a bad thing because servers may act slightly different (like in this case).

Oh well, I don’t think there is an easy solution that will work in all cases.

I got your zip endpoint working though eventually Stockage CDN/proxy du fichier de référence Insee BDM idbanks / Tom Larkworthy / Observable

1 Like

Thank you very much Tom for your debugging help!

1 Like

I just redid the backend routing in preparation for streaming support. It now has a custom domain “webcode.run” backed with a more sophisticated load balancer.

Performance seems to have increased across the board, particularly non-US regions. Our old routing was pointlessly moving traffic to the US but I think the traffic kinks are out now.

Furthermore, by default, traffic is routed to the nearest region to you, so US people should see a substantial performance performance increase as I think no-one actually selected the appropriate region, and now you don’t even need to!

Even for those that had the optimal region selected, the load balancer is just plain faster. WARM latencies in US have gone from 750ms → 600ms. Warm latencies in east asia (the worst) have gone from 2secs to 1sec. Asia tends to be worst latency in general as traffic to Observablehq origins is worst case, but the gap has been dramatically improved.

Full stats here: Realtime Serverless Cell Latency Monitor / Tom Larkworthy / Observable

2 Likes

Serverless cells now support streamed responses!

use res.write() to send a chunk, and res.end() to finish a response.

CORS Proxy fetchp / Tom Larkworthy / Observable has been upgraded to exploit the new semantic.

Now you can proxy > 32MB payloads, this is useful for transforming datasets live off websites.

Time to first byte on large requests should be much better. To really exploit the benefits you need to read them in chunks too (e.g. response.body.getReader()).

1 Like

August is dedicated focus to serverless cell upgrades

I just release a much simpler URL scheme, which does not have the weird modifiers or secrets in the URL.

This makes the devtools network trace much easier to read, but also makes forking secured services much easier as the secret names are not mixed in.

If anyone has ideas for improvement now would be a great time to send me some suggestions.

Barring external ideas I will probably look to try and improve the UX of uploading secrets. I think people struggle with hiding API keys and the technology is here but I don’t think it’s easy enough to use yet.

1 Like

Hi Tom, you called for ideas on room to improve. I see a URL written into the image file above. If I navigate to ‘https://webcode.run/’ I get ‘Request not handled’.

Did you mean to link us to a notebook or web resources, but forget to add a URL reference?

yeah that’s maybe a bit devoid of context, though the folks that complained about the ugly URL will know what I mean. I have “creating a landing page for webcode.run” on the Kanban also which would make navigating to webcode.run directly a bit more interesting.

I am upgrading all the docs Observablehq side too but that’s a WIP so its not worth looking at right now.

1 Like

Yesterday I caused a 5 hour outage by mistake. So I have added alerting so I get informed quicker if this happens again. You can see this outage in the latency notebook, you can also see I caused a performance regression a few days ago which I also fixed. ( It was a classic case of JS truthy weirdness causing unintended cache misses for negative results, GETS ME EVERY TIME! )

There are lots of very exciting workflow and performance upgrades in the works, hence the instabilities. I will have a mini launch soon. Make sure you have notifications for this thread turned on if you want to participate.

3 Likes

Version 2 of serverless cells is HERE!!! Now called webcode. It’s a backwards compatible drop in replacement!

I have taken the learnings of the first version and tried to fix the development experience. Namely, it was too hard to fix bugs in the first version, now it’s magic because of a couple of big features

This version has INSTANT deploys, so you can rapidly iterate at the speed of Observbale Dataflow.

The second thing is you can attach true DevTools and even use DevTools REPL to intercept production traffic.

These features are possible because the webcode load balancer will route traffic TO YOUR BROWSER, so you actually end up being the server when the live coding tunnel is active.

Authentication has been greatly simplified by using login-with-comment. So you do not need to register with a 3rd party service, it can figure out who you are based on you posting a code to Observablehq. No registration means there is no personal information stored anywhere.

Because we have authentication, you can upload environment secrets and it will inject them into the runtime. There is a UI to help make that simpler now. Nobody else can access your secrets. Only the code deployed in your notebooks in webcode handlers can access your secrets, and nobody else can mess with that code because only you have write access to your notebooks! Its secure.

This is truly an improvement over existing function-as-a-service offerings because of the ability to code live and attach debuggers. It exploits the amazing affordances of Observable to do so. It is preserving the best aspects of Observable and making it work with backend development.

The big drawback is latency is a big high ATM (500ms), but I have a plan to get that to 100ms but I want to get this out first. Please give it a try. All the serverless cell examples are still compatible, we have just upgraded the development workflow, it’s not an API change.

5 Likes

I am refactored the old URL handler (endpointservices.app/notebooks/…) to reuse the new URL format (webcode.run/observablehq/…) via an internal redirect. It should be backwards compatible and I have tested it, but maybe there is some case I have overlooked so let me know if you have problems (especially if you saved an old URL offline).

I am cleaning up so I can work on another performance optimization I hope brings webcode to performance parity of Lambda (but of course it’s better because its all web based tooling with instant deploys and prod debugging and no additional signup required).

3 Likes

Wow, I think we achieved a 10X improvement on latency. Cold starts are now faster than the previous warm starts! A warm start now can be a low as 30ms!!!

Wow, we can actually build decent services with these latency numbers! I am slowly upgrading old services to take advantage of it.

EDIT: fetchp the CORS proxy has been upgraded and now is fast!

3 Likes

I have the observable.run domain, but I’m not doing anything with it, maybe you’d want to use that?

1 Like