Observable Runtime and Parsing JS

Community Help
1

I’m trying to build my own observable UI for an experimental tool at work. Right now I have the observable runtime working, where I have a module with variables added to it, and they update the DOM etc.

The issue I’m seeing is this: it looks like variables always take either a constant value, or a function, i.e.

myModule.variable(...).define('blah', [], () => 'hello world')

But I want to, just like observablehq, allow the user to create their own cells in the browser. If I give them, for example, textareas to type their code into, what I get out of that is a String, and it doesn’t look like you can pass a string to variable.define and have it parsed into JavaScript. I want to be doing something like this:

myModule.variable(...).define('blah', ['html'], html => "html

Hello world!

")

Similarly, it’s not clear to me how to calculate the dependencies (in the example above, html) based on the string I get back from a textarea. I see that there’s a parser in the observable codebase here https://github.com/observablehq/parser which returns useful things like whether a chunk of code is a generator or is async etc., but it doesn’t look like it does anything about calculating deps, and anyway it’s not returning me JS that I can use.

Sorry if this is obvious, but I’m just wondering what black magic the observable team are doing to turn strings in the cells/variables in the browser into JavaScript. Are you taking the contents of a cell and then dynamically adding it via script tag, wrapped in module export or something?

For some background, basically I’m trying to build a similar UI to observablehq for a tool at work, and because it would have special access to our API’s etc. it would need to be self-hosted. In other words, I know you provide some “Teams” stuff, but we wouldn’t be allowed to use that here. I also realize that this might be part of your secret sauce for observablehq, so I understand if you don’t want to divulge your secrets…

Thanks again for the awesome tool and I appreciate any help in advance!

2

After some more digging, it seems like this might be exactly what I want: https://observablehq.com/@asg017/an-unofficial-observablehq-compiler

I’m going to start playing around with this, but if people have suggestions I’m all ears!

1 Like
3

Hi @davertron,

You’re on the right track — as the Observable runtime, standard library and parser are currently open source, but the compiler is not.

But, of course, we would much rather be able to provide a solution where you can simply use Observable notebooks at work, instead of forcing you to try to reimplement parts of Observable yourself. Is this simply a matter of hosted code? If we offered an enterprise version of Observable that your company could run on its own servers, would you prefer to use that instead? Or are you building something truly custom?

1 Like
4

I think an enterprise self-hosted solution would be exactly what we’d need. I basically want observablehq with a bunch of custom libs “baked in” as if they’re part of the standard library for fetching proprietary data etc. (although we could probably use existing import as needed) and custom visualizations of that data. I’m playing with this stuff right now to see if a) there’s any interest in this as a tool here and b) because it’s interesting and I just want to understand how it all works a little better, so I don’t mind building my own crappy compiler to learn :smile:

5

Sounds good.

Just as a heads up, in case you haven’t already seen it — Observable Teams currently support working with proprietary data safely via Database Connections and Secrets.

You can self-host the database proxy for connecting to data on internal intranets, and provide a notebook that offers a uniform interface to access the data, over a REST API, or directly with SQL. In this way, the actual data never touches Observable servers, and stays in your browser. (The visualization code, however, does.)

We use this approach ourselves for writing team-wide notebooks that visualize private data from our internal database.

Good luck!

1 Like
6

I will definitely check out the Teams stuff. I think the main barrier with going that route is educating people to understand that just because they see our data on a page not hosted by us, it doesn’t mean that organization has access to it. I’m betting it would still be a tough sell, but we’ll see if I can make a compelling case first before I go down that road.