Noob: Can you run your own Observable server

I actually really resonate with what claude is saying.
The power of ipython/jupyter was:

  • use your own data
  • host is on your machine or on your internal private network
  • work offline
  • it’s open source

I expect that devs can already build up with what’s out there a very simple editor for observable.
I think there is still venue for observablehq.com to be the platform where packages are host and open source is written, teams can collaborate etc…

2 Likes

hear hear! also, I’d like to see this happen too!

1 Like

I thought this was related. Link to running observable headless by @jflatow :

Chiming in late here, I’d love to be able to use Observable with my team at Stitch Fix, but it’s a non-starter unless it’s on-prem. We’d be happy to pay for that privilege.

7 Likes

It’s a shame there isn’t an offline offering. Without it, it’s unlikely observable will gain as much adoption as Jupyter. That adoption could eventually drive the cloud-based tool adoption.

4 Likes

I believe the goal is to try and keep the community in one place (here) from a data analytics and and visualization perspective, while providing the runtime + parser as an open source project for folks who want to host notebooks elsewhere.

There are some “offline” and “interpreter” editors available (the following are some of my takes):

Ideally we would have an API to facilitate the uploading and publishing of notebooks back here…

2 Likes

Honestly considering there is so much content on observablehq the community will never switch from observablehq, but allowing you to make your own server would probably make it way way more popular. Besides sometimes you just don’t have internet connection!!

3 Likes

Indeed on all the on premise comments.

Our cyber security team doesn’t allow us to run on cloud hosting for what we consider critical workloads - and that then includes the data we want to put in observable

Hi @MarkGStacey ! I am also very alert to security issues, and I similarly face restrictions for the use of third-party data stores for work-related information. In case this helps you gain greater flexibility in your use of Observable, it might help to point out that Observable doesn’t even need to touch your data! That is, if you supply data into Observable via a database connection, or if you use methods like hosting data on S3 and expositing it only via URL to authenticated users, you can use Observable to execute your visualization code, and your actual data is never revealed to anyone except your intended users. It doesn’t get hosted on Observable, and Observable cannot see it in any way.

I imagine that folks working at Observable may have some notebooks somewhere that better outline code execution, security, etc. Regrettably I am not able to remember a great official example to point at on this exact topic. To give you an idea of how the S3 connection works, I made and shared this notebook as a proof of concept for S3 connections. The cells that look ‘broken’ at the bottom are simply waiting for authentication. In effect, this is an example of a notebook ‘without’ data (because my data are never exposed… even when authenticated, the only end point that sees my data is my own browser.)

Hope this helps!

Thanks Aaron!

It’s a good point : you keep your API behind your firewall, restrict access etc

However, in our world (hedge funds), where we trade on data, even identifiers and names are considered top secret and knowing what we’re looking at even if you can’t see it can give competitors a leg up.

It’s something we struggle with all the time - and the theft of IP does happen in our industry so it’s not just our cyber team being over vigilant unfortunately

2 Likes

What I’m worried about is Observable going out of business; Will I be able to still run my notebooks? If not, Observable is not worth the investment.

2 Likes

Totally seconded. I believe a commercial offering allowing you to host on kubernetes etc would be a great way to commercialise the product

Hi @MarkGStacey and @NightMachinary - Thanks for this conversation! I’ll respond with as best I can on both concerns. Please note, too, that I am not an Observable team member:

Security

I work mostly with private, government-owned data. I operate in a context where I cannot install software on my computer, and I cannot use most third-party sites and tools for development and testing. I can, however, use Observable, because it does answer Mark’s concern. Here’s how:

Using teams, I can collaborate with colleagues in completely non-public, but shared coding environments. Using secrets within Teams, I can rest assured that if anyone ever tired to expose our working notebook publicly, that connection would immediately break. While I don’t use the Observable database connectors, it’s a similar principle: your host and port properties are not exposed, but you can open access to whatever data that you wish to evaluate. In my S3 notebook, I agree that it’s something of an ‘exposure’ that my bucket name and file name is there in the config, and I fortunately don’t have to worry about exposing details at this level (as our security team is comfortable with IAM roles and S3 bucket permissions, and we can avoid unwanted access through these protocols). I could very easily, however, just expose text-input fields for bucket name and file name and never would these values be exposed either: access would require a user knows where to point the connection. However, I imagine that database connections should get you where you’re going.

edit: I just caught your concerns here and the points you raise are a bit beyond my capacities to address. Thank you for the clear statement of your challenges! I appreciate learning these pain points as I am slowly building the case to integrate Observable more centrally within my organization.

Longevity

@NightMachinary - This is also a very valid concern. I have a couple of points for your consideration. One is that Observable-as-a-platform is different than Observable-as-a-runtine. Just as you may not be concerned that React.js will ‘go out of business’ (because it’s a technology, not a company per se), so too should you not have to worry about Observable going out of business. As long as the runtime works, you can host and work with your code entirely independently of the platform itself. Now with respect too all of these shared secret protocols and database connectors connected to the Teams working environment: Yes, if the company fails, then it becomes more difficult to work privately (and in a real time co-editing environment). This is part of the advantage of Observable-as-a-platform and it’s also something that I am thinking about a lot in terms of how my organization can use Observable. But at least I am assured that, as long the JS packages that I use work, so too will my notebooks, regardless of the success of Observable-as-a-platform (though I wish the team great success)!

… As a last point of reflection, I would like to point out that it’s sorta similar to Git vs. GitHub. Git is a technology. GitHub is a platform / interface leveraging that tech and adding a lot of extras. There’s not a necessary or direct connection between Git and GitHub (and in many respects, the centralization of code on GitHub is mis-aligned with the essence of Git), and so too you can carry on all day in Git without ever needing GitHub (which is also how I work, for the most part).

Hope this helps! Happy to continue the discussion!

2 Likes

There is something called starboard… As far as I know you can host it yourself. Show HN: Starboard Observable – An open source ObservableHQ notebook editor | Hacker News

1 Like

Here’s a new self hosted Observable notebook project called Dataflow:

Outside of any other considerations of whether or not the technical aspects work - as an enterprise, we are 100% restricted from using any cloud tools that could potentially leak data.
We have to justify and get approval for any cloud system (and it does happen - e.g. big things like Salesforce or Snowflake) but it is months of due diligence questionnaires, security inspections)

Something like “you can access local data sources directly, but if you upload a file it goes to our servers” is the sort of hole that would be flagged as a security risk.

Not being able to bring audit trails of who is accessing the notebooks would be problematic. Integration to our security systems would be problematic.

Having a commercial version (at a large premium, charged per server instead of per user, or whatever model suits you) would work for us, and I’m sure the many other commmercial orgs in the same boat and similar industries

1 Like

@Cobus - Any enterprise solutions on the horizon?

@MarkGStacey - I just last week requested help within my institution about initiating a security review. I’ll let you know how it goes (if it goes :wink: )

1 Like

Mark, it would be great to hear some more about your organization’s security needs. Happy to assist in conversations with your security team.

As Aaron mentioned, we are working on an Enterprise offering with a focus on security, authentication, controls, audit-ability and support, so your thoughts would be much appreciated. Please reach out to teams-support@observablehq.com so we can discuss some specifics of your organization’s needs.

1 Like

A. The community could create a custom open sourced editor for observablehq, based on the runtime

and maybe

a) as a Jupyter kernel or

b) as a Jupyter extension or

c) as a Jupyter cellmagic (would allow to mix observablehq cells with other cells in the same jupyter notebook)

Here is a first draft for a custom observer that allows to edit cells:

B. Here is some discussion about reactive Jupyter Notebooks:

However, those attempts all seem to be in experimental/outdated state.

C. Unfortunately, observablehq does not seem to allow importing a notebook that has been exported before and edited elsewhere. Please correct me, if notebooks can be re-imported to observablehq from files.

D. Related:
JOB: Jupyter ObservableHQ Bridge

Plugin for starboard to support observablehq cells, based on unofficial-observablehq-compiler:

https://starboard.gg/gz/open-source-observablehq-nfwK2VA