Noob: Can you run your own Observable server

Unfortunately in-house only, and hehehe band from anything aws. :wink: We have our own internal github, npm, etc. We would need to be certain that our algorithms that make us who we are and let us do what we do stay ours. We share other things like you said in fact many years back one of my first good videos on node.js came from a WM developer. :slight_smile: I contribute when i can to ag-grid and the angular community, but I dig what you guys are doing as I made something similar many years back I called it one field (I am a former unix systems engineer, so i have special love for command line) … a solution that would maybe work would be we fire up oN (observableNotebook?) :slight_smile: . and it lets me point to internal source to just pull in maybe the json or whatever voodoo thats under the hood, and then the chocolate and peanut butter collide. Sort of how vsCode gives you the source, and instead of hoarding plugins and such we as a community not only build the extensions and provide constant fixes and feedback, others in community invent stack-blitz. I want to embed your notebooks right in my app, so data scientists can tweak there d3 visualizations through your interface and the results are instantly there for our internal customers. I want to provide living style guide for my developers. Once I saw your stuff in action, I was like this is a game changer, and why isn’t this the only topic afoot? Then when I figured out i can’t do it in the privacy of my own network, i was very sad. Though that hasn’t stopped me from getting peoples hopes up with the possibilities and the hope you might find a way… If you can’t you can’t I mean I can see the $ potential there, so I get it, worse case scenario I’ll do my best to recreate what I can, just for the sake of seeing how close I can get :wink: Were are the jobs of making break-throughs like this? Gotta be like Mike and make your own hehehe, thanks for the response, seeing how it works and playing with it already has my brain spinning off out of control… I dropped some three.js in there and a few controls, and that gave me a new idea on an angular app I am building, I should it to my data scientist and he was like ‘I Wants’, so again way to take a different slant and spur some interesting new tech. -Chris

1 Like

We may offer some on-premise (self-hosted) enterprise solution in the future, but our current focus is on developing cloud-hosted private collaboration for teams.

4 Likes

Hey @mbostock, awesome work.

I really would like to use this internally in my company - we have private code and multiple people working on the same visualizations at the same time.

Is this something that is on the timeline?
or should we just continue writing d3+git for the time being :slight_smile:

I would like to second the idea that internal hosting is the only option for many companies. I first ran into this tool today while following a trail via TensorFlow.js. I believe there would bean opportunity for teams to develop notebooks using Observable in-house, if this were supported. For many companies, the idea of hosting IP outside the organizations, no matter how strong the security may seem, is a show-stopper.

I was immediately struck, when looking at this, at the idea that editors could be extended (with JS, HTML, CSS-specific code highlighting, for example) and notebooks could also be edited in tools like Visual Studio Code (with suitable plugins) to make the process so much richer.

I would encourage you to evaluate the possibility of offering expertise (for a price) rather than a specific technology foundation. This approach encourages wider adoption, more solid technology foundations and is a proven business model that doesn’t restrict access to a single web site. This may not be the current plan but a lean startup strategy may be suggesting this direction, given the feedback I’ve read here. This is a potential huge adoption gap for a large part of the community you are trying to reach (for IP reasons).

PS: Thanks for developing this. I plan to dive in and learn more. This will likely fit my personal needs but I am a community of 1 and cannot realistically evangelize inside the company I work for given the hosting limitations,

5 Likes

I actually really resonate with what claude is saying.
The power of ipython/jupyter was:

  • use your own data
  • host is on your machine or on your internal private network
  • work offline
  • it’s open source

I expect that devs can already build up with what’s out there a very simple editor for observable.
I think there is still venue for observablehq.com to be the platform where packages are host and open source is written, teams can collaborate etc…

2 Likes

hear hear! also, I’d like to see this happen too!

1 Like

I thought this was related. Link to running observable headless by @jflatow :

Chiming in late here, I’d love to be able to use Observable with my team at Stitch Fix, but it’s a non-starter unless it’s on-prem. We’d be happy to pay for that privilege.

7 Likes

It’s a shame there isn’t an offline offering. Without it, it’s unlikely observable will gain as much adoption as Jupyter. That adoption could eventually drive the cloud-based tool adoption.

4 Likes

I believe the goal is to try and keep the community in one place (here) from a data analytics and and visualization perspective, while providing the runtime + parser as an open source project for folks who want to host notebooks elsewhere.

There are some “offline” and “interpreter” editors available (the following are some of my takes):

Ideally we would have an API to facilitate the uploading and publishing of notebooks back here…

2 Likes

Honestly considering there is so much content on observablehq the community will never switch from observablehq, but allowing you to make your own server would probably make it way way more popular. Besides sometimes you just don’t have internet connection!!

3 Likes

Indeed on all the on premise comments.

Our cyber security team doesn’t allow us to run on cloud hosting for what we consider critical workloads - and that then includes the data we want to put in observable

Hi @MarkGStacey ! I am also very alert to security issues, and I similarly face restrictions for the use of third-party data stores for work-related information. In case this helps you gain greater flexibility in your use of Observable, it might help to point out that Observable doesn’t even need to touch your data! That is, if you supply data into Observable via a database connection, or if you use methods like hosting data on S3 and expositing it only via URL to authenticated users, you can use Observable to execute your visualization code, and your actual data is never revealed to anyone except your intended users. It doesn’t get hosted on Observable, and Observable cannot see it in any way.

I imagine that folks working at Observable may have some notebooks somewhere that better outline code execution, security, etc. Regrettably I am not able to remember a great official example to point at on this exact topic. To give you an idea of how the S3 connection works, I made and shared this notebook as a proof of concept for S3 connections. The cells that look ‘broken’ at the bottom are simply waiting for authentication. In effect, this is an example of a notebook ‘without’ data (because my data are never exposed… even when authenticated, the only end point that sees my data is my own browser.)

Hope this helps!

Thanks Aaron!

It’s a good point : you keep your API behind your firewall, restrict access etc

However, in our world (hedge funds), where we trade on data, even identifiers and names are considered top secret and knowing what we’re looking at even if you can’t see it can give competitors a leg up.

It’s something we struggle with all the time - and the theft of IP does happen in our industry so it’s not just our cyber team being over vigilant unfortunately

2 Likes

What I’m worried about is Observable going out of business; Will I be able to still run my notebooks? If not, Observable is not worth the investment.

2 Likes

Totally seconded. I believe a commercial offering allowing you to host on kubernetes etc would be a great way to commercialise the product

Hi @MarkGStacey and @NightMachinary - Thanks for this conversation! I’ll respond with as best I can on both concerns. Please note, too, that I am not an Observable team member:

Security

I work mostly with private, government-owned data. I operate in a context where I cannot install software on my computer, and I cannot use most third-party sites and tools for development and testing. I can, however, use Observable, because it does answer Mark’s concern. Here’s how:

Using teams, I can collaborate with colleagues in completely non-public, but shared coding environments. Using secrets within Teams, I can rest assured that if anyone ever tired to expose our working notebook publicly, that connection would immediately break. While I don’t use the Observable database connectors, it’s a similar principle: your host and port properties are not exposed, but you can open access to whatever data that you wish to evaluate. In my S3 notebook, I agree that it’s something of an ‘exposure’ that my bucket name and file name is there in the config, and I fortunately don’t have to worry about exposing details at this level (as our security team is comfortable with IAM roles and S3 bucket permissions, and we can avoid unwanted access through these protocols). I could very easily, however, just expose text-input fields for bucket name and file name and never would these values be exposed either: access would require a user knows where to point the connection. However, I imagine that database connections should get you where you’re going.

edit: I just caught your concerns here and the points you raise are a bit beyond my capacities to address. Thank you for the clear statement of your challenges! I appreciate learning these pain points as I am slowly building the case to integrate Observable more centrally within my organization.

Longevity

@NightMachinary - This is also a very valid concern. I have a couple of points for your consideration. One is that Observable-as-a-platform is different than Observable-as-a-runtine. Just as you may not be concerned that React.js will ‘go out of business’ (because it’s a technology, not a company per se), so too should you not have to worry about Observable going out of business. As long as the runtime works, you can host and work with your code entirely independently of the platform itself. Now with respect too all of these shared secret protocols and database connectors connected to the Teams working environment: Yes, if the company fails, then it becomes more difficult to work privately (and in a real time co-editing environment). This is part of the advantage of Observable-as-a-platform and it’s also something that I am thinking about a lot in terms of how my organization can use Observable. But at least I am assured that, as long the JS packages that I use work, so too will my notebooks, regardless of the success of Observable-as-a-platform (though I wish the team great success)!

… As a last point of reflection, I would like to point out that it’s sorta similar to Git vs. GitHub. Git is a technology. GitHub is a platform / interface leveraging that tech and adding a lot of extras. There’s not a necessary or direct connection between Git and GitHub (and in many respects, the centralization of code on GitHub is mis-aligned with the essence of Git), and so too you can carry on all day in Git without ever needing GitHub (which is also how I work, for the most part).

Hope this helps! Happy to continue the discussion!

2 Likes

There is something called starboard… As far as I know you can host it yourself. Show HN: Starboard Observable – An open source ObservableHQ notebook editor | Hacker News

1 Like

Here’s a new self hosted Observable notebook project called Dataflow:

Outside of any other considerations of whether or not the technical aspects work - as an enterprise, we are 100% restricted from using any cloud tools that could potentially leak data.
We have to justify and get approval for any cloud system (and it does happen - e.g. big things like Salesforce or Snowflake) but it is months of due diligence questionnaires, security inspections)

Something like “you can access local data sources directly, but if you upload a file it goes to our servers” is the sort of hole that would be flagged as a security risk.

Not being able to bring audit trails of who is accessing the notebooks would be problematic. Integration to our security systems would be problematic.

Having a commercial version (at a large premium, charged per server instead of per user, or whatever model suits you) would work for us, and I’m sure the many other commmercial orgs in the same boat and similar industries

1 Like