Integrating ganja.js - thoughts

Community Help
1

Hi I uploaded my library (ganja.js) to npm so I could include it more easily in observablehq. Getting an error now without line numbers or context - anyway I could get to see what I’m doing wrong ? (works fine in node, the browser and npm’s preview tool)

Algebra = import("ganja.js@1.0.3/ganja.js")

Also, ganja.js uses DOMparser in one of its (browser targetted) functions to parse an SVG string - but in observablehq this interface is not available ?

Apologies for the noob questions - thanks in advance for any help.

1 Like
2

A lot of people will be happy to see you!

A trick I use to test my modules is to load them locally:
Algebra = importrequire("http://localhost/ganja.js/build/ganja.js")

strangely it works (from Chrome at least) which allows all sorts of dirty debugging with console.log or other tools.

3

Ah excellent - should’ve thought of that :slight_smile: - trying now - thanks Fil !

4

Ok - if I use require instead of import - it loads fine :+1:

Algebra = require("ganja.js/ganja.js")

Next question - perhaps I should start a new topic … my graph function returns either a canvas or an svg object depending on what it needs to graph. If I return a canvas object it works fine …

Algebra(0,()=>this.graph(x=>Math.sin(x*Math.PI)/2))

It gets displayed correctly as the cell output.

if however I return an svg object, it simply shows SVGSVGElement … no image.

I created a quick notebook to demonstrate

Should I not expect the SVG element to be displayed as image ?

any help appreciated !

5

sorry I should have spotted this.

Your example can be fixed with

{
  var div = DOM.element("div");
  div.appendChild( Algebra(2,0,1,()=>this.graph([1e12,"o",1e1,"x",1e2,"y"])) );
  return div;
}
1 Like
6

The reason for that is something I demonstrate in this notebook - Ganga appends the SVG element to the page by itself as well as returns it as a value. Observable can’t show the same element in two places, so the only way to cleanly display it is to detach it, or as Fil points out, attach it to a different container.

1 Like
7

Hi Tom,

That´s a cool notebook !

ganja.js does not append to the page. The graph function only returns the svg element. A simplified testcase is this :

(function(){
  return new DOMParser().parseFromString(`<svg height="100" width="100"><circle cx="50" cy="50" r="40" stroke="black" stroke-width="3" fill="red"/></svg> `,'text/html').body.firstChild;
})()

I would expect that to simply return the svg element - you can paste that in the console - it seems to do just that. (setting innerHTML on svg elements does not work - which is why I’m using DOMParser).

Although pasting that in observable gives an error on DOMParser being not defined.
(Which I saw earlier when trying to include ganja, but using require seems to have resolved that to)

Again, sorry for the noob stuff - I’ve only skimmed through the intros - I really should dive into the docs first.

Once I get past these basics I plan on thinking about what we can do to better combine the idea of transpiled Javascript and observablehq. (as you and others have noticed, you lose some lexical scoping and I want to play around to see if there’s a workaround possible - or which model would fit best if not)

Thanks for the quick reply,

Cheers !

(btw I’ve also made a notebook with an easier introduction on ganja.js - some simple complex numbers and funny array stuff … in case you haven’t seen it yet)

8

The domParser.parseFromString call returns a new document with a document element and a body. So from the perspective of Observable, the returned SVG element is already attached to a document—it just so happens to be a different document than the main one. You need to remove the SVG element from its parent before returning it from the cell.

Observable does not append an element that already has a parent because doing so would remove it from its existing parent; this is a destructive side effect and we generally try to avoid side effects. Observable only appends detached elements.

1 Like
9

Gotcha !

github & npm are updated … ganja.js can now cleanly be included :

Algebra = require('ganja.js')

and all graphing works as expected

Algebra(2,0,1,()=>this.graph([1e12,1e1,1e2]))

Thanks for the help !

:+1::+1::+1:

4 Likes
10

Right on! I look forward to more ganja.js notebooks! :smile:

11

Mike,

Have you got any ideas on how to better integrate transpiled javascript into observablehq ?

ganja.js translates javascript to javascript, keeping unknown keywords (say ‘mutable’) in place, so it has the potential of integrating very neatly.

Just freewheeling here, and I’m totally open to suggestions - but two options come to mind. One option would be a way to tell observable to transpile javascript … this would potentially enable smooth integration with other js to js transpilers.

Another option would be for observable to provide a ‘scoped eval’ call so that you could tell ganja to use that call to evaluate its translated functions.

The first option, where observable knows it needs to transpile javascript :

observablehq.setTranspiler(Algebra(2,0,1).inline);  // just freewheeling

after which cells could contain full math like so :

xy = 1e12

followed by e.g.

z = 1e123 * xy

or

even=[1,2,3,4,5,z]*2;

In the second scenario they would look like so :

PGA2D = Algebra({p:2,r:1,eval:observablHQ.scopedEval})

p = PGA2D.inline(()=>1e12)()

z = PGA2D.inline(()=>1e123*xy)()

even = PGA2D.inline(()=>[1,2,3,4,5,z]*2)()

Ofcourse, chances are you already have a better model in place, looking forward to hearing your thoughts.

12

Haha, that would be a bit crazy, but crazy cool :slight_smile:

The whole lexical scoping issue for ‘inline functions’ can of course be swept under the rug from the perspective of an Observable notebook writer, because Observable ‘invents’ this scope outside of the single cell anyway.

I’m not sure if Tom/Jeremy/Mike are so keen on adding transpilers, but this would be a case where I’d just love it :slight_smile:

1 Like
13

Very intriguing — and especially your “matrix” multiplication example is a very compelling taste of just how useful an “include my transpiler” directive might be.

One thing I wanted to mention is that: Although we don’t yet have concrete plans about how to (or whether to) implement this sort of thing, we are hoping to open-source the Observable transpiler (mutable, viewof, cell references), and the runtime in the near future. And once those pieces of the puzzle are available, maybe you folks can use them to explore some of the most interesting possibilities for pre-compilation of cell bodies…

14

Our current approach for transpilation (dialects or languages other than JavaScript) is based on tagged template literals. We do this for Markdown (md), HTML (html, svg) and LaTeX (tex).

You could do something similar for ganja. Sketching:

e2 = Algebra(2, 0, 1)

xy = e2`1e12`

z = e2`1e123 * ${xy}`

even = e2`[1, 2, 3, 4, 5, ${z}] * 2`

You’d lose the syntax highlighting, of course, and you’d probably need to reparse and regenerate the code when a reactive reference changes (e.g., z in even), but maybe we can think of a clever way to avoid that, say using this values.

15

To clarify a subtle point: the embedded expressions within the template literals needn’t be string-coerced. That is how, for example, you can embed a DOM element or an array of DOM elements within an html template literal. See the source here:

I imagine you could do something similar for ganja where the embedded expressions become placeholders (arguments) that you can pass in during evaluation.

16

Hi Mike,

There’s a notebook here that give this a spin. Missing syntax highlighting is unfortunate, and for complex expressions, the extra ${} can add quite a bit of clutter.

the original ganja.js code :

  var orthocenter = (B&C<<A)^(C&A<<B);

which calculates the orthocenter of a triangle given the three vertices A,B,C. It’s quite readable if you know PGA (from left to right : find the line orthogonal to the line BC and through A and intersect it with the line orthogonal to the line CA and through B). now looks like this if you want it in a separate cell :

  orthocenter = PGA2D`(${B}&${C}<<${A})^(${C}&${A}<<${B})`

It all works but feels like a bit of a shame since a transpiler that works before observable would allow to drop all the clutter. For a project like ganja that is all about clean-lean-mean-math-syntax … I’ll keep hoping for a better way …

17

Thanks for exploring this idea. I agree the additional syntax for embedded expressions are cumbersome in this case (since the DSL is already so close to JavaScript).

Our tagged template literals allow arbitrary languages to be embedded in vanilla JavaScript. These literals are interpreted by their tag function after Observable transpiles the cell definition to vanilla JavaScript. It seems like what we want here is a transformation prior to Observable transpiling the cell definition to JavaScript, rather than after. That way, Observable can parse the transformed cell definition and extract the references to build the dataflow graph.

So for example, say you have the following cell definition, and Observable somehow knows to apply a PGA2D transformation:

orthocenter = (B&C<<A)^(C&A<<B)

The transformation would run, converting this into something like:

orthocenter = CC.Wedge(CC.Dot(CC.Vee(C, A), B), CC.Dot(CC.Vee(A, B), C))

This is now (Observable) JavaScript, and can be parsed and executed normally by the Observable runtime.

But there are several challenges to this approach:

  • How do you define and register a transpiler?

  • How do you declare that a cell should be transformed by a given transpiler? (I don’t think we want this operation to be global to the notebook; their either needs to be some sort of pragma or metadata associated with each cell that specifies the transpiler.)

  • How do transpilers inject runtime values, such as the CC Algebra instance above? Are these injected values reactive?

  • When a cell is transpiled, how does syntax highlighting and autocomplete work? (Possibly these are simply disabled when transpilation is active; otherwise we’d have to define extension mechanisms for these features, too.)

Ideally, the transpiler is defined in a cell, and maybe there’s a tiny bit of syntax to apply it to a cell, like:

PGA2D = (await require("ganja.js"))(2, 0, 1)

orthocenter = PGA2D: (B&C<<A)^(C&A<<B)

Anyway, this isn’t an easy problem, sorry! But enabling extensible transpilation could be a very powerful feature, so we’re going to keep thinking about it.

18

Another possibility is that the transpiler skips the Observable JavaScript representation and goes straight to vanilla JavaScript. Then the transpiler would be responsible for extracting the references from the code and returning an array of named inputs and a function body. (We’re going to open-source the runtime soon, which will make this API a bit more clear) As in:

({
  inputs: ["A", "B", "C"],
  value: (A, B, C) => CC.Wedge(CC.Dot(CC.Vee(C, A), B), CC.Dot(CC.Vee(A, B), C))
})

That lets the transpiler use closure for its runtime values (CC).

19

Hi Mike,

Thanks for the feedback. The key is indeed that this type of transformation can only truly deliver its power if it sits before observable. Your assessment is spot-on. I’m fine with either option. (although ganja.js currently does not know what variables are from what scope and simply doesn’t touch any of them).

For now, I’ve upgraded ganja so that inline can also be used as template function - it makes the template approach as simple as I can get it :

Algebra = require("ganja.js")

p2 = Algebra(2,0,1).inline

some_point = p2`1e12+.5e02`

some_expression = p2`2*${some_point}`

For those libs that are true js->js (like ganja or say prepack) it would really be killer to be able to keep syntax highlighting. (ganja’s operators and syntax have been painfully restricted to keep editor features valid)

I have no brilliant suggestions for how that would be most elegantly specified in observable - one idea is to augment template string functions with extra properties. (i.e. specify for a template function what syntax highlighter to use, and if its a ‘before’ or ‘after’ template)

var templateFunc = Algebra(2,0,1).inline;
templateFunc.syntaxHighlight = 'javascript';
templateFunc.beforeObservable = true;

This would give an ‘in’ to also do syntax highlighting on html,svg,tex, …

Just a thought.

Cheers,

S.