Overcoming difficulties with JSON data loading (or another mistake)

Community Help
1

Dear Observerable Team and Community Members,

I am trying to convert Mike’s BiLevel Donut Chart from Bl.ocks to Observable.

Here’s where I am at:

I’ve made a few adjustments: updating bits here and there to update from d3v3 to d3v4, attempting to add in a <svg> container to which the data can bind in an effort to learn from Tom’s earlier feedback. Most everything appears to be in order (at least I am not getting errors with my cells), but still the visualization isn’t loading.

I suspect that there’s an issue with how I am referencing the json file. In the example, the data were stored in a json data file. I tried using d3.json to load in the data, and “named” the cell as jsondata=. I tried adjusting the example to call in these data, but the chart is drawing as an empty container.

Towards solving this on my own, I referred to Mike’s Introduction to Data notebook, but this didn’t address issues of naming a dataset (which I am guessing is my issue). I also tried re-organizing the example in code in a manner similar to Mike’s donut chart example… which I unfortunately didn’t fork into a separate notebook for reference, but which was giving me a different error wherein the svg element was being returned in a weird way. Lastly, I watched Tom’s YouTube Video on Converting D3.js to work in Observable. Still no dice.

Unfortunately, I don’t know if my mistake is actually with the data loading, or if I am making another mistake in rendering the chart.

I would be happy for some help and guidance. Thanks so much to Tom and Mike for all the tutorials and problem-specific help and insights that you’ve shared with me already!!!

Sincerely,

Aaron

2

You’re loading D3 v4; in Observable you really want to use D3 v5, which replaces the callback-style API of d3-request with the promise-based API of d3-fetch. (See CHANGES for more.) Observable, and today’s JavaScript in general with async and await, have standardized on promises for asynchronous code.

So first, replace this:

d3 = require("https://d3js.org/d3.v4.min.js")

With this:

d3 = require("https://d3js.org/d3.v5.min.js")

(You might also use the non-minified d3.v5.js when developing for easier debugging.)

Then replace this callback:

d3.json("jsondata", function(error, root) {
  …
})

With either promise.then:

d3.json("jsondata").then(function(root) {
  …
})

Or await:

{
  const data = await d3.json("jsondata");
  …
}

And in Observable, if the value of a cell is a promise, then referencing that value from another cell will implicitly await the promise, so the best approach is to say:

data = d3.json("jsondata")

See Introduction to Promises for more on this topic.

But none of these examples will work yet :anger: , because per the d3-fetch README, the first argument to d3.json is a URL, not a name, and not a JSON object. The above cells are all looking for a file named “jsondata” relative to the notebook, and obviously that file does not exist. And similarly, when you say:

d3.json({"name": "flare", "children": […]})

It’ll trying to load a file named “[object Object]”… and yeah, that’s not going to work. :grin:

Per the Introduction to Data notebook, two currently-recommended approaches are to host your data on GitHub, say as a gist, or to inline the data directly in your notebook if it’s small. With the latter approach, that would simply be:

data = ({
  "name": "flare",
  "children": [
    …
  ]
})

Or, to load from GitHub, using one of my existing gists of the Flare dataset:

data = d3.json("https://gist.githubusercontent.com/mbostock/1093025/raw/b40b9fc5b53b40836ead8aa4b4a17d948b491126/flare.json")

We’ve even made a require-able version of this dataset for convenience:

data = require("@observablehq/flare")

You can browse our example datasets on GitHub.

5 Likes
3

Also, a high-level recommendation:

Don’t try to write an Observable notebook by copy-pasting a complex (non-Observable) example all in one go. Instead, break apart the source example into smaller pieces and build up the notebook incrementally. For example, start with the dataset, then define a scale or two, then a layout, etc.

If you put all the code in a single cell, it greatly reduces the benefits of Observable because you have no visibility into the state of your program. That’s especially true when your code is asynchronous (loading data, event listeners, transitions).

If your code is in separate cells, then the value of each cell is shown next to it. This lets you debug each small piece independently, getting immediate and local feedback on each piece of code. By building your notebook up from small pieces, it’s easier to understand how it works and fix something if it goes wrong.

4 Likes
4

Thank you, Mike! I will work on this one some more and will try breaking it down into pieces, per your recommendation. I’ll report back later after I’ve had a chance to work through it.

I really appreciate your care and all the time and attention you give to helping others! It’s inspiring to see a person with not only your technical expertise, but also your social grace and commitment to helping others.

And I shall continue to learn. I have a way still to go, but the pieces are slowly coming together. Thanks again!

1 Like
5

Thanks again for all the help. I am still stuck on a how (or if) I can load JSON data in-line.

As you note in your reply and in the Introduction to Data notebook, it is possible to in-line data into Observable. Although your @observablehq/flare dataset is not small, I wanted to in-line it for this example so that I can play around with the example dataset. My purpose in so doing is to learn more about how the JSON file works, and ideally to adapt this visualization to a much smaller dataset. With a working Observable notebook, I’ll have immediate feedback as I manipulate the JSON file, which will be great :slight_smile:

From what I understand of your comment and the d3-fetch README, however, I can’t combine the d3.json approach to call on in-line data, as this module requires that my data be specified as data = d3.json("jsondata"), where jsondata is a URL (or at least a path to a JSON file).

What would be the alternative module for working with JSON data loaded in-line?

Regarding your suggestion for breaking down examples into bits: I will continue to work on this. I continue to have difficulties in understanding how bits of JavaScript are connected, and where and how to separate out components. This is actually one of the parts that I most appreciate about Observable – that it allows me to play around with JavaScript to learn how the bits work. My newbie approach has been first to get a bl.ocks example working, then to try to manipulate it so as a better understand the parts. The learning curve is steep…

6

You don’t need d3.json to parse inline JSON. You can just use an object literal, as I showed above:

data = ({
  "name": "flare",
  "children": [
    …
  ]
})

d3.json is for fetching and parsing JSON. You can parse a JSON string directly using JSON.parse. Or you can use an object literal (as shown above) and you don’t need parsing or fetching at all—it’s just an object you’re directly defining in code.

1 Like
7

Thanks for the immediate reply – though I must admit that I don’t fully understand it.

I read in my data as:

jsondata = ({
  "name": "flare",
  "children": [
    …
  ]
})

Where I am struggling is how to reference these data in the chart. I tried the promises method:

d3.json("jsondata").then(function(root) {

But the chart still fails to show up. My reading of your previous discussion was that the d3.json module was inappropriate in this context, as it’s still looking for a for a file named “jsondata” relative to the notebook, which does not exist.

Is it correct that, to solve this, I need to figure out how a callback that references the in-line object literal? Is this what you mean in stating (?):

…in Observable, if the value of a cell is a promise, then referencing that value from another cell will implicitly await the promise, so the best approach is to say:

data = d3.json("jsondata")

… I’ve played around a bit in trying to do this, but haven’t yet succeeded.

I hope these questions are appropriate. I feel like I should be reading and studying more before asking. :confused: I’ll take a few steps back and work again through intro to JavaScript tutorials before going much further. I’ve been reading and working through your tutorials on Observable, which are helpful, but I suspect my challenges are more rudimentary.

8

You literally say jsondata. :smile: You don’t need anything else. And that’s true regardless of whether you define the jsondata inline or fetched from a remote server as a promise, as with d3.json.

The idea is that a cell is responsible for defining its value, and other cells can reference that value without worrying about how it’s defined. This decoupling, where referencing cells care only about values and not implementations, is the purpose of Observable’s dataflow (reactive) runtime.

So for example, if you say:

data = ({name: "Fred"})

In another cell, you can say:

md`Hello, ${data.name}!`

And it will say “Hello, Fred!”

Likewise, if you change the data cell to be defined as:

data = d3.json("https://gist.githubusercontent.com/mbostock/1093025/raw/b40b9fc5b53b40836ead8aa4b4a17d948b491126/flare.json")

Then the “Hello, Fred!” will automatically change to say “Hello, flare!”

9

Thanks again for all the help and pointers. I’ve re-marked your original answer has having solved my question on data loading. I still haven’t succeeded in getting the chart to render. :frowning:

Specifically, when replacing

d3.json("jsondata").then(function(root) {

with

const data = jsondata;

I start down another road of challenges for which my current JavaScript prowess is too limited. In this case, I create a runtime error because root isn’t defined. I’ll keep at it.

I took your advice and tried again at breaking down the example into component pieces:

I hit another snag with the way that margins were defined in the example. Seems like I’ll have several more challenges before I figure this one out :wink:

I really appreciate all the support!

With sincere thanks and kind regards,

Aaron

10

The margin syntax error is being you are missing parentheses. An Observable cell looks like a normal assignment statement in JavaScript, but it isn’t. In normal JavaScript, you can say:

const margin = {top: 20, right: 30, bottom: 30, left: 40};

But in an Observable cell, you need parentheses:

margin = ({top: 20, right: 30, bottom: 30, left: 40})

That’s because an Observable cell is more akin to the body of an arrow function: it can be either an expression or a block statement. If you were writing an arrow function, and you said:

const margin = () => {top: 20, right: 30, bottom: 30, left: 40};

You’d get a syntax error. But if you say:

const margin = () => ({top: 20, right: 30, bottom: 30, left: 40});

Then the margin function returns the expected object. Observable cells are the same way—it’s just that the Observable runtime calls your function (your cell definition) automatically.

As for the other error, I think you need to replace:

d3.json("jsondata").then(function(root) {
  … code that references root goes here …
});

With

const root = jsondata;

… code that references root goes here …

That, or just rename your jsondata cell to root, and then you don’t need a local variable, and you can just reference the cell value directly.

Take a look at my sunburst notebook for a related example:

11

Also, thanks to your motivation!, I have now posted my own Observable port of my zoomable / bilevel sunburst diagram:

3 Likes
12

Awesome! Thank you so much! I haven’t had much time since yesterday, but made some progress. Stuck now with on the partition function, so I’m excited to see how you solved it.

I really can’t express how thankful I am for your time and encouragement. I really appreciate all that you do!

2 Likes
13

Hello Sir,

Please post it in plain JavaScript format too. I am really facing issues in implementing it with PHP and javascript. I have been struggling since last 2 weeks and I am unable to give a similar output.
I have tried your latest code for d3 zoomable sunburst bilevel partition with texts, but somehow, I am unable to implement it. I think the code which you have posted is in Node.js. Can you please show a similar implementation via plain JS please?
Thanks
Atish Agrawal

14

Hi Atish,

Can you share where you’re starting from? Mike’s original version of this chart is in JavaScript running D3.js. My notebook didn’t quite accomplish a self-challenge of re-creating the notebook to work on this platform. Mike, however, demonstrated an elegant solution for the purpose of exploring further JavaScript customization. Unless I am mistaken, this involves no PHP. Node.js is a runtime on which Observable runs.

I don’t know that this helps you, If you can share what you’re working on it would be easier to gain insight into the challenges you’re facing.

Hope you figure it out!

Aaron

15

Hi, This thread sent me in a few directions. Reference CORS and using API JSON

Background-situation:

We have an api it serves data in JSON format, we are using Django RESTFramework. The ideas is to use the data (which changes every week or so) to create visualisations on Observable.
We have created a section of the website for that “hammerdirt labs Python/JS”.

Here is a search for sampling sites in one city (it is only four object):

http://mwshovel.pythonanywhere.com/dirt/beaches/Vevey/?format=json

I cannot access that in Observable notebook. This is a CORS issue with Django REST api (the extension is not installed yet).

Question: What domain needs to be white listed in the CORS setting to allow anybody using Observable to access data and create visualisations?

Another way to put that question: IF I want anybody who is using an Observable notebook from observablehq.com how what is the domain to specify to the CORS setting?

Thanks

1 Like
16

Do you consider your API public or private?

It looks public, in the sense that I can click on that link that load it in my browser. So if you want your data to be publicly-accessible (including from Observable), then you need two things:

  1. You need to host it on HTTPS. Observable is hosted on HTTPS, and the browser only allows you to fetch from other HTTPS sites; it doesn’t allow you to fetch from HTTP sites. (There are workarounds for this, such as JSONP, but I don’t recommend them.)

  2. You need to enable CORS. Specifically, you need to add Access-Control-Allow-Origin: * in the response headers so that other domains can fetch from your site and read the response.

If you want your data to be accessible only to certain authors on Observable, then you can whitelist just their domains and use the Origin request header when determining how to set the CORS response headers. For example, my origin is:

https://mbostock.static.observableusercontent.com

Replace mbostock with the username of the person you want to grant access. That said, restricting CORS to specific domains isn’t sufficient to make it private (since anyone can still make a request from the command line with whatever Origin header they want); to make your API truly private, you also need to uses cookies, tokens or similar credentials to authenticate the request.

2 Likes
17

To be clear, technically you could enable CORS just for Observable if you want to, by looking for any *.static.observableusercontent.com Origin requests. But my recommendation is that you either make your API CORS-accessible to any domain (*), or you limit it to specific Observable authors.

1 Like
18

Thanks, It looks like it will be “accessible to any domain” for now. I will get on that tonight!

19

you might find this thread on cors useful too, to understand why things are set up the way they are for cors on observable and use my simple fetch.js notebook for testing when you enable it on your end for observable notebooks hitting your api

20

Yeah I read that. However the idea is to get the lab up and running for some students(who I do not know who they are yet) then together we will build some cool things that already exist in python and Matplotlib… so as long as it works for everybody it will work for the group.