Overcoming difficulties with JSON data loading (or another mistake)

Community Help
#6

You don’t need d3.json to parse inline JSON. You can just use an object literal, as I showed above:

data = ({
  "name": "flare",
  "children": [
    …
  ]
})

d3.json is for fetching and parsing JSON. You can parse a JSON string directly using JSON.parse. Or you can use an object literal (as shown above) and you don’t need parsing or fetching at all—it’s just an object you’re directly defining in code.

1 Like
#7

Thanks for the immediate reply – though I must admit that I don’t fully understand it.

I read in my data as:

jsondata = ({
  "name": "flare",
  "children": [
    …
  ]
})

Where I am struggling is how to reference these data in the chart. I tried the promises method:

d3.json("jsondata").then(function(root) {

But the chart still fails to show up. My reading of your previous discussion was that the d3.json module was inappropriate in this context, as it’s still looking for a for a file named “jsondata” relative to the notebook, which does not exist.

Is it correct that, to solve this, I need to figure out how a callback that references the in-line object literal? Is this what you mean in stating (?):

…in Observable, if the value of a cell is a promise, then referencing that value from another cell will implicitly await the promise, so the best approach is to say:

data = d3.json("jsondata")

… I’ve played around a bit in trying to do this, but haven’t yet succeeded.

I hope these questions are appropriate. I feel like I should be reading and studying more before asking. :confused: I’ll take a few steps back and work again through intro to JavaScript tutorials before going much further. I’ve been reading and working through your tutorials on Observable, which are helpful, but I suspect my challenges are more rudimentary.

#8

You literally say jsondata. :smile: You don’t need anything else. And that’s true regardless of whether you define the jsondata inline or fetched from a remote server as a promise, as with d3.json.

The idea is that a cell is responsible for defining its value, and other cells can reference that value without worrying about how it’s defined. This decoupling, where referencing cells care only about values and not implementations, is the purpose of Observable’s dataflow (reactive) runtime.

So for example, if you say:

data = ({name: "Fred"})

In another cell, you can say:

md`Hello, ${data.name}!`

And it will say “Hello, Fred!”

Likewise, if you change the data cell to be defined as:

data = d3.json("https://gist.githubusercontent.com/mbostock/1093025/raw/b40b9fc5b53b40836ead8aa4b4a17d948b491126/flare.json")

Then the “Hello, Fred!” will automatically change to say “Hello, flare!”

#9

Thanks again for all the help and pointers. I’ve re-marked your original answer has having solved my question on data loading. I still haven’t succeeded in getting the chart to render. :frowning:

Specifically, when replacing

d3.json("jsondata").then(function(root) {

with

const data = jsondata;

I start down another road of challenges for which my current JavaScript prowess is too limited. In this case, I create a runtime error because root isn’t defined. I’ll keep at it.

I took your advice and tried again at breaking down the example into component pieces:

I hit another snag with the way that margins were defined in the example. Seems like I’ll have several more challenges before I figure this one out :wink:

I really appreciate all the support!

With sincere thanks and kind regards,

Aaron

#10

The margin syntax error is being you are missing parentheses. An Observable cell looks like a normal assignment statement in JavaScript, but it isn’t. In normal JavaScript, you can say:

const margin = {top: 20, right: 30, bottom: 30, left: 40};

But in an Observable cell, you need parentheses:

margin = ({top: 20, right: 30, bottom: 30, left: 40})

That’s because an Observable cell is more akin to the body of an arrow function: it can be either an expression or a block statement. If you were writing an arrow function, and you said:

const margin = () => {top: 20, right: 30, bottom: 30, left: 40};

You’d get a syntax error. But if you say:

const margin = () => ({top: 20, right: 30, bottom: 30, left: 40});

Then the margin function returns the expected object. Observable cells are the same way—it’s just that the Observable runtime calls your function (your cell definition) automatically.

As for the other error, I think you need to replace:

d3.json("jsondata").then(function(root) {
  … code that references root goes here …
});

With

const root = jsondata;

… code that references root goes here …

That, or just rename your jsondata cell to root, and then you don’t need a local variable, and you can just reference the cell value directly.

Take a look at my sunburst notebook for a related example:

#11

Also, thanks to your motivation!, I have now posted my own Observable port of my zoomable / bilevel sunburst diagram:

3 Likes
#12

Awesome! Thank you so much! I haven’t had much time since yesterday, but made some progress. Stuck now with on the partition function, so I’m excited to see how you solved it.

I really can’t express how thankful I am for your time and encouragement. I really appreciate all that you do!

2 Likes
#13

Hello Sir,

Please post it in plain JavaScript format too. I am really facing issues in implementing it with PHP and javascript. I have been struggling since last 2 weeks and I am unable to give a similar output.
I have tried your latest code for d3 zoomable sunburst bilevel partition with texts, but somehow, I am unable to implement it. I think the code which you have posted is in Node.js. Can you please show a similar implementation via plain JS please?
Thanks
Atish Agrawal

#14

Hi Atish,

Can you share where you’re starting from? Mike’s original version of this chart is in JavaScript running D3.js. My notebook didn’t quite accomplish a self-challenge of re-creating the notebook to work on this platform. Mike, however, demonstrated an elegant solution for the purpose of exploring further JavaScript customization. Unless I am mistaken, this involves no PHP. Node.js is a runtime on which Observable runs.

I don’t know that this helps you, If you can share what you’re working on it would be easier to gain insight into the challenges you’re facing.

Hope you figure it out!

Aaron

#15

Hi, This thread sent me in a few directions. Reference CORS and using API JSON

Background-situation:

We have an api it serves data in JSON format, we are using Django RESTFramework. The ideas is to use the data (which changes every week or so) to create visualisations on Observable.
We have created a section of the website for that “hammerdirt labs Python/JS”.

Here is a search for sampling sites in one city (it is only four object):

http://mwshovel.pythonanywhere.com/dirt/beaches/Vevey/?format=json

I cannot access that in Observable notebook. This is a CORS issue with Django REST api (the extension is not installed yet).

Question: What domain needs to be white listed in the CORS setting to allow anybody using Observable to access data and create visualisations?

Another way to put that question: IF I want anybody who is using an Observable notebook from observablehq.com how what is the domain to specify to the CORS setting?

Thanks

1 Like
#16

Do you consider your API public or private?

It looks public, in the sense that I can click on that link that load it in my browser. So if you want your data to be publicly-accessible (including from Observable), then you need two things:

  1. You need to host it on HTTPS. Observable is hosted on HTTPS, and the browser only allows you to fetch from other HTTPS sites; it doesn’t allow you to fetch from HTTP sites. (There are workarounds for this, such as JSONP, but I don’t recommend them.)

  2. You need to enable CORS. Specifically, you need to add Access-Control-Allow-Origin: * in the response headers so that other domains can fetch from your site and read the response.

If you want your data to be accessible only to certain authors on Observable, then you can whitelist just their domains and use the Origin request header when determining how to set the CORS response headers. For example, my origin is:

https://mbostock.static.observableusercontent.com

Replace mbostock with the username of the person you want to grant access. That said, restricting CORS to specific domains isn’t sufficient to make it private (since anyone can still make a request from the command line with whatever Origin header they want); to make your API truly private, you also need to uses cookies, tokens or similar credentials to authenticate the request.

2 Likes
#17

To be clear, technically you could enable CORS just for Observable if you want to, by looking for any *.static.observableusercontent.com Origin requests. But my recommendation is that you either make your API CORS-accessible to any domain (*), or you limit it to specific Observable authors.

1 Like
#18

Thanks, It looks like it will be “accessible to any domain” for now. I will get on that tonight!

#19

you might find this thread on cors useful too, to understand why things are set up the way they are for cors on observable and use my simple fetch.js notebook for testing when you enable it on your end for observable notebooks hitting your api

#20

Yeah I read that. However the idea is to get the lab up and running for some students(who I do not know who they are yet) then together we will build some cool things that already exist in python and Matplotlib… so as long as it works for everybody it will work for the group.

#21

observable sends origin and referer request headers. just use that for restricting your data end points to observable notebooks via access control allow origin response header without opening up wide for the world with * (not recommended unless your data enpoints are truly public for any web client to consume, and those are best done with user/app keys to secure and control that data flow :slight_smile: )

1 Like
#22

No the last question but there is a format difference in the examples:

For our API it the JSON request looks like this:
https://mwshovel.pythonanywhere.com/dirt/beaches/Vevey/?format=json

and for the examples it looks like this:
https://mwshovel.pythonanywhere.com/dirt/beaches/Vevey.json

The first I associate with a query type and the second a file request is that going to make a difference when I do :

data_x = d3.json(“https://mwshovel.pythonanywhere.com/dirt/beaches/Vevey/?format=json”)

#23

Ok this is settled for the case “CORS Allow all” I published the notebook with the resulting query. Very cool !

Now for the case of just observable and our domain. I do not understand your reply. In django I can:

CORS_ORIGIN_REGEX_WHITELIST

A list of regexes that match origin regex list of origin hostnames that are authorized to make cross-site HTTP requests. Defaults to [] . Useful when CORS_ORIGIN_WHITELIST is impractical, such as when you have a large number of subdomains.

Example:

CORS_ORIGIN_REGEX_WHITELIST = (r’^(https?://)?(\w+.)?google.com$’, )

This seems like it may be appropriate.

#24

Probably something like this:

CORS_ORIGIN_REGEX_WHITELIST = (r’^https://(\w+.)static.observableusercontent.com$’, )
1 Like
#25

that or you can just pull Referer and origin request headers, check for .observableusercontent.com domain and repost them in allow access control origin response header or issue 401 or 403 response instead for web clients from other sites.

not sure how it’s done in django, but I bet similar to express.js and others you have programmatic access to request/response pipeline and headers.

I would still recommend user/app key route for public data api’s since those request headers can be easily spoofed too.

1 Like