🏠 back to Observable

Help parsing JSON for Notebook of Latest COVID-19 Papers

Hi, I’m having trouble parsing the JSON output from this page (list of latest papers on COVID-19): http://connect.biorxiv.org/relate/content/181

in this notebook: https://observablehq.com/@ismms-himc/covid-19-sars-cov-2-preprints-from-medrxiv-and-biorxiv

I’m trying to adapt from the example https://observablehq.com/@jashkenas/earthquakes

Any help would be greatly appreciated!

Those don’t appear to be CORS-accessible APIs, so you’ll either need to download the JSON and attach it to your notebook as a file, or you’ll needs a CORS proxy to make them accessible. (Or you can write bioRxiv and ask them to enable CORS.)

1 Like

Thanks, that makes sense!

Hi @mbostock, Biorxiv got back to me but they wanted clarification. I’m paraphrasing below:

“Can you please help us understand what exactly the JSON feed for our COVID-19 does not do relative to what you are trying to accomplish? What kind of auto-updating are you trying to do that cannot be done by loading the JSON feed as it is? Please help us understand what you need.”

This was my response

"
Thanks for getting back to me. As far as I understand we want to have cross origin requests from the URL http://connect.biorxiv.org/relate/collection_json.php?grp=181

The page is blocking our front-end requests. Would you be able to ask for clarification on the discourse page (Help parsing JSON for Notebook of Latest COVID-19 Papers) so Mike could clarify? I’m happy to paraphrase your question. "

As far as I understand their site just needs to allow for cross origin requests but I’m not sure if I’m missing something. Thanks for your help!

You can send them the server recipes on how to enable CORS https://enable-cors.org/server.html
very important the header shall allow everywebsite

'Access-Control-Allow-Origin' '*';

ps. meanwhile you could use https://cors-anywhere.herokuapp.com/ on your notebook

url = "https://cors-anywhere.herokuapp.com/https://connect.biorxiv.org/relate/collection_json.php?grp=181"`
3 Likes

@cornhundred actually I just look your notebook and seems like the requests to to biorxiv.org is working with CORS enable, however the JSON response is coming with a weird JSON format start and ending with () parenthesis

image

So if you fetch is as a text and remove the parenthesis, then you can JSON parse the string

here is how a did

data = d3.text(url).then(text => JSON.parse(text.substring(1, text.length - 1)))
2 Likes

Thanks @radames!

I have another related question (really just a Javascript mapping question I think). So now I have an array of arrays data (one for each paper) and the first element of each sub-array contains the object of interest (information on the specific paper). However, I need to make a new array of that assembles the objects into a single array so I can visualize as a table (e.g. https://observablehq.com/@tmcw/fancy-tables).

I tried something like clean_data = data.map(x=>x[0]) but I was getting the error cannot read property '0' of null

If I try to visualize using renderTable(data[0]) I’m only able to see a single row - since I’m only giving a single array element, including data fails since it is an array of single-length arrays.

I figured it out. One of the array elements must have been null, so now I just ran data.filter and data.map. Thanks all for the help!