Help parsing JSON for Notebook of Latest COVID-19 Papers

Community Help
1

Hi, I’m having trouble parsing the JSON output from this page (list of latest papers on COVID-19): http://connect.biorxiv.org/relate/content/181

in this notebook: https://observablehq.com/@ismms-himc/covid-19-sars-cov-2-preprints-from-medrxiv-and-biorxiv

I’m trying to adapt from the example https://observablehq.com/@jashkenas/earthquakes

Any help would be greatly appreciated!

2

Those don’t appear to be CORS-accessible APIs, so you’ll either need to download the JSON and attach it to your notebook as a file, or you’ll needs a CORS proxy to make them accessible. (Or you can write bioRxiv and ask them to enable CORS.)

1 Like
3

Thanks, that makes sense!

4

Hi @mbostock, Biorxiv got back to me but they wanted clarification. I’m paraphrasing below:

“Can you please help us understand what exactly the JSON feed for our COVID-19 does not do relative to what you are trying to accomplish? What kind of auto-updating are you trying to do that cannot be done by loading the JSON feed as it is? Please help us understand what you need.”

This was my response

"
Thanks for getting back to me. As far as I understand we want to have cross origin requests from the URL http://connect.biorxiv.org/relate/collection_json.php?grp=181

The page is blocking our front-end requests. Would you be able to ask for clarification on the discourse page (Help parsing JSON for Notebook of Latest COVID-19 Papers) so Mike could clarify? I’m happy to paraphrase your question. "

As far as I understand their site just needs to allow for cross origin requests but I’m not sure if I’m missing something. Thanks for your help!

5

You can send them the server recipes on how to enable CORS https://enable-cors.org/server.html
very important the header shall allow everywebsite

'Access-Control-Allow-Origin' '*';

ps. meanwhile you could use https://cors-anywhere.herokuapp.com/ on your notebook

url = "https://cors-anywhere.herokuapp.com/https://connect.biorxiv.org/relate/collection_json.php?grp=181"`
3 Likes
6

@cornhundred actually I just look your notebook and seems like the requests to to biorxiv.org is working with CORS enable, however the JSON response is coming with a weird JSON format start and ending with () parenthesis

image

So if you fetch is as a text and remove the parenthesis, then you can JSON parse the string

here is how a did

data = d3.text(url).then(text => JSON.parse(text.substring(1, text.length - 1)))
2 Likes
7

Thanks @radames!

8

I have another related question (really just a Javascript mapping question I think). So now I have an array of arrays data (one for each paper) and the first element of each sub-array contains the object of interest (information on the specific paper). However, I need to make a new array of that assembles the objects into a single array so I can visualize as a table (e.g. https://observablehq.com/@tmcw/fancy-tables).

I tried something like clean_data = data.map(x=>x[0]) but I was getting the error cannot read property '0' of null

If I try to visualize using renderTable(data[0]) I’m only able to see a single row - since I’m only giving a single array element, including data fails since it is an array of single-length arrays.

9

I figured it out. One of the array elements must have been null, so now I just ran data.filter and data.map. Thanks all for the help!