Hi, I’m having trouble parsing the JSON output from this page (list of latest papers on COVID-19): http://connect.biorxiv.org/relate/content/181
in this notebook: https://observablehq.com/@ismms-himc/covid-19-sars-cov-2-preprints-from-medrxiv-and-biorxiv
I’m trying to adapt from the example https://observablehq.com/@jashkenas/earthquakes
Any help would be greatly appreciated!
Those don’t appear to be CORS-accessible APIs, so you’ll either need to download the JSON and attach it to your notebook as a file, or you’ll needs a CORS proxy to make them accessible. (Or you can write bioRxiv and ask them to enable CORS.)
Thanks, that makes sense!
Hi @mbostock, Biorxiv got back to me but they wanted clarification. I’m paraphrasing below:
“Can you please help us understand what exactly the JSON feed for our COVID-19 does not do relative to what you are trying to accomplish? What kind of auto-updating are you trying to do that cannot be done by loading the JSON feed as it is? Please help us understand what you need.”
This was my response
Thanks for getting back to me. As far as I understand we want to have cross origin requests from the URL http://connect.biorxiv.org/relate/collection_json.php?grp=181
The page is blocking our front-end requests. Would you be able to ask for clarification on the discourse page (Help parsing JSON for Notebook of Latest COVID-19 Papers) so Mike could clarify? I’m happy to paraphrase your question. "
As far as I understand their site just needs to allow for cross origin requests but I’m not sure if I’m missing something. Thanks for your help!
You can send them the server recipes on how to enable CORS https://enable-cors.org/server.html
very important the header shall allow everywebsite
ps. meanwhile you could use https://cors-anywhere.herokuapp.com/ on your notebook
url = "https://cors-anywhere.herokuapp.com/https://connect.biorxiv.org/relate/collection_json.php?grp=181"`
@cornhundred actually I just look your notebook and seems like the requests to to biorxiv.org is working with CORS enable, however the JSON response is coming with a weird JSON format start and ending with () parenthesis
So if you fetch is as a text and remove the parenthesis, then you can JSON parse the string
here is how a did
data = d3.text(url).then(text => JSON.parse(text.substring(1, text.length - 1)))
data (one for each paper) and the first element of each sub-array contains the object of interest (information on the specific paper). However, I need to make a new array of that assembles the objects into a single array so I can visualize as a table (e.g. https://observablehq.com/@tmcw/fancy-tables).
I tried something like
clean_data = data.map(x=>x) but I was getting the error
cannot read property '0' of null
If I try to visualize using
renderTable(data) I’m only able to see a single row - since I’m only giving a single array element, including
data fails since it is an array of single-length arrays.
I figured it out. One of the array elements must have been null, so now I just ran data.filter and data.map. Thanks all for the help!