Loading JSON data from mulitple URLS and combining

Community Help
1

Hi!

I am trying to load and combine data from mulitple URLS from the same API.

For each URL a location summary can be downloaded.

So if I want the summaries from three locations:

var datas = [“https://mwshovel.pythonanywhere.com/dirt/summary/Veveyse/?format=json”,
“https://mwshovel.pythonanywhere.com/dirt/summary/Arabie/?format=json”,
“https://mwshovel.pythonanywhere.com/dirt/summary/Anarchy-Beach/?format=json”
];

Where each URL gives a version of the following:

{ave_dense: “3.9384”, first: “2015-11”, last: “2017-01”, location: “Veveyse”, max_dense: “6.7909”, min_dense: “0.9813”, num_lakes: 1, num_locs: 1, num_rivers: 0, num_samps: 17, seven_five: “4.6791”, stan_dev: “1.5542”, total: 3436, two_five: “2.7360”}

I can get it all in an array like this:

var x_three = ;
datas.forEach(function(url) {
var x_one = d3.json(url);
var x_two = x_one.then(function(data) {x_three.push(data)});
});

Then with console.log(x_three):

​
0: Object { num_locs: 1, total: 3436, num_samps: 17, … }​
1: Object { num_locs: 1, total: 3417, num_samps: 5, … }​
2: Object { num_locs: 1, total: 1013, num_samps: 1, … }​
length: 3​
: Array

All is good up to this point, this is where I don’t get it:

If I call console.log(x_three[0]) from the script: undefined

But console.log(x_three[0]) from the browser console: result as expected

d3.nest() behaves the sameway:
in the script:

d3.nest().key(function(d) {return d.location}).entries(x_three) = undefined

In the browser console: as expected

For now the idea was just to pull the value “element.location” from each result and append to a list.

>   <div id="div-five"><ul id="list-two"></ul></div>
>           <script>
>           var datas = ["https://mwshovel.pythonanywhere.com/dirt/summary/Veveyse/?format=json",
>                        "https://mwshovel.pythonanywhere.com/dirt/summary/Arabie/?format=json",
>                        "https://mwshovel.pythonanywhere.com/dirt/summary/Anarchy-Beach/?format=json"
>                      ];
>           var x_three = [];
>           datas.forEach(function(url) {
>             var x_one = d3.json(url);
>             var x_two = x_one.then(function(data) {x_three.push(data)});
>             });
>           x_three.forEach(function(element){
>             console.log(element.location);//this does not work
>           });
>           console.log(x_three);
>           d3.select("#list-two")
>               .style("color", "blue")
>               .selectAll("li")
>               .data(x_three)
>               .enter()
>               .append("li")
>               .append("text").style("color", "red").text((d,[i]) => d[i].location);      
>          </script>

So how do I do that? Combine the data from multiple URLS and use in a D3 viz and other output for the application?

Thanks, Let me know if this would be better on StackOverflow

2

Hi!
I imagine it has something to do with promises and responses that I don’t understand. This code creates the simple list but the array of objects is still not accessible outside of the function. Even thought the array is initiated outside of the function and I used the “fetch” to itterate through the list of URLS:

  <script>
          var datas = ["https://mwshovel.pythonanywhere.com/dirt/summary/Veveyse/?format=json",
                       "https://mwshovel.pythonanywhere.com/dirt/summary/Arabie/?format=json",
                       "https://mwshovel.pythonanywhere.com/dirt/summary/Anarchy-Beach/?format=json"
                     ];
          var xData = []
          datas.forEach(function(url) {
            fetch(url)
            .then(function(response) {
              return response.json()})
            .then(function(data) {
              console.log(data.location);
              xData.push(data.location);
              d3.select("#list-three")
              .style("color", "blue")
              .selectAll("li")
              .data(xData)
              .enter()
              .append("li")
              .append("text").style("color", "red").text((d) => d);
            })
            });
          console.log(xData[0]); <----- this still does not work
          </script>
3

Hi,

I’m not fully sure of whether this question is about code in Observable or not - but regardless, here are some of the key bits:

First off, check out the Introduction to Promises notebook, that covers most of what you’re referring to here. The key understandings are:

  • To get the value of a promises, you’ll need to use await, or put the promise value in a separate cell.
  • Otherwise, promise values aren’t going to be accessible. As you’ve found, appending data in a loop just won’t work: the idea of asynchrony in JavaScript is that code doesn’t wait for asynchronous operations unless you ask it to, using .then(), await, or in the case of Observable, a cell.

In terms of resolving your specific issue, here’s an example notebook: https://beta.observablehq.com/d/09bb868a89fbdefb

data = Promise.all(
  [
    "https://mwshovel.pythonanywhere.com/dirt/summary/Veveyse/?format=json",
    "https://mwshovel.pythonanywhere.com/dirt/summary/Arabie/?format=json",
    "https://mwshovel.pythonanywhere.com/dirt/summary/Anarchy-Beach/?format=json"
  ].map(url => d3.json(url))
)

Or, in one cell:

{
  const data = await Promise.all(
    [
      "https://mwshovel.pythonanywhere.com/dirt/summary/Veveyse/?format=json",
      "https://mwshovel.pythonanywhere.com/dirt/summary/Arabie/?format=json",
      "https://mwshovel.pythonanywhere.com/dirt/summary/Anarchy-Beach/?format=json"
    ].map(url => d3.json(url))
  );
  // any code that relies on data
  return data[0].num_locs;
}

The key fixes are using Promise.all where you need to coordinate multiple asynchronous tasks, and putting async values in a separate cell, or using await to refer to them in the same cell.

4

Thanks Tom,

This the key:

Now I know where to go with it.

The series I am working on will be hosted on Observable, but tonight I was trying to work it out on my local server!

Thanks again!