Non-reactive fetch issue: Notebook needs to be manually updated to display fetched results

Community Help
1

I have a few fetch requests being made via d3.json as following:

getSummaries = function( ){

  const queries = [ url1, url2, url3 ]

  Promise.all( queries.map( query => {
      const result = d3.json( query, {
        method: "GET",
        headers: {"content-type": "application/json"},
        mode: "cors",
      }).then( response => {
        const tldrText = response.tldr.text
        const returnedPaperId = response.paperId
        tldrs.push( { paperId:returnedPaperId, tldr: tldrText } )
      } )
    }) 
  )
  return tldrs
}

When I call this function from a different cell and assign it to a variable, this results in an empty array:
summaries = getSummaries()
// summaries = :arrow_forward:Array()[0]

However, if I expand the array by clicking on it (e.g., on the triangle), it suddenly gets populated with three items.

The same behaviour is also the case when I call Inputs.table(summaries). It starts with no results, but if the cell is re-run, it is populated with three items.

So, although I can manually fix the issue, what I would rather like is these values to be loaded when the notebook initializes.

I suspect the issue has to do with my poor knowledge of promises and not handling them correctly in the map loop. Indeed, the problem does not occur when I make the queries one by one.

I isolated the behavior by reproducing a minimal version of the code in this notebook.

I made quite a few searches on this form and on StackOverflow to no avail, and I would appreciate any advice you can provide. Thank you very much in advance!

2

Your function must return a promise:

getSummaries = function( ){
  
  const queries = [
    'https://api.semanticscholar.org/graph/v1/paper/2519ed73f8084b993664e5a0c240e2dd37ba7349?fields=tldr',
    'https://api.semanticscholar.org/graph/v1/paper/63da33e250e57a09dfe21545782d2ec6249bd62f?fields=tldr',
    'https://api.semanticscholar.org/graph/v1/paper/a7862e14b4c20cefd6dc4f611f8aa866fabf130b?fields=tldr'
  ]
  
  return Promise.all( queries.map( query => d3.json(query).then( response => ({
        paperId: response.paperId,
        tldr: response.tldr.text
      }))
  ));
}

What you see in your notebook is:

  1. getSummaries() returns a reference to an empty array
  2. The cell where you view the array’s content only runs once, when the array is still empty. Observable’s Inspector lazily creates an HTML representation of the top level of this empty array.
  3. Internally the function populates the array, but doesn’t inform the Observable Runtime about the changes.
  4. When you expand the array, the Inspector looks into the array (to which it still has a reference) to fetch the next level of properties and render them. But the array has changed in the meantime.

When a cell returns a promise, Observable’s Runtime won’t run dependent cells until the promise has resolved. The value that is returned by the promise gets passed on to other cells.

3

I’d also consider dropping the function and doing it like so:

summaries = {
  
  const queries = [
    'https://api.semanticscholar.org/graph/v1/paper/2519ed73f8084b993664e5a0c240e2dd37ba7349?fields=tldr',
    'https://api.semanticscholar.org/graph/v1/paper/63da33e250e57a09dfe21545782d2ec6249bd62f?fields=tldr',
    'https://api.semanticscholar.org/graph/v1/paper/a7862e14b4c20cefd6dc4f611f8aa866fabf130b?fields=tldr'
  ]
  
  return Promise.all( queries.map( query => d3.json(query).then( response => ({
        paperId: response.paperId,
        tldr: response.tldr.text
      }))
  ));
}
4

It works like a charm!
Indeed, not much use in making a promise if you’re not going to return it :slight_smile:
Thanks a million for the fix, the more elegant way of fetching, and for the explanation of the logic.
Much appreciated!

1 Like
5

Thank you Mike! The notebook I linked is a simplified version of the original code. In the original, there are some arguments being passed and a wish to push the complexity to the Appendix, hence the function. Good to know your preference on these sorts of cases, though!