Many calls to an API

Community Help
1

In the following notebook (Polls in Wikipedia in French / PAC | Observable), I want to fetch data from an API for 300 pages. Of course, I’ve got some network error/timeout issue.

I 'd like to know if there any strategy to overcome this issue. I’ve read that Workers may do the job but I don’t how to implement them in my use case.

Do you have any clue?

2

You have to ensure that you only make a certain number of requests at any given time. The simplest way to do that is to iterate over them one by one in an async/await loop:

results = {
  let aborted = false;
  invalidation.then(() => aborted = true);
  const results = [];
  for(const {title} of all_closed_polls_concat.slice(0, 10)) {
    if(aborted) break;
    results.push(await get_articleinfo({
      article: title,
      wikipedia: "fr.wikipedia.org"
    }));
  }
  return results;
}

(Note that I limited the results to 10 items for testing.)

1 Like
3

Fantastic. Thank you very much. This takes some time but it works.

4

Here is the result (in French) Coup d'œil sur les sondages de la Wikipédia francophone / PAC | Observable

1 Like