How can I get all the results with a paginated API

Community Help
1

For @pac02/look-at-your-list-of-created-articles-with-the-xtools-page-ap,
I need to retrieve the list of articles created by a user on Wikipedia.

I use xtools pages created API (XTools/API/User - MediaWiki).

Some users have created more than 1,000 articles. In that case, the API is paginated.

I’m don’t know how to iterate until the continue parameter of the API is null.

Here is a notebook with an example Retrieve the list of created pages / PAC / Observable

2

I looked at Pinakpani for a random example with a “count”: 1566

The first 1000 pages are returned when requesting https://xtools.wmflabs.org/api/user/pages/en.wikipedia/Pinakpani

await fetch('https://xtools.wmflabs.org/api/user/pages/en.wikipedia/Pinakpani').then((d) => d.json())

image
image1049×488 40.4 KB

“continue”: “2019-04-27T17:45:04”,

Your get_prose() function could check for that.

/api/user/pages/{project}/{username}/{namespace}/{redirects}/{deleted}/{start}/{end}/{offset}

The newest pages are first. So 1000 from:
https://xtools.wmflabs.org/api/user/pages/en.wikipedia/Pinakpani
That returns the continue:date to continue from.

https://xtools.wmflabs.org/api/user/pages/en.wikipedia/Pinakpani/0/noredirects/all/2019-04-27/2000-01-01
That returns the other 566 entries. If there are more than 1000, the continue is present.

I hope that helps.

1 Like
3

thanks a lot for your answer. My problem is that I don’t know how to code this in my function. I’ve tried a recursive function but it doesn’t seem to work in Observable:

get_allpagescreated = async function (offset = "") {
  const data = await get_pagescreated("");
  if (data.continue === undefined) {
    return data;
  } else {
    return await get_allpagescreated((offset = data.continue));
  }
}

I guess that mutable would help me but I don’t really know how to use it.

4

Named functions can recurse on themselves, you don’t need to use more than one cell

for example:

function multiplyRecusiveUnsigned(a, b, sum = 0) {
  if (a == 0) return sum;
  return multiplyRecusiveUnsigned(a - 1, b, sum + b);
}

results in a recursive function called ‘multiplyRecusiveUnsigned’. You can’t do this with lambda syntax!

2 Likes
5

Instead of recursively loading, you could also do it iteratively which would allow you to start returning data immediately using Observable’s generators. I made a quick demo of how to do that. Sorry it’s not using the original API, but a fake one that I made for this demo.

3 Likes
6

Thanks a lot for all your answers.

I’ve implemented the sequential solution here Hello Created pages API / PAC / Observable

It seems to work.

2 Likes