Pagination: walking a result that arrives in pages · Python, From Zero, For AI

Nobody returns ten thousand rows

Ask an API for "all messages" and you receive the first fifty and a hint about how to get the next fifty. This is pagination, and it exists because a single response carrying a million items would be slow to build, slow to send and would fall over if the connection dropped at item 999,000. Every listing endpoint on every serious API is paginated. The three styles differ in how they say "next".

Offset pagination

The oldest style: you ask for page 3, or for items starting at offset 200.

python

def all_items(session, url):
    page = 1
    while True:
        r = session.get(url, params={"page": page, "per_page": 100}, timeout=30)
        r.raise_for_status()
        items = r.json()
        if not items:
            return
        yield from items
        page += 1

Simple, and it has a flaw you will meet in production. The pages are positions in a list, and the list is changing while you walk it. If someone inserts an item before your position between page 2 and page 3, the last item of page 2 becomes the first item of page 3 and you see it twice. If someone deletes one, an item slides from page 3 into page 2 after you have read page 2, and you never see it. On a busy dataset a long walk by offset is guaranteed to be slightly wrong, and nothing in the responses tells you.

Cursor pagination

The fix: instead of a page number, the server returns an opaque token that means "the position after the last thing I gave you".

python

def all_items(session, url):
    cursor = None
    while True:
        params = {"limit": 100}
        if cursor:
            params["after"] = cursor
        r = session.get(url, params=params, timeout=30)
        r.raise_for_status()
        body = r.json()
        yield from body["data"]
        if not body.get("has_more"):
            return
        cursor = body["data"][-1]["id"]

That is the shape of OpenAI's list endpoints: data, has_more, and the ID of the last item as the next cursor. Anthropic's use first_id, last_id and has_more. Others return a next_cursor field, or a full next URL you fetch as-is. Read the documentation for the three names; the loop is otherwise identical.

A cursor is a bookmark into the actual sequence, so insertions and deletions elsewhere do not shift it. The trade-off is that you cannot jump to page 40; you can only go forward. For a program that wants everything, that is no loss.

Treat the cursor as opaque. Some are just IDs; some are base64 blobs encoding a timestamp and a position. Do not parse them, do not construct them, and do not store them for long — many expire.

Link headers

GitHub and some others put the next URL in a response header rather than the body:

Link: <https://api.github.com/repos/x/y/issues?page=2>; rel="next",
      <https://api.github.com/repos/x/y/issues?page=9>; rel="last"

requests parses this for you:

python

while url:
    r = session.get(url, timeout=30)
    r.raise_for_status()
    yield from r.json()
    url = r.links.get("next", {}).get("url")

r.links is a dict keyed by rel. When there is no next, the loop ends. This is the cleanest style to consume, because you never construct a URL yourself.

Why a generator

Each function above uses yield, so the caller sees a plain sequence of items and never thinks about pages:

python

for item in all_items(session, url):
    if item["created_at"] < cutoff:
        break
    process(item)

Because a generator is lazy, the break stops fetching. A version that built a full list first would fetch every page before returning, including all the ones after the cutoff. When a listing has 40,000 items and you want the newest 50, that is the difference between one request and four hundred.

You also get to stop for other reasons — a budget of requests, a time limit, an error in process — without any of that logic living inside the fetching code.

Things that go wrong

Off by one at the boundary. If the API says "items after cursor X", the item X itself is not returned again. If it says "items from X", it is. Test the join between two pages once, by hand, and look for a duplicated or missing item there.

The total is a lie. Some responses carry a total count. It was true when the server computed it. Do not allocate or loop on it; loop on has_more.

Rate limits scale with pages. A walk of 400 pages is 400 requests. Use the Session from the previous lesson, and the retry-with-backoff from the one before it, or the 429 on page 180 kills the whole walk and you start again from the beginning.

Restarting from the start. If a long walk can fail, write the cursor somewhere as you go — a file, a database row — so a restart resumes rather than repeats. For offset pagination there is no reliable resume, which is one more reason to prefer cursors when the API offers them.

Try this now

GitHub's issues endpoint needs no key for a public repository. Walk https://api.github.com/repos/python/cpython/issues?per_page=100 with the r.links version, count how many pages you fetched before the first 500 issues, and add a break at 500 to confirm the fetching stops with it.