{"course":{"slug":"python-for-ai","title":"Python, from Zero, for AI","url":"https://addaly.com/learn/python-for-ai","level":"beginner","track":"build"},"lesson":{"slug":"pagination-and-cursors","title":"Pagination: walking a result that arrives in pages","url":"https://addaly.com/learn/python-for-ai/pagination-and-cursors","position":53,"of":89,"minutes":8,"answer":"An API never hands you all of anything; it hands you a page and a way to ask for the next.","keyIdea":"An API never hands you all of anything; it hands you a page and a way to ask for the next, and a generator that yields items and hides the page boundary is the cleanest way to consume it."},"body":{"format":"markdown","text":"## Nobody returns ten thousand rows\n\nAsk an API for \"all messages\" and you receive the first fifty and a hint about how to get the next fifty. This is pagination, and it exists because a single response carrying a million items would be slow to build, slow to send and would fall over if the connection dropped at item 999,000. Every listing endpoint on every serious API is paginated. The three styles differ in how they say \"next\".\n\n## Offset pagination\n\nThe oldest style: you ask for page 3, or for items starting at offset 200.\n\n```python\ndef all_items(session, url):\n    page = 1\n    while True:\n        r = session.get(url, params={\"page\": page, \"per_page\": 100}, timeout=30)\n        r.raise_for_status()\n        items = r.json()\n        if not items:\n            return\n        yield from items\n        page += 1\n```\n\nSimple, and it has a flaw you will meet in production. The pages are positions in a list, and the list is changing while you walk it. If someone inserts an item before your position between page 2 and page 3, the last item of page 2 becomes the first item of page 3 and you see it twice. If someone deletes one, an item slides from page 3 into page 2 after you have read page 2, and you never see it. On a busy dataset a long walk by offset is guaranteed to be slightly wrong, and nothing in the responses tells you.\n\n## Cursor pagination\n\nThe fix: instead of a page number, the server returns an opaque token that means \"the position after the last thing I gave you\".\n\n```python\ndef all_items(session, url):\n    cursor = None\n    while True:\n        params = {\"limit\": 100}\n        if cursor:\n            params[\"after\"] = cursor\n        r = session.get(url, params=params, timeout=30)\n        r.raise_for_status()\n        body = r.json()\n        yield from body[\"data\"]\n        if not body.get(\"has_more\"):\n            return\n        cursor = body[\"data\"][-1][\"id\"]\n```\n\nThat is the shape of OpenAI's list endpoints: `data`, `has_more`, and the ID of the last item as the next cursor. Anthropic's use `first_id`, `last_id` and `has_more`. Others return a `next_cursor` field, or a full `next` URL you fetch as-is. Read the documentation for the three names; the loop is otherwise identical.\n\nA cursor is a bookmark into the actual sequence, so insertions and deletions elsewhere do not shift it. The trade-off is that you cannot jump to page 40; you can only go forward. For a program that wants everything, that is no loss.\n\nTreat the cursor as opaque. Some are just IDs; some are base64 blobs encoding a timestamp and a position. Do not parse them, do not construct them, and do not store them for long: many expire.\n\n## Link headers\n\nGitHub and some others put the next URL in a response header rather than the body:\n\n```\nLink: <https://api.github.com/repos/x/y/issues?page=2>; rel=\"next\",\n      <https://api.github.com/repos/x/y/issues?page=9>; rel=\"last\"\n```\n\n`requests` parses this for you:\n\n```python\ndef all_items(session, url):\n    while url:\n        r = session.get(url, timeout=30)\n        r.raise_for_status()\n        yield from r.json()\n        url = r.links.get(\"next\", {}).get(\"url\")\n```\n\n`r.links` is a dict keyed by `rel`. When there is no `next`, the loop ends. This is the cleanest style to consume, because you never construct a URL yourself.\n\n## Why a generator\n\nEach function above uses `yield`, so the caller sees a plain sequence of items and never thinks about pages:\n\n```python\nfor item in all_items(session, url):\n    if item[\"created_at\"] < cutoff:\n        break\n    process(item)\n```\n\nBecause a generator is lazy, the `break` stops fetching. A version that built a full list first would fetch every page before returning, including all the ones after the cutoff. When a listing has 40,000 items and you want the newest 50, that is the difference between one request and four hundred.\n\nYou also get to stop for other reasons (a budget of requests, a time limit, an error in `process`) without any of that logic living inside the fetching code.\n\n## Things that go wrong\n\n**Off by one at the boundary.** If the API says \"items after cursor X\", the item X itself is not returned again. If it says \"items from X\", it is. Test the join between two pages once, by hand, and look for a duplicated or missing item there.\n\n**The total is a lie.** Some responses carry a `total` count. It was true when the server computed it. Do not allocate or loop on it; loop on `has_more`.\n\n**Rate limits scale with pages.** A walk of 400 pages is 400 requests. Use the Session from the previous lesson, and the retry-with-backoff from the one before it, or the 429 on page 180 kills the whole walk and you start again from the beginning.\n\n**Restarting from the start.** If a long walk can fail, write the cursor somewhere as you go (a file, a database row) so a restart resumes rather than repeats. For offset pagination there is no reliable resume, which is one more reason to prefer cursors when the API offers them.\n\n## Try this now\n\nGitHub's issues endpoint needs no key for a public repository. Walk `https://api.github.com/repos/python/cpython/issues?per_page=100` with the `r.links` version, count how many pages you fetched before the first 500 issues, and add a `break` at 500 to confirm the fetching stops with it."},"check":{"question":"A script fetches a list of files from an API using `page=1`, `page=2` and so on, stopping when a page comes back with fewer than 100 items. It runs for six minutes while other users are uploading and deleting files, and afterwards a few files appear twice in the output while others are missing. What is the mechanism?","options":[{"text":"The API rate-limited the script mid-run and silently returned cached pages from a previous request.","correct":false,"tempting":"A rate limit produces a 429 the script would see, not silently substituted data."},{"text":"The final page had exactly 100 items, so the stop condition never fired and the loop wrapped around to page 1.","correct":false,"tempting":"A page beyond the end returns zero items, which is fewer than 100 and stops the loop; nothing wraps around."},{"text":"JSON parsing of large pages drops list elements that straddle a network packet boundary and re-adds the previous element to compensate, which explains both the duplicates and the gaps.","correct":false,"tempting":"JSON is parsed after the whole body has been received; packet boundaries are invisible to the parser."},{"text":"Offset pages are positions in a list other users were changing, so an insertion shifts an item into the next page and a deletion shifts one out.","correct":true}]},"prev":{"slug":"timeouts-retries-and-backoff","title":"Timeouts and retries: the code that decides whether you pay twice","url":"https://addaly.com/learn/python-for-ai/timeouts-retries-and-backoff"},"next":{"slug":"sdks-versus-raw-requests","title":"A provider's SDK: what it does for you, what it hides, and how to read it","url":"https://addaly.com/learn/python-for-ai/sdks-versus-raw-requests"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Pagination: walking a result that arrives in pages', in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/pagination-and-cursors (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Pagination: walking a result that arrives in pages","courseTitle":"Python, from Zero, for AI","module":{"index":6,"slug":"talking-to-a-service","title":"Talking to a service: HTTP from Python, done properly"},"answer":null,"keyIdea":"An API never hands you all of anything; it hands you a page and a way to ask for the next, and a generator that yields items and hides the page boundary is the cleanest way to consume it.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"nobody-returns-ten-thousand-rows","runs":[{"text":"Nobody returns ten thousand rows"}]},{"t":"p","runs":[{"text":"Ask an API for “all messages” and you receive the first fifty and a hint about how to get the next fifty. This is pagination, and it exists because a single response carrying a million items would be slow to build, slow to send and would fall over if the connection dropped at item 999,000. Every listing endpoint on every serious API is paginated. The three styles differ in how they say “next”."}]},{"t":"h2","id":"offset-pagination","runs":[{"text":"Offset pagination"}]},{"t":"p","runs":[{"text":"The oldest style: you ask for page 3, or for items starting at offset 200."}]},{"t":"code","lang":"python","label":"Python","text":"def all_items(session, url):\n    page = 1\n    while True:\n        r = session.get(url, params={\"page\": page, \"per_page\": 100}, timeout=30)\n        r.raise_for_status()\n        items = r.json()\n        if not items:\n            return\n        yield from items\n        page += 1"},{"t":"p","runs":[{"text":"Simple, and it has a flaw you will meet in production. The pages are positions in a list, and the list is changing while you walk it. If someone inserts an item before your position between page 2 and page 3, the last item of page 2 becomes the first item of page 3 and you see it twice. If someone deletes one, an item slides from page 3 into page 2 after you have read page 2, and you never see it. On a busy dataset a long walk by offset is guaranteed to be slightly wrong, and nothing in the responses tells you."}]},{"t":"h2","id":"cursor-pagination","runs":[{"text":"Cursor pagination"}]},{"t":"p","runs":[{"text":"The fix: instead of a page number, the server returns an opaque token that means “the position after the last thing I gave you”."}]},{"t":"code","lang":"python","label":"Python","text":"def all_items(session, url):\n    cursor = None\n    while True:\n        params = {\"limit\": 100}\n        if cursor:\n            params[\"after\"] = cursor\n        r = session.get(url, params=params, timeout=30)\n        r.raise_for_status()\n        body = r.json()\n        yield from body[\"data\"]\n        if not body.get(\"has_more\"):\n            return\n        cursor = body[\"data\"][-1][\"id\"]"},{"t":"p","runs":[{"text":"That is the shape of OpenAI’s list endpoints: "},{"text":"data","code":true},{"text":", "},{"text":"has_more","code":true},{"text":", and the ID of the last item as the next cursor. Anthropic’s use "},{"text":"first_id","code":true},{"text":", "},{"text":"last_id","code":true},{"text":" and "},{"text":"has_more","code":true},{"text":". Others return a "},{"text":"next_cursor","code":true},{"text":" field, or a full "},{"text":"next","code":true},{"text":" URL you fetch as-is. Read the documentation for the three names; the loop is otherwise identical."}]},{"t":"p","runs":[{"text":"A cursor is a bookmark into the actual sequence, so insertions and deletions elsewhere do not shift it. The trade-off is that you cannot jump to page 40; you can only go forward. For a program that wants everything, that is no loss."}]},{"t":"p","runs":[{"text":"Treat the cursor as opaque. Some are just IDs; some are base64 blobs encoding a timestamp and a position. Do not parse them, do not construct them, and do not store them for long: many expire."}]},{"t":"h2","id":"link-headers","runs":[{"text":"Link headers"}]},{"t":"p","runs":[{"text":"GitHub and some others put the next URL in a response header rather than the body:"}]},{"t":"code","lang":"","label":"Text","text":"Link: <https://api.github.com/repos/x/y/issues?page=2>; rel=\"next\",\n      <https://api.github.com/repos/x/y/issues?page=9>; rel=\"last\""},{"t":"p","runs":[{"text":"requests","code":true},{"text":" parses this for you:"}]},{"t":"code","lang":"python","label":"Python","text":"def all_items(session, url):\n    while url:\n        r = session.get(url, timeout=30)\n        r.raise_for_status()\n        yield from r.json()\n        url = r.links.get(\"next\", {}).get(\"url\")"},{"t":"p","runs":[{"text":"r.links","code":true},{"text":" is a dict keyed by "},{"text":"rel","code":true},{"text":". When there is no "},{"text":"next","code":true},{"text":", the loop ends. This is the cleanest style to consume, because you never construct a URL yourself."}]},{"t":"h2","id":"why-a-generator","runs":[{"text":"Why a generator"}]},{"t":"p","runs":[{"text":"Each function above uses "},{"text":"yield","code":true},{"text":", so the caller sees a plain sequence of items and never thinks about pages:"}]},{"t":"code","lang":"python","label":"Python","text":"for item in all_items(session, url):\n    if item[\"created_at\"] < cutoff:\n        break\n    process(item)"},{"t":"p","runs":[{"text":"Because a generator is lazy, the "},{"text":"break","code":true},{"text":" stops fetching. A version that built a full list first would fetch every page before returning, including all the ones after the cutoff. When a listing has 40,000 items and you want the newest 50, that is the difference between one request and four hundred."}]},{"t":"p","runs":[{"text":"You also get to stop for other reasons (a budget of requests, a time limit, an error in "},{"text":"process","code":true},{"text":") without any of that logic living inside the fetching code."}]},{"t":"h2","id":"things-that-go-wrong","runs":[{"text":"Things that go wrong"}]},{"t":"p","runs":[{"text":"Off by one at the boundary.","b":true},{"text":" If the API says “items after cursor X”, the item X itself is not returned again. If it says “items from X”, it is. Test the join between two pages once, by hand, and look for a duplicated or missing item there."}]},{"t":"p","runs":[{"text":"The total is a lie.","b":true},{"text":" Some responses carry a "},{"text":"total","code":true},{"text":" count. It was true when the server computed it. Do not allocate or loop on it; loop on "},{"text":"has_more","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Rate limits scale with pages.","b":true},{"text":" A walk of 400 pages is 400 requests. Use the Session from the previous lesson, and the retry-with-backoff from the one before it, or the 429 on page 180 kills the whole walk and you start again from the beginning."}]},{"t":"p","runs":[{"text":"Restarting from the start.","b":true},{"text":" If a long walk can fail, write the cursor somewhere as you go (a file, a database row) so a restart resumes rather than repeats. For offset pagination there is no reliable resume, which is one more reason to prefer cursors when the API offers them."}]},{"t":"h2","id":"try-this-now","runs":[{"text":"Try this now"}],"role":"try"},{"t":"p","runs":[{"text":"GitHub’s issues endpoint needs no key for a public repository. Walk "},{"text":"https://api.github.com/repos/python/cpython/issues?per_page=100","code":true},{"text":" with the "},{"text":"r.links","code":true},{"text":" version, count how many pages you fetched before the first 500 issues, and add a "},{"text":"break","code":true},{"text":" at 500 to confirm the fetching stops with it."}]}],"check":{"question":[{"text":"A script fetches a list of files from an API using "},{"text":"page=1","code":true},{"text":", "},{"text":"page=2","code":true},{"text":" and so on, stopping when a page comes back with fewer than 100 items. It runs for six minutes while other users are uploading and deleting files, and afterwards a few files appear twice in the output while others are missing. What is the mechanism?"}],"options":[{"text":[{"text":"The API rate-limited the script mid-run and silently returned cached pages from a previous request."}],"correct":false,"tempting":[{"text":"A rate limit produces a 429 the script would see, not silently substituted data."}]},{"text":[{"text":"The final page had exactly 100 items, so the stop condition never fired and the loop wrapped around to page 1."}],"correct":false,"tempting":[{"text":"A page beyond the end returns zero items, which is fewer than 100 and stops the loop; nothing wraps around."}]},{"text":[{"text":"JSON parsing of large pages drops list elements that straddle a network packet boundary and re-adds the previous element to compensate, which explains both the duplicates and the gaps."}],"correct":false,"tempting":[{"text":"JSON is parsed after the whole body has been received; packet boundaries are invisible to the parser."}]},{"text":[{"text":"Offset pages are positions in a list other users were changing, so an insertion shifts an item into the next page and a deletion shifts one out."}],"correct":true,"tempting":null}],"order":[0,2,3,1],"why":null},"prev":{"slug":"timeouts-retries-and-backoff","title":"Timeouts and retries: the code that decides whether you pay twice","minutes":9},"next":{"slug":"sdks-versus-raw-requests","title":"A provider’s SDK: what it does for you, what it hides, and how to read it","minutes":8}}}