{"course":{"slug":"python-for-ai","title":"Python, from Zero, for AI","url":"https://addaly.com/learn/python-for-ai","level":"beginner","track":"build"},"lesson":{"slug":"the-iterator-protocol","title":"The iterator protocol: what for actually does, and why a generator is empty the second time","url":"https://addaly.com/learn/python-for-ai/the-iterator-protocol","position":64,"of":89,"minutes":9,"answer":"for calls iter() once and next() repeatedly until StopIteration; a list gives a fresh iterator each time but a generator is its own iterator and is used up after one pass.","keyIdea":"for calls iter() once and next() repeatedly until StopIteration; a list gives a fresh iterator each time but a generator is its own iterator and is used up after one pass, which is why iterating it twice yields nothing the second time."},"body":{"format":"markdown","text":"## Two calls under every loop\n\n```python\nfor item in things:\n    ...\n```\n\nis shorthand for:\n\n```python\nit = iter(things)\nwhile True:\n    try:\n        item = next(it)\n    except StopIteration:\n        break\n    ...\n```\n\n`iter(things)` calls `things.__iter__()` and gets back an **iterator**: an object with a `__next__` method. `next(it)` calls it. When there is nothing left, `__next__` raises `StopIteration` and the loop ends. That is the entire protocol, and every `for`, every comprehension, every `sum`, `list`, `max`, `zip` and `enumerate` runs on it.\n\nTwo roles are in play. An **iterable** is anything `iter()` accepts: a list, a string, a dict, a file, your `Conversation` with its `__iter__`. An **iterator** is the thing that hands out items one at a time and remembers where it is. The distinction seems academic until it bites.\n\n## Lists give you a fresh iterator; generators are the iterator\n\n```python\nnums = [1, 2, 3]\nprint(sum(nums), sum(nums))       # 6 6\n\ngen = (n for n in [1, 2, 3])\nprint(sum(gen), sum(gen))         # 6 0\n```\n\nEach `iter(nums)` returns a **new** iterator starting at position 0, so a list can be walked any number of times. A generator object is its own iterator: `iter(gen)` returns `gen` itself, and it remembers that it already reached the end. The second `sum` asks for `next`, gets `StopIteration` immediately, and returns 0.\n\n> Figure: What a for loop actually does. Steps: 1. for x in things (Two function calls inside a hidden while); 2. iter(things), once (Calls __iter__ and gets an iterator); 3. next(it), each pass (Runs to the next yield and binds the value to x); 4. StopIteration (Caught by for itself, which is why you never see it); 5. A second loop? (A list hands out a fresh iterator; a generator is used up). Because a generator is its own iterator, the second loop over it starts where the first one stopped, which is at the end. There is no error and no warning; the loop body simply never runs. If you need the data twice, materialise it once with list(), or write a function you can call again.\n\nNo error, no warning. The second pass is just empty. This is the single most common surprise for people who have started using generators, and you have been using them since module 5's streaming and module 6's pagination. The pattern that triggers it: count something, then process it, over the same generator. Or pass a generator to a function that iterates it twice internally, which `zip` does not but a few library functions do.\n\nThe fix is to decide what you want. If you need to walk the data twice, materialise it once: `lines = list(gen)`. If the data is too large for that (the reason you used a generator) restructure so that one pass does both jobs, or create the generator twice by calling the function that makes it twice.\n\n## Writing an iterator by hand\n\nYou rarely need to, because `yield` does it, but seeing one makes the protocol concrete:\n\n```python\nclass Countdown:\n    def __init__(self, n):\n        self.n = n\n    def __iter__(self):\n        return self\n    def __next__(self):\n        if self.n <= 0:\n            raise StopIteration\n        self.n -= 1\n        return self.n + 1\n```\n\n`__iter__` returning `self` is what makes it an iterator rather than merely iterable, and is what makes it single-use, for the same reason a generator is. A generator function is this class written as a function:\n\n```python\ndef countdown(n):\n    while n > 0:\n        yield n\n        n -= 1\n```\n\nEach `next()` runs the function up to the next `yield`, hands out the value, and freezes the frame (local variables, position, everything) until the following `next()`. When the function returns, `StopIteration` is raised for you. Laziness comes for free: nothing after the current `yield` has run yet.\n\n## `next()` with a default, and peeking\n\n```python\nfirst = next(iter(things), None)\n```\n\ngives you the first item or `None` without a loop and without `IndexError`. It works on any iterable, including a generator of unknown length. To look at the first few of a large stream without consuming all of it:\n\n```python\nfrom itertools import islice\nhead = list(islice(gen, 5))\n```\n\n`islice` takes the first five and stops; the rest of `gen` is still there for a later loop. That is one of the few ways to sample a generator without materialising it, and it is what the \"peek at the first rows\" idiom in module 5 was doing.\n\n## `itertools` pieces you will use\n\n- `chain(a, b)`: iterate `a` then `b` as one sequence, without building a combined list.\n- `batched(it, 20)` (3.12+): groups of twenty, the natural shape for sending twenty prompts per request or per thread. Before 3.12, write a small `yield` loop that fills a list and yields it when full.\n- `groupby(sorted_items, key=...)`: consecutive runs with the same key. It only groups **adjacent** items, so sort first or it silently produces many small groups.\n- `count()`: infinite integers, for numbering a stream of unknown length alongside `zip`.\n\nEvery one of these is lazy. A pipeline of generators (read lines, strip them, filter, batch, send) processes one item at a time end to end, and its memory use does not grow with the file.\n\n## `zip` and unequal lengths\n\n`zip(a, b)` stops at the shorter. If `a` has 100 items and `b` has 99 because of an off-by-one upstream, `zip` drops the last silently. `zip(a, b, strict=True)` (3.10+) raises instead. Use it whenever the two sequences are supposed to match, such as prompts and their results.\n\n## Where this leaves you\n\nYou now know what `for` does, why a file can be looped once, why `iter_lines` in module 6 could be handed to any function that takes an iterable, and why `results = list(gen)` appears at the end of so many pipelines: it is the moment the data stops being lazy and becomes something you can walk twice.\n\n## Try this now\n\nWrite a generator that yields chunks of 100 items from any iterable. Feed it a `range(1050)`, check that the last chunk has 50, then consume it with `sum(len(c) for c in chunks)` and try to loop over `chunks` again."},"check":{"question":"A function returns `(line.strip() for line in open(path))`. The caller does `n = sum(1 for _ in lines)` to count them and then `for line in lines: process(line)`, and nothing is processed. What is the mechanism?","options":[{"text":"`sum` closes the underlying file when it finishes iterating, so the second loop finds a closed handle and stops immediately without raising, because closed files iterate as empty.","correct":false,"tempting":"The file stays open; the generator object is what was exhausted, and it would be just as empty over a list of strings."},{"text":"Generator expressions can be consumed only by `for`, and `sum` consumed it in an invalid way that left it broken.","correct":false,"tempting":"`sum` iterates exactly as `for` does, calling next() until StopIteration; nothing about it is invalid."},{"text":"The generator is its own iterator, so the count ran it to StopIteration and the second loop asked an exhausted iterator for more.","correct":true},{"text":"`strip()` returns a new string each time, and the second pass compares stripped strings to the originals and finds no matches.","correct":false,"tempting":"No comparison is taking place; the loop body never runs because there is nothing left to yield."}]},"prev":{"slug":"dunder-methods","title":"Dunder methods: making your own objects work with len, for, in and ==","url":"https://addaly.com/learn/python-for-ai/dunder-methods"},"next":{"slug":"context-managers","title":"Context managers: the cleanup that runs even when the block raises","url":"https://addaly.com/learn/python-for-ai/context-managers"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The iterator protocol: what for actually does, and why a generator is empty the second time', in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/the-iterator-protocol (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The iterator protocol: what for actually does, and why a generator is empty the second time","courseTitle":"Python, from Zero, for AI","module":{"index":7,"slug":"objects-and-the-shape-of-programs","title":"Objects, generators and the shape a larger program takes"},"answer":null,"keyIdea":"for calls iter() once and next() repeatedly until StopIteration; a list gives a fresh iterator each time but a generator is its own iterator and is used up after one pass, which is why iterating it twice yields nothing the second time.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"caption":"Because a generator is its own iterator, the second loop over it starts where the first one stopped, which is at the end. There is no error and no warning; the loop body simply never runs. If you need the data twice, materialise it once with list(), or write a function you can call again.","steps":[{"label":"for x in things","detail":"Two function calls inside a hidden while"},{"label":"iter(things), once","detail":"Calls __iter__ and gets an iterator"},{"label":"next(it), each pass","detail":"Runs to the next yield and binds the value to x"},{"label":"StopIteration","detail":"Caught by for itself, which is why you never see it"},{"label":"A second loop?","detail":"A list hands out a fresh iterator; a generator is used up"}],"title":"What a for loop actually does","type":"flow"},"alt":"Figure: What a for loop actually does. Steps: 1. for x in things (Two function calls inside a hidden while); 2. iter(things), once (Calls __iter__ and gets an iterator); 3. next(it), each pass (Runs to the next yield and binds the value to x); 4. StopIteration (Caught by for itself, which is why you never see it); 5. A second loop? (A list hands out a fresh iterator; a generator is used up). Because a generator is its own iterator, the second loop over it starts where the first one stopped, which is at the end. There is no error and no warning; the loop body simply never runs. If you need the data twice, materialise it once with list(), or write a function you can call again."}},"blocks":[{"t":"h2","id":"two-calls-under-every-loop","runs":[{"text":"Two calls under every loop"}]},{"t":"code","lang":"python","label":"Python","text":"for item in things:\n    ..."},{"t":"p","runs":[{"text":"is shorthand for:"}]},{"t":"code","lang":"python","label":"Python","text":"it = iter(things)\nwhile True:\n    try:\n        item = next(it)\n    except StopIteration:\n        break\n    ..."},{"t":"p","runs":[{"text":"iter(things)","code":true},{"text":" calls "},{"text":"things.__iter__()","code":true},{"text":" and gets back an "},{"text":"iterator","b":true},{"text":": an object with a "},{"text":"__next__","code":true},{"text":" method. "},{"text":"next(it)","code":true},{"text":" calls it. When there is nothing left, "},{"text":"__next__","code":true},{"text":" raises "},{"text":"StopIteration","code":true},{"text":" and the loop ends. That is the entire protocol, and every "},{"text":"for","code":true},{"text":", every comprehension, every "},{"text":"sum","code":true},{"text":", "},{"text":"list","code":true},{"text":", "},{"text":"max","code":true},{"text":", "},{"text":"zip","code":true},{"text":" and "},{"text":"enumerate","code":true},{"text":" runs on it."}]},{"t":"p","runs":[{"text":"Two roles are in play. An "},{"text":"iterable","b":true},{"text":" is anything "},{"text":"iter()","code":true},{"text":" accepts: a list, a string, a dict, a file, your "},{"text":"Conversation","code":true},{"text":" with its "},{"text":"__iter__","code":true},{"text":". An "},{"text":"iterator","b":true},{"text":" is the thing that hands out items one at a time and remembers where it is. The distinction seems academic until it bites."}]},{"t":"h2","id":"lists-give-you-a-fresh-iterator-generators-are-the-iterator","runs":[{"text":"Lists give you a fresh iterator; generators are the iterator"}]},{"t":"code","lang":"python","label":"Python","text":"nums = [1, 2, 3]\nprint(sum(nums), sum(nums))       # 6 6\n\ngen = (n for n in [1, 2, 3])\nprint(sum(gen), sum(gen))         # 6 0"},{"t":"p","runs":[{"text":"Each "},{"text":"iter(nums)","code":true},{"text":" returns a "},{"text":"new","b":true},{"text":" iterator starting at position 0, so a list can be walked any number of times. A generator object is its own iterator: "},{"text":"iter(gen)","code":true},{"text":" returns "},{"text":"gen","code":true},{"text":" itself, and it remembers that it already reached the end. The second "},{"text":"sum","code":true},{"text":" asks for "},{"text":"next","code":true},{"text":", gets "},{"text":"StopIteration","code":true},{"text":" immediately, and returns 0."}]},{"t":"p","runs":[{"text":"No error, no warning. The second pass is just empty. This is the single most common surprise for people who have started using generators, and you have been using them since module 5’s streaming and module 6’s pagination. The pattern that triggers it: count something, then process it, over the same generator. Or pass a generator to a function that iterates it twice internally, which "},{"text":"zip","code":true},{"text":" does not but a few library functions do."}]},{"t":"p","runs":[{"text":"The fix is to decide what you want. If you need to walk the data twice, materialise it once: "},{"text":"lines = list(gen)","code":true},{"text":". If the data is too large for that (the reason you used a generator) restructure so that one pass does both jobs, or create the generator twice by calling the function that makes it twice."}]},{"t":"h2","id":"writing-an-iterator-by-hand","runs":[{"text":"Writing an iterator by hand"}]},{"t":"p","runs":[{"text":"You rarely need to, because "},{"text":"yield","code":true},{"text":" does it, but seeing one makes the protocol concrete:"}]},{"t":"code","lang":"python","label":"Python","text":"class Countdown:\n    def __init__(self, n):\n        self.n = n\n    def __iter__(self):\n        return self\n    def __next__(self):\n        if self.n <= 0:\n            raise StopIteration\n        self.n -= 1\n        return self.n + 1"},{"t":"p","runs":[{"text":"__iter__","code":true},{"text":" returning "},{"text":"self","code":true},{"text":" is what makes it an iterator rather than merely iterable, and is what makes it single-use, for the same reason a generator is. A generator function is this class written as a function:"}]},{"t":"code","lang":"python","label":"Python","text":"def countdown(n):\n    while n > 0:\n        yield n\n        n -= 1"},{"t":"p","runs":[{"text":"Each "},{"text":"next()","code":true},{"text":" runs the function up to the next "},{"text":"yield","code":true},{"text":", hands out the value, and freezes the frame (local variables, position, everything) until the following "},{"text":"next()","code":true},{"text":". When the function returns, "},{"text":"StopIteration","code":true},{"text":" is raised for you. Laziness comes for free: nothing after the current "},{"text":"yield","code":true},{"text":" has run yet."}]},{"t":"h2","id":"next-with-a-default-and-peeking","runs":[{"text":"next()","code":true},{"text":" with a default, and peeking"}]},{"t":"code","lang":"python","label":"Python","text":"first = next(iter(things), None)"},{"t":"p","runs":[{"text":"gives you the first item or "},{"text":"None","code":true},{"text":" without a loop and without "},{"text":"IndexError","code":true},{"text":". It works on any iterable, including a generator of unknown length. To look at the first few of a large stream without consuming all of it:"}]},{"t":"code","lang":"python","label":"Python","text":"from itertools import islice\nhead = list(islice(gen, 5))"},{"t":"p","runs":[{"text":"islice","code":true},{"text":" takes the first five and stops; the rest of "},{"text":"gen","code":true},{"text":" is still there for a later loop. That is one of the few ways to sample a generator without materialising it, and it is what the “peek at the first rows” idiom in module 5 was doing."}]},{"t":"h2","id":"itertools-pieces-you-will-use","runs":[{"text":"itertools","code":true},{"text":" pieces you will use"}]},{"t":"ul","items":[[{"text":"chain(a, b)","code":true},{"text":": iterate "},{"text":"a","code":true},{"text":" then "},{"text":"b","code":true},{"text":" as one sequence, without building a combined list."}],[{"text":"batched(it, 20)","code":true},{"text":" (3.12+): groups of twenty, the natural shape for sending twenty prompts per request or per thread. Before 3.12, write a small "},{"text":"yield","code":true},{"text":" loop that fills a list and yields it when full."}],[{"text":"groupby(sorted_items, key=...)","code":true},{"text":": consecutive runs with the same key. It only groups "},{"text":"adjacent","b":true},{"text":" items, so sort first or it silently produces many small groups."}],[{"text":"count()","code":true},{"text":": infinite integers, for numbering a stream of unknown length alongside "},{"text":"zip","code":true},{"text":"."}]]},{"t":"p","runs":[{"text":"Every one of these is lazy. A pipeline of generators (read lines, strip them, filter, batch, send) processes one item at a time end to end, and its memory use does not grow with the file."}]},{"t":"h2","id":"zip-and-unequal-lengths","runs":[{"text":"zip","code":true},{"text":" and unequal lengths"}]},{"t":"p","runs":[{"text":"zip(a, b)","code":true},{"text":" stops at the shorter. If "},{"text":"a","code":true},{"text":" has 100 items and "},{"text":"b","code":true},{"text":" has 99 because of an off-by-one upstream, "},{"text":"zip","code":true},{"text":" drops the last silently. "},{"text":"zip(a, b, strict=True)","code":true},{"text":" (3.10+) raises instead. Use it whenever the two sequences are supposed to match, such as prompts and their results."}]},{"t":"h2","id":"where-this-leaves-you","runs":[{"text":"Where this leaves you"}]},{"t":"p","runs":[{"text":"You now know what "},{"text":"for","code":true},{"text":" does, why a file can be looped once, why "},{"text":"iter_lines","code":true},{"text":" in module 6 could be handed to any function that takes an iterable, and why "},{"text":"results = list(gen)","code":true},{"text":" appears at the end of so many pipelines: it is the moment the data stops being lazy and becomes something you can walk twice."}]},{"t":"h2","id":"try-this-now","runs":[{"text":"Try this now"}],"role":"try"},{"t":"p","runs":[{"text":"Write a generator that yields chunks of 100 items from any iterable. Feed it a "},{"text":"range(1050)","code":true},{"text":", check that the last chunk has 50, then consume it with "},{"text":"sum(len(c) for c in chunks)","code":true},{"text":" and try to loop over "},{"text":"chunks","code":true},{"text":" again."}]}],"check":{"question":[{"text":"A function returns "},{"text":"(line.strip() for line in open(path))","code":true},{"text":". The caller does "},{"text":"n = sum(1 for _ in lines)","code":true},{"text":" to count them and then "},{"text":"for line in lines: process(line)","code":true},{"text":", and nothing is processed. What is the mechanism?"}],"options":[{"text":[{"text":"sum","code":true},{"text":" closes the underlying file when it finishes iterating, so the second loop finds a closed handle and stops immediately without raising, because closed files iterate as empty."}],"correct":false,"tempting":[{"text":"The file stays open; the generator object is what was exhausted, and it would be just as empty over a list of strings."}]},{"text":[{"text":"Generator expressions can be consumed only by "},{"text":"for","code":true},{"text":", and "},{"text":"sum","code":true},{"text":" consumed it in an invalid way that left it broken."}],"correct":false,"tempting":[{"text":"sum","code":true},{"text":" iterates exactly as "},{"text":"for","code":true},{"text":" does, calling next() until StopIteration; nothing about it is invalid."}]},{"text":[{"text":"The generator is its own iterator, so the count ran it to StopIteration and the second loop asked an exhausted iterator for more."}],"correct":true,"tempting":null},{"text":[{"text":"strip()","code":true},{"text":" returns a new string each time, and the second pass compares stripped strings to the originals and finds no matches."}],"correct":false,"tempting":[{"text":"No comparison is taking place; the loop body never runs because there is nothing left to yield."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"dunder-methods","title":"Dunder methods: making your own objects work with len, for, in and ==","minutes":9},"next":{"slug":"context-managers","title":"Context managers: the cleanup that runs even when the block raises","minutes":8}}}