{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"the-agent-loop-grows","title":"Why a twenty-step agent run costs what it does","url":"https://addaly.com/learn/context-engineering/the-agent-loop-grows","position":65,"of":87,"minutes":10,"answer":"An agent loop resends its entire accumulated transcript on every step.","keyIdea":"An agent loop resends its entire accumulated transcript on every step, so cost grows with the square of the number of steps and most of what is being resent is observations whose purpose has already been served."},"body":{"format":"markdown","text":"## Do the arithmetic once and you will never forget it\n\nAn agent loop is not a conversation with a model. It is a series of separate requests, each of which contains everything that came before.\n\nTake a plausible run. System prompt and tool schemas: 6,000 tokens. Each step adds an assistant message with a tool call (about 150 tokens) and a tool result (say 1,800 tokens after truncation). So each step grows the context by roughly 2,000 tokens.\n\nStep 1 sends about 8,000 tokens. Step 25 sends about 56,000. The total input billed across the run is the sum of every step's context:\n\n```\n  total ≈ 25 × (8,000 + 56,000) / 2 ≈ 800,000 input tokens\n```\n\nEight hundred thousand input tokens for a task whose entire accumulated material is 56,000. The ratio is not a bug in the design; it is the design. Every step re-reads the whole history because the model is stateless.\n\n> Figure: What each step of an agent run sends. Line chart of thousands of input tokens sent on this step against step number: The whole transcript, resent (1, 8) (5, 15.8) (10, 25.5) (15, 35.3) (20, 45) (25, 56); The new material on that step alone (1, 1.95) (5, 1.95) (10, 1.95) (15, 1.95) (20, 1.95) (25, 1.95). Six thousand tokens of system prompt and tool schemas, then about 1,950 tokens of new material a step. The run bills roughly eight hundred thousand input tokens for a transcript that ends at fifty-six thousand. Caching changes the price of those tokens and never the count, and anything that rewrites an earlier part of the transcript throws the discount away from that point onwards, which is why stubbing happens in batches rather than on every step.\n\nThis is the quadratic growth the conversation module described, and in an agent loop it is far more pronounced, because agent steps produce large observations while conversational turns produce small ones.\n\n## Caching changes the price, not the count\n\nPrefix caching is the first and largest mitigation, and it works well here for a structural reason: an agent transcript is append-only. Each step's context is the previous step's context plus new material at the end. That is precisely the shape prefix caching wants.\n\nWith a warm cache most of those 800,000 tokens bill at the cached rate, commonly a small fraction of the standard input price. The tokens still exist, the latency of reading them still partly exists, but the bill falls a long way.\n\nWhich leads to the rule that governs everything else in this lesson: **anything that rewrites earlier parts of the transcript throws that discount away from the point of the change onwards.** Hold onto it, because the obvious optimisation violates it.\n\n## Most of what you are resending is spent\n\nLook at a long trace and sort the tokens by what they are. Directory listings from step 2. The full text of a file that was read at step 4 and edited at step 5. A search result whose one useful line was acted on immediately. A schema dump consulted once.\n\nBy step twenty, the great majority of the context is observations whose purpose has been served. They are not merely wasted spend. They are also distractors competing for attention with the material that still matters, and they push the current instruction further from the end of the window.\n\nThe principle: **an observation's value expires when the action it informed has completed and its outcome is recorded elsewhere.** A file's contents matter until you have edited the file. After that, what matters is the edit.\n\n## Stubbing, and the mistake everybody makes first\n\nThe conversation module already introduced the remedy: replace a spent result with a short note standing in for it. In an agent loop it is the single highest-value move available, and there is one specific way of implementing it that costs more than it saves.\n\nThe note itself looks like this:\n\n```\n<tool_result id=\"7\" tool=\"read_file\" status=\"stubbed\">\nRead src/auth/session.py (412 lines). Content omitted — it was edited at\nstep 9 and is now stale. Call read_file again if you need the current version.\n</tool_result>\n```\n\nFifty tokens instead of two thousand. Note that it says the content is *stale*, not merely absent. A model that believes it still knows what is in a file it has since changed will make decisions on the old version, which is worse than not knowing.\n\nNow the mistake. The natural implementation is to stub the oldest observation on every step, keeping a rolling window. That rewrites the prefix on every single step, so every single step is a complete cache miss. You have replaced a large discounted bill with a smaller undiscounted one, and the undiscounted one is usually bigger.\n\nStub in batches instead. Let the transcript grow untouched until it crosses a threshold, say 60 percent of your budget, then stub aggressively in one pass, accept one cache miss, and grow cleanly again from the new prefix. One miss every fifteen steps rather than one every step.\n\nThe same reasoning explains why the popular advice to \"just drop old messages\" is worse than it looks. Dropping is a rewrite. Its cost is not the information lost but the cache destroyed, and that cost is invisible unless you are tracking hit rate.\n\n## Keep the full trace somewhere else\n\nStubbing is about the window, never about the record. Write every observation to storage as it arrives, keyed by step. The window holds what the model needs now; the store holds what you need when someone asks why the agent did what it did on Tuesday.\n\nThis also gives you the repair path. If the model stubs its way into needing something it no longer has, the tool that fetches it again reads from your store rather than re-running the original call: cheaper, and guaranteed to return what it returned the first time.\n\n## Bound the loop in code\n\nThe last defence is the crudest and the most necessary: a hard cap on steps, and a hard cap on cumulative tokens for the run. Not in the prompt: in the loop.\n\nAgents fail by repetition more often than by error. A model that cannot find a file will look for it in eleven places, each look costing a full context. A step cap turns that from an open-ended bill into a bounded one, and the message you return when it trips should tell the user what was attempted rather than stopping, because a run that halts silently at step 40 is indistinguishable from a crash."},"check":{"question":"An implementation stubs the oldest tool result on every step to keep the context small. Why does this often increase the bill?","options":[{"text":"Stub markers are themselves expensive, so replacing many results adds more tokens than it removes.","correct":false,"tempting":"A stub is tens of tokens against thousands removed; the token arithmetic is strongly favourable and is not where the loss is."},{"text":"The model re-reads the stubbed files, so each stub causes an extra tool call.","correct":false,"tempting":"It sometimes does, but that is occasional and task-dependent, whereas the cache loss is incurred on every single step by construction."},{"text":"Rewriting the transcript every step means every step is a full cache miss on the whole prefix.","correct":true},{"text":"Shorter contexts are billed at the standard rate while long ones qualify for volume discounts.","correct":false,"tempting":"No provider prices input by context length in that way; the cached-token rate depends on a prefix match, not on size."}]},"prev":{"slug":"too-many-tools","title":"When the toolset outgrows the window","url":"https://addaly.com/learn/context-engineering/too-many-tools"},"next":{"slug":"sub-agents-and-window-isolation","title":"Sub-agents buy you a window and charge you a boundary","url":"https://addaly.com/learn/context-engineering/sub-agents-and-window-isolation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why a twenty-step agent run costs what it does', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/the-agent-loop-grows (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why a twenty-step agent run costs what it does","courseTitle":"Context Engineering","module":{"index":7,"slug":"tools-schemas-and-the-agent-window","title":"Tools, schemas and the agent’s window"},"answer":null,"keyIdea":"An agent loop resends its entire accumulated transcript on every step, so cost grows with the square of the number of steps and most of what is being resent is observations whose purpose has already been served.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"What each step of an agent run sends","series":[{"label":"The whole transcript, resent","points":[[1,8],[5,15.8],[10,25.5],[15,35.3],[20,45],[25,56]]},{"label":"The new material on that step alone","points":[[1,1.95],[5,1.95],[10,1.95],[15,1.95],[20,1.95],[25,1.95]]}],"xLabel":"step number","yLabel":"thousands of input tokens sent on this step","caption":"Six thousand tokens of system prompt and tool schemas, then about 1,950 tokens of new material a step. The run bills roughly eight hundred thousand input tokens for a transcript that ends at fifty-six thousand. Caching changes the price of those tokens and never the count, and anything that rewrites an earlier part of the transcript throws the discount away from that point onwards, which is why stubbing happens in batches rather than on every step."},"alt":"Figure: What each step of an agent run sends. Line chart of thousands of input tokens sent on this step against step number: The whole transcript, resent (1, 8) (5, 15.8) (10, 25.5) (15, 35.3) (20, 45) (25, 56); The new material on that step alone (1, 1.95) (5, 1.95) (10, 1.95) (15, 1.95) (20, 1.95) (25, 1.95). Six thousand tokens of system prompt and tool schemas, then about 1,950 tokens of new material a step. The run bills roughly eight hundred thousand input tokens for a transcript that ends at fifty-six thousand. Caching changes the price of those tokens and never the count, and anything that rewrites an earlier part of the transcript throws the discount away from that point onwards, which is why stubbing happens in batches rather than on every step."}},"blocks":[{"t":"h2","id":"do-the-arithmetic-once-and-you-will-never-forget-it","runs":[{"text":"Do the arithmetic once and you will never forget it"}]},{"t":"p","runs":[{"text":"An agent loop is not a conversation with a model. It is a series of separate requests, each of which contains everything that came before."}]},{"t":"p","runs":[{"text":"Take a plausible run. System prompt and tool schemas: 6,000 tokens. Each step adds an assistant message with a tool call (about 150 tokens) and a tool result (say 1,800 tokens after truncation). So each step grows the context by roughly 2,000 tokens."}]},{"t":"p","runs":[{"text":"Step 1 sends about 8,000 tokens. Step 25 sends about 56,000. The total input billed across the run is the sum of every step’s context:"}]},{"t":"code","lang":"","label":"Text","text":"  total ≈ 25 × (8,000 + 56,000) / 2 ≈ 800,000 input tokens"},{"t":"p","runs":[{"text":"Eight hundred thousand input tokens for a task whose entire accumulated material is 56,000. The ratio is not a bug in the design; it is the design. Every step re-reads the whole history because the model is stateless."}]},{"t":"p","runs":[{"text":"This is the quadratic growth the conversation module described, and in an agent loop it is far more pronounced, because agent steps produce large observations while conversational turns produce small ones."}]},{"t":"h2","id":"caching-changes-the-price-not-the-count","runs":[{"text":"Caching changes the price, not the count"}]},{"t":"p","runs":[{"text":"Prefix caching is the first and largest mitigation, and it works well here for a structural reason: an agent transcript is append-only. Each step’s context is the previous step’s context plus new material at the end. That is precisely the shape prefix caching wants."}]},{"t":"p","runs":[{"text":"With a warm cache most of those 800,000 tokens bill at the cached rate, commonly a small fraction of the standard input price. The tokens still exist, the latency of reading them still partly exists, but the bill falls a long way."}]},{"t":"p","runs":[{"text":"Which leads to the rule that governs everything else in this lesson: "},{"text":"anything that rewrites earlier parts of the transcript throws that discount away from the point of the change onwards.","b":true},{"text":" Hold onto it, because the obvious optimisation violates it."}]},{"t":"h2","id":"most-of-what-you-are-resending-is-spent","runs":[{"text":"Most of what you are resending is spent"}]},{"t":"p","runs":[{"text":"Look at a long trace and sort the tokens by what they are. Directory listings from step 2. The full text of a file that was read at step 4 and edited at step 5. A search result whose one useful line was acted on immediately. A schema dump consulted once."}]},{"t":"p","runs":[{"text":"By step twenty, the great majority of the context is observations whose purpose has been served. They are not merely wasted spend. They are also distractors competing for attention with the material that still matters, and they push the current instruction further from the end of the window."}]},{"t":"p","runs":[{"text":"The principle: "},{"text":"an observation’s value expires when the action it informed has completed and its outcome is recorded elsewhere.","b":true},{"text":" A file’s contents matter until you have edited the file. After that, what matters is the edit."}]},{"t":"h2","id":"stubbing-and-the-mistake-everybody-makes-first","runs":[{"text":"Stubbing, and the mistake everybody makes first"}]},{"t":"p","runs":[{"text":"The conversation module already introduced the remedy: replace a spent result with a short note standing in for it. In an agent loop it is the single highest-value move available, and there is one specific way of implementing it that costs more than it saves."}]},{"t":"p","runs":[{"text":"The note itself looks like this:"}]},{"t":"code","lang":"","label":"Text","text":"<tool_result id=\"7\" tool=\"read_file\" status=\"stubbed\">\nRead src/auth/session.py (412 lines). Content omitted — it was edited at\nstep 9 and is now stale. Call read_file again if you need the current version.\n</tool_result>"},{"t":"p","runs":[{"text":"Fifty tokens instead of two thousand. Note that it says the content is "},{"text":"stale","i":true},{"text":", not merely absent. A model that believes it still knows what is in a file it has since changed will make decisions on the old version, which is worse than not knowing."}]},{"t":"p","runs":[{"text":"Now the mistake. The natural implementation is to stub the oldest observation on every step, keeping a rolling window. That rewrites the prefix on every single step, so every single step is a complete cache miss. You have replaced a large discounted bill with a smaller undiscounted one, and the undiscounted one is usually bigger."}]},{"t":"p","runs":[{"text":"Stub in batches instead. Let the transcript grow untouched until it crosses a threshold, say 60 percent of your budget, then stub aggressively in one pass, accept one cache miss, and grow cleanly again from the new prefix. One miss every fifteen steps rather than one every step."}]},{"t":"p","runs":[{"text":"The same reasoning explains why the popular advice to “just drop old messages” is worse than it looks. Dropping is a rewrite. Its cost is not the information lost but the cache destroyed, and that cost is invisible unless you are tracking hit rate."}]},{"t":"h2","id":"keep-the-full-trace-somewhere-else","runs":[{"text":"Keep the full trace somewhere else"}]},{"t":"p","runs":[{"text":"Stubbing is about the window, never about the record. Write every observation to storage as it arrives, keyed by step. The window holds what the model needs now; the store holds what you need when someone asks why the agent did what it did on Tuesday."}]},{"t":"p","runs":[{"text":"This also gives you the repair path. If the model stubs its way into needing something it no longer has, the tool that fetches it again reads from your store rather than re-running the original call: cheaper, and guaranteed to return what it returned the first time."}]},{"t":"h2","id":"bound-the-loop-in-code","runs":[{"text":"Bound the loop in code"}]},{"t":"p","runs":[{"text":"The last defence is the crudest and the most necessary: a hard cap on steps, and a hard cap on cumulative tokens for the run. Not in the prompt: in the loop."}]},{"t":"p","runs":[{"text":"Agents fail by repetition more often than by error. A model that cannot find a file will look for it in eleven places, each look costing a full context. A step cap turns that from an open-ended bill into a bounded one, and the message you return when it trips should tell the user what was attempted rather than stopping, because a run that halts silently at step 40 is indistinguishable from a crash."}]}],"check":{"question":[{"text":"An implementation stubs the oldest tool result on every step to keep the context small. Why does this often increase the bill?"}],"options":[{"text":[{"text":"Stub markers are themselves expensive, so replacing many results adds more tokens than it removes."}],"correct":false,"tempting":[{"text":"A stub is tens of tokens against thousands removed; the token arithmetic is strongly favourable and is not where the loss is."}]},{"text":[{"text":"The model re-reads the stubbed files, so each stub causes an extra tool call."}],"correct":false,"tempting":[{"text":"It sometimes does, but that is occasional and task-dependent, whereas the cache loss is incurred on every single step by construction."}]},{"text":[{"text":"Rewriting the transcript every step means every step is a full cache miss on the whole prefix."}],"correct":true,"tempting":null},{"text":[{"text":"Shorter contexts are billed at the standard rate while long ones qualify for volume discounts."}],"correct":false,"tempting":[{"text":"No provider prices input by context length in that way; the cached-token rate depends on a prefix match, not on size."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"too-many-tools","title":"When the toolset outgrows the window","minutes":9},"next":{"slug":"sub-agents-and-window-isolation","title":"Sub-agents buy you a window and charge you a boundary","minutes":9}}}