Why a twenty-step agent run costs what it does

Do the arithmetic once and you will never forget it

An agent loop is not a conversation with a model. It is a series of separate requests, each of which contains everything that came before.

Take a plausible run. System prompt and tool schemas: 6,000 tokens. Each step adds an assistant message with a tool call (about 150 tokens) and a tool result (say 1,800 tokens after truncation). So each step grows the context by roughly 2,000 tokens.

Step 1 sends about 8,000 tokens. Step 25 sends about 56,000. The total input billed across the run is the sum of every step’s context:

  total ≈ 25 × (8,000 + 56,000) / 2 ≈ 800,000 input tokens

Eight hundred thousand input tokens for a task whose entire accumulated material is 56,000. The ratio is not a bug in the design; it is the design. Every step re-reads the whole history because the model is stateless.

This is the quadratic growth the conversation module described, and in an agent loop it is far more pronounced, because agent steps produce large observations while conversational turns produce small ones.

Caching changes the price, not the count

Prefix caching is the first and largest mitigation, and it works well here for a structural reason: an agent transcript is append-only. Each step’s context is the previous step’s context plus new material at the end. That is precisely the shape prefix caching wants.

With a warm cache most of those 800,000 tokens bill at the cached rate, commonly a small fraction of the standard input price. The tokens still exist, the latency of reading them still partly exists, but the bill falls a long way.

Which leads to the rule that governs everything else in this lesson: anything that rewrites earlier parts of the transcript throws that discount away from the point of the change onwards. Hold onto it, because the obvious optimisation violates it.

Most of what you are resending is spent

Look at a long trace and sort the tokens by what they are. Directory listings from step 2. The full text of a file that was read at step 4 and edited at step 5. A search result whose one useful line was acted on immediately. A schema dump consulted once.

By step twenty, the great majority of the context is observations whose purpose has been served. They are not merely wasted spend. They are also distractors competing for attention with the material that still matters, and they push the current instruction further from the end of the window.

The principle: an observation’s value expires when the action it informed has completed and its outcome is recorded elsewhere. A file’s contents matter until you have edited the file. After that, what matters is the edit.

Stubbing, and the mistake everybody makes first

The conversation module already introduced the remedy: replace a spent result with a short note standing in for it. In an agent loop it is the single highest-value move available, and there is one specific way of implementing it that costs more than it saves.

The note itself looks like this:

<tool_result id="7" tool="read_file" status="stubbed">
Read src/auth/session.py (412 lines). Content omitted — it was edited at
step 9 and is now stale. Call read_file again if you need the current version.
</tool_result>

Fifty tokens instead of two thousand. Note that it says the content is stale, not merely absent. A model that believes it still knows what is in a file it has since changed will make decisions on the old version, which is worse than not knowing.

Now the mistake. The natural implementation is to stub the oldest observation on every step, keeping a rolling window. That rewrites the prefix on every single step, so every single step is a complete cache miss. You have replaced a large discounted bill with a smaller undiscounted one, and the undiscounted one is usually bigger.

Stub in batches instead. Let the transcript grow untouched until it crosses a threshold, say 60 percent of your budget, then stub aggressively in one pass, accept one cache miss, and grow cleanly again from the new prefix. One miss every fifteen steps rather than one every step.

The same reasoning explains why the popular advice to “just drop old messages” is worse than it looks. Dropping is a rewrite. Its cost is not the information lost but the cache destroyed, and that cost is invisible unless you are tracking hit rate.

Keep the full trace somewhere else

Stubbing is about the window, never about the record. Write every observation to storage as it arrives, keyed by step. The window holds what the model needs now; the store holds what you need when someone asks why the agent did what it did on Tuesday.

This also gives you the repair path. If the model stubs its way into needing something it no longer has, the tool that fetches it again reads from your store rather than re-running the original call: cheaper, and guaranteed to return what it returned the first time.

Bound the loop in code

The last defence is the crudest and the most necessary: a hard cap on steps, and a hard cap on cumulative tokens for the run. Not in the prompt: in the loop.

Agents fail by repetition more often than by error. A model that cannot find a file will look for it in eleven places, each look costing a full context. A step cap turns that from an open-ended bill into a bounded one, and the message you return when it trips should tell the user what was attempted rather than stopping, because a run that halts silently at step 40 is indistinguishable from a crash.