What to throw away, and when

The arithmetic nobody does first

Every turn sends the entire transcript again. If each turn adds roughly two thousand tokens, then turn one sends two thousand, turn ten sends twenty thousand, turn twenty sends forty thousand. The total input across a twenty-turn run is the sum of that series (around four hundred and twenty thousand tokens) for a task where the model only ever produced perhaps eight thousand tokens of output.

That is the single most important fact about running agents at any volume. Cost is quadratic in the number of steps. Latency follows the same curve, because time to first token grows with the prompt. A run that is twice as long is roughly four times as expensive, which is why the honest advice throughout this course is to make the chain shorter rather than the model bigger.

Caching, and the mistake that switches it off

Providers cache a prefix of the prompt. When the next request begins with byte-identical content, the cached portion is charged at a large discount: commonly around a tenth of the normal input price, though the figure and the minimum cacheable length vary by provider and change often, so check the current terms rather than trusting a number in a course.

The mechanism gives you one hard rule: the prefix must not change. Which means:

• Never put the current time, a request id, a random session token or a rotating greeting at the top of the system prompt. One volatile value at position zero invalidates the cache for the entire run and quietly multiplies your bill.
• Append, do not edit. Rewriting message three on turn twelve invalidates everything from message three onward.
• Put volatile context in the last message, where it costs a cache miss on the tail only.

Teams discover this backwards, by watching costs rise after a harmless-looking change. The timestamp is the classic.

Four ways to keep the window small

Trim tool results, keep the calls. The fact that search_orders was called with these arguments is small and important. The eight thousand tokens it returned are usually not needed once the relevant three lines have been used. Replace the old result with a one-line stand-in that says what it was and that it was trimmed. Silence is the danger: an agent that cannot tell truncation from emptiness will confidently reason about a subset.

Pass handles, not blobs. Write the large thing to disk or to a store and pass /runs/8842/invoice.txt plus a summary. Give the agent a read_file tool with an offset and a length. This turns a fixed cost on every subsequent turn into a cost paid only when the content is needed again.

Compact. At a threshold, summarise turns one to k into a short handover note (what was learned, what was done, what remains) and start the transcript again from that note. This is lossy on purpose. Write the summariser’s prompt as carefully as the agent’s, because everything it drops is gone.

Restart. For long-running work, finishing a phase and opening a fresh transcript with an explicit handover is often better than compacting a sprawling one. It is also easier to debug, because each transcript is one coherent phase.

Long context does not dissolve the problem

Two reasons. Cost and latency scale with tokens whatever the advertised limit. And retrieval from the middle of a long context is measurably worse than from the beginning or the end: the “lost in the middle” effect has been reported across several studies and holds up broadly, though how strongly it bites varies a lot by model and by how the task is posed. Accuracy on long-context benchmarks typically starts sliding well before the stated maximum.

The practical version: position matters. The current instruction and the most recent evidence belong near the end of the prompt, not buried in the middle of a large dump of retrieved material.

Measure it

Count tokens before you send them. tiktoken or the Hugging Face tokenizers library will do this locally and free, and both are close enough for budgeting even when the tokeniser is not an exact match for the model. Log per-turn input tokens, output tokens and the cumulative total on every run, and put the cumulative number in your trace viewer. The first time you look at that curve for a real run, you will find one tool returning a thousand-line payload on every call, and fixing that one tool will do more for your bill than any other change you make this month.