Every turn carries all the previous ones

The shape of the problem

A model has no memory between calls. What looks like a conversation is your code resending the whole transcript on every turn, plus one new message. That single mechanical fact produces four consequences, and every technique in this module is a response to one of them.

The bill grows with the square. Turn ten resends turns one through nine. The cost lesson did the arithmetic: ten turns of 500 tokens is 27,500 billed input tokens, not 5,000.

The window fills. At some point the transcript exceeds the history budget, and something has to give. If you have not decided what, a framework will decide for you, usually by dropping from the front.

Attention dilutes. By turn twenty the user’s original instruction is thousands of tokens from where generation begins, competing with everything since. The position module explains why this weakens it.

Errors persist. This is the one people underestimate. A wrong statement the model made on turn three is, on turn nine, part of the context: sitting in the assistant role, phrased confidently, indistinguishable from an established fact. Models are consistent with their own prior output, so the error does not just survive, it gets built on. A conversation that has gone wrong tends to stay wrong, and adding a correction later is weaker than the original mistake because the mistake has been repeated and elaborated.

Where the growth actually comes from

Measure it on your own traffic before optimising. The three sources are unequal:

• Assistant turns are usually the largest. They are longer than user turns and there are just as many of them. A verbose assistant is a compounding cost.
• Tool results are spiky. Most turns add nothing; one search result adds 12,000 tokens and permanently changes the shape of the conversation.
• User turns are the smallest and the most variable: three words or a pasted 40-page document.

So the highest-return length control is often on the assistant’s own output, which is also the expensive token rate. “Be concise unless asked otherwise” is a cost lever, a latency lever and a context lever at once.

The numbers you need from your logs

Four, and they take an hour to extract:

1. Median and 95th percentile turns per conversation. Almost always a long tail: most conversations are two turns and some are ninety.
2. Median and 95th percentile tokens per turn, split by role.
3. The distribution of total conversation tokens. This decides whether compaction is a real requirement or a hypothetical one.
4. What fraction of conversations exceed your history budget at all. If it is 2%, you need a correct-but-simple policy. If it is 40%, compaction is a core feature and deserves design.

Teams commonly discover that the 95th-percentile conversation is ten times the median, and that all of their cost and all of their quality complaints live in that tail.

Set the history budget as a cap, not a hope

From the budget ledger in module one: history gets a number. When the transcript exceeds it, a policy runs. The policy is the subject of the next three lessons, and the important part is that it exists and is yours.

One rule to fix now, because it is where naive implementations break: the system prompt is not part of the history window. Pin it outside. A messages[-20:] that includes the system message removes it on turn twenty-one, and the assistant changes personality mid-conversation with no error anywhere.

Two things worth doing before compaction

Before building summarisation machinery, two cheaper interventions often solve the problem.

Cache the history prefix. History is append-only, so everything up to the last turn is a stable prefix. A cache breakpoint after it turns the quadratic bill into something much flatter without changing behaviour at all. This is the single highest-value change for a chat product and it requires no summarisation.

Drop tool results that have been used. A search result that produced an answer three turns ago is usually dead weight. Replacing it with a one-line note (“searched for X, 8 results, used [3] and [5]”) recovers most of a large block with almost no loss. This is compaction, but of the easiest kind, on the material with the worst value-per-token in the whole transcript.

Do both first. Then measure again, and see whether you still need the harder thing.