Compacting history: sliding windows, summaries and what gets lost

Why dropping the oldest is not enough

The previous lesson trimmed from the oldest turn. For a support chat that is fine. For a long working session it is not. At turn three the user said “the budget is under 20,000 rupees”. By turn thirty that message is gone, the model suggests something at 35,000, and the user, who can see turn three on their own screen, concludes the product does not listen.

The cost side pushes the same way. A conversation whose turns average 300 tokens sends 15,000 tokens at turn fifty. Summed across the conversation that is roughly 375,000 input tokens for fifty answers: the quadratic growth the agent-loop lesson warned about, now in a chat.

Sliding window plus running summary

Keep the last N turns verbatim, and replace everything older with a summary placed directly after the system prompt. When the window overflows again, summarise the old summary together with the turns being dropped, so the summary is updated incrementally rather than regenerated from scratch. Each compaction is one model call with a few hundred output tokens, and it happens every few turns, not every turn.

With a ten-turn window and a 500-token summary, each request sends about 3,500 tokens instead of 15,000 at the far end, and the fifty-turn conversation costs roughly half what it did. With prompt caching on the stable prefix, less again.

What to ask the summariser for

Not “summarise this conversation”. A summary written freely drifts towards narrative (“the user asked about laptops and the assistant suggested some options”) which preserves the story and drops everything a later turn needs. Ask for structure, using the structured-output pattern from the earlier module:

Produce JSON with: goal (one sentence), constraints (list, verbatim
where stated), decisions_made (list), open_questions (list),
facts_user_stated (list of name: value). Do not paraphrase numbers.

What gets lost, and why

Summarisation is compression, and compression discards what looks least important to the compressor. Specifics are the first casualty: “under 20,000” becomes “discussed budget”; “not the blue one” becomes “discussed colour”; a product code becomes “a product”. Negations and numbers are short and easy to smooth away, and they are exactly what the user cared about.

The fix is to stop relying on the summary for them. Maintain a separate list of hard facts and constraints (extracted with a small structured call as each user turn arrives) that is only ever appended to, never summarised, and always placed in the context. The summary carries the shape of the conversation; the fact list carries the details. Two structures, because they have different failure modes.

Pinning

Some messages must never leave the context: the user’s original brief, the key facts extracted from a document they uploaded, a correction they made emphatically. Let a message be pinned, in the data model and optionally in the interface, and exempt pinned messages from trimming. Budget for them; three pinned messages of 800 tokens is a fixed 2,400 tokens on every request.

Retrieval over the conversation itself

For very long sessions there is a third tool: embed each past turn, and retrieve the ones relevant to the current message, using the retrieval module’s pipeline pointed at the conversation instead of a document set. It answers “what did I say about the delivery address?” well. It does not help with questions that need the whole arc, and it adds a retrieval step to every turn. Use it when conversations run to hundreds of turns, which is rarer than people expect.

Tell the user

When compaction has happened, say so, briefly: “I’ve condensed the earlier part of our conversation; key details are kept.” The marker in the interface, at the point where verbatim history stops, does the same job visually. The alternative (silently changing what the model can see) produces the “it forgot” complaint with no explanation available to anyone.

Test it, because it fails quietly

Nothing errors when compaction loses a fact. Build evaluation cases for it: a constraint stated at turn two, forty turns of unrelated work, and a question at turn forty-two whose correct answer depends on the constraint. Run them on every change to the summary prompt, the window size or the fact extractor. The evaluation module will make this a habit; this lesson is where the cases come from.