Compacting without losing the thread

Trigger on tokens, not on turns

Compact when the history block crosses a fraction of its budget (70% is a reasonable default) not after a fixed number of messages. Turns vary from four tokens to fourteen thousand, so a turn count is a proxy for the thing you actually care about, and a bad one.

Compacting at 70% rather than 100% matters. It leaves room for the next few turns, so you are not compacting on every single message once you reach the ceiling, which would be both expensive and behaviourally jarring.

if tokens(history) > 0.7 * BUDGET["history"]:
    head, middle, tail = split(history, keep_first=1, keep_last=K)
    history = [head] + [compact(middle)] + tail

Ask for structure, not for prose

The single biggest quality difference between a good compaction and a bad one is what you ask for. A request to “summarise the conversation so far” produces fluent narrative that loses exactly the things you needed: numbers, names, constraints, and what was ruled out.

Ask for fields:

Produce a compact record of this conversation with these headings.
Use the exact wording for names, numbers, ids and quoted text.

TASK: what the user is trying to achieve, in their words
CONSTRAINTS: every requirement or restriction stated, one per line
DECISIONS: what has been settled, and what was rejected and why
FACTS: specific values established (ids, dates, amounts, names)
TRIED: approaches attempted and their outcomes
OPEN: questions still unanswered

Two of those headings do most of the work. DECISIONS including rejections stops the conversation looping back to an option already dismissed, which is the most visible symptom of bad compaction. TRIED stops the model re-attempting something that failed: the same reason it is the most valuable line in a handover note.

Compact from the raw transcript, not from the last summary

A summary of a summary loses more at each step, and the loss is not random: specifics go first, then qualifiers, then the distinction between what was decided and what was merely discussed. After four rounds you have a general description of a conversation that no longer constrains anything.

Since you kept the full transcript, re-derive each compaction from the original messages rather than from the previous summary. It costs more tokens per compaction and it stops the drift. Where a conversation is enormous, compact in segments from raw and concatenate the segment records, rather than recursively compressing a compression.

Expect a discontinuity, and place it well

Compaction rewrites the middle of the prefix. Two things follow.

The cache is invalidated from the compaction point onwards. You will pay a cache write on the next call. This is a good reason not to compact every turn and a good reason to compact a large chunk when you do.

The behaviour visibly changes. The model has a different, thinner view of the conversation than it did a moment ago, and users notice as a shift in tone or a suddenly repeated question. Where you can, compact at a natural boundary (after a task completes, when the user changes subject, at the start of a session) rather than mid-thread.

Verify the compaction

It is model output, so it can be wrong, and it is about to become the only record the model has. Two cheap checks worth running.

Containment. Every id, number and date in the record should appear somewhere in the source transcript. A regex over both catches a fabricated figure immediately.

for tok in re.findall(r"\b[A-Z]{2,}-\d+|\b\d[\d,\.]*\b", record):
    assert tok in raw_transcript, f"invented value: {tok}"

Coverage. Did each constraint the user stated survive? If your structured state (the next lesson) already tracks constraints as fields, this is a set comparison rather than a judgement.

What compaction cannot do

Be honest about the trade. Compaction is lossy by definition; the only question is which loss you choose. Structured extraction loses the texture and the reasoning; prose summary loses the specifics; dropping loses everything.

There is no configuration that makes a 200,000-token conversation fit in 6,000 tokens without loss. What you can do is choose to lose the things that will not be needed, keep a pointer to everything, and make the loss visible to the user rather than silent. The last lesson in this module is about that visibility, and it is the part that turns an engineering limitation into an acceptable product.