Searching the transcript instead of carrying it

Bounded context, unbounded conversation

Compaction compresses history. The alternative is to stop carrying it at all: index the transcript, and retrieve only the parts relevant to the current message.

This inverts the cost curve. History stops growing in the window; a conversation of five hundred turns costs the same per call as one of twenty. It also crosses sessions naturally: a message from three weeks ago is as retrievable as one from three minutes ago.

It is also harder than it looks, for reasons that are specific to conversational text.

Why conversation retrieves badly

Turns are short and full of pronouns. “Yes, do that” is four tokens and means nothing on its own. Embedding it produces a vector that matches nothing useful.

Meaning is distributed across turns. A decision is often a user message and an assistant reply together; retrieving either half gives you a fragment.

Similarity across a transcript is uniformly high. Everything in one conversation is about the same subject, so cosine similarity discriminates poorly: every turn looks somewhat relevant.

The fixes are the same ones from the retrieval module, applied to a different corpus.

Segment by topic, not by turn. Group consecutive turns into episodes that share a subject, and index the episode. A simple heuristic (a gap in time, an explicit change of subject, a completed task) works better than a fixed turn count.

Augment each segment at write time. The same contextual-augmentation trick: store a one-line description of what this exchange was about, in standalone language. “The user asked whether the refund could go to a different card; agreed to keep the original method.” That sentence retrieves; “Yes, do that” does not.

Resolve references before indexing. Rewrite each user turn into a standalone form as you store it, using the same rewriting machinery as the query lesson. You are paying for the rewrite anyway on the way in.

Always keep the tail verbatim

The critical design rule: retrieval supplements the recent window, it does not replace it.

A system that retrieves only semantically similar history and drops the last few turns will lose the thread of the immediate exchange, because the most recent turn is often not the most semantically similar to the current message. The user says “actually, make it shorter”; nothing in that retrieves the draft it refers to.

So the layout is additive:

[structured state]         always
[retrieved episodes]       relevant older exchanges, chronologically ordered
[last k turns]             always, verbatim
[current message]

Restore chronology, and label it

Retrieved episodes come back ranked by relevance. Present them in time order, as the ordering lesson argued, because a conversation is a sequence and reading it out of order produces a wrong account of what was decided when.

And label them as what they are:

Earlier in this conversation (17 August, turns 12-15):

Without the label, the model treats a retrieved fragment from three weeks ago as part of the current exchange, and answers as though a resolved issue is live. With the date, it can at least reason about sequence.

When this is worth building

Be honest about the threshold. For a product where conversations are under thirty turns, this machinery is not worth it: pinned head, compacted middle, verbatim tail is simpler and works.

It becomes worth it when conversations are long-running and continuous: an assistant somebody uses daily for a year, a support relationship spanning months, an agent working a task over weeks. There, the transcript is a corpus, and treating it as one is the right call.

What it costs to write

Indexing a transcript is work on the write path: segmenting, rewriting each turn into standalone form, embedding, and inserting. That is a model call or two per exchange, which is a real cost on a chatty product and is paid whether or not anybody ever searches that conversation again.

Run it asynchronously, after the reply has been sent, so it never sits in the user’s latency path. And skip it for conversations under a handful of turns, which are the majority and which will never be retrieved from.

The evaluation is the same as any retrieval

Build a gold set of the form given this message, which earlier exchange should be retrieved, from real conversations where the model failed for lack of history. Measure recall@k on it. Every technique from the retrieval module applies unchanged: hybrid search matters here too, because users refer back to exact strings, names and numbers far more often than they paraphrase.

The common finding: keyword search over the transcript outperforms embeddings for the “what did we say about X” query, which is most of the traffic. Run both.