Giving each chunk its own context

The chunk that means nothing alone

Here is a real chunk, taken verbatim from the middle of a policy document:

This does not apply where the item was purchased during a promotional period, or where the customer has already received a replacement under clause 4.

Nothing in that text says what this is, which product line it covers, or which of your four policies clause 4 belongs to. It will not be retrieved for the query it answers, because it shares no distinctive vocabulary with it. And if it is retrieved, the model cannot use it, because the referent is missing.

This is the dominant retrieval failure in long structured documents and it is not fixed by a better embedding model. The information the chunk needs was in the document and got left behind at the split.

Three ways to put the context back, in ascending cost

1. The structural header: free. Carry the document title and heading path into the chunk at ingestion time.

Refund policy > Sale items > Exceptions
This does not apply where the item was purchased during...

Eight to fifteen tokens, no model call, and it fixes a large share of the problem. Embed the chunk with this header so the header influences retrieval, not only reading. This should be your default before you consider anything else.

2. The generated context sentence: one small-model call per chunk. Ask a cheap model to write one or two sentences situating this chunk inside the whole document, and prepend them.

This passage is from the Acme refunds policy, revised March 2026, in the
section on sale items. It lists the exceptions to the 30-day refund window.
This does not apply where the item was purchased during...

Anthropic published results for this technique (contextual embeddings plus contextual keyword indexing) reporting roughly a halving of top-20 retrieval failures on their test corpora, and a larger reduction again when combined with a reranker. Treat the exact figures as vendor-reported on vendor-chosen data, and the direction as well supported: independent teams reproduce a clear improvement on documents with heavy internal cross-reference, and a small one on corpora of short standalone pages.

3. Overlapping windows and parent expansion. Retrieve on small chunks, return the larger surrounding block. This is the technique the chunking lesson covers, and it composes with the other two rather than replacing them.

The cost, and how it collapses

Generating context per chunk sounds expensive: one call per chunk, and a large corpus has millions. The arithmetic changes completely once you use prompt caching.

The pattern: put the whole document in the prompt once, cache it, then iterate over its chunks asking for a context sentence for each. Every call after the first reads the document from cache at a fraction of the input price. On a 50-page document with 120 chunks you pay full price for the document once and a cached rate 119 times, plus roughly 100 output tokens each.

That turns a prohibitive cost into a small one: commonly a dollar or two per thousand pages with a cheap model. It is also entirely an ingestion-time cost, so it adds nothing to request latency, and it is idempotent: re-run it only when the document changes.

What to put in the generated sentence

Specific beats fluent. Ask for:

• What document this is and, if dated, when.
• Where in the document this passage sits.
• What the passage is about, in terms that do not depend on reading it.
• Any referent the passage uses without defining: what “this”, “the product” or “clause 4” refers to.

Ask explicitly for the last one. It is the highest-value item and the one a model omits unless prompted, because a fluent summary naturally reuses the document’s own pronouns.

Keep it to two sentences. The context is paid for on every retrieval of that chunk, forever, and a five-sentence preamble on a 300-token chunk is a poor ratio.

Index it in both places

The augmented text should go into the vector index and the keyword index. Contextual keyword search matters more than people expect: the generated sentence contains the document title and section names, which are exactly the rare terms a user’s query often contains and the chunk body does not.

Store the original chunk text separately from the augmented version. What you show the model, cite from and highlight in a user interface should normally be the original: the generated context is a retrieval aid, and it is model output, which means it can be wrong. Sending it to the model as though it were source text puts a guess in the evidence position.

Measure it before believing it

The experiment is clean because it does not touch anything else: index the corpus twice, once plain and once augmented, and run the same queries with the same k. Report recall@k for both.

The result varies enormously by corpus. Long documents with heavy internal reference gain a lot. A knowledge base of short, standalone, well-titled articles gains almost nothing, because each chunk already carries its context in its own words, and in that case the structural header alone was the right answer and you have just saved yourself the ingestion pipeline.