Long context instead of retrieval: the arithmetic of skipping the pipeline

The window is now bigger than most corpora

Frontier models accept 200,000 tokens as standard and some accept a million or more. A 300-page book is roughly 100,000 tokens. A company’s entire policy handbook, a product’s full documentation, a customer’s complete contract set, a mid-sized codebase: each of these fits in one request. When it fits, the question is legitimate: why build a retrieval pipeline at all?

Sometimes you should not. This lesson is the arithmetic for deciding.

The cost of sending everything

Suppose the corpus is 100,000 tokens and you place all of it in every request. At an input price of $3 per million tokens, each question costs $0.30 before the model has written a word. A thousand questions a day is $300 a day, about $9,000 a month.

A retrieval pipeline sending five chunks of 400 tokens plus a modest system prompt spends perhaps 2,500 input tokens per question: under a cent. Long context, uncached, is fifty times the cost per query.

Caching changes the picture. The prompt-caching lesson in the second module explained the rule: a stable prefix, identical across requests, is read from cache at a large discount, typically around a tenth of the normal input price. Put the corpus first, the question last, and after the first request each question costs about $0.03 of cached input rather than $0.30. That is still several times the retrieval cost, but it is in the range where you may prefer to pay it rather than build and maintain a pipeline. The catch is the cache lifetime: entries expire after minutes of inactivity on most providers, so a corpus queried a few times an hour is re-cached repeatedly, and a corpus queried every few seconds stays warm. Your traffic shape decides how much of the discount you actually get.

Quality

Needle-in-a-haystack tests (hide one sentence in a long document and ask for it) are now passed almost perfectly by current models. That is not the same as your task. The harder measurement is multi-fact reasoning: questions whose answer requires combining details from three places in a long context. On those, published evaluations show accuracy declining as the context grows, sometimes steeply past a few hundred thousand tokens, and the middle of the context is used less reliably than the ends. The models are far better at this than they were, and they are not as accurate on a hundred thousand tokens as on five thousand relevant ones. Run your evaluation set both ways before believing either side.

Latency

Time to first token grows with the input, roughly linearly. A hundred thousand uncached tokens means several seconds of silence before the answer starts, which a user reads as broken. With a warm cache the prefix is skipped and the response begins in well under a second. So long context is only comfortable interactively when the cache is warm, which again depends on traffic.

The decision

Prefer long context when the corpus is under a couple of hundred thousand tokens, changes rarely, is queried often enough to stay cached, and the questions need synthesis across the whole of it: “what is inconsistent between these three contracts?” is a question retrieval answers badly and a full window answers well. Prefer it also when the corpus is per user: one person’s documents, loaded for one session, is exactly the shape a window suits, and no shared index is needed.

Prefer retrieval when the corpus is large or grows, when it changes often, when different users may see different documents (permissions in a single stuffed prompt are impossible) when query volume is high, and when answers must cite a precise location.

The middle path

Retrieve documents, not chunks. Use the pipeline to choose the three or four documents that matter, then place those whole into the context (twenty or thirty thousand tokens) rather than five fragments. The model reads coherent text with its headings and tables intact, citations point at documents, and the cost sits between the two extremes. For many products this is the best of the three options and the least discussed.

A worked example

A support assistant over a 400-page manual, about 150,000 tokens, answering 2,000 questions a day.

• Everything in context, uncached: 300 million input tokens a day, roughly $900 a day at $3 per million. Not viable.
• Everything in context, cached, traffic steady enough to stay warm: about a tenth of that, $90 a day, with no pipeline to build and full-manual reasoning.
• Retrieval with five chunks: about 5 million input tokens a day, $15, plus the engineering and upkeep of the index, and answers that occasionally miss a fact that was two sections away.
• Retrieve the two relevant chapters whole, about 25,000 tokens: $150 a day, with the reasoning quality of long context on the part that matters.

None of those is the right answer in general. Each is the right answer for a particular product, and the numbers are what let you say which.