The claim that keeps returning
Every time context windows grow, someone declares retrieval obsolete. It happened at 32k, at 200k, at 1M. It has not happened, and it is worth being precise about why, because the reasons tell you when long context does win.
What long context is actually good at
Global questions. “What changed in tone across these 40 board minutes?” has no retrievable answer. There is no chunk that contains it. Retrieval can only return passages; if the answer is a property of the whole, you need the whole.
Small corpora. A 60,000-token codebase, a contract, one patient’s record, a single company’s filings for a year. Below roughly 100k tokens, building a retrieval pipeline costs more engineering than it saves.
Cached and repeated. If the same 80k-token document is queried 500 times a day, prefix caching drops the marginal cost by something like 90% and the economics flip. Lesson 6 covers this.
Reasoning that needs the joins. Cross-referencing clause 4.2 against annex C against a definitions section is exactly where top-k retrieval drops one of the three pieces.
What retrieval is actually good at
Scale. A support corpus of 40 million tokens does not fit in any window at any price. This is the unglamorous, decisive reason retrieval persists.
Per-request economics. Six chunks of 500 tokens is 3,000 tokens. At $3 per million that is $0.009. A 1M-token call at the same rate is $3.00: 333 times more, per question. Embedding the corpus once costs about $0.02 per million tokens with common embedding models.
Freshness. A price changed an hour ago. Reindex one document, or rebuild an 800k-token prompt.
Permissions. This is underrated and often decisive. A finance user may see the payroll folder; a contractor may not. Retrieval filters by ACL at query time. A long-context prompt containing everything has already leaked, and no instruction fixes that.
Citations. Retrieval hands you a document id per passage. You can render a link the user can check. Long context gives you an assertion.
The version that is actually dying
Not retrieval. The naive 2023 pipeline: fixed 512-token chunks, one dense embedding model, cosine similarity, top 5, no reranking. That is being replaced, and it should be.
What replaced it:
Hybrid search. Dense embeddings handle paraphrase: “my payment bounced” finding a page titled “declined transactions”. They are poor at rare literal strings, because an embedding is a lossy compression and a part number carries almost no semantic signal. Ask a support bot about error CU-4102 or part MX-880-B and dense-only retrieval misses. BM25 matches it exactly. Run both, fuse the rankings (reciprocal rank fusion is four lines of code and works), and you stop losing identifiers.
Reranking. Retrieve 50 candidates cheaply, then score each against the query with a cross-encoder that reads query and passage together. It is slower per item and far more accurate, and 50 candidates is a small enough set to afford it.
Retrieval that feeds long context. These are not opposites. Retrieve 40 chunks instead of 5, rerank to 15, spend 10k tokens instead of 2k. Bigger windows made retrieval budgets more generous rather than making retrieval unnecessary.
How to choose
| Situation | Take |
|---|---|
| Corpus under ~100k tokens, stable | Long context. Cache the prefix. |
| Corpus over ~1M tokens | Retrieval. No real choice. |
| Query names an exact identifier | Hybrid, with BM25 carrying it |
| Answer is a property of the whole | Long context, or a map-reduce pass |
| Per-user permissions | Retrieval, filtered at query time |
| Content changes hourly | Retrieval |
| High volume, one shared document | Long context plus caching |
| You need clickable sources | Retrieval |
The measurement that settles arguments
Build 50 real questions with known answers. Run three configurations: retrieval only, full context, and retrieval into a generous context. Record accuracy, p95 latency, and cost per query for each.
Most teams find the third wins on accuracy and the first wins on cost, and then they have an actual decision to make instead of an opinion to defend.
Every lesson here is open. Your progress is recorded once you pass the Data, SQL and Getting to the Answer exam.
Check your understanding
A hardware company’s support assistant uses dense-vector retrieval over 200,000 manual pages. It answers conceptual questions well but fails when customers paste exact fault codes such as E-2214 or part numbers such as MX-880-B, returning pages about vaguely similar components. What is the underlying cause?
Pick the one you would defend. Your answer stays private.
Stuck on something?Ask Sayali, your AI tutor