{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"retrieval-versus-long-context","title":"Retrieval versus long context: the real trade","url":"https://addaly.com/learn/context-engineering/retrieval-versus-long-context","position":11,"of":87,"minutes":9,"answer":"Corpus size, permissions and freshness pick the architecture; window size only changes the budget.","keyIdea":"Corpus size, permissions and freshness pick the architecture; window size only changes the budget."},"body":{"format":"markdown","text":"## The claim that keeps returning\n\nEvery time context windows grow, someone declares retrieval obsolete. It happened at 32k, at 200k, at 1M. It has not happened, and it is worth being precise about why, because the reasons tell you when long context does win.\n\n## What long context is actually good at\n\n**Global questions.** \"What changed in tone across these 40 board minutes?\" has no retrievable answer. There is no chunk that contains it. Retrieval can only return passages; if the answer is a property of the whole, you need the whole.\n\n**Small corpora.** A 60,000-token codebase, a contract, one patient's record, a single company's filings for a year. Below roughly 100k tokens, building a retrieval pipeline costs more engineering than it saves.\n\n**Cached and repeated.** If the same 80k-token document is queried 500 times a day, prefix caching drops the marginal cost by something like 90% and the economics flip. Lesson 6 covers this.\n\n**Reasoning that needs the joins.** Cross-referencing clause 4.2 against annex C against a definitions section is exactly where top-k retrieval drops one of the three pieces.\n\n## What retrieval is actually good at\n\n**Scale.** A support corpus of 40 million tokens does not fit in any window at any price. This is the unglamorous, decisive reason retrieval persists.\n\n**Per-request economics.** Six chunks of 500 tokens is 3,000 tokens. At $3 per million that is $0.009. A 1M-token call at the same rate is $3.00: 333 times more, per question. Embedding the corpus once costs about $0.02 per million tokens with common embedding models.\n\n**Freshness.** A price changed an hour ago. Reindex one document, or rebuild an 800k-token prompt.\n\n**Permissions.** This is underrated and often decisive. A finance user may see the payroll folder; a contractor may not. Retrieval filters by ACL at query time. A long-context prompt containing everything has already leaked, and no instruction fixes that.\n\n**Citations.** Retrieval hands you a document id per passage. You can render a link the user can check. Long context gives you an assertion.\n\n> Figure: What actually picks the architecture. A two by two grid, rows Corpus under 100k and Corpus over 10M, columns All readers equal and Per-user permissions: Corpus under 100k and All readers equal: Long context; Corpus under 100k and Per-user permissions: Long context, per user; Corpus over 10M and All readers equal: Retrieval; Corpus over 10M and Per-user permissions: Retrieval, filtered. Window size only changes the budget. Corpus size, permissions and freshness choose the shape, and a permission boundary settles it fastest: a long-context prompt containing everything the user might be allowed to see is a data leak waiting for the wrong prompt.\n\n## The version that is actually dying\n\nNot retrieval. The naive 2023 pipeline: fixed 512-token chunks, one dense embedding model, cosine similarity, top 5, no reranking. That is being replaced, and it should be.\n\nWhat replaced it:\n\n**Hybrid search.** Dense embeddings handle paraphrase: \"my payment bounced\" finding a page titled \"declined transactions\". They are poor at rare literal strings, because an embedding is a lossy compression and a part number carries almost no semantic signal. Ask a support bot about error `CU-4102` or part `MX-880-B` and dense-only retrieval misses. BM25 matches it exactly. Run both, fuse the rankings (reciprocal rank fusion is four lines of code and works), and you stop losing identifiers.\n\n**Reranking.** Retrieve 50 candidates cheaply, then score each against the query with a cross-encoder that reads query and passage together. It is slower per item and far more accurate, and 50 candidates is a small enough set to afford it.\n\n**Retrieval that feeds long context.** These are not opposites. Retrieve 40 chunks instead of 5, rerank to 15, spend 10k tokens instead of 2k. Bigger windows made retrieval budgets more generous rather than making retrieval unnecessary.\n\n## How to choose\n\n| Situation | Take |\n|---|---|\n| Corpus under ~100k tokens, stable | Long context. Cache the prefix. |\n| Corpus over ~1M tokens | Retrieval. No real choice. |\n| Query names an exact identifier | Hybrid, with BM25 carrying it |\n| Answer is a property of the whole | Long context, or a map-reduce pass |\n| Per-user permissions | Retrieval, filtered at query time |\n| Content changes hourly | Retrieval |\n| High volume, one shared document | Long context plus caching |\n| You need clickable sources | Retrieval |\n\n## The measurement that settles arguments\n\nBuild 50 real questions with known answers. Run three configurations: retrieval only, full context, and retrieval into a generous context. Record accuracy, p95 latency, and cost per query for each.\n\nMost teams find the third wins on accuracy and the first wins on cost, and then they have an actual decision to make instead of an opinion to defend."},"check":{"question":"A hardware company's support assistant uses dense-vector retrieval over 200,000 manual pages. It answers conceptual questions well but fails when customers paste exact fault codes such as `E-2214` or part numbers such as `MX-880-B`, returning pages about vaguely similar components. What is the underlying cause?","options":[{"text":"The chunks are too large, so the embedding of a chunk containing the code is dominated by surrounding text.","correct":false,"tempting":"Chunk size genuinely dilutes embeddings, so this explains part of the effect, but shrinking chunks does not make an embedding represent an arbitrary literal string."},{"text":"An embedding compresses meaning, and a rare identifier carries almost no semantic signal to compress, so exact-match lookup has to come from a keyword index.","correct":true},{"text":"The embedding model's tokeniser cannot represent alphanumeric codes, so they are dropped before encoding.","correct":false,"tempting":"Tokenisers do split codes into odd fragments, which sounds like loss, but the fragments are encoded normally rather than discarded."},{"text":"The index needs rebuilding with a larger embedding dimension to hold the extra detail.","correct":false,"tempting":"Higher dimensions do carry more information, so scaling up feels like the fix, but the failure is a mismatch of retrieval mechanism rather than a capacity limit."}]},"prev":{"slug":"what-breaks-when-it-overflows","title":"What actually happens when it does not fit","url":"https://addaly.com/learn/context-engineering/what-breaks-when-it-overflows"},"next":{"slug":"relevance-is-not-similarity","title":"Similar is not the same as relevant","url":"https://addaly.com/learn/context-engineering/relevance-is-not-similarity"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Retrieval versus long context: the real trade', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/retrieval-versus-long-context (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Retrieval versus long context: the real trade","courseTitle":"Context Engineering","module":{"index":2,"slug":"what-earns-a-place","title":"Selection: what earns a place"},"answer":null,"keyIdea":"Corpus size, permissions and freshness pick the architecture; window size only changes the budget.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 2.1","figure":{"cols":["All readers equal","Per-user permissions"],"rows":["Corpus under 100k","Corpus over 10M"],"type":"matrix","cells":[["Long context","Long context, per user"],["Retrieval","Retrieval, filtered"]],"title":"What actually picks the architecture","caption":"Window size only changes the budget. Corpus size, permissions and freshness choose the shape, and a permission boundary settles it fastest: a long-context prompt containing everything the user might be allowed to see is a data leak waiting for the wrong prompt.","cellLabels":[["cache it and query it all day","assemble each window separately"],["no window holds it at any price","filter inside the query, not after"]]},"alt":"Figure: What actually picks the architecture. A two by two grid, rows Corpus under 100k and Corpus over 10M, columns All readers equal and Per-user permissions: Corpus under 100k and All readers equal: Long context; Corpus under 100k and Per-user permissions: Long context, per user; Corpus over 10M and All readers equal: Retrieval; Corpus over 10M and Per-user permissions: Retrieval, filtered. Window size only changes the budget. Corpus size, permissions and freshness choose the shape, and a permission boundary settles it fastest: a long-context prompt containing everything the user might be allowed to see is a data leak waiting for the wrong prompt."}},"blocks":[{"t":"h2","id":"the-claim-that-keeps-returning","runs":[{"text":"The claim that keeps returning"}]},{"t":"p","runs":[{"text":"Every time context windows grow, someone declares retrieval obsolete. It happened at 32k, at 200k, at 1M. It has not happened, and it is worth being precise about why, because the reasons tell you when long context does win."}]},{"t":"h2","id":"what-long-context-is-actually-good-at","runs":[{"text":"What long context is actually good at"}]},{"t":"p","runs":[{"text":"Global questions.","b":true},{"text":" “What changed in tone across these 40 board minutes?” has no retrievable answer. There is no chunk that contains it. Retrieval can only return passages; if the answer is a property of the whole, you need the whole."}]},{"t":"p","runs":[{"text":"Small corpora.","b":true},{"text":" A 60,000-token codebase, a contract, one patient’s record, a single company’s filings for a year. Below roughly 100k tokens, building a retrieval pipeline costs more engineering than it saves."}]},{"t":"p","runs":[{"text":"Cached and repeated.","b":true},{"text":" If the same 80k-token document is queried 500 times a day, prefix caching drops the marginal cost by something like 90% and the economics flip. Lesson 6 covers this."}]},{"t":"p","runs":[{"text":"Reasoning that needs the joins.","b":true},{"text":" Cross-referencing clause 4.2 against annex C against a definitions section is exactly where top-k retrieval drops one of the three pieces."}]},{"t":"h2","id":"what-retrieval-is-actually-good-at","runs":[{"text":"What retrieval is actually good at"}]},{"t":"p","runs":[{"text":"Scale.","b":true},{"text":" A support corpus of 40 million tokens does not fit in any window at any price. This is the unglamorous, decisive reason retrieval persists."}]},{"t":"p","runs":[{"text":"Per-request economics.","b":true},{"text":" Six chunks of 500 tokens is 3,000 tokens. At $3 per million that is $0.009. A 1M-token call at the same rate is $3.00: 333 times more, per question. Embedding the corpus once costs about $0.02 per million tokens with common embedding models."}]},{"t":"p","runs":[{"text":"Freshness.","b":true},{"text":" A price changed an hour ago. Reindex one document, or rebuild an 800k-token prompt."}]},{"t":"p","runs":[{"text":"Permissions.","b":true},{"text":" This is underrated and often decisive. A finance user may see the payroll folder; a contractor may not. Retrieval filters by ACL at query time. A long-context prompt containing everything has already leaked, and no instruction fixes that."}]},{"t":"p","runs":[{"text":"Citations.","b":true},{"text":" Retrieval hands you a document id per passage. You can render a link the user can check. Long context gives you an assertion."}]},{"t":"h2","id":"the-version-that-is-actually-dying","runs":[{"text":"The version that is actually dying"}]},{"t":"p","runs":[{"text":"Not retrieval. The naive 2023 pipeline: fixed 512-token chunks, one dense embedding model, cosine similarity, top 5, no reranking. That is being replaced, and it should be."}]},{"t":"p","runs":[{"text":"What replaced it:"}]},{"t":"p","runs":[{"text":"Hybrid search.","b":true},{"text":" Dense embeddings handle paraphrase: “my payment bounced” finding a page titled “declined transactions”. They are poor at rare literal strings, because an embedding is a lossy compression and a part number carries almost no semantic signal. Ask a support bot about error "},{"text":"CU-4102","code":true},{"text":" or part "},{"text":"MX-880-B","code":true},{"text":" and dense-only retrieval misses. BM25 matches it exactly. Run both, fuse the rankings (reciprocal rank fusion is four lines of code and works), and you stop losing identifiers."}]},{"t":"p","runs":[{"text":"Reranking.","b":true},{"text":" Retrieve 50 candidates cheaply, then score each against the query with a cross-encoder that reads query and passage together. It is slower per item and far more accurate, and 50 candidates is a small enough set to afford it."}]},{"t":"p","runs":[{"text":"Retrieval that feeds long context.","b":true},{"text":" These are not opposites. Retrieve 40 chunks instead of 5, rerank to 15, spend 10k tokens instead of 2k. Bigger windows made retrieval budgets more generous rather than making retrieval unnecessary."}]},{"t":"h2","id":"how-to-choose","runs":[{"text":"How to choose"}]},{"t":"table","head":[[{"text":"Situation"}],[{"text":"Take"}]],"align":["left","left"],"rows":[[[{"text":"Corpus under ~100k tokens, stable"}],[{"text":"Long context. Cache the prefix."}]],[[{"text":"Corpus over ~1M tokens"}],[{"text":"Retrieval. No real choice."}]],[[{"text":"Query names an exact identifier"}],[{"text":"Hybrid, with BM25 carrying it"}]],[[{"text":"Answer is a property of the whole"}],[{"text":"Long context, or a map-reduce pass"}]],[[{"text":"Per-user permissions"}],[{"text":"Retrieval, filtered at query time"}]],[[{"text":"Content changes hourly"}],[{"text":"Retrieval"}]],[[{"text":"High volume, one shared document"}],[{"text":"Long context plus caching"}]],[[{"text":"You need clickable sources"}],[{"text":"Retrieval"}]]]},{"t":"h2","id":"the-measurement-that-settles-arguments","runs":[{"text":"The measurement that settles arguments"}]},{"t":"p","runs":[{"text":"Build 50 real questions with known answers. Run three configurations: retrieval only, full context, and retrieval into a generous context. Record accuracy, p95 latency, and cost per query for each."}]},{"t":"p","runs":[{"text":"Most teams find the third wins on accuracy and the first wins on cost, and then they have an actual decision to make instead of an opinion to defend."}]}],"check":{"question":[{"text":"A hardware company’s support assistant uses dense-vector retrieval over 200,000 manual pages. It answers conceptual questions well but fails when customers paste exact fault codes such as "},{"text":"E-2214","code":true},{"text":" or part numbers such as "},{"text":"MX-880-B","code":true},{"text":", returning pages about vaguely similar components. What is the underlying cause?"}],"options":[{"text":[{"text":"The chunks are too large, so the embedding of a chunk containing the code is dominated by surrounding text."}],"correct":false,"tempting":[{"text":"Chunk size genuinely dilutes embeddings, so this explains part of the effect, but shrinking chunks does not make an embedding represent an arbitrary literal string."}]},{"text":[{"text":"An embedding compresses meaning, and a rare identifier carries almost no semantic signal to compress, so exact-match lookup has to come from a keyword index."}],"correct":true,"tempting":null},{"text":[{"text":"The embedding model’s tokeniser cannot represent alphanumeric codes, so they are dropped before encoding."}],"correct":false,"tempting":[{"text":"Tokenisers do split codes into odd fragments, which sounds like loss, but the fragments are encoded normally rather than discarded."}]},{"text":[{"text":"The index needs rebuilding with a larger embedding dimension to hold the extra detail."}],"correct":false,"tempting":[{"text":"Higher dimensions do carry more information, so scaling up feels like the fix, but the failure is a mismatch of retrieval mechanism rather than a capacity limit."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"what-breaks-when-it-overflows","title":"What actually happens when it does not fit","minutes":8},"next":{"slug":"relevance-is-not-similarity","title":"Similar is not the same as relevant","minutes":9}}}