{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"long-context-instead-of-retrieval","title":"Long context instead of retrieval: the arithmetic of skipping the pipeline","url":"https://addaly.com/learn/building-with-ai/long-context-instead-of-retrieval","position":42,"of":80,"minutes":9,"answer":"A million-token window can replace a retrieval pipeline for a small, stable corpus, and whether it should is decided by arithmetic (tokens per query times queries per day.","keyIdea":"A million-token window can replace a retrieval pipeline for a small, stable corpus, and whether it should is decided by arithmetic (tokens per query times queries per day, with and without prompt caching) plus a measured check of multi-fact accuracy, not by the size of the window."},"body":{"format":"markdown","text":"## The window is now bigger than most corpora\n\nFrontier models accept 200,000 tokens as standard and some accept a million or more. A 300-page book is roughly 100,000 tokens. A company's entire policy handbook, a product's full documentation, a customer's complete contract set, a mid-sized codebase: each of these fits in one request. When it fits, the question is legitimate: why build a retrieval pipeline at all?\n\nSometimes you should not. This lesson is the arithmetic for deciding.\n\n## The cost of sending everything\n\nSuppose the corpus is 100,000 tokens and you place all of it in every request. At an input price of $3 per million tokens, each question costs $0.30 before the model has written a word. A thousand questions a day is $300 a day, about $9,000 a month.\n\nA retrieval pipeline sending five chunks of 400 tokens plus a modest system prompt spends perhaps 2,500 input tokens per question: under a cent. Long context, uncached, is fifty times the cost per query.\n\nCaching changes the picture. The prompt-caching lesson in the second module explained the rule: a stable prefix, identical across requests, is read from cache at a large discount, typically around a tenth of the normal input price. Put the corpus first, the question last, and after the first request each question costs about $0.03 of cached input rather than $0.30. That is still several times the retrieval cost, but it is in the range where you may prefer to pay it rather than build and maintain a pipeline. The catch is the cache lifetime: entries expire after minutes of inactivity on most providers, so a corpus queried a few times an hour is re-cached repeatedly, and a corpus queried every few seconds stays warm. Your traffic shape decides how much of the discount you actually get.\n\n## Quality\n\nNeedle-in-a-haystack tests (hide one sentence in a long document and ask for it) are now passed almost perfectly by current models. That is not the same as your task. The harder measurement is multi-fact reasoning: questions whose answer requires combining details from three places in a long context. On those, published evaluations show accuracy declining as the context grows, sometimes steeply past a few hundred thousand tokens, and the middle of the context is used less reliably than the ends. The models are far better at this than they were, and they are not as accurate on a hundred thousand tokens as on five thousand relevant ones. Run your evaluation set both ways before believing either side.\n\n## Latency\n\nTime to first token grows with the input, roughly linearly. A hundred thousand uncached tokens means several seconds of silence before the answer starts, which a user reads as broken. With a warm cache the prefix is skipped and the response begins in well under a second. So long context is only comfortable interactively when the cache is warm, which again depends on traffic.\n\n## The decision\n\nPrefer long context when the corpus is under a couple of hundred thousand tokens, changes rarely, is queried often enough to stay cached, and the questions need synthesis across the whole of it: \"what is inconsistent between these three contracts?\" is a question retrieval answers badly and a full window answers well. Prefer it also when the corpus is per user: one person's documents, loaded for one session, is exactly the shape a window suits, and no shared index is needed.\n\nPrefer retrieval when the corpus is large or grows, when it changes often, when different users may see different documents (permissions in a single stuffed prompt are impossible) when query volume is high, and when answers must cite a precise location.\n\n## The middle path\n\nRetrieve documents, not chunks. Use the pipeline to choose the three or four documents that matter, then place those whole into the context (twenty or thirty thousand tokens) rather than five fragments. The model reads coherent text with its headings and tables intact, citations point at documents, and the cost sits between the two extremes. For many products this is the best of the three options and the least discussed.\n\n## A worked example\n\nA support assistant over a 400-page manual, about 150,000 tokens, answering 2,000 questions a day.\n\n- Everything in context, uncached: 300 million input tokens a day, roughly $900 a day at $3 per million. Not viable.\n- Everything in context, cached, traffic steady enough to stay warm: about a tenth of that, $90 a day, with no pipeline to build and full-manual reasoning.\n- Retrieval with five chunks: about 5 million input tokens a day, $15, plus the engineering and upkeep of the index, and answers that occasionally miss a fact that was two sections away.\n- Retrieve the two relevant chapters whole, about 25,000 tokens: $150 a day, with the reasoning quality of long context on the part that matters.\n\nNone of those is the right answer in general. Each is the right answer for a particular product, and the numbers are what let you say which.\n\n> Figure: A 400-page manual, 2,000 questions a day. Bars (dollars a day at $3 per million input tokens): Whole manual in context, uncached 900; Two relevant chapters, whole 150; Whole manual in context, cache warm 90; Five retrieved chunks 15. None of these is the right answer in general. The cached figure assumes traffic steady enough to keep a five-minute window warm; at a few questions an hour the prefix is re-paid each time and the top bar is what you get. Retrieval is cheapest and occasionally misses a fact two sections away."},"check":{"question":"A team places a 100,000-token corpus in every request, with the question at the end, and pays $0.30 per query in input tokens. What would bring that closest to $0.03 without changing the corpus?","options":[{"text":"Move the question to the start of the prompt so that the model reads it first and can skip over the sections of the corpus that are irrelevant to it.","correct":false,"tempting":"Models do not skip input, and putting the varying text first breaks the stable prefix a cache depends on."},{"text":"Keep the corpus as an identical prefix and rely on prompt caching, with traffic frequent enough to keep the cache warm.","correct":true},{"text":"Switch to a model with a larger context window, since larger windows are priced lower per token.","correct":false,"tempting":"Window size and price per token are unrelated; the cost comes from tokens sent, not tokens allowed."},{"text":"Compress the corpus with a summarisation pass so that the model receives a tenth of the tokens.","correct":false,"tempting":"Reduces cost but changes the corpus; the question said without changing it, and summaries lose the details the questions need."}]},"prev":{"slug":"permissions-freshness-and-metadata","title":"Permissions, freshness and metadata: the boring half that leaks data","url":"https://addaly.com/learn/building-with-ai/permissions-freshness-and-metadata"},"next":{"slug":"where-the-conversation-lives","title":"Where the conversation lives","url":"https://addaly.com/learn/building-with-ai/where-the-conversation-lives"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Long context instead of retrieval: the arithmetic of skipping the pipeline', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/long-context-instead-of-retrieval (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Long context instead of retrieval: the arithmetic of skipping the pipeline","courseTitle":"Building with AI","module":{"index":4,"slug":"retrieval-and-the-knowledge-problem","title":"Retrieval and the knowledge problem"},"answer":null,"keyIdea":"A million-token window can replace a retrieval pipeline for a small, stable corpus, and whether it should is decided by arithmetic (tokens per query times queries per day, with and without prompt caching) plus a measured check of multi-fact accuracy, not by the size of the window.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"dollars a day at $3 per million input tokens","title":"A 400-page manual, 2,000 questions a day","series":[{"label":"Whole manual in context, uncached","value":900},{"label":"Two relevant chapters, whole","value":150},{"label":"Whole manual in context, cache warm","value":90},{"label":"Five retrieved chunks","value":15,"highlight":true}],"caption":"None of these is the right answer in general. The cached figure assumes traffic steady enough to keep a five-minute window warm; at a few questions an hour the prefix is re-paid each time and the top bar is what you get. Retrieval is cheapest and occasionally misses a fact two sections away."},"alt":"Figure: A 400-page manual, 2,000 questions a day. Bars (dollars a day at $3 per million input tokens): Whole manual in context, uncached 900; Two relevant chapters, whole 150; Whole manual in context, cache warm 90; Five retrieved chunks 15. None of these is the right answer in general. The cached figure assumes traffic steady enough to keep a five-minute window warm; at a few questions an hour the prefix is re-paid each time and the top bar is what you get. Retrieval is cheapest and occasionally misses a fact two sections away."}},"blocks":[{"t":"h2","id":"the-window-is-now-bigger-than-most-corpora","runs":[{"text":"The window is now bigger than most corpora"}]},{"t":"p","runs":[{"text":"Frontier models accept 200,000 tokens as standard and some accept a million or more. A 300-page book is roughly 100,000 tokens. A company’s entire policy handbook, a product’s full documentation, a customer’s complete contract set, a mid-sized codebase: each of these fits in one request. When it fits, the question is legitimate: why build a retrieval pipeline at all?"}]},{"t":"p","runs":[{"text":"Sometimes you should not. This lesson is the arithmetic for deciding."}]},{"t":"h2","id":"the-cost-of-sending-everything","runs":[{"text":"The cost of sending everything"}]},{"t":"p","runs":[{"text":"Suppose the corpus is 100,000 tokens and you place all of it in every request. At an input price of $3 per million tokens, each question costs $0.30 before the model has written a word. A thousand questions a day is $300 a day, about $9,000 a month."}]},{"t":"p","runs":[{"text":"A retrieval pipeline sending five chunks of 400 tokens plus a modest system prompt spends perhaps 2,500 input tokens per question: under a cent. Long context, uncached, is fifty times the cost per query."}]},{"t":"p","runs":[{"text":"Caching changes the picture. The prompt-caching lesson in the second module explained the rule: a stable prefix, identical across requests, is read from cache at a large discount, typically around a tenth of the normal input price. Put the corpus first, the question last, and after the first request each question costs about $0.03 of cached input rather than $0.30. That is still several times the retrieval cost, but it is in the range where you may prefer to pay it rather than build and maintain a pipeline. The catch is the cache lifetime: entries expire after minutes of inactivity on most providers, so a corpus queried a few times an hour is re-cached repeatedly, and a corpus queried every few seconds stays warm. Your traffic shape decides how much of the discount you actually get."}]},{"t":"h2","id":"quality","runs":[{"text":"Quality"}]},{"t":"p","runs":[{"text":"Needle-in-a-haystack tests (hide one sentence in a long document and ask for it) are now passed almost perfectly by current models. That is not the same as your task. The harder measurement is multi-fact reasoning: questions whose answer requires combining details from three places in a long context. On those, published evaluations show accuracy declining as the context grows, sometimes steeply past a few hundred thousand tokens, and the middle of the context is used less reliably than the ends. The models are far better at this than they were, and they are not as accurate on a hundred thousand tokens as on five thousand relevant ones. Run your evaluation set both ways before believing either side."}]},{"t":"h2","id":"latency","runs":[{"text":"Latency"}]},{"t":"p","runs":[{"text":"Time to first token grows with the input, roughly linearly. A hundred thousand uncached tokens means several seconds of silence before the answer starts, which a user reads as broken. With a warm cache the prefix is skipped and the response begins in well under a second. So long context is only comfortable interactively when the cache is warm, which again depends on traffic."}]},{"t":"h2","id":"the-decision","runs":[{"text":"The decision"}]},{"t":"p","runs":[{"text":"Prefer long context when the corpus is under a couple of hundred thousand tokens, changes rarely, is queried often enough to stay cached, and the questions need synthesis across the whole of it: “what is inconsistent between these three contracts?” is a question retrieval answers badly and a full window answers well. Prefer it also when the corpus is per user: one person’s documents, loaded for one session, is exactly the shape a window suits, and no shared index is needed."}]},{"t":"p","runs":[{"text":"Prefer retrieval when the corpus is large or grows, when it changes often, when different users may see different documents (permissions in a single stuffed prompt are impossible) when query volume is high, and when answers must cite a precise location."}]},{"t":"h2","id":"the-middle-path","runs":[{"text":"The middle path"}]},{"t":"p","runs":[{"text":"Retrieve documents, not chunks. Use the pipeline to choose the three or four documents that matter, then place those whole into the context (twenty or thirty thousand tokens) rather than five fragments. The model reads coherent text with its headings and tables intact, citations point at documents, and the cost sits between the two extremes. For many products this is the best of the three options and the least discussed."}]},{"t":"h2","id":"a-worked-example","runs":[{"text":"A worked example"}]},{"t":"p","runs":[{"text":"A support assistant over a 400-page manual, about 150,000 tokens, answering 2,000 questions a day."}]},{"t":"ul","items":[[{"text":"Everything in context, uncached: 300 million input tokens a day, roughly $900 a day at $3 per million. Not viable."}],[{"text":"Everything in context, cached, traffic steady enough to stay warm: about a tenth of that, $90 a day, with no pipeline to build and full-manual reasoning."}],[{"text":"Retrieval with five chunks: about 5 million input tokens a day, $15, plus the engineering and upkeep of the index, and answers that occasionally miss a fact that was two sections away."}],[{"text":"Retrieve the two relevant chapters whole, about 25,000 tokens: $150 a day, with the reasoning quality of long context on the part that matters."}]]},{"t":"p","runs":[{"text":"None of those is the right answer in general. Each is the right answer for a particular product, and the numbers are what let you say which."}]}],"check":{"question":[{"text":"A team places a 100,000-token corpus in every request, with the question at the end, and pays $0.30 per query in input tokens. What would bring that closest to $0.03 without changing the corpus?"}],"options":[{"text":[{"text":"Move the question to the start of the prompt so that the model reads it first and can skip over the sections of the corpus that are irrelevant to it."}],"correct":false,"tempting":[{"text":"Models do not skip input, and putting the varying text first breaks the stable prefix a cache depends on."}]},{"text":[{"text":"Keep the corpus as an identical prefix and rely on prompt caching, with traffic frequent enough to keep the cache warm."}],"correct":true,"tempting":null},{"text":[{"text":"Switch to a model with a larger context window, since larger windows are priced lower per token."}],"correct":false,"tempting":[{"text":"Window size and price per token are unrelated; the cost comes from tokens sent, not tokens allowed."}]},{"text":[{"text":"Compress the corpus with a summarisation pass so that the model receives a tenth of the tokens."}],"correct":false,"tempting":[{"text":"Reduces cost but changes the corpus; the question said without changing it, and summaries lose the details the questions need."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"permissions-freshness-and-metadata","title":"Permissions, freshness and metadata: the boring half that leaks data","minutes":9},"next":{"slug":"where-the-conversation-lives","title":"Where the conversation lives","minutes":9}}}