{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"effective-context-versus-advertised","title":"The advertised window and the usable one","url":"https://addaly.com/learn/context-engineering/effective-context-versus-advertised","position":8,"of":87,"minutes":10,"answer":"The advertised context length is an API limit, not a quality guarantee: measure the accuracy curve on your own material with real distractors, and budget below the point where it bends.","keyIdea":"The advertised context length is an API limit, not a quality guarantee: measure the accuracy curve on your own material with real distractors, and budget below the point where it bends."},"body":{"format":"markdown","text":"## Two numbers, one of which is marketing\n\nA model advertises a context window of 200,000 or a million tokens. That number is a hard limit enforced by the API. It is not a promise about quality, and the gap between the limit and the length at which the model still works well is one of the most consequential unstated facts in the field.\n\nThe pattern, repeated across families and generations: accuracy on tasks that require finding and combining information holds steady for a while and then falls, often long before the advertised limit. How much earlier depends on the model and enormously on the task.\n\n## Why the easy test is misleading\n\nThe test everybody runs is needle-in-a-haystack: hide a distinctive sentence in a long document and ask for it back. Modern models score close to perfect on this at very long lengths, and vendors publish the resulting green grids.\n\nThe test is easy for a mechanical reason. The needle is lexically distinctive: it shares rare tokens with the question, so attention has an easy target. Nothing else in the haystack competes.\n\nChange one thing and the picture changes. Paraphrase the needle so it shares no distinctive words with the question, and scores fall sharply at long lengths on models that scored perfectly on the literal version. Require two needles to be combined, and they fall further. Add distractors that are topically similar but wrong, and they fall further still.\n\nThe harder public suites (the ones that vary needle count, paraphrase, and add distractors) consistently show usable performance somewhere well below the advertised number for tasks with any reasoning in them. The honest summary is a shape rather than a constant: retrieval of a distinctive string degrades late, retrieval of a paraphrased fact degrades earlier, and combining several facts scattered through the window degrades earliest of all.\n\n> Figure: The same model on two tests as the window fills. Line chart of answers correct, per cent against thousands of input tokens in the window: Distinctive needle, no distractors (4, 99) (16, 99) (64, 98) (128, 97) (200, 96); Paraphrased answer among real distractors (4, 88) (16, 81) (64, 69) (128, 57) (200, 48). The easy test stays flat because the needle shares rare tokens with the question and nothing else competes for attention. Paraphrase it and add real distractors and the curve bends a long way below the advertised limit. Budget below the knee, and measure the knee on your own material: fifty questions at four lengths costs a few dollars and settles the argument.\n\n## The mechanisms, so the shape is not a surprise\n\nThree things are going on, and knowing them tells you when to expect trouble.\n\n**Attention dilution.** Softmax attention distributes a fixed total of one across all positions. With 200 candidate positions, a relevant one can hold a large share. With 200,000, the same absolute score is a much smaller share of the total, and weakly relevant material in aggregate can outweigh it.\n\n**Training length distribution.** Long-context ability is largely trained in, and long training documents are scarce and expensive. Much of the very long capability comes from a comparatively short extension phase, so the model has seen far fewer examples of reasoning across 300,000 tokens than across 3,000.\n\n**Position encoding extrapolation.** Techniques that extend a model beyond its trained length (interpolating or scaling rotary embeddings, for instance) work, and they degrade gracefully rather than cliff-edge. Graceful degradation is still degradation.\n\n## What this means for your budget\n\nDo not plan a system around the advertised number. Plan it around a measured one.\n\nThe measurement is not hard and you should run it on your own material, because the answer depends on your task far more than on any published average:\n\n1. Take 50 real questions whose answers you know are in one specific passage.\n2. Build contexts at several lengths (4k, 16k, 64k, 128k) where the true passage is present and the rest is filled with your own real but irrelevant material, not lorem ipsum. Distractors from the same domain are the point.\n3. Place the true passage at a random position each time, so you are measuring length and not position.\n4. Score exact-match or a simple containment check on the answer.\n5. Plot accuracy against length.\n\nYou now have your model's effective context for your task, expressed as a curve rather than a marketing number. Budget below the knee. The exercise costs a few dollars of inference and settles arguments that otherwise run for months.\n\n## Cost and latency do not degrade: they grow\n\nQuality falls off gradually. The bill does not. Input tokens are charged linearly, and prefill time is roughly linear in input length too, with attention cost growing faster than linearly in the underlying computation even where serving optimisations hide much of it.\n\nSo the trade at 200,000 tokens is: you pay perhaps twenty times the price and several times the time-to-first-token of a 10,000-token request, for a context in which the model is measurably worse at combining facts. That is a bad trade to make by default, and a reasonable one to make deliberately when the alternative is missing the relevant material entirely.\n\n## Where long context wins\n\nNone of this argues against long windows. It argues against filling them without measurement. Long context earns its cost when:\n\n- The corpus is small enough to fit entirely and retrieval would risk missing the one relevant line: a single contract, one codebase module, one patient record.\n- The task needs global structure that chunking destroys: \"is this document internally consistent\", \"summarise the argument\", \"find every place the tone changes\".\n- You are prototyping and want to establish an upper bound on quality before spending a week on a retrieval pipeline. This is an excellent use of a big window and the most underrated one.\n\nThe common thread: use the whole window when the whole thing is relevant, not when you could not decide what was."},"check":{"question":"A model scores 99% on a needle-in-a-haystack test at 500,000 tokens but 61% on a real task at 120,000 tokens. Which explanation fits the mechanism?","options":[{"text":"The real task exceeded the trained context length, while the needle test stayed within it.","correct":false,"tempting":"Gets the direction backwards; the needle test ran at the longer length, so trained length cannot explain the gap."},{"text":"The needle shares rare tokens with the question and has no competitors, whereas the real task requires combining paraphrased facts among similar distractors.","correct":true},{"text":"Needle tests use synthetic filler that compresses better, so the effective token count was much lower.","correct":false,"tempting":"Imagines compression inside the window; filler is tokenised and attended like any other text regardless of how repetitive it is."},{"text":"The real task included a system prompt, which competes with retrieval for attention and invalidates the comparison.","correct":false,"tempting":"Blames a small fixed block for a large gap; a system prompt is a tiny share of a 120,000-token window."}]},"prev":{"slug":"the-budget-ledger","title":"Writing the budget down","url":"https://addaly.com/learn/context-engineering/the-budget-ledger"},"next":{"slug":"context-assembly-as-a-function","title":"Making the assembler testable","url":"https://addaly.com/learn/context-engineering/context-assembly-as-a-function"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The advertised window and the usable one', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/effective-context-versus-advertised (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The advertised window and the usable one","courseTitle":"Context Engineering","module":{"index":1,"slug":"the-window-as-a-surface","title":"The window as an engineering surface"},"answer":null,"keyIdea":"The advertised context length is an API limit, not a quality guarantee: measure the accuracy curve on your own material with real distractors, and budget below the point where it bends.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"The same model on two tests as the window fills","series":[{"label":"Distinctive needle, no distractors","points":[[4,99],[16,99],[64,98],[128,97],[200,96]]},{"label":"Paraphrased answer among real distractors","points":[[4,88],[16,81],[64,69],[128,57],[200,48]]}],"xLabel":"thousands of input tokens in the window","yLabel":"answers correct, per cent","caption":"The easy test stays flat because the needle shares rare tokens with the question and nothing else competes for attention. Paraphrase it and add real distractors and the curve bends a long way below the advertised limit. Budget below the knee, and measure the knee on your own material: fifty questions at four lengths costs a few dollars and settles the argument."},"alt":"Figure: The same model on two tests as the window fills. Line chart of answers correct, per cent against thousands of input tokens in the window: Distinctive needle, no distractors (4, 99) (16, 99) (64, 98) (128, 97) (200, 96); Paraphrased answer among real distractors (4, 88) (16, 81) (64, 69) (128, 57) (200, 48). The easy test stays flat because the needle shares rare tokens with the question and nothing else competes for attention. Paraphrase it and add real distractors and the curve bends a long way below the advertised limit. Budget below the knee, and measure the knee on your own material: fifty questions at four lengths costs a few dollars and settles the argument."}},"blocks":[{"t":"h2","id":"two-numbers-one-of-which-is-marketing","runs":[{"text":"Two numbers, one of which is marketing"}]},{"t":"p","runs":[{"text":"A model advertises a context window of 200,000 or a million tokens. That number is a hard limit enforced by the API. It is not a promise about quality, and the gap between the limit and the length at which the model still works well is one of the most consequential unstated facts in the field."}]},{"t":"p","runs":[{"text":"The pattern, repeated across families and generations: accuracy on tasks that require finding and combining information holds steady for a while and then falls, often long before the advertised limit. How much earlier depends on the model and enormously on the task."}]},{"t":"h2","id":"why-the-easy-test-is-misleading","runs":[{"text":"Why the easy test is misleading"}]},{"t":"p","runs":[{"text":"The test everybody runs is needle-in-a-haystack: hide a distinctive sentence in a long document and ask for it back. Modern models score close to perfect on this at very long lengths, and vendors publish the resulting green grids."}]},{"t":"p","runs":[{"text":"The test is easy for a mechanical reason. The needle is lexically distinctive: it shares rare tokens with the question, so attention has an easy target. Nothing else in the haystack competes."}]},{"t":"p","runs":[{"text":"Change one thing and the picture changes. Paraphrase the needle so it shares no distinctive words with the question, and scores fall sharply at long lengths on models that scored perfectly on the literal version. Require two needles to be combined, and they fall further. Add distractors that are topically similar but wrong, and they fall further still."}]},{"t":"p","runs":[{"text":"The harder public suites (the ones that vary needle count, paraphrase, and add distractors) consistently show usable performance somewhere well below the advertised number for tasks with any reasoning in them. The honest summary is a shape rather than a constant: retrieval of a distinctive string degrades late, retrieval of a paraphrased fact degrades earlier, and combining several facts scattered through the window degrades earliest of all."}]},{"t":"h2","id":"the-mechanisms-so-the-shape-is-not-a-surprise","runs":[{"text":"The mechanisms, so the shape is not a surprise"}]},{"t":"p","runs":[{"text":"Three things are going on, and knowing them tells you when to expect trouble."}]},{"t":"p","runs":[{"text":"Attention dilution.","b":true},{"text":" Softmax attention distributes a fixed total of one across all positions. With 200 candidate positions, a relevant one can hold a large share. With 200,000, the same absolute score is a much smaller share of the total, and weakly relevant material in aggregate can outweigh it."}]},{"t":"p","runs":[{"text":"Training length distribution.","b":true},{"text":" Long-context ability is largely trained in, and long training documents are scarce and expensive. Much of the very long capability comes from a comparatively short extension phase, so the model has seen far fewer examples of reasoning across 300,000 tokens than across 3,000."}]},{"t":"p","runs":[{"text":"Position encoding extrapolation.","b":true},{"text":" Techniques that extend a model beyond its trained length (interpolating or scaling rotary embeddings, for instance) work, and they degrade gracefully rather than cliff-edge. Graceful degradation is still degradation."}]},{"t":"h2","id":"what-this-means-for-your-budget","runs":[{"text":"What this means for your budget"}]},{"t":"p","runs":[{"text":"Do not plan a system around the advertised number. Plan it around a measured one."}]},{"t":"p","runs":[{"text":"The measurement is not hard and you should run it on your own material, because the answer depends on your task far more than on any published average:"}]},{"t":"ol","start":1,"items":[[{"text":"Take 50 real questions whose answers you know are in one specific passage."}],[{"text":"Build contexts at several lengths (4k, 16k, 64k, 128k) where the true passage is present and the rest is filled with your own real but irrelevant material, not lorem ipsum. Distractors from the same domain are the point."}],[{"text":"Place the true passage at a random position each time, so you are measuring length and not position."}],[{"text":"Score exact-match or a simple containment check on the answer."}],[{"text":"Plot accuracy against length."}]]},{"t":"p","runs":[{"text":"You now have your model’s effective context for your task, expressed as a curve rather than a marketing number. Budget below the knee. The exercise costs a few dollars of inference and settles arguments that otherwise run for months."}]},{"t":"h2","id":"cost-and-latency-do-not-degrade-they-grow","runs":[{"text":"Cost and latency do not degrade: they grow"}]},{"t":"p","runs":[{"text":"Quality falls off gradually. The bill does not. Input tokens are charged linearly, and prefill time is roughly linear in input length too, with attention cost growing faster than linearly in the underlying computation even where serving optimisations hide much of it."}]},{"t":"p","runs":[{"text":"So the trade at 200,000 tokens is: you pay perhaps twenty times the price and several times the time-to-first-token of a 10,000-token request, for a context in which the model is measurably worse at combining facts. That is a bad trade to make by default, and a reasonable one to make deliberately when the alternative is missing the relevant material entirely."}]},{"t":"h2","id":"where-long-context-wins","runs":[{"text":"Where long context wins"}]},{"t":"p","runs":[{"text":"None of this argues against long windows. It argues against filling them without measurement. Long context earns its cost when:"}]},{"t":"ul","items":[[{"text":"The corpus is small enough to fit entirely and retrieval would risk missing the one relevant line: a single contract, one codebase module, one patient record."}],[{"text":"The task needs global structure that chunking destroys: “is this document internally consistent”, “summarise the argument”, “find every place the tone changes”."}],[{"text":"You are prototyping and want to establish an upper bound on quality before spending a week on a retrieval pipeline. This is an excellent use of a big window and the most underrated one."}]]},{"t":"p","runs":[{"text":"The common thread: use the whole window when the whole thing is relevant, not when you could not decide what was."}]}],"check":{"question":[{"text":"A model scores 99% on a needle-in-a-haystack test at 500,000 tokens but 61% on a real task at 120,000 tokens. Which explanation fits the mechanism?"}],"options":[{"text":[{"text":"The real task exceeded the trained context length, while the needle test stayed within it."}],"correct":false,"tempting":[{"text":"Gets the direction backwards; the needle test ran at the longer length, so trained length cannot explain the gap."}]},{"text":[{"text":"The needle shares rare tokens with the question and has no competitors, whereas the real task requires combining paraphrased facts among similar distractors."}],"correct":true,"tempting":null},{"text":[{"text":"Needle tests use synthetic filler that compresses better, so the effective token count was much lower."}],"correct":false,"tempting":[{"text":"Imagines compression inside the window; filler is tokenised and attended like any other text regardless of how repetitive it is."}]},{"text":[{"text":"The real task included a system prompt, which competes with retrieval for attention and invalidates the comparison."}],"correct":false,"tempting":[{"text":"Blames a small fixed block for a large gap; a system prompt is a tiny share of a 120,000-token window."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"the-budget-ledger","title":"Writing the budget down","minutes":8},"next":{"slug":"context-assembly-as-a-function","title":"Making the assembler testable","minutes":9}}}