{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"long-context-in-practice","title":"Long context in practice","url":"https://addaly.com/learn/running-models-yourself/long-context-in-practice","position":52,"of":82,"minutes":9,"answer":"Advertised context is a capacity and not a competence: measure your own working limit with a multi-fact test, and prefer five retrieved passages to a hundred pages even when everything fits.","keyIdea":"Advertised context is a capacity and not a competence: measure your own working limit with a multi-fact test, and prefer five retrieved passages to a hundred pages even when everything fits, because irrelevant text dilutes attention."},"body":{"format":"markdown","text":"## The advertised number is a capacity, not a competence\n\nA model advertised at 128k will accept 128,000 tokens without error. Whether it can use them is a separate question with a much less flattering answer.\n\nThe standard demonstration (hide one sentence in a long document and ask for it) is the easiest possible version of the test. Models pass it well past the point where they fail at anything harder. Treat a passing result on that test as evidence of nothing much.\n\n## What actually degrades\n\nThree distinct failures, and they arrive at different lengths.\n\n**Position effects.** Material at the start and end of a long context is used more reliably than material in the middle. Put the important passage in the middle of forty pages and it is measurably less likely to be found.\n\n**Multi-fact tasks.** Retrieving one fact is far easier than retrieving three from different places, or comparing two, or noticing that a later statement contradicts an earlier one. Models in the 4B-to-32B band typically start failing these somewhere between 8k and 32k, well short of the advertised figure.\n\n**Distractor sensitivity.** Text that resembles the answer but is not the answer pulls the model towards it. A contract with five similar clauses is much harder than a contract with one, at the same length.\n\nA model does not announce any of this. It produces a fluent answer built from whatever it managed to attend to, which is exactly the failure that is hardest to notice.\n\n## Measure your own working limit\n\nTwenty minutes, and the result is worth more than any published number because it uses your documents and your questions.\n\n1. Take a real document at your real length.\n2. Insert three facts you know the answer to, at the start, the middle and the end.\n3. Ask for all three in one question. Then ask a question requiring two of them together.\n4. Repeat at 4k, 8k, 16k, 32k by truncating or padding.\n5. Note where the answers start to go wrong.\n\nThat length is your working limit for this model on this kind of document. It will be shorter than the advertised context and probably shorter than you hoped.\n\n## The cost side\n\nLong context is expensive in three ways simultaneously.\n\n**Memory.** The KV cache grows linearly. At 128k on a modern 8B, the cache is several times the size of the weights.\n\n**Prefill time.** Processing 100,000 tokens takes real seconds even on good hardware, and a minute or more on a CPU. This lands entirely before the first output token.\n\n**Attention cost per token.** Each generated token attends over the whole cache, so generation slows as the context fills. A conversation that started at 50 tokens per second can be at 25 by the time it is deep into a long document.\n\n## Retrieval beats stuffing, almost always\n\nThe instinct is to paste everything and let the model sort it out. Five well-chosen passages beat a hundred pages on quality, cost and speed at the same time, and the quality part surprises people, because it seems as though more information should help.\n\nIt does not, for a mechanical reason: attention is a limited resource spread across the context, and irrelevant material dilutes it. Adding forty pages of unrelated text around the relevant paragraph makes the relevant paragraph harder to use, not easier.\n\nSo the pattern from the retrieval lesson is not a workaround for small context windows. It is the better design even when everything fits.\n\n## Practical management\n\n**Cap conversation history.** Chat interfaces accumulate. By turn twenty you may be sending 20,000 tokens for a one-line question, paying for it in latency on every turn.\n\n**Summarise rather than truncate.** Dropping the oldest turns loses established facts. Replacing them with a short summary keeps them, and costs one extra call. Note that this breaks prefix caching for that conversation, so do it on a schedule rather than every turn.\n\n**Put the question last.** Both because instructions at the end are followed best, and because it keeps the document prefix stable and cacheable.\n\n**Keep the document ordering stable** across queries, for the same caching reason.\n\n**Check the token count you are sending.** The `usage.prompt_tokens` field on every response, against your configured window and against your measured working limit. Two numbers, and they explain most unexpected answers on long inputs."},"check":{"question":"A team's document assistant answers accurately when given a single relevant page, but produces confident wrong answers when given the whole 60-page manual, even though it fits in the context window. What is the mechanism?","options":[{"text":"The KV cache is being quantised to 4 bits at that length to fit, which is degrading long-range recall.","correct":false,"tempting":"Would require a cache setting that changes with input length, which engines do not do; the type is fixed at load."},{"text":"The context exceeds the model's trained length and rope scaling is degrading positional precision.","correct":false,"tempting":"Assumes the window was extended; the premise says the input fits, and the failure occurs well inside typical trained lengths."},{"text":"Attention is spread across everything present, so the relevant passage competes with fifty-nine pages of similar-looking distractors and is used less reliably.","correct":true},{"text":"The document is being truncated silently at the server's configured window, so the relevant section is never seen.","correct":false,"tempting":"Would produce answers missing information rather than confident wrong ones drawn from the wrong section, and is ruled out by the premise that it fits."}]},"prev":{"slug":"tool-calling-locally","title":"Tool calling that survives contact with a small model","url":"https://addaly.com/learn/running-models-yourself/tool-calling-locally"},"next":{"slug":"determinism-and-reproducibility","title":"Why the same seed gives different answers","url":"https://addaly.com/learn/running-models-yourself/determinism-and-reproducibility"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Long context in practice', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/long-context-in-practice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Long context in practice","courseTitle":"Running Models Yourself","module":{"index":6,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"Advertised context is a capacity and not a competence: measure your own working limit with a multi-fact test, and prefer five retrieved passages to a hundred pages even when everything fits, because irrelevant text dilutes attention.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-advertised-number-is-a-capacity-not-a-competence","runs":[{"text":"The advertised number is a capacity, not a competence"}]},{"t":"p","runs":[{"text":"A model advertised at 128k will accept 128,000 tokens without error. Whether it can use them is a separate question with a much less flattering answer."}]},{"t":"p","runs":[{"text":"The standard demonstration (hide one sentence in a long document and ask for it) is the easiest possible version of the test. Models pass it well past the point where they fail at anything harder. Treat a passing result on that test as evidence of nothing much."}]},{"t":"h2","id":"what-actually-degrades","runs":[{"text":"What actually degrades"}]},{"t":"p","runs":[{"text":"Three distinct failures, and they arrive at different lengths."}]},{"t":"p","runs":[{"text":"Position effects.","b":true},{"text":" Material at the start and end of a long context is used more reliably than material in the middle. Put the important passage in the middle of forty pages and it is measurably less likely to be found."}]},{"t":"p","runs":[{"text":"Multi-fact tasks.","b":true},{"text":" Retrieving one fact is far easier than retrieving three from different places, or comparing two, or noticing that a later statement contradicts an earlier one. Models in the 4B-to-32B band typically start failing these somewhere between 8k and 32k, well short of the advertised figure."}]},{"t":"p","runs":[{"text":"Distractor sensitivity.","b":true},{"text":" Text that resembles the answer but is not the answer pulls the model towards it. A contract with five similar clauses is much harder than a contract with one, at the same length."}]},{"t":"p","runs":[{"text":"A model does not announce any of this. It produces a fluent answer built from whatever it managed to attend to, which is exactly the failure that is hardest to notice."}]},{"t":"h2","id":"measure-your-own-working-limit","runs":[{"text":"Measure your own working limit"}]},{"t":"p","runs":[{"text":"Twenty minutes, and the result is worth more than any published number because it uses your documents and your questions."}]},{"t":"ol","start":1,"items":[[{"text":"Take a real document at your real length."}],[{"text":"Insert three facts you know the answer to, at the start, the middle and the end."}],[{"text":"Ask for all three in one question. Then ask a question requiring two of them together."}],[{"text":"Repeat at 4k, 8k, 16k, 32k by truncating or padding."}],[{"text":"Note where the answers start to go wrong."}]]},{"t":"p","runs":[{"text":"That length is your working limit for this model on this kind of document. It will be shorter than the advertised context and probably shorter than you hoped."}]},{"t":"h2","id":"the-cost-side","runs":[{"text":"The cost side"}]},{"t":"p","runs":[{"text":"Long context is expensive in three ways simultaneously."}]},{"t":"p","runs":[{"text":"Memory.","b":true},{"text":" The KV cache grows linearly. At 128k on a modern 8B, the cache is several times the size of the weights."}]},{"t":"p","runs":[{"text":"Prefill time.","b":true},{"text":" Processing 100,000 tokens takes real seconds even on good hardware, and a minute or more on a CPU. This lands entirely before the first output token."}]},{"t":"p","runs":[{"text":"Attention cost per token.","b":true},{"text":" Each generated token attends over the whole cache, so generation slows as the context fills. A conversation that started at 50 tokens per second can be at 25 by the time it is deep into a long document."}]},{"t":"h2","id":"retrieval-beats-stuffing-almost-always","runs":[{"text":"Retrieval beats stuffing, almost always"}]},{"t":"p","runs":[{"text":"The instinct is to paste everything and let the model sort it out. Five well-chosen passages beat a hundred pages on quality, cost and speed at the same time, and the quality part surprises people, because it seems as though more information should help."}]},{"t":"p","runs":[{"text":"It does not, for a mechanical reason: attention is a limited resource spread across the context, and irrelevant material dilutes it. Adding forty pages of unrelated text around the relevant paragraph makes the relevant paragraph harder to use, not easier."}]},{"t":"p","runs":[{"text":"So the pattern from the retrieval lesson is not a workaround for small context windows. It is the better design even when everything fits."}]},{"t":"h2","id":"practical-management","runs":[{"text":"Practical management"}]},{"t":"p","runs":[{"text":"Cap conversation history.","b":true},{"text":" Chat interfaces accumulate. By turn twenty you may be sending 20,000 tokens for a one-line question, paying for it in latency on every turn."}]},{"t":"p","runs":[{"text":"Summarise rather than truncate.","b":true},{"text":" Dropping the oldest turns loses established facts. Replacing them with a short summary keeps them, and costs one extra call. Note that this breaks prefix caching for that conversation, so do it on a schedule rather than every turn."}]},{"t":"p","runs":[{"text":"Put the question last.","b":true},{"text":" Both because instructions at the end are followed best, and because it keeps the document prefix stable and cacheable."}]},{"t":"p","runs":[{"text":"Keep the document ordering stable","b":true},{"text":" across queries, for the same caching reason."}]},{"t":"p","runs":[{"text":"Check the token count you are sending.","b":true},{"text":" The "},{"text":"usage.prompt_tokens","code":true},{"text":" field on every response, against your configured window and against your measured working limit. Two numbers, and they explain most unexpected answers on long inputs."}]}],"check":{"question":[{"text":"A team’s document assistant answers accurately when given a single relevant page, but produces confident wrong answers when given the whole 60-page manual, even though it fits in the context window. What is the mechanism?"}],"options":[{"text":[{"text":"The KV cache is being quantised to 4 bits at that length to fit, which is degrading long-range recall."}],"correct":false,"tempting":[{"text":"Would require a cache setting that changes with input length, which engines do not do; the type is fixed at load."}]},{"text":[{"text":"The context exceeds the model’s trained length and rope scaling is degrading positional precision."}],"correct":false,"tempting":[{"text":"Assumes the window was extended; the premise says the input fits, and the failure occurs well inside typical trained lengths."}]},{"text":[{"text":"Attention is spread across everything present, so the relevant passage competes with fifty-nine pages of similar-looking distractors and is used less reliably."}],"correct":true,"tempting":null},{"text":[{"text":"The document is being truncated silently at the server’s configured window, so the relevant section is never seen."}],"correct":false,"tempting":[{"text":"Would produce answers missing information rather than confident wrong ones drawn from the wrong section, and is ruled out by the premise that it fits."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"tool-calling-locally","title":"Tool calling that survives contact with a small model","minutes":9},"next":{"slug":"determinism-and-reproducibility","title":"Why the same seed gives different answers","minutes":8}}}