{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"context-length-in-practice","title":"What a 128k context window really means","url":"https://addaly.com/learn/open-models/context-length-in-practice","position":26,"of":71,"minutes":9,"answer":"Advertised context comes from extended positional encodings, but the effective length on tasks harder than finding one sentence is often a fraction of it, and prefill cost grows with the square of prompt length.","keyIdea":"Advertised context comes from extended positional encodings, but the effective length on tasks harder than finding one sentence is often a fraction of it, and prefill cost grows with the square of prompt length."},"body":{"format":"markdown","text":"## Advertised, supported, effective\n\nThree different numbers travel under the same name.\n\n**Advertised** is what the announcement says: 128k, 256k, a million. It comes from `max_position_embeddings` and from whatever context extension the lab applied.\n\n**Supported** is what your serving stack will actually run, which is set by your memory. The previous lesson's arithmetic decides this, and for most people it is between 8k and 32k regardless of what the model claims.\n\n**Effective** is the length at which the model still does the task well. It is usually far below the advertised number, and it is the only one that matters.\n\n## Why the gap exists\n\nPosition information in current models comes from rotary embeddings, which rotate the query and key vectors by an angle proportional to position. The rotation frequencies are set by `rope_theta`. A model trained with `rope_theta` at 10,000 over 4k-token documents has never seen the rotations that correspond to position 90,000, and its behaviour there is extrapolation.\n\nLabs extend context in one of three ways. **Linear interpolation** squeezes positions into the trained range, at some cost to fine positional resolution. **NTK-aware scaling** and **YaRN** adjust the frequencies unevenly so that high-frequency components, which carry local ordering, are disturbed least. All of them are usually followed by a short continued-pretraining run on long documents, and the amount of that training decides how real the extension is.\n\nSo a model can honestly advertise 128k and be trained on very few long documents. Nothing breaks at 100k; it just gets vaguer.\n\n## Needle tests pass, and prove less than they look\n\nThe standard demonstration hides a sentence in a long document and asks the model to find it. Modern models pass this at very long lengths, and the resulting green charts are what the announcements show.\n\nRetrieving one distinctive sentence is close to the easiest possible long-context task. Harder evaluations (multiple needles, tracking a variable through a long trace, aggregating facts spread across the document, or answering when the answer is absent) show much shorter effective lengths. Published work along these lines has repeatedly found models whose effective length on demanding tasks is a quarter or less of what they advertise.\n\nThere is also a well-documented positional effect: material at the start and end of a long context is used more reliably than material in the middle. Put the instruction and the most important document at the edges, not buried at 60%.\n\n## The costs nobody quotes\n\n**Prefill is quadratic in attention.** Processing a 100,000-token prompt is not twelve times a 8,000-token prompt; the attention part grows with the square. Time to first token on a very long prompt can be tens of seconds on hardware where short prompts feel instant.\n\n**The cache is linear and large.** 128k of context on the 8B model from the last lesson is about 17 GB of KV cache, which is more than the weights.\n\n**Rented long context is expensive.** You pay per input token, every request. A 50,000-token prompt sent a thousand times a day is 50 million input tokens daily, and at typical prices that is a real monthly bill for something a retrieval step would have reduced to 3,000 tokens.\n\n## What to do instead, mostly\n\nFor nearly every application, retrieving the right 3,000 tokens beats sending 100,000. It is cheaper, faster, more accurate, and it gives you a citation trail. The exceptions are genuine: a single document that must be reasoned over as a whole, a long code file, a legal contract where any clause may matter. Those are real, and they are rarer than long-context marketing implies.\n\nWhen you do need it, three habits help. Put instructions at the end as well as the beginning. Number or label the sections, so the model can refer to them. Ask for the location of the evidence along with the answer, which both improves accuracy and gives you something to check.\n\n## Measure your own effective length\n\nTake five real documents at your longest realistic size. Ask three questions of each whose answers sit at different depths: near the start, in the middle, near the end. Score the answers. Then halve the document length and repeat.\n\nIf accuracy at half the length is much better, your effective length is below where you are operating, and no amount of prompt work will fix that. This takes an hour and is the only version of this number that describes your material."},"check":{"question":"A model advertised at 128k passes a single-needle retrieval test at 120k but fails a task requiring facts from four separate places in a 60k document. What does this indicate?","options":[{"text":"The model's positional encoding is broken above 60k, so it should be run at half the advertised length.","correct":false,"tempting":"Would predict failure on the 120k needle test too, which passed."},{"text":"The prompt is too long for the KV cache, so earlier tokens are being evicted.","correct":false,"tempting":"Eviction is a serving configuration, and it would degrade the single-needle result at 120k first."},{"text":"Single-needle retrieval is a much easier task than aggregating dispersed facts, so the effective length for real work is well below the advertised figure.","correct":true},{"text":"The four facts are being lost to tokenisation differences across the document.","correct":false,"tempting":"Tokenisation is uniform across a document and does not explain a task-difficulty gap at the same length."}]},"prev":{"slug":"attention-and-the-kv-cache","title":"Attention variants, and the cache that eats your memory","url":"https://addaly.com/learn/open-models/attention-and-the-kv-cache"},"next":{"slug":"mixture-of-experts","title":"Mixture of experts, and the memory it still needs","url":"https://addaly.com/learn/open-models/mixture-of-experts"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What a 128k context window really means', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/context-length-in-practice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What a 128k context window really means","courseTitle":"The Open Model Ecosystem","module":{"index":3,"slug":"inside-the-artefact","title":"Inside the files"},"answer":null,"keyIdea":"Advertised context comes from extended positional encodings, but the effective length on tasks harder than finding one sentence is often a fraction of it, and prefill cost grows with the square of prompt length.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"advertised-supported-effective","runs":[{"text":"Advertised, supported, effective"}]},{"t":"p","runs":[{"text":"Three different numbers travel under the same name."}]},{"t":"p","runs":[{"text":"Advertised","b":true},{"text":" is what the announcement says: 128k, 256k, a million. It comes from "},{"text":"max_position_embeddings","code":true},{"text":" and from whatever context extension the lab applied."}]},{"t":"p","runs":[{"text":"Supported","b":true},{"text":" is what your serving stack will actually run, which is set by your memory. The previous lesson’s arithmetic decides this, and for most people it is between 8k and 32k regardless of what the model claims."}]},{"t":"p","runs":[{"text":"Effective","b":true},{"text":" is the length at which the model still does the task well. It is usually far below the advertised number, and it is the only one that matters."}]},{"t":"h2","id":"why-the-gap-exists","runs":[{"text":"Why the gap exists"}]},{"t":"p","runs":[{"text":"Position information in current models comes from rotary embeddings, which rotate the query and key vectors by an angle proportional to position. The rotation frequencies are set by "},{"text":"rope_theta","code":true},{"text":". A model trained with "},{"text":"rope_theta","code":true},{"text":" at 10,000 over 4k-token documents has never seen the rotations that correspond to position 90,000, and its behaviour there is extrapolation."}]},{"t":"p","runs":[{"text":"Labs extend context in one of three ways. "},{"text":"Linear interpolation","b":true},{"text":" squeezes positions into the trained range, at some cost to fine positional resolution. "},{"text":"NTK-aware scaling","b":true},{"text":" and "},{"text":"YaRN","b":true},{"text":" adjust the frequencies unevenly so that high-frequency components, which carry local ordering, are disturbed least. All of them are usually followed by a short continued-pretraining run on long documents, and the amount of that training decides how real the extension is."}]},{"t":"p","runs":[{"text":"So a model can honestly advertise 128k and be trained on very few long documents. Nothing breaks at 100k; it just gets vaguer."}]},{"t":"h2","id":"needle-tests-pass-and-prove-less-than-they-look","runs":[{"text":"Needle tests pass, and prove less than they look"}]},{"t":"p","runs":[{"text":"The standard demonstration hides a sentence in a long document and asks the model to find it. Modern models pass this at very long lengths, and the resulting green charts are what the announcements show."}]},{"t":"p","runs":[{"text":"Retrieving one distinctive sentence is close to the easiest possible long-context task. Harder evaluations (multiple needles, tracking a variable through a long trace, aggregating facts spread across the document, or answering when the answer is absent) show much shorter effective lengths. Published work along these lines has repeatedly found models whose effective length on demanding tasks is a quarter or less of what they advertise."}]},{"t":"p","runs":[{"text":"There is also a well-documented positional effect: material at the start and end of a long context is used more reliably than material in the middle. Put the instruction and the most important document at the edges, not buried at 60%."}]},{"t":"h2","id":"the-costs-nobody-quotes","runs":[{"text":"The costs nobody quotes"}]},{"t":"p","runs":[{"text":"Prefill is quadratic in attention.","b":true},{"text":" Processing a 100,000-token prompt is not twelve times a 8,000-token prompt; the attention part grows with the square. Time to first token on a very long prompt can be tens of seconds on hardware where short prompts feel instant."}]},{"t":"p","runs":[{"text":"The cache is linear and large.","b":true},{"text":" 128k of context on the 8B model from the last lesson is about 17 GB of KV cache, which is more than the weights."}]},{"t":"p","runs":[{"text":"Rented long context is expensive.","b":true},{"text":" You pay per input token, every request. A 50,000-token prompt sent a thousand times a day is 50 million input tokens daily, and at typical prices that is a real monthly bill for something a retrieval step would have reduced to 3,000 tokens."}]},{"t":"h2","id":"what-to-do-instead-mostly","runs":[{"text":"What to do instead, mostly"}]},{"t":"p","runs":[{"text":"For nearly every application, retrieving the right 3,000 tokens beats sending 100,000. It is cheaper, faster, more accurate, and it gives you a citation trail. The exceptions are genuine: a single document that must be reasoned over as a whole, a long code file, a legal contract where any clause may matter. Those are real, and they are rarer than long-context marketing implies."}]},{"t":"p","runs":[{"text":"When you do need it, three habits help. Put instructions at the end as well as the beginning. Number or label the sections, so the model can refer to them. Ask for the location of the evidence along with the answer, which both improves accuracy and gives you something to check."}]},{"t":"h2","id":"measure-your-own-effective-length","runs":[{"text":"Measure your own effective length"}]},{"t":"p","runs":[{"text":"Take five real documents at your longest realistic size. Ask three questions of each whose answers sit at different depths: near the start, in the middle, near the end. Score the answers. Then halve the document length and repeat."}]},{"t":"p","runs":[{"text":"If accuracy at half the length is much better, your effective length is below where you are operating, and no amount of prompt work will fix that. This takes an hour and is the only version of this number that describes your material."}]}],"check":{"question":[{"text":"A model advertised at 128k passes a single-needle retrieval test at 120k but fails a task requiring facts from four separate places in a 60k document. What does this indicate?"}],"options":[{"text":[{"text":"The model’s positional encoding is broken above 60k, so it should be run at half the advertised length."}],"correct":false,"tempting":[{"text":"Would predict failure on the 120k needle test too, which passed."}]},{"text":[{"text":"The prompt is too long for the KV cache, so earlier tokens are being evicted."}],"correct":false,"tempting":[{"text":"Eviction is a serving configuration, and it would degrade the single-needle result at 120k first."}]},{"text":[{"text":"Single-needle retrieval is a much easier task than aggregating dispersed facts, so the effective length for real work is well below the advertised figure."}],"correct":true,"tempting":null},{"text":[{"text":"The four facts are being lost to tokenisation differences across the document."}],"correct":false,"tempting":[{"text":"Tokenisation is uniform across a document and does not explain a task-difficulty gap at the same length."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"attention-and-the-kv-cache","title":"Attention variants, and the cache that eats your memory","minutes":10},"next":{"slug":"mixture-of-experts","title":"Mixture of experts, and the memory it still needs","minutes":8}}}