{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"position-bias-and-effective-context","title":"Where in a long prompt things get missed","url":"https://addaly.com/learn/how-llms-work/position-bias-and-effective-context","position":32,"of":78,"minutes":9,"answer":"A model's advertised context length is a capacity, and the length over which it actually uses information reliably is a separate number you have to measure.","keyIdea":"A model's advertised context length is a capacity, and the length over which it actually uses information reliably is a separate number you have to measure."},"body":{"format":"markdown","text":"## The finding\n\nIn 2023 researchers gave models a question and a set of documents, exactly one of which contained the answer, and moved that document through every position. Accuracy was highest when the answer sat at the beginning, high again at the end, and lowest in the middle: a U-shaped curve.\n\nThe effect was substantial. On some configurations, performance with the answer in the middle of a long input fell below performance with **no documents at all**, which is a striking result: adding the correct information in the wrong place made things worse than omitting it.\n\n> Figure: Accuracy against where in the prompt the answer sits. Line chart of Answers correct, % against Depth of the answer in the prompt, % of the way through: Short prompt, about 4,000 tokens (10, 88) (30, 84) (50, 82) (70, 84) (90, 87); Long prompt, about 64,000 tokens (10, 76) (30, 62) (50, 54) (70, 60) (90, 71); No documents at all, answered from memory (10, 57) (50, 57) (90, 57). The U is the shape everybody reproduces: strongest at the start, strong again at the end, weakest in the middle, and worse as the prompt grows. On the long prompt the correct document, placed in the middle, scores below giving the model nothing at all. Measure your own curve; do not borrow this one.\n\nThe paper is titled \"Lost in the Middle\" and the name stuck. Later models have reduced the effect and none has eliminated it.\n\n## The needle test, and why it flatters\n\nThe popular demonstration of long context is the needle-in-a-haystack test: hide one odd sentence in 100,000 tokens of unrelated text and ask for it. Models score near-perfectly, and the charts get shared widely.\n\nBe careful about what that measures. The needle is *lexically distinctive*: it stands out against the haystack, so a single retrieval-like operation finds it. Real tasks rarely look like that.\n\nHarder variants tell a different story:\n\n- **Multiple needles.** Hide seven facts and ask for all seven. Accuracy falls, and falls further as the count rises.\n- **Needles that require combination.** Two facts in different places that must be joined to answer. Much harder than either alone.\n- **Distractors.** Plant several passages that look relevant but are not. Performance drops sharply.\n- **Absence.** Ask about something that is *not* in the haystack. Models frequently produce a confident answer anyway.\n\nThe gap between the single-needle chart and these results is the gap between capacity and usable context.\n\n## Effective context length\n\nThe useful concept: the length beyond which a model's accuracy on your task degrades materially. It is task-dependent and it is usually well short of the advertised window. Public long-context benchmarks that use realistic multi-fact tasks routinely find sharp declines at a fraction of the stated maximum.\n\nThere is no way to look this up for your case. There is a straightforward way to measure it, and it is an hour of work:\n\n1. Take 30 real questions from your application, each with a known answer in a known document.\n2. Build prompts at 4K, 16K, 64K and 128K by padding with plausible but irrelevant material from your own corpus.\n3. At each length, place the answer at 10, 50 and 90 per cent depth.\n4. Score. Plot accuracy against length and depth.\n\nYou now have your own curve, and it will inform a decision (how much to retrieve, whether to rerank, whether to chunk the task) that would otherwise be guesswork.\n\n## Working with the bias rather than against it\n\nSeveral practical moves follow directly:\n\n- **Put the instruction at the end**, after long context. If the task statement is buried above 50,000 tokens of documents, it sits in the weak region. Restating it after the documents is a cheap and reliably effective change.\n- **Order retrieved passages by relevance, most relevant first and second-most last**: placing the strongest material where recall is best. Some teams alternate outward from the ends for the same reason.\n- **Retrieve less.** Six good passages beat forty mediocre ones, and not only for cost. Distractors measurably reduce accuracy, so a precision-oriented retriever outperforms a recall-oriented one at the point where the model reads.\n- **Split the task.** Ask about 20,000 tokens at a time and combine the answers. More calls, better accuracy, and each call is individually checkable.\n\n## Why the bias exists\n\nHonestly: this is not fully settled. Contributing factors that have support include the distribution of document lengths in training data, attention patterns that concentrate on the first few tokens (the \"attention sink\" effect, where early positions absorb weight that has nowhere better to go) and recency effects from the causal structure. Position-encoding extension methods add their own distortions.\n\nWhat is not in dispute is the empirical shape, which is robust across model families. Treat any claim of a uniformly usable million-token window as a hypothesis about a specific model that you should test on your own task before designing around it."},"check":{"question":"A document question-answering system performs well at 8,000 tokens of context and poorly at 60,000, though the model advertises 200,000. What is the most informative next step?","options":[{"text":"Upgrade to a model with a larger advertised window.","correct":false,"tempting":"A bigger window is the intuitive response to a length problem and is what most vendors' documentation implies."},{"text":"Increase the number of retrieved passages so the answer is definitely included.","correct":false,"tempting":"Recall does matter and missing evidence is a real failure, so retrieving more feels like insurance."},{"text":"Lower the temperature, since long contexts increase the chance of drifting off topic.","correct":false,"tempting":"Temperature is the standard reliability dial, and long inputs do produce more variable output."},{"text":"Measure accuracy against context length and answer depth on the team's own questions, to find where this model's usable context actually ends.","correct":true}]},"prev":{"slug":"what-fills-the-window","title":"Budgeting the window, line by line","url":"https://addaly.com/learn/how-llms-work/what-fills-the-window"},"next":{"slug":"prompt-caching","title":"Prompt caching, and the ordering it demands","url":"https://addaly.com/learn/how-llms-work/prompt-caching"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where in a long prompt things get missed', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/position-bias-and-effective-context (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where in a long prompt things get missed","courseTitle":"How a Language Model Actually Works","module":{"index":4,"slug":"the-context-window","title":"What the model can see"},"answer":null,"keyIdea":"A model’s advertised context length is a capacity, and the length over which it actually uses information reliably is a separate number you have to measure.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Accuracy against where in the prompt the answer sits","series":[{"label":"Short prompt, about 4,000 tokens","points":[[10,88],[30,84],[50,82],[70,84],[90,87]]},{"label":"Long prompt, about 64,000 tokens","points":[[10,76],[30,62],[50,54],[70,60],[90,71]]},{"label":"No documents at all, answered from memory","points":[[10,57],[50,57],[90,57]]}],"xLabel":"Depth of the answer in the prompt, % of the way through","yLabel":"Answers correct, %","caption":"The U is the shape everybody reproduces: strongest at the start, strong again at the end, weakest in the middle, and worse as the prompt grows. On the long prompt the correct document, placed in the middle, scores below giving the model nothing at all. Measure your own curve; do not borrow this one."},"alt":"Figure: Accuracy against where in the prompt the answer sits. Line chart of Answers correct, % against Depth of the answer in the prompt, % of the way through: Short prompt, about 4,000 tokens (10, 88) (30, 84) (50, 82) (70, 84) (90, 87); Long prompt, about 64,000 tokens (10, 76) (30, 62) (50, 54) (70, 60) (90, 71); No documents at all, answered from memory (10, 57) (50, 57) (90, 57). The U is the shape everybody reproduces: strongest at the start, strong again at the end, weakest in the middle, and worse as the prompt grows. On the long prompt the correct document, placed in the middle, scores below giving the model nothing at all. Measure your own curve; do not borrow this one."}},"blocks":[{"t":"h2","id":"the-finding","runs":[{"text":"The finding"}]},{"t":"p","runs":[{"text":"In 2023 researchers gave models a question and a set of documents, exactly one of which contained the answer, and moved that document through every position. Accuracy was highest when the answer sat at the beginning, high again at the end, and lowest in the middle: a U-shaped curve."}]},{"t":"p","runs":[{"text":"The effect was substantial. On some configurations, performance with the answer in the middle of a long input fell below performance with "},{"text":"no documents at all","b":true},{"text":", which is a striking result: adding the correct information in the wrong place made things worse than omitting it."}]},{"t":"p","runs":[{"text":"The paper is titled “Lost in the Middle” and the name stuck. Later models have reduced the effect and none has eliminated it."}]},{"t":"h2","id":"the-needle-test-and-why-it-flatters","runs":[{"text":"The needle test, and why it flatters"}]},{"t":"p","runs":[{"text":"The popular demonstration of long context is the needle-in-a-haystack test: hide one odd sentence in 100,000 tokens of unrelated text and ask for it. Models score near-perfectly, and the charts get shared widely."}]},{"t":"p","runs":[{"text":"Be careful about what that measures. The needle is "},{"text":"lexically distinctive","i":true},{"text":": it stands out against the haystack, so a single retrieval-like operation finds it. Real tasks rarely look like that."}]},{"t":"p","runs":[{"text":"Harder variants tell a different story:"}]},{"t":"ul","items":[[{"text":"Multiple needles.","b":true},{"text":" Hide seven facts and ask for all seven. Accuracy falls, and falls further as the count rises."}],[{"text":"Needles that require combination.","b":true},{"text":" Two facts in different places that must be joined to answer. Much harder than either alone."}],[{"text":"Distractors.","b":true},{"text":" Plant several passages that look relevant but are not. Performance drops sharply."}],[{"text":"Absence.","b":true},{"text":" Ask about something that is "},{"text":"not","i":true},{"text":" in the haystack. Models frequently produce a confident answer anyway."}]]},{"t":"p","runs":[{"text":"The gap between the single-needle chart and these results is the gap between capacity and usable context."}]},{"t":"h2","id":"effective-context-length","runs":[{"text":"Effective context length"}]},{"t":"p","runs":[{"text":"The useful concept: the length beyond which a model’s accuracy on your task degrades materially. It is task-dependent and it is usually well short of the advertised window. Public long-context benchmarks that use realistic multi-fact tasks routinely find sharp declines at a fraction of the stated maximum."}]},{"t":"p","runs":[{"text":"There is no way to look this up for your case. There is a straightforward way to measure it, and it is an hour of work:"}]},{"t":"ol","start":1,"items":[[{"text":"Take 30 real questions from your application, each with a known answer in a known document."}],[{"text":"Build prompts at 4K, 16K, 64K and 128K by padding with plausible but irrelevant material from your own corpus."}],[{"text":"At each length, place the answer at 10, 50 and 90 per cent depth."}],[{"text":"Score. Plot accuracy against length and depth."}]]},{"t":"p","runs":[{"text":"You now have your own curve, and it will inform a decision (how much to retrieve, whether to rerank, whether to chunk the task) that would otherwise be guesswork."}]},{"t":"h2","id":"working-with-the-bias-rather-than-against-it","runs":[{"text":"Working with the bias rather than against it"}]},{"t":"p","runs":[{"text":"Several practical moves follow directly:"}]},{"t":"ul","items":[[{"text":"Put the instruction at the end","b":true},{"text":", after long context. If the task statement is buried above 50,000 tokens of documents, it sits in the weak region. Restating it after the documents is a cheap and reliably effective change."}],[{"text":"Order retrieved passages by relevance, most relevant first and second-most last","b":true},{"text":": placing the strongest material where recall is best. Some teams alternate outward from the ends for the same reason."}],[{"text":"Retrieve less.","b":true},{"text":" Six good passages beat forty mediocre ones, and not only for cost. Distractors measurably reduce accuracy, so a precision-oriented retriever outperforms a recall-oriented one at the point where the model reads."}],[{"text":"Split the task.","b":true},{"text":" Ask about 20,000 tokens at a time and combine the answers. More calls, better accuracy, and each call is individually checkable."}]]},{"t":"h2","id":"why-the-bias-exists","runs":[{"text":"Why the bias exists"}]},{"t":"p","runs":[{"text":"Honestly: this is not fully settled. Contributing factors that have support include the distribution of document lengths in training data, attention patterns that concentrate on the first few tokens (the “attention sink” effect, where early positions absorb weight that has nowhere better to go) and recency effects from the causal structure. Position-encoding extension methods add their own distortions."}]},{"t":"p","runs":[{"text":"What is not in dispute is the empirical shape, which is robust across model families. Treat any claim of a uniformly usable million-token window as a hypothesis about a specific model that you should test on your own task before designing around it."}]}],"check":{"question":[{"text":"A document question-answering system performs well at 8,000 tokens of context and poorly at 60,000, though the model advertises 200,000. What is the most informative next step?"}],"options":[{"text":[{"text":"Upgrade to a model with a larger advertised window."}],"correct":false,"tempting":[{"text":"A bigger window is the intuitive response to a length problem and is what most vendors’ documentation implies."}]},{"text":[{"text":"Increase the number of retrieved passages so the answer is definitely included."}],"correct":false,"tempting":[{"text":"Recall does matter and missing evidence is a real failure, so retrieving more feels like insurance."}]},{"text":[{"text":"Lower the temperature, since long contexts increase the chance of drifting off topic."}],"correct":false,"tempting":[{"text":"Temperature is the standard reliability dial, and long inputs do produce more variable output."}]},{"text":[{"text":"Measure accuracy against context length and answer depth on the team’s own questions, to find where this model’s usable context actually ends."}],"correct":true,"tempting":null}],"order":[1,3,0,2],"why":null},"prev":{"slug":"what-fills-the-window","title":"Budgeting the window, line by line","minutes":8},"next":{"slug":"prompt-caching","title":"Prompt caching, and the ordering it demands","minutes":8}}}