{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"ordering-and-the-middle","title":"Ordering, and the fact that models skim","url":"https://addaly.com/learn/context-engineering/ordering-and-the-middle","position":32,"of":87,"minutes":8,"answer":"Position changes whether a fact gets used, so measure the position curve instead of trusting rank order.","keyIdea":"Position changes whether a fact gets used, so measure the position curve instead of trusting rank order."},"body":{"format":"markdown","text":"## The U curve\n\nLiu et al. published *Lost in the Middle* in 2023: place a needed fact at different positions in a long context, hold everything else constant, measure accuracy. The result was a U. Accuracy is highest when the fact is near the beginning or near the end, and sags in the middle: in some settings badly enough that a model given 20 documents did worse than the same model given none.\n\nThat was 2023. The curve has flattened considerably on frontier models. It has not gone away, and it reappears whenever the task needs more than one fact, or the distractors are similar to the answer, or the context gets long enough. Any specific claim about how bad it is on a specific model this quarter is out of date. The engineering response is not to memorise a number; it is to stop assuming position is neutral and start measuring it.\n\n## What this changes in your assembly code\n\n**Put the instruction after the long document.** If you ask the question, then paste 60,000 tokens, the question is now very far away from where the answer gets generated. Ask after. Better, ask twice: a short framing before, the precise instruction after. Costs a hundred tokens.\n\n```\n[brief framing: \"You will read a lease. You will answer one question.\"]\n[the 60,000-token lease]\n[the actual question, the output schema, the constraints]\n```\n\n**Stop concatenating by relevance rank.** Retrieval hands you chunks ranked 1 to 20. Pasting them in that order puts rank 1 in the strong opening position and buries ranks 2 through 6 (often the ones carrying the supporting detail) in the weak middle. Reordering so that the highest-ranked items sit at both ends is a cheap experiment. Be honest that the evidence is mixed on newer models: it helped a lot in 2023, it helps less now, and it is a one-line change you can A/B in an afternoon.\n\n**Recency for conversation.** Newest turns closest to the generation point. If you summarise old turns to save budget, the summary goes above the verbatim recent ones, not below.\n\n**Deduplicate aggressively.** Two near-identical chunks make a claim look corroborated. If the corpus contains the 2024 policy and the 2026 policy and both are retrieved, the model has no principled way to pick, and it will pick the one positioned better. Filter by version at retrieval time; it is not the model's job.\n\n## Number the chunks and demand citations\n\n```\n[1] (handbook.md, Leave > Parental) Employees may take...\n[2] (handbook.md, Leave > Sick) ...\n[3] (policy-2026.pdf, §4.2) ...\n```\n\nThen require a `sources` array of integers in your schema. This buys you three things. The model has to attend across the set rather than latching onto the first plausible passage. Users get something checkable. And you get a distribution: if position 1 and 2 are cited on 80% of queries and positions 9 through 20 are never cited, you now know something concrete. Either your reranker is good and you should retrieve fewer chunks, or your ordering is burying useful material. Both are actionable; \"the model seems inattentive\" is not.\n\n## Measure it directly\n\nDo not reason about this. Run the sweep.\n\n```python\ndef position_sweep(question, answer_chunk, distractors, positions):\n    results = {}\n    for p in positions:\n        ctx = distractors[:p] + [answer_chunk] + distractors[p:]\n        acc = mean(is_correct(ask(question, ctx)) for _ in range(10))\n        results[p] = acc\n    return results\n\n# {0: 0.95, 5: 0.80, 10: 0.70, 15: 0.75, 19: 0.90}  <- you have a U\n```\n\nA flat line means your ordering is fine and you can spend your effort elsewhere. A U means retrieving more is actively hurting you, and the fix is a reranker that lets you retrieve fewer, not a bigger model.\n\n## The counterintuitive result to expect\n\nTeams raise `top_k` from 5 to 30, watch retriever recall@k climb, and find end-to-end accuracy *falls*. Both measurements are correct. The right chunk is now in the context: sitting at position 22, surrounded by 29 confident-sounding near-misses.\n\nRetrieval recall and answer accuracy are different metrics, and optimising the first without reranking degrades the second. More candidates, then aggressive reranking down to a handful, then careful placement. That sequence is the whole technique."},"check":{"question":"An engineering team raises retrieval from top-5 to top-30. Their retriever metric, recall@k, improves from 0.71 to 0.94: the correct passage is now almost always present. End-to-end answer accuracy drops from 78% to 69%. Total context is 15,000 tokens, well within the window. What is going on?","options":[{"text":"Recall@k is computed against a different ground truth than answer accuracy, so the two numbers are not comparable and one of them is measuring the wrong thing.","correct":false,"tempting":"Metric mismatch is a real and common bug, which makes this an appealing sceptical answer, but both numbers can be correct and still move in opposite directions."},{"text":"The context is long enough that the model is now summarising rather than answering, and needs an explicit instruction to be precise.","correct":false,"tempting":"Long contexts do shift response style, so this connects a real behaviour to the symptom, but 15,000 tokens is not where that transition happens."},{"text":"The correct passage is present but buried among 29 similar-looking distractors in the weakest positional region, so presence in the context stopped implying use.","correct":true},{"text":"The extra 25 chunks pushed the prompt past the model's effective attention span, which is shorter than its advertised window.","correct":false,"tempting":"Effective context length below the advertised window is a genuine phenomenon, so the vocabulary is right, but 15,000 tokens sits far inside any current model's reliable range."}]},"prev":{"slug":"lost-in-the-middle-the-experiment","title":"The experiment that named the problem","url":"https://addaly.com/learn/context-engineering/lost-in-the-middle-the-experiment"},"next":{"slug":"building-a-position-probe","title":"Measuring your own position curve","url":"https://addaly.com/learn/context-engineering/building-a-position-probe"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Ordering, and the fact that models skim', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/ordering-and-the-middle (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Ordering, and the fact that models skim","courseTitle":"Context Engineering","module":{"index":4,"slug":"position-and-order","title":"Position, order and what the model actually reads"},"answer":null,"keyIdea":"Position changes whether a fact gets used, so measure the position curve instead of trusting rank order.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-u-curve","runs":[{"text":"The U curve"}]},{"t":"p","runs":[{"text":"Liu et al. published "},{"text":"Lost in the Middle","i":true},{"text":" in 2023: place a needed fact at different positions in a long context, hold everything else constant, measure accuracy. The result was a U. Accuracy is highest when the fact is near the beginning or near the end, and sags in the middle: in some settings badly enough that a model given 20 documents did worse than the same model given none."}]},{"t":"p","runs":[{"text":"That was 2023. The curve has flattened considerably on frontier models. It has not gone away, and it reappears whenever the task needs more than one fact, or the distractors are similar to the answer, or the context gets long enough. Any specific claim about how bad it is on a specific model this quarter is out of date. The engineering response is not to memorise a number; it is to stop assuming position is neutral and start measuring it."}]},{"t":"h2","id":"what-this-changes-in-your-assembly-code","runs":[{"text":"What this changes in your assembly code"}]},{"t":"p","runs":[{"text":"Put the instruction after the long document.","b":true},{"text":" If you ask the question, then paste 60,000 tokens, the question is now very far away from where the answer gets generated. Ask after. Better, ask twice: a short framing before, the precise instruction after. Costs a hundred tokens."}]},{"t":"code","lang":"","label":"Text","text":"[brief framing: \"You will read a lease. You will answer one question.\"]\n[the 60,000-token lease]\n[the actual question, the output schema, the constraints]"},{"t":"p","runs":[{"text":"Stop concatenating by relevance rank.","b":true},{"text":" Retrieval hands you chunks ranked 1 to 20. Pasting them in that order puts rank 1 in the strong opening position and buries ranks 2 through 6 (often the ones carrying the supporting detail) in the weak middle. Reordering so that the highest-ranked items sit at both ends is a cheap experiment. Be honest that the evidence is mixed on newer models: it helped a lot in 2023, it helps less now, and it is a one-line change you can A/B in an afternoon."}]},{"t":"p","runs":[{"text":"Recency for conversation.","b":true},{"text":" Newest turns closest to the generation point. If you summarise old turns to save budget, the summary goes above the verbatim recent ones, not below."}]},{"t":"p","runs":[{"text":"Deduplicate aggressively.","b":true},{"text":" Two near-identical chunks make a claim look corroborated. If the corpus contains the 2024 policy and the 2026 policy and both are retrieved, the model has no principled way to pick, and it will pick the one positioned better. Filter by version at retrieval time; it is not the model’s job."}]},{"t":"h2","id":"number-the-chunks-and-demand-citations","runs":[{"text":"Number the chunks and demand citations"}]},{"t":"code","lang":"","label":"Text","text":"[1] (handbook.md, Leave > Parental) Employees may take...\n[2] (handbook.md, Leave > Sick) ...\n[3] (policy-2026.pdf, §4.2) ..."},{"t":"p","runs":[{"text":"Then require a "},{"text":"sources","code":true},{"text":" array of integers in your schema. This buys you three things. The model has to attend across the set rather than latching onto the first plausible passage. Users get something checkable. And you get a distribution: if position 1 and 2 are cited on 80% of queries and positions 9 through 20 are never cited, you now know something concrete. Either your reranker is good and you should retrieve fewer chunks, or your ordering is burying useful material. Both are actionable; “the model seems inattentive” is not."}]},{"t":"h2","id":"measure-it-directly","runs":[{"text":"Measure it directly"}]},{"t":"p","runs":[{"text":"Do not reason about this. Run the sweep."}]},{"t":"code","lang":"python","label":"Python","text":"def position_sweep(question, answer_chunk, distractors, positions):\n    results = {}\n    for p in positions:\n        ctx = distractors[:p] + [answer_chunk] + distractors[p:]\n        acc = mean(is_correct(ask(question, ctx)) for _ in range(10))\n        results[p] = acc\n    return results\n\n# {0: 0.95, 5: 0.80, 10: 0.70, 15: 0.75, 19: 0.90}  <- you have a U"},{"t":"p","runs":[{"text":"A flat line means your ordering is fine and you can spend your effort elsewhere. A U means retrieving more is actively hurting you, and the fix is a reranker that lets you retrieve fewer, not a bigger model."}]},{"t":"h2","id":"the-counterintuitive-result-to-expect","runs":[{"text":"The counterintuitive result to expect"}]},{"t":"p","runs":[{"text":"Teams raise "},{"text":"top_k","code":true},{"text":" from 5 to 30, watch retriever recall@k climb, and find end-to-end accuracy "},{"text":"falls","i":true},{"text":". Both measurements are correct. The right chunk is now in the context: sitting at position 22, surrounded by 29 confident-sounding near-misses."}]},{"t":"p","runs":[{"text":"Retrieval recall and answer accuracy are different metrics, and optimising the first without reranking degrades the second. More candidates, then aggressive reranking down to a handful, then careful placement. That sequence is the whole technique."}]}],"check":{"question":[{"text":"An engineering team raises retrieval from top-5 to top-30. Their retriever metric, recall@k, improves from 0.71 to 0.94: the correct passage is now almost always present. End-to-end answer accuracy drops from 78% to 69%. Total context is 15,000 tokens, well within the window. What is going on?"}],"options":[{"text":[{"text":"Recall@k is computed against a different ground truth than answer accuracy, so the two numbers are not comparable and one of them is measuring the wrong thing."}],"correct":false,"tempting":[{"text":"Metric mismatch is a real and common bug, which makes this an appealing sceptical answer, but both numbers can be correct and still move in opposite directions."}]},{"text":[{"text":"The context is long enough that the model is now summarising rather than answering, and needs an explicit instruction to be precise."}],"correct":false,"tempting":[{"text":"Long contexts do shift response style, so this connects a real behaviour to the symptom, but 15,000 tokens is not where that transition happens."}]},{"text":[{"text":"The correct passage is present but buried among 29 similar-looking distractors in the weakest positional region, so presence in the context stopped implying use."}],"correct":true,"tempting":null},{"text":[{"text":"The extra 25 chunks pushed the prompt past the model’s effective attention span, which is shorter than its advertised window."}],"correct":false,"tempting":[{"text":"Effective context length below the advertised window is a genuine phenomenon, so the vocabulary is right, but 15,000 tokens sits far inside any current model’s reliable range."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"lost-in-the-middle-the-experiment","title":"The experiment that named the problem","minutes":9},"next":{"slug":"building-a-position-probe","title":"Measuring your own position curve","minutes":9}}}