Ordering, and the fact that models skim

The U curve

Liu et al. published Lost in the Middle in 2023: place a needed fact at different positions in a long context, hold everything else constant, measure accuracy. The result was a U. Accuracy is highest when the fact is near the beginning or near the end, and sags in the middle: in some settings badly enough that a model given 20 documents did worse than the same model given none.

That was 2023. The curve has flattened considerably on frontier models. It has not gone away, and it reappears whenever the task needs more than one fact, or the distractors are similar to the answer, or the context gets long enough. Any specific claim about how bad it is on a specific model this quarter is out of date. The engineering response is not to memorise a number; it is to stop assuming position is neutral and start measuring it.

What this changes in your assembly code

Put the instruction after the long document. If you ask the question, then paste 60,000 tokens, the question is now very far away from where the answer gets generated. Ask after. Better, ask twice: a short framing before, the precise instruction after. Costs a hundred tokens.

[brief framing: "You will read a lease. You will answer one question."]
[the 60,000-token lease]
[the actual question, the output schema, the constraints]

Stop concatenating by relevance rank. Retrieval hands you chunks ranked 1 to 20. Pasting them in that order puts rank 1 in the strong opening position and buries ranks 2 through 6 (often the ones carrying the supporting detail) in the weak middle. Reordering so that the highest-ranked items sit at both ends is a cheap experiment. Be honest that the evidence is mixed on newer models: it helped a lot in 2023, it helps less now, and it is a one-line change you can A/B in an afternoon.

Recency for conversation. Newest turns closest to the generation point. If you summarise old turns to save budget, the summary goes above the verbatim recent ones, not below.

Deduplicate aggressively. Two near-identical chunks make a claim look corroborated. If the corpus contains the 2024 policy and the 2026 policy and both are retrieved, the model has no principled way to pick, and it will pick the one positioned better. Filter by version at retrieval time; it is not the model’s job.

Number the chunks and demand citations

[1] (handbook.md, Leave > Parental) Employees may take...
[2] (handbook.md, Leave > Sick) ...
[3] (policy-2026.pdf, §4.2) ...

Then require a sources array of integers in your schema. This buys you three things. The model has to attend across the set rather than latching onto the first plausible passage. Users get something checkable. And you get a distribution: if position 1 and 2 are cited on 80% of queries and positions 9 through 20 are never cited, you now know something concrete. Either your reranker is good and you should retrieve fewer chunks, or your ordering is burying useful material. Both are actionable; “the model seems inattentive” is not.

Measure it directly

Do not reason about this. Run the sweep.

def position_sweep(question, answer_chunk, distractors, positions):
    results = {}
    for p in positions:
        ctx = distractors[:p] + [answer_chunk] + distractors[p:]
        acc = mean(is_correct(ask(question, ctx)) for _ in range(10))
        results[p] = acc
    return results

# {0: 0.95, 5: 0.80, 10: 0.70, 15: 0.75, 19: 0.90}  <- you have a U

A flat line means your ordering is fine and you can spend your effort elsewhere. A U means retrieving more is actively hurting you, and the fix is a reranker that lets you retrieve fewer, not a bigger model.

The counterintuitive result to expect

Teams raise top_k from 5 to 30, watch retriever recall@k climb, and find end-to-end accuracy falls. Both measurements are correct. The right chunk is now in the context: sitting at position 22, surrounded by 29 confident-sounding near-misses.

Retrieval recall and answer accuracy are different metrics, and optimising the first without reranking degrades the second. More candidates, then aggressive reranking down to a handful, then careful placement. That sequence is the whole technique.