Where in a long prompt things get missed

The finding

In 2023 researchers gave models a question and a set of documents, exactly one of which contained the answer, and moved that document through every position. Accuracy was highest when the answer sat at the beginning, high again at the end, and lowest in the middle: a U-shaped curve.

The effect was substantial. On some configurations, performance with the answer in the middle of a long input fell below performance with no documents at all, which is a striking result: adding the correct information in the wrong place made things worse than omitting it.

The paper is titled “Lost in the Middle” and the name stuck. Later models have reduced the effect and none has eliminated it.

The needle test, and why it flatters

The popular demonstration of long context is the needle-in-a-haystack test: hide one odd sentence in 100,000 tokens of unrelated text and ask for it. Models score near-perfectly, and the charts get shared widely.

Be careful about what that measures. The needle is lexically distinctive: it stands out against the haystack, so a single retrieval-like operation finds it. Real tasks rarely look like that.

Harder variants tell a different story:

• Multiple needles. Hide seven facts and ask for all seven. Accuracy falls, and falls further as the count rises.
• Needles that require combination. Two facts in different places that must be joined to answer. Much harder than either alone.
• Distractors. Plant several passages that look relevant but are not. Performance drops sharply.
• Absence. Ask about something that is not in the haystack. Models frequently produce a confident answer anyway.

The gap between the single-needle chart and these results is the gap between capacity and usable context.

Effective context length

The useful concept: the length beyond which a model’s accuracy on your task degrades materially. It is task-dependent and it is usually well short of the advertised window. Public long-context benchmarks that use realistic multi-fact tasks routinely find sharp declines at a fraction of the stated maximum.

There is no way to look this up for your case. There is a straightforward way to measure it, and it is an hour of work:

1. Take 30 real questions from your application, each with a known answer in a known document.
2. Build prompts at 4K, 16K, 64K and 128K by padding with plausible but irrelevant material from your own corpus.
3. At each length, place the answer at 10, 50 and 90 per cent depth.
4. Score. Plot accuracy against length and depth.

You now have your own curve, and it will inform a decision (how much to retrieve, whether to rerank, whether to chunk the task) that would otherwise be guesswork.

Working with the bias rather than against it

Several practical moves follow directly:

• Put the instruction at the end, after long context. If the task statement is buried above 50,000 tokens of documents, it sits in the weak region. Restating it after the documents is a cheap and reliably effective change.
• Order retrieved passages by relevance, most relevant first and second-most last: placing the strongest material where recall is best. Some teams alternate outward from the ends for the same reason.
• Retrieve less. Six good passages beat forty mediocre ones, and not only for cost. Distractors measurably reduce accuracy, so a precision-oriented retriever outperforms a recall-oriented one at the point where the model reads.
• Split the task. Ask about 20,000 tokens at a time and combine the answers. More calls, better accuracy, and each call is individually checkable.

Why the bias exists

Honestly: this is not fully settled. Contributing factors that have support include the distribution of document lengths in training data, attention patterns that concentrate on the first few tokens (the “attention sink” effect, where early positions absorb weight that has nowhere better to go) and recency effects from the causal structure. Position-encoding extension methods add their own distortions.

What is not in dispute is the empirical shape, which is robust across model families. Treat any claim of a uniformly usable million-token window as a hypothesis about a specific model that you should test on your own task before designing around it.