The experiment that named the problem

The setup, because the details matter

In 2023 a group at Stanford, Berkeley and Samaya ran an experiment that gave the field a phrase. The design was deliberately plain.

Take a question with a known answer. Assemble a context of ten, twenty or thirty documents, exactly one of which contains the answer; the rest are real passages retrieved for the same question, so they are on-topic and wrong. Then vary one thing only: where in the sequence the answer document sits. First, fifth, tenth, last. Ask the question, score the answer, repeat across a large question set.

If position were irrelevant, the accuracy line would be flat. It was not flat. It was a U.

Accuracy was highest when the answer document was first, fell as it moved towards the middle, and rose again towards the end. The dip was not small. In the twenty-document setting the middle positions cost a large fraction of the accuracy available at the ends, and in the most striking result, accuracy with the answer buried in the middle fell below the model’s score with no documents at all: the retrieved material had made things worse than an unaided guess.

They repeated it with a synthetic key-value lookup task, where there is no reasoning to blame, and got the same shape. That is the part that makes it a mechanism story rather than a task story.

Why a U

Three explanations, none of them complete on their own, all of them worth knowing.

The training data has the same shape. Instructions come at the beginning of documents and requests; questions come at the end. A model trained on that distribution learns that the extremes carry the load.

Attention sinks. Softmax attention has to allocate a total of one across all positions. When nothing in particular is relevant, models dump the excess onto the earliest tokens, which acquire a structurally privileged role. A later lesson in this module covers the mechanism; the effect here is a strong pull towards the start of the sequence.

Recency, from causal attention. Every token attends only backwards, so tokens near the end have the shortest path to the question and to the point where generation begins.

The middle has none of these advantages and all of the competition.

What has changed since, and what has not

Models have improved. Curves measured on current long-context models are flatter than the 2023 ones, and some are close to flat on easy retrieval at moderate lengths.

What has not changed is that the curve is a property of a specific model, a specific task and a specific length, and that it flattens least on the tasks that matter most: combining several facts, resolving a contradiction, noticing an exception. The harder the use of the material, the more position matters.

So the useful conclusion from the experiment is not “put things at the end”. It is: position is a variable in your system, it has an effect large enough to change outcomes, and you do not know its size until you measure it. Any team that reorders passages on the basis of a paper without re-running it is making the same mistake as a team that ignores the paper.

What it implies immediately

Even before measurement, three things follow.

Fewer passages is a positional argument as well as a cost one. With five passages there is barely a middle. The k-curve lesson gave you the cost and distraction reasons; this is the third.

The ranking your retriever produces is not automatically the order to present. Rank decides what gets in. Order in the window is a separate decision, made for a different reason, and the ordering lesson later in this module covers the options.

A fact in the window is not a fact the model has. This is the sentence to remember. “It’s in the context” is a statement about your assembler, not about the answer. Verification (a citation check, a containment assertion) is what turns the first into the second.

The limitation

The original experiment used a particular kind of task: extractive question answering over retrieved passages. Generalising from it to every use of a long context is over-reading it. Summarising a whole document, checking a contract for internal consistency, or refactoring a file all use the window differently, and the position effect for those is less well characterised.

What travels is the method rather than the number. Vary one thing, hold everything else fixed, and plot. That is the next lesson.