How the model knows what order the words were in

Attention is a bag of tokens

Here is a fact that surprises people: the attention operation, on its own, does not know the order of its inputs. Shuffle the tokens and, without a positional signal, each output is a weighted sum over the same set of values: the same set, in a different arrangement. “Dog bites man” and “man bites dog” would be identical.

Something has to supply order. That something is the positional encoding, and the choices made there explain a large part of why long context was hard.

Three generations of answer

Absolute encodings. The original transformer added a fixed pattern of sines and cosines of different frequencies to each token’s embedding, so position 5 gets a different additive signature from position 400. GPT-2 replaced the fixed pattern with a learned table: one vector per position, up to a maximum. Simple, and it has a hard edge: position 1,025 in a model trained to 1,024 has no vector at all.

Relative encodings. Later work observed that what usually matters is not “this is token 400” but “this token is 3 places before that one”. Encoding relative offsets generalises better to lengths not seen in training.

Rotary position embeddings, or RoPE, which is what nearly every current open model uses. Instead of adding anything, RoPE rotates the query and key vectors by an angle proportional to their position, in each two-dimensional slice of the vector. The elegant part: when you take the dot product of a query at position m and a key at position n, the rotations partly cancel and what survives depends on m − n. Absolute rotation in, relative distance out.

You do not need the trigonometry. You need one consequence: RoPE gives the model a smooth, continuous notion of distance, which means it can be stretched.

Stretching the window

The stretch is why a model trained on 8,192 tokens can be served with a 128,000-token window. Two families of technique do it:

• Position interpolation: squash the position indices so that position 100,000 is presented as if it were position 6,250, keeping every angle inside the range the model saw in training. Cheap, and it costs some fine-grained resolution over short distances.
• NTK-aware scaling and YaRN: interpolate the low-frequency components while leaving the high-frequency ones alone, on the argument that fine local distinctions matter more than global ones. In practice these need a short fine-tuning run on long documents to work well.

This is why you should read a model card carefully. A “128K context” model may have been trained at 8K and extended, in which case its behaviour at 100K is a different and usually weaker thing than its behaviour at 8K. The number in the marketing is a capacity, not a promise of quality.

Why this matters for prompts

Two practical points come straight out of the mechanism.

First, relative position is what the model is sensitive to, so inserting a paragraph at the top of a long prompt shifts everything below it. If you rely on prompt caching, covered in module four, that shift invalidates the cache. Put the stable material first and the variable material last, and you get both better caching and more predictable behaviour.

Second, there is nothing special about the number 1. The model has no notion of a document beginning except the patterns it learned. Instructions “at the top” are not privileged by architecture; if they seem to be obeyed more, that is a learned habit from training data and it is empirically uneven, which is the subject of the lost-in-the-middle lesson.

A test you can run

Take a model you can run locally. Give it a 30-token instruction, then 20,000 tokens of filler, then a question whose answer sits in the filler. Move the answer through five positions (start, quarter, middle, three-quarters, end) and record accuracy at each. Twenty minutes, no GPU needed with a small quantised model, and you will have measured your own model’s position bias rather than reading somebody else’s chart.

That is the honest way to use any claim about long context: turn it into a five-line experiment on the thing you are actually going to deploy.