{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"position-and-order","title":"How the model knows what order the words were in","url":"https://addaly.com/learn/how-llms-work/position-and-order","position":7,"of":78,"minutes":8,"answer":"Attention has no built-in sense of order, so position is injected as extra information, and the way it is injected sets the ceiling on how far a model can usefully read.","keyIdea":"Attention has no built-in sense of order, so position is injected as extra information, and the way it is injected sets the ceiling on how far a model can usefully read."},"body":{"format":"markdown","text":"## Attention is a bag of tokens\n\nHere is a fact that surprises people: the attention operation, on its own, does not know the order of its inputs. Shuffle the tokens and, without a positional signal, each output is a weighted sum over the same set of values: the same set, in a different arrangement. \"Dog bites man\" and \"man bites dog\" would be identical.\n\nSomething has to supply order. That something is the positional encoding, and the choices made there explain a large part of why long context was hard.\n\n## Three generations of answer\n\n**Absolute encodings.** The original transformer added a fixed pattern of sines and cosines of different frequencies to each token's embedding, so position 5 gets a different additive signature from position 400. GPT-2 replaced the fixed pattern with a *learned* table: one vector per position, up to a maximum. Simple, and it has a hard edge: position 1,025 in a model trained to 1,024 has no vector at all.\n\n**Relative encodings.** Later work observed that what usually matters is not \"this is token 400\" but \"this token is 3 places before that one\". Encoding relative offsets generalises better to lengths not seen in training.\n\n**Rotary position embeddings, or RoPE**, which is what nearly every current open model uses. Instead of adding anything, RoPE *rotates* the query and key vectors by an angle proportional to their position, in each two-dimensional slice of the vector. The elegant part: when you take the dot product of a query at position *m* and a key at position *n*, the rotations partly cancel and what survives depends on *m − n*. Absolute rotation in, relative distance out.\n\nYou do not need the trigonometry. You need one consequence: RoPE gives the model a smooth, continuous notion of distance, which means it can be stretched.\n\n## Stretching the window\n\nThe stretch is why a model trained on 8,192 tokens can be served with a 128,000-token window. Two families of technique do it:\n\n- **Position interpolation**: squash the position indices so that position 100,000 is presented as if it were position 6,250, keeping every angle inside the range the model saw in training. Cheap, and it costs some fine-grained resolution over short distances.\n- **NTK-aware scaling and YaRN**: interpolate the low-frequency components while leaving the high-frequency ones alone, on the argument that fine local distinctions matter more than global ones. In practice these need a short fine-tuning run on long documents to work well.\n\nThis is why you should read a model card carefully. A \"128K context\" model may have been *trained* at 8K and extended, in which case its behaviour at 100K is a different and usually weaker thing than its behaviour at 8K. The number in the marketing is a capacity, not a promise of quality.\n\n## Why this matters for prompts\n\nTwo practical points come straight out of the mechanism.\n\nFirst, **relative position is what the model is sensitive to**, so inserting a paragraph at the top of a long prompt shifts everything below it. If you rely on prompt caching, covered in module four, that shift invalidates the cache. Put the stable material first and the variable material last, and you get both better caching and more predictable behaviour.\n\nSecond, **there is nothing special about the number 1**. The model has no notion of a document beginning except the patterns it learned. Instructions \"at the top\" are not privileged by architecture; if they seem to be obeyed more, that is a learned habit from training data and it is empirically uneven, which is the subject of the lost-in-the-middle lesson.\n\n## A test you can run\n\nTake a model you can run locally. Give it a 30-token instruction, then 20,000 tokens of filler, then a question whose answer sits in the filler. Move the answer through five positions (start, quarter, middle, three-quarters, end) and record accuracy at each. Twenty minutes, no GPU needed with a small quantised model, and you will have measured your own model's position bias rather than reading somebody else's chart.\n\nThat is the honest way to use any claim about long context: turn it into a five-line experiment on the thing you are actually going to deploy."},"check":{"question":"A model card says the model was pretrained with a 4,096-token context and released with a 131,072-token window using RoPE scaling. What should you expect?","options":[{"text":"The window is a marketing figure and the model will refuse inputs beyond 4,096 tokens.","correct":false,"tempting":"Scaled models really are often oversold, so scepticism is warranted, but the extended window does accept and process the tokens."},{"text":"Behaviour is identical at all lengths, because RoPE encodes relative distance and distance is distance.","correct":false,"tempting":"RoPE's relative property is genuinely elegant, which makes it tempting to conclude that length no longer matters at all."},{"text":"It will accept and process very long inputs, but its accuracy at 100,000 tokens is a separate empirical question from its accuracy at 4,000, and worth measuring yourself.","correct":true},{"text":"Long inputs will be silently truncated to the pretraining length before the model sees them.","correct":false,"tempting":"Silent truncation does happen in some client libraries, so this is a real-world habit that misfires here."}]},"prev":{"slug":"embedding-space-in-practice","title":"Working in embedding space, with free tools","url":"https://addaly.com/learn/how-llms-work/embedding-space-in-practice"},"next":{"slug":"the-shape-of-a-model","title":"Counting the parameters yourself","url":"https://addaly.com/learn/how-llms-work/the-shape-of-a-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'How the model knows what order the words were in', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/position-and-order (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"How the model knows what order the words were in","courseTitle":"How a Language Model Actually Works","module":{"index":1,"slug":"from-characters-to-vectors","title":"From characters to vectors"},"answer":null,"keyIdea":"Attention has no built-in sense of order, so position is injected as extra information, and the way it is injected sets the ceiling on how far a model can usefully read.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"attention-is-a-bag-of-tokens","runs":[{"text":"Attention is a bag of tokens"}]},{"t":"p","runs":[{"text":"Here is a fact that surprises people: the attention operation, on its own, does not know the order of its inputs. Shuffle the tokens and, without a positional signal, each output is a weighted sum over the same set of values: the same set, in a different arrangement. “Dog bites man” and “man bites dog” would be identical."}]},{"t":"p","runs":[{"text":"Something has to supply order. That something is the positional encoding, and the choices made there explain a large part of why long context was hard."}]},{"t":"h2","id":"three-generations-of-answer","runs":[{"text":"Three generations of answer"}]},{"t":"p","runs":[{"text":"Absolute encodings.","b":true},{"text":" The original transformer added a fixed pattern of sines and cosines of different frequencies to each token’s embedding, so position 5 gets a different additive signature from position 400. GPT-2 replaced the fixed pattern with a "},{"text":"learned","i":true},{"text":" table: one vector per position, up to a maximum. Simple, and it has a hard edge: position 1,025 in a model trained to 1,024 has no vector at all."}]},{"t":"p","runs":[{"text":"Relative encodings.","b":true},{"text":" Later work observed that what usually matters is not “this is token 400” but “this token is 3 places before that one”. Encoding relative offsets generalises better to lengths not seen in training."}]},{"t":"p","runs":[{"text":"Rotary position embeddings, or RoPE","b":true},{"text":", which is what nearly every current open model uses. Instead of adding anything, RoPE "},{"text":"rotates","i":true},{"text":" the query and key vectors by an angle proportional to their position, in each two-dimensional slice of the vector. The elegant part: when you take the dot product of a query at position "},{"text":"m","i":true},{"text":" and a key at position "},{"text":"n","i":true},{"text":", the rotations partly cancel and what survives depends on "},{"text":"m − n","i":true},{"text":". Absolute rotation in, relative distance out."}]},{"t":"p","runs":[{"text":"You do not need the trigonometry. You need one consequence: RoPE gives the model a smooth, continuous notion of distance, which means it can be stretched."}]},{"t":"h2","id":"stretching-the-window","runs":[{"text":"Stretching the window"}]},{"t":"p","runs":[{"text":"The stretch is why a model trained on 8,192 tokens can be served with a 128,000-token window. Two families of technique do it:"}]},{"t":"ul","items":[[{"text":"Position interpolation","b":true},{"text":": squash the position indices so that position 100,000 is presented as if it were position 6,250, keeping every angle inside the range the model saw in training. Cheap, and it costs some fine-grained resolution over short distances."}],[{"text":"NTK-aware scaling and YaRN","b":true},{"text":": interpolate the low-frequency components while leaving the high-frequency ones alone, on the argument that fine local distinctions matter more than global ones. In practice these need a short fine-tuning run on long documents to work well."}]]},{"t":"p","runs":[{"text":"This is why you should read a model card carefully. A “128K context” model may have been "},{"text":"trained","i":true},{"text":" at 8K and extended, in which case its behaviour at 100K is a different and usually weaker thing than its behaviour at 8K. The number in the marketing is a capacity, not a promise of quality."}]},{"t":"h2","id":"why-this-matters-for-prompts","runs":[{"text":"Why this matters for prompts"}]},{"t":"p","runs":[{"text":"Two practical points come straight out of the mechanism."}]},{"t":"p","runs":[{"text":"First, "},{"text":"relative position is what the model is sensitive to","b":true},{"text":", so inserting a paragraph at the top of a long prompt shifts everything below it. If you rely on prompt caching, covered in module four, that shift invalidates the cache. Put the stable material first and the variable material last, and you get both better caching and more predictable behaviour."}]},{"t":"p","runs":[{"text":"Second, "},{"text":"there is nothing special about the number 1","b":true},{"text":". The model has no notion of a document beginning except the patterns it learned. Instructions “at the top” are not privileged by architecture; if they seem to be obeyed more, that is a learned habit from training data and it is empirically uneven, which is the subject of the lost-in-the-middle lesson."}]},{"t":"h2","id":"a-test-you-can-run","runs":[{"text":"A test you can run"}]},{"t":"p","runs":[{"text":"Take a model you can run locally. Give it a 30-token instruction, then 20,000 tokens of filler, then a question whose answer sits in the filler. Move the answer through five positions (start, quarter, middle, three-quarters, end) and record accuracy at each. Twenty minutes, no GPU needed with a small quantised model, and you will have measured your own model’s position bias rather than reading somebody else’s chart."}]},{"t":"p","runs":[{"text":"That is the honest way to use any claim about long context: turn it into a five-line experiment on the thing you are actually going to deploy."}]}],"check":{"question":[{"text":"A model card says the model was pretrained with a 4,096-token context and released with a 131,072-token window using RoPE scaling. What should you expect?"}],"options":[{"text":[{"text":"The window is a marketing figure and the model will refuse inputs beyond 4,096 tokens."}],"correct":false,"tempting":[{"text":"Scaled models really are often oversold, so scepticism is warranted, but the extended window does accept and process the tokens."}]},{"text":[{"text":"Behaviour is identical at all lengths, because RoPE encodes relative distance and distance is distance."}],"correct":false,"tempting":[{"text":"RoPE’s relative property is genuinely elegant, which makes it tempting to conclude that length no longer matters at all."}]},{"text":[{"text":"It will accept and process very long inputs, but its accuracy at 100,000 tokens is a separate empirical question from its accuracy at 4,000, and worth measuring yourself."}],"correct":true,"tempting":null},{"text":[{"text":"Long inputs will be silently truncated to the pretraining length before the model sees them."}],"correct":false,"tempting":[{"text":"Silent truncation does happen in some client libraries, so this is a real-world habit that misfires here."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"embedding-space-in-practice","title":"Working in embedding space, with free tools","minutes":10},"next":{"slug":"the-shape-of-a-model","title":"Counting the parameters yourself","minutes":9}}}