{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"the-block","title":"What one transformer block actually does","url":"https://addaly.com/learn/how-llms-work/the-block","position":13,"of":78,"minutes":8,"answer":"Every layer adds to a running total rather than replacing it, and most parameters sit in the feed-forward half.","keyIdea":"Every layer adds to a running total rather than replacing it, and most parameters sit in the feed-forward half."},"body":{"format":"markdown","text":"## The same block, dozens of times\n\nA large language model is one small design repeated. A 7-billion-parameter model might stack 32 identical blocks; larger ones run 80 or more. Each block has two halves and looks, in essence, like this:\n\n```python\ndef block(x):\n    x = x + attention(norm(x))   # mix information across positions\n    x = x + mlp(norm(x))         # process each position on its own\n    return x\n```\n\nFour lines carry almost everything worth understanding about the architecture. Take them one at a time.\n\n## The residual stream: everything adds, nothing replaces\n\nLook at `x = x + ...`. The sublayer does not produce a new `x`. It produces something that gets **added** to the existing `x`.\n\nSo the vector at a given position is best pictured as a running total, or a shared notepad that travels through the model. It starts as the token's embedding. Block 1 reads it and writes a contribution. Block 2 reads the sum and writes another. By block 20, the notepad holds the original embedding plus everything the previous 19 blocks decided to add.\n\nThis is called the residual stream, and it explains several observations. Gradients reach the early layers during training, because there is a straight path from the output back to the input. Later layers can undo or sharpen what earlier ones wrote, because addition is reversible in a way that overwriting is not. And removing one middle block from a trained model often degrades output only mildly, which would be inexplicable if each block were a required stage in a pipeline.\n\n> Figure: The residual stream through one block. Layers, top to bottom: The vector arriving from block N−1 (A running total, not a fresh value: the embedding plus everything written so far); + attention(norm(x)) (The only step in the whole block where positions talk to each other); + mlp(norm(x)) (Two-thirds of the parameters, applied at each position in ignorance of its neighbours); The vector leaving for block N+1 (The same notepad, with two more contributions on it). Each sublayer adds; nothing overwrites. That is why gradients reach the first layer during training, why a later block can sharpen or undo what an earlier one wrote, and why removing one middle block of a trained model usually degrades the output only mildly.\n\n## The attention half\n\nCovered in the previous lesson: it gathers information from earlier positions and adds it in. It is the only part of the block where positions talk to each other at all. Everything else operates on one position at a time, in complete ignorance of its neighbours.\n\n## The feed-forward half, where the parameters are\n\nThe second sublayer is a small network applied identically and independently at every position. It projects the vector up to roughly four times its width, applies a nonlinearity, and projects back down.\n\nThat is around two-thirds of the model's parameters. Not attention. This half.\n\nWhat does it do? The best current account, from interpretability work including model-editing experiments, is that it behaves somewhat like a large set of key-value lookups: certain patterns in the incoming vector trigger certain stored contributions, and this is where much of the model's factual association appears to live. Researchers have located and edited specific factual associations in these layers with some success. Treat that as a well-supported direction of evidence, not a settled account: the picture is contested in its details and there is no clean map from a parameter to a fact.\n\n## Normalisation\n\nThe `norm` call rescales the vector before each sublayer reads it, keeping numbers in a workable range. It is essential for training stability and it is not conceptually interesting. Skip it in your mental model.\n\n## The end of the stack\n\nAfter the final block, the notepad at the last position is compared against the embedding table (one dot product per vocabulary entry) producing one raw score, a logit, for every possible next token. That comparison step is the unembedding, and lesson seven is about what happens to those scores.\n\nA useful and slightly unnerving trick: you can apply that same final comparison to an intermediate layer and read off what the model would say if it stopped there. Do it layer by layer on a factual question and you often watch the answer come into focus over the upper half of the stack. It is suggestive. It is not proof that the layers form neat stages, and the blocks carry no labels for syntax, semantics or reasoning. They are all the same shape, doing the same two things, over and over."},"check":{"question":"A model has 40 identical blocks. What best describes what block 20 receives at a given position?","options":[{"text":"A fresh representation built by block 19; the original token embedding is no longer part of it.","correct":false,"tempting":"Layers are naturally pictured as a relay, where each one hands a finished new object to the next."},{"text":"A rough draft of the answer in words, which each block makes more fluent.","correct":false,"tempting":"The final layer produces words, so it is easy to assume the intermediate layers hold rougher words."},{"text":"Block 19's output, plus the prompt text, which every block can re-read as needed.","correct":false,"tempting":"The prompt is obviously available to the model, so assuming it stays readable throughout feels safe."},{"text":"A running sum: the original embedding plus every contribution the earlier blocks chose to add, which block 20 reads and adds to in turn.","correct":true}]},"prev":{"slug":"the-causal-mask-and-parallel-training","title":"Why training is parallel and generation is not","url":"https://addaly.com/learn/how-llms-work/the-causal-mask-and-parallel-training"},"next":{"slug":"many-heads-and-what-they-do","title":"Many heads, and what has actually been found inside them","url":"https://addaly.com/learn/how-llms-work/many-heads-and-what-they-do"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What one transformer block actually does', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/the-block (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What one transformer block actually does","courseTitle":"How a Language Model Actually Works","module":{"index":2,"slug":"inside-the-block","title":"Inside the block"},"answer":null,"keyIdea":"Every layer adds to a running total rather than replacing it, and most parameters sit in the feed-forward half.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"The residual stream through one block","layers":[{"label":"The vector arriving from block N−1","detail":"A running total, not a fresh value: the embedding plus everything written so far"},{"label":"+ attention(norm(x))","detail":"The only step in the whole block where positions talk to each other"},{"label":"+ mlp(norm(x))","detail":"Two-thirds of the parameters, applied at each position in ignorance of its neighbours"},{"label":"The vector leaving for block N+1","detail":"The same notepad, with two more contributions on it"}],"caption":"Each sublayer adds; nothing overwrites. That is why gradients reach the first layer during training, why a later block can sharpen or undo what an earlier one wrote, and why removing one middle block of a trained model usually degrades the output only mildly."},"alt":"Figure: The residual stream through one block. Layers, top to bottom: The vector arriving from block N−1 (A running total, not a fresh value: the embedding plus everything written so far); + attention(norm(x)) (The only step in the whole block where positions talk to each other); + mlp(norm(x)) (Two-thirds of the parameters, applied at each position in ignorance of its neighbours); The vector leaving for block N+1 (The same notepad, with two more contributions on it). Each sublayer adds; nothing overwrites. That is why gradients reach the first layer during training, why a later block can sharpen or undo what an earlier one wrote, and why removing one middle block of a trained model usually degrades the output only mildly."}},"blocks":[{"t":"h2","id":"the-same-block-dozens-of-times","runs":[{"text":"The same block, dozens of times"}]},{"t":"p","runs":[{"text":"A large language model is one small design repeated. A 7-billion-parameter model might stack 32 identical blocks; larger ones run 80 or more. Each block has two halves and looks, in essence, like this:"}]},{"t":"code","lang":"python","label":"Python","text":"def block(x):\n    x = x + attention(norm(x))   # mix information across positions\n    x = x + mlp(norm(x))         # process each position on its own\n    return x"},{"t":"p","runs":[{"text":"Four lines carry almost everything worth understanding about the architecture. Take them one at a time."}]},{"t":"h2","id":"the-residual-stream-everything-adds-nothing-replaces","runs":[{"text":"The residual stream: everything adds, nothing replaces"}]},{"t":"p","runs":[{"text":"Look at "},{"text":"x = x + ...","code":true},{"text":". The sublayer does not produce a new "},{"text":"x","code":true},{"text":". It produces something that gets "},{"text":"added","b":true},{"text":" to the existing "},{"text":"x","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"So the vector at a given position is best pictured as a running total, or a shared notepad that travels through the model. It starts as the token’s embedding. Block 1 reads it and writes a contribution. Block 2 reads the sum and writes another. By block 20, the notepad holds the original embedding plus everything the previous 19 blocks decided to add."}]},{"t":"p","runs":[{"text":"This is called the residual stream, and it explains several observations. Gradients reach the early layers during training, because there is a straight path from the output back to the input. Later layers can undo or sharpen what earlier ones wrote, because addition is reversible in a way that overwriting is not. And removing one middle block from a trained model often degrades output only mildly, which would be inexplicable if each block were a required stage in a pipeline."}]},{"t":"h2","id":"the-attention-half","runs":[{"text":"The attention half"}]},{"t":"p","runs":[{"text":"Covered in the previous lesson: it gathers information from earlier positions and adds it in. It is the only part of the block where positions talk to each other at all. Everything else operates on one position at a time, in complete ignorance of its neighbours."}]},{"t":"h2","id":"the-feed-forward-half-where-the-parameters-are","runs":[{"text":"The feed-forward half, where the parameters are"}]},{"t":"p","runs":[{"text":"The second sublayer is a small network applied identically and independently at every position. It projects the vector up to roughly four times its width, applies a nonlinearity, and projects back down."}]},{"t":"p","runs":[{"text":"That is around two-thirds of the model’s parameters. Not attention. This half."}]},{"t":"p","runs":[{"text":"What does it do? The best current account, from interpretability work including model-editing experiments, is that it behaves somewhat like a large set of key-value lookups: certain patterns in the incoming vector trigger certain stored contributions, and this is where much of the model’s factual association appears to live. Researchers have located and edited specific factual associations in these layers with some success. Treat that as a well-supported direction of evidence, not a settled account: the picture is contested in its details and there is no clean map from a parameter to a fact."}]},{"t":"h2","id":"normalisation","runs":[{"text":"Normalisation"}]},{"t":"p","runs":[{"text":"The "},{"text":"norm","code":true},{"text":" call rescales the vector before each sublayer reads it, keeping numbers in a workable range. It is essential for training stability and it is not conceptually interesting. Skip it in your mental model."}]},{"t":"h2","id":"the-end-of-the-stack","runs":[{"text":"The end of the stack"}]},{"t":"p","runs":[{"text":"After the final block, the notepad at the last position is compared against the embedding table (one dot product per vocabulary entry) producing one raw score, a logit, for every possible next token. That comparison step is the unembedding, and lesson seven is about what happens to those scores."}]},{"t":"p","runs":[{"text":"A useful and slightly unnerving trick: you can apply that same final comparison to an intermediate layer and read off what the model would say if it stopped there. Do it layer by layer on a factual question and you often watch the answer come into focus over the upper half of the stack. It is suggestive. It is not proof that the layers form neat stages, and the blocks carry no labels for syntax, semantics or reasoning. They are all the same shape, doing the same two things, over and over."}]}],"check":{"question":[{"text":"A model has 40 identical blocks. What best describes what block 20 receives at a given position?"}],"options":[{"text":[{"text":"A fresh representation built by block 19; the original token embedding is no longer part of it."}],"correct":false,"tempting":[{"text":"Layers are naturally pictured as a relay, where each one hands a finished new object to the next."}]},{"text":[{"text":"A rough draft of the answer in words, which each block makes more fluent."}],"correct":false,"tempting":[{"text":"The final layer produces words, so it is easy to assume the intermediate layers hold rougher words."}]},{"text":[{"text":"Block 19’s output, plus the prompt text, which every block can re-read as needed."}],"correct":false,"tempting":[{"text":"The prompt is obviously available to the model, so assuming it stays readable throughout feels safe."}]},{"text":[{"text":"A running sum: the original embedding plus every contribution the earlier blocks chose to add, which block 20 reads and adds to in turn."}],"correct":true,"tempting":null}],"order":[0,3,2,1],"why":null},"prev":{"slug":"the-causal-mask-and-parallel-training","title":"Why training is parallel and generation is not","minutes":8},"next":{"slug":"many-heads-and-what-they-do","title":"Many heads, and what has actually been found inside them","minutes":9}}}