{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"the-feed-forward-half","title":"The feed-forward half, where most of the model is","url":"https://addaly.com/learn/how-llms-work/the-feed-forward-half","position":15,"of":78,"minutes":9,"answer":"Two-thirds of the parameters sit in a per-position network that acts something like a large set of pattern-triggered lookups, and its neurons respond to many unrelated things at once.","keyIdea":"Two-thirds of the parameters sit in a per-position network that acts something like a large set of pattern-triggered lookups, and its neurons respond to many unrelated things at once."},"body":{"format":"markdown","text":"## The bigger half, and the less discussed one\n\nAttention gets the attention. But in the model you counted in module one, attention was 42 million parameters per layer and the feed-forward network was 176 million. Roughly two-thirds of a transformer is this second half, and it operates on each position entirely on its own, with no knowledge of its neighbours.\n\nIts structure is simple:\n\n```python\ndef mlp(x):                        # x is one position's vector, width 4096\n    a = gelu(x @ W_up)             # project up to 14336\n    return a @ W_down              # project back down to 4096\n```\n\nUp, nonlinearity, down. Modern models use a gated variant, SwiGLU, which computes two up-projections and multiplies them element-wise before the down-projection: three matrices instead of two, and a consistent small quality gain that nobody has a fully satisfying theory for.\n\n## Reading it as a key-value store\n\nThe most useful way to think about this half, and the one supported by the strongest evidence, is that the up-projection acts like a set of pattern detectors and the down-projection like the contributions those patterns write back.\n\nEach row of `W_up` is compared against the incoming vector by a dot product. If the vector matches that row's pattern, the corresponding neuron activates. Each activated neuron then adds its own row of `W_down` into the residual stream. With 14,336 neurons per layer and 32 layers, that is about 460,000 pattern-to-contribution pairs.\n\nWork on locating and editing factual associations supports this reading. Researchers have identified specific mid-layer feed-forward weights that, when modified, change a stored association: making a model say the Eiffel Tower is in Rome, consistently, across paraphrases, without retraining. That is real evidence for facts living here in some form.\n\nState the limits with equal firmness. The editing methods work on simple subject-relation-object facts, they do not always generalise the way you would want, they sometimes damage nearby knowledge, and there is no map from a parameter to a fact. \"This is where facts live\" is a well-supported direction, not a solved account.\n\n## Neurons are polysemantic\n\nThe tempting next step is to look for the \"Paris neuron\". Do it and you will be disappointed. Individual neurons respond to collections of apparently unrelated things: one might fire on legal citations, chess notation, and a particular kind of German compound.\n\nThe leading explanation is **superposition**. A model needs to represent far more distinct features than it has neurons, so it stores them in overlapping directions that are almost, but not quite, independent. Each neuron participates in many features and each feature is spread across many neurons.\n\nThe practical response has been sparse autoencoders: train a wider, sparse layer to reconstruct the model's activations, on the argument that the wider basis can separate what the narrow one entangled. Applied to production-scale models, this has produced large dictionaries of features that look far more interpretable than raw neurons, including features that can be amplified to change behaviour in a legible way. It is a genuine advance and it is not a complete solution: reconstruction is imperfect, feature counts are chosen rather than discovered, and how much of the model's computation these dictionaries capture remains open.\n\n## Why the width is 4×\n\nThe up-projection is conventionally about four times the model width. Nobody has a principled derivation for four; it emerged empirically and stuck. Mixture-of-experts models, covered shortly, change this ratio radically, which is itself evidence that the number is a convention rather than a law.\n\n## What follows for practice\n\nThree things.\n\n**Knowledge and processing are not cleanly separable.** Attention decides what to look at; the feed-forward half decides what to do with it, including supplying associations from the weights. Neither half alone \"knows\" anything.\n\n**You cannot edit a fact by prompting.** The association lives in weights that your prompt does not touch. You can *override* it with context, which is what retrieval does, but the weights are unchanged and the model may still fall back on them.\n\n**Model size buys storage as much as it buys reasoning.** Much of what a larger model gives you is a bigger set of these pattern-to-contribution pairs. That is part of why retrieval can let a small model match a large one on knowledge-heavy tasks and not on tasks that need many steps of computation."},"check":{"question":"A model insists a discontinued product is still available, even after you paste the current catalogue into the prompt. What does the mechanism suggest is happening?","options":[{"text":"The catalogue was placed too late in the prompt for the model to attend to it.","correct":false,"tempting":"Position effects on long prompts are real and often are the cause of ignored context, so this is a fair operational guess."},{"text":"The model has cached its earlier answer and is repeating it.","correct":false,"tempting":"Response caching exists in many deployments, which makes a stale-cache theory reasonable at the system level."},{"text":"The association is stored in the feed-forward weights, and context can compete with it but does not replace it, so the model can still fall back on what the weights say.","correct":true},{"text":"The catalogue was tokenised as data rather than instructions, so the model treated it as inert.","correct":false,"tempting":"There is no such distinction in the input, but the intuition that models separate instructions from data is widespread."}]},"prev":{"slug":"many-heads-and-what-they-do","title":"Many heads, and what has actually been found inside them","url":"https://addaly.com/learn/how-llms-work/many-heads-and-what-they-do"},"next":{"slug":"normalisation-and-stability","title":"Normalisation, and why training is fragile","url":"https://addaly.com/learn/how-llms-work/normalisation-and-stability"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The feed-forward half, where most of the model is', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/the-feed-forward-half (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The feed-forward half, where most of the model is","courseTitle":"How a Language Model Actually Works","module":{"index":2,"slug":"inside-the-block","title":"Inside the block"},"answer":null,"keyIdea":"Two-thirds of the parameters sit in a per-position network that acts something like a large set of pattern-triggered lookups, and its neurons respond to many unrelated things at once.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-bigger-half-and-the-less-discussed-one","runs":[{"text":"The bigger half, and the less discussed one"}]},{"t":"p","runs":[{"text":"Attention gets the attention. But in the model you counted in module one, attention was 42 million parameters per layer and the feed-forward network was 176 million. Roughly two-thirds of a transformer is this second half, and it operates on each position entirely on its own, with no knowledge of its neighbours."}]},{"t":"p","runs":[{"text":"Its structure is simple:"}]},{"t":"code","lang":"python","label":"Python","text":"def mlp(x):                        # x is one position's vector, width 4096\n    a = gelu(x @ W_up)             # project up to 14336\n    return a @ W_down              # project back down to 4096"},{"t":"p","runs":[{"text":"Up, nonlinearity, down. Modern models use a gated variant, SwiGLU, which computes two up-projections and multiplies them element-wise before the down-projection: three matrices instead of two, and a consistent small quality gain that nobody has a fully satisfying theory for."}]},{"t":"h2","id":"reading-it-as-a-key-value-store","runs":[{"text":"Reading it as a key-value store"}]},{"t":"p","runs":[{"text":"The most useful way to think about this half, and the one supported by the strongest evidence, is that the up-projection acts like a set of pattern detectors and the down-projection like the contributions those patterns write back."}]},{"t":"p","runs":[{"text":"Each row of "},{"text":"W_up","code":true},{"text":" is compared against the incoming vector by a dot product. If the vector matches that row’s pattern, the corresponding neuron activates. Each activated neuron then adds its own row of "},{"text":"W_down","code":true},{"text":" into the residual stream. With 14,336 neurons per layer and 32 layers, that is about 460,000 pattern-to-contribution pairs."}]},{"t":"p","runs":[{"text":"Work on locating and editing factual associations supports this reading. Researchers have identified specific mid-layer feed-forward weights that, when modified, change a stored association: making a model say the Eiffel Tower is in Rome, consistently, across paraphrases, without retraining. That is real evidence for facts living here in some form."}]},{"t":"p","runs":[{"text":"State the limits with equal firmness. The editing methods work on simple subject-relation-object facts, they do not always generalise the way you would want, they sometimes damage nearby knowledge, and there is no map from a parameter to a fact. “This is where facts live” is a well-supported direction, not a solved account."}]},{"t":"h2","id":"neurons-are-polysemantic","runs":[{"text":"Neurons are polysemantic"}]},{"t":"p","runs":[{"text":"The tempting next step is to look for the “Paris neuron”. Do it and you will be disappointed. Individual neurons respond to collections of apparently unrelated things: one might fire on legal citations, chess notation, and a particular kind of German compound."}]},{"t":"p","runs":[{"text":"The leading explanation is "},{"text":"superposition","b":true},{"text":". A model needs to represent far more distinct features than it has neurons, so it stores them in overlapping directions that are almost, but not quite, independent. Each neuron participates in many features and each feature is spread across many neurons."}]},{"t":"p","runs":[{"text":"The practical response has been sparse autoencoders: train a wider, sparse layer to reconstruct the model’s activations, on the argument that the wider basis can separate what the narrow one entangled. Applied to production-scale models, this has produced large dictionaries of features that look far more interpretable than raw neurons, including features that can be amplified to change behaviour in a legible way. It is a genuine advance and it is not a complete solution: reconstruction is imperfect, feature counts are chosen rather than discovered, and how much of the model’s computation these dictionaries capture remains open."}]},{"t":"h2","id":"why-the-width-is-4","runs":[{"text":"Why the width is 4×"}]},{"t":"p","runs":[{"text":"The up-projection is conventionally about four times the model width. Nobody has a principled derivation for four; it emerged empirically and stuck. Mixture-of-experts models, covered shortly, change this ratio radically, which is itself evidence that the number is a convention rather than a law."}]},{"t":"h2","id":"what-follows-for-practice","runs":[{"text":"What follows for practice"}]},{"t":"p","runs":[{"text":"Three things."}]},{"t":"p","runs":[{"text":"Knowledge and processing are not cleanly separable.","b":true},{"text":" Attention decides what to look at; the feed-forward half decides what to do with it, including supplying associations from the weights. Neither half alone “knows” anything."}]},{"t":"p","runs":[{"text":"You cannot edit a fact by prompting.","b":true},{"text":" The association lives in weights that your prompt does not touch. You can "},{"text":"override","i":true},{"text":" it with context, which is what retrieval does, but the weights are unchanged and the model may still fall back on them."}]},{"t":"p","runs":[{"text":"Model size buys storage as much as it buys reasoning.","b":true},{"text":" Much of what a larger model gives you is a bigger set of these pattern-to-contribution pairs. That is part of why retrieval can let a small model match a large one on knowledge-heavy tasks and not on tasks that need many steps of computation."}]}],"check":{"question":[{"text":"A model insists a discontinued product is still available, even after you paste the current catalogue into the prompt. What does the mechanism suggest is happening?"}],"options":[{"text":[{"text":"The catalogue was placed too late in the prompt for the model to attend to it."}],"correct":false,"tempting":[{"text":"Position effects on long prompts are real and often are the cause of ignored context, so this is a fair operational guess."}]},{"text":[{"text":"The model has cached its earlier answer and is repeating it."}],"correct":false,"tempting":[{"text":"Response caching exists in many deployments, which makes a stale-cache theory reasonable at the system level."}]},{"text":[{"text":"The association is stored in the feed-forward weights, and context can compete with it but does not replace it, so the model can still fall back on what the weights say."}],"correct":true,"tempting":null},{"text":[{"text":"The catalogue was tokenised as data rather than instructions, so the model treated it as inert."}],"correct":false,"tempting":[{"text":"There is no such distinction in the input, but the intuition that models separate instructions from data is widespread."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"many-heads-and-what-they-do","title":"Many heads, and what has actually been found inside them","minutes":9},"next":{"slug":"normalisation-and-stability","title":"Normalisation, and why training is fragile","minutes":8}}}