The feed-forward half, where most of the model is

The bigger half, and the less discussed one

Attention gets the attention. But in the model you counted in module one, attention was 42 million parameters per layer and the feed-forward network was 176 million. Roughly two-thirds of a transformer is this second half, and it operates on each position entirely on its own, with no knowledge of its neighbours.

Its structure is simple:

def mlp(x):                        # x is one position's vector, width 4096
    a = gelu(x @ W_up)             # project up to 14336
    return a @ W_down              # project back down to 4096

Up, nonlinearity, down. Modern models use a gated variant, SwiGLU, which computes two up-projections and multiplies them element-wise before the down-projection: three matrices instead of two, and a consistent small quality gain that nobody has a fully satisfying theory for.

Reading it as a key-value store

The most useful way to think about this half, and the one supported by the strongest evidence, is that the up-projection acts like a set of pattern detectors and the down-projection like the contributions those patterns write back.

Each row of W_up is compared against the incoming vector by a dot product. If the vector matches that row’s pattern, the corresponding neuron activates. Each activated neuron then adds its own row of W_down into the residual stream. With 14,336 neurons per layer and 32 layers, that is about 460,000 pattern-to-contribution pairs.

Work on locating and editing factual associations supports this reading. Researchers have identified specific mid-layer feed-forward weights that, when modified, change a stored association: making a model say the Eiffel Tower is in Rome, consistently, across paraphrases, without retraining. That is real evidence for facts living here in some form.

State the limits with equal firmness. The editing methods work on simple subject-relation-object facts, they do not always generalise the way you would want, they sometimes damage nearby knowledge, and there is no map from a parameter to a fact. “This is where facts live” is a well-supported direction, not a solved account.

Neurons are polysemantic

The tempting next step is to look for the “Paris neuron”. Do it and you will be disappointed. Individual neurons respond to collections of apparently unrelated things: one might fire on legal citations, chess notation, and a particular kind of German compound.

The leading explanation is superposition. A model needs to represent far more distinct features than it has neurons, so it stores them in overlapping directions that are almost, but not quite, independent. Each neuron participates in many features and each feature is spread across many neurons.

The practical response has been sparse autoencoders: train a wider, sparse layer to reconstruct the model’s activations, on the argument that the wider basis can separate what the narrow one entangled. Applied to production-scale models, this has produced large dictionaries of features that look far more interpretable than raw neurons, including features that can be amplified to change behaviour in a legible way. It is a genuine advance and it is not a complete solution: reconstruction is imperfect, feature counts are chosen rather than discovered, and how much of the model’s computation these dictionaries capture remains open.

Why the width is 4×

The up-projection is conventionally about four times the model width. Nobody has a principled derivation for four; it emerged empirically and stuck. Mixture-of-experts models, covered shortly, change this ratio radically, which is itself evidence that the number is a convention rather than a law.

What follows for practice

Three things.

Knowledge and processing are not cleanly separable. Attention decides what to look at; the feed-forward half decides what to do with it, including supplying associations from the weights. Neither half alone “knows” anything.

You cannot edit a fact by prompting. The association lives in weights that your prompt does not touch. You can override it with context, which is what retrieval does, but the weights are unchanged and the model may still fall back on them.

Model size buys storage as much as it buys reasoning. Much of what a larger model gives you is a bigger set of these pattern-to-contribution pairs. That is part of why retrieval can let a small model match a large one on knowledge-heavy tasks and not on tasks that need many steps of computation.