What one transformer block actually does

The same block, dozens of times

A large language model is one small design repeated. A 7-billion-parameter model might stack 32 identical blocks; larger ones run 80 or more. Each block has two halves and looks, in essence, like this:

def block(x):
    x = x + attention(norm(x))   # mix information across positions
    x = x + mlp(norm(x))         # process each position on its own
    return x

Four lines carry almost everything worth understanding about the architecture. Take them one at a time.

The residual stream: everything adds, nothing replaces

Look at x = x + .... The sublayer does not produce a new x. It produces something that gets added to the existing x.

So the vector at a given position is best pictured as a running total, or a shared notepad that travels through the model. It starts as the token's embedding. Block 1 reads it and writes a contribution. Block 2 reads the sum and writes another. By block 20, the notepad holds the original embedding plus everything the previous 19 blocks decided to add.

This is called the residual stream, and it explains several observations. Gradients reach the early layers during training, because there is a straight path from the output back to the input. Later layers can undo or sharpen what earlier ones wrote, because addition is reversible in a way that overwriting is not. And removing one middle block from a trained model often degrades output only mildly, which would be inexplicable if each block were a required stage in a pipeline.

The attention half

Covered in the previous lesson: it gathers information from earlier positions and adds it in. It is the only part of the block where positions talk to each other at all. Everything else operates on one position at a time, in complete ignorance of its neighbours.

The feed-forward half, where the parameters are

The second sublayer is a small network applied identically and independently at every position. It projects the vector up to roughly four times its width, applies a nonlinearity, and projects back down.

That is around two-thirds of the model's parameters. Not attention. This half.

What does it do? The best current account, from interpretability work including model-editing experiments, is that it behaves somewhat like a large set of key-value lookups: certain patterns in the incoming vector trigger certain stored contributions, and this is where much of the model's factual association appears to live. Researchers have located and edited specific factual associations in these layers with some success. Treat that as a well-supported direction of evidence, not a settled account — the picture is contested in its details and there is no clean map from a parameter to a fact.

Normalisation

The norm call rescales the vector before each sublayer reads it, keeping numbers in a workable range. It is essential for training stability and it is not conceptually interesting. Skip it in your mental model.

The end of the stack

After the final block, the notepad at the last position is compared against the embedding table — one dot product per vocabulary entry — producing one raw score, a logit, for every possible next token. That comparison step is the unembedding, and lesson seven is about what happens to those scores.

A useful and slightly unnerving trick: you can apply that same final comparison to an intermediate layer and read off what the model would say if it stopped there. Do it layer by layer on a factual question and you often watch the answer come into focus over the upper half of the stack. It is suggestive. It is not proof that the layers form neat stages, and the blocks carry no labels for syntax, semantics or reasoning. They are all the same shape, doing the same two things, over and over.