Normalisation, and why training is fragile

The boring layer that decides whether training works

In the four-line block, norm looked like a detail worth skipping. For understanding what the model computes, it is. For understanding why training a large model is difficult, it is central.

The problem

Stack 80 blocks, each adding contributions to a running total, and the numbers in that total drift. If they grow, activations saturate and gradients explode; a single bad batch can send the loss to infinity and the run is dead. If they shrink, gradients vanish and the lower layers stop learning. Neither failure announces itself politely.

Normalisation rescales the vector at each position so its numbers sit in a predictable range before the next sublayer reads it.

LayerNorm subtracts the mean and divides by the standard deviation across the vector’s dimensions, then applies a learned scale and shift. RMSNorm, used by most current models, skips the mean subtraction and divides by the root mean square only. It is cheaper, and it works about as well, which suggests the re-centring was never the important part.

Note what these do not do: they operate across the features of a single position, not across the batch. Batch normalisation, which does normalise across examples, is standard in vision and almost unused here, because it behaves badly with variable-length sequences and makes each example’s output depend on which other examples happened to share its batch.

Pre-norm versus post-norm

The original 2017 transformer put the normalisation after each sublayer and its residual addition. Training those models needed a learning-rate warmup, and deep ones were unstable.

Nearly everything since normalises before each sublayer:

x = x + attention(norm(x))     # pre-norm: the residual path is untouched
x = norm(x + attention(x))     # post-norm: the residual path passes through norm

The difference looks cosmetic and is not. In pre-norm, there is a clean additive path from input to output with no normalisation in the way, so gradients reach the early layers directly. That is what makes 80-layer models trainable without heroics.

The cost is a known one: pre-norm models tend to see the residual stream’s magnitude grow steadily with depth, and there is evidence the later layers contribute proportionally less as a result. Variants that normalise in both places, or scale the residual branch, exist to claw some of that back. This is an area where practice is still moving.

What a training run actually looks like

Read a published training log (several large open models have released theirs), and you will see something the marketing never shows: loss spikes. The loss is descending smoothly, then jumps, sometimes catastrophically. Teams respond by rolling back to an earlier checkpoint, skipping the offending batch of data, and continuing.

That is a normal part of training a very large model. It is caused by some combination of unlucky data, numerical overflow and optimiser state, and mitigations include gradient clipping, careful initialisation, and computing certain operations in higher precision.

The practical lesson for anyone fine-tuning: if your loss spikes and does not recover, lower the learning rate first, then check for pathological examples in your data. Both are far more likely than a subtle bug in your model code.

Precision and where it must be kept high

Weights are usually stored in bf16, a 16-bit format with the same exponent range as fp32 and fewer mantissa bits: the range matters more than the precision for stability, which is why bf16 largely replaced fp16 for training.

Some operations still run in fp32 regardless: the softmax, the normalisation statistics, and the optimiser’s accumulators. Doing them in 16-bit is a well-known way to produce a run that trains for two days and then quietly diverges.

Why you should care as a user

You will not tune these. But when a provider’s model behaves differently after an update, or a fine-tune degrades in a way that looks like nonsense, the cause is often numerical rather than conceptual. And when you read that a lab needed several restarts to complete a training run, you now know it is a description of ordinary engineering rather than an admission of incompetence.