Reading a training failure from the numbers
Four failures and their signatures
Training goes wrong in a small number of recognisable ways. Each has a mechanism from the earlier lessons in this module, and each has a signature you can see if you log the right two numbers.
Log these from the first run, always:
grad_norm = sum(p.grad.pow(2).sum() for p in model.parameters() if p.grad is not None).sqrt()plus the loss, and if you can, the mean absolute activation at a couple of layers. Three numbers, and they separate the four cases below.
Failure 1: the loss becomes NaN
Signature. Loss is fine, then infinite or NaN, usually within the first few hundred steps, often abruptly.
Mechanism. Either the gradient exploded — the chain-rule product of factors above 1 — or something in the forward pass overflowed, most often a log(0) or an exp of a large number. In 16-bit floats the maximum representable value is 65,504, which is not a large number at all: a sum of a few hundred squared activations can reach it.
What to do. Gradient clipping, which rescales the whole gradient vector when its norm exceeds a threshold, typically 1.0:
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)This preserves direction and caps magnitude. Add warmup so the first steps are small. If you are training in 16-bit, use bf16 rather than fp16 where the hardware allows: bf16 has the same exponent range as fp32 and simply cannot overflow where fp32 would not.
Failure 2: the loss does not move at all
Signature. A flat line from step one. Gradient norm very small, or exactly zero.
Mechanism. Several candidates, distinguishable by the numbers. If the gradient norm is exactly zero, something is detached from the graph or every ReLU is dead. If it is very small but non-zero, you are in the vanishing-gradient case: a deep stack of saturating activations multiplying small factors together.
Dying ReLU deserves its own note because it is common and invisible. A ReLU unit whose input is negative for every example in the data outputs zero always, and its gradient is zero always, so it never recovers. A large learning rate early in training can kill a large fraction of units in one step. The symptom is a network with far less capacity than its parameter count suggests, and the diagnostic is to count how many units are zero across a batch. Leaky ReLU and GELU exist partly to remove this failure, since neither has an exactly-zero-gradient region.
Failure 3: the loss decreases then explodes
Signature. Healthy descent for a while, then a sharp spike, sometimes recovering, sometimes not.
Mechanism. The optimiser has walked into a region of much sharper curvature, where the learning rate that was fine is now above the stability threshold. This is common in language-model training and is the main reason large runs use a decaying schedule: the same learning rate that is right at step 1,000 is too large at step 100,000.
What to do. Clip gradients, decay the learning rate, and if a spike destroys a long run, restart from the most recent checkpoint with a lower rate and skip the batch that caused it — a genuinely standard practice in large-scale training, not a hack.
Failure 4: training loss falls, validation loss rises
Signature. The two curves separate and the gap widens.
Mechanism. Not an optimisation failure at all. The model is fitting the training set, including its noise. This is the only one of the four that gradient statistics will not diagnose, because nothing is going wrong with the descent — it is succeeding at the wrong objective.
What to do. Early stopping, more data, augmentation, regularisation, a smaller model. Note the order: more data is the most reliable and least often available.
The normalisation layers, in one paragraph
Layer normalisation subtracts the mean and divides by the standard deviation across each token's feature vector, then applies a learned scale and shift. Its effect on this module's subject is direct: it keeps the input to each layer in a range where activation derivatives are not tiny, so the chain-rule factors stay near 1. It also makes the loss surface measurably better conditioned, which is why models with normalisation tolerate much higher learning rates than models without.
RMSNorm, used in most recent large models, drops the mean subtraction and divides by the root-mean-square only. It is slightly cheaper and works about as well, which is a fair summary of the evidence.
What to do first, in order
- Overfit ten examples. Before anything else, check the model can drive the loss on ten samples to near zero. If it cannot, there is a bug, and no amount of tuning will help.
- Check the data pipeline. A shuffled label column produces a model that trains perfectly and predicts noise, and it is more common than any of the failures above.
- Then tune the learning rate, which is worth more than every other hyperparameter combined.
- Then everything else.
Most people do these in reverse order, and lose a week.
The rule to keep
Log the gradient norm from the first run. A norm of zero, a norm in the thousands and a norm that is fine while the loss diverges are three different problems with three different fixes, and the loss curve alone cannot tell them apart.
The one thing to keep
Almost every training failure announces itself in the gradient norm, the activation statistics or the loss curve before it announces itself as a bad model, and each pattern has a specific mechanism behind it.
Before you move on
A network trains normally for 2,000 steps, then the loss spikes and never recovers. Gradient norms were around 0.8 throughout, then hit 400 at the spike. Which action addresses the mechanism most directly?
Pick the one you would defend. Nobody sees your answer.