Diagnosing a model from the shape of its loss curve

The plot you should make every time

Two lines, x-axis is epochs or steps, y-axis is loss. Training loss and validation loss on the same axes. This one plot answers more questions than any other artefact in machine learning, and it takes four lines of code.

import matplotlib.pyplot as plt
plt.plot(history["train_loss"], label="train")
plt.plot(history["val_loss"], label="validation")
plt.legend(); plt.xlabel("epoch"); plt.ylabel("loss")

For scikit-learn models that do not record a history, HistGradientBoostingClassifier exposes validation_score_ per iteration, and SGDClassifier can be driven with partial_fit in a loop to build one manually. For anything in PyTorch you write it yourself, and you should.

Seven shapes, and what each one means

Both lines falling, still falling at the end. Undertrained. Train longer. The single most common mistake in a first project is stopping here and concluding the model is weak.

Both fall, then flatten together with a small gap. This is the healthy result. The remaining gap is normal; a gap of exactly zero is suspicious and usually means the model is too simple to have fitted anything specific.

Train falls, validation flattens and then rises. Overfitting, in its textbook form. The point where validation turns is where you should have stopped, and early_stopping does exactly that automatically.

Both flat and high from step one. Nothing is learning. Check that the learning rate is not absurdly small, that the features actually reach the model, and that the labels are not shuffled relative to the features. A model that predicts the base rate forever usually has a plumbing fault, not a modelling one.

Loss becomes NaN. Learning rate far too high, or a division by zero somewhere in the features, or a log of zero in a custom loss. Drop the learning rate by a factor of ten first; it is the cause about eight times out of ten.

Validation loss below training loss. Surprising, and it has two innocent causes and one guilty one. Innocent: regularisation such as dropout is active during training and off during evaluation, so training is measured on a handicapped model; and training loss is often averaged over the epoch while validation is measured at the end, so training looks worse by the amount the model improved during that epoch. Guilty: your validation set is easier than your training set, which happens when the split is not random.

A sudden cliff downward. Usually a learning rate schedule stepping down, as described in the previous lesson. Sometimes an unfreezing event in a fine-tuning run. Rarely a real discovery.

The noise is information too

A validation curve that jumps around wildly from epoch to epoch is telling you the validation set is small. The measurement noise on a set of 200 rows is large; with 5,000 rows the same model gives a smooth line. If your curve looks like static, do not tune against individual points: you will be chasing noise, and you will pick the epoch that got lucky.

This matters because early stopping picks the minimum of a noisy curve, which is biased low. That is fine for choosing when to stop and misleading if you then report that minimum as your score. Report the test set, once, afterwards.

Learning curve is a different plot

Worth separating now because the names get used loosely. The plot above has epochs on the x-axis. A learning curve has training set size on the x-axis: fit on 10% of the data, then 20%, and so on, plotting train and validation score at each size.

That second plot answers a different and very expensive question: would more data help? If the validation curve is still climbing at 100% of your data, collecting more will help. If it flattened at 40%, more of the same data will not, and you need better features or a different model. We build it properly in module 4; the distinction is worth having now so the two are never confused.

Save the history

A small habit with a large payoff: store the loss history alongside the model, with the hyperparameters, the seed, and the data version. Three weeks later someone asks why the model changed, and the answer is almost always visible in the two curves side by side. Reconstructing it from memory is not possible, and re-running it is often not either, because the data has moved on.