Keeping a large network from memorising

Five tools, in order of how much they matter

1. Early stopping. Watch validation loss and keep the weights from its best epoch. Free, effective, and it is the one to implement first. Everything else is refinement on top of it.

2. More data, or augmentation. The most effective regulariser there is. For images, augmentation is nearly free performance:

from torchvision import transforms
transforms.Compose([
    transforms.RandomResizedCrop(224, scale=(0.7, 1.0)),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(0.2, 0.2, 0.2),
])

The transformations must preserve the label. Horizontal flip is fine for cats and wrong for handwritten digits, where a flipped 2 is not a 2, and wrong for medical images where laterality is diagnostic. Think before adding one.

For text, augmentation is harder because small edits change meaning; back-translation and synonym replacement are the usual attempts and their gains are modest. For tabular data there is no reliable equivalent, as module 3 said of SMOTE.

3. Weight decay. L2 from module 2, applied through AdamW. 0.01 to 0.1 is the usual range for transformers, 1e-4 for vision networks. Consistently helpful, rarely dramatic.

4. Dropout. During training, randomly zero a fraction of activations in a layer.

nn.Sequential(nn.Linear(512, 256), nn.ReLU(), nn.Dropout(0.3))

The mechanism: no unit can rely on any particular other unit being present, so the network cannot build fragile chains of co-adapted features. It is also usefully read as training an ensemble of many thinned networks that share weights, and averaging them at inference.

At inference dropout must be switched off and the activations scaled to compensate. In PyTorch that is model.train(False), often written as the evaluation-mode call, and forgetting it is one of the most common bugs in PyTorch code: the model returns different predictions on each call and nobody can work out why.

A note on current practice: dropout was essential in the era of large fully connected layers. In convolutional networks batch norm largely replaced it, and in transformers rates are typically low: 0.1 rather than the 0.5 of older tutorials. It remains useful and it is no longer the first thing to reach for.

5. Label smoothing. Instead of training towards a target of exactly 1.0 for the correct class, train towards 0.9, spreading the remaining 0.1 across the others.

This stops the network from driving its logits towards infinity to squeeze the last fraction out of the loss, which is where over-confidence comes from. It improves calibration meaningfully and often improves accuracy slightly.

nn.CrossEntropyLoss(label_smoothing=0.1)

Diagnosing which you need

The same reading as module 4, applied to a training curve.

• Training loss falls, validation loss rises: overfitting. Add regularisation, or stop earlier, or get more data.
• Both plateau high: underfitting. Reduce regularisation, or increase capacity, or train longer.
• Training loss is unstable or spikes: learning rate or gradient clipping, not regularisation.

The second case is worth flagging because the instinct is always to add regularisation, and adding it to a network that is underfitting makes things worse while looking like diligence.

The unfashionable first move

Before any of the above: overfit a tiny subset deliberately.

Take 20 examples and train until the model gets all 20 exactly right. If it cannot, you have a bug: a broken loss, a label misalignment, a learning rate so small nothing moves, a layer not connected. No amount of regularisation fixes a bug, and this test takes two minutes and finds most of them.

Only once the model can memorise 20 examples does it make sense to give it the full dataset and start controlling how much it memorises. Karpathy’s recipe puts this step first for a reason: it separates “the model cannot learn” from “the model learns the wrong thing”, and those need completely different responses.

Regularisation is not free, and the price is legible

One closing point, because it is easy to treat these five tools as costless improvements. Every one of them deliberately makes the model worse on the training data in exchange for a smaller gap. That is the trade from module 4 stated in a new place, and it means there is always a level at which more regularisation stops helping and starts hurting.

The way to find it is not intuition. Fix everything else, sweep one setting (dropout at 0.0, 0.1, 0.3, 0.5), and plot validation loss against it. The curve is almost always U-shaped, the bottom is usually lower than the default you started with, and the whole sweep on a small network takes minutes.

Do them one at a time. Dropout, weight decay and augmentation all reduce effective capacity, so turning up all three together often overshoots into underfitting while each individual setting still looks reasonable. When a heavily regularised network’s training and validation losses have converged at a poor level, the correct move is to remove regularisation, and almost nobody’s first instinct is to try that.