{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"regularising-a-network","title":"Keeping a large network from memorising","url":"https://addaly.com/learn/machine-learning-foundations/regularising-a-network","position":79,"of":92,"minutes":8,"answer":"Early stopping and more data outrank every other regulariser, dropout must be disabled at inference or predictions become random.","keyIdea":"Early stopping and more data outrank every other regulariser, dropout must be disabled at inference or predictions become random, and before regularising anything you should confirm the network can deliberately memorise twenty examples: otherwise you are tuning around a bug."},"body":{"format":"markdown","text":"## Five tools, in order of how much they matter\n\n**1. Early stopping.** Watch validation loss and keep the weights from its best epoch. Free, effective, and it is the one to implement first. Everything else is refinement on top of it.\n\n**2. More data, or augmentation.** The most effective regulariser there is. For images, augmentation is nearly free performance:\n\n```python\nfrom torchvision import transforms\ntransforms.Compose([\n    transforms.RandomResizedCrop(224, scale=(0.7, 1.0)),\n    transforms.RandomHorizontalFlip(),\n    transforms.ColorJitter(0.2, 0.2, 0.2),\n])\n```\n\nThe transformations must preserve the label. Horizontal flip is fine for cats and wrong for handwritten digits, where a flipped 2 is not a 2, and wrong for medical images where laterality is diagnostic. Think before adding one.\n\nFor text, augmentation is harder because small edits change meaning; back-translation and synonym replacement are the usual attempts and their gains are modest. For tabular data there is no reliable equivalent, as module 3 said of SMOTE.\n\n**3. Weight decay.** L2 from module 2, applied through AdamW. 0.01 to 0.1 is the usual range for transformers, 1e-4 for vision networks. Consistently helpful, rarely dramatic.\n\n**4. Dropout.** During training, randomly zero a fraction of activations in a layer.\n\n```python\nnn.Sequential(nn.Linear(512, 256), nn.ReLU(), nn.Dropout(0.3))\n```\n\nThe mechanism: no unit can rely on any particular other unit being present, so the network cannot build fragile chains of co-adapted features. It is also usefully read as training an ensemble of many thinned networks that share weights, and averaging them at inference.\n\nAt inference dropout must be switched off and the activations scaled to compensate. In PyTorch that is `model.train(False)`, often written as the evaluation-mode call, and **forgetting it is one of the most common bugs in PyTorch code**: the model returns different predictions on each call and nobody can work out why.\n\nA note on current practice: dropout was essential in the era of large fully connected layers. In convolutional networks batch norm largely replaced it, and in transformers rates are typically low: 0.1 rather than the 0.5 of older tutorials. It remains useful and it is no longer the first thing to reach for.\n\n**5. Label smoothing.** Instead of training towards a target of exactly 1.0 for the correct class, train towards 0.9, spreading the remaining 0.1 across the others.\n\nThis stops the network from driving its logits towards infinity to squeeze the last fraction out of the loss, which is where over-confidence comes from. It improves calibration meaningfully and often improves accuracy slightly.\n\n```python\nnn.CrossEntropyLoss(label_smoothing=0.1)\n```\n\n## Diagnosing which you need\n\nThe same reading as module 4, applied to a training curve.\n\n- Training loss falls, validation loss rises: overfitting. Add regularisation, or stop earlier, or get more data.\n- Both plateau high: underfitting. **Reduce** regularisation, or increase capacity, or train longer.\n- Training loss is unstable or spikes: learning rate or gradient clipping, not regularisation.\n\nThe second case is worth flagging because the instinct is always to add regularisation, and adding it to a network that is underfitting makes things worse while looking like diligence.\n\n## The unfashionable first move\n\nBefore any of the above: **overfit a tiny subset deliberately.**\n\nTake 20 examples and train until the model gets all 20 exactly right. If it cannot, you have a bug: a broken loss, a label misalignment, a learning rate so small nothing moves, a layer not connected. No amount of regularisation fixes a bug, and this test takes two minutes and finds most of them.\n\nOnly once the model can memorise 20 examples does it make sense to give it the full dataset and start controlling how much it memorises. Karpathy's recipe puts this step first for a reason: it separates \"the model cannot learn\" from \"the model learns the wrong thing\", and those need completely different responses.\n\n## Regularisation is not free, and the price is legible\n\nOne closing point, because it is easy to treat these five tools as costless improvements. Every one of them deliberately makes the model worse on the training data in exchange for a smaller gap. That is the trade from module 4 stated in a new place, and it means there is always a level at which more regularisation stops helping and starts hurting.\n\nThe way to find it is not intuition. Fix everything else, sweep one setting (dropout at 0.0, 0.1, 0.3, 0.5), and plot validation loss against it. The curve is almost always U-shaped, the bottom is usually lower than the default you started with, and the whole sweep on a small network takes minutes.\n\nDo them one at a time. Dropout, weight decay and augmentation all reduce effective capacity, so turning up all three together often overshoots into underfitting while each individual setting still looks reasonable. When a heavily regularised network's training and validation losses have converged at a poor level, the correct move is to remove regularisation, and almost nobody's first instinct is to try that."},"check":{"question":"A network's training loss falls to 0.02 while validation loss plateaus at 0.31 and slowly rises. A colleague suggests increasing model width. What is wrong with that suggestion?","options":[{"text":"Wider layers require a lower learning rate, so training would become unstable.","correct":false,"tempting":"Width does interact with the learning rate through initialisation scaling, but that is a training-stability point rather than the reason the suggestion misreads the curves."},{"text":"The curves show overfitting (the model already fits the training data almost perfectly) so adding capacity increases the gap rather than closing it.","correct":true},{"text":"Width should only be increased alongside depth, or the network becomes unbalanced.","correct":false,"tempting":"Invents an architectural rule; width and depth are tuned independently and neither is required to accompany the other."},{"text":"Wider layers need batch normalisation to train, which the network may not have.","correct":false,"tempting":"Normalisation helps wide and deep networks train, but its absence is not what makes added capacity the wrong response here."}]},"prev":{"slug":"optimisers-beyond-sgd","title":"Momentum, Adam, and what the optimiser is doing for you","url":"https://addaly.com/learn/machine-learning-foundations/optimisers-beyond-sgd"},"next":{"slug":"convolutions","title":"Convolution: the same detector, everywhere in the image","url":"https://addaly.com/learn/machine-learning-foundations/convolutions"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Keeping a large network from memorising', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/regularising-a-network (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Keeping a large network from memorising","courseTitle":"Machine Learning, Foundations","module":{"index":8,"slug":"neural-networks","title":"Neural networks, built from what you already know"},"answer":null,"keyIdea":"Early stopping and more data outrank every other regulariser, dropout must be disabled at inference or predictions become random, and before regularising anything you should confirm the network can deliberately memorise twenty examples: otherwise you are tuning around a bug.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"five-tools-in-order-of-how-much-they-matter","runs":[{"text":"Five tools, in order of how much they matter"}]},{"t":"p","runs":[{"text":"1. Early stopping.","b":true},{"text":" Watch validation loss and keep the weights from its best epoch. Free, effective, and it is the one to implement first. Everything else is refinement on top of it."}]},{"t":"p","runs":[{"text":"2. More data, or augmentation.","b":true},{"text":" The most effective regulariser there is. For images, augmentation is nearly free performance:"}]},{"t":"code","lang":"python","label":"Python","text":"from torchvision import transforms\ntransforms.Compose([\n    transforms.RandomResizedCrop(224, scale=(0.7, 1.0)),\n    transforms.RandomHorizontalFlip(),\n    transforms.ColorJitter(0.2, 0.2, 0.2),\n])"},{"t":"p","runs":[{"text":"The transformations must preserve the label. Horizontal flip is fine for cats and wrong for handwritten digits, where a flipped 2 is not a 2, and wrong for medical images where laterality is diagnostic. Think before adding one."}]},{"t":"p","runs":[{"text":"For text, augmentation is harder because small edits change meaning; back-translation and synonym replacement are the usual attempts and their gains are modest. For tabular data there is no reliable equivalent, as module 3 said of SMOTE."}]},{"t":"p","runs":[{"text":"3. Weight decay.","b":true},{"text":" L2 from module 2, applied through AdamW. 0.01 to 0.1 is the usual range for transformers, 1e-4 for vision networks. Consistently helpful, rarely dramatic."}]},{"t":"p","runs":[{"text":"4. Dropout.","b":true},{"text":" During training, randomly zero a fraction of activations in a layer."}]},{"t":"code","lang":"python","label":"Python","text":"nn.Sequential(nn.Linear(512, 256), nn.ReLU(), nn.Dropout(0.3))"},{"t":"p","runs":[{"text":"The mechanism: no unit can rely on any particular other unit being present, so the network cannot build fragile chains of co-adapted features. It is also usefully read as training an ensemble of many thinned networks that share weights, and averaging them at inference."}]},{"t":"p","runs":[{"text":"At inference dropout must be switched off and the activations scaled to compensate. In PyTorch that is "},{"text":"model.train(False)","code":true},{"text":", often written as the evaluation-mode call, and "},{"text":"forgetting it is one of the most common bugs in PyTorch code","b":true},{"text":": the model returns different predictions on each call and nobody can work out why."}]},{"t":"p","runs":[{"text":"A note on current practice: dropout was essential in the era of large fully connected layers. In convolutional networks batch norm largely replaced it, and in transformers rates are typically low: 0.1 rather than the 0.5 of older tutorials. It remains useful and it is no longer the first thing to reach for."}]},{"t":"p","runs":[{"text":"5. Label smoothing.","b":true},{"text":" Instead of training towards a target of exactly 1.0 for the correct class, train towards 0.9, spreading the remaining 0.1 across the others."}]},{"t":"p","runs":[{"text":"This stops the network from driving its logits towards infinity to squeeze the last fraction out of the loss, which is where over-confidence comes from. It improves calibration meaningfully and often improves accuracy slightly."}]},{"t":"code","lang":"python","label":"Python","text":"nn.CrossEntropyLoss(label_smoothing=0.1)"},{"t":"h2","id":"diagnosing-which-you-need","runs":[{"text":"Diagnosing which you need"}]},{"t":"p","runs":[{"text":"The same reading as module 4, applied to a training curve."}]},{"t":"ul","items":[[{"text":"Training loss falls, validation loss rises: overfitting. Add regularisation, or stop earlier, or get more data."}],[{"text":"Both plateau high: underfitting. "},{"text":"Reduce","b":true},{"text":" regularisation, or increase capacity, or train longer."}],[{"text":"Training loss is unstable or spikes: learning rate or gradient clipping, not regularisation."}]]},{"t":"p","runs":[{"text":"The second case is worth flagging because the instinct is always to add regularisation, and adding it to a network that is underfitting makes things worse while looking like diligence."}]},{"t":"h2","id":"the-unfashionable-first-move","runs":[{"text":"The unfashionable first move"}]},{"t":"p","runs":[{"text":"Before any of the above: "},{"text":"overfit a tiny subset deliberately.","b":true}]},{"t":"p","runs":[{"text":"Take 20 examples and train until the model gets all 20 exactly right. If it cannot, you have a bug: a broken loss, a label misalignment, a learning rate so small nothing moves, a layer not connected. No amount of regularisation fixes a bug, and this test takes two minutes and finds most of them."}]},{"t":"p","runs":[{"text":"Only once the model can memorise 20 examples does it make sense to give it the full dataset and start controlling how much it memorises. Karpathy’s recipe puts this step first for a reason: it separates “the model cannot learn” from “the model learns the wrong thing”, and those need completely different responses."}]},{"t":"h2","id":"regularisation-is-not-free-and-the-price-is-legible","runs":[{"text":"Regularisation is not free, and the price is legible"}]},{"t":"p","runs":[{"text":"One closing point, because it is easy to treat these five tools as costless improvements. Every one of them deliberately makes the model worse on the training data in exchange for a smaller gap. That is the trade from module 4 stated in a new place, and it means there is always a level at which more regularisation stops helping and starts hurting."}]},{"t":"p","runs":[{"text":"The way to find it is not intuition. Fix everything else, sweep one setting (dropout at 0.0, 0.1, 0.3, 0.5), and plot validation loss against it. The curve is almost always U-shaped, the bottom is usually lower than the default you started with, and the whole sweep on a small network takes minutes."}]},{"t":"p","runs":[{"text":"Do them one at a time. Dropout, weight decay and augmentation all reduce effective capacity, so turning up all three together often overshoots into underfitting while each individual setting still looks reasonable. When a heavily regularised network’s training and validation losses have converged at a poor level, the correct move is to remove regularisation, and almost nobody’s first instinct is to try that."}]}],"check":{"question":[{"text":"A network’s training loss falls to 0.02 while validation loss plateaus at 0.31 and slowly rises. A colleague suggests increasing model width. What is wrong with that suggestion?"}],"options":[{"text":[{"text":"Wider layers require a lower learning rate, so training would become unstable."}],"correct":false,"tempting":[{"text":"Width does interact with the learning rate through initialisation scaling, but that is a training-stability point rather than the reason the suggestion misreads the curves."}]},{"text":[{"text":"The curves show overfitting (the model already fits the training data almost perfectly) so adding capacity increases the gap rather than closing it."}],"correct":true,"tempting":null},{"text":[{"text":"Width should only be increased alongside depth, or the network becomes unbalanced."}],"correct":false,"tempting":[{"text":"Invents an architectural rule; width and depth are tuned independently and neither is required to accompany the other."}]},{"text":[{"text":"Wider layers need batch normalisation to train, which the network may not have."}],"correct":false,"tempting":[{"text":"Normalisation helps wide and deep networks train, but its absence is not what makes added capacity the wrong response here."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"optimisers-beyond-sgd","title":"Momentum, Adam, and what the optimiser is doing for you","minutes":8},"next":{"slug":"convolutions","title":"Convolution: the same detector, everywhere in the image","minutes":9}}}