{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"when-more-parameters-stopped-being-worse","title":"Double descent, and the picture that stopped being complete","url":"https://addaly.com/learn/machine-learning-foundations/when-more-parameters-stopped-being-worse","position":41,"of":92,"minutes":8,"answer":"Test error can fall again past the point where a model fits the training data exactly.","keyIdea":"Test error can fall again past the point where a model fits the training data exactly, because extra capacity gives the optimiser many perfect fits to choose among and gradient descent prefers smooth ones: an observation that is well replicated and not yet fully explained."},"body":{"format":"markdown","text":"## What everybody was taught\n\nThe classical result says: as you add capacity, test error falls, bottoms out, and rises. Past the interpolation point (where the model can fit the training data exactly) it should be fitting noise and generalising terribly. Choose the bottom of the U.\n\nThat picture is correct for the models in this course. It is correct for polynomial regression, for decision trees, for k-NN. It is why `max_depth` and `alpha` exist.\n\nAnd it does not describe large neural networks, which is a problem, because large neural networks are the reason most people are reading this.\n\n## The observation\n\nA network with 100 million parameters trained on 50,000 images has 2,000 parameters per example. By the classical account it should memorise the training set perfectly and be useless on anything else.\n\nIt is not. It generalises well, often better than a smaller network that sits at the classical optimum.\n\nWhen people plotted test error against model size carefully across the whole range, the curve turned out to have a second descent. Error falls, rises to a peak exactly where the model becomes just barely able to fit the training data, and then falls again (often below the first minimum) as capacity keeps increasing. Belkin and colleagues named it **double descent** in 2019, and Nakkiran and colleagues showed the same shape appears as a function of training time and of dataset size, not only model size.\n\n## Why the peak is where it is\n\nThe most useful intuition concerns the region right at the interpolation threshold, and it is worth having because it explains why the peak is a peak rather than a plateau.\n\nA model with *just barely* enough capacity to fit the training data has exactly one way to do it. That solution is forced, it is pinned to every noisy point, and it is wildly unstable: this is the classical high-variance disaster.\n\nA model with *far more* capacity than needed has infinitely many ways to fit the data perfectly. Which one it lands on is decided by the optimiser, and gradient descent from a small random initialisation happens to prefer solutions with small weights and smooth behaviour between the training points. That preference is not written down anywhere in the loss function; it is a property of the search procedure. The field calls it **implicit regularisation**.\n\nSo the extra capacity is not being used to fit more noise. It is being used to give the optimiser room to choose a well-behaved solution among the many that fit.\n\n## What is unsettled\n\nThis is an area to describe carefully, because it is easy to overstate in either direction.\n\nThe empirical phenomenon is solid and reproduced widely. The explanation is not settled. There are rigorous results for specific simplified settings (linear regression in high dimensions, random feature models) and there is no complete theory covering deep networks trained with the optimisers people actually use. \"Implicit regularisation by gradient descent\" is a description with growing formal support behind it, not a finished proof.\n\nIt is also not a licence. Double descent does not mean regularisation is obsolete or that bigger is always better. Weight decay, dropout and early stopping still improve real models, and the second descent requires enough data and enough training; a large model on 300 rows overfits exactly as the classical picture predicts.\n\n## What to actually do\n\nFor the models in this course (linear, trees, boosting) use the classical picture. It is accurate for them, and gradient boosting does get worse with too many trees at too high a learning rate. Tune capacity and trust the U.\n\nFor neural networks, the practical advice that follows from all of this is: **do not choose a small model out of fear of overfitting.** Take a large one, regularise it, use early stopping against a validation set, and let the measurement decide. The measurement is the authority in both worlds; what changed is that the prior belief \"more parameters than rows must be bad\" is no longer a safe default.\n\n## Why this belongs in a foundations course\n\nBecause the most common way to be wrong in this field is to hold a rule whose conditions you never learned. \"More parameters than data means overfitting\" was taught as a law for decades and is now known to be a description of one regime. Learning where a rule's edge is, rather than only the rule, is what separates someone who can apply a method from someone who can tell when it has stopped applying."},"check":{"question":"A colleague argues that since a network has more parameters than training examples, it must be memorising and should be shrunk. What is the strongest correction?","options":[{"text":"Parameter count is irrelevant to generalisation, so the observation carries no information at all.","correct":false,"tempting":"Overcorrects: parameter count still matters, particularly at small data sizes where the classical picture holds firmly."},{"text":"Networks are regularised by dropout, so overfitting is impossible regardless of size.","correct":false,"tempting":"Dropout helps but does not make overfitting impossible; a large network on very little data still overfits badly."},{"text":"The parameter-to-data ratio predicts overfitting only near the interpolation threshold; far beyond it, many perfect fits exist and gradient descent tends to select smooth ones, so measured validation error should decide.","correct":true},{"text":"The classical result was disproved, so the bias-variance trade-off no longer applies to any model.","correct":false,"tempting":"Overstates the finding: the trade-off remains accurate for trees, linear models and boosting, which is most of what this course covers."}]},"prev":{"slug":"how-much-data-is-enough","title":"How much data do I need?","url":"https://addaly.com/learn/machine-learning-foundations/how-much-data-is-enough"},"next":{"slug":"when-accuracy-lies","title":"When accuracy lies","url":"https://addaly.com/learn/machine-learning-foundations/when-accuracy-lies"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Double descent, and the picture that stopped being complete', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/when-more-parameters-stopped-being-worse (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Double descent, and the picture that stopped being complete","courseTitle":"Machine Learning, Foundations","module":{"index":4,"slug":"generalisation","title":"Generalisation, and the discipline of held-out data"},"answer":null,"keyIdea":"Test error can fall again past the point where a model fits the training data exactly, because extra capacity gives the optimiser many perfect fits to choose among and gradient descent prefers smooth ones: an observation that is well replicated and not yet fully explained.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-everybody-was-taught","runs":[{"text":"What everybody was taught"}]},{"t":"p","runs":[{"text":"The classical result says: as you add capacity, test error falls, bottoms out, and rises. Past the interpolation point (where the model can fit the training data exactly) it should be fitting noise and generalising terribly. Choose the bottom of the U."}]},{"t":"p","runs":[{"text":"That picture is correct for the models in this course. It is correct for polynomial regression, for decision trees, for k-NN. It is why "},{"text":"max_depth","code":true},{"text":" and "},{"text":"alpha","code":true},{"text":" exist."}]},{"t":"p","runs":[{"text":"And it does not describe large neural networks, which is a problem, because large neural networks are the reason most people are reading this."}]},{"t":"h2","id":"the-observation","runs":[{"text":"The observation"}]},{"t":"p","runs":[{"text":"A network with 100 million parameters trained on 50,000 images has 2,000 parameters per example. By the classical account it should memorise the training set perfectly and be useless on anything else."}]},{"t":"p","runs":[{"text":"It is not. It generalises well, often better than a smaller network that sits at the classical optimum."}]},{"t":"p","runs":[{"text":"When people plotted test error against model size carefully across the whole range, the curve turned out to have a second descent. Error falls, rises to a peak exactly where the model becomes just barely able to fit the training data, and then falls again (often below the first minimum) as capacity keeps increasing. Belkin and colleagues named it "},{"text":"double descent","b":true},{"text":" in 2019, and Nakkiran and colleagues showed the same shape appears as a function of training time and of dataset size, not only model size."}]},{"t":"h2","id":"why-the-peak-is-where-it-is","runs":[{"text":"Why the peak is where it is"}]},{"t":"p","runs":[{"text":"The most useful intuition concerns the region right at the interpolation threshold, and it is worth having because it explains why the peak is a peak rather than a plateau."}]},{"t":"p","runs":[{"text":"A model with "},{"text":"just barely","i":true},{"text":" enough capacity to fit the training data has exactly one way to do it. That solution is forced, it is pinned to every noisy point, and it is wildly unstable: this is the classical high-variance disaster."}]},{"t":"p","runs":[{"text":"A model with "},{"text":"far more","i":true},{"text":" capacity than needed has infinitely many ways to fit the data perfectly. Which one it lands on is decided by the optimiser, and gradient descent from a small random initialisation happens to prefer solutions with small weights and smooth behaviour between the training points. That preference is not written down anywhere in the loss function; it is a property of the search procedure. The field calls it "},{"text":"implicit regularisation","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"So the extra capacity is not being used to fit more noise. It is being used to give the optimiser room to choose a well-behaved solution among the many that fit."}]},{"t":"h2","id":"what-is-unsettled","runs":[{"text":"What is unsettled"}]},{"t":"p","runs":[{"text":"This is an area to describe carefully, because it is easy to overstate in either direction."}]},{"t":"p","runs":[{"text":"The empirical phenomenon is solid and reproduced widely. The explanation is not settled. There are rigorous results for specific simplified settings (linear regression in high dimensions, random feature models) and there is no complete theory covering deep networks trained with the optimisers people actually use. “Implicit regularisation by gradient descent” is a description with growing formal support behind it, not a finished proof."}]},{"t":"p","runs":[{"text":"It is also not a licence. Double descent does not mean regularisation is obsolete or that bigger is always better. Weight decay, dropout and early stopping still improve real models, and the second descent requires enough data and enough training; a large model on 300 rows overfits exactly as the classical picture predicts."}]},{"t":"h2","id":"what-to-actually-do","runs":[{"text":"What to actually do"}]},{"t":"p","runs":[{"text":"For the models in this course (linear, trees, boosting) use the classical picture. It is accurate for them, and gradient boosting does get worse with too many trees at too high a learning rate. Tune capacity and trust the U."}]},{"t":"p","runs":[{"text":"For neural networks, the practical advice that follows from all of this is: "},{"text":"do not choose a small model out of fear of overfitting.","b":true},{"text":" Take a large one, regularise it, use early stopping against a validation set, and let the measurement decide. The measurement is the authority in both worlds; what changed is that the prior belief “more parameters than rows must be bad” is no longer a safe default."}]},{"t":"h2","id":"why-this-belongs-in-a-foundations-course","runs":[{"text":"Why this belongs in a foundations course"}]},{"t":"p","runs":[{"text":"Because the most common way to be wrong in this field is to hold a rule whose conditions you never learned. “More parameters than data means overfitting” was taught as a law for decades and is now known to be a description of one regime. Learning where a rule’s edge is, rather than only the rule, is what separates someone who can apply a method from someone who can tell when it has stopped applying."}]}],"check":{"question":[{"text":"A colleague argues that since a network has more parameters than training examples, it must be memorising and should be shrunk. What is the strongest correction?"}],"options":[{"text":[{"text":"Parameter count is irrelevant to generalisation, so the observation carries no information at all."}],"correct":false,"tempting":[{"text":"Overcorrects: parameter count still matters, particularly at small data sizes where the classical picture holds firmly."}]},{"text":[{"text":"Networks are regularised by dropout, so overfitting is impossible regardless of size."}],"correct":false,"tempting":[{"text":"Dropout helps but does not make overfitting impossible; a large network on very little data still overfits badly."}]},{"text":[{"text":"The parameter-to-data ratio predicts overfitting only near the interpolation threshold; far beyond it, many perfect fits exist and gradient descent tends to select smooth ones, so measured validation error should decide."}],"correct":true,"tempting":null},{"text":[{"text":"The classical result was disproved, so the bias-variance trade-off no longer applies to any model."}],"correct":false,"tempting":[{"text":"Overstates the finding: the trade-off remains accurate for trees, linear models and boosting, which is most of what this course covers."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"how-much-data-is-enough","title":"How much data do I need?","minutes":8},"next":{"slug":"when-accuracy-lies","title":"When accuracy lies","minutes":9}}}