Double descent, and the picture that stopped being complete

What everybody was taught

The classical result says: as you add capacity, test error falls, bottoms out, and rises. Past the interpolation point (where the model can fit the training data exactly) it should be fitting noise and generalising terribly. Choose the bottom of the U.

That picture is correct for the models in this course. It is correct for polynomial regression, for decision trees, for k-NN. It is why max_depth and alpha exist.

And it does not describe large neural networks, which is a problem, because large neural networks are the reason most people are reading this.

The observation

A network with 100 million parameters trained on 50,000 images has 2,000 parameters per example. By the classical account it should memorise the training set perfectly and be useless on anything else.

It is not. It generalises well, often better than a smaller network that sits at the classical optimum.

When people plotted test error against model size carefully across the whole range, the curve turned out to have a second descent. Error falls, rises to a peak exactly where the model becomes just barely able to fit the training data, and then falls again (often below the first minimum) as capacity keeps increasing. Belkin and colleagues named it double descent in 2019, and Nakkiran and colleagues showed the same shape appears as a function of training time and of dataset size, not only model size.

Why the peak is where it is

The most useful intuition concerns the region right at the interpolation threshold, and it is worth having because it explains why the peak is a peak rather than a plateau.

A model with just barely enough capacity to fit the training data has exactly one way to do it. That solution is forced, it is pinned to every noisy point, and it is wildly unstable: this is the classical high-variance disaster.

A model with far more capacity than needed has infinitely many ways to fit the data perfectly. Which one it lands on is decided by the optimiser, and gradient descent from a small random initialisation happens to prefer solutions with small weights and smooth behaviour between the training points. That preference is not written down anywhere in the loss function; it is a property of the search procedure. The field calls it implicit regularisation.

So the extra capacity is not being used to fit more noise. It is being used to give the optimiser room to choose a well-behaved solution among the many that fit.

What is unsettled

This is an area to describe carefully, because it is easy to overstate in either direction.

The empirical phenomenon is solid and reproduced widely. The explanation is not settled. There are rigorous results for specific simplified settings (linear regression in high dimensions, random feature models) and there is no complete theory covering deep networks trained with the optimisers people actually use. “Implicit regularisation by gradient descent” is a description with growing formal support behind it, not a finished proof.

It is also not a licence. Double descent does not mean regularisation is obsolete or that bigger is always better. Weight decay, dropout and early stopping still improve real models, and the second descent requires enough data and enough training; a large model on 300 rows overfits exactly as the classical picture predicts.

What to actually do

For the models in this course (linear, trees, boosting) use the classical picture. It is accurate for them, and gradient boosting does get worse with too many trees at too high a learning rate. Tune capacity and trust the U.

For neural networks, the practical advice that follows from all of this is: do not choose a small model out of fear of overfitting. Take a large one, regularise it, use early stopping against a validation set, and let the measurement decide. The measurement is the authority in both worlds; what changed is that the prior belief “more parameters than rows must be bad” is no longer a safe default.

Why this belongs in a foundations course

Because the most common way to be wrong in this field is to hold a rule whose conditions you never learned. “More parameters than data means overfitting” was taught as a law for decades and is now known to be a description of one regime. Learning where a rule’s edge is, rather than only the rule, is what separates someone who can apply a method from someone who can tell when it has stopped applying.