Overfitting, underfitting, and how to tell

Two ways to fail

A student who understands the subject does well on a new exam. A student who memorised last year’s paper does well on last year’s paper. A student who skimmed the syllabus does badly on both.

Models fail in exactly these two directions.

Overfitting: the model has captured accidents of the training data: this particular noise, these particular rows. Training error is low, new-data error is much higher.

Underfitting: the model is not expressive enough, or the features do not carry the signal. Both errors are high.

Capacity

The dial between them is capacity: how many distinguishable functions the model family can express.

Fit polynomials to 20 noisy points.

• Degree 1, a straight line. Training error 0.31, validation error 0.33. It cannot follow the shape.
• Degree 3. Training error 0.09, validation 0.11. About right.
• Degree 15. Training error 0.002: the curve threads every point exactly. Validation error 0.94, worse than the straight line, because between the points it swings wildly.

Training error falls monotonically with capacity. Validation error falls, bottoms out, and then rises. That U is the whole story, and every capacity knob you meet is a way of moving along it: tree depth, number of trees, number of layers, number of features, how long you train.

Diagnosis takes two numbers

You cannot diagnose from one score. You need training error and validation error side by side.

That last row is not a joke. Validation beating training usually means leakage, a broken split, or dropout being left on during evaluation.

The common mistake is reading row one as row two. “Overfitting” has drifted in casual use to mean “the model is bad”, so a disappointing score attracts the word regardless of the gap. If training accuracy is 71% and validation is 70%, nothing is being memorised. The model is not seeing enough, and cutting capacity will make it worse.

The cures, in order of how well they work

More data. The most reliable fix and usually the least available. Doubling the rows moves overfitting more than any hyperparameter will.

Less capacity. Shallower trees, fewer components, a smaller network.

Regularization: a penalty added to the loss that discourages complexity:

• L2 (ridge): add the sum of squared weights to the loss. Weights shrink toward zero and no single feature dominates.
• L1 (lasso): add the sum of absolute weights. Weights are driven exactly to zero, which selects features as a side effect.
• Early stopping: watch validation error each epoch and stop when it turns upward. Free, and it works.
• Dropout: during training, randomly zero out some fraction of a network’s units so no unit can rely on any other.
• Augmentation: manufacture more training data by transforming what you have: flipping, cropping, adding noise. For images this is the strongest tool on the list.

All of these have a knob, and every knob is chosen on validation. Not on test.

An honest complication

The U-curve is the classical picture and it holds for the tabular models most people build. Very large neural networks break it. Push capacity far past the point of fitting the training data perfectly and validation error sometimes falls again: the phenomenon is called double descent, and a network with more parameters than training examples can still generalise well.

Why that happens is still argued over. It does not mean overfitting was a myth; it means capacity alone is a cruder predictor of generalisation than the textbook curve suggests. For a gradient-boosted model on 50,000 rows, keep using the U. For a 70-billion-parameter network, the intuitions in this lesson stop being reliable.