Machine Learning, Foundations · Chapter 04

Two numbers, not one

1:49 · Narrated with a synthetic voice. Captions are available in the player.

Read the transcript

You cannot diagnose a model from one score. You need training error and validation error side by side, and it is the gap between them that names the problem. Both high and close together means the model is not seeing enough, so the remedy is more capacity, better features or longer training. Training low and validation much higher means it has memorised, so the remedy is more data or less freedom. The word overfitting has drifted in casual use to mean the model is bad, so any disappointing score attracts it. If training accuracy is seventy-one per cent and validation is seventy, nothing is being memorised. Cutting the capacity there will make both numbers worse. Now the part nobody warns you about. You split off a validation set, you never trained on it, and then over six weeks you compared three hundred and forty models against it. Every comparison used that set to make a decision, and the winner scored highest partly because its particular errors happened to miss those particular rows. The maximum of many noisy estimates is biased upward. With a two thousand row validation set and a hundred comparisons, the winner is inflated by roughly two and a half points, which is enough to turn no real improvement into a clear win. Keep a test set that nobody has touched, and report how many times you looked.

Sources and further reading (1)