Machine Learning, Foundations · Chapter 02
Walking downhill in fog
1:58 · Narrated with a synthetic voice. Captions are available in the player.
Read the transcript
You are on a hillside in thick fog. You cannot see the bottom, but you can feel which way the ground slopes under your feet, so you step downhill and feel again. That is gradient descent, complete. The hill is the loss. The step size is the learning rate, and it is the single knob that decides whether the walk helps you or destroys you. Too small and it crawls. Too large and each step overshoots the bottom, lands further up the far side, and within thirty steps your loss prints as not a number. When training dies, lower the learning rate by a factor of ten and look again. That is the cause about eight times out of ten. A model overfits by using large opposing coefficients to thread through individual training points. Regularisation makes that expensive. Add the size of the coefficients to the loss, and the optimiser has to justify every unit of coefficient it spends with a matching reduction in error. Ridge charges the square of a coefficient, so the push it exerts fades away as the coefficient shrinks and nothing quite reaches zero. Lasso charges the absolute value, whose push is exactly the same however small the coefficient gets, and that constant push drives small coefficients to zero and holds them there. That is why lasso selects features and ridge does not. Standardise your columns first, or the penalty lands on units of measurement rather than on contribution.