The picture
You are on a hillside in thick fog. You want the bottom of the valley. You cannot see it. But you can feel which way the ground slopes under your feet, so you take a step in the downhill direction, feel again, and step again.
That is gradient descent, complete. The hill is the loss as a function of the parameters. Your position is the current parameter values. The slope is the gradient. The step size is the learning rate.
An actual walk
One parameter, and a loss with a known bottom:
L(w) = (w - 3)²The slope at any w is 2(w - 3). Start at w = 0 with learning rate 0.1. The rule is: new w = old w − learning rate × slope.
| Step | w | Slope | New w | |---|---|---|---| | 1 | 0.000 | −6.00 | 0.600 | | 2 | 0.600 | −4.80 | 1.080 | | 3 | 1.080 | −3.84 | 1.464 | | 4 | 1.464 | −3.07 | 1.771 | | … | | | | | 20 | 2.965 | −0.07 | 2.972 |
It closes in on 3, taking smaller steps as the slope flattens, because near the bottom there is less slope to push it. Nobody told it the answer was 3. It only ever felt the ground where it stood.
Now set the learning rate to 1.1 and start again:
| Step | w | Slope | New w | |---|---|---|---| | 1 | 0.00 | −6.00 | 6.60 | | 2 | 6.60 | 7.20 | −1.32 | | 3 | −1.32 | −8.64 | 8.18 | | 4 | 8.18 | 10.37 | −3.22 |
Each step overshoots the bottom, lands further up the far side, and the next slope is steeper still. Within thirty steps the numbers exceed what a float can hold and your loss prints as nan. This is not an exotic failure. It is the single most common way training dies, and the fix is to lower the learning rate — usually by a factor of ten and then look again.
Too large diverges. Too small crawls: at a learning rate of 0.0001 the walk above needs tens of thousands of steps. There is no formula for the right value. You try 0.1, 0.01, 0.001 on validation and watch the loss curve.
Real models have many parameters
With two parameters the loss is a surface and the gradient is a pair of numbers, one slope per parameter, each saying how much the loss changes when that one parameter moves. With ten million parameters it is a vector of ten million slopes, and the update rule is the same line of arithmetic applied to each. The mental picture does not need to scale; the arithmetic does, and it does.
How much data per step
Computing the exact slope means running every training row through the model. With eight million rows that is one step per several minutes, which is unusable.
Mini-batch gradient descent estimates the slope from a random sample — 32, 128, 512 rows — and steps on that estimate. The estimate is noisy, so the path wanders rather than descending smoothly. That turns out to help: the noise shakes the walk out of shallow dips it would otherwise settle in, and it lets you take thousands of steps in the time one exact step would cost. Nearly all modern training is mini-batch.
Getting stuck, honestly
The standard worry is local minima: a small dip that is not the true bottom, where the slope is zero in every direction and the walk stops.
For the classic models this cannot happen. Linear regression, logistic regression and linear SVMs have convex losses — a single bowl, one bottom, and gradient descent finds it from anywhere.
For neural networks there are many minima, and for a long time this was assumed to be the central difficulty. It largely was not. In high dimensions the common flat spots are saddle points, where the surface rises in some directions and falls in others, and a walk can be slowed near one without being trapped. And when a large network does settle in a minimum, the minima it reaches tend to be about equally good, so which one you land in matters less than expected.
What helps in practice is momentum — keep some of your previous direction, so you roll through flat regions instead of stalling — and adaptive methods like Adam, which give each parameter its own effective step size. Both are refinements of the same walk. Feel the slope, take a step, repeat.
Before you move on