Slopes, and why training is walking downhill

The only question a derivative answers

If I nudge this number a tiny bit, how much does that number move, and in which direction?

That is a derivative. Not a limit, not a rule about x^n becoming nx^(n-1). A sensitivity.

Make it concrete. You are training a model. The loss right now is 2.6400. You increase one weight by 0.001 and recompute. The loss becomes 2.6431. It went up by 0.0031 when the weight went up by 0.001.

slope = 0.0031 / 0.001 = 3.1

The derivative of the loss with respect to that weight is 3.1. Read it in units: loss per unit of weight. Positive means increasing the weight increases the loss, so to reduce the loss you should decrease the weight.

That is the entire content of the idea. Everything else is machinery for computing it fast.

The gradient is just all of them at once

A model has more than one weight. GPT-2 small has 124 million. Do the same nudge test for each one — hold everything else still, wiggle this one, see what the loss does — and you get 124 million slopes.

Pack them into a list. That list is the gradient. It is a vector with one slot per parameter, and slot 7 means "how sensitive is the loss to parameter 7, right now, right here".

Nobody computes these by nudging. Backpropagation gets all of them in roughly the cost of one forward pass, using the chain rule. That is an engineering triumph and it does not change what the numbers mean.

Why walk instead of solve

In school you found a minimum by setting the derivative to zero and solving. Why not do that here?

Because you cannot. The loss of a neural network is a function of hundreds of millions of parameters, wrapped in nonlinearities, averaged over data. There is no formula to set to zero and rearrange. No closed form exists, and nobody expects one.

So you do the only thing left. Stand where you are, look at which way is downhill, take a step, look again.

w = 0.4200
grad = 3.1        # d(loss)/dw at this point
lr = 3e-4         # learning rate: how big a step

w = w - lr * grad # 0.41907

That is gradient descent, complete. The minus sign is the whole idea: the gradient points uphill, so you move against it. Then you do it again. Training a model is that line, executed a few hundred thousand times, on every parameter at once.

The learning rate, and why it is fiddly

A derivative is honest only very near where you measured it. It tells you the slope under your feet, not the shape of the valley.

Step too small and training crawls; you burn a week of GPU time getting nowhere. Step too big and you leap past the bottom and land higher up the far slope, then leap back further, and the loss goes to nan in about forty steps. Typical values for transformers land near 3e-4 for pretraining and 1e-5 or lower for fine-tuning, and those numbers are folklore refined by thousands of failed runs, not derived from anything.

What the picture gets wrong

Every tutorial draws a smooth bowl with a ball rolling to the bottom. That drawing is doing you a small amount of harm.

The real loss surface has flat plateaus, narrow ravines where the gradient points across the ravine instead of along it, and vast numbers of points where the slope is near zero without being any kind of bottom. On top of that, you never see the true gradient. You see an estimate from one batch of examples, which is noisy — the next lesson but one is about exactly how noisy.

And the goal is not the lowest point. A model that reached the true minimum of its training loss has memorised the training set and will be worse on anything new. You are walking downhill on a surface you cannot see, using a noisy compass, and you want to stop somewhere reasonable rather than at the bottom. It works far better than it has any right to, and no one fully agrees on why.