Many knobs, one derivative each

Freeze everything else

A loss depends on millions of weights, not one. The extension is simpler than it sounds: to find the partial derivative with respect to one variable, treat every other variable as a constant and differentiate as usual.

Take f(x, y) = x^2 + 3xy + y^3.

For the partial with respect to x, pretend y is a fixed number:

d/dx of x^2   = 2x
d/dx of 3xy   = 3y        (y is a constant, so this is 3y times x)
d/dx of y^3   = 0         (no x in it, so it is a constant)

df/dx = 2x + 3y

Now the same for y, pretending x is fixed:

df/dy = 0 + 3x + 3y^2 = 3x + 3y^2

Evaluate both at the point (2, 1):

df/dx = 2(2) + 3(1) = 7
df/dy = 3(2) + 3(1) = 9

So at (2, 1), nudging x up increases f at rate 7, and nudging y up increases it at rate 9.

The gradient is the collection

Stack the partials into a vector and you have the gradient, written ∇f:

grad f at (2, 1) = [7, 9]

This vector has two properties that carry the whole of gradient-based training.

It points in the direction of steepest increase. Not "a direction that increases f" — the steepest one, out of all possible directions. Move against it and you descend as fast as locally possible.

Its length says how steep. ||[7, 9]|| = sqrt(49 + 81) = 11.4. A gradient norm of 11.4 means the function is changing quickly here. A norm near zero means you are somewhere flat.

That second quantity is the one to watch during training. grad_norm in a training log is exactly this, computed over every parameter at once, and its behaviour tells you more about what is going wrong than the loss does.

Reading the direction claim carefully

Why is the gradient the steepest direction rather than merely an uphill one? Because the rate of change in any unit direction u is the dot product ∇f · u, and from the module on vectors, a dot product with a fixed vector is largest when u points the same way as that vector. The gradient is not defined as steepest ascent; it turns out to be it.

The consequence used everywhere: to go down fastest, step along −∇f. That is gradient descent, and the whole of it is

w  <-  w - learning_rate * grad

repeated. With the gradient [7, 9] and a learning rate of 0.1, you move from (2, 1) to (2 − 0.7, 1 − 0.9) = (1.3, 0.1).

Steepest is local, not global

The word to hold onto is locally. The gradient describes the surface at the exact point you are standing on and says nothing about what is over the hill. In a long, narrow valley — the common case in real loss landscapes — the steepest direction points at the near wall, not along the valley floor. You cross the valley, overshoot slightly, cross back, and progress along the floor is slow.

That zigzag is the standard picture of ill-conditioning. It is caused by curvature differing wildly between directions, and it is the problem that momentum, Adam and learning-rate schedules all exist to reduce. Nothing is broken when it happens; the gradient is doing exactly what it promised, which is only ever a local promise.

The scale of it in a real model

For a model with 7 billion parameters, the gradient is a vector with 7 billion entries — one partial derivative per weight, recomputed on every batch. Stored as 32-bit floats that is 28 GB, which is why the memory arithmetic of training is dominated by gradients and optimiser state rather than by the weights themselves.

Nobody computes those partials one at a time. Automatic differentiation gets all of them in roughly the cost of one extra forward pass, which is the subject of the backpropagation lesson two along from here. It is worth pausing on how surprising that is: seven billion derivatives for the price of about two function evaluations.

Checking your intuition on a real surface

import numpy as np
def f(v): x, y = v; return x**2 + 3*x*y + y**3
def grad(v): x, y = v; return np.array([2*x + 3*y, 3*x + 3*y**2])

v = np.array([2.0, 1.0])
for step in range(5):
    g = grad(v)
    v = v - 0.05 * g
    print(step, v.round(3), f(v).round(4), np.linalg.norm(g).round(3))

Watch the third column fall and the fourth shrink. When the gradient norm approaches zero you have arrived somewhere flat — which may be a minimum, and may not be.

The rule to keep

One partial derivative per parameter, collected into a vector that points uphill fastest. Its length is the number to watch: a gradient norm that collapses towards zero or explodes towards thousands is telling you about your training long before the loss curve does.