The chain rule, which is the whole trick

Rates multiply

If a car's fuel use depends on speed, and speed depends on the accelerator position, how does fuel use depend on the accelerator? Multiply the two rates. That is the chain rule, and it is the only calculus fact you genuinely have to internalise for machine learning.

Formally, if y = f(u) and u = g(x):

dy/dx = (dy/du) * (du/dx)

Worked. Let y = (3x + 1)^2. Set u = 3x + 1, so y = u^2.

dy/du = 2u = 2(3x + 1)
du/dx = 3
dy/dx = 2(3x + 1) * 3 = 6(3x + 1) = 18x + 6

Check it by expanding first: y = 9x² + 6x + 1, derivative 18x + 6. Same answer. The chain rule saved you the expansion, and in a network the expansion is not available at any price.

Longer chains

Nesting three deep just multiplies three factors:

y = f(g(h(x)))
dy/dx = f'(g(h(x))) * g'(h(x)) * h'(x)

A neural network with 40 layers is a function nested 40 deep — plus an activation function inside each layer, so more like 80. The derivative of the loss with respect to a first-layer weight is a product of around 80 terms.

That single sentence explains two of the most important phenomena in deep learning.

Why gradients vanish, in arithmetic

Suppose each of those factors is around 0.5 — entirely plausible for a sigmoid, whose derivative maxes out at 0.25.

0.5^10 = 0.00098
0.5^20 = 0.00000095
0.5^40 = 9.1e-13

By layer 40 the gradient reaching the first layer is a millionth of a millionth of what reaches the last. In 32-bit floating point it is indistinguishable from zero. The early layers receive no signal and do not learn. This is the vanishing gradient problem, and it is not a bug in anyone's code — it is a product of small numbers, behaving exactly as multiplication does.

Now suppose each factor is 1.5:

1.5^10 = 57.7
1.5^20 = 3,325
1.5^40 = 11,057,332

That is the exploding gradient, which shows up as a loss that suddenly becomes NaN.

The architectural responses all attack the factors directly. ReLU has derivative exactly 1 for positive inputs, so it contributes a factor of 1 rather than 0.25. A residual connection computes x + F(x), whose derivative is 1 + F'(x) — a path through which the gradient passes with factor 1 no matter what the layer does. Normalisation layers keep activations in a range where derivatives are not tiny. Every one of these is a fix for a product of eighty numbers.

The multivariable version

When a variable influences the output through several paths, the chain rule sums over the paths as well as multiplying along them:

dL/dx = sum over paths of (product of derivatives along that path)

This is what automatic differentiation implements. The computation is a graph; each edge has a local derivative; the derivative of the output with respect to any input is the sum over all routes of the product along each route. Backpropagation is an efficient ordering of that sum, not a different idea.

Worked on a tiny network

One neuron, sigmoid activation, squared loss. Forward:

z = w x + b
a = sigmoid(z)
L = (a - y)^2

Backward, one factor at a time:

dL/da = 2(a - y)
da/dz = a(1 - a)
dz/dw = x

dL/dw = 2(a - y) * a(1 - a) * x

Put numbers in: x = 2, w = 0.5, b = 0, y = 1.

z = 1.0
a = 1/(1 + e^-1) = 0.7311
L = (0.7311 - 1)^2 = 0.0723
dL/dw = 2(-0.2689) * (0.7311)(0.2689) * 2 = -0.2115

Negative, so increasing w decreases the loss — which is right, since a needs to rise towards 1. Notice the middle factor, a(1 − a) = 0.197. If a had been 0.99, that factor would be 0.0099 and the gradient would be twenty times smaller. A confident sigmoid learns slowly even when it is confidently wrong, which is the specific reason cross-entropy loss replaced squared error for classification: the a(1 − a) term cancels out of the algebra.

The rule to keep

Nested functions multiply their rates. Chains of many small numbers vanish and chains of many large ones explode, and almost every architectural trick in deep learning is an attempt to keep those factors near one.