Backpropagation, traced through one tiny network

The question backpropagation answers

A network has thousands of weights. Change one, and the loss changes by some amount. Gradient descent needs that amount for every weight, and computing it by nudging each one and re-running the network would take one forward pass per weight: impossible at any real scale.

Backpropagation computes all of them in one backward pass, at roughly the cost of one forward pass. That efficiency is the reason deep learning exists at all.

The smallest possible network

One input, one hidden unit, one output. Three weights and no biases, to keep the arithmetic visible.

x = 2.0
w1 = 0.5      h_pre = w1 * x        = 1.0
              h     = ReLU(1.0)     = 1.0
w2 = -0.3     y_pre = w2 * h        = -0.3
              y     = y_pre         = -0.3
target = 1.0
loss  = (y - target)^2 = (-1.3)^2   = 1.69

Backwards, one step at a time

We want how much the loss changes per unit change in w1 and in w2. Work backwards from the loss, carrying one number at each stage.

Step 1: loss with respect to the output.

d(loss)/dy = 2 * (y - target) = 2 * (-1.3) = -2.6

So increasing y by a small amount reduces the loss at a rate of 2.6 per unit. Good; y is below the target.

Step 2: loss with respect to w2. Since y = w2 * h, changing w2 changes y at a rate of h:

d(loss)/dw2 = d(loss)/dy * dy/dw2 = -2.6 * 1.0 = -2.6

Step 3: loss with respect to h. Changing h changes y at a rate of w2:

d(loss)/dh = -2.6 * (-0.3) = 0.78

Step 4: through the ReLU. h_pre was 1.0, which is positive, so ReLU’s derivative is 1 and the gradient passes through unchanged.

d(loss)/d(h_pre) = 0.78 * 1 = 0.78

Had h_pre been negative, this would be 0.78 * 0 = 0, and nothing below would have learned from this example. That is the dying-ReLU mechanism, visible in one line of arithmetic.

Step 5: loss with respect to w1. Since h_pre = w1 * x:

d(loss)/dw1 = 0.78 * 2.0 = 1.56

Update, learning rate 0.1:

w2 = -0.3 - 0.1 * (-2.6) = -0.04
w1 =  0.5 - 0.1 * ( 1.56) = 0.344

Rerun the forward pass with those values and the loss falls from 1.69 to about 1.10. One step, and it worked.

The pattern

Every step is the same shape: the gradient arriving from above, multiplied by the local derivative of this operation. That is the chain rule, applied mechanically, and it is why the whole thing costs one pass: each node needs only the number handed to it and its own local rule.

Two consequences follow directly.

Multiplication compounds. Ten layers means ten multiplications. If the local derivatives are consistently under 1, the gradient vanishes; if consistently over 1, it explodes into NaN. This is the same arithmetic that killed sigmoid networks, seen from the other direction.

You must store the forward values. Step 5 needed x, step 2 needed h. Every intermediate activation from the forward pass has to be kept until the backward pass uses it, which is why training uses far more memory than inference, and why the standard fix when a model will not fit is to reduce the batch size.

In practice

You will never write this. Every framework builds a graph of operations during the forward pass and walks it backwards automatically.

import torch
x = torch.tensor([2.0])
w1 = torch.tensor([0.5], requires_grad=True)
w2 = torch.tensor([-0.3], requires_grad=True)

loss = (torch.relu(w1 * x) * w2 - 1.0) ** 2
loss.backward()
w1.grad, w2.grad        # tensor([1.5600]), tensor([-2.6000])

The same numbers. requires_grad=True is what tells PyTorch to record the operations, and .backward() is the walk.

Why do it by hand once

Because the errors you will actually meet are all visible in the trace above. A gradient that is zero because a ReLU was inactive. A gradient that vanished through many small multiplications. A gradient that exploded through many large ones. Memory exhausted because activations are retained. A .backward() that fails because something in the chain was converted to NumPy and broke the graph.

Every one of those is obvious once you have seen the mechanism and opaque if you have not. Twenty minutes with a pen buys you that.