{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"backpropagation-by-hand","title":"Backpropagation, traced through one tiny network","url":"https://addaly.com/learn/machine-learning-foundations/backpropagation-by-hand","position":76,"of":92,"minutes":10,"answer":"Backpropagation multiplies the gradient arriving from above by each operation's local derivative, which computes every weight's gradient in one backward pass, and explains why gradients vanish, why dead ReLUs stop learning.","keyIdea":"Backpropagation multiplies the gradient arriving from above by each operation's local derivative, which computes every weight's gradient in one backward pass, and explains why gradients vanish, why dead ReLUs stop learning, and why training needs far more memory than inference."},"body":{"format":"markdown","text":"## The question backpropagation answers\n\nA network has thousands of weights. Change one, and the loss changes by some amount. Gradient descent needs that amount for every weight, and computing it by nudging each one and re-running the network would take one forward pass per weight: impossible at any real scale.\n\nBackpropagation computes all of them in one backward pass, at roughly the cost of one forward pass. That efficiency is the reason deep learning exists at all.\n\n## The smallest possible network\n\nOne input, one hidden unit, one output. Three weights and no biases, to keep the arithmetic visible.\n\n```text\nx = 2.0\nw1 = 0.5      h_pre = w1 * x        = 1.0\n              h     = ReLU(1.0)     = 1.0\nw2 = -0.3     y_pre = w2 * h        = -0.3\n              y     = y_pre         = -0.3\ntarget = 1.0\nloss  = (y - target)^2 = (-1.3)^2   = 1.69\n```\n\n## Backwards, one step at a time\n\nWe want how much the loss changes per unit change in `w1` and in `w2`. Work backwards from the loss, carrying one number at each stage.\n\n**Step 1: loss with respect to the output.**\n\n```text\nd(loss)/dy = 2 * (y - target) = 2 * (-1.3) = -2.6\n```\n\nSo increasing `y` by a small amount reduces the loss at a rate of 2.6 per unit. Good; `y` is below the target.\n\n**Step 2: loss with respect to w2.** Since `y = w2 * h`, changing `w2` changes `y` at a rate of `h`:\n\n```text\nd(loss)/dw2 = d(loss)/dy * dy/dw2 = -2.6 * 1.0 = -2.6\n```\n\n**Step 3: loss with respect to h.** Changing `h` changes `y` at a rate of `w2`:\n\n```text\nd(loss)/dh = -2.6 * (-0.3) = 0.78\n```\n\n**Step 4: through the ReLU.** `h_pre` was 1.0, which is positive, so ReLU's derivative is 1 and the gradient passes through unchanged.\n\n```text\nd(loss)/d(h_pre) = 0.78 * 1 = 0.78\n```\n\nHad `h_pre` been negative, this would be 0.78 * 0 = 0, and nothing below would have learned from this example. That is the dying-ReLU mechanism, visible in one line of arithmetic.\n\n**Step 5: loss with respect to w1.** Since `h_pre = w1 * x`:\n\n```text\nd(loss)/dw1 = 0.78 * 2.0 = 1.56\n```\n\n**Update, learning rate 0.1:**\n\n```text\nw2 = -0.3 - 0.1 * (-2.6) = -0.04\nw1 =  0.5 - 0.1 * ( 1.56) = 0.344\n```\n\nRerun the forward pass with those values and the loss falls from 1.69 to about 1.10. One step, and it worked.\n\n> Figure: One backward pass through the three-weight network. Steps: 1. d loss / d y = −2.6 (2 × (y − target), with y = −0.3 and target 1); 2. d loss / d w2 = −2.6 (× h, and h = 1.0); 3. d loss / d h = 0.78 (× w2 = −0.3); 4. Through ReLU: 0.78 (h_pre was positive, so × 1. Negative would have given 0 and w1 learns nothing.); 5. d loss / d w1 = 1.56 (× x = 2.0). Read right to left in time, left to right on the page: each box is the number arriving from the layer above multiplied by one local derivative. Ten layers would be ten such multiplications, which is exactly where gradients vanish or explode.\n\n## The pattern\n\nEvery step is the same shape: **the gradient arriving from above, multiplied by the local derivative of this operation.** That is the chain rule, applied mechanically, and it is why the whole thing costs one pass: each node needs only the number handed to it and its own local rule.\n\nTwo consequences follow directly.\n\n**Multiplication compounds.** Ten layers means ten multiplications. If the local derivatives are consistently under 1, the gradient vanishes; if consistently over 1, it explodes into `NaN`. This is the same arithmetic that killed sigmoid networks, seen from the other direction.\n\n**You must store the forward values.** Step 5 needed `x`, step 2 needed `h`. Every intermediate activation from the forward pass has to be kept until the backward pass uses it, which is why training uses far more memory than inference, and why the standard fix when a model will not fit is to reduce the batch size.\n\n## In practice\n\nYou will never write this. Every framework builds a graph of operations during the forward pass and walks it backwards automatically.\n\n```python\nimport torch\nx = torch.tensor([2.0])\nw1 = torch.tensor([0.5], requires_grad=True)\nw2 = torch.tensor([-0.3], requires_grad=True)\n\nloss = (torch.relu(w1 * x) * w2 - 1.0) ** 2\nloss.backward()\nw1.grad, w2.grad        # tensor([1.5600]), tensor([-2.6000])\n```\n\nThe same numbers. `requires_grad=True` is what tells PyTorch to record the operations, and `.backward()` is the walk.\n\n## Why do it by hand once\n\nBecause the errors you will actually meet are all visible in the trace above. A gradient that is zero because a ReLU was inactive. A gradient that vanished through many small multiplications. A gradient that exploded through many large ones. Memory exhausted because activations are retained. A `.backward()` that fails because something in the chain was converted to NumPy and broke the graph.\n\nEvery one of those is obvious once you have seen the mechanism and opaque if you have not. Twenty minutes with a pen buys you that."},"check":{"question":"In the traced network, suppose w1 had been -0.5 instead of 0.5, making h_pre negative. What would the gradient on w1 be, and why?","options":[{"text":"Larger in magnitude, because the error would be bigger with a negative hidden pre-activation.","correct":false,"tempting":"The loss would indeed differ, but the size of the loss does not determine whether the gradient reaches w1 through a ReLU that is inactive."},{"text":"The same, because ReLU's derivative does not depend on the sign of its input.","correct":false,"tempting":"ReLU's derivative is precisely a function of the input's sign: 1 when positive and 0 when negative."},{"text":"Negative rather than positive, because the sign of the pre-activation flips the sign of the gradient.","correct":false,"tempting":"Sign flipping happens when a weight in the chain is negative, but ReLU zeroes the negative side rather than reflecting it."},{"text":"Exactly zero, because ReLU's derivative is 0 for negative inputs, so the gradient arriving from above is multiplied by 0 and nothing below the ReLU updates for this example.","correct":true}]},"prev":{"slug":"activations","title":"The bends: ReLU, and why sigmoid stopped being used inside networks","url":"https://addaly.com/learn/machine-learning-foundations/activations"},"next":{"slug":"initialisation-and-normalisation","title":"Starting well, and staying well-behaved","url":"https://addaly.com/learn/machine-learning-foundations/initialisation-and-normalisation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Backpropagation, traced through one tiny network', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/backpropagation-by-hand (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Backpropagation, traced through one tiny network","courseTitle":"Machine Learning, Foundations","module":{"index":8,"slug":"neural-networks","title":"Neural networks, built from what you already know"},"answer":null,"keyIdea":"Backpropagation multiplies the gradient arriving from above by each operation’s local derivative, which computes every weight’s gradient in one backward pass, and explains why gradients vanish, why dead ReLUs stop learning, and why training needs far more memory than inference.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"d loss / d y = −2.6","detail":"2 × (y − target), with y = −0.3 and target 1"},{"label":"d loss / d w2 = −2.6","detail":"× h, and h = 1.0"},{"label":"d loss / d h = 0.78","detail":"× w2 = −0.3"},{"label":"Through ReLU: 0.78","detail":"h_pre was positive, so × 1. Negative would have given 0 and w1 learns nothing."},{"label":"d loss / d w1 = 1.56","detail":"× x = 2.0"}],"title":"One backward pass through the three-weight network","caption":"Read right to left in time, left to right on the page: each box is the number arriving from the layer above multiplied by one local derivative. Ten layers would be ten such multiplications, which is exactly where gradients vanish or explode."},"alt":"Figure: One backward pass through the three-weight network. Steps: 1. d loss / d y = −2.6 (2 × (y − target), with y = −0.3 and target 1); 2. d loss / d w2 = −2.6 (× h, and h = 1.0); 3. d loss / d h = 0.78 (× w2 = −0.3); 4. Through ReLU: 0.78 (h_pre was positive, so × 1. Negative would have given 0 and w1 learns nothing.); 5. d loss / d w1 = 1.56 (× x = 2.0). Read right to left in time, left to right on the page: each box is the number arriving from the layer above multiplied by one local derivative. Ten layers would be ten such multiplications, which is exactly where gradients vanish or explode."}},"blocks":[{"t":"h2","id":"the-question-backpropagation-answers","runs":[{"text":"The question backpropagation answers"}]},{"t":"p","runs":[{"text":"A network has thousands of weights. Change one, and the loss changes by some amount. Gradient descent needs that amount for every weight, and computing it by nudging each one and re-running the network would take one forward pass per weight: impossible at any real scale."}]},{"t":"p","runs":[{"text":"Backpropagation computes all of them in one backward pass, at roughly the cost of one forward pass. That efficiency is the reason deep learning exists at all."}]},{"t":"h2","id":"the-smallest-possible-network","runs":[{"text":"The smallest possible network"}]},{"t":"p","runs":[{"text":"One input, one hidden unit, one output. Three weights and no biases, to keep the arithmetic visible."}]},{"t":"code","lang":"text","label":"Text","text":"x = 2.0\nw1 = 0.5      h_pre = w1 * x        = 1.0\n              h     = ReLU(1.0)     = 1.0\nw2 = -0.3     y_pre = w2 * h        = -0.3\n              y     = y_pre         = -0.3\ntarget = 1.0\nloss  = (y - target)^2 = (-1.3)^2   = 1.69"},{"t":"h2","id":"backwards-one-step-at-a-time","runs":[{"text":"Backwards, one step at a time"}]},{"t":"p","runs":[{"text":"We want how much the loss changes per unit change in "},{"text":"w1","code":true},{"text":" and in "},{"text":"w2","code":true},{"text":". Work backwards from the loss, carrying one number at each stage."}]},{"t":"p","runs":[{"text":"Step 1: loss with respect to the output.","b":true}]},{"t":"code","lang":"text","label":"Text","text":"d(loss)/dy = 2 * (y - target) = 2 * (-1.3) = -2.6"},{"t":"p","runs":[{"text":"So increasing "},{"text":"y","code":true},{"text":" by a small amount reduces the loss at a rate of 2.6 per unit. Good; "},{"text":"y","code":true},{"text":" is below the target."}]},{"t":"p","runs":[{"text":"Step 2: loss with respect to w2.","b":true},{"text":" Since "},{"text":"y = w2 * h","code":true},{"text":", changing "},{"text":"w2","code":true},{"text":" changes "},{"text":"y","code":true},{"text":" at a rate of "},{"text":"h","code":true},{"text":":"}]},{"t":"code","lang":"text","label":"Text","text":"d(loss)/dw2 = d(loss)/dy * dy/dw2 = -2.6 * 1.0 = -2.6"},{"t":"p","runs":[{"text":"Step 3: loss with respect to h.","b":true},{"text":" Changing "},{"text":"h","code":true},{"text":" changes "},{"text":"y","code":true},{"text":" at a rate of "},{"text":"w2","code":true},{"text":":"}]},{"t":"code","lang":"text","label":"Text","text":"d(loss)/dh = -2.6 * (-0.3) = 0.78"},{"t":"p","runs":[{"text":"Step 4: through the ReLU.","b":true},{"text":" "},{"text":"h_pre","code":true},{"text":" was 1.0, which is positive, so ReLU’s derivative is 1 and the gradient passes through unchanged."}]},{"t":"code","lang":"text","label":"Text","text":"d(loss)/d(h_pre) = 0.78 * 1 = 0.78"},{"t":"p","runs":[{"text":"Had "},{"text":"h_pre","code":true},{"text":" been negative, this would be 0.78 * 0 = 0, and nothing below would have learned from this example. That is the dying-ReLU mechanism, visible in one line of arithmetic."}]},{"t":"p","runs":[{"text":"Step 5: loss with respect to w1.","b":true},{"text":" Since "},{"text":"h_pre = w1 * x","code":true},{"text":":"}]},{"t":"code","lang":"text","label":"Text","text":"d(loss)/dw1 = 0.78 * 2.0 = 1.56"},{"t":"p","runs":[{"text":"Update, learning rate 0.1:","b":true}]},{"t":"code","lang":"text","label":"Text","text":"w2 = -0.3 - 0.1 * (-2.6) = -0.04\nw1 =  0.5 - 0.1 * ( 1.56) = 0.344"},{"t":"p","runs":[{"text":"Rerun the forward pass with those values and the loss falls from 1.69 to about 1.10. One step, and it worked."}]},{"t":"h2","id":"the-pattern","runs":[{"text":"The pattern"}]},{"t":"p","runs":[{"text":"Every step is the same shape: "},{"text":"the gradient arriving from above, multiplied by the local derivative of this operation.","b":true},{"text":" That is the chain rule, applied mechanically, and it is why the whole thing costs one pass: each node needs only the number handed to it and its own local rule."}]},{"t":"p","runs":[{"text":"Two consequences follow directly."}]},{"t":"p","runs":[{"text":"Multiplication compounds.","b":true},{"text":" Ten layers means ten multiplications. If the local derivatives are consistently under 1, the gradient vanishes; if consistently over 1, it explodes into "},{"text":"NaN","code":true},{"text":". This is the same arithmetic that killed sigmoid networks, seen from the other direction."}]},{"t":"p","runs":[{"text":"You must store the forward values.","b":true},{"text":" Step 5 needed "},{"text":"x","code":true},{"text":", step 2 needed "},{"text":"h","code":true},{"text":". Every intermediate activation from the forward pass has to be kept until the backward pass uses it, which is why training uses far more memory than inference, and why the standard fix when a model will not fit is to reduce the batch size."}]},{"t":"h2","id":"in-practice","runs":[{"text":"In practice"}]},{"t":"p","runs":[{"text":"You will never write this. Every framework builds a graph of operations during the forward pass and walks it backwards automatically."}]},{"t":"code","lang":"python","label":"Python","text":"import torch\nx = torch.tensor([2.0])\nw1 = torch.tensor([0.5], requires_grad=True)\nw2 = torch.tensor([-0.3], requires_grad=True)\n\nloss = (torch.relu(w1 * x) * w2 - 1.0) ** 2\nloss.backward()\nw1.grad, w2.grad        # tensor([1.5600]), tensor([-2.6000])"},{"t":"p","runs":[{"text":"The same numbers. "},{"text":"requires_grad=True","code":true},{"text":" is what tells PyTorch to record the operations, and "},{"text":".backward()","code":true},{"text":" is the walk."}]},{"t":"h2","id":"why-do-it-by-hand-once","runs":[{"text":"Why do it by hand once"}]},{"t":"p","runs":[{"text":"Because the errors you will actually meet are all visible in the trace above. A gradient that is zero because a ReLU was inactive. A gradient that vanished through many small multiplications. A gradient that exploded through many large ones. Memory exhausted because activations are retained. A "},{"text":".backward()","code":true},{"text":" that fails because something in the chain was converted to NumPy and broke the graph."}]},{"t":"p","runs":[{"text":"Every one of those is obvious once you have seen the mechanism and opaque if you have not. Twenty minutes with a pen buys you that."}]}],"check":{"question":[{"text":"In the traced network, suppose w1 had been -0.5 instead of 0.5, making h_pre negative. What would the gradient on w1 be, and why?"}],"options":[{"text":[{"text":"Larger in magnitude, because the error would be bigger with a negative hidden pre-activation."}],"correct":false,"tempting":[{"text":"The loss would indeed differ, but the size of the loss does not determine whether the gradient reaches w1 through a ReLU that is inactive."}]},{"text":[{"text":"The same, because ReLU’s derivative does not depend on the sign of its input."}],"correct":false,"tempting":[{"text":"ReLU’s derivative is precisely a function of the input’s sign: 1 when positive and 0 when negative."}]},{"text":[{"text":"Negative rather than positive, because the sign of the pre-activation flips the sign of the gradient."}],"correct":false,"tempting":[{"text":"Sign flipping happens when a weight in the chain is negative, but ReLU zeroes the negative side rather than reflecting it."}]},{"text":[{"text":"Exactly zero, because ReLU’s derivative is 0 for negative inputs, so the gradient arriving from above is multiplied by 0 and nothing below the ReLU updates for this example."}],"correct":true,"tempting":null}],"order":[2,3,1,0],"why":null},"prev":{"slug":"activations","title":"The bends: ReLU, and why sigmoid stopped being used inside networks","minutes":8},"next":{"slug":"initialisation-and-normalisation","title":"Starting well, and staying well-behaved","minutes":9}}}