{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"activations","title":"The bends: ReLU, and why sigmoid stopped being used inside networks","url":"https://addaly.com/learn/machine-learning-foundations/activations","position":75,"of":92,"minutes":8,"answer":"Sigmoid's derivative peaks at 0.25, so backpropagation shrinks the gradient by at least a factor of four per layer and deep networks stopped learning; ReLU's derivative is exactly 1 where it is active.","keyIdea":"Sigmoid's derivative peaks at 0.25, so backpropagation shrinks the gradient by at least a factor of four per layer and deep networks stopped learning; ReLU's derivative is exactly 1 where it is active, which is why a crude hinge replaced a smooth curve."},"body":{"format":"markdown","text":"## Sigmoid, and the problem it caused\n\nEarly networks used the sigmoid, `1 / (1 + exp(-x))`, inside hidden layers. It squashes any input into (0, 1) and it is smooth, which felt right.\n\nIts derivative is `s(x) * (1 - s(x))`, which peaks at 0.25 when `x` is 0 and approaches zero as `x` moves away in either direction. That maximum of 0.25 is the whole problem.\n\nBackpropagation multiplies derivatives layer by layer. With sigmoids, each layer multiplies the gradient by at most 0.25:\n\n```text\n1 layer:   0.25\n5 layers:  0.25^5  = 0.00098\n10 layers: 0.25^10 = 0.00000095\n```\n\nBy ten layers the gradient reaching the first layer is a millionth of what arrived at the last. The early layers effectively stop learning. This is the **vanishing gradient problem**, and it is the main reason deep networks did not work before about 2010. It was not a shortage of ideas; it was that the signal did not reach the bottom.\n\n**tanh** is the same shape centred on zero, with a maximum derivative of 1. Better, and still saturating, so it only postpones the problem.\n\n## ReLU\n\n```text\nReLU(x) = max(0, x)\n```\n\nA hinge. Negative input gives zero, positive input passes through unchanged.\n\nIts derivative is exactly 1 for positive inputs. Multiply by 1 through fifty layers and nothing shrinks. That single property is most of why deep networks became trainable, and it is why a function this crude replaced a smooth one that had a century of mathematical respectability behind it.\n\n> Figure: Why a hinge replaced a smooth curve. Sigmoid in a hidden layer: Derivative peaks at 0.25 and falls towards zero on either side; Ten layers: 0.25 to the tenth, about a millionth of the gradient reaches the bottom; Early layers stop learning; deep networks did not work before about 2010; An exponential per unit. ReLU, max(0, x): Derivative is exactly 1 wherever the input is positive; Fifty layers of multiplying by 1: nothing shrinks; Half the units output zero for any input, so the representation is sparse; A comparison; costs almost nothing; The price: a unit whose input is always negative is dead for good. Backpropagation multiplies one local derivative per layer. A factor of at most 0.25 per layer starves the early layers; a factor of exactly 1 leaves the gradient intact. The sigmoid did not disappear: it moved to the output layer, where one of it is applied and saturation is the point.\n\nTwo further benefits. It is cheap: a comparison, not an exponential. And it produces genuine zeros, so a typical layer has half its units inactive for any given input, which makes the representation sparse and, in practice, easier to optimise.\n\n## Dying ReLU\n\nThe cost. If a unit's input is negative for every training example, its gradient is zero for every example, so its weights never update. The unit is dead permanently and contributes nothing for the rest of training.\n\nThis happens most often after a large learning-rate step pushes the bias strongly negative. With a badly chosen learning rate, 40% of units can die in the first epoch and the network never recovers that capacity.\n\nThe variants that address it:\n\n- **Leaky ReLU**: `max(0.01 * x, x)`. A small slope on the negative side means the gradient is never exactly zero, so a unit can come back.\n- **ELU** and **SELU**: smooth on the negative side, with outputs that push activations towards zero mean.\n- **GELU**: a smooth approximation to ReLU weighted by a Gaussian. Standard in transformers, including BERT and the GPT family.\n- **Swish / SiLU**: `x * sigmoid(x)`. Found by automated search, marginally better than ReLU on deep vision networks.\n\nThe honest summary: on most problems the differences between these are under a point. ReLU is the right default. GELU if you are building a transformer, because that is what everything else in that family uses.\n\n## The output layer is a separate decision\n\nHidden layers and the output layer answer different questions, and the output is dictated by the task rather than chosen by preference.\n\n- **Regression**: no activation. You want an unbounded real number.\n- **Binary classification**: sigmoid, giving one probability. Paired with binary cross-entropy.\n- **Multi-class, one label**: softmax across the classes, from module 2. Paired with categorical cross-entropy.\n- **Multi-label**: independent sigmoids, one per label. Also from module 2, and the distinction matters as much here as it did there.\n- **Bounded positive output** such as a count: softplus, or exponentiate, or predict the log and transform back.\n\nSo sigmoid did not disappear. It moved from the hidden layers, where its vanishing derivative was fatal, to the output layer, where exactly one of them is applied and the saturation is a feature rather than a defect.\n\n## The numerical detail worth knowing\n\nEvery serious framework computes the sigmoid or softmax *together with* the loss in one fused operation (`BCEWithLogitsLoss` in PyTorch, `from_logits=True` in Keras) rather than applying the activation and then the loss separately.\n\nThe reason is precision. A sigmoid output of 0.9999999 rounds to 1.0 in float32, and `log(1 - 1.0)` is negative infinity. Computing the loss directly from the pre-activation values avoids the rounding entirely. Use the fused version; the separate version will eventually produce a `NaN` you cannot explain."},"check":{"question":"After a few epochs, 45% of a network's ReLU units output zero for every input in the validation set and their weights have stopped changing. What most likely happened?","options":[{"text":"The network is overfitting, and the dead units are a form of automatic regularisation.","correct":false,"tempting":"Sparse activations do have a mild regularising effect, but permanently dead units are lost capacity rather than a controlled constraint."},{"text":"ReLU saturates for large positive inputs, so those units have reached their maximum and stopped responding.","correct":false,"tempting":"Inverts ReLU's behaviour: it is unbounded above and saturates only on the negative side, at exactly zero."},{"text":"A large learning-rate step pushed those units' biases strongly negative, so their inputs are always negative, their gradient is exactly zero, and they can never update again.","correct":true},{"text":"The inputs were not standardised, so half the features are negative and ReLU zeroes them.","correct":false,"tempting":"Negative inputs are ordinary and do not kill units by themselves; a unit dies only when its own pre-activation is negative for every example."}]},"prev":{"slug":"why-one-layer-is-not-enough","title":"XOR, and why the bend is the whole idea","url":"https://addaly.com/learn/machine-learning-foundations/why-one-layer-is-not-enough"},"next":{"slug":"backpropagation-by-hand","title":"Backpropagation, traced through one tiny network","url":"https://addaly.com/learn/machine-learning-foundations/backpropagation-by-hand"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The bends: ReLU, and why sigmoid stopped being used inside networks', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/activations (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The bends: ReLU, and why sigmoid stopped being used inside networks","courseTitle":"Machine Learning, Foundations","module":{"index":8,"slug":"neural-networks","title":"Neural networks, built from what you already know"},"answer":null,"keyIdea":"Sigmoid’s derivative peaks at 0.25, so backpropagation shrinks the gradient by at least a factor of four per layer and deep networks stopped learning; ReLU’s derivative is exactly 1 where it is active, which is why a crude hinge replaced a smooth curve.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Sigmoid in a hidden layer","items":["Derivative peaks at 0.25 and falls towards zero on either side","Ten layers: 0.25 to the tenth, about a millionth of the gradient reaches the bottom","Early layers stop learning; deep networks did not work before about 2010","An exponential per unit"]},"type":"compare","right":{"title":"ReLU, max(0, x)","items":["Derivative is exactly 1 wherever the input is positive","Fifty layers of multiplying by 1: nothing shrinks","Half the units output zero for any input, so the representation is sparse","A comparison; costs almost nothing","The price: a unit whose input is always negative is dead for good"]},"title":"Why a hinge replaced a smooth curve","caption":"Backpropagation multiplies one local derivative per layer. A factor of at most 0.25 per layer starves the early layers; a factor of exactly 1 leaves the gradient intact. The sigmoid did not disappear: it moved to the output layer, where one of it is applied and saturation is the point."},"alt":"Figure: Why a hinge replaced a smooth curve. Sigmoid in a hidden layer: Derivative peaks at 0.25 and falls towards zero on either side; Ten layers: 0.25 to the tenth, about a millionth of the gradient reaches the bottom; Early layers stop learning; deep networks did not work before about 2010; An exponential per unit. ReLU, max(0, x): Derivative is exactly 1 wherever the input is positive; Fifty layers of multiplying by 1: nothing shrinks; Half the units output zero for any input, so the representation is sparse; A comparison; costs almost nothing; The price: a unit whose input is always negative is dead for good. Backpropagation multiplies one local derivative per layer. A factor of at most 0.25 per layer starves the early layers; a factor of exactly 1 leaves the gradient intact. The sigmoid did not disappear: it moved to the output layer, where one of it is applied and saturation is the point."}},"blocks":[{"t":"h2","id":"sigmoid-and-the-problem-it-caused","runs":[{"text":"Sigmoid, and the problem it caused"}]},{"t":"p","runs":[{"text":"Early networks used the sigmoid, "},{"text":"1 / (1 + exp(-x))","code":true},{"text":", inside hidden layers. It squashes any input into (0, 1) and it is smooth, which felt right."}]},{"t":"p","runs":[{"text":"Its derivative is "},{"text":"s(x) * (1 - s(x))","code":true},{"text":", which peaks at 0.25 when "},{"text":"x","code":true},{"text":" is 0 and approaches zero as "},{"text":"x","code":true},{"text":" moves away in either direction. That maximum of 0.25 is the whole problem."}]},{"t":"p","runs":[{"text":"Backpropagation multiplies derivatives layer by layer. With sigmoids, each layer multiplies the gradient by at most 0.25:"}]},{"t":"code","lang":"text","label":"Text","text":"1 layer:   0.25\n5 layers:  0.25^5  = 0.00098\n10 layers: 0.25^10 = 0.00000095"},{"t":"p","runs":[{"text":"By ten layers the gradient reaching the first layer is a millionth of what arrived at the last. The early layers effectively stop learning. This is the "},{"text":"vanishing gradient problem","b":true},{"text":", and it is the main reason deep networks did not work before about 2010. It was not a shortage of ideas; it was that the signal did not reach the bottom."}]},{"t":"p","runs":[{"text":"tanh","b":true},{"text":" is the same shape centred on zero, with a maximum derivative of 1. Better, and still saturating, so it only postpones the problem."}]},{"t":"h2","id":"relu","runs":[{"text":"ReLU"}]},{"t":"code","lang":"text","label":"Text","text":"ReLU(x) = max(0, x)"},{"t":"p","runs":[{"text":"A hinge. Negative input gives zero, positive input passes through unchanged."}]},{"t":"p","runs":[{"text":"Its derivative is exactly 1 for positive inputs. Multiply by 1 through fifty layers and nothing shrinks. That single property is most of why deep networks became trainable, and it is why a function this crude replaced a smooth one that had a century of mathematical respectability behind it."}]},{"t":"p","runs":[{"text":"Two further benefits. It is cheap: a comparison, not an exponential. And it produces genuine zeros, so a typical layer has half its units inactive for any given input, which makes the representation sparse and, in practice, easier to optimise."}]},{"t":"h2","id":"dying-relu","runs":[{"text":"Dying ReLU"}]},{"t":"p","runs":[{"text":"The cost. If a unit’s input is negative for every training example, its gradient is zero for every example, so its weights never update. The unit is dead permanently and contributes nothing for the rest of training."}]},{"t":"p","runs":[{"text":"This happens most often after a large learning-rate step pushes the bias strongly negative. With a badly chosen learning rate, 40% of units can die in the first epoch and the network never recovers that capacity."}]},{"t":"p","runs":[{"text":"The variants that address it:"}]},{"t":"ul","items":[[{"text":"Leaky ReLU","b":true},{"text":": "},{"text":"max(0.01 * x, x)","code":true},{"text":". A small slope on the negative side means the gradient is never exactly zero, so a unit can come back."}],[{"text":"ELU","b":true},{"text":" and "},{"text":"SELU","b":true},{"text":": smooth on the negative side, with outputs that push activations towards zero mean."}],[{"text":"GELU","b":true},{"text":": a smooth approximation to ReLU weighted by a Gaussian. Standard in transformers, including BERT and the GPT family."}],[{"text":"Swish / SiLU","b":true},{"text":": "},{"text":"x * sigmoid(x)","code":true},{"text":". Found by automated search, marginally better than ReLU on deep vision networks."}]]},{"t":"p","runs":[{"text":"The honest summary: on most problems the differences between these are under a point. ReLU is the right default. GELU if you are building a transformer, because that is what everything else in that family uses."}]},{"t":"h2","id":"the-output-layer-is-a-separate-decision","runs":[{"text":"The output layer is a separate decision"}]},{"t":"p","runs":[{"text":"Hidden layers and the output layer answer different questions, and the output is dictated by the task rather than chosen by preference."}]},{"t":"ul","items":[[{"text":"Regression","b":true},{"text":": no activation. You want an unbounded real number."}],[{"text":"Binary classification","b":true},{"text":": sigmoid, giving one probability. Paired with binary cross-entropy."}],[{"text":"Multi-class, one label","b":true},{"text":": softmax across the classes, from module 2. Paired with categorical cross-entropy."}],[{"text":"Multi-label","b":true},{"text":": independent sigmoids, one per label. Also from module 2, and the distinction matters as much here as it did there."}],[{"text":"Bounded positive output","b":true},{"text":" such as a count: softplus, or exponentiate, or predict the log and transform back."}]]},{"t":"p","runs":[{"text":"So sigmoid did not disappear. It moved from the hidden layers, where its vanishing derivative was fatal, to the output layer, where exactly one of them is applied and the saturation is a feature rather than a defect."}]},{"t":"h2","id":"the-numerical-detail-worth-knowing","runs":[{"text":"The numerical detail worth knowing"}]},{"t":"p","runs":[{"text":"Every serious framework computes the sigmoid or softmax "},{"text":"together with","i":true},{"text":" the loss in one fused operation ("},{"text":"BCEWithLogitsLoss","code":true},{"text":" in PyTorch, "},{"text":"from_logits=True","code":true},{"text":" in Keras) rather than applying the activation and then the loss separately."}]},{"t":"p","runs":[{"text":"The reason is precision. A sigmoid output of 0.9999999 rounds to 1.0 in float32, and "},{"text":"log(1 - 1.0)","code":true},{"text":" is negative infinity. Computing the loss directly from the pre-activation values avoids the rounding entirely. Use the fused version; the separate version will eventually produce a "},{"text":"NaN","code":true},{"text":" you cannot explain."}]}],"check":{"question":[{"text":"After a few epochs, 45% of a network’s ReLU units output zero for every input in the validation set and their weights have stopped changing. What most likely happened?"}],"options":[{"text":[{"text":"The network is overfitting, and the dead units are a form of automatic regularisation."}],"correct":false,"tempting":[{"text":"Sparse activations do have a mild regularising effect, but permanently dead units are lost capacity rather than a controlled constraint."}]},{"text":[{"text":"ReLU saturates for large positive inputs, so those units have reached their maximum and stopped responding."}],"correct":false,"tempting":[{"text":"Inverts ReLU’s behaviour: it is unbounded above and saturates only on the negative side, at exactly zero."}]},{"text":[{"text":"A large learning-rate step pushed those units’ biases strongly negative, so their inputs are always negative, their gradient is exactly zero, and they can never update again."}],"correct":true,"tempting":null},{"text":[{"text":"The inputs were not standardised, so half the features are negative and ReLU zeroes them."}],"correct":false,"tempting":[{"text":"Negative inputs are ordinary and do not kill units by themselves; a unit dies only when its own pre-activation is negative for every example."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"why-one-layer-is-not-enough","title":"XOR, and why the bend is the whole idea","minutes":8},"next":{"slug":"backpropagation-by-hand","title":"Backpropagation, traced through one tiny network","minutes":10}}}