The bends: ReLU, and why sigmoid stopped being used inside networks

Sigmoid, and the problem it caused

Early networks used the sigmoid, 1 / (1 + exp(-x)), inside hidden layers. It squashes any input into (0, 1) and it is smooth, which felt right.

Its derivative is s(x) * (1 - s(x)), which peaks at 0.25 when x is 0 and approaches zero as x moves away in either direction. That maximum of 0.25 is the whole problem.

Backpropagation multiplies derivatives layer by layer. With sigmoids, each layer multiplies the gradient by at most 0.25:

1 layer:   0.25
5 layers:  0.25^5  = 0.00098
10 layers: 0.25^10 = 0.00000095

By ten layers the gradient reaching the first layer is a millionth of what arrived at the last. The early layers effectively stop learning. This is the vanishing gradient problem, and it is the main reason deep networks did not work before about 2010. It was not a shortage of ideas; it was that the signal did not reach the bottom.

tanh is the same shape centred on zero, with a maximum derivative of 1. Better, and still saturating, so it only postpones the problem.

ReLU

ReLU(x) = max(0, x)

A hinge. Negative input gives zero, positive input passes through unchanged.

Its derivative is exactly 1 for positive inputs. Multiply by 1 through fifty layers and nothing shrinks. That single property is most of why deep networks became trainable, and it is why a function this crude replaced a smooth one that had a century of mathematical respectability behind it.

Two further benefits. It is cheap: a comparison, not an exponential. And it produces genuine zeros, so a typical layer has half its units inactive for any given input, which makes the representation sparse and, in practice, easier to optimise.

Dying ReLU

The cost. If a unit’s input is negative for every training example, its gradient is zero for every example, so its weights never update. The unit is dead permanently and contributes nothing for the rest of training.

This happens most often after a large learning-rate step pushes the bias strongly negative. With a badly chosen learning rate, 40% of units can die in the first epoch and the network never recovers that capacity.

The variants that address it:

• Leaky ReLU: max(0.01 * x, x). A small slope on the negative side means the gradient is never exactly zero, so a unit can come back.
• ELU and SELU: smooth on the negative side, with outputs that push activations towards zero mean.
• GELU: a smooth approximation to ReLU weighted by a Gaussian. Standard in transformers, including BERT and the GPT family.
• Swish / SiLU: x * sigmoid(x). Found by automated search, marginally better than ReLU on deep vision networks.

The honest summary: on most problems the differences between these are under a point. ReLU is the right default. GELU if you are building a transformer, because that is what everything else in that family uses.

The output layer is a separate decision

Hidden layers and the output layer answer different questions, and the output is dictated by the task rather than chosen by preference.

• Regression: no activation. You want an unbounded real number.
• Binary classification: sigmoid, giving one probability. Paired with binary cross-entropy.
• Multi-class, one label: softmax across the classes, from module 2. Paired with categorical cross-entropy.
• Multi-label: independent sigmoids, one per label. Also from module 2, and the distinction matters as much here as it did there.
• Bounded positive output such as a count: softplus, or exponentiate, or predict the log and transform back.

So sigmoid did not disappear. It moved from the hidden layers, where its vanishing derivative was fatal, to the output layer, where exactly one of them is applied and the saturation is a feature rather than a defect.

The numerical detail worth knowing

Every serious framework computes the sigmoid or softmax together with the loss in one fused operation (BCEWithLogitsLoss in PyTorch, from_logits=True in Keras) rather than applying the activation and then the loss separately.

The reason is precision. A sigmoid output of 0.9999999 rounds to 1.0 in float32, and log(1 - 1.0) is negative infinity. Computing the loss directly from the pre-activation values avoids the rounding entirely. Use the fused version; the separate version will eventually produce a NaN you cannot explain.