Starting well, and staying well-behaved

Why not zero

Initialise every weight to zero and every unit in a layer computes the same output, receives the same gradient, and updates identically. The layer behaves as one unit forever, no matter how wide it is. This is the symmetry problem, and it is why initialisation must be random.

Why not any random values

Random is necessary and not sufficient. The scale matters enormously, and the reasoning is the same compounding arithmetic as backpropagation.

Consider a 20-layer network with weights drawn from a normal distribution.

• Standard deviation too small, say 0.01. Each layer’s outputs are smaller than its inputs. After twenty layers the activations are effectively zero, the gradients are effectively zero, and nothing learns.
• Too large, say 1.0 with wide layers. Each layer amplifies. Activations grow layer by layer, saturating any bounded activation and overflowing to NaN with unbounded ones.

You want the variance of the activations to stay roughly constant as the signal passes through, in both directions.

The two standard schemes

Xavier / Glorot: variance 2 / (fan_in + fan_out). Derived assuming a symmetric activation like tanh. Use it with tanh or sigmoid.

He / Kaiming: variance 2 / fan_in. The extra factor of 2 compensates for ReLU zeroing half the inputs, which otherwise halves the variance at every layer. Use it with ReLU and its relatives.

import torch.nn as nn
layer = nn.Linear(256, 128)
nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")
nn.init.zeros_(layer.bias)

Frameworks default to something sensible, so you rarely set this by hand. It matters when you write a custom layer, and it matters that you know why a network that will not train at all might have this as its cause.

Biases start at zero, which is fine: there is no symmetry problem, because the weights already differ.

Batch normalisation

Initialisation fixes the start. Normalisation keeps things well-behaved throughout training.

Batch norm standardises each feature across the current mini-batch to mean 0 and variance 1, then applies two learned parameters (a scale and a shift) so the network can undo the normalisation if that is what it needs.

nn.Sequential(nn.Linear(256, 128), nn.BatchNorm1d(128), nn.ReLU())

What it buys is substantial: much higher learning rates become safe, training is faster and more stable, and it acts as a mild regulariser because each example’s normalisation depends on which other examples happened to be in its batch.

What it costs is worth knowing before you use it. Behaviour differs between training, where batch statistics are used, and inference, where a running average is used, so a model can train well and behave differently at prediction time. It performs badly with small batches, since the statistics are estimated from too few examples. And it creates a dependence between examples in a batch, which is awkward for some architectures and for reproducibility.

The original 2015 explanation was that it reduces “internal covariate shift”. Later work argued the real effect is that it smooths the loss surface, making it easier to descend. The mechanism is still debated; the practical benefit is not.

Layer normalisation

Normalise across the features of each individual example rather than across the batch.

This removes every problem in the list above: no dependence between examples, identical behaviour in training and inference, works with a batch size of one. It is the standard in transformers and in any sequence model, where sequences have different lengths and batch statistics are ill-defined.

The rough division: batch norm for convolutional vision networks, layer norm for transformers and anything sequential.

Residual connections

The third piece, and arguably the most important of the three.

output = x + F(x)

Add the layer’s input to its output. On the backward pass, the gradient flows through both branches, and the x path passes it through unchanged: no multiplication, so nothing shrinks. The vanishing gradient problem has a direct route around it.

This is what made networks of 50, 100 and 1,000 layers trainable. Before residual connections, adding layers past about 20 made networks worse on the training set, which is not overfitting but a failure to optimise. He and colleagues introduced them in 2015 and depth stopped being the constraint.

A useful second reading: with a residual connection, a layer that learns nothing outputs zero and the block becomes the identity. So adding layers cannot make the architecture less expressive, and the network can effectively choose its own depth.

The order to check

When a network will not train, the checks in order of frequency: learning rate too high, inputs not standardised, initialisation wrong for the activation, no normalisation in a deep stack, no residual connections past about 20 layers. Four of those five are one line each.