{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"initialisation-and-normalisation","title":"Starting well, and staying well-behaved","url":"https://addaly.com/learn/machine-learning-foundations/initialisation-and-normalisation","position":77,"of":92,"minutes":9,"answer":"Initialisation scale keeps activation variance stable across layers, normalisation keeps it stable across training, and a residual connection gives the gradient an unmultiplied path backwards.","keyIdea":"Initialisation scale keeps activation variance stable across layers, normalisation keeps it stable across training, and a residual connection gives the gradient an unmultiplied path backwards, which is what made networks deeper than about twenty layers trainable at all."},"body":{"format":"markdown","text":"## Why not zero\n\nInitialise every weight to zero and every unit in a layer computes the same output, receives the same gradient, and updates identically. The layer behaves as one unit forever, no matter how wide it is. This is the symmetry problem, and it is why initialisation must be random.\n\n## Why not any random values\n\nRandom is necessary and not sufficient. The scale matters enormously, and the reasoning is the same compounding arithmetic as backpropagation.\n\nConsider a 20-layer network with weights drawn from a normal distribution.\n\n- **Standard deviation too small**, say 0.01. Each layer's outputs are smaller than its inputs. After twenty layers the activations are effectively zero, the gradients are effectively zero, and nothing learns.\n- **Too large**, say 1.0 with wide layers. Each layer amplifies. Activations grow layer by layer, saturating any bounded activation and overflowing to `NaN` with unbounded ones.\n\nYou want the variance of the activations to stay roughly constant as the signal passes through, in both directions.\n\n## The two standard schemes\n\n**Xavier / Glorot**: variance `2 / (fan_in + fan_out)`. Derived assuming a symmetric activation like tanh. Use it with tanh or sigmoid.\n\n**He / Kaiming**: variance `2 / fan_in`. The extra factor of 2 compensates for ReLU zeroing half the inputs, which otherwise halves the variance at every layer. Use it with ReLU and its relatives.\n\n```python\nimport torch.nn as nn\nlayer = nn.Linear(256, 128)\nnn.init.kaiming_normal_(layer.weight, nonlinearity=\"relu\")\nnn.init.zeros_(layer.bias)\n```\n\nFrameworks default to something sensible, so you rarely set this by hand. It matters when you write a custom layer, and it matters that you know why a network that will not train at all might have this as its cause.\n\nBiases start at zero, which is fine: there is no symmetry problem, because the weights already differ.\n\n## Batch normalisation\n\nInitialisation fixes the start. Normalisation keeps things well-behaved throughout training.\n\nBatch norm standardises each feature across the current mini-batch to mean 0 and variance 1, then applies two learned parameters (a scale and a shift) so the network can undo the normalisation if that is what it needs.\n\n```python\nnn.Sequential(nn.Linear(256, 128), nn.BatchNorm1d(128), nn.ReLU())\n```\n\nWhat it buys is substantial: much higher learning rates become safe, training is faster and more stable, and it acts as a mild regulariser because each example's normalisation depends on which other examples happened to be in its batch.\n\nWhat it costs is worth knowing before you use it. Behaviour differs between training, where batch statistics are used, and inference, where a running average is used, so a model can train well and behave differently at prediction time. It performs badly with small batches, since the statistics are estimated from too few examples. And it creates a dependence between examples in a batch, which is awkward for some architectures and for reproducibility.\n\nThe original 2015 explanation was that it reduces \"internal covariate shift\". Later work argued the real effect is that it smooths the loss surface, making it easier to descend. The mechanism is still debated; the practical benefit is not.\n\n## Layer normalisation\n\nNormalise across the *features* of each individual example rather than across the batch.\n\nThis removes every problem in the list above: no dependence between examples, identical behaviour in training and inference, works with a batch size of one. It is the standard in transformers and in any sequence model, where sequences have different lengths and batch statistics are ill-defined.\n\nThe rough division: batch norm for convolutional vision networks, layer norm for transformers and anything sequential.\n\n## Residual connections\n\nThe third piece, and arguably the most important of the three.\n\n```text\noutput = x + F(x)\n```\n\nAdd the layer's input to its output. On the backward pass, the gradient flows through both branches, and the `x` path passes it through **unchanged**: no multiplication, so nothing shrinks. The vanishing gradient problem has a direct route around it.\n\nThis is what made networks of 50, 100 and 1,000 layers trainable. Before residual connections, adding layers past about 20 made networks *worse on the training set*, which is not overfitting but a failure to optimise. He and colleagues introduced them in 2015 and depth stopped being the constraint.\n\nA useful second reading: with a residual connection, a layer that learns nothing outputs zero and the block becomes the identity. So adding layers cannot make the architecture less expressive, and the network can effectively choose its own depth.\n\n## The order to check\n\nWhen a network will not train, the checks in order of frequency: learning rate too high, inputs not standardised, initialisation wrong for the activation, no normalisation in a deep stack, no residual connections past about 20 layers. Four of those five are one line each."},"check":{"question":"A 40-layer plain network without residual connections has higher training error than a 20-layer version of the same architecture. What does this indicate?","options":[{"text":"An optimisation failure rather than a capacity problem: gradients degrade through 40 multiplications, so the deeper network cannot even fit what the shallower one fits, which is the problem residual connections solve.","correct":true},{"text":"The deeper network needs a larger batch size for its batch normalisation statistics to be reliable.","correct":false,"tempting":"Small-batch statistics do destabilise batch norm, but that would affect both depths rather than degrading specifically with added layers."},{"text":"40 layers exceeds the universal approximation requirement, so the extra layers add noise to the function being represented.","correct":false,"tempting":"Misapplies the theorem, which concerns what a network can represent rather than imposing an upper limit on useful depth."},{"text":"Overfitting, since the deeper model has more parameters and memorises the training data.","correct":false,"tempting":"Overfitting produces lower training error with higher validation error; here the training error itself is worse."}]},"prev":{"slug":"backpropagation-by-hand","title":"Backpropagation, traced through one tiny network","url":"https://addaly.com/learn/machine-learning-foundations/backpropagation-by-hand"},"next":{"slug":"optimisers-beyond-sgd","title":"Momentum, Adam, and what the optimiser is doing for you","url":"https://addaly.com/learn/machine-learning-foundations/optimisers-beyond-sgd"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Starting well, and staying well-behaved', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/initialisation-and-normalisation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Starting well, and staying well-behaved","courseTitle":"Machine Learning, Foundations","module":{"index":8,"slug":"neural-networks","title":"Neural networks, built from what you already know"},"answer":null,"keyIdea":"Initialisation scale keeps activation variance stable across layers, normalisation keeps it stable across training, and a residual connection gives the gradient an unmultiplied path backwards, which is what made networks deeper than about twenty layers trainable at all.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-not-zero","runs":[{"text":"Why not zero"}]},{"t":"p","runs":[{"text":"Initialise every weight to zero and every unit in a layer computes the same output, receives the same gradient, and updates identically. The layer behaves as one unit forever, no matter how wide it is. This is the symmetry problem, and it is why initialisation must be random."}]},{"t":"h2","id":"why-not-any-random-values","runs":[{"text":"Why not any random values"}]},{"t":"p","runs":[{"text":"Random is necessary and not sufficient. The scale matters enormously, and the reasoning is the same compounding arithmetic as backpropagation."}]},{"t":"p","runs":[{"text":"Consider a 20-layer network with weights drawn from a normal distribution."}]},{"t":"ul","items":[[{"text":"Standard deviation too small","b":true},{"text":", say 0.01. Each layer’s outputs are smaller than its inputs. After twenty layers the activations are effectively zero, the gradients are effectively zero, and nothing learns."}],[{"text":"Too large","b":true},{"text":", say 1.0 with wide layers. Each layer amplifies. Activations grow layer by layer, saturating any bounded activation and overflowing to "},{"text":"NaN","code":true},{"text":" with unbounded ones."}]]},{"t":"p","runs":[{"text":"You want the variance of the activations to stay roughly constant as the signal passes through, in both directions."}]},{"t":"h2","id":"the-two-standard-schemes","runs":[{"text":"The two standard schemes"}]},{"t":"p","runs":[{"text":"Xavier / Glorot","b":true},{"text":": variance "},{"text":"2 / (fan_in + fan_out)","code":true},{"text":". Derived assuming a symmetric activation like tanh. Use it with tanh or sigmoid."}]},{"t":"p","runs":[{"text":"He / Kaiming","b":true},{"text":": variance "},{"text":"2 / fan_in","code":true},{"text":". The extra factor of 2 compensates for ReLU zeroing half the inputs, which otherwise halves the variance at every layer. Use it with ReLU and its relatives."}]},{"t":"code","lang":"python","label":"Python","text":"import torch.nn as nn\nlayer = nn.Linear(256, 128)\nnn.init.kaiming_normal_(layer.weight, nonlinearity=\"relu\")\nnn.init.zeros_(layer.bias)"},{"t":"p","runs":[{"text":"Frameworks default to something sensible, so you rarely set this by hand. It matters when you write a custom layer, and it matters that you know why a network that will not train at all might have this as its cause."}]},{"t":"p","runs":[{"text":"Biases start at zero, which is fine: there is no symmetry problem, because the weights already differ."}]},{"t":"h2","id":"batch-normalisation","runs":[{"text":"Batch normalisation"}]},{"t":"p","runs":[{"text":"Initialisation fixes the start. Normalisation keeps things well-behaved throughout training."}]},{"t":"p","runs":[{"text":"Batch norm standardises each feature across the current mini-batch to mean 0 and variance 1, then applies two learned parameters (a scale and a shift) so the network can undo the normalisation if that is what it needs."}]},{"t":"code","lang":"python","label":"Python","text":"nn.Sequential(nn.Linear(256, 128), nn.BatchNorm1d(128), nn.ReLU())"},{"t":"p","runs":[{"text":"What it buys is substantial: much higher learning rates become safe, training is faster and more stable, and it acts as a mild regulariser because each example’s normalisation depends on which other examples happened to be in its batch."}]},{"t":"p","runs":[{"text":"What it costs is worth knowing before you use it. Behaviour differs between training, where batch statistics are used, and inference, where a running average is used, so a model can train well and behave differently at prediction time. It performs badly with small batches, since the statistics are estimated from too few examples. And it creates a dependence between examples in a batch, which is awkward for some architectures and for reproducibility."}]},{"t":"p","runs":[{"text":"The original 2015 explanation was that it reduces “internal covariate shift”. Later work argued the real effect is that it smooths the loss surface, making it easier to descend. The mechanism is still debated; the practical benefit is not."}]},{"t":"h2","id":"layer-normalisation","runs":[{"text":"Layer normalisation"}]},{"t":"p","runs":[{"text":"Normalise across the "},{"text":"features","i":true},{"text":" of each individual example rather than across the batch."}]},{"t":"p","runs":[{"text":"This removes every problem in the list above: no dependence between examples, identical behaviour in training and inference, works with a batch size of one. It is the standard in transformers and in any sequence model, where sequences have different lengths and batch statistics are ill-defined."}]},{"t":"p","runs":[{"text":"The rough division: batch norm for convolutional vision networks, layer norm for transformers and anything sequential."}]},{"t":"h2","id":"residual-connections","runs":[{"text":"Residual connections"}]},{"t":"p","runs":[{"text":"The third piece, and arguably the most important of the three."}]},{"t":"code","lang":"text","label":"Text","text":"output = x + F(x)"},{"t":"p","runs":[{"text":"Add the layer’s input to its output. On the backward pass, the gradient flows through both branches, and the "},{"text":"x","code":true},{"text":" path passes it through "},{"text":"unchanged","b":true},{"text":": no multiplication, so nothing shrinks. The vanishing gradient problem has a direct route around it."}]},{"t":"p","runs":[{"text":"This is what made networks of 50, 100 and 1,000 layers trainable. Before residual connections, adding layers past about 20 made networks "},{"text":"worse on the training set","i":true},{"text":", which is not overfitting but a failure to optimise. He and colleagues introduced them in 2015 and depth stopped being the constraint."}]},{"t":"p","runs":[{"text":"A useful second reading: with a residual connection, a layer that learns nothing outputs zero and the block becomes the identity. So adding layers cannot make the architecture less expressive, and the network can effectively choose its own depth."}]},{"t":"h2","id":"the-order-to-check","runs":[{"text":"The order to check"}]},{"t":"p","runs":[{"text":"When a network will not train, the checks in order of frequency: learning rate too high, inputs not standardised, initialisation wrong for the activation, no normalisation in a deep stack, no residual connections past about 20 layers. Four of those five are one line each."}]}],"check":{"question":[{"text":"A 40-layer plain network without residual connections has higher training error than a 20-layer version of the same architecture. What does this indicate?"}],"options":[{"text":[{"text":"An optimisation failure rather than a capacity problem: gradients degrade through 40 multiplications, so the deeper network cannot even fit what the shallower one fits, which is the problem residual connections solve."}],"correct":true,"tempting":null},{"text":[{"text":"The deeper network needs a larger batch size for its batch normalisation statistics to be reliable."}],"correct":false,"tempting":[{"text":"Small-batch statistics do destabilise batch norm, but that would affect both depths rather than degrading specifically with added layers."}]},{"text":[{"text":"40 layers exceeds the universal approximation requirement, so the extra layers add noise to the function being represented."}],"correct":false,"tempting":[{"text":"Misapplies the theorem, which concerns what a network can represent rather than imposing an upper limit on useful depth."}]},{"text":[{"text":"Overfitting, since the deeper model has more parameters and memorises the training data."}],"correct":false,"tempting":[{"text":"Overfitting produces lower training error with higher validation error; here the training error itself is worse."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"backpropagation-by-hand","title":"Backpropagation, traced through one tiny network","minutes":10},"next":{"slug":"optimisers-beyond-sgd","title":"Momentum, Adam, and what the optimiser is doing for you","minutes":8}}}