{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"normalisation-and-stability","title":"Normalisation, and why training is fragile","url":"https://addaly.com/learn/how-llms-work/normalisation-and-stability","position":16,"of":78,"minutes":8,"answer":"Normalisation layers exist to keep numbers in a workable range, and where they are placed determines whether a very deep model can be trained at all.","keyIdea":"Normalisation layers exist to keep numbers in a workable range, and where they are placed determines whether a very deep model can be trained at all."},"body":{"format":"markdown","text":"## The boring layer that decides whether training works\n\nIn the four-line block, `norm` looked like a detail worth skipping. For understanding what the model computes, it is. For understanding why training a large model is difficult, it is central.\n\n## The problem\n\nStack 80 blocks, each adding contributions to a running total, and the numbers in that total drift. If they grow, activations saturate and gradients explode; a single bad batch can send the loss to infinity and the run is dead. If they shrink, gradients vanish and the lower layers stop learning. Neither failure announces itself politely.\n\nNormalisation rescales the vector at each position so its numbers sit in a predictable range before the next sublayer reads it.\n\n**LayerNorm** subtracts the mean and divides by the standard deviation across the vector's dimensions, then applies a learned scale and shift. **RMSNorm**, used by most current models, skips the mean subtraction and divides by the root mean square only. It is cheaper, and it works about as well, which suggests the re-centring was never the important part.\n\nNote what these do not do: they operate across the *features* of a single position, not across the batch. Batch normalisation, which does normalise across examples, is standard in vision and almost unused here, because it behaves badly with variable-length sequences and makes each example's output depend on which other examples happened to share its batch.\n\n## Pre-norm versus post-norm\n\nThe original 2017 transformer put the normalisation *after* each sublayer and its residual addition. Training those models needed a learning-rate warmup, and deep ones were unstable.\n\nNearly everything since normalises *before* each sublayer:\n\n```python\nx = x + attention(norm(x))     # pre-norm: the residual path is untouched\nx = norm(x + attention(x))     # post-norm: the residual path passes through norm\n```\n\nThe difference looks cosmetic and is not. In pre-norm, there is a clean additive path from input to output with no normalisation in the way, so gradients reach the early layers directly. That is what makes 80-layer models trainable without heroics.\n\nThe cost is a known one: pre-norm models tend to see the residual stream's magnitude grow steadily with depth, and there is evidence the later layers contribute proportionally less as a result. Variants that normalise in both places, or scale the residual branch, exist to claw some of that back. This is an area where practice is still moving.\n\n## What a training run actually looks like\n\nRead a published training log (several large open models have released theirs), and you will see something the marketing never shows: **loss spikes**. The loss is descending smoothly, then jumps, sometimes catastrophically. Teams respond by rolling back to an earlier checkpoint, skipping the offending batch of data, and continuing.\n\nThat is a normal part of training a very large model. It is caused by some combination of unlucky data, numerical overflow and optimiser state, and mitigations include gradient clipping, careful initialisation, and computing certain operations in higher precision.\n\nThe practical lesson for anyone fine-tuning: if your loss spikes and does not recover, lower the learning rate first, then check for pathological examples in your data. Both are far more likely than a subtle bug in your model code.\n\n## Precision and where it must be kept high\n\nWeights are usually stored in bf16, a 16-bit format with the same exponent range as fp32 and fewer mantissa bits: the range matters more than the precision for stability, which is why bf16 largely replaced fp16 for training.\n\nSome operations still run in fp32 regardless: the softmax, the normalisation statistics, and the optimiser's accumulators. Doing them in 16-bit is a well-known way to produce a run that trains for two days and then quietly diverges.\n\n## Why you should care as a user\n\nYou will not tune these. But when a provider's model behaves differently after an update, or a fine-tune degrades in a way that looks like nonsense, the cause is often numerical rather than conceptual. And when you read that a lab needed several restarts to complete a training run, you now know it is a description of ordinary engineering rather than an admission of incompetence."},"check":{"question":"A team fine-tunes an open model and the loss falls smoothly for two hours, then spikes to a very large value and never recovers. What is the first thing to try?","options":[{"text":"Roll back to the last good checkpoint, reduce the learning rate, and inspect the batch of data that preceded the spike.","correct":true},{"text":"Switch from RMSNorm to LayerNorm, since the mean-subtraction improves stability.","correct":false,"tempting":"Norm choice does affect stability, and the extra term sounds like it should help, which makes this a credible-sounding intervention."},{"text":"Increase the batch size so each update averages over more examples.","correct":false,"tempting":"Larger batches genuinely do reduce gradient noise, so this is a real technique aimed at the wrong failure."},{"text":"Move the normalisation after each sublayer, as in the original transformer.","correct":false,"tempting":"Post-norm is a real architectural option, and citing the original paper gives it authority, but it is the less stable arrangement."}]},"prev":{"slug":"the-feed-forward-half","title":"The feed-forward half, where most of the model is","url":"https://addaly.com/learn/how-llms-work/the-feed-forward-half"},"next":{"slug":"from-final-vector-to-logits","title":"From the final vector to a score for every word","url":"https://addaly.com/learn/how-llms-work/from-final-vector-to-logits"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Normalisation, and why training is fragile', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/normalisation-and-stability (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Normalisation, and why training is fragile","courseTitle":"How a Language Model Actually Works","module":{"index":2,"slug":"inside-the-block","title":"Inside the block"},"answer":null,"keyIdea":"Normalisation layers exist to keep numbers in a workable range, and where they are placed determines whether a very deep model can be trained at all.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-boring-layer-that-decides-whether-training-works","runs":[{"text":"The boring layer that decides whether training works"}]},{"t":"p","runs":[{"text":"In the four-line block, "},{"text":"norm","code":true},{"text":" looked like a detail worth skipping. For understanding what the model computes, it is. For understanding why training a large model is difficult, it is central."}]},{"t":"h2","id":"the-problem","runs":[{"text":"The problem"}]},{"t":"p","runs":[{"text":"Stack 80 blocks, each adding contributions to a running total, and the numbers in that total drift. If they grow, activations saturate and gradients explode; a single bad batch can send the loss to infinity and the run is dead. If they shrink, gradients vanish and the lower layers stop learning. Neither failure announces itself politely."}]},{"t":"p","runs":[{"text":"Normalisation rescales the vector at each position so its numbers sit in a predictable range before the next sublayer reads it."}]},{"t":"p","runs":[{"text":"LayerNorm","b":true},{"text":" subtracts the mean and divides by the standard deviation across the vector’s dimensions, then applies a learned scale and shift. "},{"text":"RMSNorm","b":true},{"text":", used by most current models, skips the mean subtraction and divides by the root mean square only. It is cheaper, and it works about as well, which suggests the re-centring was never the important part."}]},{"t":"p","runs":[{"text":"Note what these do not do: they operate across the "},{"text":"features","i":true},{"text":" of a single position, not across the batch. Batch normalisation, which does normalise across examples, is standard in vision and almost unused here, because it behaves badly with variable-length sequences and makes each example’s output depend on which other examples happened to share its batch."}]},{"t":"h2","id":"pre-norm-versus-post-norm","runs":[{"text":"Pre-norm versus post-norm"}]},{"t":"p","runs":[{"text":"The original 2017 transformer put the normalisation "},{"text":"after","i":true},{"text":" each sublayer and its residual addition. Training those models needed a learning-rate warmup, and deep ones were unstable."}]},{"t":"p","runs":[{"text":"Nearly everything since normalises "},{"text":"before","i":true},{"text":" each sublayer:"}]},{"t":"code","lang":"python","label":"Python","text":"x = x + attention(norm(x))     # pre-norm: the residual path is untouched\nx = norm(x + attention(x))     # post-norm: the residual path passes through norm"},{"t":"p","runs":[{"text":"The difference looks cosmetic and is not. In pre-norm, there is a clean additive path from input to output with no normalisation in the way, so gradients reach the early layers directly. That is what makes 80-layer models trainable without heroics."}]},{"t":"p","runs":[{"text":"The cost is a known one: pre-norm models tend to see the residual stream’s magnitude grow steadily with depth, and there is evidence the later layers contribute proportionally less as a result. Variants that normalise in both places, or scale the residual branch, exist to claw some of that back. This is an area where practice is still moving."}]},{"t":"h2","id":"what-a-training-run-actually-looks-like","runs":[{"text":"What a training run actually looks like"}]},{"t":"p","runs":[{"text":"Read a published training log (several large open models have released theirs), and you will see something the marketing never shows: "},{"text":"loss spikes","b":true},{"text":". The loss is descending smoothly, then jumps, sometimes catastrophically. Teams respond by rolling back to an earlier checkpoint, skipping the offending batch of data, and continuing."}]},{"t":"p","runs":[{"text":"That is a normal part of training a very large model. It is caused by some combination of unlucky data, numerical overflow and optimiser state, and mitigations include gradient clipping, careful initialisation, and computing certain operations in higher precision."}]},{"t":"p","runs":[{"text":"The practical lesson for anyone fine-tuning: if your loss spikes and does not recover, lower the learning rate first, then check for pathological examples in your data. Both are far more likely than a subtle bug in your model code."}]},{"t":"h2","id":"precision-and-where-it-must-be-kept-high","runs":[{"text":"Precision and where it must be kept high"}]},{"t":"p","runs":[{"text":"Weights are usually stored in bf16, a 16-bit format with the same exponent range as fp32 and fewer mantissa bits: the range matters more than the precision for stability, which is why bf16 largely replaced fp16 for training."}]},{"t":"p","runs":[{"text":"Some operations still run in fp32 regardless: the softmax, the normalisation statistics, and the optimiser’s accumulators. Doing them in 16-bit is a well-known way to produce a run that trains for two days and then quietly diverges."}]},{"t":"h2","id":"why-you-should-care-as-a-user","runs":[{"text":"Why you should care as a user"}]},{"t":"p","runs":[{"text":"You will not tune these. But when a provider’s model behaves differently after an update, or a fine-tune degrades in a way that looks like nonsense, the cause is often numerical rather than conceptual. And when you read that a lab needed several restarts to complete a training run, you now know it is a description of ordinary engineering rather than an admission of incompetence."}]}],"check":{"question":[{"text":"A team fine-tunes an open model and the loss falls smoothly for two hours, then spikes to a very large value and never recovers. What is the first thing to try?"}],"options":[{"text":[{"text":"Roll back to the last good checkpoint, reduce the learning rate, and inspect the batch of data that preceded the spike."}],"correct":true,"tempting":null},{"text":[{"text":"Switch from RMSNorm to LayerNorm, since the mean-subtraction improves stability."}],"correct":false,"tempting":[{"text":"Norm choice does affect stability, and the extra term sounds like it should help, which makes this a credible-sounding intervention."}]},{"text":[{"text":"Increase the batch size so each update averages over more examples."}],"correct":false,"tempting":[{"text":"Larger batches genuinely do reduce gradient noise, so this is a real technique aimed at the wrong failure."}]},{"text":[{"text":"Move the normalisation after each sublayer, as in the original transformer."}],"correct":false,"tempting":[{"text":"Post-norm is a real architectural option, and citing the original paper gives it authority, but it is the less stable arrangement."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"the-feed-forward-half","title":"The feed-forward half, where most of the model is","minutes":9},"next":{"slug":"from-final-vector-to-logits","title":"From the final vector to a score for every word","minutes":8}}}