{"course":{"slug":"maths-you-need","title":"The Maths You Actually Need","url":"https://addaly.com/learn/maths-you-need","level":"intermediate","track":"deep"},"lesson":{"slug":"distributions-and-the-normal","title":"Distributions, and why the normal one keeps showing up","url":"https://addaly.com/learn/maths-you-need/distributions-and-the-normal","position":34,"of":76,"minutes":8,"answer":"The normal shape comes from many small independent effects adding up, and most real quantities are not built that way.","keyIdea":"The normal shape comes from many small independent effects adding up, and most real quantities are not built that way."},"body":{"format":"markdown","text":"## A distribution is belief spread over options\n\nOne probability answers one question. A distribution answers all of them at once: for every possible value, how much belief goes there.\n\nWhen the options are countable, that is a list. A language model's next-token output is a distribution over 50,000 tokens, each with its share, all adding to 1. You have already met it.\n\nWhen the values are continuous (a delivery time, a price in naira, a person's height) you cannot list them, because there are infinitely many. Instead you describe the shape: where belief piles up, how wide the spread is, how heavy the ends are. A histogram is that shape, drawn from data.\n\n## The normal one\n\nThe bell curve gets more airtime than every other distribution combined, and mostly for one solid reason.\n\nWhen a quantity is built by **adding up many small, independent effects**, none of which dominates, the total tends toward a normal shape. That is the central limit theorem, and it is close to a miracle: it does not care what the individual effects look like. Add enough of them and you get a bell.\n\nAdult height is roughly normal because it is thousands of genetic and nutritional contributions summed. Measurement error is roughly normal because it is many tiny independent disturbances summed. Average anything over a decent sample, and the average is roughly normal even when the raw values are not.\n\nA normal distribution needs exactly two numbers: the mean (where the peak sits) and the standard deviation (how wide it is). From those, the rough guides everyone uses: about 68% of values fall within one standard deviation of the mean, about 95% within two.\n\n## Where it appears in a model\n\n**Weight initialisation.** Before training, every weight is drawn from a narrow normal centred on zero, with a standard deviation scaled to the layer's input size: commonly `sqrt(2 / fan_in)`. That scaling is not decoration. Too wide and activations explode through the layers; too narrow and the signal fades to nothing by layer 20. The distribution you start from decides whether the network trains at all.\n\n**Diffusion models.** The forward process adds normal noise to an image, step by step, until it is indistinguishable from static. Generation runs it backwards: start from a fresh draw of pure normal noise and remove a little at a time. The whole method is built on that distribution specifically, because normal noise added to normal noise stays normal, which makes the maths tractable.\n\n**Dropout, sampling, augmentation.** Wherever a model needs randomness with no preferred direction, this is the default draw.\n\n## Where it does not apply, which is most places\n\nHere is the part that gets skipped, and it causes more damage than everything above combined.\n\nWord frequencies are not normal. They follow Zipf's law: the most common word appears about twice as often as the second, three times as often as the third. \"The\" makes up around 5% of English text on its own. Plot it and you get a cliff, not a bell, which is exactly why tokenizers and vocabulary cutoffs are designed the way they are.\n\nIncome is not normal. It is right-skewed with a long tail, which is why the mean income in any city sits well above what most people earn, and why the median is the honest number to quote.\n\nAPI latency is not normal. Requests cannot take negative time, most are fast, and a few hit a retry or a cold start and take 40 times the median. Server latency, file sizes, city populations, revenue per customer, time between failures: all skewed, all with tails much heavier than a bell allows.\n\nThe test is the sentence from earlier: **is this quantity made of many small independent things added together?** Height, yes. Latency, no: it comes from queueing and multiplication, which produce a different shape entirely. When the answer is no, borrowed normal-shaped rules such as \"two standard deviations covers 95%\" stop being true, and no error tells you."},"check":{"question":"A team assumes API latency is normal, sets an alert at the mean plus two standard deviations, and expects to page on about 2 to 3% of requests. In production the alert fires on 8%. What is the best explanation?","options":[{"text":"The sample used to compute the mean and standard deviation was too small, so both estimates are off.","correct":false,"tempting":"Treats a wrong shape as a sampling problem, which more data will not fix."},{"text":"Latency is not many small independent effects added together: queueing and retries make it right-skewed with a heavy tail, so the two-sigma guide does not hold.","correct":true},{"text":"Two standard deviations leaves 5% outside, not 2.5%, so the expected rate was wrong from the start.","correct":false,"tempting":"Argues about the one-sided versus two-sided convention instead of the assumption, and still leaves the 8% unexplained."},{"text":"Latency is normal, but its mean drifts through the day, so a fixed threshold misfires.","correct":false,"tempting":"Keeps the normal assumption alive by blaming drift, which would push the alert rate both up and down rather than only up."}]},"prev":{"slug":"random-variables-and-simulation","title":"When you cannot solve it, simulate it","url":"https://addaly.com/learn/maths-you-need/random-variables-and-simulation"},"next":{"slug":"the-distributions-you-will-meet","title":"Six distributions, and the process each one describes","url":"https://addaly.com/learn/maths-you-need/the-distributions-you-will-meet"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Distributions, and why the normal one keeps showing up', in The Maths You Actually Need. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/maths-you-need/distributions-and-the-normal (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Distributions, and why the normal one keeps showing up","courseTitle":"The Maths You Actually Need","module":{"index":4,"slug":"probability-you-can-rely-on","title":"Probability you can rely on"},"answer":null,"keyIdea":"The normal shape comes from many small independent effects adding up, and most real quantities are not built that way.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-distribution-is-belief-spread-over-options","runs":[{"text":"A distribution is belief spread over options"}]},{"t":"p","runs":[{"text":"One probability answers one question. A distribution answers all of them at once: for every possible value, how much belief goes there."}]},{"t":"p","runs":[{"text":"When the options are countable, that is a list. A language model’s next-token output is a distribution over 50,000 tokens, each with its share, all adding to 1. You have already met it."}]},{"t":"p","runs":[{"text":"When the values are continuous (a delivery time, a price in naira, a person’s height) you cannot list them, because there are infinitely many. Instead you describe the shape: where belief piles up, how wide the spread is, how heavy the ends are. A histogram is that shape, drawn from data."}]},{"t":"h2","id":"the-normal-one","runs":[{"text":"The normal one"}]},{"t":"p","runs":[{"text":"The bell curve gets more airtime than every other distribution combined, and mostly for one solid reason."}]},{"t":"p","runs":[{"text":"When a quantity is built by "},{"text":"adding up many small, independent effects","b":true},{"text":", none of which dominates, the total tends toward a normal shape. That is the central limit theorem, and it is close to a miracle: it does not care what the individual effects look like. Add enough of them and you get a bell."}]},{"t":"p","runs":[{"text":"Adult height is roughly normal because it is thousands of genetic and nutritional contributions summed. Measurement error is roughly normal because it is many tiny independent disturbances summed. Average anything over a decent sample, and the average is roughly normal even when the raw values are not."}]},{"t":"p","runs":[{"text":"A normal distribution needs exactly two numbers: the mean (where the peak sits) and the standard deviation (how wide it is). From those, the rough guides everyone uses: about 68% of values fall within one standard deviation of the mean, about 95% within two."}]},{"t":"h2","id":"where-it-appears-in-a-model","runs":[{"text":"Where it appears in a model"}]},{"t":"p","runs":[{"text":"Weight initialisation.","b":true},{"text":" Before training, every weight is drawn from a narrow normal centred on zero, with a standard deviation scaled to the layer’s input size: commonly "},{"text":"sqrt(2 / fan_in)","code":true},{"text":". That scaling is not decoration. Too wide and activations explode through the layers; too narrow and the signal fades to nothing by layer 20. The distribution you start from decides whether the network trains at all."}]},{"t":"p","runs":[{"text":"Diffusion models.","b":true},{"text":" The forward process adds normal noise to an image, step by step, until it is indistinguishable from static. Generation runs it backwards: start from a fresh draw of pure normal noise and remove a little at a time. The whole method is built on that distribution specifically, because normal noise added to normal noise stays normal, which makes the maths tractable."}]},{"t":"p","runs":[{"text":"Dropout, sampling, augmentation.","b":true},{"text":" Wherever a model needs randomness with no preferred direction, this is the default draw."}]},{"t":"h2","id":"where-it-does-not-apply-which-is-most-places","runs":[{"text":"Where it does not apply, which is most places"}]},{"t":"p","runs":[{"text":"Here is the part that gets skipped, and it causes more damage than everything above combined."}]},{"t":"p","runs":[{"text":"Word frequencies are not normal. They follow Zipf’s law: the most common word appears about twice as often as the second, three times as often as the third. “The” makes up around 5% of English text on its own. Plot it and you get a cliff, not a bell, which is exactly why tokenizers and vocabulary cutoffs are designed the way they are."}]},{"t":"p","runs":[{"text":"Income is not normal. It is right-skewed with a long tail, which is why the mean income in any city sits well above what most people earn, and why the median is the honest number to quote."}]},{"t":"p","runs":[{"text":"API latency is not normal. Requests cannot take negative time, most are fast, and a few hit a retry or a cold start and take 40 times the median. Server latency, file sizes, city populations, revenue per customer, time between failures: all skewed, all with tails much heavier than a bell allows."}]},{"t":"p","runs":[{"text":"The test is the sentence from earlier: "},{"text":"is this quantity made of many small independent things added together?","b":true},{"text":" Height, yes. Latency, no: it comes from queueing and multiplication, which produce a different shape entirely. When the answer is no, borrowed normal-shaped rules such as “two standard deviations covers 95%” stop being true, and no error tells you."}]}],"check":{"question":[{"text":"A team assumes API latency is normal, sets an alert at the mean plus two standard deviations, and expects to page on about 2 to 3% of requests. In production the alert fires on 8%. What is the best explanation?"}],"options":[{"text":[{"text":"The sample used to compute the mean and standard deviation was too small, so both estimates are off."}],"correct":false,"tempting":[{"text":"Treats a wrong shape as a sampling problem, which more data will not fix."}]},{"text":[{"text":"Latency is not many small independent effects added together: queueing and retries make it right-skewed with a heavy tail, so the two-sigma guide does not hold."}],"correct":true,"tempting":null},{"text":[{"text":"Two standard deviations leaves 5% outside, not 2.5%, so the expected rate was wrong from the start."}],"correct":false,"tempting":[{"text":"Argues about the one-sided versus two-sided convention instead of the assumption, and still leaves the 8% unexplained."}]},{"text":[{"text":"Latency is normal, but its mean drifts through the day, so a fixed threshold misfires."}],"correct":false,"tempting":[{"text":"Keeps the normal assumption alive by blaming drift, which would push the alert rate both up and down rather than only up."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"random-variables-and-simulation","title":"When you cannot solve it, simulate it","minutes":9},"next":{"slug":"the-distributions-you-will-meet","title":"Six distributions, and the process each one describes","minutes":10}}}