A distribution is belief spread over options
One probability answers one question. A distribution answers all of them at once: for every possible value, how much belief goes there.
When the options are countable, that is a list. A language model's next-token output is a distribution over 50,000 tokens, each with its share, all adding to 1. You have already met it.
When the values are continuous — a delivery time, a price in naira, a person's height — you cannot list them, because there are infinitely many. Instead you describe the shape: where belief piles up, how wide the spread is, how heavy the ends are. A histogram is that shape, drawn from data.
The normal one
The bell curve gets more airtime than every other distribution combined, and mostly for one solid reason.
When a quantity is built by adding up many small, independent effects, none of which dominates, the total tends toward a normal shape. That is the central limit theorem, and it is close to a miracle: it does not care what the individual effects look like. Add enough of them and you get a bell.
Adult height is roughly normal because it is thousands of genetic and nutritional contributions summed. Measurement error is roughly normal because it is many tiny independent disturbances summed. Average anything over a decent sample, and the average is roughly normal even when the raw values are not.
A normal distribution needs exactly two numbers: the mean (where the peak sits) and the standard deviation (how wide it is). From those, the rough guides everyone uses: about 68% of values fall within one standard deviation of the mean, about 95% within two.
Where it appears in a model
Weight initialisation. Before training, every weight is drawn from a narrow normal centred on zero, with a standard deviation scaled to the layer's input size — commonly sqrt(2 / fan_in). That scaling is not decoration. Too wide and activations explode through the layers; too narrow and the signal fades to nothing by layer 20. The distribution you start from decides whether the network trains at all.
Diffusion models. The forward process adds normal noise to an image, step by step, until it is indistinguishable from static. Generation runs it backwards: start from a fresh draw of pure normal noise and remove a little at a time. The whole method is built on that distribution specifically, because normal noise added to normal noise stays normal, which makes the maths tractable.
Dropout, sampling, augmentation. Wherever a model needs randomness with no preferred direction, this is the default draw.
Where it does not apply, which is most places
Here is the part that gets skipped, and it causes more damage than everything above combined.
Word frequencies are not normal. They follow Zipf's law: the most common word appears about twice as often as the second, three times as often as the third. "The" makes up around 5% of English text on its own. Plot it and you get a cliff, not a bell — which is exactly why tokenizers and vocabulary cutoffs are designed the way they are.
Income is not normal. It is right-skewed with a long tail, which is why the mean income in any city sits well above what most people earn, and why the median is the honest number to quote.
API latency is not normal. Requests cannot take negative time, most are fast, and a few hit a retry or a cold start and take 40 times the median. Server latency, file sizes, city populations, revenue per customer, time between failures — all skewed, all with tails much heavier than a bell allows.
The test is the sentence from earlier: is this quantity made of many small independent things added together? Height, yes. Latency, no — it comes from queueing and multiplication, which produce a different shape entirely. When the answer is no, borrowed normal-shaped rules such as "two standard deviations covers 95%" quietly stop being true, and no error tells you.
Before you move on