Expectation and variance in plain terms

Expectation is a weighted average

Every possible value, multiplied by how likely it is, all added up.

A street vendor in Jakarta sells 40 bowls on a good day (60% of days) and 15 on a bad one (40%).

expected sales = 0.6 * 40 + 0.4 * 15 = 24 + 6 = 30 bowls

Thirty is the expectation. Now notice what it is not: he never sells 30 bowls. Not once, ever. The expectation is not a value you expect to see. It is what the average settles near over many days.

This trips people constantly. The expected value of a die roll is 3.5. Average household size in a country might be 4.2 people. Neither is a possible outcome. An expectation is a summary of a distribution, and summaries are allowed to be things that never happen.

Every loss you have seen is an expectation

The loss you actually want to minimise is the average over all data your model will ever meet. You cannot compute that — you do not have all the data. You compute it over your training set, which is itself a stand-in. And in practice you compute it over one batch of 32 or 512 examples, which is a stand-in for the stand-in.

So the gradient you step with is an estimate. It is roughly right, and it wobbles. That wobble is not a flaw to be engineered away; it is the "stochastic" in stochastic gradient descent, and a small amount of it helps by shaking the model out of bad spots.

Variance is how much things wobble

Variance measures spread: how far values sit from their mean, on average, with the distances squared so they do not cancel out. Standard deviation is the square root of that, which puts it back into the original units — bowls, rupees, seconds — so it is the one worth reading.

The fact that matters most in machine learning is this one: when you average n independent things, the spread of that average shrinks by the square root of n.

Square root, not n. This shows up everywhere and disappoints everyone.

Go from batch size 32 to batch size 128 — four times the compute per step — and the gradient noise does not drop to a quarter. It halves. To halve it again you need batch size 512. The returns are brutal, and this is exactly why very large batches stop helping long before they stop costing.

The number that should change how you read results

You evaluate two models on a 500-question benchmark. Model A scores 71.2%. Model B scores 71.8%. B wins, ship it.

No. Work out how noisy a 500-question score is. For a proportion near 0.7 with n = 500, the standard deviation of the measurement is:

sqrt(0.7 * 0.3 / 500) = sqrt(0.00042) ≈ 0.0205

About 2 percentage points. Run the same model on a different random 500 questions and you would routinely see 69% or 73%. The 0.6 point gap between A and B is a quarter of one standard deviation. It is noise. You have measured nothing.

To resolve a 0.6 point difference reliably you need on the order of 20,000 questions, or a paired test that compares the two models question by question, which cancels out much of the shared difficulty and is far more efficient. Neither is what most benchmark tables do.

This is why leaderboards where the top eight entries sit within a point of each other are ranking noise, and why a model that gains half a point on a small evaluation has demonstrated nothing at all. Once you know the square root rule, you cannot unsee it — and you will start asking for error bars on results that never had any.