{"course":{"slug":"maths-you-need","title":"The Maths You Actually Need","url":"https://addaly.com/learn/maths-you-need","level":"intermediate","track":"deep"},"lesson":{"slug":"expectation-and-variance","title":"Expectation and variance in plain terms","url":"https://addaly.com/learn/maths-you-need/expectation-and-variance","position":36,"of":76,"minutes":8,"answer":"An expectation is a weighted average that may never occur, and its noise falls with the square root of the sample size.","keyIdea":"An expectation is a weighted average that may never occur, and its noise falls with the square root of the sample size."},"body":{"format":"markdown","text":"## Expectation is a weighted average\n\nEvery possible value, multiplied by how likely it is, all added up.\n\nA street vendor in Jakarta sells 40 bowls on a good day (60% of days) and 15 on a bad one (40%).\n\n```\nexpected sales = 0.6 * 40 + 0.4 * 15 = 24 + 6 = 30 bowls\n```\n\nThirty is the expectation. Now notice what it is not: he never sells 30 bowls. Not once, ever. The expectation is not a value you expect to see. It is what the average settles near over many days.\n\nThis trips people constantly. The expected value of a die roll is 3.5. Average household size in a country might be 4.2 people. Neither is a possible outcome. An expectation is a summary of a distribution, and summaries are allowed to be things that never happen.\n\n## Every loss you have seen is an expectation\n\nThe loss you actually want to minimise is the average over all data your model will ever meet. You cannot compute that: you do not have all the data. You compute it over your training set, which is itself a stand-in. And in practice you compute it over one batch of 32 or 512 examples, which is a stand-in for the stand-in.\n\nSo the gradient you step with is an estimate. It is roughly right, and it wobbles. That wobble is not a flaw to be engineered away; it is the \"stochastic\" in stochastic gradient descent, and a small amount of it helps by shaking the model out of bad spots.\n\n## Variance is how much things wobble\n\nVariance measures spread: how far values sit from their mean, on average, with the distances squared so they do not cancel out. Standard deviation is the square root of that, which puts it back into the original units (bowls, rupees, seconds) so it is the one worth reading.\n\nThe fact that matters most in machine learning is this one: **when you average n independent things, the spread of that average shrinks by the square root of n.**\n\nSquare root, not n. This shows up everywhere and disappoints everyone.\n\nGo from batch size 32 to batch size 128 (four times the compute per step) and the gradient noise does not drop to a quarter. It halves. To halve it again you need batch size 512. The returns are brutal, and this is exactly why very large batches stop helping long before they stop costing.\n\n## The number that should change how you read results\n\nYou evaluate two models on a 500-question benchmark. Model A scores 71.2%. Model B scores 71.8%. B wins, ship it.\n\nNo. Work out how noisy a 500-question score is. For a proportion near 0.7 with n = 500, the standard deviation of the measurement is:\n\n```\nsqrt(0.7 * 0.3 / 500) = sqrt(0.00042) ≈ 0.0205\n```\n\nAbout 2 percentage points. Run the same model on a different random 500 questions and you would routinely see 69% or 73%. The 0.6 point gap between A and B is a quarter of one standard deviation. It is noise. You have measured nothing.\n\nTo resolve a 0.6 point difference reliably you need on the order of 20,000 questions, or a paired test that compares the two models question by question, which cancels out much of the shared difficulty and is far more efficient. Neither is what most benchmark tables do.\n\nThis is why leaderboards where the top eight entries sit within a point of each other are ranking noise, and why a model that gains half a point on a small evaluation has demonstrated nothing at all. Once you know the square root rule, you cannot unsee it, and you will start asking for error bars on results that never had any."},"check":{"question":"You raise the batch size from 32 to 128, hoping for a less noisy gradient. What happens to the noise?","options":[{"text":"It stays the same. A gradient is an average, and averaging more examples does not change what the average is.","correct":false,"tempting":"Confuses the value an estimate is centred on, which does not change, with how much it wobbles, which does."},{"text":"It falls to a quarter, since the error shrinks in proportion to the number of examples.","correct":false,"tempting":"Assumes the error shrinks linearly with sample size rather than with its square root."},{"text":"It roughly halves, because the spread of an average shrinks with the square root of the count.","correct":true},{"text":"It depends on the learning rate, since that is what controls how far each step goes.","correct":false,"tempting":"Mixes up how big your steps are with how noisy the direction of those steps is."}]},"prev":{"slug":"the-distributions-you-will-meet","title":"Six distributions, and the process each one describes","url":"https://addaly.com/learn/maths-you-need/the-distributions-you-will-meet"},"next":{"slug":"covariance-and-the-shape-of-a-cloud","title":"Covariance, and the shape of a cloud of points","url":"https://addaly.com/learn/maths-you-need/covariance-and-the-shape-of-a-cloud"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Expectation and variance in plain terms', in The Maths You Actually Need. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/maths-you-need/expectation-and-variance (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Expectation and variance in plain terms","courseTitle":"The Maths You Actually Need","module":{"index":4,"slug":"probability-you-can-rely-on","title":"Probability you can rely on"},"answer":null,"keyIdea":"An expectation is a weighted average that may never occur, and its noise falls with the square root of the sample size.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"expectation-is-a-weighted-average","runs":[{"text":"Expectation is a weighted average"}]},{"t":"p","runs":[{"text":"Every possible value, multiplied by how likely it is, all added up."}]},{"t":"p","runs":[{"text":"A street vendor in Jakarta sells 40 bowls on a good day (60% of days) and 15 on a bad one (40%)."}]},{"t":"code","lang":"","label":"Text","text":"expected sales = 0.6 * 40 + 0.4 * 15 = 24 + 6 = 30 bowls"},{"t":"p","runs":[{"text":"Thirty is the expectation. Now notice what it is not: he never sells 30 bowls. Not once, ever. The expectation is not a value you expect to see. It is what the average settles near over many days."}]},{"t":"p","runs":[{"text":"This trips people constantly. The expected value of a die roll is 3.5. Average household size in a country might be 4.2 people. Neither is a possible outcome. An expectation is a summary of a distribution, and summaries are allowed to be things that never happen."}]},{"t":"h2","id":"every-loss-you-have-seen-is-an-expectation","runs":[{"text":"Every loss you have seen is an expectation"}]},{"t":"p","runs":[{"text":"The loss you actually want to minimise is the average over all data your model will ever meet. You cannot compute that: you do not have all the data. You compute it over your training set, which is itself a stand-in. And in practice you compute it over one batch of 32 or 512 examples, which is a stand-in for the stand-in."}]},{"t":"p","runs":[{"text":"So the gradient you step with is an estimate. It is roughly right, and it wobbles. That wobble is not a flaw to be engineered away; it is the “stochastic” in stochastic gradient descent, and a small amount of it helps by shaking the model out of bad spots."}]},{"t":"h2","id":"variance-is-how-much-things-wobble","runs":[{"text":"Variance is how much things wobble"}]},{"t":"p","runs":[{"text":"Variance measures spread: how far values sit from their mean, on average, with the distances squared so they do not cancel out. Standard deviation is the square root of that, which puts it back into the original units (bowls, rupees, seconds) so it is the one worth reading."}]},{"t":"p","runs":[{"text":"The fact that matters most in machine learning is this one: "},{"text":"when you average n independent things, the spread of that average shrinks by the square root of n.","b":true}]},{"t":"p","runs":[{"text":"Square root, not n. This shows up everywhere and disappoints everyone."}]},{"t":"p","runs":[{"text":"Go from batch size 32 to batch size 128 (four times the compute per step) and the gradient noise does not drop to a quarter. It halves. To halve it again you need batch size 512. The returns are brutal, and this is exactly why very large batches stop helping long before they stop costing."}]},{"t":"h2","id":"the-number-that-should-change-how-you-read-results","runs":[{"text":"The number that should change how you read results"}]},{"t":"p","runs":[{"text":"You evaluate two models on a 500-question benchmark. Model A scores 71.2%. Model B scores 71.8%. B wins, ship it."}]},{"t":"p","runs":[{"text":"No. Work out how noisy a 500-question score is. For a proportion near 0.7 with n = 500, the standard deviation of the measurement is:"}]},{"t":"code","lang":"","label":"Text","text":"sqrt(0.7 * 0.3 / 500) = sqrt(0.00042) ≈ 0.0205"},{"t":"p","runs":[{"text":"About 2 percentage points. Run the same model on a different random 500 questions and you would routinely see 69% or 73%. The 0.6 point gap between A and B is a quarter of one standard deviation. It is noise. You have measured nothing."}]},{"t":"p","runs":[{"text":"To resolve a 0.6 point difference reliably you need on the order of 20,000 questions, or a paired test that compares the two models question by question, which cancels out much of the shared difficulty and is far more efficient. Neither is what most benchmark tables do."}]},{"t":"p","runs":[{"text":"This is why leaderboards where the top eight entries sit within a point of each other are ranking noise, and why a model that gains half a point on a small evaluation has demonstrated nothing at all. Once you know the square root rule, you cannot unsee it, and you will start asking for error bars on results that never had any."}]}],"check":{"question":[{"text":"You raise the batch size from 32 to 128, hoping for a less noisy gradient. What happens to the noise?"}],"options":[{"text":[{"text":"It stays the same. A gradient is an average, and averaging more examples does not change what the average is."}],"correct":false,"tempting":[{"text":"Confuses the value an estimate is centred on, which does not change, with how much it wobbles, which does."}]},{"text":[{"text":"It falls to a quarter, since the error shrinks in proportion to the number of examples."}],"correct":false,"tempting":[{"text":"Assumes the error shrinks linearly with sample size rather than with its square root."}]},{"text":[{"text":"It roughly halves, because the spread of an average shrinks with the square root of the count."}],"correct":true,"tempting":null},{"text":[{"text":"It depends on the learning rate, since that is what controls how far each step goes."}],"correct":false,"tempting":[{"text":"Mixes up how big your steps are with how noisy the direction of those steps is."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"the-distributions-you-will-meet","title":"Six distributions, and the process each one describes","minutes":10},"next":{"slug":"covariance-and-the-shape-of-a-cloud","title":"Covariance, and the shape of a cloud of points","minutes":9}}}