The law of large numbers, and how fast it works

The law, and the mechanism behind it

The average of many independent draws settles toward the expected value. Everyone knows this; fewer people know how fast, and fewer still know when it stops being true. Both come from three lines of algebra.

Each draw has some variance σ². For independent draws, variances add, so the sum of n of them has variance n σ². The average is the sum divided by n, and dividing a quantity by n divides its variance by n². So

Var(average) = n σ² / n² = σ² / n
SD(average)  = σ / √n

The spread of the average shrinks with the square root of the sample size, and the word doing the work is independent. If the draws were correlated, the covariances would add too, and the n in the denominator would be smaller than it looks.

How slow the square root is

A coin flip has σ = 0.5. The average of 100 flips has spread 0.05; of 10,000 flips, 0.005. Ten times the precision costs a hundred times the samples. Module 4 stated this; here it is derived, and it is worth feeling the weight of it. An accuracy estimate you want to sharpen from ±3 points to ±1 point needs nine times the test items. There is no cleverness that changes the exponent for a plain average.

A quick sizing rule follows. For a proportion, σ is at most 0.5, so to get within ε with about 95 per cent confidence you need roughly n = (2 × 0.5 / ε)² = 1 / ε². Within one point, ε = 0.01, gives n = 10,000. Within five points, 400. The course evaluating-ai turns this into a procedure; the arithmetic is only this.

A guarantee with no assumptions

The square-root law says how the spread shrinks. Chebyshev's inequality turns it into a promise that needs no assumption about the shape of the distribution at all:

P(|average − μ| ≥ k × σ/√n) ≤ 1 / k²

For 400 coin flips, σ/√n = 0.025. The chance the average is off by 0.125 or more, five spreads, is at most 1/25 = 4 per cent. The true chance is far smaller, under one in a million, because the bell curve of the next lesson has much thinner tails than Chebyshev allows for. But Chebyshev holds for any distribution with finite variance, and "at most four per cent, whatever the shape" is sometimes the sentence you need.

When the law fails: infinite variance

The derivation assumed σ² exists. Some distributions have none. The Cauchy distribution, which is what you get from the ratio of two normal variables, has a bell-shaped middle and tails so heavy that its variance, and even its mean, are undefined.

import numpy as np
rng = np.random.default_rng(0)
for _ in range(4):
    print(rng.standard_cauchy(200_000).mean())
# 2.67   0.24   -0.01   -1.66

Four averages of two hundred thousand draws each, and they disagree by more than four units. The average of a million Cauchy draws is exactly as spread out as a single draw. Every so often a single value of a hundred thousand arrives and drags the whole average with it. The law of large numbers does not fail slowly here; it does not apply.

Real quantities are never exactly Cauchy, but the heavy-tailed ones from module 4, response times, incomes, file sizes, the length of the longest document, live near that edge. Their averages converge, if at all, far more slowly than 1/√n promises, and the estimate keeps jumping as rarer extremes turn up. That is the mechanism behind the advice to report percentiles: a percentile has a finite variance even when the data do not.

When the law fails: correlated draws

The second assumption was independence. Suppose an evaluation set has 2,000 answers, but they come from 80 users with 25 answers each, and answers from the same user tend to agree: a correlation of ρ = 0.5 within a user. The variance of the average is inflated by the design effect:

design effect = 1 + (m − 1) ρ = 1 + 24 × 0.5 = 13
effective n   = 2,000 / 13 ≈ 154

Two thousand rows carry the information of 154 independent ones. An interval computed as if n = 2000 is 3.6 times too narrow. This is the most common way a confident number turns out to be wrong: not a bad formula, but a sample that was less independent than it was counted as. Sessions, documents, days, batches, and anything generated by the same prompt are the usual culprits.

The third failure: the thing moved

Averaging assumes there is one fixed quantity to converge to. An average of a model's accuracy across six months of traffic converges to the average of a changing thing, which describes no month in particular. The law of large numbers is exact about a stationary world and silent about any other.

The rule to keep

Spread falls as σ/√n, provided the variance is finite and the draws are independent. Check both before trusting the n. When the variance is not finite, average something else. When the draws are clustered, count the clusters.