Getting more signal from the items you already have

The other lever

Sample size is one way to see smaller differences. Reducing the noise in each observation is the other, and it is usually cheaper. Every technique here works by removing variation the change did not cause, so that what remains is the change.

Hold everything else still

The largest single gain is running both systems on the same items, which the course has already established. The rest of the principle is the same idea applied to everything else that wobbles:

• Fix seeds and set temperature to 0 for the comparison run, even if production runs hotter. You are measuring a difference between configurations, not simulating production.
• Freeze the retrieval corpus. A vector index that ingested 2,000 new documents overnight makes yesterday’s numbers incomparable, and this is the most common cause of a mysterious drift in a RAG evaluation.
• Pin the model version and the tokeniser, and record both.
• Use the same ordering and the same batching, since provider-side batching can change outputs subtly.

This is the common-random-numbers idea from simulation: when two systems share their random draws, the shared part cancels in the difference and the comparison gets sharper. It costs nothing but discipline.

Stratify

If a known factor drives most of the variance (language, input length, intent) allocate the set proportionally across it and compute a stratified estimate rather than a single average. The between-stratum variance disappears from the estimate’s error, leaving only the within-stratum part. When language accounts for, say, 40% of the total variance, stratifying removes that 40% from the noise for free.

Adjust with a covariate

This is the technique most teams have never tried offline, and it is the strongest one available.

If you have any per-item number that correlates with the outcome but is unaffected by your change, you can subtract its influence. Online, that number is usually the user’s behaviour in a pre-experiment period: the method known as CUPED. Offline, it can be the score of a cheap baseline on the same item, an item difficulty rating, or the previous release’s score.

Y_adjusted = Y - theta * (X - mean(X))       theta = cov(Y, X) / var(X)

The adjusted metric has the same expectation and a variance reduced by a factor of 1 - rho², where rho is the correlation between Y and the covariate. At rho = 0.7 that is a 51% reduction: the same precision as doubling the number of items, at the cost of eight lines of code.

Use a score with more information in it

A pass or fail throws away magnitude. An answer that misses one required field and one that is entirely wrong both score 0, and the difference between them is invisible to the metric. A graded rubric (four binary checks summed to a 0-to-4 score) carries more information per item and needs fewer items for the same resolution. This is one reason the rubric lesson later in the course insists on several small checks rather than one holistic rating.

Repeats, and a result that surprises people

Suppose the model is non-deterministic, so each item has run-to-run noise on top of item-to-item noise. With n items and k runs each, the variance of the overall mean is:

(item_variance + run_variance / k) / n

Fix a budget of n * k calls. Substituting shows the variance is minimised at k = 1: for estimating an average, one run on many items beats several runs on few. Repeats buy precision only on the individual item, not on the mean.

So use one run per item when the question is “which system is better overall”, and use repeats when the question is “is this specific item reliably passing”: a regression suite, a safety case, a customer’s reported bug. Two different questions, two different designs, and teams routinely spend their budget on the wrong one.

What none of this fixes

Variance reduction makes an estimate more precise. It does nothing about bias. A set drawn from the wrong traffic, labelled against a confused guide, or contaminated by training data will give you a tighter interval around the wrong number, which is worse than a loose interval around it, because it is more convincing. Fix the sampling and the labels first, then sharpen.