{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"variance-reduction","title":"Getting more signal from the items you already have","url":"https://addaly.com/learn/evaluating-ai/variance-reduction","position":28,"of":76,"minutes":9,"answer":"Fix every source of noise the change did not cause (same items, same seeds, same retrieval snapshot, plus a covariate adjustment) and a set you cannot afford to enlarge starts detecting differences half the size.","keyIdea":"Fix every source of noise the change did not cause (same items, same seeds, same retrieval snapshot, plus a covariate adjustment) and a set you cannot afford to enlarge starts detecting differences half the size."},"body":{"format":"markdown","text":"## The other lever\n\nSample size is one way to see smaller differences. Reducing the noise in each observation is the other, and it is usually cheaper. Every technique here works by removing variation the change did not cause, so that what remains is the change.\n\n## Hold everything else still\n\nThe largest single gain is running both systems on the same items, which the course has already established. The rest of the principle is the same idea applied to everything else that wobbles:\n\n- **Fix seeds and set temperature to 0** for the comparison run, even if production runs hotter. You are measuring a difference between configurations, not simulating production.\n- **Freeze the retrieval corpus.** A vector index that ingested 2,000 new documents overnight makes yesterday's numbers incomparable, and this is the most common cause of a mysterious drift in a RAG evaluation.\n- **Pin the model version and the tokeniser**, and record both.\n- **Use the same ordering and the same batching**, since provider-side batching can change outputs subtly.\n\nThis is the common-random-numbers idea from simulation: when two systems share their random draws, the shared part cancels in the difference and the comparison gets sharper. It costs nothing but discipline.\n\n## Stratify\n\nIf a known factor drives most of the variance (language, input length, intent) allocate the set proportionally across it and compute a stratified estimate rather than a single average. The between-stratum variance disappears from the estimate's error, leaving only the within-stratum part. When language accounts for, say, 40% of the total variance, stratifying removes that 40% from the noise for free.\n\n## Adjust with a covariate\n\nThis is the technique most teams have never tried offline, and it is the strongest one available.\n\nIf you have any per-item number that correlates with the outcome but is unaffected by your change, you can subtract its influence. Online, that number is usually the user's behaviour in a pre-experiment period: the method known as CUPED. Offline, it can be the score of a cheap baseline on the same item, an item difficulty rating, or the previous release's score.\n\n```\nY_adjusted = Y - theta * (X - mean(X))       theta = cov(Y, X) / var(X)\n```\n\nThe adjusted metric has the same expectation and a variance reduced by a factor of `1 - rho²`, where `rho` is the correlation between `Y` and the covariate. At `rho = 0.7` that is a 51% reduction: the same precision as doubling the number of items, at the cost of eight lines of code.\n\n## Use a score with more information in it\n\nA pass or fail throws away magnitude. An answer that misses one required field and one that is entirely wrong both score 0, and the difference between them is invisible to the metric. A graded rubric (four binary checks summed to a 0-to-4 score) carries more information per item and needs fewer items for the same resolution. This is one reason the rubric lesson later in the course insists on several small checks rather than one holistic rating.\n\n## Repeats, and a result that surprises people\n\nSuppose the model is non-deterministic, so each item has run-to-run noise on top of item-to-item noise. With `n` items and `k` runs each, the variance of the overall mean is:\n\n```\n(item_variance + run_variance / k) / n\n```\n\nFix a budget of `n * k` calls. Substituting shows the variance is minimised at `k = 1`: for estimating an average, one run on many items beats several runs on few. Repeats buy precision only on the individual item, not on the mean.\n\nSo use one run per item when the question is \"which system is better overall\", and use repeats when the question is \"is this specific item reliably passing\": a regression suite, a safety case, a customer's reported bug. Two different questions, two different designs, and teams routinely spend their budget on the wrong one.\n\n## What none of this fixes\n\nVariance reduction makes an estimate more precise. It does nothing about bias. A set drawn from the wrong traffic, labelled against a confused guide, or contaminated by training data will give you a tighter interval around the wrong number, which is worse than a loose interval around it, because it is more convincing. Fix the sampling and the labels first, then sharpen."},"check":{"question":"A team has a fixed budget of 1,200 model calls to compare two prompts, and the model is non-deterministic. Which allocation gives the most precise estimate of the difference in mean quality?","options":[{"text":"1,200 items run once each, since for estimating a mean the run-to-run noise is divided by the item count anyway.","correct":true},{"text":"400 items run three times each, because averaging three runs per item removes the non-determinism before the comparison.","correct":false,"tempting":"Averaging does reduce per-item noise, but it costs items three for one, and the arithmetic shows the mean's variance is minimised at one run per item."},{"text":"240 items run five times each, so every item's score is stable enough to trust individually.","correct":false,"tempting":"Optimises for per-item reliability, which is the right goal for a regression suite and the wrong one for estimating an overall difference."},{"text":"600 items run twice each, as a compromise between item coverage and run stability.","correct":false,"tempting":"Splits the difference rather than following the arithmetic; any k above one trades items for precision the mean does not need."}]},"prev":{"slug":"peeking-and-stopping-rules","title":"Watching the dashboard until it says yes","url":"https://addaly.com/learn/evaluating-ai/peeking-and-stopping-rules"},"next":{"slug":"simpsons-paradox","title":"Better in every group and worse overall","url":"https://addaly.com/learn/evaluating-ai/simpsons-paradox"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Getting more signal from the items you already have', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/variance-reduction (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Getting more signal from the items you already have","courseTitle":"Knowing If It Works","module":{"index":3,"slug":"deciding-if-it-is-real","title":"Deciding whether a difference is real"},"answer":null,"keyIdea":"Fix every source of noise the change did not cause (same items, same seeds, same retrieval snapshot, plus a covariate adjustment) and a set you cannot afford to enlarge starts detecting differences half the size.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-other-lever","runs":[{"text":"The other lever"}]},{"t":"p","runs":[{"text":"Sample size is one way to see smaller differences. Reducing the noise in each observation is the other, and it is usually cheaper. Every technique here works by removing variation the change did not cause, so that what remains is the change."}]},{"t":"h2","id":"hold-everything-else-still","runs":[{"text":"Hold everything else still"}]},{"t":"p","runs":[{"text":"The largest single gain is running both systems on the same items, which the course has already established. The rest of the principle is the same idea applied to everything else that wobbles:"}]},{"t":"ul","items":[[{"text":"Fix seeds and set temperature to 0","b":true},{"text":" for the comparison run, even if production runs hotter. You are measuring a difference between configurations, not simulating production."}],[{"text":"Freeze the retrieval corpus.","b":true},{"text":" A vector index that ingested 2,000 new documents overnight makes yesterday’s numbers incomparable, and this is the most common cause of a mysterious drift in a RAG evaluation."}],[{"text":"Pin the model version and the tokeniser","b":true},{"text":", and record both."}],[{"text":"Use the same ordering and the same batching","b":true},{"text":", since provider-side batching can change outputs subtly."}]]},{"t":"p","runs":[{"text":"This is the common-random-numbers idea from simulation: when two systems share their random draws, the shared part cancels in the difference and the comparison gets sharper. It costs nothing but discipline."}]},{"t":"h2","id":"stratify","runs":[{"text":"Stratify"}]},{"t":"p","runs":[{"text":"If a known factor drives most of the variance (language, input length, intent) allocate the set proportionally across it and compute a stratified estimate rather than a single average. The between-stratum variance disappears from the estimate’s error, leaving only the within-stratum part. When language accounts for, say, 40% of the total variance, stratifying removes that 40% from the noise for free."}]},{"t":"h2","id":"adjust-with-a-covariate","runs":[{"text":"Adjust with a covariate"}]},{"t":"p","runs":[{"text":"This is the technique most teams have never tried offline, and it is the strongest one available."}]},{"t":"p","runs":[{"text":"If you have any per-item number that correlates with the outcome but is unaffected by your change, you can subtract its influence. Online, that number is usually the user’s behaviour in a pre-experiment period: the method known as CUPED. Offline, it can be the score of a cheap baseline on the same item, an item difficulty rating, or the previous release’s score."}]},{"t":"code","lang":"","label":"Text","text":"Y_adjusted = Y - theta * (X - mean(X))       theta = cov(Y, X) / var(X)"},{"t":"p","runs":[{"text":"The adjusted metric has the same expectation and a variance reduced by a factor of "},{"text":"1 - rho²","code":true},{"text":", where "},{"text":"rho","code":true},{"text":" is the correlation between "},{"text":"Y","code":true},{"text":" and the covariate. At "},{"text":"rho = 0.7","code":true},{"text":" that is a 51% reduction: the same precision as doubling the number of items, at the cost of eight lines of code."}]},{"t":"h2","id":"use-a-score-with-more-information-in-it","runs":[{"text":"Use a score with more information in it"}]},{"t":"p","runs":[{"text":"A pass or fail throws away magnitude. An answer that misses one required field and one that is entirely wrong both score 0, and the difference between them is invisible to the metric. A graded rubric (four binary checks summed to a 0-to-4 score) carries more information per item and needs fewer items for the same resolution. This is one reason the rubric lesson later in the course insists on several small checks rather than one holistic rating."}]},{"t":"h2","id":"repeats-and-a-result-that-surprises-people","runs":[{"text":"Repeats, and a result that surprises people"}]},{"t":"p","runs":[{"text":"Suppose the model is non-deterministic, so each item has run-to-run noise on top of item-to-item noise. With "},{"text":"n","code":true},{"text":" items and "},{"text":"k","code":true},{"text":" runs each, the variance of the overall mean is:"}]},{"t":"code","lang":"","label":"Text","text":"(item_variance + run_variance / k) / n"},{"t":"p","runs":[{"text":"Fix a budget of "},{"text":"n * k","code":true},{"text":" calls. Substituting shows the variance is minimised at "},{"text":"k = 1","code":true},{"text":": for estimating an average, one run on many items beats several runs on few. Repeats buy precision only on the individual item, not on the mean."}]},{"t":"p","runs":[{"text":"So use one run per item when the question is “which system is better overall”, and use repeats when the question is “is this specific item reliably passing”: a regression suite, a safety case, a customer’s reported bug. Two different questions, two different designs, and teams routinely spend their budget on the wrong one."}]},{"t":"h2","id":"what-none-of-this-fixes","runs":[{"text":"What none of this fixes"}]},{"t":"p","runs":[{"text":"Variance reduction makes an estimate more precise. It does nothing about bias. A set drawn from the wrong traffic, labelled against a confused guide, or contaminated by training data will give you a tighter interval around the wrong number, which is worse than a loose interval around it, because it is more convincing. Fix the sampling and the labels first, then sharpen."}]}],"check":{"question":[{"text":"A team has a fixed budget of 1,200 model calls to compare two prompts, and the model is non-deterministic. Which allocation gives the most precise estimate of the difference in mean quality?"}],"options":[{"text":[{"text":"1,200 items run once each, since for estimating a mean the run-to-run noise is divided by the item count anyway."}],"correct":true,"tempting":null},{"text":[{"text":"400 items run three times each, because averaging three runs per item removes the non-determinism before the comparison."}],"correct":false,"tempting":[{"text":"Averaging does reduce per-item noise, but it costs items three for one, and the arithmetic shows the mean’s variance is minimised at one run per item."}]},{"text":[{"text":"240 items run five times each, so every item’s score is stable enough to trust individually."}],"correct":false,"tempting":[{"text":"Optimises for per-item reliability, which is the right goal for a regression suite and the wrong one for estimating an overall difference."}]},{"text":[{"text":"600 items run twice each, as a compromise between item coverage and run stability."}],"correct":false,"tempting":[{"text":"Splits the difference rather than following the arithmetic; any k above one trades items for precision the mean does not need."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"peeking-and-stopping-rules","title":"Watching the dashboard until it says yes","minutes":9},"next":{"slug":"simpsons-paradox","title":"Better in every group and worse overall","minutes":8}}}