{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"is-the-difference-real","title":"Three points better, or three points of noise","url":"https://addaly.com/learn/context-engineering/is-the-difference-real","position":82,"of":87,"minutes":10,"answer":"Two variants must be compared on the same inputs so that per-item differences can be counted.","keyIdea":"Two variants must be compared on the same inputs so that per-item differences can be counted, because a paired design detects far smaller effects than independent samples and because most reported prompt improvements are within the noise of the evaluation that produced them."},"body":{"format":"markdown","text":"## The number that ships bad changes\n\nYou run the golden set on the current prompt: 78 percent. You run it on the new one: 81 percent. Three points better. You ship it.\n\nA fortnight later someone re-runs the old prompt and gets 80. Nothing changed. The three points were never there.\n\nThis is the most common failure in applied prompt work, and it is not a statistics problem so much as a habit problem: the run produced a number, the number was bigger, and nobody asked what the measurement's own variability was.\n\n## Pair everything\n\nThe first and largest improvement costs nothing. Run both variants on **the same inputs**, and compare item by item rather than comparing two aggregate percentages.\n\nWhy it matters so much: most of the variance in an evaluation comes from the items, not the variants. Some questions are hard and both variants fail them; some are easy and both pass. Those items contribute nothing to the comparison and, in an unpaired design, they contribute a great deal of noise.\n\nPairing cancels them. What is left is only the **discordant** items (where one variant passed and the other failed) and those are the entire evidence.\n\n```\n            new passes   new fails\nold passes      142          9       <- b = 9\nold fails        17         32       <- c = 17\n```\n\nThe 142 and the 32 are irrelevant to the comparison. The question is whether 17 versus 9 is more than chance would produce. McNemar's test answers exactly that, and for these numbers it is not convincing: with 26 discordant pairs split 17 to 9, the two-sided p-value is around 0.17. You are looking at a plausible three-point improvement with no evidence behind it.\n\nA paired bootstrap gives the same answer with no formula to remember: resample the items with replacement a few thousand times, recompute the difference each time, and read off the interval. Twenty lines of NumPy, free, and it works for any metric rather than just binary pass or fail.\n\n## Roughly how many you need\n\nThe honest shape, without pretending to precision:\n\n- **50 items**: only very large effects, roughly ten points or more, are detectable. Most of what you test here will be inconclusive.\n- **200 items**: differences of about five points become visible, depending on how often the variants actually disagree.\n- **1,000 items**: two-point differences come within reach.\n\nThe exact numbers depend on the discordant rate, which you can only know after running. The practical consequence is worth stating plainly: **if your set has 100 items, you cannot resolve a two-point difference, and no amount of re-running will change that.** Either accept that small changes are unmeasurable and decide on other grounds (cost, latency, simplicity) or grow the set.\n\nThat is not a counsel of despair. It is permission to stop agonising over changes your instrument cannot see, and to spend the effort on changes large enough to matter.\n\n## Sampling noise is separate and also real\n\nEven with fixed inputs, the model does not return identical output twice.\n\nSet temperature to 0 for comparisons. This removes most of the variance and it does not remove all of it: floating-point non-associativity means results can differ with batch composition and hardware, so identical requests occasionally produce different tokens even at zero temperature. It is a small effect and it is not zero, and it is why a rerun of an unchanged configuration does not always reproduce exactly.\n\nIf your product runs at a higher temperature, evaluate there and run each item several times, scoring the mean. That multiplies your evaluation cost by the number of samples, which is the real reason people skip it.\n\n## Three practices that cost nothing\n\n**Run the baseline twice.** Same configuration, two runs. The difference between them is your noise floor, measured rather than assumed. Any candidate improvement smaller than that gap is not an improvement. This single habit prevents most bad ships and takes one extra run.\n\n**Change one thing.** A variant that alters the prompt and k and the reranker tells you the bundle differs, and nothing about which part. When the bundle regresses six months later you will have no way back.\n\n**Report the interval.** \"81 percent (95% CI 76–86)\" against \"78 percent (73–83)\" makes the situation legible to everyone in the room, including the person who wants to ship. A bare percentage invites a decision the data does not support, and the interval is two lines of bootstrap away.\n\n## The uncomfortable implication\n\nMost published prompt-engineering advice (including techniques with names, diagrams and enthusiastic write-ups) has been evaluated on small sets, at one temperature, on one model, without a paired test or an interval.\n\nThat does not make the advice wrong. It makes it unverified, which is a different thing and should produce a different response: try it on your own set, with a baseline run for the noise floor, and see. Roughly half of what you try will turn out to do nothing measurable on your task, and knowing which half is worth more than any of the techniques individually."},"check":{"question":"Two prompts are compared on the same 180 items. They disagree on 26: the new one wins 17 and loses 9. Why is this weak evidence despite a three-point aggregate gain?","options":[{"text":"Because 180 items is below the minimum at which any paired comparison can be run.","correct":false,"tempting":"Paired tests work at any size; what changes with size is the smallest effect they can resolve, not their validity."},{"text":"Because the 154 agreeing items dilute the difference, pulling the aggregate towards no effect.","correct":false,"tempting":"Agreeing items are excluded from the paired test precisely so they cannot dilute anything; they affect only the headline percentage."},{"text":"Because a 17 to 9 split of 26 pairs is well within what chance produces.","correct":true},{"text":"Because the losses matter more than the wins, so an improvement needs a wider margin to justify the regressions it causes.","correct":false,"tempting":"Asymmetric costs are a product judgement worth making separately; they are not why this particular split fails to be evidence."}]},"prev":{"slug":"ablation-earns-the-block","title":"Take it out and see what happens","url":"https://addaly.com/learn/context-engineering/ablation-earns-the-block"},"next":{"slug":"judging-with-a-model","title":"Using a model to mark the homework","url":"https://addaly.com/learn/context-engineering/judging-with-a-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Three points better, or three points of noise', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/is-the-difference-real (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Three points better, or three points of noise","courseTitle":"Context Engineering","module":{"index":9,"slug":"shipping-and-measuring","title":"Shipping context, and proving it got better"},"answer":null,"keyIdea":"Two variants must be compared on the same inputs so that per-item differences can be counted, because a paired design detects far smaller effects than independent samples and because most reported prompt improvements are within the noise of the evaluation that produced them.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-number-that-ships-bad-changes","runs":[{"text":"The number that ships bad changes"}]},{"t":"p","runs":[{"text":"You run the golden set on the current prompt: 78 percent. You run it on the new one: 81 percent. Three points better. You ship it."}]},{"t":"p","runs":[{"text":"A fortnight later someone re-runs the old prompt and gets 80. Nothing changed. The three points were never there."}]},{"t":"p","runs":[{"text":"This is the most common failure in applied prompt work, and it is not a statistics problem so much as a habit problem: the run produced a number, the number was bigger, and nobody asked what the measurement’s own variability was."}]},{"t":"h2","id":"pair-everything","runs":[{"text":"Pair everything"}]},{"t":"p","runs":[{"text":"The first and largest improvement costs nothing. Run both variants on "},{"text":"the same inputs","b":true},{"text":", and compare item by item rather than comparing two aggregate percentages."}]},{"t":"p","runs":[{"text":"Why it matters so much: most of the variance in an evaluation comes from the items, not the variants. Some questions are hard and both variants fail them; some are easy and both pass. Those items contribute nothing to the comparison and, in an unpaired design, they contribute a great deal of noise."}]},{"t":"p","runs":[{"text":"Pairing cancels them. What is left is only the "},{"text":"discordant","b":true},{"text":" items (where one variant passed and the other failed) and those are the entire evidence."}]},{"t":"code","lang":"","label":"Text","text":"            new passes   new fails\nold passes      142          9       <- b = 9\nold fails        17         32       <- c = 17"},{"t":"p","runs":[{"text":"The 142 and the 32 are irrelevant to the comparison. The question is whether 17 versus 9 is more than chance would produce. McNemar’s test answers exactly that, and for these numbers it is not convincing: with 26 discordant pairs split 17 to 9, the two-sided p-value is around 0.17. You are looking at a plausible three-point improvement with no evidence behind it."}]},{"t":"p","runs":[{"text":"A paired bootstrap gives the same answer with no formula to remember: resample the items with replacement a few thousand times, recompute the difference each time, and read off the interval. Twenty lines of NumPy, free, and it works for any metric rather than just binary pass or fail."}]},{"t":"h2","id":"roughly-how-many-you-need","runs":[{"text":"Roughly how many you need"}]},{"t":"p","runs":[{"text":"The honest shape, without pretending to precision:"}]},{"t":"ul","items":[[{"text":"50 items","b":true},{"text":": only very large effects, roughly ten points or more, are detectable. Most of what you test here will be inconclusive."}],[{"text":"200 items","b":true},{"text":": differences of about five points become visible, depending on how often the variants actually disagree."}],[{"text":"1,000 items","b":true},{"text":": two-point differences come within reach."}]]},{"t":"p","runs":[{"text":"The exact numbers depend on the discordant rate, which you can only know after running. The practical consequence is worth stating plainly: "},{"text":"if your set has 100 items, you cannot resolve a two-point difference, and no amount of re-running will change that.","b":true},{"text":" Either accept that small changes are unmeasurable and decide on other grounds (cost, latency, simplicity) or grow the set."}]},{"t":"p","runs":[{"text":"That is not a counsel of despair. It is permission to stop agonising over changes your instrument cannot see, and to spend the effort on changes large enough to matter."}]},{"t":"h2","id":"sampling-noise-is-separate-and-also-real","runs":[{"text":"Sampling noise is separate and also real"}]},{"t":"p","runs":[{"text":"Even with fixed inputs, the model does not return identical output twice."}]},{"t":"p","runs":[{"text":"Set temperature to 0 for comparisons. This removes most of the variance and it does not remove all of it: floating-point non-associativity means results can differ with batch composition and hardware, so identical requests occasionally produce different tokens even at zero temperature. It is a small effect and it is not zero, and it is why a rerun of an unchanged configuration does not always reproduce exactly."}]},{"t":"p","runs":[{"text":"If your product runs at a higher temperature, evaluate there and run each item several times, scoring the mean. That multiplies your evaluation cost by the number of samples, which is the real reason people skip it."}]},{"t":"h2","id":"three-practices-that-cost-nothing","runs":[{"text":"Three practices that cost nothing"}]},{"t":"p","runs":[{"text":"Run the baseline twice.","b":true},{"text":" Same configuration, two runs. The difference between them is your noise floor, measured rather than assumed. Any candidate improvement smaller than that gap is not an improvement. This single habit prevents most bad ships and takes one extra run."}]},{"t":"p","runs":[{"text":"Change one thing.","b":true},{"text":" A variant that alters the prompt and k and the reranker tells you the bundle differs, and nothing about which part. When the bundle regresses six months later you will have no way back."}]},{"t":"p","runs":[{"text":"Report the interval.","b":true},{"text":" “81 percent (95% CI 76–86)” against “78 percent (73–83)” makes the situation legible to everyone in the room, including the person who wants to ship. A bare percentage invites a decision the data does not support, and the interval is two lines of bootstrap away."}]},{"t":"h2","id":"the-uncomfortable-implication","runs":[{"text":"The uncomfortable implication"}]},{"t":"p","runs":[{"text":"Most published prompt-engineering advice (including techniques with names, diagrams and enthusiastic write-ups) has been evaluated on small sets, at one temperature, on one model, without a paired test or an interval."}]},{"t":"p","runs":[{"text":"That does not make the advice wrong. It makes it unverified, which is a different thing and should produce a different response: try it on your own set, with a baseline run for the noise floor, and see. Roughly half of what you try will turn out to do nothing measurable on your task, and knowing which half is worth more than any of the techniques individually."}]}],"check":{"question":[{"text":"Two prompts are compared on the same 180 items. They disagree on 26: the new one wins 17 and loses 9. Why is this weak evidence despite a three-point aggregate gain?"}],"options":[{"text":[{"text":"Because 180 items is below the minimum at which any paired comparison can be run."}],"correct":false,"tempting":[{"text":"Paired tests work at any size; what changes with size is the smallest effect they can resolve, not their validity."}]},{"text":[{"text":"Because the 154 agreeing items dilute the difference, pulling the aggregate towards no effect."}],"correct":false,"tempting":[{"text":"Agreeing items are excluded from the paired test precisely so they cannot dilute anything; they affect only the headline percentage."}]},{"text":[{"text":"Because a 17 to 9 split of 26 pairs is well within what chance produces."}],"correct":true,"tempting":null},{"text":[{"text":"Because the losses matter more than the wins, so an improvement needs a wider margin to justify the regressions it causes."}],"correct":false,"tempting":[{"text":"Asymmetric costs are a product judgement worth making separately; they are not why this particular split fails to be evidence."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"ablation-earns-the-block","title":"Take it out and see what happens","minutes":9},"next":{"slug":"judging-with-a-model","title":"Using a model to mark the homework","minutes":9}}}