{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"simulating-the-null","title":"Asking what pure chance would have produced","url":"https://addaly.com/learn/evaluating-ai/simulating-the-null","position":21,"of":76,"minutes":9,"answer":"Shuffle which system produced which result ten thousand times and see how often chance beats your observed gap: one loop replaces a shelf of named tests and encodes your design honestly.","keyIdea":"Shuffle which system produced which result ten thousand times and see how often chance beats your observed gap: one loop replaces a shelf of named tests and encodes your design honestly."},"body":{"format":"markdown","text":"## The question under every comparison\n\nPrompt B scores 5 points above prompt A on the same 200 items. Before anything else, one question has to be answered: if the two prompts were identical in quality, how often would a gap that big appear anyway?\n\nThere are two ways to answer it. You can look up the right named test, check its assumptions, and hope you chose correctly. Or you can build the world where the null is true and count. The second is a permutation test, it takes ten lines, and it is very hard to get wrong in a way you would not notice.\n\n## The engine\n\nYou have per-item results for both systems. Under the null hypothesis, the label saying which result came from A and which from B carries no information, so swapping them within an item should change nothing systematic. Do exactly that, many times.\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\n\na = np.array(scores_a)          # one score per item, same order\nb = np.array(scores_b)\nobserved = b.mean() - a.mean()  # e.g. +0.050\n\n> Figure: A permutation test in five steps. Steps: 1. Per-item scores (Both systems, the same items, the same order.); 2. Observed gap (Mean of B minus mean of A, say +0.050.); 3. Flip signs at random (Each item's difference times ±1: the same as swapping which system is called A.); 4. Repeat 10,000 times (The spread of gaps that chance alone produces.); 5. Count the tail (The share of shuffles at least as large as the observed gap.). For 200 items the null gaps spread about ±0.021 either way, so an observed +0.050 is beaten by roughly 1.5% of shuffles. That share is the p-value, with no table and no normality assumption.\n\ndiffs = b - a\nnull = []\nfor _ in range(10_000):\n    signs = rng.choice([-1, 1], size=len(diffs))\n    null.append((diffs * signs).mean())\n\nnull = np.array(null)\np = (np.abs(null) >= abs(observed)).mean()\nprint(f\"gap {observed:+.3f}, p = {p:.3f}\")\n```\n\nFlipping the sign of each item's difference is the same as swapping which system is called A. Run it and you get a distribution of gaps that chance alone produces: for 200 items with this much spread, typically a standard deviation around 0.021. Against that, an observed +0.050 sits well out in the tail, and roughly 1.5% of shuffles exceed it. That is your p-value, computed from your own data, with no assumption of normality and no table.\n\n## Why this beats remembering tests\n\nThe permutation approach works for any statistic you can compute. Median latency. Recall at 5. A weighted composite of three rubric checks. Cost per successful task. There is no closed-form test for most of these, and you do not need one: you only need to be able to compute the number and to know what the null would scramble.\n\nIt also forces you to state your design, which is the part that most often goes wrong. **What you shuffle is your claim about what is exchangeable.**\n\n- Paired evaluation, same items through both systems: flip signs within each item, as above.\n- Two separate groups of users: shuffle the group labels across users.\n- Items grouped by customer: shuffle at the customer level, keeping each customer's rows together.\n\nShuffling rows when the rows are grouped gives a confidently wrong answer, and the next lessons cover why. The virtue of writing the shuffle by hand is that the mistake becomes visible in code rather than hiding in a function's assumptions.\n\n## The A/A test, which you should run first\n\nBefore trusting any comparison machinery, point it at two copies of the same system. Run configuration A twice, feed both result sets into the test, and see what it says.\n\nThe result should be an unremarkable gap and a p-value scattered anywhere between 0 and 1. If A against A produces a \"significant\" difference, you have found a bug, and it is nearly always one of five:\n\n- The two runs used different subsets, because an item errored out in one of them and was silently dropped.\n- Results were compared in the wrong order, so item 41's A score sits against item 42's B score.\n- A cache served the first run and not the second.\n- The set is ordered, and a truncated run covers a different mix.\n- Non-determinism is larger than you assumed, which is not a bug but is a finding.\n\nAn A/A test costs one extra run and it has caught bad harnesses at essentially every team that has bothered.\n\n## What it cannot do\n\nA simulation answers exactly the question you encoded, and nothing more.\n\n- It cannot rescue a biased sample. If your 200 items over-represent one intent, permutation gives you a precise p-value about a set that does not resemble your traffic.\n- It cannot tell you the effect is important. A gap can be reliably real and commercially worthless; report the size and its interval next to the p-value.\n- It assumes the outcomes you fed it are comparable. Two runs at temperature 0.7, compared without accounting for run-to-run variance, will produce a small p for a difference that vanishes tomorrow.\n\nTen lines, `numpy`, and a laptop. For small sets you can do the same thing in a spreadsheet with a column of random signs, and it is worth doing once by hand to see the null distribution appear."},"check":{"question":"A team runs a permutation test by shuffling all 600 result rows freely between the two systems, but the 600 rows come from 120 customers with 5 rows each. What does that shuffle get wrong?","options":[{"text":"Nothing, provided the number of rows assigned to each system stays equal after each shuffle.","correct":false,"tempting":"Treats balance as the only requirement; the problem is not how many rows each side gets but which rows are treated as independently exchangeable."},{"text":"It breaks the pairing, so the test loses power and will fail to detect a real difference.","correct":false,"tempting":"Names a real cost of unpaired shuffling, but the more serious error here is in the opposite direction: the p-value comes out too small, not too large."},{"text":"It treats correlated rows from one customer as independent, so the null distribution is narrower than reality and the p-value comes out too small.","correct":true},{"text":"It requires the two systems to have been run on identical inputs, which grouped data cannot guarantee.","correct":false,"tempting":"Confuses a data-collection question with the exchangeability assumption the shuffle encodes; the inputs here are identical and the grouping is still the problem."}]},"prev":{"slug":"the-benchmark-you-read-about","title":"Public benchmarks, and what they are for","url":"https://addaly.com/learn/evaluating-ai/the-benchmark-you-read-about"},"next":{"slug":"p-values-and-what-they-do-not-say","title":"What a p-value is, and the four things it is not","url":"https://addaly.com/learn/evaluating-ai/p-values-and-what-they-do-not-say"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Asking what pure chance would have produced', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/simulating-the-null (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Asking what pure chance would have produced","courseTitle":"Knowing If It Works","module":{"index":3,"slug":"deciding-if-it-is-real","title":"Deciding whether a difference is real"},"answer":null,"keyIdea":"Shuffle which system produced which result ten thousand times and see how often chance beats your observed gap: one loop replaces a shelf of named tests and encodes your design honestly.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 3.1","figure":{"type":"flow","steps":[{"label":"Per-item scores","detail":"Both systems, the same items, the same order."},{"label":"Observed gap","detail":"Mean of B minus mean of A, say +0.050."},{"label":"Flip signs at random","detail":"Each item’s difference times ±1: the same as swapping which system is called A."},{"label":"Repeat 10,000 times","detail":"The spread of gaps that chance alone produces."},{"label":"Count the tail","detail":"The share of shuffles at least as large as the observed gap."}],"title":"A permutation test in five steps","caption":"For 200 items the null gaps spread about ±0.021 either way, so an observed +0.050 is beaten by roughly 1.5% of shuffles. That share is the p-value, with no table and no normality assumption."},"alt":"Figure: A permutation test in five steps. Steps: 1. Per-item scores (Both systems, the same items, the same order.); 2. Observed gap (Mean of B minus mean of A, say +0.050.); 3. Flip signs at random (Each item’s difference times ±1: the same as swapping which system is called A.); 4. Repeat 10,000 times (The spread of gaps that chance alone produces.); 5. Count the tail (The share of shuffles at least as large as the observed gap.). For 200 items the null gaps spread about ±0.021 either way, so an observed +0.050 is beaten by roughly 1.5% of shuffles. That share is the p-value, with no table and no normality assumption."}},"blocks":[{"t":"h2","id":"the-question-under-every-comparison","runs":[{"text":"The question under every comparison"}]},{"t":"p","runs":[{"text":"Prompt B scores 5 points above prompt A on the same 200 items. Before anything else, one question has to be answered: if the two prompts were identical in quality, how often would a gap that big appear anyway?"}]},{"t":"p","runs":[{"text":"There are two ways to answer it. You can look up the right named test, check its assumptions, and hope you chose correctly. Or you can build the world where the null is true and count. The second is a permutation test, it takes ten lines, and it is very hard to get wrong in a way you would not notice."}]},{"t":"h2","id":"the-engine","runs":[{"text":"The engine"}]},{"t":"p","runs":[{"text":"You have per-item results for both systems. Under the null hypothesis, the label saying which result came from A and which from B carries no information, so swapping them within an item should change nothing systematic. Do exactly that, many times."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nrng = np.random.default_rng(0)\n\na = np.array(scores_a)          # one score per item, same order\nb = np.array(scores_b)\nobserved = b.mean() - a.mean()  # e.g. +0.050\n\ndiffs = b - a\nnull = []\nfor _ in range(10_000):\n    signs = rng.choice([-1, 1], size=len(diffs))\n    null.append((diffs * signs).mean())\n\nnull = np.array(null)\np = (np.abs(null) >= abs(observed)).mean()\nprint(f\"gap {observed:+.3f}, p = {p:.3f}\")"},{"t":"p","runs":[{"text":"Flipping the sign of each item’s difference is the same as swapping which system is called A. Run it and you get a distribution of gaps that chance alone produces: for 200 items with this much spread, typically a standard deviation around 0.021. Against that, an observed +0.050 sits well out in the tail, and roughly 1.5% of shuffles exceed it. That is your p-value, computed from your own data, with no assumption of normality and no table."}]},{"t":"h2","id":"why-this-beats-remembering-tests","runs":[{"text":"Why this beats remembering tests"}]},{"t":"p","runs":[{"text":"The permutation approach works for any statistic you can compute. Median latency. Recall at 5. A weighted composite of three rubric checks. Cost per successful task. There is no closed-form test for most of these, and you do not need one: you only need to be able to compute the number and to know what the null would scramble."}]},{"t":"p","runs":[{"text":"It also forces you to state your design, which is the part that most often goes wrong. "},{"text":"What you shuffle is your claim about what is exchangeable.","b":true}]},{"t":"ul","items":[[{"text":"Paired evaluation, same items through both systems: flip signs within each item, as above."}],[{"text":"Two separate groups of users: shuffle the group labels across users."}],[{"text":"Items grouped by customer: shuffle at the customer level, keeping each customer’s rows together."}]]},{"t":"p","runs":[{"text":"Shuffling rows when the rows are grouped gives a confidently wrong answer, and the next lessons cover why. The virtue of writing the shuffle by hand is that the mistake becomes visible in code rather than hiding in a function’s assumptions."}]},{"t":"h2","id":"the-a-a-test-which-you-should-run-first","runs":[{"text":"The A/A test, which you should run first"}]},{"t":"p","runs":[{"text":"Before trusting any comparison machinery, point it at two copies of the same system. Run configuration A twice, feed both result sets into the test, and see what it says."}]},{"t":"p","runs":[{"text":"The result should be an unremarkable gap and a p-value scattered anywhere between 0 and 1. If A against A produces a “significant” difference, you have found a bug, and it is nearly always one of five:"}]},{"t":"ul","items":[[{"text":"The two runs used different subsets, because an item errored out in one of them and was silently dropped."}],[{"text":"Results were compared in the wrong order, so item 41’s A score sits against item 42’s B score."}],[{"text":"A cache served the first run and not the second."}],[{"text":"The set is ordered, and a truncated run covers a different mix."}],[{"text":"Non-determinism is larger than you assumed, which is not a bug but is a finding."}]]},{"t":"p","runs":[{"text":"An A/A test costs one extra run and it has caught bad harnesses at essentially every team that has bothered."}]},{"t":"h2","id":"what-it-cannot-do","runs":[{"text":"What it cannot do"}]},{"t":"p","runs":[{"text":"A simulation answers exactly the question you encoded, and nothing more."}]},{"t":"ul","items":[[{"text":"It cannot rescue a biased sample. If your 200 items over-represent one intent, permutation gives you a precise p-value about a set that does not resemble your traffic."}],[{"text":"It cannot tell you the effect is important. A gap can be reliably real and commercially worthless; report the size and its interval next to the p-value."}],[{"text":"It assumes the outcomes you fed it are comparable. Two runs at temperature 0.7, compared without accounting for run-to-run variance, will produce a small p for a difference that vanishes tomorrow."}]]},{"t":"p","runs":[{"text":"Ten lines, "},{"text":"numpy","code":true},{"text":", and a laptop. For small sets you can do the same thing in a spreadsheet with a column of random signs, and it is worth doing once by hand to see the null distribution appear."}]}],"check":{"question":[{"text":"A team runs a permutation test by shuffling all 600 result rows freely between the two systems, but the 600 rows come from 120 customers with 5 rows each. What does that shuffle get wrong?"}],"options":[{"text":[{"text":"Nothing, provided the number of rows assigned to each system stays equal after each shuffle."}],"correct":false,"tempting":[{"text":"Treats balance as the only requirement; the problem is not how many rows each side gets but which rows are treated as independently exchangeable."}]},{"text":[{"text":"It breaks the pairing, so the test loses power and will fail to detect a real difference."}],"correct":false,"tempting":[{"text":"Names a real cost of unpaired shuffling, but the more serious error here is in the opposite direction: the p-value comes out too small, not too large."}]},{"text":[{"text":"It treats correlated rows from one customer as independent, so the null distribution is narrower than reality and the p-value comes out too small."}],"correct":true,"tempting":null},{"text":[{"text":"It requires the two systems to have been run on identical inputs, which grouped data cannot guarantee."}],"correct":false,"tempting":[{"text":"Confuses a data-collection question with the exchangeability assumption the shuffle encodes; the inputs here are identical and the grouping is still the problem."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"the-benchmark-you-read-about","title":"Public benchmarks, and what they are for","minutes":8},"next":{"slug":"p-values-and-what-they-do-not-say","title":"What a p-value is, and the four things it is not","minutes":9}}}