{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"p-values-and-what-they-do-not-say","title":"What a p-value is, and the four things it is not","url":"https://addaly.com/learn/evaluating-ai/p-values-and-what-they-do-not-say","position":22,"of":76,"minutes":9,"answer":"A p-value is the chance of seeing a gap this large if there were no effect. It is not the probability you are right.","keyIdea":"A p-value is the chance of seeing a gap this large if there were no effect. It is not the probability you are right, and when few of your ideas work most of your significant results are still wrong."},"body":{"format":"markdown","text":"## The definition, stated carefully\n\nA p-value is the probability of observing a result at least as extreme as the one you got, **assuming the null hypothesis is true**. That is the entire content. Everything difficult about p-values comes from that conditional clause, which people drop within seconds of reading it.\n\nFour things it is not:\n\n- **Not the probability that the null is true.** A p of 0.03 does not mean a 3% chance the systems are equal, nor a 97% chance B is better. Getting from one to the other needs a prior, which the p-value does not contain.\n- **Not a measure of size.** A tiny p can accompany a difference of 0.2 points.\n- **Not a measure of importance.** Those are different words for a reason.\n- **Not repeatable.** A p-value is itself a random quantity. Rerun a genuine experiment with 50% power and the p bounces between 0.001 and 0.4 across replications. Treating 0.049 and 0.051 as different states of the world is a category error.\n\n## The arithmetic that changes how you ship\n\nHere is the calculation that matters most for a team pushing prompt changes, and it is rarely done.\n\nSuppose you try 100 prompt variations over a quarter. Realistically, perhaps 10 of them help. Your tests use the usual 5% threshold, and with your set size you have about 50% power to detect the effects you care about.\n\n- Of the 10 true improvements, you detect **5**.\n- Of the 90 changes that do nothing, 5% come out significant anyway: **4.5** false alarms.\n\nSo you finish the quarter with about 9.5 \"wins\", of which nearly half are noise, and you shipped all of them. Nothing was done incorrectly; each individual test behaved exactly as advertised. The false-discovery rate depends on how often your ideas are right, and nobody's ideas are right most of the time.\n\nThree consequences follow:\n\n1. **Raise power before lowering alpha.** Going from 50% to 80% power turns 5 true detections into 8 and leaves the 4.5 false alarms unchanged.\n2. **Replicate anything surprising.** A second, independent run on fresh items costs a day and removes most of the false alarms, because a fluke rarely repeats.\n3. **Prefer changes with a mechanism.** A change you can explain is drawn from a population with a much higher true-positive rate than a change found by trying things.\n\n## Significant and worthless\n\nOnline, the opposite failure dominates. With 200,000 sessions, a difference of 0.15 percentage points in click-through will be significant at any threshold you like. The correct response is not to celebrate; it is to ask whether 0.15 points is worth the maintenance cost of the change, the extra latency, and the new failure mode.\n\nThis is why the pre-registered decision threshold from earlier in the course exists. Write down before the experiment: \"we will ship if the improvement is at least 2 points, and not otherwise\". Then significance becomes a check on whether you can see 2 points, rather than the decision itself.\n\n## Non-significant does not mean no difference\n\nThe most consequential misreading in practice. \"p = 0.31, so the models are the same\" is wrong; the correct statement is that the experiment could not distinguish them. Report the interval instead:\n\n> B is 2 points better (95% CI −3 to +7). We can rule out a regression larger than 3 points and an improvement larger than 7.\n\nThat sentence supports a decision. \"Not significant\" does not. If the interval is so wide that both a serious regression and a large gain are inside it, the honest conclusion is that you ran the wrong-sized experiment, and the next lesson is about avoiding that.\n\n## Where 0.05 came from\n\nFisher suggested it in the 1920s as a convenient convention and said explicitly that no fixed level should be used for all purposes. It has no mathematical standing. Choose from consequences instead:\n\n- Changing a prompt on a low-risk internal tool: 0.10 is fine; the cost of a wrong call is a revert.\n- Changing a safety threshold: use a much stricter level, several independent runs, and human review regardless of the number.\n\nState the level you chose, before you look, and do not change it afterwards. Moving the threshold after seeing the result is how a decision procedure quietly becomes a story."},"check":{"question":"A team runs 100 prompt experiments in a quarter at the 5% level with roughly 50% power, and about one in ten of their ideas genuinely helps. Roughly what share of their significant results are false alarms?","options":[{"text":"About 5%, since that is the significance level they chose.","correct":false,"tempting":"Reads alpha as the false-discovery rate; 5% is the error rate among the ideas that do nothing, not among the results that come out significant."},{"text":"About 50%, because half the true effects are missed and the missed half is replaced by false positives.","correct":false,"tempting":"Lands near the right answer by the wrong route: the halving comes from power applied to true effects, while the false alarms come from applying 5% to the 90 null experiments."},{"text":"Roughly 4.5 false alarms against 5 true detections, so nearly half.","correct":true},{"text":"None, provided each experiment was correctly designed and pre-registered.","correct":false,"tempting":"Assumes correct procedure removes false positives; a 5% error rate applied 90 times produces false positives no matter how carefully each test is run."}]},"prev":{"slug":"simulating-the-null","title":"Asking what pure chance would have produced","url":"https://addaly.com/learn/evaluating-ai/simulating-the-null"},"next":{"slug":"how-many-items-do-you-need","title":"Sizing the set before you run it","url":"https://addaly.com/learn/evaluating-ai/how-many-items-do-you-need"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What a p-value is, and the four things it is not', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/p-values-and-what-they-do-not-say (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What a p-value is, and the four things it is not","courseTitle":"Knowing If It Works","module":{"index":3,"slug":"deciding-if-it-is-real","title":"Deciding whether a difference is real"},"answer":null,"keyIdea":"A p-value is the chance of seeing a gap this large if there were no effect. It is not the probability you are right, and when few of your ideas work most of your significant results are still wrong.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-definition-stated-carefully","runs":[{"text":"The definition, stated carefully"}]},{"t":"p","runs":[{"text":"A p-value is the probability of observing a result at least as extreme as the one you got, "},{"text":"assuming the null hypothesis is true","b":true},{"text":". That is the entire content. Everything difficult about p-values comes from that conditional clause, which people drop within seconds of reading it."}]},{"t":"p","runs":[{"text":"Four things it is not:"}]},{"t":"ul","items":[[{"text":"Not the probability that the null is true.","b":true},{"text":" A p of 0.03 does not mean a 3% chance the systems are equal, nor a 97% chance B is better. Getting from one to the other needs a prior, which the p-value does not contain."}],[{"text":"Not a measure of size.","b":true},{"text":" A tiny p can accompany a difference of 0.2 points."}],[{"text":"Not a measure of importance.","b":true},{"text":" Those are different words for a reason."}],[{"text":"Not repeatable.","b":true},{"text":" A p-value is itself a random quantity. Rerun a genuine experiment with 50% power and the p bounces between 0.001 and 0.4 across replications. Treating 0.049 and 0.051 as different states of the world is a category error."}]]},{"t":"h2","id":"the-arithmetic-that-changes-how-you-ship","runs":[{"text":"The arithmetic that changes how you ship"}]},{"t":"p","runs":[{"text":"Here is the calculation that matters most for a team pushing prompt changes, and it is rarely done."}]},{"t":"p","runs":[{"text":"Suppose you try 100 prompt variations over a quarter. Realistically, perhaps 10 of them help. Your tests use the usual 5% threshold, and with your set size you have about 50% power to detect the effects you care about."}]},{"t":"ul","items":[[{"text":"Of the 10 true improvements, you detect "},{"text":"5","b":true},{"text":"."}],[{"text":"Of the 90 changes that do nothing, 5% come out significant anyway: "},{"text":"4.5","b":true},{"text":" false alarms."}]]},{"t":"p","runs":[{"text":"So you finish the quarter with about 9.5 “wins”, of which nearly half are noise, and you shipped all of them. Nothing was done incorrectly; each individual test behaved exactly as advertised. The false-discovery rate depends on how often your ideas are right, and nobody’s ideas are right most of the time."}]},{"t":"p","runs":[{"text":"Three consequences follow:"}]},{"t":"ol","start":1,"items":[[{"text":"Raise power before lowering alpha.","b":true},{"text":" Going from 50% to 80% power turns 5 true detections into 8 and leaves the 4.5 false alarms unchanged."}],[{"text":"Replicate anything surprising.","b":true},{"text":" A second, independent run on fresh items costs a day and removes most of the false alarms, because a fluke rarely repeats."}],[{"text":"Prefer changes with a mechanism.","b":true},{"text":" A change you can explain is drawn from a population with a much higher true-positive rate than a change found by trying things."}]]},{"t":"h2","id":"significant-and-worthless","runs":[{"text":"Significant and worthless"}]},{"t":"p","runs":[{"text":"Online, the opposite failure dominates. With 200,000 sessions, a difference of 0.15 percentage points in click-through will be significant at any threshold you like. The correct response is not to celebrate; it is to ask whether 0.15 points is worth the maintenance cost of the change, the extra latency, and the new failure mode."}]},{"t":"p","runs":[{"text":"This is why the pre-registered decision threshold from earlier in the course exists. Write down before the experiment: “we will ship if the improvement is at least 2 points, and not otherwise”. Then significance becomes a check on whether you can see 2 points, rather than the decision itself."}]},{"t":"h2","id":"non-significant-does-not-mean-no-difference","runs":[{"text":"Non-significant does not mean no difference"}]},{"t":"p","runs":[{"text":"The most consequential misreading in practice. “p = 0.31, so the models are the same” is wrong; the correct statement is that the experiment could not distinguish them. Report the interval instead:"}]},{"t":"quote","runs":[{"text":"B is 2 points better (95% CI −3 to +7). We can rule out a regression larger than 3 points and an improvement larger than 7."}]},{"t":"p","runs":[{"text":"That sentence supports a decision. “Not significant” does not. If the interval is so wide that both a serious regression and a large gain are inside it, the honest conclusion is that you ran the wrong-sized experiment, and the next lesson is about avoiding that."}]},{"t":"h2","id":"where-0-05-came-from","runs":[{"text":"Where 0.05 came from"}]},{"t":"p","runs":[{"text":"Fisher suggested it in the 1920s as a convenient convention and said explicitly that no fixed level should be used for all purposes. It has no mathematical standing. Choose from consequences instead:"}]},{"t":"ul","items":[[{"text":"Changing a prompt on a low-risk internal tool: 0.10 is fine; the cost of a wrong call is a revert."}],[{"text":"Changing a safety threshold: use a much stricter level, several independent runs, and human review regardless of the number."}]]},{"t":"p","runs":[{"text":"State the level you chose, before you look, and do not change it afterwards. Moving the threshold after seeing the result is how a decision procedure quietly becomes a story."}]}],"check":{"question":[{"text":"A team runs 100 prompt experiments in a quarter at the 5% level with roughly 50% power, and about one in ten of their ideas genuinely helps. Roughly what share of their significant results are false alarms?"}],"options":[{"text":[{"text":"About 5%, since that is the significance level they chose."}],"correct":false,"tempting":[{"text":"Reads alpha as the false-discovery rate; 5% is the error rate among the ideas that do nothing, not among the results that come out significant."}]},{"text":[{"text":"About 50%, because half the true effects are missed and the missed half is replaced by false positives."}],"correct":false,"tempting":[{"text":"Lands near the right answer by the wrong route: the halving comes from power applied to true effects, while the false alarms come from applying 5% to the 90 null experiments."}]},{"text":[{"text":"Roughly 4.5 false alarms against 5 true detections, so nearly half."}],"correct":true,"tempting":null},{"text":[{"text":"None, provided each experiment was correctly designed and pre-registered."}],"correct":false,"tempting":[{"text":"Assumes correct procedure removes false positives; a 5% error rate applied 90 times produces false positives no matter how carefully each test is run."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"simulating-the-null","title":"Asking what pure chance would have produced","minutes":9},"next":{"slug":"how-many-items-do-you-need","title":"Sizing the set before you run it","minutes":9}}}