{"course":{"slug":"ai-careers","title":"Working in AI","url":"https://addaly.com/learn/ai-careers","level":"beginner","track":"work"},"lesson":{"slug":"statistics-you-will-be-asked-about","title":"The statistics that come up, and the ones that never do","url":"https://addaly.com/learn/ai-careers/statistics-you-will-be-asked-about","position":24,"of":85,"minutes":10,"answer":"Precision improves only with the square root of sample size, so most business tests were never large enough to detect the effect they were run to find.","keyIdea":"Precision improves only with the square root of sample size, so most business tests were never large enough to detect the effect they were run to find."},"body":{"format":"markdown","text":"## A short list, learned properly\n\nInterviewers rarely ask you to derive anything. They ask whether you will notice that a number is meaningless. Six ideas cover almost every case, and each has an arithmetic form you can do on paper.\n\n## 1. Uncertainty in a proportion\n\nYou measure a conversion rate of 12% on 400 visitors. How much could that move by chance alone?\n\n```\nstandard error = sqrt( p * (1 - p) / n )\n              = sqrt( 0.12 * 0.88 / 400 ) = 0.0162\n95% interval  ≈ p ± 2 * SE = 12% ± 3.2%\n```\n\nSo the true rate is somewhere around 8.8% to 15.2%. If someone reports that the rate \"rose from 11% to 12%\", nothing has been observed.\n\nThe consequence worth memorising: precision improves with the **square root** of sample size. Four times the data halves the error. That is why a survey of 100 people has a margin of roughly ±10 points and why \"we asked 40 users\" is almost never evidence of a small effect.\n\n> Figure: Four times the visitors, half the error. Line chart of Plus or minus, in points against Visitors measured: Measured rate near 50%, the widest case (100, 10) (400, 5) (1600, 2.5) (6400, 1.25); Measured rate near 12%, the lesson's example (100, 6.5) (400, 3.2) (1600, 1.6) (6400, 0.8). The 95 per cent interval around a measured proportion shrinks with the square root of sample size. From 400 to 1,600 visitors halves it; a survey of 100 people is roughly plus or minus ten points, which is why forty users is never evidence of a small effect.\n\n## 2. Power, and why most tests cannot see what they are looking for\n\nBefore running a test, ask what size of difference it could detect. A serviceable rule for 80% power at the usual 5% threshold:\n\n```\nn per group ≈ 16 * p * (1 - p) / (difference)^2\n```\n\nBaseline conversion 5%, hoping for a 10% relative lift (5.0% → 5.5%, a difference of 0.005):\n\n```\nn ≈ 16 * 0.05 * 0.95 / 0.000025 ≈ 30,400 per group\n```\n\nNow the same baseline with a 1% relative lift (difference 0.0005): about **3 million per group**. Most companies do not have that traffic, which means most of their tests were never capable of detecting the effect they were run to find. Saying this politely, early, before the test runs, is one of the more valuable things a junior analyst can do.\n\n## 3. Simpson's paradox\n\nA pattern in every subgroup can reverse when the groups are combined. Two treatments for kidney stones, real published shape:\n\n- Small stones: treatment A cures 81 of 87 (93%); treatment B cures 234 of 270 (87%).\n- Large stones: A cures 192 of 263 (73%); B cures 55 of 80 (69%).\n- Combined: A cures 273 of 350 (78%); B cures 289 of 350 (83%).\n\nA wins both groups and loses overall, because doctors sent the hard cases to A. The lesson is not that aggregates lie. It is that a comparison is only meaningful once you know what determined who ended up in which group.\n\n## 4. Regression to the mean\n\nTake your worst-performing shops, intervene, and measure again. They improve. So would they have without you, because part of being the worst last month was bad luck, and luck does not repeat.\n\nAny evaluation that starts by selecting the extreme of a distribution needs a control group selected the same way. Without one, you will report an effect that is partly an artefact of the selection, and the size of that artefact is not small.\n\n## 5. Multiple comparisons\n\nSlice a result by twenty segments at the conventional 5% threshold and you should expect about one \"significant\" finding from pure noise. Slice by a hundred and you expect five.\n\nThis is how dashboards produce a weekly false discovery. The fix is not sophisticated: decide the segments before you look, or state plainly that a post-hoc slice is a hypothesis rather than a finding.\n\n## 6. Correlation, causation, and the third thing\n\nIce cream sales and drowning both rise in summer. The useful habit is not reciting the slogan but naming the candidate confounder out loud. When you cannot name one, say so; when you can, the conversation improves immediately.\n\nThe three routes to a causal claim, in descending order of strength: a randomised experiment; a natural experiment where something outside the system changed for reasons unrelated to the outcome; and adjustment for measured confounders, which only handles the ones you thought of.\n\n## What is not asked\n\nDerivations. Obscure distributions. Proofs. In seven years of ordinary data work you may never use a t-test by hand, and no interviewer will ask you to integrate anything.\n\n## Free ways to learn this properly\n\n*OpenIntro Statistics* is a full university textbook, free as a PDF, and it is better than most paid courses. Seeing Theory (Brown University) animates every idea above in a browser. For computation, `scipy.stats` and `statsmodels` in Python are free, R is free, and a spreadsheet handles most of it. If you want the mathematics behind the arithmetic, MIT OpenCourseWare publishes complete statistics courses at no cost.\n\n## The limitation\n\nStatistical significance is not importance. A test on ten million users can return a real, robust, thoroughly significant improvement of 0.02% that is worth nothing and costs a month of engineering. The number that matters is the effect size and what it is worth in currency; the p-value only tells you whether you are looking at noise. Report both, always in that order."},"check":{"question":"A team wants to test whether a new checkout screen improves conversion. Baseline conversion is 4%, they expect roughly a 5% relative improvement, and the site gets 9,000 visitors a week. What should be said before the test starts?","options":[{"text":"Run it for four weeks; 36,000 visitors is a substantial sample and will settle the question.","correct":false,"tempting":"Judges sample size by whether it feels large rather than against the difference being sought, which is the mistake the power calculation exists to prevent."},{"text":"Detecting a difference of 0.2 percentage points at this baseline needs roughly 150,000 visitors per arm, so at this traffic the test would run for years: the honest options are to seek a larger change, use a more sensitive metric, or accept the decision cannot be made this way.","correct":true},{"text":"Use a one-sided test to increase sensitivity, which roughly halves the required sample.","correct":false,"tempting":"Buys a small factor by narrowing the question, leaving the test short by more than an order of magnitude."},{"text":"Skip the test and ship it, since a 5% relative lift is small enough that the risk is low either way.","correct":false,"tempting":"Reaches a defensible action through the wrong reasoning, and gives up the chance to state what evidence would have been needed."}]},"prev":{"slug":"python-that-earns","title":"The employable half of Python","url":"https://addaly.com/learn/ai-careers/python-that-earns"},"next":{"slug":"version-control-in-public","title":"Git, and what a reviewer sees before your code","url":"https://addaly.com/learn/ai-careers/version-control-in-public"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The statistics that come up, and the ones that never do', in Working in AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-careers/statistics-you-will-be-asked-about (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The statistics that come up, and the ones that never do","courseTitle":"Working in AI","module":{"index":3,"slug":"the-evidence","title":"The evidence: what gets tested, and how to prove you have it"},"answer":null,"keyIdea":"Precision improves only with the square root of sample size, so most business tests were never large enough to detect the effect they were run to find.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Four times the visitors, half the error","series":[{"label":"Measured rate near 50%, the widest case","points":[[100,10],[400,5],[1600,2.5],[6400,1.25]]},{"label":"Measured rate near 12%, the lesson’s example","points":[[100,6.5],[400,3.2],[1600,1.6],[6400,0.8]]}],"xLabel":"Visitors measured","yLabel":"Plus or minus, in points","caption":"The 95 per cent interval around a measured proportion shrinks with the square root of sample size. From 400 to 1,600 visitors halves it; a survey of 100 people is roughly plus or minus ten points, which is why forty users is never evidence of a small effect."},"alt":"Figure: Four times the visitors, half the error. Line chart of Plus or minus, in points against Visitors measured: Measured rate near 50%, the widest case (100, 10) (400, 5) (1600, 2.5) (6400, 1.25); Measured rate near 12%, the lesson’s example (100, 6.5) (400, 3.2) (1600, 1.6) (6400, 0.8). The 95 per cent interval around a measured proportion shrinks with the square root of sample size. From 400 to 1,600 visitors halves it; a survey of 100 people is roughly plus or minus ten points, which is why forty users is never evidence of a small effect."}},"blocks":[{"t":"h2","id":"a-short-list-learned-properly","runs":[{"text":"A short list, learned properly"}]},{"t":"p","runs":[{"text":"Interviewers rarely ask you to derive anything. They ask whether you will notice that a number is meaningless. Six ideas cover almost every case, and each has an arithmetic form you can do on paper."}]},{"t":"h2","id":"1-uncertainty-in-a-proportion","runs":[{"text":"1. Uncertainty in a proportion"}]},{"t":"p","runs":[{"text":"You measure a conversion rate of 12% on 400 visitors. How much could that move by chance alone?"}]},{"t":"code","lang":"","label":"Text","text":"standard error = sqrt( p * (1 - p) / n )\n              = sqrt( 0.12 * 0.88 / 400 ) = 0.0162\n95% interval  ≈ p ± 2 * SE = 12% ± 3.2%"},{"t":"p","runs":[{"text":"So the true rate is somewhere around 8.8% to 15.2%. If someone reports that the rate “rose from 11% to 12%”, nothing has been observed."}]},{"t":"p","runs":[{"text":"The consequence worth memorising: precision improves with the "},{"text":"square root","b":true},{"text":" of sample size. Four times the data halves the error. That is why a survey of 100 people has a margin of roughly ±10 points and why “we asked 40 users” is almost never evidence of a small effect."}]},{"t":"h2","id":"2-power-and-why-most-tests-cannot-see-what-they-are-looking-","runs":[{"text":"2. Power, and why most tests cannot see what they are looking for"}]},{"t":"p","runs":[{"text":"Before running a test, ask what size of difference it could detect. A serviceable rule for 80% power at the usual 5% threshold:"}]},{"t":"code","lang":"","label":"Text","text":"n per group ≈ 16 * p * (1 - p) / (difference)^2"},{"t":"p","runs":[{"text":"Baseline conversion 5%, hoping for a 10% relative lift (5.0% → 5.5%, a difference of 0.005):"}]},{"t":"code","lang":"","label":"Text","text":"n ≈ 16 * 0.05 * 0.95 / 0.000025 ≈ 30,400 per group"},{"t":"p","runs":[{"text":"Now the same baseline with a 1% relative lift (difference 0.0005): about "},{"text":"3 million per group","b":true},{"text":". Most companies do not have that traffic, which means most of their tests were never capable of detecting the effect they were run to find. Saying this politely, early, before the test runs, is one of the more valuable things a junior analyst can do."}]},{"t":"h2","id":"3-simpson-s-paradox","runs":[{"text":"3. Simpson’s paradox"}]},{"t":"p","runs":[{"text":"A pattern in every subgroup can reverse when the groups are combined. Two treatments for kidney stones, real published shape:"}]},{"t":"ul","items":[[{"text":"Small stones: treatment A cures 81 of 87 (93%); treatment B cures 234 of 270 (87%)."}],[{"text":"Large stones: A cures 192 of 263 (73%); B cures 55 of 80 (69%)."}],[{"text":"Combined: A cures 273 of 350 (78%); B cures 289 of 350 (83%)."}]]},{"t":"p","runs":[{"text":"A wins both groups and loses overall, because doctors sent the hard cases to A. The lesson is not that aggregates lie. It is that a comparison is only meaningful once you know what determined who ended up in which group."}]},{"t":"h2","id":"4-regression-to-the-mean","runs":[{"text":"4. Regression to the mean"}]},{"t":"p","runs":[{"text":"Take your worst-performing shops, intervene, and measure again. They improve. So would they have without you, because part of being the worst last month was bad luck, and luck does not repeat."}]},{"t":"p","runs":[{"text":"Any evaluation that starts by selecting the extreme of a distribution needs a control group selected the same way. Without one, you will report an effect that is partly an artefact of the selection, and the size of that artefact is not small."}]},{"t":"h2","id":"5-multiple-comparisons","runs":[{"text":"5. Multiple comparisons"}]},{"t":"p","runs":[{"text":"Slice a result by twenty segments at the conventional 5% threshold and you should expect about one “significant” finding from pure noise. Slice by a hundred and you expect five."}]},{"t":"p","runs":[{"text":"This is how dashboards produce a weekly false discovery. The fix is not sophisticated: decide the segments before you look, or state plainly that a post-hoc slice is a hypothesis rather than a finding."}]},{"t":"h2","id":"6-correlation-causation-and-the-third-thing","runs":[{"text":"6. Correlation, causation, and the third thing"}]},{"t":"p","runs":[{"text":"Ice cream sales and drowning both rise in summer. The useful habit is not reciting the slogan but naming the candidate confounder out loud. When you cannot name one, say so; when you can, the conversation improves immediately."}]},{"t":"p","runs":[{"text":"The three routes to a causal claim, in descending order of strength: a randomised experiment; a natural experiment where something outside the system changed for reasons unrelated to the outcome; and adjustment for measured confounders, which only handles the ones you thought of."}]},{"t":"h2","id":"what-is-not-asked","runs":[{"text":"What is not asked"}]},{"t":"p","runs":[{"text":"Derivations. Obscure distributions. Proofs. In seven years of ordinary data work you may never use a t-test by hand, and no interviewer will ask you to integrate anything."}]},{"t":"h2","id":"free-ways-to-learn-this-properly","runs":[{"text":"Free ways to learn this properly"}]},{"t":"p","runs":[{"text":"OpenIntro Statistics","i":true},{"text":" is a full university textbook, free as a PDF, and it is better than most paid courses. Seeing Theory (Brown University) animates every idea above in a browser. For computation, "},{"text":"scipy.stats","code":true},{"text":" and "},{"text":"statsmodels","code":true},{"text":" in Python are free, R is free, and a spreadsheet handles most of it. If you want the mathematics behind the arithmetic, MIT OpenCourseWare publishes complete statistics courses at no cost."}]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"Statistical significance is not importance. A test on ten million users can return a real, robust, thoroughly significant improvement of 0.02% that is worth nothing and costs a month of engineering. The number that matters is the effect size and what it is worth in currency; the p-value only tells you whether you are looking at noise. Report both, always in that order."}]}],"check":{"question":[{"text":"A team wants to test whether a new checkout screen improves conversion. Baseline conversion is 4%, they expect roughly a 5% relative improvement, and the site gets 9,000 visitors a week. What should be said before the test starts?"}],"options":[{"text":[{"text":"Run it for four weeks; 36,000 visitors is a substantial sample and will settle the question."}],"correct":false,"tempting":[{"text":"Judges sample size by whether it feels large rather than against the difference being sought, which is the mistake the power calculation exists to prevent."}]},{"text":[{"text":"Detecting a difference of 0.2 percentage points at this baseline needs roughly 150,000 visitors per arm, so at this traffic the test would run for years: the honest options are to seek a larger change, use a more sensitive metric, or accept the decision cannot be made this way."}],"correct":true,"tempting":null},{"text":[{"text":"Use a one-sided test to increase sensitivity, which roughly halves the required sample."}],"correct":false,"tempting":[{"text":"Buys a small factor by narrowing the question, leaving the test short by more than an order of magnitude."}]},{"text":[{"text":"Skip the test and ship it, since a 5% relative lift is small enough that the risk is low either way."}],"correct":false,"tempting":[{"text":"Reaches a defensible action through the wrong reasoning, and gives up the chance to state what evidence would have been needed."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"python-that-earns","title":"The employable half of Python","minutes":9},"next":{"slug":"version-control-in-public","title":"Git, and what a reviewer sees before your code","minutes":8}}}