The statistics that come up, and the ones that never do

A short list, learned properly

Interviewers rarely ask you to derive anything. They ask whether you will notice that a number is meaningless. Six ideas cover almost every case, and each has an arithmetic form you can do on paper.

1. Uncertainty in a proportion

You measure a conversion rate of 12% on 400 visitors. How much could that move by chance alone?

standard error = sqrt( p * (1 - p) / n )
              = sqrt( 0.12 * 0.88 / 400 ) = 0.0162
95% interval  ≈ p ± 2 * SE = 12% ± 3.2%

So the true rate is somewhere around 8.8% to 15.2%. If someone reports that the rate “rose from 11% to 12%”, nothing has been observed.

The consequence worth memorising: precision improves with the square root of sample size. Four times the data halves the error. That is why a survey of 100 people has a margin of roughly ±10 points and why “we asked 40 users” is almost never evidence of a small effect.

2. Power, and why most tests cannot see what they are looking for

Before running a test, ask what size of difference it could detect. A serviceable rule for 80% power at the usual 5% threshold:

n per group ≈ 16 * p * (1 - p) / (difference)^2

Baseline conversion 5%, hoping for a 10% relative lift (5.0% → 5.5%, a difference of 0.005):

n ≈ 16 * 0.05 * 0.95 / 0.000025 ≈ 30,400 per group

Now the same baseline with a 1% relative lift (difference 0.0005): about 3 million per group. Most companies do not have that traffic, which means most of their tests were never capable of detecting the effect they were run to find. Saying this politely, early, before the test runs, is one of the more valuable things a junior analyst can do.

3. Simpson’s paradox

A pattern in every subgroup can reverse when the groups are combined. Two treatments for kidney stones, real published shape:

• Small stones: treatment A cures 81 of 87 (93%); treatment B cures 234 of 270 (87%).
• Large stones: A cures 192 of 263 (73%); B cures 55 of 80 (69%).
• Combined: A cures 273 of 350 (78%); B cures 289 of 350 (83%).

A wins both groups and loses overall, because doctors sent the hard cases to A. The lesson is not that aggregates lie. It is that a comparison is only meaningful once you know what determined who ended up in which group.

4. Regression to the mean

Take your worst-performing shops, intervene, and measure again. They improve. So would they have without you, because part of being the worst last month was bad luck, and luck does not repeat.

Any evaluation that starts by selecting the extreme of a distribution needs a control group selected the same way. Without one, you will report an effect that is partly an artefact of the selection, and the size of that artefact is not small.

5. Multiple comparisons

Slice a result by twenty segments at the conventional 5% threshold and you should expect about one “significant” finding from pure noise. Slice by a hundred and you expect five.

This is how dashboards produce a weekly false discovery. The fix is not sophisticated: decide the segments before you look, or state plainly that a post-hoc slice is a hypothesis rather than a finding.

6. Correlation, causation, and the third thing

Ice cream sales and drowning both rise in summer. The useful habit is not reciting the slogan but naming the candidate confounder out loud. When you cannot name one, say so; when you can, the conversation improves immediately.

The three routes to a causal claim, in descending order of strength: a randomised experiment; a natural experiment where something outside the system changed for reasons unrelated to the outcome; and adjustment for measured confounders, which only handles the ones you thought of.

What is not asked

Derivations. Obscure distributions. Proofs. In seven years of ordinary data work you may never use a t-test by hand, and no interviewer will ask you to integrate anything.

Free ways to learn this properly

OpenIntro Statistics is a full university textbook, free as a PDF, and it is better than most paid courses. Seeing Theory (Brown University) animates every idea above in a browser. For computation, scipy.stats and statsmodels in Python are free, R is free, and a spreadsheet handles most of it. If you want the mathematics behind the arithmetic, MIT OpenCourseWare publishes complete statistics courses at no cost.

The limitation

Statistical significance is not importance. A test on ten million users can return a real, robust, thoroughly significant improvement of 0.02% that is worth nothing and costs a month of engineering. The number that matters is the effect size and what it is worth in currency; the p-value only tells you whether you are looking at noise. Report both, always in that order.