Zero failures in 300 tries, and what that does not prove

The most common wrong sentence in a release note

"We ran it 300 times and it never failed." The sentence is true, and the conclusion people draw from it, that the failure rate is zero or near enough, is not. Zero observed failures is evidence, and the question is how much. The answer fits on one line.

Deriving the rule

Suppose the true failure rate is p and the trials are independent. The chance of seeing no failures in n trials is (1 − p)^n. Ask: what is the largest p for which zero failures would still be unsurprising, say at least 5 per cent likely?

(1 − p)^n = 0.05
n × ln(1 − p) = ln 0.05 = −3.0
n × (−p) ≈ −3.0          (since ln(1 − p) ≈ −p for small p)
p ≈ 3 / n

That is the rule of three: after n trials with no failures, the 95 per cent upper bound on the failure rate is about 3/n.

0 failures in     30  →  rate could be up to 10%
0 failures in    300  →  up to 1%
0 failures in  3,000  →  up to 0.1%
0 failures in 30,000  →  up to 0.01%

Three hundred clean runs are consistent with a one-in-a-hundred failure rate. If the system will handle a thousand requests a day, that is ten failures a day the test could not have seen. Fifty clean runs, which is what a demo usually amounts to, are consistent with six per cent.

The other direction: how many trials to see one

The same algebra answers the planning question. To have a 95 per cent chance of observing at least one failure of rate p, you need n ≈ 3/p trials. A failure that happens once in a thousand requests needs three thousand trials to show up reliably. One that happens once in a hundred thousand needs three hundred thousand. If your test budget is a thousand runs, you are blind to anything rarer than about one in three hundred, and you should say so rather than report a zero.

Seeing a few failures

Once failures do appear, the count over many trials is approximately Poisson, and the 95 per cent upper bounds on the expected count are:

observed 0  →  at most 3.0
observed 1  →  at most 4.7
observed 2  →  at most 6.3
observed 3  →  at most 7.8

Divide by n for the rate. Two failures in 1,000 runs means the rate is 0.2 per cent as a point estimate but could be as high as 0.63 per cent. Note how slowly the bound tightens: the upper bound for three observed failures is not much more than double that for zero. A handful of failures is a handful of information.

Estimating a rare rate to a given precision

Seeing a failure is one thing; measuring its rate is another. The standard error of a proportion, relative to the proportion itself, is about 1/√(np) when p is small. To pin a rate to within ±50 per cent of itself, two standard errors, you need 1/√(np) = 0.25, so np = 16: sixteen observed failures, whatever the rate. To within ±10 per cent you need np = 400, four hundred failures. A failure rate of 0.1 per cent therefore needs 16,000 trials to estimate coarsely and 400,000 to estimate well. This is why rare-event rates in published evaluations are so often quoted with one decimal place and no interval: the interval would swallow the number.

Where the rule is used, and misused

• Safety evaluations. "No jailbreak in 200 attempts" bounds the success rate for attacks of that kind at about 1.5 per cent. It says nothing about attacks the red team did not think of, which is a different limitation, covered in evaluating-ai.
• Clinical trials. A drug tested on 1,500 patients with no case of a side effect cannot rule out that it strikes one patient in 500. This is the standard textbook example, and it is why rare side effects are found after approval.
• Hardware and infrastructure. A component with no failures across 10,000 hours of testing may fail once per 3,300 hours.

The misuse is always the same: treating the point estimate of zero as the answer, when the honest answer is a bound.

Two assumptions to check

The trials must be independent, and they must be drawn from the deployment distribution. Three hundred test prompts written by one engineer in one afternoon are neither: they share that engineer's blind spots, and they were not sampled from what users will send. The rule of three then bounds the failure rate on prompts like those, which is a smaller claim than it sounds. The arithmetic is exact; the sample is where the honesty goes.

The sentence to write instead

Not "it never failed", but "in 300 independent trials we saw no failures, which bounds the rate below about 1 per cent at 95 per cent confidence for inputs of this kind." Longer, and true.