A suite that lies intermittently

The arithmetic that explains the despair

A flaky test is one that passes and fails on identical code. One of them is an annoyance. The problem is that they compound, and the compounding is multiplicative.

Suppose each test in a 500-test suite has a 0.1% chance of failing spuriously. The probability that a whole run is clean is 0.999^500, which is about 61%. So two runs in five are red for no reason.

At 0.5% per test, 0.995^500 is about 8%. Nine runs in ten are red on correct code.

That is the mechanism behind the thing every engineer has watched happen: a team reaches the point where a red build carries no information, and the rational response (re-run it) becomes the norm. From then on the suite has stopped being a gate. It is a delay with a random duration.

Where the flakes come from, in order of frequency

Time. datetime.now() in the code or the test. A test that computes “30 days from today” and fails when the month boundary lands wrong. Anything with sleep() in it, which is a guess about how long something takes, and guesses fail on a loaded CI machine.

Order dependence. Test A leaves a row behind and test B passes only because of it. Runs green in file order, red under pytest -p randomly or a parallel runner. The fix is to run randomised locally on purpose so you find it before CI does.

Shared state. A module-level cache, a singleton, an environment variable, a temporary file with a fixed name, a database not rolled back between tests, a port already in use.

The network. Any test that reaches a real service is flaky by construction, because the internet is. This includes package registries during the build.

Concurrency. The hard category. A race that manifests once in two hundred runs is telling you about a race in the production system, and it deserves investigation rather than a retry.

Why generated tests flake more

This is worth naming specifically, because it is a real and consistent pattern.

Generated tests reach for sleep() to wait for asynchronous work, because that is what appears most in the world’s example code. They use the real clock rather than an injected one. They assume dictionary and set iteration order. They construct fixtures with fixed identifiers that collide under parallel execution. And they are produced in volume, so a suite of 300 generated tests carries more of these patterns than a suite of 40 written ones, and each one is a lottery ticket.

The review check is simple and worth automating: grep new test files for sleep(, now(), random, localhost: and hard-coded ports, and ask about every hit.

A policy that works

The key move is that a flaky test must be removed from the gate immediately and fixed on a deadline, and those are two separate decisions. Teams that only do the first accumulate a quarantine nobody empties. Teams that only do the second leave a broken gate in place for a week.

1. Detect automatically. Re-run the suite on the merge commit nightly. Any test that has both passed and failed on identical code in the last thirty runs is flaky, by definition, and no argument is needed.
2. Quarantine on detection. Mark it, exclude it from the required check, and open a ticket with the failure output attached.
3. Cap the quarantine. A fixed number (ten is a common choice) and when it is full, the next flaky test cannot be quarantined until one is fixed or deleted. Without a cap, quarantine is a graveyard.
4. Set a deadline. Two weeks. After that the test is deleted and its ticket becomes “write a reliable test for this behaviour”. Deleting a test feels wrong and is correct: an unreliable test is worse than none, because it consumes attention and provides no signal.
5. Never add a blanket retry. --reruns 3 at the suite level hides every flake including the ones that are real concurrency bugs in production. If you retry, retry a named list, visibly, with the count reported.

The one that is not a flake

A test that fails once in two hundred runs because of a genuine race in the code under test is not a flaky test. It is the only detector you have for a bug that will happen in production at a rate proportional to your traffic, and quarantining it is deleting your smoke alarm because it went off.

The way to tell them apart: fix the test’s own nondeterminism first: inject the clock, remove the sleep, isolate the state. If it still fails intermittently after that, the nondeterminism is in the system, and you have found something important.