{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"flaky-tests","title":"A suite that lies intermittently","url":"https://addaly.com/learn/how-software-changed/flaky-tests","position":27,"of":87,"minutes":9,"answer":"Flake probability compounds across a suite, so a small per-test failure rate makes a red build meaningless, and the fix is automatic detection, a capped quarantine with a deletion deadline, and never a blanket retry.","keyIdea":"Flake probability compounds across a suite, so a small per-test failure rate makes a red build meaningless, and the fix is automatic detection, a capped quarantine with a deletion deadline, and never a blanket retry."},"body":{"format":"markdown","text":"## The arithmetic that explains the despair\n\nA flaky test is one that passes and fails on identical code. One of them is an annoyance. The problem is that they compound, and the compounding is multiplicative.\n\nSuppose each test in a 500-test suite has a 0.1% chance of failing spuriously. The probability that a whole run is clean is `0.999^500`, which is about 61%. So two runs in five are red for no reason.\n\nAt 0.5% per test, `0.995^500` is about 8%. Nine runs in ten are red on correct code.\n\n> Figure: What a small flake rate does to a suite of 500. Line chart of chance the whole run is clean, per cent against chance any one test fails spuriously, per cent: A suite of 500 independent tests (0, 100) (0.1, 61) (0.2, 37) (0.5, 8) (1, 0.7). At a tenth of a per cent per test, two runs in five are red for no reason. At half a per cent, nine in ten are. That is the point where re-running becomes the rational response and the suite stops being a gate.\n\nThat is the mechanism behind the thing every engineer has watched happen: a team reaches the point where a red build carries no information, and the rational response (re-run it) becomes the norm. From then on the suite has stopped being a gate. It is a delay with a random duration.\n\n## Where the flakes come from, in order of frequency\n\n**Time.** `datetime.now()` in the code or the test. A test that computes \"30 days from today\" and fails when the month boundary lands wrong. Anything with `sleep()` in it, which is a guess about how long something takes, and guesses fail on a loaded CI machine.\n\n**Order dependence.** Test A leaves a row behind and test B passes only because of it. Runs green in file order, red under `pytest -p randomly` or a parallel runner. The fix is to run randomised locally on purpose so you find it before CI does.\n\n**Shared state.** A module-level cache, a singleton, an environment variable, a temporary file with a fixed name, a database not rolled back between tests, a port already in use.\n\n**The network.** Any test that reaches a real service is flaky by construction, because the internet is. This includes package registries during the build.\n\n**Concurrency.** The hard category. A race that manifests once in two hundred runs is telling you about a race in the production system, and it deserves investigation rather than a retry.\n\n## Why generated tests flake more\n\nThis is worth naming specifically, because it is a real and consistent pattern.\n\nGenerated tests reach for `sleep()` to wait for asynchronous work, because that is what appears most in the world's example code. They use the real clock rather than an injected one. They assume dictionary and set iteration order. They construct fixtures with fixed identifiers that collide under parallel execution. And they are produced in volume, so a suite of 300 generated tests carries more of these patterns than a suite of 40 written ones, and each one is a lottery ticket.\n\nThe review check is simple and worth automating: grep new test files for `sleep(`, `now()`, `random`, `localhost:` and hard-coded ports, and ask about every hit.\n\n## A policy that works\n\nThe key move is that a flaky test must be **removed from the gate immediately** and **fixed on a deadline**, and those are two separate decisions. Teams that only do the first accumulate a quarantine nobody empties. Teams that only do the second leave a broken gate in place for a week.\n\n1. **Detect automatically.** Re-run the suite on the merge commit nightly. Any test that has both passed and failed on identical code in the last thirty runs is flaky, by definition, and no argument is needed.\n2. **Quarantine on detection.** Mark it, exclude it from the required check, and open a ticket with the failure output attached.\n3. **Cap the quarantine.** A fixed number (ten is a common choice) and when it is full, the next flaky test cannot be quarantined until one is fixed or deleted. Without a cap, quarantine is a graveyard.\n4. **Set a deadline.** Two weeks. After that the test is deleted and its ticket becomes \"write a reliable test for this behaviour\". Deleting a test feels wrong and is correct: an unreliable test is worse than none, because it consumes attention and provides no signal.\n5. **Never add a blanket retry.** `--reruns 3` at the suite level hides every flake including the ones that are real concurrency bugs in production. If you retry, retry a named list, visibly, with the count reported.\n\n## The one that is not a flake\n\nA test that fails once in two hundred runs because of a genuine race in the code under test is not a flaky test. It is the only detector you have for a bug that will happen in production at a rate proportional to your traffic, and quarantining it is deleting your smoke alarm because it went off.\n\nThe way to tell them apart: fix the test's own nondeterminism first: inject the clock, remove the sleep, isolate the state. If it still fails intermittently after that, the nondeterminism is in the system, and you have found something important."},"check":{"question":"A 500-test suite has a per-test spurious failure rate of about 0.5%. The team adds `--reruns 3` so red builds retry automatically. Builds are green again. What has been lost?","options":[{"text":"Build speed, since every flaky run now costs up to four times as long and the pipeline becomes the bottleneck.","correct":false,"tempting":"A real cost, but a slower green pipeline is a nuisance rather than the loss of a safety property."},{"text":"Nothing important, provided the team tracks which tests needed retries and fixes the worst offenders over time.","correct":false,"tempting":"Describes a discipline that almost never survives, because a green build removes the pressure that would have driven the fixing."},{"text":"The ability to distinguish a spurious failure from a real intermittent bug in the system.","correct":true},{"text":"Coverage, because retried tests may take different code paths on each attempt and the reported figures become unreliable.","correct":false,"tempting":"Invents a measurement artefact; coverage is aggregated across the run and is not what retries damage."}]},"prev":{"slug":"adversarial-inputs","title":"Inputs designed to hurt","url":"https://addaly.com/learn/how-software-changed/adversarial-inputs"},"next":{"slug":"ci-as-the-gate","title":"Designing a pipeline that is actually a gate","url":"https://addaly.com/learn/how-software-changed/ci-as-the-gate"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A suite that lies intermittently', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/flaky-tests (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A suite that lies intermittently","courseTitle":"How Software Development Changed","module":{"index":3,"slug":"oracles-and-verification","title":"Building oracles that stay ahead"},"answer":null,"keyIdea":"Flake probability compounds across a suite, so a small per-test failure rate makes a red build meaningless, and the fix is automatic detection, a capped quarantine with a deletion deadline, and never a blanket retry.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"What a small flake rate does to a suite of 500","series":[{"label":"A suite of 500 independent tests","points":[[0,100],[0.1,61],[0.2,37],[0.5,8],[1,0.7]]}],"xLabel":"chance any one test fails spuriously, per cent","yLabel":"chance the whole run is clean, per cent","caption":"At a tenth of a per cent per test, two runs in five are red for no reason. At half a per cent, nine in ten are. That is the point where re-running becomes the rational response and the suite stops being a gate."},"alt":"Figure: What a small flake rate does to a suite of 500. Line chart of chance the whole run is clean, per cent against chance any one test fails spuriously, per cent: A suite of 500 independent tests (0, 100) (0.1, 61) (0.2, 37) (0.5, 8) (1, 0.7). At a tenth of a per cent per test, two runs in five are red for no reason. At half a per cent, nine in ten are. That is the point where re-running becomes the rational response and the suite stops being a gate."}},"blocks":[{"t":"h2","id":"the-arithmetic-that-explains-the-despair","runs":[{"text":"The arithmetic that explains the despair"}]},{"t":"p","runs":[{"text":"A flaky test is one that passes and fails on identical code. One of them is an annoyance. The problem is that they compound, and the compounding is multiplicative."}]},{"t":"p","runs":[{"text":"Suppose each test in a 500-test suite has a 0.1% chance of failing spuriously. The probability that a whole run is clean is "},{"text":"0.999^500","code":true},{"text":", which is about 61%. So two runs in five are red for no reason."}]},{"t":"p","runs":[{"text":"At 0.5% per test, "},{"text":"0.995^500","code":true},{"text":" is about 8%. Nine runs in ten are red on correct code."}]},{"t":"p","runs":[{"text":"That is the mechanism behind the thing every engineer has watched happen: a team reaches the point where a red build carries no information, and the rational response (re-run it) becomes the norm. From then on the suite has stopped being a gate. It is a delay with a random duration."}]},{"t":"h2","id":"where-the-flakes-come-from-in-order-of-frequency","runs":[{"text":"Where the flakes come from, in order of frequency"}]},{"t":"p","runs":[{"text":"Time.","b":true},{"text":" "},{"text":"datetime.now()","code":true},{"text":" in the code or the test. A test that computes “30 days from today” and fails when the month boundary lands wrong. Anything with "},{"text":"sleep()","code":true},{"text":" in it, which is a guess about how long something takes, and guesses fail on a loaded CI machine."}]},{"t":"p","runs":[{"text":"Order dependence.","b":true},{"text":" Test A leaves a row behind and test B passes only because of it. Runs green in file order, red under "},{"text":"pytest -p randomly","code":true},{"text":" or a parallel runner. The fix is to run randomised locally on purpose so you find it before CI does."}]},{"t":"p","runs":[{"text":"Shared state.","b":true},{"text":" A module-level cache, a singleton, an environment variable, a temporary file with a fixed name, a database not rolled back between tests, a port already in use."}]},{"t":"p","runs":[{"text":"The network.","b":true},{"text":" Any test that reaches a real service is flaky by construction, because the internet is. This includes package registries during the build."}]},{"t":"p","runs":[{"text":"Concurrency.","b":true},{"text":" The hard category. A race that manifests once in two hundred runs is telling you about a race in the production system, and it deserves investigation rather than a retry."}]},{"t":"h2","id":"why-generated-tests-flake-more","runs":[{"text":"Why generated tests flake more"}]},{"t":"p","runs":[{"text":"This is worth naming specifically, because it is a real and consistent pattern."}]},{"t":"p","runs":[{"text":"Generated tests reach for "},{"text":"sleep()","code":true},{"text":" to wait for asynchronous work, because that is what appears most in the world’s example code. They use the real clock rather than an injected one. They assume dictionary and set iteration order. They construct fixtures with fixed identifiers that collide under parallel execution. And they are produced in volume, so a suite of 300 generated tests carries more of these patterns than a suite of 40 written ones, and each one is a lottery ticket."}]},{"t":"p","runs":[{"text":"The review check is simple and worth automating: grep new test files for "},{"text":"sleep(","code":true},{"text":", "},{"text":"now()","code":true},{"text":", "},{"text":"random","code":true},{"text":", "},{"text":"localhost:","code":true},{"text":" and hard-coded ports, and ask about every hit."}]},{"t":"h2","id":"a-policy-that-works","runs":[{"text":"A policy that works"}]},{"t":"p","runs":[{"text":"The key move is that a flaky test must be "},{"text":"removed from the gate immediately","b":true},{"text":" and "},{"text":"fixed on a deadline","b":true},{"text":", and those are two separate decisions. Teams that only do the first accumulate a quarantine nobody empties. Teams that only do the second leave a broken gate in place for a week."}]},{"t":"ol","start":1,"items":[[{"text":"Detect automatically.","b":true},{"text":" Re-run the suite on the merge commit nightly. Any test that has both passed and failed on identical code in the last thirty runs is flaky, by definition, and no argument is needed."}],[{"text":"Quarantine on detection.","b":true},{"text":" Mark it, exclude it from the required check, and open a ticket with the failure output attached."}],[{"text":"Cap the quarantine.","b":true},{"text":" A fixed number (ten is a common choice) and when it is full, the next flaky test cannot be quarantined until one is fixed or deleted. Without a cap, quarantine is a graveyard."}],[{"text":"Set a deadline.","b":true},{"text":" Two weeks. After that the test is deleted and its ticket becomes “write a reliable test for this behaviour”. Deleting a test feels wrong and is correct: an unreliable test is worse than none, because it consumes attention and provides no signal."}],[{"text":"Never add a blanket retry.","b":true},{"text":" "},{"text":"--reruns 3","code":true},{"text":" at the suite level hides every flake including the ones that are real concurrency bugs in production. If you retry, retry a named list, visibly, with the count reported."}]]},{"t":"h2","id":"the-one-that-is-not-a-flake","runs":[{"text":"The one that is not a flake"}]},{"t":"p","runs":[{"text":"A test that fails once in two hundred runs because of a genuine race in the code under test is not a flaky test. It is the only detector you have for a bug that will happen in production at a rate proportional to your traffic, and quarantining it is deleting your smoke alarm because it went off."}]},{"t":"p","runs":[{"text":"The way to tell them apart: fix the test’s own nondeterminism first: inject the clock, remove the sleep, isolate the state. If it still fails intermittently after that, the nondeterminism is in the system, and you have found something important."}]}],"check":{"question":[{"text":"A 500-test suite has a per-test spurious failure rate of about 0.5%. The team adds "},{"text":"--reruns 3","code":true},{"text":" so red builds retry automatically. Builds are green again. What has been lost?"}],"options":[{"text":[{"text":"Build speed, since every flaky run now costs up to four times as long and the pipeline becomes the bottleneck."}],"correct":false,"tempting":[{"text":"A real cost, but a slower green pipeline is a nuisance rather than the loss of a safety property."}]},{"text":[{"text":"Nothing important, provided the team tracks which tests needed retries and fixes the worst offenders over time."}],"correct":false,"tempting":[{"text":"Describes a discipline that almost never survives, because a green build removes the pressure that would have driven the fixing."}]},{"text":[{"text":"The ability to distinguish a spurious failure from a real intermittent bug in the system."}],"correct":true,"tempting":null},{"text":[{"text":"Coverage, because retried tests may take different code paths on each attempt and the reported figures become unreliable."}],"correct":false,"tempting":[{"text":"Invents a measurement artefact; coverage is aggregated across the run and is not what retries damage."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"adversarial-inputs","title":"Inputs designed to hurt","minutes":9},"next":{"slug":"ci-as-the-gate","title":"Designing a pipeline that is actually a gate","minutes":9}}}