{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"mutation-testing","title":"Proving the suite bites","url":"https://addaly.com/learn/how-software-changed/mutation-testing","position":25,"of":87,"minutes":9,"answer":"Mutation testing measures whether a suite would notice if the code were wrong, which is the question coverage cannot answer, and the informative comparison is a module with high coverage and a low mutation score.","keyIdea":"Mutation testing measures whether a suite would notice if the code were wrong, which is the question coverage cannot answer, and the informative comparison is a module with high coverage and a low mutation score."},"body":{"format":"markdown","text":"## Coverage answers the wrong question\n\nCoverage measures which lines executed while the tests ran. It does not measure whether anything would have noticed had those lines been wrong. Every one of the weak test shapes (the assert-nothing test, the tautology, the mirror) scores well on coverage, because they all execute the code.\n\nThat gap always existed. It matters more now because a large generated suite produces high coverage effortlessly, so the number that used to be weak evidence of care has become no evidence at all.\n\nMutation testing asks the right question directly: **if I break the code, does a test go red?**\n\n## How it works\n\nThe tool makes small, systematic edits to your source (mutants) and runs the suite against each one.\n\n- Change `<` to `<=`\n- Change `+` to `-`\n- Replace a return value with a constant\n- Delete a statement\n- Negate a condition\n- Replace a string literal with an empty string\n\nIf the suite fails, the mutant is *killed* and your tests are doing their job on that line. If the suite passes, the mutant *survived*, and you have found a line whose behaviour no test constrains.\n\nThe mutation score is the proportion killed. Unlike coverage, it cannot be gamed by executing code without asserting on it.\n\n## Running it\n\nAll of these are free and open source.\n\n```bash\n# Python\npip install mutmut && mutmut run --paths-to-mutate billing/\nmutmut results\nmutmut show 42          # see the exact surviving mutant\n\n# JavaScript / TypeScript\nnpx stryker run\n\n# Java: PIT.  Rust: cargo-mutants.  Go: go-mutesting.  C#: Stryker.NET.\n```\n\nThe output that matters is not the score. It is the list of surviving mutants, because each one is a specific sentence: *this line could be changed to that and nothing in your suite would object.* Read five of them and you will learn more about your tests than from any dashboard.\n\n## The cost, and how to make it affordable\n\nMutation testing is slow. It runs your test suite once per mutant, and a moderate module generates hundreds. A suite that takes 40 seconds becomes a two-hour run. This is the reason most teams try it once and abandon it.\n\nThe fix is to stop running it over everything.\n\n**Run it on the diff only.** Mutate the lines this pull request changed. That is usually 50 to 200 lines, the run takes a few minutes, and it answers exactly the question review needs: do the new tests actually test the new code?\n\n```bash\n# stryker supports this directly\nnpx stryker run --since=main\n\n# mutmut, roughly\nmutmut run --paths-to-mutate \"$(git diff --name-only origin/main...HEAD \\\n  | grep '\\.py$' | tr '\\n' ',')\"\n```\n\n**Run it nightly on the highest-value module.** Billing, authentication, permissions. One module, overnight, results in the morning.\n\n**Run it once, manually, on a suite you have inherited and do not trust.** This is the highest-value single use. You will get a number and a list, and the list will tell you whether the suite is load-bearing or decorative.\n\n## Reading the score honestly\n\nDo not chase 100%. Some surviving mutants are equivalent: the mutated code behaves identically, so no test could kill it. Some are on code where the behaviour does not matter, like a log message. Chasing the last fifteen percent produces tests written to kill mutants, which is a new way of writing tests that assert nothing meaningful.\n\nUseful bands, in practice: below 50% on important code means the suite is decoration. 60 to 80% is a normal, healthy suite. Above 90% is either an exceptional suite or a team that has started gaming the metric.\n\nAnd the comparison worth making is not against an absolute target but against your own coverage number. A module with 95% coverage and a 40% mutation score is the specific, common, dangerous case: fully exercised and almost entirely unchecked. That single comparison is the reason to run this at all.\n\n> Figure: Coverage against mutation score. A two by two grid, rows High coverage and Low coverage, columns High mutation score and Low mutation score: High coverage and High mutation score: Healthy; High coverage and Low mutation score: Exercised, unchecked; Low coverage and High mutation score: Narrow but real; Low coverage and Low mutation score: Decoration. The comparison worth making is against your own coverage number rather than an absolute target. A module at 95 per cent coverage with a 40 per cent mutation score is fully exercised and almost entirely unchecked.\n\n## The manual version, which is free and immediate\n\nIf you install nothing else today, do this once by hand on the module you are least sure about. Open the source, break one thing on purpose, run the suite.\n\n```python\n-if amount < 0:\n+if amount < -1_000_000:\n     raise ValueError(\"negative amount\")\n```\n\nGreen? You have learned in ninety seconds what a coverage report would never have told you, and you now know exactly how much to trust the next green run."},"check":{"question":"A module reports 96% line coverage and a 41% mutation score. What is the most accurate reading?","options":[{"text":"The suite is slow or flaky, so many mutants survived because their test runs timed out rather than because the tests are weak.","correct":false,"tempting":"Timeouts do inflate survival and should be checked, but they would show up as an error category rather than as a broad low score."},{"text":"Nearly all lines run during tests, but most of them could be changed without any test failing.","correct":true},{"text":"The mutation tool generated too many equivalent mutants, which is common in modules with heavy logging and configuration code.","correct":false,"tempting":"Equivalent mutants do depress scores, but a gap this large in a normal module is not explained by them."},{"text":"Coverage is measuring integration tests while mutation testing only runs unit tests, so the two numbers describe different suites.","correct":false,"tempting":"Invents a tooling split; both run whatever suite is configured, and assuming otherwise hides a real weakness."}]},"prev":{"slug":"constraints-in-the-database","title":"The database is the last honest gate","url":"https://addaly.com/learn/how-software-changed/constraints-in-the-database"},"next":{"slug":"adversarial-inputs","title":"Inputs designed to hurt","url":"https://addaly.com/learn/how-software-changed/adversarial-inputs"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Proving the suite bites', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/mutation-testing (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Proving the suite bites","courseTitle":"How Software Development Changed","module":{"index":3,"slug":"oracles-and-verification","title":"Building oracles that stay ahead"},"answer":null,"keyIdea":"Mutation testing measures whether a suite would notice if the code were wrong, which is the question coverage cannot answer, and the informative comparison is a module with high coverage and a low mutation score.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["High mutation score","Low mutation score"],"rows":["High coverage","Low coverage"],"type":"matrix","cells":[["Healthy","Exercised, unchecked"],["Narrow but real","Decoration"]],"title":"Coverage against mutation score","caption":"The comparison worth making is against your own coverage number rather than an absolute target. A module at 95 per cent coverage with a 40 per cent mutation score is fully exercised and almost entirely unchecked.","cellLabels":[["the tests run the code and object when it is wrong","the common dangerous case: every line runs and almost nothing is asserted"],["what it covers, it covers properly","neither run nor checked, and honest about it"]]},"alt":"Figure: Coverage against mutation score. A two by two grid, rows High coverage and Low coverage, columns High mutation score and Low mutation score: High coverage and High mutation score: Healthy; High coverage and Low mutation score: Exercised, unchecked; Low coverage and High mutation score: Narrow but real; Low coverage and Low mutation score: Decoration. The comparison worth making is against your own coverage number rather than an absolute target. A module at 95 per cent coverage with a 40 per cent mutation score is fully exercised and almost entirely unchecked."}},"blocks":[{"t":"h2","id":"coverage-answers-the-wrong-question","runs":[{"text":"Coverage answers the wrong question"}]},{"t":"p","runs":[{"text":"Coverage measures which lines executed while the tests ran. It does not measure whether anything would have noticed had those lines been wrong. Every one of the weak test shapes (the assert-nothing test, the tautology, the mirror) scores well on coverage, because they all execute the code."}]},{"t":"p","runs":[{"text":"That gap always existed. It matters more now because a large generated suite produces high coverage effortlessly, so the number that used to be weak evidence of care has become no evidence at all."}]},{"t":"p","runs":[{"text":"Mutation testing asks the right question directly: "},{"text":"if I break the code, does a test go red?","b":true}]},{"t":"h2","id":"how-it-works","runs":[{"text":"How it works"}]},{"t":"p","runs":[{"text":"The tool makes small, systematic edits to your source (mutants) and runs the suite against each one."}]},{"t":"ul","items":[[{"text":"Change "},{"text":"<","code":true},{"text":" to "},{"text":"<=","code":true}],[{"text":"Change "},{"text":"+","code":true},{"text":" to "},{"text":"-","code":true}],[{"text":"Replace a return value with a constant"}],[{"text":"Delete a statement"}],[{"text":"Negate a condition"}],[{"text":"Replace a string literal with an empty string"}]]},{"t":"p","runs":[{"text":"If the suite fails, the mutant is "},{"text":"killed","i":true},{"text":" and your tests are doing their job on that line. If the suite passes, the mutant "},{"text":"survived","i":true},{"text":", and you have found a line whose behaviour no test constrains."}]},{"t":"p","runs":[{"text":"The mutation score is the proportion killed. Unlike coverage, it cannot be gamed by executing code without asserting on it."}]},{"t":"h2","id":"running-it","runs":[{"text":"Running it"}]},{"t":"p","runs":[{"text":"All of these are free and open source."}]},{"t":"code","lang":"bash","label":"Bash","text":"# Python\npip install mutmut && mutmut run --paths-to-mutate billing/\nmutmut results\nmutmut show 42          # see the exact surviving mutant\n\n# JavaScript / TypeScript\nnpx stryker run\n\n# Java: PIT.  Rust: cargo-mutants.  Go: go-mutesting.  C#: Stryker.NET."},{"t":"p","runs":[{"text":"The output that matters is not the score. It is the list of surviving mutants, because each one is a specific sentence: "},{"text":"this line could be changed to that and nothing in your suite would object.","i":true},{"text":" Read five of them and you will learn more about your tests than from any dashboard."}]},{"t":"h2","id":"the-cost-and-how-to-make-it-affordable","runs":[{"text":"The cost, and how to make it affordable"}]},{"t":"p","runs":[{"text":"Mutation testing is slow. It runs your test suite once per mutant, and a moderate module generates hundreds. A suite that takes 40 seconds becomes a two-hour run. This is the reason most teams try it once and abandon it."}]},{"t":"p","runs":[{"text":"The fix is to stop running it over everything."}]},{"t":"p","runs":[{"text":"Run it on the diff only.","b":true},{"text":" Mutate the lines this pull request changed. That is usually 50 to 200 lines, the run takes a few minutes, and it answers exactly the question review needs: do the new tests actually test the new code?"}]},{"t":"code","lang":"bash","label":"Bash","text":"# stryker supports this directly\nnpx stryker run --since=main\n\n# mutmut, roughly\nmutmut run --paths-to-mutate \"$(git diff --name-only origin/main...HEAD \\\n  | grep '\\.py$' | tr '\\n' ',')\""},{"t":"p","runs":[{"text":"Run it nightly on the highest-value module.","b":true},{"text":" Billing, authentication, permissions. One module, overnight, results in the morning."}]},{"t":"p","runs":[{"text":"Run it once, manually, on a suite you have inherited and do not trust.","b":true},{"text":" This is the highest-value single use. You will get a number and a list, and the list will tell you whether the suite is load-bearing or decorative."}]},{"t":"h2","id":"reading-the-score-honestly","runs":[{"text":"Reading the score honestly"}]},{"t":"p","runs":[{"text":"Do not chase 100%. Some surviving mutants are equivalent: the mutated code behaves identically, so no test could kill it. Some are on code where the behaviour does not matter, like a log message. Chasing the last fifteen percent produces tests written to kill mutants, which is a new way of writing tests that assert nothing meaningful."}]},{"t":"p","runs":[{"text":"Useful bands, in practice: below 50% on important code means the suite is decoration. 60 to 80% is a normal, healthy suite. Above 90% is either an exceptional suite or a team that has started gaming the metric."}]},{"t":"p","runs":[{"text":"And the comparison worth making is not against an absolute target but against your own coverage number. A module with 95% coverage and a 40% mutation score is the specific, common, dangerous case: fully exercised and almost entirely unchecked. That single comparison is the reason to run this at all."}]},{"t":"h2","id":"the-manual-version-which-is-free-and-immediate","runs":[{"text":"The manual version, which is free and immediate"}]},{"t":"p","runs":[{"text":"If you install nothing else today, do this once by hand on the module you are least sure about. Open the source, break one thing on purpose, run the suite."}]},{"t":"code","lang":"python","label":"Python","text":"-if amount < 0:\n+if amount < -1_000_000:\n     raise ValueError(\"negative amount\")"},{"t":"p","runs":[{"text":"Green? You have learned in ninety seconds what a coverage report would never have told you, and you now know exactly how much to trust the next green run."}]}],"check":{"question":[{"text":"A module reports 96% line coverage and a 41% mutation score. What is the most accurate reading?"}],"options":[{"text":[{"text":"The suite is slow or flaky, so many mutants survived because their test runs timed out rather than because the tests are weak."}],"correct":false,"tempting":[{"text":"Timeouts do inflate survival and should be checked, but they would show up as an error category rather than as a broad low score."}]},{"text":[{"text":"Nearly all lines run during tests, but most of them could be changed without any test failing."}],"correct":true,"tempting":null},{"text":[{"text":"The mutation tool generated too many equivalent mutants, which is common in modules with heavy logging and configuration code."}],"correct":false,"tempting":[{"text":"Equivalent mutants do depress scores, but a gap this large in a normal module is not explained by them."}]},{"text":[{"text":"Coverage is measuring integration tests while mutation testing only runs unit tests, so the two numbers describe different suites."}],"correct":false,"tempting":[{"text":"Invents a tooling split; both run whatever suite is configured, and assuming otherwise hides a real weakness."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"constraints-in-the-database","title":"The database is the last honest gate","minutes":9},"next":{"slug":"adversarial-inputs","title":"Inputs designed to hurt","minutes":9}}}