Proving the suite bites

Coverage answers the wrong question

Coverage measures which lines executed while the tests ran. It does not measure whether anything would have noticed had those lines been wrong. Every one of the weak test shapes (the assert-nothing test, the tautology, the mirror) scores well on coverage, because they all execute the code.

That gap always existed. It matters more now because a large generated suite produces high coverage effortlessly, so the number that used to be weak evidence of care has become no evidence at all.

Mutation testing asks the right question directly: if I break the code, does a test go red?

How it works

The tool makes small, systematic edits to your source (mutants) and runs the suite against each one.

• Change < to <=
• Change + to -
• Replace a return value with a constant
• Delete a statement
• Negate a condition
• Replace a string literal with an empty string

If the suite fails, the mutant is killed and your tests are doing their job on that line. If the suite passes, the mutant survived, and you have found a line whose behaviour no test constrains.

The mutation score is the proportion killed. Unlike coverage, it cannot be gamed by executing code without asserting on it.

Running it

All of these are free and open source.

# Python
pip install mutmut && mutmut run --paths-to-mutate billing/
mutmut results
mutmut show 42          # see the exact surviving mutant

# JavaScript / TypeScript
npx stryker run

# Java: PIT.  Rust: cargo-mutants.  Go: go-mutesting.  C#: Stryker.NET.

The output that matters is not the score. It is the list of surviving mutants, because each one is a specific sentence: this line could be changed to that and nothing in your suite would object. Read five of them and you will learn more about your tests than from any dashboard.

The cost, and how to make it affordable

Mutation testing is slow. It runs your test suite once per mutant, and a moderate module generates hundreds. A suite that takes 40 seconds becomes a two-hour run. This is the reason most teams try it once and abandon it.

The fix is to stop running it over everything.

Run it on the diff only. Mutate the lines this pull request changed. That is usually 50 to 200 lines, the run takes a few minutes, and it answers exactly the question review needs: do the new tests actually test the new code?

# stryker supports this directly
npx stryker run --since=main

# mutmut, roughly
mutmut run --paths-to-mutate "$(git diff --name-only origin/main...HEAD \
  | grep '\.py$' | tr '\n' ',')"

Run it nightly on the highest-value module. Billing, authentication, permissions. One module, overnight, results in the morning.

Run it once, manually, on a suite you have inherited and do not trust. This is the highest-value single use. You will get a number and a list, and the list will tell you whether the suite is load-bearing or decorative.

Reading the score honestly

Do not chase 100%. Some surviving mutants are equivalent: the mutated code behaves identically, so no test could kill it. Some are on code where the behaviour does not matter, like a log message. Chasing the last fifteen percent produces tests written to kill mutants, which is a new way of writing tests that assert nothing meaningful.

Useful bands, in practice: below 50% on important code means the suite is decoration. 60 to 80% is a normal, healthy suite. Above 90% is either an exceptional suite or a team that has started gaming the metric.

And the comparison worth making is not against an absolute target but against your own coverage number. A module with 95% coverage and a 40% mutation score is the specific, common, dangerous case: fully exercised and almost entirely unchecked. That single comparison is the reason to run this at all.

The manual version, which is free and immediate

If you install nothing else today, do this once by hand on the module you are least sure about. Open the source, break one thing on purpose, run the suite.

-if amount < 0:
+if amount < -1_000_000:
     raise ValueError("negative amount")

Green? You have learned in ninety seconds what a coverage report would never have told you, and you now know exactly how much to trust the next green run.