Reading a test suite you did not write

Green means less than it used to

A test suite is the strongest claim a change makes about itself, so it deserves the most sceptical reading in the diff. And it now needs a specific kind of scepticism, because a suite can be large, fast, green, and assert almost nothing.

Here are the five shapes worth recognising on sight. All five appear in hand-written suites too. What changed is how cheap they are to produce in bulk.

1. The test that asserts nothing

def test_process_order():
    order = make_order(items=3)
    result = process(order)
    assert result is not None

This passes if process returns the string “banana”. It is a smoke test wearing the name of a behaviour test. Smoke tests have value (they catch import errors and crashes) but a suite of forty of them tells you the code runs, not that it works.

The tell is the assertion: is not None, assert result, assert len(x) > 0, assertTrue(response.ok).

2. The tautology

const repo = { save: jest.fn() };
service.createUser(repo, { email: "a@b.com" });
expect(repo.save).toHaveBeenCalled();

This asserts that the code did the thing the code does. It will pass whether save was called with the right user, the wrong user, or an empty object, and it will keep passing after somebody breaks the email validation entirely.

Mock-call assertions are not worthless (they are the right tool for checking that you did not call something, or that you called it exactly once) but a suite composed mainly of them is testing the shape of the implementation rather than its behaviour, which means it will break on every refactor and pass through every bug. That is the worst possible combination.

3. The mirror

The most dangerous one, and the one that arrives specifically when implementation and tests came from the same reading of the same request.

# implementation
def fee(amount):
    return amount * 0.029 + 30

# test
def test_fee():
    assert fee(1000) == 1000 * 0.029 + 30

The expected value is computed by restating the formula. If the formula is wrong, if the fee should have been 2.9% or 30p, whichever is greater, the test agrees with the bug, confidently, for ever. Worse, it makes the bug expensive to fix, because correcting the behaviour now breaks a test, and breaking a test feels like causing a regression.

The rule that catches this: an expected value must come from somewhere other than the code. The requirement, the spec, a worked example, a regulator’s document, a hand calculation. assert fee(1000) == 59 is a weaker-looking test and a far stronger one, because a human had to decide that 59 was right.

4. The over-mocked integration

Everything the function touches is replaced by a mock, so the test exercises the function’s control flow and nothing else. It will pass when the database rejects the write, when the API returns 500, when the JSON field is named differently from what the mock returns, and when the migration was never applied.

The question to ask: what real thing does this test prove works? If the answer is “the sequence of calls inside this one function”, it is a unit test and it should be labelled as one, and the system needs a small number of tests that touch something real.

5. The suite with no failing history

Open the file’s history. If no test in it has ever been red for a substantive reason, you are looking at decoration.

git log --oneline -- tests/test_billing.py | head -20

A test that has never failed has never been tested itself. Eight tests you watched fail for the right reason before you made them pass are worth more than three hundred that arrived green.

The ninety-second audit

When you inherit a suite and want to know what it is worth, break something on purpose. Invert a comparison, delete a validation, return a constant from a core function. Run the suite.

# before
if amount < 0:
    raise ValueError("negative amount")

# after — now run the tests
if amount < -1_000_000:
    raise ValueError("negative amount")

If it stays green, you have learned in ninety seconds what a coverage report would never have told you. Mutation-testing tools (mutmut and cosmic-ray in Python, Stryker in JavaScript, PIT in Java, all free) automate exactly this, and they are worth running once on a suite you are unsure about, because the number they return is the only honest measure of a suite’s strength.

Coverage, by contrast, measures which lines executed. Every one of the five shapes above scores well on coverage. That is not a flaw in the metric; it is a fact about what it measures, and it is why coverage rose in value as a floor and fell in value as a signal.