{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"reading-the-tests","title":"Reading a test suite you did not write","url":"https://addaly.com/learn/how-software-changed/reading-the-tests","position":15,"of":87,"minutes":10,"answer":"A test's strength lies in where its expected value came from, so a suite whose assertions restate the implementation will pass through every bug and break on every refactor.","keyIdea":"A test's strength lies in where its expected value came from, so a suite whose assertions restate the implementation will pass through every bug and break on every refactor, and the cheapest way to find out is to break the code on purpose and watch."},"body":{"format":"markdown","text":"## Green means less than it used to\n\nA test suite is the strongest claim a change makes about itself, so it deserves the most sceptical reading in the diff. And it now needs a specific kind of scepticism, because a suite can be large, fast, green, and assert almost nothing.\n\nHere are the five shapes worth recognising on sight. All five appear in hand-written suites too. What changed is how cheap they are to produce in bulk.\n\n## 1. The test that asserts nothing\n\n```python\ndef test_process_order():\n    order = make_order(items=3)\n    result = process(order)\n    assert result is not None\n```\n\nThis passes if `process` returns the string \"banana\". It is a smoke test wearing the name of a behaviour test. Smoke tests have value (they catch import errors and crashes) but a suite of forty of them tells you the code runs, not that it works.\n\nThe tell is the assertion: `is not None`, `assert result`, `assert len(x) > 0`, `assertTrue(response.ok)`.\n\n## 2. The tautology\n\n```javascript\nconst repo = { save: jest.fn() };\nservice.createUser(repo, { email: \"a@b.com\" });\nexpect(repo.save).toHaveBeenCalled();\n```\n\nThis asserts that the code did the thing the code does. It will pass whether `save` was called with the right user, the wrong user, or an empty object, and it will keep passing after somebody breaks the email validation entirely.\n\nMock-call assertions are not worthless (they are the right tool for checking that you did *not* call something, or that you called it exactly once) but a suite composed mainly of them is testing the shape of the implementation rather than its behaviour, which means it will break on every refactor and pass through every bug. That is the worst possible combination.\n\n## 3. The mirror\n\nThe most dangerous one, and the one that arrives specifically when implementation and tests came from the same reading of the same request.\n\n```python\n# implementation\ndef fee(amount):\n    return amount * 0.029 + 30\n\n# test\ndef test_fee():\n    assert fee(1000) == 1000 * 0.029 + 30\n```\n\nThe expected value is computed by restating the formula. If the formula is wrong, if the fee should have been 2.9% *or* 30p, whichever is greater, the test agrees with the bug, confidently, for ever. Worse, it makes the bug expensive to fix, because correcting the behaviour now breaks a test, and breaking a test feels like causing a regression.\n\nThe rule that catches this: **an expected value must come from somewhere other than the code.** The requirement, the spec, a worked example, a regulator's document, a hand calculation. `assert fee(1000) == 59` is a weaker-looking test and a far stronger one, because a human had to decide that 59 was right.\n\n## 4. The over-mocked integration\n\nEverything the function touches is replaced by a mock, so the test exercises the function's control flow and nothing else. It will pass when the database rejects the write, when the API returns 500, when the JSON field is named differently from what the mock returns, and when the migration was never applied.\n\nThe question to ask: **what real thing does this test prove works?** If the answer is \"the sequence of calls inside this one function\", it is a unit test and it should be labelled as one, and the system needs a small number of tests that touch something real.\n\n## 5. The suite with no failing history\n\nOpen the file's history. If no test in it has ever been red for a substantive reason, you are looking at decoration.\n\n```bash\ngit log --oneline -- tests/test_billing.py | head -20\n```\n\nA test that has never failed has never been tested itself. Eight tests you watched fail for the right reason before you made them pass are worth more than three hundred that arrived green.\n\n## The ninety-second audit\n\nWhen you inherit a suite and want to know what it is worth, break something on purpose. Invert a comparison, delete a validation, return a constant from a core function. Run the suite.\n\n```python\n# before\nif amount < 0:\n    raise ValueError(\"negative amount\")\n\n# after — now run the tests\nif amount < -1_000_000:\n    raise ValueError(\"negative amount\")\n```\n\nIf it stays green, you have learned in ninety seconds what a coverage report would never have told you. Mutation-testing tools (`mutmut` and `cosmic-ray` in Python, Stryker in JavaScript, PIT in Java, all free) automate exactly this, and they are worth running once on a suite you are unsure about, because the number they return is the only honest measure of a suite's strength.\n\nCoverage, by contrast, measures which lines executed. Every one of the five shapes above scores well on coverage. That is not a flaw in the metric; it is a fact about what it measures, and it is why coverage rose in value as a floor and fell in value as a signal."},"check":{"question":"A pull request adds a fee calculation and 40 tests. Coverage rises to 96%, every test passes, and the reviewer confirms the assertions match the implementation exactly. What has been established?","options":[{"text":"That the implementation is internally consistent, but nothing about whether the fee rule is correct.","correct":true},{"text":"That the fee logic is well tested, since 96% coverage with matching assertions is the standard the industry uses.","correct":false,"tempting":"Accepts coverage plus agreement as evidence, which is the exact combination that a mirrored suite produces automatically."},{"text":"Nothing at all, because tests written alongside an implementation can never provide evidence about it.","correct":false,"tempting":"Over-corrects: tests written with an implementation are weak evidence about intent but still catch crashes, regressions and future edits."},{"text":"That the code is correct for the inputs tested, and the remaining risk is confined to the 4% of uncovered lines.","correct":false,"tempting":"Treats uncovered lines as the residual risk, when the risk here lives in covered lines whose expected values were derived from the code."}]},"prev":{"slug":"diff-size-economics","title":"Why the diff has to be small","url":"https://addaly.com/learn/how-software-changed/diff-size-economics"},"next":{"slug":"safety-that-went-missing","title":"Safety that quietly went missing","url":"https://addaly.com/learn/how-software-changed/safety-that-went-missing"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Reading a test suite you did not write', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/reading-the-tests (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Reading a test suite you did not write","courseTitle":"How Software Development Changed","module":{"index":2,"slug":"reading-what-you-did-not-write","title":"Reading what you did not write"},"answer":null,"keyIdea":"A test’s strength lies in where its expected value came from, so a suite whose assertions restate the implementation will pass through every bug and break on every refactor, and the cheapest way to find out is to break the code on purpose and watch.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"green-means-less-than-it-used-to","runs":[{"text":"Green means less than it used to"}]},{"t":"p","runs":[{"text":"A test suite is the strongest claim a change makes about itself, so it deserves the most sceptical reading in the diff. And it now needs a specific kind of scepticism, because a suite can be large, fast, green, and assert almost nothing."}]},{"t":"p","runs":[{"text":"Here are the five shapes worth recognising on sight. All five appear in hand-written suites too. What changed is how cheap they are to produce in bulk."}]},{"t":"h2","id":"1-the-test-that-asserts-nothing","runs":[{"text":"1. The test that asserts nothing"}]},{"t":"code","lang":"python","label":"Python","text":"def test_process_order():\n    order = make_order(items=3)\n    result = process(order)\n    assert result is not None"},{"t":"p","runs":[{"text":"This passes if "},{"text":"process","code":true},{"text":" returns the string “banana”. It is a smoke test wearing the name of a behaviour test. Smoke tests have value (they catch import errors and crashes) but a suite of forty of them tells you the code runs, not that it works."}]},{"t":"p","runs":[{"text":"The tell is the assertion: "},{"text":"is not None","code":true},{"text":", "},{"text":"assert result","code":true},{"text":", "},{"text":"assert len(x) > 0","code":true},{"text":", "},{"text":"assertTrue(response.ok)","code":true},{"text":"."}]},{"t":"h2","id":"2-the-tautology","runs":[{"text":"2. The tautology"}]},{"t":"code","lang":"javascript","label":"JavaScript","text":"const repo = { save: jest.fn() };\nservice.createUser(repo, { email: \"a@b.com\" });\nexpect(repo.save).toHaveBeenCalled();"},{"t":"p","runs":[{"text":"This asserts that the code did the thing the code does. It will pass whether "},{"text":"save","code":true},{"text":" was called with the right user, the wrong user, or an empty object, and it will keep passing after somebody breaks the email validation entirely."}]},{"t":"p","runs":[{"text":"Mock-call assertions are not worthless (they are the right tool for checking that you did "},{"text":"not","i":true},{"text":" call something, or that you called it exactly once) but a suite composed mainly of them is testing the shape of the implementation rather than its behaviour, which means it will break on every refactor and pass through every bug. That is the worst possible combination."}]},{"t":"h2","id":"3-the-mirror","runs":[{"text":"3. The mirror"}]},{"t":"p","runs":[{"text":"The most dangerous one, and the one that arrives specifically when implementation and tests came from the same reading of the same request."}]},{"t":"code","lang":"python","label":"Python","text":"# implementation\ndef fee(amount):\n    return amount * 0.029 + 30\n\n# test\ndef test_fee():\n    assert fee(1000) == 1000 * 0.029 + 30"},{"t":"p","runs":[{"text":"The expected value is computed by restating the formula. If the formula is wrong, if the fee should have been 2.9% "},{"text":"or","i":true},{"text":" 30p, whichever is greater, the test agrees with the bug, confidently, for ever. Worse, it makes the bug expensive to fix, because correcting the behaviour now breaks a test, and breaking a test feels like causing a regression."}]},{"t":"p","runs":[{"text":"The rule that catches this: "},{"text":"an expected value must come from somewhere other than the code.","b":true},{"text":" The requirement, the spec, a worked example, a regulator’s document, a hand calculation. "},{"text":"assert fee(1000) == 59","code":true},{"text":" is a weaker-looking test and a far stronger one, because a human had to decide that 59 was right."}]},{"t":"h2","id":"4-the-over-mocked-integration","runs":[{"text":"4. The over-mocked integration"}]},{"t":"p","runs":[{"text":"Everything the function touches is replaced by a mock, so the test exercises the function’s control flow and nothing else. It will pass when the database rejects the write, when the API returns 500, when the JSON field is named differently from what the mock returns, and when the migration was never applied."}]},{"t":"p","runs":[{"text":"The question to ask: "},{"text":"what real thing does this test prove works?","b":true},{"text":" If the answer is “the sequence of calls inside this one function”, it is a unit test and it should be labelled as one, and the system needs a small number of tests that touch something real."}]},{"t":"h2","id":"5-the-suite-with-no-failing-history","runs":[{"text":"5. The suite with no failing history"}]},{"t":"p","runs":[{"text":"Open the file’s history. If no test in it has ever been red for a substantive reason, you are looking at decoration."}]},{"t":"code","lang":"bash","label":"Bash","text":"git log --oneline -- tests/test_billing.py | head -20"},{"t":"p","runs":[{"text":"A test that has never failed has never been tested itself. Eight tests you watched fail for the right reason before you made them pass are worth more than three hundred that arrived green."}]},{"t":"h2","id":"the-ninety-second-audit","runs":[{"text":"The ninety-second audit"}]},{"t":"p","runs":[{"text":"When you inherit a suite and want to know what it is worth, break something on purpose. Invert a comparison, delete a validation, return a constant from a core function. Run the suite."}]},{"t":"code","lang":"python","label":"Python","text":"# before\nif amount < 0:\n    raise ValueError(\"negative amount\")\n\n# after — now run the tests\nif amount < -1_000_000:\n    raise ValueError(\"negative amount\")"},{"t":"p","runs":[{"text":"If it stays green, you have learned in ninety seconds what a coverage report would never have told you. Mutation-testing tools ("},{"text":"mutmut","code":true},{"text":" and "},{"text":"cosmic-ray","code":true},{"text":" in Python, Stryker in JavaScript, PIT in Java, all free) automate exactly this, and they are worth running once on a suite you are unsure about, because the number they return is the only honest measure of a suite’s strength."}]},{"t":"p","runs":[{"text":"Coverage, by contrast, measures which lines executed. Every one of the five shapes above scores well on coverage. That is not a flaw in the metric; it is a fact about what it measures, and it is why coverage rose in value as a floor and fell in value as a signal."}]}],"check":{"question":[{"text":"A pull request adds a fee calculation and 40 tests. Coverage rises to 96%, every test passes, and the reviewer confirms the assertions match the implementation exactly. What has been established?"}],"options":[{"text":[{"text":"That the implementation is internally consistent, but nothing about whether the fee rule is correct."}],"correct":true,"tempting":null},{"text":[{"text":"That the fee logic is well tested, since 96% coverage with matching assertions is the standard the industry uses."}],"correct":false,"tempting":[{"text":"Accepts coverage plus agreement as evidence, which is the exact combination that a mirrored suite produces automatically."}]},{"text":[{"text":"Nothing at all, because tests written alongside an implementation can never provide evidence about it."}],"correct":false,"tempting":[{"text":"Over-corrects: tests written with an implementation are weak evidence about intent but still catch crashes, regressions and future edits."}]},{"text":[{"text":"That the code is correct for the inputs tested, and the remaining risk is confined to the 4% of uncovered lines."}],"correct":false,"tempting":[{"text":"Treats uncovered lines as the residual risk, when the risk here lives in covered lines whose expected values were derived from the code."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"diff-size-economics","title":"Why the diff has to be small","minutes":9},"next":{"slug":"safety-that-went-missing","title":"Safety that quietly went missing","minutes":9}}}