{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"verification-is-the-bottleneck","title":"Testing when generating is free","url":"https://addaly.com/learn/how-software-changed/verification-is-the-bottleneck","position":20,"of":87,"minutes":9,"answer":"A test only counts if you have seen it fail for the right reason.","keyIdea":"A test only counts if you have seen it fail for the right reason."},"body":{"format":"markdown","text":"## The scarce thing is the oracle\n\nIf you can produce five candidate implementations in the time it used to take to produce one, the interesting question stops being *how do I write this* and becomes *how do I tell which of these is correct*.\n\nThe name for the thing that answers that is an oracle: any mechanism that says yes or no about behaviour. A test, a type, a database constraint, an assertion, a known-good previous version, a human who understands the domain. Oracles were always the real asset in a codebase. Now they are the only asset that is scarce.\n\n## The trap that catches careful people\n\nAsk for an implementation, then ask for tests for it, and you will get tests that pass.\n\nOf course you will. They were derived from the same reading of the same request. If the implementation misunderstood the requirement, the tests encode the same misunderstanding and assert it confidently. You now have a green suite that certifies the bug and makes it harder to fix, because changing the behaviour breaks tests and breaking tests feels like regression.\n\nA test that asserts wrong behaviour is worse than no test at all. No test leaves you uncertain, which is accurate. A wrong test makes you certain, which is not.\n\nThe practical split: generate the scaffolding, write the assertions. Fixtures, parametrisation, mocks, the boring setup: generating those saves real time and the failure modes are visible. The line that says what the answer should be is yours, and it should come from the requirement rather than from the code.\n\n## Oracles that stay ahead of cheap generation\n\n**Properties instead of examples.** An example test says `total([100, 250]) == 350`. A property says something that must be true of every input:\n\n```python\nfrom hypothesis import given, strategies as st\n\n@given(st.lists(st.integers(min_value=0, max_value=10**9)))\ndef test_total_is_order_independent(amounts):\n    assert total(amounts) == total(list(reversed(amounts)))\n\n@given(st.lists(st.integers(min_value=0, max_value=10**9)))\ndef test_total_never_exceeds_sum_of_parts(amounts):\n    assert total(amounts) <= sum(amounts)\n```\n\nProperties are cheap for you to state and expensive for wrong code to satisfy by accident. Round-trip (`decode(encode(x)) == x`), idempotence (applying twice equals applying once), order independence, invariants that must hold before and after. Hypothesis in Python, fast-check in JavaScript, and equivalents almost everywhere.\n\n**Differential testing.** Run the old implementation and the new one against the same inputs and compare. This is the strongest position you can be in, and it makes refactoring the single best-suited task for these tools: the requirement is exactly \"behave identically\", and you have a perfect oracle sitting in git history.\n\n**Real traffic.** Replay a sample of production inputs through both versions and diff the outputs. A day of real requests finds edge cases no one would think to write down: the customer whose name is a single character, the order with 4,000 line items, the address field containing a newline.\n\n**Cheap oracles that always run.** Types, non-null constraints, foreign keys, check constraints, assertions at function boundaries. A `CHECK (amount >= 0)` in the database catches a class of bugs forever, for one line, regardless of what generated the code above it.\n\n## The green suite tells you less than you think\n\nHere is a question worth asking about any test: **have I seen this fail for the right reason?**\n\nA suite of 340 generated tests that runs in 12 seconds and has never been red is decorative. Eight tests you wrote, each of which you watched fail before you made it pass, are load-bearing. That is what test-driven development was buying, and it is worth more now than it was, because a passing test is now much cheaper to produce than a correct one.\n\nCheap way to check a suite you inherited: break something on purpose. Invert a comparison, delete a validation, return a constant. Run the tests. If the suite stays green, you have learned something important about it in ninety seconds. Mutation testing tools automate this idea, but the manual version is enough to start.\n\n## Budget for it\n\nIf you take one operational thing from this lesson: verification capacity is now the number that determines your throughput, so it should appear in planning as an explicit line rather than as a residue of whatever time is left.\n\nWhen someone says a feature will take two days, the useful follow-up is not \"can it be one\". It is \"how will we know it works, who will confirm that, and is that person free\"."},"check":{"question":"A teammate ships a change with 60 new tests, all passing, taking coverage from 62% to 88%. What is the most useful thing to ask before treating this as increased confidence?","options":[{"text":"Whether 88% is high enough, given that the remaining 12% is untested and could contain anything.","correct":false,"tempting":"Chases a coverage number, which measures which lines ran, not whether anything would have objected if they behaved wrongly."},{"text":"Whether the tests run fast enough to stay in the pull request pipeline as the suite keeps growing.","correct":false,"tempting":"A real long-term concern, but speed of a suite that certifies nothing just means being wrong more quickly."},{"text":"Whether the tests are unit or integration, since integration tests give more confidence per test.","correct":false,"tempting":"Argues about the level of the test while skipping the prior question of whether any of them can distinguish correct behaviour from incorrect."},{"text":"Whether any of them would fail if the behaviour were broken: pick one, break the code it covers, and watch what happens.","correct":true}]},"prev":{"slug":"machine-assisted-review","title":"Letting a machine help you review","url":"https://addaly.com/learn/how-software-changed/machine-assisted-review"},"next":{"slug":"properties-not-examples","title":"Properties, not examples","url":"https://addaly.com/learn/how-software-changed/properties-not-examples"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Testing when generating is free', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/verification-is-the-bottleneck (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Testing when generating is free","courseTitle":"How Software Development Changed","module":{"index":3,"slug":"oracles-and-verification","title":"Building oracles that stay ahead"},"answer":null,"keyIdea":"A test only counts if you have seen it fail for the right reason.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-scarce-thing-is-the-oracle","runs":[{"text":"The scarce thing is the oracle"}]},{"t":"p","runs":[{"text":"If you can produce five candidate implementations in the time it used to take to produce one, the interesting question stops being "},{"text":"how do I write this","i":true},{"text":" and becomes "},{"text":"how do I tell which of these is correct","i":true},{"text":"."}]},{"t":"p","runs":[{"text":"The name for the thing that answers that is an oracle: any mechanism that says yes or no about behaviour. A test, a type, a database constraint, an assertion, a known-good previous version, a human who understands the domain. Oracles were always the real asset in a codebase. Now they are the only asset that is scarce."}]},{"t":"h2","id":"the-trap-that-catches-careful-people","runs":[{"text":"The trap that catches careful people"}]},{"t":"p","runs":[{"text":"Ask for an implementation, then ask for tests for it, and you will get tests that pass."}]},{"t":"p","runs":[{"text":"Of course you will. They were derived from the same reading of the same request. If the implementation misunderstood the requirement, the tests encode the same misunderstanding and assert it confidently. You now have a green suite that certifies the bug and makes it harder to fix, because changing the behaviour breaks tests and breaking tests feels like regression."}]},{"t":"p","runs":[{"text":"A test that asserts wrong behaviour is worse than no test at all. No test leaves you uncertain, which is accurate. A wrong test makes you certain, which is not."}]},{"t":"p","runs":[{"text":"The practical split: generate the scaffolding, write the assertions. Fixtures, parametrisation, mocks, the boring setup: generating those saves real time and the failure modes are visible. The line that says what the answer should be is yours, and it should come from the requirement rather than from the code."}]},{"t":"h2","id":"oracles-that-stay-ahead-of-cheap-generation","runs":[{"text":"Oracles that stay ahead of cheap generation"}]},{"t":"p","runs":[{"text":"Properties instead of examples.","b":true},{"text":" An example test says "},{"text":"total([100, 250]) == 350","code":true},{"text":". A property says something that must be true of every input:"}]},{"t":"code","lang":"python","label":"Python","text":"from hypothesis import given, strategies as st\n\n@given(st.lists(st.integers(min_value=0, max_value=10**9)))\ndef test_total_is_order_independent(amounts):\n    assert total(amounts) == total(list(reversed(amounts)))\n\n@given(st.lists(st.integers(min_value=0, max_value=10**9)))\ndef test_total_never_exceeds_sum_of_parts(amounts):\n    assert total(amounts) <= sum(amounts)"},{"t":"p","runs":[{"text":"Properties are cheap for you to state and expensive for wrong code to satisfy by accident. Round-trip ("},{"text":"decode(encode(x)) == x","code":true},{"text":"), idempotence (applying twice equals applying once), order independence, invariants that must hold before and after. Hypothesis in Python, fast-check in JavaScript, and equivalents almost everywhere."}]},{"t":"p","runs":[{"text":"Differential testing.","b":true},{"text":" Run the old implementation and the new one against the same inputs and compare. This is the strongest position you can be in, and it makes refactoring the single best-suited task for these tools: the requirement is exactly “behave identically”, and you have a perfect oracle sitting in git history."}]},{"t":"p","runs":[{"text":"Real traffic.","b":true},{"text":" Replay a sample of production inputs through both versions and diff the outputs. A day of real requests finds edge cases no one would think to write down: the customer whose name is a single character, the order with 4,000 line items, the address field containing a newline."}]},{"t":"p","runs":[{"text":"Cheap oracles that always run.","b":true},{"text":" Types, non-null constraints, foreign keys, check constraints, assertions at function boundaries. A "},{"text":"CHECK (amount >= 0)","code":true},{"text":" in the database catches a class of bugs forever, for one line, regardless of what generated the code above it."}]},{"t":"h2","id":"the-green-suite-tells-you-less-than-you-think","runs":[{"text":"The green suite tells you less than you think"}]},{"t":"p","runs":[{"text":"Here is a question worth asking about any test: "},{"text":"have I seen this fail for the right reason?","b":true}]},{"t":"p","runs":[{"text":"A suite of 340 generated tests that runs in 12 seconds and has never been red is decorative. Eight tests you wrote, each of which you watched fail before you made it pass, are load-bearing. That is what test-driven development was buying, and it is worth more now than it was, because a passing test is now much cheaper to produce than a correct one."}]},{"t":"p","runs":[{"text":"Cheap way to check a suite you inherited: break something on purpose. Invert a comparison, delete a validation, return a constant. Run the tests. If the suite stays green, you have learned something important about it in ninety seconds. Mutation testing tools automate this idea, but the manual version is enough to start."}]},{"t":"h2","id":"budget-for-it","runs":[{"text":"Budget for it"}]},{"t":"p","runs":[{"text":"If you take one operational thing from this lesson: verification capacity is now the number that determines your throughput, so it should appear in planning as an explicit line rather than as a residue of whatever time is left."}]},{"t":"p","runs":[{"text":"When someone says a feature will take two days, the useful follow-up is not “can it be one”. It is “how will we know it works, who will confirm that, and is that person free”."}]}],"check":{"question":[{"text":"A teammate ships a change with 60 new tests, all passing, taking coverage from 62% to 88%. What is the most useful thing to ask before treating this as increased confidence?"}],"options":[{"text":[{"text":"Whether 88% is high enough, given that the remaining 12% is untested and could contain anything."}],"correct":false,"tempting":[{"text":"Chases a coverage number, which measures which lines ran, not whether anything would have objected if they behaved wrongly."}]},{"text":[{"text":"Whether the tests run fast enough to stay in the pull request pipeline as the suite keeps growing."}],"correct":false,"tempting":[{"text":"A real long-term concern, but speed of a suite that certifies nothing just means being wrong more quickly."}]},{"text":[{"text":"Whether the tests are unit or integration, since integration tests give more confidence per test."}],"correct":false,"tempting":[{"text":"Argues about the level of the test while skipping the prior question of whether any of them can distinguish correct behaviour from incorrect."}]},{"text":[{"text":"Whether any of them would fail if the behaviour were broken: pick one, break the code it covers, and watch what happens."}],"correct":true,"tempting":null}],"order":[2,1,3,0],"why":null},"prev":{"slug":"machine-assisted-review","title":"Letting a machine help you review","minutes":9},"next":{"slug":"properties-not-examples","title":"Properties, not examples","minutes":10}}}