{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"differential-testing","title":"The old version is your best oracle","url":"https://addaly.com/learn/how-software-changed/differential-testing","position":22,"of":87,"minutes":10,"answer":"When the requirement is that behaviour must not change, the previous version is a perfect oracle.","keyIdea":"When the requirement is that behaviour must not change, the previous version is a perfect oracle, so refactors and migrations should be verified by running both implementations over real traffic rather than by reading the diff."},"body":{"format":"markdown","text":"## The strongest position you can be in\n\nWhen the required behaviour is \"exactly what it did before\", verification is close to free. You have a reference implementation, it is in git, and you can run both.\n\nThis is why refactoring is the best-suited task for generated code and why it is undersold in every discussion of these tools. The specification is perfect, the oracle is perfect, and the failure mode, a subtle semantic change, is precisely what differential testing catches and human review does not.\n\n## The basic shape\n\n```python\nimport old_billing, new_billing\nfrom hypothesis import given, strategies as st\n\n@given(invoices())          # your own strategy for realistic invoices\ndef test_same_behaviour(inv):\n    assert new_billing.total(inv) == old_billing.total(inv)\n```\n\nKeep the old module beside the new one for the length of the migration, run this in CI, and delete it when you are confident. Two hundred lines of temporary scaffolding is a small price for knowing.\n\nWhere the function is not pure, compare effects instead of return values: the rows written, the requests made, the messages emitted. Capture them and diff.\n\n## Real traffic beats generated inputs\n\nA day of production requests contains edge cases nobody would invent: the customer whose name is one character, the order with 4,000 line items, the address field with an embedded newline, the request that arrives twice within the same millisecond.\n\n**Log-and-replay** is the cheapest version. Sample inputs from production, store them, run both implementations over the file, diff the outputs.\n\n```bash\n# 10,000 sampled requests, both versions, structural diff\njq -c '.' sampled_requests.jsonl \\\n  | parallel --pipe -N100 'python run_both.py' \\\n  > diffs.jsonl\nwc -l diffs.jsonl\n```\n\n**Shadow traffic** is the stronger version. Send every live request to both implementations, serve the response from the old one, and record where they disagree. The new implementation is exercised at full production volume and scale with zero user-visible risk. This is how large migrations are done safely and it is available to small teams too: a few lines in a proxy or a middleware.\n\nOne rule: **the shadow path must not have side effects.** Writing to the database twice, sending two emails, charging a card twice. Either point the shadow at a copy or make the write path a no-op, and check this before you enable it rather than afterwards.\n\n## Golden files, and their trap\n\nApproval testing (run the code, save the output, commit it, fail when it changes) is the cheapest differential technique and the easiest to get wrong.\n\n```bash\npytest --snapshot-update      # regenerate\ngit diff tests/__snapshots__  # review as a human\n```\n\nThe trap is that regenerating is one flag away. Under time pressure a failing snapshot gets updated rather than investigated, and the snapshot is now a record of the bug. Two things prevent it: the updated file must appear in the diff and be reviewed like code, and the pipeline must never update snapshots automatically. A team that adopts approval testing without that discipline has adopted a slower way of doing nothing.\n\n## Handling the things that legitimately differ\n\nMost differential comparisons fail on the first run for reasons that are not bugs. Deal with them explicitly rather than by loosening the comparison.\n\n- **Time.** Inject a clock. A function that calls `datetime.now()` internally cannot be differentially tested and should not be written that way regardless.\n- **Randomness.** Seed it, or inject the generator.\n- **Ordering.** Dictionary iteration, set iteration, `GROUP BY` without `ORDER BY`, parallel result collection. Normalise before comparing (sort the output, canonicalise the JSON key order) rather than declaring the difference acceptable.\n- **IDs and timestamps in payloads.** Strip them, or replace with a placeholder, in one shared normalisation function so the rule is visible.\n- **Floating point.** Compare with a tolerance you chose and can defend, not with the default of whatever library you are using. And if this is money, the real bug is the float.\n\nEach of these is a place where the temptation is to widen the comparison until it passes. Every widening is a hole, and holes accumulate. Write the normalisation, name it, and keep the comparison exact.\n\n## What it cannot do\n\nDifferential testing verifies that the new thing matches the old thing. It says nothing about whether the old thing was right.\n\nSo it is exactly the correct tool for a refactor, a rewrite, a language migration, a performance optimisation, or replacing a library. It is the wrong tool for a behaviour change, and the most common misuse is running it during a change that was supposed to fix a bug, then \"accepting\" the differences one by one until the fix is indistinguishable from a regression.\n\nWhen behaviour is meant to change, split the work: one change that is provably identical, verified this way, and a second, tiny change that alters behaviour and is reviewed by a human. That split is worth doing for its own sake and this is one more reason for it."},"check":{"question":"A team shadow-runs a rewritten pricing service against live traffic. Outputs match on 99.4% of requests. They investigate the 0.6% and find all of them involve orders placed within a second of a rate change. What should they conclude?","options":[{"text":"The rewrite is safe to ship, since a 0.6% divergence confined to a rare timing window is within acceptable tolerance.","correct":false,"tempting":"Treats rarity as safety; a rate-boundary divergence in pricing is a money defect whose frequency says nothing about its cost."},{"text":"The comparison is invalid because time-dependent behaviour cannot be shadow-tested, so a different verification approach is needed.","correct":false,"tempting":"Discards a method that worked: the shadow run is precisely what surfaced the timing window in the first place."},{"text":"The divergence is a real behavioural difference at a boundary, and shipping requires deciding which version is correct.","correct":true},{"text":"The old service is the reference, so the new one should be adjusted until the divergence disappears and the match reaches 100%.","correct":false,"tempting":"Assumes the reference is correct, which differential testing never establishes; the old service may be the one mishandling the boundary."}]},"prev":{"slug":"properties-not-examples","title":"Properties, not examples","url":"https://addaly.com/learn/how-software-changed/properties-not-examples"},"next":{"slug":"types-as-always-on-oracles","title":"The type checker got more valuable","url":"https://addaly.com/learn/how-software-changed/types-as-always-on-oracles"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The old version is your best oracle', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/differential-testing (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The old version is your best oracle","courseTitle":"How Software Development Changed","module":{"index":3,"slug":"oracles-and-verification","title":"Building oracles that stay ahead"},"answer":null,"keyIdea":"When the requirement is that behaviour must not change, the previous version is a perfect oracle, so refactors and migrations should be verified by running both implementations over real traffic rather than by reading the diff.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-strongest-position-you-can-be-in","runs":[{"text":"The strongest position you can be in"}]},{"t":"p","runs":[{"text":"When the required behaviour is “exactly what it did before”, verification is close to free. You have a reference implementation, it is in git, and you can run both."}]},{"t":"p","runs":[{"text":"This is why refactoring is the best-suited task for generated code and why it is undersold in every discussion of these tools. The specification is perfect, the oracle is perfect, and the failure mode, a subtle semantic change, is precisely what differential testing catches and human review does not."}]},{"t":"h2","id":"the-basic-shape","runs":[{"text":"The basic shape"}]},{"t":"code","lang":"python","label":"Python","text":"import old_billing, new_billing\nfrom hypothesis import given, strategies as st\n\n@given(invoices())          # your own strategy for realistic invoices\ndef test_same_behaviour(inv):\n    assert new_billing.total(inv) == old_billing.total(inv)"},{"t":"p","runs":[{"text":"Keep the old module beside the new one for the length of the migration, run this in CI, and delete it when you are confident. Two hundred lines of temporary scaffolding is a small price for knowing."}]},{"t":"p","runs":[{"text":"Where the function is not pure, compare effects instead of return values: the rows written, the requests made, the messages emitted. Capture them and diff."}]},{"t":"h2","id":"real-traffic-beats-generated-inputs","runs":[{"text":"Real traffic beats generated inputs"}]},{"t":"p","runs":[{"text":"A day of production requests contains edge cases nobody would invent: the customer whose name is one character, the order with 4,000 line items, the address field with an embedded newline, the request that arrives twice within the same millisecond."}]},{"t":"p","runs":[{"text":"Log-and-replay","b":true},{"text":" is the cheapest version. Sample inputs from production, store them, run both implementations over the file, diff the outputs."}]},{"t":"code","lang":"bash","label":"Bash","text":"# 10,000 sampled requests, both versions, structural diff\njq -c '.' sampled_requests.jsonl \\\n  | parallel --pipe -N100 'python run_both.py' \\\n  > diffs.jsonl\nwc -l diffs.jsonl"},{"t":"p","runs":[{"text":"Shadow traffic","b":true},{"text":" is the stronger version. Send every live request to both implementations, serve the response from the old one, and record where they disagree. The new implementation is exercised at full production volume and scale with zero user-visible risk. This is how large migrations are done safely and it is available to small teams too: a few lines in a proxy or a middleware."}]},{"t":"p","runs":[{"text":"One rule: "},{"text":"the shadow path must not have side effects.","b":true},{"text":" Writing to the database twice, sending two emails, charging a card twice. Either point the shadow at a copy or make the write path a no-op, and check this before you enable it rather than afterwards."}]},{"t":"h2","id":"golden-files-and-their-trap","runs":[{"text":"Golden files, and their trap"}]},{"t":"p","runs":[{"text":"Approval testing (run the code, save the output, commit it, fail when it changes) is the cheapest differential technique and the easiest to get wrong."}]},{"t":"code","lang":"bash","label":"Bash","text":"pytest --snapshot-update      # regenerate\ngit diff tests/__snapshots__  # review as a human"},{"t":"p","runs":[{"text":"The trap is that regenerating is one flag away. Under time pressure a failing snapshot gets updated rather than investigated, and the snapshot is now a record of the bug. Two things prevent it: the updated file must appear in the diff and be reviewed like code, and the pipeline must never update snapshots automatically. A team that adopts approval testing without that discipline has adopted a slower way of doing nothing."}]},{"t":"h2","id":"handling-the-things-that-legitimately-differ","runs":[{"text":"Handling the things that legitimately differ"}]},{"t":"p","runs":[{"text":"Most differential comparisons fail on the first run for reasons that are not bugs. Deal with them explicitly rather than by loosening the comparison."}]},{"t":"ul","items":[[{"text":"Time.","b":true},{"text":" Inject a clock. A function that calls "},{"text":"datetime.now()","code":true},{"text":" internally cannot be differentially tested and should not be written that way regardless."}],[{"text":"Randomness.","b":true},{"text":" Seed it, or inject the generator."}],[{"text":"Ordering.","b":true},{"text":" Dictionary iteration, set iteration, "},{"text":"GROUP BY","code":true},{"text":" without "},{"text":"ORDER BY","code":true},{"text":", parallel result collection. Normalise before comparing (sort the output, canonicalise the JSON key order) rather than declaring the difference acceptable."}],[{"text":"IDs and timestamps in payloads.","b":true},{"text":" Strip them, or replace with a placeholder, in one shared normalisation function so the rule is visible."}],[{"text":"Floating point.","b":true},{"text":" Compare with a tolerance you chose and can defend, not with the default of whatever library you are using. And if this is money, the real bug is the float."}]]},{"t":"p","runs":[{"text":"Each of these is a place where the temptation is to widen the comparison until it passes. Every widening is a hole, and holes accumulate. Write the normalisation, name it, and keep the comparison exact."}]},{"t":"h2","id":"what-it-cannot-do","runs":[{"text":"What it cannot do"}]},{"t":"p","runs":[{"text":"Differential testing verifies that the new thing matches the old thing. It says nothing about whether the old thing was right."}]},{"t":"p","runs":[{"text":"So it is exactly the correct tool for a refactor, a rewrite, a language migration, a performance optimisation, or replacing a library. It is the wrong tool for a behaviour change, and the most common misuse is running it during a change that was supposed to fix a bug, then “accepting” the differences one by one until the fix is indistinguishable from a regression."}]},{"t":"p","runs":[{"text":"When behaviour is meant to change, split the work: one change that is provably identical, verified this way, and a second, tiny change that alters behaviour and is reviewed by a human. That split is worth doing for its own sake and this is one more reason for it."}]}],"check":{"question":[{"text":"A team shadow-runs a rewritten pricing service against live traffic. Outputs match on 99.4% of requests. They investigate the 0.6% and find all of them involve orders placed within a second of a rate change. What should they conclude?"}],"options":[{"text":[{"text":"The rewrite is safe to ship, since a 0.6% divergence confined to a rare timing window is within acceptable tolerance."}],"correct":false,"tempting":[{"text":"Treats rarity as safety; a rate-boundary divergence in pricing is a money defect whose frequency says nothing about its cost."}]},{"text":[{"text":"The comparison is invalid because time-dependent behaviour cannot be shadow-tested, so a different verification approach is needed."}],"correct":false,"tempting":[{"text":"Discards a method that worked: the shadow run is precisely what surfaced the timing window in the first place."}]},{"text":[{"text":"The divergence is a real behavioural difference at a boundary, and shipping requires deciding which version is correct."}],"correct":true,"tempting":null},{"text":[{"text":"The old service is the reference, so the new one should be adjusted until the divergence disappears and the match reaches 100%."}],"correct":false,"tempting":[{"text":"Assumes the reference is correct, which differential testing never establishes; the old service may be the one mishandling the boundary."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"properties-not-examples","title":"Properties, not examples","minutes":10},"next":{"slug":"types-as-always-on-oracles","title":"The type checker got more valuable","minutes":9}}}