The old version is your best oracle

The strongest position you can be in

When the required behaviour is “exactly what it did before”, verification is close to free. You have a reference implementation, it is in git, and you can run both.

This is why refactoring is the best-suited task for generated code and why it is undersold in every discussion of these tools. The specification is perfect, the oracle is perfect, and the failure mode, a subtle semantic change, is precisely what differential testing catches and human review does not.

The basic shape

import old_billing, new_billing
from hypothesis import given, strategies as st

@given(invoices())          # your own strategy for realistic invoices
def test_same_behaviour(inv):
    assert new_billing.total(inv) == old_billing.total(inv)

Keep the old module beside the new one for the length of the migration, run this in CI, and delete it when you are confident. Two hundred lines of temporary scaffolding is a small price for knowing.

Where the function is not pure, compare effects instead of return values: the rows written, the requests made, the messages emitted. Capture them and diff.

Real traffic beats generated inputs

A day of production requests contains edge cases nobody would invent: the customer whose name is one character, the order with 4,000 line items, the address field with an embedded newline, the request that arrives twice within the same millisecond.

Log-and-replay is the cheapest version. Sample inputs from production, store them, run both implementations over the file, diff the outputs.

# 10,000 sampled requests, both versions, structural diff
jq -c '.' sampled_requests.jsonl \
  | parallel --pipe -N100 'python run_both.py' \
  > diffs.jsonl
wc -l diffs.jsonl

Shadow traffic is the stronger version. Send every live request to both implementations, serve the response from the old one, and record where they disagree. The new implementation is exercised at full production volume and scale with zero user-visible risk. This is how large migrations are done safely and it is available to small teams too: a few lines in a proxy or a middleware.

One rule: the shadow path must not have side effects. Writing to the database twice, sending two emails, charging a card twice. Either point the shadow at a copy or make the write path a no-op, and check this before you enable it rather than afterwards.

Golden files, and their trap

Approval testing (run the code, save the output, commit it, fail when it changes) is the cheapest differential technique and the easiest to get wrong.

pytest --snapshot-update      # regenerate
git diff tests/__snapshots__  # review as a human

The trap is that regenerating is one flag away. Under time pressure a failing snapshot gets updated rather than investigated, and the snapshot is now a record of the bug. Two things prevent it: the updated file must appear in the diff and be reviewed like code, and the pipeline must never update snapshots automatically. A team that adopts approval testing without that discipline has adopted a slower way of doing nothing.

Handling the things that legitimately differ

Most differential comparisons fail on the first run for reasons that are not bugs. Deal with them explicitly rather than by loosening the comparison.

• Time. Inject a clock. A function that calls datetime.now() internally cannot be differentially tested and should not be written that way regardless.
• Randomness. Seed it, or inject the generator.
• Ordering. Dictionary iteration, set iteration, GROUP BY without ORDER BY, parallel result collection. Normalise before comparing (sort the output, canonicalise the JSON key order) rather than declaring the difference acceptable.
• IDs and timestamps in payloads. Strip them, or replace with a placeholder, in one shared normalisation function so the rule is visible.
• Floating point. Compare with a tolerance you chose and can defend, not with the default of whatever library you are using. And if this is money, the real bug is the float.

Each of these is a place where the temptation is to widen the comparison until it passes. Every widening is a hole, and holes accumulate. Write the normalisation, name it, and keep the comparison exact.

What it cannot do

Differential testing verifies that the new thing matches the old thing. It says nothing about whether the old thing was right.

So it is exactly the correct tool for a refactor, a rewrite, a language migration, a performance optimisation, or replacing a library. It is the wrong tool for a behaviour change, and the most common misuse is running it during a change that was supposed to fix a bug, then “accepting” the differences one by one until the fix is indistinguishable from a regression.

When behaviour is meant to change, split the work: one change that is provably identical, verified this way, and a second, tiny change that alters behaviour and is reviewed by a human. That split is worth doing for its own sake and this is one more reason for it.