Testing when generating is free

The scarce thing is the oracle

If you can produce five candidate implementations in the time it used to take to produce one, the interesting question stops being *how do I write this* and becomes *how do I tell which of these is correct*.

The name for the thing that answers that is an oracle: any mechanism that says yes or no about behaviour. A test, a type, a database constraint, an assertion, a known-good previous version, a human who understands the domain. Oracles were always the real asset in a codebase. Now they are the only asset that is scarce.

The trap that catches careful people

Ask for an implementation, then ask for tests for it, and you will get tests that pass.

Of course you will. They were derived from the same reading of the same request. If the implementation misunderstood the requirement, the tests encode the same misunderstanding and assert it confidently. You now have a green suite that certifies the bug and makes it harder to fix, because changing the behaviour breaks tests and breaking tests feels like regression.

A test that asserts wrong behaviour is worse than no test at all. No test leaves you uncertain, which is accurate. A wrong test makes you certain, which is not.

The practical split: generate the scaffolding, write the assertions. Fixtures, parametrisation, mocks, the boring setup — generating those saves real time and the failure modes are visible. The line that says what the answer should be is yours, and it should come from the requirement rather than from the code.

Oracles that stay ahead of cheap generation

Properties instead of examples. An example test says total([100, 250]) == 350. A property says something that must be true of every input:

from hypothesis import given, strategies as st

@given(st.lists(st.integers(min_value=0, max_value=10**9)))
def test_total_is_order_independent(amounts):
    assert total(amounts) == total(list(reversed(amounts)))

@given(st.lists(st.integers(min_value=0, max_value=10**9)))
def test_total_never_exceeds_sum_of_parts(amounts):
    assert total(amounts) <= sum(amounts)

Properties are cheap for you to state and expensive for wrong code to satisfy by accident. Round-trip (decode(encode(x)) == x), idempotence (applying twice equals applying once), order independence, invariants that must hold before and after. Hypothesis in Python, fast-check in JavaScript, and equivalents almost everywhere.

Differential testing. Run the old implementation and the new one against the same inputs and compare. This is the strongest position you can be in, and it makes refactoring the single best-suited task for these tools: the requirement is exactly "behave identically", and you have a perfect oracle sitting in git history.

Real traffic. Replay a sample of production inputs through both versions and diff the outputs. A day of real requests finds edge cases no one would think to write down: the customer whose name is a single character, the order with 4,000 line items, the address field containing a newline.

Cheap oracles that always run. Types, non-null constraints, foreign keys, check constraints, assertions at function boundaries. A CHECK (amount >= 0) in the database catches a class of bugs forever, for one line, regardless of what generated the code above it.

The green suite tells you less than you think

Here is a question worth asking about any test: have I seen this fail for the right reason?

A suite of 340 generated tests that runs in 12 seconds and has never been red is decorative. Eight tests you wrote, each of which you watched fail before you made it pass, are load-bearing. That is what test-driven development was really buying, and it is worth more now than it was, because a passing test is now much cheaper to produce than a correct one.

Cheap way to check a suite you inherited: break something on purpose. Invert a comparison, delete a validation, return a constant. Run the tests. If the suite stays green, you have learned something important about it in ninety seconds. Mutation testing tools automate this idea, but the manual version is enough to start.

Budget for it

If you take one operational thing from this lesson: verification capacity is now the number that determines your throughput, so it should appear in planning as an explicit line rather than as a residue of whatever time is left.

When someone says a feature will take two days, the useful follow-up is not "can it be one". It is "how will we know it works, who will confirm that, and is that person free".