Properties, not examples

Why examples stopped being enough

An example test pins one input to one output. total([100, 250]) == 350. It is easy to write, easy to read, and easy to satisfy, including by code that is wrong everywhere except that point.

That last property used to be a minor theoretical objection. It is now the main event, because code that satisfies the visible examples and fails elsewhere is exactly what cheap generation produces. An implementation pattern-matched from a thousand similar functions will handle the shapes those functions handled and diverge at the edges nobody wrote down.

A property test says something that must be true of every input, and then the machine searches for a counterexample. That reverses the economics: cheap for you to state, expensive for wrong code to satisfy by accident.

The four families, with examples

Round-trip. Encoding and decoding must return you to where you started.

from hypothesis import given, strategies as st

@given(st.text())
def test_slug_roundtrip(s):
    assert unslugify(slugify(s)) == s.strip()

This one finds Unicode problems within about four seconds, every time. Serialisation, compression, parsing and formatting, URL encoding, currency formatting: all round-trips.

Idempotence. Applying twice equals applying once.

@given(st.text())
def test_normalise_is_idempotent(s):
    assert normalise(normalise(s)) == normalise(s)

This is the property behind every retry-safe operation, every migration you might have to re-run, and every reconciliation job.

Invariants. Something true before and after, regardless of the operation.

@given(st.lists(st.integers(min_value=0, max_value=10**9)), st.integers())
def test_transfer_conserves_total(balances, amount):
    before = sum(balances)
    after = sum(apply_transfer(balances, amount))
    assert before == after

Money is conserved. A sorted list stays the same length. A queue never loses an item. Invariants are where the domain knowledge goes and they are the most valuable family.

Comparison against a reference. A slow, obviously-correct implementation versus the fast one.

@given(st.lists(st.integers()))
def test_fast_median_matches_naive(xs):
    assume(xs)
    assert fast_median(xs) == sorted(xs)[len(xs)//2] if len(xs) % 2 else True

This is the strongest position available for optimisation work, and it is the same idea as differential testing in the next lesson.

Shrinking is the part that makes it usable

When a property fails, the framework does not hand you the 400-element list that broke it. It repeatedly simplifies the failing input until it cannot be simplified further, and gives you the minimal case.

So instead of debugging [8123, -4, 0, 991, ...] you get [0, 0], and the bug is obvious. Shrinking is what turns a random-input generator into a debugging tool, and it is the reason property testing is worth the setup cost.

The free tooling, all of it

Hypothesis (Python), fast-check (JavaScript and TypeScript), jqwik (Java), PropCheck and StreamData (Elixir), QuickCheck (Haskell, the original), proptest and quickcheck (Rust), gopter and the built-in testing/quick plus Go’s native fuzzing. Every one is free and open source. Installation is one line and the first useful property takes about ten minutes.

Where properties are hard, stated honestly

They are not a universal replacement, and pretending otherwise wastes people’s afternoons.

• Stateful systems. Properties over a sequence of operations against a database or a service are possible (Hypothesis has RuleBasedStateMachine for exactly this) and they are considerably more work than a single-function property.
• Things with no clean invariant. “The email looks right” is not a property. Rendering, layout, natural language output and anything judged by a human resist this technique.
• Slow functions. A property runs a hundred times by default. If one call takes two seconds, that is a three-minute test and it will get deleted.
• Finding the property is the work. Writing @given is trivial. Working out what must always be true about your discount engine is the hard, valuable part, and it is exactly the specification thinking that the next module is about.

The habit worth building

When you add an example test, ask one question: is this example an instance of something more general?

Often it is not, and an example is the right answer. When it is (when the example is “a discount never makes the total negative” or “the parser accepts everything the formatter produces”) write the general form instead. It takes two more minutes, and it is the version that catches the input neither you nor the generator thought of.