The tests that need no model at all

Cheap tests catch the expensive bugs

Evaluating answers costs money, takes minutes, and is noisy. Checking the request costs nothing, takes milliseconds, and is exact.

And a surprising share of production incidents in this area are request-shaped rather than answer-shaped: a timestamp that crept into the cached prefix, a delimiter that stopped being escaped, an instruction block that silently doubled because two code paths both added it. None of these needs a model to detect. All of them are invisible in code review.

If you made the assembler a pure function (same inputs, same bytes out, no clock and no network) this lesson is a morning’s work. If you did not, this is the argument for doing it.

The snapshot diff, which is worth the other six together

Render the request for a fixed set of inputs, commit the result, and fail the build when it changes unexpectedly.

def test_request_snapshot(snapshot):
    req = assemble(fixture_query, fixture_passages, config=CONFIG_V7, now=FIXED_TIME)
    snapshot.assert_match(render(req), "request_standard_query.txt")

What this gives you is a diff of the prompt, and it changes the character of prompt work entirely. Someone edits a sentence for clarity; the diff shows they also moved the block above the cache breakpoint. Someone upgrades a library; the diff shows the chat template now emits a different tool serialisation. Someone adds a field to a chunk header; the diff shows it costs 40 tokens and is repeated eight times per request.

None of that is visible in the source diff, because the source diff shows a template while the snapshot shows the artefact. Review the artefact.

Keep three or four snapshots covering different shapes (a plain query, a long conversation, a tool-calling turn, a turn with untrusted content) and accept updates deliberately rather than regenerating them by reflex when the build goes red.

The assertions

Budget. The assembled request fits, with the output reserve subtracted first, at the largest realistic inputs rather than the fixture ones.

assert count(req) + RESERVE <= LIMIT, f"{count(req)} + {RESERVE} > {LIMIT}"

Determinism. Assemble twice from identical inputs and compare bytes. This catches dictionary ordering, set iteration, an unsorted tool list, an unseeded shuffle: every one of which quietly destroys the prefix cache and none of which raises an error.

Prefix stability. Hash everything up to the cache breakpoint. Assert it is unchanged across requests that differ only in the user’s message. A test that fails here has found a cost regression before it reached the bill.

Escaping. For every untrusted block, assert the closing delimiter does not appear inside it. The security module explained why; this is the check that keeps it true after someone refactors.

Provenance. Every block carries a trust label. Fail on a missing one rather than defaulting, so a new source added later cannot slip in unlabelled.

Citation handles. Every handle rendered into the window resolves to a passage in the manifest, and no two passages share a handle. Duplicate handles are an unglamorous bug that makes every downstream citation check meaningless.

No secrets in the prefix. Assert the cached block contains no pattern matching your key formats and no user-identifying fields. Cached prefixes are shared infrastructure and are the wrong place for anything per-user.

Property tests for the packer

The packing and truncation code is where off-by-one errors live, and it is pleasantly testable with generated inputs.

Generate random passage sets and assert the invariants hold every time: the result never exceeds the budget; no chunk is included partially; a pinned passage is always present; the order of equal-scoring items is stable. hypothesis does this in Python for free, and it finds the empty-list case and the single-oversized-chunk case within seconds, which are the two that reach production.

What these cannot tell you

They verify the request is what you meant. They say nothing about whether what you meant is any good.

A request can pass every check here and produce poor answers, because the instruction is unclear or the retrieved passages are wrong. That is what the golden set is for, and the division of labour is worth keeping crisp: offline checks defend the mechanism; evaluation defends the outcome. Teams that have only the second spend their time re-discovering mechanical bugs through expensive noisy experiments, which is a slow and demoralising way to find a stray timestamp.