{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"offline-checks-on-the-assembler","title":"The tests that need no model at all","url":"https://addaly.com/learn/context-engineering/offline-checks-on-the-assembler","position":80,"of":87,"minutes":8,"answer":"Most context bugs are properties of the assembled request rather than of the model's answer, so they can be caught by assertions and a snapshot diff that run in milliseconds, cost nothing.","keyIdea":"Most context bugs are properties of the assembled request rather than of the model's answer, so they can be caught by assertions and a snapshot diff that run in milliseconds, cost nothing, and belong in continuous integration."},"body":{"format":"markdown","text":"## Cheap tests catch the expensive bugs\n\nEvaluating answers costs money, takes minutes, and is noisy. Checking the request costs nothing, takes milliseconds, and is exact.\n\nAnd a surprising share of production incidents in this area are request-shaped rather than answer-shaped: a timestamp that crept into the cached prefix, a delimiter that stopped being escaped, an instruction block that silently doubled because two code paths both added it. None of these needs a model to detect. All of them are invisible in code review.\n\nIf you made the assembler a pure function (same inputs, same bytes out, no clock and no network) this lesson is a morning's work. If you did not, this is the argument for doing it.\n\n## The snapshot diff, which is worth the other six together\n\nRender the request for a fixed set of inputs, commit the result, and fail the build when it changes unexpectedly.\n\n```python\ndef test_request_snapshot(snapshot):\n    req = assemble(fixture_query, fixture_passages, config=CONFIG_V7, now=FIXED_TIME)\n    snapshot.assert_match(render(req), \"request_standard_query.txt\")\n```\n\nWhat this gives you is a **diff of the prompt**, and it changes the character of prompt work entirely. Someone edits a sentence for clarity; the diff shows they also moved the block above the cache breakpoint. Someone upgrades a library; the diff shows the chat template now emits a different tool serialisation. Someone adds a field to a chunk header; the diff shows it costs 40 tokens and is repeated eight times per request.\n\nNone of that is visible in the source diff, because the source diff shows a template while the snapshot shows the artefact. Review the artefact.\n\nKeep three or four snapshots covering different shapes (a plain query, a long conversation, a tool-calling turn, a turn with untrusted content) and accept updates deliberately rather than regenerating them by reflex when the build goes red.\n\n## The assertions\n\n**Budget.** The assembled request fits, with the output reserve subtracted first, at the largest realistic inputs rather than the fixture ones.\n\n```python\nassert count(req) + RESERVE <= LIMIT, f\"{count(req)} + {RESERVE} > {LIMIT}\"\n```\n\n**Determinism.** Assemble twice from identical inputs and compare bytes. This catches dictionary ordering, set iteration, an unsorted tool list, an unseeded shuffle: every one of which quietly destroys the prefix cache and none of which raises an error.\n\n**Prefix stability.** Hash everything up to the cache breakpoint. Assert it is unchanged across requests that differ only in the user's message. A test that fails here has found a cost regression before it reached the bill.\n\n**Escaping.** For every untrusted block, assert the closing delimiter does not appear inside it. The security module explained why; this is the check that keeps it true after someone refactors.\n\n**Provenance.** Every block carries a trust label. Fail on a missing one rather than defaulting, so a new source added later cannot slip in unlabelled.\n\n**Citation handles.** Every handle rendered into the window resolves to a passage in the manifest, and no two passages share a handle. Duplicate handles are an unglamorous bug that makes every downstream citation check meaningless.\n\n**No secrets in the prefix.** Assert the cached block contains no pattern matching your key formats and no user-identifying fields. Cached prefixes are shared infrastructure and are the wrong place for anything per-user.\n\n## Property tests for the packer\n\nThe packing and truncation code is where off-by-one errors live, and it is pleasantly testable with generated inputs.\n\nGenerate random passage sets and assert the invariants hold every time: the result never exceeds the budget; no chunk is included partially; a pinned passage is always present; the order of equal-scoring items is stable. `hypothesis` does this in Python for free, and it finds the empty-list case and the single-oversized-chunk case within seconds, which are the two that reach production.\n\n## What these cannot tell you\n\nThey verify the request is what you meant. They say nothing about whether what you meant is any good.\n\nA request can pass every check here and produce poor answers, because the instruction is unclear or the retrieved passages are wrong. That is what the golden set is for, and the division of labour is worth keeping crisp: **offline checks defend the mechanism; evaluation defends the outcome.** Teams that have only the second spend their time re-discovering mechanical bugs through expensive noisy experiments, which is a slow and demoralising way to find a stray timestamp."},"check":{"question":"An assembler builds its tool list from a Python set. A determinism test assembles twice and compares bytes. What does this catch that evaluation would not?","options":[{"text":"A quality regression from tools being presented in an order the model finds harder to read.","correct":false,"tempting":"Order can affect selection a little, but the test detects instability rather than quality, and evaluation is where quality questions belong."},{"text":"A schema error in one of the tools, which only appears when the set happens to serialise it first.","correct":false,"tempting":"Serialisation order does not change whether a schema is valid; a malformed tool fails identically in any position."},{"text":"Silent cache destruction, because a prefix that varies between runs never matches.","correct":true},{"text":"A budget overflow, since unordered iteration can include more tools than the limit allows.","correct":false,"tempting":"Set iteration changes order rather than membership, so the token total is the same however it is ordered."}]},"prev":{"slug":"the-golden-set","title":"The set of examples you are allowed to trust","url":"https://addaly.com/learn/context-engineering/the-golden-set"},"next":{"slug":"ablation-earns-the-block","title":"Take it out and see what happens","url":"https://addaly.com/learn/context-engineering/ablation-earns-the-block"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The tests that need no model at all', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/offline-checks-on-the-assembler (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The tests that need no model at all","courseTitle":"Context Engineering","module":{"index":9,"slug":"shipping-and-measuring","title":"Shipping context, and proving it got better"},"answer":null,"keyIdea":"Most context bugs are properties of the assembled request rather than of the model’s answer, so they can be caught by assertions and a snapshot diff that run in milliseconds, cost nothing, and belong in continuous integration.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"cheap-tests-catch-the-expensive-bugs","runs":[{"text":"Cheap tests catch the expensive bugs"}]},{"t":"p","runs":[{"text":"Evaluating answers costs money, takes minutes, and is noisy. Checking the request costs nothing, takes milliseconds, and is exact."}]},{"t":"p","runs":[{"text":"And a surprising share of production incidents in this area are request-shaped rather than answer-shaped: a timestamp that crept into the cached prefix, a delimiter that stopped being escaped, an instruction block that silently doubled because two code paths both added it. None of these needs a model to detect. All of them are invisible in code review."}]},{"t":"p","runs":[{"text":"If you made the assembler a pure function (same inputs, same bytes out, no clock and no network) this lesson is a morning’s work. If you did not, this is the argument for doing it."}]},{"t":"h2","id":"the-snapshot-diff-which-is-worth-the-other-six-together","runs":[{"text":"The snapshot diff, which is worth the other six together"}]},{"t":"p","runs":[{"text":"Render the request for a fixed set of inputs, commit the result, and fail the build when it changes unexpectedly."}]},{"t":"code","lang":"python","label":"Python","text":"def test_request_snapshot(snapshot):\n    req = assemble(fixture_query, fixture_passages, config=CONFIG_V7, now=FIXED_TIME)\n    snapshot.assert_match(render(req), \"request_standard_query.txt\")"},{"t":"p","runs":[{"text":"What this gives you is a "},{"text":"diff of the prompt","b":true},{"text":", and it changes the character of prompt work entirely. Someone edits a sentence for clarity; the diff shows they also moved the block above the cache breakpoint. Someone upgrades a library; the diff shows the chat template now emits a different tool serialisation. Someone adds a field to a chunk header; the diff shows it costs 40 tokens and is repeated eight times per request."}]},{"t":"p","runs":[{"text":"None of that is visible in the source diff, because the source diff shows a template while the snapshot shows the artefact. Review the artefact."}]},{"t":"p","runs":[{"text":"Keep three or four snapshots covering different shapes (a plain query, a long conversation, a tool-calling turn, a turn with untrusted content) and accept updates deliberately rather than regenerating them by reflex when the build goes red."}]},{"t":"h2","id":"the-assertions","runs":[{"text":"The assertions"}]},{"t":"p","runs":[{"text":"Budget.","b":true},{"text":" The assembled request fits, with the output reserve subtracted first, at the largest realistic inputs rather than the fixture ones."}]},{"t":"code","lang":"python","label":"Python","text":"assert count(req) + RESERVE <= LIMIT, f\"{count(req)} + {RESERVE} > {LIMIT}\""},{"t":"p","runs":[{"text":"Determinism.","b":true},{"text":" Assemble twice from identical inputs and compare bytes. This catches dictionary ordering, set iteration, an unsorted tool list, an unseeded shuffle: every one of which quietly destroys the prefix cache and none of which raises an error."}]},{"t":"p","runs":[{"text":"Prefix stability.","b":true},{"text":" Hash everything up to the cache breakpoint. Assert it is unchanged across requests that differ only in the user’s message. A test that fails here has found a cost regression before it reached the bill."}]},{"t":"p","runs":[{"text":"Escaping.","b":true},{"text":" For every untrusted block, assert the closing delimiter does not appear inside it. The security module explained why; this is the check that keeps it true after someone refactors."}]},{"t":"p","runs":[{"text":"Provenance.","b":true},{"text":" Every block carries a trust label. Fail on a missing one rather than defaulting, so a new source added later cannot slip in unlabelled."}]},{"t":"p","runs":[{"text":"Citation handles.","b":true},{"text":" Every handle rendered into the window resolves to a passage in the manifest, and no two passages share a handle. Duplicate handles are an unglamorous bug that makes every downstream citation check meaningless."}]},{"t":"p","runs":[{"text":"No secrets in the prefix.","b":true},{"text":" Assert the cached block contains no pattern matching your key formats and no user-identifying fields. Cached prefixes are shared infrastructure and are the wrong place for anything per-user."}]},{"t":"h2","id":"property-tests-for-the-packer","runs":[{"text":"Property tests for the packer"}]},{"t":"p","runs":[{"text":"The packing and truncation code is where off-by-one errors live, and it is pleasantly testable with generated inputs."}]},{"t":"p","runs":[{"text":"Generate random passage sets and assert the invariants hold every time: the result never exceeds the budget; no chunk is included partially; a pinned passage is always present; the order of equal-scoring items is stable. "},{"text":"hypothesis","code":true},{"text":" does this in Python for free, and it finds the empty-list case and the single-oversized-chunk case within seconds, which are the two that reach production."}]},{"t":"h2","id":"what-these-cannot-tell-you","runs":[{"text":"What these cannot tell you"}]},{"t":"p","runs":[{"text":"They verify the request is what you meant. They say nothing about whether what you meant is any good."}]},{"t":"p","runs":[{"text":"A request can pass every check here and produce poor answers, because the instruction is unclear or the retrieved passages are wrong. That is what the golden set is for, and the division of labour is worth keeping crisp: "},{"text":"offline checks defend the mechanism; evaluation defends the outcome.","b":true},{"text":" Teams that have only the second spend their time re-discovering mechanical bugs through expensive noisy experiments, which is a slow and demoralising way to find a stray timestamp."}]}],"check":{"question":[{"text":"An assembler builds its tool list from a Python set. A determinism test assembles twice and compares bytes. What does this catch that evaluation would not?"}],"options":[{"text":[{"text":"A quality regression from tools being presented in an order the model finds harder to read."}],"correct":false,"tempting":[{"text":"Order can affect selection a little, but the test detects instability rather than quality, and evaluation is where quality questions belong."}]},{"text":[{"text":"A schema error in one of the tools, which only appears when the set happens to serialise it first."}],"correct":false,"tempting":[{"text":"Serialisation order does not change whether a schema is valid; a malformed tool fails identically in any position."}]},{"text":[{"text":"Silent cache destruction, because a prefix that varies between runs never matches."}],"correct":true,"tempting":null},{"text":[{"text":"A budget overflow, since unordered iteration can include more tools than the limit allows."}],"correct":false,"tempting":[{"text":"Set iteration changes order rather than membership, so the token total is the same however it is ordered."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"the-golden-set","title":"The set of examples you are allowed to trust","minutes":9},"next":{"slug":"ablation-earns-the-block","title":"Take it out and see what happens","minutes":9}}}