{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"assertions-before-judges","title":"Assertions before judges: the checks that cost nothing","url":"https://addaly.com/learn/building-with-ai/assertions-before-judges","position":54,"of":80,"minutes":8,"answer":"Most regressions in an AI feature are caught by deterministic checks (schema validity, forbidden strings, resolvable citations, length, language, code that runs.","keyIdea":"Most regressions in an AI feature are caught by deterministic checks (schema validity, forbidden strings, resolvable citations, length, language, code that runs, cost and latency bounds) which cost nothing per run and fail for a stated reason, so they go underneath any model judge, never instead of one."},"body":{"format":"markdown","text":"## The cheap layer\n\nBefore you ask a model whether an answer is good, ask code whether it is even acceptable. A surprisingly large share of the ways an AI feature breaks are mechanical: the JSON stopped validating, a citation points nowhere, the answer came back in the wrong language, the refusal phrase appeared on a question that should have been answered, the reply is three times longer than last week. None of these needs judgement. All of them can be checked in microseconds, for free, and each failure names its own cause.\n\nThe evaluation course builds this layer as part of a full framework. This lesson is the working list for a feature you are shipping.\n\n## The catalogue\n\n**Shape.** Valid JSON. Conforms to the schema. Required fields present. Enumerated fields hold allowed values. Numbers within range. Dates parse.\n\n**Content.** Must-contain strings from the case file. Must-not-contain strings: competitor names, a placeholder like \"[insert name]\", the phrase \"as an AI\" where the product's voice forbids it, a raw URL where the product renders links.\n\n**Citations.** Every cited id exists in the set that was retrieved for this request. The permissions lesson made the point: a model given five chunks will sometimes cite a sixth.\n\n**Language.** A language detector on the output, compared with the language of the input. Models drift into English when a prompt is in English and the user is not, and nothing else catches it.\n\n**Length.** A lower bound, because an empty or one-line answer is usually a failure, and an upper bound, because verbosity creeps in with every prompt tweak and nobody notices until the interface overflows.\n\n**Refusals.** The refusal phrase must appear on the cases tagged `should-refuse` and must not appear on any other. Both directions; over-refusal is the more common regression.\n\n**Code.** If the feature writes code, it must parse. If there are tests, they must pass in a sandbox: the running-code lesson from the agents module showed how to run them safely. A syntax error is a failure with a line number.\n\n**Personal data.** A pattern check for emails, phone numbers and card-like digit runs in the output, when the feature should never emit them.\n\n**Cost and latency.** Token usage per case against a budget, and wall-clock time against a limit. A prompt change that doubles output length passes every quality check and doubles the bill.\n\n## Writing them\n\nAny test runner does this. In Python:\n\n```python\nimport json, pytest\nfrom feature import answer\n\ncases = [json.loads(l) for l in open(\"cases.jsonl\")]\n\n@pytest.mark.parametrize(\"case\", cases, ids=lambda c: c[\"id\"])\ndef test_case(case):\n    out = answer(**case[\"input\"])\n    for s in case[\"expect\"].get(\"must_contain\", []):\n        assert s in out.text, f\"missing {s!r}\"\n    for s in case[\"expect\"].get(\"must_not_contain\", []):\n        assert s not in out.text, f\"found {s!r}\"\n    for cid in case[\"expect\"].get(\"cites\", []):\n        assert cid in out.cited_ids\n    assert set(out.cited_ids) <= set(out.retrieved_ids), \"cited an unretrieved chunk\"\n    assert out.usage.output_tokens < 600\n```\n\n`promptfoo` is a free tool that does the same from a YAML file with built-in assertion types, and it is a good choice when several people who are not all engineers maintain the cases.\n\n## Why this layer comes first\n\nThree reasons, in order of importance. It is free: no model call, no cost per run, so it runs on every commit. It is precise: a failure says \"cited chunk 7, which was not retrieved\", not \"score 6/10\". And it is stable: the same output always gets the same verdict, which the next two lessons will show a model judge cannot promise.\n\nIn practice, on a mature feature, a majority of regressions caught in CI are caught here. The judge layer above it exists for the failures this layer cannot see: an answer that is well-formed, cites correctly, is the right length and is wrong.\n\n## What assertions cannot do\n\nThey cannot tell you an answer is good. A response can pass every check above and be confidently incorrect, subtly off-topic, or technically accurate and useless. Assertions rule out the mechanically broken; they do not rule in the correct. That is the judge's job, and the reason the next lesson exists. The mistake to avoid is the one that follows from a green assertion suite: treating \"nothing mechanical failed\" as \"it works\".\n\n## The habit that keeps the list growing\n\nEvery production failure that could have been caught by a string check becomes a string check. A user reports that the assistant answered in English; a language assertion is added. A citation was fabricated; the citation check is tightened. The list in this lesson is a starting point, and the version a team has after a year is longer, specific to the product, and the cheapest quality instrument they own.\n\n> Figure: Where one quarter's regressions were actually caught. Bars (% of regressions caught, one quarter): Deterministic assertions 61; Model judge 24; Retrieval recall check 9; Found in production 6. The free layer catches most of them, and each failure names its own cause: cited chunk 7, which was not retrieved. The judge is not redundant (it exists for the answer that is well-formed, cites correctly, is the right length and is wrong) but running it first would cost money to learn less."},"check":{"question":"A team's CI runs only a model judge, at some cost per run, and it occasionally passes an answer that cites a chunk which was never retrieved. What is the most direct fix?","options":[{"text":"Add a deterministic check that every cited id is in the retrieved set, and keep the judge for what code cannot see.","correct":true},{"text":"Give the judge the retrieved set and instruct it to check every citation carefully against that list before it produces a score.","correct":false,"tempting":"Asks a probabilistic judge to do a mechanical comparison it will sometimes get wrong, at a cost, when code does it perfectly for nothing."},{"text":"Lower the judge's passing threshold so borderline answers with citation problems fail more often.","correct":false,"tempting":"A threshold change affects every dimension at once and does not target the specific mechanical failure."},{"text":"Run the judge three times per case and take the majority verdict to reduce misses.","correct":false,"tempting":"Reduces noise on subjective calls, but a judge that does not compare ids will miss the same fabricated citation three times."}]},"prev":{"slug":"the-first-fifty-cases","title":"The first fifty cases, and where they come from","url":"https://addaly.com/learn/building-with-ai/the-first-fifty-cases"},"next":{"slug":"a-judge-you-have-checked","title":"A model judge you have checked against people","url":"https://addaly.com/learn/building-with-ai/a-judge-you-have-checked"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Assertions before judges: the checks that cost nothing', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/assertions-before-judges (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Assertions before judges: the checks that cost nothing","courseTitle":"Building with AI","module":{"index":6,"slug":"evaluating-and-iterating","title":"Evaluating and iterating"},"answer":null,"keyIdea":"Most regressions in an AI feature are caught by deterministic checks (schema validity, forbidden strings, resolvable citations, length, language, code that runs, cost and latency bounds) which cost nothing per run and fail for a stated reason, so they go underneath any model judge, never instead of one.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% of regressions caught, one quarter","title":"Where one quarter’s regressions were actually caught","series":[{"label":"Deterministic assertions","value":61,"highlight":true},{"label":"Model judge","value":24},{"label":"Retrieval recall check","value":9},{"label":"Found in production","value":6}],"caption":"The free layer catches most of them, and each failure names its own cause: cited chunk 7, which was not retrieved. The judge is not redundant (it exists for the answer that is well-formed, cites correctly, is the right length and is wrong) but running it first would cost money to learn less."},"alt":"Figure: Where one quarter’s regressions were actually caught. Bars (% of regressions caught, one quarter): Deterministic assertions 61; Model judge 24; Retrieval recall check 9; Found in production 6. The free layer catches most of them, and each failure names its own cause: cited chunk 7, which was not retrieved. The judge is not redundant (it exists for the answer that is well-formed, cites correctly, is the right length and is wrong) but running it first would cost money to learn less."}},"blocks":[{"t":"h2","id":"the-cheap-layer","runs":[{"text":"The cheap layer"}]},{"t":"p","runs":[{"text":"Before you ask a model whether an answer is good, ask code whether it is even acceptable. A surprisingly large share of the ways an AI feature breaks are mechanical: the JSON stopped validating, a citation points nowhere, the answer came back in the wrong language, the refusal phrase appeared on a question that should have been answered, the reply is three times longer than last week. None of these needs judgement. All of them can be checked in microseconds, for free, and each failure names its own cause."}]},{"t":"p","runs":[{"text":"The evaluation course builds this layer as part of a full framework. This lesson is the working list for a feature you are shipping."}]},{"t":"h2","id":"the-catalogue","runs":[{"text":"The catalogue"}]},{"t":"p","runs":[{"text":"Shape.","b":true},{"text":" Valid JSON. Conforms to the schema. Required fields present. Enumerated fields hold allowed values. Numbers within range. Dates parse."}]},{"t":"p","runs":[{"text":"Content.","b":true},{"text":" Must-contain strings from the case file. Must-not-contain strings: competitor names, a placeholder like “[insert name]”, the phrase “as an AI” where the product’s voice forbids it, a raw URL where the product renders links."}]},{"t":"p","runs":[{"text":"Citations.","b":true},{"text":" Every cited id exists in the set that was retrieved for this request. The permissions lesson made the point: a model given five chunks will sometimes cite a sixth."}]},{"t":"p","runs":[{"text":"Language.","b":true},{"text":" A language detector on the output, compared with the language of the input. Models drift into English when a prompt is in English and the user is not, and nothing else catches it."}]},{"t":"p","runs":[{"text":"Length.","b":true},{"text":" A lower bound, because an empty or one-line answer is usually a failure, and an upper bound, because verbosity creeps in with every prompt tweak and nobody notices until the interface overflows."}]},{"t":"p","runs":[{"text":"Refusals.","b":true},{"text":" The refusal phrase must appear on the cases tagged "},{"text":"should-refuse","code":true},{"text":" and must not appear on any other. Both directions; over-refusal is the more common regression."}]},{"t":"p","runs":[{"text":"Code.","b":true},{"text":" If the feature writes code, it must parse. If there are tests, they must pass in a sandbox: the running-code lesson from the agents module showed how to run them safely. A syntax error is a failure with a line number."}]},{"t":"p","runs":[{"text":"Personal data.","b":true},{"text":" A pattern check for emails, phone numbers and card-like digit runs in the output, when the feature should never emit them."}]},{"t":"p","runs":[{"text":"Cost and latency.","b":true},{"text":" Token usage per case against a budget, and wall-clock time against a limit. A prompt change that doubles output length passes every quality check and doubles the bill."}]},{"t":"h2","id":"writing-them","runs":[{"text":"Writing them"}]},{"t":"p","runs":[{"text":"Any test runner does this. In Python:"}]},{"t":"code","lang":"python","label":"Python","text":"import json, pytest\nfrom feature import answer\n\ncases = [json.loads(l) for l in open(\"cases.jsonl\")]\n\n@pytest.mark.parametrize(\"case\", cases, ids=lambda c: c[\"id\"])\ndef test_case(case):\n    out = answer(**case[\"input\"])\n    for s in case[\"expect\"].get(\"must_contain\", []):\n        assert s in out.text, f\"missing {s!r}\"\n    for s in case[\"expect\"].get(\"must_not_contain\", []):\n        assert s not in out.text, f\"found {s!r}\"\n    for cid in case[\"expect\"].get(\"cites\", []):\n        assert cid in out.cited_ids\n    assert set(out.cited_ids) <= set(out.retrieved_ids), \"cited an unretrieved chunk\"\n    assert out.usage.output_tokens < 600"},{"t":"p","runs":[{"text":"promptfoo","code":true},{"text":" is a free tool that does the same from a YAML file with built-in assertion types, and it is a good choice when several people who are not all engineers maintain the cases."}]},{"t":"h2","id":"why-this-layer-comes-first","runs":[{"text":"Why this layer comes first"}]},{"t":"p","runs":[{"text":"Three reasons, in order of importance. It is free: no model call, no cost per run, so it runs on every commit. It is precise: a failure says “cited chunk 7, which was not retrieved”, not “score 6/10”. And it is stable: the same output always gets the same verdict, which the next two lessons will show a model judge cannot promise."}]},{"t":"p","runs":[{"text":"In practice, on a mature feature, a majority of regressions caught in CI are caught here. The judge layer above it exists for the failures this layer cannot see: an answer that is well-formed, cites correctly, is the right length and is wrong."}]},{"t":"h2","id":"what-assertions-cannot-do","runs":[{"text":"What assertions cannot do"}]},{"t":"p","runs":[{"text":"They cannot tell you an answer is good. A response can pass every check above and be confidently incorrect, subtly off-topic, or technically accurate and useless. Assertions rule out the mechanically broken; they do not rule in the correct. That is the judge’s job, and the reason the next lesson exists. The mistake to avoid is the one that follows from a green assertion suite: treating “nothing mechanical failed” as “it works”."}]},{"t":"h2","id":"the-habit-that-keeps-the-list-growing","runs":[{"text":"The habit that keeps the list growing"}]},{"t":"p","runs":[{"text":"Every production failure that could have been caught by a string check becomes a string check. A user reports that the assistant answered in English; a language assertion is added. A citation was fabricated; the citation check is tightened. The list in this lesson is a starting point, and the version a team has after a year is longer, specific to the product, and the cheapest quality instrument they own."}]}],"check":{"question":[{"text":"A team’s CI runs only a model judge, at some cost per run, and it occasionally passes an answer that cites a chunk which was never retrieved. What is the most direct fix?"}],"options":[{"text":[{"text":"Add a deterministic check that every cited id is in the retrieved set, and keep the judge for what code cannot see."}],"correct":true,"tempting":null},{"text":[{"text":"Give the judge the retrieved set and instruct it to check every citation carefully against that list before it produces a score."}],"correct":false,"tempting":[{"text":"Asks a probabilistic judge to do a mechanical comparison it will sometimes get wrong, at a cost, when code does it perfectly for nothing."}]},{"text":[{"text":"Lower the judge’s passing threshold so borderline answers with citation problems fail more often."}],"correct":false,"tempting":[{"text":"A threshold change affects every dimension at once and does not target the specific mechanical failure."}]},{"text":[{"text":"Run the judge three times per case and take the majority verdict to reduce misses."}],"correct":false,"tempting":[{"text":"Reduces noise on subjective calls, but a judge that does not compare ids will miss the same fabricated citation three times."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"the-first-fifty-cases","title":"The first fifty cases, and where they come from","minutes":8},"next":{"slug":"a-judge-you-have-checked","title":"A model judge you have checked against people","minutes":9}}}