{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"cheap-checks-first","title":"The checks that cost nothing and catch most of it","url":"https://addaly.com/learn/evaluating-ai/cheap-checks-first","position":45,"of":76,"minutes":8,"answer":"Deterministic checks are free, exact and repeatable, so run them first and move every check you can down into code before spending a judge call on it.","keyIdea":"Deterministic checks are free, exact and repeatable, so run them first and move every check you can down into code before spending a judge call on it."},"body":{"format":"markdown","text":"## Order your checks by price\n\nThere are three tiers of checking, and they differ by four orders of magnitude in cost:\n\n- **Deterministic code.** Microseconds, free, perfectly repeatable.\n- **Model judge.** Roughly a second and a fraction of a penny per call, repeatable to within its own noise.\n- **Human.** Two to five minutes and real money, and the only one that can settle a new question.\n\nRun them in that order and let each tier filter for the next. A judge asked to assess the quality of an answer that is not valid JSON has burned a call to tell you something `json.loads` knew for free.\n\nMost teams build the middle tier first, because it is the interesting one. Then they discover their judge's most common verdict is \"the output was truncated\".\n\n> Figure: Three tiers of checking, four orders of magnitude apart. Layers, top to bottom: Human review (two to five minutes and real money; the only tier that settles a new question); Model judge (about a second and a fraction of a penny; repeatable to within its own noise); Deterministic code (microseconds, free, perfectly repeatable; runs first on everything). Run from the bottom up, and let each tier filter for the one above it. A judge asked about an output that is not valid JSON has spent a call to learn what json.loads knew for free.\n\n## What code can check\n\nMore than people expect. Every item here is a real production failure mode, and every one is a few lines:\n\n- **Structure.** Does it parse as JSON? Validate against a JSON Schema, in `jsonschema` or Pydantic. Are all required fields present, of the right type, and is every enum value in the allowed set?\n- **Grounding, cheaply.** Every quoted span the model claims to have taken from the source is a substring of the source. Every cited document id exists. Every URL is one from the supplied context, not one the model composed.\n- **Numbers.** Sums add up. Line items total the stated total. Dates are valid, in range, and in the right order. A percentage is between 0 and 100. VAT is what the arithmetic says.\n- **Language.** The output language matches the input's, via `langdetect` or `fasttext`: a free 900-language model that runs in milliseconds.\n- **Leftovers.** No `[INSERT NAME]`, `Lorem ipsum`, `As an AI language model`, `TODO`, or the system prompt echoed back.\n- **Bounds.** Length between a floor and a ceiling. Truncation detection: did it stop because it finished, or because it hit the token limit? The finish reason is in the API response and going unchecked is one of the most common silent failures in production.\n- **Privacy.** No email address, phone number or card-shaped digit string that was not in the input.\n- **Repetition.** The same sentence three times, or a trailing loop: a cheap n-gram check catches degeneracy that a judge often rates as \"thorough\".\n\nWrite them as ordinary tests. `pytest` over a JSONL file is a perfectly good eval harness for this tier, it runs in CI, and it is free.\n\n## The first number to get\n\nBefore any quality work, measure **what fraction of outputs fail a structural check**. It is a ten-minute job and it is routinely much higher than the team's guess: the guess is usually near zero, because the team has only ever looked at outputs that rendered.\n\nWhen that number is meaningful, fix it before anything else, and fix it in the right place. Structured-output or JSON-mode support from the provider, a constrained decoder such as `outlines` or `guidance`, or one retry on a parse failure will each do more for your quality metric than a week of prompt rewriting. And they are boring, which is why they get skipped.\n\n## Assertions live next to the data\n\nKeep the checks per item, not per suite. Some items have specific expectations: this invoice's total is 4,320; this query must return the Delhi office's address; this refusal must not name a competitor. Store them with the item:\n\n```jsonl\n{\"id\": \"inv-041\",\n \"input\": \"...\",\n \"assert\": [{\"type\": \"json_field\", \"path\": \"total\", \"equals\": 4320},\n            {\"type\": \"not_contains\", \"value\": \"approximately\"},\n            {\"type\": \"language\", \"equals\": \"input\"}]}\n```\n\nThis is exactly how **promptfoo** models an eval, and if you want the pattern without writing it, promptfoo is MIT-licensed, runs from a YAML file, and takes about twenty minutes to get running. **Inspect** from the UK AI Safety Institute is the more programmable free option. **DeepEval** wraps the same ideas around pytest. Commercial platforms (Braintrust, Langfuse's paid tiers, LangSmith) add hosted dashboards and team features on top of the same fundamentals.\n\nStart with a JSONL file and a for-loop anyway. You will understand what any platform is doing, and moving 200 items into a tool later is an hour's work, while adopting a platform before you know what you are measuring shapes your evaluation around the tool's opinions.\n\n## The rule of thumb\n\nBefore adding any judge call, ask whether code could decide it. If the answer is \"code could decide it if the output were structured\", change the output to be structured. Every check you move down a tier makes your evaluation faster, cheaper, perfectly repeatable, and immune to the judge drifting under you."},"check":{"question":"A team's LLM judge reports that 18% of their extraction outputs are \"low quality\". What should they check before tuning the prompt?","options":[{"text":"Whether the judge itself agrees with human labels on those 18%.","correct":false,"tempting":"Is the right move once the outputs are known to be well-formed, but spends judge-calibration effort on items that may have failed for a mechanical reason."},{"text":"Whether a stronger judge model gives a different figure, to establish whether 18% is real.","correct":false,"tempting":"Escalates to a more expensive judge to confirm a number, when a free deterministic check may explain most of it in ten minutes."},{"text":"What fraction of those outputs fail to parse, are missing required fields, or hit the token limit and were truncated.","correct":true},{"text":"Whether the 18% cluster in particular input types, using the slice tags from the eval set.","correct":false,"tempting":"Slicing is genuinely valuable, but it partitions a quality signal that may not yet be a quality signal at all."}]},"prev":{"slug":"ranking-systems-by-preference","title":"Turning pairwise wins into a ranking you can defend","url":"https://addaly.com/learn/evaluating-ai/ranking-systems-by-preference"},"next":{"slug":"groundedness-checks","title":"Checking an answer against the passages it was given","url":"https://addaly.com/learn/evaluating-ai/groundedness-checks"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The checks that cost nothing and catch most of it', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/cheap-checks-first (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The checks that cost nothing and catch most of it","courseTitle":"Knowing If It Works","module":{"index":5,"slug":"automating-judgement","title":"Automating judgement"},"answer":null,"keyIdea":"Deterministic checks are free, exact and repeatable, so run them first and move every check you can down into code before spending a judge call on it.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Three tiers of checking, four orders of magnitude apart","layers":[{"label":"Human review","detail":"two to five minutes and real money; the only tier that settles a new question"},{"label":"Model judge","detail":"about a second and a fraction of a penny; repeatable to within its own noise"},{"label":"Deterministic code","detail":"microseconds, free, perfectly repeatable; runs first on everything"}],"caption":"Run from the bottom up, and let each tier filter for the one above it. A judge asked about an output that is not valid JSON has spent a call to learn what json.loads knew for free."},"alt":"Figure: Three tiers of checking, four orders of magnitude apart. Layers, top to bottom: Human review (two to five minutes and real money; the only tier that settles a new question); Model judge (about a second and a fraction of a penny; repeatable to within its own noise); Deterministic code (microseconds, free, perfectly repeatable; runs first on everything). Run from the bottom up, and let each tier filter for the one above it. A judge asked about an output that is not valid JSON has spent a call to learn what json.loads knew for free."}},"blocks":[{"t":"h2","id":"order-your-checks-by-price","runs":[{"text":"Order your checks by price"}]},{"t":"p","runs":[{"text":"There are three tiers of checking, and they differ by four orders of magnitude in cost:"}]},{"t":"ul","items":[[{"text":"Deterministic code.","b":true},{"text":" Microseconds, free, perfectly repeatable."}],[{"text":"Model judge.","b":true},{"text":" Roughly a second and a fraction of a penny per call, repeatable to within its own noise."}],[{"text":"Human.","b":true},{"text":" Two to five minutes and real money, and the only one that can settle a new question."}]]},{"t":"p","runs":[{"text":"Run them in that order and let each tier filter for the next. A judge asked to assess the quality of an answer that is not valid JSON has burned a call to tell you something "},{"text":"json.loads","code":true},{"text":" knew for free."}]},{"t":"p","runs":[{"text":"Most teams build the middle tier first, because it is the interesting one. Then they discover their judge’s most common verdict is “the output was truncated”."}]},{"t":"h2","id":"what-code-can-check","runs":[{"text":"What code can check"}]},{"t":"p","runs":[{"text":"More than people expect. Every item here is a real production failure mode, and every one is a few lines:"}]},{"t":"ul","items":[[{"text":"Structure.","b":true},{"text":" Does it parse as JSON? Validate against a JSON Schema, in "},{"text":"jsonschema","code":true},{"text":" or Pydantic. Are all required fields present, of the right type, and is every enum value in the allowed set?"}],[{"text":"Grounding, cheaply.","b":true},{"text":" Every quoted span the model claims to have taken from the source is a substring of the source. Every cited document id exists. Every URL is one from the supplied context, not one the model composed."}],[{"text":"Numbers.","b":true},{"text":" Sums add up. Line items total the stated total. Dates are valid, in range, and in the right order. A percentage is between 0 and 100. VAT is what the arithmetic says."}],[{"text":"Language.","b":true},{"text":" The output language matches the input’s, via "},{"text":"langdetect","code":true},{"text":" or "},{"text":"fasttext","code":true},{"text":": a free 900-language model that runs in milliseconds."}],[{"text":"Leftovers.","b":true},{"text":" No "},{"text":"[INSERT NAME]","code":true},{"text":", "},{"text":"Lorem ipsum","code":true},{"text":", "},{"text":"As an AI language model","code":true},{"text":", "},{"text":"TODO","code":true},{"text":", or the system prompt echoed back."}],[{"text":"Bounds.","b":true},{"text":" Length between a floor and a ceiling. Truncation detection: did it stop because it finished, or because it hit the token limit? The finish reason is in the API response and going unchecked is one of the most common silent failures in production."}],[{"text":"Privacy.","b":true},{"text":" No email address, phone number or card-shaped digit string that was not in the input."}],[{"text":"Repetition.","b":true},{"text":" The same sentence three times, or a trailing loop: a cheap n-gram check catches degeneracy that a judge often rates as “thorough”."}]]},{"t":"p","runs":[{"text":"Write them as ordinary tests. "},{"text":"pytest","code":true},{"text":" over a JSONL file is a perfectly good eval harness for this tier, it runs in CI, and it is free."}]},{"t":"h2","id":"the-first-number-to-get","runs":[{"text":"The first number to get"}]},{"t":"p","runs":[{"text":"Before any quality work, measure "},{"text":"what fraction of outputs fail a structural check","b":true},{"text":". It is a ten-minute job and it is routinely much higher than the team’s guess: the guess is usually near zero, because the team has only ever looked at outputs that rendered."}]},{"t":"p","runs":[{"text":"When that number is meaningful, fix it before anything else, and fix it in the right place. Structured-output or JSON-mode support from the provider, a constrained decoder such as "},{"text":"outlines","code":true},{"text":" or "},{"text":"guidance","code":true},{"text":", or one retry on a parse failure will each do more for your quality metric than a week of prompt rewriting. And they are boring, which is why they get skipped."}]},{"t":"h2","id":"assertions-live-next-to-the-data","runs":[{"text":"Assertions live next to the data"}]},{"t":"p","runs":[{"text":"Keep the checks per item, not per suite. Some items have specific expectations: this invoice’s total is 4,320; this query must return the Delhi office’s address; this refusal must not name a competitor. Store them with the item:"}]},{"t":"code","lang":"jsonl","label":"JSON Lines","text":"{\"id\": \"inv-041\",\n \"input\": \"...\",\n \"assert\": [{\"type\": \"json_field\", \"path\": \"total\", \"equals\": 4320},\n            {\"type\": \"not_contains\", \"value\": \"approximately\"},\n            {\"type\": \"language\", \"equals\": \"input\"}]}"},{"t":"p","runs":[{"text":"This is exactly how "},{"text":"promptfoo","b":true},{"text":" models an eval, and if you want the pattern without writing it, promptfoo is MIT-licensed, runs from a YAML file, and takes about twenty minutes to get running. "},{"text":"Inspect","b":true},{"text":" from the UK AI Safety Institute is the more programmable free option. "},{"text":"DeepEval","b":true},{"text":" wraps the same ideas around pytest. Commercial platforms (Braintrust, Langfuse’s paid tiers, LangSmith) add hosted dashboards and team features on top of the same fundamentals."}]},{"t":"p","runs":[{"text":"Start with a JSONL file and a for-loop anyway. You will understand what any platform is doing, and moving 200 items into a tool later is an hour’s work, while adopting a platform before you know what you are measuring shapes your evaluation around the tool’s opinions."}]},{"t":"h2","id":"the-rule-of-thumb","runs":[{"text":"The rule of thumb"}]},{"t":"p","runs":[{"text":"Before adding any judge call, ask whether code could decide it. If the answer is “code could decide it if the output were structured”, change the output to be structured. Every check you move down a tier makes your evaluation faster, cheaper, perfectly repeatable, and immune to the judge drifting under you."}]}],"check":{"question":[{"text":"A team’s LLM judge reports that 18% of their extraction outputs are “low quality”. What should they check before tuning the prompt?"}],"options":[{"text":[{"text":"Whether the judge itself agrees with human labels on those 18%."}],"correct":false,"tempting":[{"text":"Is the right move once the outputs are known to be well-formed, but spends judge-calibration effort on items that may have failed for a mechanical reason."}]},{"text":[{"text":"Whether a stronger judge model gives a different figure, to establish whether 18% is real."}],"correct":false,"tempting":[{"text":"Escalates to a more expensive judge to confirm a number, when a free deterministic check may explain most of it in ten minutes."}]},{"text":[{"text":"What fraction of those outputs fail to parse, are missing required fields, or hit the token limit and were truncated."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the 18% cluster in particular input types, using the slice tags from the eval set."}],"correct":false,"tempting":[{"text":"Slicing is genuinely valuable, but it partitions a quality signal that may not yet be a quality signal at all."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"ranking-systems-by-preference","title":"Turning pairwise wins into a ranking you can defend","minutes":9},"next":{"slug":"groundedness-checks","title":"Checking an answer against the passages it was given","minutes":9}}}