{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"the-eval-harness","title":"The harness, and why you should build the small one first","url":"https://addaly.com/learn/evaluating-ai/the-eval-harness","position":50,"of":76,"minutes":9,"answer":"A run is reproducible only if the dataset hash, prompt version and model version are recorded with it, and the useful output is a diff against the previous run, not a number.","keyIdea":"A run is reproducible only if the dataset hash, prompt version and model version are recorded with it, and the useful output is a diff against the previous run, not a number."},"body":{"format":"markdown","text":"## What a run is\n\nEverything in this course reduces to one operation: take a dataset, run a system over it, score each item, summarise. The harness is the code that does that reproducibly. It is smaller than you think (a first version is about eighty lines) and getting it right early is what makes every later lesson cheap.\n\nA **run** is identified by everything that could change the answer:\n\n```json\n{\"run_id\": \"2026-09-06T11:04Z-a17c\",\n \"dataset\": \"support-v4\", \"dataset_sha\": \"9f21c0\",\n \"system\": \"reply-agent\", \"git_sha\": \"e4b1aa\",\n \"prompt_version\": \"reply-v9\",\n \"model\": \"claude-haiku-4-5-20260101\", \"temperature\": 0.2,\n \"judge_model\": \"...\", \"judge_prompt_version\": \"rubric-v3\",\n \"seed\": 7, \"n\": 100}\n```\n\nIf any of those is missing, two runs that disagree cannot be explained, and you will spend an afternoon finding out that somebody edited the prompt. The dataset hash matters as much as the git sha: a changed set with an unchanged name is the most confusing bug in this whole discipline.\n\n## Store per item, summarise later\n\nWrite one record per item, not just a total:\n\n```jsonl\n{\"run_id\": \"...\", \"item_id\": \"tkt-0041\", \"output\": \"...\",\n \"checks\": {\"json_valid\": true, \"language_match\": true,\n            \"r1_no_commitment\": false},\n \"pass\": false, \"latency_ms\": 1840, \"tokens_in\": 1204,\n \"tokens_out\": 210, \"cost_usd\": 0.0031}\n```\n\nEverything else is a query over these. Per-slice pass rates, the bootstrap interval from module one, the flip counts between two runs: all of them need per-item records, and none of them can be recovered from a stored average. JSONL files in a directory are a perfectly good store for the first year; a table in Postgres once you want to query across runs.\n\n## Cache, or you will not run it\n\nThe difference between an eval you run on every change and one you run monthly is how long it takes. Cache on a hash of everything that determines an output:\n\n```python\nkey = sha256(f\"{model}|{temperature}|{prompt_version}|{item_id}|{input}\")\n```\n\nChange the prompt and every item re-runs. Add ten items and only those ten cost anything. Re-run to regenerate a report and it is instant and free. A cache turns a twelve-minute, four-dollar run into a nine-second one, and that is the difference between a tool people use and a ritual people avoid.\n\nKeep an explicit `--no-cache` flag, and use it when you want to measure run-to-run variance.\n\n## Concurrency, retries and the failures that look like quality\n\nRun items concurrently (twenty at a time is usually safe) and the wall clock drops from twelve minutes to under one. Two disciplines go with it:\n\n- **Retry transient errors** (429, 500, timeouts) with backoff, and **count them separately**. An item that failed because of a rate limit is not a quality failure, and silently scoring it zero drops your number for reasons that have nothing to do with the system.\n- **Never retry a bad answer.** Retrying until something passes is not evaluation.\n\n## Where it runs\n\nThree tiers, matching the cost tiers from the previous module:\n\n- **Every pull request:** the deterministic checks on the full dev set. Seconds, free, blocking. If schema validity drops, the build fails.\n- **Nightly, or on merge to main:** the judge set. Minutes, a few dollars. Posts a comparison against the previous run: the total, and which items flipped.\n- **Monthly or per release:** the test set, plus whatever human review you have committed to. Slow, expensive, decisive.\n\nThe first tier is the one that changes behaviour, because it is the only one that catches a mistake before it is merged.\n\n## Report the diff, not the number\n\nThe output of a run should be a comparison, because a number alone provokes no action:\n\n```\nreply-v9 vs reply-v8   (support-v4, n=100, 3 runs)\n\n  pass rate   82.0% -> 86.3%   (+4.3, CI on diff +0.8 to +7.9)\n  improved    9 items\n  regressed   4 items   <- read these\n  R1 gate     0 -> 0 violations\n  p95 latency 2.9s -> 4.4s     <- got slower\n  cost/item   $0.0031 -> $0.0068\n```\n\nFour regressions on a run that improved overall is exactly the information a total hides, and the latency and cost lines stop a quality win from quietly becoming a product loss.\n\n## Build the small one first\n\nYou will be tempted to adopt a platform. The free ones are good: **promptfoo** (MIT, YAML-driven, up in twenty minutes), **Inspect** (UK AISI, MIT, the most programmable), **DeepEval** (pytest-shaped), **Langfuse** (self-hostable tracing plus evals). The commercial ones (Braintrust, LangSmith) add hosted dashboards and collaboration.\n\nThe advice is still to write the eighty lines first. Not because the tools are bad, but because a platform adopted before you know what you are measuring quietly decides what you measure: you will use the metrics it ships with. Write your own loop, learn what your system's failures actually look like, and then adopt a tool to stop maintaining plumbing. Migrating a JSONL set into any of these is an hour."},"check":{"question":"A nightly eval drops from 87% to 79% overnight with no deployment. What does a well-built harness let you check first?","options":[{"text":"Whether the dataset hash, prompt version and model version string are identical between the two runs.","correct":true},{"text":"Whether the failures cluster in one slice of the eval set.","correct":false,"tempting":"Is a good second step, but it assumes the two runs measured the same thing, which is precisely what has not been established."},{"text":"Whether the provider had an incident during the run window.","correct":false,"tempting":"Chases an external cause before ruling out the far more common one that something in the run's own configuration changed."},{"text":"Whether the judge model has drifted, by re-running the calibration set.","correct":false,"tempting":"Suspects the instrument, which is worth doing on a schedule, but skips the cheaper check of whether anything in the run definition moved."}]},"prev":{"slug":"evaluating-agents","title":"When the output is a sequence of actions","url":"https://addaly.com/learn/evaluating-ai/evaluating-agents"},"next":{"slug":"evals-in-ci","title":"An evaluation that runs on every change without going red at random","url":"https://addaly.com/learn/evaluating-ai/evals-in-ci"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The harness, and why you should build the small one first', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/the-eval-harness (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The harness, and why you should build the small one first","courseTitle":"Knowing If It Works","module":{"index":6,"slug":"evaluation-in-the-loop","title":"Evaluation in the working loop"},"answer":null,"keyIdea":"A run is reproducible only if the dataset hash, prompt version and model version are recorded with it, and the useful output is a diff against the previous run, not a number.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-a-run-is","runs":[{"text":"What a run is"}]},{"t":"p","runs":[{"text":"Everything in this course reduces to one operation: take a dataset, run a system over it, score each item, summarise. The harness is the code that does that reproducibly. It is smaller than you think (a first version is about eighty lines) and getting it right early is what makes every later lesson cheap."}]},{"t":"p","runs":[{"text":"A "},{"text":"run","b":true},{"text":" is identified by everything that could change the answer:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"run_id\": \"2026-09-06T11:04Z-a17c\",\n \"dataset\": \"support-v4\", \"dataset_sha\": \"9f21c0\",\n \"system\": \"reply-agent\", \"git_sha\": \"e4b1aa\",\n \"prompt_version\": \"reply-v9\",\n \"model\": \"claude-haiku-4-5-20260101\", \"temperature\": 0.2,\n \"judge_model\": \"...\", \"judge_prompt_version\": \"rubric-v3\",\n \"seed\": 7, \"n\": 100}"},{"t":"p","runs":[{"text":"If any of those is missing, two runs that disagree cannot be explained, and you will spend an afternoon finding out that somebody edited the prompt. The dataset hash matters as much as the git sha: a changed set with an unchanged name is the most confusing bug in this whole discipline."}]},{"t":"h2","id":"store-per-item-summarise-later","runs":[{"text":"Store per item, summarise later"}]},{"t":"p","runs":[{"text":"Write one record per item, not just a total:"}]},{"t":"code","lang":"jsonl","label":"JSON Lines","text":"{\"run_id\": \"...\", \"item_id\": \"tkt-0041\", \"output\": \"...\",\n \"checks\": {\"json_valid\": true, \"language_match\": true,\n            \"r1_no_commitment\": false},\n \"pass\": false, \"latency_ms\": 1840, \"tokens_in\": 1204,\n \"tokens_out\": 210, \"cost_usd\": 0.0031}"},{"t":"p","runs":[{"text":"Everything else is a query over these. Per-slice pass rates, the bootstrap interval from module one, the flip counts between two runs: all of them need per-item records, and none of them can be recovered from a stored average. JSONL files in a directory are a perfectly good store for the first year; a table in Postgres once you want to query across runs."}]},{"t":"h2","id":"cache-or-you-will-not-run-it","runs":[{"text":"Cache, or you will not run it"}]},{"t":"p","runs":[{"text":"The difference between an eval you run on every change and one you run monthly is how long it takes. Cache on a hash of everything that determines an output:"}]},{"t":"code","lang":"python","label":"Python","text":"key = sha256(f\"{model}|{temperature}|{prompt_version}|{item_id}|{input}\")"},{"t":"p","runs":[{"text":"Change the prompt and every item re-runs. Add ten items and only those ten cost anything. Re-run to regenerate a report and it is instant and free. A cache turns a twelve-minute, four-dollar run into a nine-second one, and that is the difference between a tool people use and a ritual people avoid."}]},{"t":"p","runs":[{"text":"Keep an explicit "},{"text":"--no-cache","code":true},{"text":" flag, and use it when you want to measure run-to-run variance."}]},{"t":"h2","id":"concurrency-retries-and-the-failures-that-look-like-quality","runs":[{"text":"Concurrency, retries and the failures that look like quality"}]},{"t":"p","runs":[{"text":"Run items concurrently (twenty at a time is usually safe) and the wall clock drops from twelve minutes to under one. Two disciplines go with it:"}]},{"t":"ul","items":[[{"text":"Retry transient errors","b":true},{"text":" (429, 500, timeouts) with backoff, and "},{"text":"count them separately","b":true},{"text":". An item that failed because of a rate limit is not a quality failure, and silently scoring it zero drops your number for reasons that have nothing to do with the system."}],[{"text":"Never retry a bad answer.","b":true},{"text":" Retrying until something passes is not evaluation."}]]},{"t":"h2","id":"where-it-runs","runs":[{"text":"Where it runs"}]},{"t":"p","runs":[{"text":"Three tiers, matching the cost tiers from the previous module:"}]},{"t":"ul","items":[[{"text":"Every pull request:","b":true},{"text":" the deterministic checks on the full dev set. Seconds, free, blocking. If schema validity drops, the build fails."}],[{"text":"Nightly, or on merge to main:","b":true},{"text":" the judge set. Minutes, a few dollars. Posts a comparison against the previous run: the total, and which items flipped."}],[{"text":"Monthly or per release:","b":true},{"text":" the test set, plus whatever human review you have committed to. Slow, expensive, decisive."}]]},{"t":"p","runs":[{"text":"The first tier is the one that changes behaviour, because it is the only one that catches a mistake before it is merged."}]},{"t":"h2","id":"report-the-diff-not-the-number","runs":[{"text":"Report the diff, not the number"}]},{"t":"p","runs":[{"text":"The output of a run should be a comparison, because a number alone provokes no action:"}]},{"t":"code","lang":"","label":"Text","text":"reply-v9 vs reply-v8   (support-v4, n=100, 3 runs)\n\n  pass rate   82.0% -> 86.3%   (+4.3, CI on diff +0.8 to +7.9)\n  improved    9 items\n  regressed   4 items   <- read these\n  R1 gate     0 -> 0 violations\n  p95 latency 2.9s -> 4.4s     <- got slower\n  cost/item   $0.0031 -> $0.0068"},{"t":"p","runs":[{"text":"Four regressions on a run that improved overall is exactly the information a total hides, and the latency and cost lines stop a quality win from quietly becoming a product loss."}]},{"t":"h2","id":"build-the-small-one-first","runs":[{"text":"Build the small one first"}]},{"t":"p","runs":[{"text":"You will be tempted to adopt a platform. The free ones are good: "},{"text":"promptfoo","b":true},{"text":" (MIT, YAML-driven, up in twenty minutes), "},{"text":"Inspect","b":true},{"text":" (UK AISI, MIT, the most programmable), "},{"text":"DeepEval","b":true},{"text":" (pytest-shaped), "},{"text":"Langfuse","b":true},{"text":" (self-hostable tracing plus evals). The commercial ones (Braintrust, LangSmith) add hosted dashboards and collaboration."}]},{"t":"p","runs":[{"text":"The advice is still to write the eighty lines first. Not because the tools are bad, but because a platform adopted before you know what you are measuring quietly decides what you measure: you will use the metrics it ships with. Write your own loop, learn what your system’s failures actually look like, and then adopt a tool to stop maintaining plumbing. Migrating a JSONL set into any of these is an hour."}]}],"check":{"question":[{"text":"A nightly eval drops from 87% to 79% overnight with no deployment. What does a well-built harness let you check first?"}],"options":[{"text":[{"text":"Whether the dataset hash, prompt version and model version string are identical between the two runs."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the failures cluster in one slice of the eval set."}],"correct":false,"tempting":[{"text":"Is a good second step, but it assumes the two runs measured the same thing, which is precisely what has not been established."}]},{"text":[{"text":"Whether the provider had an incident during the run window."}],"correct":false,"tempting":[{"text":"Chases an external cause before ruling out the far more common one that something in the run’s own configuration changed."}]},{"text":[{"text":"Whether the judge model has drifted, by re-running the calibration set."}],"correct":false,"tempting":[{"text":"Suspects the instrument, which is worth doing on a schedule, but skips the cheaper check of whether anything in the run definition moved."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"evaluating-agents","title":"When the output is a sequence of actions","minutes":10},"next":{"slug":"evals-in-ci","title":"An evaluation that runs on every change without going red at random","minutes":9}}}