{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evals-in-ci","title":"An evaluation that runs on every change without going red at random","url":"https://addaly.com/learn/evaluating-ai/evals-in-ci","position":51,"of":76,"minutes":9,"answer":"Block merges only on deterministic checks and on a band around a rolling baseline, quarantine flaky items instead of retrying them, and let the slow full run open an issue rather than stop the queue.","keyIdea":"Block merges only on deterministic checks and on a band around a rolling baseline, quarantine flaky items instead of retrying them, and let the slow full run open an issue rather than stop the queue."},"body":{"format":"markdown","text":"## Why the obvious design fails\n\nThe obvious design is: run the eval set on every pull request, fail the build if the score drops. It survives about a fortnight.\n\nA stochastic model produces a different score on identical code. The build goes red on a change to a README. Somebody adds a retry. Somebody else raises the threshold. Within a month the gate is either ignored or disabled, and the team is back to shipping on feeling, which is worse than never having built it, because everyone believes a gate exists.\n\nThe fix is to separate what can be a hard gate from what cannot.\n\n## Three tiers\n\n**Tier one: deterministic checks, blocking, under sixty seconds.** Schema validity, forbidden strings, prompt templates rendering without missing variables, citations resolving, no secrets in outputs, the judge rubric file parsing. These are unit tests. They are reproducible, they never flake, and they catch a large share of real breakages.\n\n**Tier two: a small core set, blocking on a band.** 150 to 300 items, one run, temperature 0, pinned model. Do not compare against a fixed threshold; compare against a rolling baseline from the last several runs on the main branch, and fail only when the drop exceeds a band derived from the measured run-to-run spread:\n\n```\nfail if  pass_rate < baseline - max(3 * run_to_run_sd, minimum_band)\n```\n\nMeasure that spread once, the A/A test from the statistics block, and write it into the config with a comment saying when it was measured. A gate whose sensitivity is derived from data survives arguments; a gate set at \"must be above 85%\" does not.\n\n**Tier three: the full set, nightly, non-blocking.** It runs on a schedule, posts its result, and opens an issue when it regresses. It is where the number you report comes from. It never stops the merge queue, because a two-hour job that blocks a queue gets deleted.\n\n> Figure: Three tiers, and only two may block. Layers, top to bottom: Tier one: deterministic checks (blocking, under sixty seconds: schema, forbidden strings, citations resolve, templates render); Tier two: core set of 150 to 300 items (blocking on a band around a rolling baseline, one run, temperature 0, pinned model); Tier three: the full set, nightly (non-blocking: posts the result, opens an issue on a regression, never stops the merge queue). Tier two fails only when pass_rate < baseline − max(3 × run_to_run_sd, minimum_band). A gate set at 'must be above 85%' is disabled within a month; one derived from the measured spread survives arguments.\n\n## Flakiness: quarantine, never retry\n\nThere is one rule here worth being strict about. **Never re-run a failing evaluation item until it passes.** That is optional stopping wearing a CI badge, and it converts your gate into a machine for confirming whatever you hoped.\n\nRetries are legitimate for transport failures only (a 500, a timeout, a rate limit) and those should be retried by the client, counted, and reported separately, because a rising retry count is itself a signal.\n\nFor items that flip between runs, quarantine them. An item that changes verdict twice in a week moves to a quarantine list: still run, still reported, no longer blocking. Review the list weekly. A growing quarantine list means either your prompt is unstable or your rubric has an ambiguous case, and both are worth an hour.\n\n## Cost and time budgets\n\nSet both explicitly, in the config, as numbers the job enforces:\n\n- Wall clock for the blocking tiers: five minutes. Beyond that, people stop waiting and start merging around it.\n- Cost per pull request: a stated ceiling, enforced by the harness, with the run aborting and reporting rather than quietly spending.\n\nCaching by output hash does most of the work here: a pull request that changes documentation should cost nothing to evaluate. Recorded fixtures do the rest: capture real model responses once and replay them for the deterministic tier, so most CI runs need no network access, no API key, and no vendor availability.\n\n## Report a diff, not a number\n\nThe pull request comment should say which items changed verdict, not what the score was:\n\n```\ncore set: 268/300 (baseline 271, band -9)  OK\n  3 items regressed:  refund-window-hinglish, quote-with-emoji, empty-input\n  1 item improved:    long-thread-truncation\n  cost $0.31, cache hit 82%, 2 transport retries\n```\n\nThree named items are something a reviewer can open and read. A drop from 90.3% to 89.4% is something a reviewer scrolls past.\n\n## Write the override policy down\n\nSometimes a real regression should ship anyway: a deliberate trade, a fix for something worse. Decide in advance who may override, and require the override to be recorded in the pull request with a reason. An unrecorded override is indistinguishable from a broken gate three months later, when somebody is trying to work out when the quality dropped.\n\nAnd keep the honest limit in view: a green CI run says the set did not detect a regression. It says nothing about behaviour the set does not contain, which is most behaviour. That gap is what the nightly run, the production monitoring and the incident-to-item pipeline are for."},"check":{"question":"An evaluation job in CI is failing intermittently on the same three items. Which response is defensible?","options":[{"text":"Add a retry loop so each item is attempted up to three times and passes if any attempt succeeds.","correct":false,"tempting":"Retrying until success is optional stopping: it guarantees a green build regardless of quality and destroys the gate's meaning."},{"text":"Raise the pass threshold band until the intermittent failures no longer trip it.","correct":false,"tempting":"Hides a real instability by desensitising the gate, which also blinds it to the genuine regressions the band was sized to catch."},{"text":"Move those items to a quarantine list that still runs and reports but does not block, and review why they flip.","correct":true},{"text":"Remove the three items from the set, since unstable items cannot contribute to a reliable score.","correct":false,"tempting":"Deleting the evidence of instability makes the number look better and removes the signal; quarantine keeps the measurement while unblocking the queue."}]},"prev":{"slug":"the-eval-harness","title":"The harness, and why you should build the small one first","url":"https://addaly.com/learn/evaluating-ai/the-eval-harness"},"next":{"slug":"tracing-and-logging","title":"Recording enough to diagnose a failure you did not see happen","url":"https://addaly.com/learn/evaluating-ai/tracing-and-logging"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'An evaluation that runs on every change without going red at random', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evals-in-ci (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"An evaluation that runs on every change without going red at random","courseTitle":"Knowing If It Works","module":{"index":6,"slug":"evaluation-in-the-loop","title":"Evaluation in the working loop"},"answer":null,"keyIdea":"Block merges only on deterministic checks and on a band around a rolling baseline, quarantine flaky items instead of retrying them, and let the slow full run open an issue rather than stop the queue.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Three tiers, and only two may block","layers":[{"label":"Tier one: deterministic checks","detail":"blocking, under sixty seconds: schema, forbidden strings, citations resolve, templates render"},{"label":"Tier two: core set of 150 to 300 items","detail":"blocking on a band around a rolling baseline, one run, temperature 0, pinned model"},{"label":"Tier three: the full set, nightly","detail":"non-blocking: posts the result, opens an issue on a regression, never stops the merge queue"}],"caption":"Tier two fails only when pass_rate < baseline − max(3 × run_to_run_sd, minimum_band). A gate set at ‘must be above 85%’ is disabled within a month; one derived from the measured spread survives arguments."},"alt":"Figure: Three tiers, and only two may block. Layers, top to bottom: Tier one: deterministic checks (blocking, under sixty seconds: schema, forbidden strings, citations resolve, templates render); Tier two: core set of 150 to 300 items (blocking on a band around a rolling baseline, one run, temperature 0, pinned model); Tier three: the full set, nightly (non-blocking: posts the result, opens an issue on a regression, never stops the merge queue). Tier two fails only when pass_rate < baseline − max(3 × run_to_run_sd, minimum_band). A gate set at ‘must be above 85%’ is disabled within a month; one derived from the measured spread survives arguments."}},"blocks":[{"t":"h2","id":"why-the-obvious-design-fails","runs":[{"text":"Why the obvious design fails"}]},{"t":"p","runs":[{"text":"The obvious design is: run the eval set on every pull request, fail the build if the score drops. It survives about a fortnight."}]},{"t":"p","runs":[{"text":"A stochastic model produces a different score on identical code. The build goes red on a change to a README. Somebody adds a retry. Somebody else raises the threshold. Within a month the gate is either ignored or disabled, and the team is back to shipping on feeling, which is worse than never having built it, because everyone believes a gate exists."}]},{"t":"p","runs":[{"text":"The fix is to separate what can be a hard gate from what cannot."}]},{"t":"h2","id":"three-tiers","runs":[{"text":"Three tiers"}]},{"t":"p","runs":[{"text":"Tier one: deterministic checks, blocking, under sixty seconds.","b":true},{"text":" Schema validity, forbidden strings, prompt templates rendering without missing variables, citations resolving, no secrets in outputs, the judge rubric file parsing. These are unit tests. They are reproducible, they never flake, and they catch a large share of real breakages."}]},{"t":"p","runs":[{"text":"Tier two: a small core set, blocking on a band.","b":true},{"text":" 150 to 300 items, one run, temperature 0, pinned model. Do not compare against a fixed threshold; compare against a rolling baseline from the last several runs on the main branch, and fail only when the drop exceeds a band derived from the measured run-to-run spread:"}]},{"t":"code","lang":"","label":"Text","text":"fail if  pass_rate < baseline - max(3 * run_to_run_sd, minimum_band)"},{"t":"p","runs":[{"text":"Measure that spread once, the A/A test from the statistics block, and write it into the config with a comment saying when it was measured. A gate whose sensitivity is derived from data survives arguments; a gate set at “must be above 85%” does not."}]},{"t":"p","runs":[{"text":"Tier three: the full set, nightly, non-blocking.","b":true},{"text":" It runs on a schedule, posts its result, and opens an issue when it regresses. It is where the number you report comes from. It never stops the merge queue, because a two-hour job that blocks a queue gets deleted."}]},{"t":"h2","id":"flakiness-quarantine-never-retry","runs":[{"text":"Flakiness: quarantine, never retry"}]},{"t":"p","runs":[{"text":"There is one rule here worth being strict about. "},{"text":"Never re-run a failing evaluation item until it passes.","b":true},{"text":" That is optional stopping wearing a CI badge, and it converts your gate into a machine for confirming whatever you hoped."}]},{"t":"p","runs":[{"text":"Retries are legitimate for transport failures only (a 500, a timeout, a rate limit) and those should be retried by the client, counted, and reported separately, because a rising retry count is itself a signal."}]},{"t":"p","runs":[{"text":"For items that flip between runs, quarantine them. An item that changes verdict twice in a week moves to a quarantine list: still run, still reported, no longer blocking. Review the list weekly. A growing quarantine list means either your prompt is unstable or your rubric has an ambiguous case, and both are worth an hour."}]},{"t":"h2","id":"cost-and-time-budgets","runs":[{"text":"Cost and time budgets"}]},{"t":"p","runs":[{"text":"Set both explicitly, in the config, as numbers the job enforces:"}]},{"t":"ul","items":[[{"text":"Wall clock for the blocking tiers: five minutes. Beyond that, people stop waiting and start merging around it."}],[{"text":"Cost per pull request: a stated ceiling, enforced by the harness, with the run aborting and reporting rather than quietly spending."}]]},{"t":"p","runs":[{"text":"Caching by output hash does most of the work here: a pull request that changes documentation should cost nothing to evaluate. Recorded fixtures do the rest: capture real model responses once and replay them for the deterministic tier, so most CI runs need no network access, no API key, and no vendor availability."}]},{"t":"h2","id":"report-a-diff-not-a-number","runs":[{"text":"Report a diff, not a number"}]},{"t":"p","runs":[{"text":"The pull request comment should say which items changed verdict, not what the score was:"}]},{"t":"code","lang":"","label":"Text","text":"core set: 268/300 (baseline 271, band -9)  OK\n  3 items regressed:  refund-window-hinglish, quote-with-emoji, empty-input\n  1 item improved:    long-thread-truncation\n  cost $0.31, cache hit 82%, 2 transport retries"},{"t":"p","runs":[{"text":"Three named items are something a reviewer can open and read. A drop from 90.3% to 89.4% is something a reviewer scrolls past."}]},{"t":"h2","id":"write-the-override-policy-down","runs":[{"text":"Write the override policy down"}]},{"t":"p","runs":[{"text":"Sometimes a real regression should ship anyway: a deliberate trade, a fix for something worse. Decide in advance who may override, and require the override to be recorded in the pull request with a reason. An unrecorded override is indistinguishable from a broken gate three months later, when somebody is trying to work out when the quality dropped."}]},{"t":"p","runs":[{"text":"And keep the honest limit in view: a green CI run says the set did not detect a regression. It says nothing about behaviour the set does not contain, which is most behaviour. That gap is what the nightly run, the production monitoring and the incident-to-item pipeline are for."}]}],"check":{"question":[{"text":"An evaluation job in CI is failing intermittently on the same three items. Which response is defensible?"}],"options":[{"text":[{"text":"Add a retry loop so each item is attempted up to three times and passes if any attempt succeeds."}],"correct":false,"tempting":[{"text":"Retrying until success is optional stopping: it guarantees a green build regardless of quality and destroys the gate’s meaning."}]},{"text":[{"text":"Raise the pass threshold band until the intermittent failures no longer trip it."}],"correct":false,"tempting":[{"text":"Hides a real instability by desensitising the gate, which also blinds it to the genuine regressions the band was sized to catch."}]},{"text":[{"text":"Move those items to a quarantine list that still runs and reports but does not block, and review why they flip."}],"correct":true,"tempting":null},{"text":[{"text":"Remove the three items from the set, since unstable items cannot contribute to a reliable score."}],"correct":false,"tempting":[{"text":"Deleting the evidence of instability makes the number look better and removes the signal; quarantine keeps the measurement while unblocking the queue."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"the-eval-harness","title":"The harness, and why you should build the small one first","minutes":9},"next":{"slug":"tracing-and-logging","title":"Recording enough to diagnose a failure you did not see happen","minutes":9}}}