An evaluation that runs on every change without going red at random

Why the obvious design fails

The obvious design is: run the eval set on every pull request, fail the build if the score drops. It survives about a fortnight.

A stochastic model produces a different score on identical code. The build goes red on a change to a README. Somebody adds a retry. Somebody else raises the threshold. Within a month the gate is either ignored or disabled, and the team is back to shipping on feeling, which is worse than never having built it, because everyone believes a gate exists.

The fix is to separate what can be a hard gate from what cannot.

Three tiers

Tier one: deterministic checks, blocking, under sixty seconds. Schema validity, forbidden strings, prompt templates rendering without missing variables, citations resolving, no secrets in outputs, the judge rubric file parsing. These are unit tests. They are reproducible, they never flake, and they catch a large share of real breakages.

Tier two: a small core set, blocking on a band. 150 to 300 items, one run, temperature 0, pinned model. Do not compare against a fixed threshold; compare against a rolling baseline from the last several runs on the main branch, and fail only when the drop exceeds a band derived from the measured run-to-run spread:

fail if  pass_rate < baseline - max(3 * run_to_run_sd, minimum_band)

Measure that spread once, the A/A test from the statistics block, and write it into the config with a comment saying when it was measured. A gate whose sensitivity is derived from data survives arguments; a gate set at “must be above 85%” does not.

Tier three: the full set, nightly, non-blocking. It runs on a schedule, posts its result, and opens an issue when it regresses. It is where the number you report comes from. It never stops the merge queue, because a two-hour job that blocks a queue gets deleted.

Flakiness: quarantine, never retry

There is one rule here worth being strict about. Never re-run a failing evaluation item until it passes. That is optional stopping wearing a CI badge, and it converts your gate into a machine for confirming whatever you hoped.

Retries are legitimate for transport failures only (a 500, a timeout, a rate limit) and those should be retried by the client, counted, and reported separately, because a rising retry count is itself a signal.

For items that flip between runs, quarantine them. An item that changes verdict twice in a week moves to a quarantine list: still run, still reported, no longer blocking. Review the list weekly. A growing quarantine list means either your prompt is unstable or your rubric has an ambiguous case, and both are worth an hour.

Cost and time budgets

Set both explicitly, in the config, as numbers the job enforces:

• Wall clock for the blocking tiers: five minutes. Beyond that, people stop waiting and start merging around it.
• Cost per pull request: a stated ceiling, enforced by the harness, with the run aborting and reporting rather than quietly spending.

Caching by output hash does most of the work here: a pull request that changes documentation should cost nothing to evaluate. Recorded fixtures do the rest: capture real model responses once and replay them for the deterministic tier, so most CI runs need no network access, no API key, and no vendor availability.

Report a diff, not a number

The pull request comment should say which items changed verdict, not what the score was:

core set: 268/300 (baseline 271, band -9)  OK
  3 items regressed:  refund-window-hinglish, quote-with-emoji, empty-input
  1 item improved:    long-thread-truncation
  cost $0.31, cache hit 82%, 2 transport retries

Three named items are something a reviewer can open and read. A drop from 90.3% to 89.4% is something a reviewer scrolls past.

Write the override policy down

Sometimes a real regression should ship anyway: a deliberate trade, a fix for something worse. Decide in advance who may override, and require the override to be recorded in the pull request with a reason. An unrecorded override is indistinguishable from a broken gate three months later, when somebody is trying to work out when the quality dropped.

And keep the honest limit in view: a green CI run says the set did not detect a regression. It says nothing about behaviour the set does not contain, which is most behaviour. That gap is what the nightly run, the production monitoring and the incident-to-item pipeline are for.