The harness in CI, and why temperature zero is not deterministic

What “run on every change” requires

You have a case file, an assertion layer, a judge with a calibration figure, and a retrieval gold set. A harness is those four wired to one command that exits zero or non-zero, run automatically on every commit that touches a prompt, the retrieval configuration, a tool definition or the model name. That is the whole idea, and the difficulty is not the wiring. It is that the thing under test is not deterministic, and a test suite that fails at random is disabled within a month.

Temperature zero is not determinism

Setting temperature to zero makes the sampler pick the highest-probability token every step. It does not make the probabilities the same every run. Three mechanisms, each real.

GPU arithmetic is not associative in floating point: adding the same numbers in a different order gives a slightly different result, and the order depends on how the provider batched your request with other people’s. A batch of eight and a batch of thirty-two run different kernels with different reduction orders. When two candidate tokens are close in probability, that last decimal place flips the choice, and once one token differs the rest of the output diverges.

Mixture-of-experts models route tokens to different sub-networks, and that routing can shift with the batch in the same way.

And providers update models under the same name. The alias that meant one set of weights on Monday can mean another on Thursday, with no change in your code.

The practical shape: outputs are usually identical for the first few dozen tokens and diverge later, more often on long outputs, and a case that passed a hundred times fails once. A suite that treats that one failure as red teaches the team to ignore red.

Pass rates, not pass or fail

Run each case several times (three is a working number) and count a case as passing if it passes at least two. Then judge the suite by its pass rate against a threshold rather than by any single case: 94 per cent, say, with the number chosen from a week of runs on a known-good version so you know its natural variation. Golden cases are the exception: a golden failure in two of three runs blocks the merge, because golden cases define correct behaviour and a change that breaks one is wrong.

Trend the pass rate across commits. A drift from 97 to 93 over a month, with no single red run, is a regression that per-run thresholds never show.

Cache the responses

Key each model response by a hash of the rendered prompt, the model identifier and the parameters, and store it. A commit that changes one case, or one line of a prompt used by ten cases, re-runs only what the hash says changed. A run that touches nothing is free. Invalidate the cache when the model identifier changes, and pin the identifier to a dated snapshot rather than a moving alias, so the cache and the results mean the same thing tomorrow.

The cost of a run

Fifty cases, three runs each, about 2,500 tokens a case: 375,000 tokens, which is a few dollars with a frontier model and well under one with a mid-sized model. The judge adds a similar amount. A team making twenty prompt changes a week spends tens of dollars a month on the harness, and with response caching most of that is spent only on the cases that changed. It is the cheapest insurance in the product; budget it explicitly so nobody turns it off to save a bill they never looked at.

What blocks a merge

• A golden case failing in a majority of runs.
• A regression case failing: a bug that was fixed has returned, and the case id says which.
• The pass rate falling more than a few points below the base branch, with the diff of failing case ids in the report.
• Cost per case rising more than a fifth, or output length rising sharply: a prompt change that doubled verbosity passes every quality check and doubles the bill.
• Latency at the 95th percentile rising sharply.

Everything else is a warning in the report, not a block.

Flakiness is the enemy

Track the flake rate: cases whose verdict changed with no change to the system. If a case flakes more than one run in ten, quarantine it with an issue attached, and fix it: usually by tightening a vague rubric or replacing an exact-string check with a pattern. Have one rule the team does not bend: nobody merges with the suite skipped. The first skip “just this once” is the end of the harness.

The report

Every run produces the same page: pass rate by tag, failing case ids with the assertion or rubric criterion that failed, cost and latency totals, and the diff against the base branch. That page is what the reviewer reads; a green tick without it tells nobody why they should believe it.