{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"the-harness-in-ci","title":"The harness in CI, and why temperature zero is not deterministic","url":"https://addaly.com/learn/building-with-ai/the-harness-in-ci","position":57,"of":80,"minutes":9,"answer":"A model at temperature zero still returns different tokens on different runs, because GPU arithmetic under batching is not bit-for-bit repeatable and providers change models under the same name.","keyIdea":"A model at temperature zero still returns different tokens on different runs, because GPU arithmetic under batching is not bit-for-bit repeatable and providers change models under the same name, so a CI harness must judge by pass rate over repeated runs with cached responses for unchanged cases, or it goes red at random and gets switched off."},"body":{"format":"markdown","text":"## What \"run on every change\" requires\n\nYou have a case file, an assertion layer, a judge with a calibration figure, and a retrieval gold set. A harness is those four wired to one command that exits zero or non-zero, run automatically on every commit that touches a prompt, the retrieval configuration, a tool definition or the model name. That is the whole idea, and the difficulty is not the wiring. It is that the thing under test is not deterministic, and a test suite that fails at random is disabled within a month.\n\n## Temperature zero is not determinism\n\nSetting temperature to zero makes the sampler pick the highest-probability token every step. It does not make the probabilities the same every run. Three mechanisms, each real.\n\nGPU arithmetic is not associative in floating point: adding the same numbers in a different order gives a slightly different result, and the order depends on how the provider batched your request with other people's. A batch of eight and a batch of thirty-two run different kernels with different reduction orders. When two candidate tokens are close in probability, that last decimal place flips the choice, and once one token differs the rest of the output diverges.\n\nMixture-of-experts models route tokens to different sub-networks, and that routing can shift with the batch in the same way.\n\nAnd providers update models under the same name. The alias that meant one set of weights on Monday can mean another on Thursday, with no change in your code.\n\nThe practical shape: outputs are usually identical for the first few dozen tokens and diverge later, more often on long outputs, and a case that passed a hundred times fails once. A suite that treats that one failure as red teaches the team to ignore red.\n\n## Pass rates, not pass or fail\n\nRun each case several times (three is a working number) and count a case as passing if it passes at least two. Then judge the suite by its pass rate against a threshold rather than by any single case: 94 per cent, say, with the number chosen from a week of runs on a known-good version so you know its natural variation. Golden cases are the exception: a golden failure in two of three runs blocks the merge, because golden cases define correct behaviour and a change that breaks one is wrong.\n\nTrend the pass rate across commits. A drift from 97 to 93 over a month, with no single red run, is a regression that per-run thresholds never show.\n\n## Cache the responses\n\nKey each model response by a hash of the rendered prompt, the model identifier and the parameters, and store it. A commit that changes one case, or one line of a prompt used by ten cases, re-runs only what the hash says changed. A run that touches nothing is free. Invalidate the cache when the model identifier changes, and pin the identifier to a dated snapshot rather than a moving alias, so the cache and the results mean the same thing tomorrow.\n\n## The cost of a run\n\nFifty cases, three runs each, about 2,500 tokens a case: 375,000 tokens, which is a few dollars with a frontier model and well under one with a mid-sized model. The judge adds a similar amount. A team making twenty prompt changes a week spends tens of dollars a month on the harness, and with response caching most of that is spent only on the cases that changed. It is the cheapest insurance in the product; budget it explicitly so nobody turns it off to save a bill they never looked at.\n\n## What blocks a merge\n\n- A golden case failing in a majority of runs.\n- A regression case failing: a bug that was fixed has returned, and the case id says which.\n- The pass rate falling more than a few points below the base branch, with the diff of failing case ids in the report.\n- Cost per case rising more than a fifth, or output length rising sharply: a prompt change that doubled verbosity passes every quality check and doubles the bill.\n- Latency at the 95th percentile rising sharply.\n\nEverything else is a warning in the report, not a block.\n\n## Flakiness is the enemy\n\nTrack the flake rate: cases whose verdict changed with no change to the system. If a case flakes more than one run in ten, quarantine it with an issue attached, and fix it: usually by tightening a vague rubric or replacing an exact-string check with a pattern. Have one rule the team does not bend: nobody merges with the suite skipped. The first skip \"just this once\" is the end of the harness.\n\n## The report\n\nEvery run produces the same page: pass rate by tag, failing case ids with the assertion or rubric criterion that failed, cost and latency totals, and the diff against the base branch. That page is what the reviewer reads; a green tick without it tells nobody why they should believe it."},"check":{"question":"A CI suite runs each case once at temperature zero and fails the build on any single failure. One case fails roughly once a week with no code change. What is happening, and what should change?","options":[{"text":"The provider is rate-limiting the test runs and returning truncated output; add retries with backoff.","correct":false,"tempting":"Rate limits produce errors, not plausible-but-different completions; the symptom is divergence, not truncation."},{"text":"The case's expected string has a typo that only matches some phrasings; fix the expectation and keep single runs, since the rest of the suite is stable.","correct":false,"tempting":"May explain one case, but the underlying non-determinism affects every case and single runs will keep flaking."},{"text":"Temperature zero is being ignored by the API, so the parameter should be set explicitly in every request.","correct":false,"tempting":"The parameter works; it fixes the sampling rule, not the floating-point arithmetic underneath it."},{"text":"Outputs at temperature zero are not repeatable under batching, so run cases several times and gate on a pass rate.","correct":true}]},"prev":{"slug":"measuring-retrieval-on-its-own","title":"Measuring retrieval on its own","url":"https://addaly.com/learn/building-with-ai/measuring-retrieval-on-its-own"},"next":{"slug":"traces-you-can-debug-from","title":"Traces you can debug from","url":"https://addaly.com/learn/building-with-ai/traces-you-can-debug-from"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The harness in CI, and why temperature zero is not deterministic', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/the-harness-in-ci (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The harness in CI, and why temperature zero is not deterministic","courseTitle":"Building with AI","module":{"index":6,"slug":"evaluating-and-iterating","title":"Evaluating and iterating"},"answer":null,"keyIdea":"A model at temperature zero still returns different tokens on different runs, because GPU arithmetic under batching is not bit-for-bit repeatable and providers change models under the same name, so a CI harness must judge by pass rate over repeated runs with cached responses for unchanged cases, or it goes red at random and gets switched off.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-run-on-every-change-requires","runs":[{"text":"What “run on every change” requires"}]},{"t":"p","runs":[{"text":"You have a case file, an assertion layer, a judge with a calibration figure, and a retrieval gold set. A harness is those four wired to one command that exits zero or non-zero, run automatically on every commit that touches a prompt, the retrieval configuration, a tool definition or the model name. That is the whole idea, and the difficulty is not the wiring. It is that the thing under test is not deterministic, and a test suite that fails at random is disabled within a month."}]},{"t":"h2","id":"temperature-zero-is-not-determinism","runs":[{"text":"Temperature zero is not determinism"}]},{"t":"p","runs":[{"text":"Setting temperature to zero makes the sampler pick the highest-probability token every step. It does not make the probabilities the same every run. Three mechanisms, each real."}]},{"t":"p","runs":[{"text":"GPU arithmetic is not associative in floating point: adding the same numbers in a different order gives a slightly different result, and the order depends on how the provider batched your request with other people’s. A batch of eight and a batch of thirty-two run different kernels with different reduction orders. When two candidate tokens are close in probability, that last decimal place flips the choice, and once one token differs the rest of the output diverges."}]},{"t":"p","runs":[{"text":"Mixture-of-experts models route tokens to different sub-networks, and that routing can shift with the batch in the same way."}]},{"t":"p","runs":[{"text":"And providers update models under the same name. The alias that meant one set of weights on Monday can mean another on Thursday, with no change in your code."}]},{"t":"p","runs":[{"text":"The practical shape: outputs are usually identical for the first few dozen tokens and diverge later, more often on long outputs, and a case that passed a hundred times fails once. A suite that treats that one failure as red teaches the team to ignore red."}]},{"t":"h2","id":"pass-rates-not-pass-or-fail","runs":[{"text":"Pass rates, not pass or fail"}]},{"t":"p","runs":[{"text":"Run each case several times (three is a working number) and count a case as passing if it passes at least two. Then judge the suite by its pass rate against a threshold rather than by any single case: 94 per cent, say, with the number chosen from a week of runs on a known-good version so you know its natural variation. Golden cases are the exception: a golden failure in two of three runs blocks the merge, because golden cases define correct behaviour and a change that breaks one is wrong."}]},{"t":"p","runs":[{"text":"Trend the pass rate across commits. A drift from 97 to 93 over a month, with no single red run, is a regression that per-run thresholds never show."}]},{"t":"h2","id":"cache-the-responses","runs":[{"text":"Cache the responses"}]},{"t":"p","runs":[{"text":"Key each model response by a hash of the rendered prompt, the model identifier and the parameters, and store it. A commit that changes one case, or one line of a prompt used by ten cases, re-runs only what the hash says changed. A run that touches nothing is free. Invalidate the cache when the model identifier changes, and pin the identifier to a dated snapshot rather than a moving alias, so the cache and the results mean the same thing tomorrow."}]},{"t":"h2","id":"the-cost-of-a-run","runs":[{"text":"The cost of a run"}]},{"t":"p","runs":[{"text":"Fifty cases, three runs each, about 2,500 tokens a case: 375,000 tokens, which is a few dollars with a frontier model and well under one with a mid-sized model. The judge adds a similar amount. A team making twenty prompt changes a week spends tens of dollars a month on the harness, and with response caching most of that is spent only on the cases that changed. It is the cheapest insurance in the product; budget it explicitly so nobody turns it off to save a bill they never looked at."}]},{"t":"h2","id":"what-blocks-a-merge","runs":[{"text":"What blocks a merge"}]},{"t":"ul","items":[[{"text":"A golden case failing in a majority of runs."}],[{"text":"A regression case failing: a bug that was fixed has returned, and the case id says which."}],[{"text":"The pass rate falling more than a few points below the base branch, with the diff of failing case ids in the report."}],[{"text":"Cost per case rising more than a fifth, or output length rising sharply: a prompt change that doubled verbosity passes every quality check and doubles the bill."}],[{"text":"Latency at the 95th percentile rising sharply."}]]},{"t":"p","runs":[{"text":"Everything else is a warning in the report, not a block."}]},{"t":"h2","id":"flakiness-is-the-enemy","runs":[{"text":"Flakiness is the enemy"}]},{"t":"p","runs":[{"text":"Track the flake rate: cases whose verdict changed with no change to the system. If a case flakes more than one run in ten, quarantine it with an issue attached, and fix it: usually by tightening a vague rubric or replacing an exact-string check with a pattern. Have one rule the team does not bend: nobody merges with the suite skipped. The first skip “just this once” is the end of the harness."}]},{"t":"h2","id":"the-report","runs":[{"text":"The report"}]},{"t":"p","runs":[{"text":"Every run produces the same page: pass rate by tag, failing case ids with the assertion or rubric criterion that failed, cost and latency totals, and the diff against the base branch. That page is what the reviewer reads; a green tick without it tells nobody why they should believe it."}]}],"check":{"question":[{"text":"A CI suite runs each case once at temperature zero and fails the build on any single failure. One case fails roughly once a week with no code change. What is happening, and what should change?"}],"options":[{"text":[{"text":"The provider is rate-limiting the test runs and returning truncated output; add retries with backoff."}],"correct":false,"tempting":[{"text":"Rate limits produce errors, not plausible-but-different completions; the symptom is divergence, not truncation."}]},{"text":[{"text":"The case’s expected string has a typo that only matches some phrasings; fix the expectation and keep single runs, since the rest of the suite is stable."}],"correct":false,"tempting":[{"text":"May explain one case, but the underlying non-determinism affects every case and single runs will keep flaking."}]},{"text":[{"text":"Temperature zero is being ignored by the API, so the parameter should be set explicitly in every request."}],"correct":false,"tempting":[{"text":"The parameter works; it fixes the sampling rule, not the floating-point arithmetic underneath it."}]},{"text":[{"text":"Outputs at temperature zero are not repeatable under batching, so run cases several times and gate on a pass rate."}],"correct":true,"tempting":null}],"order":[0,2,3,1],"why":null},"prev":{"slug":"measuring-retrieval-on-its-own","title":"Measuring retrieval on its own","minutes":8},"next":{"slug":"traces-you-can-debug-from","title":"Traces you can debug from","minutes":9}}}