The harness, and why you should build the small one first

What a run is

Everything in this course reduces to one operation: take a dataset, run a system over it, score each item, summarise. The harness is the code that does that reproducibly. It is smaller than you think (a first version is about eighty lines) and getting it right early is what makes every later lesson cheap.

A run is identified by everything that could change the answer:

{"run_id": "2026-09-06T11:04Z-a17c",
 "dataset": "support-v4", "dataset_sha": "9f21c0",
 "system": "reply-agent", "git_sha": "e4b1aa",
 "prompt_version": "reply-v9",
 "model": "claude-haiku-4-5-20260101", "temperature": 0.2,
 "judge_model": "...", "judge_prompt_version": "rubric-v3",
 "seed": 7, "n": 100}

If any of those is missing, two runs that disagree cannot be explained, and you will spend an afternoon finding out that somebody edited the prompt. The dataset hash matters as much as the git sha: a changed set with an unchanged name is the most confusing bug in this whole discipline.

Store per item, summarise later

Write one record per item, not just a total:

{"run_id": "...", "item_id": "tkt-0041", "output": "...",
 "checks": {"json_valid": true, "language_match": true,
            "r1_no_commitment": false},
 "pass": false, "latency_ms": 1840, "tokens_in": 1204,
 "tokens_out": 210, "cost_usd": 0.0031}

Everything else is a query over these. Per-slice pass rates, the bootstrap interval from module one, the flip counts between two runs: all of them need per-item records, and none of them can be recovered from a stored average. JSONL files in a directory are a perfectly good store for the first year; a table in Postgres once you want to query across runs.

Cache, or you will not run it

The difference between an eval you run on every change and one you run monthly is how long it takes. Cache on a hash of everything that determines an output:

key = sha256(f"{model}|{temperature}|{prompt_version}|{item_id}|{input}")

Change the prompt and every item re-runs. Add ten items and only those ten cost anything. Re-run to regenerate a report and it is instant and free. A cache turns a twelve-minute, four-dollar run into a nine-second one, and that is the difference between a tool people use and a ritual people avoid.

Keep an explicit --no-cache flag, and use it when you want to measure run-to-run variance.

Concurrency, retries and the failures that look like quality

Run items concurrently (twenty at a time is usually safe) and the wall clock drops from twelve minutes to under one. Two disciplines go with it:

• Retry transient errors (429, 500, timeouts) with backoff, and count them separately. An item that failed because of a rate limit is not a quality failure, and silently scoring it zero drops your number for reasons that have nothing to do with the system.
• Never retry a bad answer. Retrying until something passes is not evaluation.

Where it runs

Three tiers, matching the cost tiers from the previous module:

• Every pull request: the deterministic checks on the full dev set. Seconds, free, blocking. If schema validity drops, the build fails.
• Nightly, or on merge to main: the judge set. Minutes, a few dollars. Posts a comparison against the previous run: the total, and which items flipped.
• Monthly or per release: the test set, plus whatever human review you have committed to. Slow, expensive, decisive.

The first tier is the one that changes behaviour, because it is the only one that catches a mistake before it is merged.

Report the diff, not the number

The output of a run should be a comparison, because a number alone provokes no action:

reply-v9 vs reply-v8   (support-v4, n=100, 3 runs)

  pass rate   82.0% -> 86.3%   (+4.3, CI on diff +0.8 to +7.9)
  improved    9 items
  regressed   4 items   <- read these
  R1 gate     0 -> 0 violations
  p95 latency 2.9s -> 4.4s     <- got slower
  cost/item   $0.0031 -> $0.0068

Four regressions on a run that improved overall is exactly the information a total hides, and the latency and cost lines stop a quality win from quietly becoming a product loss.

Build the small one first

You will be tempted to adopt a platform. The free ones are good: promptfoo (MIT, YAML-driven, up in twenty minutes), Inspect (UK AISI, MIT, the most programmable), DeepEval (pytest-shaped), Langfuse (self-hostable tracing plus evals). The commercial ones (Braintrust, LangSmith) add hosted dashboards and collaboration.

The advice is still to write the eighty lines first. Not because the tools are bad, but because a platform adopted before you know what you are measuring quietly decides what you measure: you will use the metrics it ships with. Write your own loop, learn what your system’s failures actually look like, and then adopt a tool to stop maintaining plumbing. Migrating a JSONL set into any of these is an hour.