A small harness that tells you whether a change helped

Impressions do not survive a week

Every lesson in this block ends with “measure it”. This one is the measuring apparatus: thirty items, a script, and a file of results. It takes an afternoon to build and it is the difference between knowing whether the new model is better and believing it is.

Without it, every question in this course becomes an argument: is Q4 worse than Q5, did the prompt rewrite help, is the new release better, did the fine-tune work. With it, each is a ten-minute run.

The shape

A JSON Lines file, one item per line:

{"id": "ext-001", "kind": "extract", "prompt": "...", "expect": {"total": 4500}}
{"id": "cls-014", "kind": "classify", "prompt": "...", "expect": "billing"}
{"id": "sum-007", "kind": "summary", "prompt": "...", "must_mention": ["refund", "14 days"]}

Three kinds of check, in descending order of how much you should rely on them:

Exact or structural. Does the JSON parse, does the field equal the expected value, does the code compile, does the label match. Objective, free to run, and where most of your items should live.

Property-based. Does the summary mention these two facts, is the output under 100 words, does it avoid a forbidden phrase, is the number in range. Cheap, robust to wording, and the right tool for prose.

Human graded. Three-point scale: usable, usable after a small edit, not usable. Reserve for the items where the first two cannot say anything, because this is the expensive one.

Resist the urge to use a model to grade the outputs at this scale. With thirty items you can read them yourself in fifteen minutes, and a judge model on a local setup adds a second unreliable component to a measurement you are trying to make trustworthy.

The runner

import json, time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="x")
CONFIG = {"model": "local", "temperature": 0, "max_tokens": 512}

rows = [json.loads(l) for l in open("evalset.jsonl")]
out = []
for r in rows:
    t0 = time.time()
    resp = client.chat.completions.create(
        messages=[{"role": "user", "content": r["prompt"]}], **CONFIG)
    out.append({**r,
                "got": resp.choices[0].message.content,
                "secs": round(time.time() - t0, 2),
                "in_tok": resp.usage.prompt_tokens,
                "out_tok": resp.usage.completion_tokens})

json.dump(out, open("run-2026-09-08-q4.json", "w"), indent=1)

Thirty lines including the scorer. Name the output file after the configuration, because in three weeks you will not remember which run was which.

Record latency and token counts alongside quality. Half the decisions in this course are quality-against-speed, and a harness that measures only one of them cannot answer them.

Reading a small sample honestly

Thirty items is a small number and it is worth being clear about what it can detect.

A change of one or two items is noise. A change of eight is real. In between, you have not learned much; and the honest response is either to add items or to accept that the difference is below what you can measure and choose on another basis, such as speed or memory.

When two configurations come out within a couple of items of each other, that is a genuine result: it says the difference does not matter for your work. Take the cheaper one.

Keeping it useful

Add every real failure. When something goes wrong in production, that input becomes item thirty-one. The set grows in exactly the places where your system is weak, which is the correct growth pattern.

Freeze the items. Do not edit an item because a model failed it. If the expected answer was wrong, fix it and note that you did; if the model was wrong, leave it.

Never tune against the whole set. Keep ten items you do not look at while iterating. Otherwise you will optimise a prompt for thirty specific inputs and learn nothing about the thirty-first.

Re-run after every engine or driver upgrade. Regressions happen, and this is how you find out in ten minutes rather than through a user report a fortnight later.

What this connects to

This is the smallest useful version of a much larger subject (evaluation design, statistical significance, judge models, per-group reporting) which the evaluation course on this site covers properly. What matters here is that you have something, run consistently, written down. Thirty items and a text file beat any amount of careful reasoning about which quant ought to be better.