Evaluating your own AI feature

The problem with trying it a few times

You change a prompt, try three examples, they look better, you ship. Two weeks later something else is worse and nobody can say when it broke. A prompt is program logic with no type system and no test suite, and you just deployed it on the strength of three anecdotes.

The fix is not sophisticated. It is fifty saved examples and a script.

Build the golden set this afternoon

Take fifty real inputs from your logs. Not fifty you invented — real ones, including the boring ones and the ones you already know go wrong. For each, write down what a good output requires. Sometimes that is an exact answer. More often it is a set of properties: names the right policy, stays under 80 words, does not invent a phone number.

Fifty is not statistically impressive and it is enormously better than zero. Every bug you fix from now on gets added, so the suite grows into a memory of everything you have already broken.

Prefer checks that cannot argue back

Rank your checks by how much they can lie to you.

Deterministic assertions first. They are free, instant and never drift.

def checks(case, out):
    yield "parses", is_valid_json(out)
    yield "cites_real_doc", out.get("doc_id") in case["allowed_docs"]
    yield "length", len(out["reply"].split()) <= 80
    yield "no_phone", not re.search(r"\+?\d[\d \-]{7,}", out["reply"])
    yield "quote_grounded", out["source_quote"] in case["input"]

You can check far more than people expect this way: format, grounding, refusal on the cases that should be refused, absence of banned strings, whether the right tool was called.

Component metrics second. If you built retrieval, measure recall@k on the golden questions separately from answer quality. A single end-to-end number tells you the feature got worse; two numbers tell you which half to fix.

A model judge last, for the genuinely fuzzy part.

cases = json.load(open("evals/support.json"))
failed = collections.defaultdict(list)
for c in cases:
    out = feature(c["input"])
    for name, ok in checks(c, out):
        if not ok:
            failed[name].append(c["id"])
for name, ids in sorted(failed.items()):
    print(f"{name:16} {len(ids):3} failures  {ids[:5]}")

Be honest about model judges

Using a model to grade another model's output works, and it has known biases you must design around:

• Judges prefer long answers. Almost every naive judge rewards thoroughness, so any change that makes replies longer raises your score without making anything better for users.
• Absolute scores drift. A 1-to-5 rating means something slightly different each time you touch the judge prompt, so scores are not comparable across weeks.
• Pairwise is steadier. Show the judge output A and output B for the same input and ask which better satisfies a specific criterion. Swap the order on half the cases, because judges also favour whichever answer came first.
• Calibrate before trusting. Label thirty cases yourself, run the judge on them, and see how often it agrees. If it agrees 60% of the time, it is not measuring your feature, it is adding noise. Re-check whenever you edit the judge prompt.

Measure cost and latency in the same run

Record tokens and wall time for every case. A change that improves quality and triples cost is a decision for someone to make, not a win to announce. Print it next to the pass rate so the trade-off is visible at the moment of choosing.

Make it a gate

Run the suite before and after every prompt change, and in CI if you can. The bar is not perfection. The bar is: you know the number, you know which cases moved, and you can say why you accepted it.