{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-your-feature","title":"Evaluating your own AI feature","url":"https://addaly.com/learn/building-with-ai/evaluating-your-feature","position":52,"of":80,"minutes":8,"answer":"A prompt change is a code change; if you cannot re-run fifty saved cases, you are guessing.","keyIdea":"A prompt change is a code change; if you cannot re-run fifty saved cases, you are guessing."},"body":{"format":"markdown","text":"## The problem with trying it a few times\n\nYou change a prompt, try three examples, they look better, you ship. Two weeks later something else is worse and nobody can say when it broke. A prompt is program logic with no type system and no test suite, and you just deployed it on the strength of three anecdotes.\n\nThe fix is not sophisticated. It is fifty saved examples and a script.\n\n## Build the golden set this afternoon\n\nTake fifty real inputs from your logs. Not fifty you invented: real ones, including the boring ones and the ones you already know go wrong. For each, write down what a good output requires. Sometimes that is an exact answer. More often it is a set of properties: names the right policy, stays under 80 words, does not invent a phone number.\n\nFifty is not statistically impressive and it is enormously better than zero. Every bug you fix from now on gets added, so the suite grows into a memory of everything you have already broken.\n\n## Prefer checks that cannot argue back\n\nRank your checks by how much they can lie to you.\n\n**Deterministic assertions first.** They are free, instant and never drift.\n\n```python\ndef checks(case, out):\n    yield \"parses\", is_valid_json(out)\n    yield \"cites_real_doc\", out.get(\"doc_id\") in case[\"allowed_docs\"]\n    yield \"length\", len(out[\"reply\"].split()) <= 80\n    yield \"no_phone\", not re.search(r\"\\+?\\d[\\d \\-]{7,}\", out[\"reply\"])\n    yield \"quote_grounded\", out[\"source_quote\"] in case[\"input\"]\n```\n\nYou can check far more than people expect this way: format, grounding, refusal on the cases that should be refused, absence of banned strings, whether the right tool was called.\n\n**Component metrics second.** If you built retrieval, measure recall@k on the golden questions separately from answer quality. A single end-to-end number tells you the feature got worse; two numbers tell you which half to fix.\n\n**A model judge last**, for the fuzzy part.\n\n```python\ncases = json.load(open(\"evals/support.json\"))\nfailed = collections.defaultdict(list)\nfor c in cases:\n    out = feature(c[\"input\"])\n    for name, ok in checks(c, out):\n        if not ok:\n            failed[name].append(c[\"id\"])\nfor name, ids in sorted(failed.items()):\n    print(f\"{name:16} {len(ids):3} failures  {ids[:5]}\")\n```\n\n## Be honest about model judges\n\nUsing a model to grade another model's output works, and it has known biases you must design around:\n\n- **Judges prefer long answers.** Almost every naive judge rewards thoroughness, so any change that makes replies longer raises your score without making anything better for users.\n- **Absolute scores drift.** A 1-to-5 rating means something slightly different each time you touch the judge prompt, so scores are not comparable across weeks.\n- **Pairwise is steadier.** Show the judge output A and output B for the same input and ask which better satisfies a specific criterion. Swap the order on half the cases, because judges also favour whichever answer came first.\n- **Calibrate before trusting.** Label thirty cases yourself, run the judge on them, and see how often it agrees. If it agrees 60% of the time, it is not measuring your feature, it is adding noise. Re-check whenever you edit the judge prompt.\n\n## Measure cost and latency in the same run\n\nRecord tokens and wall time for every case. A change that improves quality and triples cost is a decision for someone to make, not a win to announce. Print it next to the pass rate so the trade-off is visible at the moment of choosing.\n\n## Make it a gate\n\nRun the suite before and after every prompt change, and in CI if you can. The bar is not perfection. The bar is: you know the number, you know which cases moved, and you can say why you accepted it."},"check":{"question":"A team grades their support bot with a model judge scoring 1 to 5. After a prompt change the average rises from 3.6 to 4.1, so they ship. Within a week users complain the answers have become waffly and hard to skim. What most likely happened?","options":[{"text":"The judge model is not capable enough to notice waffle; a stronger judge would have caught it.","correct":false,"tempting":"Judging feels like a capability problem, so a smarter grader sounds like the obvious upgrade rather than a criteria problem."},{"text":"Fifty cases is too small a sample; more examples would have revealed the drop.","correct":false,"tempting":"Small samples are the reflex explanation for any misleading number, and sometimes they genuinely are the cause."},{"text":"Users are reacting to slower responses rather than worse ones, and the judge measured quality correctly.","correct":false,"tempting":"Longer answers really are slower to arrive and to read, so the complaint can plausibly be redescribed as a latency issue."},{"text":"The judge rewarded length and thoroughness, which the prompt change increased, so the score moved without the thing users cared about moving.","correct":true}]},"prev":{"slug":"feedback-signals-and-what-they-measure","title":"Thumbs up, thumbs down, and what the buttons actually measure","url":"https://addaly.com/learn/building-with-ai/feedback-signals-and-what-they-measure"},"next":{"slug":"the-first-fifty-cases","title":"The first fifty cases, and where they come from","url":"https://addaly.com/learn/building-with-ai/the-first-fifty-cases"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Evaluating your own AI feature', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/evaluating-your-feature (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Evaluating your own AI feature","courseTitle":"Building with AI","module":{"index":6,"slug":"evaluating-and-iterating","title":"Evaluating and iterating"},"answer":null,"keyIdea":"A prompt change is a code change; if you cannot re-run fifty saved cases, you are guessing.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-problem-with-trying-it-a-few-times","runs":[{"text":"The problem with trying it a few times"}]},{"t":"p","runs":[{"text":"You change a prompt, try three examples, they look better, you ship. Two weeks later something else is worse and nobody can say when it broke. A prompt is program logic with no type system and no test suite, and you just deployed it on the strength of three anecdotes."}]},{"t":"p","runs":[{"text":"The fix is not sophisticated. It is fifty saved examples and a script."}]},{"t":"h2","id":"build-the-golden-set-this-afternoon","runs":[{"text":"Build the golden set this afternoon"}]},{"t":"p","runs":[{"text":"Take fifty real inputs from your logs. Not fifty you invented: real ones, including the boring ones and the ones you already know go wrong. For each, write down what a good output requires. Sometimes that is an exact answer. More often it is a set of properties: names the right policy, stays under 80 words, does not invent a phone number."}]},{"t":"p","runs":[{"text":"Fifty is not statistically impressive and it is enormously better than zero. Every bug you fix from now on gets added, so the suite grows into a memory of everything you have already broken."}]},{"t":"h2","id":"prefer-checks-that-cannot-argue-back","runs":[{"text":"Prefer checks that cannot argue back"}]},{"t":"p","runs":[{"text":"Rank your checks by how much they can lie to you."}]},{"t":"p","runs":[{"text":"Deterministic assertions first.","b":true},{"text":" They are free, instant and never drift."}]},{"t":"code","lang":"python","label":"Python","text":"def checks(case, out):\n    yield \"parses\", is_valid_json(out)\n    yield \"cites_real_doc\", out.get(\"doc_id\") in case[\"allowed_docs\"]\n    yield \"length\", len(out[\"reply\"].split()) <= 80\n    yield \"no_phone\", not re.search(r\"\\+?\\d[\\d \\-]{7,}\", out[\"reply\"])\n    yield \"quote_grounded\", out[\"source_quote\"] in case[\"input\"]"},{"t":"p","runs":[{"text":"You can check far more than people expect this way: format, grounding, refusal on the cases that should be refused, absence of banned strings, whether the right tool was called."}]},{"t":"p","runs":[{"text":"Component metrics second.","b":true},{"text":" If you built retrieval, measure recall@k on the golden questions separately from answer quality. A single end-to-end number tells you the feature got worse; two numbers tell you which half to fix."}]},{"t":"p","runs":[{"text":"A model judge last","b":true},{"text":", for the fuzzy part."}]},{"t":"code","lang":"python","label":"Python","text":"cases = json.load(open(\"evals/support.json\"))\nfailed = collections.defaultdict(list)\nfor c in cases:\n    out = feature(c[\"input\"])\n    for name, ok in checks(c, out):\n        if not ok:\n            failed[name].append(c[\"id\"])\nfor name, ids in sorted(failed.items()):\n    print(f\"{name:16} {len(ids):3} failures  {ids[:5]}\")"},{"t":"h2","id":"be-honest-about-model-judges","runs":[{"text":"Be honest about model judges"}]},{"t":"p","runs":[{"text":"Using a model to grade another model’s output works, and it has known biases you must design around:"}]},{"t":"ul","items":[[{"text":"Judges prefer long answers.","b":true},{"text":" Almost every naive judge rewards thoroughness, so any change that makes replies longer raises your score without making anything better for users."}],[{"text":"Absolute scores drift.","b":true},{"text":" A 1-to-5 rating means something slightly different each time you touch the judge prompt, so scores are not comparable across weeks."}],[{"text":"Pairwise is steadier.","b":true},{"text":" Show the judge output A and output B for the same input and ask which better satisfies a specific criterion. Swap the order on half the cases, because judges also favour whichever answer came first."}],[{"text":"Calibrate before trusting.","b":true},{"text":" Label thirty cases yourself, run the judge on them, and see how often it agrees. If it agrees 60% of the time, it is not measuring your feature, it is adding noise. Re-check whenever you edit the judge prompt."}]]},{"t":"h2","id":"measure-cost-and-latency-in-the-same-run","runs":[{"text":"Measure cost and latency in the same run"}]},{"t":"p","runs":[{"text":"Record tokens and wall time for every case. A change that improves quality and triples cost is a decision for someone to make, not a win to announce. Print it next to the pass rate so the trade-off is visible at the moment of choosing."}]},{"t":"h2","id":"make-it-a-gate","runs":[{"text":"Make it a gate"}]},{"t":"p","runs":[{"text":"Run the suite before and after every prompt change, and in CI if you can. The bar is not perfection. The bar is: you know the number, you know which cases moved, and you can say why you accepted it."}]}],"check":{"question":[{"text":"A team grades their support bot with a model judge scoring 1 to 5. After a prompt change the average rises from 3.6 to 4.1, so they ship. Within a week users complain the answers have become waffly and hard to skim. What most likely happened?"}],"options":[{"text":[{"text":"The judge model is not capable enough to notice waffle; a stronger judge would have caught it."}],"correct":false,"tempting":[{"text":"Judging feels like a capability problem, so a smarter grader sounds like the obvious upgrade rather than a criteria problem."}]},{"text":[{"text":"Fifty cases is too small a sample; more examples would have revealed the drop."}],"correct":false,"tempting":[{"text":"Small samples are the reflex explanation for any misleading number, and sometimes they genuinely are the cause."}]},{"text":[{"text":"Users are reacting to slower responses rather than worse ones, and the judge measured quality correctly."}],"correct":false,"tempting":[{"text":"Longer answers really are slower to arrive and to read, so the complaint can plausibly be redescribed as a latency issue."}]},{"text":[{"text":"The judge rewarded length and thoroughness, which the prompt change increased, so the score moved without the thing users cared about moving."}],"correct":true,"tempting":null}],"order":[0,1,3,2],"why":null},"prev":{"slug":"feedback-signals-and-what-they-measure","title":"Thumbs up, thumbs down, and what the buttons actually measure","minutes":8},"next":{"slug":"the-first-fifty-cases","title":"The first fifty cases, and where they come from","minutes":8}}}