{"course":{"slug":"ai-careers","title":"Working in AI","url":"https://addaly.com/learn/ai-careers","level":"beginner","track":"work"},"lesson":{"slug":"evaluation-the-scarce-skill","title":"Evaluation: deciding what \"correct\" means, in advance","url":"https://addaly.com/learn/ai-careers/evaluation-the-scarce-skill","position":28,"of":85,"minutes":10,"answer":"Anyone can produce an AI output; the scarce and durable skill is defining what counts as correct before you look, and then measuring it.","keyIdea":"Anyone can produce an AI output; the scarce and durable skill is defining what counts as correct before you look, and then measuring it."},"body":{"format":"markdown","text":"## Why this is the skill worth having\n\nWork that is well specified, abundantly exampled and quickly checkable is the work that gets automated first. Deciding what \"good enough\" means is none of those things. It is a judgement someone has to be accountable for, and organisations do not hand accountability to a tool.\n\nThat is the strategic argument. The practical one is simpler: almost no beginner shows evidence of having evaluated their own work, so doing it makes your portfolio unusual in the one way that matters.\n\n## Build the set by hand, and build it small\n\nThirty to fifty cases. Not three thousand. You are going to read every one of them personally, several times, and a set you will not read is a set you will not learn from.\n\nWhere the cases come from:\n\n- **Real questions people actually asked.** Support tickets, search logs, the questions in a team's chat channel. These are worth ten invented ones.\n- **The edge cases the team argues about.** Every team has three examples they disagree on. Those are the most valuable cases you own.\n- **Three cases that should fail.** A question the system has no information to answer, a request it should refuse, and a question in another language or dialect. What a system does when it should say \"I don't know\" is where most of the risk lives.\n\n## The rubric, and why three points beats five\n\nFor each case, write down what a correct answer must contain, before you see any output. Then grade.\n\nUse binary correct/incorrect wherever you can. Where you cannot, use three levels: correct, partly correct, wrong. Resist the five-point scale: graders do not agree on the difference between a 3 and a 4, agreement between them collapses, and once two people scoring the same output produce different numbers, the average stops meaning anything. Three coarse levels that people agree on beat five fine ones they do not.\n\n## What a record looks like\n\n```\ncase_id: 017\nquestion: \"What is the refund window for orders shipped to Nigeria?\"\nmust_contain: [\"30 days\", \"counted from delivery, not order date\"]\nanswer: \"You can request a refund within 14 days of ordering.\"\nverdict: wrong\nnote: retrieved the UK policy page; the Nigeria page exists\n      but is titled 'West Africa returns' and did not match\n```\n\nThe `note` field is where the value is. \"Wrong\" is a number. \"Retrieved the wrong document because the title uses a regional name\" is a fix.\n\n## The two numbers to report, and the one to lead with\n\nReport accuracy on the set (*33 of 40 correct*) and report the **worst failure** by name. A single named failure moves a decision more than any average, because a stakeholder can picture it. \"It confidently gave a customer in Lagos the UK refund window\" is a sentence that gets a bug fixed. \"82 per cent accuracy\" is a sentence people nod at.\n\n## The baseline almost nobody builds\n\nBefore you report that your system scores 82 per cent, score the dumb version on the same forty cases. Keyword search over the documents. Or a fixed reply saying \"contact support\". Or the previous process.\n\nIf plain keyword search scores 74, your honest headline is *eight points above search*, and that changes what the thing is worth. Reporting 82 with no baseline is not lying, but it lets everyone in the room assume the baseline was zero, which it never is.\n\n> Figure: An evaluation, start to finish. Steps: 1. 30 to 50 real cases (tickets, logs, the three the team argues about, three that should fail); 2. Write what correct must contain (before seeing any output); 3. Run, grade, write the note (correct / partly / wrong, and why); 4. Score the dumb baseline (keyword search, a fixed reply, the old process); 5. Report (33 of 40, and the worst failure by name). The rubric is written before any output is seen, and the dumb baseline is scored on the same forty cases. That is what turns 'eighty-two per cent' into 'eight points above keyword search', which is what the thing is actually worth.\n\n## Using a model as a judge, carefully\n\nGrading by hand does not scale past a few hundred cases, so people have models grade outputs. It works, with two known biases: judges prefer longer answers, and they prefer text that reads like their own output. Both inflate scores in ways unrelated to correctness.\n\nThe discipline: hand-label about fifty cases, run the judge on the same fifty, and measure how often they agree. If agreement is poor, fix the rubric before you trust any large run. Re-check whenever the judge's model version changes, because a provider updating a model silently changes your measuring instrument.\n\n## The free path\n\nYou do not need an evaluation platform. A CSV of cases, a script that runs each one, and a column for the verdict is a complete evaluation harness, and it is what most teams actually use for the first year.\n\nThe named paid products (LangSmith, Braintrust, Weights & Biases Weave) buy you tracing, dashboards and collaboration when many people grade together. The free equivalents are **promptfoo** (open source, config-file driven), the open evaluation harnesses on Hugging Face, or sixty lines of Python. Start with the sixty lines; you will understand what the product is doing when you eventually need one.\n\n## The limitation that catches everyone\n\nAn evaluation set built once goes stale. You fix the cases in it, the score climbs to 95 per cent, and the system is no better in production, because you have optimised against forty fixed examples rather than against reality. This is overfitting to your test set, and it happens quietly.\n\nThe fix is a refresh habit: every month, sample fresh cases from real traffic and add them, retiring some old ones. Keep a small held-out set you never look at while developing. And treat any score above about 95 per cent on a hand-built set as a sign that the set has become too easy, not that the work is done."},"check":{"question":"Your assistant scores 82 per cent on a 40-case set you wrote. Your manager asks whether it is ready to put in front of customers. Which response is strongest?","options":[{"text":"\"Yes: 82 per cent is above the 80 per cent threshold we agreed as the bar for launch.\"","correct":false,"tempting":"Treats an agreed number as sufficient without asking what the errors were, which is how a system ships with a small number of severe failures."},{"text":"\"Not yet: I need a much larger test set before the figure is statistically reliable.\"","correct":false,"tempting":"Reaches for sample size, which matters, while ignoring that the existing failures are already readable and already decisive."},{"text":"\"Yes, with a disclaimer on the interface telling customers the answers may be inaccurate.\"","correct":false,"tempting":"Moves the risk onto the customer rather than examining it, and a disclaimer does not reduce the harm of a confidently wrong answer."},{"text":"\"Plain keyword search scores 74 on the same cases, so we are eight points better than search. Of the seven failures, five are one bug: it retrieves the UK policy for African customers. I would fix that, re-run, and add twenty fresh cases sampled from real questions before deciding.\"","correct":true}]},"prev":{"slug":"the-spreadsheet-nobody-lists","title":"The spreadsheet skill nobody puts on a syllabus","url":"https://addaly.com/learn/ai-careers/the-spreadsheet-nobody-lists"},"next":{"slug":"portfolio-from-zero","title":"A portfolio when you have never been paid for this","url":"https://addaly.com/learn/ai-careers/portfolio-from-zero"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Evaluation: deciding what \"correct\" means, in advance', in Working in AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-careers/evaluation-the-scarce-skill (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Evaluation: deciding what “correct” means, in advance","courseTitle":"Working in AI","module":{"index":3,"slug":"the-evidence","title":"The evidence: what gets tested, and how to prove you have it"},"answer":null,"keyIdea":"Anyone can produce an AI output; the scarce and durable skill is defining what counts as correct before you look, and then measuring it.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"30 to 50 real cases","detail":"tickets, logs, the three the team argues about, three that should fail"},{"label":"Write what correct must contain","detail":"before seeing any output"},{"label":"Run, grade, write the note","detail":"correct / partly / wrong, and why"},{"label":"Score the dumb baseline","detail":"keyword search, a fixed reply, the old process"},{"label":"Report","detail":"33 of 40, and the worst failure by name"}],"title":"An evaluation, start to finish","caption":"The rubric is written before any output is seen, and the dumb baseline is scored on the same forty cases. That is what turns ‘eighty-two per cent’ into ‘eight points above keyword search’, which is what the thing is actually worth."},"alt":"Figure: An evaluation, start to finish. Steps: 1. 30 to 50 real cases (tickets, logs, the three the team argues about, three that should fail); 2. Write what correct must contain (before seeing any output); 3. Run, grade, write the note (correct / partly / wrong, and why); 4. Score the dumb baseline (keyword search, a fixed reply, the old process); 5. Report (33 of 40, and the worst failure by name). The rubric is written before any output is seen, and the dumb baseline is scored on the same forty cases. That is what turns ‘eighty-two per cent’ into ‘eight points above keyword search’, which is what the thing is actually worth."}},"blocks":[{"t":"h2","id":"why-this-is-the-skill-worth-having","runs":[{"text":"Why this is the skill worth having"}]},{"t":"p","runs":[{"text":"Work that is well specified, abundantly exampled and quickly checkable is the work that gets automated first. Deciding what “good enough” means is none of those things. It is a judgement someone has to be accountable for, and organisations do not hand accountability to a tool."}]},{"t":"p","runs":[{"text":"That is the strategic argument. The practical one is simpler: almost no beginner shows evidence of having evaluated their own work, so doing it makes your portfolio unusual in the one way that matters."}]},{"t":"h2","id":"build-the-set-by-hand-and-build-it-small","runs":[{"text":"Build the set by hand, and build it small"}]},{"t":"p","runs":[{"text":"Thirty to fifty cases. Not three thousand. You are going to read every one of them personally, several times, and a set you will not read is a set you will not learn from."}]},{"t":"p","runs":[{"text":"Where the cases come from:"}]},{"t":"ul","items":[[{"text":"Real questions people actually asked.","b":true},{"text":" Support tickets, search logs, the questions in a team’s chat channel. These are worth ten invented ones."}],[{"text":"The edge cases the team argues about.","b":true},{"text":" Every team has three examples they disagree on. Those are the most valuable cases you own."}],[{"text":"Three cases that should fail.","b":true},{"text":" A question the system has no information to answer, a request it should refuse, and a question in another language or dialect. What a system does when it should say “I don’t know” is where most of the risk lives."}]]},{"t":"h2","id":"the-rubric-and-why-three-points-beats-five","runs":[{"text":"The rubric, and why three points beats five"}]},{"t":"p","runs":[{"text":"For each case, write down what a correct answer must contain, before you see any output. Then grade."}]},{"t":"p","runs":[{"text":"Use binary correct/incorrect wherever you can. Where you cannot, use three levels: correct, partly correct, wrong. Resist the five-point scale: graders do not agree on the difference between a 3 and a 4, agreement between them collapses, and once two people scoring the same output produce different numbers, the average stops meaning anything. Three coarse levels that people agree on beat five fine ones they do not."}]},{"t":"h2","id":"what-a-record-looks-like","runs":[{"text":"What a record looks like"}]},{"t":"code","lang":"","label":"Text","text":"case_id: 017\nquestion: \"What is the refund window for orders shipped to Nigeria?\"\nmust_contain: [\"30 days\", \"counted from delivery, not order date\"]\nanswer: \"You can request a refund within 14 days of ordering.\"\nverdict: wrong\nnote: retrieved the UK policy page; the Nigeria page exists\n      but is titled 'West Africa returns' and did not match"},{"t":"p","runs":[{"text":"The "},{"text":"note","code":true},{"text":" field is where the value is. “Wrong” is a number. “Retrieved the wrong document because the title uses a regional name” is a fix."}]},{"t":"h2","id":"the-two-numbers-to-report-and-the-one-to-lead-with","runs":[{"text":"The two numbers to report, and the one to lead with"}]},{"t":"p","runs":[{"text":"Report accuracy on the set ("},{"text":"33 of 40 correct","i":true},{"text":") and report the "},{"text":"worst failure","b":true},{"text":" by name. A single named failure moves a decision more than any average, because a stakeholder can picture it. “It confidently gave a customer in Lagos the UK refund window” is a sentence that gets a bug fixed. “82 per cent accuracy” is a sentence people nod at."}]},{"t":"h2","id":"the-baseline-almost-nobody-builds","runs":[{"text":"The baseline almost nobody builds"}]},{"t":"p","runs":[{"text":"Before you report that your system scores 82 per cent, score the dumb version on the same forty cases. Keyword search over the documents. Or a fixed reply saying “contact support”. Or the previous process."}]},{"t":"p","runs":[{"text":"If plain keyword search scores 74, your honest headline is "},{"text":"eight points above search","i":true},{"text":", and that changes what the thing is worth. Reporting 82 with no baseline is not lying, but it lets everyone in the room assume the baseline was zero, which it never is."}]},{"t":"h2","id":"using-a-model-as-a-judge-carefully","runs":[{"text":"Using a model as a judge, carefully"}]},{"t":"p","runs":[{"text":"Grading by hand does not scale past a few hundred cases, so people have models grade outputs. It works, with two known biases: judges prefer longer answers, and they prefer text that reads like their own output. Both inflate scores in ways unrelated to correctness."}]},{"t":"p","runs":[{"text":"The discipline: hand-label about fifty cases, run the judge on the same fifty, and measure how often they agree. If agreement is poor, fix the rubric before you trust any large run. Re-check whenever the judge’s model version changes, because a provider updating a model silently changes your measuring instrument."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"You do not need an evaluation platform. A CSV of cases, a script that runs each one, and a column for the verdict is a complete evaluation harness, and it is what most teams actually use for the first year."}]},{"t":"p","runs":[{"text":"The named paid products (LangSmith, Braintrust, Weights & Biases Weave) buy you tracing, dashboards and collaboration when many people grade together. The free equivalents are "},{"text":"promptfoo","b":true},{"text":" (open source, config-file driven), the open evaluation harnesses on Hugging Face, or sixty lines of Python. Start with the sixty lines; you will understand what the product is doing when you eventually need one."}]},{"t":"h2","id":"the-limitation-that-catches-everyone","runs":[{"text":"The limitation that catches everyone"}],"role":"limit"},{"t":"p","runs":[{"text":"An evaluation set built once goes stale. You fix the cases in it, the score climbs to 95 per cent, and the system is no better in production, because you have optimised against forty fixed examples rather than against reality. This is overfitting to your test set, and it happens quietly."}]},{"t":"p","runs":[{"text":"The fix is a refresh habit: every month, sample fresh cases from real traffic and add them, retiring some old ones. Keep a small held-out set you never look at while developing. And treat any score above about 95 per cent on a hand-built set as a sign that the set has become too easy, not that the work is done."}]}],"check":{"question":[{"text":"Your assistant scores 82 per cent on a 40-case set you wrote. Your manager asks whether it is ready to put in front of customers. Which response is strongest?"}],"options":[{"text":[{"text":"“Yes: 82 per cent is above the 80 per cent threshold we agreed as the bar for launch.”"}],"correct":false,"tempting":[{"text":"Treats an agreed number as sufficient without asking what the errors were, which is how a system ships with a small number of severe failures."}]},{"text":[{"text":"“Not yet: I need a much larger test set before the figure is statistically reliable.”"}],"correct":false,"tempting":[{"text":"Reaches for sample size, which matters, while ignoring that the existing failures are already readable and already decisive."}]},{"text":[{"text":"“Yes, with a disclaimer on the interface telling customers the answers may be inaccurate.”"}],"correct":false,"tempting":[{"text":"Moves the risk onto the customer rather than examining it, and a disclaimer does not reduce the harm of a confidently wrong answer."}]},{"text":[{"text":"“Plain keyword search scores 74 on the same cases, so we are eight points better than search. Of the seven failures, five are one bug: it retrieves the UK policy for African customers. I would fix that, re-run, and add twenty fresh cases sampled from real questions before deciding.”"}],"correct":true,"tempting":null}],"order":[0,2,1,3],"why":null},"prev":{"slug":"the-spreadsheet-nobody-lists","title":"The spreadsheet skill nobody puts on a syllabus","minutes":7},"next":{"slug":"portfolio-from-zero","title":"A portfolio when you have never been paid for this","minutes":7}}}