{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"the-hundred-examples","title":"A hundred examples, made by hand","url":"https://addaly.com/learn/evaluating-ai/the-hundred-examples","position":3,"of":76,"minutes":8,"answer":"A hundred real labelled examples resolves fifteen-point differences and settles arguments; it cannot resolve three-point ones.","keyIdea":"A hundred real labelled examples resolves fifteen-point differences and settles arguments; it cannot resolve three-point ones."},"body":{"format":"markdown","text":"## Where the examples come from\n\nReal traffic, if you have it. Pull a week of production inputs, sample 100, and label them. That is the whole method and it beats every clever alternative.\n\nIf you have no traffic yet, you still have sources that are not your imagination: support tickets, the search box on your existing site, WhatsApp messages your sales person forwards you, the spreadsheet the ops team maintains by hand today. A used-car marketplace in Bogotá built its first eval set from 100 rows of the WhatsApp export its two agents had been answering manually for a year. Those 100 rows contained things nobody would have invented: customers sending voice-note transcriptions, prices written as \"18 palos\", three different spellings of the same neighbourhood.\n\n## Stratify, then label\n\nDo not sample 100 items and hope the hard cases show up. Decide the slices you care about first, then fill each one:\n\n- the common case, in proportion (usually 50 to 60 items)\n- the known-hard cases: other languages, very long inputs, missing information, ambiguous requests\n- the ones that already broke in production, every single one\n- a few that *should* be refused: abuse, off-topic, requests you do not serve\n\nTag every item with its slice. Overall pass rate is the headline; pass rate per slice is where you actually learn something.\n\nKeep it in the repository as JSONL, next to the code, in version control:\n\n```jsonl\n{\"id\":\"addr-014\",\n \"input\":\"opposite blue mosque, after 2nd speed bump, ikeja\",\n \"expected\":{\"area\":\"Ikeja\",\"landmark\":\"blue mosque\",\"street\":null},\n \"tags\":[\"landmark-only\",\"no-street-name\"]}\n```\n\n## Why a hundred\n\nBecause of what it can and cannot see. A 100-item set measures a pass rate to roughly plus or minus ten points. It will reliably show you a 15-point change. It will not show you a 3-point change, and you should stop pretending otherwise: teams routinely announce improvements far smaller than their set can resolve.\n\n> Figure: What a set of a given size can resolve. Line chart of 95% interval half-width, points against Items in the set: Pass rate near 50% (the widest case) (25, 20) (50, 14) (100, 10) (200, 7) (400, 5) (1000, 3.2). A hundred items gives about plus or minus ten points: enough to see a fifteen-point change, not a three-point one. Halving the interval costs four times the items.\n\nSo: 100 is enough to make decisions about prompt rewrites, model swaps, and \"is this feature ready\". It is not enough to tune. If you need to detect small differences, you need thousands of items, which means automated labelling, which means the next lesson's problems. Most teams never need that.\n\nA hundred items also takes about three hours to label properly. Three hours is a price you can pay this week.\n\n## Two people, twenty items\n\nBefore you label the rest, have a colleague independently label the same 20 items. Then compare.\n\nIf you disagree on 6 of 20, your *criterion* is broken, not the model. Two competent people cannot agree on what \"a helpful reply\" means, so no model can be scored against it and no judge can automate it. Rewrite the criterion until two humans agree at least 90% of the time, then label the rest. This single step catches more bad evaluation than anything else in this course.\n\n## Freeze it, then grow it\n\nOnce labelled, the set is frozen. You do not edit it to make a number look better. When production produces a failure your set did not predict, that failure becomes a new item: this is how the set earns its keep over a year.\n\nOne legitimate reason to touch existing items: your own labels were wrong. That happens, often to 5 or 10 items in a first set. Fix them, write down what you changed, and re-run every previous version against the corrected set so your history stays comparable. A corrected label with a note is honest. A quietly deleted hard item is how a set slowly becomes a set of things you already pass."},"check":{"question":"Reviewing a first eval run, you find eight items where the model's answer was right and your own \"expected\" answer was wrong. What is the sound move?","options":[{"text":"Correct the eight labels, note the change, and re-run the earlier prompt versions so the old and new numbers still compare.","correct":true},{"text":"Leave them as they are: changing the set after seeing the results is exactly how people fool themselves.","correct":false,"tempting":"Applies the rule against tuning a set to your results so literally that known-wrong labels are preserved as truth."},{"text":"Remove those eight items, since ambiguous cases only add noise to the score.","correct":false,"tempting":"Mistakes hard-but-real cases for noise, and makes the set quietly easier every time it happens."},{"text":"Keep the labels and mark those eight as passes for this run, since the model was in fact right.","correct":false,"tempting":"Patches the score instead of the data, so the next person to run the set inherits the same eight wrong labels."}]},"prev":{"slug":"the-question-before-the-metric","title":"The decision the number has to serve","url":"https://addaly.com/learn/evaluating-ai/the-question-before-the-metric"},"next":{"slug":"where-examples-come-from","title":"Where the examples come from","url":"https://addaly.com/learn/evaluating-ai/where-examples-come-from"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A hundred examples, made by hand', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/the-hundred-examples (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A hundred examples, made by hand","courseTitle":"Knowing If It Works","module":{"index":1,"slug":"evidence","title":"What counts as evidence"},"answer":null,"keyIdea":"A hundred real labelled examples resolves fifteen-point differences and settles arguments; it cannot resolve three-point ones.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"What a set of a given size can resolve","series":[{"label":"Pass rate near 50% (the widest case)","points":[[25,20],[50,14],[100,10],[200,7],[400,5],[1000,3.2]]}],"xLabel":"Items in the set","yLabel":"95% interval half-width, points","caption":"A hundred items gives about plus or minus ten points: enough to see a fifteen-point change, not a three-point one. Halving the interval costs four times the items."},"alt":"Figure: What a set of a given size can resolve. Line chart of 95% interval half-width, points against Items in the set: Pass rate near 50% (the widest case) (25, 20) (50, 14) (100, 10) (200, 7) (400, 5) (1000, 3.2). A hundred items gives about plus or minus ten points: enough to see a fifteen-point change, not a three-point one. Halving the interval costs four times the items."}},"blocks":[{"t":"h2","id":"where-the-examples-come-from","runs":[{"text":"Where the examples come from"}]},{"t":"p","runs":[{"text":"Real traffic, if you have it. Pull a week of production inputs, sample 100, and label them. That is the whole method and it beats every clever alternative."}]},{"t":"p","runs":[{"text":"If you have no traffic yet, you still have sources that are not your imagination: support tickets, the search box on your existing site, WhatsApp messages your sales person forwards you, the spreadsheet the ops team maintains by hand today. A used-car marketplace in Bogotá built its first eval set from 100 rows of the WhatsApp export its two agents had been answering manually for a year. Those 100 rows contained things nobody would have invented: customers sending voice-note transcriptions, prices written as “18 palos”, three different spellings of the same neighbourhood."}]},{"t":"h2","id":"stratify-then-label","runs":[{"text":"Stratify, then label"}]},{"t":"p","runs":[{"text":"Do not sample 100 items and hope the hard cases show up. Decide the slices you care about first, then fill each one:"}]},{"t":"ul","items":[[{"text":"the common case, in proportion (usually 50 to 60 items)"}],[{"text":"the known-hard cases: other languages, very long inputs, missing information, ambiguous requests"}],[{"text":"the ones that already broke in production, every single one"}],[{"text":"a few that "},{"text":"should","i":true},{"text":" be refused: abuse, off-topic, requests you do not serve"}]]},{"t":"p","runs":[{"text":"Tag every item with its slice. Overall pass rate is the headline; pass rate per slice is where you actually learn something."}]},{"t":"p","runs":[{"text":"Keep it in the repository as JSONL, next to the code, in version control:"}]},{"t":"code","lang":"jsonl","label":"JSON Lines","text":"{\"id\":\"addr-014\",\n \"input\":\"opposite blue mosque, after 2nd speed bump, ikeja\",\n \"expected\":{\"area\":\"Ikeja\",\"landmark\":\"blue mosque\",\"street\":null},\n \"tags\":[\"landmark-only\",\"no-street-name\"]}"},{"t":"h2","id":"why-a-hundred","runs":[{"text":"Why a hundred"}]},{"t":"p","runs":[{"text":"Because of what it can and cannot see. A 100-item set measures a pass rate to roughly plus or minus ten points. It will reliably show you a 15-point change. It will not show you a 3-point change, and you should stop pretending otherwise: teams routinely announce improvements far smaller than their set can resolve."}]},{"t":"p","runs":[{"text":"So: 100 is enough to make decisions about prompt rewrites, model swaps, and “is this feature ready”. It is not enough to tune. If you need to detect small differences, you need thousands of items, which means automated labelling, which means the next lesson’s problems. Most teams never need that."}]},{"t":"p","runs":[{"text":"A hundred items also takes about three hours to label properly. Three hours is a price you can pay this week."}]},{"t":"h2","id":"two-people-twenty-items","runs":[{"text":"Two people, twenty items"}]},{"t":"p","runs":[{"text":"Before you label the rest, have a colleague independently label the same 20 items. Then compare."}]},{"t":"p","runs":[{"text":"If you disagree on 6 of 20, your "},{"text":"criterion","i":true},{"text":" is broken, not the model. Two competent people cannot agree on what “a helpful reply” means, so no model can be scored against it and no judge can automate it. Rewrite the criterion until two humans agree at least 90% of the time, then label the rest. This single step catches more bad evaluation than anything else in this course."}]},{"t":"h2","id":"freeze-it-then-grow-it","runs":[{"text":"Freeze it, then grow it"}]},{"t":"p","runs":[{"text":"Once labelled, the set is frozen. You do not edit it to make a number look better. When production produces a failure your set did not predict, that failure becomes a new item: this is how the set earns its keep over a year."}]},{"t":"p","runs":[{"text":"One legitimate reason to touch existing items: your own labels were wrong. That happens, often to 5 or 10 items in a first set. Fix them, write down what you changed, and re-run every previous version against the corrected set so your history stays comparable. A corrected label with a note is honest. A quietly deleted hard item is how a set slowly becomes a set of things you already pass."}]}],"check":{"question":[{"text":"Reviewing a first eval run, you find eight items where the model’s answer was right and your own “expected” answer was wrong. What is the sound move?"}],"options":[{"text":[{"text":"Correct the eight labels, note the change, and re-run the earlier prompt versions so the old and new numbers still compare."}],"correct":true,"tempting":null},{"text":[{"text":"Leave them as they are: changing the set after seeing the results is exactly how people fool themselves."}],"correct":false,"tempting":[{"text":"Applies the rule against tuning a set to your results so literally that known-wrong labels are preserved as truth."}]},{"text":[{"text":"Remove those eight items, since ambiguous cases only add noise to the score."}],"correct":false,"tempting":[{"text":"Mistakes hard-but-real cases for noise, and makes the set quietly easier every time it happens."}]},{"text":[{"text":"Keep the labels and mark those eight as passes for this run, since the model was in fact right."}],"correct":false,"tempting":[{"text":"Patches the score instead of the data, so the next person to run the set inherits the same eight wrong labels."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"the-question-before-the-metric","title":"The decision the number has to serve","minutes":8},"next":{"slug":"where-examples-come-from","title":"Where the examples come from","minutes":9}}}