{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"thirty-examples-of-your-own","title":"The evaluation that decides it","url":"https://addaly.com/learn/hugging-face/thirty-examples-of-your-own","position":69,"of":75,"minutes":9,"answer":"A leaderboard narrows the field and thirty examples of your own decide it, because only your set contains your languages, formats and hard cases.","keyIdea":"A leaderboard narrows the field and thirty examples of your own decide it, because only your set contains your languages, formats and hard cases, and at that size it reliably detects large differences and must never be used to justify small ones."},"body":{"format":"markdown","text":"## Why thirty, and why yours\n\nA leaderboard narrows the field to three candidates. Nothing public can pick the winner, because no public benchmark contains your text, your languages, your formats or your users' phrasing. The thing that decides it is a small set of your own examples, and it takes an afternoon to build.\n\nThirty is chosen deliberately. It is few enough to read every failure, which is where the information is, and enough to detect the differences worth acting on: a model that cannot handle your language, a configuration that broke structured output, a prompt change that lost a capability. It will **not** detect a two-point difference, and treating a 26-out-of-30 against a 27-out-of-30 as a result is how people convince themselves of noise.\n\n## Building it\n\n```json\n{\"id\": \"t001\", \"input\": \"मेरा ऑर्डर कैंसिल करना है\", \"expect_label\": \"cancellation\", \"note\": \"hindi\"}\n{\"id\": \"t002\", \"input\": \"where's my stuff??\", \"expect_label\": \"delivery\", \"note\": \"informal\"}\n{\"id\": \"t003\", \"input\": \"invoice INV-2024-8823 wrong amount\", \"expect_label\": \"billing\", \"note\": \"has id\"}\n```\n\nFour rules for choosing the thirty:\n\n1. **Take them from real inputs**, not invented ones. Invented examples are always cleaner than reality.\n2. **Cover the shape of your traffic**, including the rare classes you care about. This is stratified sampling, and it means you cannot quote a single headline rate as if it were production performance.\n3. **Include the hard cases deliberately**: the other language, the misspelling, the empty input, the extremely long input, the one with an identifier in it, the ambiguous one.\n4. **Include three where the right answer is \"I do not know\"**, so you can see whether the model invents something.\n\nRedact personal data before this file exists anywhere. It is a copy of real user text, and it will end up in a repository.\n\n## Scoring what can be scored\n\n```python\nimport evaluate, json\n\nrows = [json.loads(l) for l in open(\"eval.jsonl\")]\nacc = evaluate.load(\"accuracy\")\n\ndef run(model_fn):\n    preds, refs, fails = [], [], []\n    for r in rows:\n        got = model_fn(r[\"input\"])\n        preds.append(got); refs.append(r[\"expect_label\"])\n        if got != r[\"expect_label\"]:\n            fails.append((r[\"id\"], r[\"note\"], r[\"input\"][:60], got))\n    return acc.compute(predictions=preds, references=refs), fails\n```\n\nMore of your task is exactly scorable than you expect: is the JSON valid, is the label right, is the number correct, is the date in the right format, is the answer in the right language, is it under the length limit. Score all of those automatically and cheaply.\n\nFor open-ended output, read it. Thirty is a deliberate size because thirty is readable. A model scoring another model is a real technique with real biases (position effects, a preference for its own style, a preference for longer answers) and at thirty items it buys you nothing you cannot get by reading.\n\n## Comparing two models honestly\n\nRun both on the identical set with identical prompts and identical generation settings. Differences of ten points or more at thirty items are worth acting on. Smaller ones are not.\n\nTwo confounders that ruin more comparisons than anything else:\n\n- **Different chat templates or prompts.** You are then comparing prompts, not models.\n- **Inherited generation defaults.** Module two's point: two models can appear to differ in quality when their `generation_config.json` files differ in temperature.\n\nAnd always include a baseline that is not a large model: a keyword rule, a small fine-tuned classifier, the current system. A great many projects discover that a 40-line rule scores within a few points of the model, which is a finding worth having in week one rather than month three.\n\n## The tools, when you want more than a script\n\n`evaluate` loads standard metric implementations so everybody's F1 is the same F1. `lighteval` runs standard benchmark suites against a model, locally or through a provider, if you want to reproduce a leaderboard number yourself rather than trust it, which is occasionally illuminating, because reproducing a published score is harder than it sounds and the gap is informative.\n\nNeither replaces the thirty. They are for the questions that are about the field; the thirty is for the question that is about you.\n\n## Keep it, and grow it\n\nStore the file in your repository, versioned. Add a row every time production surprises you: an input that produced a wrong answer becomes a permanent test. Over a year the set becomes a record of everything you have learned about your own problem, and it is the single most valuable artefact a small team builds.\n\nWhen you change model, prompt, quantization or library version, run it. That is the whole discipline.\n\n## Do this now\n\nOpen a file and write ten rows from real inputs in the next fifteen minutes. Not thirty: ten. Run your current system against them and read every failure. You will find something in the first ten, and that is the point."},"check":{"question":"Two candidate models score 26/30 and 27/30 on a team's own evaluation set. What is the correct conclusion?","options":[{"text":"The one-point difference is within noise at this sample size, so the choice should rest on other grounds.","correct":true},{"text":"The second model is better and should be adopted, since the comparison used identical inputs and settings.","correct":false,"tempting":"Controlling the conditions removes confounders without making a one-item difference statistically meaningful."},{"text":"The set is too easy, since both models pass most items, and harder examples should replace it entirely.","correct":false,"tempting":"High scores may reflect a well-matched task rather than an inadequate set, and replacing it loses comparability."},{"text":"Neither result is usable because thirty items cannot support any conclusion about model quality.","correct":false,"tempting":"Overcorrects: thirty items detect large differences reliably, which is what the set is for."}]},"prev":{"slug":"leaderboards-lie-a-little","title":"The leaderboard is not measuring your task","url":"https://addaly.com/learn/hugging-face/leaderboards-lie-a-little"},"next":{"slug":"inference-providers","title":"Renting the compute instead of owning it","url":"https://addaly.com/learn/hugging-face/inference-providers"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The evaluation that decides it', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/thirty-examples-of-your-own (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The evaluation that decides it","courseTitle":"Hugging Face, End to End","module":{"index":8,"slug":"depending-on-it-and-publishing","title":"Depending on it, and publishing so others can"},"answer":null,"keyIdea":"A leaderboard narrows the field and thirty examples of your own decide it, because only your set contains your languages, formats and hard cases, and at that size it reliably detects large differences and must never be used to justify small ones.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-thirty-and-why-yours","runs":[{"text":"Why thirty, and why yours"}]},{"t":"p","runs":[{"text":"A leaderboard narrows the field to three candidates. Nothing public can pick the winner, because no public benchmark contains your text, your languages, your formats or your users’ phrasing. The thing that decides it is a small set of your own examples, and it takes an afternoon to build."}]},{"t":"p","runs":[{"text":"Thirty is chosen deliberately. It is few enough to read every failure, which is where the information is, and enough to detect the differences worth acting on: a model that cannot handle your language, a configuration that broke structured output, a prompt change that lost a capability. It will "},{"text":"not","b":true},{"text":" detect a two-point difference, and treating a 26-out-of-30 against a 27-out-of-30 as a result is how people convince themselves of noise."}]},{"t":"h2","id":"building-it","runs":[{"text":"Building it"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"id\": \"t001\", \"input\": \"मेरा ऑर्डर कैंसिल करना है\", \"expect_label\": \"cancellation\", \"note\": \"hindi\"}\n{\"id\": \"t002\", \"input\": \"where's my stuff??\", \"expect_label\": \"delivery\", \"note\": \"informal\"}\n{\"id\": \"t003\", \"input\": \"invoice INV-2024-8823 wrong amount\", \"expect_label\": \"billing\", \"note\": \"has id\"}"},{"t":"p","runs":[{"text":"Four rules for choosing the thirty:"}]},{"t":"ol","start":1,"items":[[{"text":"Take them from real inputs","b":true},{"text":", not invented ones. Invented examples are always cleaner than reality."}],[{"text":"Cover the shape of your traffic","b":true},{"text":", including the rare classes you care about. This is stratified sampling, and it means you cannot quote a single headline rate as if it were production performance."}],[{"text":"Include the hard cases deliberately","b":true},{"text":": the other language, the misspelling, the empty input, the extremely long input, the one with an identifier in it, the ambiguous one."}],[{"text":"Include three where the right answer is “I do not know”","b":true},{"text":", so you can see whether the model invents something."}]]},{"t":"p","runs":[{"text":"Redact personal data before this file exists anywhere. It is a copy of real user text, and it will end up in a repository."}]},{"t":"h2","id":"scoring-what-can-be-scored","runs":[{"text":"Scoring what can be scored"}]},{"t":"code","lang":"python","label":"Python","text":"import evaluate, json\n\nrows = [json.loads(l) for l in open(\"eval.jsonl\")]\nacc = evaluate.load(\"accuracy\")\n\ndef run(model_fn):\n    preds, refs, fails = [], [], []\n    for r in rows:\n        got = model_fn(r[\"input\"])\n        preds.append(got); refs.append(r[\"expect_label\"])\n        if got != r[\"expect_label\"]:\n            fails.append((r[\"id\"], r[\"note\"], r[\"input\"][:60], got))\n    return acc.compute(predictions=preds, references=refs), fails"},{"t":"p","runs":[{"text":"More of your task is exactly scorable than you expect: is the JSON valid, is the label right, is the number correct, is the date in the right format, is the answer in the right language, is it under the length limit. Score all of those automatically and cheaply."}]},{"t":"p","runs":[{"text":"For open-ended output, read it. Thirty is a deliberate size because thirty is readable. A model scoring another model is a real technique with real biases (position effects, a preference for its own style, a preference for longer answers) and at thirty items it buys you nothing you cannot get by reading."}]},{"t":"h2","id":"comparing-two-models-honestly","runs":[{"text":"Comparing two models honestly"}]},{"t":"p","runs":[{"text":"Run both on the identical set with identical prompts and identical generation settings. Differences of ten points or more at thirty items are worth acting on. Smaller ones are not."}]},{"t":"p","runs":[{"text":"Two confounders that ruin more comparisons than anything else:"}]},{"t":"ul","items":[[{"text":"Different chat templates or prompts.","b":true},{"text":" You are then comparing prompts, not models."}],[{"text":"Inherited generation defaults.","b":true},{"text":" Module two’s point: two models can appear to differ in quality when their "},{"text":"generation_config.json","code":true},{"text":" files differ in temperature."}]]},{"t":"p","runs":[{"text":"And always include a baseline that is not a large model: a keyword rule, a small fine-tuned classifier, the current system. A great many projects discover that a 40-line rule scores within a few points of the model, which is a finding worth having in week one rather than month three."}]},{"t":"h2","id":"the-tools-when-you-want-more-than-a-script","runs":[{"text":"The tools, when you want more than a script"}]},{"t":"p","runs":[{"text":"evaluate","code":true},{"text":" loads standard metric implementations so everybody’s F1 is the same F1. "},{"text":"lighteval","code":true},{"text":" runs standard benchmark suites against a model, locally or through a provider, if you want to reproduce a leaderboard number yourself rather than trust it, which is occasionally illuminating, because reproducing a published score is harder than it sounds and the gap is informative."}]},{"t":"p","runs":[{"text":"Neither replaces the thirty. They are for the questions that are about the field; the thirty is for the question that is about you."}]},{"t":"h2","id":"keep-it-and-grow-it","runs":[{"text":"Keep it, and grow it"}]},{"t":"p","runs":[{"text":"Store the file in your repository, versioned. Add a row every time production surprises you: an input that produced a wrong answer becomes a permanent test. Over a year the set becomes a record of everything you have learned about your own problem, and it is the single most valuable artefact a small team builds."}]},{"t":"p","runs":[{"text":"When you change model, prompt, quantization or library version, run it. That is the whole discipline."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Open a file and write ten rows from real inputs in the next fifteen minutes. Not thirty: ten. Run your current system against them and read every failure. You will find something in the first ten, and that is the point."}]}],"check":{"question":[{"text":"Two candidate models score 26/30 and 27/30 on a team’s own evaluation set. What is the correct conclusion?"}],"options":[{"text":[{"text":"The one-point difference is within noise at this sample size, so the choice should rest on other grounds."}],"correct":true,"tempting":null},{"text":[{"text":"The second model is better and should be adopted, since the comparison used identical inputs and settings."}],"correct":false,"tempting":[{"text":"Controlling the conditions removes confounders without making a one-item difference statistically meaningful."}]},{"text":[{"text":"The set is too easy, since both models pass most items, and harder examples should replace it entirely."}],"correct":false,"tempting":[{"text":"High scores may reflect a well-matched task rather than an inadequate set, and replacing it loses comparability."}]},{"text":[{"text":"Neither result is usable because thirty items cannot support any conclusion about model quality."}],"correct":false,"tempting":[{"text":"Overcorrects: thirty items detect large differences reliably, which is what the set is for."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"leaderboards-lie-a-little","title":"The leaderboard is not measuring your task","minutes":9},"next":{"slug":"inference-providers","title":"Renting the compute instead of owning it","minutes":8}}}