{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"a-demo-is-not-a-measurement","title":"A demo is not a measurement","url":"https://addaly.com/learn/evaluating-ai/a-demo-is-not-a-measurement","position":1,"of":76,"minutes":7,"answer":"A result is a number, on a fixed set, next to a comparison: anything else is an anecdote.","keyIdea":"A result is a number, on a fixed set, next to a comparison: anything else is an anecdote."},"body":{"format":"markdown","text":"## The bot that worked perfectly on twelve messages\n\nA small team in Lagos built a WhatsApp assistant that pulls delivery addresses out of customer messages. Before launch they pasted in twelve messages and read the outputs. All twelve were right. It looked good. They shipped.\n\nIn the first week the assistant handled about 3,000 messages and got roughly one in five wrong, not slightly wrong, but wrong in the way that sends a rider to the other side of the city. The twelve test messages had been written by the team, and the team writes addresses like \"14 Adeola Odeku Street, Victoria Island\". Customers write \"opposite the blue mosque, after the second speed bump, call when you reach\".\n\nThe model did not fail. The measurement failed. There was no measurement.\n\n## Three things \"it looks good\" cannot tell you\n\n**How often.** Twelve passes feels like proof. It is not. If you see zero failures in twelve tries, a reasonable upper bound on your true failure rate is about three in twelve: one in four. A system failing a quarter of the time will still hand you twelve clean passes fairly often. Small informal samples can rule out \"catastrophic\". They cannot separate \"very good\" from \"roughly okay\", and that is exactly the distinction you need.\n\n**Compared to what.** Better than the previous prompt? Better than a regular expression? Better than the cheaper model that costs an eighth as much? \"Looks good\" contains no comparison, so it cannot support a decision. Every real result is a difference between two things.\n\n**Read by whom.** You are the worst possible reader of your own system's output. You know what the answer should be, so you fill in the gaps as you read. Hand the same output to someone who does not know the intended answer and watch how much of the \"obviously fine\" becomes \"wait, which invoice is this about\".\n\n## What a result actually looks like\n\nA result has four parts: a number, the fixed set it was computed on, something to compare against, and enough version detail to reproduce it.\n\n```\naddress-extract v3\n  set:     address-eval-100  (frozen 2026-08-14)\n  metric:  exact match on area + landmark\n  score:   78%   (v2 on same set: 61%)   n=100\n  model:   pinned id, temperature 0\n```\n\nThat fits on four lines and it settles arguments. \"I tried it and it seems better\" starts them.\n\n## Why teams skip this\n\nNot laziness. Three real reasons.\n\nIt feels slow. Building the set is an afternoon of unglamorous labelling while your colleague is shipping features. It feels premature. \"We're still exploring, the prompt changes daily.\" That is precisely when you need it: you are making a change a day with no way to tell forward from backward. And it feels like admitting you might be wrong, because a number can disappoint you and a demo never will.\n\nThe afternoon pays for itself the first time two people disagree about whether a prompt change helped. Without a set, that argument is decided by seniority. With a set, it is decided in ten minutes.\n\n## The caveat\n\nEvaluation does not make you right. It makes you *checkable*. A well-built eval set with a badly chosen metric will give you a confident number about something you do not care about, and the rest of this course is largely about that failure mode: measuring the wrong thing precisely.\n\nStart anyway. A rough number on real examples beats a strong opinion on imagined ones, and you can improve a metric once you have one. You cannot improve a vibe."},"check":{"question":"A team edits their support-reply prompt. They run the old version and the new version on the same 30 examples they have been using since January. Both versions produce a correct reply on all 30. What have they learned?","options":[{"text":"Very little: a set where both versions pass everything contains no example capable of telling them apart.","correct":true},{"text":"The new prompt is at least as good as the old one, since it did not lose on any example.","correct":false,"tempting":"Reads a tie on examples neither version fails as evidence that the two versions are equivalent."},{"text":"The change is not worth shipping, since it produced no improvement on the test set.","correct":false,"tempting":"Treats absence of a measured gain as evidence the change is useless, when nothing was capable of measuring a gain."},{"text":"The problem is the 30 examples were hand-picked; 30 randomly sampled examples would have given a valid answer.","correct":false,"tempting":"Assumes random sampling is the missing ingredient, when the deeper issue is that the set has no failures left to move."}]},"prev":null,"next":{"slug":"the-question-before-the-metric","title":"The decision the number has to serve","url":"https://addaly.com/learn/evaluating-ai/the-question-before-the-metric"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A demo is not a measurement', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/a-demo-is-not-a-measurement (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A demo is not a measurement","courseTitle":"Knowing If It Works","module":{"index":1,"slug":"evidence","title":"What counts as evidence"},"answer":null,"keyIdea":"A result is a number, on a fixed set, next to a comparison: anything else is an anecdote.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-bot-that-worked-perfectly-on-twelve-messages","runs":[{"text":"The bot that worked perfectly on twelve messages"}]},{"t":"p","runs":[{"text":"A small team in Lagos built a WhatsApp assistant that pulls delivery addresses out of customer messages. Before launch they pasted in twelve messages and read the outputs. All twelve were right. It looked good. They shipped."}]},{"t":"p","runs":[{"text":"In the first week the assistant handled about 3,000 messages and got roughly one in five wrong, not slightly wrong, but wrong in the way that sends a rider to the other side of the city. The twelve test messages had been written by the team, and the team writes addresses like “14 Adeola Odeku Street, Victoria Island”. Customers write “opposite the blue mosque, after the second speed bump, call when you reach”."}]},{"t":"p","runs":[{"text":"The model did not fail. The measurement failed. There was no measurement."}]},{"t":"h2","id":"three-things-it-looks-good-cannot-tell-you","runs":[{"text":"Three things “it looks good” cannot tell you"}]},{"t":"p","runs":[{"text":"How often.","b":true},{"text":" Twelve passes feels like proof. It is not. If you see zero failures in twelve tries, a reasonable upper bound on your true failure rate is about three in twelve: one in four. A system failing a quarter of the time will still hand you twelve clean passes fairly often. Small informal samples can rule out “catastrophic”. They cannot separate “very good” from “roughly okay”, and that is exactly the distinction you need."}]},{"t":"p","runs":[{"text":"Compared to what.","b":true},{"text":" Better than the previous prompt? Better than a regular expression? Better than the cheaper model that costs an eighth as much? “Looks good” contains no comparison, so it cannot support a decision. Every real result is a difference between two things."}]},{"t":"p","runs":[{"text":"Read by whom.","b":true},{"text":" You are the worst possible reader of your own system’s output. You know what the answer should be, so you fill in the gaps as you read. Hand the same output to someone who does not know the intended answer and watch how much of the “obviously fine” becomes “wait, which invoice is this about”."}]},{"t":"h2","id":"what-a-result-actually-looks-like","runs":[{"text":"What a result actually looks like"}]},{"t":"p","runs":[{"text":"A result has four parts: a number, the fixed set it was computed on, something to compare against, and enough version detail to reproduce it."}]},{"t":"code","lang":"","label":"Text","text":"address-extract v3\n  set:     address-eval-100  (frozen 2026-08-14)\n  metric:  exact match on area + landmark\n  score:   78%   (v2 on same set: 61%)   n=100\n  model:   pinned id, temperature 0"},{"t":"p","runs":[{"text":"That fits on four lines and it settles arguments. “I tried it and it seems better” starts them."}]},{"t":"h2","id":"why-teams-skip-this","runs":[{"text":"Why teams skip this"}]},{"t":"p","runs":[{"text":"Not laziness. Three real reasons."}]},{"t":"p","runs":[{"text":"It feels slow. Building the set is an afternoon of unglamorous labelling while your colleague is shipping features. It feels premature. “We’re still exploring, the prompt changes daily.” That is precisely when you need it: you are making a change a day with no way to tell forward from backward. And it feels like admitting you might be wrong, because a number can disappoint you and a demo never will."}]},{"t":"p","runs":[{"text":"The afternoon pays for itself the first time two people disagree about whether a prompt change helped. Without a set, that argument is decided by seniority. With a set, it is decided in ten minutes."}]},{"t":"h2","id":"the-caveat","runs":[{"text":"The caveat"}]},{"t":"p","runs":[{"text":"Evaluation does not make you right. It makes you "},{"text":"checkable","i":true},{"text":". A well-built eval set with a badly chosen metric will give you a confident number about something you do not care about, and the rest of this course is largely about that failure mode: measuring the wrong thing precisely."}]},{"t":"p","runs":[{"text":"Start anyway. A rough number on real examples beats a strong opinion on imagined ones, and you can improve a metric once you have one. You cannot improve a vibe."}]}],"check":{"question":[{"text":"A team edits their support-reply prompt. They run the old version and the new version on the same 30 examples they have been using since January. Both versions produce a correct reply on all 30. What have they learned?"}],"options":[{"text":[{"text":"Very little: a set where both versions pass everything contains no example capable of telling them apart."}],"correct":true,"tempting":null},{"text":[{"text":"The new prompt is at least as good as the old one, since it did not lose on any example."}],"correct":false,"tempting":[{"text":"Reads a tie on examples neither version fails as evidence that the two versions are equivalent."}]},{"text":[{"text":"The change is not worth shipping, since it produced no improvement on the test set."}],"correct":false,"tempting":[{"text":"Treats absence of a measured gain as evidence the change is useless, when nothing was capable of measuring a gain."}]},{"text":[{"text":"The problem is the 30 examples were hand-picked; 30 randomly sampled examples would have given a valid answer."}],"correct":false,"tempting":[{"text":"Assumes random sampling is the missing ingredient, when the deeper issue is that the set has no failures left to move."}]}],"order":[1,3,0,2],"why":null},"prev":null,"next":{"slug":"the-question-before-the-metric","title":"The decision the number has to serve","minutes":8}}}