{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"reporting-a-number-honestly","title":"Writing the result down","url":"https://addaly.com/learn/evaluating-ai/reporting-a-number-honestly","position":75,"of":76,"minutes":8,"answer":"A result travels without you, so write one page carrying the set, the interval, the slices, the cost, what was not measured and the decision it supports.","keyIdea":"A result travels without you, so write one page carrying the set, the interval, the slices, the cost, what was not measured and the decision it supports."},"body":{"format":"markdown","text":"## The number leaves the room without you\n\nYou will say \"about 86%, on a hundred items, plus or minus eight, and the Bengali slice is much worse\". What travels to the next meeting is \"86%\". By the third retelling it is \"nearly 90% accurate\" and it is in a slide to a customer.\n\nThe only defence is a written artefact that goes wherever the number goes. One page. It takes fifteen minutes and it is the difference between a measurement and a rumour.\n\n## The result card\n\n> **Support reply quality: reply-v9**\n> *2026-09-06 · owner: Aditi · dataset support-v4 (sha 9f21c0)*\n>\n> **What was measured.** Whether a generated reply answers the question asked, makes no commitment absent from the policy text, and asks rather than assumes when information is missing. Six binary rules; two are gates.\n>\n> **Result.** 86% of items passed all six rules (95% CI 79 to 91, n = 100, 3 runs, spread 2 points). Previous version, same items: 82% (74 to 89). 9 items improved, 4 regressed.\n>\n> **Gates.** 0 policy-commitment violations in 300 runs. 1 account-fact violation, item tkt-0087, fixed and retained as a regression test.\n>\n> **By slice.** English 91% (n=61) · Hindi 84% (n=25) · Bengali 64% (n=14, interval too wide to act on, needs more items).\n>\n> **Cost and speed.** $0.0068 per reply, up from $0.0031. p95 latency 4.4s, up from 2.9s.\n>\n> **What this does not measure.** Whether customers were satisfied; whether the reply was appropriate in tone; anything about voice or attachments; anything in a language other than the three above.\n>\n> **Decision it supports.** Ship to the English and Hindi queues. Do not ship to Bengali until that slice has 40 items and clears 80%.\n\nSix headings. Every one of them exists because a team somewhere got burned by leaving it out.\n\n## The four sentences that make it honest\n\n**\"On these hundred items.\"** Never \"the model is 86% accurate\". Always \"86% on this set\". The set is the scope of the claim and it should be inseparable from it.\n\n**\"Plus or minus eight.\"** From module one. Any number without an interval invites comparison against another number without an interval, and that is how a team ships a three-point regression believing it is a three-point improvement.\n\n**\"We did not measure X.\"** The most valuable line on the page, and the hardest to write, because it reads as weakness and is the opposite. It stops somebody assuming you covered voice input, or Bengali, or tone.\n\n**\"We cannot tell.\"** For the Bengali slice with 14 items, the interval is roughly ±25 points. The honest report is not 64%. It is that the number could be anywhere from 40% to 88% and the only fix is more items. Reporting \"we cannot tell yet\" once buys more credibility than five confident numbers, because it demonstrates that the confident ones mean something.\n\n## Talking to people who will not read a confidence interval\n\nTranslate, do not simplify away:\n\n- \"About 8 in 10 replies are right, and we could be off by about 1 in 10 either way.\"\n- \"It's better than the old version on this set, by roughly 4 points, and 4 items got worse; we read those.\"\n- \"It works noticeably less well in Bengali, and we don't have enough Bengali examples yet to say how much less.\"\n\nWhat not to do: report a mean judge score. \"The assistant scores 8.4 out of 10\" is meaningless outside your repository, it is not comparable to anything, it drifts when the judge model updates, and it will be quoted at you for a year. Judge scores are internal, relative, same-run comparisons. Keep them there.\n\n## Refusing to give a number\n\nSometimes the right answer to \"what's the accuracy?\" is \"we don't have a measurement I would stand behind, and here is what it would take to get one: two days and a hundred labelled examples\".\n\nThis is the hardest sentence in the course to say and the one that most defines whether the evaluation function in a company is trusted. A number produced under pressure without a set behind it is a guess wearing a decimal point, and it will be treated as fact by everybody who was not in the room.\n\nSay the two days. In almost every case, somebody will find them."},"check":{"question":"An executive asks for a single headline accuracy figure for a slide. Your set is 100 items, the pass rate is 86%, and one language slice of 14 items sits at 64%. What is the best response?","options":[{"text":"Give 86% and mention the language gap verbally, since a slide cannot carry caveats.","correct":false,"tempting":"Attaches the caveat to a conversation while the number goes onto a slide that will be forwarded without it."},{"text":"Give 82% instead, discounting for the weak slice, so the figure is conservative and defensible.","correct":false,"tempting":"Invents a number by informal adjustment, which is less defensible than the measured one because nothing supports the discount."},{"text":"Decline to give any figure until the weak slice has enough items to be measured properly.","correct":false,"tempting":"Treats an incomplete measurement as no measurement, withholding a result that is genuinely sound for the two slices it covers."},{"text":"Give \"86% on 100 support tickets, plus or minus 8\", state that it covers English and Hindi only, and say the third language cannot yet be measured.","correct":true}]},"prev":{"slug":"incident-to-eval-item","title":"Turning a complaint into a permanent test","url":"https://addaly.com/learn/evaluating-ai/incident-to-eval-item"},"next":{"slug":"how-much-evaluation-is-enough","title":"How much of this to actually do","url":"https://addaly.com/learn/evaluating-ai/how-much-evaluation-is-enough"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Writing the result down', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/reporting-a-number-honestly (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Writing the result down","courseTitle":"Knowing If It Works","module":{"index":8,"slug":"living-with-it","title":"Living with it"},"answer":null,"keyIdea":"A result travels without you, so write one page carrying the set, the interval, the slices, the cost, what was not measured and the decision it supports.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-number-leaves-the-room-without-you","runs":[{"text":"The number leaves the room without you"}]},{"t":"p","runs":[{"text":"You will say “about 86%, on a hundred items, plus or minus eight, and the Bengali slice is much worse”. What travels to the next meeting is “86%”. By the third retelling it is “nearly 90% accurate” and it is in a slide to a customer."}]},{"t":"p","runs":[{"text":"The only defence is a written artefact that goes wherever the number goes. One page. It takes fifteen minutes and it is the difference between a measurement and a rumour."}]},{"t":"h2","id":"the-result-card","runs":[{"text":"The result card"}]},{"t":"quote","runs":[{"text":"Support reply quality: reply-v9","b":true}]},{"t":"quote","runs":[{"text":"2026-09-06 · owner: Aditi · dataset support-v4 (sha 9f21c0)","i":true}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"What was measured.","b":true},{"text":" Whether a generated reply answers the question asked, makes no commitment absent from the policy text, and asks rather than assumes when information is missing. Six binary rules; two are gates."}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"Result.","b":true},{"text":" 86% of items passed all six rules (95% CI 79 to 91, n = 100, 3 runs, spread 2 points). Previous version, same items: 82% (74 to 89). 9 items improved, 4 regressed."}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"Gates.","b":true},{"text":" 0 policy-commitment violations in 300 runs. 1 account-fact violation, item tkt-0087, fixed and retained as a regression test."}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"By slice.","b":true},{"text":" English 91% (n=61) · Hindi 84% (n=25) · Bengali 64% (n=14, interval too wide to act on, needs more items)."}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"Cost and speed.","b":true},{"text":" $0.0068 per reply, up from $0.0031. p95 latency 4.4s, up from 2.9s."}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"What this does not measure.","b":true},{"text":" Whether customers were satisfied; whether the reply was appropriate in tone; anything about voice or attachments; anything in a language other than the three above."}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"Decision it supports.","b":true},{"text":" Ship to the English and Hindi queues. Do not ship to Bengali until that slice has 40 items and clears 80%."}]},{"t":"p","runs":[{"text":"Six headings. Every one of them exists because a team somewhere got burned by leaving it out."}]},{"t":"h2","id":"the-four-sentences-that-make-it-honest","runs":[{"text":"The four sentences that make it honest"}]},{"t":"p","runs":[{"text":"”On these hundred items.”","b":true},{"text":" Never “the model is 86% accurate”. Always “86% on this set”. The set is the scope of the claim and it should be inseparable from it."}]},{"t":"p","runs":[{"text":"”Plus or minus eight.”","b":true},{"text":" From module one. Any number without an interval invites comparison against another number without an interval, and that is how a team ships a three-point regression believing it is a three-point improvement."}]},{"t":"p","runs":[{"text":"”We did not measure X.”","b":true},{"text":" The most valuable line on the page, and the hardest to write, because it reads as weakness and is the opposite. It stops somebody assuming you covered voice input, or Bengali, or tone."}]},{"t":"p","runs":[{"text":"”We cannot tell.”","b":true},{"text":" For the Bengali slice with 14 items, the interval is roughly ±25 points. The honest report is not 64%. It is that the number could be anywhere from 40% to 88% and the only fix is more items. Reporting “we cannot tell yet” once buys more credibility than five confident numbers, because it demonstrates that the confident ones mean something."}]},{"t":"h2","id":"talking-to-people-who-will-not-read-a-confidence-interval","runs":[{"text":"Talking to people who will not read a confidence interval"}]},{"t":"p","runs":[{"text":"Translate, do not simplify away:"}]},{"t":"ul","items":[[{"text":"“About 8 in 10 replies are right, and we could be off by about 1 in 10 either way.”"}],[{"text":"“It’s better than the old version on this set, by roughly 4 points, and 4 items got worse; we read those.”"}],[{"text":"“It works noticeably less well in Bengali, and we don’t have enough Bengali examples yet to say how much less.”"}]]},{"t":"p","runs":[{"text":"What not to do: report a mean judge score. “The assistant scores 8.4 out of 10” is meaningless outside your repository, it is not comparable to anything, it drifts when the judge model updates, and it will be quoted at you for a year. Judge scores are internal, relative, same-run comparisons. Keep them there."}]},{"t":"h2","id":"refusing-to-give-a-number","runs":[{"text":"Refusing to give a number"}]},{"t":"p","runs":[{"text":"Sometimes the right answer to “what’s the accuracy?” is “we don’t have a measurement I would stand behind, and here is what it would take to get one: two days and a hundred labelled examples”."}]},{"t":"p","runs":[{"text":"This is the hardest sentence in the course to say and the one that most defines whether the evaluation function in a company is trusted. A number produced under pressure without a set behind it is a guess wearing a decimal point, and it will be treated as fact by everybody who was not in the room."}]},{"t":"p","runs":[{"text":"Say the two days. In almost every case, somebody will find them."}]}],"check":{"question":[{"text":"An executive asks for a single headline accuracy figure for a slide. Your set is 100 items, the pass rate is 86%, and one language slice of 14 items sits at 64%. What is the best response?"}],"options":[{"text":[{"text":"Give 86% and mention the language gap verbally, since a slide cannot carry caveats."}],"correct":false,"tempting":[{"text":"Attaches the caveat to a conversation while the number goes onto a slide that will be forwarded without it."}]},{"text":[{"text":"Give 82% instead, discounting for the weak slice, so the figure is conservative and defensible."}],"correct":false,"tempting":[{"text":"Invents a number by informal adjustment, which is less defensible than the measured one because nothing supports the discount."}]},{"text":[{"text":"Decline to give any figure until the weak slice has enough items to be measured properly."}],"correct":false,"tempting":[{"text":"Treats an incomplete measurement as no measurement, withholding a result that is genuinely sound for the two slices it covers."}]},{"text":[{"text":"Give “86% on 100 support tickets, plus or minus 8”, state that it covers English and Hindi only, and say the third language cannot yet be measured."}],"correct":true,"tempting":null}],"order":[0,2,3,1],"why":null},"prev":{"slug":"incident-to-eval-item","title":"Turning a complaint into a permanent test","minutes":8},"next":{"slug":"how-much-evaluation-is-enough","title":"How much of this to actually do","minutes":8}}}