Writing the result down

The number leaves the room without you

You will say “about 86%, on a hundred items, plus or minus eight, and the Bengali slice is much worse”. What travels to the next meeting is “86%”. By the third retelling it is “nearly 90% accurate” and it is in a slide to a customer.

The only defence is a written artefact that goes wherever the number goes. One page. It takes fifteen minutes and it is the difference between a measurement and a rumour.

The result card

Support reply quality: reply-v9

2026-09-06 · owner: Aditi · dataset support-v4 (sha 9f21c0)

What was measured. Whether a generated reply answers the question asked, makes no commitment absent from the policy text, and asks rather than assumes when information is missing. Six binary rules; two are gates.

Result. 86% of items passed all six rules (95% CI 79 to 91, n = 100, 3 runs, spread 2 points). Previous version, same items: 82% (74 to 89). 9 items improved, 4 regressed.

Gates. 0 policy-commitment violations in 300 runs. 1 account-fact violation, item tkt-0087, fixed and retained as a regression test.

By slice. English 91% (n=61) · Hindi 84% (n=25) · Bengali 64% (n=14, interval too wide to act on, needs more items).

Cost and speed. $0.0068 per reply, up from $0.0031. p95 latency 4.4s, up from 2.9s.

What this does not measure. Whether customers were satisfied; whether the reply was appropriate in tone; anything about voice or attachments; anything in a language other than the three above.

Decision it supports. Ship to the English and Hindi queues. Do not ship to Bengali until that slice has 40 items and clears 80%.

Six headings. Every one of them exists because a team somewhere got burned by leaving it out.

The four sentences that make it honest

”On these hundred items.” Never “the model is 86% accurate”. Always “86% on this set”. The set is the scope of the claim and it should be inseparable from it.

”Plus or minus eight.” From module one. Any number without an interval invites comparison against another number without an interval, and that is how a team ships a three-point regression believing it is a three-point improvement.

”We did not measure X.” The most valuable line on the page, and the hardest to write, because it reads as weakness and is the opposite. It stops somebody assuming you covered voice input, or Bengali, or tone.

”We cannot tell.” For the Bengali slice with 14 items, the interval is roughly ±25 points. The honest report is not 64%. It is that the number could be anywhere from 40% to 88% and the only fix is more items. Reporting “we cannot tell yet” once buys more credibility than five confident numbers, because it demonstrates that the confident ones mean something.

Talking to people who will not read a confidence interval

Translate, do not simplify away:

• “About 8 in 10 replies are right, and we could be off by about 1 in 10 either way.”
• “It’s better than the old version on this set, by roughly 4 points, and 4 items got worse; we read those.”
• “It works noticeably less well in Bengali, and we don’t have enough Bengali examples yet to say how much less.”

What not to do: report a mean judge score. “The assistant scores 8.4 out of 10” is meaningless outside your repository, it is not comparable to anything, it drifts when the judge model updates, and it will be quoted at you for a year. Judge scores are internal, relative, same-run comparisons. Keep them there.

Refusing to give a number

Sometimes the right answer to “what’s the accuracy?” is “we don’t have a measurement I would stand behind, and here is what it would take to get one: two days and a hundred labelled examples”.

This is the hardest sentence in the course to say and the one that most defines whether the evaluation function in a company is trusted. A number produced under pressure without a set behind it is a guess wearing a decimal point, and it will be treated as fact by everybody who was not in the room.

Say the two days. In almost every case, somebody will find them.