What a wrong answer costs, and who finds it

Two questions, not one

Before you automate anything, ask what a wrong output costs, and ask who would find out. They are separate questions and the second is the one people skip.

Put them on a grid.

Cheap and visible. A support ticket tagged into the wrong queue. The agent who opens it sees immediately that it is not theirs and moves it. Automate freely; the correction path already exists.

Expensive and visible. A payment sent to the wrong supplier. Somebody will certainly find out, but the money has gone and getting it back involves two banks and a month. Automate behind a gate: a limit in code, a second pair of eyes, or a dry run.

Cheap and invisible. A summary that drops the one clause about the renewal date. Individually trivial, and nobody checks. This needs sampling, because the only way you will ever know the rate is by deliberately looking.

Expensive and invisible. A pricing rule applied to the wrong customer segment, discovered at the audit. Do not automate this until you have built the detector. The detector is the project; the automation is the easy part afterwards.

Silent failure is the real danger

A nightly job at a logistics firm read three input files and produced a depot report. One of the three stopped arriving after a supplier changed a filename. The job did not crash; the reader found no rows and carried on. For six weeks the report was correctly formatted, plausibly shaped, and missing a third of the country.

Nobody noticed because people check format and plausibility, not completeness. A well-formed wrong answer passes every casual inspection, and language models are exceptionally good at producing well-formed answers. This is the mechanism: your reviewer is running a shape check, and the failure is not a shape failure.

The fix is not more reading. It is assertions on quantities the eye cannot hold: row counts against yesterday, date coverage, totals reconciled to the source system, the number of distinct customers. Fail the run when they disagree, loudly.

The reviewer decays

“A human will check it” is not a control unless you design the human’s job.

At a 2 per cent error rate the reviewer sees roughly forty-nine correct items between errors. Attention in monotonous checking tasks falls off within the first half hour, and detection gets worse as the thing you are looking for gets rarer: this is a robust finding in quality-control and screening research, though the size of the effect varies a lot by task and how it is measured. What is not in dispute is the direction: a rare fault in a long stream is the hardest thing for a person to catch.

Two practical consequences. Keep review batches small and varied. And seed known-bad items: put five deliberately wrong records into a hundred and count how many come back. That number is the honest measure of your control. Most teams that do this the first time discover the reviewer is catching one or two, and stop describing human review as a safety net.

Arithmetic before opinion

Two per cent of 500 runs a day is ten wrong outputs a day, two hundred a month. Now ask what one wrong output costs and who pays it. Two per cent is an excellent error rate for a model and an unacceptable one for a payment instruction. The same number is a success or a scandal depending entirely on which column it lands in, and the only way to know is to write the multiplication down.

Bias the system towards the cheaper mistake

Errors are rarely symmetric. A triage bot that escalates when unsure costs you an escalation; one that closes the ticket when unsure costs you a customer. So build the abstain path deliberately: “I am not sure, send this to a person” is a first-class output, not a failure.

It needs a budget. If 40 per cent of cases abstain, you have not automated anything, you have added a queue. Track the abstain rate as carefully as the error rate; the two move against each other, and choosing where to sit between them is the actual design decision.

Write the error budget down

Three lines, next to the code:

• The error rate we accept, stated as a number.
• Who detects an error, and how.
• The time within which it must be detected.

If the third line reads “at the annual audit”, you have found the quadrant you are in, and the answer is to build the detector first.

None of this needs tooling. A sampling ledger is a spreadsheet (LibreOffice Calc or Google Sheets) with twenty random rows a week, a column for verdict, and a running rate at the bottom. The discipline is doing it in the weeks when nothing appears to be wrong.