{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"the-cost-of-being-wrong","title":"What a wrong answer costs, and who finds it","url":"https://addaly.com/learn/agents-and-automation/the-cost-of-being-wrong","position":5,"of":72,"minutes":9,"answer":"Classify work by the cost of an error and by whether anyone would notice it; the expensive-and-invisible quadrant needs a detector built before the automation, not after.","keyIdea":"Classify work by the cost of an error and by whether anyone would notice it; the expensive-and-invisible quadrant needs a detector built before the automation, not after."},"body":{"format":"markdown","text":"## Two questions, not one\n\nBefore you automate anything, ask what a wrong output costs, and ask who would find out. They are separate questions and the second is the one people skip.\n\nPut them on a grid.\n\n**Cheap and visible.** A support ticket tagged into the wrong queue. The agent who opens it sees immediately that it is not theirs and moves it. Automate freely; the correction path already exists.\n\n**Expensive and visible.** A payment sent to the wrong supplier. Somebody will certainly find out, but the money has gone and getting it back involves two banks and a month. Automate behind a gate: a limit in code, a second pair of eyes, or a dry run.\n\n**Cheap and invisible.** A summary that drops the one clause about the renewal date. Individually trivial, and nobody checks. This needs sampling, because the only way you will ever know the rate is by deliberately looking.\n\n**Expensive and invisible.** A pricing rule applied to the wrong customer segment, discovered at the audit. Do not automate this until you have built the detector. The detector is the project; the automation is the easy part afterwards.\n\n> Figure: What a wrong answer costs, and who would find out. A two by two grid, rows Cheap error and Expensive error, columns Someone notices and Nobody notices: Cheap error and Someone notices: Automate freely; Cheap error and Nobody notices: Sample deliberately; Expensive error and Someone notices: Gate every one; Expensive error and Nobody notices: Build the detector first. The second question is the one people skip. Expensive-and-invisible is not an automation project yet: the detector is the project, and the automation is the easy part afterwards.\n\n## Silent failure is the real danger\n\nA nightly job at a logistics firm read three input files and produced a depot report. One of the three stopped arriving after a supplier changed a filename. The job did not crash; the reader found no rows and carried on. For six weeks the report was correctly formatted, plausibly shaped, and missing a third of the country.\n\nNobody noticed because people check format and plausibility, not completeness. A well-formed wrong answer passes every casual inspection, and language models are exceptionally good at producing well-formed answers. This is the mechanism: your reviewer is running a shape check, and the failure is not a shape failure.\n\nThe fix is not more reading. It is assertions on quantities the eye cannot hold: row counts against yesterday, date coverage, totals reconciled to the source system, the number of distinct customers. Fail the run when they disagree, loudly.\n\n## The reviewer decays\n\n\"A human will check it\" is not a control unless you design the human's job.\n\nAt a 2 per cent error rate the reviewer sees roughly forty-nine correct items between errors. Attention in monotonous checking tasks falls off within the first half hour, and detection gets worse as the thing you are looking for gets rarer: this is a robust finding in quality-control and screening research, though the size of the effect varies a lot by task and how it is measured. What is not in dispute is the direction: a rare fault in a long stream is the hardest thing for a person to catch.\n\nTwo practical consequences. Keep review batches small and varied. And seed known-bad items: put five deliberately wrong records into a hundred and count how many come back. That number is the honest measure of your control. Most teams that do this the first time discover the reviewer is catching one or two, and stop describing human review as a safety net.\n\n## Arithmetic before opinion\n\nTwo per cent of 500 runs a day is ten wrong outputs a day, two hundred a month. Now ask what one wrong output costs and who pays it. Two per cent is an excellent error rate for a model and an unacceptable one for a payment instruction. The same number is a success or a scandal depending entirely on which column it lands in, and the only way to know is to write the multiplication down.\n\n## Bias the system towards the cheaper mistake\n\nErrors are rarely symmetric. A triage bot that escalates when unsure costs you an escalation; one that closes the ticket when unsure costs you a customer. So build the abstain path deliberately: \"I am not sure, send this to a person\" is a first-class output, not a failure.\n\nIt needs a budget. If 40 per cent of cases abstain, you have not automated anything, you have added a queue. Track the abstain rate as carefully as the error rate; the two move against each other, and choosing where to sit between them is the actual design decision.\n\n## Write the error budget down\n\nThree lines, next to the code:\n\n- The error rate we accept, stated as a number.\n- Who detects an error, and how.\n- The time within which it must be detected.\n\nIf the third line reads \"at the annual audit\", you have found the quadrant you are in, and the answer is to build the detector first.\n\nNone of this needs tooling. A sampling ledger is a spreadsheet (LibreOffice Calc or Google Sheets) with twenty random rows a week, a column for verdict, and a running rate at the bottom. The discipline is doing it in the weeks when nothing appears to be wrong."},"check":{"question":"A nightly extraction runs at a measured 2 per cent error rate, and a person spot-checks ten outputs each morning. Which change most improves the chance that a genuinely bad week is caught?","options":[{"text":"Increase the spot check to twenty outputs each morning.","correct":false,"tempting":"Doubles the sample but not the method; at 2 per cent most mornings still show nothing, and the reviewer learns to expect nothing."},{"text":"Have the model rate its confidence on each row and route the low-confidence rows to the reviewer.","correct":false,"tempting":"Treats stated confidence as calibrated; it is generated text produced the same way as the answer, and it does not track correctness."},{"text":"Move to a larger model to push the error rate below one per cent.","correct":false,"tempting":"Reduces how often errors happen while leaving detection exactly as weak; the failure that hurts is the one nobody sees, not the one that is rare."},{"text":"Assert on totals (row counts, date coverage and sums reconciled against the source) and fail the run when they disagree.","correct":true}]},"prev":{"slug":"what-work-is-automatable","title":"Which work is worth automating","url":"https://addaly.com/learn/agents-and-automation/what-work-is-automatable"},"next":{"slug":"why-demos-fail","title":"Why the demo worked and production did not","url":"https://addaly.com/learn/agents-and-automation/why-demos-fail"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What a wrong answer costs, and who finds it', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/the-cost-of-being-wrong (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What a wrong answer costs, and who finds it","courseTitle":"Agents and Automation","module":{"index":1,"slug":"deciding-what-to-build","title":"Before you build: script, workflow, or agent"},"answer":null,"keyIdea":"Classify work by the cost of an error and by whether anyone would notice it; the expensive-and-invisible quadrant needs a detector built before the automation, not after.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Someone notices","Nobody notices"],"rows":["Cheap error","Expensive error"],"type":"matrix","cells":[["Automate freely","Sample deliberately"],["Gate every one","Build the detector first"]],"title":"What a wrong answer costs, and who would find out","caption":"The second question is the one people skip. Expensive-and-invisible is not an automation project yet: the detector is the project, and the automation is the easy part afterwards.","cellLabels":[["ticket sent to the wrong queue","a summary that drops the renewal clause"],["payment to the wrong supplier","wrong pricing rule, found at the audit"]]},"alt":"Figure: What a wrong answer costs, and who would find out. A two by two grid, rows Cheap error and Expensive error, columns Someone notices and Nobody notices: Cheap error and Someone notices: Automate freely; Cheap error and Nobody notices: Sample deliberately; Expensive error and Someone notices: Gate every one; Expensive error and Nobody notices: Build the detector first. The second question is the one people skip. Expensive-and-invisible is not an automation project yet: the detector is the project, and the automation is the easy part afterwards."}},"blocks":[{"t":"h2","id":"two-questions-not-one","runs":[{"text":"Two questions, not one"}]},{"t":"p","runs":[{"text":"Before you automate anything, ask what a wrong output costs, and ask who would find out. They are separate questions and the second is the one people skip."}]},{"t":"p","runs":[{"text":"Put them on a grid."}]},{"t":"p","runs":[{"text":"Cheap and visible.","b":true},{"text":" A support ticket tagged into the wrong queue. The agent who opens it sees immediately that it is not theirs and moves it. Automate freely; the correction path already exists."}]},{"t":"p","runs":[{"text":"Expensive and visible.","b":true},{"text":" A payment sent to the wrong supplier. Somebody will certainly find out, but the money has gone and getting it back involves two banks and a month. Automate behind a gate: a limit in code, a second pair of eyes, or a dry run."}]},{"t":"p","runs":[{"text":"Cheap and invisible.","b":true},{"text":" A summary that drops the one clause about the renewal date. Individually trivial, and nobody checks. This needs sampling, because the only way you will ever know the rate is by deliberately looking."}]},{"t":"p","runs":[{"text":"Expensive and invisible.","b":true},{"text":" A pricing rule applied to the wrong customer segment, discovered at the audit. Do not automate this until you have built the detector. The detector is the project; the automation is the easy part afterwards."}]},{"t":"h2","id":"silent-failure-is-the-real-danger","runs":[{"text":"Silent failure is the real danger"}]},{"t":"p","runs":[{"text":"A nightly job at a logistics firm read three input files and produced a depot report. One of the three stopped arriving after a supplier changed a filename. The job did not crash; the reader found no rows and carried on. For six weeks the report was correctly formatted, plausibly shaped, and missing a third of the country."}]},{"t":"p","runs":[{"text":"Nobody noticed because people check format and plausibility, not completeness. A well-formed wrong answer passes every casual inspection, and language models are exceptionally good at producing well-formed answers. This is the mechanism: your reviewer is running a shape check, and the failure is not a shape failure."}]},{"t":"p","runs":[{"text":"The fix is not more reading. It is assertions on quantities the eye cannot hold: row counts against yesterday, date coverage, totals reconciled to the source system, the number of distinct customers. Fail the run when they disagree, loudly."}]},{"t":"h2","id":"the-reviewer-decays","runs":[{"text":"The reviewer decays"}]},{"t":"p","runs":[{"text":"“A human will check it” is not a control unless you design the human’s job."}]},{"t":"p","runs":[{"text":"At a 2 per cent error rate the reviewer sees roughly forty-nine correct items between errors. Attention in monotonous checking tasks falls off within the first half hour, and detection gets worse as the thing you are looking for gets rarer: this is a robust finding in quality-control and screening research, though the size of the effect varies a lot by task and how it is measured. What is not in dispute is the direction: a rare fault in a long stream is the hardest thing for a person to catch."}]},{"t":"p","runs":[{"text":"Two practical consequences. Keep review batches small and varied. And seed known-bad items: put five deliberately wrong records into a hundred and count how many come back. That number is the honest measure of your control. Most teams that do this the first time discover the reviewer is catching one or two, and stop describing human review as a safety net."}]},{"t":"h2","id":"arithmetic-before-opinion","runs":[{"text":"Arithmetic before opinion"}]},{"t":"p","runs":[{"text":"Two per cent of 500 runs a day is ten wrong outputs a day, two hundred a month. Now ask what one wrong output costs and who pays it. Two per cent is an excellent error rate for a model and an unacceptable one for a payment instruction. The same number is a success or a scandal depending entirely on which column it lands in, and the only way to know is to write the multiplication down."}]},{"t":"h2","id":"bias-the-system-towards-the-cheaper-mistake","runs":[{"text":"Bias the system towards the cheaper mistake"}]},{"t":"p","runs":[{"text":"Errors are rarely symmetric. A triage bot that escalates when unsure costs you an escalation; one that closes the ticket when unsure costs you a customer. So build the abstain path deliberately: “I am not sure, send this to a person” is a first-class output, not a failure."}]},{"t":"p","runs":[{"text":"It needs a budget. If 40 per cent of cases abstain, you have not automated anything, you have added a queue. Track the abstain rate as carefully as the error rate; the two move against each other, and choosing where to sit between them is the actual design decision."}]},{"t":"h2","id":"write-the-error-budget-down","runs":[{"text":"Write the error budget down"}]},{"t":"p","runs":[{"text":"Three lines, next to the code:"}]},{"t":"ul","items":[[{"text":"The error rate we accept, stated as a number."}],[{"text":"Who detects an error, and how."}],[{"text":"The time within which it must be detected."}]]},{"t":"p","runs":[{"text":"If the third line reads “at the annual audit”, you have found the quadrant you are in, and the answer is to build the detector first."}]},{"t":"p","runs":[{"text":"None of this needs tooling. A sampling ledger is a spreadsheet (LibreOffice Calc or Google Sheets) with twenty random rows a week, a column for verdict, and a running rate at the bottom. The discipline is doing it in the weeks when nothing appears to be wrong."}]}],"check":{"question":[{"text":"A nightly extraction runs at a measured 2 per cent error rate, and a person spot-checks ten outputs each morning. Which change most improves the chance that a genuinely bad week is caught?"}],"options":[{"text":[{"text":"Increase the spot check to twenty outputs each morning."}],"correct":false,"tempting":[{"text":"Doubles the sample but not the method; at 2 per cent most mornings still show nothing, and the reviewer learns to expect nothing."}]},{"text":[{"text":"Have the model rate its confidence on each row and route the low-confidence rows to the reviewer."}],"correct":false,"tempting":[{"text":"Treats stated confidence as calibrated; it is generated text produced the same way as the answer, and it does not track correctness."}]},{"text":[{"text":"Move to a larger model to push the error rate below one per cent."}],"correct":false,"tempting":[{"text":"Reduces how often errors happen while leaving detection exactly as weak; the failure that hurts is the one nobody sees, not the one that is rare."}]},{"text":[{"text":"Assert on totals (row counts, date coverage and sums reconciled against the source) and fail the run when they disagree."}],"correct":true,"tempting":null}],"order":[0,3,1,2],"why":null},"prev":{"slug":"what-work-is-automatable","title":"Which work is worth automating","minutes":8},"next":{"slug":"why-demos-fail","title":"Why the demo worked and production did not","minutes":8}}}