{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"when-the-failure-is-invisible","title":"Failure without an error message","url":"https://addaly.com/learn/ai-and-you/when-the-failure-is-invisible","position":5,"of":73,"minutes":3,"answer":"A model that cannot handle an input still answers, so failure makes no sound.","keyIdea":"A model that cannot handle an input still answers, so failure makes no sound."},"body":{"format":"markdown","text":"## A score every fifteen minutes\n\nEpic, whose record systems run hundreds of American hospitals, supplies a model that flags patients who may be developing sepsis. It computes a score every fifteen minutes. Epic reported that the model separated sepsis from other cases with an AUC of 0.76 to 0.83. An AUC of 1 is perfect. An AUC of 0.5 is a coin toss.\n\nIn 2021 a team at Michigan Medicine checked it on their own patients: 27,697 adults across 38,455 hospital stays. They published the result in *JAMA Internal Medicine*.\n\nThe AUC was 0.63. The model missed 1,709 of the 2,552 patients who developed sepsis, two in three. It still raised an alert on 6,971 stays, 18% of the total. Throughout, it produced a normal-looking score on schedule. Nothing on any screen said it was failing.\n\n> Figure: Michigan Medicine: 2,552 patients with sepsis, and what the alert added. Bars (patients with sepsis): Never flagged by the model 1709; Flagged by the model 843; Flagged, and not yet given timely antibiotics 183. Wong and colleagues, JAMA Internal Medicine, 2021. Of the sepsis patients it did flag, most had already had antibiotics on time. Only 183 were flagged before staff had acted.\n\n## Silence reads as success\n\nOrdinary software makes a noise when it cannot do the job. A spreadsheet shows #REF!. A program that cannot open a file stops and says so.\n\nA trained model has no such signal. Hand it a patient unlike the ones it learned from and it still returns a score, of the usual shape, on time.\n\nThree versions are common.\n\n**Out of distribution.** A model is reliable on inputs like its training data. A new hospital's patients, a new scanner or a photographed page are different, and the model answers anyway.\n\n**Drift.** The world moves after launch. Data drift is the inputs changing: new slang, new products, new fraud. Concept drift is the right answer changing for the same input, as when a churn model built before a competitor arrived meets a market that has one.\n\n**Quiet patching.** Staff start fixing what the system produces and say nothing. The dashboard stays flat because people are absorbing the decline. The edit distance between what the system proposed and what a person sent is the number that shows it.\n\n## Three ways to hear a silent failure\n\n**Re-run a known set on a schedule.** Take the small test set of your own real cases from [what a benchmark score leaves out](/learn/ai-and-you/the-benchmark-and-the-world) and run it every week. When the answers change, something changed, including a model update nobody announced.\n\n**Build in a way to abstain.** A system that can say \"not confident, send this to a person\" fails where you can see it. It makes a demo look worse, which is why it is often skipped.\n\n**Log inputs and outputs.** You cannot investigate what you did not record. Logs carry a privacy cost, covered in this course's module on data, but with no log you first hear of a failure from the person it harmed.\n\nThe Michigan team found the gap only because they looked."},"check":{"question":"A document-reading system has shown the same accuracy for a year, but the operations team now quietly retypes about a fifth of its fields. What does this most likely mean?","options":[{"text":"Accuracy is scored on old tests, and staff absorb a real decline","correct":true},{"text":"The system is fine and the team has grown more cautious over time","correct":false,"tempting":"Takes the flat accuracy figure at face value, when the team's corrections are what keep it flat."},{"text":"The team needs retraining, since they distrust correct output","correct":false,"tempting":"Assumes the people are wrong without checking, which throws away the only signal the system gives."},{"text":"The model is drifting and will soon show errors, so wait for them","correct":false,"tempting":"Expects an error message to arrive. These systems do not produce one. The rate of human correction is the signal."}],"why":"The dashboard only sees the original test set. The retyping is the failure showing through, and it needs counting."},"prev":{"slug":"the-benchmark-and-the-world","title":"What a benchmark score leaves out","url":"https://addaly.com/learn/ai-and-you/the-benchmark-and-the-world"},"next":{"slug":"the-demo-and-the-machinery","title":"How a demo is built","url":"https://addaly.com/learn/ai-and-you/the-demo-and-the-machinery"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Failure without an error message', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/when-the-failure-is-invisible (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Failure without an error message","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":1,"slug":"how-things-fail","title":"How these systems fail"},"answer":null,"keyIdea":"A model that cannot handle an input still answers, so failure makes no sound.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"caption":"Wong and colleagues, JAMA Internal Medicine, 2021. Of the sepsis patients it did flag, most had already had antibiotics on time. Only 183 were flagged before staff had acted.","series":[{"label":"Never flagged by the model","value":1709},{"label":"Flagged by the model","value":843},{"highlight":true,"label":"Flagged, and not yet given timely antibiotics","value":183}],"title":"Michigan Medicine: 2,552 patients with sepsis, and what the alert added","type":"bars","unit":"patients with sepsis"},"alt":"Figure: Michigan Medicine: 2,552 patients with sepsis, and what the alert added. Bars (patients with sepsis): Never flagged by the model 1709; Flagged by the model 843; Flagged, and not yet given timely antibiotics 183. Wong and colleagues, JAMA Internal Medicine, 2021. Of the sepsis patients it did flag, most had already had antibiotics on time. Only 183 were flagged before staff had acted."}},"blocks":[{"t":"h2","id":"a-score-every-fifteen-minutes","runs":[{"text":"A score every fifteen minutes"}]},{"t":"p","runs":[{"text":"Epic, whose record systems run hundreds of American hospitals, supplies a model that flags patients who may be developing sepsis. It computes a score every fifteen minutes. Epic reported that the model separated sepsis from other cases with an AUC of 0.76 to 0.83. An AUC of 1 is perfect. An AUC of 0.5 is a coin toss."}]},{"t":"p","runs":[{"text":"In 2021 a team at Michigan Medicine checked it on their own patients: 27,697 adults across 38,455 hospital stays. They published the result in "},{"text":"JAMA Internal Medicine","i":true},{"text":"."}]},{"t":"quick","prompt":"How did the model do on Michigan’s patients?","options":["As reported, 0.76 or better","A little lower, about 0.72","Much lower, about 0.63"],"answer":2,"reveal":"Much lower, about 0.63. And in all that time it never raised an error."},{"t":"p","runs":[{"text":"The AUC was 0.63. The model missed 1,709 of the 2,552 patients who developed sepsis, two in three. It still raised an alert on 6,971 stays, 18% of the total. Throughout, it produced a normal-looking score on schedule. Nothing on any screen said it was failing."}]},{"t":"h2","id":"silence-reads-as-success","runs":[{"text":"Silence reads as success"}]},{"t":"p","runs":[{"text":"Ordinary software makes a noise when it cannot do the job. A spreadsheet shows #REF!. A program that cannot open a file stops and says so."}]},{"t":"p","runs":[{"text":"A trained model has no such signal. Hand it a patient unlike the ones it learned from and it still returns a score, of the usual shape, on time."}]},{"t":"p","runs":[{"text":"Three versions are common."}]},{"t":"p","runs":[{"text":"Out of distribution.","b":true},{"text":" A model is reliable on inputs like its training data. A new hospital’s patients, a new scanner or a photographed page are different, and the model answers anyway."}]},{"t":"p","runs":[{"text":"Drift.","b":true},{"text":" The world moves after launch. Data drift is the inputs changing: new slang, new products, new fraud. Concept drift is the right answer changing for the same input, as when a churn model built before a competitor arrived meets a market that has one."}]},{"t":"p","runs":[{"text":"Quiet patching.","b":true},{"text":" Staff start fixing what the system produces and say nothing. The dashboard stays flat because people are absorbing the decline. The edit distance between what the system proposed and what a person sent is the number that shows it."}]},{"t":"h2","id":"three-ways-to-hear-a-silent-failure","runs":[{"text":"Three ways to hear a silent failure"}]},{"t":"p","runs":[{"text":"Re-run a known set on a schedule.","b":true},{"text":" Take the small test set of your own real cases from "},{"text":"what a benchmark score leaves out","href":"https://addaly.com/learn/ai-and-you/the-benchmark-and-the-world"},{"text":" and run it every week. When the answers change, something changed, including a model update nobody announced."}]},{"t":"p","runs":[{"text":"Build in a way to abstain.","b":true},{"text":" A system that can say “not confident, send this to a person” fails where you can see it. It makes a demo look worse, which is why it is often skipped."}]},{"t":"p","runs":[{"text":"Log inputs and outputs.","b":true},{"text":" You cannot investigate what you did not record. Logs carry a privacy cost, covered in this course’s module on data, but with no log you first hear of a failure from the person it harmed."}]},{"t":"p","runs":[{"text":"The Michigan team found the gap only because they looked."}]}],"check":{"question":[{"text":"A document-reading system has shown the same accuracy for a year, but the operations team now quietly retypes about a fifth of its fields. What does this most likely mean?"}],"options":[{"text":[{"text":"Accuracy is scored on old tests, and staff absorb a real decline"}],"correct":true,"tempting":null},{"text":[{"text":"The system is fine and the team has grown more cautious over time"}],"correct":false,"tempting":[{"text":"Takes the flat accuracy figure at face value, when the team’s corrections are what keep it flat."}]},{"text":[{"text":"The team needs retraining, since they distrust correct output"}],"correct":false,"tempting":[{"text":"Assumes the people are wrong without checking, which throws away the only signal the system gives."}]},{"text":[{"text":"The model is drifting and will soon show errors, so wait for them"}],"correct":false,"tempting":[{"text":"Expects an error message to arrive. These systems do not produce one. The rate of human correction is the signal."}]}],"order":[2,0,3,1],"why":[{"text":"The dashboard only sees the original test set. The retyping is the failure showing through, and it needs counting."}]},"prev":{"slug":"the-benchmark-and-the-world","title":"What a benchmark score leaves out","minutes":3},"next":{"slug":"the-demo-and-the-machinery","title":"How a demo is built","minutes":3}}}