A score every fifteen minutes
Epic, whose record systems run hundreds of American hospitals, supplies a model that flags patients who may be developing sepsis. It computes a score every fifteen minutes. Epic reported that the model separated sepsis from other cases with an AUC of 0.76 to 0.83. An AUC of 1 is perfect. An AUC of 0.5 is a coin toss.
In 2021 a team at Michigan Medicine checked it on their own patients: 27,697 adults across 38,455 hospital stays. They published the result in JAMA Internal Medicine.
Quick check
How did the model do on Michigan’s patients?
The AUC was 0.63. The model missed 1,709 of the 2,552 patients who developed sepsis, two in three. It still raised an alert on 6,971 stays, 18% of the total. Throughout, it produced a normal-looking score on schedule. Nothing on any screen said it was failing.
Silence reads as success
Ordinary software makes a noise when it cannot do the job. A spreadsheet shows #REF!. A program that cannot open a file stops and says so.
A trained model has no such signal. Hand it a patient unlike the ones it learned from and it still returns a score, of the usual shape, on time.
Three versions are common.
Out of distribution. A model is reliable on inputs like its training data. A new hospital’s patients, a new scanner or a photographed page are different, and the model answers anyway.
Drift. The world moves after launch. Data drift is the inputs changing: new slang, new products, new fraud. Concept drift is the right answer changing for the same input, as when a churn model built before a competitor arrived meets a market that has one.
Quiet patching. Staff start fixing what the system produces and say nothing. The dashboard stays flat because people are absorbing the decline. The edit distance between what the system proposed and what a person sent is the number that shows it.
Three ways to hear a silent failure
Re-run a known set on a schedule. Take the small test set of your own real cases from what a benchmark score leaves out and run it every week. When the answers change, something changed, including a model update nobody announced.
Build in a way to abstain. A system that can say “not confident, send this to a person” fails where you can see it. It makes a demo look worse, which is why it is often skipped.
Log inputs and outputs. You cannot investigate what you did not record. Logs carry a privacy cost, covered in this course’s module on data, but with no log you first hear of a failure from the person it harmed.
The Michigan team found the gap only because they looked.
Check your understanding
A document-reading system has shown the same accuracy for a year, but the operations team now quietly retypes about a fifth of its fields. What does this most likely mean?
Pick the one you would defend. Your answer stays private.
Your place and first choices stay on this browser. Classroom credit is recorded separately.
Stuck on something?Ask Rukmini, your AI tutor