Failure without an error message

Software used to tell you

When a spreadsheet formula breaks, you get #REF!. When a program cannot open a file, it stops and says so. Decades of computing habit rest on a simple property: when the machine cannot do the thing, it makes a noise.

Machine learning systems do not have that property, and this single difference is responsible for more real-world harm than any exotic risk. A model handed an input far outside anything it has seen does not stop. It produces an output of exactly the usual shape, with the usual formatting and the usual tone. The failure is silent, and silence in software reads as success.

Three specific versions of this are worth being able to name.

Out of distribution

A model is reliable on inputs resembling its training data. Hand it something genuinely different and it still answers, because answering is all it does.

A medical imaging model trained on scans from three hospitals meets a fourth hospital's scanner with different settings, and its accuracy drops without anything appearing on a dashboard. A document reader trained on clean PDFs meets a photographed, slightly rotated page from a Xerox machine and starts inventing digits rather than reporting that it cannot read them. A speech system trained on studio audio meets a call from a market street.

The useful habit: before trusting a system on a case, ask whether this case resembles the cases it was tested on. If you cannot answer, that is itself the answer.

Drift

A system can be right on the day it launches and wrong six months later without one line of code changing, because the world moved.

Two distinct things move. Data drift is the inputs changing: new slang, a new product line, a new document format, a new fraud pattern. Concept drift is the relationship changing: the same input should now produce a different answer. A model predicting which customers will churn was built when a competitor did not exist; now one does.

Covid-era retail forecasting is the textbook case — every demand model in the world was trained on a world that stopped existing in a fortnight, and none of them raised an alarm. They kept predicting, confidently, using the old relationship.

Drift is why a deployed model needs monitoring rather than a launch. The minimum useful monitor is not clever: track the distribution of the inputs and the distribution of the outputs over time, and alert when either shifts. You do not need labels for that, which is the point — labels are the expensive part.

Degradation you adapt to

The subtlest version. The system gets slightly worse, and the humans around it compensate without noticing they are compensating. Support staff quietly rewrite the drafts. Clinicians ignore certain alerts. Warehouse workers learn which recommendations to override. Six months later the system's reported accuracy is unchanged, because the humans are patching the difference, and nobody has measured how much patching is happening.

This is measurable if you decide to measure it: track the override rate and the edit distance between what the system proposed and what the human sent. A rising override rate is a failing system telling you the only way it can.

What to do about silence

Three practices, in increasing order of effort.

Keep a canary set. Twenty or thirty inputs whose correct output you know, run on a schedule — weekly is plenty for most things. When the answers change, something changed. This costs almost nothing and catches provider-side model updates you were not told about.

Make abstention possible. A system that can say I am not confident, route this to a person fails visibly. Building that route in is a design decision, and it is usually skipped because it makes the demo look worse.

Log inputs and outputs. You cannot investigate what you did not record. Retention has a privacy cost, covered later in this course, and the balance is real — but a system with no log is a system whose failures you will hear about first from the person harmed.

The general rule: with conventional software you find out because it breaks; with these systems you only find out because you looked.