The models that have been working since 2002

The quiet success story

Long before anyone had heard of a chatbot, machine learning was already doing a job well: keeping rubbish out of your inbox and stopping your card being used by somebody else. These two are worth studying precisely because they are unglamorous and they work.

Spam filtering as we know it dates to about 2002, when Paul Graham’s essay A Plan for Spam popularised a statistical approach. The method was simple enough to explain in a paragraph: count how often each word appears in spam and in real mail, then combine those counts to estimate the probability that a new message is spam. It was hand-buildable, it needed no graphics card, and it dropped spam volumes in inboxes dramatically.

Modern filters are far more elaborate (sender reputation, network patterns, gradient-boosted trees, and neural models over message text) but the shape is unchanged: examples in, probability out, threshold applied.

The thing that makes these different

A chatbot’s problem does not fight back. Spam and fraud do.

This is called an adversarial setting, and it changes everything about how the system is maintained. The moment a filter learns that a particular phrasing signals spam, spammers stop using it. Deliberate misspellings, images instead of text, hijacked legitimate accounts, and lately fluent generated prose are all responses to filters getting better.

So these models are retrained constantly, daily or faster, while a language model may sit unchanged for a year. If you ever hear that a fraud model was “deployed and finished”, something is wrong. In adversarial settings, a model that is not being retrained is degrading.

Why the threshold is a business decision

Every such system produces a score, and somebody has to choose the line above which it acts. That choice is not technical.

Set the line low and you catch more fraud, and you also decline more genuine purchases. A declined card at a supermarket till costs the bank almost nothing directly and costs you your afternoon. Set the line high and losses rise but customers are undisturbed. The number chosen encodes how the operator values your inconvenience against its money, and it is chosen by people, not learned.

This is the first appearance of an idea that runs through the rest of this course: the model produces a score, a human chooses what the score means. When something is described as “the algorithm’s decision”, there is nearly always a threshold behind it that somebody set.

The trap in the numbers: how rare fraud is

Here is the arithmetic that surprises everybody, and it needs no maths beyond counting.

Suppose one transaction in a thousand is fraudulent. You have a model that is 99% accurate on both kinds. Run it over a million transactions.

• Of the 1,000 fraudulent ones, it catches 990.
• Of the 999,000 genuine ones, it wrongly flags 1%: about 9,990.

So the queue of flagged transactions contains roughly 990 real frauds and 9,990 false alarms. Nine out of every ten alerts are wrong, from a model that is 99% accurate.

Nothing is broken. The rarity of the thing being detected does this. Any time you hear an accuracy figure for a rare event (fraud, disease, extremism, benefit cheating) this arithmetic is waiting underneath, and the honest question is not “how accurate” but “of the cases it flags, how many are real”.

What this means for you

Three practical consequences.

Your card being declined abroad is usually this: your normal pattern changed, the score crossed the line, and the bank preferred a false alarm to a loss. Telling the bank your travel dates helps, because it feeds the model context it does not otherwise have.

A legitimate email of yours landing in spam is usually about the sending domain’s reputation rather than your words. If it happens repeatedly, the fix is at the sending end.

And when a fraud team says “the system flagged you”, you are entitled to ask what happens next, because in a well-run system a flag is a trigger for review, not a verdict. Where a flag is treated as a verdict, and there are national examples in a later lesson, the false-alarm arithmetic above turns into thousands of wrongly accused people.