{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"spam-fraud-and-the-oldest-ai","title":"The models that have been working since 2002","url":"https://addaly.com/learn/ai-actually-explained/spam-fraud-and-the-oldest-ai","position":12,"of":92,"minutes":9,"answer":"When the thing being detected is rare, even a 99% accurate model produces mostly false alarms, so ask what fraction of flagged cases are real rather than how accurate the model is.","keyIdea":"When the thing being detected is rare, even a 99% accurate model produces mostly false alarms, so ask what fraction of flagged cases are real rather than how accurate the model is."},"body":{"format":"markdown","text":"## The quiet success story\n\nLong before anyone had heard of a chatbot, machine learning was already doing a job well: keeping rubbish out of your inbox and stopping your card being used by somebody else. These two are worth studying precisely because they are unglamorous and they work.\n\nSpam filtering as we know it dates to about 2002, when Paul Graham's essay *A Plan for Spam* popularised a statistical approach. The method was simple enough to explain in a paragraph: count how often each word appears in spam and in real mail, then combine those counts to estimate the probability that a new message is spam. It was hand-buildable, it needed no graphics card, and it dropped spam volumes in inboxes dramatically.\n\nModern filters are far more elaborate (sender reputation, network patterns, gradient-boosted trees, and neural models over message text) but the shape is unchanged: examples in, probability out, threshold applied.\n\n## The thing that makes these different\n\nA chatbot's problem does not fight back. Spam and fraud do.\n\nThis is called an **adversarial** setting, and it changes everything about how the system is maintained. The moment a filter learns that a particular phrasing signals spam, spammers stop using it. Deliberate misspellings, images instead of text, hijacked legitimate accounts, and lately fluent generated prose are all responses to filters getting better.\n\nSo these models are retrained constantly, daily or faster, while a language model may sit unchanged for a year. If you ever hear that a fraud model was \"deployed and finished\", something is wrong. In adversarial settings, a model that is not being retrained is degrading.\n\n## Why the threshold is a business decision\n\nEvery such system produces a score, and somebody has to choose the line above which it acts. That choice is not technical.\n\nSet the line low and you catch more fraud, and you also decline more genuine purchases. A declined card at a supermarket till costs the bank almost nothing directly and costs you your afternoon. Set the line high and losses rise but customers are undisturbed. The number chosen encodes how the operator values your inconvenience against its money, and it is chosen by people, not learned.\n\nThis is the first appearance of an idea that runs through the rest of this course: **the model produces a score, a human chooses what the score means.** When something is described as \"the algorithm's decision\", there is nearly always a threshold behind it that somebody set.\n\n## The trap in the numbers: how rare fraud is\n\nHere is the arithmetic that surprises everybody, and it needs no maths beyond counting.\n\nSuppose one transaction in a thousand is fraudulent. You have a model that is 99% accurate on both kinds. Run it over a million transactions.\n\n- Of the 1,000 fraudulent ones, it catches 990.\n- Of the 999,000 genuine ones, it wrongly flags 1%: about 9,990.\n\nSo the queue of flagged transactions contains roughly 990 real frauds and 9,990 false alarms. Nine out of every ten alerts are wrong, from a model that is 99% accurate.\n\n> Figure: A 99% accurate fraud model over a million transactions. A two by two grid, rows Actually fraud (1,000) and Actually genuine (999,000), columns Model flags and Model clears: Actually fraud (1,000) and Model flags: 990; Actually fraud (1,000) and Model clears: 10; Actually genuine (999,000) and Model flags: 9,990; Actually genuine (999,000) and Model clears: 989,010. One transaction in a thousand is fraud. The model catches 990 of the 1,000 and wrongly flags 1% of the 999,000 genuine ones. Nine of every ten alerts are false, from a model that is 99% accurate, which is why the honest question is what fraction of flagged cases are real.\n\nNothing is broken. The rarity of the thing being detected does this. Any time you hear an accuracy figure for a rare event (fraud, disease, extremism, benefit cheating) this arithmetic is waiting underneath, and the honest question is not \"how accurate\" but \"of the cases it flags, how many are real\".\n\n## What this means for you\n\nThree practical consequences.\n\nYour card being declined abroad is usually this: your normal pattern changed, the score crossed the line, and the bank preferred a false alarm to a loss. Telling the bank your travel dates helps, because it feeds the model context it does not otherwise have.\n\nA legitimate email of yours landing in spam is usually about the sending domain's reputation rather than your words. If it happens repeatedly, the fix is at the sending end.\n\nAnd when a fraud team says \"the system flagged you\", you are entitled to ask what happens next, because in a well-run system a flag is a trigger for review, not a verdict. Where a flag is treated as a verdict, and there are national examples in a later lesson, the false-alarm arithmetic above turns into thousands of wrongly accused people."},"check":{"question":"A benefits agency reports that its new fraud model is 98% accurate, and that fraud occurs in about 1 in 200 claims. A minister concludes that flagged claimants are almost certainly cheating. What is wrong with that conclusion?","options":[{"text":"Accuracy figures are always inflated by vendors, so the true rate is much lower than 98%.","correct":false,"tempting":"Attacks the honesty of the number instead of the reasoning, which fails even if the 98% is entirely truthful."},{"text":"The model was trained on past cases, so it cannot detect new fraud methods.","correct":false,"tempting":"Names a genuine problem in adversarial settings, but it is not what makes this particular inference invalid."},{"text":"98% accuracy is too low for government use and the threshold should simply be raised.","correct":false,"tempting":"Treats it as a tuning problem; raising the threshold trades one error for the other and does not fix the inference."},{"text":"Because genuine claims outnumber fraudulent ones about 200 to 1, the small error rate on that huge majority produces far more false flags than real ones.","correct":true}]},"prev":{"slug":"the-feed-that-learns-you","title":"The feed is a prediction machine","url":"https://addaly.com/learn/ai-actually-explained/the-feed-that-learns-you"},"next":{"slug":"the-camera-in-your-pocket","title":"Your photographs are partly predicted","url":"https://addaly.com/learn/ai-actually-explained/the-camera-in-your-pocket"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The models that have been working since 2002', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/spam-fraud-and-the-oldest-ai (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The models that have been working since 2002","courseTitle":"AI, Actually Explained","module":{"index":2,"slug":"where-it-already-is","title":"Where it already is, without a label"},"answer":null,"keyIdea":"When the thing being detected is rare, even a 99% accurate model produces mostly false alarms, so ask what fraction of flagged cases are real rather than how accurate the model is.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Model flags","Model clears"],"rows":["Actually fraud (1,000)","Actually genuine (999,000)"],"type":"matrix","cells":[["990","10"],["9,990","989,010"]],"title":"A 99% accurate fraud model over a million transactions","caption":"One transaction in a thousand is fraud. The model catches 990 of the 1,000 and wrongly flags 1% of the 999,000 genuine ones. Nine of every ten alerts are false, from a model that is 99% accurate, which is why the honest question is what fraction of flagged cases are real.","cellLabels":[["caught","missed"],["false alarm","correctly cleared"]]},"alt":"Figure: A 99% accurate fraud model over a million transactions. A two by two grid, rows Actually fraud (1,000) and Actually genuine (999,000), columns Model flags and Model clears: Actually fraud (1,000) and Model flags: 990; Actually fraud (1,000) and Model clears: 10; Actually genuine (999,000) and Model flags: 9,990; Actually genuine (999,000) and Model clears: 989,010. One transaction in a thousand is fraud. The model catches 990 of the 1,000 and wrongly flags 1% of the 999,000 genuine ones. Nine of every ten alerts are false, from a model that is 99% accurate, which is why the honest question is what fraction of flagged cases are real."}},"blocks":[{"t":"h2","id":"the-quiet-success-story","runs":[{"text":"The quiet success story"}]},{"t":"p","runs":[{"text":"Long before anyone had heard of a chatbot, machine learning was already doing a job well: keeping rubbish out of your inbox and stopping your card being used by somebody else. These two are worth studying precisely because they are unglamorous and they work."}]},{"t":"p","runs":[{"text":"Spam filtering as we know it dates to about 2002, when Paul Graham’s essay "},{"text":"A Plan for Spam","i":true},{"text":" popularised a statistical approach. The method was simple enough to explain in a paragraph: count how often each word appears in spam and in real mail, then combine those counts to estimate the probability that a new message is spam. It was hand-buildable, it needed no graphics card, and it dropped spam volumes in inboxes dramatically."}]},{"t":"p","runs":[{"text":"Modern filters are far more elaborate (sender reputation, network patterns, gradient-boosted trees, and neural models over message text) but the shape is unchanged: examples in, probability out, threshold applied."}]},{"t":"h2","id":"the-thing-that-makes-these-different","runs":[{"text":"The thing that makes these different"}]},{"t":"p","runs":[{"text":"A chatbot’s problem does not fight back. Spam and fraud do."}]},{"t":"p","runs":[{"text":"This is called an "},{"text":"adversarial","b":true},{"text":" setting, and it changes everything about how the system is maintained. The moment a filter learns that a particular phrasing signals spam, spammers stop using it. Deliberate misspellings, images instead of text, hijacked legitimate accounts, and lately fluent generated prose are all responses to filters getting better."}]},{"t":"p","runs":[{"text":"So these models are retrained constantly, daily or faster, while a language model may sit unchanged for a year. If you ever hear that a fraud model was “deployed and finished”, something is wrong. In adversarial settings, a model that is not being retrained is degrading."}]},{"t":"h2","id":"why-the-threshold-is-a-business-decision","runs":[{"text":"Why the threshold is a business decision"}]},{"t":"p","runs":[{"text":"Every such system produces a score, and somebody has to choose the line above which it acts. That choice is not technical."}]},{"t":"p","runs":[{"text":"Set the line low and you catch more fraud, and you also decline more genuine purchases. A declined card at a supermarket till costs the bank almost nothing directly and costs you your afternoon. Set the line high and losses rise but customers are undisturbed. The number chosen encodes how the operator values your inconvenience against its money, and it is chosen by people, not learned."}]},{"t":"p","runs":[{"text":"This is the first appearance of an idea that runs through the rest of this course: "},{"text":"the model produces a score, a human chooses what the score means.","b":true},{"text":" When something is described as “the algorithm’s decision”, there is nearly always a threshold behind it that somebody set."}]},{"t":"h2","id":"the-trap-in-the-numbers-how-rare-fraud-is","runs":[{"text":"The trap in the numbers: how rare fraud is"}]},{"t":"p","runs":[{"text":"Here is the arithmetic that surprises everybody, and it needs no maths beyond counting."}]},{"t":"p","runs":[{"text":"Suppose one transaction in a thousand is fraudulent. You have a model that is 99% accurate on both kinds. Run it over a million transactions."}]},{"t":"ul","items":[[{"text":"Of the 1,000 fraudulent ones, it catches 990."}],[{"text":"Of the 999,000 genuine ones, it wrongly flags 1%: about 9,990."}]]},{"t":"p","runs":[{"text":"So the queue of flagged transactions contains roughly 990 real frauds and 9,990 false alarms. Nine out of every ten alerts are wrong, from a model that is 99% accurate."}]},{"t":"p","runs":[{"text":"Nothing is broken. The rarity of the thing being detected does this. Any time you hear an accuracy figure for a rare event (fraud, disease, extremism, benefit cheating) this arithmetic is waiting underneath, and the honest question is not “how accurate” but “of the cases it flags, how many are real”."}]},{"t":"h2","id":"what-this-means-for-you","runs":[{"text":"What this means for you"}]},{"t":"p","runs":[{"text":"Three practical consequences."}]},{"t":"p","runs":[{"text":"Your card being declined abroad is usually this: your normal pattern changed, the score crossed the line, and the bank preferred a false alarm to a loss. Telling the bank your travel dates helps, because it feeds the model context it does not otherwise have."}]},{"t":"p","runs":[{"text":"A legitimate email of yours landing in spam is usually about the sending domain’s reputation rather than your words. If it happens repeatedly, the fix is at the sending end."}]},{"t":"p","runs":[{"text":"And when a fraud team says “the system flagged you”, you are entitled to ask what happens next, because in a well-run system a flag is a trigger for review, not a verdict. Where a flag is treated as a verdict, and there are national examples in a later lesson, the false-alarm arithmetic above turns into thousands of wrongly accused people."}]}],"check":{"question":[{"text":"A benefits agency reports that its new fraud model is 98% accurate, and that fraud occurs in about 1 in 200 claims. A minister concludes that flagged claimants are almost certainly cheating. What is wrong with that conclusion?"}],"options":[{"text":[{"text":"Accuracy figures are always inflated by vendors, so the true rate is much lower than 98%."}],"correct":false,"tempting":[{"text":"Attacks the honesty of the number instead of the reasoning, which fails even if the 98% is entirely truthful."}]},{"text":[{"text":"The model was trained on past cases, so it cannot detect new fraud methods."}],"correct":false,"tempting":[{"text":"Names a genuine problem in adversarial settings, but it is not what makes this particular inference invalid."}]},{"text":[{"text":"98% accuracy is too low for government use and the threshold should simply be raised."}],"correct":false,"tempting":[{"text":"Treats it as a tuning problem; raising the threshold trades one error for the other and does not fix the inference."}]},{"text":[{"text":"Because genuine claims outnumber fraudulent ones about 200 to 1, the small error rate on that huge majority produces far more false flags than real ones."}],"correct":true,"tempting":null}],"order":[2,1,0,3],"why":null},"prev":{"slug":"the-feed-that-learns-you","title":"The feed is a prediction machine","minutes":9},"next":{"slug":"the-camera-in-your-pocket","title":"Your photographs are partly predicted","minutes":9}}}