Anomaly detection, and the base rate that ruins it

A different problem shape

Classification learns the boundary between two classes you have examples of. Anomaly detection learns what normal looks like and flags departures, using mostly or entirely normal data.

That framing is right when the anomalies are too rare to learn from, too varied to have a consistent pattern, or when tomorrow’s anomaly will not resemble today’s. Equipment failures, novel intrusions, new fraud schemes: in each case the positive class is not one thing.

Four methods

Isolation Forest. Build random trees by picking a random feature and a random split point. Anomalies get isolated in fewer splits, because they sit in sparse regions where a random cut is likely to separate them. The score is the average path length to isolate a point.

from sklearn.ensemble import IsolationForest
iso = IsolationForest(contamination=0.01, random_state=0).fit(X)
scores = -iso.score_samples(X_new)     # higher means more anomalous

Fast, handles mixed distributions, needs little tuning, scales well. The usual first choice.

Local Outlier Factor. Compares a point’s local density with the density of its neighbours. Its advantage is finding local anomalies: a point that is unremarkable globally but strange for its immediate neighbourhood. A ₹50,000 transaction is ordinary overall and extraordinary for a customer who has never exceeded ₹2,000.

One-Class SVM. Fits a boundary around the normal region. Sensitive to its parameters and slow above tens of thousands of rows, but useful when you have clean normal-only data.

Reconstruction error. Fit PCA or an autoencoder on normal data, reconstruct a new point, and measure the error. Points the model cannot rebuild are unlike what it learned. Works well with images and sensor data, and PCA reconstruction error is a perfectly serviceable version that needs no neural network.

The contamination parameter

contamination tells scikit-learn what fraction of the data to treat as anomalous, and it sets the threshold. Setting it is the same decision as choosing a threshold in module 5, and it is frequently set at the default without thought.

If you have any labelled anomalies at all, use them to pick this by expected cost rather than guessing. If you have none, set it by capacity: how many alerts can somebody actually look at per day?

The arithmetic that sinks most deployments

This is the lesson’s real content, and it follows from the base rate discussion in module 5.

One million transactions a day. 100 are fraudulent. Your detector catches 90% of them and has a 1% false positive rate, which sounds excellent.

True positives  = 90
False positives = 0.01 * 999,900 = 9,999
Precision = 90 / 10,089 = 0.9%

Ten thousand alerts a day, of which 99.1% are wrong. No team can work that queue, and within two weeks they will ignore it entirely, which is worse than having no detector, because everyone believes the problem is covered.

To get to 50% precision you would need a false positive rate near 0.01%, a hundred times better. That is the actual requirement, and it should be stated before the project starts rather than discovered after.

What to do about it

Narrow the population. Do not score all transactions. Score transactions above ₹10,000 from accounts under 30 days old, where the base rate might be 2% instead of 0.01%. Precision rises by orders of magnitude for free, because the arithmetic changes.

Cascade. A cheap fast filter passes 1% of traffic to an expensive accurate model. Each stage only needs to be good enough not to discard true positives.

Rank, do not alert. Give the team the top 200 by score every morning. Now the metric is precision at 200 and the queue is finite by construction.

Aggregate. One anomalous transaction is weak evidence; five from the same account in an hour is strong. Alerting on entities rather than events raises the base rate of the thing you alert on, which is the same trick as narrowing the population.

And use labels the moment you have any

A supervised model beats an unsupervised one whenever labels exist, and it is not close. Anomaly detection is what you do while you have no labels, or for anomalies unlike anything you have seen. The moment your investigators have confirmed 200 cases, train a classifier on them, and keep the anomaly detector running alongside for the novel cases the classifier has no examples of. The two catch different things, and running both is normal.