ROC and precision-recall: every threshold at once

Two curves, two questions

Rather than pick one threshold, plot the model’s behaviour at all of them.

The ROC curve plots true positive rate (recall) against false positive rate. It sweeps from the top-right corner (flag everything, catch everything, alarm on everything) to the bottom-left, where you flag nothing. A diagonal line is a model with no information.

The precision-recall curve plots precision against recall over the same sweep. Its baseline is not a diagonal but a horizontal line at the positive class’s prevalence, which is what a random model achieves.

from sklearn.metrics import roc_auc_score, average_precision_score
roc_auc_score(y_val, p)
average_precision_score(y_val, p)

What AUC actually means

The area under the ROC curve has an interpretation that is far more useful than “area under a curve”:

AUC is the probability that a randomly chosen positive case gets a higher score than a randomly chosen negative case.

So 0.5 is a coin flip, 1.0 is perfect ranking, and 0.83 means that if you pick one fraud and one legitimate transaction at random, the fraud scores higher 83% of the time.

That framing tells you what AUC is for and what it is not. It is entirely about ranking, so it is unchanged if you apply any monotonic transformation to the scores: a model whose probabilities are all badly wrong can still have an AUC of 0.95, as long as the order is right. AUC says nothing about calibration, and it does not depend on a threshold, which is both its convenience and its limitation.

Why AUC misleads on rare positives

The false positive rate divides by the number of negatives. When negatives outnumber positives 1,000 to 1, a large absolute number of false positives is a small false positive rate, and the ROC curve barely notices.

Concretely: 100,000 transactions, 100 fraudulent. Your model flags 900 legitimate transactions as fraud. That is a false positive rate of 0.9%: the ROC curve looks excellent. It is also 900 investigations for at most 100 real cases, so precision is under 10% and the fraud team’s day is ruined.

The precision-recall curve shows this immediately, because precision divides by the number flagged rather than by the number of negatives. For imbalanced problems, report average precision. Report AUC too if you like, but do not let it be the headline.

A useful reference point: on a dataset with 1% positives, average precision of 0.30 is a strong model: it is thirty times the random baseline of 0.01, even though 0.30 reads like a bad number to anyone used to accuracy.

Reading the shape, not just the area

Two models can have identical AUC and completely different curves. One is excellent at the high-score end and mediocre elsewhere; the other is uniformly average. If you are going to operate at high precision (investigating the top 200) the first is much better and the summary number cannot tell them apart.

So look at the curve in the region you will actually operate in. A model chosen by AUC alone is a model chosen by its average behaviour across thresholds you will never use.

Partial AUC and top-k metrics

When the operating region is known, measure it directly.

• Partial AUC restricts the area to a range of false positive rates, say 0 to 0.1.
• Precision at k: of the top 200 by score, how many are positive. This is the metric for any queue with fixed capacity.
• Recall at k: of all positives, how many appear in the top 200. This is the same measurement viewed from the other side, and it is the one to quote when the question is coverage.

These are less standard and far more decision-relevant than either curve’s area.

The comparison trap

AUC computed on two different datasets is not comparable, and this catches people constantly. A model scoring 0.91 on one population and 0.84 on another has not necessarily got worse; the second population may be harder, or have a different mix of easy and hard cases. Comparisons are only meaningful on the same evaluation set.

The same applies across time. A model whose AUC drops from 0.88 to 0.85 between quarters may be degrading, or the quarter may have contained a different mix of customers. Investigate before concluding, and keep a fixed reference set that you rescore each period precisely so that you have one comparison that is like-for-like.