What a curve says, and what its one number leaves out

AUC is a statement about pairs

Everybody quotes ROC-AUC and few people can say what it means. It has an exact and useful interpretation: pick one positive item and one negative item at random; AUC is the probability that the model gives the positive the higher score.

You can compute it by counting. Suppose three positives score 0.9, 0.6, 0.4, and four negatives score 0.8, 0.5, 0.3, 0.1. There are 3 × 4 = 12 pairs. Count the pairs where the positive is higher:

• 0.9 beats all four.
• 0.6 beats 0.5, 0.3, 0.1: three.
• 0.4 beats 0.3, 0.1: two.

Nine of twelve, so AUC = 0.75. No curve required. This is the Wilcoxon–Mann–Whitney statistic, and knowing it is the same thing as AUC explains everything else in this lesson.

Three consequences that follow immediately

AUC ignores calibration. Pass every score through any function that preserves order (square them, take logs, add 0.4), and every pair comparison is unchanged, so AUC is unchanged. A model with beautiful AUC can emit scores that mean nothing as probabilities. That is a separate property, measured separately, in the next lesson.

AUC averages over thresholds you will never use. The curve includes the region where you flag 80% of all traffic. If nobody would ever run there, the model’s skill in that region is being counted towards the number you are judging it by. Two models can tie at 0.86 while one is better where you actually operate and worse everywhere else.

AUC is insensitive to the positive rate. The false-positive-rate axis has all the negatives in its denominator, so with 4% positives an extra hundred false alarms moves the curve imperceptibly while quintupling your review queue. This is why average precision (the area under the precision–recall curve) is the honest summary for rare positives.

Curves cross, and the crossing is the decision

Plot both models on the same precision–recall axes. It is common for model A to hold higher precision in the high-recall region while model B is better when you only want the top few per cent. A single scalar cannot represent that, and whichever scalar you quote will pick a winner by accident.

So quote the number that matches your operating point:

• Recall at fixed precision. “At 90% precision, A catches 41% of cases and B catches 33%.” This is the right framing when a false positive has a fixed unacceptable cost.
• Precision at a fixed budget. “Reviewing 200 items a day, A’s queue is 62% real and B’s is 51%.” This is the right framing whenever a human queue exists, because the queue length is the real constraint.
• False positive rate at a fixed recall. Common in fraud and safety, where the recall target is set by policy.

from sklearn.metrics import precision_recall_curve
p, r, t = precision_recall_curve(y_true, scores)
i = (p >= 0.90).argmax()          # first point meeting the precision target
print(f"recall {r[i]:.2f} at precision {p[i]:.2f}, threshold {t[i]:.3f}")

Four lines, free, and it produces a sentence a product manager can act on rather than a number only you can interpret.

The uncertainty nobody prints

A curve drawn from 40 positives is mostly noise, and average precision computed on it swings several points between resamples. The mechanism is direct: precision at the top of the ranking is computed from a handful of items, so one item moving changes it a lot.

Bootstrap the whole curve: resample items with replacement, recompute the metric, repeat 1,000 times, take the 2.5th and 97.5th percentiles. That is the next module’s tool, and this is the first place you need it. If the two models’ intervals overlap heavily, you do not yet have a result, however clean the plot looks.

Plot it once, even if never again

Print the curve for your own system at least once. Three shapes are diagnostic and instantly readable:

• A curve hugging the top-left means a threshold exists that gives you both precision and recall. Take it.
• A curve that falls off a cliff at 20% recall means the model is confident about a small easy subset and guessing thereafter: often a sign that an easy pattern dominates the training data.
• A curve near the diagonal means no threshold will save you, and no amount of threshold tuning is worth doing until the model changes.

Ten minutes with matplotlib settles which of those three situations you are in, and each has a different next action.