{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"reading-the-curve","title":"What a curve says, and what its one number leaves out","url":"https://addaly.com/learn/evaluating-ai/reading-the-curve","position":14,"of":76,"minutes":9,"answer":"AUC is the probability that a random positive outranks a random negative, so it measures ranking across thresholds you would never run: quote recall at a fixed precision, or precision at your real review budget.","keyIdea":"AUC is the probability that a random positive outranks a random negative, so it measures ranking across thresholds you would never run: quote recall at a fixed precision, or precision at your real review budget, instead."},"body":{"format":"markdown","text":"## AUC is a statement about pairs\n\nEverybody quotes ROC-AUC and few people can say what it means. It has an exact and useful interpretation: **pick one positive item and one negative item at random; AUC is the probability that the model gives the positive the higher score.**\n\nYou can compute it by counting. Suppose three positives score 0.9, 0.6, 0.4, and four negatives score 0.8, 0.5, 0.3, 0.1. There are 3 × 4 = 12 pairs. Count the pairs where the positive is higher:\n\n- 0.9 beats all four.\n- 0.6 beats 0.5, 0.3, 0.1: three.\n- 0.4 beats 0.3, 0.1: two.\n\nNine of twelve, so AUC = 0.75. No curve required. This is the Wilcoxon–Mann–Whitney statistic, and knowing it is the same thing as AUC explains everything else in this lesson.\n\n## Three consequences that follow immediately\n\n**AUC ignores calibration.** Pass every score through any function that preserves order (square them, take logs, add 0.4), and every pair comparison is unchanged, so AUC is unchanged. A model with beautiful AUC can emit scores that mean nothing as probabilities. That is a separate property, measured separately, in the next lesson.\n\n**AUC averages over thresholds you will never use.** The curve includes the region where you flag 80% of all traffic. If nobody would ever run there, the model's skill in that region is being counted towards the number you are judging it by. Two models can tie at 0.86 while one is better where you actually operate and worse everywhere else.\n\n**AUC is insensitive to the positive rate.** The false-positive-rate axis has all the negatives in its denominator, so with 4% positives an extra hundred false alarms moves the curve imperceptibly while quintupling your review queue. This is why average precision (the area under the precision–recall curve) is the honest summary for rare positives.\n\n## Curves cross, and the crossing is the decision\n\nPlot both models on the same precision–recall axes. It is common for model A to hold higher precision in the high-recall region while model B is better when you only want the top few per cent. A single scalar cannot represent that, and whichever scalar you quote will pick a winner by accident.\n\nSo quote the number that matches your operating point:\n\n- **Recall at fixed precision.** \"At 90% precision, A catches 41% of cases and B catches 33%.\" This is the right framing when a false positive has a fixed unacceptable cost.\n- **Precision at a fixed budget.** \"Reviewing 200 items a day, A's queue is 62% real and B's is 51%.\" This is the right framing whenever a human queue exists, because the queue length is the real constraint.\n- **False positive rate at a fixed recall.** Common in fraud and safety, where the recall target is set by policy.\n\n```python\nfrom sklearn.metrics import precision_recall_curve\np, r, t = precision_recall_curve(y_true, scores)\ni = (p >= 0.90).argmax()          # first point meeting the precision target\nprint(f\"recall {r[i]:.2f} at precision {p[i]:.2f}, threshold {t[i]:.3f}\")\n```\n\nFour lines, free, and it produces a sentence a product manager can act on rather than a number only you can interpret.\n\n## The uncertainty nobody prints\n\nA curve drawn from 40 positives is mostly noise, and average precision computed on it swings several points between resamples. The mechanism is direct: precision at the top of the ranking is computed from a handful of items, so one item moving changes it a lot.\n\nBootstrap the whole curve: resample items with replacement, recompute the metric, repeat 1,000 times, take the 2.5th and 97.5th percentiles. That is the next module's tool, and this is the first place you need it. If the two models' intervals overlap heavily, you do not yet have a result, however clean the plot looks.\n\n## Plot it once, even if never again\n\nPrint the curve for your own system at least once. Three shapes are diagnostic and instantly readable:\n\n- A curve hugging the top-left means a threshold exists that gives you both precision and recall. Take it.\n- A curve that falls off a cliff at 20% recall means the model is confident about a small easy subset and guessing thereafter: often a sign that an easy pattern dominates the training data.\n- A curve near the diagonal means no threshold will save you, and no amount of threshold tuning is worth doing until the model changes.\n\nTen minutes with `matplotlib` settles which of those three situations you are in, and each has a different next action."},"check":{"question":"Two spam classifiers both score ROC-AUC 0.91 on the same 5,000-message set with 3% spam. Which follow-up tells you most about which to ship?","options":[{"text":"Compute F1 for both at the default 0.5 threshold and pick the higher one.","correct":false,"tempting":"Reads a single arbitrary threshold as the operating point; 0.5 has no special status, and the better model at 0.5 is often the worse model at the threshold you would actually run."},{"text":"Compare precision at the volume your review queue can actually absorb, and plot both precision-recall curves to see where they cross.","correct":true},{"text":"Compare their accuracies, since AUC ignores the base rate and accuracy does not.","correct":false,"tempting":"Reaches for the metric that class imbalance ruins: at 3% spam, a model flagging nothing scores 97%."},{"text":"Prefer the model with better-calibrated probabilities, since equal AUC means equal ranking quality.","correct":false,"tempting":"Calibration is worth knowing but does not follow from AUC, and equal AUC does not mean equal ranking at the top of the list, which is the region that decides the product."}]},"prev":{"slug":"multi-label-and-partial-credit","title":"When an item has several right answers","url":"https://addaly.com/learn/evaluating-ai/multi-label-and-partial-credit"},"next":{"slug":"thresholds-and-the-cost-of-being-wrong","title":"Moving the threshold is a business decision","url":"https://addaly.com/learn/evaluating-ai/thresholds-and-the-cost-of-being-wrong"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What a curve says, and what its one number leaves out', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/reading-the-curve (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What a curve says, and what its one number leaves out","courseTitle":"Knowing If It Works","module":{"index":2,"slug":"metrics-that-mean-something","title":"Metrics that mean something"},"answer":null,"keyIdea":"AUC is the probability that a random positive outranks a random negative, so it measures ranking across thresholds you would never run: quote recall at a fixed precision, or precision at your real review budget, instead.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"auc-is-a-statement-about-pairs","runs":[{"text":"AUC is a statement about pairs"}]},{"t":"p","runs":[{"text":"Everybody quotes ROC-AUC and few people can say what it means. It has an exact and useful interpretation: "},{"text":"pick one positive item and one negative item at random; AUC is the probability that the model gives the positive the higher score.","b":true}]},{"t":"p","runs":[{"text":"You can compute it by counting. Suppose three positives score 0.9, 0.6, 0.4, and four negatives score 0.8, 0.5, 0.3, 0.1. There are 3 × 4 = 12 pairs. Count the pairs where the positive is higher:"}]},{"t":"ul","items":[[{"text":"0.9 beats all four."}],[{"text":"0.6 beats 0.5, 0.3, 0.1: three."}],[{"text":"0.4 beats 0.3, 0.1: two."}]]},{"t":"p","runs":[{"text":"Nine of twelve, so AUC = 0.75. No curve required. This is the Wilcoxon–Mann–Whitney statistic, and knowing it is the same thing as AUC explains everything else in this lesson."}]},{"t":"h2","id":"three-consequences-that-follow-immediately","runs":[{"text":"Three consequences that follow immediately"}]},{"t":"p","runs":[{"text":"AUC ignores calibration.","b":true},{"text":" Pass every score through any function that preserves order (square them, take logs, add 0.4), and every pair comparison is unchanged, so AUC is unchanged. A model with beautiful AUC can emit scores that mean nothing as probabilities. That is a separate property, measured separately, in the next lesson."}]},{"t":"p","runs":[{"text":"AUC averages over thresholds you will never use.","b":true},{"text":" The curve includes the region where you flag 80% of all traffic. If nobody would ever run there, the model’s skill in that region is being counted towards the number you are judging it by. Two models can tie at 0.86 while one is better where you actually operate and worse everywhere else."}]},{"t":"p","runs":[{"text":"AUC is insensitive to the positive rate.","b":true},{"text":" The false-positive-rate axis has all the negatives in its denominator, so with 4% positives an extra hundred false alarms moves the curve imperceptibly while quintupling your review queue. This is why average precision (the area under the precision–recall curve) is the honest summary for rare positives."}]},{"t":"h2","id":"curves-cross-and-the-crossing-is-the-decision","runs":[{"text":"Curves cross, and the crossing is the decision"}]},{"t":"p","runs":[{"text":"Plot both models on the same precision–recall axes. It is common for model A to hold higher precision in the high-recall region while model B is better when you only want the top few per cent. A single scalar cannot represent that, and whichever scalar you quote will pick a winner by accident."}]},{"t":"p","runs":[{"text":"So quote the number that matches your operating point:"}]},{"t":"ul","items":[[{"text":"Recall at fixed precision.","b":true},{"text":" “At 90% precision, A catches 41% of cases and B catches 33%.” This is the right framing when a false positive has a fixed unacceptable cost."}],[{"text":"Precision at a fixed budget.","b":true},{"text":" “Reviewing 200 items a day, A’s queue is 62% real and B’s is 51%.” This is the right framing whenever a human queue exists, because the queue length is the real constraint."}],[{"text":"False positive rate at a fixed recall.","b":true},{"text":" Common in fraud and safety, where the recall target is set by policy."}]]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.metrics import precision_recall_curve\np, r, t = precision_recall_curve(y_true, scores)\ni = (p >= 0.90).argmax()          # first point meeting the precision target\nprint(f\"recall {r[i]:.2f} at precision {p[i]:.2f}, threshold {t[i]:.3f}\")"},{"t":"p","runs":[{"text":"Four lines, free, and it produces a sentence a product manager can act on rather than a number only you can interpret."}]},{"t":"h2","id":"the-uncertainty-nobody-prints","runs":[{"text":"The uncertainty nobody prints"}]},{"t":"p","runs":[{"text":"A curve drawn from 40 positives is mostly noise, and average precision computed on it swings several points between resamples. The mechanism is direct: precision at the top of the ranking is computed from a handful of items, so one item moving changes it a lot."}]},{"t":"p","runs":[{"text":"Bootstrap the whole curve: resample items with replacement, recompute the metric, repeat 1,000 times, take the 2.5th and 97.5th percentiles. That is the next module’s tool, and this is the first place you need it. If the two models’ intervals overlap heavily, you do not yet have a result, however clean the plot looks."}]},{"t":"h2","id":"plot-it-once-even-if-never-again","runs":[{"text":"Plot it once, even if never again"}]},{"t":"p","runs":[{"text":"Print the curve for your own system at least once. Three shapes are diagnostic and instantly readable:"}]},{"t":"ul","items":[[{"text":"A curve hugging the top-left means a threshold exists that gives you both precision and recall. Take it."}],[{"text":"A curve that falls off a cliff at 20% recall means the model is confident about a small easy subset and guessing thereafter: often a sign that an easy pattern dominates the training data."}],[{"text":"A curve near the diagonal means no threshold will save you, and no amount of threshold tuning is worth doing until the model changes."}]]},{"t":"p","runs":[{"text":"Ten minutes with "},{"text":"matplotlib","code":true},{"text":" settles which of those three situations you are in, and each has a different next action."}]}],"check":{"question":[{"text":"Two spam classifiers both score ROC-AUC 0.91 on the same 5,000-message set with 3% spam. Which follow-up tells you most about which to ship?"}],"options":[{"text":[{"text":"Compute F1 for both at the default 0.5 threshold and pick the higher one."}],"correct":false,"tempting":[{"text":"Reads a single arbitrary threshold as the operating point; 0.5 has no special status, and the better model at 0.5 is often the worse model at the threshold you would actually run."}]},{"text":[{"text":"Compare precision at the volume your review queue can actually absorb, and plot both precision-recall curves to see where they cross."}],"correct":true,"tempting":null},{"text":[{"text":"Compare their accuracies, since AUC ignores the base rate and accuracy does not."}],"correct":false,"tempting":[{"text":"Reaches for the metric that class imbalance ruins: at 3% spam, a model flagging nothing scores 97%."}]},{"text":[{"text":"Prefer the model with better-calibrated probabilities, since equal AUC means equal ranking quality."}],"correct":false,"tempting":[{"text":"Calibration is worth knowing but does not follow from AUC, and equal AUC does not mean equal ranking at the top of the list, which is the region that decides the product."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"multi-label-and-partial-credit","title":"When an item has several right answers","minutes":8},"next":{"slug":"thresholds-and-the-cost-of-being-wrong","title":"Moving the threshold is a business decision","minutes":10}}}