{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"roc-and-pr-curves","title":"ROC and precision-recall: every threshold at once","url":"https://addaly.com/learn/machine-learning-foundations/roc-and-pr-curves","position":45,"of":92,"minutes":9,"answer":"AUC is the probability that a random positive outscores a random negative, so it measures ranking only and is insensitive to a large absolute number of false positives when negatives vastly outnumber positives.","keyIdea":"AUC is the probability that a random positive outscores a random negative, so it measures ranking only and is insensitive to a large absolute number of false positives when negatives vastly outnumber positives, which is why imbalanced problems should be reported with average precision."},"body":{"format":"markdown","text":"## Two curves, two questions\n\nRather than pick one threshold, plot the model's behaviour at all of them.\n\n**The ROC curve** plots true positive rate (recall) against false positive rate. It sweeps from the top-right corner (flag everything, catch everything, alarm on everything) to the bottom-left, where you flag nothing. A diagonal line is a model with no information.\n\n**The precision-recall curve** plots precision against recall over the same sweep. Its baseline is not a diagonal but a horizontal line at the positive class's prevalence, which is what a random model achieves.\n\n```python\nfrom sklearn.metrics import roc_auc_score, average_precision_score\nroc_auc_score(y_val, p)\naverage_precision_score(y_val, p)\n```\n\n## What AUC actually means\n\nThe area under the ROC curve has an interpretation that is far more useful than \"area under a curve\":\n\n> **AUC is the probability that a randomly chosen positive case gets a higher score than a randomly chosen negative case.**\n\nSo 0.5 is a coin flip, 1.0 is perfect ranking, and 0.83 means that if you pick one fraud and one legitimate transaction at random, the fraud scores higher 83% of the time.\n\nThat framing tells you what AUC is for and what it is not. It is entirely about **ranking**, so it is unchanged if you apply any monotonic transformation to the scores: a model whose probabilities are all badly wrong can still have an AUC of 0.95, as long as the order is right. AUC says nothing about calibration, and it does not depend on a threshold, which is both its convenience and its limitation.\n\n## Why AUC misleads on rare positives\n\nThe false positive rate divides by the number of negatives. When negatives outnumber positives 1,000 to 1, a large absolute number of false positives is a small false positive rate, and the ROC curve barely notices.\n\nConcretely: 100,000 transactions, 100 fraudulent. Your model flags 900 legitimate transactions as fraud. That is a false positive rate of 0.9%: the ROC curve looks excellent. It is also 900 investigations for at most 100 real cases, so precision is under 10% and the fraud team's day is ruined.\n\nThe precision-recall curve shows this immediately, because precision divides by the number flagged rather than by the number of negatives. **For imbalanced problems, report average precision.** Report AUC too if you like, but do not let it be the headline.\n\nA useful reference point: on a dataset with 1% positives, average precision of 0.30 is a strong model: it is thirty times the random baseline of 0.01, even though 0.30 reads like a bad number to anyone used to accuracy.\n\n## Reading the shape, not just the area\n\nTwo models can have identical AUC and completely different curves. One is excellent at the high-score end and mediocre elsewhere; the other is uniformly average. If you are going to operate at high precision (investigating the top 200) the first is much better and the summary number cannot tell them apart.\n\nSo look at the curve in the region you will actually operate in. A model chosen by AUC alone is a model chosen by its average behaviour across thresholds you will never use.\n\n## Partial AUC and top-k metrics\n\nWhen the operating region is known, measure it directly.\n\n- **Partial AUC** restricts the area to a range of false positive rates, say 0 to 0.1.\n- **Precision at k**: of the top 200 by score, how many are positive. This is the metric for any queue with fixed capacity.\n- **Recall at k**: of all positives, how many appear in the top 200. This is the same measurement viewed from the other side, and it is the one to quote when the question is coverage.\n\nThese are less standard and far more decision-relevant than either curve's area.\n\n## The comparison trap\n\nAUC computed on two different datasets is not comparable, and this catches people constantly. A model scoring 0.91 on one population and 0.84 on another has not necessarily got worse; the second population may be harder, or have a different mix of easy and hard cases. Comparisons are only meaningful on the same evaluation set.\n\nThe same applies across time. A model whose AUC drops from 0.88 to 0.85 between quarters may be degrading, or the quarter may have contained a different mix of customers. Investigate before concluding, and keep a fixed reference set that you rescore each period precisely so that you have one comparison that is like-for-like."},"check":{"question":"Two models have identical ROC AUC of 0.88 on a 1% positive dataset, but model A has average precision 0.34 and model B has 0.11. What does this tell you?","options":[{"text":"One of the two metrics has been computed incorrectly, since both summarise the same ranking.","correct":false,"tempting":"Both do summarise the same ranking, but they weight regions of it differently, so identical AUC with different average precision is entirely consistent."},{"text":"Model B is better calibrated, which is why its precision-based metric is lower.","correct":false,"tempting":"Neither metric measures calibration; both depend only on the ordering of scores."},{"text":"Model A ranks the true positives much higher within the top of the list, which average precision rewards and ROC AUC largely averages away across the vast negative class.","correct":true},{"text":"Model A must have been evaluated at a different threshold, producing the higher score.","correct":false,"tempting":"Both metrics are threshold-free summaries over the whole score range, so no threshold is involved in either."}]},"prev":{"slug":"choosing-the-threshold","title":"The threshold is a business decision, not a default","url":"https://addaly.com/learn/machine-learning-foundations/choosing-the-threshold"},"next":{"slug":"probability-calibration","title":"Does 0.7 mean seventy per cent?","url":"https://addaly.com/learn/machine-learning-foundations/probability-calibration"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'ROC and precision-recall: every threshold at once', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/roc-and-pr-curves (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"ROC and precision-recall: every threshold at once","courseTitle":"Machine Learning, Foundations","module":{"index":5,"slug":"measuring-honestly","title":"Measuring a model honestly"},"answer":null,"keyIdea":"AUC is the probability that a random positive outscores a random negative, so it measures ranking only and is insensitive to a large absolute number of false positives when negatives vastly outnumber positives, which is why imbalanced problems should be reported with average precision.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-curves-two-questions","runs":[{"text":"Two curves, two questions"}]},{"t":"p","runs":[{"text":"Rather than pick one threshold, plot the model’s behaviour at all of them."}]},{"t":"p","runs":[{"text":"The ROC curve","b":true},{"text":" plots true positive rate (recall) against false positive rate. It sweeps from the top-right corner (flag everything, catch everything, alarm on everything) to the bottom-left, where you flag nothing. A diagonal line is a model with no information."}]},{"t":"p","runs":[{"text":"The precision-recall curve","b":true},{"text":" plots precision against recall over the same sweep. Its baseline is not a diagonal but a horizontal line at the positive class’s prevalence, which is what a random model achieves."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.metrics import roc_auc_score, average_precision_score\nroc_auc_score(y_val, p)\naverage_precision_score(y_val, p)"},{"t":"h2","id":"what-auc-actually-means","runs":[{"text":"What AUC actually means"}]},{"t":"p","runs":[{"text":"The area under the ROC curve has an interpretation that is far more useful than “area under a curve”:"}]},{"t":"quote","runs":[{"text":"AUC is the probability that a randomly chosen positive case gets a higher score than a randomly chosen negative case.","b":true}]},{"t":"p","runs":[{"text":"So 0.5 is a coin flip, 1.0 is perfect ranking, and 0.83 means that if you pick one fraud and one legitimate transaction at random, the fraud scores higher 83% of the time."}]},{"t":"p","runs":[{"text":"That framing tells you what AUC is for and what it is not. It is entirely about "},{"text":"ranking","b":true},{"text":", so it is unchanged if you apply any monotonic transformation to the scores: a model whose probabilities are all badly wrong can still have an AUC of 0.95, as long as the order is right. AUC says nothing about calibration, and it does not depend on a threshold, which is both its convenience and its limitation."}]},{"t":"h2","id":"why-auc-misleads-on-rare-positives","runs":[{"text":"Why AUC misleads on rare positives"}]},{"t":"p","runs":[{"text":"The false positive rate divides by the number of negatives. When negatives outnumber positives 1,000 to 1, a large absolute number of false positives is a small false positive rate, and the ROC curve barely notices."}]},{"t":"p","runs":[{"text":"Concretely: 100,000 transactions, 100 fraudulent. Your model flags 900 legitimate transactions as fraud. That is a false positive rate of 0.9%: the ROC curve looks excellent. It is also 900 investigations for at most 100 real cases, so precision is under 10% and the fraud team’s day is ruined."}]},{"t":"p","runs":[{"text":"The precision-recall curve shows this immediately, because precision divides by the number flagged rather than by the number of negatives. "},{"text":"For imbalanced problems, report average precision.","b":true},{"text":" Report AUC too if you like, but do not let it be the headline."}]},{"t":"p","runs":[{"text":"A useful reference point: on a dataset with 1% positives, average precision of 0.30 is a strong model: it is thirty times the random baseline of 0.01, even though 0.30 reads like a bad number to anyone used to accuracy."}]},{"t":"h2","id":"reading-the-shape-not-just-the-area","runs":[{"text":"Reading the shape, not just the area"}]},{"t":"p","runs":[{"text":"Two models can have identical AUC and completely different curves. One is excellent at the high-score end and mediocre elsewhere; the other is uniformly average. If you are going to operate at high precision (investigating the top 200) the first is much better and the summary number cannot tell them apart."}]},{"t":"p","runs":[{"text":"So look at the curve in the region you will actually operate in. A model chosen by AUC alone is a model chosen by its average behaviour across thresholds you will never use."}]},{"t":"h2","id":"partial-auc-and-top-k-metrics","runs":[{"text":"Partial AUC and top-k metrics"}]},{"t":"p","runs":[{"text":"When the operating region is known, measure it directly."}]},{"t":"ul","items":[[{"text":"Partial AUC","b":true},{"text":" restricts the area to a range of false positive rates, say 0 to 0.1."}],[{"text":"Precision at k","b":true},{"text":": of the top 200 by score, how many are positive. This is the metric for any queue with fixed capacity."}],[{"text":"Recall at k","b":true},{"text":": of all positives, how many appear in the top 200. This is the same measurement viewed from the other side, and it is the one to quote when the question is coverage."}]]},{"t":"p","runs":[{"text":"These are less standard and far more decision-relevant than either curve’s area."}]},{"t":"h2","id":"the-comparison-trap","runs":[{"text":"The comparison trap"}]},{"t":"p","runs":[{"text":"AUC computed on two different datasets is not comparable, and this catches people constantly. A model scoring 0.91 on one population and 0.84 on another has not necessarily got worse; the second population may be harder, or have a different mix of easy and hard cases. Comparisons are only meaningful on the same evaluation set."}]},{"t":"p","runs":[{"text":"The same applies across time. A model whose AUC drops from 0.88 to 0.85 between quarters may be degrading, or the quarter may have contained a different mix of customers. Investigate before concluding, and keep a fixed reference set that you rescore each period precisely so that you have one comparison that is like-for-like."}]}],"check":{"question":[{"text":"Two models have identical ROC AUC of 0.88 on a 1% positive dataset, but model A has average precision 0.34 and model B has 0.11. What does this tell you?"}],"options":[{"text":[{"text":"One of the two metrics has been computed incorrectly, since both summarise the same ranking."}],"correct":false,"tempting":[{"text":"Both do summarise the same ranking, but they weight regions of it differently, so identical AUC with different average precision is entirely consistent."}]},{"text":[{"text":"Model B is better calibrated, which is why its precision-based metric is lower."}],"correct":false,"tempting":[{"text":"Neither metric measures calibration; both depend only on the ordering of scores."}]},{"text":[{"text":"Model A ranks the true positives much higher within the top of the list, which average precision rewards and ROC AUC largely averages away across the vast negative class."}],"correct":true,"tempting":null},{"text":[{"text":"Model A must have been evaluated at a different threshold, producing the higher score."}],"correct":false,"tempting":[{"text":"Both metrics are threshold-free summaries over the whole score range, so no threshold is involved in either."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"choosing-the-threshold","title":"The threshold is a business decision, not a default","minutes":9},"next":{"slug":"probability-calibration","title":"Does 0.7 mean seventy per cent?","minutes":9}}}