{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"probability-calibration","title":"Does 0.7 mean seventy per cent?","url":"https://addaly.com/learn/machine-learning-foundations/probability-calibration","position":46,"of":92,"minutes":9,"answer":"Calibration asks whether a predicted 0.7 corresponds to 70% of cases occurring, is separate from ranking quality, and is fixed by a monotonic post-processing step that leaves AUC unchanged.","keyIdea":"Calibration asks whether a predicted 0.7 corresponds to 70% of cases occurring, is separate from ranking quality, and is fixed by a monotonic post-processing step that leaves AUC unchanged."},"body":{"format":"markdown","text":"## The question nobody asks\n\nYour model outputs 0.7 for a thousand customers. Do about 700 of them convert?\n\nIf yes, the model is **calibrated** and the number is a probability you can compute with. If no, if 400 convert, then 0.7 is a ranking score wearing a probability's clothes, and every expected-value calculation built on it is wrong.\n\nThis matters whenever the output feeds arithmetic rather than a simple ordering: expected revenue, expected loss, an insurance premium, a threshold chosen by cost ratio, or any case where a human reads the number and forms a belief.\n\n## Measuring it\n\nBin the predictions and compare the mean predicted probability in each bin with the actual observed rate.\n\n```python\nfrom sklearn.calibration import calibration_curve\nprob_true, prob_pred = calibration_curve(y_val, p, n_bins=10,\n                                         strategy=\"quantile\")\n```\n\nPlot `prob_true` against `prob_pred`. Perfect calibration is the diagonal. Below it means over-confident: you predicted 0.8 and observed 0.6. Above it means under-confident.\n\n> Figure: Reliability diagram for a 500-tree random forest. Line chart of Observed fraction of positives against Mean predicted probability in the bin: Random forest (0.1, 0.03) (0.3, 0.22) (0.5, 0.5) (0.7, 0.78) (0.9, 0.97); Perfect calibration (0, 0) (1, 1). The forest rarely reaches 0 or 1, because all 500 trees seldom agree, so its curve is steeper than the diagonal: a predicted 0.9 comes true 97 per cent of the time and a predicted 0.1 only 3 per cent. Platt or isotonic scaling straightens the curve without changing a single ranking.\n\nUse `strategy=\"quantile\"` rather than uniform bins. With uniform bins on an imbalanced problem, nine of the ten bins contain almost no data and the plot is mostly noise.\n\nThe summary number is the **Brier score**, the mean squared error of the probabilities. It combines calibration and discrimination into one figure, so a poor Brier score does not tell you which is at fault, but it is a reasonable single metric when you need one.\n\n## Which models are calibrated, and why\n\n**Logistic regression is well calibrated by construction.** It minimises log loss, and log loss is what statisticians call a proper scoring rule: it is minimised exactly when the predicted probabilities equal the true ones. The model has no incentive to distort them.\n\n**Naive Bayes is badly calibrated.** It assumes features are independent, which they are not, so it multiplies correlated evidence repeatedly and ends up pushing predictions towards 0 and 1. Its ranking can be perfectly good while its probabilities are almost all above 0.99 or below 0.01.\n\n**Random forests are under-confident at the extremes.** A forest's prediction is the fraction of trees voting positive, and for a case that is certain, getting all 500 trees to agree is unlikely, so predictions rarely reach 0 or 1. The output tends to bunch between about 0.1 and 0.9.\n\n**SVMs produce distances, not probabilities.** Whatever `predict_proba` returns for an SVM comes from a calibration step bolted on afterwards.\n\n**Boosted trees are usually reasonable but drift with more trees.** Training past the point of best log loss tends to push probabilities outward.\n\n**Deep networks are typically over-confident**, and increasingly so as they get larger, a finding from Guo and colleagues in 2017 that has held up. Modern networks routinely predict 0.99 on cases they get wrong.\n\n## Fixing it\n\nCalibration is a post-processing step, fitted on data the model did not train on.\n\n```python\nfrom sklearn.calibration import CalibratedClassifierCV\ncal = CalibratedClassifierCV(base_model, method=\"isotonic\", cv=5)\ncal.fit(X_train, y_train)\n```\n\n**Platt scaling** (`method=\"sigmoid\"`) fits a one-parameter logistic function mapping scores to probabilities. Two parameters, so it works on small calibration sets, and it can only apply an S-shaped correction.\n\n**Isotonic regression** (`method=\"isotonic\"`) fits any non-decreasing function. More flexible, needs more data (a few thousand rows at least) and it overfits on small sets, producing a staircase that memorises the calibration data.\n\nRule of thumb: under about 1,000 calibration rows, use sigmoid; above, isotonic is usually better.\n\nOne property worth knowing: both methods are monotonic, so **calibration cannot change the ranking and therefore cannot change AUC.** It changes the numbers and leaves the order alone. If someone reports that calibration improved their AUC, something else changed too.\n\n## What breaks calibration in production\n\n**Class weighting and resampling.** Both deliberately misrepresent the base rate to the model, and the probabilities come out shifted. This is the side effect flagged in module 3, and it is why threshold-moving is preferable when you need meaningful probabilities.\n\n**Base rate shift.** A model calibrated when fraud was 1% is over-predicting when fraud falls to 0.4%. The ranking may be untouched; the probabilities are all wrong. This is the most common cause in practice and the fix is recalibration on recent data, not retraining.\n\n**Time.** Calibration decays faster than discrimination. A model can keep its AUC for a year while its probabilities become unusable in three months. Monitor both separately; they fail on different schedules and for different reasons."},"check":{"question":"A random forest has excellent AUC but its predicted probabilities cluster between 0.15 and 0.85, never reaching the extremes. An analyst multiplies all probabilities by 1.4 to spread them out. What is wrong with that fix?","options":[{"text":"Multiplying is monotonic, so it cannot change the AUC, which means the fix does nothing at all.","correct":false,"tempting":"Correctly notes the transformation is monotonic, but the goal was never to change AUC: it was to fix the probabilities, and the problem with the fix lies elsewhere."},{"text":"It pushes values above 1 and applies one linear stretch everywhere, whereas the miscalibration is a non-linear compression that needs a fitted mapping such as isotonic regression on held-out data.","correct":true},{"text":"It would work, but only if applied to the log-odds rather than the probabilities.","correct":false,"tempting":"Working on the log-odds scale is a sensible instinct, but a single fixed multiplier is still an unfitted guess at a correction that should be estimated from data."},{"text":"Random forest probabilities cannot be calibrated at all, because they are vote fractions rather than probability estimates.","correct":false,"tempting":"Vote fractions are exactly the kind of score that post-hoc calibration handles well; forests calibrate readily."}]},"prev":{"slug":"roc-and-pr-curves","title":"ROC and precision-recall: every threshold at once","url":"https://addaly.com/learn/machine-learning-foundations/roc-and-pr-curves"},"next":{"slug":"regression-metrics","title":"Metrics for a number, and what each one hides","url":"https://addaly.com/learn/machine-learning-foundations/regression-metrics"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Does 0.7 mean seventy per cent?', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/probability-calibration (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Does 0.7 mean seventy per cent?","courseTitle":"Machine Learning, Foundations","module":{"index":5,"slug":"measuring-honestly","title":"Measuring a model honestly"},"answer":null,"keyIdea":"Calibration asks whether a predicted 0.7 corresponds to 70% of cases occurring, is separate from ranking quality, and is fixed by a monotonic post-processing step that leaves AUC unchanged.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Reliability diagram for a 500-tree random forest","series":[{"label":"Random forest","points":[[0.1,0.03],[0.3,0.22],[0.5,0.5],[0.7,0.78],[0.9,0.97]]},{"label":"Perfect calibration","points":[[0,0],[1,1]]}],"xLabel":"Mean predicted probability in the bin","yLabel":"Observed fraction of positives","caption":"The forest rarely reaches 0 or 1, because all 500 trees seldom agree, so its curve is steeper than the diagonal: a predicted 0.9 comes true 97 per cent of the time and a predicted 0.1 only 3 per cent. Platt or isotonic scaling straightens the curve without changing a single ranking."},"alt":"Figure: Reliability diagram for a 500-tree random forest. Line chart of Observed fraction of positives against Mean predicted probability in the bin: Random forest (0.1, 0.03) (0.3, 0.22) (0.5, 0.5) (0.7, 0.78) (0.9, 0.97); Perfect calibration (0, 0) (1, 1). The forest rarely reaches 0 or 1, because all 500 trees seldom agree, so its curve is steeper than the diagonal: a predicted 0.9 comes true 97 per cent of the time and a predicted 0.1 only 3 per cent. Platt or isotonic scaling straightens the curve without changing a single ranking."}},"blocks":[{"t":"h2","id":"the-question-nobody-asks","runs":[{"text":"The question nobody asks"}]},{"t":"p","runs":[{"text":"Your model outputs 0.7 for a thousand customers. Do about 700 of them convert?"}]},{"t":"p","runs":[{"text":"If yes, the model is "},{"text":"calibrated","b":true},{"text":" and the number is a probability you can compute with. If no, if 400 convert, then 0.7 is a ranking score wearing a probability’s clothes, and every expected-value calculation built on it is wrong."}]},{"t":"p","runs":[{"text":"This matters whenever the output feeds arithmetic rather than a simple ordering: expected revenue, expected loss, an insurance premium, a threshold chosen by cost ratio, or any case where a human reads the number and forms a belief."}]},{"t":"h2","id":"measuring-it","runs":[{"text":"Measuring it"}]},{"t":"p","runs":[{"text":"Bin the predictions and compare the mean predicted probability in each bin with the actual observed rate."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.calibration import calibration_curve\nprob_true, prob_pred = calibration_curve(y_val, p, n_bins=10,\n                                         strategy=\"quantile\")"},{"t":"p","runs":[{"text":"Plot "},{"text":"prob_true","code":true},{"text":" against "},{"text":"prob_pred","code":true},{"text":". Perfect calibration is the diagonal. Below it means over-confident: you predicted 0.8 and observed 0.6. Above it means under-confident."}]},{"t":"p","runs":[{"text":"Use "},{"text":"strategy=\"quantile\"","code":true},{"text":" rather than uniform bins. With uniform bins on an imbalanced problem, nine of the ten bins contain almost no data and the plot is mostly noise."}]},{"t":"p","runs":[{"text":"The summary number is the "},{"text":"Brier score","b":true},{"text":", the mean squared error of the probabilities. It combines calibration and discrimination into one figure, so a poor Brier score does not tell you which is at fault, but it is a reasonable single metric when you need one."}]},{"t":"h2","id":"which-models-are-calibrated-and-why","runs":[{"text":"Which models are calibrated, and why"}]},{"t":"p","runs":[{"text":"Logistic regression is well calibrated by construction.","b":true},{"text":" It minimises log loss, and log loss is what statisticians call a proper scoring rule: it is minimised exactly when the predicted probabilities equal the true ones. The model has no incentive to distort them."}]},{"t":"p","runs":[{"text":"Naive Bayes is badly calibrated.","b":true},{"text":" It assumes features are independent, which they are not, so it multiplies correlated evidence repeatedly and ends up pushing predictions towards 0 and 1. Its ranking can be perfectly good while its probabilities are almost all above 0.99 or below 0.01."}]},{"t":"p","runs":[{"text":"Random forests are under-confident at the extremes.","b":true},{"text":" A forest’s prediction is the fraction of trees voting positive, and for a case that is certain, getting all 500 trees to agree is unlikely, so predictions rarely reach 0 or 1. The output tends to bunch between about 0.1 and 0.9."}]},{"t":"p","runs":[{"text":"SVMs produce distances, not probabilities.","b":true},{"text":" Whatever "},{"text":"predict_proba","code":true},{"text":" returns for an SVM comes from a calibration step bolted on afterwards."}]},{"t":"p","runs":[{"text":"Boosted trees are usually reasonable but drift with more trees.","b":true},{"text":" Training past the point of best log loss tends to push probabilities outward."}]},{"t":"p","runs":[{"text":"Deep networks are typically over-confident","b":true},{"text":", and increasingly so as they get larger, a finding from Guo and colleagues in 2017 that has held up. Modern networks routinely predict 0.99 on cases they get wrong."}]},{"t":"h2","id":"fixing-it","runs":[{"text":"Fixing it"}]},{"t":"p","runs":[{"text":"Calibration is a post-processing step, fitted on data the model did not train on."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.calibration import CalibratedClassifierCV\ncal = CalibratedClassifierCV(base_model, method=\"isotonic\", cv=5)\ncal.fit(X_train, y_train)"},{"t":"p","runs":[{"text":"Platt scaling","b":true},{"text":" ("},{"text":"method=\"sigmoid\"","code":true},{"text":") fits a one-parameter logistic function mapping scores to probabilities. Two parameters, so it works on small calibration sets, and it can only apply an S-shaped correction."}]},{"t":"p","runs":[{"text":"Isotonic regression","b":true},{"text":" ("},{"text":"method=\"isotonic\"","code":true},{"text":") fits any non-decreasing function. More flexible, needs more data (a few thousand rows at least) and it overfits on small sets, producing a staircase that memorises the calibration data."}]},{"t":"p","runs":[{"text":"Rule of thumb: under about 1,000 calibration rows, use sigmoid; above, isotonic is usually better."}]},{"t":"p","runs":[{"text":"One property worth knowing: both methods are monotonic, so "},{"text":"calibration cannot change the ranking and therefore cannot change AUC.","b":true},{"text":" It changes the numbers and leaves the order alone. If someone reports that calibration improved their AUC, something else changed too."}]},{"t":"h2","id":"what-breaks-calibration-in-production","runs":[{"text":"What breaks calibration in production"}]},{"t":"p","runs":[{"text":"Class weighting and resampling.","b":true},{"text":" Both deliberately misrepresent the base rate to the model, and the probabilities come out shifted. This is the side effect flagged in module 3, and it is why threshold-moving is preferable when you need meaningful probabilities."}]},{"t":"p","runs":[{"text":"Base rate shift.","b":true},{"text":" A model calibrated when fraud was 1% is over-predicting when fraud falls to 0.4%. The ranking may be untouched; the probabilities are all wrong. This is the most common cause in practice and the fix is recalibration on recent data, not retraining."}]},{"t":"p","runs":[{"text":"Time.","b":true},{"text":" Calibration decays faster than discrimination. A model can keep its AUC for a year while its probabilities become unusable in three months. Monitor both separately; they fail on different schedules and for different reasons."}]}],"check":{"question":[{"text":"A random forest has excellent AUC but its predicted probabilities cluster between 0.15 and 0.85, never reaching the extremes. An analyst multiplies all probabilities by 1.4 to spread them out. What is wrong with that fix?"}],"options":[{"text":[{"text":"Multiplying is monotonic, so it cannot change the AUC, which means the fix does nothing at all."}],"correct":false,"tempting":[{"text":"Correctly notes the transformation is monotonic, but the goal was never to change AUC: it was to fix the probabilities, and the problem with the fix lies elsewhere."}]},{"text":[{"text":"It pushes values above 1 and applies one linear stretch everywhere, whereas the miscalibration is a non-linear compression that needs a fitted mapping such as isotonic regression on held-out data."}],"correct":true,"tempting":null},{"text":[{"text":"It would work, but only if applied to the log-odds rather than the probabilities."}],"correct":false,"tempting":[{"text":"Working on the log-odds scale is a sensible instinct, but a single fixed multiplier is still an unfitted guess at a correction that should be estimated from data."}]},{"text":[{"text":"Random forest probabilities cannot be calibrated at all, because they are vote fractions rather than probability estimates."}],"correct":false,"tempting":[{"text":"Vote fractions are exactly the kind of score that post-hoc calibration handles well; forests calibrate readily."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"roc-and-pr-curves","title":"ROC and precision-recall: every threshold at once","minutes":9},"next":{"slug":"regression-metrics","title":"Metrics for a number, and what each one hides","minutes":8}}}