Does 0.7 mean seventy per cent?

The question nobody asks

Your model outputs 0.7 for a thousand customers. Do about 700 of them convert?

If yes, the model is calibrated and the number is a probability you can compute with. If no, if 400 convert, then 0.7 is a ranking score wearing a probability’s clothes, and every expected-value calculation built on it is wrong.

This matters whenever the output feeds arithmetic rather than a simple ordering: expected revenue, expected loss, an insurance premium, a threshold chosen by cost ratio, or any case where a human reads the number and forms a belief.

Measuring it

Bin the predictions and compare the mean predicted probability in each bin with the actual observed rate.

from sklearn.calibration import calibration_curve
prob_true, prob_pred = calibration_curve(y_val, p, n_bins=10,
                                         strategy="quantile")

Plot prob_true against prob_pred. Perfect calibration is the diagonal. Below it means over-confident: you predicted 0.8 and observed 0.6. Above it means under-confident.

Use strategy="quantile" rather than uniform bins. With uniform bins on an imbalanced problem, nine of the ten bins contain almost no data and the plot is mostly noise.

The summary number is the Brier score, the mean squared error of the probabilities. It combines calibration and discrimination into one figure, so a poor Brier score does not tell you which is at fault, but it is a reasonable single metric when you need one.

Which models are calibrated, and why

Logistic regression is well calibrated by construction. It minimises log loss, and log loss is what statisticians call a proper scoring rule: it is minimised exactly when the predicted probabilities equal the true ones. The model has no incentive to distort them.

Naive Bayes is badly calibrated. It assumes features are independent, which they are not, so it multiplies correlated evidence repeatedly and ends up pushing predictions towards 0 and 1. Its ranking can be perfectly good while its probabilities are almost all above 0.99 or below 0.01.

Random forests are under-confident at the extremes. A forest’s prediction is the fraction of trees voting positive, and for a case that is certain, getting all 500 trees to agree is unlikely, so predictions rarely reach 0 or 1. The output tends to bunch between about 0.1 and 0.9.

SVMs produce distances, not probabilities. Whatever predict_proba returns for an SVM comes from a calibration step bolted on afterwards.

Boosted trees are usually reasonable but drift with more trees. Training past the point of best log loss tends to push probabilities outward.

Deep networks are typically over-confident, and increasingly so as they get larger, a finding from Guo and colleagues in 2017 that has held up. Modern networks routinely predict 0.99 on cases they get wrong.

Fixing it

Calibration is a post-processing step, fitted on data the model did not train on.

from sklearn.calibration import CalibratedClassifierCV
cal = CalibratedClassifierCV(base_model, method="isotonic", cv=5)
cal.fit(X_train, y_train)

Platt scaling (method="sigmoid") fits a one-parameter logistic function mapping scores to probabilities. Two parameters, so it works on small calibration sets, and it can only apply an S-shaped correction.

Isotonic regression (method="isotonic") fits any non-decreasing function. More flexible, needs more data (a few thousand rows at least) and it overfits on small sets, producing a staircase that memorises the calibration data.

Rule of thumb: under about 1,000 calibration rows, use sigmoid; above, isotonic is usually better.

One property worth knowing: both methods are monotonic, so calibration cannot change the ranking and therefore cannot change AUC. It changes the numbers and leaves the order alone. If someone reports that calibration improved their AUC, something else changed too.

What breaks calibration in production

Class weighting and resampling. Both deliberately misrepresent the base rate to the model, and the probabilities come out shifted. This is the side effect flagged in module 3, and it is why threshold-moving is preferable when you need meaningful probabilities.

Base rate shift. A model calibrated when fraud was 1% is over-predicting when fraud falls to 0.4%. The ranking may be untouched; the probabilities are all wrong. This is the most common cause in practice and the fix is recalibration on recent data, not retraining.

Time. Calibration decays faster than discrimination. A model can keep its AUC for a year while its probabilities become unusable in three months. Monitor both separately; they fail on different schedules and for different reasons.