Making a 0.8 mean eighty per cent

The moment it matters

Two systems score identically on ranking. One of them lets you write this line:

expected cost = p * cost_of_acting_wrongly + (1 - p) * cost_of_not_acting

The other does not, because its p is a number-shaped noise. Any decision built on expected cost, any threshold derived from money, any triage that says “route everything above 70% risk to a human” depends on the score meaning what it says. That property is calibration, and it is measured, not assumed.

Measure it with bins

Take a held-out labelled set, sort by predicted score, cut into ten bins, and compare each bin’s mean predicted score with the observed fraction of positives in it.

A typical neural network gives you something like this:

• bin 0.0 to 0.1: predicted 0.05, observed 0.04: near enough
• bin 0.5 to 0.6: predicted 0.55, observed 0.41: over-confident
• bin 0.8 to 0.9: predicted 0.85, observed 0.62: badly over-confident
• bin 0.9 to 1.0: predicted 0.96, observed 0.78: badly over-confident

Plotted, that is the reliability diagram, and modern classifiers usually bulge below the diagonal: they are more sure than they should be, because the training objective rewards pushing probabilities to the extremes.

Expected calibration error summarises it as the count-weighted mean of the gaps.

import numpy as np
def ece(p, y, bins=10):
    edges = np.linspace(0, 1, bins + 1)
    idx = np.clip(np.digitize(p, edges) - 1, 0, bins - 1)
    return sum(np.mean(idx == b) * abs(p[idx == b].mean() - y[idx == b].mean())
               for b in range(bins) if (idx == b).any())

An ECE of 0.03 means your probabilities are off by three points on average. An ECE of 0.15 means the word “probability” is doing no work.

Two proper scores complement it. Brier score is the mean squared error of the probabilities. Log loss is the mean of -ln(p_assigned_to_the_truth) and punishes confident mistakes savagely: being wrong at 0.9 costs -ln(0.1) = 2.30, being wrong at 0.6 costs -ln(0.4) = 0.92. If your product suffers most from confident errors, log loss is the metric that agrees with you.

Fixing it is cheap

Fit a small monotone function from raw score to calibrated probability on a held-out slice.

• Platt scaling fits a one-parameter logistic curve. It needs only a few hundred labelled points and is hard to overfit. It assumes the miscalibration has a sigmoid shape, which it usually does.
• Isotonic regression fits any non-decreasing step function. It corrects shapes Platt cannot, and it needs a few thousand points or it memorises the calibration set.
• Temperature scaling is Platt’s single-parameter cousin applied to logits, standard for neural networks: divide the logits by one learned number.

from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(model, method="sigmoid", cv="prefit").fit(X_cal, y_cal)

Here is the property that makes this safe and also limits it: all three are monotone, so they change no rankings at all. AUC, average precision, and every recall-at-precision figure stay exactly where they were. Calibration makes the number usable; it cannot make a weak model strong, and a perfectly calibrated model that always outputs the base rate is perfectly calibrated and perfectly useless.

Two ways calibration breaks after you fix it

It is not uniform across groups. A model calibrated overall can be over-confident for one language and under-confident for another, and the two errors cancel in the average. Compute ECE per slice, not once. This is the same arithmetic that makes group fairness hard, and it appears again later in the course.

It moves when the base rate moves. Calibrate at 4% prevalence, deploy where prevalence is 12%, and every probability is wrong. If only the prevalence has shifted, correct the odds directly rather than refitting:

new_odds = old_odds * (0.12 / 0.88) / (0.04 / 0.96)

Prevalence drifts with season, with a marketing campaign, with a new market. Re-check calibration on fresh labelled data on a schedule, the same way you re-check the model.

The special case of language models

A model asked “how confident are you, 0 to 100” produces text, not a probability, and that text clusters around 80 to 95 almost regardless of correctness: the phrasing of confident answers is what the training data rewarded. Where the provider exposes token log-probabilities for the answer tokens, those can be calibrated with the tools above and behave sensibly. Where it does not, treat any stated confidence as a style choice, and get your uncertainty from agreement across several samples instead.