Metrics for a number, and what each one hides

Four metrics, four different claims

When the target is a number rather than a class, the metric choice is quieter but no less consequential.

MAE: mean absolute error. Average size of the error, in the target’s units. Interpretable directly: “we are off by 4,200 rupees on average”. Treats every rupee of error the same.

RMSE: root mean squared error. Also in the target’s units, but errors are squared before averaging, so large errors dominate. RMSE is always at least as large as MAE, and the gap between them tells you how uneven your errors are. RMSE 12,000 with MAE 4,200 means a few very large errors; RMSE 4,600 with MAE 4,200 means errors of consistent size. Reporting both is more informative than reporting either.

MAPE: mean absolute percentage error. Scale-free, so it compares across products or regions. Two serious defects: it is undefined when the true value is zero, and it is asymmetric: over-predicting is capped at 100% error while under-predicting is unbounded, so a model tuned on MAPE learns to under-predict systematically. Widely used in demand forecasting and widely criticised there for exactly this.

R²: the fraction of the target’s variance the model explains. 0 means you match the mean; 1 means perfect; negative means worse than predicting the mean, which happens more often than people expect on held-out data. Its weakness is that it depends on the variance of the evaluation set, so an R² of 0.9 on a varied population and 0.3 on a narrow one can come from the same model with the same absolute errors.

Choosing

Ask what an error costs.

• Cost proportional to the size of the error: MAE.
• Large errors disproportionately bad: RMSE.
• A 10% error means the same at every scale: MAPE, or better, fit on the log of the target and use RMSE there, which gives you the same intent without MAPE’s asymmetry.
• Explaining to a non-technical audience how much better than nothing you are: R², alongside the baseline.

And remember from module 1 that the metric and the training loss need not match. Train on Huber for robustness and report MAE; there is nothing wrong with that as long as you say so.

The metric hides the shape

Every one of these is an average, and averages conceal structure. Two diagnostics take a minute each and are worth more than the summary numbers.

Plot residuals against predictions. What you want is a formless cloud centred on zero. What you often get:

• A funnel widening to the right: errors grow with the prediction, which means a log transform on the target is likely to help.
• A curve: the model has systematic bias in a region, which means a missing non-linearity.
• A slope: the classic regression-to-the-mean pattern, where the model over-predicts low values and under-predicts high ones.

Plot the error distribution. A long tail on one side tells you the model fails asymmetrically, which matters enormously if the two directions have different costs. Under-predicting demand means a stockout; over-predicting means holding stock. Those are not the same problem, and no symmetric metric distinguishes them.

Quantile loss, for when you need a range

Sometimes a single number is the wrong output. “Delivery in 34 minutes” is a promise; “90% of deliveries arrive within 48 minutes” is a promise you can keep.

Quantile regression fits a chosen quantile rather than the mean, by using an asymmetric loss that penalises under-prediction and over-prediction differently.

from sklearn.ensemble import GradientBoostingRegressor
GradientBoostingRegressor(loss="quantile", alpha=0.9)

Fit three models at 0.1, 0.5 and 0.9 and you have a median with an interval around it. For anything where a customer is given a commitment, this is usually more useful than a point estimate, and it is the honest way to express what the model does and does not know.

Two mistakes to avoid

Comparing RMSE across different target scales. RMSE of 400 on house prices and 400 on delivery minutes are not comparable numbers. Normalise, or use R², or report against a baseline.

Averaging percentage errors across items of wildly different size. One tiny item with a large percentage error can dominate a MAPE computed over a thousand products. Weight by volume or value if the business cares about total error rather than average per-item error, and say which you did.