When the output is a quantity

The same question in a different costume

Plenty of AI features output a number: predicted delivery time, forecast demand, an estimated price, a risk score in rupees, a count extracted from a document. The evaluation question is unchanged (what decision does this serve) but the metrics differ, and the differences are not cosmetic.

Start with five errors: 1, 1, 1, 1, 10 units.

MAE  = (1+1+1+1+10) / 5              = 2.8
RMSE = sqrt((1+1+1+1+100) / 5)       = sqrt(20.8) = 4.6

Mean absolute error is the average miss, in the units of the thing. Root mean squared error squares first, so the single large error dominates. That is the whole choice, and it is a business question rather than a statistical one:

• If ten small delays cost the same as one big delay, use MAE.
• If one delivery three hours late costs more than ten deliveries eighteen minutes late (because it triggers a refund, a call, a lost customer) use RMSE, which will steer the model away from large misses.

Quote MAE alongside anything else, because it is the only one a non-specialist reads correctly: “on average we are 2.8 minutes out” is a true sentence about MAE and a false one about RMSE.

Why MAPE keeps getting chosen and keeps causing trouble

Mean absolute percentage error is popular because it is unit-free and comparable across products. It has two mechanical defects that change decisions.

It explodes near zero. Actual 2, predicted 4: the absolute error is 2 and the percentage error is 100%. Actual 100, predicted 98: the same absolute error is 2%. On any series with small values (a slow-moving product, a quiet night) MAPE is dominated by items nobody cares about.

It is asymmetric. Over-forecasting can never exceed 100% error when the actual is fixed, while under-forecasting is unbounded. So a model trained or selected on MAPE learns to forecast low. This is not a subtlety; it shows up as systematic under-stocking in real inventory systems.

Use WAPE instead, which is the total absolute error divided by the total actual volume:

WAPE = sum(|actual - predicted|) / sum(actual)

It is unit-free, it does not explode, and it weights by size, which is usually what the business means by “how far off were we”.

Compare against the naive forecast, always

For anything with a time dimension, the baseline is not the mean. It is last value (“tomorrow equals today”) or seasonal naive (“this Tuesday equals last Tuesday”). Both are free and both are hard to beat.

MASE makes the comparison a single number: your MAE divided by the naive method’s MAE on the same data.

• MASE 0.8: you are 20% better than repeating last week.
• MASE 1.0: you have built a machine that repeats last week.
• MASE above 1.0: you would do better deleting the model.

R² carries the same idea for non-time-series regression: it compares your squared error against always predicting the mean, so a negative R² means you are worse than a constant. Both numbers exist to stop a team celebrating a forecast that a spreadsheet formula beats.

A point estimate is often the wrong output

Many decisions need a range, not a number: how much stock to hold, how much buffer to promise, whether to escalate. Then the model should emit an interval, and the evaluation is a coverage check.

Take the stated 90% interval and count how often the truth fell inside it. If it is 71%, your intervals are too narrow and every plan built on them is under-buffered. If it is 99%, they are too wide to be useful. This is exactly calibration from the previous lesson, applied to quantities, and quantile loss (also called pinball loss) is the training-time version of the same idea.

Report error by size, not once

Errors are almost never uniform. A 5% error on a ₹200 order and a 5% error on a ₹2,00,000 order are different businesses. Bucket by magnitude and print MAE and WAPE for each bucket. The usual finding is that the headline is carried by the many small items while the money sits in the few large ones, and that the model is worst exactly there, because large values are rare in training data.

Everything here runs in sklearn.metrics, in statsmodels, or in a spreadsheet with four columns. None of it needs a GPU, and the naive baseline needs no model at all.