{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"metrics-for-numbers","title":"When the output is a quantity","url":"https://addaly.com/learn/evaluating-ai/metrics-for-numbers","position":17,"of":76,"minutes":9,"answer":"Choose between MAE and RMSE by deciding whether one large error is worse than several small ones, avoid MAPE because its asymmetry biases model selection towards under-forecasting, and score every forecast against the naive baseline.","keyIdea":"Choose between MAE and RMSE by deciding whether one large error is worse than several small ones, avoid MAPE because its asymmetry biases model selection towards under-forecasting, and score every forecast against the naive baseline."},"body":{"format":"markdown","text":"## The same question in a different costume\n\nPlenty of AI features output a number: predicted delivery time, forecast demand, an estimated price, a risk score in rupees, a count extracted from a document. The evaluation question is unchanged (what decision does this serve) but the metrics differ, and the differences are not cosmetic.\n\nStart with five errors: 1, 1, 1, 1, 10 units.\n\n```\nMAE  = (1+1+1+1+10) / 5              = 2.8\nRMSE = sqrt((1+1+1+1+100) / 5)       = sqrt(20.8) = 4.6\n```\n\n**Mean absolute error** is the average miss, in the units of the thing. **Root mean squared error** squares first, so the single large error dominates. That is the whole choice, and it is a business question rather than a statistical one:\n\n- If ten small delays cost the same as one big delay, use MAE.\n- If one delivery three hours late costs more than ten deliveries eighteen minutes late (because it triggers a refund, a call, a lost customer) use RMSE, which will steer the model away from large misses.\n\nQuote MAE alongside anything else, because it is the only one a non-specialist reads correctly: \"on average we are 2.8 minutes out\" is a true sentence about MAE and a false one about RMSE.\n\n## Why MAPE keeps getting chosen and keeps causing trouble\n\nMean absolute percentage error is popular because it is unit-free and comparable across products. It has two mechanical defects that change decisions.\n\n**It explodes near zero.** Actual 2, predicted 4: the absolute error is 2 and the percentage error is 100%. Actual 100, predicted 98: the same absolute error is 2%. On any series with small values (a slow-moving product, a quiet night) MAPE is dominated by items nobody cares about.\n\n**It is asymmetric.** Over-forecasting can never exceed 100% error when the actual is fixed, while under-forecasting is unbounded. So a model trained or selected on MAPE learns to forecast low. This is not a subtlety; it shows up as systematic under-stocking in real inventory systems.\n\nUse **WAPE** instead, which is the total absolute error divided by the total actual volume:\n\n```\nWAPE = sum(|actual - predicted|) / sum(actual)\n```\n\nIt is unit-free, it does not explode, and it weights by size, which is usually what the business means by \"how far off were we\".\n\n## Compare against the naive forecast, always\n\nFor anything with a time dimension, the baseline is not the mean. It is **last value** (\"tomorrow equals today\") or **seasonal naive** (\"this Tuesday equals last Tuesday\"). Both are free and both are hard to beat.\n\n**MASE** makes the comparison a single number: your MAE divided by the naive method's MAE on the same data.\n\n- MASE 0.8: you are 20% better than repeating last week.\n- MASE 1.0: you have built a machine that repeats last week.\n- MASE above 1.0: you would do better deleting the model.\n\nR² carries the same idea for non-time-series regression: it compares your squared error against always predicting the mean, so a negative R² means you are worse than a constant. Both numbers exist to stop a team celebrating a forecast that a spreadsheet formula beats.\n\n## A point estimate is often the wrong output\n\nMany decisions need a range, not a number: how much stock to hold, how much buffer to promise, whether to escalate. Then the model should emit an interval, and the evaluation is a coverage check.\n\nTake the stated 90% interval and count how often the truth fell inside it. If it is 71%, your intervals are too narrow and every plan built on them is under-buffered. If it is 99%, they are too wide to be useful. This is exactly calibration from the previous lesson, applied to quantities, and quantile loss (also called pinball loss) is the training-time version of the same idea.\n\n## Report error by size, not once\n\nErrors are almost never uniform. A 5% error on a ₹200 order and a 5% error on a ₹2,00,000 order are different businesses. Bucket by magnitude and print MAE and WAPE for each bucket. The usual finding is that the headline is carried by the many small items while the money sits in the few large ones, and that the model is worst exactly there, because large values are rare in training data.\n\nEverything here runs in `sklearn.metrics`, in `statsmodels`, or in a spreadsheet with four columns. None of it needs a GPU, and the naive baseline needs no model at all."},"check":{"question":"A demand forecast is selected by lowest MAPE. Six months later the warehouse is persistently under-stocked. What is the most likely mechanism?","options":[{"text":"MAPE weights all products equally, so the model optimised for the many small products and neglected the large ones.","correct":false,"tempting":"A real effect of unit-free averaging, but it would produce errors in both directions rather than the one-sided shortfall described."},{"text":"MAPE penalises under-forecasting less than over-forecasting for a given absolute error, because over-forecast error is bounded relative to the actual while under-forecast error is not, so selection favoured models that forecast low.","correct":true},{"text":"MAPE is undefined when the actual is zero, so those days were dropped and the model never learned about stock-outs.","correct":false,"tempting":"Names a genuine nuisance in MAPE, but dropping zero-demand days would not systematically bias forecasts downwards across ordinary days."},{"text":"RMSE should have been used, since squaring errors is the only way to penalise large misses.","correct":false,"tempting":"RMSE would help with large misses, but it is not the only option and it does not explain why the bias was consistently in the under-stocking direction."}]},"prev":{"slug":"probability-calibration","title":"Making a 0.8 mean eighty per cent","url":"https://addaly.com/learn/evaluating-ai/probability-calibration"},"next":{"slug":"retrieval-metrics","title":"Scoring a ranked list","url":"https://addaly.com/learn/evaluating-ai/retrieval-metrics"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When the output is a quantity', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/metrics-for-numbers (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When the output is a quantity","courseTitle":"Knowing If It Works","module":{"index":2,"slug":"metrics-that-mean-something","title":"Metrics that mean something"},"answer":null,"keyIdea":"Choose between MAE and RMSE by deciding whether one large error is worse than several small ones, avoid MAPE because its asymmetry biases model selection towards under-forecasting, and score every forecast against the naive baseline.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-same-question-in-a-different-costume","runs":[{"text":"The same question in a different costume"}]},{"t":"p","runs":[{"text":"Plenty of AI features output a number: predicted delivery time, forecast demand, an estimated price, a risk score in rupees, a count extracted from a document. The evaluation question is unchanged (what decision does this serve) but the metrics differ, and the differences are not cosmetic."}]},{"t":"p","runs":[{"text":"Start with five errors: 1, 1, 1, 1, 10 units."}]},{"t":"code","lang":"","label":"Text","text":"MAE  = (1+1+1+1+10) / 5              = 2.8\nRMSE = sqrt((1+1+1+1+100) / 5)       = sqrt(20.8) = 4.6"},{"t":"p","runs":[{"text":"Mean absolute error","b":true},{"text":" is the average miss, in the units of the thing. "},{"text":"Root mean squared error","b":true},{"text":" squares first, so the single large error dominates. That is the whole choice, and it is a business question rather than a statistical one:"}]},{"t":"ul","items":[[{"text":"If ten small delays cost the same as one big delay, use MAE."}],[{"text":"If one delivery three hours late costs more than ten deliveries eighteen minutes late (because it triggers a refund, a call, a lost customer) use RMSE, which will steer the model away from large misses."}]]},{"t":"p","runs":[{"text":"Quote MAE alongside anything else, because it is the only one a non-specialist reads correctly: “on average we are 2.8 minutes out” is a true sentence about MAE and a false one about RMSE."}]},{"t":"h2","id":"why-mape-keeps-getting-chosen-and-keeps-causing-trouble","runs":[{"text":"Why MAPE keeps getting chosen and keeps causing trouble"}]},{"t":"p","runs":[{"text":"Mean absolute percentage error is popular because it is unit-free and comparable across products. It has two mechanical defects that change decisions."}]},{"t":"p","runs":[{"text":"It explodes near zero.","b":true},{"text":" Actual 2, predicted 4: the absolute error is 2 and the percentage error is 100%. Actual 100, predicted 98: the same absolute error is 2%. On any series with small values (a slow-moving product, a quiet night) MAPE is dominated by items nobody cares about."}]},{"t":"p","runs":[{"text":"It is asymmetric.","b":true},{"text":" Over-forecasting can never exceed 100% error when the actual is fixed, while under-forecasting is unbounded. So a model trained or selected on MAPE learns to forecast low. This is not a subtlety; it shows up as systematic under-stocking in real inventory systems."}]},{"t":"p","runs":[{"text":"Use "},{"text":"WAPE","b":true},{"text":" instead, which is the total absolute error divided by the total actual volume:"}]},{"t":"code","lang":"","label":"Text","text":"WAPE = sum(|actual - predicted|) / sum(actual)"},{"t":"p","runs":[{"text":"It is unit-free, it does not explode, and it weights by size, which is usually what the business means by “how far off were we”."}]},{"t":"h2","id":"compare-against-the-naive-forecast-always","runs":[{"text":"Compare against the naive forecast, always"}]},{"t":"p","runs":[{"text":"For anything with a time dimension, the baseline is not the mean. It is "},{"text":"last value","b":true},{"text":" (“tomorrow equals today”) or "},{"text":"seasonal naive","b":true},{"text":" (“this Tuesday equals last Tuesday”). Both are free and both are hard to beat."}]},{"t":"p","runs":[{"text":"MASE","b":true},{"text":" makes the comparison a single number: your MAE divided by the naive method’s MAE on the same data."}]},{"t":"ul","items":[[{"text":"MASE 0.8: you are 20% better than repeating last week."}],[{"text":"MASE 1.0: you have built a machine that repeats last week."}],[{"text":"MASE above 1.0: you would do better deleting the model."}]]},{"t":"p","runs":[{"text":"R² carries the same idea for non-time-series regression: it compares your squared error against always predicting the mean, so a negative R² means you are worse than a constant. Both numbers exist to stop a team celebrating a forecast that a spreadsheet formula beats."}]},{"t":"h2","id":"a-point-estimate-is-often-the-wrong-output","runs":[{"text":"A point estimate is often the wrong output"}]},{"t":"p","runs":[{"text":"Many decisions need a range, not a number: how much stock to hold, how much buffer to promise, whether to escalate. Then the model should emit an interval, and the evaluation is a coverage check."}]},{"t":"p","runs":[{"text":"Take the stated 90% interval and count how often the truth fell inside it. If it is 71%, your intervals are too narrow and every plan built on them is under-buffered. If it is 99%, they are too wide to be useful. This is exactly calibration from the previous lesson, applied to quantities, and quantile loss (also called pinball loss) is the training-time version of the same idea."}]},{"t":"h2","id":"report-error-by-size-not-once","runs":[{"text":"Report error by size, not once"}]},{"t":"p","runs":[{"text":"Errors are almost never uniform. A 5% error on a ₹200 order and a 5% error on a ₹2,00,000 order are different businesses. Bucket by magnitude and print MAE and WAPE for each bucket. The usual finding is that the headline is carried by the many small items while the money sits in the few large ones, and that the model is worst exactly there, because large values are rare in training data."}]},{"t":"p","runs":[{"text":"Everything here runs in "},{"text":"sklearn.metrics","code":true},{"text":", in "},{"text":"statsmodels","code":true},{"text":", or in a spreadsheet with four columns. None of it needs a GPU, and the naive baseline needs no model at all."}]}],"check":{"question":[{"text":"A demand forecast is selected by lowest MAPE. Six months later the warehouse is persistently under-stocked. What is the most likely mechanism?"}],"options":[{"text":[{"text":"MAPE weights all products equally, so the model optimised for the many small products and neglected the large ones."}],"correct":false,"tempting":[{"text":"A real effect of unit-free averaging, but it would produce errors in both directions rather than the one-sided shortfall described."}]},{"text":[{"text":"MAPE penalises under-forecasting less than over-forecasting for a given absolute error, because over-forecast error is bounded relative to the actual while under-forecast error is not, so selection favoured models that forecast low."}],"correct":true,"tempting":null},{"text":[{"text":"MAPE is undefined when the actual is zero, so those days were dropped and the model never learned about stock-outs."}],"correct":false,"tempting":[{"text":"Names a genuine nuisance in MAPE, but dropping zero-demand days would not systematically bias forecasts downwards across ordinary days."}]},{"text":[{"text":"RMSE should have been used, since squaring errors is the only way to penalise large misses."}],"correct":false,"tempting":[{"text":"RMSE would help with large misses, but it is not the only option and it does not explain why the bias was consistently in the under-stocking direction."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"probability-calibration","title":"Making a 0.8 mean eighty per cent","minutes":9},"next":{"slug":"retrieval-metrics","title":"Scoring a ranked list","minutes":9}}}