{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"regression-metrics","title":"Metrics for a number, and what each one hides","url":"https://addaly.com/learn/machine-learning-foundations/regression-metrics","position":47,"of":92,"minutes":8,"answer":"Report MAE and RMSE together because their gap measures how uneven the errors are, and plot residuals against predictions: the shape of the failure carries information no summary metric retains.","keyIdea":"Report MAE and RMSE together because their gap measures how uneven the errors are, and plot residuals against predictions: the shape of the failure carries information no summary metric retains."},"body":{"format":"markdown","text":"## Four metrics, four different claims\n\nWhen the target is a number rather than a class, the metric choice is quieter but no less consequential.\n\n**MAE**: mean absolute error. Average size of the error, in the target's units. Interpretable directly: \"we are off by 4,200 rupees on average\". Treats every rupee of error the same.\n\n**RMSE**: root mean squared error. Also in the target's units, but errors are squared before averaging, so large errors dominate. RMSE is always at least as large as MAE, and the *gap between them tells you how uneven your errors are*. RMSE 12,000 with MAE 4,200 means a few very large errors; RMSE 4,600 with MAE 4,200 means errors of consistent size. Reporting both is more informative than reporting either.\n\n**MAPE**: mean absolute percentage error. Scale-free, so it compares across products or regions. Two serious defects: it is undefined when the true value is zero, and it is asymmetric: over-predicting is capped at 100% error while under-predicting is unbounded, so a model tuned on MAPE learns to under-predict systematically. Widely used in demand forecasting and widely criticised there for exactly this.\n\n**R²**: the fraction of the target's variance the model explains. 0 means you match the mean; 1 means perfect; negative means worse than predicting the mean, which happens more often than people expect on held-out data. Its weakness is that it depends on the variance of the evaluation set, so an R² of 0.9 on a varied population and 0.3 on a narrow one can come from the same model with the same absolute errors.\n\n## Choosing\n\nAsk what an error costs.\n\n- Cost proportional to the size of the error: **MAE**.\n- Large errors disproportionately bad: **RMSE**.\n- A 10% error means the same at every scale: **MAPE**, or better, fit on the log of the target and use RMSE there, which gives you the same intent without MAPE's asymmetry.\n- Explaining to a non-technical audience how much better than nothing you are: **R²**, alongside the baseline.\n\nAnd remember from module 1 that the metric and the training loss need not match. Train on Huber for robustness and report MAE; there is nothing wrong with that as long as you say so.\n\n## The metric hides the shape\n\nEvery one of these is an average, and averages conceal structure. Two diagnostics take a minute each and are worth more than the summary numbers.\n\n**Plot residuals against predictions.** What you want is a formless cloud centred on zero. What you often get:\n\n- A funnel widening to the right: errors grow with the prediction, which means a log transform on the target is likely to help.\n- A curve: the model has systematic bias in a region, which means a missing non-linearity.\n- A slope: the classic regression-to-the-mean pattern, where the model over-predicts low values and under-predicts high ones.\n\n**Plot the error distribution.** A long tail on one side tells you the model fails asymmetrically, which matters enormously if the two directions have different costs. Under-predicting demand means a stockout; over-predicting means holding stock. Those are not the same problem, and no symmetric metric distinguishes them.\n\n## Quantile loss, for when you need a range\n\nSometimes a single number is the wrong output. \"Delivery in 34 minutes\" is a promise; \"90% of deliveries arrive within 48 minutes\" is a promise you can keep.\n\nQuantile regression fits a chosen quantile rather than the mean, by using an asymmetric loss that penalises under-prediction and over-prediction differently.\n\n```python\nfrom sklearn.ensemble import GradientBoostingRegressor\nGradientBoostingRegressor(loss=\"quantile\", alpha=0.9)\n```\n\nFit three models at 0.1, 0.5 and 0.9 and you have a median with an interval around it. For anything where a customer is given a commitment, this is usually more useful than a point estimate, and it is the honest way to express what the model does and does not know.\n\n## Two mistakes to avoid\n\n**Comparing RMSE across different target scales.** RMSE of 400 on house prices and 400 on delivery minutes are not comparable numbers. Normalise, or use R², or report against a baseline.\n\n**Averaging percentage errors across items of wildly different size.** One tiny item with a large percentage error can dominate a MAPE computed over a thousand products. Weight by volume or value if the business cares about total error rather than average per-item error, and say which you did."},"check":{"question":"A demand model is tuned to minimise MAPE and consistently under-forecasts, causing stockouts. What property of MAPE explains the direction of the bias?","options":[{"text":"MAPE is undefined at zero, so days with no demand were dropped, biasing the remaining sample downward.","correct":false,"tempting":"The zero problem is real and does distort MAPE, but excluding zero-demand days does not systematically push forecasts below actual demand."},{"text":"MAPE weights every item equally regardless of volume, so small-volume items dominate and pull forecasts down.","correct":false,"tempting":"Equal weighting is a genuine MAPE weakness, but it distorts which items the model attends to rather than the direction of the error."},{"text":"MAPE divides by the actual value, so it is asymmetric: over-predicting can cost at most 100% while under-predicting is unbounded, so the optimiser learns that forecasting low is the safer error.","correct":true},{"text":"MAPE is a squared metric, so it over-penalises large errors and the model shrinks all forecasts toward zero to avoid them.","correct":false,"tempting":"MAPE uses absolute rather than squared errors, so the described mechanism is not present."}]},"prev":{"slug":"probability-calibration","title":"Does 0.7 mean seventy per cent?","url":"https://addaly.com/learn/machine-learning-foundations/probability-calibration"},"next":{"slug":"comparing-two-models","title":"Is that improvement real?","url":"https://addaly.com/learn/machine-learning-foundations/comparing-two-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Metrics for a number, and what each one hides', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/regression-metrics (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Metrics for a number, and what each one hides","courseTitle":"Machine Learning, Foundations","module":{"index":5,"slug":"measuring-honestly","title":"Measuring a model honestly"},"answer":null,"keyIdea":"Report MAE and RMSE together because their gap measures how uneven the errors are, and plot residuals against predictions: the shape of the failure carries information no summary metric retains.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"four-metrics-four-different-claims","runs":[{"text":"Four metrics, four different claims"}]},{"t":"p","runs":[{"text":"When the target is a number rather than a class, the metric choice is quieter but no less consequential."}]},{"t":"p","runs":[{"text":"MAE","b":true},{"text":": mean absolute error. Average size of the error, in the target’s units. Interpretable directly: “we are off by 4,200 rupees on average”. Treats every rupee of error the same."}]},{"t":"p","runs":[{"text":"RMSE","b":true},{"text":": root mean squared error. Also in the target’s units, but errors are squared before averaging, so large errors dominate. RMSE is always at least as large as MAE, and the "},{"text":"gap between them tells you how uneven your errors are","i":true},{"text":". RMSE 12,000 with MAE 4,200 means a few very large errors; RMSE 4,600 with MAE 4,200 means errors of consistent size. Reporting both is more informative than reporting either."}]},{"t":"p","runs":[{"text":"MAPE","b":true},{"text":": mean absolute percentage error. Scale-free, so it compares across products or regions. Two serious defects: it is undefined when the true value is zero, and it is asymmetric: over-predicting is capped at 100% error while under-predicting is unbounded, so a model tuned on MAPE learns to under-predict systematically. Widely used in demand forecasting and widely criticised there for exactly this."}]},{"t":"p","runs":[{"text":"R²","b":true},{"text":": the fraction of the target’s variance the model explains. 0 means you match the mean; 1 means perfect; negative means worse than predicting the mean, which happens more often than people expect on held-out data. Its weakness is that it depends on the variance of the evaluation set, so an R² of 0.9 on a varied population and 0.3 on a narrow one can come from the same model with the same absolute errors."}]},{"t":"h2","id":"choosing","runs":[{"text":"Choosing"}]},{"t":"p","runs":[{"text":"Ask what an error costs."}]},{"t":"ul","items":[[{"text":"Cost proportional to the size of the error: "},{"text":"MAE","b":true},{"text":"."}],[{"text":"Large errors disproportionately bad: "},{"text":"RMSE","b":true},{"text":"."}],[{"text":"A 10% error means the same at every scale: "},{"text":"MAPE","b":true},{"text":", or better, fit on the log of the target and use RMSE there, which gives you the same intent without MAPE’s asymmetry."}],[{"text":"Explaining to a non-technical audience how much better than nothing you are: "},{"text":"R²","b":true},{"text":", alongside the baseline."}]]},{"t":"p","runs":[{"text":"And remember from module 1 that the metric and the training loss need not match. Train on Huber for robustness and report MAE; there is nothing wrong with that as long as you say so."}]},{"t":"h2","id":"the-metric-hides-the-shape","runs":[{"text":"The metric hides the shape"}]},{"t":"p","runs":[{"text":"Every one of these is an average, and averages conceal structure. Two diagnostics take a minute each and are worth more than the summary numbers."}]},{"t":"p","runs":[{"text":"Plot residuals against predictions.","b":true},{"text":" What you want is a formless cloud centred on zero. What you often get:"}]},{"t":"ul","items":[[{"text":"A funnel widening to the right: errors grow with the prediction, which means a log transform on the target is likely to help."}],[{"text":"A curve: the model has systematic bias in a region, which means a missing non-linearity."}],[{"text":"A slope: the classic regression-to-the-mean pattern, where the model over-predicts low values and under-predicts high ones."}]]},{"t":"p","runs":[{"text":"Plot the error distribution.","b":true},{"text":" A long tail on one side tells you the model fails asymmetrically, which matters enormously if the two directions have different costs. Under-predicting demand means a stockout; over-predicting means holding stock. Those are not the same problem, and no symmetric metric distinguishes them."}]},{"t":"h2","id":"quantile-loss-for-when-you-need-a-range","runs":[{"text":"Quantile loss, for when you need a range"}]},{"t":"p","runs":[{"text":"Sometimes a single number is the wrong output. “Delivery in 34 minutes” is a promise; “90% of deliveries arrive within 48 minutes” is a promise you can keep."}]},{"t":"p","runs":[{"text":"Quantile regression fits a chosen quantile rather than the mean, by using an asymmetric loss that penalises under-prediction and over-prediction differently."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.ensemble import GradientBoostingRegressor\nGradientBoostingRegressor(loss=\"quantile\", alpha=0.9)"},{"t":"p","runs":[{"text":"Fit three models at 0.1, 0.5 and 0.9 and you have a median with an interval around it. For anything where a customer is given a commitment, this is usually more useful than a point estimate, and it is the honest way to express what the model does and does not know."}]},{"t":"h2","id":"two-mistakes-to-avoid","runs":[{"text":"Two mistakes to avoid"}]},{"t":"p","runs":[{"text":"Comparing RMSE across different target scales.","b":true},{"text":" RMSE of 400 on house prices and 400 on delivery minutes are not comparable numbers. Normalise, or use R², or report against a baseline."}]},{"t":"p","runs":[{"text":"Averaging percentage errors across items of wildly different size.","b":true},{"text":" One tiny item with a large percentage error can dominate a MAPE computed over a thousand products. Weight by volume or value if the business cares about total error rather than average per-item error, and say which you did."}]}],"check":{"question":[{"text":"A demand model is tuned to minimise MAPE and consistently under-forecasts, causing stockouts. What property of MAPE explains the direction of the bias?"}],"options":[{"text":[{"text":"MAPE is undefined at zero, so days with no demand were dropped, biasing the remaining sample downward."}],"correct":false,"tempting":[{"text":"The zero problem is real and does distort MAPE, but excluding zero-demand days does not systematically push forecasts below actual demand."}]},{"text":[{"text":"MAPE weights every item equally regardless of volume, so small-volume items dominate and pull forecasts down."}],"correct":false,"tempting":[{"text":"Equal weighting is a genuine MAPE weakness, but it distorts which items the model attends to rather than the direction of the error."}]},{"text":[{"text":"MAPE divides by the actual value, so it is asymmetric: over-predicting can cost at most 100% while under-predicting is unbounded, so the optimiser learns that forecasting low is the safer error."}],"correct":true,"tempting":null},{"text":[{"text":"MAPE is a squared metric, so it over-penalises large errors and the model shrinks all forecasts toward zero to avoid them."}],"correct":false,"tempting":[{"text":"MAPE uses absolute rather than squared errors, so the described mechanism is not present."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"probability-calibration","title":"Does 0.7 mean seventy per cent?","minutes":9},"next":{"slug":"comparing-two-models","title":"Is that improvement real?","minutes":9}}}