{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"what-a-loss-function-encodes","title":"The loss is where you state what a mistake costs","url":"https://addaly.com/learn/machine-learning-foundations/what-a-loss-function-encodes","position":4,"of":92,"minutes":9,"answer":"Squared error fits the mean and fears outliers, absolute error fits the median and ignores them, log loss punishes confident mistakes without limit, and none of these is the metric you report.","keyIdea":"Squared error fits the mean and fears outliers, absolute error fits the median and ignores them, log loss punishes confident mistakes without limit, and none of these is the metric you report."},"body":{"format":"markdown","text":"## Fitting means minimising something you chose\n\nA model has parameters. Training searches for parameter values that make one number small. That number is the **loss**, and the choice of loss is not a technicality: it is the sentence \"this is what being wrong costs me\", written in maths so an optimiser can act on it.\n\nTwo people with the same data and the same algorithm can get different models because they encoded different costs. Neither is wrong. Only one matches the situation.\n\n## Squared error punishes the outlier\n\nFor a numeric target, the default is mean squared error:\n\n```python\nmse = ((y_true - y_pred) ** 2).mean()\n```\n\nSquaring does something specific. An error of 10 costs 100; an error of 100 costs 10,000. One badly wrong row can outweigh a hundred nearly right ones. Fit a line to house prices with MSE and a single mansion drags the whole line upward, worsening every ordinary prediction, because moving 5 units towards the mansion buys more loss reduction than 5 units of error added across many small houses.\n\nMean absolute error does not do that:\n\n```python\nmae = np.abs(y_true - y_pred).mean()\n```\n\nAn error of 100 costs 100. Outliers get no special weight.\n\nThere is a clean way to remember which you are asking for. **Minimising squared error fits the mean; minimising absolute error fits the median.** If your data is `[1, 2, 3, 4, 100]` and you must predict one constant, MSE gives you 22, MAE gives you 3. Ask yourself which of those two answers you actually want a delivery-time estimate to be.\n\nHuber loss is the compromise: squared near zero, linear far away, with a threshold you set. It is a good default for regression on messy real data and it is one line in scikit-learn.\n\n## For classification, the loss is about probabilities\n\nYou might think classification should minimise the number of wrong labels. It does not, and the reason is mechanical: the count of errors is a step function. Nudge a parameter slightly and the count either does not move at all or jumps by one. There is no slope to follow, so gradient descent has nothing to work with.\n\nSo classification minimises **log loss**, also called cross-entropy, over predicted probabilities:\n\n```python\nlogloss = -(y * np.log(p) + (1 - y) * np.log(1 - p)).mean()\n```\n\nRead what this does at two rows. The true label is 1. You predict 0.9, and the loss is `-log(0.9) = 0.105`. You predict 0.6, and it is `0.511`. You predict 0.01, and it is `4.6`. Now the ugly one: you predict 0.0001 and the truth is 1, and the loss is `9.2`. Log loss is asymmetric in a way that matters: being confidently wrong is punished savagely, and as your prediction approaches 0 for a positive case the loss goes to infinity. That is the property that makes models trained on log loss produce probabilities you can partly trust, and it is why a single confident mistake can dominate a validation score.\n\n> Figure: What log loss charges for one row whose true label is 1. Bars (log loss for a positive row at that predicted probability): Predicted 0.9 0.105; Predicted 0.6 0.511; Predicted 0.01 4.6; Predicted 0.0001 9.2. The scale is not linear. Moving a prediction from 0.9 down to 0.6 costs about half a unit; moving from 0.01 down to 0.0001 costs another 4.6. One confidently wrong row can outweigh dozens of nearly right ones, which is why a model trained on log loss learns to hedge.\n\n## The loss is not the metric\n\nThis trips up almost everyone once. The loss is what training optimises; the metric is what you report and decide with. They are often different, deliberately.\n\nYou cannot train on \"precision at a threshold of 0.7\": it is not differentiable and the threshold is not part of the model. So you train on log loss to get honest probabilities, and then you choose the threshold afterwards using precision and recall, which is exactly what module 5 does. The gap between loss and metric is normal. What is not normal is failing to notice that closing the gap in loss did nothing for the metric, which happens more often than the textbooks admit.\n\n## Weighting is how you say some rows matter more\n\nEvery major library accepts `sample_weight`. Multiply each row's loss contribution by a number and the optimiser will trade accuracy on cheap rows for accuracy on expensive ones.\n\nThis is the cleanest tool available for imbalance and for unequal costs. If missing a fraudulent transaction costs 200 times what a false alarm costs, that is not a threshold trick: you can say it directly, in the loss, and the model will spend its capacity accordingly. `class_weight='balanced'` in scikit-learn is the same mechanism with the weights set to the inverse class frequency.\n\n## The limitation nobody puts on the slide\n\nA loss function is a scalar. Real decisions have costs that do not collapse into one number, and when you force them to, you have made an ethical claim with a division sign. A model that decides who gets a loan is trading false rejections against false approvals, and those two harms land on different people. Writing `sample_weight=200` does not resolve that; it records your answer to it in a place where nobody will read it again.\n\nSo state the loss in words next to the code. \"A missed fraud costs us roughly 200 times a false alarm, based on average chargeback of ₹18,000 against about ₹90 of review time.\" Someone can then argue with the 200. They cannot argue with a number they never saw."},"check":{"question":"A delivery firm predicts arrival times. Most trips take 20 to 40 minutes, but a handful take three hours because of accidents. Trained with MSE, the model consistently over-predicts ordinary trips. What is happening?","options":[{"text":"The three-hour trips are rare enough to be treated as noise, so the model is ignoring them and drifting toward the minimum trip time.","correct":false,"tempting":"Inverts the effect: squared error makes rare extreme values more influential, not less, so they cannot be quietly ignored."},{"text":"The model has too few features to distinguish accident days, so it underfits and predicts the overall average for everything.","correct":false,"tempting":"Would produce a flat prediction, but does not explain the specific upward bias, which comes from how squared error weights the long trips."},{"text":"Squared error makes the three-hour errors enormously costly, so the fitted line moves upward to reduce them, adding a little error to every normal trip.","correct":true},{"text":"Arrival time is not normally distributed, and MSE requires a normal target, so the optimiser fails to converge.","correct":false,"tempting":"Treats a distributional assumption as a hard requirement; MSE converges fine on skewed targets, it just fits the mean of that skewed distribution."}]},"prev":{"slug":"the-shape-of-a-dataset","title":"What a row is, and why that decision is the whole project","url":"https://addaly.com/learn/machine-learning-foundations/the-shape-of-a-dataset"},"next":{"slug":"the-baseline-you-must-beat","title":"The number your model has to beat before anyone should care","url":"https://addaly.com/learn/machine-learning-foundations/the-baseline-you-must-beat"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The loss is where you state what a mistake costs', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/what-a-loss-function-encodes (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The loss is where you state what a mistake costs","courseTitle":"Machine Learning, Foundations","module":{"index":1,"slug":"what-learning-is","title":"What learning is, and what it is not"},"answer":null,"keyIdea":"Squared error fits the mean and fears outliers, absolute error fits the median and ignores them, log loss punishes confident mistakes without limit, and none of these is the metric you report.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"log loss for a positive row at that predicted probability","title":"What log loss charges for one row whose true label is 1","series":[{"label":"Predicted 0.9","value":0.105},{"label":"Predicted 0.6","value":0.511},{"label":"Predicted 0.01","value":4.6},{"label":"Predicted 0.0001","value":9.2,"highlight":true}],"caption":"The scale is not linear. Moving a prediction from 0.9 down to 0.6 costs about half a unit; moving from 0.01 down to 0.0001 costs another 4.6. One confidently wrong row can outweigh dozens of nearly right ones, which is why a model trained on log loss learns to hedge."},"alt":"Figure: What log loss charges for one row whose true label is 1. Bars (log loss for a positive row at that predicted probability): Predicted 0.9 0.105; Predicted 0.6 0.511; Predicted 0.01 4.6; Predicted 0.0001 9.2. The scale is not linear. Moving a prediction from 0.9 down to 0.6 costs about half a unit; moving from 0.01 down to 0.0001 costs another 4.6. One confidently wrong row can outweigh dozens of nearly right ones, which is why a model trained on log loss learns to hedge."}},"blocks":[{"t":"h2","id":"fitting-means-minimising-something-you-chose","runs":[{"text":"Fitting means minimising something you chose"}]},{"t":"p","runs":[{"text":"A model has parameters. Training searches for parameter values that make one number small. That number is the "},{"text":"loss","b":true},{"text":", and the choice of loss is not a technicality: it is the sentence “this is what being wrong costs me”, written in maths so an optimiser can act on it."}]},{"t":"p","runs":[{"text":"Two people with the same data and the same algorithm can get different models because they encoded different costs. Neither is wrong. Only one matches the situation."}]},{"t":"h2","id":"squared-error-punishes-the-outlier","runs":[{"text":"Squared error punishes the outlier"}]},{"t":"p","runs":[{"text":"For a numeric target, the default is mean squared error:"}]},{"t":"code","lang":"python","label":"Python","text":"mse = ((y_true - y_pred) ** 2).mean()"},{"t":"p","runs":[{"text":"Squaring does something specific. An error of 10 costs 100; an error of 100 costs 10,000. One badly wrong row can outweigh a hundred nearly right ones. Fit a line to house prices with MSE and a single mansion drags the whole line upward, worsening every ordinary prediction, because moving 5 units towards the mansion buys more loss reduction than 5 units of error added across many small houses."}]},{"t":"p","runs":[{"text":"Mean absolute error does not do that:"}]},{"t":"code","lang":"python","label":"Python","text":"mae = np.abs(y_true - y_pred).mean()"},{"t":"p","runs":[{"text":"An error of 100 costs 100. Outliers get no special weight."}]},{"t":"p","runs":[{"text":"There is a clean way to remember which you are asking for. "},{"text":"Minimising squared error fits the mean; minimising absolute error fits the median.","b":true},{"text":" If your data is "},{"text":"[1, 2, 3, 4, 100]","code":true},{"text":" and you must predict one constant, MSE gives you 22, MAE gives you 3. Ask yourself which of those two answers you actually want a delivery-time estimate to be."}]},{"t":"p","runs":[{"text":"Huber loss is the compromise: squared near zero, linear far away, with a threshold you set. It is a good default for regression on messy real data and it is one line in scikit-learn."}]},{"t":"h2","id":"for-classification-the-loss-is-about-probabilities","runs":[{"text":"For classification, the loss is about probabilities"}]},{"t":"p","runs":[{"text":"You might think classification should minimise the number of wrong labels. It does not, and the reason is mechanical: the count of errors is a step function. Nudge a parameter slightly and the count either does not move at all or jumps by one. There is no slope to follow, so gradient descent has nothing to work with."}]},{"t":"p","runs":[{"text":"So classification minimises "},{"text":"log loss","b":true},{"text":", also called cross-entropy, over predicted probabilities:"}]},{"t":"code","lang":"python","label":"Python","text":"logloss = -(y * np.log(p) + (1 - y) * np.log(1 - p)).mean()"},{"t":"p","runs":[{"text":"Read what this does at two rows. The true label is 1. You predict 0.9, and the loss is "},{"text":"-log(0.9) = 0.105","code":true},{"text":". You predict 0.6, and it is "},{"text":"0.511","code":true},{"text":". You predict 0.01, and it is "},{"text":"4.6","code":true},{"text":". Now the ugly one: you predict 0.0001 and the truth is 1, and the loss is "},{"text":"9.2","code":true},{"text":". Log loss is asymmetric in a way that matters: being confidently wrong is punished savagely, and as your prediction approaches 0 for a positive case the loss goes to infinity. That is the property that makes models trained on log loss produce probabilities you can partly trust, and it is why a single confident mistake can dominate a validation score."}]},{"t":"h2","id":"the-loss-is-not-the-metric","runs":[{"text":"The loss is not the metric"}]},{"t":"p","runs":[{"text":"This trips up almost everyone once. The loss is what training optimises; the metric is what you report and decide with. They are often different, deliberately."}]},{"t":"p","runs":[{"text":"You cannot train on “precision at a threshold of 0.7”: it is not differentiable and the threshold is not part of the model. So you train on log loss to get honest probabilities, and then you choose the threshold afterwards using precision and recall, which is exactly what module 5 does. The gap between loss and metric is normal. What is not normal is failing to notice that closing the gap in loss did nothing for the metric, which happens more often than the textbooks admit."}]},{"t":"h2","id":"weighting-is-how-you-say-some-rows-matter-more","runs":[{"text":"Weighting is how you say some rows matter more"}]},{"t":"p","runs":[{"text":"Every major library accepts "},{"text":"sample_weight","code":true},{"text":". Multiply each row’s loss contribution by a number and the optimiser will trade accuracy on cheap rows for accuracy on expensive ones."}]},{"t":"p","runs":[{"text":"This is the cleanest tool available for imbalance and for unequal costs. If missing a fraudulent transaction costs 200 times what a false alarm costs, that is not a threshold trick: you can say it directly, in the loss, and the model will spend its capacity accordingly. "},{"text":"class_weight='balanced'","code":true},{"text":" in scikit-learn is the same mechanism with the weights set to the inverse class frequency."}]},{"t":"h2","id":"the-limitation-nobody-puts-on-the-slide","runs":[{"text":"The limitation nobody puts on the slide"}],"role":"limit"},{"t":"p","runs":[{"text":"A loss function is a scalar. Real decisions have costs that do not collapse into one number, and when you force them to, you have made an ethical claim with a division sign. A model that decides who gets a loan is trading false rejections against false approvals, and those two harms land on different people. Writing "},{"text":"sample_weight=200","code":true},{"text":" does not resolve that; it records your answer to it in a place where nobody will read it again."}]},{"t":"p","runs":[{"text":"So state the loss in words next to the code. “A missed fraud costs us roughly 200 times a false alarm, based on average chargeback of ₹18,000 against about ₹90 of review time.” Someone can then argue with the 200. They cannot argue with a number they never saw."}]}],"check":{"question":[{"text":"A delivery firm predicts arrival times. Most trips take 20 to 40 minutes, but a handful take three hours because of accidents. Trained with MSE, the model consistently over-predicts ordinary trips. What is happening?"}],"options":[{"text":[{"text":"The three-hour trips are rare enough to be treated as noise, so the model is ignoring them and drifting toward the minimum trip time."}],"correct":false,"tempting":[{"text":"Inverts the effect: squared error makes rare extreme values more influential, not less, so they cannot be quietly ignored."}]},{"text":[{"text":"The model has too few features to distinguish accident days, so it underfits and predicts the overall average for everything."}],"correct":false,"tempting":[{"text":"Would produce a flat prediction, but does not explain the specific upward bias, which comes from how squared error weights the long trips."}]},{"text":[{"text":"Squared error makes the three-hour errors enormously costly, so the fitted line moves upward to reduce them, adding a little error to every normal trip."}],"correct":true,"tempting":null},{"text":[{"text":"Arrival time is not normally distributed, and MSE requires a normal target, so the optimiser fails to converge."}],"correct":false,"tempting":[{"text":"Treats a distributional assumption as a hard requirement; MSE converges fine on skewed targets, it just fits the mean of that skewed distribution."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"the-shape-of-a-dataset","title":"What a row is, and why that decision is the whole project","minutes":9},"next":{"slug":"the-baseline-you-must-beat","title":"The number your model has to beat before anyone should care","minutes":8}}}