{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"slice-based-error-analysis","title":"The average is hiding the failure","url":"https://addaly.com/learn/machine-learning-foundations/slice-based-error-analysis","position":52,"of":92,"minutes":9,"answer":"A headline metric is a weighted average that hides its worst population, so slice by every dimension you can name, read fifty errors by hand to find the causes.","keyIdea":"A headline metric is a weighted average that hides its worst population, so slice by every dimension you can name, read fifty errors by hand to find the causes, and turn each failure mode into a fixed test set that runs on every change."},"body":{"format":"markdown","text":"## One number, many populations\n\nA model reports 0.86 accuracy. Inside that number:\n\n- 0.94 on desktop users, 0.71 on mobile.\n- 0.89 on English text, 0.52 on Hindi.\n- 0.91 on customers older than a year, 0.63 on those in their first month.\n- 0.88 on weekdays, 0.74 on weekends.\n\nThe average is the weighted mean of these, dominated by whichever slice is largest. If mobile is 15% of traffic, a catastrophic mobile failure moves the headline number by three points and is invisible.\n\nThis is not an exotic scenario. It is the normal condition of a deployed model, and slicing is the routine that finds it.\n\n## How to slice\n\nStart with the dimensions you would use to describe a user or an input, then add the ones specific to your system.\n\n- Platform, device, browser, app version\n- Language, region, time zone\n- Tenure, account age, first-time versus returning\n- Volume band: small orders versus large\n- Time: hour of day, weekday, month, before and after a product change\n- Data quality: rows with missing fields versus complete rows\n- Any group with a fairness dimension\n\n```python\nfor col in slice_columns:\n    for value, sub in df.groupby(col):\n        if len(sub) < 30:\n            continue\n        acc = (sub[\"pred\"] == sub[\"y\"]).mean()\n        print(f\"{col}={value}: {acc:.3f}  n={len(sub)}\")\n```\n\nThe `len(sub) < 30` guard matters. Slice enough ways and you will find a group of 11 rows where accuracy is 0.45, and it will be noise. With a hundred slices, several will look bad by chance alone: this is the multiple comparisons problem from module 4 in a new costume. Confirm any finding on a second period or a second sample before acting on it.\n\n## Looking at the errors themselves\n\nAutomated slicing finds where the model fails. Reading errors tells you why, and there is no substitute for it.\n\nSort the validation set by loss, take the 50 worst, and read them. Not skim: read. For each, write one line about what went wrong. Then group your lines and count.\n\nWhat comes out of an hour of this is a list like:\n\n- 18 of 50: the label is wrong. The model was right.\n- 12: the input is ambiguous; two humans would disagree.\n- 9: a specific pattern the model has never seen: a new product category.\n- 7: an input format the pipeline mangles: a currency symbol becoming a null.\n- 4: hard.\n\nThat distribution is a work plan, and it is completely different from what the metrics suggested. Thirty of the fifty are data problems. Tuning the model addresses four of them.\n\nAndrew Ng has made this argument for years and it remains the highest-value hour available to anyone with a model that is not good enough. Almost nobody does it, because it is tedious and there is always a hyperparameter to try instead.\n\n## Worst-group performance as a target\n\nOnce you have slices, you can optimise for the worst one rather than the average.\n\nThis is a real change in objective. A model that scores 0.86 average with a worst slice at 0.71 may be worse, for your purposes, than one scoring 0.84 average with a worst slice at 0.80. Which you prefer depends on whether the slice is a group of people who will be systematically failed.\n\nThe methods (group distributionally robust optimisation and its relatives) are beyond this course. The reporting habit is not: **put the worst slice in the summary, next to the average.** A dashboard that shows only the mean is a dashboard designed to look good.\n\n## Turning findings into tests\n\nThe last step, and the one that makes this durable. Every failure mode you find becomes a fixed set of examples that runs on every model change.\n\nTwenty Hindi inputs. Twenty mobile-format rows. Twenty rows with missing income. Ten known-hard cases with hand-checked labels. Score each set separately on every candidate model.\n\nNow an improvement that raises the average while breaking Hindi is visible before it ships, rather than after a user reports it. This is the machine learning equivalent of a regression test suite, it takes an afternoon to build, and teams that have one ship far fewer surprises than teams that do not."},"check":{"question":"An analyst slices a model 80 ways and finds four slices with markedly poor accuracy, each with 15 to 25 rows. What is the right next step?","options":[{"text":"Report all four immediately, since even small slices represent real users being failed.","correct":false,"tempting":"Small slices do represent real users, but reporting unconfirmed noise as findings wastes the team's credibility and its time."},{"text":"Retrain with those four slices upweighted, so the model attends to them more.","correct":false,"tempting":"Acts on the findings before establishing they are real, which risks degrading the model to fit twenty rows of noise."},{"text":"Discard them, since any slice under 30 rows is statistically meaningless by convention.","correct":false,"tempting":"Applies a rule mechanically; a genuinely failing small slice matters, and the correct response is confirmation rather than dismissal."},{"text":"Treat them as hypotheses and check the same slices on a separate period or sample, because 80 comparisons produce several poor-looking small slices by chance alone.","correct":true}]},"prev":{"slug":"the-impossibility-result","title":"Why you cannot have all three","url":"https://addaly.com/learn/machine-learning-foundations/the-impossibility-result"},"next":{"slug":"the-classic-algorithms","title":"The classic algorithms, and when each is right","url":"https://addaly.com/learn/machine-learning-foundations/the-classic-algorithms"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The average is hiding the failure', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/slice-based-error-analysis (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The average is hiding the failure","courseTitle":"Machine Learning, Foundations","module":{"index":5,"slug":"measuring-honestly","title":"Measuring a model honestly"},"answer":null,"keyIdea":"A headline metric is a weighted average that hides its worst population, so slice by every dimension you can name, read fifty errors by hand to find the causes, and turn each failure mode into a fixed test set that runs on every change.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"one-number-many-populations","runs":[{"text":"One number, many populations"}]},{"t":"p","runs":[{"text":"A model reports 0.86 accuracy. Inside that number:"}]},{"t":"ul","items":[[{"text":"0.94 on desktop users, 0.71 on mobile."}],[{"text":"0.89 on English text, 0.52 on Hindi."}],[{"text":"0.91 on customers older than a year, 0.63 on those in their first month."}],[{"text":"0.88 on weekdays, 0.74 on weekends."}]]},{"t":"p","runs":[{"text":"The average is the weighted mean of these, dominated by whichever slice is largest. If mobile is 15% of traffic, a catastrophic mobile failure moves the headline number by three points and is invisible."}]},{"t":"p","runs":[{"text":"This is not an exotic scenario. It is the normal condition of a deployed model, and slicing is the routine that finds it."}]},{"t":"h2","id":"how-to-slice","runs":[{"text":"How to slice"}]},{"t":"p","runs":[{"text":"Start with the dimensions you would use to describe a user or an input, then add the ones specific to your system."}]},{"t":"ul","items":[[{"text":"Platform, device, browser, app version"}],[{"text":"Language, region, time zone"}],[{"text":"Tenure, account age, first-time versus returning"}],[{"text":"Volume band: small orders versus large"}],[{"text":"Time: hour of day, weekday, month, before and after a product change"}],[{"text":"Data quality: rows with missing fields versus complete rows"}],[{"text":"Any group with a fairness dimension"}]]},{"t":"code","lang":"python","label":"Python","text":"for col in slice_columns:\n    for value, sub in df.groupby(col):\n        if len(sub) < 30:\n            continue\n        acc = (sub[\"pred\"] == sub[\"y\"]).mean()\n        print(f\"{col}={value}: {acc:.3f}  n={len(sub)}\")"},{"t":"p","runs":[{"text":"The "},{"text":"len(sub) < 30","code":true},{"text":" guard matters. Slice enough ways and you will find a group of 11 rows where accuracy is 0.45, and it will be noise. With a hundred slices, several will look bad by chance alone: this is the multiple comparisons problem from module 4 in a new costume. Confirm any finding on a second period or a second sample before acting on it."}]},{"t":"h2","id":"looking-at-the-errors-themselves","runs":[{"text":"Looking at the errors themselves"}]},{"t":"p","runs":[{"text":"Automated slicing finds where the model fails. Reading errors tells you why, and there is no substitute for it."}]},{"t":"p","runs":[{"text":"Sort the validation set by loss, take the 50 worst, and read them. Not skim: read. For each, write one line about what went wrong. Then group your lines and count."}]},{"t":"p","runs":[{"text":"What comes out of an hour of this is a list like:"}]},{"t":"ul","items":[[{"text":"18 of 50: the label is wrong. The model was right."}],[{"text":"12: the input is ambiguous; two humans would disagree."}],[{"text":"9: a specific pattern the model has never seen: a new product category."}],[{"text":"7: an input format the pipeline mangles: a currency symbol becoming a null."}],[{"text":"4: hard."}]]},{"t":"p","runs":[{"text":"That distribution is a work plan, and it is completely different from what the metrics suggested. Thirty of the fifty are data problems. Tuning the model addresses four of them."}]},{"t":"p","runs":[{"text":"Andrew Ng has made this argument for years and it remains the highest-value hour available to anyone with a model that is not good enough. Almost nobody does it, because it is tedious and there is always a hyperparameter to try instead."}]},{"t":"h2","id":"worst-group-performance-as-a-target","runs":[{"text":"Worst-group performance as a target"}]},{"t":"p","runs":[{"text":"Once you have slices, you can optimise for the worst one rather than the average."}]},{"t":"p","runs":[{"text":"This is a real change in objective. A model that scores 0.86 average with a worst slice at 0.71 may be worse, for your purposes, than one scoring 0.84 average with a worst slice at 0.80. Which you prefer depends on whether the slice is a group of people who will be systematically failed."}]},{"t":"p","runs":[{"text":"The methods (group distributionally robust optimisation and its relatives) are beyond this course. The reporting habit is not: "},{"text":"put the worst slice in the summary, next to the average.","b":true},{"text":" A dashboard that shows only the mean is a dashboard designed to look good."}]},{"t":"h2","id":"turning-findings-into-tests","runs":[{"text":"Turning findings into tests"}]},{"t":"p","runs":[{"text":"The last step, and the one that makes this durable. Every failure mode you find becomes a fixed set of examples that runs on every model change."}]},{"t":"p","runs":[{"text":"Twenty Hindi inputs. Twenty mobile-format rows. Twenty rows with missing income. Ten known-hard cases with hand-checked labels. Score each set separately on every candidate model."}]},{"t":"p","runs":[{"text":"Now an improvement that raises the average while breaking Hindi is visible before it ships, rather than after a user reports it. This is the machine learning equivalent of a regression test suite, it takes an afternoon to build, and teams that have one ship far fewer surprises than teams that do not."}]}],"check":{"question":[{"text":"An analyst slices a model 80 ways and finds four slices with markedly poor accuracy, each with 15 to 25 rows. What is the right next step?"}],"options":[{"text":[{"text":"Report all four immediately, since even small slices represent real users being failed."}],"correct":false,"tempting":[{"text":"Small slices do represent real users, but reporting unconfirmed noise as findings wastes the team’s credibility and its time."}]},{"text":[{"text":"Retrain with those four slices upweighted, so the model attends to them more."}],"correct":false,"tempting":[{"text":"Acts on the findings before establishing they are real, which risks degrading the model to fit twenty rows of noise."}]},{"text":[{"text":"Discard them, since any slice under 30 rows is statistically meaningless by convention."}],"correct":false,"tempting":[{"text":"Applies a rule mechanically; a genuinely failing small slice matters, and the correct response is confirmation rather than dismissal."}]},{"text":[{"text":"Treat them as hypotheses and check the same slices on a separate period or sample, because 80 comparisons produce several poor-looking small slices by chance alone."}],"correct":true,"tempting":null}],"order":[0,1,3,2],"why":null},"prev":{"slug":"the-impossibility-result","title":"Why you cannot have all three","minutes":8},"next":{"slug":"the-classic-algorithms","title":"The classic algorithms, and when each is right","minutes":10}}}