The average is hiding the failure

One number, many populations

A model reports 0.86 accuracy. Inside that number:

• 0.94 on desktop users, 0.71 on mobile.
• 0.89 on English text, 0.52 on Hindi.
• 0.91 on customers older than a year, 0.63 on those in their first month.
• 0.88 on weekdays, 0.74 on weekends.

The average is the weighted mean of these, dominated by whichever slice is largest. If mobile is 15% of traffic, a catastrophic mobile failure moves the headline number by three points and is invisible.

This is not an exotic scenario. It is the normal condition of a deployed model, and slicing is the routine that finds it.

How to slice

Start with the dimensions you would use to describe a user or an input, then add the ones specific to your system.

• Platform, device, browser, app version
• Language, region, time zone
• Tenure, account age, first-time versus returning
• Volume band: small orders versus large
• Time: hour of day, weekday, month, before and after a product change
• Data quality: rows with missing fields versus complete rows
• Any group with a fairness dimension

for col in slice_columns:
    for value, sub in df.groupby(col):
        if len(sub) < 30:
            continue
        acc = (sub["pred"] == sub["y"]).mean()
        print(f"{col}={value}: {acc:.3f}  n={len(sub)}")

The len(sub) < 30 guard matters. Slice enough ways and you will find a group of 11 rows where accuracy is 0.45, and it will be noise. With a hundred slices, several will look bad by chance alone: this is the multiple comparisons problem from module 4 in a new costume. Confirm any finding on a second period or a second sample before acting on it.

Looking at the errors themselves

Automated slicing finds where the model fails. Reading errors tells you why, and there is no substitute for it.

Sort the validation set by loss, take the 50 worst, and read them. Not skim: read. For each, write one line about what went wrong. Then group your lines and count.

What comes out of an hour of this is a list like:

• 18 of 50: the label is wrong. The model was right.
• 12: the input is ambiguous; two humans would disagree.
• 9: a specific pattern the model has never seen: a new product category.
• 7: an input format the pipeline mangles: a currency symbol becoming a null.
• 4: hard.

That distribution is a work plan, and it is completely different from what the metrics suggested. Thirty of the fifty are data problems. Tuning the model addresses four of them.

Andrew Ng has made this argument for years and it remains the highest-value hour available to anyone with a model that is not good enough. Almost nobody does it, because it is tedious and there is always a hyperparameter to try instead.

Worst-group performance as a target

Once you have slices, you can optimise for the worst one rather than the average.

This is a real change in objective. A model that scores 0.86 average with a worst slice at 0.71 may be worse, for your purposes, than one scoring 0.84 average with a worst slice at 0.80. Which you prefer depends on whether the slice is a group of people who will be systematically failed.

The methods (group distributionally robust optimisation and its relatives) are beyond this course. The reporting habit is not: put the worst slice in the summary, next to the average. A dashboard that shows only the mean is a dashboard designed to look good.

Turning findings into tests

The last step, and the one that makes this durable. Every failure mode you find becomes a fixed set of examples that runs on every model change.

Twenty Hindi inputs. Twenty mobile-format rows. Twenty rows with missing income. Ten known-hard cases with hand-checked labels. Score each set separately on every candidate model.

Now an improvement that raises the average while breaking Hindi is visible before it ships, rather than after a user reports it. This is the machine learning equivalent of a regression test suite, it takes an afternoon to build, and teams that have one ship far fewer surprises than teams that do not.