Fairness as arithmetic, and the choice you cannot avoid

The overall number is an average over people

A system at 86% is not 86% for everybody. It is 91% for one group and 68% for another, and the average conceals which. That is not an ethical observation yet; it is a measurement failure, and it has an entirely concrete fix: compute the metric per group before you compute it overall.

What counts as a group depends on what you can legitimately observe. Often you can use non-sensitive attributes that carry most of the signal anyway:

• language and script: this is the big one for most products, and it is usually recorded
• region or city tier
• device and connection quality: long, garbled inputs from a cheap keyboard on a slow line
• account age or tenure
• channel: web, WhatsApp, phone transcript

Do not infer protected characteristics from names in order to measure them. Name-based inference is unreliable, and building the inference creates a dataset you then have to protect. Where legally collected demographic data exists, use it under the same rules as any other sensitive field. Where it does not, slice by what you have and say clearly which groups you could not check.

A finding worth expecting: a support assistant at 88% in English and 61% in Bengali is not one system with a quality problem. It is two systems, one of which was never evaluated.

Three definitions, and they conflict

When a system makes decisions about people (approving, flagging, prioritising, ranking) “fair” has to be made precise before it can be measured, and there is more than one way to do it.

Demographic parity. The positive rate is the same across groups. If 12% of group A is approved, 12% of group B is approved. Ignores whether the underlying rates differ.

Equalised odds. The error rates match across groups: the same true positive rate and the same false positive rate. Among people who should be approved, the same fraction are, in both groups.

Calibration within groups. A score of 0.8 means an 80% chance of the outcome, in every group. Everyone with the same score gets the same treatment.

All three sound like fairness. Here is the part that people find surprising the first time.

You cannot have all three

This is a mathematical result, not a matter of effort or engineering skill. Kleinberg, Mullainathan and Raghavan (2016) and Chouldechova (2016) showed independently that unless the base rates are identical across groups, or the classifier is perfect, calibration and equalised odds cannot both hold.

A sketch of why. Precision, recall and base rate are linked: FPR = (1 - precision)/precision × (base rate)/(1 - base rate) × TPR, roughly. Fix the base rates unequal and hold two of the fairness properties, and the third is forced to be violated. There is no clever model that escapes it.

The consequence is practical and it is the point of this lesson. Fairness is a choice you state, not a box you tick. You pick the definition that matches the harm you most want to avoid, you write down which one you chose and why, and you report the others so the trade-off is visible rather than hidden. A team that says “our system is fair” without naming a definition has not measured anything.

Which to choose, in practice: if a false positive is the harm (being wrongly flagged, wrongly refused, wrongly investigated) equalised odds is usually the definition you want, because it is about who bears the errors. If the score is handed to a human who will act on it as a probability, calibration matters more, because a score that means different things for different groups misleads every decision made from it.

Do it in practice

For a language-model feature, the whole exercise is often just this:

1. Tag every eval item with the slices you can observe.
2. Compute pass rate per slice, with an interval, and require at least 30 items per slice or say you cannot tell.
3. Publish the slice table in every report, worst slice first.
4. Set a floor: no slice below some stated level ships. A floor is a much better instrument than an average, because it cannot be met by improving the group that is already doing well.

That fourth point changes behaviour more than any statistical machinery. An average rewards work on the majority group; a floor forces work on the minority one.

What data can and cannot fix

Some gaps close with data: if a group is underrepresented in your retrieval corpus, few-shot examples or fine-tuning set, adding representative material helps and is the first thing to try.

Some gaps do not. If your labels themselves carry a historical bias (past decisions that were unfair, now encoded as ground truth) then a model that fits the labels perfectly reproduces the unfairness perfectly, and every metric will say it is working. This is the case where evaluation cannot save you, because your ruler is the problem. The only route through is to audit the labels themselves against a criterion that does not come from the same history, and that is a slow, human, contested piece of work.

Say so when it applies. “We measured X and cannot measure Y” is a much more useful sentence than a fairness dashboard implying you have covered it.

The free tools

Fairlearn (Microsoft, MIT) and AI Fairness 360 (IBM, Apache 2.0) both compute the standard group metrics and disparity measures out of the box, and both are free. For a language-model feature, honestly, a groupby over your per-item results gives you most of the value in five lines, which is another reason to store per-item records rather than summaries.