Fairness as arithmetic: the definitions and what each one demands

Moving from principle to number

Everybody agrees a model should be fair. The agreement dissolves the moment you write down what that means numerically, and it dissolves in a specific, well-understood way that is worth knowing precisely.

Take a hiring screen with two groups, A and B. Here are three definitions, all reasonable, all in use.

Demographic parity

The model selects the same proportion of each group.

P(predicted positive | group A) = P(predicted positive | group B)

This is the criterion behind the four-fifths rule used in US employment discrimination screening: if group B’s selection rate is under 80% of group A’s, that is treated as evidence of adverse impact worth investigating.

What it demands: equal outcomes at the group level, regardless of anything else.

What it ignores: whether the groups differ in whatever the model is predicting. If they differ, enforcing parity means using different effective standards, which is exactly the objection its critics raise and exactly the point its defenders make.

Equalised odds

The model is equally accurate for each group, in both directions.

P(predicted positive | actually positive, group A)
  = P(predicted positive | actually positive, group B)      # equal recall
P(predicted positive | actually negative, group A)
  = P(predicted positive | actually negative, group B)      # equal false positive rate

This is the criterion that COMPAS, a US recidivism prediction tool, was found to violate in the 2016 ProPublica analysis. Black defendants who did not reoffend were labelled high-risk at roughly twice the rate of white defendants who did not reoffend: unequal false positive rates.

Equal opportunity is the weaker version requiring only equal recall, on the argument that failing to identify a qualified candidate is the harm that matters most.

Calibration within groups

A predicted 0.7 means the same thing whichever group you are in.

P(actually positive | predicted 0.7, group A)
  = P(actually positive | predicted 0.7, group B)

This is what Northpointe, the company behind COMPAS, argued in response: their scores were calibrated within groups, so a score of 7 meant the same recidivism rate for everyone. That claim was also true.

Both analyses were arithmetically correct. They measured different things.

Measuring it

def group_metrics(y_true, y_pred, group):
    for g in np.unique(group):
        m = group == g
        tn, fp, fn, tp = confusion_matrix(y_true[m], y_pred[m]).ravel()
        print(f"{g}: rate {(tp+fp)/m.sum():.3f} "
              f"recall {tp/(tp+fn):.3f} fpr {fp/(fp+tn):.3f} "
              f"precision {tp/(tp+fp):.3f}  n={m.sum()}")

Print n per group, always. A recall of 0.41 computed on 23 people is not a finding; it is a wide interval. The most common error in fairness reporting is treating a small-group number as though it had the precision of a large-group one, and the fix is to bootstrap the per-group metrics and show intervals.

The free library for this work is fairlearn, which computes these metrics and implements several mitigation methods.

Removing the protected attribute does not work

The first instinct is to drop the column. It fails, reliably, and the mechanism is worth understanding rather than accepting.

The protected attribute is encoded in other features. Postcode correlates with ethnicity in most countries with residential segregation. First name correlates with gender and often ethnicity. Which school, which sports, which employer, which shopping patterns. A model with 60 features has many routes to reconstruct what you deleted, and it will use them because they are predictive.

Worse, removing the column removes your ability to measure the disparity. The model still discriminates and you can no longer see it. This is why several regulators now require collecting protected attributes specifically for auditing, held separately from the modelling data: a distinction between using an attribute for prediction and using it for measurement that is important and frequently misunderstood.

Three places to intervene

Pre-processing: reweight or transform the training data to remove the association. Model-agnostic, and you may lose accuracy you cannot account for.

In-processing: add a fairness constraint to the training objective. Most effective, requires a model you can modify.

Post-processing: use different thresholds per group to equalise a chosen metric. Simple, effective, and legally fraught: in several jurisdictions applying a different decision threshold by race or sex is itself unlawful, regardless of intent. This is a real tension between what the mathematics recommends and what the law permits, it differs by country, and it is unresolved. It is also not something to resolve from a lesson: the point here is to know that the tension exists before you build something that runs into it.

The limit of all of this

Every definition above compares model outputs against recorded labels. If the labels themselves encode past discrimination (arrests rather than crimes, promotions granted by biased managers, loans approved by a biased process) then a model that is perfectly fair by every metric on this page faithfully reproduces the injustice in the labels.

No amount of arithmetic detects that. It requires knowing how the labels were made, which is a question for the people who made them.