{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"fairness-you-can-compute","title":"Fairness as arithmetic: the definitions and what each one demands","url":"https://addaly.com/learn/machine-learning-foundations/fairness-you-can-compute","position":50,"of":92,"minutes":10,"answer":"Demographic parity, equalised odds and within-group calibration are three precise and different demands, dropping the protected column removes your ability to measure disparity without removing the disparity.","keyIdea":"Demographic parity, equalised odds and within-group calibration are three precise and different demands, dropping the protected column removes your ability to measure disparity without removing the disparity, and every one of these criteria is silent about bias already inside the labels."},"body":{"format":"markdown","text":"## Moving from principle to number\n\nEverybody agrees a model should be fair. The agreement dissolves the moment you write down what that means numerically, and it dissolves in a specific, well-understood way that is worth knowing precisely.\n\nTake a hiring screen with two groups, A and B. Here are three definitions, all reasonable, all in use.\n\n## Demographic parity\n\n> The model selects the same proportion of each group.\n\n```text\nP(predicted positive | group A) = P(predicted positive | group B)\n```\n\nThis is the criterion behind the four-fifths rule used in US employment discrimination screening: if group B's selection rate is under 80% of group A's, that is treated as evidence of adverse impact worth investigating.\n\nWhat it demands: equal outcomes at the group level, regardless of anything else.\n\nWhat it ignores: whether the groups differ in whatever the model is predicting. If they differ, enforcing parity means using different effective standards, which is exactly the objection its critics raise and exactly the point its defenders make.\n\n## Equalised odds\n\n> The model is equally accurate for each group, in both directions.\n\n```text\nP(predicted positive | actually positive, group A)\n  = P(predicted positive | actually positive, group B)      # equal recall\nP(predicted positive | actually negative, group A)\n  = P(predicted positive | actually negative, group B)      # equal false positive rate\n```\n\nThis is the criterion that COMPAS, a US recidivism prediction tool, was found to violate in the 2016 ProPublica analysis. Black defendants who did not reoffend were labelled high-risk at roughly twice the rate of white defendants who did not reoffend: unequal false positive rates.\n\n**Equal opportunity** is the weaker version requiring only equal recall, on the argument that failing to identify a qualified candidate is the harm that matters most.\n\n## Calibration within groups\n\n> A predicted 0.7 means the same thing whichever group you are in.\n\n```text\nP(actually positive | predicted 0.7, group A)\n  = P(actually positive | predicted 0.7, group B)\n```\n\nThis is what Northpointe, the company behind COMPAS, argued in response: their scores were calibrated within groups, so a score of 7 meant the same recidivism rate for everyone. That claim was also true.\n\nBoth analyses were arithmetically correct. They measured different things.\n\n## Measuring it\n\n```python\ndef group_metrics(y_true, y_pred, group):\n    for g in np.unique(group):\n        m = group == g\n        tn, fp, fn, tp = confusion_matrix(y_true[m], y_pred[m]).ravel()\n        print(f\"{g}: rate {(tp+fp)/m.sum():.3f} \"\n              f\"recall {tp/(tp+fn):.3f} fpr {fp/(fp+tn):.3f} \"\n              f\"precision {tp/(tp+fp):.3f}  n={m.sum()}\")\n```\n\nPrint `n` per group, always. A recall of 0.41 computed on 23 people is not a finding; it is a wide interval. The most common error in fairness reporting is treating a small-group number as though it had the precision of a large-group one, and the fix is to bootstrap the per-group metrics and show intervals.\n\nThe free library for this work is `fairlearn`, which computes these metrics and implements several mitigation methods.\n\n## Removing the protected attribute does not work\n\nThe first instinct is to drop the column. It fails, reliably, and the mechanism is worth understanding rather than accepting.\n\nThe protected attribute is encoded in other features. Postcode correlates with ethnicity in most countries with residential segregation. First name correlates with gender and often ethnicity. Which school, which sports, which employer, which shopping patterns. A model with 60 features has many routes to reconstruct what you deleted, and it will use them because they are predictive.\n\nWorse, removing the column removes your ability to *measure* the disparity. The model still discriminates and you can no longer see it. This is why several regulators now require collecting protected attributes specifically for auditing, held separately from the modelling data: a distinction between using an attribute for prediction and using it for measurement that is important and frequently misunderstood.\n\n## Three places to intervene\n\n**Pre-processing**: reweight or transform the training data to remove the association. Model-agnostic, and you may lose accuracy you cannot account for.\n\n**In-processing**: add a fairness constraint to the training objective. Most effective, requires a model you can modify.\n\n**Post-processing**: use different thresholds per group to equalise a chosen metric. Simple, effective, and legally fraught: in several jurisdictions applying a different decision threshold by race or sex is itself unlawful, regardless of intent. This is a real tension between what the mathematics recommends and what the law permits, it differs by country, and it is unresolved. It is also not something to resolve from a lesson: the point here is to know that the tension exists before you build something that runs into it.\n\n## The limit of all of this\n\nEvery definition above compares model outputs against recorded labels. If the labels themselves encode past discrimination (arrests rather than crimes, promotions granted by biased managers, loans approved by a biased process) then a model that is perfectly fair by every metric on this page faithfully reproduces the injustice in the labels.\n\nNo amount of arithmetic detects that. It requires knowing how the labels were made, which is a question for the people who made them."},"check":{"question":"A lender removes ethnicity from its model and reports that the model 'cannot discriminate because it never sees the attribute'. What is the flaw?","options":[{"text":"The model still needs ethnicity to satisfy demographic parity, so removing it makes the model less fair by construction.","correct":false,"tempting":"Parity is one criterion among several and is not automatically the goal; the deeper flaw is about what the remaining features encode."},{"text":"Correlated features such as postcode, name and employer reconstruct the attribute, so disparities persist, and without the column the lender can no longer measure them.","correct":true},{"text":"Removing a feature always reduces accuracy, and a less accurate model produces more errors in every group.","correct":false,"tempting":"Accuracy may fall slightly, but a uniformly less accurate model is not what makes the fairness claim wrong."},{"text":"Legally, ethnicity must be retained in credit models in most jurisdictions, so the removal itself is non-compliant.","correct":false,"tempting":"Requirements vary and often concern collection for auditing rather than use in the model, so this misstates the legal position and misses the mechanism."}]},"prev":{"slug":"cost-sensitive-decisions","title":"From a probability to a decision worth making","url":"https://addaly.com/learn/machine-learning-foundations/cost-sensitive-decisions"},"next":{"slug":"the-impossibility-result","title":"Why you cannot have all three","url":"https://addaly.com/learn/machine-learning-foundations/the-impossibility-result"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Fairness as arithmetic: the definitions and what each one demands', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/fairness-you-can-compute (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Fairness as arithmetic: the definitions and what each one demands","courseTitle":"Machine Learning, Foundations","module":{"index":5,"slug":"measuring-honestly","title":"Measuring a model honestly"},"answer":null,"keyIdea":"Demographic parity, equalised odds and within-group calibration are three precise and different demands, dropping the protected column removes your ability to measure disparity without removing the disparity, and every one of these criteria is silent about bias already inside the labels.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"moving-from-principle-to-number","runs":[{"text":"Moving from principle to number"}]},{"t":"p","runs":[{"text":"Everybody agrees a model should be fair. The agreement dissolves the moment you write down what that means numerically, and it dissolves in a specific, well-understood way that is worth knowing precisely."}]},{"t":"p","runs":[{"text":"Take a hiring screen with two groups, A and B. Here are three definitions, all reasonable, all in use."}]},{"t":"h2","id":"demographic-parity","runs":[{"text":"Demographic parity"}]},{"t":"quote","runs":[{"text":"The model selects the same proportion of each group."}]},{"t":"code","lang":"text","label":"Text","text":"P(predicted positive | group A) = P(predicted positive | group B)"},{"t":"p","runs":[{"text":"This is the criterion behind the four-fifths rule used in US employment discrimination screening: if group B’s selection rate is under 80% of group A’s, that is treated as evidence of adverse impact worth investigating."}]},{"t":"p","runs":[{"text":"What it demands: equal outcomes at the group level, regardless of anything else."}]},{"t":"p","runs":[{"text":"What it ignores: whether the groups differ in whatever the model is predicting. If they differ, enforcing parity means using different effective standards, which is exactly the objection its critics raise and exactly the point its defenders make."}]},{"t":"h2","id":"equalised-odds","runs":[{"text":"Equalised odds"}]},{"t":"quote","runs":[{"text":"The model is equally accurate for each group, in both directions."}]},{"t":"code","lang":"text","label":"Text","text":"P(predicted positive | actually positive, group A)\n  = P(predicted positive | actually positive, group B)      # equal recall\nP(predicted positive | actually negative, group A)\n  = P(predicted positive | actually negative, group B)      # equal false positive rate"},{"t":"p","runs":[{"text":"This is the criterion that COMPAS, a US recidivism prediction tool, was found to violate in the 2016 ProPublica analysis. Black defendants who did not reoffend were labelled high-risk at roughly twice the rate of white defendants who did not reoffend: unequal false positive rates."}]},{"t":"p","runs":[{"text":"Equal opportunity","b":true},{"text":" is the weaker version requiring only equal recall, on the argument that failing to identify a qualified candidate is the harm that matters most."}]},{"t":"h2","id":"calibration-within-groups","runs":[{"text":"Calibration within groups"}]},{"t":"quote","runs":[{"text":"A predicted 0.7 means the same thing whichever group you are in."}]},{"t":"code","lang":"text","label":"Text","text":"P(actually positive | predicted 0.7, group A)\n  = P(actually positive | predicted 0.7, group B)"},{"t":"p","runs":[{"text":"This is what Northpointe, the company behind COMPAS, argued in response: their scores were calibrated within groups, so a score of 7 meant the same recidivism rate for everyone. That claim was also true."}]},{"t":"p","runs":[{"text":"Both analyses were arithmetically correct. They measured different things."}]},{"t":"h2","id":"measuring-it","runs":[{"text":"Measuring it"}]},{"t":"code","lang":"python","label":"Python","text":"def group_metrics(y_true, y_pred, group):\n    for g in np.unique(group):\n        m = group == g\n        tn, fp, fn, tp = confusion_matrix(y_true[m], y_pred[m]).ravel()\n        print(f\"{g}: rate {(tp+fp)/m.sum():.3f} \"\n              f\"recall {tp/(tp+fn):.3f} fpr {fp/(fp+tn):.3f} \"\n              f\"precision {tp/(tp+fp):.3f}  n={m.sum()}\")"},{"t":"p","runs":[{"text":"Print "},{"text":"n","code":true},{"text":" per group, always. A recall of 0.41 computed on 23 people is not a finding; it is a wide interval. The most common error in fairness reporting is treating a small-group number as though it had the precision of a large-group one, and the fix is to bootstrap the per-group metrics and show intervals."}]},{"t":"p","runs":[{"text":"The free library for this work is "},{"text":"fairlearn","code":true},{"text":", which computes these metrics and implements several mitigation methods."}]},{"t":"h2","id":"removing-the-protected-attribute-does-not-work","runs":[{"text":"Removing the protected attribute does not work"}]},{"t":"p","runs":[{"text":"The first instinct is to drop the column. It fails, reliably, and the mechanism is worth understanding rather than accepting."}]},{"t":"p","runs":[{"text":"The protected attribute is encoded in other features. Postcode correlates with ethnicity in most countries with residential segregation. First name correlates with gender and often ethnicity. Which school, which sports, which employer, which shopping patterns. A model with 60 features has many routes to reconstruct what you deleted, and it will use them because they are predictive."}]},{"t":"p","runs":[{"text":"Worse, removing the column removes your ability to "},{"text":"measure","i":true},{"text":" the disparity. The model still discriminates and you can no longer see it. This is why several regulators now require collecting protected attributes specifically for auditing, held separately from the modelling data: a distinction between using an attribute for prediction and using it for measurement that is important and frequently misunderstood."}]},{"t":"h2","id":"three-places-to-intervene","runs":[{"text":"Three places to intervene"}]},{"t":"p","runs":[{"text":"Pre-processing","b":true},{"text":": reweight or transform the training data to remove the association. Model-agnostic, and you may lose accuracy you cannot account for."}]},{"t":"p","runs":[{"text":"In-processing","b":true},{"text":": add a fairness constraint to the training objective. Most effective, requires a model you can modify."}]},{"t":"p","runs":[{"text":"Post-processing","b":true},{"text":": use different thresholds per group to equalise a chosen metric. Simple, effective, and legally fraught: in several jurisdictions applying a different decision threshold by race or sex is itself unlawful, regardless of intent. This is a real tension between what the mathematics recommends and what the law permits, it differs by country, and it is unresolved. It is also not something to resolve from a lesson: the point here is to know that the tension exists before you build something that runs into it."}]},{"t":"h2","id":"the-limit-of-all-of-this","runs":[{"text":"The limit of all of this"}],"role":"limit"},{"t":"p","runs":[{"text":"Every definition above compares model outputs against recorded labels. If the labels themselves encode past discrimination (arrests rather than crimes, promotions granted by biased managers, loans approved by a biased process) then a model that is perfectly fair by every metric on this page faithfully reproduces the injustice in the labels."}]},{"t":"p","runs":[{"text":"No amount of arithmetic detects that. It requires knowing how the labels were made, which is a question for the people who made them."}]}],"check":{"question":[{"text":"A lender removes ethnicity from its model and reports that the model ‘cannot discriminate because it never sees the attribute’. What is the flaw?"}],"options":[{"text":[{"text":"The model still needs ethnicity to satisfy demographic parity, so removing it makes the model less fair by construction."}],"correct":false,"tempting":[{"text":"Parity is one criterion among several and is not automatically the goal; the deeper flaw is about what the remaining features encode."}]},{"text":[{"text":"Correlated features such as postcode, name and employer reconstruct the attribute, so disparities persist, and without the column the lender can no longer measure them."}],"correct":true,"tempting":null},{"text":[{"text":"Removing a feature always reduces accuracy, and a less accurate model produces more errors in every group."}],"correct":false,"tempting":[{"text":"Accuracy may fall slightly, but a uniformly less accurate model is not what makes the fairness claim wrong."}]},{"text":[{"text":"Legally, ethnicity must be retained in credit models in most jurisdictions, so the removal itself is non-compliant."}],"correct":false,"tempting":[{"text":"Requirements vary and often concern collection for auditing rather than use in the model, so this misstates the legal position and misses the mechanism."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"cost-sensitive-decisions","title":"From a probability to a decision worth making","minutes":8},"next":{"slug":"the-impossibility-result","title":"Why you cannot have all three","minutes":8}}}