{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"fairness-you-can-measure","title":"Fairness as arithmetic, and the choice you cannot avoid","url":"https://addaly.com/learn/evaluating-ai/fairness-you-can-measure","position":73,"of":76,"minutes":10,"answer":"Calibration and equalised odds cannot both hold when base rates differ, so fairness is a definition you choose and state, and a per-slice floor forces the work an average lets you avoid.","keyIdea":"Calibration and equalised odds cannot both hold when base rates differ, so fairness is a definition you choose and state, and a per-slice floor forces the work an average lets you avoid."},"body":{"format":"markdown","text":"## The overall number is an average over people\n\nA system at 86% is not 86% for everybody. It is 91% for one group and 68% for another, and the average conceals which. That is not an ethical observation yet; it is a measurement failure, and it has an entirely concrete fix: compute the metric per group before you compute it overall.\n\nWhat counts as a group depends on what you can legitimately observe. Often you can use non-sensitive attributes that carry most of the signal anyway:\n\n- **language** and script: this is the big one for most products, and it is usually recorded\n- **region or city tier**\n- **device and connection quality**: long, garbled inputs from a cheap keyboard on a slow line\n- **account age or tenure**\n- **channel**: web, WhatsApp, phone transcript\n\nDo not infer protected characteristics from names in order to measure them. Name-based inference is unreliable, and building the inference creates a dataset you then have to protect. Where legally collected demographic data exists, use it under the same rules as any other sensitive field. Where it does not, slice by what you have and say clearly which groups you could not check.\n\nA finding worth expecting: a support assistant at 88% in English and 61% in Bengali is not one system with a quality problem. It is two systems, one of which was never evaluated.\n\n> Figure: One average, three systems. Bars (% pass rate): Overall 86; English (60% of items) 91; Hindi (25% of items) 84; Bengali (15% of items) 68. A floor of 80 fails this release; the average passes it. A floor cannot be met by improving the group that is already doing well, which is why it changes behaviour more than any statistical machinery.\n\n## Three definitions, and they conflict\n\nWhen a system makes decisions about people (approving, flagging, prioritising, ranking) \"fair\" has to be made precise before it can be measured, and there is more than one way to do it.\n\n**Demographic parity.** The positive rate is the same across groups. If 12% of group A is approved, 12% of group B is approved. Ignores whether the underlying rates differ.\n\n**Equalised odds.** The error rates match across groups: the same true positive rate and the same false positive rate. Among people who should be approved, the same fraction are, in both groups.\n\n**Calibration within groups.** A score of 0.8 means an 80% chance of the outcome, in every group. Everyone with the same score gets the same treatment.\n\nAll three sound like fairness. Here is the part that people find surprising the first time.\n\n## You cannot have all three\n\nThis is a mathematical result, not a matter of effort or engineering skill. Kleinberg, Mullainathan and Raghavan (2016) and Chouldechova (2016) showed independently that unless the base rates are identical across groups, or the classifier is perfect, calibration and equalised odds cannot both hold.\n\nA sketch of why. Precision, recall and base rate are linked: `FPR = (1 - precision)/precision × (base rate)/(1 - base rate) × TPR`, roughly. Fix the base rates unequal and hold two of the fairness properties, and the third is forced to be violated. There is no clever model that escapes it.\n\nThe consequence is practical and it is the point of this lesson. **Fairness is a choice you state, not a box you tick.** You pick the definition that matches the harm you most want to avoid, you write down which one you chose and why, and you report the others so the trade-off is visible rather than hidden. A team that says \"our system is fair\" without naming a definition has not measured anything.\n\nWhich to choose, in practice: if a false positive is the harm (being wrongly flagged, wrongly refused, wrongly investigated) equalised odds is usually the definition you want, because it is about who bears the errors. If the score is handed to a human who will act on it as a probability, calibration matters more, because a score that means different things for different groups misleads every decision made from it.\n\n## Do it in practice\n\nFor a language-model feature, the whole exercise is often just this:\n\n1. Tag every eval item with the slices you can observe.\n2. Compute pass rate per slice, with an interval, and require at least 30 items per slice or say you cannot tell.\n3. Publish the slice table in every report, worst slice first.\n4. Set a floor: no slice below some stated level ships. A floor is a much better instrument than an average, because it cannot be met by improving the group that is already doing well.\n\nThat fourth point changes behaviour more than any statistical machinery. An average rewards work on the majority group; a floor forces work on the minority one.\n\n## What data can and cannot fix\n\nSome gaps close with data: if a group is underrepresented in your retrieval corpus, few-shot examples or fine-tuning set, adding representative material helps and is the first thing to try.\n\nSome gaps do not. If your labels themselves carry a historical bias (past decisions that were unfair, now encoded as ground truth) then a model that fits the labels perfectly reproduces the unfairness perfectly, and every metric will say it is working. This is the case where evaluation cannot save you, because your ruler is the problem. The only route through is to audit the labels themselves against a criterion that does not come from the same history, and that is a slow, human, contested piece of work.\n\nSay so when it applies. \"We measured X and cannot measure Y\" is a much more useful sentence than a fairness dashboard implying you have covered it.\n\n## The free tools\n\n**Fairlearn** (Microsoft, MIT) and **AI Fairness 360** (IBM, Apache 2.0) both compute the standard group metrics and disparity measures out of the box, and both are free. For a language-model feature, honestly, a `groupby` over your per-item results gives you most of the value in five lines, which is another reason to store per-item records rather than summaries."},"check":{"question":"A lending model is calibrated within both groups (a score of 0.7 means a 70% repayment rate for each), but has a higher false positive rate for group B. A reviewer says this proves the model is biased and calibration should be kept while equalising the error rates. What is the accurate response?","options":[{"text":"With unequal base rates, calibration and equalised odds cannot both hold, so the team must choose which to prioritise and state the choice and its cost.","correct":true},{"text":"The disparity is a data problem; adding more training examples from group B will bring both properties into line.","correct":false,"tempting":"Treats an impossibility result as a sample-size problem, when the conflict persists at any amount of data whenever base rates differ."},{"text":"Calibration is the stronger property, so a calibrated model is fair by definition and the false positive gap is not a fairness issue.","correct":false,"tempting":"Elevates one definition to the definition, which is exactly the move that hides the trade-off rather than resolving it."},{"text":"The thresholds should be adjusted per group until the false positive rates match, which resolves the conflict.","correct":false,"tempting":"Proposes a real technique for equalising odds without acknowledging that it necessarily breaks the calibration the reviewer wants to keep."}]},"prev":{"slug":"when-the-eval-set-goes-stale","title":"Refreshing a set without breaking every comparison","url":"https://addaly.com/learn/evaluating-ai/when-the-eval-set-goes-stale"},"next":{"slug":"incident-to-eval-item","title":"Turning a complaint into a permanent test","url":"https://addaly.com/learn/evaluating-ai/incident-to-eval-item"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Fairness as arithmetic, and the choice you cannot avoid', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/fairness-you-can-measure (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Fairness as arithmetic, and the choice you cannot avoid","courseTitle":"Knowing If It Works","module":{"index":8,"slug":"living-with-it","title":"Living with it"},"answer":null,"keyIdea":"Calibration and equalised odds cannot both hold when base rates differ, so fairness is a definition you choose and state, and a per-slice floor forces the work an average lets you avoid.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% pass rate","title":"One average, three systems","series":[{"label":"Overall","value":86},{"label":"English (60% of items)","value":91},{"label":"Hindi (25% of items)","value":84},{"label":"Bengali (15% of items)","value":68,"highlight":true}],"caption":"A floor of 80 fails this release; the average passes it. A floor cannot be met by improving the group that is already doing well, which is why it changes behaviour more than any statistical machinery."},"alt":"Figure: One average, three systems. Bars (% pass rate): Overall 86; English (60% of items) 91; Hindi (25% of items) 84; Bengali (15% of items) 68. A floor of 80 fails this release; the average passes it. A floor cannot be met by improving the group that is already doing well, which is why it changes behaviour more than any statistical machinery."}},"blocks":[{"t":"h2","id":"the-overall-number-is-an-average-over-people","runs":[{"text":"The overall number is an average over people"}]},{"t":"p","runs":[{"text":"A system at 86% is not 86% for everybody. It is 91% for one group and 68% for another, and the average conceals which. That is not an ethical observation yet; it is a measurement failure, and it has an entirely concrete fix: compute the metric per group before you compute it overall."}]},{"t":"p","runs":[{"text":"What counts as a group depends on what you can legitimately observe. Often you can use non-sensitive attributes that carry most of the signal anyway:"}]},{"t":"ul","items":[[{"text":"language","b":true},{"text":" and script: this is the big one for most products, and it is usually recorded"}],[{"text":"region or city tier","b":true}],[{"text":"device and connection quality","b":true},{"text":": long, garbled inputs from a cheap keyboard on a slow line"}],[{"text":"account age or tenure","b":true}],[{"text":"channel","b":true},{"text":": web, WhatsApp, phone transcript"}]]},{"t":"p","runs":[{"text":"Do not infer protected characteristics from names in order to measure them. Name-based inference is unreliable, and building the inference creates a dataset you then have to protect. Where legally collected demographic data exists, use it under the same rules as any other sensitive field. Where it does not, slice by what you have and say clearly which groups you could not check."}]},{"t":"p","runs":[{"text":"A finding worth expecting: a support assistant at 88% in English and 61% in Bengali is not one system with a quality problem. It is two systems, one of which was never evaluated."}]},{"t":"h2","id":"three-definitions-and-they-conflict","runs":[{"text":"Three definitions, and they conflict"}]},{"t":"p","runs":[{"text":"When a system makes decisions about people (approving, flagging, prioritising, ranking) “fair” has to be made precise before it can be measured, and there is more than one way to do it."}]},{"t":"p","runs":[{"text":"Demographic parity.","b":true},{"text":" The positive rate is the same across groups. If 12% of group A is approved, 12% of group B is approved. Ignores whether the underlying rates differ."}]},{"t":"p","runs":[{"text":"Equalised odds.","b":true},{"text":" The error rates match across groups: the same true positive rate and the same false positive rate. Among people who should be approved, the same fraction are, in both groups."}]},{"t":"p","runs":[{"text":"Calibration within groups.","b":true},{"text":" A score of 0.8 means an 80% chance of the outcome, in every group. Everyone with the same score gets the same treatment."}]},{"t":"p","runs":[{"text":"All three sound like fairness. Here is the part that people find surprising the first time."}]},{"t":"h2","id":"you-cannot-have-all-three","runs":[{"text":"You cannot have all three"}]},{"t":"p","runs":[{"text":"This is a mathematical result, not a matter of effort or engineering skill. Kleinberg, Mullainathan and Raghavan (2016) and Chouldechova (2016) showed independently that unless the base rates are identical across groups, or the classifier is perfect, calibration and equalised odds cannot both hold."}]},{"t":"p","runs":[{"text":"A sketch of why. Precision, recall and base rate are linked: "},{"text":"FPR = (1 - precision)/precision × (base rate)/(1 - base rate) × TPR","code":true},{"text":", roughly. Fix the base rates unequal and hold two of the fairness properties, and the third is forced to be violated. There is no clever model that escapes it."}]},{"t":"p","runs":[{"text":"The consequence is practical and it is the point of this lesson. "},{"text":"Fairness is a choice you state, not a box you tick.","b":true},{"text":" You pick the definition that matches the harm you most want to avoid, you write down which one you chose and why, and you report the others so the trade-off is visible rather than hidden. A team that says “our system is fair” without naming a definition has not measured anything."}]},{"t":"p","runs":[{"text":"Which to choose, in practice: if a false positive is the harm (being wrongly flagged, wrongly refused, wrongly investigated) equalised odds is usually the definition you want, because it is about who bears the errors. If the score is handed to a human who will act on it as a probability, calibration matters more, because a score that means different things for different groups misleads every decision made from it."}]},{"t":"h2","id":"do-it-in-practice","runs":[{"text":"Do it in practice"}]},{"t":"p","runs":[{"text":"For a language-model feature, the whole exercise is often just this:"}]},{"t":"ol","start":1,"items":[[{"text":"Tag every eval item with the slices you can observe."}],[{"text":"Compute pass rate per slice, with an interval, and require at least 30 items per slice or say you cannot tell."}],[{"text":"Publish the slice table in every report, worst slice first."}],[{"text":"Set a floor: no slice below some stated level ships. A floor is a much better instrument than an average, because it cannot be met by improving the group that is already doing well."}]]},{"t":"p","runs":[{"text":"That fourth point changes behaviour more than any statistical machinery. An average rewards work on the majority group; a floor forces work on the minority one."}]},{"t":"h2","id":"what-data-can-and-cannot-fix","runs":[{"text":"What data can and cannot fix"}]},{"t":"p","runs":[{"text":"Some gaps close with data: if a group is underrepresented in your retrieval corpus, few-shot examples or fine-tuning set, adding representative material helps and is the first thing to try."}]},{"t":"p","runs":[{"text":"Some gaps do not. If your labels themselves carry a historical bias (past decisions that were unfair, now encoded as ground truth) then a model that fits the labels perfectly reproduces the unfairness perfectly, and every metric will say it is working. This is the case where evaluation cannot save you, because your ruler is the problem. The only route through is to audit the labels themselves against a criterion that does not come from the same history, and that is a slow, human, contested piece of work."}]},{"t":"p","runs":[{"text":"Say so when it applies. “We measured X and cannot measure Y” is a much more useful sentence than a fairness dashboard implying you have covered it."}]},{"t":"h2","id":"the-free-tools","runs":[{"text":"The free tools"}]},{"t":"p","runs":[{"text":"Fairlearn","b":true},{"text":" (Microsoft, MIT) and "},{"text":"AI Fairness 360","b":true},{"text":" (IBM, Apache 2.0) both compute the standard group metrics and disparity measures out of the box, and both are free. For a language-model feature, honestly, a "},{"text":"groupby","code":true},{"text":" over your per-item results gives you most of the value in five lines, which is another reason to store per-item records rather than summaries."}]}],"check":{"question":[{"text":"A lending model is calibrated within both groups (a score of 0.7 means a 70% repayment rate for each), but has a higher false positive rate for group B. A reviewer says this proves the model is biased and calibration should be kept while equalising the error rates. What is the accurate response?"}],"options":[{"text":[{"text":"With unequal base rates, calibration and equalised odds cannot both hold, so the team must choose which to prioritise and state the choice and its cost."}],"correct":true,"tempting":null},{"text":[{"text":"The disparity is a data problem; adding more training examples from group B will bring both properties into line."}],"correct":false,"tempting":[{"text":"Treats an impossibility result as a sample-size problem, when the conflict persists at any amount of data whenever base rates differ."}]},{"text":[{"text":"Calibration is the stronger property, so a calibrated model is fair by definition and the false positive gap is not a fairness issue."}],"correct":false,"tempting":[{"text":"Elevates one definition to the definition, which is exactly the move that hides the trade-off rather than resolving it."}]},{"text":[{"text":"The thresholds should be adjusted per group until the false positive rates match, which resolves the conflict."}],"correct":false,"tempting":[{"text":"Proposes a real technique for equalising odds without acknowledging that it necessarily breaks the calibration the reviewer wants to keep."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"when-the-eval-set-goes-stale","title":"Refreshing a set without breaking every comparison","minutes":8},"next":{"slug":"incident-to-eval-item","title":"Turning a complaint into a permanent test","minutes":8}}}