{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"where-bias-comes-from","title":"Where bias comes from","url":"https://addaly.com/learn/ai-and-you/where-bias-comes-from","position":10,"of":73,"minutes":9,"answer":"You cannot delete bias by deleting the sensitive field; other columns rebuild it and you lose the ability to measure.","keyIdea":"You cannot delete bias by deleting the sensitive field; other columns rebuild it and you lose the ability to measure."},"body":{"format":"markdown","text":"\"The AI is biased\" is true and nearly useless, because it does not say where the bias entered. There are at least four doors, and they need different responses.\n\n## Door one: the world was unequal, and the data recorded it\n\nA model trained on twenty years of hiring decisions at a company where engineering was 90% men learns that engineers look like men. It is describing the past accurately. The problem is that you asked it to predict the future.\n\nThis is the hardest kind, because the data is not wrong. Amazon abandoned an internal CV-screening tool in 2018 after finding it downgraded CVs containing the word \"women's\", as in \"women's chess club captain\". Nobody put that rule in. It was learned from what past success had looked like.\n\n## Door two: who is in the data at all\n\nA skin cancer detector trained mostly on light skin does worse on dark skin. A voice assistant trained mostly on American and British English does worse on Nigerian, Scottish, Telugu-accented or Filipino English. Speech recognition studies have found error rates roughly twice as high for Black American speakers as for white speakers on the same words.\n\nThis is a sampling problem, and it is the most fixable of the four. You can go and collect the missing data. It costs money, which is why it often does not happen.\n\n## Door three: what you chose to measure\n\nA famous US healthcare algorithm ranked patients for extra care using **past healthcare spending** as a stand-in for **how sick you are**. But Black patients had historically had less money spent on them for the same illness, so at any given level of real sickness they scored as healthier. Researchers found that at a given risk score, Black patients were considerably sicker than white patients. Fixing the target variable (predicting illness directly rather than cost) cut the gap sharply.\n\nSomeone picked a convenient proxy. That is the whole story, and it repeats constantly, because the thing you care about is usually hard to measure and something correlated with it is sitting right there in the database.\n\n## Door four: how the output gets used\n\nA model outputs a probability. A human turns it into a yes or no by picking a cutoff. Where you put the cutoff decides who gets hurt. Set it to catch more fraud and you falsely accuse more innocent people. Set it to accuse fewer innocents and you miss more fraud. There is no setting that avoids the choice, and the choice is a policy decision, not a technical one, however much it looks like a number in a config file.\n\n## Why \"just remove the bias\" is not a plan\n\nThe obvious fix is to delete the sensitive field. Do not ask for gender, caste, race, religion. Then the model cannot discriminate.\n\nIt can. Postcode encodes race and caste in most countries. Name encodes religion and ethnicity. School attended encodes class. Height and weight partly encode sex. A model with enough other variables reconstructs the one you removed: this is called *proxy discrimination*, and removing the protected field mainly removes your ability to *detect* the problem.\n\nThe deeper trouble is that \"fair\" has several definitions that are mathematically incompatible. Should the same score mean the same risk for every group? Should the error rates be equal across groups? Should the acceptance rates be equal? Except in unusual cases you cannot have all three at once. This is a proven result, not an engineering shortfall.\n\nSo bias work is not removal. It is: measure the gaps, decide which fairness definition your situation demands, accept what that costs you elsewhere, and write down why. That last part (the writing down) is what makes it reviewable by someone who is not you."},"check":{"question":"A lending team removes caste, religion and gender from its model inputs and reports that the model is now fair. Six months later, approval rates still differ sharply by community. What best explains this?","options":[{"text":"Remaining inputs such as postcode, name and school stand in for the removed ones","correct":true},{"text":"Someone must have hard-coded a bias into the model when it was being built","correct":false,"tempting":"Persistent unfairness feels as if it needs an author. Learned correlations produce the same outcome with no rule anywhere."},{"text":"The model needs more training data before the gap between communities evens out","correct":false,"tempting":"More data fixes sampling gaps. More data drawn from the same unequal world strengthens the proxies instead of diluting them."},{"text":"Equalising approval rates as well would make the model fair on every definition","correct":false,"tempting":"Equal outcomes sound like the complete answer. The main statistical definitions of fairness conflict, so meeting one usually breaks another."}],"why":"Removing a column removes a label, not the pattern. Correlated inputs rebuild it."},"prev":{"slug":"risk-by-consequence","title":"Triage: sorting risk by what it costs","url":"https://addaly.com/learn/ai-and-you/risk-by-consequence"},"next":{"slug":"the-confusion-matrix-by-group","title":"Counting the four ways to be right and wrong","url":"https://addaly.com/learn/ai-and-you/the-confusion-matrix-by-group"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where bias comes from', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/where-bias-comes-from (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where bias comes from","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":2,"slug":"bias-and-fairness","title":"Bias, and how to measure it"},"answer":null,"keyIdea":"You cannot delete bias by deleting the sensitive field; other columns rebuild it and you lose the ability to measure.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"p","runs":[{"text":"“The AI is biased” is true and nearly useless, because it does not say where the bias entered. There are at least four doors, and they need different responses."}],"lede":true},{"t":"h2","id":"door-one-the-world-was-unequal-and-the-data-recorded-it","runs":[{"text":"Door one: the world was unequal, and the data recorded it"}]},{"t":"p","runs":[{"text":"A model trained on twenty years of hiring decisions at a company where engineering was 90% men learns that engineers look like men. It is describing the past accurately. The problem is that you asked it to predict the future."}]},{"t":"p","runs":[{"text":"This is the hardest kind, because the data is not wrong. Amazon abandoned an internal CV-screening tool in 2018 after finding it downgraded CVs containing the word “women’s”, as in “women’s chess club captain”. Nobody put that rule in. It was learned from what past success had looked like."}]},{"t":"h2","id":"door-two-who-is-in-the-data-at-all","runs":[{"text":"Door two: who is in the data at all"}]},{"t":"p","runs":[{"text":"A skin cancer detector trained mostly on light skin does worse on dark skin. A voice assistant trained mostly on American and British English does worse on Nigerian, Scottish, Telugu-accented or Filipino English. Speech recognition studies have found error rates roughly twice as high for Black American speakers as for white speakers on the same words."}]},{"t":"p","runs":[{"text":"This is a sampling problem, and it is the most fixable of the four. You can go and collect the missing data. It costs money, which is why it often does not happen."}]},{"t":"h2","id":"door-three-what-you-chose-to-measure","runs":[{"text":"Door three: what you chose to measure"}]},{"t":"p","runs":[{"text":"A famous US healthcare algorithm ranked patients for extra care using "},{"text":"past healthcare spending","b":true},{"text":" as a stand-in for "},{"text":"how sick you are","b":true},{"text":". But Black patients had historically had less money spent on them for the same illness, so at any given level of real sickness they scored as healthier. Researchers found that at a given risk score, Black patients were considerably sicker than white patients. Fixing the target variable (predicting illness directly rather than cost) cut the gap sharply."}]},{"t":"p","runs":[{"text":"Someone picked a convenient proxy. That is the whole story, and it repeats constantly, because the thing you care about is usually hard to measure and something correlated with it is sitting right there in the database."}]},{"t":"h2","id":"door-four-how-the-output-gets-used","runs":[{"text":"Door four: how the output gets used"}]},{"t":"p","runs":[{"text":"A model outputs a probability. A human turns it into a yes or no by picking a cutoff. Where you put the cutoff decides who gets hurt. Set it to catch more fraud and you falsely accuse more innocent people. Set it to accuse fewer innocents and you miss more fraud. There is no setting that avoids the choice, and the choice is a policy decision, not a technical one, however much it looks like a number in a config file."}]},{"t":"h2","id":"why-just-remove-the-bias-is-not-a-plan","runs":[{"text":"Why “just remove the bias” is not a plan"}]},{"t":"p","runs":[{"text":"The obvious fix is to delete the sensitive field. Do not ask for gender, caste, race, religion. Then the model cannot discriminate."}]},{"t":"p","runs":[{"text":"It can. Postcode encodes race and caste in most countries. Name encodes religion and ethnicity. School attended encodes class. Height and weight partly encode sex. A model with enough other variables reconstructs the one you removed: this is called "},{"text":"proxy discrimination","i":true},{"text":", and removing the protected field mainly removes your ability to "},{"text":"detect","i":true},{"text":" the problem."}]},{"t":"p","runs":[{"text":"The deeper trouble is that “fair” has several definitions that are mathematically incompatible. Should the same score mean the same risk for every group? Should the error rates be equal across groups? Should the acceptance rates be equal? Except in unusual cases you cannot have all three at once. This is a proven result, not an engineering shortfall."}]},{"t":"p","runs":[{"text":"So bias work is not removal. It is: measure the gaps, decide which fairness definition your situation demands, accept what that costs you elsewhere, and write down why. That last part (the writing down) is what makes it reviewable by someone who is not you."}]}],"check":{"question":[{"text":"A lending team removes caste, religion and gender from its model inputs and reports that the model is now fair. Six months later, approval rates still differ sharply by community. What best explains this?"}],"options":[{"text":[{"text":"Remaining inputs such as postcode, name and school stand in for the removed ones"}],"correct":true,"tempting":null},{"text":[{"text":"Someone must have hard-coded a bias into the model when it was being built"}],"correct":false,"tempting":[{"text":"Persistent unfairness feels as if it needs an author. Learned correlations produce the same outcome with no rule anywhere."}]},{"text":[{"text":"The model needs more training data before the gap between communities evens out"}],"correct":false,"tempting":[{"text":"More data fixes sampling gaps. More data drawn from the same unequal world strengthens the proxies instead of diluting them."}]},{"text":[{"text":"Equalising approval rates as well would make the model fair on every definition"}],"correct":false,"tempting":[{"text":"Equal outcomes sound like the complete answer. The main statistical definitions of fairness conflict, so meeting one usually breaks another."}]}],"order":[2,1,3,0],"why":[{"text":"Removing a column removes a label, not the pattern. Correlated inputs rebuild it."}]},"prev":{"slug":"risk-by-consequence","title":"Triage: sorting risk by what it costs","minutes":4},"next":{"slug":"the-confusion-matrix-by-group","title":"Counting the four ways to be right and wrong","minutes":9}}}