{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"bias-in-the-output","title":"Bias, and the difference between fixing it and hiding it","url":"https://addaly.com/learn/ai-actually-explained/bias-in-the-output","position":36,"of":92,"minutes":10,"answer":"Safety training removed overt bias while covert associations triggered by dialect and names survived, which shows that fixing what you measure moves the problem into what you do not measure.","keyIdea":"Safety training removed overt bias while covert associations triggered by dialect and names survived, which shows that fixing what you measure moves the problem into what you do not measure."},"body":{"format":"markdown","text":"## Four doors it comes in through\n\nBias in a model is not one thing arriving from one place. It enters at four separate points, and they need different responses.\n\n**The corpus.** Text written by people carries the assumptions of the people who wrote it, and the web over-represents some populations enormously. Occupations, nationalities and names all carry statistical associations that the model absorbs as ordinary patterns.\n\n**The labels.** Every judgement in fine-tuning was made by a person following a guideline. Who they were and what the guideline said is baked in.\n\n**The filters.** Quality classifiers decide what counts as good text. There is documented evidence that some corpus filters remove text written in non-standard dialects at higher rates, which means those voices are thinned before training begins.\n\n**The preference tuning.** Raters preferred some answers over others, and their preferences are not neutral on questions about people.\n\n> Figure: Four doors bias comes in through, in the order it arrives. Steps: 1. The corpus (Text written by people, over-representing some populations enormously. Occupations, names and nationalities carry statistical associations); 2. The quality filters (A classifier decides what counts as good text. Some filters remove non-standard dialects at higher rates, thinning those voices before training starts); 3. The demonstrations (Every fine-tuning example was written by a person following a guideline. Who they were is baked in); 4. The preference tuning (Raters preferred some answers to others, and their preferences are not neutral on questions about people); 5. The deployed model (Overt statements can be patched. The statistical residue is distributed across billions of weights and cannot be removed by adding a rule). Each stage needs a different response, and none of them is a rule somebody wrote. Fixing what you measure at the last stage moves the association into a channel nobody is testing, which is exactly what the 2024 dialect study found.\n\n## What it looks like in practice\n\nThree measured examples, not anecdotes.\n\n**Names on CVs.** A 2024 Bloomberg investigation had a leading model rank identical CVs that differed only in the name attached. Names associated with different racial groups received systematically different rankings across thousands of trials, and the pattern varied by job. Nobody wrote a rule about names. The association came from the text.\n\n**Occupational gender in translation.** Translating from a language without grammatical gender into one with it forces a choice, and the choice follows the training text: engineers male, nurses female. Providers have patched short phrases; longer passages still show it.\n\n**Dialect.** This is the most important of the three. A 2024 study published in *Nature* examined how models responded to text written in African American English compared with Standardised American English expressing the same content. The models produced markedly more negative associations for the dialect text, and this appeared in consequential judgements: hypothetical employability and, in one experimental setting, severity of sentence.\n\nThe finding that matters: the same models showed *little* overt bias when asked directly about race. Safety training had successfully removed the explicit expression. The covert association, triggered by dialect rather than by any stated group, was not removed, and in some measures it was strongest in the models that had received the most safety training.\n\n## The lesson underneath\n\n**Fixing what you measure does not fix what you do not measure.**\n\nThe explicit tests were run, the model passed them, and the association moved to a channel nobody was testing. This is a general property of optimisation against a metric, and it will recur throughout your dealings with these systems.\n\nIt also explains a frustrating experience many people report: a model that is scrupulously even-handed when the subject is named, and subtly different in tone, thoroughness or generosity when the same subject is merely implied by a name, a dialect, a place or a phrasing.\n\n## What can and cannot be fixed\n\nHonestly:\n\n**Fixable.** Overt statements. Refusal to produce slurs. Balanced treatment when a group is explicitly named. Gender alternatives for short translations. These have improved, and dismissing that improvement is unfair to the work behind it.\n\n**Not fixable by the same means.** The statistical residue of a corpus: the fact that in the text, engineers were mostly men and certain names appeared in certain contexts. You cannot remove it by adding a rule, because it is not represented as a rule. It is distributed across billions of weights.\n\nThere are partial approaches: rebalancing the corpus, targeted fine-tuning, filtering outputs, and running audits by group. None of them is complete, and each has a cost: most obviously that a model heavily patched on one axis often becomes strange on adjacent ones.\n\n## What to actually do\n\n**Do not use a general-purpose model to make decisions about people.** Not shortlisting, not scoring, not triaging applicants. Not because it is worse than a human panel on every axis (that is arguable) but because it applies whatever it has at scale, invisibly, with no explanation and no appeal.\n\n**If you use it to assist, blind the input.** Strip names, addresses, schools, photographs and anything that signals a group before the model sees it. This is simple, it is effective against the specific mechanism above, and it is worth doing for human reviewers too.\n\n**Audit by group.** If a system is used at scale, measure its outputs separately for each group affected. An overall accuracy figure hides exactly the failure you are looking for.\n\n**Watch for the covert channel.** When you test for bias, test with the signal implicit (a name, a dialect, a location) not only with it stated. Testing only the explicit case measures the patch, not the model."},"check":{"question":"A model shows no measurable bias when asked directly about candidates from different ethnic groups, but rates identical applications differently when only the applicant's name changes. What does this pattern indicate?","options":[{"text":"The bias is in the evaluation method rather than the model, since two tests of the same property should agree.","correct":false,"tempting":"Assumes both tests probe the same channel; the explicit and implicit channels are demonstrably separable."},{"text":"The model is applying a deliberate rule about names that was introduced during safety tuning.","correct":false,"tempting":"Attributes to an explicit rule what arises from distributed statistical association in the corpus."},{"text":"Explicit bias was trained out because it was the thing being measured, while the association carried by names was never targeted and remains.","correct":true},{"text":"Name-based differences reflect genuine variation in the training CVs, so the model is reporting a real pattern.","correct":false,"tempting":"Treats a learned social association as a valid signal, which is precisely the reasoning that makes such systems harmful at scale."}]},"prev":{"slug":"the-cut-off-and-the-clock","title":"It has no clock, and its knowledge thins before it stops","url":"https://addaly.com/learn/ai-actually-explained/the-cut-off-and-the-clock"},"next":{"slug":"the-reasoning-models","title":"Models that think out loud, and what that does not prove","url":"https://addaly.com/learn/ai-actually-explained/the-reasoning-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Bias, and the difference between fixing it and hiding it', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/bias-in-the-output (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Bias, and the difference between fixing it and hiding it","courseTitle":"AI, Actually Explained","module":{"index":4,"slug":"why-it-goes-wrong","title":"Why it gets things wrong"},"answer":null,"keyIdea":"Safety training removed overt bias while covert associations triggered by dialect and names survived, which shows that fixing what you measure moves the problem into what you do not measure.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"The corpus","detail":"Text written by people, over-representing some populations enormously. Occupations, names and nationalities carry statistical associations"},{"label":"The quality filters","detail":"A classifier decides what counts as good text. Some filters remove non-standard dialects at higher rates, thinning those voices before training starts"},{"label":"The demonstrations","detail":"Every fine-tuning example was written by a person following a guideline. Who they were is baked in"},{"label":"The preference tuning","detail":"Raters preferred some answers to others, and their preferences are not neutral on questions about people"},{"label":"The deployed model","detail":"Overt statements can be patched. The statistical residue is distributed across billions of weights and cannot be removed by adding a rule"}],"title":"Four doors bias comes in through, in the order it arrives","caption":"Each stage needs a different response, and none of them is a rule somebody wrote. Fixing what you measure at the last stage moves the association into a channel nobody is testing, which is exactly what the 2024 dialect study found."},"alt":"Figure: Four doors bias comes in through, in the order it arrives. Steps: 1. The corpus (Text written by people, over-representing some populations enormously. Occupations, names and nationalities carry statistical associations); 2. The quality filters (A classifier decides what counts as good text. Some filters remove non-standard dialects at higher rates, thinning those voices before training starts); 3. The demonstrations (Every fine-tuning example was written by a person following a guideline. Who they were is baked in); 4. The preference tuning (Raters preferred some answers to others, and their preferences are not neutral on questions about people); 5. The deployed model (Overt statements can be patched. The statistical residue is distributed across billions of weights and cannot be removed by adding a rule). Each stage needs a different response, and none of them is a rule somebody wrote. Fixing what you measure at the last stage moves the association into a channel nobody is testing, which is exactly what the 2024 dialect study found."}},"blocks":[{"t":"h2","id":"four-doors-it-comes-in-through","runs":[{"text":"Four doors it comes in through"}]},{"t":"p","runs":[{"text":"Bias in a model is not one thing arriving from one place. It enters at four separate points, and they need different responses."}]},{"t":"p","runs":[{"text":"The corpus.","b":true},{"text":" Text written by people carries the assumptions of the people who wrote it, and the web over-represents some populations enormously. Occupations, nationalities and names all carry statistical associations that the model absorbs as ordinary patterns."}]},{"t":"p","runs":[{"text":"The labels.","b":true},{"text":" Every judgement in fine-tuning was made by a person following a guideline. Who they were and what the guideline said is baked in."}]},{"t":"p","runs":[{"text":"The filters.","b":true},{"text":" Quality classifiers decide what counts as good text. There is documented evidence that some corpus filters remove text written in non-standard dialects at higher rates, which means those voices are thinned before training begins."}]},{"t":"p","runs":[{"text":"The preference tuning.","b":true},{"text":" Raters preferred some answers over others, and their preferences are not neutral on questions about people."}]},{"t":"h2","id":"what-it-looks-like-in-practice","runs":[{"text":"What it looks like in practice"}]},{"t":"p","runs":[{"text":"Three measured examples, not anecdotes."}]},{"t":"p","runs":[{"text":"Names on CVs.","b":true},{"text":" A 2024 Bloomberg investigation had a leading model rank identical CVs that differed only in the name attached. Names associated with different racial groups received systematically different rankings across thousands of trials, and the pattern varied by job. Nobody wrote a rule about names. The association came from the text."}]},{"t":"p","runs":[{"text":"Occupational gender in translation.","b":true},{"text":" Translating from a language without grammatical gender into one with it forces a choice, and the choice follows the training text: engineers male, nurses female. Providers have patched short phrases; longer passages still show it."}]},{"t":"p","runs":[{"text":"Dialect.","b":true},{"text":" This is the most important of the three. A 2024 study published in "},{"text":"Nature","i":true},{"text":" examined how models responded to text written in African American English compared with Standardised American English expressing the same content. The models produced markedly more negative associations for the dialect text, and this appeared in consequential judgements: hypothetical employability and, in one experimental setting, severity of sentence."}]},{"t":"p","runs":[{"text":"The finding that matters: the same models showed "},{"text":"little","i":true},{"text":" overt bias when asked directly about race. Safety training had successfully removed the explicit expression. The covert association, triggered by dialect rather than by any stated group, was not removed, and in some measures it was strongest in the models that had received the most safety training."}]},{"t":"h2","id":"the-lesson-underneath","runs":[{"text":"The lesson underneath"}]},{"t":"p","runs":[{"text":"Fixing what you measure does not fix what you do not measure.","b":true}]},{"t":"p","runs":[{"text":"The explicit tests were run, the model passed them, and the association moved to a channel nobody was testing. This is a general property of optimisation against a metric, and it will recur throughout your dealings with these systems."}]},{"t":"p","runs":[{"text":"It also explains a frustrating experience many people report: a model that is scrupulously even-handed when the subject is named, and subtly different in tone, thoroughness or generosity when the same subject is merely implied by a name, a dialect, a place or a phrasing."}]},{"t":"h2","id":"what-can-and-cannot-be-fixed","runs":[{"text":"What can and cannot be fixed"}]},{"t":"p","runs":[{"text":"Honestly:"}]},{"t":"p","runs":[{"text":"Fixable.","b":true},{"text":" Overt statements. Refusal to produce slurs. Balanced treatment when a group is explicitly named. Gender alternatives for short translations. These have improved, and dismissing that improvement is unfair to the work behind it."}]},{"t":"p","runs":[{"text":"Not fixable by the same means.","b":true},{"text":" The statistical residue of a corpus: the fact that in the text, engineers were mostly men and certain names appeared in certain contexts. You cannot remove it by adding a rule, because it is not represented as a rule. It is distributed across billions of weights."}]},{"t":"p","runs":[{"text":"There are partial approaches: rebalancing the corpus, targeted fine-tuning, filtering outputs, and running audits by group. None of them is complete, and each has a cost: most obviously that a model heavily patched on one axis often becomes strange on adjacent ones."}]},{"t":"h2","id":"what-to-actually-do","runs":[{"text":"What to actually do"}]},{"t":"p","runs":[{"text":"Do not use a general-purpose model to make decisions about people.","b":true},{"text":" Not shortlisting, not scoring, not triaging applicants. Not because it is worse than a human panel on every axis (that is arguable) but because it applies whatever it has at scale, invisibly, with no explanation and no appeal."}]},{"t":"p","runs":[{"text":"If you use it to assist, blind the input.","b":true},{"text":" Strip names, addresses, schools, photographs and anything that signals a group before the model sees it. This is simple, it is effective against the specific mechanism above, and it is worth doing for human reviewers too."}]},{"t":"p","runs":[{"text":"Audit by group.","b":true},{"text":" If a system is used at scale, measure its outputs separately for each group affected. An overall accuracy figure hides exactly the failure you are looking for."}]},{"t":"p","runs":[{"text":"Watch for the covert channel.","b":true},{"text":" When you test for bias, test with the signal implicit (a name, a dialect, a location) not only with it stated. Testing only the explicit case measures the patch, not the model."}]}],"check":{"question":[{"text":"A model shows no measurable bias when asked directly about candidates from different ethnic groups, but rates identical applications differently when only the applicant’s name changes. What does this pattern indicate?"}],"options":[{"text":[{"text":"The bias is in the evaluation method rather than the model, since two tests of the same property should agree."}],"correct":false,"tempting":[{"text":"Assumes both tests probe the same channel; the explicit and implicit channels are demonstrably separable."}]},{"text":[{"text":"The model is applying a deliberate rule about names that was introduced during safety tuning."}],"correct":false,"tempting":[{"text":"Attributes to an explicit rule what arises from distributed statistical association in the corpus."}]},{"text":[{"text":"Explicit bias was trained out because it was the thing being measured, while the association carried by names was never targeted and remains."}],"correct":true,"tempting":null},{"text":[{"text":"Name-based differences reflect genuine variation in the training CVs, so the model is reporting a real pattern."}],"correct":false,"tempting":[{"text":"Treats a learned social association as a valid signal, which is precisely the reasoning that makes such systems harmful at scale."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"the-cut-off-and-the-clock","title":"It has no clock, and its knowledge thins before it stops","minutes":8},"next":{"slug":"the-reasoning-models","title":"Models that think out loud, and what that does not prove","minutes":9}}}