Bias, and the difference between fixing it and hiding it

Four doors it comes in through

Bias in a model is not one thing arriving from one place. It enters at four separate points, and they need different responses.

The corpus. Text written by people carries the assumptions of the people who wrote it, and the web over-represents some populations enormously. Occupations, nationalities and names all carry statistical associations that the model absorbs as ordinary patterns.

The labels. Every judgement in fine-tuning was made by a person following a guideline. Who they were and what the guideline said is baked in.

The filters. Quality classifiers decide what counts as good text. There is documented evidence that some corpus filters remove text written in non-standard dialects at higher rates, which means those voices are thinned before training begins.

The preference tuning. Raters preferred some answers over others, and their preferences are not neutral on questions about people.

What it looks like in practice

Three measured examples, not anecdotes.

Names on CVs. A 2024 Bloomberg investigation had a leading model rank identical CVs that differed only in the name attached. Names associated with different racial groups received systematically different rankings across thousands of trials, and the pattern varied by job. Nobody wrote a rule about names. The association came from the text.

Occupational gender in translation. Translating from a language without grammatical gender into one with it forces a choice, and the choice follows the training text: engineers male, nurses female. Providers have patched short phrases; longer passages still show it.

Dialect. This is the most important of the three. A 2024 study published in Nature examined how models responded to text written in African American English compared with Standardised American English expressing the same content. The models produced markedly more negative associations for the dialect text, and this appeared in consequential judgements: hypothetical employability and, in one experimental setting, severity of sentence.

The finding that matters: the same models showed little overt bias when asked directly about race. Safety training had successfully removed the explicit expression. The covert association, triggered by dialect rather than by any stated group, was not removed, and in some measures it was strongest in the models that had received the most safety training.

The lesson underneath

Fixing what you measure does not fix what you do not measure.

The explicit tests were run, the model passed them, and the association moved to a channel nobody was testing. This is a general property of optimisation against a metric, and it will recur throughout your dealings with these systems.

It also explains a frustrating experience many people report: a model that is scrupulously even-handed when the subject is named, and subtly different in tone, thoroughness or generosity when the same subject is merely implied by a name, a dialect, a place or a phrasing.

What can and cannot be fixed

Honestly:

Fixable. Overt statements. Refusal to produce slurs. Balanced treatment when a group is explicitly named. Gender alternatives for short translations. These have improved, and dismissing that improvement is unfair to the work behind it.

Not fixable by the same means. The statistical residue of a corpus: the fact that in the text, engineers were mostly men and certain names appeared in certain contexts. You cannot remove it by adding a rule, because it is not represented as a rule. It is distributed across billions of weights.

There are partial approaches: rebalancing the corpus, targeted fine-tuning, filtering outputs, and running audits by group. None of them is complete, and each has a cost: most obviously that a model heavily patched on one axis often becomes strange on adjacent ones.

What to actually do

Do not use a general-purpose model to make decisions about people. Not shortlisting, not scoring, not triaging applicants. Not because it is worse than a human panel on every axis (that is arguable) but because it applies whatever it has at scale, invisibly, with no explanation and no appeal.

If you use it to assist, blind the input. Strip names, addresses, schools, photographs and anything that signals a group before the model sees it. This is simple, it is effective against the specific mechanism above, and it is worth doing for human reviewers too.

Audit by group. If a system is used at scale, measure its outputs separately for each group affected. An overall accuracy figure hides exactly the failure you are looking for.

Watch for the covert channel. When you test for bias, test with the signal implicit (a name, a dialect, a location) not only with it stated. Testing only the explicit case measures the patch, not the model.