When a model decides something that matters

A different category

Everything so far in this block costs you seconds when it fails. A bad recommendation, a mistyped word, an over-processed photograph. This lesson is about the other kind, where a model’s output determines whether you get credit, a job interview, a benefit payment, a visa or a place at a school.

The technology is not more sophisticated. Often it is much simpler: a logistic regression or a tree model over a few dozen fields. What differs is that the person the model is wrong about has no way to argue with it and often does not know it exists.

Three cases that are documented, not hypothetical

The Netherlands, childcare benefits. From around 2013 the Dutch tax authority used a risk-scoring system to flag childcare-benefit claims for fraud investigation. Having a second nationality was among the factors used. Tens of thousands of families were wrongly accused, ordered to repay large sums, and driven into debt; more than a thousand children were taken into care. The scandal broke fully in 2020 and the entire Dutch government resigned in January 2021.

Australia, Robodebt. From 2016 an automated scheme compared welfare recipients’ reported income with annual tax data, averaged that annual figure across fortnights, and issued debt notices where the averaged figure disagreed. The averaging step was invalid for anyone with irregular work, that is, for most of the affected population. Around 470,000 wrongful debts were raised. A Royal Commission reported in 2023 in scathing terms, and the scheme was unlawful.

Amazon’s CV screener. Built in the mid-2010s and trained on a decade of the company’s own hiring decisions, it learned to downgrade CVs containing the word “women’s”, as in “women’s chess club captain”. It was never deployed at scale and was abandoned. It is the cleanest textbook case of a model reproducing the past accurately and thereby doing something nobody intended.

Note that Robodebt was not even machine learning. It was arithmetic. That matters, because the harm did not come from the sophistication of the method. It came from automating a decision at scale, presuming the output correct, and putting the burden of disproof on the person.

The four features that make these dangerous

Look for these together, because it is the combination that does the damage:

1. Scale. A caseworker makes a hundred wrong decisions a year. A model makes them at the rate of the queue.
2. Presumption of correctness. The output is treated as a finding rather than a flag for review. Robodebt inverted the burden of proof: the citizen had to disprove the computer.
3. No explanation. The affected person cannot see which factors moved their score, so cannot challenge them.
4. Asymmetric cost. A false positive costs you your income. It costs the operator almost nothing, which is why the threshold drifts towards catching more.

What rights you have

This differs by country, and it is worth knowing your own position rather than assuming.

In the EU and UK, Article 22 of the GDPR restricts decisions based solely on automated processing that produce legal or similarly significant effects, and where such processing is permitted it comes with a right to obtain human intervention and to contest the decision. The EU’s AI Act, which entered into force in 2024 and phases in through 2026 and 2027, additionally classifies uses such as credit scoring, employment screening and access to essential services as high risk, with obligations attached.

In India, the Digital Personal Data Protection Act 2023 governs personal data but contains no equivalent automated-decision provision, so the protection you have is different in kind. In the United States the position varies by state and by sector, with credit decisions covered by long-standing law requiring adverse-action notices, and hiring covered by newer local rules such as New York City’s bias-audit requirement.

This course is not legal advice and cannot be. The practical point is that a right to a human review exists in some places and not others, and knowing which you are in changes what you can ask for.

What to actually do

If an automated decision goes against you, three requests are usually available and often effective:

• Ask whether a human reviewed it, and if not, ask for a human review by name of the process.
• Ask what information was used. In credit, the specific reasons for a refusal are required in several jurisdictions, and the answer is often a correctable error in a data file rather than a judgement about you.
• Ask for the decision in writing. Automated processes generate paper trails, and the request itself frequently routes the case to a person.

The common thread in every documented disaster above is that appeal routes existed on paper and were made practically impossible to use. The systems were not defeated by better technology. They were defeated by journalists, auditors and courts insisting on the paper trail.