{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"decisions-made-about-you","title":"When a model decides something that matters","url":"https://addaly.com/learn/ai-actually-explained/decisions-made-about-you","position":17,"of":92,"minutes":10,"answer":"The damage in automated decisions comes from scale, presumed correctness, no explanation and asymmetric cost, not from the sophistication of the method: Robodebt was arithmetic.","keyIdea":"The damage in automated decisions comes from scale, presumed correctness, no explanation and asymmetric cost, not from the sophistication of the method: Robodebt was arithmetic."},"body":{"format":"markdown","text":"## A different category\n\nEverything so far in this block costs you seconds when it fails. A bad recommendation, a mistyped word, an over-processed photograph. This lesson is about the other kind, where a model's output determines whether you get credit, a job interview, a benefit payment, a visa or a place at a school.\n\nThe technology is not more sophisticated. Often it is much simpler: a logistic regression or a tree model over a few dozen fields. What differs is that the person the model is wrong about has no way to argue with it and often does not know it exists.\n\n## Three cases that are documented, not hypothetical\n\n**The Netherlands, childcare benefits.** From around 2013 the Dutch tax authority used a risk-scoring system to flag childcare-benefit claims for fraud investigation. Having a second nationality was among the factors used. Tens of thousands of families were wrongly accused, ordered to repay large sums, and driven into debt; more than a thousand children were taken into care. The scandal broke fully in 2020 and the entire Dutch government resigned in January 2021.\n\n**Australia, Robodebt.** From 2016 an automated scheme compared welfare recipients' reported income with annual tax data, averaged that annual figure across fortnights, and issued debt notices where the averaged figure disagreed. The averaging step was invalid for anyone with irregular work, that is, for most of the affected population. Around 470,000 wrongful debts were raised. A Royal Commission reported in 2023 in scathing terms, and the scheme was unlawful.\n\n**Amazon's CV screener.** Built in the mid-2010s and trained on a decade of the company's own hiring decisions, it learned to downgrade CVs containing the word \"women's\", as in \"women's chess club captain\". It was never deployed at scale and was abandoned. It is the cleanest textbook case of a model reproducing the past accurately and thereby doing something nobody intended.\n\nNote that Robodebt was not even machine learning. It was arithmetic. That matters, because the harm did not come from the sophistication of the method. It came from automating a decision at scale, presuming the output correct, and putting the burden of disproof on the person.\n\n> Figure: Three automated decisions, and how long the appeal took. Timeline: 2013, The Dutch tax authority begins risk-scoring childcare benefit claims; second nationality is among the factors; 2016, Australia's Robodebt averages annual tax data across fortnights and issues automated debt notices; mid-2010s, Amazon's CV screener, trained on a decade of its own hiring, downgrades CVs containing 'women's'; abandoned; 2020, The Dutch scandal breaks fully: tens of thousands of families wrongly accused, over a thousand children taken into care; Jan 2021, The entire Dutch government resigns over it; 2023, An Australian Royal Commission reports; around 470,000 debts were wrongful and the scheme unlawful. Robodebt used arithmetic, not machine learning. The harm came from scale, presumed correctness and an inverted burden of proof, not from the sophistication of the method. In every case the paper trail, not better technology, ended it.\n\n## The four features that make these dangerous\n\nLook for these together, because it is the combination that does the damage:\n\n1. **Scale.** A caseworker makes a hundred wrong decisions a year. A model makes them at the rate of the queue.\n2. **Presumption of correctness.** The output is treated as a finding rather than a flag for review. Robodebt inverted the burden of proof: the citizen had to disprove the computer.\n3. **No explanation.** The affected person cannot see which factors moved their score, so cannot challenge them.\n4. **Asymmetric cost.** A false positive costs you your income. It costs the operator almost nothing, which is why the threshold drifts towards catching more.\n\n## What rights you have\n\nThis differs by country, and it is worth knowing your own position rather than assuming.\n\nIn the **EU and UK**, Article 22 of the GDPR restricts decisions based solely on automated processing that produce legal or similarly significant effects, and where such processing is permitted it comes with a right to obtain human intervention and to contest the decision. The EU's **AI Act**, which entered into force in 2024 and phases in through 2026 and 2027, additionally classifies uses such as credit scoring, employment screening and access to essential services as high risk, with obligations attached.\n\nIn **India**, the Digital Personal Data Protection Act 2023 governs personal data but contains no equivalent automated-decision provision, so the protection you have is different in kind. In the **United States** the position varies by state and by sector, with credit decisions covered by long-standing law requiring adverse-action notices, and hiring covered by newer local rules such as New York City's bias-audit requirement.\n\nThis course is not legal advice and cannot be. The practical point is that a right to a human review exists in some places and not others, and knowing which you are in changes what you can ask for.\n\n## What to actually do\n\nIf an automated decision goes against you, three requests are usually available and often effective:\n\n- Ask **whether a human reviewed it**, and if not, ask for a human review by name of the process.\n- Ask **what information was used**. In credit, the specific reasons for a refusal are required in several jurisdictions, and the answer is often a correctable error in a data file rather than a judgement about you.\n- Ask for the decision **in writing**. Automated processes generate paper trails, and the request itself frequently routes the case to a person.\n\nThe common thread in every documented disaster above is that appeal routes existed on paper and were made practically impossible to use. The systems were not defeated by better technology. They were defeated by journalists, auditors and courts insisting on the paper trail."},"check":{"question":"Why is Australia's Robodebt scheme, which used simple income averaging rather than machine learning, relevant to a course about AI?","options":[{"text":"Because income averaging is a form of statistical model, so it qualifies as machine learning after all.","correct":false,"tempting":"Argues about classification rather than engaging with why the harm occurred."},{"text":"Because it shows that harm at scale comes from automating a decision, presuming its output correct and shifting the burden of proof, independently of how clever the method is.","correct":true},{"text":"Because it demonstrates that machine learning would have produced more accurate debts and should have been used instead.","correct":false,"tempting":"Assumes a better model fixes the problem, when the failure was in how the output was treated rather than in its accuracy."},{"text":"Because it was the first case in which a government was found to have used an algorithm unlawfully.","correct":false,"tempting":"Treats legal novelty as the lesson; the transferable point is structural, not chronological."}]},"prev":{"slug":"speech-to-text","title":"Machines that write down what you said","url":"https://addaly.com/learn/ai-actually-explained/speech-to-text"},"next":{"slug":"factories-fields-and-clinics","title":"The uses that are not on a screen","url":"https://addaly.com/learn/ai-actually-explained/factories-fields-and-clinics"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When a model decides something that matters', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/decisions-made-about-you (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When a model decides something that matters","courseTitle":"AI, Actually Explained","module":{"index":2,"slug":"where-it-already-is","title":"Where it already is, without a label"},"answer":null,"keyIdea":"The damage in automated decisions comes from scale, presumed correctness, no explanation and asymmetric cost, not from the sophistication of the method: Robodebt was arithmetic.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"timeline","title":"Three automated decisions, and how long the appeal took","events":[{"when":"2013","what":"The Dutch tax authority begins risk-scoring childcare benefit claims; second nationality is among the factors"},{"when":"2016","what":"Australia’s Robodebt averages annual tax data across fortnights and issues automated debt notices"},{"when":"mid-2010s","what":"Amazon’s CV screener, trained on a decade of its own hiring, downgrades CVs containing ‘women’s’; abandoned"},{"when":"2020","what":"The Dutch scandal breaks fully: tens of thousands of families wrongly accused, over a thousand children taken into care"},{"when":"Jan 2021","what":"The entire Dutch government resigns over it"},{"when":"2023","what":"An Australian Royal Commission reports; around 470,000 debts were wrongful and the scheme unlawful"}],"caption":"Robodebt used arithmetic, not machine learning. The harm came from scale, presumed correctness and an inverted burden of proof, not from the sophistication of the method. In every case the paper trail, not better technology, ended it."},"alt":"Figure: Three automated decisions, and how long the appeal took. Timeline: 2013, The Dutch tax authority begins risk-scoring childcare benefit claims; second nationality is among the factors; 2016, Australia’s Robodebt averages annual tax data across fortnights and issues automated debt notices; mid-2010s, Amazon’s CV screener, trained on a decade of its own hiring, downgrades CVs containing ‘women’s’; abandoned; 2020, The Dutch scandal breaks fully: tens of thousands of families wrongly accused, over a thousand children taken into care; Jan 2021, The entire Dutch government resigns over it; 2023, An Australian Royal Commission reports; around 470,000 debts were wrongful and the scheme unlawful. Robodebt used arithmetic, not machine learning. The harm came from scale, presumed correctness and an inverted burden of proof, not from the sophistication of the method. In every case the paper trail, not better technology, ended it."}},"blocks":[{"t":"h2","id":"a-different-category","runs":[{"text":"A different category"}]},{"t":"p","runs":[{"text":"Everything so far in this block costs you seconds when it fails. A bad recommendation, a mistyped word, an over-processed photograph. This lesson is about the other kind, where a model’s output determines whether you get credit, a job interview, a benefit payment, a visa or a place at a school."}]},{"t":"p","runs":[{"text":"The technology is not more sophisticated. Often it is much simpler: a logistic regression or a tree model over a few dozen fields. What differs is that the person the model is wrong about has no way to argue with it and often does not know it exists."}]},{"t":"h2","id":"three-cases-that-are-documented-not-hypothetical","runs":[{"text":"Three cases that are documented, not hypothetical"}]},{"t":"p","runs":[{"text":"The Netherlands, childcare benefits.","b":true},{"text":" From around 2013 the Dutch tax authority used a risk-scoring system to flag childcare-benefit claims for fraud investigation. Having a second nationality was among the factors used. Tens of thousands of families were wrongly accused, ordered to repay large sums, and driven into debt; more than a thousand children were taken into care. The scandal broke fully in 2020 and the entire Dutch government resigned in January 2021."}]},{"t":"p","runs":[{"text":"Australia, Robodebt.","b":true},{"text":" From 2016 an automated scheme compared welfare recipients’ reported income with annual tax data, averaged that annual figure across fortnights, and issued debt notices where the averaged figure disagreed. The averaging step was invalid for anyone with irregular work, that is, for most of the affected population. Around 470,000 wrongful debts were raised. A Royal Commission reported in 2023 in scathing terms, and the scheme was unlawful."}]},{"t":"p","runs":[{"text":"Amazon’s CV screener.","b":true},{"text":" Built in the mid-2010s and trained on a decade of the company’s own hiring decisions, it learned to downgrade CVs containing the word “women’s”, as in “women’s chess club captain”. It was never deployed at scale and was abandoned. It is the cleanest textbook case of a model reproducing the past accurately and thereby doing something nobody intended."}]},{"t":"p","runs":[{"text":"Note that Robodebt was not even machine learning. It was arithmetic. That matters, because the harm did not come from the sophistication of the method. It came from automating a decision at scale, presuming the output correct, and putting the burden of disproof on the person."}]},{"t":"h2","id":"the-four-features-that-make-these-dangerous","runs":[{"text":"The four features that make these dangerous"}]},{"t":"p","runs":[{"text":"Look for these together, because it is the combination that does the damage:"}]},{"t":"ol","start":1,"items":[[{"text":"Scale.","b":true},{"text":" A caseworker makes a hundred wrong decisions a year. A model makes them at the rate of the queue."}],[{"text":"Presumption of correctness.","b":true},{"text":" The output is treated as a finding rather than a flag for review. Robodebt inverted the burden of proof: the citizen had to disprove the computer."}],[{"text":"No explanation.","b":true},{"text":" The affected person cannot see which factors moved their score, so cannot challenge them."}],[{"text":"Asymmetric cost.","b":true},{"text":" A false positive costs you your income. It costs the operator almost nothing, which is why the threshold drifts towards catching more."}]]},{"t":"h2","id":"what-rights-you-have","runs":[{"text":"What rights you have"}]},{"t":"p","runs":[{"text":"This differs by country, and it is worth knowing your own position rather than assuming."}]},{"t":"p","runs":[{"text":"In the "},{"text":"EU and UK","b":true},{"text":", Article 22 of the GDPR restricts decisions based solely on automated processing that produce legal or similarly significant effects, and where such processing is permitted it comes with a right to obtain human intervention and to contest the decision. The EU’s "},{"text":"AI Act","b":true},{"text":", which entered into force in 2024 and phases in through 2026 and 2027, additionally classifies uses such as credit scoring, employment screening and access to essential services as high risk, with obligations attached."}]},{"t":"p","runs":[{"text":"In "},{"text":"India","b":true},{"text":", the Digital Personal Data Protection Act 2023 governs personal data but contains no equivalent automated-decision provision, so the protection you have is different in kind. In the "},{"text":"United States","b":true},{"text":" the position varies by state and by sector, with credit decisions covered by long-standing law requiring adverse-action notices, and hiring covered by newer local rules such as New York City’s bias-audit requirement."}]},{"t":"p","runs":[{"text":"This course is not legal advice and cannot be. The practical point is that a right to a human review exists in some places and not others, and knowing which you are in changes what you can ask for."}]},{"t":"h2","id":"what-to-actually-do","runs":[{"text":"What to actually do"}]},{"t":"p","runs":[{"text":"If an automated decision goes against you, three requests are usually available and often effective:"}]},{"t":"ul","items":[[{"text":"Ask "},{"text":"whether a human reviewed it","b":true},{"text":", and if not, ask for a human review by name of the process."}],[{"text":"Ask "},{"text":"what information was used","b":true},{"text":". In credit, the specific reasons for a refusal are required in several jurisdictions, and the answer is often a correctable error in a data file rather than a judgement about you."}],[{"text":"Ask for the decision "},{"text":"in writing","b":true},{"text":". Automated processes generate paper trails, and the request itself frequently routes the case to a person."}]]},{"t":"p","runs":[{"text":"The common thread in every documented disaster above is that appeal routes existed on paper and were made practically impossible to use. The systems were not defeated by better technology. They were defeated by journalists, auditors and courts insisting on the paper trail."}]}],"check":{"question":[{"text":"Why is Australia’s Robodebt scheme, which used simple income averaging rather than machine learning, relevant to a course about AI?"}],"options":[{"text":[{"text":"Because income averaging is a form of statistical model, so it qualifies as machine learning after all."}],"correct":false,"tempting":[{"text":"Argues about classification rather than engaging with why the harm occurred."}]},{"text":[{"text":"Because it shows that harm at scale comes from automating a decision, presuming its output correct and shifting the burden of proof, independently of how clever the method is."}],"correct":true,"tempting":null},{"text":[{"text":"Because it demonstrates that machine learning would have produced more accurate debts and should have been used instead."}],"correct":false,"tempting":[{"text":"Assumes a better model fixes the problem, when the failure was in how the output was treated rather than in its accuracy."}]},{"text":[{"text":"Because it was the first case in which a government was found to have used an algorithm unlawfully."}],"correct":false,"tempting":[{"text":"Treats legal novelty as the lesson; the transferable point is structural, not chronological."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"speech-to-text","title":"Machines that write down what you said","minutes":8},"next":{"slug":"factories-fields-and-clinics","title":"The uses that are not on a screen","minutes":9}}}