{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"risk-by-consequence","title":"Triage: sorting risk by what it costs","url":"https://addaly.com/learn/ai-and-you/risk-by-consequence","position":9,"of":73,"minutes":4,"answer":"Set verification by reversibility, who pays, how often it repeats, and whether appeal exists.","keyIdea":"Set verification by reversibility, who pays, how often it repeats, and whether appeal exists."},"body":{"format":"markdown","text":"## One formula, 280,000 grades\n\nIn spring 2020 England cancelled its A-level exams because of Covid. Teachers estimated a grade for every entry. Ofqual, the exams regulator, then ran a statistical model. For larger classes it kept each teacher's ranking of their students and fitted it to the school's results in earlier years. For the smallest classes it used the teachers' grades, and it blended them in for small ones, which favoured schools with small classes.\n\nResults came out on 13 August. Of 718,000 entries, 39.1% were lower than the teacher's estimate: 35.6% by one grade, 3.3% by two, 0.2% by three. That is about 280,000 entries, and university places rested on them.\n\n> Figure: England, 2020: A-level grades after the model. Bars (% of entries): No change 58.7; Down one grade 35.6; Down two grades 3.3; Up one grade 2.2; Down three grades 0.2. Ofqual's own figures, 718,000 entries, all ages. A further 0.06% went up by two or three grades. Four days after results day the adjusted grades were abandoned.\n\nOn 17 August the government abandoned the adjusted grades and gave students their teachers' estimates. Run the case through four questions.\n\n## Four questions, thirty seconds\n\n**1. If this is wrong, can it be undone?** A wrong sentence in a draft is fixed by editing. A lost university place in August is hard to get back by September. A sent message, a payment, a public post and a dose are one-way doors. They deserve a pause that two-way doors do not.\n\n**2. Who bears the cost, and did they choose it?** If it lands on you, you may accept the risk. The students chose nothing. They could not see the model and had no say in it. When the person harmed did not choose the tool, the bar rises sharply.\n\n**3. How many times will this happen?** One wrong grade is an incident. The same rule applied to 280,000 entries is a policy. A human marker with a bias affects the scripts they mark. A model with the same bias affects everyone, identically, at once.\n\n**4. Is there a route to appeal, and does anyone answer it?** On 15 August Ofqual published its criteria for appeals based on mock exams and suspended them within hours. A system with a working appeal can be wrong and still recoverable.\n\n## Three tiers\n\nMost uses land in one of three tiers.\n\n**Low.** Reversible, affects only you, happens once. Drafting, brainstorming, explaining a concept. Verification: skim, and save your attention for the other tiers.\n\n**Medium.** Reversible with effort, or it affects colleagues, or it repeats. Emails to clients, reports someone will act on, code going into a system. Verification: check every name, number and claim, and know what the answer should be before you read the output.\n\n**High.** Irreversible, or it affects someone who did not choose it, or it runs at scale, or there is no appeal. Anything medical, legal or financial. Anything that decides about a person. Verification: a primary source you open yourself, a second reader, and a record of what was checked.\n\n> Figure: Two of the four questions, and where they put a task. A two by two grid, rows Reversible and Irreversible, columns The cost falls on you and The cost falls on someone who did not choose: Reversible and The cost falls on you: Low: skim; Reversible and The cost falls on someone who did not choose: High: primary source, second reader, keep the record; Irreversible and The cost falls on you: High: primary source, second reader, keep the record; Irreversible and The cost falls on someone who did not choose: High: primary source, second reader, keep the record. Either irreversibility or a cost borne by someone who did not choose makes a task High under this lesson's rule. Scale or no appeal also makes it High.\n\nSome uses sit outside the tiers, because the answer is not to use the tool: a crisis conversation, or a decision you are licensed to make in person.\n\n## Decide on a calm afternoon\n\nRisk judged in the moment follows convenience. At six on a Thursday evening everything looks low risk. Write the rule down while you are calm: anything that goes to a client is checked against the source, and anything about a person gets a second reader.\n\nAn organisation needs the same thing on one page: which tasks may use which tools, what each tier requires, and whom to tell when something goes wrong. A policy that only says \"use AI responsibly\" hands the whole question back to the person least able to answer it at 6pm."},"check":{"question":"Two uses: drafting a blog post you will edit before publishing, and writing a standard rejection sent automatically, from an address that takes no replies, to 3,000 job applicants. Why does the second need far more care?","options":[{"text":"Rejection letters are regulated by law, and blog posts are not","correct":false,"tempting":"Reaches for a rule that mostly does not exist. The difference is structural, not a statute about rejections."},{"text":"Any text written for other people is riskier than text for you","correct":false,"tempting":"A rough pattern stated as a rule. It hides the factors doing the work and fails on plenty of low-stakes text."},{"text":"It runs at scale, on people who chose nothing, with no appeal","correct":true},{"text":"The blog is reviewed by you first, so any tool use there is safe","correct":false,"tempting":"Right that the blog is reversible. Review is not a complete defence, and it misses what makes the rejections different."}],"why":"Three of the four questions point the same way. Scale, who pays and the missing appeal each raise the tier."},"prev":{"slug":"the-liability-question","title":"Who is responsible when it is wrong","url":"https://addaly.com/learn/ai-and-you/the-liability-question"},"next":{"slug":"where-bias-comes-from","title":"Where bias comes from","url":"https://addaly.com/learn/ai-and-you/where-bias-comes-from"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Triage: sorting risk by what it costs', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/risk-by-consequence (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Triage: sorting risk by what it costs","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":1,"slug":"how-things-fail","title":"How these systems fail"},"answer":null,"keyIdea":"Set verification by reversibility, who pays, how often it repeats, and whether appeal exists.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","title":"England, 2020: A-level grades after the model","caption":"Ofqual’s own figures, 718,000 entries, all ages. A further 0.06% went up by two or three grades. Four days after results day the adjusted grades were abandoned.","unit":"% of entries","series":[{"label":"No change","value":58.7},{"label":"Down one grade","value":35.6,"highlight":true},{"label":"Down two grades","value":3.3},{"label":"Up one grade","value":2.2},{"label":"Down three grades","value":0.2}]},"alt":"Figure: England, 2020: A-level grades after the model. Bars (% of entries): No change 58.7; Down one grade 35.6; Down two grades 3.3; Up one grade 2.2; Down three grades 0.2. Ofqual’s own figures, 718,000 entries, all ages. A further 0.06% went up by two or three grades. Four days after results day the adjusted grades were abandoned."}},"blocks":[{"t":"h2","id":"one-formula-280-000-grades","runs":[{"text":"One formula, 280,000 grades"}]},{"t":"p","runs":[{"text":"In spring 2020 England cancelled its A-level exams because of Covid. Teachers estimated a grade for every entry. Ofqual, the exams regulator, then ran a statistical model. For larger classes it kept each teacher’s ranking of their students and fitted it to the school’s results in earlier years. For the smallest classes it used the teachers’ grades, and it blended them in for small ones, which favoured schools with small classes."}]},{"t":"quick","prompt":"Guess first: what made the 2020 model most dangerous?","options":["It was mathematically complicated","It was wrong for everyone","It ran one rule across a country"],"answer":2,"reveal":"It ran one rule across a whole country, on people who never chose it. The four questions below show why that matters."},{"t":"p","runs":[{"text":"Results came out on 13 August. Of 718,000 entries, 39.1% were lower than the teacher’s estimate: 35.6% by one grade, 3.3% by two, 0.2% by three. That is about 280,000 entries, and university places rested on them."}]},{"t":"p","runs":[{"text":"On 17 August the government abandoned the adjusted grades and gave students their teachers’ estimates. Run the case through four questions."}]},{"t":"h2","id":"four-questions-thirty-seconds","runs":[{"text":"Four questions, thirty seconds"}]},{"t":"p","runs":[{"text":"1. If this is wrong, can it be undone?","b":true},{"text":" A wrong sentence in a draft is fixed by editing. A lost university place in August is hard to get back by September. A sent message, a payment, a public post and a dose are one-way doors. They deserve a pause that two-way doors do not."}]},{"t":"p","runs":[{"text":"2. Who bears the cost, and did they choose it?","b":true},{"text":" If it lands on you, you may accept the risk. The students chose nothing. They could not see the model and had no say in it. When the person harmed did not choose the tool, the bar rises sharply."}]},{"t":"p","runs":[{"text":"3. How many times will this happen?","b":true},{"text":" One wrong grade is an incident. The same rule applied to 280,000 entries is a policy. A human marker with a bias affects the scripts they mark. A model with the same bias affects everyone, identically, at once."}]},{"t":"p","runs":[{"text":"4. Is there a route to appeal, and does anyone answer it?","b":true},{"text":" On 15 August Ofqual published its criteria for appeals based on mock exams and suspended them within hours. A system with a working appeal can be wrong and still recoverable."}]},{"t":"h2","id":"three-tiers","runs":[{"text":"Three tiers"}]},{"t":"p","runs":[{"text":"Most uses land in one of three tiers."}]},{"t":"p","runs":[{"text":"Low.","b":true},{"text":" Reversible, affects only you, happens once. Drafting, brainstorming, explaining a concept. Verification: skim, and save your attention for the other tiers."}]},{"t":"p","runs":[{"text":"Medium.","b":true},{"text":" Reversible with effort, or it affects colleagues, or it repeats. Emails to clients, reports someone will act on, code going into a system. Verification: check every name, number and claim, and know what the answer should be before you read the output."}]},{"t":"p","runs":[{"text":"High.","b":true},{"text":" Irreversible, or it affects someone who did not choose it, or it runs at scale, or there is no appeal. Anything medical, legal or financial. Anything that decides about a person. Verification: a primary source you open yourself, a second reader, and a record of what was checked."}]},{"t":"figure","label":"Figure","figure":{"type":"matrix","title":"Two of the four questions, and where they put a task","caption":"Either irreversibility or a cost borne by someone who did not choose makes a task High under this lesson’s rule. Scale or no appeal also makes it High.","rows":["Reversible","Irreversible"],"cols":["The cost falls on you","The cost falls on someone who did not choose"],"cells":[["Low: skim","High: primary source, second reader, keep the record"],["High: primary source, second reader, keep the record","High: primary source, second reader, keep the record"]],"cellLabels":[["a draft, a brainstorm","a report a colleague will act on"],["a payment, a sent message, a public post","a dose, a filing, a decision about a person"]]},"alt":"Figure: Two of the four questions, and where they put a task. A two by two grid, rows Reversible and Irreversible, columns The cost falls on you and The cost falls on someone who did not choose: Reversible and The cost falls on you: Low: skim; Reversible and The cost falls on someone who did not choose: High: primary source, second reader, keep the record; Irreversible and The cost falls on you: High: primary source, second reader, keep the record; Irreversible and The cost falls on someone who did not choose: High: primary source, second reader, keep the record. Either irreversibility or a cost borne by someone who did not choose makes a task High under this lesson’s rule. Scale or no appeal also makes it High."},{"t":"p","runs":[{"text":"Some uses sit outside the tiers, because the answer is not to use the tool: a crisis conversation, or a decision you are licensed to make in person."}]},{"t":"h2","id":"decide-on-a-calm-afternoon","runs":[{"text":"Decide on a calm afternoon"}]},{"t":"p","runs":[{"text":"Risk judged in the moment follows convenience. At six on a Thursday evening everything looks low risk. Write the rule down while you are calm: anything that goes to a client is checked against the source, and anything about a person gets a second reader."}]},{"t":"p","runs":[{"text":"An organisation needs the same thing on one page: which tasks may use which tools, what each tier requires, and whom to tell when something goes wrong. A policy that only says “use AI responsibly” hands the whole question back to the person least able to answer it at 6pm."}]}],"check":{"question":[{"text":"Two uses: drafting a blog post you will edit before publishing, and writing a standard rejection sent automatically, from an address that takes no replies, to 3,000 job applicants. Why does the second need far more care?"}],"options":[{"text":[{"text":"Rejection letters are regulated by law, and blog posts are not"}],"correct":false,"tempting":[{"text":"Reaches for a rule that mostly does not exist. The difference is structural, not a statute about rejections."}]},{"text":[{"text":"Any text written for other people is riskier than text for you"}],"correct":false,"tempting":[{"text":"A rough pattern stated as a rule. It hides the factors doing the work and fails on plenty of low-stakes text."}]},{"text":[{"text":"It runs at scale, on people who chose nothing, with no appeal"}],"correct":true,"tempting":null},{"text":[{"text":"The blog is reviewed by you first, so any tool use there is safe"}],"correct":false,"tempting":[{"text":"Right that the blog is reversible. Review is not a complete defence, and it misses what makes the rejections different."}]}],"order":[3,0,2,1],"why":[{"text":"Three of the four questions point the same way. Scale, who pays and the missing appeal each raise the tier."}]},"prev":{"slug":"the-liability-question","title":"Who is responsible when it is wrong","minutes":4},"next":{"slug":"where-bias-comes-from","title":"Where bias comes from","minutes":9}}}