{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-for-harm","title":"Measuring the worst case, not the average one","url":"https://addaly.com/learn/evaluating-ai/evaluating-for-harm","position":63,"of":76,"minutes":9,"answer":"Rank failures by severity, reversibility and whether the person can tell they were harmed, then test each harm category with cases written for it, because harm concentrates exactly where a user cannot check the answer.","keyIdea":"Rank failures by severity, reversibility and whether the person can tell they were harmed, then test each harm category with cases written for it, because harm concentrates exactly where a user cannot check the answer."},"body":{"format":"markdown","text":"## Error rate and harm are different quantities\n\nA system with a 20% error rate can be harmless: the errors are obvious, the user notices immediately, the cost is a repeated request. A system with a 0.1% error rate can be dangerous: the errors are invisible, they land on people who cannot verify them, and they are irreversible.\n\nAveraging over items treats those the same. Something else is needed.\n\n> Figure: Error rate and harm are different quantities. A 20% error rate that is harmless: The error is obvious; The user notices immediately; The cost is a repeated request; It can be undone by anyone. A 0.1% error rate that is dangerous: The error is invisible; The user cannot verify it; Acting on it is irreversible; It lands on the people least able to appeal. Averaging over items treats these two systems the same. Describe each failure mode by severity, reversibility, detectability and distribution before you compute anything.\n\n## Describe each failure mode along four axes\n\nFor every way the system can go wrong, write down:\n\n- **Severity.** The worst realistic outcome, stated concretely. Not \"poor user experience\", \"a customer misses a filing deadline and pays a penalty\".\n- **Reversibility.** Can it be undone, and by whom, and at what cost. A wrong draft is reversible; a sent message, a submitted form, a deleted record, a disclosed secret are not.\n- **Detectability by the person affected.** Can the user tell it was wrong. This is the axis most often left out and it dominates the others: an error a user can spot is a nuisance, and the same error where verification is impossible is a harm.\n- **Distribution.** Who bears it. If the failure concentrates on one language, one region, or people with less ability to appeal, the average is actively misleading.\n\nThat description is the input to everything else. Fifteen minutes of writing it usually reorders the team's priorities.\n\n## Sample the tail on purpose\n\nRandom sampling finds common failures. It finds the rare catastrophic one at the rate it occurs, which is to say almost never, on the day before it occurs in production.\n\nSo sample deliberately from where harm lives:\n\n- The lowest-scoring outputs by your judge.\n- Requests in high-consequence categories (medical, legal, financial, safety, self-harm, immigration status) identified by a cheap classifier.\n- Requests from users in the vulnerable slices you identified.\n- The cases where a guardrail fired.\n\nThen have a person read them. A hundred deliberately-chosen items reviewed by a human tells you more about harm than ten thousand sampled ones scored by a model.\n\n## Write the cases; do not wait for them\n\nFor each harm category, write fifty test cases. This is the same discipline as the safety and red-teaming lesson, applied to consequences rather than to attacks:\n\n- A user describing symptoms and asking what to take.\n- A user asking whether their contract lets them leave.\n- A user in evident distress.\n- A question whose honest answer is \"I do not know, and here is who does\".\n\nScore these on a severity-weighted basis and report the score separately from the general quality metric. And be explicit that the weights are a judgement: \"we weight an irreversible financial error at fifty times a formatting error\" is a value statement made visible, which is the most that can be asked of it. Hiding the weights inside an average does not make the judgement go away; it makes it unexaminable.\n\n## Design changes usually beat model changes here\n\nThe most effective response to a harm is often not a better model. Three interventions reduce harm without touching quality at all, and they should be on the table in the same conversation:\n\n- **Remove the irreversibility.** A draft the user sends beats a message the system sends. A queued action with a ten-second undo beats an immediate one.\n- **Restore detectability.** Show the source beside the answer, so verification takes two seconds instead of being impossible.\n- **Refuse and route.** For the highest-consequence categories, the best output is a short answer that names where the real help is.\n\nEach is measurable: undo rate, source click-through, and the proportion of high-consequence requests that were routed rather than answered.\n\n## Near misses are the leading indicator\n\nCount the cases where a guardrail caught something before it reached a user, and track that rate over time. Near misses rise before incidents do, they are far more numerous, and they cost nothing to collect since the guardrail already fires.\n\nA doubling in near misses after a prompt change is a finding, even if no harm reached anybody, and it is the earliest warning available.\n\n## The limit that matters most\n\nYou cannot enumerate harms in advance. Every system in production surprises its builders, usually because a group of users has a context nobody in the room shared.\n\nThe mitigations are procedural rather than statistical: a visible, easy way for people to report a problem; somebody whose job is to read those reports weekly; and a route from a report into the evaluation set, so the same harm cannot recur silently. A team that measures harm carefully and has no channel for the harms it did not predict is measuring the part of the problem it already understood."},"check":{"question":"Which failure most deserves priority, given equal frequency of 0.5%?","options":[{"text":"An incorrect summary of an article the user is reading, shown alongside the article.","correct":false,"tempting":"Wrong and immediately checkable, since the source sits next to the output, so the user detects and corrects it at once."},{"text":"A formatting error that makes a response hard to read on a small screen.","correct":false,"tempting":"Highly visible and fully reversible; irritating rather than harmful, and a candidate for the ordinary quality backlog."},{"text":"An incorrect eligibility answer that leads a user to abandon a benefits claim they were entitled to.","correct":true},{"text":"A wrong product recommendation in a list of five suggestions.","correct":false,"tempting":"Low severity and easily ignored by the user, since the other four options remain visible and nothing irreversible follows."}]},"prev":{"slug":"standards-and-frameworks","title":"What the emerging rules ask for, and what they do not settle","url":"https://addaly.com/learn/evaluating-ai/standards-and-frameworks"},"next":{"slug":"human-oversight-that-is-real","title":"Measuring the human in the loop","url":"https://addaly.com/learn/evaluating-ai/human-oversight-that-is-real"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Measuring the worst case, not the average one', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-for-harm (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Measuring the worst case, not the average one","courseTitle":"Knowing If It Works","module":{"index":7,"slug":"assurance","title":"Evidence other people can act on"},"answer":null,"keyIdea":"Rank failures by severity, reversibility and whether the person can tell they were harmed, then test each harm category with cases written for it, because harm concentrates exactly where a user cannot check the answer.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"A 20% error rate that is harmless","items":["The error is obvious","The user notices immediately","The cost is a repeated request","It can be undone by anyone"]},"type":"compare","right":{"title":"A 0.1% error rate that is dangerous","items":["The error is invisible","The user cannot verify it","Acting on it is irreversible","It lands on the people least able to appeal"]},"title":"Error rate and harm are different quantities","caption":"Averaging over items treats these two systems the same. Describe each failure mode by severity, reversibility, detectability and distribution before you compute anything."},"alt":"Figure: Error rate and harm are different quantities. A 20% error rate that is harmless: The error is obvious; The user notices immediately; The cost is a repeated request; It can be undone by anyone. A 0.1% error rate that is dangerous: The error is invisible; The user cannot verify it; Acting on it is irreversible; It lands on the people least able to appeal. Averaging over items treats these two systems the same. Describe each failure mode by severity, reversibility, detectability and distribution before you compute anything."}},"blocks":[{"t":"h2","id":"error-rate-and-harm-are-different-quantities","runs":[{"text":"Error rate and harm are different quantities"}]},{"t":"p","runs":[{"text":"A system with a 20% error rate can be harmless: the errors are obvious, the user notices immediately, the cost is a repeated request. A system with a 0.1% error rate can be dangerous: the errors are invisible, they land on people who cannot verify them, and they are irreversible."}]},{"t":"p","runs":[{"text":"Averaging over items treats those the same. Something else is needed."}]},{"t":"h2","id":"describe-each-failure-mode-along-four-axes","runs":[{"text":"Describe each failure mode along four axes"}]},{"t":"p","runs":[{"text":"For every way the system can go wrong, write down:"}]},{"t":"ul","items":[[{"text":"Severity.","b":true},{"text":" The worst realistic outcome, stated concretely. Not “poor user experience”, “a customer misses a filing deadline and pays a penalty”."}],[{"text":"Reversibility.","b":true},{"text":" Can it be undone, and by whom, and at what cost. A wrong draft is reversible; a sent message, a submitted form, a deleted record, a disclosed secret are not."}],[{"text":"Detectability by the person affected.","b":true},{"text":" Can the user tell it was wrong. This is the axis most often left out and it dominates the others: an error a user can spot is a nuisance, and the same error where verification is impossible is a harm."}],[{"text":"Distribution.","b":true},{"text":" Who bears it. If the failure concentrates on one language, one region, or people with less ability to appeal, the average is actively misleading."}]]},{"t":"p","runs":[{"text":"That description is the input to everything else. Fifteen minutes of writing it usually reorders the team’s priorities."}]},{"t":"h2","id":"sample-the-tail-on-purpose","runs":[{"text":"Sample the tail on purpose"}]},{"t":"p","runs":[{"text":"Random sampling finds common failures. It finds the rare catastrophic one at the rate it occurs, which is to say almost never, on the day before it occurs in production."}]},{"t":"p","runs":[{"text":"So sample deliberately from where harm lives:"}]},{"t":"ul","items":[[{"text":"The lowest-scoring outputs by your judge."}],[{"text":"Requests in high-consequence categories (medical, legal, financial, safety, self-harm, immigration status) identified by a cheap classifier."}],[{"text":"Requests from users in the vulnerable slices you identified."}],[{"text":"The cases where a guardrail fired."}]]},{"t":"p","runs":[{"text":"Then have a person read them. A hundred deliberately-chosen items reviewed by a human tells you more about harm than ten thousand sampled ones scored by a model."}]},{"t":"h2","id":"write-the-cases-do-not-wait-for-them","runs":[{"text":"Write the cases; do not wait for them"}]},{"t":"p","runs":[{"text":"For each harm category, write fifty test cases. This is the same discipline as the safety and red-teaming lesson, applied to consequences rather than to attacks:"}]},{"t":"ul","items":[[{"text":"A user describing symptoms and asking what to take."}],[{"text":"A user asking whether their contract lets them leave."}],[{"text":"A user in evident distress."}],[{"text":"A question whose honest answer is “I do not know, and here is who does”."}]]},{"t":"p","runs":[{"text":"Score these on a severity-weighted basis and report the score separately from the general quality metric. And be explicit that the weights are a judgement: “we weight an irreversible financial error at fifty times a formatting error” is a value statement made visible, which is the most that can be asked of it. Hiding the weights inside an average does not make the judgement go away; it makes it unexaminable."}]},{"t":"h2","id":"design-changes-usually-beat-model-changes-here","runs":[{"text":"Design changes usually beat model changes here"}]},{"t":"p","runs":[{"text":"The most effective response to a harm is often not a better model. Three interventions reduce harm without touching quality at all, and they should be on the table in the same conversation:"}]},{"t":"ul","items":[[{"text":"Remove the irreversibility.","b":true},{"text":" A draft the user sends beats a message the system sends. A queued action with a ten-second undo beats an immediate one."}],[{"text":"Restore detectability.","b":true},{"text":" Show the source beside the answer, so verification takes two seconds instead of being impossible."}],[{"text":"Refuse and route.","b":true},{"text":" For the highest-consequence categories, the best output is a short answer that names where the real help is."}]]},{"t":"p","runs":[{"text":"Each is measurable: undo rate, source click-through, and the proportion of high-consequence requests that were routed rather than answered."}]},{"t":"h2","id":"near-misses-are-the-leading-indicator","runs":[{"text":"Near misses are the leading indicator"}]},{"t":"p","runs":[{"text":"Count the cases where a guardrail caught something before it reached a user, and track that rate over time. Near misses rise before incidents do, they are far more numerous, and they cost nothing to collect since the guardrail already fires."}]},{"t":"p","runs":[{"text":"A doubling in near misses after a prompt change is a finding, even if no harm reached anybody, and it is the earliest warning available."}]},{"t":"h2","id":"the-limit-that-matters-most","runs":[{"text":"The limit that matters most"}],"role":"limit"},{"t":"p","runs":[{"text":"You cannot enumerate harms in advance. Every system in production surprises its builders, usually because a group of users has a context nobody in the room shared."}]},{"t":"p","runs":[{"text":"The mitigations are procedural rather than statistical: a visible, easy way for people to report a problem; somebody whose job is to read those reports weekly; and a route from a report into the evaluation set, so the same harm cannot recur silently. A team that measures harm carefully and has no channel for the harms it did not predict is measuring the part of the problem it already understood."}]}],"check":{"question":[{"text":"Which failure most deserves priority, given equal frequency of 0.5%?"}],"options":[{"text":[{"text":"An incorrect summary of an article the user is reading, shown alongside the article."}],"correct":false,"tempting":[{"text":"Wrong and immediately checkable, since the source sits next to the output, so the user detects and corrects it at once."}]},{"text":[{"text":"A formatting error that makes a response hard to read on a small screen."}],"correct":false,"tempting":[{"text":"Highly visible and fully reversible; irritating rather than harmful, and a candidate for the ordinary quality backlog."}]},{"text":[{"text":"An incorrect eligibility answer that leads a user to abandon a benefits claim they were entitled to."}],"correct":true,"tempting":null},{"text":[{"text":"A wrong product recommendation in a list of five suggestions."}],"correct":false,"tempting":[{"text":"Low severity and easily ignored by the user, since the other four options remain visible and nothing irreversible follows."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"standards-and-frameworks","title":"What the emerging rules ask for, and what they do not settle","minutes":9},"next":{"slug":"human-oversight-that-is-real","title":"Measuring the human in the loop","minutes":9}}}