Measuring the worst case, not the average one

Error rate and harm are different quantities

A system with a 20% error rate can be harmless: the errors are obvious, the user notices immediately, the cost is a repeated request. A system with a 0.1% error rate can be dangerous: the errors are invisible, they land on people who cannot verify them, and they are irreversible.

Averaging over items treats those the same. Something else is needed.

Describe each failure mode along four axes

For every way the system can go wrong, write down:

• Severity. The worst realistic outcome, stated concretely. Not “poor user experience”, “a customer misses a filing deadline and pays a penalty”.
• Reversibility. Can it be undone, and by whom, and at what cost. A wrong draft is reversible; a sent message, a submitted form, a deleted record, a disclosed secret are not.
• Detectability by the person affected. Can the user tell it was wrong. This is the axis most often left out and it dominates the others: an error a user can spot is a nuisance, and the same error where verification is impossible is a harm.
• Distribution. Who bears it. If the failure concentrates on one language, one region, or people with less ability to appeal, the average is actively misleading.

That description is the input to everything else. Fifteen minutes of writing it usually reorders the team’s priorities.

Sample the tail on purpose

Random sampling finds common failures. It finds the rare catastrophic one at the rate it occurs, which is to say almost never, on the day before it occurs in production.

So sample deliberately from where harm lives:

• The lowest-scoring outputs by your judge.
• Requests in high-consequence categories (medical, legal, financial, safety, self-harm, immigration status) identified by a cheap classifier.
• Requests from users in the vulnerable slices you identified.
• The cases where a guardrail fired.

Then have a person read them. A hundred deliberately-chosen items reviewed by a human tells you more about harm than ten thousand sampled ones scored by a model.

Write the cases; do not wait for them

For each harm category, write fifty test cases. This is the same discipline as the safety and red-teaming lesson, applied to consequences rather than to attacks:

• A user describing symptoms and asking what to take.
• A user asking whether their contract lets them leave.
• A user in evident distress.
• A question whose honest answer is “I do not know, and here is who does”.

Score these on a severity-weighted basis and report the score separately from the general quality metric. And be explicit that the weights are a judgement: “we weight an irreversible financial error at fifty times a formatting error” is a value statement made visible, which is the most that can be asked of it. Hiding the weights inside an average does not make the judgement go away; it makes it unexaminable.

Design changes usually beat model changes here

The most effective response to a harm is often not a better model. Three interventions reduce harm without touching quality at all, and they should be on the table in the same conversation:

• Remove the irreversibility. A draft the user sends beats a message the system sends. A queued action with a ten-second undo beats an immediate one.
• Restore detectability. Show the source beside the answer, so verification takes two seconds instead of being impossible.
• Refuse and route. For the highest-consequence categories, the best output is a short answer that names where the real help is.

Each is measurable: undo rate, source click-through, and the proportion of high-consequence requests that were routed rather than answered.

Near misses are the leading indicator

Count the cases where a guardrail caught something before it reached a user, and track that rate over time. Near misses rise before incidents do, they are far more numerous, and they cost nothing to collect since the guardrail already fires.

A doubling in near misses after a prompt change is a finding, even if no harm reached anybody, and it is the earliest warning available.

The limit that matters most

You cannot enumerate harms in advance. Every system in production surprises its builders, usually because a group of users has a context nobody in the room shared.

The mitigations are procedural rather than statistical: a visible, easy way for people to report a problem; somebody whose job is to read those reports weekly; and a route from a report into the evaluation set, so the same harm cannot recur silently. A team that measures harm carefully and has no channel for the harms it did not predict is measuring the part of the problem it already understood.