"A human approves everything" is not a safety design
It is a person doing the job slowly, with extra clicks. And it degrades on its own: show someone the twentieth near-identical approval dialog and they stop reading it. Approval fatigue is not a character flaw, it is what attention does. A gate that gets rubber-stamped is worse than no gate, because everyone now believes the work was checked.
So the design question is not *whether* to include a human. It is which decisions, and at what moment.
Gate on what cannot be undone
The useful axis is reversibility and blast radius. Not model confidence — more on that shortly.
| | Small blast radius | Large blast radius | |---|---|---| | Reversible | Let it run. (Draft a reply into a folder.) | Let it run, log loudly, one-click undo. (Tag 400 tickets.) | | Irreversible | Batch review. (Send one customer email.) | Hard gate, every time. (Issue refunds, delete records, publish, pay.) |
This is a better guide than instinct, because instinct gates on how scary the action *sounds*. Deleting 12,000 rows from a table with point-in-time restore is genuinely fine. Sending one WhatsApp message to a customer is not undoable and deserves more care than it usually gets.
Three placements, three costs
Before the act — approval. The person sees the proposed action and says yes. Highest friction, highest assurance. Reserve it for the bottom-right box.
After the act — review queue plus undo. The agent acts, everything lands in a queue, someone skims and reverses what is wrong. Far cheaper, and it scales. It only works where undo is genuinely real. "We can email a correction" is not undo.
On the exception — the agent escalates itself. The best ratio when it works: the agent runs its own check, and hands over only what fails. The honest caveat is that this requires the agent to detect its own errors, which is precisely the thing it is worst at. Use it where the check is external and mechanical — arithmetic that must balance, an ID that must exist, a total that must match a receipt — and not where the check is the model marking its own homework.
Make the approval readable in three seconds
The shape of the request determines whether it is read.
Bad:
Agent wants to run refund_order. Approve? [y/N]Good:
Refund ₹4,200 to arjun@example.com — order ORD-48213
Reason: item never shipped (courier shows no scan since 12 Aug)
This customer has had 0 previous refunds.
[Approve] [Reject] [Open the order]The second one contains the facts a person needs to disagree. The first one only offers a decision, and a decision without evidence becomes a habit.
The trust ratchet
Start with everything gated. Log every single approval decision — what the agent proposed, what the human chose. After a few hundred, look at where the human agreed every time. Those are the categories you can safely ungate; you have evidence, not a feeling. Where the human disagreed even occasionally, the gate stays and you have just found your highest-value bug report.
Ratchet one direction only, and slowly. Removing a gate should require the same seriousness as deploying.
Do not gate on confidence
One trap worth naming directly. It is very tempting to have the model rate its own output and route the low scores to a human. It reads as elegant and it does not work, because self-reported confidence is not a measurement of correctness. A model is often most fluent, most assured and most detailed exactly when it is inventing — that is what confident error looks like from the inside. Route on category and consequence, which you can observe. Not on a number the model made up about itself.
Before you move on