{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"human-in-the-loop","title":"Where the person goes","url":"https://addaly.com/learn/agents-and-automation/human-in-the-loop","position":44,"of":72,"minutes":8,"answer":"Gate on what cannot be undone, never on how confident the model sounds.","keyIdea":"Gate on what cannot be undone, never on how confident the model sounds."},"body":{"format":"markdown","text":"## \"A human approves everything\" is not a safety design\n\nIt is a person doing the job slowly, with extra clicks. And it degrades on its own: show someone the twentieth near-identical approval dialog and they stop reading it. Approval fatigue is not a character flaw, it is what attention does. A gate that gets rubber-stamped is worse than no gate, because everyone now believes the work was checked.\n\nSo the design question is not *whether* to include a human. It is **which decisions**, and **at what moment**.\n\n## Gate on what cannot be undone\n\nThe useful axis is reversibility and blast radius. Not model confidence, more on that shortly.\n\n| | **Small blast radius** | **Large blast radius** |\n|---|---|---|\n| **Reversible** | Let it run. (Draft a reply into a folder.) | Let it run, log loudly, one-click undo. (Tag 400 tickets.) |\n| **Irreversible** | Batch review. (Send one customer email.) | Hard gate, every time. (Issue refunds, delete records, publish, pay.) |\n\nThis is a better guide than instinct, because instinct gates on how scary the action *sounds*. Deleting 12,000 rows from a table with point-in-time restore is fine. Sending one WhatsApp message to a customer is not undoable and deserves more care than it usually gets.\n\n## Three placements, three costs\n\n**Before the act: approval.** The person sees the proposed action and says yes. Highest friction, highest assurance. Reserve it for the bottom-right box.\n\n> Figure: Where the person goes: reversibility against blast radius. A two by two grid, rows Reversible and Irreversible, columns Small blast radius and Large blast radius: Reversible and Small blast radius: Let it run; Reversible and Large blast radius: Run it, log loudly, one-click undo; Irreversible and Small blast radius: Batch review; Irreversible and Large blast radius: Hard gate, every time. Gate on what cannot be undone, never on how confident the model sounds. Deleting twelve thousand rows from a table with point-in-time restore is genuinely fine; one message to a customer is not.\n\n**After the act: review queue plus undo.** The agent acts, everything lands in a queue, someone skims and reverses what is wrong. Far cheaper, and it scales. It only works where undo is real. \"We can email a correction\" is not undo.\n\n**On the exception: the agent escalates itself.** The best ratio when it works: the agent runs its own check, and hands over only what fails. The honest caveat is that this requires the agent to detect its own errors, which is precisely the thing it is worst at. Use it where the check is external and mechanical (arithmetic that must balance, an ID that must exist, a total that must match a receipt) and not where the check is the model marking its own homework.\n\n## Make the approval readable in three seconds\n\nThe shape of the request determines whether it is read.\n\nBad:\n\n```\nAgent wants to run refund_order. Approve? [y/N]\n```\n\nGood:\n\n```\nRefund ₹4,200 to arjun@example.com — order ORD-48213\nReason: item never shipped (courier shows no scan since 12 Aug)\nThis customer has had 0 previous refunds.\n\n[Approve]  [Reject]  [Open the order]\n```\n\nThe second one contains the facts a person needs to disagree. The first one only offers a decision, and a decision without evidence becomes a habit.\n\n## The trust ratchet\n\nStart with everything gated. Log every single approval decision: what the agent proposed, what the human chose. After a few hundred, look at where the human agreed every time. Those are the categories you can safely ungate; you have evidence, not a feeling. Where the human disagreed even occasionally, the gate stays and you have just found your highest-value bug report.\n\nRatchet one direction only, and slowly. Removing a gate should require the same seriousness as deploying.\n\n## Do not gate on confidence\n\nOne trap worth naming directly. It is very tempting to have the model rate its own output and route the low scores to a human. It reads as elegant and it does not work, because self-reported confidence is not a measurement of correctness. A model is often most fluent, most assured and most detailed exactly when it is inventing: that is what confident error looks like from the inside. Route on category and consequence, which you can observe. Not on a number the model made up about itself."},"check":{"question":"A content agent drafts and publishes posts. The team gates on self-assessed quality: anything the model scores below 8/10 goes to a human, anything above publishes automatically. Six weeks in, the worst published post (full of invented citations) was self-scored 9/10. What went wrong?","options":[{"text":"Self-reported confidence does not measure correctness, and fluent invention scores high, so the gate systematically passed the failures it was meant to catch.","correct":true},{"text":"The threshold was set too low; moving it to 9.5 would have routed this post to a human.","correct":false,"tempting":"Treats a miscalibrated signal as merely mis-thresholded: moving the line on a bad signal only changes which errors slip past."},{"text":"The scoring prompt was too vague; a detailed rubric would make the self-score trustworthy.","correct":false,"tempting":"Assumes calibration is a prompting problem, when a rubric improves consistency without giving the model access to what it does not know."},{"text":"The human reviewers were the weak link: they approved posts they should have rejected.","correct":false,"tempting":"Blames the reviewers, but this post was never routed to one; the routing rule is what failed."}]},"prev":{"slug":"secrets-and-agent-identity","title":"The agent gets its own account","url":"https://addaly.com/learn/agents-and-automation/secrets-and-agent-identity"},"next":{"slug":"irreversible-actions","title":"Money, messages and deletions","url":"https://addaly.com/learn/agents-and-automation/irreversible-actions"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where the person goes', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/human-in-the-loop (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where the person goes","courseTitle":"Agents and Automation","module":{"index":5,"slug":"safety-and-blast-radius","title":"Permissions, injection and blast radius"},"answer":null,"keyIdea":"Gate on what cannot be undone, never on how confident the model sounds.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Small blast radius","Large blast radius"],"rows":["Reversible","Irreversible"],"type":"matrix","cells":[["Let it run","Run it, log loudly, one-click undo"],["Batch review","Hard gate, every time"]],"title":"Where the person goes: reversibility against blast radius","caption":"Gate on what cannot be undone, never on how confident the model sounds. Deleting twelve thousand rows from a table with point-in-time restore is genuinely fine; one message to a customer is not.","cellLabels":[["draft a reply into a folder","tag four hundred tickets"],["send one customer email","issue refunds, delete records, publish, pay"]]},"alt":"Figure: Where the person goes: reversibility against blast radius. A two by two grid, rows Reversible and Irreversible, columns Small blast radius and Large blast radius: Reversible and Small blast radius: Let it run; Reversible and Large blast radius: Run it, log loudly, one-click undo; Irreversible and Small blast radius: Batch review; Irreversible and Large blast radius: Hard gate, every time. Gate on what cannot be undone, never on how confident the model sounds. Deleting twelve thousand rows from a table with point-in-time restore is genuinely fine; one message to a customer is not."}},"blocks":[{"t":"h2","id":"a-human-approves-everything-is-not-a-safety-design","runs":[{"text":"“A human approves everything” is not a safety design"}]},{"t":"p","runs":[{"text":"It is a person doing the job slowly, with extra clicks. And it degrades on its own: show someone the twentieth near-identical approval dialog and they stop reading it. Approval fatigue is not a character flaw, it is what attention does. A gate that gets rubber-stamped is worse than no gate, because everyone now believes the work was checked."}]},{"t":"p","runs":[{"text":"So the design question is not "},{"text":"whether","i":true},{"text":" to include a human. It is "},{"text":"which decisions","b":true},{"text":", and "},{"text":"at what moment","b":true},{"text":"."}]},{"t":"h2","id":"gate-on-what-cannot-be-undone","runs":[{"text":"Gate on what cannot be undone"}]},{"t":"p","runs":[{"text":"The useful axis is reversibility and blast radius. Not model confidence, more on that shortly."}]},{"t":"table","head":[[],[{"text":"Small blast radius","b":true}],[{"text":"Large blast radius","b":true}]],"align":["left","left","left"],"rows":[[[{"text":"Reversible","b":true}],[{"text":"Let it run. (Draft a reply into a folder.)"}],[{"text":"Let it run, log loudly, one-click undo. (Tag 400 tickets.)"}]],[[{"text":"Irreversible","b":true}],[{"text":"Batch review. (Send one customer email.)"}],[{"text":"Hard gate, every time. (Issue refunds, delete records, publish, pay.)"}]]]},{"t":"p","runs":[{"text":"This is a better guide than instinct, because instinct gates on how scary the action "},{"text":"sounds","i":true},{"text":". Deleting 12,000 rows from a table with point-in-time restore is fine. Sending one WhatsApp message to a customer is not undoable and deserves more care than it usually gets."}]},{"t":"h2","id":"three-placements-three-costs","runs":[{"text":"Three placements, three costs"}]},{"t":"p","runs":[{"text":"Before the act: approval.","b":true},{"text":" The person sees the proposed action and says yes. Highest friction, highest assurance. Reserve it for the bottom-right box."}]},{"t":"p","runs":[{"text":"After the act: review queue plus undo.","b":true},{"text":" The agent acts, everything lands in a queue, someone skims and reverses what is wrong. Far cheaper, and it scales. It only works where undo is real. “We can email a correction” is not undo."}]},{"t":"p","runs":[{"text":"On the exception: the agent escalates itself.","b":true},{"text":" The best ratio when it works: the agent runs its own check, and hands over only what fails. The honest caveat is that this requires the agent to detect its own errors, which is precisely the thing it is worst at. Use it where the check is external and mechanical (arithmetic that must balance, an ID that must exist, a total that must match a receipt) and not where the check is the model marking its own homework."}]},{"t":"h2","id":"make-the-approval-readable-in-three-seconds","runs":[{"text":"Make the approval readable in three seconds"}]},{"t":"p","runs":[{"text":"The shape of the request determines whether it is read."}]},{"t":"p","runs":[{"text":"Bad:"}]},{"t":"code","lang":"","label":"Text","text":"Agent wants to run refund_order. Approve? [y/N]"},{"t":"p","runs":[{"text":"Good:"}]},{"t":"code","lang":"","label":"Text","text":"Refund ₹4,200 to arjun@example.com — order ORD-48213\nReason: item never shipped (courier shows no scan since 12 Aug)\nThis customer has had 0 previous refunds.\n\n[Approve]  [Reject]  [Open the order]"},{"t":"p","runs":[{"text":"The second one contains the facts a person needs to disagree. The first one only offers a decision, and a decision without evidence becomes a habit."}]},{"t":"h2","id":"the-trust-ratchet","runs":[{"text":"The trust ratchet"}]},{"t":"p","runs":[{"text":"Start with everything gated. Log every single approval decision: what the agent proposed, what the human chose. After a few hundred, look at where the human agreed every time. Those are the categories you can safely ungate; you have evidence, not a feeling. Where the human disagreed even occasionally, the gate stays and you have just found your highest-value bug report."}]},{"t":"p","runs":[{"text":"Ratchet one direction only, and slowly. Removing a gate should require the same seriousness as deploying."}]},{"t":"h2","id":"do-not-gate-on-confidence","runs":[{"text":"Do not gate on confidence"}]},{"t":"p","runs":[{"text":"One trap worth naming directly. It is very tempting to have the model rate its own output and route the low scores to a human. It reads as elegant and it does not work, because self-reported confidence is not a measurement of correctness. A model is often most fluent, most assured and most detailed exactly when it is inventing: that is what confident error looks like from the inside. Route on category and consequence, which you can observe. Not on a number the model made up about itself."}]}],"check":{"question":[{"text":"A content agent drafts and publishes posts. The team gates on self-assessed quality: anything the model scores below 8/10 goes to a human, anything above publishes automatically. Six weeks in, the worst published post (full of invented citations) was self-scored 9/10. What went wrong?"}],"options":[{"text":[{"text":"Self-reported confidence does not measure correctness, and fluent invention scores high, so the gate systematically passed the failures it was meant to catch."}],"correct":true,"tempting":null},{"text":[{"text":"The threshold was set too low; moving it to 9.5 would have routed this post to a human."}],"correct":false,"tempting":[{"text":"Treats a miscalibrated signal as merely mis-thresholded: moving the line on a bad signal only changes which errors slip past."}]},{"text":[{"text":"The scoring prompt was too vague; a detailed rubric would make the self-score trustworthy."}],"correct":false,"tempting":[{"text":"Assumes calibration is a prompting problem, when a rubric improves consistency without giving the model access to what it does not know."}]},{"text":[{"text":"The human reviewers were the weak link: they approved posts they should have rejected."}],"correct":false,"tempting":[{"text":"Blames the reviewers, but this post was never routed to one; the routing rule is what failed."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"secrets-and-agent-identity","title":"The agent gets its own account","minutes":9},"next":{"slug":"irreversible-actions","title":"Money, messages and deletions","minutes":8}}}