{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"proving-the-value","title":"Did it actually help, and how would you know","url":"https://addaly.com/learn/agents-and-automation/proving-the-value","position":66,"of":72,"minutes":9,"answer":"Hours saved is the weakest claim available, because a saved hour only becomes value if you can say what it turned into.","keyIdea":"Hours saved is the weakest claim available, because a saved hour only becomes value if you can say what it turned into."},"body":{"format":"markdown","text":"## Before-and-after is contaminated\n\nThe default method is to compare the month before with the month after. It is also the method that cannot support a conclusion, because the month after differs in every other way too: seasonality, a staffing change, a product launch, one large customer leaving, a policy that changed in the same fortnight.\n\nThree better designs, all cheap:\n\n**Shadow.** Run the automation alongside the humans, changing nothing, and compare outputs. Gives you accuracy on live traffic with no risk. It does not tell you about time saved, because nothing was saved.\n\n**Holdout.** Apply the automation to 80 per cent of items and deliberately leave 20 per cent alone, split by hash of a stable id. Both groups live in the same week, with the same staff and the same weather. This is the strongest evidence available for the effort, and the objection (\"we're deliberately not helping some cases\") is worth answering with the honest reply that you do not yet know it helps.\n\n**Staged rollout.** Region by region or team by team, with the not-yet-rolled-out groups as the comparison. Weaker, because the groups differ, and often the only politically possible option.\n\n## The freed-hour question\n\nSomebody will report that the automation saves twelve hours a week. Ask what those twelve hours became.\n\nSometimes the answer is concrete: the same team now handles 30 per cent more volume without hiring, or a two-day turnaround became four hours. That is value, and it can be stated.\n\nSometimes the answer is that the work was spread across eleven people at an hour each, and each of them now has an hour that has not produced anything visible. That is real relief for eleven people and it is not a saving the business can bank. Both facts can be said in the same sentence, and saying them is what makes the rest of your numbers believable.\n\nThe claim that ages worst is a headcount saving nobody ever made. Do not write it down unless somebody is actually planning to act on it, because it will be quoted back.\n\n## Measures that survive scrutiny\n\n- **Throughput at constant headcount.** Items handled per week with the same team.\n- **Cycle time.** Arrival to resolution, at the median and the 90th percentile. The 90th is where the pain lives and where automation usually helps most.\n- **Backlog age.** The oldest unresolved item.\n- **Error and rework rate.** Items that came back.\n- **Cost per item**, all in: model spend, infrastructure, and the review time the automation created.\n\nThat last term is the one teams omit. An automation that saves four minutes of processing and adds ninety seconds of review has saved two and a half minutes, and the honest arithmetic is the one that survives the first challenge.\n\n## Watch the denominator\n\nIf volume grew 20 per cent while your cost per item fell 15 per cent, some of that is fixed costs spread wider and none of it is necessarily your automation. Compare like periods, hold the mix constant where you can, and say which effects you have not separated. A stated limitation makes a report more credible, not less.\n\n## Measure the quality change, not only the speed change\n\nFaster and worse is a common outcome and an easy one to miss, because speed is instrumented automatically and quality is not. Pick one quality measure that existed before the automation (complaints per thousand items, rework rate, the proportion of cases reopened within a week) and track it alongside throughput from the first day.\n\nThe comparison you want is a pair: what moved, and what did not get worse. A report showing cycle time down 40 per cent and rework flat is persuasive in a way that cycle time alone never is, because the first question anybody sensible asks is what you gave up to get it.\n\n## Publish the method with the number\n\nWrite it as: what we measured, over what period, against what comparison, with what we could not control for. One paragraph.\n\nA value claim nobody can check is a claim that will be quietly disbelieved the first time the system misbehaves, and at that moment the difference between \"they said it saved twelve hours\" and \"they showed the method and the caveats\" is the difference between losing the automation and fixing it.\n\n## And be willing to conclude nothing happened\n\nSometimes the honest finding is that cycle time did not move. That is a real result, it is worth more than an invented one, and a team that can report it is a team whose next set of numbers will be believed."},"check":{"question":"A team reports that an automation saves 12 hours a week, spread across eleven people. What is the right way to state this?","options":[{"text":"As 12 hours of capacity released, valued at the loaded hourly cost of those staff.","correct":false,"tempting":"Converts scattered minutes into a currency figure that no budget will ever show, which is the claim most likely to be challenged."},{"text":"As a headcount saving of roughly a third of a full-time role, since 12 hours is close to that.","correct":false,"tempting":"Aggregates time that is not fungible into a post nobody will remove, and commits the project to a reduction no one planned."},{"text":"As relief distributed across eleven people, stating plainly that it has not yet converted into throughput or turnaround the business can measure.","correct":true},{"text":"As unproven until a holdout study is run, with no interim claim made at all.","correct":false,"tempting":"Over-corrects into silence; the time is genuinely being saved and the honest description of it is available now."}]},"prev":{"slug":"who-owns-it-when-it-breaks","title":"Whose is it, at 3am, in eight months","url":"https://addaly.com/learn/agents-and-automation/who-owns-it-when-it-breaks"},"next":{"slug":"the-bill-and-who-reads-it","title":"What it costs, and where the cost hides","url":"https://addaly.com/learn/agents-and-automation/the-bill-and-who-reads-it"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Did it actually help, and how would you know', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/proving-the-value (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Did it actually help, and how would you know","courseTitle":"Agents and Automation","module":{"index":8,"slug":"living-with-it","title":"The year after it ships"},"answer":null,"keyIdea":"Hours saved is the weakest claim available, because a saved hour only becomes value if you can say what it turned into.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"before-and-after-is-contaminated","runs":[{"text":"Before-and-after is contaminated"}]},{"t":"p","runs":[{"text":"The default method is to compare the month before with the month after. It is also the method that cannot support a conclusion, because the month after differs in every other way too: seasonality, a staffing change, a product launch, one large customer leaving, a policy that changed in the same fortnight."}]},{"t":"p","runs":[{"text":"Three better designs, all cheap:"}]},{"t":"p","runs":[{"text":"Shadow.","b":true},{"text":" Run the automation alongside the humans, changing nothing, and compare outputs. Gives you accuracy on live traffic with no risk. It does not tell you about time saved, because nothing was saved."}]},{"t":"p","runs":[{"text":"Holdout.","b":true},{"text":" Apply the automation to 80 per cent of items and deliberately leave 20 per cent alone, split by hash of a stable id. Both groups live in the same week, with the same staff and the same weather. This is the strongest evidence available for the effort, and the objection (“we’re deliberately not helping some cases”) is worth answering with the honest reply that you do not yet know it helps."}]},{"t":"p","runs":[{"text":"Staged rollout.","b":true},{"text":" Region by region or team by team, with the not-yet-rolled-out groups as the comparison. Weaker, because the groups differ, and often the only politically possible option."}]},{"t":"h2","id":"the-freed-hour-question","runs":[{"text":"The freed-hour question"}]},{"t":"p","runs":[{"text":"Somebody will report that the automation saves twelve hours a week. Ask what those twelve hours became."}]},{"t":"p","runs":[{"text":"Sometimes the answer is concrete: the same team now handles 30 per cent more volume without hiring, or a two-day turnaround became four hours. That is value, and it can be stated."}]},{"t":"p","runs":[{"text":"Sometimes the answer is that the work was spread across eleven people at an hour each, and each of them now has an hour that has not produced anything visible. That is real relief for eleven people and it is not a saving the business can bank. Both facts can be said in the same sentence, and saying them is what makes the rest of your numbers believable."}]},{"t":"p","runs":[{"text":"The claim that ages worst is a headcount saving nobody ever made. Do not write it down unless somebody is actually planning to act on it, because it will be quoted back."}]},{"t":"h2","id":"measures-that-survive-scrutiny","runs":[{"text":"Measures that survive scrutiny"}]},{"t":"ul","items":[[{"text":"Throughput at constant headcount.","b":true},{"text":" Items handled per week with the same team."}],[{"text":"Cycle time.","b":true},{"text":" Arrival to resolution, at the median and the 90th percentile. The 90th is where the pain lives and where automation usually helps most."}],[{"text":"Backlog age.","b":true},{"text":" The oldest unresolved item."}],[{"text":"Error and rework rate.","b":true},{"text":" Items that came back."}],[{"text":"Cost per item","b":true},{"text":", all in: model spend, infrastructure, and the review time the automation created."}]]},{"t":"p","runs":[{"text":"That last term is the one teams omit. An automation that saves four minutes of processing and adds ninety seconds of review has saved two and a half minutes, and the honest arithmetic is the one that survives the first challenge."}]},{"t":"h2","id":"watch-the-denominator","runs":[{"text":"Watch the denominator"}]},{"t":"p","runs":[{"text":"If volume grew 20 per cent while your cost per item fell 15 per cent, some of that is fixed costs spread wider and none of it is necessarily your automation. Compare like periods, hold the mix constant where you can, and say which effects you have not separated. A stated limitation makes a report more credible, not less."}]},{"t":"h2","id":"measure-the-quality-change-not-only-the-speed-change","runs":[{"text":"Measure the quality change, not only the speed change"}]},{"t":"p","runs":[{"text":"Faster and worse is a common outcome and an easy one to miss, because speed is instrumented automatically and quality is not. Pick one quality measure that existed before the automation (complaints per thousand items, rework rate, the proportion of cases reopened within a week) and track it alongside throughput from the first day."}]},{"t":"p","runs":[{"text":"The comparison you want is a pair: what moved, and what did not get worse. A report showing cycle time down 40 per cent and rework flat is persuasive in a way that cycle time alone never is, because the first question anybody sensible asks is what you gave up to get it."}]},{"t":"h2","id":"publish-the-method-with-the-number","runs":[{"text":"Publish the method with the number"}]},{"t":"p","runs":[{"text":"Write it as: what we measured, over what period, against what comparison, with what we could not control for. One paragraph."}]},{"t":"p","runs":[{"text":"A value claim nobody can check is a claim that will be quietly disbelieved the first time the system misbehaves, and at that moment the difference between “they said it saved twelve hours” and “they showed the method and the caveats” is the difference between losing the automation and fixing it."}]},{"t":"h2","id":"and-be-willing-to-conclude-nothing-happened","runs":[{"text":"And be willing to conclude nothing happened"}]},{"t":"p","runs":[{"text":"Sometimes the honest finding is that cycle time did not move. That is a real result, it is worth more than an invented one, and a team that can report it is a team whose next set of numbers will be believed."}]}],"check":{"question":[{"text":"A team reports that an automation saves 12 hours a week, spread across eleven people. What is the right way to state this?"}],"options":[{"text":[{"text":"As 12 hours of capacity released, valued at the loaded hourly cost of those staff."}],"correct":false,"tempting":[{"text":"Converts scattered minutes into a currency figure that no budget will ever show, which is the claim most likely to be challenged."}]},{"text":[{"text":"As a headcount saving of roughly a third of a full-time role, since 12 hours is close to that."}],"correct":false,"tempting":[{"text":"Aggregates time that is not fungible into a post nobody will remove, and commits the project to a reduction no one planned."}]},{"text":[{"text":"As relief distributed across eleven people, stating plainly that it has not yet converted into throughput or turnaround the business can measure."}],"correct":true,"tempting":null},{"text":[{"text":"As unproven until a holdout study is run, with no interim claim made at all."}],"correct":false,"tempting":[{"text":"Over-corrects into silence; the time is genuinely being saved and the honest description of it is available now."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"who-owns-it-when-it-breaks","title":"Whose is it, at 3am, in eight months","minutes":9},"next":{"slug":"the-bill-and-who-reads-it","title":"What it costs, and where the cost hides","minutes":9}}}