Did it actually help, and how would you know

Before-and-after is contaminated

The default method is to compare the month before with the month after. It is also the method that cannot support a conclusion, because the month after differs in every other way too: seasonality, a staffing change, a product launch, one large customer leaving, a policy that changed in the same fortnight.

Three better designs, all cheap:

Shadow. Run the automation alongside the humans, changing nothing, and compare outputs. Gives you accuracy on live traffic with no risk. It does not tell you about time saved, because nothing was saved.

Holdout. Apply the automation to 80 per cent of items and deliberately leave 20 per cent alone, split by hash of a stable id. Both groups live in the same week, with the same staff and the same weather. This is the strongest evidence available for the effort, and the objection (“we’re deliberately not helping some cases”) is worth answering with the honest reply that you do not yet know it helps.

Staged rollout. Region by region or team by team, with the not-yet-rolled-out groups as the comparison. Weaker, because the groups differ, and often the only politically possible option.

The freed-hour question

Somebody will report that the automation saves twelve hours a week. Ask what those twelve hours became.

Sometimes the answer is concrete: the same team now handles 30 per cent more volume without hiring, or a two-day turnaround became four hours. That is value, and it can be stated.

Sometimes the answer is that the work was spread across eleven people at an hour each, and each of them now has an hour that has not produced anything visible. That is real relief for eleven people and it is not a saving the business can bank. Both facts can be said in the same sentence, and saying them is what makes the rest of your numbers believable.

The claim that ages worst is a headcount saving nobody ever made. Do not write it down unless somebody is actually planning to act on it, because it will be quoted back.

Measures that survive scrutiny

• Throughput at constant headcount. Items handled per week with the same team.
• Cycle time. Arrival to resolution, at the median and the 90th percentile. The 90th is where the pain lives and where automation usually helps most.
• Backlog age. The oldest unresolved item.
• Error and rework rate. Items that came back.
• Cost per item, all in: model spend, infrastructure, and the review time the automation created.

That last term is the one teams omit. An automation that saves four minutes of processing and adds ninety seconds of review has saved two and a half minutes, and the honest arithmetic is the one that survives the first challenge.

Watch the denominator

If volume grew 20 per cent while your cost per item fell 15 per cent, some of that is fixed costs spread wider and none of it is necessarily your automation. Compare like periods, hold the mix constant where you can, and say which effects you have not separated. A stated limitation makes a report more credible, not less.

Measure the quality change, not only the speed change

Faster and worse is a common outcome and an easy one to miss, because speed is instrumented automatically and quality is not. Pick one quality measure that existed before the automation (complaints per thousand items, rework rate, the proportion of cases reopened within a week) and track it alongside throughput from the first day.

The comparison you want is a pair: what moved, and what did not get worse. A report showing cycle time down 40 per cent and rework flat is persuasive in a way that cycle time alone never is, because the first question anybody sensible asks is what you gave up to get it.

Publish the method with the number

Write it as: what we measured, over what period, against what comparison, with what we could not control for. One paragraph.

A value claim nobody can check is a claim that will be quietly disbelieved the first time the system misbehaves, and at that moment the difference between “they said it saved twelve hours” and “they showed the method and the caveats” is the difference between losing the automation and fixing it.

And be willing to conclude nothing happened

Sometimes the honest finding is that cycle time did not move. That is a real result, it is worth more than an invented one, and a team that can report it is a team whose next set of numbers will be believed.