Measuring whether it actually helped

Why the question is hard

Everybody has an impression and the impressions conflict. Enthusiasts report enormous gains; sceptics report none; both are describing their own work honestly. The reason is that the effect differs by task, and an average across a team hides everything interesting.

There is also a specific bias to correct for. These tools are most impressive on the tasks people find least pleasant, which makes relief feel like productivity. Getting through a dreaded email in four minutes instead of twenty is a real gain in wellbeing and may be a smaller gain in time than it feels.

The measurement that is worth doing

Two weeks, one line a day, per person:

minutes saved   |  minutes spent checking  |  one thing it could not do

That is it. It takes ninety seconds a day, and after two weeks you have something to look at instead of an argument.

Two refinements that make it much better. Record the task, not just the time, so you can see which kinds of work produced the gains. And record it on the day, because reconstructing a fortnight from memory produces the impression you already had.

What to measure, and what not to

Worth measuring: time to a usable draft, time spent checking, number of tasks completed that would otherwise have been deferred, errors caught before they went out, and the tasks that got done at all because the barrier dropped.

Not worth measuring: messages sent to the assistant, lines of code produced, words written, logins. Every one of these can be increased without producing anything of value, and measuring them reliably causes people to increase them. A team measured on lines of code will produce more lines of code.

The distinction is between measuring output and measuring work completed to a standard. Only the second is worth anything, and it is harder to count, which is precisely why the easy metrics get used.

The honest complications

Quality change is harder to see than time change. A draft produced in a quarter of the time might be slightly worse, and slightly worse across a hundred documents is a real cost that no time measurement captures. Where quality matters, somebody has to read a sample of before and after.

The learning curve confounds the first month. People are slower while learning and faster afterwards. A measurement taken in week one understates; one taken in month six on the tasks that survived is closer to the truth.

Selection effects. People use the tool where it works and stop where it does not, which means measured tasks are the favourable ones. This is fine (it is how tools are supposed to be used) as long as nobody extrapolates from it to tasks nobody chose to try.

Displaced work. Time saved on drafting sometimes reappears as time spent reviewing somebody else’s AI-assisted draft. At an organisational level that can net to nothing while every individual honestly reports a saving.

Comparing against the right baseline

One error makes every measurement flattering. The comparison is not against doing the task perfectly by hand with unlimited time. It is against what would actually have happened: the email that would have been sent two days later, the report that would have been shorter, the analysis nobody would have done at all.

Some of the largest genuine gains are in that last category: work that was not worth the effort before and is now. Those do not appear as time saved, because no time was being spent, and a measurement that only counts minutes saved misses them entirely. Note them in the third column and count them separately.

What a good result looks like

The typical honest finding, from teams that have measured properly, is not a uniform improvement. It is a wide distribution: a few tasks transformed, many modestly improved, a few made worse by the checking burden.

That is a useful result, and it points somewhere specific: do more of the first category, stop doing the third, and stop arguing about the average. An organisation that knows which of its tasks are which is in a much better position than one with a single headline number, whichever direction that number points.