{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"measuring-whether-it-helped","title":"Measuring whether it actually helped","url":"https://addaly.com/learn/choosing-your-tools/measuring-whether-it-helped","position":73,"of":76,"minutes":8,"answer":"Record minutes saved, minutes spent checking and one thing it could not do, on the day, per task, and never measure messages, words or lines produced.","keyIdea":"Record minutes saved, minutes spent checking and one thing it could not do, on the day, per task, and never measure messages, words or lines produced, because those rise without anything being completed to a standard."},"body":{"format":"markdown","text":"## Why the question is hard\n\nEverybody has an impression and the impressions conflict. Enthusiasts report enormous gains; sceptics report none; both are describing their own work honestly. The reason is that the effect differs by task, and an average across a team hides everything interesting.\n\nThere is also a specific bias to correct for. These tools are most impressive on the tasks people find least pleasant, which makes relief feel like productivity. Getting through a dreaded email in four minutes instead of twenty is a real gain in wellbeing and may be a smaller gain in time than it feels.\n\n## The measurement that is worth doing\n\nTwo weeks, one line a day, per person:\n\n```\nminutes saved   |  minutes spent checking  |  one thing it could not do\n```\n\nThat is it. It takes ninety seconds a day, and after two weeks you have something to look at instead of an argument.\n\nTwo refinements that make it much better. Record the *task*, not just the time, so you can see which kinds of work produced the gains. And record it on the day, because reconstructing a fortnight from memory produces the impression you already had.\n\n## What to measure, and what not to\n\n**Worth measuring:** time to a usable draft, time spent checking, number of tasks completed that would otherwise have been deferred, errors caught before they went out, and the tasks that got done at all because the barrier dropped.\n\n**Not worth measuring:** messages sent to the assistant, lines of code produced, words written, logins. Every one of these can be increased without producing anything of value, and measuring them reliably causes people to increase them. A team measured on lines of code will produce more lines of code.\n\nThe distinction is between measuring output and measuring work completed to a standard. Only the second is worth anything, and it is harder to count, which is precisely why the easy metrics get used.\n\n## The honest complications\n\n**Quality change is harder to see than time change.** A draft produced in a quarter of the time might be slightly worse, and slightly worse across a hundred documents is a real cost that no time measurement captures. Where quality matters, somebody has to read a sample of before and after.\n\n**The learning curve confounds the first month.** People are slower while learning and faster afterwards. A measurement taken in week one understates; one taken in month six on the tasks that survived is closer to the truth.\n\n**Selection effects.** People use the tool where it works and stop where it does not, which means measured tasks are the favourable ones. This is fine (it is how tools are supposed to be used) as long as nobody extrapolates from it to tasks nobody chose to try.\n\n**Displaced work.** Time saved on drafting sometimes reappears as time spent reviewing somebody else's AI-assisted draft. At an organisational level that can net to nothing while every individual honestly reports a saving.\n\n## Comparing against the right baseline\n\nOne error makes every measurement flattering. The comparison is not against doing the task perfectly by hand with unlimited time. It is against what would actually have happened: the email that would have been sent two days later, the report that would have been shorter, the analysis nobody would have done at all.\n\nSome of the largest genuine gains are in that last category: work that was not worth the effort before and is now. Those do not appear as time saved, because no time was being spent, and a measurement that only counts minutes saved misses them entirely. Note them in the third column and count them separately.\n\n## What a good result looks like\n\nThe typical honest finding, from teams that have measured properly, is not a uniform improvement. It is a wide distribution: a few tasks transformed, many modestly improved, a few made worse by the checking burden.\n\nThat is a useful result, and it points somewhere specific: do more of the first category, stop doing the third, and stop arguing about the average. An organisation that knows which of its tasks are which is in a much better position than one with a single headline number, whichever direction that number points."},"check":{"question":"A team measures adoption by counting assistant messages per person, and the number rises steadily. Why is this poor evidence of benefit?","options":[{"text":"Message counts vary with task type, so comparing across roles produces misleading rankings.","correct":false,"tempting":"Notes a comparability problem while accepting that the count would mean something within a single role."},{"text":"Message counts do not distinguish long conversations from short ones, so the unit is inconsistent between users.","correct":false,"tempting":"Refines the unit rather than questioning whether any count of interactions tracks work completed."},{"text":"Some staff use the tool through an API or a front-end that does not report messages, so the count is incomplete.","correct":false,"tempting":"Raises a coverage gap, which would matter only if the metric were sound to begin with."},{"text":"Messages rise when people struggle for a usable answer, so the count grows with difficulty too.","correct":true}]},"prev":{"slug":"introducing-a-tool-to-a-team","title":"Introducing a tool to other people","url":"https://addaly.com/learn/choosing-your-tools/introducing-a-tool-to-a-team"},"next":{"slug":"keeping-up-without-drowning","title":"Keeping up without drowning","url":"https://addaly.com/learn/choosing-your-tools/keeping-up-without-drowning"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Measuring whether it actually helped', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/measuring-whether-it-helped (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Measuring whether it actually helped","courseTitle":"Choosing and Using the Tools","module":{"index":8,"slug":"living-with-it","title":"Living with it"},"answer":null,"keyIdea":"Record minutes saved, minutes spent checking and one thing it could not do, on the day, per task, and never measure messages, words or lines produced, because those rise without anything being completed to a standard.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-the-question-is-hard","runs":[{"text":"Why the question is hard"}]},{"t":"p","runs":[{"text":"Everybody has an impression and the impressions conflict. Enthusiasts report enormous gains; sceptics report none; both are describing their own work honestly. The reason is that the effect differs by task, and an average across a team hides everything interesting."}]},{"t":"p","runs":[{"text":"There is also a specific bias to correct for. These tools are most impressive on the tasks people find least pleasant, which makes relief feel like productivity. Getting through a dreaded email in four minutes instead of twenty is a real gain in wellbeing and may be a smaller gain in time than it feels."}]},{"t":"h2","id":"the-measurement-that-is-worth-doing","runs":[{"text":"The measurement that is worth doing"}]},{"t":"p","runs":[{"text":"Two weeks, one line a day, per person:"}]},{"t":"code","lang":"","label":"Text","text":"minutes saved   |  minutes spent checking  |  one thing it could not do"},{"t":"p","runs":[{"text":"That is it. It takes ninety seconds a day, and after two weeks you have something to look at instead of an argument."}]},{"t":"p","runs":[{"text":"Two refinements that make it much better. Record the "},{"text":"task","i":true},{"text":", not just the time, so you can see which kinds of work produced the gains. And record it on the day, because reconstructing a fortnight from memory produces the impression you already had."}]},{"t":"h2","id":"what-to-measure-and-what-not-to","runs":[{"text":"What to measure, and what not to"}]},{"t":"p","runs":[{"text":"Worth measuring:","b":true},{"text":" time to a usable draft, time spent checking, number of tasks completed that would otherwise have been deferred, errors caught before they went out, and the tasks that got done at all because the barrier dropped."}]},{"t":"p","runs":[{"text":"Not worth measuring:","b":true},{"text":" messages sent to the assistant, lines of code produced, words written, logins. Every one of these can be increased without producing anything of value, and measuring them reliably causes people to increase them. A team measured on lines of code will produce more lines of code."}]},{"t":"p","runs":[{"text":"The distinction is between measuring output and measuring work completed to a standard. Only the second is worth anything, and it is harder to count, which is precisely why the easy metrics get used."}]},{"t":"h2","id":"the-honest-complications","runs":[{"text":"The honest complications"}]},{"t":"p","runs":[{"text":"Quality change is harder to see than time change.","b":true},{"text":" A draft produced in a quarter of the time might be slightly worse, and slightly worse across a hundred documents is a real cost that no time measurement captures. Where quality matters, somebody has to read a sample of before and after."}]},{"t":"p","runs":[{"text":"The learning curve confounds the first month.","b":true},{"text":" People are slower while learning and faster afterwards. A measurement taken in week one understates; one taken in month six on the tasks that survived is closer to the truth."}]},{"t":"p","runs":[{"text":"Selection effects.","b":true},{"text":" People use the tool where it works and stop where it does not, which means measured tasks are the favourable ones. This is fine (it is how tools are supposed to be used) as long as nobody extrapolates from it to tasks nobody chose to try."}]},{"t":"p","runs":[{"text":"Displaced work.","b":true},{"text":" Time saved on drafting sometimes reappears as time spent reviewing somebody else’s AI-assisted draft. At an organisational level that can net to nothing while every individual honestly reports a saving."}]},{"t":"h2","id":"comparing-against-the-right-baseline","runs":[{"text":"Comparing against the right baseline"}]},{"t":"p","runs":[{"text":"One error makes every measurement flattering. The comparison is not against doing the task perfectly by hand with unlimited time. It is against what would actually have happened: the email that would have been sent two days later, the report that would have been shorter, the analysis nobody would have done at all."}]},{"t":"p","runs":[{"text":"Some of the largest genuine gains are in that last category: work that was not worth the effort before and is now. Those do not appear as time saved, because no time was being spent, and a measurement that only counts minutes saved misses them entirely. Note them in the third column and count them separately."}]},{"t":"h2","id":"what-a-good-result-looks-like","runs":[{"text":"What a good result looks like"}]},{"t":"p","runs":[{"text":"The typical honest finding, from teams that have measured properly, is not a uniform improvement. It is a wide distribution: a few tasks transformed, many modestly improved, a few made worse by the checking burden."}]},{"t":"p","runs":[{"text":"That is a useful result, and it points somewhere specific: do more of the first category, stop doing the third, and stop arguing about the average. An organisation that knows which of its tasks are which is in a much better position than one with a single headline number, whichever direction that number points."}]}],"check":{"question":[{"text":"A team measures adoption by counting assistant messages per person, and the number rises steadily. Why is this poor evidence of benefit?"}],"options":[{"text":[{"text":"Message counts vary with task type, so comparing across roles produces misleading rankings."}],"correct":false,"tempting":[{"text":"Notes a comparability problem while accepting that the count would mean something within a single role."}]},{"text":[{"text":"Message counts do not distinguish long conversations from short ones, so the unit is inconsistent between users."}],"correct":false,"tempting":[{"text":"Refines the unit rather than questioning whether any count of interactions tracks work completed."}]},{"text":[{"text":"Some staff use the tool through an API or a front-end that does not report messages, so the count is incomplete."}],"correct":false,"tempting":[{"text":"Raises a coverage gap, which would matter only if the metric were sound to begin with."}]},{"text":[{"text":"Messages rise when people struggle for a usable answer, so the count grows with difficulty too."}],"correct":true,"tempting":null}],"order":[0,2,1,3],"why":null},"prev":{"slug":"introducing-a-tool-to-a-team","title":"Introducing a tool to other people","minutes":8},"next":{"slug":"keeping-up-without-drowning","title":"Keeping up without drowning","minutes":7}}}