{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"what-the-experiments-found","title":"What the controlled experiments found","url":"https://addaly.com/learn/ai-actually-explained/what-the-experiments-found","position":74,"of":92,"minutes":10,"answer":"When people were randomly given the tools and measured, output rose by a seventh to a half on well-specified tasks, quality rose.","keyIdea":"When people were randomly given the tools and measured, output rose by a seventh to a half on well-specified tasks, quality rose, and the gains went mostly to the least experienced: on tasks inside the frontier."},"body":{"format":"markdown","text":"## Beyond the surveys\n\nExposure studies ask what a model could do. A different set of studies asked what happened when real people were given one. Four of them, from 2023, are the basis of nearly every claim about productivity you will read, and they are worth knowing in detail because they are quoted loosely.\n\n## Customer support\n\nA software company gave a chatbot-based assistant to some of its customer-support agents (just over five thousand agents in total, mostly in the Philippines) and researchers compared those who got it with those who did not. The assistant suggested responses and pointed to documentation; the agent decided what to send.\n\nIssues resolved per hour rose by about fourteen per cent on average. The average hid the finding that matters: agents with the least experience improved by about a third, and the most experienced improved hardly at all. Customers were happier, agents were less likely to quit, and new agents reached the productivity of veterans in months rather than the better part of a year. The mechanism the researchers proposed is the one this course would predict: the model had learned the patterns of the best agents' replies and was handing them to the worst.\n\n## Professional writing\n\nSeveral hundred college-educated professionals were given realistic writing tasks (press releases, short reports, delicate emails) and half were allowed a chatbot. Time taken fell by about forty per cent, from around twenty-seven minutes to seventeen. Quality, as rated by blind evaluators, rose by about eighteen per cent. And the spread between the best and worst writers narrowed: the weaker writers gained most, and most of the time saved came from the model producing the first draft and the person editing rather than composing.\n\n## Programming\n\nNinety-five developers were asked to build a small web server. Those given a code-completion tool finished about fifty-six per cent faster. It was a short, well-specified task in a common language, which is worth remembering when the next lesson arrives.\n\n## Consulting, and the frontier\n\nThe largest was run with seven hundred and fifty-eight consultants at a major firm, on eighteen realistic tasks. On tasks the researchers had chosen to sit inside what the model could do, consultants with the tool completed about twelve per cent more tasks, twenty-five per cent faster, at quality rated forty per cent higher. The below-average consultants gained about forty-three per cent; the above-average, about seventeen.\n\nOne task was chosen to sit just outside: it required combining a spreadsheet with interview notes in a way the model's data did not support. On that task the consultants with the tool were nineteen percentage points less likely to reach the correct recommendation, because the tool was fluent and wrong and they trusted it. This is the jagged frontier from the module on putting it to work, measured.\n\n> Figure: Who gained, in the 2023 controlled experiments. Bars (% more work completed with the tool): Support agents, all 14; Support agents, least experienced 34; Consultants, above average 17; Consultants, below average 43. The averages hide the finding. The model supplies the median of competent practice, so the people furthest below that median gain most and the people above it gain least. All of these were measured on tasks chosen to sit inside what the model could do.\n\n## What the four have in common\n\nRead them together and a pattern is unmistakable, and it is the pattern the course predicts. The gains are large on tasks that are well specified, language-shaped, short and abundantly documented. The gains go disproportionately to the least experienced, because the model supplies the median good practice they lacked. Quality rises on average because the floor rises. And the danger is where the task is outside the model's support and the person cannot tell.\n\n## What they do not show\n\n**They measure tasks over hours, not careers over years.** Whether the novice who is helped today becomes the expert of tomorrow, or never develops the skill the model supplied, is the subject of a lesson to come.\n\n**They measure the tool's first year.** Novelty effects are real, and so is the possibility that the tools of 2025 are better than the ones tested.\n\n**They measure consenting, monitored workers on chosen tasks.** The support agents knew they were studied; the consultants were volunteers; the tasks were designed to be scorable. A tool's effect on the unscorable parts of a job (the judgement, the relationships, the responsibility) was not in any of them.\n\n**And they measure gains where the model was in-frontier.** The next lesson is about a study that was run where it was not, and found the opposite sign.\n\nQuote the numbers when you need them. Fourteen per cent, thirty-four for novices, forty per cent faster writing, nineteen points worse outside the line. They are the best evidence there is, and they are evidence about a narrow thing."},"check":{"question":"Across the 2023 experiments, which group gained most from being given the tool, and why?","options":[{"text":"The most experienced workers, because they knew how to prompt the model well and had the judgement to use its output.","correct":false,"tempting":"Reverses the finding; veterans gained little, because the model was supplying what they already had."},{"text":"The least experienced, because the model handed them the patterns that veterans already had.","correct":true},{"text":"Everyone equally, because the tool speeds up typing regardless of skill.","correct":false,"tempting":"Ignores the spread; the gains narrowed the gap between best and worst rather than shifting everyone alike."},{"text":"The workers on the hardest tasks, because that is where the model had the most to add.","correct":false,"tempting":"Inverts the frontier result; on the task outside the model's support, the tool made people worse."}]},"prev":{"slug":"the-exposure-studies","title":"Measuring exposure, and what the numbers do not say","url":"https://addaly.com/learn/ai-actually-explained/the-exposure-studies"},"next":{"slug":"the-study-that-found-a-slowdown","title":"The study where it made experts slower","url":"https://addaly.com/learn/ai-actually-explained/the-study-that-found-a-slowdown"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What the controlled experiments found', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/what-the-experiments-found (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What the controlled experiments found","courseTitle":"AI, Actually Explained","module":{"index":8,"slug":"ai-and-jobs","title":"AI and work, with the evidence"},"answer":null,"keyIdea":"When people were randomly given the tools and measured, output rose by a seventh to a half on well-specified tasks, quality rose, and the gains went mostly to the least experienced: on tasks inside the frontier.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% more work completed with the tool","title":"Who gained, in the 2023 controlled experiments","series":[{"label":"Support agents, all","value":14},{"label":"Support agents, least experienced","value":34,"highlight":true},{"label":"Consultants, above average","value":17},{"label":"Consultants, below average","value":43}],"caption":"The averages hide the finding. The model supplies the median of competent practice, so the people furthest below that median gain most and the people above it gain least. All of these were measured on tasks chosen to sit inside what the model could do."},"alt":"Figure: Who gained, in the 2023 controlled experiments. Bars (% more work completed with the tool): Support agents, all 14; Support agents, least experienced 34; Consultants, above average 17; Consultants, below average 43. The averages hide the finding. The model supplies the median of competent practice, so the people furthest below that median gain most and the people above it gain least. All of these were measured on tasks chosen to sit inside what the model could do."}},"blocks":[{"t":"h2","id":"beyond-the-surveys","runs":[{"text":"Beyond the surveys"}]},{"t":"p","runs":[{"text":"Exposure studies ask what a model could do. A different set of studies asked what happened when real people were given one. Four of them, from 2023, are the basis of nearly every claim about productivity you will read, and they are worth knowing in detail because they are quoted loosely."}]},{"t":"h2","id":"customer-support","runs":[{"text":"Customer support"}]},{"t":"p","runs":[{"text":"A software company gave a chatbot-based assistant to some of its customer-support agents (just over five thousand agents in total, mostly in the Philippines) and researchers compared those who got it with those who did not. The assistant suggested responses and pointed to documentation; the agent decided what to send."}]},{"t":"p","runs":[{"text":"Issues resolved per hour rose by about fourteen per cent on average. The average hid the finding that matters: agents with the least experience improved by about a third, and the most experienced improved hardly at all. Customers were happier, agents were less likely to quit, and new agents reached the productivity of veterans in months rather than the better part of a year. The mechanism the researchers proposed is the one this course would predict: the model had learned the patterns of the best agents’ replies and was handing them to the worst."}]},{"t":"h2","id":"professional-writing","runs":[{"text":"Professional writing"}]},{"t":"p","runs":[{"text":"Several hundred college-educated professionals were given realistic writing tasks (press releases, short reports, delicate emails) and half were allowed a chatbot. Time taken fell by about forty per cent, from around twenty-seven minutes to seventeen. Quality, as rated by blind evaluators, rose by about eighteen per cent. And the spread between the best and worst writers narrowed: the weaker writers gained most, and most of the time saved came from the model producing the first draft and the person editing rather than composing."}]},{"t":"h2","id":"programming","runs":[{"text":"Programming"}]},{"t":"p","runs":[{"text":"Ninety-five developers were asked to build a small web server. Those given a code-completion tool finished about fifty-six per cent faster. It was a short, well-specified task in a common language, which is worth remembering when the next lesson arrives."}]},{"t":"h2","id":"consulting-and-the-frontier","runs":[{"text":"Consulting, and the frontier"}]},{"t":"p","runs":[{"text":"The largest was run with seven hundred and fifty-eight consultants at a major firm, on eighteen realistic tasks. On tasks the researchers had chosen to sit inside what the model could do, consultants with the tool completed about twelve per cent more tasks, twenty-five per cent faster, at quality rated forty per cent higher. The below-average consultants gained about forty-three per cent; the above-average, about seventeen."}]},{"t":"p","runs":[{"text":"One task was chosen to sit just outside: it required combining a spreadsheet with interview notes in a way the model’s data did not support. On that task the consultants with the tool were nineteen percentage points less likely to reach the correct recommendation, because the tool was fluent and wrong and they trusted it. This is the jagged frontier from the module on putting it to work, measured."}]},{"t":"h2","id":"what-the-four-have-in-common","runs":[{"text":"What the four have in common"}]},{"t":"p","runs":[{"text":"Read them together and a pattern is unmistakable, and it is the pattern the course predicts. The gains are large on tasks that are well specified, language-shaped, short and abundantly documented. The gains go disproportionately to the least experienced, because the model supplies the median good practice they lacked. Quality rises on average because the floor rises. And the danger is where the task is outside the model’s support and the person cannot tell."}]},{"t":"h2","id":"what-they-do-not-show","runs":[{"text":"What they do not show"}]},{"t":"p","runs":[{"text":"They measure tasks over hours, not careers over years.","b":true},{"text":" Whether the novice who is helped today becomes the expert of tomorrow, or never develops the skill the model supplied, is the subject of a lesson to come."}]},{"t":"p","runs":[{"text":"They measure the tool’s first year.","b":true},{"text":" Novelty effects are real, and so is the possibility that the tools of 2025 are better than the ones tested."}]},{"t":"p","runs":[{"text":"They measure consenting, monitored workers on chosen tasks.","b":true},{"text":" The support agents knew they were studied; the consultants were volunteers; the tasks were designed to be scorable. A tool’s effect on the unscorable parts of a job (the judgement, the relationships, the responsibility) was not in any of them."}]},{"t":"p","runs":[{"text":"And they measure gains where the model was in-frontier.","b":true},{"text":" The next lesson is about a study that was run where it was not, and found the opposite sign."}]},{"t":"p","runs":[{"text":"Quote the numbers when you need them. Fourteen per cent, thirty-four for novices, forty per cent faster writing, nineteen points worse outside the line. They are the best evidence there is, and they are evidence about a narrow thing."}]}],"check":{"question":[{"text":"Across the 2023 experiments, which group gained most from being given the tool, and why?"}],"options":[{"text":[{"text":"The most experienced workers, because they knew how to prompt the model well and had the judgement to use its output."}],"correct":false,"tempting":[{"text":"Reverses the finding; veterans gained little, because the model was supplying what they already had."}]},{"text":[{"text":"The least experienced, because the model handed them the patterns that veterans already had."}],"correct":true,"tempting":null},{"text":[{"text":"Everyone equally, because the tool speeds up typing regardless of skill."}],"correct":false,"tempting":[{"text":"Ignores the spread; the gains narrowed the gap between best and worst rather than shifting everyone alike."}]},{"text":[{"text":"The workers on the hardest tasks, because that is where the model had the most to add."}],"correct":false,"tempting":[{"text":"Inverts the frontier result; on the task outside the model’s support, the tool made people worse."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"the-exposure-studies","title":"Measuring exposure, and what the numbers do not say","minutes":9},"next":{"slug":"the-study-that-found-a-slowdown","title":"The study where it made experts slower","minutes":9}}}