What the controlled experiments found

Beyond the surveys

Exposure studies ask what a model could do. A different set of studies asked what happened when real people were given one. Four of them, from 2023, are the basis of nearly every claim about productivity you will read, and they are worth knowing in detail because they are quoted loosely.

Customer support

A software company gave a chatbot-based assistant to some of its customer-support agents (just over five thousand agents in total, mostly in the Philippines) and researchers compared those who got it with those who did not. The assistant suggested responses and pointed to documentation; the agent decided what to send.

Issues resolved per hour rose by about fourteen per cent on average. The average hid the finding that matters: agents with the least experience improved by about a third, and the most experienced improved hardly at all. Customers were happier, agents were less likely to quit, and new agents reached the productivity of veterans in months rather than the better part of a year. The mechanism the researchers proposed is the one this course would predict: the model had learned the patterns of the best agents’ replies and was handing them to the worst.

Professional writing

Several hundred college-educated professionals were given realistic writing tasks (press releases, short reports, delicate emails) and half were allowed a chatbot. Time taken fell by about forty per cent, from around twenty-seven minutes to seventeen. Quality, as rated by blind evaluators, rose by about eighteen per cent. And the spread between the best and worst writers narrowed: the weaker writers gained most, and most of the time saved came from the model producing the first draft and the person editing rather than composing.

Programming

Ninety-five developers were asked to build a small web server. Those given a code-completion tool finished about fifty-six per cent faster. It was a short, well-specified task in a common language, which is worth remembering when the next lesson arrives.

Consulting, and the frontier

The largest was run with seven hundred and fifty-eight consultants at a major firm, on eighteen realistic tasks. On tasks the researchers had chosen to sit inside what the model could do, consultants with the tool completed about twelve per cent more tasks, twenty-five per cent faster, at quality rated forty per cent higher. The below-average consultants gained about forty-three per cent; the above-average, about seventeen.

One task was chosen to sit just outside: it required combining a spreadsheet with interview notes in a way the model’s data did not support. On that task the consultants with the tool were nineteen percentage points less likely to reach the correct recommendation, because the tool was fluent and wrong and they trusted it. This is the jagged frontier from the module on putting it to work, measured.

What the four have in common

Read them together and a pattern is unmistakable, and it is the pattern the course predicts. The gains are large on tasks that are well specified, language-shaped, short and abundantly documented. The gains go disproportionately to the least experienced, because the model supplies the median good practice they lacked. Quality rises on average because the floor rises. And the danger is where the task is outside the model’s support and the person cannot tell.

What they do not show

They measure tasks over hours, not careers over years. Whether the novice who is helped today becomes the expert of tomorrow, or never develops the skill the model supplied, is the subject of a lesson to come.

They measure the tool’s first year. Novelty effects are real, and so is the possibility that the tools of 2025 are better than the ones tested.

They measure consenting, monitored workers on chosen tasks. The support agents knew they were studied; the consultants were volunteers; the tasks were designed to be scorable. A tool’s effect on the unscorable parts of a job (the judgement, the relationships, the responsibility) was not in any of them.

And they measure gains where the model was in-frontier. The next lesson is about a study that was run where it was not, and found the opposite sign.

Quote the numbers when you need them. Fourteen per cent, thirty-four for novices, forty per cent faster writing, nineteen points worse outside the line. They are the best evidence there is, and they are evidence about a narrow thing.