The study where it made experts slower

A different kind of task

The previous lesson’s experiments were run on tasks chosen to be scorable, short and inside the model’s support, with participants who were often new to the work. In 2025 a research group ran the experiment the other way round.

They recruited sixteen experienced open-source developers, each working on a large project they had contributed to for years: codebases of a million lines and more, with tens of thousands of users. The developers supplied their own real tasks, two hundred and forty-six in all: bug fixes, features, refactors, the ordinary work of maintaining serious software. Each task was randomly assigned to be done with or without AI tools, which meant the leading code assistant of the day with a frontier model behind it. Time was recorded from screen capture.

The result

With the tools, the developers took about nineteen per cent longer.

Before the study, the developers had predicted the tools would make them about twenty-four per cent faster. After it, having just done the tasks, with the times available to them, they believed the tools had made them about twenty per cent faster. The measured effect was a slowdown of about the same size as the speed-up they were sure they had felt.

Why

The researchers’ analysis pointed at several mechanisms, every one of which this course has already given you.

The tasks were outside the frontier. Not because they were hard in the abstract but because they depended on knowledge that was not in the code: why a function was written that way in 2019, what the maintainers would accept, which module was about to be replaced. The developers held that context in their heads. The model had the repository and none of the history.

Fluent suggestions cost time to reject. The tool produced plausible code quickly. The developers then had to read it, decide whether it was right, and usually fix or discard it. Reviewing a fluent wrong answer is slower than writing a right one when you already know the answer, which is exactly the situation an expert on their own codebase is in.

Experts have little median to gain. The previous lesson’s gains went to novices because the model supplied the median good practice they lacked. These developers were far above the median on their own projects. There was nothing for the model to raise.

The context was too large for the desk. A million-line project does not fit in a context window, and the tool’s retrieval fetched the wrong pieces often enough to mislead.

The result that matters most is the belief

The nineteen per cent is interesting. The gap between the nineteen per cent and the felt twenty per cent gain is the finding. It means that a person’s sense of whether a tool is helping is not a measurement of whether it is. The feeling tracked how fluent and fast the suggestions were; the stopwatch tracked how long the task took; and they came apart by forty percentage points.

That is the module on keeping your own judgement, measured on experts. It also means that every survey in which workers report that AI has made them more productive (and there are many, with large numbers) is measuring the feeling. The controlled studies with stopwatches are fewer, and they disagree with each other by task.

What it does not show

The authors were careful and it is worth being as careful. Sixteen developers is a small group. They were unusually expert on unusually large projects. Several had little prior experience with the tools, and the tools of the following year were better. The study says nothing about junior developers, small projects, new codebases, or anything outside software, and the previous lesson’s programming study, on a small task in a common language, found a large speed-up. Both are true. They are true about different places on the frontier.

A large industry survey the previous year had found a related pattern at the scale of whole teams: a rise in AI adoption was associated with a small fall in how much software teams delivered and a larger fall in how stable it was. A survey is not an experiment, but the direction agreed.

How to hold it

Put the four 2023 experiments and this one on the same map. Inside the frontier, on specified tasks, novices gain a lot and experts a little. Outside it, on tasks whose difficulty lives in context the model lacks, experts lose time and do not notice. The tool has not changed between the two. The task has, and so has who is holding it. Whatever you read about productivity next, ask which of those two places it was measured in, and whether it was measured at all.