{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"the-study-that-found-a-slowdown","title":"The study where it made experts slower","url":"https://addaly.com/learn/ai-actually-explained/the-study-that-found-a-slowdown","position":75,"of":92,"minutes":9,"answer":"Experienced developers on their own large codebases were nineteen per cent slower with AI tools while believing they were twenty per cent faster.","keyIdea":"Experienced developers on their own large codebases were nineteen per cent slower with AI tools while believing they were twenty per cent faster, because the tasks sat outside the tool's support and the feeling of help tracked the fluency of the suggestions, not their value."},"body":{"format":"markdown","text":"## A different kind of task\n\nThe previous lesson's experiments were run on tasks chosen to be scorable, short and inside the model's support, with participants who were often new to the work. In 2025 a research group ran the experiment the other way round.\n\nThey recruited sixteen experienced open-source developers, each working on a large project they had contributed to for years: codebases of a million lines and more, with tens of thousands of users. The developers supplied their own real tasks, two hundred and forty-six in all: bug fixes, features, refactors, the ordinary work of maintaining serious software. Each task was randomly assigned to be done with or without AI tools, which meant the leading code assistant of the day with a frontier model behind it. Time was recorded from screen capture.\n\n## The result\n\nWith the tools, the developers took about nineteen per cent longer.\n\nBefore the study, the developers had predicted the tools would make them about twenty-four per cent faster. After it, having just done the tasks, with the times available to them, they believed the tools had made them about twenty per cent faster. The measured effect was a slowdown of about the same size as the speed-up they were sure they had felt.\n\n> Figure: Sixteen expert developers, 246 real tasks, two answers. What they felt: Predicted beforehand: 24% faster; Believed afterwards: 20% faster; The suggestions arrived quickly and read well; Self-report tracked the fluency of the output. What the screen recording showed: Measured: 19% slower; Time went on reading and rejecting plausible code; The tasks turned on history not in the repository; A million-line project does not fit on the desk. The nineteen per cent is interesting. The forty-point gap between the felt gain and the measured loss is the finding: a person's sense of whether a tool is helping is not a measurement of whether it is. Every survey in which workers report being more productive is measuring the left-hand column.\n\n## Why\n\nThe researchers' analysis pointed at several mechanisms, every one of which this course has already given you.\n\n**The tasks were outside the frontier.** Not because they were hard in the abstract but because they depended on knowledge that was not in the code: why a function was written that way in 2019, what the maintainers would accept, which module was about to be replaced. The developers held that context in their heads. The model had the repository and none of the history.\n\n**Fluent suggestions cost time to reject.** The tool produced plausible code quickly. The developers then had to read it, decide whether it was right, and usually fix or discard it. Reviewing a fluent wrong answer is slower than writing a right one when you already know the answer, which is exactly the situation an expert on their own codebase is in.\n\n**Experts have little median to gain.** The previous lesson's gains went to novices because the model supplied the median good practice they lacked. These developers were far above the median on their own projects. There was nothing for the model to raise.\n\n**The context was too large for the desk.** A million-line project does not fit in a context window, and the tool's retrieval fetched the wrong pieces often enough to mislead.\n\n## The result that matters most is the belief\n\nThe nineteen per cent is interesting. The gap between the nineteen per cent and the felt twenty per cent gain is the finding. It means that a person's sense of whether a tool is helping is not a measurement of whether it is. The feeling tracked how fluent and fast the suggestions were; the stopwatch tracked how long the task took; and they came apart by forty percentage points.\n\nThat is the module on keeping your own judgement, measured on experts. It also means that every survey in which workers report that AI has made them more productive (and there are many, with large numbers) is measuring the feeling. The controlled studies with stopwatches are fewer, and they disagree with each other by task.\n\n## What it does not show\n\nThe authors were careful and it is worth being as careful. Sixteen developers is a small group. They were unusually expert on unusually large projects. Several had little prior experience with the tools, and the tools of the following year were better. The study says nothing about junior developers, small projects, new codebases, or anything outside software, and the previous lesson's programming study, on a small task in a common language, found a large speed-up. Both are true. They are true about different places on the frontier.\n\nA large industry survey the previous year had found a related pattern at the scale of whole teams: a rise in AI adoption was associated with a small fall in how much software teams delivered and a larger fall in how stable it was. A survey is not an experiment, but the direction agreed.\n\n## How to hold it\n\nPut the four 2023 experiments and this one on the same map. Inside the frontier, on specified tasks, novices gain a lot and experts a little. Outside it, on tasks whose difficulty lives in context the model lacks, experts lose time and do not notice. The tool has not changed between the two. The task has, and so has who is holding it. Whatever you read about productivity next, ask which of those two places it was measured in, and whether it was measured at all."},"check":{"question":"The developers in the 2025 study believed the tools had sped them up by about 20 per cent when they had in fact been slowed by about 19 per cent. What does that gap show?","options":[{"text":"That the developers were careless in reporting their times.","correct":false,"tempting":"Blames the participants; the times were recorded from screen capture, and the belief was measured separately."},{"text":"That the tools were speeding up the parts of the work the developers noticed and slowing the parts they did not.","correct":false,"tempting":"Reaches for a partial truth; the study's point is that the belief tracked fluency, not any measured component."},{"text":"That the sense of being helped tracks the fluency of the suggestions, not the measured effect on the task.","correct":true},{"text":"That experienced developers are biased against AI and under-reported its benefits.","correct":false,"tempting":"Gets the direction backwards; the developers over-reported the benefit, not under."}]},"prev":{"slug":"what-the-experiments-found","title":"What the controlled experiments found","url":"https://addaly.com/learn/ai-actually-explained/what-the-experiments-found"},"next":{"slug":"who-gains-first","title":"Novices gain first, and what that means for everyone else","url":"https://addaly.com/learn/ai-actually-explained/who-gains-first"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The study where it made experts slower', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/the-study-that-found-a-slowdown (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The study where it made experts slower","courseTitle":"AI, Actually Explained","module":{"index":8,"slug":"ai-and-jobs","title":"AI and work, with the evidence"},"answer":null,"keyIdea":"Experienced developers on their own large codebases were nineteen per cent slower with AI tools while believing they were twenty per cent faster, because the tasks sat outside the tool’s support and the feeling of help tracked the fluency of the suggestions, not their value.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"What they felt","items":["Predicted beforehand: 24% faster","Believed afterwards: 20% faster","The suggestions arrived quickly and read well","Self-report tracked the fluency of the output"]},"type":"compare","right":{"title":"What the screen recording showed","items":["Measured: 19% slower","Time went on reading and rejecting plausible code","The tasks turned on history not in the repository","A million-line project does not fit on the desk"]},"title":"Sixteen expert developers, 246 real tasks, two answers","caption":"The nineteen per cent is interesting. The forty-point gap between the felt gain and the measured loss is the finding: a person’s sense of whether a tool is helping is not a measurement of whether it is. Every survey in which workers report being more productive is measuring the left-hand column."},"alt":"Figure: Sixteen expert developers, 246 real tasks, two answers. What they felt: Predicted beforehand: 24% faster; Believed afterwards: 20% faster; The suggestions arrived quickly and read well; Self-report tracked the fluency of the output. What the screen recording showed: Measured: 19% slower; Time went on reading and rejecting plausible code; The tasks turned on history not in the repository; A million-line project does not fit on the desk. The nineteen per cent is interesting. The forty-point gap between the felt gain and the measured loss is the finding: a person’s sense of whether a tool is helping is not a measurement of whether it is. Every survey in which workers report being more productive is measuring the left-hand column."}},"blocks":[{"t":"h2","id":"a-different-kind-of-task","runs":[{"text":"A different kind of task"}]},{"t":"p","runs":[{"text":"The previous lesson’s experiments were run on tasks chosen to be scorable, short and inside the model’s support, with participants who were often new to the work. In 2025 a research group ran the experiment the other way round."}]},{"t":"p","runs":[{"text":"They recruited sixteen experienced open-source developers, each working on a large project they had contributed to for years: codebases of a million lines and more, with tens of thousands of users. The developers supplied their own real tasks, two hundred and forty-six in all: bug fixes, features, refactors, the ordinary work of maintaining serious software. Each task was randomly assigned to be done with or without AI tools, which meant the leading code assistant of the day with a frontier model behind it. Time was recorded from screen capture."}]},{"t":"h2","id":"the-result","runs":[{"text":"The result"}]},{"t":"p","runs":[{"text":"With the tools, the developers took about nineteen per cent longer."}]},{"t":"p","runs":[{"text":"Before the study, the developers had predicted the tools would make them about twenty-four per cent faster. After it, having just done the tasks, with the times available to them, they believed the tools had made them about twenty per cent faster. The measured effect was a slowdown of about the same size as the speed-up they were sure they had felt."}]},{"t":"h2","id":"why","runs":[{"text":"Why"}]},{"t":"p","runs":[{"text":"The researchers’ analysis pointed at several mechanisms, every one of which this course has already given you."}]},{"t":"p","runs":[{"text":"The tasks were outside the frontier.","b":true},{"text":" Not because they were hard in the abstract but because they depended on knowledge that was not in the code: why a function was written that way in 2019, what the maintainers would accept, which module was about to be replaced. The developers held that context in their heads. The model had the repository and none of the history."}]},{"t":"p","runs":[{"text":"Fluent suggestions cost time to reject.","b":true},{"text":" The tool produced plausible code quickly. The developers then had to read it, decide whether it was right, and usually fix or discard it. Reviewing a fluent wrong answer is slower than writing a right one when you already know the answer, which is exactly the situation an expert on their own codebase is in."}]},{"t":"p","runs":[{"text":"Experts have little median to gain.","b":true},{"text":" The previous lesson’s gains went to novices because the model supplied the median good practice they lacked. These developers were far above the median on their own projects. There was nothing for the model to raise."}]},{"t":"p","runs":[{"text":"The context was too large for the desk.","b":true},{"text":" A million-line project does not fit in a context window, and the tool’s retrieval fetched the wrong pieces often enough to mislead."}]},{"t":"h2","id":"the-result-that-matters-most-is-the-belief","runs":[{"text":"The result that matters most is the belief"}]},{"t":"p","runs":[{"text":"The nineteen per cent is interesting. The gap between the nineteen per cent and the felt twenty per cent gain is the finding. It means that a person’s sense of whether a tool is helping is not a measurement of whether it is. The feeling tracked how fluent and fast the suggestions were; the stopwatch tracked how long the task took; and they came apart by forty percentage points."}]},{"t":"p","runs":[{"text":"That is the module on keeping your own judgement, measured on experts. It also means that every survey in which workers report that AI has made them more productive (and there are many, with large numbers) is measuring the feeling. The controlled studies with stopwatches are fewer, and they disagree with each other by task."}]},{"t":"h2","id":"what-it-does-not-show","runs":[{"text":"What it does not show"}]},{"t":"p","runs":[{"text":"The authors were careful and it is worth being as careful. Sixteen developers is a small group. They were unusually expert on unusually large projects. Several had little prior experience with the tools, and the tools of the following year were better. The study says nothing about junior developers, small projects, new codebases, or anything outside software, and the previous lesson’s programming study, on a small task in a common language, found a large speed-up. Both are true. They are true about different places on the frontier."}]},{"t":"p","runs":[{"text":"A large industry survey the previous year had found a related pattern at the scale of whole teams: a rise in AI adoption was associated with a small fall in how much software teams delivered and a larger fall in how stable it was. A survey is not an experiment, but the direction agreed."}]},{"t":"h2","id":"how-to-hold-it","runs":[{"text":"How to hold it"}]},{"t":"p","runs":[{"text":"Put the four 2023 experiments and this one on the same map. Inside the frontier, on specified tasks, novices gain a lot and experts a little. Outside it, on tasks whose difficulty lives in context the model lacks, experts lose time and do not notice. The tool has not changed between the two. The task has, and so has who is holding it. Whatever you read about productivity next, ask which of those two places it was measured in, and whether it was measured at all."}]}],"check":{"question":[{"text":"The developers in the 2025 study believed the tools had sped them up by about 20 per cent when they had in fact been slowed by about 19 per cent. What does that gap show?"}],"options":[{"text":[{"text":"That the developers were careless in reporting their times."}],"correct":false,"tempting":[{"text":"Blames the participants; the times were recorded from screen capture, and the belief was measured separately."}]},{"text":[{"text":"That the tools were speeding up the parts of the work the developers noticed and slowing the parts they did not."}],"correct":false,"tempting":[{"text":"Reaches for a partial truth; the study’s point is that the belief tracked fluency, not any measured component."}]},{"text":[{"text":"That the sense of being helped tracks the fluency of the suggestions, not the measured effect on the task."}],"correct":true,"tempting":null},{"text":[{"text":"That experienced developers are biased against AI and under-reported its benefits."}],"correct":false,"tempting":[{"text":"Gets the direction backwards; the developers over-reported the benefit, not under."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"what-the-experiments-found","title":"What the controlled experiments found","minutes":10},"next":{"slug":"who-gains-first","title":"Novices gain first, and what that means for everyone else","minutes":9}}}