How to read a claim about developer productivity

Three study designs, three different claims

Almost every number you will be shown comes from one of three designs, and each can support a different sentence. Knowing which one you are looking at does most of the work.

A randomised trial on a fixed task. Recruit developers, give everyone the same problem, randomly give half the tool, measure completion. The widely quoted example is the 2023 GitHub and Microsoft Research experiment in which participants implemented an HTTP server in JavaScript; the group with the assistant completed it around 55% faster.

What that establishes: on a self-contained, well-specified, greenfield task in a common language, with that tool and that population, completion time falls a lot. That is a real finding and it should not be dismissed.

What it cannot establish: anything about a four-year-old codebase, anything about the defect rate of what was submitted, and anything about whether the finished work was correct beyond whatever check the study applied.

A randomised trial on real work. Harder to run and more informative. The 2025 METR study is the well-known one: sixteen experienced open-source developers, working on 246 real issues in repositories they themselves maintained, with tool access randomised per issue. Measured result: they were about 19% slower with the tools. Before starting they had forecast being 24% faster, and after finishing they estimated they had been about 20% faster.

What that establishes: the sign of the effect is not universal, and practitioners’ self-assessment can be wrong by roughly forty percentage points on their own work.

What it cannot establish: that the tools slow people down generally. Sixteen unusually expert developers, in codebases they know intimately (the exact population and setting where the theoretical benefit is smallest) using a specific set of tools during a specific few months.

Observational telemetry at scale. Company dashboards, industry surveys, the DORA reports. Millions of data points, and confounded from the start, because the teams that adopt tools early differ from the ones that do not in every way that also affects delivery. Correlations here are worth reading and worth never quoting as causes.

The checklist

When a figure arrives, ask these in order. It takes about two minutes and it eliminates most of what gets circulated.

1. Who chose the task? A vendor-selected task is a demonstration, not a measurement. The most common trick is not fabrication, it is choosing a benchmark that matches the strength of the product.
2. Measured or reported? Self-reported time savings run consistently higher than clock time. If the study asked people how much time they saved, it measured belief.
3. Was quality measured at all? Speed without a defect rate is half a result. A change that arrives twice as fast and comes back once in five is not an improvement, and a study that stops at the merge button cannot tell you which you have.
4. Compared with what? No tool at all, an older tool, or a differently configured version of the same tool? A great deal of published improvement is against a baseline nobody actually uses.
5. What is n, and what is the interval? A headline of 55% with n around a hundred carries a wide interval. The point estimate is what gets quoted; the interval is what tells you whether to act.
6. Who funded it, and was the analysis fixed in advance? Preregistration is rare in this field and its presence is a strong positive signal.

The two metrics to distrust on sight

”X% of our code is now written by AI.” This counts accepted characters. It says nothing about whether those characters were necessary, correct, or later deleted. A team that accepts more verbose completions scores higher. It is a usage statistic wearing the clothes of an outcome statistic.

Acceptance rate. The proportion of suggestions a developer takes. It measures how often the tool guesses your next line, which correlates with how conventional your code is. Optimising it is optimising for writing predictable code.

Neither belongs in a decision about whether the tools are working. The measures that do are further along the pipeline: lead time from commit to production, change failure rate, and time to restore service.

The sentence you can actually defend

Put together, the current evidence supports something like this, and you should be suspicious of anyone who states it with less hedging:

On greenfield, well-specified tasks in widely used languages, measured completion time falls substantially: the effect is large and repeatedly observed. On maintenance work inside mature systems the effect is not established, varies by person and codebase, and at least one careful randomised trial found it negative. Effects on defect rates and on the cost of maintaining the result are, as of now, largely unmeasured.

That is a less exciting sentence than the ones in the press releases. It is also the one that survives contact with your own team’s numbers, which is the only study whose population is definitely yours.