{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"reading-a-productivity-study","title":"How to read a claim about developer productivity","url":"https://addaly.com/learn/how-software-changed/reading-a-productivity-study","position":4,"of":87,"minutes":10,"answer":"Match the claim to the study design that produced it: a fixed-task trial can only speak about fixed tasks, self-reported savings measure belief rather than time.","keyIdea":"Match the claim to the study design that produced it: a fixed-task trial can only speak about fixed tasks, self-reported savings measure belief rather than time, and a speed result with no defect measure is half a finding."},"body":{"format":"markdown","text":"## Three study designs, three different claims\n\nAlmost every number you will be shown comes from one of three designs, and each can support a different sentence. Knowing which one you are looking at does most of the work.\n\n**A randomised trial on a fixed task.** Recruit developers, give everyone the same problem, randomly give half the tool, measure completion. The widely quoted example is the 2023 GitHub and Microsoft Research experiment in which participants implemented an HTTP server in JavaScript; the group with the assistant completed it around 55% faster.\n\nWhat that establishes: on a self-contained, well-specified, greenfield task in a common language, with that tool and that population, completion time falls a lot. That is a real finding and it should not be dismissed.\n\nWhat it cannot establish: anything about a four-year-old codebase, anything about the defect rate of what was submitted, and anything about whether the finished work was correct beyond whatever check the study applied.\n\n**A randomised trial on real work.** Harder to run and more informative. The 2025 METR study is the well-known one: sixteen experienced open-source developers, working on 246 real issues in repositories they themselves maintained, with tool access randomised per issue. Measured result: they were about 19% *slower* with the tools. Before starting they had forecast being 24% faster, and after finishing they estimated they had been about 20% faster.\n\nWhat that establishes: the sign of the effect is not universal, and practitioners' self-assessment can be wrong by roughly forty percentage points on their own work.\n\nWhat it cannot establish: that the tools slow people down generally. Sixteen unusually expert developers, in codebases they know intimately (the exact population and setting where the theoretical benefit is smallest) using a specific set of tools during a specific few months.\n\n**Observational telemetry at scale.** Company dashboards, industry surveys, the DORA reports. Millions of data points, and confounded from the start, because the teams that adopt tools early differ from the ones that do not in every way that also affects delivery. Correlations here are worth reading and worth never quoting as causes.\n\n## The checklist\n\nWhen a figure arrives, ask these in order. It takes about two minutes and it eliminates most of what gets circulated.\n\n1. **Who chose the task?** A vendor-selected task is a demonstration, not a measurement. The most common trick is not fabrication, it is choosing a benchmark that matches the strength of the product.\n2. **Measured or reported?** Self-reported time savings run consistently higher than clock time. If the study asked people how much time they saved, it measured belief.\n3. **Was quality measured at all?** Speed without a defect rate is half a result. A change that arrives twice as fast and comes back once in five is not an improvement, and a study that stops at the merge button cannot tell you which you have.\n4. **Compared with what?** No tool at all, an older tool, or a differently configured version of the same tool? A great deal of published improvement is against a baseline nobody actually uses.\n5. **What is n, and what is the interval?** A headline of 55% with n around a hundred carries a wide interval. The point estimate is what gets quoted; the interval is what tells you whether to act.\n6. **Who funded it, and was the analysis fixed in advance?** Preregistration is rare in this field and its presence is a strong positive signal.\n\n## The two metrics to distrust on sight\n\n**\"X% of our code is now written by AI.\"** This counts accepted characters. It says nothing about whether those characters were necessary, correct, or later deleted. A team that accepts more verbose completions scores higher. It is a usage statistic wearing the clothes of an outcome statistic.\n\n**Acceptance rate.** The proportion of suggestions a developer takes. It measures how often the tool guesses your next line, which correlates with how conventional your code is. Optimising it is optimising for writing predictable code.\n\nNeither belongs in a decision about whether the tools are working. The measures that do are further along the pipeline: lead time from commit to production, change failure rate, and time to restore service.\n\n## The sentence you can actually defend\n\nPut together, the current evidence supports something like this, and you should be suspicious of anyone who states it with less hedging:\n\n> On greenfield, well-specified tasks in widely used languages, measured completion time falls substantially: the effect is large and repeatedly observed. On maintenance work inside mature systems the effect is not established, varies by person and codebase, and at least one careful randomised trial found it negative. Effects on defect rates and on the cost of maintaining the result are, as of now, largely unmeasured.\n\nThat is a less exciting sentence than the ones in the press releases. It is also the one that survives contact with your own team's numbers, which is the only study whose population is definitely yours."},"check":{"question":"A vendor cites a randomised trial showing 55% faster completion, and a sceptic cites a randomised trial showing 19% slower completion. Both are real. What is the most defensible way to hold both?","options":[{"text":"One of the two must have a methodological flaw, and the larger study should be preferred until the flaw is identified.","correct":false,"tempting":"Assumes contradictory results require an error, when the two studies measured different tasks in different settings and can both be sound."},{"text":"They measured different work: a greenfield exercise versus real issues in the participants' own codebases.","correct":true},{"text":"Randomised trials are unsuitable for developer productivity, so both should be set aside in favour of large-scale telemetry.","correct":false,"tempting":"Discards the only designs that support causal claims in favour of the one that is confounded by who adopts the tools."},{"text":"The difference is explained by tool improvement between the two dates, so the more recent number is the current truth.","correct":false,"tempting":"Reaches for a timeline explanation when the studies differ on task type and participant expertise, which are the variables the designs actually manipulated."}]},"prev":{"slug":"induced-demand-for-software","title":"Cheaper software means more software","url":"https://addaly.com/learn/how-software-changed/induced-demand-for-software"},"next":{"slug":"the-perception-gap","title":"Why it feels faster than it measures","url":"https://addaly.com/learn/how-software-changed/the-perception-gap"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'How to read a claim about developer productivity', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/reading-a-productivity-study (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"How to read a claim about developer productivity","courseTitle":"How Software Development Changed","module":{"index":1,"slug":"where-the-cost-went","title":"Where the cost actually went"},"answer":null,"keyIdea":"Match the claim to the study design that produced it: a fixed-task trial can only speak about fixed tasks, self-reported savings measure belief rather than time, and a speed result with no defect measure is half a finding.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-study-designs-three-different-claims","runs":[{"text":"Three study designs, three different claims"}]},{"t":"p","runs":[{"text":"Almost every number you will be shown comes from one of three designs, and each can support a different sentence. Knowing which one you are looking at does most of the work."}]},{"t":"p","runs":[{"text":"A randomised trial on a fixed task.","b":true},{"text":" Recruit developers, give everyone the same problem, randomly give half the tool, measure completion. The widely quoted example is the 2023 GitHub and Microsoft Research experiment in which participants implemented an HTTP server in JavaScript; the group with the assistant completed it around 55% faster."}]},{"t":"p","runs":[{"text":"What that establishes: on a self-contained, well-specified, greenfield task in a common language, with that tool and that population, completion time falls a lot. That is a real finding and it should not be dismissed."}]},{"t":"p","runs":[{"text":"What it cannot establish: anything about a four-year-old codebase, anything about the defect rate of what was submitted, and anything about whether the finished work was correct beyond whatever check the study applied."}]},{"t":"p","runs":[{"text":"A randomised trial on real work.","b":true},{"text":" Harder to run and more informative. The 2025 METR study is the well-known one: sixteen experienced open-source developers, working on 246 real issues in repositories they themselves maintained, with tool access randomised per issue. Measured result: they were about 19% "},{"text":"slower","i":true},{"text":" with the tools. Before starting they had forecast being 24% faster, and after finishing they estimated they had been about 20% faster."}]},{"t":"p","runs":[{"text":"What that establishes: the sign of the effect is not universal, and practitioners’ self-assessment can be wrong by roughly forty percentage points on their own work."}]},{"t":"p","runs":[{"text":"What it cannot establish: that the tools slow people down generally. Sixteen unusually expert developers, in codebases they know intimately (the exact population and setting where the theoretical benefit is smallest) using a specific set of tools during a specific few months."}]},{"t":"p","runs":[{"text":"Observational telemetry at scale.","b":true},{"text":" Company dashboards, industry surveys, the DORA reports. Millions of data points, and confounded from the start, because the teams that adopt tools early differ from the ones that do not in every way that also affects delivery. Correlations here are worth reading and worth never quoting as causes."}]},{"t":"h2","id":"the-checklist","runs":[{"text":"The checklist"}]},{"t":"p","runs":[{"text":"When a figure arrives, ask these in order. It takes about two minutes and it eliminates most of what gets circulated."}]},{"t":"ol","start":1,"items":[[{"text":"Who chose the task?","b":true},{"text":" A vendor-selected task is a demonstration, not a measurement. The most common trick is not fabrication, it is choosing a benchmark that matches the strength of the product."}],[{"text":"Measured or reported?","b":true},{"text":" Self-reported time savings run consistently higher than clock time. If the study asked people how much time they saved, it measured belief."}],[{"text":"Was quality measured at all?","b":true},{"text":" Speed without a defect rate is half a result. A change that arrives twice as fast and comes back once in five is not an improvement, and a study that stops at the merge button cannot tell you which you have."}],[{"text":"Compared with what?","b":true},{"text":" No tool at all, an older tool, or a differently configured version of the same tool? A great deal of published improvement is against a baseline nobody actually uses."}],[{"text":"What is n, and what is the interval?","b":true},{"text":" A headline of 55% with n around a hundred carries a wide interval. The point estimate is what gets quoted; the interval is what tells you whether to act."}],[{"text":"Who funded it, and was the analysis fixed in advance?","b":true},{"text":" Preregistration is rare in this field and its presence is a strong positive signal."}]]},{"t":"h2","id":"the-two-metrics-to-distrust-on-sight","runs":[{"text":"The two metrics to distrust on sight"}]},{"t":"p","runs":[{"text":"”X% of our code is now written by AI.”","b":true},{"text":" This counts accepted characters. It says nothing about whether those characters were necessary, correct, or later deleted. A team that accepts more verbose completions scores higher. It is a usage statistic wearing the clothes of an outcome statistic."}]},{"t":"p","runs":[{"text":"Acceptance rate.","b":true},{"text":" The proportion of suggestions a developer takes. It measures how often the tool guesses your next line, which correlates with how conventional your code is. Optimising it is optimising for writing predictable code."}]},{"t":"p","runs":[{"text":"Neither belongs in a decision about whether the tools are working. The measures that do are further along the pipeline: lead time from commit to production, change failure rate, and time to restore service."}]},{"t":"h2","id":"the-sentence-you-can-actually-defend","runs":[{"text":"The sentence you can actually defend"}]},{"t":"p","runs":[{"text":"Put together, the current evidence supports something like this, and you should be suspicious of anyone who states it with less hedging:"}]},{"t":"quote","runs":[{"text":"On greenfield, well-specified tasks in widely used languages, measured completion time falls substantially: the effect is large and repeatedly observed. On maintenance work inside mature systems the effect is not established, varies by person and codebase, and at least one careful randomised trial found it negative. Effects on defect rates and on the cost of maintaining the result are, as of now, largely unmeasured."}]},{"t":"p","runs":[{"text":"That is a less exciting sentence than the ones in the press releases. It is also the one that survives contact with your own team’s numbers, which is the only study whose population is definitely yours."}]}],"check":{"question":[{"text":"A vendor cites a randomised trial showing 55% faster completion, and a sceptic cites a randomised trial showing 19% slower completion. Both are real. What is the most defensible way to hold both?"}],"options":[{"text":[{"text":"One of the two must have a methodological flaw, and the larger study should be preferred until the flaw is identified."}],"correct":false,"tempting":[{"text":"Assumes contradictory results require an error, when the two studies measured different tasks in different settings and can both be sound."}]},{"text":[{"text":"They measured different work: a greenfield exercise versus real issues in the participants’ own codebases."}],"correct":true,"tempting":null},{"text":[{"text":"Randomised trials are unsuitable for developer productivity, so both should be set aside in favour of large-scale telemetry."}],"correct":false,"tempting":[{"text":"Discards the only designs that support causal claims in favour of the one that is confounded by who adopts the tools."}]},{"text":[{"text":"The difference is explained by tool improvement between the two dates, so the more recent number is the current truth."}],"correct":false,"tempting":[{"text":"Reaches for a timeline explanation when the studies differ on task type and participant expertise, which are the variables the designs actually manipulated."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"induced-demand-for-software","title":"Cheaper software means more software","minutes":9},"next":{"slug":"the-perception-gap","title":"Why it feels faster than it measures","minutes":8}}}