{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"measuring-delivery-not-activity","title":"Four numbers that resist gaming","url":"https://addaly.com/learn/how-software-changed/measuring-delivery-not-activity","position":51,"of":87,"minutes":10,"answer":"Measure at the end of the pipeline with deployment frequency, lead time, change failure rate and recovery time, because those four interlock so that cheating on one degrades another.","keyIdea":"Measure at the end of the pipeline with deployment frequency, lead time, change failure rate and recovery time, because those four interlock so that cheating on one degrades another, while every activity count rises with tool use regardless of what is delivered."},"body":{"format":"markdown","text":"## Why activity metrics got worse, not just useless\n\nCommits, pull requests, lines of code, story points, suggestion acceptance rate. Every one of these rises when a team uses these tools more, whatever happens to delivery. That was always true and it did not matter much, because the correlation between typing volume and delivered value was at least weakly positive. It is now close to zero, and in a review-constrained team it can be negative.\n\nThe four measures popularised by the DORA research programme have the opposite property: they sit at the end of the pipeline, so they only improve if the whole system improves. Two are about speed and two about stability, which is what stops either being optimised alone.\n\n> Figure: Two sets of numbers. Activity counts, which all rise with tool use: Commits and pull requests opened; Lines of code; Suggestion acceptance rate; Percentage of code generated; Story points completed. Delivery measures, which move only if the system does: Deployment frequency; Lead time from commit to production; Change failure rate; Failed deployment recovery time; Reliability against whatever you promised. Each of the four on the right can be improved on its own by cheating, and every cheat degrades another one. That interlock is the design, and it is why the set is worth more than any member of it.\n\n## The four, with how to compute them from what you already have\n\n**Deployment frequency.** How often code reaches production. Count deployments per week from your CI logs. Not releases planned, deployments that happened.\n\n**Lead time for changes.** Time from a commit being made to that commit running in production. Every git host has the commit timestamp; your deployment records have the other end.\n\n```bash\n# crude but honest: median hours from commit to the deploy that carried it\ngit log --since=90.days --format='%H %ct' > commits.txt\n# join against your deploy log by commit sha, take the difference, take the median\n```\n\n**Change failure rate.** The proportion of deployments that caused a degraded service: a rollback, a hotfix, an incident. Count them. If you have no record of rollbacks, that absence is the first finding.\n\n**Failed deployment recovery time.** Median time from a bad deployment being noticed to service being restored. Your incident records have this if you keep any.\n\nA fifth, added in later research and worth tracking: **reliability**, meaning whether the service meets whatever availability or latency promise you have made.\n\n## Why these resist gaming\n\nEach one can be improved individually by cheating, and the cheat degrades another.\n\nDeploy more often by shipping without review: change failure rate rises. Reduce change failure rate by deploying rarely and carefully: deployment frequency falls and lead time rises. Reduce lead time by skipping verification: both stability measures degrade.\n\nThe only way to move all four together is to make changes smaller, verification faster and recovery easier, which is the actual goal. That interlock is the design, and it is why the set is worth more than any of its members.\n\n## What they show about these tools specifically\n\nThe expected signature of adoption without process change is precise, and it is worth watching for:\n\n- Commits and pull requests up, sometimes sharply\n- Deployment frequency flat\n- Lead time up, because the review queue lengthened\n- Change failure rate up, because review got shallower\n\nIf that is your pattern, the constraint is downstream of writing and adding more writing capacity makes it worse. Industry reports have described versions of this pattern, and you should not need a report: the four numbers from your own systems are more convincing to your own management than any survey.\n\n## What they do not measure\n\nBe honest about the limits, because a metric oversold gets discarded.\n\n**They say nothing about whether you built the right thing.** A team can deploy the wrong feature forty times a week with an excellent change failure rate. Value is a separate question and these numbers do not touch it.\n\n**They can be gamed by redefinition.** \"Incident\" is whatever your team decides it is, and a quiet narrowing of that definition improves the numbers without improving anything. Fix the definitions once, write them down, and do not change them to make a quarter look better.\n\n**They are team measures, never individual ones.** Lead time is a property of a pipeline that many people share. Attributing it to a person measures who happened to be assigned the changes that queued.\n\n**They lag.** Change failure rate over a quarter will not tell you that this week's practice has decayed. Pair them with the leading indicators from earlier in this course: median diff size, review turnaround, and how many people can explain a given service.\n\n## Start smaller than you think\n\nA team that has never measured any of this should not build a dashboard. Take a spreadsheet, take the last thirty deployments, fill in four columns by hand, and compute four medians. It takes an afternoon and it is usually the most surprising afternoon of the quarter: most teams discover their lead time is several times what they believed, and almost all of it is waiting.\n\nDo it again in three months. The comparison is the point; the absolute numbers mean less than the direction, because every team's definitions differ slightly and cross-company benchmarks are worth much less than they appear."},"check":{"question":"After adopting coding assistants, a team's pull request volume doubles, deployment frequency is unchanged, lead time rises from 3 to 6 days and change failure rate rises from 8% to 14%. What does this pattern indicate?","options":[{"text":"The tools are producing lower-quality code, so the remedy is stricter review standards on generated changes specifically.","correct":false,"tempting":"Blames output quality when the same pattern appears with good code: doubled arrival into unchanged review capacity produces both effects."},{"text":"Production rose while verification capacity did not, so the queue lengthened and review quality fell to compensate.","correct":true},{"text":"Deployment frequency is the binding constraint, and automating the release process would let the extra changes through.","correct":false,"tempting":"Deployment automation helps if deploys are the queue, but flat frequency with rising lead time points upstream to review."},{"text":"The measurements are too early to interpret, since adoption effects take two to three quarters to stabilise.","correct":false,"tempting":"Counsels waiting through a pattern that is already coherent and already costing stability."}]},"prev":{"slug":"what-got-harder","title":"The parts that got harder","url":"https://addaly.com/learn/how-software-changed/what-got-harder"},"next":{"slug":"estimation-and-the-demo-effect","title":"Why everything looks nearly done","url":"https://addaly.com/learn/how-software-changed/estimation-and-the-demo-effect"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Four numbers that resist gaming', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/measuring-delivery-not-activity (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Four numbers that resist gaming","courseTitle":"How Software Development Changed","module":{"index":6,"slug":"teams-and-delivery","title":"Teams, process and delivery"},"answer":null,"keyIdea":"Measure at the end of the pipeline with deployment frequency, lead time, change failure rate and recovery time, because those four interlock so that cheating on one degrades another, while every activity count rises with tool use regardless of what is delivered.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Activity counts, which all rise with tool use","items":["Commits and pull requests opened","Lines of code","Suggestion acceptance rate","Percentage of code generated","Story points completed"]},"type":"compare","right":{"title":"Delivery measures, which move only if the system does","items":["Deployment frequency","Lead time from commit to production","Change failure rate","Failed deployment recovery time","Reliability against whatever you promised"]},"title":"Two sets of numbers","caption":"Each of the four on the right can be improved on its own by cheating, and every cheat degrades another one. That interlock is the design, and it is why the set is worth more than any member of it."},"alt":"Figure: Two sets of numbers. Activity counts, which all rise with tool use: Commits and pull requests opened; Lines of code; Suggestion acceptance rate; Percentage of code generated; Story points completed. Delivery measures, which move only if the system does: Deployment frequency; Lead time from commit to production; Change failure rate; Failed deployment recovery time; Reliability against whatever you promised. Each of the four on the right can be improved on its own by cheating, and every cheat degrades another one. That interlock is the design, and it is why the set is worth more than any member of it."}},"blocks":[{"t":"h2","id":"why-activity-metrics-got-worse-not-just-useless","runs":[{"text":"Why activity metrics got worse, not just useless"}]},{"t":"p","runs":[{"text":"Commits, pull requests, lines of code, story points, suggestion acceptance rate. Every one of these rises when a team uses these tools more, whatever happens to delivery. That was always true and it did not matter much, because the correlation between typing volume and delivered value was at least weakly positive. It is now close to zero, and in a review-constrained team it can be negative."}]},{"t":"p","runs":[{"text":"The four measures popularised by the DORA research programme have the opposite property: they sit at the end of the pipeline, so they only improve if the whole system improves. Two are about speed and two about stability, which is what stops either being optimised alone."}]},{"t":"h2","id":"the-four-with-how-to-compute-them-from-what-you-already-have","runs":[{"text":"The four, with how to compute them from what you already have"}]},{"t":"p","runs":[{"text":"Deployment frequency.","b":true},{"text":" How often code reaches production. Count deployments per week from your CI logs. Not releases planned, deployments that happened."}]},{"t":"p","runs":[{"text":"Lead time for changes.","b":true},{"text":" Time from a commit being made to that commit running in production. Every git host has the commit timestamp; your deployment records have the other end."}]},{"t":"code","lang":"bash","label":"Bash","text":"# crude but honest: median hours from commit to the deploy that carried it\ngit log --since=90.days --format='%H %ct' > commits.txt\n# join against your deploy log by commit sha, take the difference, take the median"},{"t":"p","runs":[{"text":"Change failure rate.","b":true},{"text":" The proportion of deployments that caused a degraded service: a rollback, a hotfix, an incident. Count them. If you have no record of rollbacks, that absence is the first finding."}]},{"t":"p","runs":[{"text":"Failed deployment recovery time.","b":true},{"text":" Median time from a bad deployment being noticed to service being restored. Your incident records have this if you keep any."}]},{"t":"p","runs":[{"text":"A fifth, added in later research and worth tracking: "},{"text":"reliability","b":true},{"text":", meaning whether the service meets whatever availability or latency promise you have made."}]},{"t":"h2","id":"why-these-resist-gaming","runs":[{"text":"Why these resist gaming"}]},{"t":"p","runs":[{"text":"Each one can be improved individually by cheating, and the cheat degrades another."}]},{"t":"p","runs":[{"text":"Deploy more often by shipping without review: change failure rate rises. Reduce change failure rate by deploying rarely and carefully: deployment frequency falls and lead time rises. Reduce lead time by skipping verification: both stability measures degrade."}]},{"t":"p","runs":[{"text":"The only way to move all four together is to make changes smaller, verification faster and recovery easier, which is the actual goal. That interlock is the design, and it is why the set is worth more than any of its members."}]},{"t":"h2","id":"what-they-show-about-these-tools-specifically","runs":[{"text":"What they show about these tools specifically"}]},{"t":"p","runs":[{"text":"The expected signature of adoption without process change is precise, and it is worth watching for:"}]},{"t":"ul","items":[[{"text":"Commits and pull requests up, sometimes sharply"}],[{"text":"Deployment frequency flat"}],[{"text":"Lead time up, because the review queue lengthened"}],[{"text":"Change failure rate up, because review got shallower"}]]},{"t":"p","runs":[{"text":"If that is your pattern, the constraint is downstream of writing and adding more writing capacity makes it worse. Industry reports have described versions of this pattern, and you should not need a report: the four numbers from your own systems are more convincing to your own management than any survey."}]},{"t":"h2","id":"what-they-do-not-measure","runs":[{"text":"What they do not measure"}]},{"t":"p","runs":[{"text":"Be honest about the limits, because a metric oversold gets discarded."}]},{"t":"p","runs":[{"text":"They say nothing about whether you built the right thing.","b":true},{"text":" A team can deploy the wrong feature forty times a week with an excellent change failure rate. Value is a separate question and these numbers do not touch it."}]},{"t":"p","runs":[{"text":"They can be gamed by redefinition.","b":true},{"text":" “Incident” is whatever your team decides it is, and a quiet narrowing of that definition improves the numbers without improving anything. Fix the definitions once, write them down, and do not change them to make a quarter look better."}]},{"t":"p","runs":[{"text":"They are team measures, never individual ones.","b":true},{"text":" Lead time is a property of a pipeline that many people share. Attributing it to a person measures who happened to be assigned the changes that queued."}]},{"t":"p","runs":[{"text":"They lag.","b":true},{"text":" Change failure rate over a quarter will not tell you that this week’s practice has decayed. Pair them with the leading indicators from earlier in this course: median diff size, review turnaround, and how many people can explain a given service."}]},{"t":"h2","id":"start-smaller-than-you-think","runs":[{"text":"Start smaller than you think"}]},{"t":"p","runs":[{"text":"A team that has never measured any of this should not build a dashboard. Take a spreadsheet, take the last thirty deployments, fill in four columns by hand, and compute four medians. It takes an afternoon and it is usually the most surprising afternoon of the quarter: most teams discover their lead time is several times what they believed, and almost all of it is waiting."}]},{"t":"p","runs":[{"text":"Do it again in three months. The comparison is the point; the absolute numbers mean less than the direction, because every team’s definitions differ slightly and cross-company benchmarks are worth much less than they appear."}]}],"check":{"question":[{"text":"After adopting coding assistants, a team’s pull request volume doubles, deployment frequency is unchanged, lead time rises from 3 to 6 days and change failure rate rises from 8% to 14%. What does this pattern indicate?"}],"options":[{"text":[{"text":"The tools are producing lower-quality code, so the remedy is stricter review standards on generated changes specifically."}],"correct":false,"tempting":[{"text":"Blames output quality when the same pattern appears with good code: doubled arrival into unchanged review capacity produces both effects."}]},{"text":[{"text":"Production rose while verification capacity did not, so the queue lengthened and review quality fell to compensate."}],"correct":true,"tempting":null},{"text":[{"text":"Deployment frequency is the binding constraint, and automating the release process would let the extra changes through."}],"correct":false,"tempting":[{"text":"Deployment automation helps if deploys are the queue, but flat frequency with rising lead time points upstream to review."}]},{"text":[{"text":"The measurements are too early to interpret, since adoption effects take two to three quarters to stabilise."}],"correct":false,"tempting":[{"text":"Counsels waiting through a pattern that is already coherent and already costing stability."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"what-got-harder","title":"The parts that got harder","minutes":10},"next":{"slug":"estimation-and-the-demo-effect","title":"Why everything looks nearly done","minutes":9}}}