Four numbers that resist gaming

Why activity metrics got worse, not just useless

Commits, pull requests, lines of code, story points, suggestion acceptance rate. Every one of these rises when a team uses these tools more, whatever happens to delivery. That was always true and it did not matter much, because the correlation between typing volume and delivered value was at least weakly positive. It is now close to zero, and in a review-constrained team it can be negative.

The four measures popularised by the DORA research programme have the opposite property: they sit at the end of the pipeline, so they only improve if the whole system improves. Two are about speed and two about stability, which is what stops either being optimised alone.

The four, with how to compute them from what you already have

Deployment frequency. How often code reaches production. Count deployments per week from your CI logs. Not releases planned, deployments that happened.

Lead time for changes. Time from a commit being made to that commit running in production. Every git host has the commit timestamp; your deployment records have the other end.

# crude but honest: median hours from commit to the deploy that carried it
git log --since=90.days --format='%H %ct' > commits.txt
# join against your deploy log by commit sha, take the difference, take the median

Change failure rate. The proportion of deployments that caused a degraded service: a rollback, a hotfix, an incident. Count them. If you have no record of rollbacks, that absence is the first finding.

Failed deployment recovery time. Median time from a bad deployment being noticed to service being restored. Your incident records have this if you keep any.

A fifth, added in later research and worth tracking: reliability, meaning whether the service meets whatever availability or latency promise you have made.

Why these resist gaming

Each one can be improved individually by cheating, and the cheat degrades another.

Deploy more often by shipping without review: change failure rate rises. Reduce change failure rate by deploying rarely and carefully: deployment frequency falls and lead time rises. Reduce lead time by skipping verification: both stability measures degrade.

The only way to move all four together is to make changes smaller, verification faster and recovery easier, which is the actual goal. That interlock is the design, and it is why the set is worth more than any of its members.

What they show about these tools specifically

The expected signature of adoption without process change is precise, and it is worth watching for:

• Commits and pull requests up, sometimes sharply
• Deployment frequency flat
• Lead time up, because the review queue lengthened
• Change failure rate up, because review got shallower

If that is your pattern, the constraint is downstream of writing and adding more writing capacity makes it worse. Industry reports have described versions of this pattern, and you should not need a report: the four numbers from your own systems are more convincing to your own management than any survey.

What they do not measure

Be honest about the limits, because a metric oversold gets discarded.

They say nothing about whether you built the right thing. A team can deploy the wrong feature forty times a week with an excellent change failure rate. Value is a separate question and these numbers do not touch it.

They can be gamed by redefinition. “Incident” is whatever your team decides it is, and a quiet narrowing of that definition improves the numbers without improving anything. Fix the definitions once, write them down, and do not change them to make a quarter look better.

They are team measures, never individual ones. Lead time is a property of a pipeline that many people share. Attributing it to a person measures who happened to be assigned the changes that queued.

They lag. Change failure rate over a quarter will not tell you that this week’s practice has decayed. Pair them with the leading indicators from earlier in this course: median diff size, review turnaround, and how many people can explain a given service.

Start smaller than you think

A team that has never measured any of this should not build a dashboard. Take a spreadsheet, take the last thirty deployments, fill in four columns by hand, and compute four medians. It takes an afternoon and it is usually the most surprising afternoon of the quarter: most teams discover their lead time is several times what they believed, and almost all of it is waiting.

Do it again in three months. The comparison is the point; the absolute numbers mean less than the direction, because every team’s definitions differ slightly and cross-company benchmarks are worth much less than they appear.