{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"the-honest-ledger","title":"Measuring the effect on yourself","url":"https://addaly.com/learn/how-software-changed/the-honest-ledger","position":9,"of":87,"minutes":9,"answer":"A two-week self-experiment with coin-flip allocation, a merged-and-deployed end point and a thirty-day defect column will tell you more about your own practice than any published study.","keyIdea":"A two-week self-experiment with coin-flip allocation, a merged-and-deployed end point and a thirty-day defect column will tell you more about your own practice than any published study, provided you write the decision rule before you see the data."},"body":{"format":"markdown","text":"## A fortnight, designed properly\n\nYou will not settle the general question. You can settle the question about you, in your codebase, with your habits, and that is the only version that should govern your working day. Here is a design that costs about ten minutes a day and survives the objections in the previous lessons.\n\n**Write down the prediction and the decision rule first.** Two sentences, before you start:\n\n> I expect assisted tasks to take about 30% less time to reach merged-and-green. If the difference is under 10% either way, I will conclude I cannot tell from a sample this size and stop claiming either.\n\nThat second sentence is what stops you from reading a result into noise, and writing it before you see the data is what makes it honest. This is preregistration on a napkin, and it costs nothing.\n\n**Randomise per task.** Flip a coin: actually flip one, or use `shuf -n1 -e assisted unassisted`. Per task, never per day and never per project. Allocation by day measures your Mondays.\n\n**Define the end point as merged, green in CI, and deployed.** Not \"code written\". The entire divergence between impression and reality lives after the code was written, so an end point before that is measuring the segment you already know moved.\n\n**Log four fields per task, in a text file.** Date, condition, minutes elapsed, and one word for size. Nothing more elaborate survives a busy week.\n\n```\n2026-09-02  assisted    95   small\n2026-09-02  unassisted 140   small\n2026-09-03  assisted   310   large\n```\n\n**Follow each change for thirty days.** Add a fifth column later: did it come back? A follow-up commit that fixes it, a revert, a bug report, an incident. Speed with no defect column is half a measurement, and it is the half that flatters.\n\n## What the numbers can and cannot tell you\n\nBe realistic about power. Task durations are wildly variable (the same nominal size can differ by a factor of five depending on what you hit), so with twenty tasks you can detect a large difference and nothing else. If your two medians differ by 15%, the correct conclusion is that you cannot tell. Writing that down is a result, not a failure.\n\nUse medians, not means. One task that hit a bad deployment problem and took nine hours will dominate a mean of ten numbers and tell you nothing about the tool.\n\nRecord the obvious confounds in a margin note: interruptions, whether you knew the area, whether you were tired, whether the task turned out to be something other than what the ticket said. That last one is more common than any of the others and it is usually the real explanation for an outlier.\n\n## The three numbers worth keeping afterwards\n\nWhen the fortnight ends, keep measuring three things permanently. They are cheap and they are the ones that move with reality.\n\n1. **Median time from first commit to deployed.** Your personal lead time. It captures review, CI and deployment, which is where the waiting is.\n2. **Proportion of your changes that came back within thirty days.** Your personal change failure rate. If this rises while your lead time falls, you have not got faster, you have moved work into the future.\n3. **Proportion of your merged changes you could still explain a week later.** This one is unusual and it is the most informative of the three. Once a week, open a change you merged seven days ago, read it, and ask whether you can say why each non-obvious line is there. Score it yes or no. A falling score is the earliest available signal that you are shipping code you do not own, and it appears months before anything shows up in an incident.\n\n## What not to measure, and why it matters ethically\n\nLines of code, commits, pull requests opened, suggestion acceptance rate, percentage of code generated. All of them go up when you use the tools more, whatever the effect on delivery, which is what makes them attractive and useless.\n\nThe stronger version of this warning is about other people. If any of these numbers is used to assess an individual, the behaviour it produces is entirely predictable: more, larger, less carefully read diffs, submitted faster, into a review queue that is already the constraint. You will have used a measurement to make the bottleneck worse and called it a productivity programme.\n\nMeasure the team's flow and the system's failure rate. Measure individuals with your eyes, in review, by asking them to explain their work, which is a conversation, not a metric, and it is the only instrument that detects the failure mode this course is about."},"check":{"question":"After twenty randomised tasks, assisted work has a median of 96 minutes and unassisted 110 minutes. The developer's preregistered rule said differences under 10% would be treated as undetectable. This is 13%. What is the sound conclusion?","options":[{"text":"The rule was met, so assisted work is faster by roughly 13% for this developer on this kind of task.","correct":false,"tempting":"Applies the threshold mechanically without noticing that task durations vary by a factor of several, so twenty samples cannot resolve a gap this small."},{"text":"The rule should be revised downward now that real variability is known, since 13% is a meaningful business difference.","correct":false,"tempting":"Changes the decision rule after seeing the data, which is precisely the move preregistration exists to prevent."},{"text":"The result is uninterpretable because the defect column has not yet had thirty days to fill, and speed without it is not a result.","correct":false,"tempting":"Names a genuine gap in the measurement, but the sample-size problem makes the speed figure unreadable regardless of what the defect column later shows."},{"text":"Twenty highly variable tasks cannot separate a 13% gap from noise, so the experiment did not resolve the question.","correct":true}]},"prev":{"slug":"what-the-loop-costs","title":"What the loop costs, and the free path","url":"https://addaly.com/learn/how-software-changed/what-the-loop-costs"},"next":{"slug":"reading-at-volume","title":"Reviewing code you did not write","url":"https://addaly.com/learn/how-software-changed/reading-at-volume"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Measuring the effect on yourself', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/the-honest-ledger (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Measuring the effect on yourself","courseTitle":"How Software Development Changed","module":{"index":1,"slug":"where-the-cost-went","title":"Where the cost actually went"},"answer":null,"keyIdea":"A two-week self-experiment with coin-flip allocation, a merged-and-deployed end point and a thirty-day defect column will tell you more about your own practice than any published study, provided you write the decision rule before you see the data.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-fortnight-designed-properly","runs":[{"text":"A fortnight, designed properly"}]},{"t":"p","runs":[{"text":"You will not settle the general question. You can settle the question about you, in your codebase, with your habits, and that is the only version that should govern your working day. Here is a design that costs about ten minutes a day and survives the objections in the previous lessons."}]},{"t":"p","runs":[{"text":"Write down the prediction and the decision rule first.","b":true},{"text":" Two sentences, before you start:"}]},{"t":"quote","runs":[{"text":"I expect assisted tasks to take about 30% less time to reach merged-and-green. If the difference is under 10% either way, I will conclude I cannot tell from a sample this size and stop claiming either."}]},{"t":"p","runs":[{"text":"That second sentence is what stops you from reading a result into noise, and writing it before you see the data is what makes it honest. This is preregistration on a napkin, and it costs nothing."}]},{"t":"p","runs":[{"text":"Randomise per task.","b":true},{"text":" Flip a coin: actually flip one, or use "},{"text":"shuf -n1 -e assisted unassisted","code":true},{"text":". Per task, never per day and never per project. Allocation by day measures your Mondays."}]},{"t":"p","runs":[{"text":"Define the end point as merged, green in CI, and deployed.","b":true},{"text":" Not “code written”. The entire divergence between impression and reality lives after the code was written, so an end point before that is measuring the segment you already know moved."}]},{"t":"p","runs":[{"text":"Log four fields per task, in a text file.","b":true},{"text":" Date, condition, minutes elapsed, and one word for size. Nothing more elaborate survives a busy week."}]},{"t":"code","lang":"","label":"Text","text":"2026-09-02  assisted    95   small\n2026-09-02  unassisted 140   small\n2026-09-03  assisted   310   large"},{"t":"p","runs":[{"text":"Follow each change for thirty days.","b":true},{"text":" Add a fifth column later: did it come back? A follow-up commit that fixes it, a revert, a bug report, an incident. Speed with no defect column is half a measurement, and it is the half that flatters."}]},{"t":"h2","id":"what-the-numbers-can-and-cannot-tell-you","runs":[{"text":"What the numbers can and cannot tell you"}]},{"t":"p","runs":[{"text":"Be realistic about power. Task durations are wildly variable (the same nominal size can differ by a factor of five depending on what you hit), so with twenty tasks you can detect a large difference and nothing else. If your two medians differ by 15%, the correct conclusion is that you cannot tell. Writing that down is a result, not a failure."}]},{"t":"p","runs":[{"text":"Use medians, not means. One task that hit a bad deployment problem and took nine hours will dominate a mean of ten numbers and tell you nothing about the tool."}]},{"t":"p","runs":[{"text":"Record the obvious confounds in a margin note: interruptions, whether you knew the area, whether you were tired, whether the task turned out to be something other than what the ticket said. That last one is more common than any of the others and it is usually the real explanation for an outlier."}]},{"t":"h2","id":"the-three-numbers-worth-keeping-afterwards","runs":[{"text":"The three numbers worth keeping afterwards"}]},{"t":"p","runs":[{"text":"When the fortnight ends, keep measuring three things permanently. They are cheap and they are the ones that move with reality."}]},{"t":"ol","start":1,"items":[[{"text":"Median time from first commit to deployed.","b":true},{"text":" Your personal lead time. It captures review, CI and deployment, which is where the waiting is."}],[{"text":"Proportion of your changes that came back within thirty days.","b":true},{"text":" Your personal change failure rate. If this rises while your lead time falls, you have not got faster, you have moved work into the future."}],[{"text":"Proportion of your merged changes you could still explain a week later.","b":true},{"text":" This one is unusual and it is the most informative of the three. Once a week, open a change you merged seven days ago, read it, and ask whether you can say why each non-obvious line is there. Score it yes or no. A falling score is the earliest available signal that you are shipping code you do not own, and it appears months before anything shows up in an incident."}]]},{"t":"h2","id":"what-not-to-measure-and-why-it-matters-ethically","runs":[{"text":"What not to measure, and why it matters ethically"}]},{"t":"p","runs":[{"text":"Lines of code, commits, pull requests opened, suggestion acceptance rate, percentage of code generated. All of them go up when you use the tools more, whatever the effect on delivery, which is what makes them attractive and useless."}]},{"t":"p","runs":[{"text":"The stronger version of this warning is about other people. If any of these numbers is used to assess an individual, the behaviour it produces is entirely predictable: more, larger, less carefully read diffs, submitted faster, into a review queue that is already the constraint. You will have used a measurement to make the bottleneck worse and called it a productivity programme."}]},{"t":"p","runs":[{"text":"Measure the team’s flow and the system’s failure rate. Measure individuals with your eyes, in review, by asking them to explain their work, which is a conversation, not a metric, and it is the only instrument that detects the failure mode this course is about."}]}],"check":{"question":[{"text":"After twenty randomised tasks, assisted work has a median of 96 minutes and unassisted 110 minutes. The developer’s preregistered rule said differences under 10% would be treated as undetectable. This is 13%. What is the sound conclusion?"}],"options":[{"text":[{"text":"The rule was met, so assisted work is faster by roughly 13% for this developer on this kind of task."}],"correct":false,"tempting":[{"text":"Applies the threshold mechanically without noticing that task durations vary by a factor of several, so twenty samples cannot resolve a gap this small."}]},{"text":[{"text":"The rule should be revised downward now that real variability is known, since 13% is a meaningful business difference."}],"correct":false,"tempting":[{"text":"Changes the decision rule after seeing the data, which is precisely the move preregistration exists to prevent."}]},{"text":[{"text":"The result is uninterpretable because the defect column has not yet had thirty days to fill, and speed without it is not a result."}],"correct":false,"tempting":[{"text":"Names a genuine gap in the measurement, but the sample-size problem makes the speed figure unreadable regardless of what the defect column later shows."}]},{"text":[{"text":"Twenty highly variable tasks cannot separate a 13% gap from noise, so the experiment did not resolve the question."}],"correct":true,"tempting":null}],"order":[0,1,2,3],"why":null},"prev":{"slug":"what-the-loop-costs","title":"What the loop costs, and the free path","minutes":9},"next":{"slug":"reading-at-volume","title":"Reviewing code you did not write","minutes":9}}}