{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"debugging-did-not-change","title":"Debugging still works the old way","url":"https://addaly.com/learn/how-software-changed/debugging-did-not-change","position":43,"of":87,"minutes":9,"answer":"Debugging held its value because the evidence lives in the running system rather than in any text, so the method stays reproduce, minimise, bisect, hypothesise and instrument.","keyIdea":"Debugging held its value because the evidence lives in the running system rather than in any text, so the method stays reproduce, minimise, bisect, hypothesise and instrument, and a suggested cause is a hypothesis to test rather than a diagnosis."},"body":{"format":"markdown","text":"## Why this is the skill that held its value\n\nA model reasons over text. A bug lives in a running system: in the data, in the timing, in the load, in the interaction between two services, in the thing that only happens on Tuesdays because of a cron job somebody wrote in 2021.\n\nAlmost none of that is in any text you can supply. That is why debugging is the part of the job that got relatively more valuable, and why it is worth being deliberate about the method rather than poking until it works.\n\n## The method, in five steps\n\n**1. Reproduce it.** Nothing before this counts. If you cannot make it happen on demand you cannot know when you have fixed it, and every subsequent step is guessing. Getting a reliable reproduction is often 80% of the work and it is the part people try to skip.\n\nIf it only happens in production, your goal shifts to capturing enough state (the exact request, the exact row, the exact time, the exact configuration) that you can replay it locally.\n\n**2. Minimise it.** Cut away everything not required to trigger it. Fewer rows, fewer fields, fewer services, a smaller input. Every element you remove without the bug disappearing is a hypothesis eliminated for free.\n\nThis is what property-testing frameworks call shrinking, and doing it by hand is the same operation. A bug that reproduces with two rows and one field is a bug you can hold in your head.\n\n**3. Bisect it.** Two dimensions, and both are mechanical rather than clever.\n\nOver history:\n\n```bash\ngit bisect start HEAD v3.1.0\ngit bisect run ./repro.sh\n```\n\nOver the system: does it happen with the cache disabled? With one worker instead of eight? Against a local database rather than the shared one? Each answer halves the search space.\n\n**4. Form a hypothesis that can be wrong.** \"Something is wrong with the date handling\" is not a hypothesis. \"The reservation expiry compares a naive local datetime against an aware UTC one, so it is off by the timezone offset, which is why only Nepali users see it\" is a hypothesis, and it makes a prediction: users at UTC+00:00 will not be affected. Go and check that prediction before you fix anything.\n\nA hypothesis that cannot be wrong cannot be tested, and the whole point of this step is to stop you fixing three things and never learning which one it was.\n\n**5. Instrument.** When reasoning runs out, get more data. Log the actual values at the boundary: not \"entering function\", but the arguments, the types and the intermediate results.\n\n```python\nlogger.info(\"expiry check: now=%r reserved_at=%r tz=%r delta=%r\",\n            now, reserved_at, reserved_at.tzinfo, now - reserved_at)\n```\n\nUse `%r` rather than `%s`. It shows you the quotes, the type and the exact representation, and the difference between `\"3\"` and `3` is a large proportion of all bugs.\n\n## Where a model helps\n\nSpecifically at three points, all of them verifiable:\n\n- **Explaining an unfamiliar error.** A Rust borrow-checker message, a Java stack trace forty frames deep, a cryptic error from a build tool. Fast, useful, and you can check the explanation against the code.\n- **Generating hypotheses.** \"Here is the symptom and here is the function. List ten things that could cause this.\" A list of candidates is useful even when most are wrong, because it breaks the tunnel vision that comes from staring at one theory for an hour.\n- **Writing the instrumentation and the reproduction script.** Boring, mechanical, and immediately verifiable by running it.\n\n## Where it cannot help, and why\n\n**It cannot see the running system.** The value in memory, the query the ORM actually issued, the row that is different from all the others, the lock that was held for 400 milliseconds. Unless you paste it, it does not exist.\n\n**It will confidently name a cause.** Ask what is wrong with a piece of code and you will get an answer whether or not there is anything wrong, because that is the shape of the question. Treat every suggested cause as a hypothesis to test, never as a diagnosis, and notice how strong the pull is to skip the testing when the answer sounds right.\n\n**It has no memory of your incidents.** \"It is always the cache invalidation\" is knowledge that lives in a person who was there. This is the concrete cost of the knowledge-transfer failure from earlier in the course: a team where nobody holds that knowledge debugs everything from first principles, at 3am, under pressure.\n\n## The habit worth keeping\n\nWhen you fix a bug, write two sentences in the pull request: what the cause was, and what would have caught it earlier.\n\nThat second sentence is where the compounding is. \"A `CHECK (expires_at > created_at)` would have made this impossible.\" \"A test with a non-UTC timezone would have caught it.\" Over a year those sentences become your team's actual list of what to invest in, derived from things that happened rather than from what somebody read in a course."},"check":{"question":"A developer describes a bug to a model, receives a confident explanation naming a race condition, applies the suggested fix, and the symptom disappears. Why is this weaker evidence than it appears?","options":[{"text":"Race conditions are rarely the cause of intermittent bugs, so the diagnosis was statistically unlikely from the start.","correct":false,"tempting":"Races are a very common cause of intermittent bugs; dismissing the category avoids the actual reasoning problem."},{"text":"The fix probably introduced synchronisation that slowed the code enough to hide the timing, rather than removing the race.","correct":false,"tempting":"A real and specific failure mode, but it is one possible mechanism rather than the reason the evidence is weak in general."},{"text":"The symptom disappearing is consistent with many causes, and nothing tested the prediction the hypothesis made.","correct":true},{"text":"Without a reliable reproduction the developer cannot know the bug is gone rather than merely rarer.","correct":false,"tempting":"True and important, but it is a precondition that was violated earlier; even with a reproduction, one disappearing symptom does not confirm a cause."}]},"prev":{"slug":"agents-and-blast-radius","title":"Confining an agent","url":"https://addaly.com/learn/how-software-changed/agents-and-blast-radius"},"next":{"slug":"refactoring-and-codemods","title":"The task it is best at","url":"https://addaly.com/learn/how-software-changed/refactoring-and-codemods"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Debugging still works the old way', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/debugging-did-not-change (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Debugging still works the old way","courseTitle":"How Software Development Changed","module":{"index":5,"slug":"the-working-loop","title":"The working loop"},"answer":null,"keyIdea":"Debugging held its value because the evidence lives in the running system rather than in any text, so the method stays reproduce, minimise, bisect, hypothesise and instrument, and a suggested cause is a hypothesis to test rather than a diagnosis.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-this-is-the-skill-that-held-its-value","runs":[{"text":"Why this is the skill that held its value"}]},{"t":"p","runs":[{"text":"A model reasons over text. A bug lives in a running system: in the data, in the timing, in the load, in the interaction between two services, in the thing that only happens on Tuesdays because of a cron job somebody wrote in 2021."}]},{"t":"p","runs":[{"text":"Almost none of that is in any text you can supply. That is why debugging is the part of the job that got relatively more valuable, and why it is worth being deliberate about the method rather than poking until it works."}]},{"t":"h2","id":"the-method-in-five-steps","runs":[{"text":"The method, in five steps"}]},{"t":"p","runs":[{"text":"1. Reproduce it.","b":true},{"text":" Nothing before this counts. If you cannot make it happen on demand you cannot know when you have fixed it, and every subsequent step is guessing. Getting a reliable reproduction is often 80% of the work and it is the part people try to skip."}]},{"t":"p","runs":[{"text":"If it only happens in production, your goal shifts to capturing enough state (the exact request, the exact row, the exact time, the exact configuration) that you can replay it locally."}]},{"t":"p","runs":[{"text":"2. Minimise it.","b":true},{"text":" Cut away everything not required to trigger it. Fewer rows, fewer fields, fewer services, a smaller input. Every element you remove without the bug disappearing is a hypothesis eliminated for free."}]},{"t":"p","runs":[{"text":"This is what property-testing frameworks call shrinking, and doing it by hand is the same operation. A bug that reproduces with two rows and one field is a bug you can hold in your head."}]},{"t":"p","runs":[{"text":"3. Bisect it.","b":true},{"text":" Two dimensions, and both are mechanical rather than clever."}]},{"t":"p","runs":[{"text":"Over history:"}]},{"t":"code","lang":"bash","label":"Bash","text":"git bisect start HEAD v3.1.0\ngit bisect run ./repro.sh"},{"t":"p","runs":[{"text":"Over the system: does it happen with the cache disabled? With one worker instead of eight? Against a local database rather than the shared one? Each answer halves the search space."}]},{"t":"p","runs":[{"text":"4. Form a hypothesis that can be wrong.","b":true},{"text":" “Something is wrong with the date handling” is not a hypothesis. “The reservation expiry compares a naive local datetime against an aware UTC one, so it is off by the timezone offset, which is why only Nepali users see it” is a hypothesis, and it makes a prediction: users at UTC+00:00 will not be affected. Go and check that prediction before you fix anything."}]},{"t":"p","runs":[{"text":"A hypothesis that cannot be wrong cannot be tested, and the whole point of this step is to stop you fixing three things and never learning which one it was."}]},{"t":"p","runs":[{"text":"5. Instrument.","b":true},{"text":" When reasoning runs out, get more data. Log the actual values at the boundary: not “entering function”, but the arguments, the types and the intermediate results."}]},{"t":"code","lang":"python","label":"Python","text":"logger.info(\"expiry check: now=%r reserved_at=%r tz=%r delta=%r\",\n            now, reserved_at, reserved_at.tzinfo, now - reserved_at)"},{"t":"p","runs":[{"text":"Use "},{"text":"%r","code":true},{"text":" rather than "},{"text":"%s","code":true},{"text":". It shows you the quotes, the type and the exact representation, and the difference between "},{"text":"\"3\"","code":true},{"text":" and "},{"text":"3","code":true},{"text":" is a large proportion of all bugs."}]},{"t":"h2","id":"where-a-model-helps","runs":[{"text":"Where a model helps"}]},{"t":"p","runs":[{"text":"Specifically at three points, all of them verifiable:"}]},{"t":"ul","items":[[{"text":"Explaining an unfamiliar error.","b":true},{"text":" A Rust borrow-checker message, a Java stack trace forty frames deep, a cryptic error from a build tool. Fast, useful, and you can check the explanation against the code."}],[{"text":"Generating hypotheses.","b":true},{"text":" “Here is the symptom and here is the function. List ten things that could cause this.” A list of candidates is useful even when most are wrong, because it breaks the tunnel vision that comes from staring at one theory for an hour."}],[{"text":"Writing the instrumentation and the reproduction script.","b":true},{"text":" Boring, mechanical, and immediately verifiable by running it."}]]},{"t":"h2","id":"where-it-cannot-help-and-why","runs":[{"text":"Where it cannot help, and why"}]},{"t":"p","runs":[{"text":"It cannot see the running system.","b":true},{"text":" The value in memory, the query the ORM actually issued, the row that is different from all the others, the lock that was held for 400 milliseconds. Unless you paste it, it does not exist."}]},{"t":"p","runs":[{"text":"It will confidently name a cause.","b":true},{"text":" Ask what is wrong with a piece of code and you will get an answer whether or not there is anything wrong, because that is the shape of the question. Treat every suggested cause as a hypothesis to test, never as a diagnosis, and notice how strong the pull is to skip the testing when the answer sounds right."}]},{"t":"p","runs":[{"text":"It has no memory of your incidents.","b":true},{"text":" “It is always the cache invalidation” is knowledge that lives in a person who was there. This is the concrete cost of the knowledge-transfer failure from earlier in the course: a team where nobody holds that knowledge debugs everything from first principles, at 3am, under pressure."}]},{"t":"h2","id":"the-habit-worth-keeping","runs":[{"text":"The habit worth keeping"}]},{"t":"p","runs":[{"text":"When you fix a bug, write two sentences in the pull request: what the cause was, and what would have caught it earlier."}]},{"t":"p","runs":[{"text":"That second sentence is where the compounding is. “A "},{"text":"CHECK (expires_at > created_at)","code":true},{"text":" would have made this impossible.” “A test with a non-UTC timezone would have caught it.” Over a year those sentences become your team’s actual list of what to invest in, derived from things that happened rather than from what somebody read in a course."}]}],"check":{"question":[{"text":"A developer describes a bug to a model, receives a confident explanation naming a race condition, applies the suggested fix, and the symptom disappears. Why is this weaker evidence than it appears?"}],"options":[{"text":[{"text":"Race conditions are rarely the cause of intermittent bugs, so the diagnosis was statistically unlikely from the start."}],"correct":false,"tempting":[{"text":"Races are a very common cause of intermittent bugs; dismissing the category avoids the actual reasoning problem."}]},{"text":[{"text":"The fix probably introduced synchronisation that slowed the code enough to hide the timing, rather than removing the race."}],"correct":false,"tempting":[{"text":"A real and specific failure mode, but it is one possible mechanism rather than the reason the evidence is weak in general."}]},{"text":[{"text":"The symptom disappearing is consistent with many causes, and nothing tested the prediction the hypothesis made."}],"correct":true,"tempting":null},{"text":[{"text":"Without a reliable reproduction the developer cannot know the bug is gone rather than merely rarer."}],"correct":false,"tempting":[{"text":"True and important, but it is a precondition that was violated earlier; even with a reproduction, one disappearing symptom does not confirm a cause."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"agents-and-blast-radius","title":"Confining an agent","minutes":10},"next":{"slug":"refactoring-and-codemods","title":"The task it is best at","minutes":9}}}