{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"machine-assisted-review","title":"Letting a machine help you review","url":"https://addaly.com/learn/how-software-changed/machine-assisted-review","position":19,"of":87,"minutes":9,"answer":"Delegate review questions you can verify in under a minute and refuse the ones you cannot, and configure any review bot for precision over coverage.","keyIdea":"Delegate review questions you can verify in under a minute and refuse the ones you cannot, and configure any review bot for precision over coverage, because a checker that is usually wrong trains people to dismiss the one time it is right."},"body":{"format":"markdown","text":"## The line that decides everything\n\nUsing a model to help review is useful, and the boundary is sharp enough to state in one sentence: **use it for anything whose answer you can verify in under a minute, and for nothing else.**\n\nThat rule sorts the whole space and it holds regardless of which tool you use.\n\n**On the verifiable side, and worth doing:**\n\n- Summarise this 400-line diff in five bullet points. You check it against the file list.\n- List every call site of `Invoice.total()` in this repository. You check it with `rg`.\n- Explain what this regex matches. You check it with three inputs.\n- What does this SQL query return if `status` is null? You check it against the query.\n- Rewrite this stack trace as a sequence of what called what.\n- What files does this change touch that the ticket does not mention?\n\nEach of these reduces your reading cost on something you would otherwise do slowly, and each is falsifiable in seconds. That is a good trade and you should take it.\n\n**On the unverifiable side, and worth refusing:**\n\n- Is this change correct?\n- Is this the right design?\n- Is this safe to deploy on Friday?\n\nThose questions require what the model does not have: what this system is for, which of these code paths carries real money, what broke last quarter, what your on-call rotation can absorb at 2am, and which of the two designs fits the thing your company is about to do next. An answer will still arrive, phrased with the same confidence as the verifiable ones, and that is the trap.\n\n## The false-positive economics of a review bot\n\nAutomated review comments on pull requests are now easy to set up, and most teams that set one up turn it off within two months. The reason is a precision problem and it is worth understanding numerically, because it determines how to configure the thing.\n\nSuppose the bot leaves eight comments per pull request and two are useful. Precision is 25%. A developer reads eight, acts on two, and dismisses six. After twenty pull requests they have dismissed a hundred and twenty comments. The habit they have built is *dismiss*, and it is now applied to comment one hundred and twenty-one, which was the SQL injection.\n\nA noisy checker is not neutral. It is worse than no checker, because it trains the exact behaviour that makes checkers useless. This is the same failure as an alerting system that pages too often, and it has the same fix.\n\n**Configure for precision, not coverage.** Turn on only the categories that are almost always right, and accept that you will miss things:\n\n- Hard-coded secrets and credentials: near-perfect precision, catastrophic miss cost. Free tools: `gitleaks`, `trufflehog`, GitHub's own secret scanning.\n- Known-vulnerable dependency versions: a database lookup, not a judgement.\n- Removed test files and skipped tests: a fact about the diff.\n- Diff size and files touched outside the stated scope: facts, not opinions.\n- Missing migration for a changed model: a structural check.\n\nEverything softer (naming, design suggestions, \"consider extracting this\") should be off by default. It is exactly the category with poor precision and low value, and it is what most bots emit most of.\n\n**Make the bot post one comment, not many.** A single summary comment that updates in place is read. Fifteen inline comments are collapsed.\n\n**Never let it approve.** A bot approval satisfies the branch protection rule and provides no review. If your process can be satisfied without a human reading the change, you have automated the paperwork rather than the work.\n\n## The one new thing worth automating\n\nThere is a check that was not possible before and is now cheap: **ask whether the diff does what its description says.**\n\nGive a model the pull request description, the linked ticket, and the diff, and ask it to list anything in the diff that is not explained by the description. It is quite good at this (it is a comparison task with both sides supplied, which is the shape it handles best) and it targets the specific failure that matters most: scope creep in a change nobody scoped.\n\nThe output still goes to a human, phrased as a question rather than a verdict, and the human decides. Which is the same rule as everywhere else in this course: the machine narrows what you have to look at, and the looking stays yours."},"check":{"question":"A team's review bot leaves roughly eight comments per pull request, of which two are useful. A proposal is made to improve it by adding more rule categories so it catches more issues. What does the precision argument predict?","options":[{"text":"Coverage will rise and so will the number of useful comments, which is a net gain as long as developers read carefully.","correct":false,"tempting":"Counts the useful comments gained and ignores that reading carefully is precisely the behaviour a low-precision stream destroys."},{"text":"Nothing will change, because developers already ignore the bot, so additional categories are simply wasted configuration effort.","correct":false,"tempting":"Assumes the current state is a stable floor, when adding noise pushes teams from ignoring the bot to removing it."},{"text":"More categories mean more comments per pull request but the same absolute number of real findings, since rule-based checks overlap heavily.","correct":false,"tempting":"Invents an overlap assumption; new categories do find new things, which is why the proposal is tempting in the first place."},{"text":"Precision falls further, dismissal becomes more automatic, and the probability that a genuinely serious comment is acted on goes down.","correct":true}]},"prev":{"slug":"review-as-knowledge-transfer","title":"The second job of review, and how it broke","url":"https://addaly.com/learn/how-software-changed/review-as-knowledge-transfer"},"next":{"slug":"verification-is-the-bottleneck","title":"Testing when generating is free","url":"https://addaly.com/learn/how-software-changed/verification-is-the-bottleneck"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Letting a machine help you review', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/machine-assisted-review (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Letting a machine help you review","courseTitle":"How Software Development Changed","module":{"index":2,"slug":"reading-what-you-did-not-write","title":"Reading what you did not write"},"answer":null,"keyIdea":"Delegate review questions you can verify in under a minute and refuse the ones you cannot, and configure any review bot for precision over coverage, because a checker that is usually wrong trains people to dismiss the one time it is right.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-line-that-decides-everything","runs":[{"text":"The line that decides everything"}]},{"t":"p","runs":[{"text":"Using a model to help review is useful, and the boundary is sharp enough to state in one sentence: "},{"text":"use it for anything whose answer you can verify in under a minute, and for nothing else.","b":true}]},{"t":"p","runs":[{"text":"That rule sorts the whole space and it holds regardless of which tool you use."}]},{"t":"p","runs":[{"text":"On the verifiable side, and worth doing:","b":true}]},{"t":"ul","items":[[{"text":"Summarise this 400-line diff in five bullet points. You check it against the file list."}],[{"text":"List every call site of "},{"text":"Invoice.total()","code":true},{"text":" in this repository. You check it with "},{"text":"rg","code":true},{"text":"."}],[{"text":"Explain what this regex matches. You check it with three inputs."}],[{"text":"What does this SQL query return if "},{"text":"status","code":true},{"text":" is null? You check it against the query."}],[{"text":"Rewrite this stack trace as a sequence of what called what."}],[{"text":"What files does this change touch that the ticket does not mention?"}]]},{"t":"p","runs":[{"text":"Each of these reduces your reading cost on something you would otherwise do slowly, and each is falsifiable in seconds. That is a good trade and you should take it."}]},{"t":"p","runs":[{"text":"On the unverifiable side, and worth refusing:","b":true}]},{"t":"ul","items":[[{"text":"Is this change correct?"}],[{"text":"Is this the right design?"}],[{"text":"Is this safe to deploy on Friday?"}]]},{"t":"p","runs":[{"text":"Those questions require what the model does not have: what this system is for, which of these code paths carries real money, what broke last quarter, what your on-call rotation can absorb at 2am, and which of the two designs fits the thing your company is about to do next. An answer will still arrive, phrased with the same confidence as the verifiable ones, and that is the trap."}]},{"t":"h2","id":"the-false-positive-economics-of-a-review-bot","runs":[{"text":"The false-positive economics of a review bot"}]},{"t":"p","runs":[{"text":"Automated review comments on pull requests are now easy to set up, and most teams that set one up turn it off within two months. The reason is a precision problem and it is worth understanding numerically, because it determines how to configure the thing."}]},{"t":"p","runs":[{"text":"Suppose the bot leaves eight comments per pull request and two are useful. Precision is 25%. A developer reads eight, acts on two, and dismisses six. After twenty pull requests they have dismissed a hundred and twenty comments. The habit they have built is "},{"text":"dismiss","i":true},{"text":", and it is now applied to comment one hundred and twenty-one, which was the SQL injection."}]},{"t":"p","runs":[{"text":"A noisy checker is not neutral. It is worse than no checker, because it trains the exact behaviour that makes checkers useless. This is the same failure as an alerting system that pages too often, and it has the same fix."}]},{"t":"p","runs":[{"text":"Configure for precision, not coverage.","b":true},{"text":" Turn on only the categories that are almost always right, and accept that you will miss things:"}]},{"t":"ul","items":[[{"text":"Hard-coded secrets and credentials: near-perfect precision, catastrophic miss cost. Free tools: "},{"text":"gitleaks","code":true},{"text":", "},{"text":"trufflehog","code":true},{"text":", GitHub’s own secret scanning."}],[{"text":"Known-vulnerable dependency versions: a database lookup, not a judgement."}],[{"text":"Removed test files and skipped tests: a fact about the diff."}],[{"text":"Diff size and files touched outside the stated scope: facts, not opinions."}],[{"text":"Missing migration for a changed model: a structural check."}]]},{"t":"p","runs":[{"text":"Everything softer (naming, design suggestions, “consider extracting this”) should be off by default. It is exactly the category with poor precision and low value, and it is what most bots emit most of."}]},{"t":"p","runs":[{"text":"Make the bot post one comment, not many.","b":true},{"text":" A single summary comment that updates in place is read. Fifteen inline comments are collapsed."}]},{"t":"p","runs":[{"text":"Never let it approve.","b":true},{"text":" A bot approval satisfies the branch protection rule and provides no review. If your process can be satisfied without a human reading the change, you have automated the paperwork rather than the work."}]},{"t":"h2","id":"the-one-new-thing-worth-automating","runs":[{"text":"The one new thing worth automating"}]},{"t":"p","runs":[{"text":"There is a check that was not possible before and is now cheap: "},{"text":"ask whether the diff does what its description says.","b":true}]},{"t":"p","runs":[{"text":"Give a model the pull request description, the linked ticket, and the diff, and ask it to list anything in the diff that is not explained by the description. It is quite good at this (it is a comparison task with both sides supplied, which is the shape it handles best) and it targets the specific failure that matters most: scope creep in a change nobody scoped."}]},{"t":"p","runs":[{"text":"The output still goes to a human, phrased as a question rather than a verdict, and the human decides. Which is the same rule as everywhere else in this course: the machine narrows what you have to look at, and the looking stays yours."}]}],"check":{"question":[{"text":"A team’s review bot leaves roughly eight comments per pull request, of which two are useful. A proposal is made to improve it by adding more rule categories so it catches more issues. What does the precision argument predict?"}],"options":[{"text":[{"text":"Coverage will rise and so will the number of useful comments, which is a net gain as long as developers read carefully."}],"correct":false,"tempting":[{"text":"Counts the useful comments gained and ignores that reading carefully is precisely the behaviour a low-precision stream destroys."}]},{"text":[{"text":"Nothing will change, because developers already ignore the bot, so additional categories are simply wasted configuration effort."}],"correct":false,"tempting":[{"text":"Assumes the current state is a stable floor, when adding noise pushes teams from ignoring the bot to removing it."}]},{"text":[{"text":"More categories mean more comments per pull request but the same absolute number of real findings, since rule-based checks overlap heavily."}],"correct":false,"tempting":[{"text":"Invents an overlap assumption; new categories do find new things, which is why the proposal is tempting in the first place."}]},{"text":[{"text":"Precision falls further, dismissal becomes more automatic, and the probability that a genuinely serious comment is acted on goes down."}],"correct":true,"tempting":null}],"order":[0,2,3,1],"why":null},"prev":{"slug":"review-as-knowledge-transfer","title":"The second job of review, and how it broke","minutes":9},"next":{"slug":"verification-is-the-bottleneck","title":"Testing when generating is free","minutes":9}}}