Letting a machine help you review

The line that decides everything

Using a model to help review is useful, and the boundary is sharp enough to state in one sentence: use it for anything whose answer you can verify in under a minute, and for nothing else.

That rule sorts the whole space and it holds regardless of which tool you use.

On the verifiable side, and worth doing:

• Summarise this 400-line diff in five bullet points. You check it against the file list.
• List every call site of Invoice.total() in this repository. You check it with rg.
• Explain what this regex matches. You check it with three inputs.
• What does this SQL query return if status is null? You check it against the query.
• Rewrite this stack trace as a sequence of what called what.
• What files does this change touch that the ticket does not mention?

Each of these reduces your reading cost on something you would otherwise do slowly, and each is falsifiable in seconds. That is a good trade and you should take it.

On the unverifiable side, and worth refusing:

• Is this change correct?
• Is this the right design?
• Is this safe to deploy on Friday?

Those questions require what the model does not have: what this system is for, which of these code paths carries real money, what broke last quarter, what your on-call rotation can absorb at 2am, and which of the two designs fits the thing your company is about to do next. An answer will still arrive, phrased with the same confidence as the verifiable ones, and that is the trap.

The false-positive economics of a review bot

Automated review comments on pull requests are now easy to set up, and most teams that set one up turn it off within two months. The reason is a precision problem and it is worth understanding numerically, because it determines how to configure the thing.

Suppose the bot leaves eight comments per pull request and two are useful. Precision is 25%. A developer reads eight, acts on two, and dismisses six. After twenty pull requests they have dismissed a hundred and twenty comments. The habit they have built is dismiss, and it is now applied to comment one hundred and twenty-one, which was the SQL injection.

A noisy checker is not neutral. It is worse than no checker, because it trains the exact behaviour that makes checkers useless. This is the same failure as an alerting system that pages too often, and it has the same fix.

Configure for precision, not coverage. Turn on only the categories that are almost always right, and accept that you will miss things:

• Hard-coded secrets and credentials: near-perfect precision, catastrophic miss cost. Free tools: gitleaks, trufflehog, GitHub’s own secret scanning.
• Known-vulnerable dependency versions: a database lookup, not a judgement.
• Removed test files and skipped tests: a fact about the diff.
• Diff size and files touched outside the stated scope: facts, not opinions.
• Missing migration for a changed model: a structural check.

Everything softer (naming, design suggestions, “consider extracting this”) should be off by default. It is exactly the category with poor precision and low value, and it is what most bots emit most of.

Make the bot post one comment, not many. A single summary comment that updates in place is read. Fifteen inline comments are collapsed.

Never let it approve. A bot approval satisfies the branch protection rule and provides no review. If your process can be satisfied without a human reading the change, you have automated the paperwork rather than the work.

The one new thing worth automating

There is a check that was not possible before and is now cheap: ask whether the diff does what its description says.

Give a model the pull request description, the linked ticket, and the diff, and ask it to list anything in the diff that is not explained by the description. It is quite good at this (it is a comparison task with both sides supplied, which is the shape it handles best) and it targets the specific failure that matters most: scope creep in a change nobody scoped.

The output still goes to a human, phrased as a question rather than a verdict, and the human decides. Which is the same rule as everywhere else in this course: the machine narrows what you have to look at, and the looking stays yours.