AI Code Review Tools: What They Catch, What They Miss, and How to Deploy Them Without Annoying Everyone

The pitch for AI code review is straightforward enough. Reviews are a bottleneck, senior engineers are expensive, and a model can read a diff in seconds.

What actually happens when a team switches one on is more interesting. Week one, it catches a genuine bug and everyone is impressed. Week three, the pull request comments have become wallpaper. Week six, somebody quietly turns it off.

That trajectory is avoidable, and the fix has almost nothing to do with which tool you pick.

What these tools genuinely do well

An automated reviewer reads a diff and comments on it. Within a specific band of problems, it is reliably better than a tired human on a Friday afternoon.

Caught Unhandled null path, a secret committed in a config file, and an error swallowed without logging. All three are boring, all three are real, and all three are exactly what a human reviewer skims past.

Mechanical correctness. Null and undefined paths, off-by-one errors, unhandled promise rejections, resource leaks, error cases swallowed silently. The unglamorous defects that cause most production incidents.

Consistency at scale. Whether this diff matches how the rest of the codebase does things. A model that has read the whole repository is better at this than a reviewer who has read the parts they wrote.

The obvious security classes. Hardcoded credentials, injection-shaped string handling, missing authorisation checks on a new endpoint, dependencies with known advisories. Not a substitute for security review, but genuinely useful as a first pass.

Test gaps. Which branches the new code introduces and which of them nothing exercises. Mechanical, tedious, and something humans consistently skip.

Speed on large diffs. A 900-line pull request gets a complete read rather than the first two files and a rubber stamp. Every engineering team recognises that particular failure.

What they consistently miss

The gap matters more than the coverage, because it is where the false confidence lives.

Whether the change should exist. The most valuable review comment is often “we already do this elsewhere” or “this solves a problem we decided not to have”. That requires knowing the product, the roadmap and last quarter’s argument. A model reading a diff knows none of it.

Architectural consequence. Code that is locally correct and adds a dependency that will be painful in eighteen months. Correctness is visible in the diff. Consequence is not.

Domain rules. That this calculation must round in a particular direction because of a regulatory requirement, or that this field is nullable in theory and never in practice for reasons nobody wrote down. This is where automated reviewers are most confidently wrong.

Intent. Whether the code does what the ticket asked. A model can tell you the function is correct. It cannot tell you it is the wrong function.

Anything genuinely novel. Concurrency bugs specific to your runtime, subtle race conditions, performance characteristics that only appear at your data volumes.

There is also a category worth naming separately: plausible wrongness. A human reviewer who does not understand something says so. An automated reviewer produces a confident, well-formatted, entirely incorrect comment, and a junior engineer will act on it. This is the single largest risk in deploying one, and it is a process problem rather than a tooling problem.

Why teams abandon them

Almost always the same cause: signal-to-noise collapse.

The tool is switched on with default settings. It comments on everything. Style points the linter already covers, suggestions that ignore team conventions, restatements of what the code plainly does. Within a fortnight, engineers stop reading the comments, and once they stop reading them, the genuine catch in comment fourteen goes past unseen.

The failure is not that the tool was wrong. It is that it was right about too many things nobody needed telling.

A deployment that survives

1. 01Weeks 1 to 2Silent mode. It comments, nobody is required to act. Collect what it says and judge the quality honestly.
2. 02Week 3Cut hard. Turn off every category that duplicates the linter or produced a comment nobody acted on. Expect to disable most of it.
3. 03Week 4Teach it your conventions. Feed it the standards doc, the patterns you use and the ones you have deliberately rejected.
4. 04Weeks 5 to 6Set the rule: advisory only, never blocking, and a human still reviews every pull request. Then measure.

Four rules make the difference between a tool that lasts and one that gets muted.

Advisory, never blocking. The moment an automated comment can hold a merge, engineers optimise for satisfying the tool rather than for good code. It also means one confidently wrong comment can stop a release.

Never the only reviewer. A human still reads every pull request. The automated pass makes their read better by clearing the mechanical debris first, so their attention goes to design and intent.

Ruthless about categories. If a category has not produced a comment somebody acted on in a month, turn it off. A reviewer that raises three things and is right about three things gets read. One that raises thirty does not.

Juniors are told it can be wrong, explicitly. Say it out loud in onboarding. “This tool is confident and sometimes incorrect. If a comment does not make sense to you, ask a person before you change anything.” Without that sentence, junior engineers assume the machine knows better, and it frequently does not.

Measuring whether it is working

Most teams measure the wrong thing. Comments generated is a vanity metric, and a high number is usually a warning.

ActedShare of comments that led to an actual change. Below roughly a third and it is noise.

EscapedDefects reaching production that fall inside the tool's stated competence. The honest measure.

TimeHuman review time per pull request. It should fall. If it rises, the tool is generating work.

WrongIncorrect comments, logged by engineers in one shared place. Track the trend, not the total.

That last one requires a deliberately low-friction way to record it, a channel or a label, and a culture where flagging a bad comment is normal rather than a complaint. Without it you will never know how much plausible wrongness your team is quietly absorbing.

Choosing between tools

Less important than the deployment, but worth a short answer.

The meaningful differences are: how well the tool sees your whole repository rather than just the diff, whether it can be taught your conventions in a way that persists, how granular the category controls are, whether it integrates with the pull request flow you already have, and what its terms say about your code being retained or used for training.

That last one is not a footnote. Your source code going to a third party is a data question with contractual implications, and it belongs in your AI use register with a named owner, not in a developer’s browser tab.

Run any tool you are seriously considering against thirty of your own merged pull requests where you already know what the review found. That single afternoon tells you more than any comparison table, including this one.

The summary

AI code review is a good mechanical first pass and a poor final judgement. It catches the boring defects that cause most incidents and misses everything that requires knowing why your system is the way it is.

Deploy it as an advisory layer that clears debris before a human reads the diff, be aggressive about switching off categories that produce noise, tell your juniors out loud that it can be wrong, and measure the share of comments that lead to a change rather than the number of comments produced.

Done that way it gives a team back real time. Done by switching on the defaults and hoping, it becomes another thing people have learned to scroll past.

If you are working out where AI genuinely fits in your delivery process rather than where it can be bolted on, that assessment is what our AI readiness audit is for.