AI for Code Review: What It Catches and What It Misses

Every vendor page says the same thing: AI code review finds bugs. It does. The sentence is true and nearly useless, because it does not say which bugs, and it does not touch the thing that actually decides whether this works on your team — what happens when the bot is wrong, loudly, twelve times, on a forty-line pull request.

The Challenge

Code review is where engineering time goes to die. The PR sits, the reviewer context-switches, the author moves on, and reviews get rubber-stamped at 5pm on a Friday. The obvious move is to put a model on it.

But there are two problems here, and almost every article addresses only the first.

One: what can a reviewer that only sees a diff actually know? Structural. It does not go away with a better model.

Two: what happens when it is wrong? This is what kills rollouts, and nobody writes about it.

Problem two has a precise, published shape, worked out decades before LLMs existed. Google's static-analysis team (Sadowski et al., Communications of the ACM, 2018) gives us the metric:

We consider an issue to be an "effective false positive" if developers did not take positive action after seeing the issue.

Read that twice. A comment does not have to be wrong to be a false positive. It can be entirely correct and still count against the tool if the developer read it and did nothing — the paper is explicit that this includes true findings. And the judgment is not yours to make: "developers, not tool authors, will determine and act on a tool's perceived false-positive rate."

Google's budget: "analysis results shown during code review are allowed to include up to 10% effective false positives," and above that, "the Tricorder team disables the analyzer." Coverity's team, from twelve years of shipping a commercial analyzer (Bessey et al., CACM, 2010), reports the same physics and names the mechanism:

False positives do matter. In our experience, more than 30% easily cause problems. […] A vicious cycle starts where low trust causes complex bugs to be labeled false positives, leading to yet lower trust.

Plus the detail every team rolling out a bot should tattoo somewhere: "The initial reports matter inordinately; if the first N reports are false positives (N = 3?), people tend to utter variants on 'This tool sucks.'"

So the industry's own tool-builders converge on a tolerance budget of roughly 10-30% before engineers stop reading. Now measure the current generation against it.

A January 2026 study of LLM-based vulnerability detection at project scale ran five LLM detectors against real open-source projects and had humans hand-inspect 385 warnings over 150+ hours. Even the best-performing tool averaged an 85.3% false discovery rate; the worst hit 97.0%. Recall on C/C++: 21.09%. It misses most real bugs and the overwhelming majority of what it reports is wrong.

Atlassian's data is friendlier and still sobering — 4,000 LLM-generated review comments across 3,746 PRs in 1,007 repositories, from a tuned in-house reviewer:

Even the best category is ignored more often than acted on. Note which is worst — design. Hold that thought.

How AI Solves It

Start with the mechanism, because everything follows from it.

The reviewer sees a diff. It does not see the system.

A human reviewer's value is not that they are smarter than the model. It is that they know things that are not in the patch — the outage last March, the fact that retry_count is load-bearing for a customer whose name appears nowhere in the code, the fact that someone already wrote this function in another directory under another name. None of that is in the diff, so none of it is available to a reviewer whose entire world is the diff.

This is not hand-waving. It is the measured number-one cause of false positives: that January 2026 study categorized every false positive it found, and the largest single bucket — 37.47% — was shallow interprocedural reasoning, the model failing to trace data across the multi-hop paths a real program actually uses. It is the leading reason these tools are wrong, and it is not a defect in the model. It is the shape of the job.

What it reliably catches — everything local to the diff, where the bug and the evidence of the bug are both on screen:

• Null and undefined dereferences. The early return that skips initialization.
• Unhandled error paths. The promise with no .catch, the swallowed exception, the err returned and never checked. See error handling.
• Obvious injection. String-concatenated SQL, unescaped shell interpolation.
• Resource leaks. A file handle, socket or transaction opened on a path that can return early.
• Off-by-one errors in a visible loop, missing tests for a new branch, typos in strings and enum values — genuinely useful, genuinely boring.
• Style, naming and formatting. It is superb at this. That is precisely the problem.

These bugs reach production constantly, and a model catches them in seconds, at 3am, on every PR, without getting bored. That is worth money.

What it systematically misses — everything requiring knowledge the diff does not contain:

• Architectural violations. This module may not reach into that one. The rule lives in a design doc, or a staff engineer's head.
• Cross-file contract breaks. Behavior changed in a way that is fine here and wrong in the three places that call it.
• Correct in isolation, wrong in context. The most dangerous category, and the subject of Example 4 below.
• Business-logic errors. The code does exactly what it says. What it says is not what the business needs.
• Race conditions across modules, and "this is the third place we have implemented this."

Recall the Atlassian table: the category developers resolved least often was code design (28.6%) — the closest thing to architectural judgment. The model is least persuasive exactly where the diff tells it least.

Repository-indexing tools — Greptile builds a call graph of the whole repo, CodeRabbit clones it into a sandbox, Claude Code reads your working tree — push this boundary outward. But the boundary moves; it does not disappear. No index contains intent that was never written down, and most of what a senior reviewer knows was never written down.

So: let the model take the local, mechanical, tedious findings — the ones humans are worst at, because humans get bored — and let the human spend their attention on what only they can see. The bot handles the null check. The human handles "why are we doing this at all."

Recommended Tools & Models

Every tool here will find a null dereference. They differ in how much they see, and how much control you have over what they say.

Three things you will not find on a comparison page.

Amazon Q Developer is being retired. New signups closed May 2026, support ends April 2027, AWS points users to Kiro. Its marketing pages have not caught up. Do not start a rollout there.

Be skeptical of every vendor accuracy stat, because they measure a different thing. Vendor benchmarks typically score tools on PRs known to contain a bug — which makes noise free in the scoring, since a tool that comments on everything scores well. Greptile's own published benchmark explicitly excludes false positives from scoring; Cursor reports that 70–80% of flags get resolved before merge but publishes no false-positive rate — and "resolved" is not "was a real bug." And benchmarks inflate badly: on PrimeVul (ICSE 2025), a de-duplicated, correctly-labeled dataset, a model scoring 68.26% F1 on the older BigVul dropped to 3.09%. The benchmark was doing the work, not the model.

The best design idea in the field is to make the bot try to prove the bug. Claude Code's /code-review ultra runs a verification pass and labels each finding CONFIRMED or PLAUSIBLE, so you can triage by whether it actually reproduced. That is not the same as suppressing the unproven ones — plausible findings still reach you — but it puts the tool's own uncertainty on the page, which is more than any other reviewer here does. We covered the multi-agent version in Claude Code Introduces /ultrareview.

On models: Claude Sonnet 5 and Claude Opus 4.8 follow negative instructions well, which matters enormously — most of your prompt will be a list of things not to say. Gemini 3.5 has the context window for large diffs plus surrounding files. Claude Haiku 4.5 is cheap enough to run on every commit. The model is not your bottleneck. The scope you give it is.

Step-by-Step Implementation

1. Decide what the bot is for before you install it

Every tool defaults to "comment on everything," and that default is what gets it muted. Pick three to five categories the bot owns — null derefs and unhandled errors, resource leaks, injection, missing tests, and things the linter flags but people skip. That set is not arbitrary: it is exactly what is local to a diff.

Then write the "not yours" list, which is longer and matters more: naming, formatting, subjective structure, "consider extracting this," documentation style, praise, and summaries of what the diff does. Your team can read.

2. Wire it into CI without blocking merges

Run on pull request, advisory only — not a required status check. Do not fail the build on a finding. Never let it request changes automatically.

A blocking gate whose comments are acted on a third of the time does not raise your quality bar. It teaches engineers that the correct response to this bot is to click override — and once overriding is muscle memory, you have something worse than no gate: a gate everyone has been trained to ignore, including on the day it is right. Earn the right to block later, per category, with data.

3. Scope the review to the changed lines

The highest-leverage setting, and usually off by default. A reviewer commenting on pre-existing code is describing a decision made two years ago by someone who has left. Not wrong; not actionable; and it arrives in bulk on any PR touching a large old file. Cloudflare's production reviewer instructs its agents to ignore "issues in unchanged code that this MR doesn't affect."

4. Write the review prompt

This is where the noise problem is actually solved. Notice how much of it is about staying quiet.

No issues found within review scope.

Three parts of that prompt do the real work.

The suppression list, longer than the report list, deliberately. Most of a good reviewer's job is not saying things.

The senior-engineer filter. "Would a senior engineer say this out loud, unprompted, to a colleague they respect?" This one sentence removes more noise than any confidence threshold, because it encodes the social cost of a comment — the thing the model has no other way to represent. A staff engineer does not leave twelve naming comments on a colleague's forty-line PR; not because they did not notice, but because they know what it would cost. Same move as Garry Tan's Claude Code senior engineer prompt: telling the model who it is changes what it says.

The permission to find nothing. Not a nicety — it fixes a measured pathology. In the Atlassian study, on open-source projects, LLM reviewers returned a "no issue" verdict 0.6% of the time. Human reviewers did so 9.0% of the time. The model is roughly fifteen times less willing to say "this looks fine" than a person, because finding something looks like doing the job. Blessing "No issues found" as a success is what stops the bot manufacturing a finding on a clean PR. If you take one line from this guide, take that one.

5. Turn the volume down at the tool level too

In CodeRabbit, the .coderabbit.yaml keys that matter:

reviews:
  profile: chill          # quiet | chill | assertive. "assertive" is the nitpicky one.
  path_filters:
    - "!**/*.lock"
    - "!dist/**"
    - "!**/generated/**"
    - "!**/*.snap"
  path_instructions:
    - path: "tests/**"
      instructions: "Only flag tests that cannot fail. Ignore style entirely."

Elsewhere: strictness 1-3 in Greptile — which also learns from your downvotes, blocking a new comment when it is semantically close to at least three you have already rejected — BUGBOT.md in Cursor, custom instructions files for GitHub Copilot. Know the gap there: Copilot code review has no severity or confidence threshold at all, and instructions cannot suppress comment types. GitHub's own documentation says it "has a risk of hallucination — it may highlight problems in reviewed code that do not exist."

Exclude generated files, lock files and snapshots before the first run, not after the bot has buried a PR under two hundred comments about package-lock.json. Remember Coverity: if the first three reports are false positives, the tool is already dead. You get one first impression.

6. Reviewing AI-generated code — increasingly the actual job

GitClear's analysis of 623 million code changes (2023-2026) reports code duplication up 81%, code reuse down 70%, and legacy refactoring down 74%. (Vendor research, not peer-reviewed, but the largest dataset published.) Their CEO's summary is the practical version: "you have five different implementations of the same thing that are similar yet different." The 2025 DORA report points the same way: AI adoption now correlates with higher throughput but still correlates negatively with delivery stability.

Now see the trap. The dominant defect in AI-generated code is duplication — and duplication is precisely what a diff-scoped reviewer cannot see. "We already have this function, two directories over" is a cross-file finding. The bot is weakest exactly where the new code is weakest.

So reviewing AI-generated code requires a reviewer with repository context — the clearest case for paying for indexing. Add an explicit instruction: "Before approving any new function, search the repository for an existing function that does the same thing. If one exists, name it and stop."

A 2026 participatory-design study of practitioners reviewing multi-file LLM-generated changes frames trust calibration, not diffing, as the central challenge — and names the failure mode: "systematic misallocation of review effort toward low-risk segments and away from high-risk ones." You skim the boring 400 lines and miss the dangerous 3. For the generation side, see Harper Reed's LLM codegen workflow and the agentic coding loop. If the code arrived via vibe coding, assume no human has read any of it.

7. What to never let it auto-fix

Auto-fix is safe only where a wrong fix fails loudly and immediately.

Safe: formatting, import order, unused variables, obvious typos, lint violations. Get these wrong and the build breaks or a test goes red — you know in ninety seconds.

Never: security logic, authentication and authorization; concurrency and locking; database migrations; money arithmetic and rounding; cryptography, under any circumstances; and any file the model has not read in full.

The common thread is that a wrong fix in any of these is silent. It merges green, and you find it in an incident review. Model confidence is uncorrelated with correctness — see hallucinations.

8. Measure the action rate and prune, monthly

Google disables an analyzer that goes above a 10% effective-false-positive rate — and disables it until the owner fixes it, which is the part everyone forgets. Do the same: for each category of comment, count how many led to a code change. Anything under the bar gets switched off first and argued about afterwards. The point is that a category has to earn its place back, not that you delete it in anger.

Real-World Examples

Example 1: A vendor audits its own bot and finds 79% nits

The most honest published data in this field comes from Greptile, about Greptile.

Before. In December 2024 they audited their own review comments:

We analyzed existing Greptile comments and found that ~19% were good, 2% were flat-out incorrect, and 79% were nits.

Read that against the Coverity budget. Only 2% were wrong — the model was accurate. And 79% were noise, which under Google's definition makes them false positives all the same. Accuracy was never the problem.

After. They rebuilt around suppressing nits rather than finding more bugs, and reported their address rate — the share of comments developers act on before merge — going from 19% to 55%+.

What changed: not the model, not the bug-finding. They made it shut up, and the address rate nearly tripled because what remained was worth reading. (The obvious objection is that this metric would also go up if the bot left almost no comments at all. Correct — which is why you track action rate and escaped defects, never either alone.)

Example 2: Cloudflare, at production scale

The best operational dataset anyone has published. Over 30 days, Cloudflare's internal AI reviewer ran 131,246 reviews across 48,095 merge requests in 5,169 repositories.

The naive version — one general-purpose reviewer, told to review the diff — produces exactly the noise this guide is about. What they built instead:

1. Seven specialist agents rather than one generalist: code quality, security, performance, documentation, release management, an Engineering Codex agent for internal compliance, and an AGENTS.md reviewer. A coordinator sits above them and decides what survives.
2. A "What NOT to Flag" section in every prompt. Their security reviewer is told to ignore "theoretical risks that require unlikely preconditions," "defense-in-depth suggestions when primary defenses are adequate," and "'consider using library X' style suggestions."
3. A coordinator that filters before anything is posted — dedupe, re-categorize, then a "reasonableness filter: speculative issues, nitpicks, false positives, and convention-contradicted findings get dropped."

Result: 159,103 findings across 131,246 reviews — about 1.2 findings per review. Median 3 minutes 39 seconds, average cost $1.19.

1.2 findings per review. That is the number to aim at. Not twelve. Throwing away the model's own output before a human ever sees it is doing most of the work here, and it is the step nearly every team skips.

And note what Cloudflare then permits itself, having done that work: their rubric routes a critical item or a production safety risk to a verdict that blocks the merge. That is not a contradiction of the advisory rule below — it is the rule's endpoint. You start advisory because you have not earned anything yet. Cloudflare blocks on two narrow categories, after a coordinator has already discarded most of what seven agents produced. Earn it the same way.

Example 3: The bug it caught, exactly as advertised

The diff: a new early-return guard in a request handler.

def handle(req):
    conn = pool.acquire()
    if not req.user:
        return Error(401)        # the new line
    data = conn.query(req.id)
    conn.release()
    return data

The comment: "HIGH: conn is acquired on line 2 and released only on the success path. The new early return on line 4 leaks a pooled connection on every unauthenticated request. Under load this exhausts the pool."

Local to the diff, mechanically verifiable, names the triggering input and the runtime consequence. A tired human misses this constantly — precisely because the new line looks obviously correct. The bug is not in the line that changed; it is in the interaction between the line that changed and a line that did not. This is what you are buying.

Example 4: The bug it missed, and could never have caught

The diff: one line, in a utility function.

# before
def parse_ts(s): return datetime.fromisoformat(s)

# after
def parse_ts(s): return datetime.fromisoformat(s).replace(tzinfo=timezone.utc)

Every reviewer, human and machine, approved it. It is a strictly better function. It fixes a real naive-datetime bug. The author was right.

What broke: one of eleven callers, in another service, was already localizing the timestamp itself. It now double-applied the offset. Billing events for European customers landed an hour off for four days.

Why no AI reviewer catches this. The bug is not in the diff — the diff is correct. It exists only in the relationship between this change and a caller that is not on screen, whose behavior depends on the old contract. A repo-indexing reviewer could in principle have found that caller. But it would still have had to understand that the caller's timezone handling was deliberate rather than redundant, and that fact appears nowhere in the code. It was a decision, made in a meeting, in 2023.

What would have caught it: the reviewer who was in that meeting. The model can only reason about what is written down. The most expensive things your team knows were never written down.

Industry-Specific Applications

Open source. The best fit, and the economics are unbeatable — CodeRabbit is free forever on public repos, Greptile free for qualifying MIT/Apache projects. Two cautions: tone (a first-time contributor who receives nine automated nitpicks does not come back), and the closed loop maintainers already report — contributors having an agent answer the bot on their behalf, so two models negotiate a merge and no human reads anything.

Fintech and regulated environments. Auto-fix off, permanently. Money arithmetic and audit trails are where a silent wrong fix is unrecoverable.

Security-critical code. Calibrate hard. That January 2026 study found an 85–97% false discovery rate and 21% recall on real C/C++ projects, and LLM security reasoning is not robust: on the SecLLMHolmes benchmark (IEEE Symposium on Security and Privacy, 2024), merely renaming functions or variables flipped the evaluated models to an incorrect answer 26% of the time — and adding library functions did so in another 17%. The code did not change. The name did. AI review supplements a security program; it is not one. It will catch the string-concatenated SQL. It will not catch the auth bypass spanning four files.

Monorepos. The diff-versus-system gap is widest here, and repository indexing stops being optional. The review a payments directory needs is not the review a frontend directory needs; path_instructions is how you say so.

Startups shipping AI-generated code fast. You need this most and can afford the noise least. Start with the narrowest scope and expand only when the action rate earns it.

Best Practices

• Advisory, never blocking. Earn the right to block, per category, with data.
• Restrict to changed lines. Highest-leverage setting, usually off by default.
• Write the "do not comment on" list first. It is longer, and it is what saves the rollout.
• Apply the senior-engineer filter. If they would not say it out loud, silence.
• Bless "no issues found" explicitly. The model is ~15x less willing to say "looks fine" than a human.
• Filter the model's output before a human sees it. Cloudflare's coordinator step is the highest-value architecture in this guide.
• Demand a triggering path, a consequence, and a file:line citation. "This could be a problem" is not a review.
• Suppress low-confidence findings. Do not display them and expect humans to filter — humans will not filter, they will disengage.
• Never auto-fix anything that fails silently. Crypto, auth, concurrency, migrations, money.
• Get the first three comments right. "If the first N reports are false positives (N = 3?), people tend to utter variants on 'This tool sucks.'"

Common Pitfalls

Death by nitpick. The defining failure. Twelve comments, one bug, nobody reads any of them. The cruelty is that the bot was right — it found the leak — and still failed, because it had already spent the reader's attention on resp versus response. Noise does not dilute signal. It destroys it.

Treating "technically correct" as "successful." If the developer took no action, it was an effective false positive regardless of truth. Most teams never make this shift, and it is the whole game.

The trust death spiral. Coverity named it in 2010 and LLMs make it worse: "a vicious cycle starts where low trust causes complex bugs to be labeled false positives, leading to yet lower trust." Once your team believes the bot is noisy, they start dismissing its true findings as noise too — damage not recoverable by improving the tool, because nobody is reading it any more.

Trusting a green review on a contract change. A clean bot review means it found no problem in the diff. It does not mean there is no problem.

Believing benchmark numbers. Vendors benchmark on PRs known to contain bugs, which makes noise free in the scoring. And benchmarks inflate — 68.26% F1 to 3.09% once the data was cleaned. Yours is the only number about you.

Assuming a quiet bot is a broken bot. A well-scoped reviewer says nothing on most PRs; Cloudflare runs at 1.2 findings per review. That is the system working. "Getting more value" by loosening the scope is exactly how it gets muted again.

Measuring Success

One metric matters most, and it is not "bugs found." Action rate: the share of bot comments that result in a code change. It is the inverse of Google's effective false-positive rate, and it predicts whether anyone will still be reading the bot next quarter.

If you are under 20%, your bot is on its way to being muted whether or not anyone has said so out loud.

Then, in order:

• Findings per review. Cloudflare runs at ~1.2. If you are at 10, you do not have a code reviewer, you have a linter with a thesaurus.
• Action rate by category. What you prune on. It is normal for "resource leaks" to run at 70% and "suggest refactor" at 2%. Delete the second.
• Escaped defects in the categories you assigned it. Did a null deref reach production? That is the bot's job, and it failed. Escaped bugs outside its scope are not its failure — do not let anyone score it that way, and do not let anyone believe it was covering them.
• Whether anyone still reads it. In month three, ask the team whether they read the bot's comments or collapse them. The answer beats any dashboard.

Cost Analysis

Tools, for a 10-engineer team:

• CodeRabbit Pro: $24/dev/month annual — $240/month. Free forever on public repos.
• Greptile Pro: $30/seat/month — $300/month, plus $1 per review beyond the included 50.
• GitHub Copilot Pro: $10/user/month — $100/month, bundled into an assistant you may already pay for. Often the cheapest "good enough."
• Cursor Bugbot: usage-based since June 2026. Cursor's published figure is $1.00–$1.50 for an average run, varying with PR size and complexity; higher effort levels cost more and no cap is published. On 200 PRs a month that is roughly $200–$300 — but meter your own first month, because the variance is in your diffs, not in the price.
• Roll your own with Claude Code or an API script in CI: token cost only. Cloudflare's production reviewer averages $1.19 per review — the best public benchmark for what this costs at scale.

So: $100 to $300 a month for ten engineers. Trivial against one salary. The money is not the cost.

The real cost is attention, and nobody prices it. Ten engineers reading four junk comments per PR across 200 PRs a month is 8,000 comment-reads, most worthless. At ten seconds each that is over 20 hours a month — more than the tool costs. And the hours are not the harm. The harm is that after a few weeks they stop reading, and then the connection leak in comment eleven ships.

Setup: a day to wire in, two to four weeks to tune the noise down — the actual work, and the step every team skips.

What it buys you: a tireless reviewer that never misses a null dereference and never gets bored at 5pm on a Friday. What it does not buy you: anyone who knows why the code exists.

Related Tools

• Claude Code — /review and /code-review ultra, with a reproduction step before reporting
• GitHub Copilot — PR review bundled into a plan most teams already have
• Cursor — Bugbot, usage-priced, with repo-level rules
• Windsurf, Cline, Aider — agentic editors you can point at a diff with your own review prompt
• Ollama / LM Studio — local models, for code that may not leave your network

Related Models

• Claude Sonnet 5 — follows negative instructions well, which is most of a review prompt
• Claude Opus 4.8 — deepest reasoning for high-stakes diffs
• Claude Haiku 4.5 — cheap enough to run on every commit
• GPT-5.5 — strong generalist reviewer
• Gemini 3.5 — large context for big diffs plus surrounding files
• Composer — the model behind Cursor's agentic workflows

New to this? Start with prompt engineering — on this use case the prompt is not a detail, it is the product. The difference between a bot your team reads and a bot your team mutes is a suppression list and one sentence about what a senior engineer would actually bother to say out loud.