---
source: 'https://howaiworks.ai/use-cases/ai-for-code-review'
section: use-cases
title: 'AI for Code Review: What It Catches and What It Misses'
description: >-
  What AI code review reliably catches, what it structurally cannot, and how to
  stop the false positives that get the bot muted within two weeks.
category: Engineering & Design
subcategory: Pull Request Review
targetAudience:
  - Software Engineers
  - Engineering Managers
  - Tech Leads
  - Platform Teams
toolsUsed:
  - CodeRabbit
  - Greptile
  - GitHub Copilot
  - Cursor
  - Claude Code
modelTypes:
  - Claude Sonnet 5
  - Claude Opus 5
  - GPT-6 Astra
  - Gemini 3.8 Flash
difficulty: Intermediate
industries:
  - Software
  - SaaS
  - Fintech
  - Open Source
roles:
  - Software Engineer
  - Engineering Manager
  - Tech Lead
  - Staff Engineer
estimatedTimeToImplement: '1 day to wire in, 2-4 weeks to tune the noise down'
potentialROI: >-
  Not a throughput number. The only honest metric is the share of bot comments
  that change code: Atlassian measured 28.6-43.3% by category on 4,000 LLM
  review comments, and Greptile raised its own from 19% to 55%+ by suppressing
  nits
tags:
  - Code Review
  - Developer Tools
  - Software Engineering
  - false positives
  - pull requests
  - static analysis
  - CI/CD
  - Prompt Engineering
relatedUseCases:
  - ai-for-support-ticket-triage
  - ai-for-ui-mockups
lastUpdated: '2026-09-06'
readingTime: 13 minutes
---

# AI for Code Review: What It Catches and What It Misses

> What AI code review reliably catches, what it structurally cannot, and how to stop the false positives that get the bot muted within two weeks.

Every vendor page says the same thing: AI code review finds bugs. It does. The sentence is true and nearly useless, because it does not say *which* bugs, and it does not touch the thing that actually decides whether this works on your team — **what happens when the bot is wrong, loudly, twelve times, on a forty-line pull request.**

## The Challenge

Code review is where engineering time goes to die. The PR sits, the reviewer context-switches, the author moves on, and reviews get rubber-stamped at 5pm on a Friday. The obvious move is to put a model on it.

But there are two problems here, and almost every article addresses only the first.

**One: what can a reviewer that only sees a diff actually know?** Structural. It does not go away with a better model.

**Two: what happens when it is wrong?** This is what kills rollouts, and nobody writes about it.

Problem two has a precise, published shape, worked out decades before LLMs existed. Google's static-analysis team (Sadowski et al., *Communications of the ACM*, 2018) gives us the metric:

> We consider an issue to be an "effective false positive" if developers did not take positive action after seeing the issue.

Read that twice. **A comment does not have to be wrong to be a false positive.** It can be entirely correct and still count against the tool if the developer read it and did nothing — the paper is explicit that this includes true findings. And the judgment is not yours to make: *"developers, not tool authors, will determine and act on a tool's perceived false-positive rate."*

Google's budget: *"analysis results shown during code review are allowed to include up to 10% effective false positives,"* and above that, *"the Tricorder team disables the analyzer."* Coverity's team, from twelve years of shipping a commercial analyzer (Bessey et al., *CACM*, 2010), reports the same physics and names the mechanism:

> False positives do matter. In our experience, more than 30% easily cause problems. […] A vicious cycle starts where low trust causes complex bugs to be labeled false positives, leading to yet lower trust.

Plus the detail every team rolling out a bot should tattoo somewhere: *"The initial reports matter inordinately; if the first N reports are false positives (N = 3?), people tend to utter variants on 'This tool sucks.'"*

So the industry's own tool-builders converge on a tolerance budget of roughly **10-30% before engineers stop reading.** Now measure the current generation against it.

A January 2026 study of LLM-based vulnerability detection at project scale ran five LLM detectors against real open-source projects and had humans hand-inspect **385 warnings over 150+ hours**. Even the **best-performing** tool averaged an **85.3% false discovery rate**; the worst hit **97.0%**. Recall on C/C++: **21.09%**. It misses most real bugs *and* the overwhelming majority of what it reports is wrong.

Atlassian's data is friendlier and still sobering — **4,000 LLM-generated review comments across 3,746 PRs in 1,007 repositories**, from a tuned in-house reviewer:

| Comment type | Resolved by developers |
|---|---|
| Code readability | 43.3% |
| Code bugs | 41.9% |
| Maintainability | 36.2% |
| **Code design** | **28.6%** |

Even the best category is ignored more often than acted on. Note which is worst — design. Hold that thought.

## How AI Solves It

Start with the mechanism, because everything follows from it.

> **The reviewer sees a diff. It does not see the system.**

A human reviewer's value is not that they are smarter than the model. It is that **they know things that are not in the patch** — the outage last March, the fact that `retry_count` is load-bearing for a customer whose name appears nowhere in the code, the fact that someone already wrote this function in another directory under another name. None of that is in the diff, so none of it is available to a reviewer whose entire world is the diff.

This is not hand-waving. It is the **measured** number-one cause of false positives: that January 2026 study categorized every false positive it found, and the largest single bucket — **37.47%** — was *shallow interprocedural reasoning*, the model failing to trace data across the multi-hop paths a real program actually uses. It is the leading reason these tools are wrong, and it is not a defect in the model. It is the shape of the job.

**What it reliably catches** — everything local to the diff, where the bug and the evidence of the bug are both on screen:

- **Null and undefined dereferences.** The early return that skips initialization.
- **Unhandled error paths.** The promise with no `.catch`, the swallowed exception, the `err` returned and never checked. See [error handling](https://howaiworks.ai/glossary/error-handling).
- **Obvious injection.** String-concatenated SQL, unescaped shell interpolation.
- **Resource leaks.** A file handle, socket or transaction opened on a path that can return early.
- **Off-by-one errors in a visible loop**, missing tests for a new branch, typos in strings and enum values — genuinely useful, genuinely boring.
- **Style, naming and formatting.** It is superb at this. That is precisely the problem.

These bugs reach production constantly, and a model catches them in seconds, at 3am, on every PR, without getting bored. That is worth money.

**What it systematically misses** — everything requiring knowledge the diff does not contain:

- **Architectural violations.** This module may not reach into that one. The rule lives in a design doc, or a staff engineer's head.
- **Cross-file contract breaks.** Behavior changed in a way that is fine here and wrong in the three places that call it.
- **Correct in isolation, wrong in context.** The most dangerous category, and the subject of Example 4 below.
- **Business-logic errors.** The code does exactly what it says. What it says is not what the business needs.
- **Race conditions across modules**, and **"this is the third place we have implemented this."**

Recall the Atlassian table: the category developers resolved *least* often was **code design (28.6%)** — the closest thing to architectural judgment. The model is least persuasive exactly where the diff tells it least.

Repository-indexing tools — [Greptile](https://www.greptile.com) builds a call graph of the whole repo, CodeRabbit clones it into a sandbox, [Claude Code](https://howaiworks.ai/ai-tools/claude-code) reads your working tree — push this boundary outward. But the boundary **moves; it does not disappear.** No index contains intent that was never written down, and most of what a senior reviewer knows was never written down.

So: **let the model take the local, mechanical, tedious findings — the ones humans are worst at, because humans get bored — and let the human spend their attention on what only they can see.** The bot handles the null check. The human handles "why are we doing this at all."

## Recommended Tools & Models

Every tool here will find a null dereference. They differ in how much they *see*, and how much control you have over what they *say*.

| Tool | What it sees | Noise controls | Price (July 2026) |
|---|---|---|---|
| **[CodeRabbit](https://www.coderabbit.ai)** | Clones the full repo; fuses 50+ linters/SAST (Semgrep, ast-grep, TruffleHog) under the LLM | Strongest: `profile: quiet/chill/assertive`, `path_filters`, `path_instructions`, AST rules, natural-language "learnings" | Pro $24/dev/mo annual; Pro+ $48. **Free forever on public repos** |
| **[Greptile](https://www.greptile.com)** | Graph index of the whole repository | `strictness` 1-3; **auto-suppresses a comment type after repeated downvotes** | Pro $30/seat/mo; free tier for 1 dev; free for MIT/Apache OSS |
| **[GitHub Copilot](https://howaiworks.ai/ai-tools/github-copilot) review** | Diff, plus live grep. No static-analysis layer | Custom instructions only. **No severity or confidence threshold exists** | Included from Copilot Pro ($10/mo) up. **Not in Free** |
| **[Cursor](https://howaiworks.ai/ai-tools/cursor) Bugbot** | Full PR diff on every push | `BUGBOT.md` rules; effort level | **Usage-based since June 2026.** Cursor's own figure: **$1.00–$1.50 per run**, scaling with PR size; high-effort reviews cost more, and there is no published cap. Meter your first month |
| **[Claude Code](https://howaiworks.ai/ai-tools/claude-code)** `/review`, `/code-review ultra` | Your full local repo; `/code-review ultra` runs a **verification pass** and labels each finding CONFIRMED or PLAUSIBLE | Your own prompt — total control | Token cost |
| **Anthropic Code Review** (research preview) | Parallel specialist agents + verification against actual code behavior | `REVIEW.md`: cap nit count, define severity per repo, suppress new nits on re-review | ~$15-25 per review; Team/Enterprise |
| **Qodo / PR-Agent** | Diff with retrieval | Custom rules | Pro Team from ~$30 + credits. OSS `pr-agent` is MIT |

Three things you will not find on a comparison page.

**Amazon Q Developer is being retired.** New signups closed May 2026, support ends April 2027, AWS points users to Kiro. Its marketing pages have not caught up. Do not start a rollout there.

**Be skeptical of every vendor accuracy stat, because they measure a different thing.** Vendor benchmarks typically score tools on PRs *known to contain a bug* — which makes **noise free in the scoring**, since a tool that comments on everything scores well. Greptile's own published benchmark explicitly excludes false positives from scoring; Cursor reports that 70–80% of flags get resolved before merge but publishes no false-positive rate — and "resolved" is not "was a real bug." And benchmarks inflate badly: on **PrimeVul** (ICSE 2025), a de-duplicated, correctly-labeled dataset, a model scoring **68.26% F1** on the older BigVul dropped to **3.09%**. The benchmark was doing the work, not the model.

**The best design idea in the field is to make the bot try to prove the bug.** [Claude Code](https://howaiworks.ai/ai-tools/claude-code)'s `/code-review ultra` runs a verification pass and labels each finding **CONFIRMED** or **PLAUSIBLE**, so you can triage by whether it actually reproduced. That is not the same as suppressing the unproven ones — plausible findings still reach you — but it puts the tool's own uncertainty on the page, which is more than any other reviewer here does. We covered the multi-agent version in [Claude Code Introduces /ultrareview](https://howaiworks.ai/blog/claude-code-ultrareview-agentic-code-analysis).

On models: [Claude Sonnet 5](https://howaiworks.ai/models/claude-sonnet) and [Claude Opus 5](https://howaiworks.ai/models/claude-opus) follow *negative* instructions well, which matters enormously — most of your prompt will be a list of things **not** to say. [Gemini 3.8 Flash](https://howaiworks.ai/models/gemini) has the [context window](https://howaiworks.ai/glossary/context-window) for large diffs plus surrounding files. [Claude Haiku 4.5](https://howaiworks.ai/models/claude-haiku) is cheap enough to run on every commit. The model is not your bottleneck. **The scope you give it is.**

## Step-by-Step Implementation

### 1. Decide what the bot is for before you install it

Every tool defaults to "comment on everything," and that default is what gets it muted. **Pick three to five categories the bot owns** — null derefs and unhandled errors, resource leaks, injection, missing tests, and things the linter flags but people skip. That set is not arbitrary: it is exactly what is local to a diff.

**Then write the "not yours" list**, which is longer and matters more: naming, formatting, subjective structure, "consider extracting this," documentation style, praise, and summaries of what the diff does. Your team can read.

### 2. Wire it into CI without blocking merges

Run on pull request, **advisory only** — not a required status check. Do not fail the build on a finding. Never let it request changes automatically.

A blocking gate whose comments are acted on a third of the time does not raise your quality bar. It teaches engineers that the correct response to this bot is to click override — and once overriding is muscle memory, you have something worse than no gate: **a gate everyone has been trained to ignore, including on the day it is right.** Earn the right to block later, per category, with data.

### 3. Scope the review to the changed lines

The highest-leverage setting, and usually off by default. A reviewer commenting on pre-existing code is describing a decision made two years ago by someone who has left. Not wrong; not actionable; and it arrives in bulk on any PR touching a large old file. Cloudflare's production reviewer instructs its agents to ignore *"issues in unchanged code that this MR doesn't affect."*

### 4. Write the review prompt

This is where the noise problem is actually solved. Notice how much of it is about staying quiet.

**Prompt template — Scoped Code Review Prompt (noise-suppressing)**

````text
You are reviewing a pull request. You can see the diff below and you can read files in
the repository, but you cannot see: the design discussion, the incident history, the
product requirements, or what the author was actually asked to build.

Review accordingly. Comment only on what is verifiable from the code.

**REPORT ONLY THESE. Nothing else is in scope:**

1. Null / undefined dereference reachable from a code path in this diff
2. Unhandled error or rejected promise
3. A resource opened and not released on some path (file, socket, transaction, lock)
4. Injection: unparameterized SQL, unescaped shell or template interpolation
5. A new branch with no test covering it
6. A change to a function's contract (signature, return shape, thrown errors,
   nullability) where a CALLER OUTSIDE THIS DIFF relies on the old behavior — you must
   open the caller and cite file and line

**DO NOT COMMENT ON — this list is binding:**

- Naming, formatting, style, import order
- "Consider extracting this into a helper"
- Anything the linter or formatter already handles
- Pre-existing code this diff does not modify
- Missing docstrings or comments
- Theoretical risks requiring unlikely preconditions
- Defense-in-depth suggestions where the primary defense is adequate
- Praise, or a summary of what the diff does. I wrote it. I know.

**THE BAR — apply to every comment before you write it:**

Would a senior engineer say this out loud, unprompted, in a review, to a colleague they
respect? If it is a nitpick they would let slide, or a preference rather than a defect —
SAY NOTHING. Silence is a valid and frequently correct review.

**EVERY COMMENT MUST INCLUDE:**

- The specific input or code path that triggers the problem
- The consequence: what actually breaks, at runtime
- A confidence: HIGH (I can name the failing path) / MEDIUM (likely, unverified) /
  LOW (a hunch)

**Suppress every LOW-confidence finding entirely. Do not report it.**

A behavioral claim needs a file:line citation, not an inference from a function's name.

If you find nothing meeting this bar, reply with exactly:

```
No issues found within review scope.
```

That is a successful review, not a failed one. Do not manufacture a finding to appear
useful.

**DIFF:**
[PASTE THE DIFF]
````

Use as a custom instruction file (CodeRabbit path_instructions, Copilot custom instructions, Cursor BUGBOT.md, Anthropic REVIEW.md) or directly with Claude Code. The suppression list is not garnish — it is what makes the bot survive its first month.

Three parts of that prompt do the real work.

**The suppression list**, longer than the report list, deliberately. Most of a good reviewer's job is not saying things.

**The senior-engineer filter.** *"Would a senior engineer say this out loud, unprompted, to a colleague they respect?"* This one sentence removes more noise than any confidence threshold, because it encodes the **social cost of a comment** — the thing the model has no other way to represent. A staff engineer does not leave twelve naming comments on a colleague's forty-line PR; not because they did not notice, but because they know what it would cost. Same move as [Garry Tan's Claude Code senior engineer prompt](https://howaiworks.ai/blog/garry-tan-claude-code-senior-engineer-prompt): telling the model who it is changes what it says.

**The permission to find nothing.** Not a nicety — it fixes a measured pathology. In the Atlassian study, on open-source projects, LLM reviewers returned a "no issue" verdict **0.6% of the time. Human reviewers did so 9.0% of the time.** The model is roughly **fifteen times less willing to say "this looks fine"** than a person, because finding something looks like doing the job. Blessing "No issues found" as a success is what stops the bot manufacturing a finding on a clean PR. If you take one line from this guide, take that one.

### 5. Turn the volume down at the tool level too

In [CodeRabbit](https://www.coderabbit.ai), the `.coderabbit.yaml` keys that matter:

```yaml
reviews:
  profile: chill          # quiet | chill | assertive. "assertive" is the nitpicky one.
  path_filters:
    - "!**/*.lock"
    - "!dist/**"
    - "!**/generated/**"
    - "!**/*.snap"
  path_instructions:
    - path: "tests/**"
      instructions: "Only flag tests that cannot fail. Ignore style entirely."
```

Elsewhere: **`strictness` 1-3** in Greptile — which also learns from your downvotes, blocking a new comment when it is semantically close to at least three you have already rejected — **`BUGBOT.md`** in [Cursor](https://howaiworks.ai/ai-tools/cursor), custom instructions files for [GitHub Copilot](https://howaiworks.ai/ai-tools/github-copilot). Know the gap there: **Copilot code review has no severity or confidence threshold at all**, and instructions cannot suppress comment types. GitHub's own documentation says it *"has a risk of hallucination — it may highlight problems in reviewed code that do not exist."*

Exclude generated files, lock files and snapshots **before** the first run, not after the bot has buried a PR under two hundred comments about `package-lock.json`. Remember Coverity: if the first three reports are false positives, the tool is already dead. **You get one first impression.**

### 6. Reviewing AI-generated code — increasingly the actual job

GitClear's analysis of **623 million code changes (2023-2026)** reports code duplication **up 81%**, code reuse **down 70%**, and legacy refactoring **down 74%**. (Vendor research, not peer-reviewed, but the largest dataset published.) Their CEO's summary is the practical version: *"you have five different implementations of the same thing that are similar yet different."* The 2025 DORA report points the same way: AI adoption now correlates with higher throughput but **still correlates negatively with delivery stability.**

Now see the trap. **The dominant defect in AI-generated code is duplication — and duplication is precisely what a diff-scoped reviewer cannot see.** "We already have this function, two directories over" is a cross-file finding. **The bot is weakest exactly where the new code is weakest.**

So reviewing AI-generated code requires a reviewer with **repository context** — the clearest case for paying for indexing. Add an explicit instruction: *"Before approving any new function, search the repository for an existing function that does the same thing. If one exists, name it and stop."*

A 2026 participatory-design study of practitioners reviewing multi-file LLM-generated changes frames trust calibration, not diffing, as the central challenge — and names the failure mode: *"systematic misallocation of review effort toward low-risk segments and away from high-risk ones."* You skim the boring 400 lines and miss the dangerous 3. For the generation side, see [Harper Reed's LLM codegen workflow](https://howaiworks.ai/blog/harper-reed-llm-codegen-workflow) and the [agentic coding loop](https://howaiworks.ai/blog/geoffrey-huntley-ralph-agentic-coding-loop). If the code arrived via [vibe coding](https://howaiworks.ai/glossary/vibe-coding), assume no human has read any of it.

### 7. What to never let it auto-fix

> **Auto-fix is safe only where a wrong fix fails loudly and immediately.**

**Safe:** formatting, import order, unused variables, obvious typos, lint violations. Get these wrong and the build breaks or a test goes red — you know in ninety seconds.

**Never:** security logic, authentication and authorization; concurrency and locking; database migrations; money arithmetic and rounding; cryptography, under any circumstances; and any file the model has not read in full.

The common thread is that a wrong fix in any of these is **silent**. It merges green, and you find it in an incident review. Model confidence is uncorrelated with correctness — see [hallucinations](https://howaiworks.ai/glossary/hallucinations).

### 8. Measure the action rate and prune, monthly

Google disables an analyzer that goes above a 10% effective-false-positive rate — and disables it *until the owner fixes it*, which is the part everyone forgets. Do the same: for each **category** of comment, count how many led to a code change. Anything under the bar gets **switched off first and argued about afterwards.** The point is that a category has to earn its place back, not that you delete it in anger.

## Real-World Examples

### Example 1: A vendor audits its own bot and finds 79% nits

The most honest published data in this field comes from Greptile, about Greptile.

**Before.** In December 2024 they audited their own review comments:

> We analyzed existing Greptile comments and found that **~19% were good, 2% were flat-out incorrect, and 79% were nits.**

Read that against the Coverity budget. Only 2% were *wrong* — the model was accurate. And 79% were noise, which under Google's definition makes them false positives all the same. **Accuracy was never the problem.**

**After.** They rebuilt around suppressing nits rather than finding more bugs, and reported their address rate — the share of comments developers act on before merge — going **from 19% to 55%+**.

**What changed:** not the model, not the bug-finding. They made it shut up, and the address rate nearly tripled because what remained was worth reading. (The obvious objection is that this metric would also go up if the bot left almost no comments at all. Correct — which is why you track action rate **and** escaped defects, never either alone.)

### Example 2: Cloudflare, at production scale

The best operational dataset anyone has published. Over 30 days, Cloudflare's internal AI reviewer ran **131,246 reviews across 48,095 merge requests in 5,169 repositories.**

The naive version — one general-purpose reviewer, told to review the diff — produces exactly the noise this guide is about. What they built instead:

1. **Seven specialist agents** rather than one generalist: code quality, security, performance, documentation, release management, an Engineering Codex agent for internal compliance, and an AGENTS.md reviewer. A **coordinator** sits above them and decides what survives.
2. **A "What NOT to Flag" section in every prompt.** Their security reviewer is told to ignore *"theoretical risks that require unlikely preconditions,"* *"defense-in-depth suggestions when primary defenses are adequate,"* and *"'consider using library X' style suggestions."*
3. **A coordinator that filters before anything is posted** — dedupe, re-categorize, then a *"reasonableness filter: speculative issues, nitpicks, false positives, and convention-contradicted findings get dropped."*

**Result: 159,103 findings across 131,246 reviews — about 1.2 findings per review.** Median 3 minutes 39 seconds, average cost **$1.19**.

**1.2 findings per review.** That is the number to aim at. Not twelve. Throwing away the model's own output before a human ever sees it is doing most of the work here, and it is the step nearly every team skips.

And note what Cloudflare then permits itself, having done that work: their rubric routes a critical item or a production safety risk to a verdict that **blocks the merge**. That is not a contradiction of the advisory rule below — it is the rule's endpoint. You start advisory because you have not earned anything yet. Cloudflare blocks on two narrow categories, after a coordinator has already discarded most of what seven agents produced. Earn it the same way.

### Example 3: The bug it caught, exactly as advertised

**The diff:** a new early-return guard in a request handler.

```python
def handle(req):
    conn = pool.acquire()
    if not req.user:
        return Error(401)        # the new line
    data = conn.query(req.id)
    conn.release()
    return data
```

**The comment:** *"HIGH: `conn` is acquired on line 2 and released only on the success path. The new early return on line 4 leaks a pooled connection on every unauthenticated request. Under load this exhausts the pool."*

Local to the diff, mechanically verifiable, names the triggering input and the runtime consequence. A tired human misses this constantly — precisely because the new line *looks* obviously correct. The bug is not in the line that changed; it is in the interaction between the line that changed and a line that did not. **This is what you are buying.**

### Example 4: The bug it missed, and could never have caught

**The diff:** one line, in a utility function.

```python
# before
def parse_ts(s): return datetime.fromisoformat(s)

# after
def parse_ts(s): return datetime.fromisoformat(s).replace(tzinfo=timezone.utc)
```

**Every reviewer, human and machine, approved it.** It is a strictly better function. It fixes a real naive-datetime bug. The author was right.

**What broke:** one of eleven callers, in another service, was already localizing the timestamp itself. It now double-applied the offset. Billing events for European customers landed an hour off for four days.

**Why no AI reviewer catches this.** The bug is not in the diff — the diff is *correct*. It exists only in the relationship between this change and a caller that is not on screen, whose behavior depends on the old contract. A repo-indexing reviewer could in principle have found that caller. But it would still have had to *understand* that the caller's timezone handling was deliberate rather than redundant, and that fact appears nowhere in the code. It was a decision, made in a meeting, in 2023.

**What would have caught it:** the reviewer who was in that meeting. **The model can only reason about what is written down. The most expensive things your team knows were never written down.**

## Industry-Specific Applications

**Open source.** The best fit, and the economics are unbeatable — CodeRabbit is free forever on public repos, Greptile free for qualifying MIT/Apache projects. Two cautions: tone (a first-time contributor who receives nine automated nitpicks does not come back), and the closed loop maintainers already report — contributors having an agent answer the bot on their behalf, so two models negotiate a merge and no human reads anything.

**Fintech and regulated environments.** Auto-fix off, permanently. Money arithmetic and audit trails are where a silent wrong fix is unrecoverable.

**Security-critical code.** Calibrate hard. That January 2026 study found an **85–97% false discovery rate** and **21% recall** on real C/C++ projects, and LLM security reasoning is not robust: on the SecLLMHolmes benchmark (IEEE Symposium on Security and Privacy, 2024), *merely renaming functions or variables* flipped the evaluated models to an incorrect answer **26%** of the time — and adding library functions did so in another 17%. The code did not change. The name did. AI review supplements a security program; it is not one. It will catch the string-concatenated SQL. It will not catch the auth bypass spanning four files.

**Monorepos.** The diff-versus-system gap is widest here, and repository indexing stops being optional. The review a payments directory needs is not the review a frontend directory needs; `path_instructions` is how you say so.

**Startups shipping AI-generated code fast.** You need this most and can afford the noise least. Start with the narrowest scope and expand only when the action rate earns it.

## Best Practices

- **Advisory, never blocking.** Earn the right to block, per category, with data.
- **Restrict to changed lines.** Highest-leverage setting, usually off by default.
- **Write the "do not comment on" list first.** It is longer, and it is what saves the rollout.
- **Apply the senior-engineer filter.** If they would not say it out loud, silence.
- **Bless "no issues found" explicitly.** The model is ~15x less willing to say "looks fine" than a human.
- **Filter the model's output before a human sees it.** Cloudflare's coordinator step is the highest-value architecture in this guide.
- **Demand a triggering path, a consequence, and a file:line citation.** "This could be a problem" is not a review.
- **Suppress low-confidence findings.** Do not display them and expect humans to filter — humans will not filter, they will disengage.
- **Never auto-fix anything that fails silently.** Crypto, auth, concurrency, migrations, money.
- **Get the first three comments right.** *"If the first N reports are false positives (N = 3?), people tend to utter variants on 'This tool sucks.'"*

## Common Pitfalls

**Death by nitpick.** The defining failure. Twelve comments, one bug, nobody reads any of them. The cruelty is that the bot was *right* — it found the leak — and still failed, because it had already spent the reader's attention on `resp` versus `response`. **Noise does not dilute signal. It destroys it.**

**Treating "technically correct" as "successful."** If the developer took no action, it was an effective false positive regardless of truth. Most teams never make this shift, and it is the whole game.

**The trust death spiral.** Coverity named it in 2010 and LLMs make it worse: *"a vicious cycle starts where low trust causes complex bugs to be labeled false positives, leading to yet lower trust."* Once your team believes the bot is noisy, they start dismissing its **true** findings as noise too — damage not recoverable by improving the tool, because nobody is reading it any more.

**Trusting a green review on a contract change.** A clean bot review means it found no problem *in the diff*. It does not mean there is no problem.

**Believing benchmark numbers.** Vendors benchmark on PRs known to contain bugs, which makes noise free in the scoring. And benchmarks inflate — 68.26% F1 to 3.09% once the data was cleaned. Yours is the only number about you.

**Assuming a quiet bot is a broken bot.** A well-scoped reviewer says nothing on most PRs; Cloudflare runs at 1.2 findings per review. That is the system working. "Getting more value" by loosening the scope is exactly how it gets muted again.

## Measuring Success

One metric matters most, and it is not "bugs found." **Action rate: the share of bot comments that result in a code change.** It is the inverse of Google's effective false-positive rate, and it predicts whether anyone will still be reading the bot next quarter.

| Reference point | Action / resolution rate |
|---|---|
| Google's bar to keep a check enabled | 90%+ (under 10% effective FP) |
| Greptile, after rebuilding to suppress nits | 55%+ |
| Atlassian's in-house reviewer, best category | 43.3% |
| Atlassian, design comments | 28.6% |
| Greptile, before | 19% |

**If you are under 20%, your bot is on its way to being muted whether or not anyone has said so out loud.**

Then, in order:

- **Findings per review.** Cloudflare runs at ~1.2. If you are at 10, you do not have a code reviewer, you have a linter with a thesaurus.
- **Action rate by category.** What you prune on. It is normal for "resource leaks" to run at 70% and "suggest refactor" at 2%. Delete the second.
- **Escaped defects in the categories you assigned it.** Did a null deref reach production? That is the bot's job, and it failed. Escaped bugs *outside* its scope are not its failure — do not let anyone score it that way, and do not let anyone believe it was covering them.
- **Whether anyone still reads it.** In month three, ask the team whether they read the bot's comments or collapse them. The answer beats any dashboard.

## Cost Analysis

**Tools, for a 10-engineer team:**

- **[CodeRabbit](https://www.coderabbit.ai) Pro:** $24/dev/month annual — **$240/month**. Free forever on public repos.
- **[Greptile](https://www.greptile.com) Pro:** $30/seat/month — **$300/month**, plus $1 per review beyond the included 50.
- **[GitHub Copilot](https://howaiworks.ai/ai-tools/github-copilot) Pro:** $10/user/month — **$100/month**, bundled into an assistant you may already pay for. Often the cheapest "good enough."
- **[Cursor](https://howaiworks.ai/ai-tools/cursor) Bugbot:** usage-based since June 2026. Cursor's published figure is **$1.00–$1.50 for an average run**, varying with PR size and complexity; higher effort levels cost more and no cap is published. On 200 PRs a month that is roughly $200–$300 — but meter your own first month, because the variance is in your diffs, not in the price.
- **Roll your own** with [Claude Code](https://howaiworks.ai/ai-tools/claude-code) or an API script in CI: token cost only. Cloudflare's production reviewer averages **$1.19 per review** — the best public benchmark for what this costs at scale.

So: **$100 to $300 a month for ten engineers.** Trivial against one salary. **The money is not the cost.**

**The real cost is attention**, and nobody prices it. Ten engineers reading four junk comments per PR across 200 PRs a month is 8,000 comment-reads, most worthless. At ten seconds each that is over 20 hours a month — more than the tool costs. And the hours are not the harm. The harm is that after a few weeks they stop reading, and then the connection leak in comment eleven ships.

**Setup:** a day to wire in, **two to four weeks to tune the noise down** — the actual work, and the step every team skips.

**What it buys you:** a tireless reviewer that never misses a null dereference and never gets bored at 5pm on a Friday. **What it does not buy you:** anyone who knows why the code exists.

## Related Tools

- **[Claude Code](https://howaiworks.ai/ai-tools/claude-code)** — `/review` and `/code-review ultra`, with a reproduction step before reporting
- **[GitHub Copilot](https://howaiworks.ai/ai-tools/github-copilot)** — PR review bundled into a plan most teams already have
- **[Cursor](https://howaiworks.ai/ai-tools/cursor)** — Bugbot, usage-priced, with repo-level rules
- **[Windsurf](https://howaiworks.ai/ai-tools/windsurf)**, **[Cline](https://howaiworks.ai/ai-tools/cline)**, **[Aider](https://howaiworks.ai/ai-tools/aider)** — agentic editors you can point at a diff with your own review prompt
- **[Ollama](https://howaiworks.ai/ai-tools/ollama)** / **[LM Studio](https://howaiworks.ai/ai-tools/lm-studio)** — local models, for code that may not leave your network

## Related Models

- **[Claude Sonnet 5](https://howaiworks.ai/models/claude-sonnet)** — follows negative instructions well, which is most of a review prompt
- **[Claude Opus 5](https://howaiworks.ai/models/claude-opus)** — deepest reasoning for high-stakes diffs
- **[Claude Haiku 4.5](https://howaiworks.ai/models/claude-haiku)** — cheap enough to run on every commit
- **[GPT-6 Astra](https://howaiworks.ai/models/gpt)** — OpenAI's flagship since September 2026, replacing GPT-5.6; a strong generalist reviewer, but at $10/$50 per million tokens the cheaper GPT-5.6 Luna ($0.20/$1.20) is the one to benchmark first for per-commit runs
- **[Gemini 3.8 Flash](https://howaiworks.ai/models/gemini)** — large context for big diffs plus surrounding files
- **[Composer](https://howaiworks.ai/models/composer)** — the model behind Cursor's agentic workflows

New to this? Start with [prompt engineering](https://howaiworks.ai/glossary/prompt-engineering) — on this use case the prompt is not a detail, it is the product. The difference between a bot your team reads and a bot your team mutes is a suppression list and one sentence about what a senior engineer would actually bother to say out loud.

## Frequently Asked Questions

### Does AI code review actually find real bugs?

Yes, but a narrow class of them: bugs visible inside the diff. Null and undefined dereferences, unhandled error paths, resource leaks, obvious injection, off-by-one errors in a loop you can see. What it misses is anything requiring knowledge the diff does not contain — architectural violations, cross-file contract breaks, and code that is correct in isolation but wrong given how its caller uses it.

### Why does everyone end up ignoring their AI code reviewer?

Volume. Google's static-analysis team defines an 'effective false positive' as any issue where developers took no action after seeing it, and disables any check that exceeds 10% until its owner improves it. Most AI reviewers are nowhere near that bar — Greptile audited its own output and found 79% of its comments were nits. A bot that is right but ignorable gets muted, and a muted bot catches nothing.

### Should AI code review block merges?

Not at first, and probably not ever outside a small, proven set of categories. Make it advisory. A blocking bot whose comments are acted on a third of the time does not raise quality, it teaches people to click override — and once overriding is a habit, the gate is worth less than no gate at all.

### Can AI review catch security vulnerabilities?

Far less reliably than the marketing suggests. A January 2026 study of five LLM-based detectors on real open-source projects hand-inspected 385 warnings and found an average false discovery rate of 97%, with recall of 21% on C/C++. Serious vulnerabilities usually span files, and the reviewer is looking at a patch.

### What should I never let an AI reviewer auto-fix?

Anything where being wrong is silent: security logic, authentication and authorization, concurrency and locking, migrations, money arithmetic, and cryptography. Auto-fix is safe only for the class of problem where a wrong fix fails loudly and immediately — formatting, imports, obvious typos.

### Is AI review more useful now that AI writes the code?

It is more necessary and no more capable, which is an uncomfortable combination. The dominant defect in AI-generated code is duplication — GitClear reports code reuse down 70% and duplication up 81% across 623 million changes. 'We already have this function' is exactly the finding a diff-scoped reviewer cannot make.

---

Source: https://howaiworks.ai/use-cases/ai-for-code-review — HowAIWorks.ai