Code review tools are usually compared on how many issues they find. That is the wrong axis. Two tools reporting 40 findings each can be almost entirely non-overlapping, because they are looking for different categories of problem. The useful question is which category each one covers, and which categories nothing covers.
What are the categories of code review problem?
Four, and they need different tooling. Deterministic defects (a null dereference), security vulnerabilities (an injection path), design problems (the wrong abstraction), and context violations (this contradicts a decision the team made last year). Most tools handle one or two well and are near-useless on the others.
| Category | What catches it | What does not |
|---|---|---|
| Deterministic defects | Type checkers, linters, static analysis | LLM review, which is probabilistic and will miss some |
| Security vulnerabilities | SAST, dependency scanners, OWASP-aligned rules | General-purpose LLMs unless specifically prompted |
| Design and readability | LLM review, human review | Linters, which have no model of intent |
| Context violations | Human review, or an LLM given the team's decision history | Every tool with no access to prior decisions |
What is AI code review genuinely good at?
Explaining unfamiliar code and catching readability and design problems that linters cannot express as rules. An LLM can say 'this function does three things and the third is surprising' — a category of feedback static analysis has no vocabulary for. It is also fast on code nobody on the team wrote.
What does AI code review reliably miss?
Anything requiring context it was not given. If a pull request violates an architectural decision made a year ago, no model reviewing the diff alone will flag it, because the decision is not in the diff. This is the single largest gap in AI review today, and it is a context problem rather than a model-capability problem.
- Cross-file and cross-service consequences when only the diff is in context.
- Team conventions that are not encoded anywhere the tool can read.
- Whether the change is the right thing to build at all — a review-time question no tool asks.
- Regressions against decisions recorded outside the codebase.
The finding that matters most
DORA's research consistently associates fast, small pull requests with better delivery performance. That means a review tool's most valuable property is often latency, not thoroughness — a tool that returns useful feedback in 30 seconds changes behaviour, while one that takes 20 minutes gets routed around. Weight speed accordingly when comparing.
How should you combine tools?
Layer them by category rather than picking a winner. A type checker and linter on every commit, a security scanner on every pull request, an LLM review for explanation and design feedback, and a human for context and intent. Each layer is cheap relative to the class of bug it catches.
When is human review still the only option?
Whenever the question is 'should this exist' rather than 'is this correct'. Tools evaluate the code in front of them; only a person who knows the roadmap, the incident history and the team's prior decisions can say the change is well-formed but wrong. No current tool substitutes for that.
Frequently Asked Questions
Can AI replace code review?
No. It replaces the mechanical layer — style, obvious defects, explaining unfamiliar code — which is genuinely most of the volume. It does not replace judgement about whether a change fits the system, because that judgement depends on context the tool does not have.
Does AI code review increase or decrease review time?
It reduces time-to-first-feedback, which is usually the bottleneck, while adding triage work if the tool is noisy. Tools that report everything they notice tend to get ignored within weeks; tools that report few, high-confidence findings survive.
How do I stop an AI reviewer from being noisy?
Constrain the categories it reports on and set a confidence floor. A reviewer that surfaces three real problems is used; one that surfaces forty possible problems is muted, and the three real ones are lost with the rest.
What should I measure to know if it is working?
Change failure rate and time to restore, from the DORA metrics — not the number of findings. A tool producing more findings while defects escaping to production stay flat is producing noise.
References
- OWASP Code Review Guide — OWASP
- About pull request reviews — GitHub Docs
- DORA research program — DORA / Google Cloud
About Jishu Labs
Jishu Labs is a software development company founded in 2016. We build custom software, AI/ML systems, and full-stack web and mobile applications for clients, and we make eight AI tools for software teams.