Coding agents like Claude Code, Codex and Cursor now open pull requests that a human may have skimmed once. That gap is why teams are building an AI code review checklist: the old habit of assuming the author understood the change doesn’t hold when a model wrote it.
Why agent PRs get extra scrutiny
Several studies point the same way, though none measures your own codebase.
- CodeRabbit looked at 470 open-source pull requests. Those co-authored with AI averaged 10.83 issues, compared with 6.45 for human-only ones. Security problems ran up to 2.74 times higher. The company sells review tools and guessed authorship from co-author tags, so read the ratio loosely.
- Veracode gave more than 100 models 80 coding tasks. Nearly half the time (45%) the output contained an OWASP Top 10 flaw, and cross-site scripting defences failed in 86% of relevant cases.
- Researchers presenting at USENIX Security 2025 counted 205,474 distinct package names that models invented. Hallucination rates averaged at least 5.2% for commercial models and 21.7% for open-source ones. An attacker who registers such a name gets code installed by anyone who trusts the suggestion.
- The 2025 DORA report surveyed nearly 5,000 professionals. AI adoption went with faster delivery and with worse stability.
Reviewers are the last line of defence, and their time is limited. So the first job is deciding how much attention each PR deserves.
Sort pull requests into three risk tiers
Decide the tier from the files touched, before reading the diff.
- Tier 1: documentation, tests, internal scripts.
- Tier 2: feature code inside a tested module.
- Tier 3: authentication, payments, personal data, migrations, infrastructure, CI, dependencies and agent settings.
Who wrote the change doesn’t set the tier. An agent’s typo fix in the docs is tier 1, while a human’s two-line tweak to an auth check is tier 3.
Put the questions in the PR template
GitHub pre-fills descriptions from pull_request_template.md, stored in the repo root, docs/ or .github/. It reaches collaborators after merging into the default branch. Teams that want several templates can use a PULL_REQUEST_TEMPLATE/ folder and choose with a query parameter.
Ask the author to state the linked issue, whether a human, an agent or both wrote the code, the tier, and what they verified by hand. Two prompts earn their place. “How this was written” shows how much understanding sits behind the diff. “Files changed outside the task” exposes agents that wander into code nobody asked them to edit.
The 24 checks, by area
- Scope: the PR links its spec, stays within it, and discloses how it was produced.
- Tests: new tests fail without the change; none were deleted, skipped, loosened or re-snapshotted; nothing is hard-coded to satisfy a fixture.
- Correctness: timeouts, retries, empty input and partial failure are handled, exceptions aren’t swallowed, comments match behaviour, and no helper duplicates an existing one.
- Dependencies: each new package is real, correctly named, maintained and licensed acceptably, and the lockfile moves only as the manifest does.
- Security: no secrets anywhere, validated inputs, authorisation (not just authentication) on new endpoints, parameterised queries, clean logs.
- Agent configuration: edits to AGENTS.md, CLAUDE.md, permissions or MCP settings, and any CI workflow change, go to a code owner. Prompt injection is LLM01 in the OWASP Top 10 for LLM Applications 2025, so check that nothing in the diff obeys instructions the agent picked up from issues or web pages.
- Ownership: the reviewer can explain every line, AI review is only a first pass, and tier 3 work ships behind a flag or with a rollback and reversible migrations.
One habit is worth adopting. Before approving, ask whether the new tests would fail on main. Tests written alongside the code often just restate what it does.
Making GitHub enforce it
A checklist that is optional won’t last a quarter. A CODEOWNERS file maps paths to owners, requests their review automatically, and, with branch protection, can require their sign-off. The last matching pattern wins. Point workflows, infra, migrations, auth code, package.json and agent config files at named teams.
Keep the required approval human. Copilot posts a “Comment” review by default instead of an “Approve”, so it doesn’t count toward required approvals, and it’s best left that way.
Measure what slips through
For four weeks, log each merged PR with its tier, author type, lines changed, review minutes, issues caught and escaped defects (bugs found after merge and traced back to it). Escape rate is escaped defects divided by issues found in review plus escaped defects. Also track review minutes per 100 lines so a huge agent PR and a tiny fix can be compared.
Three results call for action. Agent-written PRs escaping more defects than human ones in the same tier means review is too shallow, so turn the commonest miss into a required check. Falling minutes per 100 lines as agent volume climbs suggests skimming, so cap PR size. Any tier 3 escape sends you back to ask whether two reviewers and a code owner signed off. The same figures feed your change failure rate.
What stays outside the checklist
It guides the human reading the diff. Automated tests, static analysis, dependency scanning and security reviews for sensitive systems still need to run, and a separate policy should decide which repos and data agents may touch at all.
Source: IdeaPlan