# SWR-Bench: LLM Code Review Comment Generation on Real Pull Requests

A benchmark paper from Peking University and Northwestern Polytechnical University (Zhengran Zeng, Ruikai Shi and colleagues), first posted September 2025 and published at FSE 2026. Earlier automated code review benchmarks score a model on single methods or diff hunks with no project context and grade it by BLEU. SWR-Bench scores review of a whole pull request against the repository it lands in, and grades by whether the review found the issues human reviewers actually raised.

## Building the benchmark

The PRs come from the 12 Python projects in SWE-bench. After dropping PRs with no review comments, more than 10 commits, or rebases, 21,350 remained. Gemini-2.5-Pro read each PR's review timeline and extracted "change-actions" (a located problem plus a proposed change), classified into 11 types that split into evolutionary changes (documentation, layout, structure) and functional ones (interface, logic, resources, checks, external support, larger defects). A PR was kept only if three runs agreed. The SZZ algorithm removed PRs where a later commit fixed something reviewers had missed, so that PRs with no change-actions can be treated as clean. Five annotators then checked labels in pairs (Cohen's kappa 0.66, disagreements resolved by discussion) and removed PRs whose only changes were formatting or documentation.

The final set is 500 PRs with change-actions (1.90 per PR on average) and 500 clean PRs, sampled to match on size so a model cannot tell them apart by line counts. On a clean PR any comment is a false positive. Filtering raised the share of functional changes from under 15% in the raw data to 31.8%.

Grading uses an LLM for matching rather than scoring: it extracts the review's individual suggestions and marks which ground-truth change-actions each one hits. Hit agreement across humans and LLM judges ran from kappa 0.71 to 0.87. When humans picked the better of two reviews, this method agreed with them at 0.53–0.62, on par with humans agreeing with each other (0.56–0.63); an LLM giving a 1–10 quality score agreed at 0.31–0.45, and BLEU agreed worse than chance. Gemini-2.5-Flash does the grading, at about $1.57 per full run.

## Results

Nothing performs well. The best combination in the main table, PR-Review (a single prompt with careful structure) on Gemini-2.5-Pro, reaches precision 16.65%, recall 23.18%, F1 19.38%. Mean F1 by tool: PR-Review 18.73%, an agentic baseline that explores the repository 12.61%, a plain prompt 12.49%, CR-Agent's two debating agents 9.22%, and Hybrid-Review, which pastes static-analysis output into the prompt, 4.87% with 6.18 false positives per PR on average. The older fine-tuned hunk-level reviewers score 6.13% and 7.22%, and they also score the highest BLEU, which is the paper's evidence that text similarity measures the wrong thing.

Across models under PR-Review, GPT-5 has the highest F1 at 20.85% (recall 35.93%). Reasoning training helps: Qwen-2.5-R1-14B reaches 15.95% against 9.01% for Qwen-2.5-14B. The ranking does not track SWE-bench or LiveCodeBench, which the authors take as a sign that review is a different skill from generation.

Functional issues are found more reliably than evolutionary ones: logic changes reach 26.20% F1, while the best evolutionary type, code organization, reaches 16.45%. Recall collapses as PRs carry more issues, from 38.35% with one issue to 8.88% with five or more, while precision stays between 29.63% and 44.12%. A manual look at 100 of the 1,101 false positives from the best setup found missing context (48%), treating any large modification as risky (17%), vague unactionable advice (16%), missing project conventions (13%) and misjudged intentional deviations from best practice (3%).

## Reviews are noisy, so pool them

Different models agreed on only 36 found change-actions, and the same model across five runs agreed on only 27. That randomness motivates Multi-Review: run the reviewer n times and have one more LLM call merge the reports, either from the same model (Self-Agg) or from several models (Multi-Agg). Gemini-2.5-Flash with Self-Agg at n = 10 reaches 21.91% F1, a 43.67% relative gain, and raises recall 118.83% to 30.44%. At n = 5, Flash reaches 20.48% and beats a single Gemini-2.5-Pro pass (19.38%) at $0.00368 against $0.00586 per PR. Pro with Self-Agg at n = 5 reaches 23.84%. Gains flatten past n = 5 while cost keeps growing linearly. Qwen-Chat-7B and 32B improve by 26.13% and 19.25% relative. The Multi-Agg results are only in a figure, and the vault copy does not carry them.

## What it does not test

The PRs were written by humans before LLM coding, and the reviewing model has no relationship to their author, so the benchmark says nothing about self-review against cross-model review, or about reviewing LLM-written code. What it contributes to that question is a baseline: a single LLM review pass on real code is low-precision and unstable from run to run, and repeated passes by the same cheap model add recall at a cost.

## Cross-references

- [[cross-model-code-review]] — how this fits with the papers that do vary who reviews whom
- [[code-review-throughput-limits]] — the human ceiling that makes automated review attractive in the first place
- [[google-code-review-looking-for]] and [[code-review-knowledge-transfer]] — the evolutionary, non-defect side of review that models find hardest
- [[roborev]] — a commit-hook reviewer with a false-positive filtering pass, the practical form of this task
- [[llm-output-variance]] — the run-to-run instability that Multi-Review exploits