# SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation DOI: [10.1145/3808144](https://doi.org/10.1145/3808144)Journal: PACMSEVolume: 3FSEFSE1377fse26mainb-p535-pCCS: Software and its engineering Maintaining software Zhengran Zeng [](https://orcid.org/0009-0009-8422-4522 "ORCID 0009-0009-8422-4522")Note: Both authors contributed equally to this research. Affiliation: Peking University, Beijing, China email: [zhengranzeng@stu.pku.edu.cn](mailto:zhengranzeng@stu.pku.edu.cn) Ruikai Shi [](https://orcid.org/0009-0001-2028-670X "ORCID 0009-0001-2028-670X")Affiliation: Peking University, Beijing, China email: [rkshi25@stu.pku.edu.cn](mailto:rkshi25@stu.pku.edu.cn) , Keke Han [](https://orcid.org/0009-0003-1843-8545 "ORCID 0009-0003-1843-8545")Affiliation: Peking University, Beijing, China email: [kkhan25@stu.pku.edu.cn](mailto:kkhan25@stu.pku.edu.cn) , Yixin Li [](https://orcid.org/0009-0005-0492-2234 "ORCID 0009-0005-0492-2234")Affiliation: Peking University, Beijing, China email: [leason\_lyx@stu.pku.edu.cn](mailto:leason_lyx@stu.pku.edu.cn) , Kaicheng Sun [](https://orcid.org/0009-0007-0856-9049 "ORCID 0009-0007-0856-9049")Affiliation: Northwestern Polytechnical University, Xian, China email: [sunkaicheng@mail.nwpu.edu.cn](mailto:sunkaicheng@mail.nwpu.edu.cn) , Yidong Wang [](https://orcid.org/0009-0007-9969-8259 "ORCID 0009-0007-9969-8259")Affiliation: Peking University, Beijing, China email: [2301110730@stu.pku.edu.cn](mailto:2301110730@stu.pku.edu.cn) , Zhuohao Yu [](https://orcid.org/0009-0000-8256-8588 "ORCID 0009-0000-8256-8588")Affiliation: Peking University, Beijing, China email: [zyu@stu.pku.edu.cn](mailto:zyu@stu.pku.edu.cn) , Rui Xie [](https://orcid.org/0000-0002-1756-7746 "ORCID 0000-0002-1756-7746")Note: Those authors are the corresponding authors. Affiliation: Peking University, Beijing, China email: [ruixie@pku.edu.cn](mailto:ruixie@pku.edu.cn) , Wei Ye [](https://orcid.org/0000-0002-9331-4716 "ORCID 0000-0002-9331-4716")Affiliation: Peking University, Beijing, China email: [wye@pku.edu.cn](mailto:wye@pku.edu.cn) and Shikun Zhang [](https://orcid.org/0000-0002-8576-2674 "ORCID 0000-0002-8576-2674")Affiliation: Peking University, Beijing, China email: [zhangsk@pku.edu.cn](mailto:zhangsk@pku.edu.cn) Received  2026-03-24 ###### Abstract. Automated Code Review (ACR) is crucial for software quality, yet existing benchmarks often fail to reflect real-world complexities, hindering the evaluation of modern Large Language Models (LLMs). Current benchmarks frequently focus on fine-grained code units, lack complete project context, and use inadequate evaluation metrics. To address these limitations, we introduce SWR-Bench, a new benchmark comprising 1000 manually verified Pull Requests (PRs) from GitHub, offering PR-centric review with full project context. SWR-Bench employs an objective LLM-based evaluation method that aligns strongly with human judgment (∼\\sim90% agreement) by verifying if issues from a structured ground truth are covered in generated reviews. Our systematic evaluation of mainstream ACR tools and LLMs on SWR-Bench reveals that current systems underperform, and ACR tools are more adept at detecting functional errors. Subsequently, we propose and validate a simple multi-review aggregation strategy that significantly boosts ACR performance, increasing F1 scores by up to 43.67%. Our contributions include the SWR-Bench benchmark, its objective evaluation method, a comprehensive study of current ACR capabilities, and an effective enhancement approach, offering valuable insights for advancing ACR research. ###### Keywords: Automated Code Review, Large Language Models, Benchmark ††cc-license: by ## 1. Introduction Code Review (CR) is an indispensable quality assurance practice in software development ([Yang et al., 2024b](#bib.bib22); [Badampudi et al., 2023](#bib.bib23); [Çetin et al., 2021a](#bib.bib1); [Bavota and Russo, 2015](#bib.bib2); [Liu et al., 2024](#bib.bib45)), aimed at identifying and rectifying potential issues in code changes before integration. While crucial for enhancing code quality, traditional manual code review faces significant hurdles in modern software development. The increasing scale and complexity of projects mean that manual reviews are time-consuming, contributing to development costs and potential delays in feature releases ([Sghaier and Sahraoui, 2024](#bib.bib3); [Denisov-Blanch et al., 2024](#bib.bib4); [Çetin et al., 2021b](#bib.bib5)). To mitigate these challenges, Automated Code Review (ACR) Comment Generation has emerged as a vital field and a prominent research area within software engineering ([Cihan et al., 2024](#bib.bib6); [Tufano et al., 2021](#bib.bib11); [Thongtanunam et al., 2022](#bib.bib12); [Tufano et al., 2022](#bib.bib13); [Li et al., 2022](#bib.bib14)), driving the creation of numerous tools and datasets. These ACR works strive to improve the efficiency and effectiveness of the code review process by shortening feedback cycles, reducing the manual burden on developers, and improving the consistency and scope of reviews, thus playing a crucial role in streamlining software development workflows and maintaining high standards of code quality. More recently, the explosive advancements in Large Language Models (LLMs) have catalyzed a significant shift in ACR research ([Sun et al., 2025](#bib.bib7); [Jaoua et al., 2025a](#bib.bib8); [Kumar and Chimalakonda, 2024](#bib.bib9); [Lu et al., 2023](#bib.bib10)), with a growing emphasis on LLM-based methodologies. ![Refer to caption](https://arxiv.org/html/2509.01494v2/hunk.png) (a) Traditional hunk-level review with limited context. ![Refer to caption](https://arxiv.org/html/2509.01494v2/prreview.png) (b) SWR-Bench PR-level review with full codebase context. Figure 1. A comparison of review paradigms. However, a substantial portion of existing ACR benchmarks ([Tufano et al., 2021](#bib.bib11); [Thongtanunam et al., 2022](#bib.bib12); [Tufano et al., 2022](#bib.bib13); [Li et al., 2022](#bib.bib14)) were established primarily for deep learning models that predate the widespread capabilities and sophisticated understanding of modern LLMs. This temporal and methodological gap presents challenges in evaluating the true potential of contemporary LLM-based ACR tools using these benchmarks. This discrepancy is particularly evident in the fundamental setup of the review task itself, as illustrated in Figure [1](#S1.F1 "Figure 1 ‣ 1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). Specifically, existing representative benchmarks ([Tufano et al., 2021](#bib.bib11); [Tufano et al., 2022](#bib.bib13); [Thongtanunam et al., 2022](#bib.bib12); [Li et al., 2022](#bib.bib14)) primarily center their evaluation on isolated code changes, or diff hunks, with limited contextual information, such as only a few related source code files (see Figure [1(a)](#S1.F1.sf1 "In Figure 1 ‣ 1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")). This hunk-based approach fails to mirror real-world developer practices, where an entire Pull Request (PR) is reviewed as a cohesive unit. Consequently, this narrow scope can lead to missing critical inter-dependency bugs.Meanwhile, these benchmarks typically do not provide the complete project codebase as necessary context, making it difficult for ACR tools to understand the global impact of code changes. In contrast, our paradigm provides the full context (Figure [1(b)](#acmlabel1 "In Figure 1 ‣ 1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")). More critically, the evaluation methodologies for these benchmarks are also problematic. They generally rely on traditional natural language generation metrics such as E​x​a​c​t​-​M​a​t​c​hExact\\text{-}Match ([Tufano et al., 2021](#bib.bib11)), B​l​e​uBleu scores ([Papineni et al., 2002](#bib.bib15)) and simple LLM-based ratings ([Gu et al., 2024](#bib.bib20); [Wang et al., 2024](#bib.bib35); [Jaoua et al., 2025a](#bib.bib8)). However, metrics like B​l​e​uBleu primarily measure textual similarity, while the reliability of LLM ratings is often questioned due to potential inherent biases; both approaches have been shown to be severely inadequate in assessing whether the crucial issues are truly identified in code review comments ([Jiang et al., 2025](#bib.bib16); [Lu et al., 2025](#bib.bib17)). Furthermore, some studies ([Sun et al., 2025](#bib.bib7); [Cihan et al., 2024](#bib.bib6)) utilize expensive human annotation, but this approach is difficult to scale. Consequently, evaluations on current benchmarks often misrepresent the true capabilities and practical value of LLMs in complex, real-world code review. To bridge this critical gap, we introduce SWR-Bench (Software Review Benchmark), a software review benchmark designed for more realistic evaluation. Specifically, SWR-Bench comprises 1000 manually verified PR instances from GitHub open-source projects. It directly confronts the issues in existing benchmarks through its core design features: - • PR-centric Review: Each code review instance is based on a complete PR. This setup is intentionally more challenging, yet more aligned with real-world practices, as we aim to assess the ability of ACR tools for end-to-end code review. This includes identifying areas requiring review, pinpointing specific issues, and generating meaningful review comments. - • Comprehensive Context: To simulate a realistic review environment, our benchmark provides all relevant code changes (commits) and a snapshot of the entire project repository for each PR, offering comprehensive context that enables tools to accurately assess the global impact of modifications. - • Objective LLM-based Evaluation: We employ an automated evaluation where an LLM objectively verifies if issues from a structured ground truth report are covered in the generated report. This method offers a more objective and semantically relevant assessment than subjective “LLM-as-judge” ratings ([Gu et al., 2024](#bib.bib20); [Wang et al., 2024](#bib.bib35); [Jaoua et al., 2025a](#bib.bib8)) or text-similarity metrics ([Tufano et al., 2021](#bib.bib11); [Papineni et al., 2002](#bib.bib15)). After establishing SWR-Bench, we first validated its evaluation method, confirming strong (∼\\sim90%) agreement with human judgment. We then systematically evaluated mainstream ACR tools and leading LLMs on SWR-Bench, uncovering key performance aspects in realistic scenarios: 1) current tools and LLMs do not yet perform sufficiently well; 2) furthermore, we found that LLMs trained with a reasoning-focused approach exhibit better code review capabilities; and finally, 3) existing tools are more adept at detecting functional errors, such as bugs, compared to identifying non-functional issues like outdated documentation. Motivated by these observations, we further introduce a simple enhancement scheme for LLM-based automated code review. This straightforward strategy empowers an LLM to synthesize feedback from multiple review sources by analyzing, filtering valid points, and discarding erroneous suggestions to produce a superior final report. Experiments demonstrate this strategy substantially boosts ACR performance (increasing issue detection F​1F1 score by up to 43.67%), suggesting promising avenues for future research. Therefore, our main contributions in this work include: 1. 1\. Benchmark: We introduce SWR-Bench, a benchmark comprising 1,000 real-world PRs with full project context, providing a more realistic and challenging evaluation platform for ACR systems. 2. 2\. Evaluation Method: We developed and validated an objective LLM-based evaluation method to objectively assess ACR quality by comparing tool outputs against human ground truth. 3. 3\. Study: We systematically assessed mainstream LLMs and ACR tools on SWR-Bench, analyzing their performance, strengths, and limitations in practical code review scenarios. 4. 4\. Approach: We developed and validated a simple multiple review strategy that significantly improves the code review performance of ACR tools on SWR-Bench. ## 2. Background and Related Work ### 2.1. Modern Code Review ![Refer to caption](https://arxiv.org/html/2509.01494v2/review-process.png) Figure 2. A simplified code review process. Modern Code Review (MCR) is a widely adopted quality assurance practice in contemporary software development ([Yang et al., 2024b](#bib.bib22); [Badampudi et al., 2023](#bib.bib23)), crucial for enhancing code quality, fostering knowledge sharing, and ensuring adherence to project standards. It typically involves developers submitting code changes, often through PRs, for scrutiny by their peers before integration into the main codebase. A simplified, yet representative, MCR process is depicted in Figure [2](#acmlabel2 "Figure 2 ‣ 2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). This process generally unfolds in three key stages: 1) reviewers scrutinize the code submitted in the pull request; 2) following their examination, reviewers provide a review report, identifying specific areas requiring modification, which we term “change-actions”; 3) the developer then verifies these change-actions and updates the PR code accordingly. The fundamental unit of actionable feedback within this process is what we term a change-action. We define an effective change-action as a complete directive that specifies the location(s) of the issue, describes the underlying problem, and proposes a tangible solution. Therefore, a single change-action precisely reflects a specific issue and its resolution. An ACR tool capable of generating correct change-actions can thus effectively guide developers in improving code quality. It is also important to note that a change-action is not strictly tied to a specific line of code; it can be conceptual and address a high-level concern spanning multiple code locations, such as an architectural flaw. We use the term ‘change‘ rather than ‘fix‘ or ‘defect‘ because reviewer intent often extends beyond rectifying bugs to include non-functional improvements in readability, maintainability, or design, which are not traditional defects but are vital for software quality ([Beller et al., 2014](#bib.bib36); [Fregnan et al., 2022](#bib.bib24)). The nature of these change-actions varies significantly, reflecting diverse reviewer considerations. To classify them, we adopt the detailed taxonomy from established literature shown in Table [1](#S2.T1 "Table 1 ‣ 2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation") ([Beller et al., 2014](#bib.bib36); [Fregnan et al., 2022](#bib.bib24)). We selected this taxonomy because it is a reliable standard that has been consistently used across multiple studies in the Automated Code Review (ACR) domain. It distinguishes between two high-level categories: Evolutionary changes (improving future maintainability) and Functional changes (altering software behavior), which are further decomposed into 11 fine-grained types. Understanding this classification is fundamental for analyzing the effectiveness of both manual and automated review processes. Furthermore, it is important to note that in this work, we focus specifically on the task of review comment generation, which we consider a critical prerequisite for end-to-end code review automation. The quality of generated comments directly determines the success of downstream tasks like automated code refinement, as precise feedback is the necessary input for any subsequent modifications. Table 1. The taxonomy of change-actions under code review, adapted from ([Beller et al., 2014](#bib.bib36); [Fregnan et al., 2022](#bib.bib24)). Note: All change-actions are classified into leaf-node categories only. ID Type & Description E Evolutionary: Improving code’s future maintainability and structure, not its function. E.1 Documentation: Modifying in-code information for better human understanding. E.1.1 Textual: Changing textual code elements such as inline comments and variable/function names. E.1.2 Language Supported: Using language features for documentation purposes. E.2 Visual Representation: Modifying code layout for better style and readability. E.3 Structure: Altering the project’s organization or architecture. E.3.1 Organization: Reorganizing code by moving or removing parts. E.3.2 Solution Approach: Altering implementation methods or adding maintainability elements (e.g., tests). F Functional: Altering the software’s behavior or interactions. F.1 Interface: Modifying interactions between codebase parts. F.2 Logic: Changing the code’s logical operations or algorithms. F.3 Resource: Altering how variables or resources are managed. F.4 Check: Modifying checks for unhandled states/conditions. F.5 Support: Adjusting interactions with external support systems or libraries. F.6 Larger Defects: PRs that were ultimately not merged (i.e., rejected suggestions), typically involving major, extensive functional issues or missing features that required fundamental rethinking. ### 2.2. Automatic Code Review While modern code review, as described above, is a vital software quality practice, manual code review faces efficiency and consistency challenges ([Sghaier and Sahraoui, 2024](#bib.bib3); [Denisov-Blanch et al., 2024](#bib.bib4); [Çetin et al., 2021b](#bib.bib5)), motivating the development of ACR, aiming to use tools to assist or replace parts of manual review, thereby improving efficiency, shortening feedback cycles, and enhancing consistency ([Cihan et al., 2024](#bib.bib6)). Specifically, ACR technology has progressed through several stages. Initially, development focused on rule-based and static analysis tools, such as PMD ([PMD, 2000](#bib.bib25)) and SonarQube ([SonarSource, 2006](#bib.bib26)), which are effective but can be rigid and prone to false positives. Subsequently, machine learning and deep learning approaches ([Tufano et al., 2021](#bib.bib11); [Tufano et al., 2022](#bib.bib13); [Thongtanunam et al., 2022](#bib.bib12); [Li et al., 2022](#bib.bib14)) emerged for tasks like code change quality assessment ([Tang et al., 2024](#bib.bib28); [Jaoua et al., 2025a](#bib.bib8); [Qodana AI, 2025](#bib.bib39); [Zeng et al., 2021](#bib.bib37); [Zeng et al., 2024](#bib.bib46)), comment generation ([Li et al., 2022](#bib.bib14)), and automated code repair ([Just et al., 2014](#bib.bib44); [35](#bib.bib48); [Xia et al., 2024](#bib.bib38)); while these methods can learn complex patterns, they may still struggle with deep code intent and complex contexts. More recently, large language model and AI agent-based methods ([Sun et al., 2025](#bib.bib7); [Jaoua et al., 2025a](#bib.bib8); [Kumar and Chimalakonda, 2024](#bib.bib9); [Lu et al., 2023](#bib.bib10); [Rasheed et al., 2024](#bib.bib27); [Tang et al., 2024](#bib.bib28)), have demonstrated strong code understanding and generation capabilities, enabling more natural comments and complex reasoning. Cutting-edge work in this area, such as CodeAgent ([Tang et al., 2024](#bib.bib28)), explores multi-agent systems for collaborative review, while commercial tools like PR-Review ([Qodana AI, 2025](#bib.bib39)) focus on providing targeted, actionable feedback that dynamically learns team norms, aiming to address new challenges in AI-assisted coding like feedback redundancy and unclear prioritization. Despite the potential of ACR, especially LLM/Agent-based methods, their real-world effectiveness in improving review quality and efficiency remains a key question. This highlights an urgent need for more effective and reliable evaluation methods and benchmarks, as current assessment approaches ([Tufano et al., 2021](#bib.bib11); [Thongtanunam et al., 2022](#bib.bib12); [Tufano et al., 2022](#bib.bib13); [Li et al., 2022](#bib.bib14); [Jaoua et al., 2025a](#bib.bib8)) may not adequately capture the performance of these advanced models in complex, realistic code review scenarios. ### 2.3. Code Review Benchmarks To evaluate various ACR techniques, the research community has constructed several specialized code review benchmarks. Table [2](#S2.T2 "Table 2 ‣ 2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation") provides a comparative overview of representative benchmarks, assessing them based on their data source, scale, fundamental review unit, the scope of provided context, and evaluation methodology. This comparison highlights a significant gap in existing resources, which our work, SWR-Bench, aims to fill. Table 2. Comparison of Representative Code Review Benchmarks. Dataset Source Size Unit Context Evaluation Trans-Review-Data ([Tufano et al., 2021](#bib.bib11)) Github, Gerrit 1,719 Method None Bleu AutoTransform-Data ([Thongtanunam et al., 2022](#bib.bib12)) Gerrit 14,750 Method None Bleu T5-Review-Data ([Tufano et al., 2022](#bib.bib13)) Github, Gerrit 17,194 Method None Bleu Code-Reviewer-Data ([Li et al., 2022](#bib.bib14)) Github 10,000 Diff Hunk None Bleu, Human Eval. CR-Agent-Data ([Tang et al., 2024](#bib.bib28)) Github 3,545 Commit Related Source Code Human Eval. Hybrid-Review-Data ([Jaoua et al., 2025a](#bib.bib8)) Github 1,245 Diff Hunk Related Source Code LLM Scoring SWR-Bench Github 1,000 Pull Request Complete Codebase Objective LLM Eval. Tufano et al.’s datasets ([Tufano et al., 2021](#bib.bib11); [Thongtanunam et al., 2022](#bib.bib12); [Tufano et al., 2022](#bib.bib13)), typically sourced from Gerrit and GitHub projects, center on method-level triplets (i.e., the submitted method, reviewer comments, and the modified method). These are used to evaluate tasks like method-level code transformation, implementing changes based on comments, and generating comments for methods. As indicated in Table [2](#S2.T2 "Table 2 ‣ 2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), these benchmarks provide no external code context and primarily rely on text-similarity metrics like B​l​e​uBleu for evaluation. Code-Reviewer-Data ([Li et al., 2022](#bib.bib14)) shifts the focus to a single diff hunk (i.e., a contiguous block of modified lines). While closer to a review task, it still operates on fragments of a change rather than the whole. Its evaluation metrics also include BLEU and Exact-Match, with some later studies ([Jaoua et al., 2025a](#bib.bib8)) adopting simple LLM-based scoring. More recent benchmarks like CR-Agent-Data ([Tang et al., 2024](#bib.bib28)) and Hybrid-Review-Data ([Jaoua et al., 2025a](#bib.bib8)) begin to incorporate related source code as context, but still fall short of providing the complete project environment. While valuable, these benchmarks have limitations compared to real-world code review. As summarized in Table [2](#S2.T2 "Table 2 ‣ 2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), they consistently lack a PR-level scope and comprehensive context, instead focusing on isolated methods or diff hunks. This hinders evaluation of an ACR tool’s ability to manage real-world complexity. Furthermore, these benchmarks predominantly rely on inadequate evaluation metrics, such as E​x​a​c​t​-​M​a​t​c​hExact\\text{-}Match and B​l​e​uBleu or simple LLM-based ratings metric, which are known to correlate poorly with human judgment ([Kulal et al., 2019](#bib.bib41); [Chen et al., 2025](#bib.bib40)), thus leading to potentially misleading assessments of tool efficacy. Beyond these public benchmarks, some studies ([Sun et al., 2025](#bib.bib7); [Cihan et al., 2024](#bib.bib6)) leverage proprietary internal data for ACR research, validating tools through manual human assessment. While such efforts provide valuable, deep insights, the private nature of this data and the prohibitive cost of manual validation make these approaches unscalable and difficult for the broader community to build upon. This leaves a clear void for a publicly available, context-rich, and scalably evaluable benchmark. In summary, as Table [2](#S2.T2 "Table 2 ‣ 2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation") illustrates, existing public benchmarks consistently lack PR-level scope, comprehensive project context, and reliable evaluation metrics, three deficiencies that limit their ability to accurately assess modern LLM-based ACR tools. Meanwhile, proprietary benchmarks with manual evaluation, though insightful, are neither scalable nor reproducible. These gaps motivate the design of SWR-Bench, which we detail in the following section. ## 3. SWR-Bench This section details our methodology for constructing the automatic code review benchmark. ### 3.1. Benchmark Construction ![Refer to caption](https://arxiv.org/html/2509.01494v2/swrbench.png) Figure 3. The SWR-Bench construction pipeline. The overall construction workflow of SWR-Bench is depicted in Figure [3](#acmlabel3 "Figure 3 ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation") and detailed below. #### Step 1: Source Data Collection and Initial Filtering The foundation of SWR-Bench lies in real-world software development practices. To ensure the benchmark’s realism and quality, we adopted the 12 open-source Python projects from the well-established SWE-Bench ([Jimenez et al., 2024](#bib.bib29)), as they represent the most popular packages on PyPI by download count. This selection guarantees their code is of high quality and their development practices are highly representative of real-world scenarios. We utilized the GitHub API to crawl all historical pull requests (PRs) from these projects, collecting comprehensive metadata for each. This included titles, descriptions, commit histories, code diffs, review comments, discussion threads, and final PR statuses (e.g., merged, closed). From this raw collection, we performed initial filtering to refine the candidate pool. We excluded PRs without any review comments, as they lack the necessary interaction for our analysis. We also removed PRs with large-scale changes (e.g., exceeding 10 commits in one PR) or those with rebase operations in their commit history. This is because extensive changes make it challenging for the LLM in the subsequent verification stage to correctly identify and label change-actions, and rebase operations complicate the construction of reproducible environments. After this initial filtering, 21,350 PRs remained for the next phase. #### Step 2: LLM-based change-actions Verification and Classification In this step, we employed a LLM to identify and extract “change-actions” from the filtered PRs. Specifically, we provided the LLM with the complete, chronologically ordered review timeline for each PR, which included its title, description, all developer-reviewer discussions, and all commit messages. A crucial instruction was for the LLM to identify each change-action (corresponding to the 11 types in Table [1](#S2.T1 "Table 1 ‣ 2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")) and link it to the specific commit that introduced and fixed the issue, with the full prompt is detailed in ([35](#bib.bib48)) due to space constraints. To ensure high-quality annotations, we employed Gemini-2.5-Pro ([Google, 2025](#bib.bib30)), a state-of-the-art (SOTA) LLM. Furthermore, we implemented a majority voting technique by making three independent requests to the LLM for each PR. A PR was considered for further processing only if all three requests yielded consistent change-actions extraction results. This requirement acts as a quality filter, excluding PRs where the LLM produced inconsistent results across runs. Lastly, those consistent PRs were then classified into two categories: Change-PRs”, which contain at least one identified change-action, and Clean-PRs”, which have no identified change-action. #### Step 3: Quality Enhancement through Filtering and Sampling To further enhance dataset quality, we performed additional filtering and sampling. First, we applied the SZZ algorithm ([Rosa et al., 2023](#bib.bib31)) to identify issues that might have been missed by human reviewers but were fixed in later commits. PRs containing such missed changes were removed. This step ensures that our Clean-PRs are genuinely free of known, non-trivial defects. Consequently, any comment an ACR tool generates for a Clean-PR is, by definition, considered a false positive. This strict protocol enables a robust measurement of the false positive rate, a critical metric for assessing a tool’s precision and its ability to avoid distracting developers with irrelevant suggestions. We acknowledge that SZZ has known limitations in precision and recall ([Rosa et al., 2023](#bib.bib31)), as it may miss a small number of latent bugs or produce false links. We therefore employed it solely as a supplementary heuristic to improve dataset completeness, not as a sole ground-truth source, and all results were subject to subsequent manual verification (Step 4). Moreover, since all tools are evaluated on the same SZZ-augmented dataset, this still provides a fair basis for comparative analysis. Next, to prevent models from relying on superficial heuristics, we addressed the statistical differences between Change-PRs and Clean-PRs. As highlighted in prior work ([Zeng et al., 2021](#bib.bib37)), models can exploit simple metrics (e.g., lines of code changed, number of commits) to distinguish between PRs needing changes and those that do not, without truly understanding the code. To mitigate this and create a more robust evaluation, we performed stratified sampling ([Wikipedia, 2025b](#bib.bib49)) on the Clean-PRs to align their distribution of key statistical properties with that of the Change-PRs. While this intentionally deviates from the natural data distribution, it creates a more challenging and meaningful benchmark that forces models to engage with code logic. #### Step 4: Manual Verification and Refinement After the preceding steps, approximately 3,500 PRs remained. Given the significant effort required for manual annotation, we randomly sampled 1,000 Change-PRs and 1,000 Clean-PRs for rigorous manual verification. This process was conducted by five experienced graduate students in computer science, each holding at least a Master’s degree and possessing over two years of software development experience. Specifically, we assigned 800 PRs to each annotator, organized such that each PR was independently annotated by exactly two annotators. The primary goal of this manual verification was to check the correctness of each LLM-identified change-action label, ensuring it conformed to our definition and could be accurately classified into one of the 11 types in Table [1](#S2.T1 "Table 1 ‣ 2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). Additionally, annotators filtered out Change-PRs containing only “trivial change-actions”, which we define as changes related solely to formatting or documentation updates (types E.1 and E.2 in Table [1](#S2.T1 "Table 1 ‣ 2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")). This was a deliberate choice to create a more challenging and realistic benchmark, as a model excelling on SWR-Bench by being forced to address substantive issues is more likely to be effective in real-world scenarios. To ensure annotation reliability, we calculated Cohen’s Kappa ([Wikipedia, 2025a](#bib.bib50)) for each pair of independent reviews, yielding a score of 66.08. While this indicates “substantial agreement,” it also reflects the inherent difficulty and ambiguity of identifying specific change-actions from PR discussions. To mitigate the impact of this ambiguity on dataset quality, all disagreements were resolved through discussion between the two assigned annotators to ensure the correctness of the final labels. This rigorous verification process was crucial for ensuring the high quality and reliability of SWR-Bench. #### Dataset Statistics The final SWR-Bench dataset consists of 500 Change-PRs and 500 Clean-PRs, randomly selected from manually verified and corrected PRs. This balanced composition is vital for robustly evaluating ACR tools, particularly their false positive rate, a common critique of existing automated review systems ([Tanzil et al., 2024](#bib.bib42); [Sun et al., 2025](#bib.bib7)). Each SWR-Bench instance offers a comprehensive dataset for the evaluation of ACR tools, capturing the state of a PR right before its first manual review. This dataset is composed of three key components: (1) the metadata data of the PR (title, description, commits made before the first manual review), (2) ground-truth change-action (only for those made prior to the first manual review) information for Change-PRs, detailing their type, description, and associated commit SHA, and (3) codebase checkpoint information, allowing for the reconstruction of the full PR codebase at the relevant commit. This latter feature is particularly important for agent-based ACR tools that require an interactive environment. Table [3](#S3.T3 "Table 3 ‣ Dataset Statistics ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation") presents the overall dataset statistics. We find that, on average, Change-PRs have similar numbers of modified files and lines of code compared to Clean-PRs. These similarities underscore the effectiveness of our resampling strategy (Step 3) in ensuring that Clean-PRs are not trivially distinguishable from Change-PRs based on such coarse-grained metrics. Furthermore, we analyzed the change-action type distribution (Figure [4](#acmlabel4 "Figure 4 ‣ Dataset Statistics ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")). In Raw-PRs (before manual verification and filtering), functional changes (e.g., logic errors, performance issues) constituted less than 15% of all identified changes, with a predominance of more evolutionary changes like documentation or style adjustments. After our manual verification process, which explicitly filtered out PRs with only trivial change-actions, the proportion of functional changes in SWR-Bench increased to 31.8%, while the proportion of simple textual changes was intentionally reduced. Additionally, earlier quality filtering steps (e.g., majority voting) also contribute to distribution shifts from real-world data. We argue that such deviations are acceptable, as our goal is to provide a fair and consistent evaluation scenario for ACR tools. A method that performs well on this intentionally difficult distribution, where simple cases are underrepresented, is highly likely to generalize well to practical scenarios. PR Type Count Avg. Commits Avg. Files Avg. Lines + Avg. Lines - Avg. Change Points Change-PR 500 4.05 6.29 123.64 60.69 1.90 Clean-PR 500 3.25 6.85 112.60 67.22 0.00 All 1000 3.65 6.57 118.12 63.96 0.95 Table 3. Overall statistics of the SWR-Bench. “Avg.” denotes average values of all instances. Figure 4. Change type distribution of SWR-Bench. ### 3.2. Evaluation Methodology Evaluating the output of ACR tools presents unique challenges. The report generated by a model is typically unstructured natural language text, containing various comments, or identified issues. Directly comparing this with a structured list of ground truth change-actions is difficult. As discussed (Section [2](#S2 "2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")), traditional text similarity metrics fail to capture semantic accuracy and practical value in code review comments ([Jiang et al., 2025](#bib.bib16); [Lu et al., 2025](#bib.bib17)). Consequently, we turn to solutions utilizing large language models as evaluation aids (LLM-as-Judge) ([Chen et al., 2025](#bib.bib40); [Wang et al., 2024](#bib.bib35)). However, we also recognize that traditional LLM-as-Judge methods often rely on the LLM’s subjective judgment (e.g., scoring comment quality), which can introduce bias and inconsistency ([Zhu et al., 2025](#bib.bib18); [Shi et al., 2024](#bib.bib19); [Gu et al., 2024](#bib.bib20); [Thakur et al., 2024](#bib.bib21)). To mitigate this, we designed an objective LLM evaluation framework that leverages ground-truth change-actions referencing. Instead of subjective scoring or ranking, our LLM performs a fact-based matching task: determining if issues in the model’s report correspond to predefined ground truth change-actions. ![Refer to caption](https://arxiv.org/html/2509.01494v2/evaluation.png) Figure 5. The objective LLM-based evaluation pipeline for predicted review report. Our evaluation process, depicted in Figure [5](#acmlabel5 "Figure 5 ‣ 3.2. Evaluation Methodology ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), unfolds as follows: The inputs to the evaluation pipeline are the code review report generated by the ACR tool and the corresponding ground truth change-actions from our SWR-Bench. First, an evaluation LLM parses the ACR tool’s unstructured report to extract distinct predicted change-actions (a single report generated by the ACT tool contains multiple change-actions). For each predicted change-action, the LLM is also prompted to determine its change-type (from 11 types in Table [1](#S2.T1 "Table 1 ‣ 2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")). These latter two are subjective LLM assessments but provide useful metadata about the review report. Next, the core objective evaluation involves the LLM performing a matching task. It compares each extracted predicted change-action against the PR’s set of ground truth change-actions. The LLM determines, for each ground truth change-action, whether it has been successfully “hit” (i.e., semantically identified) by one or more predicted change-actions. Based on this explicit matching information, we can define TP, FP, and FN. These definitions then enable the straightforward calculation of standard performance metrics like P​r​e​c​i​s​i​o​n=T​PT​P+F​PPrecision=\\frac{TP}{TP+FP}, R​e​c​a​l​l=T​PT​P+F​NRecall=\\frac{TP}{TP+FN}, and F​1=2×P​r​e​c​i​s​i​o​n×R​e​c​a​l​lP​r​e​c​i​s​i​o​n+R​e​c​a​l​lF1=2\\times\\frac{Precision\\times Recall}{Precision+Recall}: - • True Positives (TP): Ground-truth change-actions successfully hit by at least one predicted change-action. - • False Positives (FP): The number of predicted change-actions that do not hit any ground truth change-action. - • False Negatives (FN): The number of ground truth change-actions not hit by any predicted change-actions. Furthermore, as discussed in Section [3.1](#S3.SS1 "3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), functional changes constitute a critical, but smaller portion of the change-actions in SWR-Bench. To provide a more nuanced understanding of an ACR tool’s ability to detect these functional issues, we report performance metrics (P​r​e​c​i​s​i​o​nPrecision, R​e​c​a​l​lRecall, and F​1F1) specifically for functional changes in addition to the overall metrics. This specialized calculation considers only predicted change-actions that the evaluation LLM has typed as functional change and ground-truth change-actions with functional type. This objective, ground-truth-referenced LLM evaluation approach allows for a more accurate and in-depth assessment of ACR tools, overcoming traditional method limitations. The reliability of this evaluation approach is further validated through manual verification in later experiments. ## 4. Evaluation and Study ### 4.1. Research Questions Our study is guided by the following research questions: - • RQ1: How reliable is the proposed evaluation methodology? - • RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? - • RQ3: How can the performance of automated code review tools be improved? ### 4.2. Subjects of Study #### 4.2.1. Large Language Models (LLMs) To comprehensively evaluate current LLMs on ACR tasks, we selected a diverse range of models, including closed-source LLMs like the GPT series (o3, o4-mini, GPT-4o, GPT-5), Gemini series (Gemini-2.5-Pro, Gemini-2.5-Flash), and Claude series (Claude-3.7-Sonnet, Claude-4-Sonnet, Claude-4-Opus). We also include open-source LLMs such as DeepSeek series (DeepSeek-R1, DeepSeek-V3) and Qwen2.5 series (Qwen2.5-Chat-7B/14B/32B, Qwen2.5-R1-7B/14B/32B). #### 4.2.2. Automated Code Review (ACR) Baselines For a comprehensive evaluation, we benchmark against representative ACR tools and methodologies: - • LLM-Review (Prompting Baseline): Our straightforward baseline that directly feeds PR metadata and diffs to an LLM via a simple prompt, measuring vanilla review capabilities. - • SWR-Agent (Agentic Baseline): Inspired by SWE-Agent ([Yang et al., 2024a](#bib.bib32)), we built this baseline to adapt agentic workflows for code review, allowing the LLM to use tools, explore the codebase, and gather context. - • CR-Agent ([Tang et al., 2024](#bib.bib28)): A multi-agent tool using two interacting agents (focusing on formatting and functional defects, respectively) that discuss and debate to produce a final report. - • Hybrid-Review ([Jaoua et al., 2025b](#bib.bib33)): Extends the simple prompting baseline by additionally injecting static analysis reports as supplementary context. - • PR-Review ([Qodana AI, 2025](#bib.bib39)): While similar to LLM-Review, PR-Review uses more sophisticated prompt engineering. Specifically, it first consolidates code diffs from multiple commits to form the input. When faced with context length limitations, it prioritizes files based on importance, potentially excluding less critical files. Finally, it instructs the LLM to output the review report in a specific structured format, categorizing suggestions into areas such as test-related defects, security-related defects, and other areas for improvement. - • Code-Reviewer ([Li et al., 2022](#bib.bib14)): A traditional hunk-level method using a fine-tuned CodeT5 model ([Wang et al., 2021](#bib.bib47)). We adapt our PRs into diff hunks for its input and concatenate its generated hunk-level comments to form a full PR review. - • Llama-Reviewer ([Lu et al., 2023](#bib.bib10)): Fine-tunes a Llama-Base-7B model ([Touvron et al., 2023](#bib.bib51)) for hunk-level review. We evaluate it using the same hunk-to-PR concatenation protocol as Code-Reviewer. Lastly, to ensure fair comparison, all evaluated ACR tools used their official default configurations. ### RQ1: How reliable is the proposed evaluation methodology? ![Refer to caption](https://arxiv.org/html/2509.01494v2/heatmap_combined.png) Figure 6. Validation of the proposed objective evaluation methodology. (a) and (b) show the inter-rater agreement (Cohen’s Kappa) for Hit identification and Type classification, respectively. (c) compares the agreement (Cohen’s Kappa) of two LLM-as-Judge methodology and BLEU judge methodology with human preferences in a pairwise comparison task. To rigorously assess the reliability of our proposed LLM-as-Judge methodology, we conducted two validation experiments. First, we measured inter-rater and inter-model agreement for our core evaluation tasks. We randomly selected 100 code review reports from RQ2 and had them independently annotated by three human experts and three distinct LLM judges (Gemini-2.5-Pro, Gemini-2.5-Flash and Gemini-2.5-Flash with majority voting (nn=3)). We measured agreement using Cohen’s Kappa for identifying “Hit” of each ground-truth change-actions and classifying their “Type”. Hit Agreement (Figure [6](#acmlabel6 "Figure 6 ‣ RQ1: How reliable is the proposed evaluation methodology? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")a): The agreement for our primary "Hit" metric was exceptionally high, with Kappa scores ranging from 70.6 to 86.7 across all human-human, human-LLM, and LLM-LLM pairs. This indicates almost perfect agreement and confirms that identifying "Hits" is an objective task. The high consistency between Gemini-2.5-Pro and Gemini-2.5-Flash (Kappa = 86.7) demonstrates that the evaluation is stable across different models, implying that the results are not sensitive to the choice of a specific LLM judge, and therefore are unlikely to be sensitive to the randomness of multiple runs. Furthermore, the majority voting variant achieved Kappa scores comparable to the single-call Gemini-2.5-Flash and Gemini-2.5-Pro, indicating that majority voting provides only marginal improvement for this already highly consistent task. Type Agreement (Figure [6](#acmlabel6 "Figure 6 ‣ RQ1: How reliable is the proposed evaluation methodology? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")b): For "Type" classification, human-LLM kappa scores ranged from 63.4 to 87.6. The agreement between LLMs and humans was comparable to the agreement among humans themselves (human-human Kappa: 60.7 to 75.9). This suggests that the LLM judge’s performance on Type classification is consistent with that of human experts. Similarly, the majority voting variant of Gemini-2.5-Flash showed no significant improvement over the single-call version for Type classification, further confirming that a single LLM call is sufficient for reliable evaluation. Second, to demonstrate the superiority of our objective methodology over traditional approaches, we compared it against several alternative evaluation methods. We randomly selected 1,000 pairs of review reports from RQ2, where each pair containing two distinct reviews for the same PR, and asked three human experts to choose the better report in each pair. We then tasked the following automated methods (both using Gemini-2.5-Flash) to perform the same comparison: 1) LLM-Hit-Judge (ours), our proposed method, which prefers the report that achieves more “Hits” on ground-truth change-actions with fewer false positives. 2) LLM-Score-Judge, a traditional approach where the LLM assigns a holistic quality score (1-10) to each report, with the higher-scoring report being preferred. 3) Bleu ([Papineni et al., 2002](#bib.bib15)) and Rouge-L ([Lin, 2004](#bib.bib52)), traditional text similarity metrics that compare the generated review report against the ground-truth review discussion text, with the report achieving a higher score being preferred. As shown in Figure [6](#acmlabel6 "Figure 6 ‣ RQ1: How reliable is the proposed evaluation methodology? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")c, LLM-Hit-Judge achieved substantial agreement with human preferences, with Cohen’s Kappa scores ranging from 52.8 to 62.0. This level of agreement is on par with the inter-human agreement (Kappa range: 56.3 to 62.6). In contrast, the baseline LLM-Score-Judge showed significantly lower agreement with humans (Kappa range: 31.0 to 44.9), highlighting the unreliability of subjective scoring. Most notably, the traditional text similarity metrics performed drastically worse. BLEU achieved Kappa scores ranging from only -35 to -41 with human preferences, indicating agreement even worse than chance. This result validates that our objective, change-action-based evaluation aligns more closely with expert human judgment. Ultimately, Gemini-2.5-Pro, Gemini-2.5-Flash and Majority-Voting demonstrated commendable and similar agreement levels with human evaluators, particularly for the primary “Hit” metric. Given that Gemini-2.5-Flash demonstrated strong, reliable performance comparable to Gemini-2.5-Pro at a significantly lower cost (approx. $1.57 for a full SWR-Bench evaluation), we selected it for all subsequent experiments. *Conclusion 1: Our objective, change-action-based evaluation methodology is highly reliable, stable across different LLM judges (with or without majority voting), and demonstrates superior alignment with human expert judgment compared to both traditional subjective scoring methods and text similarity metrics.* ### RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? Table 4. Evaluation of studied ACR approaches on SWR-Bench. The table shows hit-based Precision, Recall, and F1. Avg. Count indicate the average number of predicted change-actions, while Avg. FP Count indicate the average false positive number. ACR Tools LLMs Overall Change-Actions Functional Change-Actions Text Similarity Precision Recall F1 Precision Recall F1 Bleu Rouge-L LLM-Review Gemini-2.5-Pro 8.02 21.29 11.65 2.47 2.28 22.60 26.86 24.54 0.35 0.27 0.58 9.64 Claude-3.7-Sonnet 9.85 14.31 11.67 1.35 1.22 24.37 16.02 19.33 0.20 0.15 0.47 10.53 DeepSeek-R1 9.79 25.58 14.16 2.44 2.20 15.53 32.02 20.92 0.61 0.52 0.50 10.19 Mean 9.22 20.39 12.49 2.09 1.90 20.83 24.97 21.60 0.39 0.31 0.52 10.12 SWR-Agent Gemini-2.5-Pro 9.93 19.14 13.07 1.80 1.62 18.11 25.84 21.30 0.42 0.35 1.95 11.45 Claude-3.7-Sonnet 8.29 22.72 12.15 2.55 2.34 17.92 30.90 22.68 0.51 0.42 1.46 11.77 Mean 9.11 20.93 12.61 2.18 1.98 18.02 28.37 21.99 0.47 0.38 1.31 11.11 CR-Agent Gemini-2.5-Pro 6.97 17.63 9.98 2.35 2.18 16.26 26.55 20.17 0.48 0.40 0.74 7.43 Claude-3.7-Sonnet 6.30 17.17 9.21 2.54 2.38 18.06 22.78 20.15 0.38 0.31 1.00 10.22 DeepSeek-R1 5.42 19.50 8.48 3.36 3.17 11.45 29.44 16.49 0.77 0.68 1.00 9.87 Mean 6.23 18.10 9.22 2.75 2.58 15.26 26.26 18.94 0.54 0.47 0.91 9.17 Hybrid-Review Gemini-2.5-Pro 2.83 20.44 4.97 6.64 6.42 11.09 29.28 16.08 0.80 0.71 0.77 7.28 Claude-3.7-Sonnet 2.20 11.23 3.68 4.63 4.52 7.95 15.64 10.55 0.59 0.54 0.39 5.30 DeepSeek-R1 3.33 28.44 5.96 7.95 7.60 10.10 47.31 16.65 1.47 1.31 1.02 8.41 Mean 2.79 20.04 4.87 6.41 6.18 9.71 30.74 14.43 0.95 0.85 0.73 7.00 PR-Review Gemini-2.5-Pro 16.65 23.18 19.38 1.32 1.10 19.38 40.72 26.26 0.65 0.52 0.40 8.92 Claude-3.7-Sonnet 14.90 23.50 18.23 1.50 1.27 14.72 40.32 21.56 0.86 0.74 0.36 8.38 DeepSeek-R1 14.61 25.50 18.58 1.66 1.41 15.55 50.62 23.79 1.06 0.89 0.31 7.70 Mean 15.39 24.06 18.73 1.49 1.26 16.55 43.89 23.87 0.86 0.72 0.36 8.34 Code-Reviewer 4.19 11.35 6.13 2.55 2.44 9.46 14.67 11.50 0.47 0.42 10.25 22.14 Llama-Reviewer 4.28 23.08 7.22 4.91 4.70 8.62 37.02 13.98 1.32 1.20 7.38 22.89 In this RQ, we first investigate the performance of studied ACR approaches on SWR-Bench, and each approaches was evaluated using three powerful LLMs, with the exception of SWR-Agent, which was not run with DeepSeek-R1 due to the latter’s lack of function call capabilities. For each ACR technique, we also report the mean performance metrics across the LLMs it was paired with. The detailed results are summarized in Table [4](#S4.T4 "Table 4 ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). The results in Table [4](#S4.T4 "Table 4 ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation") show that SOTA ACR techniques, when paired with SOTA LLMs, are not yet ready for real-world code review deployment based on their performance on SWR-Bench. For instance, the top-performing combination, PR-Review leveraged with Gemini-2.5-Pro, achieved an F​1F1 score of only 19.38%. A primary factor limiting higher F​1F1 scores for all techniques is their low precision, indicative of a high false positive rate. Specifically, the Avg. FP Count metric reveals that these approaches frequently generate multiple invalid suggestions per pull request, with some combinations (e.g., Hybrid-Review with DeepSeek-R1) producing over 7 false positives on average. This issue is more severe for other four ACR techniques, all of which exhibited precision scores below 10%. This implies that while these ACR techniques can identify some valuable issues (or “change-actions”), their overall effectiveness is severely undermined by an excessive number of false positives. Consequently, developers would need to invest considerable additional effort in verifying the validity of the generated reports, hindering practical adoption. This observation aligns with findings from previous empirical studies ([Tanzil et al., 2024](#bib.bib42)) and further validates the efficacy of SWR-Bench as a benchmark that closely mirrors realistic code review scenarios. *Conclusion 2: Current ACR approaches, even when augmented with advanced LLMs, demonstrate limited performance on the SWR-Bench, primarily constrained by high false positive rates. This significantly hinders their immediate applicability in practical code review workflows.* ##### Performance Analysis by ACR Tools A granular analysis of F​1F1 scores reveals distinct performance tiers among the ACR tools (Table [4](#S4.T4 "Table 4 ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")). PR-Review, leveraging meticulous prompt engineering, achieved the highest overall performance (O​v​e​r​a​l​l​-​F​1Overall\\text{-}F1: 18.73%). This significantly surpassed both SWR-Agent and LLM-Review (approx. 12%). Meanwhile, other approaches showed significant limitations. The multi-agent CR-Agent performed poorly (O​v​e​r​a​l​l​-​F​1Overall\\text{-}F1: 9.22%), likely due to interaction overhead and error propagation—known challenges in multi-agent systems ([Cemri et al., 2025](#bib.bib43)). The worst performer, Hybrid-Review (O​v​e​r​a​l​l​-​F​1Overall\\text{-}F1: 4.87%), suffered from extremely low precision (with the highest mean Avg. FP Count of 6.18), indicating that simply integrating raw static analysis outputs is ineffective without careful processing. Furthermore, the poor performance of traditional fine-tuned models (Code-Reviewer: 6.13% F​1F1, Llama-Reviewer: 7.22% F​1F1) is also notable. It highlights a critical flaw in prior approaches: models optimized for isolated code hunks fail at the holistic, contextual task of reviewing a full pull request. This underscores the value of SWR-Bench in providing a more realistic evaluation and guiding research toward end-to-end solutions. Table [4](#S4.T4 "Table 4 ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation") also reports Bleu ([Papineni et al., 2002](#bib.bib15)) and Rouge-L ([Lin, 2004](#bib.bib52)) scores. Interestingly, text similarity inversely correlates with F​1F1: hunk-level models achieve high Bleu but low F​1F1, whereas PR-level tools (e.g., PR-Review) show the opposite. Our manual analysis reveals this stems from formatting differences rather than review quality. PR-Review generates structured, detailed reports that differ from brief, colloquial human ground truths, yielding low Bleu despite capturing actual issues (high F​1F1). Conversely, hunk-level models mimic human chat, often generating short, ambiguous comments that inflate Bleu but fail to identify actionable defects (low F​1F1). Consistent with RQ1 (Figure [6](#acmlabel6 "Figure 6 ‣ RQ1: How reliable is the proposed evaluation methodology? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")c), this format bias confirms that Bleu and Rouge-L are unreliable proxies for review quality, validating our change-action hit-based evaluation. Lastly, comparing PR-Review and LLM-Review, both of which operate via a single-turn interaction with the LLM, PR-Review’s superior performance can be attributed to its more sophisticated prompt engineering, these refinements appear to significantly reduce false positives, thereby enhancing precision and F​1F1 score. Collectively, these results underscore that refined prompt engineering, as demonstrated by PR-Review, is currently the most effective strategy for improving ACR capabilities. Agent-based approaches require further research to overcome their architectural challenges and unlock their potential. The failure of models trained on isolated code hunks validates the necessity of holistic, end-to-end benchmarks like SWR-Bench to guide the field towards more practical solutions. *Conclusion 3: PR-Review, leveraging meticulous prompt engineering, achieves the best performance on SWR-Bench, underscoring the promise of prompt engineering for improving efficacy.* Table 5. PR-Review (Gemini-2.5-Pro) performance vs. number of ground-truth changes (NN). NN Count Precision Recall F1 1 266 30.18 38.35 33.77 2 139 35.05 24.46 28.81 3 56 34.18 16.07 21.86 4 17 29.63 11.76 16.84 5+ 22 44.12 8.88 14.78 Table 6. Performance across different change types on SWR-Bench. Change Type Precision Recall F1 Avg. Count E.1.1 Textual Changes 15.85 13.03 14.30 0.17 E.1.2 Language Features 8.28 7.59 7.85 0.02 E.2 Visual Representation 12.21 4.39 6.05 0.02 E.3.1 Organization 24.94 12.56 16.45 0.05 E.3.2 Solution Approach 12.66 19.15 15.21 0.37 F.1 Interface 16.97 40.83 23.55 0.12 F.2 Logic 17.28 54.60 26.20 0.35 F.3 Resource 15.89 53.45 24.26 0.10 F.4 Check 13.36 39.70 19.60 0.12 F.5 Support 15.95 35.80 21.74 0.16 F.6 Larger Defects 53.31 18.88 27.65 0.01 ##### Performance in Multi-Change PR Scenarios A distinctive feature of SWR-Bench is that each Change-PR contains multiple ground-truth change-actions (on average 1.90 per PR, as shown in Table [3](#S3.T3 "Table 3 ‣ Dataset Statistics ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")). To understand how baselines perform in such scenarios, we analyzed the relationship between the number of ground-truth change-actions (NN) in a PR and the ACR tool’s performance. As detailed in Table [6](#S4.T6 "Table 6 ‣ Performance Analysis by ACR Tools ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation") (using PR-Review with Gemini-2.5-Pro as a representative example), we observed that as the number of change-actions per PR increases, the Recall tends to decrease sharply (from 38.35% for N=1N=1 to 8.88% for N≥5N\\geq 5), indicating that tools struggle to identify all issues in more complex PRs. Conversely, Precision remains relatively stable (fluctuating roughly between 29.63% and 44.12%), suggesting that the quality of individual predictions does not degrade significantly with PR complexity. This analysis highlights that multi-change PRs remain a significant challenge. *Conclusion 4: ACR tools struggle to comprehensively review complex PRs. As the number of issues in a PR increases, ACR tools experience a sharp drop in Recall while Precision remains stable, highlighting their inability to comprehensively review multi-change PRs.* ##### Performance Analysis by Change Type To gain a deeper understanding of how ACR tools handle different types of change-actions, we further analyzed the average performance of PR-Review across various change types defined in Table [1](#S2.T1 "Table 1 ‣ 2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). The results, disaggregated by change type, are shown in Table [6](#S4.T6 "Table 6 ‣ Performance Analysis by ACR Tools ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). A key observation is that PR-Review demonstrates significantly stronger detection capabilities for functional change-actions compared to evolutionary change-actions. Specifically, for the F.2 Logic change, PR-Review achieved an F​1F1 score of 26.20%, with most functional change types also yielding F​1F1 scores above 21%. In stark contrast, the highest F​1F1 score for an evolutionary change type, E.3.1 Organization, was merely 16.45%. We hypothesize that this disparity arises from the inherent nature of these change categories. Evolutionary changes often represent optional or stylistic improvements, where the criteria for what constitutes a necessary change can vary significantly between human reviewers ([Beller et al., 2014](#bib.bib36)) and LLMs. For instance, with E.3.2 Solution Approach, which was the most prevalent evolutionary change type, one human reviewer might suggest an alternative implementation they deem superior, while another might find the current approach acceptable. Consequently, accurately detecting such optional evolutionary changes presents a considerable challenge. Based on these findings, we recommend that future development of ACR tools prioritize enhancing the detection capabilities for functional changes to improve the precision for functional change-actions. Furthermore, ACR tools could consider presenting reports for functional and evolutionary changes separately. *Conclusion 5: ACR tools demonstrate better performance in detecting functional changes, which likely due to the subjective nature of many evolutionary changes. Accordingly, future ACR tools could prioritize robust detection of functional changes to enhance practical utility.* Table 7. Categories and Proportions of Common False Positives Categories Description Proportion Lack of Contextual Understanding The tool applies programming rules rigidly and in isolation, ignoring crucial logical consistency established by the surrounding code within the same project or commit. 48% Over-sensitivity to Modification The tool lacks an understanding of the developer’s intent and tends to treat any large-scale code modification or refactoring as a potential risk. 17% Vague and Unactionable Feedback The review feedback provided by the tool is often too broad and generic, lacking specific, actionable steps. 16% Lack of Domain Knowledge The tool fails to recognize specialized coding conventions specific to a project or technical domain (i.e., domain knowledge), causing it to flag correct and idiomatic code as anomalous. 13% Misjudgment of "Anti-best-practices" The tool relies on superficial heuristics and makes incorrect judgments when developers intentionally deviate from conventional best practices to achieve higher-level goals like test effectiveness or readability. 3% Other A catch-all for other rare causes. 3% ##### False Positive Analysis While our previous analysis highlighted the practical utility of ACR tools is critically undermined by the generation of false positives, we therefore performed a qualitative analysis to diagnose the root causes of these inaccuracies. We manually inspected 100 functional false positives, randomly sampled from a total of 1,101 false positives generated by PR-Review (based on Gemini-2.5-Pro), categorizing them as summarized in Table [7](#S4.T7 "Table 7 ‣ Performance Analysis by Change Type ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). The taxonomy was derived through an iterative process based on Grounded Theory ([Cemri et al., 2025](#bib.bib43)). Three authors independently examined and labeled each of the 100 sampled false positives, assigning initial descriptive codes to characterize the root cause. The authors then collaboratively discussed and reconciled their labels through multiple rounds of discussion, iteratively merging, splitting, and refining categories until a stable set of themes emerged. This process resulted in the five major categories and one “Other” category presented in Table [7](#S4.T7 "Table 7 ‣ Performance Analysis by Change Type ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). As the table shows, the most prevalent issue is a Lack of Contextual Understanding (48%). These errors typically arise from the LLM’s insufficient contextual reasoning, leading to a misunderstanding of the code. For example, in astropy/astropy-1199, the tool warned that parameter unpacking (\*result) could raise a TypeError, but it failed to recognize that the variable had been explicitly constructed as a tuple within the same commit, thus ensuring the operation’s safety. Other significant sources of error are Over-sensitivity to Modification (17%) and Vague and Unactionable Feedback (16%). These stem from the model’s cautious bias, leading to low-value suggestions like "verifying" every new logic in scikit-learn/scikit-learn-25025, offering no specific flaw or actionable advice. Such issues could be mitigated through prompt engineering. Additionally, a Lack of Domain Knowledge (13%) contributes to false positives when the LLM is unaware of project-specific conventions. In sympy/sympy-17801, the idiomatic expression is S.true, used for checking symbolic boolean values, was incorrectly flagged, even though it is the standard practice in the library to avoid a TypeError. This type of error could potentially be addressed by providing additional domain knowledge in the prompt. In summary, these findings demonstrate that false positives arise from a fundamental inability to comprehend developer intent, code context, and domain-specific knowledge. Future work must therefore enhance the reasoning abilities of LLMs and optimize prompts to generate more accurate, actionable feedback. *Conclusion 6: Current ACR tools struggle to grasp three critical elements: developer intent, code context, and domain knowledge. Addressing this requires advancing LLMs’ core reasoning capabilities and employing better prompting strategies to improve the quality of review reports.* Table 8. LLMs performance evaluation for code review with PR-Review on SWR-Bench. LLMs Overall Change-Actions Functional Change-Actions Precision Recall F1 Precision Recall F1 Reasoning LLM Gemini-2.5-Pro 16.65 23.18 19.38 1.32 19.38 40.72 26.26 0.65 Gemini-2.5-Flash 16.88 13.91 15.25 0.78 17.92 24.42 20.67 0.41 DeepSeek-R1 14.61 25.5 18.58 1.66 15.55 50.62 23.79 1.06 GPT-o3 14.05 25.58 18.13 1.73 14.41 50.78 22.45 1.13 GPT-5 14.69 35.93 20.85 2.32 14.87 65.03 24.2 1.43 Qwen-2.5-R1-32B 11.69 20.86 14.98 1.69 12.65 38.19 19 0.93 Qwen-2.5-R1-14B 13.21 20.13 15.95 1.45 15.01 40.94 21.96 0.87 Qwen-2.5-R1-7B 6.83 8.33 7.51 1.16 6.8 11.18 8.46 0.5 Standard LLM Claude-4-Opus 14.94 19.68 16.99 1.25 16.02 37.54 22.45 0.74 Claude-4-Sonnet 13.84 20.76 16.61 1.42 14.81 39.45 21.54 0.87 Claude-3.7-Sonnet 14.9 23.5 18.23 1.5 14.72 40.32 21.56 0.86 GPT-4o 14.13 27.78 18.73 1.86 15.61 44.48 23.11 0.85 DeepSeek-V3 15.49 20.17 17.52 1.22 16.15 33.56 21.81 0.61 Qwen-2.5-32B 11.43 17.28 13.76 1.44 13.68 29.71 18.73 0.68 Qwen-2.5-14B 8.67 9.38 9.01 1.03 13.65 17.68 15.41 0.4 Qwen-2.5-7B 8.87 16.86 11.63 1.8 10 24.6 14.22 0.77 ##### Impact of LLMs and Reasoning Enhancement Following PR-Review’s superior performance, we evaluated its efficacy on SWR-Bench with diverse LLMs, including variants with and without reasoning-enhancement training (Table [8](#S4.T8 "Table 8 ‣ False Positive Analysis ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")). Overall, LLM performance with PR-Review remains suboptimal for practical code review (highest O​v​e​r​a​l​l​-​F​1Overall\\text{-}F1 19%). Within this context, we first noted a significant divergence: the LLM performance ranking on SWR-Bench does not consistently mirror trends observed in other SE benchmarks (e.g., SWE-Bench ([Jimenez et al., 2024](#bib.bib29)) and LiveCodeBench ([Jain et al., 2025](#bib.bib34))). This discrepancy underscores SWR-Bench’s value in capturing unique code review challenges and speculatively points to a potential misalignment between current LLM training/optimization and code review’s specific demands, where overemphasis on other SE tasks might inadvertently reduce code review proficiency. Turning to the impact of reasoning enhancement training, we found further nuances. Models like Gemini-2.5-Pro and DeepSeek-R1, known for their reasoning capabilities, generally performed better. The Qwen-2.5 series provided a clear illustration: its standard versions (without specific reasoning enhancement) yielded lower F​1F1 scores (max 13.76%). However, their reasoning-enhanced counterparts showed marked improvement. Specifically, Qwen-2.5-R1-14B achieved an F​1F1 of 15.95% (Qwen-2.5-R1-7B was an exception due to output formatting issues). This strongly suggests that reasoning-enhancement training is a crucial factor for improving LLM effectiveness in code review. *Conclusion 7: Current LLMs underperform for practical review on SWR-Bench; however, reasoning-enhanced models perform better, highlighting reasoning-enhanced as a crucial advancement path.* ### RQ3: How can the performance of automated code review tools be improved? (a) Overlap across different models. (b) Overlap across multiple runs of the same model. Figure 7. Venn diagrams illustrating the overlap of identified change-actions. Conclusion 7 indicates reasoning-enhanced LLMs perform better, the overall practical utility of current ACR approaches remains limited. This prompted a closer examination of the nature of LLM-generated reviews, particularly their reliability and consistency. Specifically, we questioned whether current LLMs provide stable and consistent feedback across different invocations or when compared to other models. To investigate this, we conducted an analysis into the overlap of specific ground truth change-actions identified by different LLMs and by multiple runs of the same LLM using Venn diagrams (Figures [7(a)](#S4.F7.sf1 "In Figure 7 ‣ RQ3: How can the performance of automated code review tools be improved? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation") and [7(b)](#acmlabel7 "In Figure 7 ‣ RQ3: How can the performance of automated code review tools be improved? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")). This analysis revealed considerable variability and instability in the sets of successfully identified defects across these models and runs. Notably, for different LLMs only 36 successfully identified change-actions overlapped (Figure [7(a)](#S4.F7.sf1 "In Figure 7 ‣ RQ3: How can the performance of automated code review tools be improved? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")), for the same LLM over five independent runs, only 27 successfully identified change-actions overlapped (Figure [7(b)](#acmlabel7 "In Figure 7 ‣ RQ3: How can the performance of automated code review tools be improved? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")). This low consistency suggests unstable detection for most change-actions, implying significant randomness in LLM identification. While this variability might partly stem from the subjective nature of many evolutionary changes, it strongly indicates that LLMs’ current grasp of code review nuances may be superficial, with detections sometimes attributable to stochastic factors rather than deep comprehension. ![Refer to caption](https://arxiv.org/html/2509.01494v2/agg.png) Figure 8. The overall workflow of the proposed Multi-Review approach. This observed variability and the implied limitations of single-pass reviews directly motivated our exploration into whether integrating diverse review outputs could yield a more comprehensive and reliable final report. To this end, building upon PR-Review, we propose an enhanced approach named Multi-Review (Figure [8](#S4.F8 "Figure 8 ‣ RQ3: How can the performance of automated code review tools be improved? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")). The core idea is to execute PR-Review (or another ACR tool) multiple times on the same code change to generate nn independent review reports, which are then aggregated into a final review report using an additional LLM call. Specifically, all nn reports are concatenated into a single input, and an LLM is prompted with a meta-instruction to synthesize them into a single, superior review. To validate Multi-Review’s efficacy, we conducted experiments on SWR-Bench with two aggregation strategies: 1) Self-Agg, which employs the same LLM to both generate and aggregate multiple independent review reports, testing whether a single model can overcome its own stochasticity by consolidating multiple internal opinions; and 2) Multi-Agg (Cross-Model Aggregation), which aggregates reports generated by multiple different LLMs, leveraging the complementary strengths of different models to produce a more complete and accurate review. (a) Performance of Multi-Review for varying numbers of aggregated reports (nn). (b) Cost-benefit analysis across different models and aggregation sizes (nn). Figure 9. Evaluation of the Multi-Review strategy. (a) shows the performance impact of aggregating varying numbers of reports, while (b) presents the trade-off between Overall F1 and average API cost per PR (in US dollars). We selected Gemini-2.5-Flash as a representative LLM to investigate the impact of aggregating varying numbers of review reports (n∈{0,1,3,5,10}n\\in\\{0,1,3,5,10\\}, where n=0n=0 represents the baseline PR-Review without aggregation). The results, presented in Figure [9(a)](#S4.F9.sf1 "In Figure 9 ‣ RQ3: How can the performance of automated code review tools be improved? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), demonstrate a significant performance uplift with Multi-Review. Regardless of whether aggregating results from different models or multiple runs of a single model, the F​1F1 scores showed marked improvement. Specifically, Gemini-2.5-Flash with Self-Agg (n=10n=10) achieved an F​1F1 of 21.91% (a 43.67% increase) and a R​e​c​a​l​lRecall of 30.44% (a 118.83% increase). This larger gain in R​e​c​a​l​lRecall indicates Multi-Review’s effectiveness in identifying more true defects. However, to further boost the F​1F1 score, enhancing P​r​e​c​i​s​i​o​nPrecision by reducing false positives remains a key priority. To further assess the generalizability of Multi-Review, we experimented with smaller, open-source LLMs from the Qwen-2.5 series (7B, 14B, 32B), investigating whether these models could achieve significant self-enhancement via Multi-Review in resource-constrained scenarios. Specifically, Multi-Review consistently boosted F​1F1 scores for the Qwen series. Notably, Qwen-Chat-7B Self-Agg (n=10n=10) improved its F​1F1 by 26.13% (to 14.67%), and Qwen-Chat-32B Self-Agg (n=10n=10) enhanced its F​1F1 by 19.25% (to 16.41%), allowing their aggregated results to approach the baseline PR-Review performance of some larger commercial LLMs. Furthermore, Figure [9(b)](#S4.F9.sf2 "In Figure 9 ‣ RQ3: How can the performance of automated code review tools be improved? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation") presents a cost-benefit analysis of the Multi-Review, including results with the larger Gemini-2.5-Pro model. A key finding is that Gemini-2.5-Flash Self-Agg (nn=5) achieves an F1 of 20.48%, surpassing the single-pass Gemini-2.5-Pro baseline (F1: 19.38%) at a lower API cost ($3.68E-03 vs. $5.86E-03). This demonstrates that multiple runs of a smaller, cheaper model can be more cost-effective than a single run of a larger, more expensive one. Meanwhile, Gemini-2.5-Pro Self-Agg (nn=5) achieves the highest F1 of 23.84% (a 23.01% improvement over its single-pass baseline), confirming that the aggregation strategy benefits models across the entire performance spectrum. However, performance gains exhibit diminishing returns beyond nn=5, while costs continue to scale linearly, suggesting that nn=5 represents a practical sweet spot. Regarding time efficiency, generating initial drafts takes 38.75 seconds per PR. This step is fully parallelizable, meaning its duration remains constant regardless of nn. The final aggregation step requires minimal additional time, taking only 4.67, 8.76, 12.80, and 24.35 seconds for nn=1, 3, 5, and 10 reports respectively. This low time overhead confirms the approach is practical for real-world deployment. *Conclusion 8: Multi-Review improves code review performance across difference model sizes by aggregating multiple reports. Notably, aggregating several runs of a smaller model can cost-effectively match or exceed the accuracy of a single run from a larger model. This highlights the substantial potential of improving current tools through systematic output integration.* ## 5. Discussion and Implications The findings from our evaluation on SWR-Bench provide concrete insights into automated code review. We summarize these implications as follow: Objective Evaluation through Fact Matching: As demonstrated in Conclusion 1, traditional metrics like text similarity fail to reflect true semantic quality. Furthermore, relying on subjective large language model scoring to evaluate generated reviews shows poor agreement with human expert preferences. Researchers should therefore evaluate tools using objective fact matching against predefined ground truth to verify semantic issues reliably. From Code Generation to Code Critique. Code review is fundamentally different from code generation. Generation relies on pattern completion, whereas review demands logical deduction, counterfactual reasoning, and cross-file dependency analysis. This distinction explains why models optimized primarily for code generation underperform on SWR-Bench, and why reasoning-enhanced models consistently achieve better results (Conclusion 7). Future work should move beyond prompt engineering and explore dedicated training objectives for code critique, such as reinforcement learning with process reward models that evaluate the logical rigor of identified issues rather than only the correctness of generated code. Mitigating False Positives in Practice. The high false positive rates of current tools (Conclusion 2) severely limit their practical adoption by overwhelming developers with invalid alerts. Since most false positives arise from insufficient contextual understanding and ignorance of project-specific conventions (Table [7](#S4.T7 "Table 7 ‣ Performance Analysis by Change Type ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation")), future systems must integrate historical repository data to learn implicit coding norms. Furthermore, tool designers should decouple feedback based on severity. Because models detect functional changes more reliably than evolutionary ones (Conclusion 5), presenting functional defects as mandatory requirements and evolutionary suggestions as optional recommendations will significantly enhance the practical value of automated reviews. Enhancing Review Performance through Report Aggregation: Single passes of large language models exhibit significant randomness and miss different defects across runs. Aggregating multiple independent review reports into a single final output effectively mitigates this stochasticity and significantly improves overall defect detection. Conclusion 8 demonstrates that this aggregation strategy also allows multiple runs of smaller models to match or exceed the performance of a single run from a larger and more expensive model. ## 6. Threats to Validity Internal Validity. Threats to internal validity mainly arise from potential bugs in our implementation and the accuracy of manual verification. To mitigate these threats, we have conducted a detailed review of the code, and our code has been made publicly available at ([35](#bib.bib48)) for independent verification. External Validity. Threats to external validity mainly arise from the LLMs, ACR tools, and the quality of the selected projects. To mitigate these threats, we chose representative tools based on an extensive literature review and selected 12 popular, well-maintained open-source GitHub projects. Construct Validity. The primary threat to construct validity centers on the accuracy and reliability of our objective LLM-based evaluation method. We mitigated this through human validation, which confirmed high consistency between our method’s results and human expert evaluations. ## 7. Conclusions This paper addressed critical limitations in existing automated code review benchmarks by introducing SWR-Bench, a novel benchmark featuring 1,000 real-world pull requests with full project context and an objective LLM-based evaluation. Our study on SWR-Bench revealed that while current LLM-based ACR systems generally underperform, they show better aptitude for functional error detection. To improve performance, we proposed and validated a multi-review aggregation strategy that significantly boosts F​1F1 scores. SWR-Bench, along with our findings and proposed enhancement, provides a more realistic platform and valuable insights for advancing practical ACR research and development. ## Data Availability The replication package for our study, containing the necessary source code and scripts to reproduce our experiments, is available at the anonymous repository ([35](#bib.bib48)). ## References - Badampudi et al. (2023) D. Badampudi, M. Unterkalmsteiner, and R. Britto Modern code reviews - survey of literature and practice. ACM Trans. Softw. Eng. Methodol. 32 (4), pp. 107:1–107:61. External Links: [Link](https://doi.org/10.1145/3585004), [Document](https://dx.doi.org/10.1145/3585004) Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.1](#S2.SS1.p1.1 "2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Bavota and Russo (2015) G. Bavota and B. Russo Four eyes are better than two: on the impact of code reviews on software quality. In 2015 IEEE International Conference on Software Maintenance and Evolution, ICSME 2015, Bremen, Germany, September 29 - October 1, 2015, R. Koschke, J. Krinke, and M. P. Robillard (Eds.), pp. 81–90. External Links: [Link](https://doi.org/10.1109/ICSM.2015.7332454), [Document](https://dx.doi.org/10.1109/ICSM.2015.7332454) Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Beller et al. (2014) M. Beller, A. Bacchelli, A. Zaidman, and E. Jürgens Modern code reviews in open-source projects: which problems do they fix?. In 11th Working Conference on Mining Software Repositories, MSR 2014, Proceedings, May 31 - June 1, 2014, Hyderabad, India, P. T. Devanbu, S. Kim, and M. Pinzger (Eds.), pp. 202–211. External Links: [Link](https://doi.org/10.1145/2597073.2597082), [Document](https://dx.doi.org/10.1145/2597073.2597082) Cited by: [§2.1](#S2.SS1.p2.1 "2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.1](#S2.SS1.p3.1 "2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [Table 1](#S2.T1 "In 2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [Table 1](#S2.T1.6 "In 2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§4](#S4.SSx2.SSS2.Px3.p2.1 "Performance Analysis by Change Type ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Cemri et al. (2025) M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. G. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica Why do multi-agent LLM systems fail?. CoRR abs/2503.13657. External Links: [Link](https://doi.org/10.48550/arXiv.2503.13657), [Document](https://dx.doi.org/10.48550/ARXIV.2503.13657), 2503.13657 Cited by: [§4](#S4.SSx2.SSS2.Px1.p2.1 "Performance Analysis by ACR Tools ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§4](#S4.SSx2.SSS2.Px4.p2.1 "False Positive Analysis ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Çetin et al. (2021a) H. A. Çetin, E. Dogan, and E. Tüzün A review of code reviewer recommendation studies: challenges and future directions. Sci. Comput. Program. 208, pp. 102652. External Links: [Link](https://doi.org/10.1016/j.scico.2021.102652), [Document](https://dx.doi.org/10.1016/J.SCICO.2021.102652) Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Çetin et al. (2021b) H. A. Çetin, E. Dogan, and E. Tüzün A review of code reviewer recommendation studies: challenges and future directions. Sci. Comput. Program. 208, pp. 102652. External Links: [Link](https://doi.org/10.1016/j.scico.2021.102652), [Document](https://dx.doi.org/10.1016/J.SCICO.2021.102652) Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p1.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Chen et al. (2025) J. Chen, A. Feng, Z. Zhao, J. Garza, G. Nurbek, C. Qin, A. Maatouk, L. Tassiulas, Y. Gao, and R. Ying MTBench: A multimodal time series benchmark for temporal reasoning and question answering. CoRR abs/2503.16858. External Links: [Link](https://doi.org/10.48550/arXiv.2503.16858), [Document](https://dx.doi.org/10.48550/ARXIV.2503.16858), 2503.16858 Cited by: [§2.3](#S2.SS3.p4.1 "2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§3.2](#S3.SS2.p1.1 "3.2. Evaluation Methodology ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Cihan et al. (2024) U. Cihan, V. Haratian, A. Içöz, M. K. Gül, Ö. Devran, E. F. Bayendur, B. M. Uçar, and E. Tüzün Automated code review in practice. CoRR abs/2412.18531. External Links: [Link](https://doi.org/10.48550/arXiv.2412.18531), [Document](https://dx.doi.org/10.48550/ARXIV.2412.18531), 2412.18531 Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p1.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.3](#S2.SS3.p5.1 "2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Denisov-Blanch et al. (2024) Y. Denisov-Blanch, I. Ciobanu, S. Obstbaum, and M. Kosinski Predicting expert evaluations in software code reviews. CoRR abs/2409.15152. External Links: [Link](https://doi.org/10.48550/arXiv.2409.15152), [Document](https://dx.doi.org/10.48550/ARXIV.2409.15152), 2409.15152 Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p1.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Fregnan et al. (2022) E. Fregnan, F. Petrulio, and A. Bacchelli The evolution of the code during review: an investigation on review changes. Empir. Softw. Eng. 27 (7), pp. 177. External Links: [Link](https://doi.org/10.1007/s10664-022-10205-7), [Document](https://dx.doi.org/10.1007/S10664-022-10205-7) Cited by: [§2.1](#S2.SS1.p2.1 "2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.1](#S2.SS1.p3.1 "2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [Table 1](#S2.T1 "In 2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [Table 1](#S2.T1.6 "In 2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Google (2025) Google Gemini 2.5: our most intelligent ai model. Note: [https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-thinking](https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-thinking) Cited by: [§3.1](#S3.SS1.SSSx2.p1.1 "Step 2: LLM-based change-actions Verification and Classification ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Gu et al. (2024) J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, Y. Wang, and J. Guo A survey on llm-as-a-judge. CoRR abs/2411.15594. External Links: [Link](https://doi.org/10.48550/arXiv.2411.15594), [Document](https://dx.doi.org/10.48550/ARXIV.2411.15594), 2411.15594 Cited by: [3rd item](#S1.I1.i3.p1.1 "In 1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§3.2](#S3.SS2.p1.1 "3.2. Evaluation Methodology ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Jain et al. (2025) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, External Links: [Link](https://openreview.net/forum?id=chfJJYC3iL) Cited by: [§4](#S4.SSx2.SSS2.Px5.p2.1 "Impact of LLMs and Reasoning Enhancement ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Jaoua et al. (2025a) I. Jaoua, O. B. Sghaier, and H. A. Sahraoui Combining large language models with static analyzers for code review generation. CoRR abs/2502.06633. External Links: [Link](https://doi.org/10.48550/arXiv.2502.06633), [Document](https://dx.doi.org/10.48550/ARXIV.2502.06633), 2502.06633 Cited by: [3rd item](#S1.I1.i3.p1.1 "In 1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.3](#S2.SS3.p3.1 "2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [Table 2](#S2.T2.5.1.7.1 "In 2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Jaoua et al. (2025b) I. Jaoua, O. B. Sghaier, and H. A. Sahraoui Combining large language models with static analyzers for code review generation. CoRR abs/2502.06633. External Links: [Link](https://doi.org/10.48550/arXiv.2502.06633), [Document](https://dx.doi.org/10.48550/ARXIV.2502.06633), 2502.06633 Cited by: [4th item](#S4.I2.i4.p1.1 "In 4.2.2. Automated Code Review (ACR) Baselines ‣ 4.2. Subjects of Study ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Jiang et al. (2025) Y. Jiang, H. Liu, T. Chen, F. Fan, C. Dong, K. Liu, and L. Zhang Deep assessment of code review generation approaches: beyond lexical similarity. CoRR abs/2501.05176. External Links: [Link](https://doi.org/10.48550/arXiv.2501.05176), [Document](https://dx.doi.org/10.48550/ARXIV.2501.05176), 2501.05176 Cited by: [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§3.2](#S3.SS2.p1.1 "3.2. Evaluation Methodology ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan SWE-bench: can language models resolve real-world github issues?. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, External Links: [Link](https://openreview.net/forum?id=VTF8yNQM66) Cited by: [§3.1](#S3.SS1.SSSx1.p1.1 "Step 1: Source Data Collection and Initial Filtering ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§4](#S4.SSx2.SSS2.Px5.p2.1 "Impact of LLMs and Reasoning Enhancement ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Just et al. (2014) R. Just, D. Jalali, and M. D. Ernst Defects4J: a database of existing faults to enable controlled testing studies for java programs. In International Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA - July 21 - 26, 2014, C. S. Pasareanu and D. Marinov (Eds.), pp. 437–440. External Links: [Link](https://doi.org/10.1145/2610384.2628055), [Document](https://dx.doi.org/10.1145/2610384.2628055) Cited by: [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Kulal et al. (2019) S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. Liang SPoC: search-based pseudocode to code. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett (Eds.), pp. 11883–11894. External Links: [Link](https://proceedings.neurips.cc/paper/2019/hash/7298332f04ac004a0ca44cc69ecf6f6b-Abstract.html) Cited by: [§2.3](#S2.SS3.p4.1 "2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Kumar and Chimalakonda (2024) J. Kumar and S. Chimalakonda Code review automation via multi-task federated LLM - an empirical study. CoRR abs/2412.15676. External Links: [Link](https://doi.org/10.48550/arXiv.2412.15676), [Document](https://dx.doi.org/10.48550/ARXIV.2412.15676), 2412.15676 Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Li et al. (2022) Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu, and N. Sundaresan Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, A. Roychoudhury, C. Cadar, and M. Kim (Eds.), pp. 1035–1047. External Links: [Link](https://doi.org/10.1145/3540250.3549081), [Document](https://dx.doi.org/10.1145/3540250.3549081) Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.3](#S2.SS3.p3.1 "2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [Table 2](#S2.T2.5.1.5.1 "In 2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [6th item](#S4.I2.i6.p1.1 "In 4.2.2. Automated Code Review (ACR) Baselines ‣ 4.2. Subjects of Study ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Lin (2004) C. Lin ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, pp. 74–81. External Links: [Link](https://aclanthology.org/W04-1013/) Cited by: [§4](#S4.SSx1.p5.1 "RQ1: How reliable is the proposed evaluation methodology? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§4](#S4.SSx2.SSS2.Px1.p4.1 "Performance Analysis by ACR Tools ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Liu et al. (2024) J. Liu, K. Wang, Y. Chen, X. Peng, Z. Chen, L. Zhang, and Y. Lou Large language model-based agents for software engineering: A survey. CoRR abs/2409.02977. External Links: [Link](https://doi.org/10.48550/arXiv.2409.02977), [Document](https://dx.doi.org/10.48550/ARXIV.2409.02977), 2409.02977 Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Lu et al. (2025) J. Lu, X. Li, Z. Hua, L. Yu, S. Cheng, L. Yang, F. Zhang, and C. Zuo DeepCRCEval: revisiting the evaluation of code review comment generation. In Fundamental Approaches to Software Engineering - 28th International Conference, FASE 2025, Held as Part of the International Joint Conferences on Theory and Practice of Software, ETAPS 2025, Hamilton, ON, Canada, May 3-8, 2025, Proceedings, A. Boronat and G. Fraser (Eds.), Lecture Notes in Computer Science, Vol. 15693, pp. 43–64. External Links: [Link](https://doi.org/10.1007/978-3-031-90900-9%5C_3), [Document](https://dx.doi.org/10.1007/978-3-031-90900-9%5F3) Cited by: [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§3.2](#S3.SS2.p1.1 "3.2. Evaluation Methodology ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Lu et al. (2023) J. Lu, L. Yu, X. Li, L. Yang, and C. Zuo LLaMA-reviewer: advancing code review automation with large language models through parameter-efficient fine-tuning. In 34th IEEE International Symposium on Software Reliability Engineering, ISSRE 2023, Florence, Italy, October 9-12, 2023, pp. 647–658. External Links: [Link](https://doi.org/10.1109/ISSRE59848.2023.00026), [Document](https://dx.doi.org/10.1109/ISSRE59848.2023.00026) Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [7th item](#S4.I2.i7.p1.1 "In 4.2.2. Automated Code Review (ACR) Baselines ‣ 4.2. Subjects of Study ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Papineni et al. (2002) K. Papineni, S. Roukos, T. Ward, and W. Zhu Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311–318. External Links: [Link](https://aclanthology.org/P02-1040/), [Document](https://dx.doi.org/10.3115/1073083.1073135) Cited by: [3rd item](#S1.I1.i3.p1.1 "In 1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§4](#S4.SSx1.p5.1 "RQ1: How reliable is the proposed evaluation methodology? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§4](#S4.SSx2.SSS2.Px1.p4.1 "Performance Analysis by ACR Tools ‣ RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - PMD (2000) PMD External Links: [Link](https://pmd.github.io/) Cited by: [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Qodana AI (2025) Qodana AIPR-Agent: AI-Powered Pull Request Agent(Website) External Links: [Link](https://github.com/qodo-ai/pr-agent) Cited by: [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [5th item](#S4.I2.i5.p1.1 "In 4.2.2. Automated Code Review (ACR) Baselines ‣ 4.2. Subjects of Study ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Rasheed et al. (2024) Z. Rasheed, M. A. Sami, M. Waseem, K. Kemell, X. Wang, A. Nguyen, K. Systä, and P. Abrahamsson AI-powered code review with llms: early results. CoRR abs/2404.18496. External Links: [Link](https://doi.org/10.48550/arXiv.2404.18496), [Document](https://dx.doi.org/10.48550/ARXIV.2404.18496), 2404.18496 Cited by: [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Rosa et al. (2023) G. Rosa, L. Pascarella, S. Scalabrino, R. Tufano, G. Bavota, M. Lanza, and R. Oliveto A comprehensive evaluation of SZZ variants through a developer-informed oracle. J. Syst. Softw. 202, pp. 111729. External Links: [Link](https://doi.org/10.1016/j.jss.2023.111729), [Document](https://dx.doi.org/10.1016/J.JSS.2023.111729) Cited by: [§3.1](#S3.SS1.SSSx3.p1.1 "Step 3: Quality Enhancement through Filtering and Sampling ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Sghaier and Sahraoui (2024) O. B. Sghaier and H. A. Sahraoui Improving the learning of code review successive tasks with cross-task knowledge distillation. Proc. ACM Softw. Eng. 1 (FSE), pp. 1086–1106. External Links: [Link](https://doi.org/10.1145/3643775), [Document](https://dx.doi.org/10.1145/3643775) Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p1.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Shi et al. (2024) L. Shi, W. Ma, and S. Vosoughi Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. CoRR abs/2406.07791. External Links: [Link](https://doi.org/10.48550/arXiv.2406.07791), [Document](https://dx.doi.org/10.48550/ARXIV.2406.07791), 2406.07791 Cited by: [§3.2](#S3.SS2.p1.1 "3.2. Evaluation Methodology ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - SonarSource (2006) SonarSource SonarQube. External Links: [Link](https://www.sonarsource.com/products/sonarqube/) Cited by: [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Sun et al. (2025) T. Sun, J. Xu, Y. Li, Z. Yan, G. Zhang, L. Xie, L. Geng, Z. Wang, Y. Chen, Q. Lin, W. Duan, and K. Sui BitsAI-cr: automated code review via LLM in practice. CoRR abs/2501.15134. External Links: [Link](https://doi.org/10.48550/arXiv.2501.15134), [Document](https://dx.doi.org/10.48550/ARXIV.2501.15134), 2501.15134 Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.3](#S2.SS3.p5.1 "2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§3.1](#S3.SS1.SSSx5.p1.1 "Dataset Statistics ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - \[35] SWR-Bench: A Benchmarking Suite for Serverless Cold Start Time Reduction. Note: [https://github.com/ZZR0/SWRench](https://github.com/ZZR0/SWRench)Accessed: 2025-05-23. Cited by: [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§3.1](#S3.SS1.SSSx2.p1.1 "Step 2: LLM-based change-actions Verification and Classification ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§6](#S6.p1.1 "6. Threats to Validity ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [Data Availability](#Sx1.p1.1 "Data Availability ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Tang et al. (2024) X. Tang, K. Kim, Y. Song, C. Lothritz, B. Li, S. Ezzini, H. Tian, J. Klein, and T. F. Bissyandé CodeAgent: autonomous communicative agents for code review. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), pp. 11279–11313. External Links: [Link](https://aclanthology.org/2024.emnlp-main.632) Cited by: [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.3](#S2.SS3.p3.1 "2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [Table 2](#S2.T2.5.1.6.1 "In 2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [3rd item](#S4.I2.i3.p1.1 "In 4.2.2. Automated Code Review (ACR) Baselines ‣ 4.2. Subjects of Study ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Tanzil et al. (2024) M. H. Tanzil, J. Y. Khan, and G. Uddin ChatGPT incorrectness detection in software reviews. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pp. 180:1–180:12. External Links: [Link](https://doi.org/10.1145/3597503.3639194), [Document](https://dx.doi.org/10.1145/3597503.3639194) Cited by: [§3.1](#S3.SS1.SSSx5.p1.1 "Dataset Statistics ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§4](#S4.SSx2.p2.1 "RQ2: How do mainstream automated code review tools and LLMs perform on SWR-Bench? ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Thakur et al. (2024) A. S. Thakur, K. Choudhary, V. S. Ramayapally, S. Vaidyanathan, and D. Hupkes Judging the judges: evaluating alignment and vulnerabilities in llms-as-judges. CoRR abs/2406.12624. External Links: [Link](https://doi.org/10.48550/arXiv.2406.12624), [Document](https://dx.doi.org/10.48550/ARXIV.2406.12624), 2406.12624 Cited by: [§3.2](#S3.SS2.p1.1 "3.2. Evaluation Methodology ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Thongtanunam et al. (2022) P. Thongtanunam, C. Pornprasit, and C. Tantithamthavorn AutoTransform: automated code transformation to support modern code review process. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 237–248. External Links: [Link](https://doi.org/10.1145/3510003.3510067), [Document](https://dx.doi.org/10.1145/3510003.3510067) Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.3](#S2.SS3.p2.1 "2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [Table 2](#S2.T2.5.1.3.1 "In 2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample LLaMA: open and efficient foundation language models. CoRR abs/2302.13971. External Links: [Link](https://doi.org/10.48550/arXiv.2302.13971), [Document](https://dx.doi.org/10.48550/ARXIV.2302.13971), 2302.13971 Cited by: [7th item](#S4.I2.i7.p1.1 "In 4.2.2. Automated Code Review (ACR) Baselines ‣ 4.2. Subjects of Study ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Tufano et al. (2022) R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota Using pre-trained models to boost code review automation. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 2291–2302. External Links: [Link](https://doi.org/10.1145/3510003.3510621), [Document](https://dx.doi.org/10.1145/3510003.3510621) Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.3](#S2.SS3.p2.1 "2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [Table 2](#S2.T2.5.1.4.1 "In 2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Tufano et al. (2021) R. Tufano, L. Pascarella, M. Tufano, D. Poshyvanyk, and G. Bavota Towards automating code review activities. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pp. 163–174. External Links: [Link](https://doi.org/10.1109/ICSE43902.2021.00027), [Document](https://dx.doi.org/10.1109/ICSE43902.2021.00027) Cited by: [3rd item](#S1.I1.i3.p1.1 "In 1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.3](#S2.SS3.p2.1 "2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [Table 2](#S2.T2.5.1.2.1 "In 2.3. Code Review Benchmarks ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Wang et al. (2024) Y. Wang, Z. Yu, W. Yao, Z. Zeng, L. Yang, C. Wang, H. Chen, C. Jiang, R. Xie, J. Wang, X. Xie, W. Ye, S. Zhang, and Y. Zhang PandaLM: an automatic evaluation benchmark for LLM instruction tuning optimization. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, External Links: [Link](https://openreview.net/forum?id=5Nn2BLV7SB) Cited by: [3rd item](#S1.I1.i3.p1.1 "In 1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§1](#S1.p2.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§3.2](#S3.SS2.p1.1 "3.2. Evaluation Methodology ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Wang et al. (2021) Y. Wang, W. Wang, S. R. Joty, and S. C. H. Hoi CodeT5: identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), pp. 8696–8708. External Links: [Link](https://doi.org/10.18653/v1/2021.emnlp-main.685), [Document](https://dx.doi.org/10.18653/V1/2021.EMNLP-MAIN.685) Cited by: [6th item](#S4.I2.i6.p1.1 "In 4.2.2. Automated Code Review (ACR) Baselines ‣ 4.2. Subjects of Study ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Wikipedia (2025a) Wikipedia Cohen’s kappa. Note: [https://en.wikipedia.org/wiki/Cohen's\_kappa](https://en.wikipedia.org/wiki/Cohen's_kappa)Accessed: 2025-09-11 Cited by: [§3.1](#S3.SS1.SSSx4.p2.1 "Step 4: Manual Verification and Refinement ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Wikipedia (2025b) Wikipedia Stratified sampling. Note: [https://en.wikipedia.org/wiki/Stratified\_sampling](https://en.wikipedia.org/wiki/Stratified_sampling)Accessed: 2025-09-11 Cited by: [§3.1](#S3.SS1.SSSx3.p2.1 "Step 3: Quality Enhancement through Filtering and Sampling ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Xia et al. (2024) C. S. Xia, Y. Deng, S. Dunn, and L. Zhang Agentless: demystifying llm-based software engineering agents. CoRR abs/2407.01489. External Links: [Link](https://doi.org/10.48550/arXiv.2407.01489), [Document](https://dx.doi.org/10.48550/ARXIV.2407.01489), 2407.01489 Cited by: [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Yang et al. (2024a) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press SWE-agent: agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang (Eds.), External Links: [Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-Abstract-Conference.html) Cited by: [2nd item](#S4.I2.i2.p1.1 "In 4.2.2. Automated Code Review (ACR) Baselines ‣ 4.2. Subjects of Study ‣ 4. Evaluation and Study ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Yang et al. (2024b) Z. Yang, C. Gao, Z. Guo, Z. Li, K. Liu, X. Xia, and Y. Zhou A survey on modern code review: progresses, challenges and opportunities. CoRR abs/2405.18216. External Links: [Link](https://doi.org/10.48550/arXiv.2405.18216), [Document](https://dx.doi.org/10.48550/ARXIV.2405.18216), 2405.18216 Cited by: [§1](#S1.p1.1 "1. Introduction ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§2.1](#S2.SS1.p1.1 "2.1. Modern Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Zeng et al. (2024) Z. Zeng, Y. Wang, R. Xie, W. Ye, and S. Zhang CoderUJB: an executable and unified java benchmark for practical programming scenarios. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024, M. Christakis and M. Pradel (Eds.), pp. 124–136. External Links: [Link](https://doi.org/10.1145/3650212.3652115), [Document](https://dx.doi.org/10.1145/3650212.3652115) Cited by: [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Zeng et al. (2021) Z. Zeng, Y. Zhang, H. Zhang, and L. Zhang Deep just-in-time defect prediction: how far are we?. In ISSTA ’21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, Denmark, July 11-17, 2021, C. Cadar and X. Zhang (Eds.), pp. 427–438. External Links: [Link](https://doi.org/10.1145/3460319.3464819), [Document](https://dx.doi.org/10.1145/3460319.3464819) Cited by: [§2.2](#S2.SS2.p2.1 "2.2. Automatic Code Review ‣ 2. Background and Related Work ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"), [§3.1](#S3.SS1.SSSx3.p2.1 "Step 3: Quality Enhancement through Filtering and Sampling ‣ 3.1. Benchmark Construction ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation"). - Zhu et al. (2025) L. Zhu, X. Wang, and X. Wang JudgeLM: fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, External Links: [Link](https://openreview.net/forum?id=xsELpEPn4A) Cited by: [§3.2](#S3.SS2.p1.1 "3.2. Evaluation Methodology ‣ 3. SWR-Bench ‣ SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation").