Paper deep dive
Understanding the Limits of Automated Evaluation for Code Review Bots in Practice
Veli Karakaya, Utku Boran Torun, Baykal Mehmet Uçar, Eray TĂŒzĂŒn
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 6/21/2026, 8:33:46 AM
Summary
This research investigates the feasibility and limitations of using LLM-based automated evaluation strategies (G-Eval and LLM-as-a-Judge) to assess the usefulness of Automated Code Review (ACR) bots in an industrial setting (Beko). By analyzing a dataset of 2,604 bot-generated pull request comments, the study compares automated scores (using both binary and 0-4 Likert scales) against human-provided labels (fixed/wontFix). The findings reveal that automated evaluators achieve only moderate alignment (0.44 to 0.62) with human labels, primarily because developer actions are heavily influenced by organizational constraints, workflow pressures, and contextual factors that static automated evaluations fail to capture.
Entities (9)
Relation Signals (5)
G-Eval â evaluates â Automated Code Review (ACR) Bot
confidence 100% · Two automated evaluation approaches, G-Eval and an LLM-as-a-Judge pipeline, are applied
LLM-as-a-judge â evaluates â Automated Code Review (ACR) Bot
confidence 100% · Two automated evaluation approaches, G-Eval and an LLM-as-a-Judge pipeline, are applied
Beko â provideddatasetfor â Automated Code Review (ACR) Bot
confidence 100% · We analyze an industrial dataset from Beko comprising 2,604 bot-generated PR comments
Developer Actions â influences â Ground Truth Labels
confidence 95% · Developer actions such as resolving or ignoring comments reflect not only comment quality, but also contextual constraints...
Gemini 2.5 Pro â usedasbackendfor â G-Eval
confidence 90% · We evaluate both automated evaluators with three LLM backends: Gemini-2.5-pro...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Automated code review (ACR) bots are increasingly used in industrial software development to assist developers during pull request (PR) review. As adoption grows, a key challenge is how to evaluate the usefulness of bot-generated comments reliably and at scale. In practice, such evaluation often relies on developer actions and annotations that are shaped by contextual and organizational factors, complicating their use as objective ground truth. We examine the feasibility and limitations of automating the evaluation of LLM-powered ACR bots in an industrial setting. We analyze an industrial dataset from Beko comprising 2,604 bot-generated PR comments, each labeled by software engineers as fixed/wontFix. Two automated evaluation approaches, G-Eval and an LLM-as-a-Judge pipeline, are applied using both binary decisions and a 0-4 Likert-scale formulation, enabling a controlled comparison against developer-provided labels. Across Gemini-2.5-pro, GPT-4.1-mini, and GPT-5.2, both evaluation strategies achieve only moderate alignment with human labels. Agreement ratios range from approximately 0.44 to 0.62, with noticeable variation across models and between binary and Likert-scale formulations, indicating sensitivity to both model choice and evaluation design. Our findings highlight practical limitations in fully automating the evaluation of ACR bot comments in industrial contexts. Developer actions such as resolving or ignoring comments reflect not only comment quality, but also contextual constraints, prioritization decisions, and workflow dynamics that are difficult to capture through static artifacts. Insights from a follow-up interview with a software engineering director further corroborate that developer labeling behavior is strongly influenced by workflow pressures and organizational constraints, reinforcing the challenges of treating such signals as objective ground truth.
Tags
Links
- Source: https://arxiv.org/abs/2604.24525v1
- Canonical: https://arxiv.org/abs/2604.24525v1
Trouble viewing inline? Open PDF directly â
Full Text
60,317 characters extracted from source content.
Expand or collapse full text
Understanding the Limits of Automated Evaluation for Code Review Bots in Practice Veli Karakaya â Utku Boran Torun â veli.karakaya@ug.bilkent.edu.tr boran.torun@bilkent.edu.tr Bilkent University Ankara, Turkey Baykal Mehmet Uçar baykal.ucar@beko.com Beko Ä°stanbul, Turkey Eray TĂŒzĂŒn eraytuzun@cs.bilkent.edu.tr Bilkent University Ankara, Turkey Abstract Automated code review (ACR) bots are increasingly used in indus- trial software development to assist developers during pull request (PR) review. As adoption grows, a key challenge is how to eval- uate the usefulness of bot-generated comments reliably and at scale. In practice, such evaluation often relies on developer actions and annotations that are shaped by contextual and organizational factors, complicating their use as objective ground truth. We ex- amine the feasibility and limitations of automating the evaluation of LLM-powered ACR bots in an industrial setting. We analyze an industrial dataset from Beko comprising 2,604 bot-generated PR comments, each labeled by software engineers as fixed/wontFix. Two automated evaluation approaches, G-Eval and an LLM-as-a- Judge pipeline, are applied using both binary decisions and a 0â4 Likert-scale formulation, enabling a controlled comparison against developer-provided labels. Across Gemini-2.5-pro, GPT-4.1-mini, and GPT-5.2, both evaluation strategies achieve only moderate alignment with human labels. Agreement ratios range from approx- imately 0.44 to 0.62, with noticeable variation across models and between binary and Likert-scale formulations, indicating sensitivity to both model choice and evaluation design. Our findings highlight practical limitations in fully automating the evaluation of ACR bot comments in industrial contexts. Developer actions such as resolv- ing or ignoring comments reflect not only comment quality, but also contextual constraints, prioritization decisions, and workflow dy- namics that are difficult to capture through static artifacts. Insights from a follow-up interview with a software engineering director further corroborate that developer labeling behavior is strongly influenced by workflow pressures and organizational constraints, reinforcing the challenges of treating such signals as objective ground truth. CCS Concepts âą Software and its engineeringâSoftware creation and man- agement. â Both authors contributed equally to this research. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. EASE â26, Glasgow, UK © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-x-x-x/Y/M https://doi.org/10.1145/n.n Keywords LLM Evaluation, Automated Evaluation, Automated Code Review Bot, Automated Code Review, Pull Request Comments ACM Reference Format: Veli Karakaya, Utku Boran Torun, Baykal Mehmet Uçar, and Eray TĂŒzĂŒn. 2026. Understanding the Limits of Automated Evaluation for Code Review Bots in Practice. In Proceedings of International Conference on Evaluation and Assessment in Software Engineering (EASE â26). ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/n.n 1 Introduction Code review plays a central role in modern software development, serving as a critical quality assurance mechanism for detecting bugs, improving maintainability, and transferring knowledge among de- velopers [1â3]. To reduce manual review overhead and accelerate development workflows, many organizations have adopted Auto- mated code review (ACR) bots such as ReviewBot [4], SonarLint [5], and BitsAI-CR [6], which generate comments on pull requests (PRs) to highlight potential issues or improvements. These bots are de- signed to augment human reviewers, yet their practical usefulness and reliability remain uncertain, particularly because organizations lack reliable ways to evaluate the value of bot-generated feedback in industrial settings. While ACR bots can provide rapid and scalable feedback on PRs [7], their practical value critically depends on how the quality of the generated review comments is assessed in practice. Prior work by Cihan et al. [4] in industrial settings shows that a 33.4% of bot comments can be incorrect, misleading, or not applicable, highlight- ing the challenges of using developer actions as a reliable proxy for comment usefulness. Developer responses to bot comments often reflect not only technical validity, but also prioritization pressures, workflow constraints, and timing considerations. Low-quality or poorly timed feedback can waste developer time, introduce unnec- essary changes, and ultimately reduce trust in review automation. As a result, organizations must solve not only how to deploy ACR bots, but also how to determine whether their feedback genuinely helps developers during PR review [1]. This observation is further supported by our interview with an director of software engineer- ing at Beko, who noted that developersâ responses to bot comments are frequently shaped by workflow pressures and prioritization constraints rather than purely technical considerations. Despite the widespread adoption of ACR bots, reliably evaluat- ing the quality and usefulness of their generated review comments remains an open problem in industrial practice [4]. Commercial ACR systems continuously produce large volumes of feedback, yet arXiv:2604.24525v1 [cs.SE] 27 Apr 2026 EASE â26, June 09â12, 2026, Glasgow, UKVeli Karakaya, Utku Boran Torun, Baykal Mehmet Uçar, and Eray TĂŒzĂŒn there is no standardized, scalable, and reliable mechanism to deter- mine whether these comments meaningfully assist developers or merely introduce review noise [8]. In current practice, effectiveness is primarily assessed through human judgment, such as manual inspection or developer feedback [1]. However, human-centered evaluation remains expensive, slow, and difficult to repeat at scale [9]. This becomes especially problematic as ACR systems evolve through model updates, prompt changes, or rule modifications [4,6]. In addition, developer actions such as fixing or ignoring comments are imperfect labels because they may reflect local pri- orities, release pressure, ownership boundaries, or timing rather than usefulness alone [4]. These challenges are further amplified by the increasing adoption of commercial ACR solutions such as CodeRabbit [10], Qodo [11], and Snyk Code [12], which makes systematic comparison across tools and versions particularly dif- ficult [13]. Figure 1 illustrates the automated evaluation setting considered in this study, where multiple evaluation strategies inde- pendently assess bot-generated comments and are compared based on their agreement with human-labeled annotations. Evaluating ACR comments is fundamentally challenging because usefulness is inherently subjective and depends heavily on the pull request context, including the code diff, developer intent, and project conventions. As a result, prior studies often rely on manual inspection, expert judgment, or partially automated procedures [4,6,14,15]. Although informative, these approaches are time- consuming, inconsistent across evaluators, and difficult to scale to large industrial repositories [4]. This creates a practical gap: in- dustry requires evaluation methods that are not only accurate, but also scalable, consistent, and cost-effective. Whether such evalu- ation can be fully automated without oversimplifying developer judgment and contextual nuance remains an open question. This challenge is closely related to the broader problem of evalu- ating LLM outputs. As recent ACR systems increasingly leverage LLMs to generate review feedback [14,15], assessing ACR useful- ness naturally depends on advances in LLM evaluation method- ologies. Recent work proposes rubric-based, reference-free evalu- ation protocols (e.g., G-Eval) [16] and investigates the reliability of LLM judges [17,18]. Benchmarking and survey studies further highlight the diversity of evaluation designs [19], while retrieval- augmented evaluation frameworks such as RAGAS emphasize the role of grounding and faithfulness [20]. Despite these advances, existing LLM evaluation frameworks are largely generic and are not specialized for software engineering tasks. Moreover, in indus- trial code review settings, human labels themselves may reflect workflow constraints, risk tolerance, and prioritization decisions, introducing additional ambiguity that automated evaluators are not designed to resolve. To the best of our knowledge, no widely adopted task-specific automated evaluation framework currently exists for industrial ACR workflows. Moreover, it remains unclear whether such stan- dardization is feasible in practice, given the inherently contextual, project-specific, and organizational nature of code review decisions. This study examines the feasibility and limitations of automated evaluation strategies for judging the usefulness of bot-generated code review comments, with the goal of understanding where and why such automation breaks down in industrial settings, while explicitly acknowledging that certain design choices such as scoring weights and decision thresholds require human intervention. We conducted this study in collaboration with Beko, a multinational home appliances company, whose software teams have adopted an LLM-based automated code review bot in their industrial PR workflows. The dataset used in this study originates from prior work by Cihan et al. [4], where a Qodo-based code review agent powered by GPT-4 Turbo [21] was deployed in Bekoâs industrial repositories. Using data collected from this real-world deployment, we investigate two automated evaluation strategies, G-Eval and an LLM-as-a-Judge pipeline, and analyze their agreement with developer-provided labels that are commonly treated as ground truth in practice. To contextualize and interpret our quantitative findings, we additionally conducted a semi-structured interview with the director of software engineering at Beko. Our objective is to assess whether evaluation can be automated. This leads us to the following research questions. Research Questions RQ1: To what extent does a rubric-based G-Eval strategy align with human-labeled annotations when assessing the usefulness of ACR bot comments? RQ2: To what extent does a plain LLM-as-a-Judge evaluation strategy align with human-labeled annotations when assessing the usefulness of ACR bot comments? Rather than assuming high alignment as a necessary or attainable objective, these research questions are designed to characterize the degree and nature of agreement between automated evaluators and developer-provided labels in an industrial setting. 2 Related Work 2.1 Automated Code Review ACR research has progressed from early ML/DL models trained on historical PRs and review feedback to recent LLM-based systems that can generate and assess review comments with richer context. However, most of this literature emphasizes generation quality or deployment outcomes, while the problem of evaluating com- ment usefulness in practice remains underexplored. We therefore organize prior work around this gap. 2.1.1 Automated Code Review with ML and DL. Tufano et al. [22] used a neural machine translation (NMT) model trained on historical PRs to learn reviewer-style code transformations, eval- uating it in both contributor-style edit prediction and comment- available refinement settings. Li et al. [23] introduced CodeReviewer, a transformer pre-trained on large multilingual code-change and review-comment data to support multiple review tasks (e.g., com- ment generation and code refinement), establishing a strong base- line across benchmark tasks. Li et al. also proposed AUGER [24], which generates review comments using pre-trained DL models and evaluates output quality with standard generation metrics and usefulness signals from human-grounded assessments. Hong et al. [25] presented CommentFinder, a retrieval-based ap- proach that recommends existing human review comments for similar code changes, emphasizing efficiency and practical deploy- ability via fast candidate retrieval and ranking. Understanding the Limits of Automated Evaluation for Code Review Bots in PracticeEASE â26, June 09â12, 2026, Glasgow, UK Figure 1: This figure illustrates how multiple evaluation strategies assess the usefulness of ACR bot comments across PRs (PR-1...PR-M) against the human labeled outcomes. Alexander et al. [26] reported industrial-scale deployment of ML systems that automate the resolution of frequent review patterns to reduce manual reviewer effort and improve throughput, focusing on operational impact in real workflows. 2.1.2 Automated Code Review with LLMs. LLMs have enabled more context-aware ACR pipelines [14,15,27], including hybrid systems, end-to-end assistants, fine-tuning methods, and retrieval- augmented prompting aimed at improving comment quality and applicability. Sun et al. [6] proposed BitsAI-CR, a two-stage hybrid pipeline that first detects issues with rule-based/static checks and then uses an LLM to validate and refine candidates, reporting precision and deployment-focused adoption metrics. Guo et al. [14] empirically studied ChatGPT for automated code refinement by comparing it against CodeReviewer on public and curated datasets, using generation/refinement metrics (e.g., EM, BLEU) and analyzing common failure modes tied to comment am- biguity and missing context. Vijayvergiya et al. [13] introduced AutoCommenter, an end-to- end LLM-backed assistant that enforces coding best practices across multiple languages at Google, evaluating success primarily through large-scale deployment and developer-rated helpfulness. Ramesh et al. [28] reports real-world experience with integrat- ing LLM-based automated code review into industrial development workflows. Their experience report emphasizes deployment con- siderations and practical challenges that arise when moving from promising model outputs to reliable, everyday code review support. Alexander et al. [26] also explored LLM-assisted review support and automated application of review feedback to code changes, emphasizing automation of fixes rather than explicit usefulness assessment of the comments. Lu et al. [29] proposed LLaMA-Reviewer, adapting LLaMA for review comment generation via parameter-efficient fine-tuning (LoRA), and evaluated it on public code review datasets to measure generation performance under low trainable-parameter budgets. Zhang et al. [30] proposed LAURA, a retrieval-augmented prompt- ing method that supplies LLMs with historical review context to improve generated comment quality, evaluating gains using correctness/helpfulness-oriented outcome measures. These studies show how recent systems increasingly incorporate richer context and stronger models, but they still treat evaluation largely as a byproduct of generation quality. 2.2 LLM Evaluation Recent work on LLM evaluation increasingly favors rubric-based and reference-free judging to better match human preferences in open-ended tasks, while also documenting systematic biases and reliability issues. Liu et al. [16] introduced G-Eval, which uses a strong LLM with rubric-guided prompting (including chain-of-thought style reason- ing) to score generated text, aiming to align more closely with human judgments than surface-form overlap metrics. Zheng et al. [31] studied the LLM-as-a-Judge paradigm at scale via MT-Bench and Chatbot Arena, examining agreement with hu- man preferences and documenting evaluation risks such as bias and prompt sensitivity that affect judgment reliability. Gao et al. [19] surveyed LLM evaluation methods and orga- nized common protocols (e.g., rubric-based, reference-free, and pair- wise comparison), providing a taxonomy that motivates judgment- oriented evaluation beyond exact-match metrics. Wang et al. [32] empirically study whether LLM-as-a-Judge can replace human evaluators across software engineering tasks (e.g., code generation/translation/summarization) and show that align- ment with human ratings is strongly task- and setup-dependent, limiting the applicability of a single judge configuration across SE settings. He et al. [33] present an SE-focused perspective on LLM evalua- tion by synthesizing evidence on LLM-as-a-Judge usage in software engineering and highlighting recurring reliability risks such as bias, context sensitivity, and calibration issues, motivating SE-specific evaluation designs. Zhou et al. [17] and Chen et al. [18] analyzed judgment reliability and the relationship between automatic metrics and human ratings, reinforcing the need for evaluation setups that better reflect human perception. Together, these works motivate rubric-based, reference- free LLM evaluators for assessing the usefulness of LLM-generated code review comments against human judgments. For code review evaluation, the most relevant studies are those that connect judge design to the ambiguity of review usefulness. EASE â26, June 09â12, 2026, Glasgow, UKVeli Karakaya, Utku Boran Torun, Baykal Mehmet Uçar, and Eray TĂŒzĂŒn Naik et al. [34] proposed CRScore, a reference-free metric for eval- uating code review comments along dimensions such as relevance and comprehensiveness, supporting the need for task-specific eval- uation metrics for LLM-based code review systems. Cihan et al. [4], in turn, are central for our problem framing because they show in an industrial setting that developer resolutions and comment outcomes are informative but imperfect signals of usefulness. Together, these studies motivate our central question: even with strong rubric-based judges, how far can automated evaluation go when the available âground truthâ is itself shaped by workflow context, prioritization, and organizational constraints? 3 Methodology 3.1 Data Source and Extraction We use an industrial dataset of 2,604 PR comments from Bekoâs repository previously analyzed by Cihan et al. [4]. The dataset con- tains bot-generated PR comments and human-labeled outcomes (fixed / wontFix) derived from developer labeled comments. There were five labels (active, pending, resolved (fixed), wontFix, and closed) for each comment. We exclude comments labeled as ac- tive and pending because they represent intermediate workflow states where no final developer decision regarding usefulness has been made. We also exclude closed comments because, in the orig- inal dataset definition, this label indicates that a comment was not implemented for reasons other than wontFix (e.g., external or workflow-related factors) rather than as a result of an explicit tech- nical judgment about usefulness [4]. Since automated evaluators and LLM-based judges do not have access to these external decision factors, treating closed as not useful would introduce additional ambiguity into the ground truth. The resulting dataset consists of 1,733 comments (âŒ66%) labeled as fixed and 871 comments (âŒ34%) labeled as wontFix. Data are accessed via the organizationâs Azure DevOps instance and Jira project. For each PR we extract: (1) PR Description, PR Title, PR diff (2) All bot-generated review comments (3) JIRA issue description if specified in PR. Human-labeled usefulness is given and considered as ground truth of this study. Human-Labeled Usefulness. The usefulness labels used in this study originate from an existing industrial code review workflow. As reported by Cihan et al. [4], developers are required to explicitly finalize the status of all review comments (e.g., fixed, wontFix, active, pending, closed) before a PR can be merged. This enforced process ensures that each review comment is deliberately assessed by developers during real code review activities. Following this practice, we map comments marked as fixed to useful, and comments marked as wontFix to not useful. The defi- nition of usefulness is given in replication package 1 . These labels reflect developersâ real decisions during production code review and are therefore treated as ground truth for evaluating automated review comment usefulness. 1 https://figshare.com/articles/conference_contribution/Understanding_the_Limits_ of_Automated_Evaluation_for_Code_Review_Bots_in_Practice/31462948?file= 62372143 Figure 2: Evaluation setup comparing G-Eval and LLM-as-a- Judge (binary vs. 0â4 Likert) for labeling bot comment use- fulness and reporting agreement and classification metrics. 3.2 Evaluation Criteria Models. We evaluate both automated evaluators with three LLM backends: Gemini-2.5-pro [35], GPT-4.1-mini [36], and GPT-5.2 [37]. To assess the usefulness of ACR comments, we evaluate each com- ment using two complementary rubric settings: (i) 0-4 Likert scale and (i) a direct binary usefulness rubric as shown in Figure 2. The Likert scale provides a structured breakdown of comment qual- ity, while the binary rubric captures an overall usefulness decision aligned with the datasetâs useful/not useful labels. 3.2.1 0-4 Likert Scale and Decision Rule. In the 0-4 Likert scale setting, the evaluator assigns integer scoresí usefulness â 0,1,2,3,4 for ACR bot comments. The full definition of these usefulness scores are given in the replication package [38]. Rationale for the 0â4 Scoring Scale. We adopt a discrete 0â4 scor- ing scale to implement a simple and interpretable evaluation rubric. This design choice is consistent with prior LLM-as-a-Judge and rubric-driven evaluation frameworks, which emphasize explicit and well-specified scoring criteria to improve interpretability for both human annotators and language models [16,39]. To enable comparison with binary developer labels, we map Likert scores to binary outcomes using a threshold of 1. In this mapping, a score of 0 denotes the absence of any perceived usefulness signal, while scores of 1 or higher indicate the presence of at least some signal, even if weak or imperfect. This interpretation is consistent with prior work showing that low non-zero Likert scores may still cor- respond to problematic or hallucinated content, yet nonetheless reflect the presence of a detectable response rather than a complete absence of information [40]. Abbreviated rubric summary 0 (Not Useful): Incorrect, misleading, or unrelated to the PR diff, . . . not actionable. 1 (Low Usefulness): Minor/generic guidance with limited PR-specific grounding, . . . limited impact. 2 (Borderline Useful): Plausibly relevant but evidence/ac- tionability in this PR is unclear, . . . . 3 (Mostly Useful): PR-specific non-obvious issue with enough evidence to act now, . . . . 4 (Fully Useful): PR-critical and evidence-backed feedback that should be handled in this PR, . . . . 3.2.2Binary Usefulness Rubric (0â1). In addition to the 0-4 Likert scale, we evaluate a direct binary usefulness decision to match the Understanding the Limits of Automated Evaluation for Code Review Bots in PracticeEASE â26, June 09â12, 2026, Glasgow, UK datasetâs ground truth framing. In this setting, the evaluator assigns a single usefulness score constrained to the set0, 1. The binary prediction is obtained deterministically by mapping the usefulness score to the label: useful when the usefulness score is 1 and, not useful otherwise. The corresponding full explanations of scores are given in the replication package [38]. Abbreviated rubric summary 0 (Not Useful): Incorrect/irrelevant/confusing feedback with no actionable value, . . . . 1 (Useful): Technically correct, relevant, clear, and actionable feedback, . . . improves code quality. 3.2.3Shared Inputs, Prompting Scope, and Decision Mapping. Both evaluation strategies (G-Eval and LLM-as-a-Judge) use the same shared evaluation setup so that differences can be attributed to the evaluation approach itself. Inputs. For each bot comment, the evaluator receives the same compact PR context: PR title, PR description, PR diff, and the bot comment. No repository-wide artifacts (e.g., project guidelines or historical discussions) are provided. Prompting Scope. In both strategies, prompts instruct the evalu- ator to ground judgments in the provided PR artifacts and apply the same usefulness criteria defined in Sections 3.2.1 and 3.2.2. The key difference from G-Eval is prompt style: LLM-as-a-Judge uses direct judge-style instructions rather than G-Evalâs rubric-metric template but the underlying evaluation criteria remain identical. Decision Mapping. For Likert-based evaluation, numeric scores are mapped to binary labels using the same thresholding rule in Section 3.2.1 (useful iffí usefulness â„1). For direct binary evaluation, outputs are deterministically mapped using Section 3.2.2 (useful iff score = 1). 3.3 G-Eval Based Evaluation Our first evaluation strategy is based on G-Eval 2 , a rubric-driven LLM-based evaluation framework designed for reference-free and subjective assessment tasks. Evaluating the usefulness of ACR bot comments exhibits two key characteristics: (i) there is no single âgoldâ reference comment for a given PR diff, and (i) usefulness is inherently context dependent, requiring reasoning over PR artifacts rather than surface-level lexical overlap. As a result, conventional reference-based automatic metrics (e.g., BLEU, ROUGE) are not applicable, and retrieval-focused evaluation frameworks such as RAGAS [20] do not directly capture the notion of comment useful- ness. G-Eval aligns with these requirements by operationalizing evalu- ation as an explicit rubric-following scoring procedure executed by an LLM. It enables structured scoring under well-specified criteria and can be applied without task-specific fine-tuning. Prior work reports that rubric-driven prompting with intermediate reasoning can improve alignment with human judgments in subjective evalu- ation settings [16]. We therefore use G-Eval as a baseline to assess 2 https://deepeval.com/docs/metrics-llm-evals whether a rubric-based evaluator can recover developersâ labels from PR artifacts alone. This strategy uses the shared setup in Section 3.2.3 for inputs, criteria, and decision mapping. 3.4 LLM-as-a-Judge Evaluation Our second evaluation strategy follows an LLM-as-a-Judge design, where an LLM is prompted to act as an evaluator and decide whether a bot-generated review comment is useful. This approach is moti- vated by the growing use of LLMs as meta-evaluators for subjective tasks, in which explicit references are unavailable and the assess- ment criteria are inherently context dependent [32]. In our setting, a reliable evaluator must (i) give a score to generated ACR bot com- ment with concrete PR context, and (i) map that interpretation to a simple usefulness label comparable to the industrial ground truth. This strategy also follows the shared setup in Section 3.2.3 for inputs, criteria, and decision mapping. 3.5 Evaluation Protocol and Metrics We evaluate each automated approach by comparing its binary usefulness prediction against the human-labeled ground truth label provided in the dataset [4]. Evaluation Unit. The unit of evaluation is a single bot-generated PR comment. Each comment is independently scored by the eval- uator and mapped to a binary decision using the corresponding binary variants. Primary Metric. Following the datasetâs framing, we report agreement ratio: Agreement = #(bíŠ= íŠ)/í, whereíŠis the human la- bel,bíŠis the automated prediction, andíis the number of evaluated comments. Aggregation. We report results across the overall agreement ratio per model and methodology approach to capture model vari- ability and evaluation difficulty. 3.6 Implementation Details We implement both automated evaluators as deterministic pipelines that consume the same comment-level inputs and differ only in rubric definition. LLM Execution Settings. All LLM calls are executed with the default temperature and a fixed system prompt. We retained vendor- recommended defaults to reflect realistic industrial deployment conditions, where models are typically used without enforced de- terministic decoding [41]. Because each evaluator was executed once per comment and no repeated sampling was performed, set- ting the temperature to 0 would have imposed artificial determinism on a single run. Preserving the modelsâ default stochastic behavior therefore provides a more representative view of practical usage, while robustness under repeated runs and alternative temperature settings remains future work. Token Budget and Truncation. To comply with model context window limitations, we truncated PR diffs only when the total input length exceeded the maximum context window size supported by the respective LLM. This affected 227 PRs (approximately 8.7% of the dataset). In such cases, we preserved the PR content in its original order up to the context window boundary and removed only the remaining tail portion that exceeded this limit. EASE â26, June 09â12, 2026, Glasgow, UKVeli Karakaya, Utku Boran Torun, Baykal Mehmet Uçar, and Eray TĂŒzĂŒn Reproducibility Notes. All intermediate artifacts (reasonings, prompts, and raw model outputs) are logged per comment to sup- port auditability and error analysis. 4 Results This section presents the empirical results of the proposed auto- mated evaluation approaches and reports their alignment with human-labeled ground truth. We first present the overall agree- ment ratio across three models and two methodologies presented in Section 3. Further, we analyze the F1 score, precision, recall, and Matthews correlation coefficient (MCC) score for one result with best agreement ratio across all evaluations. We report MCC in addition to the other metrics because the dataset is imbalanced, and MCC provides a more reliable evaluation measure under class imbalance [42]. Table 1: Agreement ratios of automated evaluators against human-labeled ground truth across models and rubric vari- ants. G-EvalLLM-as-a-Judge ModelBinary Likert Binary Likert GPT-5.20.450.590.440.58 GPT-4.1-mini0.480.620.570.61 Gemini-2.5-pro0.510.570.530.55 Table 2: Detailed evaluation metrics for the best agreement ratio achieved method (G-Eval - Likert) approach across mod- els. G-Eval (Likert) ModelAgreement F1 Precision Recall MCC GPT-5.20.590.720.670.78-0.0020 GPT-4.1-mini0.620.760.660.90-0.0590 Gemini-2.5-pro0.570.680.670.69 0.0106 4.1 Performance of G-Eval We evaluate the agreement between G-Eval and human-labeled annotations using both a direct binary rubric and a Likert-based rubric that is mapped to binary decisions via the thresholding rule described in Section 3.2.1. As summarized in Table 1, the Likert- based variant consistently achieves higher agreement with human labels than the direct binary variant across all evaluated models. For GPT-5.2, agreement increases from 0.45 under the binary rubric to 0.59 with the Likert-based formulation. Similarly, agreement for GPT-4.1-miniincreases from 0.48 to 0.62, and forGemini-2.5-pro from 0.51 to 0.57. Across models,GPT-4.1-miniachieves the highest agreement under the Likert-based G-Eval configuration (0.62), followed by GPT-5.2(0.59) andGemini-2.5-pro(0.57). While the absolute dif- ferences between models are moderate, these results indicate that the choice of evaluation backend can meaningfully affect agreement outcomes even when the evaluation rubric and input data are held constant. The consistent improvement observed with the Likert-based configuration suggests that a graded rubric allows the evaluator to express borderline cases more reliably than a single-step binary decision. The subsequent deterministic mapping from Likert scores to binary labels appears to reduce ambiguity relative to forcing an immediate binary judgment, leading to higher alignment with developer-provided annotations. Nevertheless, overall agreement remains moderate, indicating that rubric design and model choice can influence, but do not fully resolve, the challenges of automated usefulness evaluation in industrial code review settings. To further characterize the behavior of the best-performing configuration, we examine the detailed classification metrics re- ported in Table 2. For the Likert-based G-Eval configuration with GPT-4.1-mini, the evaluator achieves an F1 score of 0.76, precision of 0.66, recall of 0.90, and an MCC ofâ0.0590. Across models, preci- sion values are relatively similar, whereas recall varies substantially, ranging from 0.69 forGemini-2.5-proto 0.90 forGPT-4.1-mini. The high recall indicates that the judge frequently predicts the useful label and captures a large fraction of the fixed comments, while the lower precision reflects a higher rate of false positives on wontFix comments. Despite moderate agreement ratios and relatively strong F1 scores, the near-zero (and slightly negative) MCC values indicate weak correlation once class imbalance is taken into account. This discrepancy suggests that agreement and F1 alone may overstate evaluator reliability in this setting, as label distributions and ground- truth confounding where wontFix reflects organizational or contex- tual constraints rather than purely technical non-usefulness limit calibrated discrimination. Overall, these results reinforce that rubric choice and model selection influence G-Eval behavior, but do not fully overcome the inherent challenges of automated usefulness evaluation in industrial code review contexts. 4.2 Performance of LLM-as-a-Judge We evaluate the agreement between the LLM-as-a-Judge approach and human-labeled annotations using both a direct binary rubric and a Likert-based rubric that is mapped to binary decisions via the thresholding rule described in Section 3.2.1. As reported in Table 1, the Likert-based configuration consistently achieves higher agreement than the direct binary formulation across all evaluated models. Under the Likert rubric,GPT-4.1-miniachieves the highest agreement (0.61), followed byGPT-5.2(0.58) andGemini-2.5-pro (0.55). A similar pattern is observed when comparing binary and Likert variants directly: agreement increases from 0.44 to 0.58 for GPT-5.2, from 0.57 to 0.61 forGPT-4.1-mini, and from 0.53 to 0.55 for Gemini-2.5-pro. Across configurations,GPT-4.1-miniconsistently yields the strongest alignment with developer-provided labels for the LLM-as- a-Judge approach. While the absolute differences between models are moderate, these results indicate that the choice of judge backend affects agreement outcomes even when the evaluation rubric and input evidence are held constant. Understanding the Limits of Automated Evaluation for Code Review Bots in PracticeEASE â26, June 09â12, 2026, Glasgow, UK 5 Discussion 5.1 Implications for Practitioners Evaluating the usefulness of ACR bot comments remains a difficult task [9]. Usefulness is subjective and context dependent, and devel- oper outcomes such as fixed and wontFix can reflect non-technical constraints (e.g., prioritization and timing) in addition to comment quality. In our setting, both automated evaluators achieve only mod- erate agreement with human labels, indicating that these signals are imperfect and should not be interpreted as an objective ground truth proxy without additional validation. Our results provide sev- eral practical insights for teams that deploy ACR bots in real-world software development workflows: Automated scores should be treated as weak signals, not ground truth. Given the moderate agreement observed in Sec- tion 4, practitioners should not blindly trust automated usefulness scores as the sole basis for approving, rejecting, or ranking re- view bot comments. Instead, these scores are better positioned as heuristic indicators that require periodic sampling and human verification. Risk of false positives and wasted effort. Imperfect evalua- tors can label non-actionable or contextually inappropriate com- ments as useful. If these scores are surfaced directly to the de- velopers or used to drive automation, they may increase review noise, distract reviewers, and reduce trust in both the bot and the evaluation pipeline. Use automated evaluation for triage, not decision-making. A more conservative integration is to use automated scores to prioritize what to inspect (e.g., routing low-score or disputed cases to manual review), rather than to automatically accept or dismiss comments. This can reduce evaluation cost while limiting the harm from misclassifications. Model and rubric choices matter. Agreement varies across models and rubric variants (Table 1), implying that practitioners should not assume evaluator performance is stable across backends or prompt settings. Any deployment should include routine re- evaluation after model upgrades or prompt/rubric changes. Diagnostics require careful interpretation. Aggregate eval- uation scores (e.g., agreement, F1, precision) may hide recurring weaknesses in specific file types, programming languages, or re- view scenarios. Therefore, when using evaluator outputs to refine bot prompts or rules, teams should complement these numerical metrics with manual inspection of selected PR examples to better understand common error patterns. 5.2 Implications for Researchers Limits of weighting schemes and decision thresholds. While alternative weighting strategies and decision thresholds can influ- ence evaluation outcomes, our findings suggest that tuning these parameters alone may not resolve misalignment when human labels encode contextual and organizational factors. Future work should therefore study not only how different scoring schemes affect align- ment, but also when such alignment is fundamentally constrained by dataset characteristics and task ambiguity. Human-in-the-loop analysis as a diagnostic tool, not a sil- ver bullet. Analyzing the reasoning or justification texts produced by LLM-based judges can provide valuable insight into evaluator behavior. However, such analyses should be viewed as diagnostic in- struments for identifying systematic biases or failure modes, rather than as mechanisms that guarantee reliable automated judgment. Understanding when and why evaluator reasoning diverges from developer intent remains an open challenge. Cautious use of fine-tuned decision models. Fine-tuning decision models on labeled review comments may yield more stable or deterministic outputs. At the same time, this approach risks over- fitting to labels that themselves reflect workflow constraints rather than pure comment quality. Future research should examine how fine-tuned evaluators generalize beyond their training context and how sensitive they are to label noise and organizational practices. Interpreting behavioral signals requires contextual ground- ing. Post-merge application or non-application of bot comments can provide additional behavioral signals of usefulness. However, such signals are also shaped by factors such as scope, timing, and developer workload. Research is needed to understand how these implicit signals can be combined with explicit labels without rein- forcing existing biases. Generalization as an empirical question, not an assump- tion. Although automated evaluators are often presented as general- purpose tools, our results suggest that their behavior may be tightly coupled to specific repositories, teams, and review conventions. Future studies should treat cross-domain generalization as an em- pirical question and explicitly report where and why evaluators fail to transfer. Robustness of evaluation under limited context. LLM-based judges frequently operate under a limited context, such as partial diffs or missing cross-file dependencies. Understanding how evalua- tors behave under such constraints, and whether certain judgments are fundamentally unreliable in these settings, remains a key open research problem. Addressing this issue is essential for realistic evaluation in industrial environments. 5.3 Interview with an Industrial Practitioner To contextualize the moderate agreement levels observed between automated evaluators and developer-provided labels, we conducted a semi-structured interview with a director of software engineer- ing at Beko who has direct experience deploying and operating AI-assisted code review tools in production workflows. We ana- lyzed the transcript using open coding, an analytic process within grounded theory [43]. The first and second authors independently coded meaningful segments and iteratively grouped related codes into higher-level themes through discussion. Theme 1 â Developer labels are operational outcomes, not pure judgments of comment quality. The interviewee emphasized that labels such as fixed and wontFix often reflect delivery constraints rather than purely technical judgments about comment quality. Developers may resolve comments quickly to proceed with de- ployment, particularly under time pressure. As the interviewee explained: âEspecially on the wontFix side, developers may select it very quickly because the priority is often getting the change into production.â This observation aligns with our quantitative findings of moderate agreement. If labels encode operational priorities such as deadlines or scope limitations, they cannot be treated as fully objective ground truth for automated evaluation. EASE â26, June 09â12, 2026, Glasgow, UKVeli Karakaya, Utku Boran Torun, Baykal Mehmet Uçar, and Eray TĂŒzĂŒn Theme 2 â Diff-centered evaluation structurally limits judgment quality. The interview repeatedly highlighted the mismatch be- tween what automated systems can observe and what developers consider in practice. While evaluator pipelines typically operate over PR diffs and limited metadata, developers reason holistically about runtime behavior, architectural constraints, and safeguards beyond the visible scope of the PR. This mismatch can produce systematic discrepancies: comments that appear technically valid in isolation may be redundant or already addressed elsewhere, and evaluators may recommend defensive changes for rare edge cases that teams intentionally accept due to broader trade-offs. Theme 3 â Industrial evaluation is multi-dimensional. The inter- viewee emphasized that evaluation in practice is not reducible to accuracy alone. Practitioners consider risk reduction (e.g., prevent- ing production bugs or security issues), alert fatigue, and integration into the software development lifecycle (SDLC). Over-notification with low-priority suggestions may cause critical issues to be over- looked, while high-severity findings may justify higher operational costs. Thus, usefulness depends on severity, operational impact, and process fit rather than a single aggregate performance score. Theme 4 â Continuous monitoring is desired, but systematic eval- uation is rare without automation. The interviewee framed review bots as agents that require ongoing monitoring similar to human contributors, arguing that automated tools should be managed anal- ogously to human performance (agent-resource management). The interviewee expanded, âIf we manage human resources, we should also manage agent resources. These tools need to be monitored continuously.â At the same time, systematic evaluation is rarely sus- tained in practice because manual monitoring is time-consuming and difficult to scale. This tension reinforces our motivation: auto- mated evaluation is attractive as a scalable monitoring mechanism, yet its outputs must be interpreted as context-limited and approxi- mate signals rather than definitive ground truth. The interviewee further noted that such challenges are unlikely to be specific to a single organization, as development culture, business priorities, and acceptable quality thresholds vary across companies. 5.4 Threats to Validity Internal validity. Several factors may affect the internal validity of our findings. First, some PR titles in the dataset include embedded issue types or contextual cues. Since our evaluation pipelines ex- tract contextual information programmatically, such metadata may unintentionally bias both G-Eval and LLM-as-a-Judge by providing overly strong or misleading signals. Second, our analysis relies on developer-provided usefulness labels as ground truth. These labels are inherently subjective and may reflect workflow constraints such as time pressure, prioritization, or scope rather than the semantic usefulness of a comment, introducing noise and potential biases into the evaluation [44,45]. Third, the prompt design used to oper- ationalize the evaluation rubrics may influence model outputs, as phrasing, instruction structure, and rubric wording can shape how LLMs interpret usefulness criteria. Finally, LLM-based evaluators may exhibit model-driven biases, for example, by overweighting linguistic clarity, PR titles, or surface-level features of comments, which can skew predictions toward useful even when the underly- ing technical value is limited. External validity. The external validity of this study is con- strained by the dataset and evaluation setting. Our analysis is based on multiple industrial repositories from a single organization and focuses on one ACR bot, which limits generalization to other devel- opment environments, programming languages, review cultures, or CI/CD practices. In addition, the evaluation is conducted using a specific set of LLM backends; alternative models, versions, or rubric formulations may yield different results. Moreover, our setting in- volves an LLM evaluating outputs generated by another LLM: the ACR bot in the reference dataset is based on GPT-4 Turbo [21], while our evaluators also rely on modern LLMs. This introduces a potential coupling effect, where evaluators closer in model family or training period may better align with the generatorâs output style, while others may appear to perform worse due to systematic differences in phrasing or calibration. As a result, performance dif- ferences across evaluator backends should be interpreted cautiously and not attributed solely to superior evaluation capability. 6 Conclusion This paper examined whether the usefulness of ACR bot comments can be assessed using automated evaluators, using an industrial dataset of 2,604 bot-generated PR comments annotated by software engineers. We compared a rubric-based LLM evaluator, which is G- Eval, against the LLM-as-a-Judge pipeline; both approaches are ap- plied over the same binary decision and 0-4 Likert scale evaluation. The agreement ratios, F1 scores, precision, recall, and MCC scores are obtained by agreement with the human-labeled annotations. Across multiple LLM backends, both evaluation strategies achieved moderate agreement with human labels, with overall agreement ra- tios ranging approximately from 0.44 to 0.62 in our setting. Across these evaluations, we further analyzed the best strategy among four options: G-Eval (0-4 Likert/binary) and LLM-as-a-Judge (0-4 Likert/binary). The best-performing model achieved 0.76 F1 score, 0.66 precision, 0.90 recall, and -0.00590 MCC score. The results also show that performance varies by model and by the choice of methodology. These findings suggest that automated usefulness evaluation provides only an approximate and partial signal of human judg- ment, rather than a direct or reliable replacement. In particular, wontFix labels may reflect non-technical factors (e.g., scope, prior- itization, timing) that are not fully recoverable from PR artifacts or repository snapshots, which introduces an inherent ceiling on agreement-based evaluation. Insights from our interview further support this interpretation, highlighting that developer labeling behavior is frequently shaped by workflow pressures and organiza- tional constraints beyond the observable PR context. As a result, automated metrics should be interpreted cautiously when used for large-scale monitoring or comparative analysis of ACR systems. Acknowledgment This work has been supported by the ITEA4 GENIUS project, funded by the national funding authorities of the participating countries: https://itea4.org/project/genius.html Understanding the Limits of Automated Evaluation for Code Review Bots in PracticeEASE â26, June 09â12, 2026, Glasgow, UK References [1]Hyungjin Kim, Yonghwi Kwon, Sangwoo Joh, Hyukin Kwon, Yeonhee Ryou, and Taeksu Kim. Understanding automated code review process and developer experience in industry. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, page 1398â1407, New York, NY, USA, 2022. Association for Computing Machinery. [2]Deepika Badampudi, Ricardo Britto, and Michael Unterkalmsteiner. Modern code reviews - preliminary results of a systematic mapping study. In Proceedings of the Evaluation and Assessment on Software Engineering, EASE â19, page 340â345. ACM, April 2019. [3] Zezhou Yang, Cuiyun Gao, Zhaoqiang Guo, Zhenhao Li, Kui Liu, Xin Xia, and Yuming Zhou. A survey on modern code review: Progresses, challenges and opportunities. arXiv preprint arXiv:2405.18216, 2024. [4]Umut Cihan, Vahid Haratian, Arda Icoz, Mert Kaan Gul, Omercan Devran, Emir- can Furkan Bayendur, Baykal Mehmet Uçar, and Eray TĂŒzĂŒn. Automated code review in practice: An industrial case study. In Proceedings of the 46th Inter- national Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 1â12. ACM, 2024. [5]SonarSource. SonarQube for IDE: Real-Time AI Code Verification. https://w. sonarsource.com/products/sonarqube/ide/. Accessed: 2026-03-03. [6] Tao Sun, Jian Xu, Yuanpeng Li, Zhao Yan, Ge Zhang, Lintao Xie, Lu Geng, Zheng Wang, Yueyan Chen, Qin Lin, et al. Bitsai-cr: Automated code review via llm in practice. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, pages 274â285, 2025. [7]Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, and Marco A. Gerosa. Effects of adopting code review bots on pull requests to oss projects. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 1â11, 2020. [8] Taufiqul Islam Khan, Shaowei Wang, Haoxiang Zhang, and Tse-Hsun Chen. A survey of code review benchmarks and evaluation practices in pre-llm and llm era, 2026. [9]Utku Boran Torun, Veli Karakaya, Ali Babar, and Eray TĂŒzĂŒn. Evaluation of LLM- based software engineering tools: Practices, challenges, and future directions. In Proceedings of the 30th International Conference on Evaluation and Assessment in Software Engineering, EASE â26, New York, NY, USA, 2026. Association for Computing Machinery. [10]CodeRabbit: Ai-powered code review platform. https://w.coderabbit.ai/, 2026. [11] Qodo: Ai code review and code quality platform. https://w.qodo.ai/, 2026. [12]Snyk Code: Static application security testing (sast) documentation. https://docs. snyk.io/scan-with-snyk/snyk-code, 2026. [13]Manushree Vijayvergiya, MaĆgorzata Salawa, Ivan BudiseliÄ, Dan Zheng, Pascal Lamblin, Marko IvankoviÄ, Juanjo Carin, Mateusz Lewko, Jovan Andonov, Goran PetroviÄ, Daniel Tarlow, Petros Maniatis, and RenĂ© Just. Ai-assisted assessment of coding practices in modern code review. In Proceedings of the 1st ACM Inter- national Conference on AI-Powered Software, AIware 2024, page 85â93, New York, NY, USA, 2024. Association for Computing Machinery. [14] Qi Guo, Junming Cao, Xiaofei Xie, Shangqing Liu, Xiaohong Li, Bihuan Chen, and Xin Peng. Exploring the potential of chatgpt in automated code refinement: An empirical study. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE â24, New York, NY, USA, 2024. Association for Computing Machinery. [15]Chanathip Pornprasit and Chakkrit Tantithamthavorn. Fine-tuning and prompt engineering for large language models-based code review automation. Inf. Softw. Technol., 175(C), November 2024. [16]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511â2522, Singapore, December 2023. Association for Computational Linguistics. [17]Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39â57, 2024. [18]Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs as the judge? a study on judgement bias. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301â8327, Miami, Florida, USA, November 2024. Association for Computational Linguistics. [19]Mingqi Gao, Xinyu Hu, Xunjian Yin, Jie Ruan, Xiao Pu, and Xiaojun Wan. LLM- based NLG evaluation: Current status and challenges. Computational Linguistics, 51:661â687, June 2025. [20]Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. RAGAs: Automated evaluation of retrieval augmented generation. In Nikolaos Aletras and Orphee De Clercq, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150â158, St. Julians, Malta, March 2024. Association for Computational Linguistics. [21]OpenAI. GPT-4 Turbo Model | OpenAI API Documentation. https://developers. openai.com/api/docs/models/gpt-4-turbo. Accessed: 2026-03-03. [22] Michele Tufano, Denys Poshyvanyk, Gabriele Bavota, and Massimiliano Di Penta. Towards automating code review activities. In Proceedings of the 43rd International Conference on Software Engineering (ICSE), page 163â174, 2021. [23] Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Ma- jumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, et al. Codereviewer: Pre- training for automating code review activities. arXiv preprint arXiv:2203.09095, 2022. [24]Yucheng Li, Xin Xia, David Lo, Weiqin Wang, and Zhiqiang Chen. Auger: Auto- matically generating review comments with pre-trained models. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 1087â1099, 2022. [25]Changrong Hong, Zhiqiang Chen, Xin Xia, and David Lo. Commentfinder: A simpler, faster, more accurate code review comment recommendation. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 890â 902, 2022. [26]Alexander Froemmgen, Jacob Austin, Peter Choy, Nimesh Ghelani, Lera Kharatyan, Gabriela Surita, Elena Khrapko, Pascal Lamblin, Pierre-Antoine Man- zagol, Marcus Revaj, Maxim Tabachnyk, Daniel Tarlow, Kevin Villela, Daniel Zheng, Satish Chandra, and Petros Maniatis. Resolving code review comments with machine learning. In Proceedings of the 46th International Conference on Soft- ware Engineering: Software Engineering in Practice, ICSE-SEIP â24, page 204â215, New York, NY, USA, 2024. Association for Computing Machinery. [27]Umut Cihan, Arda İçöz, Vahid Haratian, and Eray TĂŒzĂŒn. Evaluating large language models for code review. arXiv preprint arXiv:2505.20206, 2025. [28] Shweta Ramesh, Joy Bose, Hamender Singh, A K Raghavan, Sujoy Roychowdhury, Giriprasad Sridhara, Nishrith Saini, and Ricardo Britto. Automated code review using large language models at ericsson: An experience report, 2025. [29]Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), pages 647â658. IEEE, 2023. [30] Yuxin Zhang, Yuxia Zhang, Zeyu Sun, Yanjie Jiang, and Hui Liu. Laura: Enhancing code review generation with context-enriched retrieval-augmented llm. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 2983â2995. IEEE, 2025. [31]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595â46623, 2023. [32]Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering. Proceedings of the ACM on Software Engineering, 2(IS- STA):1955â1977, June 2025. [33] Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. Llm-as-a-judge for software engineering: Literature review, vision, and the road ahead. ACM Trans. Softw. Eng. Methodol., February 2026. Just Accepted. [34] Atharva Naik, Marcus Alenius, Daniel Fried, and Carolyn Rose. Crscore: Ground- ing automated evaluation of code review comments in code claims and smells. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 9049â9076, 2025. [35] Google Cloud. Gemini 2.5 Pro | Generative AI on Vertex AI. https://docs.cloud. google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro.Accessed: 2026-03-03. [36]OpenAI. Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/. Accessed: 2026-03-03. [37]OpenAI. Introducing GPT-5.2. https://openai.com/index/introducing-gpt-5-2/. Accessed: 2026-03-03. [38]Veli Karakaya, U. Boran Torun, Baykal Mehmet Uçar, and Eray TĂŒzĂŒn. Under- standing the Limits of Automated Evaluation for Code Review Bots in Practice. 3 2026. https://doi.org/10.6084/m9.figshare.31462948. [39]Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models, 2023. [40]Kla Tantithamthavorn, Hong Yi Lin, Patanamon Thongtanunam, Wachiraphan Charoenwet, Minwoo Jeong, and Ming Wu. Hallujudge: A reference-free hallu- cination detection for context misalignment in code review automation. arXiv preprint arXiv:2601.19072, 2026. [41]George Kour, Itay Nakash, Michal Shmueli-Scheuer, and Ateret Anaby Tavor. Think again! the effect of test-time compute on preferences, opinions, and beliefs of large language models. In Proceedings of the 63rd Annual Meeting of the EASE â26, June 09â12, 2026, Glasgow, UKVeli Karakaya, Utku Boran Torun, Baykal Mehmet Uçar, and Eray TĂŒzĂŒn Association for Computational Linguistics (Volume 6: Industry Track), pages 639â 660, 2025. [42]Davide Chicco and Giuseppe Jurman. The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC genomics, 21(1):6, 2020. [43]Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. Grounded theory in software engineering research: a critical review and guidelines. In Proceedings of the 38th International Conference on Software Engineering, ICSE â16, page 120â131, New York, NY, USA, 2016. Association for Computing Machinery. [44]Eray TĂŒzĂŒn, Hakan Erdogmus, Maria Teresa Baldassarre, Michael Felderer, Robert Feldt, and Burak Turhan. Ground-truth deficiencies in software engineering: When codifying the past can be counterproductive. IEEE Software, 39(3):85â95, 2022. [45]Martin Obaidi, Marc Herrmann, Kurt Schneider, and Jil KlĂŒnder. Towards trust- worthy sentiment analysis in software engineering: Dataset characteristics and tool selection. In 2025 IEEE 33rd International Requirements Engineering Confer- ence Workshops (REW), pages 538â547. IEEE, 2025.