Paper deep dive
RefactorAssist: Agentic Refinement for Reliable Code Refactoring
Jonathan Cordeiro, Shayan Noei, Ying Zou
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Code refactoring aims to enhance the internal structure of source code without affecting its functional behavior. The recent advancements of Large Language Models (LLMs) have demonstrated potential for automating software engineering tasks, such as code refactoring. However, the refactorings produced by LLMs often introduce subtle errors, leading to functional behavior changes and failed unit tests, which limit their practical adoption. To address the limitations of LLM-generated refactorings, we analyze the root causes of their failures and develop the RefactorAssist agent to improve the functional correctness of LLM-generated refactorings. To this end, we use 10 open-source Java projects with their native test suites and manually evaluate why LLM-generated refactorings fail unit tests. We then design an agentic approach that leverages unit-test logs, error explanations, project context retrieval, and code diffs to guide the iterative refactoring. Our findings show that the main reasons for failure are context misunderstanding/hallucination (24.3%), incorrect or inconsistent renaming (15.3%), adding new functionality or variables (13.7%), code incompleteness (11.3%), syntax and structural errors (9.7%), edge cases not handled (9%), improper type handling (8.7%), and variables outside scope (8%). To make our approach cost-effective, RefactorAssist first applies a static repair step for missing imports, unbalanced brackets, and compilation errors without LLMs. For remaining failures, RefactorAssist incorporates error logs and code diffs, achieving up to a 70.8% repair rate on the remaining failures and a 94.2% cumulative pass rate under the best-performing configuration. These results indicate that static checks and test-guided, context-aware agentic repair can increase the reliability of LLM-generated refactorings, bringing them closer to practical integration within developer workflows.
Tags
Links
- Source: https://arxiv.org/abs/2608.00924v1
- Canonical: https://arxiv.org/abs/2608.00924v1
Trouble viewing inline? Open PDF directly â
Full Text
99,386 characters extracted from source content.
Expand or collapse full text
RefactorAssist: Agentic Refinement for Reliable Code Refactoring JONATHAN CORDEIRO, Queenâs University, Canada SHAYAN NOEI, Queenâs University, Canada YING ZOU, Queenâs University, Canada Code refactoring aims to enhance the internal structure of source code without affecting its functional behavior. The recent ad- vancements of Large Language Models (LLMs) have demonstrated potential for automating software engineering tasks, such as code refactoring, offering new opportunities to improve code quality automatically. However, the refactorings produced by LLMs often introduce subtle errors, leading to functional behavior changes and failed unit tests, which limit their practical adoption. To address the limitations of LLM-generated refactorings, we analyze the root causes of their failures and develop the RefactorAssist agent to improve the functional correctness of LLM-generated refactorings. To this end, we use 10 open-source Java projects with their native test suites, and we first manually evaluate why LLM-generated refactorings fail unit tests. We then design an automated agentic approach that leverages unit-test logs, error explanations, project context retrieval, and code diffs to guide the iterative refactoring process. Our findings show that the main reasons for refactorings failing unit tests are context misunderstanding/hallucination (24.3%), incorrect or inconsistent renaming (15.3%), adding new functionality or variables (13.7%), code incompleteness (11.3%), syntax and structural errors (9.7%), edge cases not handled (9%), improper type handling (8.7%), and variables outside scope (8%). To make our approach cost-effective, our RefactorAssist agent first applies a static repair step that corrects syntactic issues of the code, such as missing imports, unbalanced brackets, and compilation errors, before test execution without LLMs. For refactorings that still fail test cases, RefactorAssist incorporates error logs and code diffs, achieving up to a 70.8% repair rate on the remaining failures and up to a 94.2% cumulative pass rate under the best-performing configuration. These results indicate that static checks as well as test-guided, context-aware agentic repair can increase the reliability of LLM-generated refactorings, bringing them closer to practical integration within developer workflows. CCS Concepts:⢠Software and its engineering; Additional Key Words and Phrases: Code Refactoring, Code Quality Improvement, Large Language Models ACM Reference Format: Jonathan Cordeiro, Shayan Noei, and Ying Zou. 2018. RefactorAssist: Agentic Refinement for Reliable Code Refactoring. J. ACM 37, 4, Article 111 (August 2018), 27 pages. https://doi.org/X.X 1 Introduction Code refactoring is the process of restructuring existing code to enhance its quality while preserving its external functionality [16,17]. Effective refactoring not only improves readability and modularity but also reduces dependencies and addresses technical debt [21,38]. Although human developers perform better in code refactoring due to their contextual awareness and domain expertise [17], they also pose the risk of human errors [3,10]. Therefore, automated Authorsâ Contact Information: Jonathan Cordeiro, 19jac16@queensu.ca, Queenâs University, Kingston, Canada; Shayan Noei, s.noei@queensu.ca, Queenâs University, Kingston, Canada, ; Ying Zou, ying.zou@queensu.ca, Queenâs University, Kingston, Canada. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Š 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM Manuscript submitted to ACM1 arXiv:2608.00924v1 [cs.SE] 2 Aug 2026 2Cordeiro et al. approaches can reduce the dependence on developers and save time. Classical static tooling, including general code smell and bug detectors such as SonarQube [51], PMD [45], and SpotBugs [52], and code smell detection tools such as DesigniteJava [49], can flag design problems and suggest limited refactoring types. However, their adoption at scale is limited by noisy alerts and false positives, time-intensive configuration, and no guarantee that suggested changes preserve functional behavior. Studies report under-use or suppression of alerts in practice, sometimes accruing technical debt [26, 30, 37]. The advancement of automatic code generation has revolutionized software development, enabling developers to produce code more efficiently and consistently [14,54]. Despite the capabilities of Large Language Models (LLMs) in automating a variety of programming tasks, including code generation and bug fixing [14,59], code changes related to software design and architecture, such as refactoring, remain unreliable [31]. LLM-generated refactorings often introduce unintended behavioral changes, violate architectural constraints, or generate refactorings that fail existing test cases [6,10,11,23], highlighting the need for specialized refactoring agents designed specifically for safe and reliable refactorings. In this study, we first empirically evaluate how often LLMs fail to perform refactorings and investigate the underlying causes of these failures. We then propose and evaluate an agentic approach (i.e., RefactorAssist) for repairing failed LLM-refactored code. RefactorAssist first applies a static repair that fixes syntax and structural issues in LLM-refactored code to reduce the cost of running LLMs. It then employs an iterative agentic repair loop to resolve functional issues introduced by LLM-generated refactorings. RefactorAssist combines compilation feedback, unit-test logs, code diffs, retrieved project context, and diagnostic explanations to guide refactoring repair, and recover from failures introduced by LLM-generated refactorings. To demonstrate the effectiveness of our approach, we utilize the Microsoft Methods2Test dataset [57], a collection of real-world Java methods paired with their original unit tests extracted from mature open-source projects, to develop and test RefactorAssist. To empirically evaluate our design of RefactorAssist, we apply it to incorrect LLM-generated refactorings and measure whether it can repair the introduced functional failures while preserving the intended refactoring behavior. To minimize data leakage and better reflect real-world settings where refactoring tasks are unseen by the model, we use StarCoder2 [32] for refactoring generation. StarCoder2 is open-source and trained on a publicly available Stack v2 dataset [32], allowing us to verify that the evaluation repositories are not included in its training. In summary, our work addresses the following research questions: RQ1: How often does LLM-based refactoring lead to incorrect code behavior? Although LLMs demonstrate strong capabilities in code generation and refactoring, their functional correctness remains a significant concern. A refactoring must preserve external behavior; therefore, transformations that improve code structure but introduce compilation errors or failing tests cannot be considered correct refactorings. To evaluate this issue, we measure the unit test pass rates of refactorings generated by state-of-the-art LLMs, including StarCoder2, GPT-4o, Claude Sonnet, and Qwen2.5-Coder-32B-Instruct, under 0-shot, 1-shot, 3-shot, and 5-shot prompting settings. We compile each refactored program and execute the associated unit tests to determine whether the generated refactoring preserves behavior. Our findings show that model choice has the strongest effect on functional correctness, with GPT-4o achieving the highest overall pass rate of 80.8%. In contrast, few-shot prompting and refactoring scope have limited effects on correctness, indicating that simply adding examples or restricting the transformation scope is insufficient to ensure reliable refactoring generation. RQ2: What are the most common reasons for failures in LLM-refactored code? To understand the root causes of unit test failures, we first apply an LLM-assisted clustering approach to cluster all failure messages into distinct failure Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring3 categories. We then conduct a manual analysis on a statistically representative sample of failures to uncover recurring patterns and verify the failure patterns. Both analyses converge on the same fixed taxonomy: (1) context misunder- standing/hallucination, (2) new functionality introduced, (3) incorrect or inconsistent renaming, (4) scope/dependency conflicts, (5) improper type handling, (6) code incompleteness, (7) unhandled edge cases, and (8) syntax/structural errors. Our analysis reveals that the causes of unit test failures are context misunderstanding/hallucination (24.3%), incorrect or inconsistent renaming (15.3%), adding new functionality or variables (13.7%), code incompleteness (11.3%), syntax and structural errors (9.7%), edge cases not handled (9%), improper type handling (8.7%), and variables outside scope (8%). RQ3: Can our agentic approach improve the correctness of LLM-refactored code? The failure analysis in the second research question indicates that many incorrect refactorings stem from syntactic and structural issues, while others require deeper semantic diagnosis and repair. To address both types of failures, we introduce RefactorAssist, which combines a repair pipeline in which static fixes are applied first, followed by an agentic refactoring repair stage for the remaining failing cases. The static-fix stage preserves imports, enforces bracket matching, and performs basic type-consistency checks before test execution, while the agentic approach uses compiler and unit-test failure logs, code diffs, and an error diagnostic LLM explanation to guide targeted repair generation. Our results show that this combined approach substantially improves the overall unit test pass rate from 68.4% to 80.3% after static intervention and further rises to 93.4% after utilizing our agentic approach. RQ4: Which components of our approach contribute most to repair effectiveness? To understand which components of RefactorAssist drive the most improvement in refactoring correction, we perform an ablation study that evaluates the contribution of different components of RefactorAssist. We find that the highest-performing configuration combines the code diff, error log, and LLM explanation output from the GPT-4o error diagnostic LLM. This configuration repairs 70.8% of the refactorings, contributing to a cumulative unit test pass rate of 94.2%. In contrast, adding retrieval- augmented context has a negative effect on the cumulative pass rate. The key contributions of this work are as follows: â˘An evaluation of LLM-generated refactorings using the public Methods2Test dataset, measuring unit test failure rates to assess their reliability in preserving functional behavior. ⢠An analysis of common failure reasons in LLM-refactored code, providing insights into recurring issues along with quick static fixes. ⢠The development of an agentic approach to refine LLM refactorings, improving their functional correctness. â˘We provide a replication package to enable the reproducibility of our study. The replication package of the study can be accessed at: https://github.com/Software-Evolution-Analytics-Lab-SEAL/Automating-refactoring- correction Paper Organization. The remainder of this paper is structured as follows. Section 2 describes RefactorAssist and its repair workflow. Section 3 outlines the research approach and methodology. Section 4 provides the results of the research. Section 5 discusses the threats to validity. Section 6 summarizes existing work on automatic refactoring, testing, and automated refactoring repair. Finally, Section 7 concludes the paper by summarizing our contributions and outlining future research directions. 2 RefactorAssist RefactorAssist is designed to improve incorrect LLM-generated refactorings. RefactorAssist first applies static fixes to address dependency, syntactic, and structural errors without additional LLM calls. If the refactoring still fails, it enters Manuscript submitted to ACM 4Cordeiro et al. Agentic Refactoring RepairStatic Refactoring Repair Dependency Resolution Syntax Correction Structural Correction LLM Refactoring Refactored Code Pass Fail Recompile and Re-run Tests Run Test Cases Run Test Cases Error Diagnosis Repair Generation Short-term Memory Long-term Memory Tools Error Log Processor Code Diff Generator Context Retriever Test Runner Refactored Code Fail Pass Pass Fail Fig. 1. Overview of RefactorAssist. Failing LLM-generated refactorings first undergo static repair and, if still failing, enter an iterative agentic repair loop. an iterative agentic repair loop that uses failure evidence, project context, and diagnostic explanations to guide targeted repair generation. Figure 1 provides an overview of RefactorAssist. 2.1 Static Refactoring Repair The first repair phase of RefactorAssist applies static fixes before invoking the agentic repair loop. This phase is designed to address failure patterns that do not require deeper semantic reasoning. The static repair phase aims to repair the initial LLM-generated refactoring that fails at least one of its associated unit tests. The static fixes are conservative and syntax-preserving and are intended to correct structural issues without altering the intended program logic. Specifically, the static repair stage performs the following operations: â˘Dependency Resolution: This operation targets failures related to variables outside scope, dependency conflicts, and incomplete context, where the LLM removes imports that are still required by the refactored code. We extract the import declarations from the original file before refactoring and compare them with those in the LLM-refactored file. If an original import is removed during generation, we restore it in the refactored file as a conservative fix for omitted import statements. â˘Syntax Correction: This operation targets syntax and structural errors as well as incomplete code outputs, which are identified in the second research question as recurring causes of unit test failure. We apply a stack-based pass to ensure every , [, and ( has a corresponding closing counterpart. When the imbalance can be resolved through a single-token correction, the missing delimiter is inserted at the nearest safe location. â˘Structural Correction: This operation targets improper type handling and simple structural inconsistencies introduced during refactoring. Therefore, we check for mismatches, such as assigning aStringto anintor producing incompatible method signatures, and apply patches when the correction is syntactically clear. Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring5 2.2 Agentic Refactoring Repair The failed refactoring undergoes static refactoring repair and is re-evaluated using the associated unit tests. If the refactoring still fails, it enters the agentic repair loop of RefactorAssist, where it diagnoses the failure, generates a repair, recompiles the project, and reruns the tests until the refactoring passes or reaches the iteration limit. 2.2.1 Tools. RefactorAssist uses four dedicated tools to collect failure evidence, retrieve project context, and construct repair guidance for the code generation LLM. These tools are described below: â˘Error Log Processor: This tool transforms raw compiler and unit-test output into structured failure evidence for downstream diagnosis and repair. To this end, for each failing refactoring, we capture the complete unit-test output, including compiler diagnostics, stack traces, and failed assertions. To keep the input consistent across projects, we standardize environment-specific details by replacing absolute file paths and temporary directory names with placeholders. We then extract two failure-relevant signals for downstream analysis: (1) an error signature, including the exception type, assertion message, and failing test name, and (2) an approximate failure location, including the class, method, and line ranges referenced in the logs. ⢠Code Diff Generator: We compute a unified diff between the original file and the LLM-refactored file to explicitly capture the code changes introduced by the refactoring. The diff is used to identify candidate patch regions by selecting diff hunks that overlap with the failure locations or reference symbols mentioned in the error logs, enabling analysis and repair of the modified code regions. ⢠Context Retriever: To provide the repair process with project context that is often missing from the failing file, RefactorAssist retrieves relevant dependency information from the same repository. The retrieval query is constructed from program symbols extracted from the error logs and code diff, including class names, method signatures, package imports, and referenced fields. These symbols are used to locate repository files and code snippets that define or constrain the failing code. The retrieved context includes dependent utility classes, type declarations, interfaces, constants, and method definitions that are referenced by the failing code or test. When the retrieved content exceeds the prompt-size limit, we prioritize the snippets most directly related to the symbols appearing in the error log and diff and remove unrelated methods, comments, or file regions. The goal of this tool is to expose the dependencies that the code generation LLM must preserve when proposing a fix. â˘Test Runner: This tool compiles the modified project and executes the associated unit tests after each static-fix or repair attempt. The resulting pass/fail outcome determines whether the refactoring is accepted, passed to the agentic repair loop, or used as feedback for the next iteration. 2.2.2 Memory. RefactorAssist maintains both short-term and long-term memory. Short-term memory is used at runtime and captures failure-specific evidence from the current iteration, including the latest unit-test error log and code diff. This information is updated after each repair attempt. Long-term memory stores project-level context retrieved from the repository, such as relevant dependencies and type information, which is reused across repair iterations to provide repository-level context. 2.3 Agent Runtime At the agentic refactoring repair runtime, RefactorAssist follows the two-stage repair process shown in Figure 1 and described below. Manuscript submitted to ACM 6Cordeiro et al. 2.3.1 Error Diagnosis. The error diagnosis phase converts raw failure evidence into a structured diagnosis that the code generation LLM can use for repair. RefactorAssist first uses the Error Log Processor tool to extract compiler and test failure information from the raw logs, including error coordinates, diagnostic messages, failed assertions, and optional symbol:/location:lines. It then uses the Code Diff Generator tool to summarize the changes between the original file and the LLM-refactored file as a unified diff. Next, the Context Retriever tool retrieves relevant project context and common fix patterns from long-term memory based on the symbols and failure locations appearing in the error log and diff. Using these inputs, RefactorAssist constructs an âexplainâ prompt for the error diagnostic component. The error diagnostic component uses an LLM to explain the failure and suggest a repair plan. Its output is a tuple root_cause, hint, where the root cause summarizes why the refactoring failed, and the hint provides a concise fix suggestion containing specific symbols, such as class, method, or parameter names. This tuple is then passed to the repair prompt in the subsequent stage. 2.3.2 Repair Generation. The repair generation phase converts the error diagnosis into a revised Java source file. To this end, the failing code, the code diff, and the plan generated by the error diagnosis component are assembled into a repair prompt and passed to the code-generation LLM to produce a revised version of the source code. The repair prompt instructs the code-generation LLM to return only runnable Java source code and constrains the model to avoid unnecessary class or top-level type changes, preserve public signatures unless required by the failure, and keep edits minimal and localized to the regions implicated by the diff or error log. The goal is to convert the diagnostic explanation into a repair attempt that restores compilability and unit-test correctness while preserving the intended refactoring. The full prompt templates used for diagnosis and repair are provided in our replication package. 2.4 Recompile and Re-run Tests After each repair attempt, RefactorAssist replaces the failing file with the repaired version and uses the Test Runner tool to recompile the project and rerun the associated unit tests. If the repair succeeds, the process stops. Otherwise, the Error Log Processor tool captures the new failure evidence, short-term memory is updated with the latest log, diff, and diagnostic information, and the next repair iteration begins. This loop continues for up to 10 iterations or until the refactored code passes all associated tests. We use this iteration limit to balance repair opportunity and efficiency, allowing multiple rounds of diagnosis and correction while reducing computational cost and limiting the risk of drifting away from the original refactoring intent. 3 Experiment Setup This section describes the experimental setup of our study, including project selection, data preparation, and analysis methods. 3.1 Overview of Our Approach Figure 2 shows the overview of our study. We begin by collecting our dataset from the corresponding GitHub projects in the Methods2Test dataset [57]. For the initial refactoring generation, we utilize StarCoder2 [32] due to its open-source availability and transparent training data, which allows us to mitigate potential data leakage. Data leakage refers to the risk that the data used in our evaluation was already seen during model training, which can artificially inflate performance [5,28]. Then each generated refactoring is compiled and executed against the unit tests of the original Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring7 Dataset Creation Extract repository names Extract projects Sample methods Test Case and Method Extraction Sample test cases Refactoring Generation Model Selection Test Failure Reasons Static Fixes Unit Test Pass Rate Feedback Loop RQ1: Identify how Often LLM- Generated Code Fails Unit Tests RQ2: Identify the Reasons for Unit Test Failures RQ3: Evaluate RefactorAssist Repair Effectiveness RQ4: Ablation Study of Repair Components Verify test casesChoose repositories Scope Restriction Compile/Run Unit Tests Analysis Subset of Methods2Test Refactoring Repair Agent Fig. 2. Overview of the research approach. project to measure its functional correctness. When a refactored method fails the unit tests, we analyze the resulting test failures to understand common failure patterns and identify opportunities for improvement. Leveraging identified failure patterns, we implement a repair pipeline to rectify and refine the initial refactoring attempts. Therefore, we employ an agentic approach that first performs static fixes, then analyzes error logs and code diffs between the original and refactored code to identify and repair failures. 3.2 Dataset Creation This section describes the construction of our evaluation dataset, including filtering steps to mitigate data leakage and ensure sufficient test coverage. 3.2.1 Subject Selection. We conduct our study using repositories from the Methods2Test dataset [57], which contains 9,410 open-source Java repositories and metadata linking methods to their corresponding test cases. Since Methods2Test does not provide ground-truth refactorings required to compare against LLM-generated code, we use the repository names referenced by Methods2Test as our target projects. From these projects, we independently extract methods and their corresponding unit tests using our own parsing and mapping pipeline. Our project selection approach is as follows: (1)Data Leakage Prevention: To address potential data leakage, we extract all repository names from the Stack-v2 dataset, which forms the training corpus for StarCoder2 [32]. By cross-referencing these names against those in the Methods2Test dataset, we exclude all repositories that overlap. This filtering step reduces the dataset from 9,410 to 4,447 repositories and ensures that no evaluation repository was seen during training. (2)Test Coverage Filtering: From the remaining repositories, we select only those with at least 1,000 runnable unit tests [39]. This ensures that each selected project provides sufficient test coverage for reliable evaluation of behavioral correctness. After this filtering step, 17 repositories remain. (3) Author and Redundancy Filtering: To increase project diversity and avoid potential author or organizational bias, we remove repositories developed by the same authors or organizations (e.g., forks, template variants). This step results in a final set of 10 unique repositories. Our subject selection process aims to balance experimental rigor and evaluation scalability while preserving project diversity, test coverage, and protection against potential data leakage. In our agentic approach, each refactoring undergoes up to 10 feedback iterations across 11 ablation configurations, resulting in thousands of compile and test cycles per project. Running these experiments demands substantial multi-GPU resources, as each iteration involves repeated model inference, compilation, and unit test execution. Even when utilizing a cluster with eight NVIDIA A100 GPUs, expanding beyond this scale would be computationally infeasible within a reasonable experimental time frame. Manuscript submitted to ACM 8Cordeiro et al. 3.2.2 Instance Selection. From our selected subjects, we randomly sample 1,000 passing method-test pairs from each of the 10 repositories, which provides 95% confidence with a 5% margin of error [38,48,61], resulting in a final evaluation suite of 10,000 unique method-test pairs across 10 diverse projects. This configuration is optimized to maximize architectural diversity while maintaining computational feasibility relative to the intensive requirements of the refactoring pipeline. Furthermore, we implement a Python script that processes each methodâtest pair in the Methods2Test dataset. For each pair, we check whether the mapped test class actually corresponds to the target method and whether the test can be executed successfully in the original project environment. To do this, the script builds the project and runs the associated test class referenced by the dataset. ⢠If the repository is not already available in the local workspace, the script clones it recursively together with its submodules. If a local copy already exists, the script updates it to the required version. â˘The script then checks whether a top-levelpom.xmlfile exists. In Maven-based Java projects, thepom.xmlfile defines the project structure, dependencies, build plugins, modules, and test configuration. Therefore, we use this file as the entry point for building the project with Apache Maven [36]. When the file is present, the script builds the entire project so that all required modules and dependencies are installed locally before executing the associated test class. After setup, each test class is executed in isolation using Apache Maven, with a timeout of 600 seconds per invocation to filter out invalid or non-terminating tests. The script captures both standard output and standard errors and uses the Maven test result to determine whether the test passes. Only methodâtest pairs whose associated tests can be built and executed successfully are retained for our study. 3.3 Test Case and Method Extraction To evaluate the functional correctness of LLM-generated refactorings, we map the test cases to their corresponding methods within the selected repositories. We create a script to automate the identification and linking of test cases to their associated methods. The process to extract only valid, runnable test cases begins with cloning each of the 10 selected repositories locally. All Java files within each repository are parsed using the JavaParser library [25], which analyzes both production-code and test-code directories to extract methods and relevant structural metadata; for each production method, we record its fully qualified name, including the package name, class name, and method name, as well as its file path, method signature, and the enclosing class or interface. For each test case, we similarly record the test method name, file path, enclosing test class, package name, and any invocation relationships that link the test to a target production method. The methods in the test code are identified based on the presence of the@Testannotation, their location in files under thesrc/test/directory, or naming conventions commonly used in test frameworks, such asTestortest[27]. Once a test method is located, we examine its body to find the method calls it invokes. For example, a test method such astestAddItem()inCartTestmay contain a call tocart.addItem(product). In this case, the invoked method is resolved to its fully qualified name, such ascom.shop.Cart.addItem(Product), which we identify as the target production method under test. More generally, our goal is to determine which specific production method a test is intended to validate, which is what we refer to as the target method. To identify the target method, each method call in the test is resolved to its fully qualified name (i.e., class, package, and method signature), allowing us to accurately map a test to the exact method under test. Our extracted data contains the repository name, the fully qualified method name, Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring9 the fully qualified test name, and the file paths of both the method and the test, which are included in our replication package. 3.4 Refactoring Generation The refactoring generation stage produces the initial LLM-generated refactoring for each target method or class. For each target, the selected LLM generates one refactored candidate per prompt configuration. We then compile the modified project and run the associated unit tests to determine whether the generated refactoring preserves functional correctness. Unit tests are a widely adopted mechanism to validate functional correctness and detect regressions; therefore, we examine the unit test pass rates of the initial LLM-refactored code. A refactoring is considered correct if it compiles successfully and all associated unit tests pass. We report this outcome using the unit test pass rate, defined as follows: Pass Rate= # of refactorings that compile and pass all tests # of total refactorings (1) We evaluate four LLMs for initial refactoring generation: ⢠StarCoder2 [32] is included as an open-source code model with publicly documented training data, which makes it suitable for controlled experimentation and data-leakage mitigation. ⢠GPT-4o [40] is selected as a strong closed-source model to represent the performance of modern commercial LLMs on behavior-preserving refactoring. ⢠Claude Sonnet [2] is included as another closed-source commercial model to examine whether refactoring correctness trends remain consistent across proprietary LLMs. ⢠Qwen2.5-Coder-32B-Instruct [24] is included as a modern open-source, code-specialized model, enabling com- parison with a strong coding-focused model that is more accessible for local or reproducible deployment than proprietary systems. The selection of four LLMs allows us to compare LLM-based refactoring across open-source, open-weight, and closed- source frontier models. 3.5 Hardware Environment All experiments were conducted on a cluster of 8 NVIDIA A100 GPUs, each with 80 GB of memory, running on Ubuntu 22.04.4 LTS. 4 Results In this section, we provide the motivation, approach, and results of our research questions. 4.1 RQ1: How often does LLM-based refactoring lead to incorrect code behavior? 4.1.1 Motivation. Prior studies have shown that LLM-generated code can introduce functional errors despite ap- pearing plausible or improving code quality attributes [11,15,60]. However, most existing work has focused on code generation or bug fixing, rather than systematically evaluating whether LLM-generated refactorings preserve behavior, which is a goal of code refactoring [7,14]. A transformation that improves structure but changes external functionality cannot be considered a correct refactoring. Therefore, this research question investigates how often refactorings generated by LLMs lead to behavioral regressions, as measured by unit test outcomes. Manuscript submitted to ACM 10Cordeiro et al. Instruction You are a model specialized in refactoring Java code. Refactoring improves the internal structure, readability, and maintain- ability without changing external behavior. Output a refactored version of the input code. Prompt # unrefactored code snippet (java): code_segment_before_refactoring # refactored version of the same code snippet: Fig. 3. Zero-shot instruction used to prompt each evaluated LLM for refactoring. 4.1.2 Approach. To evaluate LLM-based refactoring, we generate refactorings using multiple LLMs under consistent prompting configurations, verify that the outputs constitute valid refactorings, and evaluate their functional correctness using unit test outcomes. We then apply statistical tests to analyze differences in pass rates across models, refactoring scopes, and prompting settings. Model Selection. We evaluate StarCoder2 [32], GPT-4o [40], Claude Sonnet [2], and Qwen2.5-Coder-32B-Instruct [24] to cover LLMs with different levels of capability, accessibility, and specialization as mentioned in Section 3.4. Functional Correctness Evaluation. To address this research question, we evaluate the functional correctness of code refactorings generated by each selected LLM on individual Java methods extracted from the target repositories, as described in Section 3.3. We report the overall pass rate (1) as well as pass rates across method-level and class-level refactorings to understand how frequently LLM-generated changes maintain functional behavior. Prompting Configuration. To examine the effect of in-context examples, we run four prompting settings: 0-shot, 1-shot, 3-shot, and 5-shot. In the few-shot settings, examples are randomly sampled from the same project but are disjoint from the target method, and no retrieval or similarity-based selection is applied. Each shot count corresponds to the number of worked refactoring examples included in the prompt before the target method. We use the same prompt template across all evaluated models to ensure a consistent comparison. Decoding parameters, including temperature, top_p, and maximum token limit, are held fixed within each model configuration to reduce variation caused by sampling and to isolate the impact of model choice and shot count. To this end, refactorings are produced using the instruction prompt shown in Figure 3, which consists of a system- level instruction defining refactoring as a behavior-preserving code transformation [17], followed by the target code snippet and a placeholder for the refactored output. Few-shot exemplars, when used, precede the target snippet in the same format. Output Verification. Before measuring test outcomes, we verify that each LLM actually performs refactoring rather than reproducing the original code. To do this, we use RefactoringMiner 2.0 (RMiner) [56], a state-of-the-art refactoring detection tool that is widely used in empirical software engineering studies and supports the detection of method- and class-level refactorings. For each generated file, we compute a unified diff between the LLM-refactored version and the original version and run RMiner to detect structural refactorings. A generated output is considered a valid refactoring only if RMiner identifies at least one refactoring operation, such asRename Method, in the diff between the original code and the LLM output. Outputs where RMiner detects no refactoring are excluded from the analysis to avoid counting identical or near-identical reproductions as successful refactorings. Statistical Evaluation of Refactoring Outcomes. After filtering non-refactored cases, we compare Pass@1 rates across the evaluated LLMs, where Pass@1 denotes the percentage of generated refactorings that successfully compile and pass all associated unit tests on the first attempt. We first compare model-level pass-rate distributions to determine Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring11 StarCoder2Qwen2.5-Coder-32B-InstructClaude SonnetGPT-4o LLM 60 65 70 75 80 85 Unit Test Pass Rate (%) 66.1% 73.1% 77.2% 80.8% Fig. 4. Distribution of unit-test pass rates across evaluated LLMs. whether some LLMs preserve behavior more reliably than others. We then compare pass rates between method-level and class-level refactorings using the Mann-Whitney U-test [34]. This non-parametric test evaluates whether two independent samples differ significantly in their median success rates without assuming a normal distribution [34]. To assess whether pass rates differ across the 0-shot, 1-shot, 3-shot, and 5-shot prompting settings, we apply the Kruskal-Wallis test [33], a non-parametric alternative to one-way ANOVA, to determine whether at least one group differs significantly from the others. We report the test statistic,í-value, and effect size( Ë í 2 )to quantify the magnitude of the overall difference. When significant differences are observed across models or prompting settings, we apply a post-hoc multiple-comparison procedure, such as Dunnâs test [12] with Holm correction, to identify which groups differ from one another. 4.1.3 Findings. LLM-generated refactorings frequently introduce behavioral regressions, even across models with larger parameter sizes and coding capabilities. Figure 4 shows the distribution of project-level Pass@1 rates across the evaluated LLMs. The median Pass@1 rates are 66.1% for StarCoder2, 80.8% for GPT-4o, 77.2% for Claude Sonnet, and 73.1% for Qwen2.5-Coder-32B-Instruct. These results show a clear model effect: frontier closed-source models (GPT-4o, Claude Sonnet) generally achieve higher pass rates than open models, while StarCoder2 is consistently lowest. However, even the strongest model still exhibits failures 19.2% of the time, indicating that behavior preservation remains a challenge for LLM-based refactoring. Although we control for this risk for StarCoder2 by excluding repositories included in its training data, the same level of verification is not possible for closed-source models because their training data is not fully disclosed. Therefore, the higher pass rates of GPT-4o and Claude Sonnet may be due to the exposure of the models to target or similar code, or a combination of both. The statistical comparison across models confirms significant differences. The KruskalâWallis test shows a significant difference in Pass@1 distributions across the evaluated LLMs (íť=25.18,í=1.4Ă10 â5 , Ë í 2 =0.62). Dunnâs post-hoc test with Holm correction shows that GPT-4o significantly outperforms Qwen2.5-Coder-32B-Instruct (í=0.039) and StarCoder2 (í<0.001), and Claude Sonnet significantly outperforms StarCoder2 (í<0.001). GPT-4o and Claude Sonnet Manuscript submitted to ACM 12Cordeiro et al. 0-shot1-shot3-shot5-shot Prompting Setting 55 60 65 70 75 80 85 90 Unit Test Pass Rate (%) 67 63 64 65 72 74 73 72 78 77 77 77 82 81 80 81 LLM StarCoder2 Qwen2.5-Coder-32B-Instruct Claude Sonnet GPT-4o Fig. 5. Distribution of unit-test pass rates across prompting settings. Method-LevelClass-Level Refactoring Scope 55 60 65 70 75 80 85 Unit Test Pass Rate (%) 65 65 73 73 77 77 80 81 LLM StarCoder2 Qwen2.5-Coder-32B-Instruct Claude Sonnet GPT-4o Fig. 6. Distribution of unit-test pass rates for method-level and class-level refactorings. are not significantly different (í=0.485), and the difference between Claude Sonnet and Qwen2.5-Coder-32B-Instruct is not significant (í=0.143). These findings suggest that models with stronger coding and reasoning capabilities improve functional correctness, but do not eliminate behavioral regressions. Few-shot examples have a limited and inconsistent impact on functional correctness. Figure 5 shows pass-rate distributions across 0-shot, 1-shot, 3-shot, and 5-shot prompting. The median project-level pass rates are 75.3%, 73.6%, 73.3%, and 73.6%, respectively. Overall, differences are small, and the grouped plot indicates no consistent monotonic gain from adding more examples across models. This observation is supported by the KruskalâWallis H-test, which reveals no statistically significant difference among the four prompting settings (íť=0.25,í=0.969, Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring13 Ë í 2 â0). These results suggest that correctness is driven more by underlying model capability than by simply increasing in-context examples. Method-level and class-level refactorings exhibit comparable functional correctness. Figure 6 compares Pass@1 rates between method-level and class-level refactorings. The median Pass@1 rate for method-level refactorings is 73.8%, while class-level refactorings achieve 74.1%. The MannâWhitney U-test shows no statistically significant difference between the two groups (í=50.0,í=1.00), indicating that refactoring scope does not substantially influence functional correctness. This result suggests that failures are not explained solely by whether edits are confined to a method or span a class. Instead, regressions likely arise from deeper issues such as incomplete context understanding, inconsistent renaming, type mismatches, or unintended behavioral changes. Our results show that LLM-based refactoring can preserve functional correctness, but behavioral regressions remain common across all evaluated models. Model choice has the strongest effect, with GPT-4o achieving the best overall functional correctness at an 80.8% pass rate; however, no model fully eliminates compilation or unit-test failures. In contrast, few-shot prompting and refactoring scope have limited and less consistent effects on correctness, suggesting that downstream validation and repair remain necessary for reliable LLM-based refactoring. 4.2 RQ2: What are the most common reasons for failures in LLM-refactored code? 4.2.1 Motivation. In the first research question, we find that a median of only 66.1% of StarCoder2-generated refactorings preserve functional correctness, and that neither few-shot prompting nor refactoring scope substantially improves unit-test pass rates. These results indicate that a significant portion of refactorings still introduce behavioral errors that are not mitigated by prompt-level adjustments alone. Therefore, understanding the root causes of unit test failures is critical to enhance LLM performance and testing methodologies. In this research question, we aim to analyze patterns in test failures and identify specific limitations in the LLM-refactored code. 4.2.2 Approach. To investigate common reasons for test failures in LLM-refactored code, we combine automated large-scale analysis with manual validation to identify recurring failure patterns and categorize them into a fixed taxonomy. For refactorings that fail unit tests, we analyze compilation errors and runtime test failures to identify common failure patterns. This analysis is performed on all failed refactorings to systematically categorize the root causes of failure. We then quantify the frequency of each failure category to understand the dominant failure modes of LLM-generated refactorings. These insights are further used to inform the design of our static-fix pipeline and guide the iterative repair process. Automated Failure Analysis. We employ an automated categorization step using GPT-4 [40], as it has demonstrated strong performance on code understanding and natural language reasoning tasks, to analyze all unit test failures. This process consists of the following stages: (1)LLM-based Summarization: For each failing case, GPT-4 produces a single-sentence natural language summary of the error log [42, 55]. (2)Embedding and Clustering: We encode all GPT-generated summaries using the all-MiniLM-L6-v2 SentenceTrans- former [58]. The resulting embeddings are clustered using KMeans [22]. (3) Cluster Optimization: To identify the most meaningful grouping of failures, we compute silhouette scores [38,47] across candidate cluster counts (2â12). The silhouette score measures how similar each data point is to others Manuscript submitted to ACM 14Cordeiro et al. 051015202530 Percentage of failures (%) Incomplete context / hallucination Incorrect / inconsistent renaming New functionality introduced Code incompleteness Syntax / structural errors Edge cases not handled Improper type handling Variables out of scope / dependency conflicts 24.3% 15.3% 13.7% 11.3% 9.7% 9.0% 8.7% 8.0% Fig. 7. Breakdown of the reasons for unit test failures from manual analysis. within its cluster compared to those in other clusters; higher scores indicate well-separated, coherent clusters. The best score (i.e., 0.34) occurs with eight clusters. Manual Verification. To evaluate the effectiveness of our automated failure analysis, we randomly select a statistical representative sample size of 350 failures, which corresponds to a 95% confidence level and a 5% margin of error [38,48, 61,62]. To this end, we select up to 35 unit test failures from each repository (for a total of 350) in our dataset to ensure balanced coverage across projects. As some repositories contain fewer than 35 failures, the final evaluation set consists of 300 unit test failures in total, while still preserving diversity across refactoring contexts and functional requirements. To validate the failure categories identified through automated analysis, we follow prior work [1,62] by employing a human evaluator and comparing the resulting categorizations with those produced by the LLM. Specifically, the first author independently analyzes each instance in the sampled set and determines its likely failure cause based on the refactored code, error logs, and test outcomes. Since our analysis is exploratory, neither the human evaluator nor the LLM assigns labels from a fixed, predefined set at the outset. Instead, both produce failure explanations that are subsequently normalized into a shared failure taxonomy for comparison. When the manual evaluation label corresponds to the same failure category as our automated approach, the case is treated as an agreement; otherwise, it is treated as a disagreement. To quantify the level of agreement between the automated and manual analyses, we compute Cohenâs kappa [8], which measures inter-rater agreement beyond chance. A high kappa score indicates that the automated clustering and labeling process is well aligned with the manually derived categories and therefore provides reliable support for identifying recurring test failure patterns at scale. 4.2.3 Findings. The manual analysis and alignment between the manually assigned labels and the GPT-4-assisted labels yielded a Cohenâs kappa of 0.78, indicating a high level of agreement [8]. This result suggests that the automatically identified clusters are largely consistent with the manually derived failure categories, supporting the reliability of our taxonomy for characterizing common unit test failure patterns in LLM-refactored code. Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring15 Many unit test failures stem from contextual misunderstandings and code-generation errors rather than semantic defects. The results of our manual evaluation reveal insights into the common reasons for unit test failures in LLM-refactored code. The findings are summarized in Figure 7, which illustrates the percent breakdown of the different failure reasons. Each failure reason is described in detail below. ⢠Incomplete context or hallucination is identified as the leading cause of failures, accounting for 24.3%. This category highlights the LLMâs inability to fully understand the broader context of the codebase during refactoring. Failures in this category often involved inappropriate or unrelated changes, such as introducing unnecessary methods or class members that were inconsistent with the original functionality. Such hallucinations reflect limitations in the modelâs ability to synthesize complex dependencies and maintain coherence within the refactored code. â˘Incorrect or inconsistent renaming contributes to 15.3% of test failures. These errors occurred when the LLM renamed variables, methods, or other identifiers inconsistently across the codebase. Inconsistent renaming resulted in broken references, logical mismatches, or runtime errors, particularly in cases where the renamed entities were used extensively within the code. â˘Code incompleteness is observed in 11.3% of the analyzed failures. These issues are characterized by refactored code outputs that are syntactically incomplete, such as missing return statements, incomplete control flow structures, or undefined variables. Such failures indicate that the model sometimes terminates refactorings prematurely or lacks the necessary mechanisms to ensure completeness in its output. â˘Edge cases not handled account for 9%. These issues arise when the refactored code does not account for specific edge cases, leading to incorrect functionality under certain input conditions. This highlights the modelâs inability to generalize effectively to less common scenarios, often due to a lack of training data covering diverse use cases. â˘Syntax and structural errors are responsible for 9.7% of failures. These errors included basic violations such as unmatched parentheses, invalid syntax, or improper formatting. â˘Adding new functionality or variables accounts for 13.7% of failures. This category involves the unintended introduction of elements that deviate from the original functionality, often leading to test failures due to unforeseen behavioral changes. These issues underscore the need for stricter constraints on the scope of LLM- generated refactorings to ensure alignment with the original code intent. â˘Variables outside scope or conflicts account for 8% of the total unit test failures analyzed. Failures in this category arise when variables are accessed outside their defined scope or when refactorings introduce conflicts with existing dependencies. These issues often result from the LLMâs lack of awareness of scoping rules and project-level dependencies during refactoring. â˘Improper type handling contributes to 8.7% (i.e., 26 out of 300 cases) of failures. These failures typically involve mismatched data types in variable assignments or method signatures, leading to compilation or runtime errors. These issues indicate that while LLMs are generally capable of inferring data types, they often struggle with maintaining type safety in complex contexts. Manuscript submitted to ACM 16Cordeiro et al. Our analysis shows that unit test failures in LLM-refactored code are dominated by contextual misunderstandings, hallucinated changes, and structural issues such as inconsistent renaming and incomplete code. These failure modes indicate that many incorrect refactorings arise from limitations in context comprehension and output completeness rather than from complex semantic errors, motivating targeted mitigation strategies such as static fixes and feedback-driven repair mechanisms. 4.3 RQ3: Can our agentic approach improve the correctness of LLM-refactored code? 4.3.1 Motivation. Our findings from the second research question reveal that many incorrect LLM-generated refac- torings fail due to syntactic and shallow structural issues, such as missing imports, unbalanced brackets, improper type handling, and incomplete code fragments. These failure modes suggest that a substantial portion of unit test failures arises from easily identifiable and potentially correctable issues introduced during refactoring generation. However, static corrections alone are insufficient to resolve all failing refactorings, as some failures stem from deeper semantic inconsistencies, contextual misunderstandings, or incorrect transformations that require LLM reasoning. Therefore, in this research question, we investigate whether an agentic approach that integrates static fixes and an iterative repair loop can improve the functional correctness of LLM-generated refactorings. By applying rule-based corrections before each compilation and test cycle, and then invoking targeted LLM-based repair only when needed, our approach aims to recover incorrect refactorings while minimizing unnecessary edits and computational overhead. 4.3.2 Approach. To evaluate whether RefactorAssist improves the correctness of LLM-generated refactorings, we apply it to refactorings that fail compilation or unit testing after the initial generation stage. As described in Section 2, RefactorAssist first applies static fixes to address common syntactic and structural issues. Refactorings that still fail are then passed to the iterative agentic repair loop, where compiler diagnostics, unit-test failures, code diffs, retrieved project context, and diagnostic explanations are used to guide repair generation. Repair Generation Model. We use StarCoder2 [32] as both the initial refactoring-generation model and the repair-generation model inside RefactorAssist. This setup helps us control for the effect of model substitution, ensuring that any improvement in correctness is attributed to RefactorAssistâs static fixes, agentic repair, and repair guidance rather than to replacing the original model with a stronger code generation model. Error Diagnostic Explanation Models. For the error diagnostic explanation component, we evaluate three LLMs: GPT-4o [40], Claude 3.5 Sonnet [2], and Llama-3 8B [19]. These models are used only to generate error explanations and repair hints; they do not directly edit the code. We select these models to compare diagnostic performance across different levels of capability, scale, and accessibility. GPT-4o and Claude 3.5 Sonnet provide strong reasoning ability and large context windows for handling long error traces and multi-file repair contexts, while Llama-3 8B serves as a lightweight, locally deployable open-source alternative. We report the results of all three models in the same evaluation. To evaluate the performance of the repair pipeline, we measure the unit test pass criteria defined in the first research question. In addition, we record the iteration at which each successful repair occurs and track the cases that remain unresolved after the iteration limit, allowing us to assess both repair effectiveness and repair efficiency over time. 4.3.3 Findings. Static fixes increase the median functional correctness of LLM-generated refactorings by 8.3 percentage points before entering the agentic refactoring repair stage of RefactorAssist. Figure 8 compares project-level unit test pass rates for original and static-fixed refactorings. Across projects, the median pass rate increases from 66.1% in the Original setting to 74.4% after applying static fixes, corresponding to a median improvement of 8.3 percentage points. After applying the static-fix step in RefactorAssist, 80.3% of all refactorings pass the unit tests. The Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring17 OriginalStatic Fix 0 20 40 60 80 100 Pass rate (%) Median: 66.1% Median: 74.4% Fig. 8. Project-level unit test pass rates for original and static-fixed refactorings. Boxes show the distribution across projects. BaseStatic12345678910 Repair Stage 60 65 70 75 80 85 90 95 100 Overall % of refactorings passing unit tests Fig. 9. Overall unit test pass rate across the full refactoring repair pipeline. The first point shows the base pass rate of the original LLM-generated refactorings, the second shows performance after the static-fix stage, and the remaining points show the cumulative improvement after each iteration of the agentic refactoring repair stage. remaining 19.7% of refactored methods and classes still fail and therefore proceed to the agentic refactoring repair stage of RefactorAssist. The combined static-fix and iterative repair pipeline substantially improves the correctness of failing LLM- generated refactorings, reaching 93.4% pass rate after 10 iterations. Figure 9 shows the cumulative percentage of failures repaired by iterationíusing the full repair pipeline, which combines static fixes, GPT-4o-based error Manuscript submitted to ACM 18Cordeiro et al. diagnosis, retrieved project context, code diffs, error logs, and StarCoder2-based repair. The base pass rate of the original LLM-generated refactorings is 68.4%. After the static-fix stage, this increases to 80.3%. The remaining failures then enter the agentic refactoring repair stage, which raises the overall pass rate further over successive iterations, reaching 85.1% after one iteration, 87.8% after two iterations, 89.7% after three iterations, and 93.4% after ten iterations. Most successful repairs occur in the early iterations of the agentic refactoring repair stage. The largest gains occur within the first three repair attempts, after which the rate of improvement slows substantially. This pattern suggests that many failures can be resolved once the model is provided with focused diagnostic evidence and minimal repair guidance, while later iterations mainly address a smaller subset of harder cases. The diminishing gains after the third iteration also support the use of a bounded repair budget, as most practically recoverable failures are resolved early in the process. Remaining failures are likely associated with deeper semantic inconsistencies. Although RefactorAssist substantially improves correctness, a portion of failures (e.g., 6.6%) remains unresolved after the iteration limit of 10. Consistent with the failure analysis in the second research question, these likely correspond to cases involving broader contextual misunderstandings, inconsistent renaming across scopes, hallucinated functionality, or other semantic issues that cannot be resolved through lightweight structural correction and localized iterative repair alone. The combined static-fix and agentic refactoring repair stage substantially improves the functional correctness of LLM-generated refactorings. The overall unit test pass rate increases from 68.4% for the original refactorings to 80.3% after static intervention, and further rises through iterative repair to 93.4%. Most successful repairs occur within the first three iterations, indicating that targeted diagnostic feedback is effective for recovering a large portion of incorrect refactorings. 4.4 RQ4: Which components of our approach contribute most to repair effectiveness? 4.4.1 Motivation. In the third research question we observe that the combined static-fix and agentic refactoring repair stage can substantially improve the correctness of failing LLM-generated refactorings. However, the repair pipeline consists of several interacting components, including code diffs, compiler and runtime error logs, explanation-LLM output, and retrieval-augmented project context. It remains unclear which of these components are most responsible for the observed repair gains, and whether stronger diagnosis LLMs consistently provide better repair guidance. Understanding the relative contribution of each component is important for both practical deployment and future agent design, as it helps identify which sources of feedback are essential and which provide only limited or situational benefit. Therefore, in this research question, we conduct an ablation study to determine how different diagnostic inputs and explanation LLMs affect repair effectiveness. 4.4.2 Approach. To understand which components of our repair pipeline drive the largest gains, we perform an ablation study over the four main inputs to the repair prompt: (1) code diff, (2) compiler/runtime error log, (3) explanation-LLM output, and (4) retrieval-augmented project context. Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring19 Table 1. Ablation Study: Cumulative percent of initial errors fixed by iterationífor each feedback-loop configuration (10 iterations). Cumulative errors fixed by iteration í (%) Error Diagnostic Agent Step123456789 10 GPTâ4oDiff + ErrLog + Expl24.338.047.854.059.663.566.569.370.5 70.8 Claude 3.5 SonnetDiff + ErrLog + Expl22.937.346.052.957.060.563.165.366.566.7 Llama 3 (8B)Diff + ErrLog + Expl19.631.739.644.749.152.154.556.457.958.1 GPTâ4oDiff + ErrLog + Expl + RAG21.734.943.850.655.859.962.865.167.067.3 Claude 3.5 SonnetDiff + ErrLog + Expl + RAG22.435.043.149.153.957.660.462.564.064.2 Llama 3 (8B)Diff + ErrLog + Expl + RAG18.729.936.842.245.848.550.752.553.653.7 Diff + ErrLog14.823.328.331.834.536.337.938.939.539.7 Diff + RAG13.120.325.829.031.833.935.336.336.836.9 GPTâ4oExpl only12.819.324.227.329.831.532.733.834.434.6 RAG only8.713.717.319.821.623.124.225.025.525.6 Diff only8.913.416.218.320.021.322.423.123.523.6 Step legend: Diff = code diff; ErrLog = compiler/runtime error log; Expl = agent-generated explanation and fix hint; RAG = retrieval-augmented context. Notes: Baseline pass rate before entering the feedback loop is 80.3%, so the initial error pool is 19.7% of methods; percentages are reported relative to this pool. Rows labeled with Expl use an LLM as an explanation agent (GPT-4o, Claude 3.5 Sonnet, or Llama 3 (8B)). Rows without Expl are signal-only baselines that do not invoke an explanation agent and are included to isolate the contribution of each feedback signal. Iteration 10 contributesâ0% additional fixes due to diminishing returns. For the explanation step, we evaluate three diagnosis LLMs: GPT-4o [40], Claude 3.5 Sonnet [2], and Llama-3 8B [19]. These models are used exclusively to produce structured failure explanations and fix hints; they do not directly edit the code. We evaluate all combinations of the four repair inputs and record, for each configuration, the percentage of failing refactorings repaired by iterationí. We also record the number of iterations required for each successful repair and track the failures that remain unresolved after the iteration limit. To compare configurations statistically, we apply the Scott-Knott ESD [13,41] test at the 0.05 level to the per-project fix-rate distributions [9]. This test partitions the evaluated configurations into statistically distinct performance groups, allowing us to identify which combinations of diagnostic inputs yield significantly different levels of repair effectiveness while accounting for multiple comparisons. 4.4.3 Findings. Explanation-aware configurations achieve the strongest repair performance, with the best configuration repairing 70.8% of failures remaining after static intervention for a total refactoring success rate of 94.2%. The cumulative results in Table 1 and Figure 10 show that configurations combining code diffs, error logs, and explanation-LLM output consistently achieve the highest repair rates. The values in Figure 10 are computed over the failures that remain after the static-fix stage, which correspond to 19.7% of all refactorings. Among the remaining failures, the strongest configuration, which uses the code diff, error log, explanation output, and GPT-4o as the error diagnostic LLM, repairs 24.3% in the first iteration and 38.0% by the second iteration. By the third iteration, it fixes 47.8% of all recoverable failures, and after ten iterations reaches a cumulative recovery rate of 70.8%. These results indicate that explicit failure explanation is a critical ingredient in effective iterative repair. The quality of the error diagnostic LLM strongly influences repair effectiveness. Claude 3.5 Sonnet and Llama 3 (8B) exhibit the same overall trend but achieve lower final recovery rates. Specifically, Claude 3.5 Sonnet reaches 66.7% after ten iterations, while Llama 3 (8B) recovers 58.1% of errors under the same configuration. The differences Manuscript submitted to ACM 20Cordeiro et al. 12345678910 Iteration (k) 0 20 40 60 80 100 Cumulative % of initial errors fixed Diff + ErrLog + Expl (GPT-4o) Diff + ErrLog + Expl (Claude 3.5 Sonnet) Diff + ErrLog + Expl (Llama 3 (8B)) Diff + ErrLog + Expl + RAG (GPT-4o) Diff + ErrLog + Expl + RAG (Claude 3.5 Sonnet) Diff + ErrLog + Expl + RAG (Llama 3 (8B)) Diff + ErrLog Expl only (GPT-4o) RAG only Diff + RAG Diff only Fig. 10. Cumulative percentage of failures remaining after the static-fix stage repaired by iterationíacross ablation configurations. These failures correspond to the 19.7% of refactorings that still fail after static intervention. indicate that stronger explanation LLMs produce more accurate root-cause diagnoses and repair hints, which directly improve downstream patch generation. Retrieval-augmented generation accelerates early fixes but is not sufficient for the final recovery. Adding retrieval (â+RAGâ) yields modest gains in the first one to two iterations (e.g., 21.7% vs. 24.3% at iteration one without RAG) but results in lower final coverage (67.3% vs. 70.8%). This suggests that retrieval primarily assists in quickly resolving familiar or surface-level issues, while later iterations benefit more from precise failure explanations than from additional context retrieval. Adding retrieval-augmented context does not yield statistically significant gains in final recovery. For GPT-4o, the Diff + ErrLog + Expl configuration (mean 70.8%) is statistically indistinguishable from its RAG-augmented variant (67.3%), confirming that retrieval primarily accelerates early fixes rather than increasing overall repair coverage. A similar pattern holds for Claude 3.5 Sonnet, where adding RAG shifts the configuration from Rank 1 to Rank 2. Explanation-aware configurations significantly outperform all other agent designs. Table 2 reports the ScottâKnott ESD grouping of agentic refactoring repair stage configurations based on the percentage of fixable errors. The top-ranked group (Rank 1) consists of configurations that combine code diff, error logs, and explanation-LLM output: GPT-4o (with and without RAG) and Claude 3.5 Sonnet without RAG. These configurations achieve the highest repair rates, with mean fixable percentages ranging from 66.7% to 70.8%, and are statistically superior to all other variants based on the Scott-Knott ESD test. Weaker explanation LLMs lead to substantially lower repair rates. Configurations using Llama 3 (8B) consistently occupy Rank 3, with mean fixable percentages between 53.7% and 58.1%, indicating a clear drop in effectiveness compared to GPT-4o and Claude 3.5 Sonnet. This gap highlights the role of high-quality failure explanations in guiding successful repairs. Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring21 Table 2. ScottâKnott ESD grouping of feedback-loop configurations based on fixable percentage (%). Lower ranks indicate significantly better performance (Rank 1 is the best rank). ConfigurationMean Median SK Rank Diff + ErrLog + Expl (GPT-4o)70.866.151 Diff + ErrLog + Expl + RAG (GPT-4o)67.371.201 Diff + ErrLog + Expl (Claude 3.5 Sonnet)66.767.301 Diff + ErrLog + Expl + RAG (Claude 3.5 Sonnet)64.266.602 Diff + ErrLog + Expl (Llama 3 (8B))58.156.403 Diff + ErrLog + Expl + RAG (Llama 3 (8B))53.758.953 Diff + ErrLog39.738.404 Diff + RAG36.938.305 Expl only (GPT-4o)34.633.905 RAG only25.626.306 Diff only23.623.657 Partial or single-input configurations perform poorly. Ablations that omit the explanation LLM, such as Diff + ErrLog (39.7%), Diff + RAG (36.9%), or Expl-only (34.6%), fall into lower ranks (Ranks 4-5). Minimal configurations using only RAG or only the diff achieve the lowest repair rates (25.6% and 23.6%, respectively), demonstrating that no single component is sufficient for effective automated repair. Finally, the sharp drop in marginal improvements after the first few iterations, together with the dominance of explanation-aware configurations, suggests that most fixable errors are shallow and diagnostic-driven, whereas remaining failures likely require deeper semantic reasoning or human intervention. RefactorAssist can substantially improve the functional correctness of failing LLM-refactored code, with most recoverable errors fixed within the first three iterations. RefactorAssist configurations that combine code diffs, error logs, and the error diagnostic LLM consistently achieve the highest repair rates, with the highest being 70.8% with GPT-4o as the error diagnostic LLM. The effectiveness of the repair process is strongly influenced by the quality of the explanation LLM, indicating that accurate failure diagnosis is critical for successful automated repair. 5 Threats to Validity This section outlines potential threats to the validity of our study and describes the measures taken to mitigate them. 5.1 Threats to Internal Validity Although we filter overlapping repositories that are excluded from the Stack-v2 dataset and not included in StarCoder2âs training data, it is possible that the model learned general patterns or styles from similar codebases. To mitigate this risk, we select 10 projects with distinct characteristics, ensuring minimal similarity with common training examples. Another threat is the inherent challenge of hallucinations in LLM-generated refactorings. These hallucinations may introduce syntactically valid but semantically incorrect changes. To address this issue, all refactored code is validated using unit tests, and failure cases are analyzed to identify and categorize recurring patterns. The mapping of test cases to methods depends on heuristic and static analysis techniques, which may overlook certain relationships, particularly in cases involving dynamically generated code or complex dependency injection frameworks. To reduce this risk, we compile and run the corresponding unit test for each method to ensure its validity before adding it to our dataset. Lastly, the selection of commits and repositories could introduce bias. Although we randomize the selection process within the curated dataset, the exclusion of certain repositories based on filtering criteria may inadvertently favor simpler Manuscript submitted to ACM 22Cordeiro et al. codebases or more well-documented projects. To mitigate this, we ensure a representative distribution of projects across various domains, sizes, and complexities. 5.2 Threats to External Validity The generalizability of our findings is limited to the scope of the study, which focuses on open-source Java projects with sufficient commit history and test coverage. These findings may not directly extend to other programming languages or proprietary software systems. Furthermore, our evaluation is based on a specific version of StarCoder2, and results may vary with updates to the model architecture or future LLMs. Additionally, the dataset primarily includes repositories with robust test coverage, which may not reflect real-world scenarios where test cases are incomplete or absent. This could lead to an overestimation of the modelâs performance in environments with limited testing infrastructure. Our evaluation is also limited by the scope of the refactoring targets. Since Methods2Test [57] provides method-level mappings between production code and unit tests, we restrict the evaluated refactoring targets to single-method and single-class changes so that behavioral outcomes can be attributed more directly to the intended transformation. This restriction helps reduce unrelated cross-file edits, build configuration changes, or modifications outside the mapped method-test relationship. However, this design may limit generalizability to multi-file refactorings, where correctness often depends on interactions among several classes, interfaces, and imports. To mitigate this limitation, RefactorAssist is not restricted to the target method or class when diagnosing failures; when compiler errors, test failures, or code diffs indicate cross-file dependencies, it can use repository context during iterative refinement. To mitigate these threats, we evaluate the model on a diverse set of real-world projects from multiple domains, use repositories with established development histories and unit tests to ensure reliable behavioral assessment, and report our experimental setup in sufficient detail to support reproducibility and future replication across other models, languages, and software settings. 5.3 Threats to Construct Validity We use unit test pass rates as the primary metric for evaluating functional correctness. While this metric provides an objective measure, it may not fully capture other aspects of code quality, such as readability, maintainability, or adherence to design principles. Subtle issues related to long-term code maintenance are not directly assessed in this study. Our statistical analyses, including significance testing and effect size reporting, rely on the sample size and distribution of test outcomes. Small sample sizes in specific failure categories or project domains could reduce the statistical power of our conclusions. To address this, we ensure adequate sample sizes and report p-values to provide clarity. Finally, the automated extraction of test-method mappings introduces potential errors. Any inaccuracies in the mapping process could affect the evaluation of refactorings. Because Methods2Test [57] links methods to associated unit tests, our evaluation assumes that the mapped tests are suitable for checking whether the refactored code behaves correctly. Inaccurate or incomplete mappings could therefore affect whether a passing test outcome fully reflects functional correctness. To mitigate this, we validate the mappings in our dataset by compiling and running the associated tests for each target before evaluation to exclude cases where the original method-test pair is not executable. 5.4 Implications Our findings provide several implications for practitioners, researchers, and tool developers working with LLM-based code refactoring. Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring23 Implications for Practitioners. Our results show that while LLMs can produce code refactorings, they frequently introduce functional regressions. This suggests that LLM-generated refactorings should not be directly integrated into production systems without validation. Practitioners should incorporate automated testing and validation pipelines when adopting LLM-assisted refactoring tools. Our findings also highlight the role of static fixes, which can eliminate common syntactic and structural issues before testing and thereby improve the correctness of generated refactorings at low cost without the invocation of LLMs. In addition, the strong gains achieved by the iterative repair mechanism suggest that agent-based validation and repair can substantially improve correctness with limited manual intervention. In a broader multi-agent software engineering ecosystem, our approach can therefore serve as a dedicated refactoring agent that proposes quality-improving transformations while coordinating with testing agents to preserve functional behavior. Implications for Researchers. This study highlights the gap between structural improvements and functional correctness in LLM-generated refactorings. Future research should focus on improving the reliability of LLMs in preserving program semantics, particularly in multi-file and dependency-aware settings. Our results also suggest that combining static fixes with iterative agentic repair is a promising direction for improving robustness, as these stages address different classes of failures and together substantially improve functional correctness. The observed failure patterns further provide a foundation for developing more targeted benchmarks, taxonomies, and evaluation frameworks for refactoring tasks. More broadly, our findings suggest that refactoring should be studied not only as a one-step code generation problem, but as a multi-stage workflow involving generation, validation, diagnosis, and repair. This opens opportunities for future work on agent-based refactoring systems, better retrieval mechanisms for project-specific context, and training or fine-tuning strategies tailored to semantics-preserving code transformation rather than general-purpose code generation. Implications for Tool Developers. The effectiveness of static fixes and iterative repair pipelines suggests that LLM-based refactoring tools should be designed as hybrid systems rather than standalone generators. Instead of relying on a single model invocation, tool builders should integrate compiler feedback, test execution, error diagnosis, and repair into the refactoring workflow to improve output reliability. Our findings also suggest the value of modular architectures in which different models or components can be assigned specialized roles, such as initial refactoring, failure diagnosis, and targeted repair. This design enables flexibility across deployment settings while making it easier to incorporate retrieval mechanisms, project-specific context, and validation feedback into the overall refactoring pipeline. 6 Related Work This section reviews key areas of related work, focusing on automated refactoring, testing LLM-generated code, and approaches to repairing failing refactorings. Static Code Refactoring. Automated code refactoring tools aim to improve software quality by restructuring code without altering its functionality [16]. Early refactoring tools, such as JRefactory [4] and Eclipseâs Refactoring Engine [53], are rule-based, relying on static analysis to apply predefined transformations. While these tools effectively handle straightforward tasks such as renaming variables or extracting methods, they often require manual intervention to address more complex, context-dependent refactoring scenarios. In contrast to these rule-based approaches, our work evaluates LLM-generated refactorings at the method-level and class-level, and focuses on producing refactorings as well as whether their functional correctness can be preserved and recovered through automated repair. Manuscript submitted to ACM 24Cordeiro et al. Automated Code Refactoring with LLMs. The advent of LLMs, such as Codex [7], GPT-4 [40], and StarCoder2 [32], has enabled more flexible and intelligent refactoring solutions. These models generate human-like code and structural changes, offering new possibilities for automated code improvement, including tasks such as Extract Method refactoring guided by LLM suggestions [46]. However, studies highlight that while LLMs excel in reducing code smells and simplifying logic [10,11], they often fail to preserve semantic correctness, frequently producing hallucinated or incorrect transformations that lead to unit test failures [50]. Unlike prior studies that primarily assess whether LLMs can generate refactorings or reduce code smells, our work systematically examines their behavioral correctness through unit testing and further investigates how incorrect refactorings can be recovered using static fixes and an iterative repair pipeline. Testing LLM-Generated Code. Unit testing plays a critical role in validating the functional correctness of LLM- generated refactorings. Tools such as EvoSuite [18] generate test cases that aim to achieve high structural coverage, but they may not capture the requirements of real-world scenarios. The disparity between automatically generated and manually developed tests is a key challenge in assessing the reliability of LLM-generated refactorings [5]. Research has also explored integrating LLMs into Test-Driven Development (TDD) workflows. For instance, LLM4TDD leverages LLMs to generate code from test cases, demonstrating improved code quality when high-quality tests are provided [44]. The approach introduced by LLM4TDD can be integrated into an iterative feedback loop to generate refactorings based on unit tests. Mutation testing, another evaluation strategy, has been used to identify flaws in refactored code by injecting artificial defects and assessing whether the test suite detects them [43]. Building on this line of work, our study uses unit testing not only as an evaluation mechanism for LLM-generated refactorings, but also as an integral feedback signal that drives iterative diagnosis and repair of incorrect transformations. LLM-Based Refactoring Repair. While LLMs can propose meaningful refactoring changes, they frequently introduce errors that require manual correction. This limitation has spurred research into automated refactoring repair mechanisms, which aim to identify and fix issues in LLM-generated refactorings. Automated Program Repair (APR) techniques, such as those based on evolutionary algorithms [29,35] and neural repair models [20], have been adapted to validate and refine LLM-generated code. These methods typically focus on applying minimal changes to pass failing tests, aligning closely with the goals of this study. Shirafuji et al. [50] explore leveraging few-shot prompting to improve the quality of LLM-generated refactorings. While effective, their approach relies heavily on prompt design and does not dynamically adapt to real-world test results. Similarly, Liu et al. [31] propose using few-shot prompts to guide LLMs, significantly improving refactoring success rates but highlighting the need for iterative refinement to address persistent errors. Our work combines failure analysis, static correction, and an agentic refactoring repair stage to automatically diagnose and recover incorrect LLM-generated refactorings based on observed compilation and test outcomes. 7 Conclusion This study examines the common problems in generating automated code refactorings using large language models and explores designing a refactoring repair agent for improving their functional correctness. Our investigation provides both empirical evidence of the current limitations in LLM-driven refactoring and enhancement strategies. We find that StarCoder2 achieves a median project-level unit test pass rate of only 66.1%, indicating that many generated refactorings fail to preserve intended functionality. Analysis of method versus class-level refactoring changes shows no statistically significant difference in pass rates, suggesting that refactoring scope alone does not meaningfully affect functional correctness. Furthermore, we show that the common failure patterns include incomplete context Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring25 understanding, inconsistent renaming, and missing edge-case handling. To address these challenges, we design a two-stage repair strategy. We first apply static fixes that target syntactic and structural issues, increasing the median unit test pass rate from 66.1% to 74.4%. Second, we introduce RefactorAssist, which is an agentic approach that integrates unit test failure logs, code diffs, and LLM-generated explanations into the repair prompt. RefactorAssist uses the error diagnostic component and incorporates both the code diff and error log, which repairs up to 70.8% of the initially failing refactorings after static treatment and produces a total cumulative pass rate of 94.2%. These results advance the understanding of LLM-based refactoring by quantifying where current models succeed and fail, offering concrete targets for training data and algorithmic improvements. By identifying common refactoring failure patterns and demonstrating an effective agentic repair approach, this work lays the foundation for more robust refactoring assistants that can be reliably deployed in real-world software engineering pipelines. Future work will explore integrating static analysis and semantic reasoning tools into the agentic refactoring repair stage, fine-tuning LLMs on refactoring-specific datasets, and scaling this framework to more diverse programming languages and project types. References [1] Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can LLMs Replace Manual Annotation of Software Engineering Artifacts?. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). 526â538. doi:10.1109/MSR66628.2025.00086 [2]Anthropic. 2024. Claude 3.5 Sonnet. Model release. https://w.anthropic.com/news/claude-3-5-sonnet Released June 20, 2024; available via Claude.ai, iOS app, Anthropic API, Amazon Bedrock, Google Vertex AI. [3]Vaibhav Anu, Wenhua Hu, Jeffrey C Carver, Gursimran S Walia, and Gary Bradshaw. 2018. Development of a human error taxonomy for software requirements: A systematic literature review. Information and Software Technology 103 (2018), 112â124. doi:10.1016/j.infsof.2018.06.011 [4] M. Bittman, R. Roos, and G.M. Kapfhammer. 2001. Creating a Free, Dependable Software Engineering Environment for Building Java Applications. In 1st Workshop on Open Source Software Engineering at ICSE 2001. [5] Jason Brownlee. 2020. Data leakage in machine learning. https://machinelearningmastery.com/data-leakage-machine-learning/ [6]Yupeng Chang et al.2024. A Survey on Evaluation of Large Language Models. ACM Trans. Intell. Syst. Technol. 15, 3, Article 39 (March 2024), 45 pages. doi:10.1145/3641289 [7] Mark Chen et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG] https://arxiv.org/abs/2107.03374 [8] Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37â46. [9] Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences (2 ed.). Routledge. doi:10.4324/9780203771587 [10] Jonathan Cordeiro, Shayan Noei, and Ying Zou. 2025. LLM-Driven Code Refactoring: Opportunities and Limitations. In 2025 IEEE/ACM Second IDE Workshop (IDE). 32â36. doi:10.1109/IDE66625.2025.00011 [11] Jonathan Cordeiro, Shayan Noei, and Ying Zou. 2026. An Empirical Study on the Code Refactoring Capability of Large Language Models. ACM Trans. Softw. Eng. Methodol. (March 2026). doi:10.1145/3801158 Just Accepted. [12] Alexis Dinno. 2015. Nonparametric pairwise multiple comparisons in independent groups using Dunnâs test. The Stata Journal 15, 1 (2015), 292â300. [13]Enio G. Jelihovschi, Jose Claudio Faria, and Ivan Bezerra Allaman. 2014. ScottKnott: A Package for Performing the Scott-Knott Clustering Algorithm in R. Trends in Applied and Computational Mathematics 15, 1 (2014), 3â17. https://tema.sbmac.org.br/tema/article/view/646/643 [14]Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). 31â53. doi:10.1109/ICSE-FoSE59343.2023.00008 [15]Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated Repair of Programs from Large Language Models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 1469â1481. doi:10.1109/ICSE48619.2023.00128 [16] Martin Fowler. 2018. Refactoring: improving the design of existing code. Addison-Wesley Professional. [17]M. Fowler, K. Beck, J. Brant, W. Opdyke, and D. Roberts. 1999. Refactoring: Improving the Design of Existing Code. Addison-Wesley Professional, Berkeley, CA, USA. [18] Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (Szeged, Hungary) (ESEC/FSE â11). Association for Computing Machinery, New York, NY, USA, 416â419. doi:10.1145/2025113.2025179 [19] Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783 [20] Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. Deepfix: Fixing common c language errors by deep learning. In Proceedings of the aaai conference on artificial intelligence, Vol. 31. Manuscript submitted to ACM 26Cordeiro et al. [21] Refactoring Guru. 2024. https://refactoring.guru/refactoring/techniques [22]John A Hartigan and Manchek A Wong. 1979. Algorithm AS 136: A k-means clustering algorithm. Journal of the royal statistical society. series c (applied statistics) 28, 1 (1979), 100â108. [23] Lei Huang et al.2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Trans. Inf. Syst. 43, 2, Article 42 (Jan. 2025), 55 pages. doi:10.1145/3703155 [24] Binyuan Hui et al. 2024. Qwen2.5-Coder Technical Report. arXiv:2409.12186 [cs.CL] https://arxiv.org/abs/2409.12186 [25] JavaParser Community. 2024. JavaParser. https://javaparser.org/ Open-source library for parsing and analyzing Java source code. [26]Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why donât software developers use static analysis tools to find bugs?. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 672â681. [27] JUnit Team. 2024. JUnit 5. https://junit.org/junit5/ Java testing framework for unit and integration testing. [28] Sayash Kapoor and Arvind Narayanan. 2023. Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4, 9 (2023). [29] Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2011. Genprog: A generic method for automatic software repair. Ieee transactions on software engineering 38, 1 (2011), 54â72. [30] Valentina Lenarduzzi, Fabiano Pecorelli, Nyyti Saarimaki, Savanna Lujan, and Fabio Palomba. 2023. A critical comparison on six static analysis tools: Detection, agreement, and precision. Journal of Systems and Software 198 (2023), 111575. doi:10.1016/j.jss.2022.111575 [31]Bo Liu, Yanjie Jiang, Yuxia Zhang, Nan Niu, Guangjie Li, and Hui Liu. 2024. An Empirical Study on the Potential of LLMs in Automated Software Refactoring. arXiv:2411.04444 [cs.SE] https://arxiv.org/abs/2411.04444 [32] Anton Lozhkov et al. 2024. StarCoder 2 and The Stack v2: The Next Generation. arXiv:2402.19173 [cs.SE] https://arxiv.org/abs/2402.19173 [33] PatrickE.McKightandJuliusNajab.2010.Kruskal-Wallis Test.JohnWiley&Sons,Ltd,1â1. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780470479216.corpsy0491 doi:10.1002/9780470479216.corpsy0491 [34]PatrickE.McKnightandJuliusNajab.2010.Mann-Whitney U Test.JohnWiley&Sons,Ltd,1â1. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780470479216.corpsy0524 doi:10.1002/9780470479216.corpsy0524 [35]Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: scalable multiline program patch synthesis via symbolic analysis. In Proceedings of the 38th International Conference on Software Engineering (Austin, Texas) (ICSE â16). Association for Computing Machinery, New York, NY, USA, 691â701. doi:10.1145/2884781.2884807 [36] Frederic P. Miller, Agnes F. Vandome, and John McBrewster. 2010. Apache Maven. https://api.semanticscholar.org/CorpusID:86808749 [37] Emerson Murphy-Hill and Andrew P Black. 2008. Refactoring tools: Fitness for purpose. IEEE software 25, 5 (2008), 38â44. [38] Shayan Noei, Heng Li, Stefanos Georgiou, and Ying Zou. 2023. An Empirical Study of Refactoring Rhythms and Tactics in the Software Development Process. IEEE Transactions on Software Engineering 49, 12 (2023), 5103â5119. doi:10.1109/TSE.2023.3326775 [39]Shayan Noei, Heng Li, and Ying Zou. 2025. An Empirical Study on Release-Wise Refactoring Patterns. Proc. ACM Softw. Eng. 2, FSE, Article FSE019 (June 2025), 22 pages. doi:10.1145/3715734 [40] OpenAI, :, Aaron Hurst, et al. 2024. GPT-4o System Card. arXiv:2410.21276 [cs.CL] https://arxiv.org/abs/2410.21276 [41]Mohamed Ouf, Shayan Noei, Zeph Van Iterson, Mariam Guizani, and Ying Zou. 2026. An Empirical Analysis of Community and Coding Patterns in OSS4SG vs. Conventional OSS. arXiv preprint arXiv:2601.03430 (2026). [42]Sheena Panthaplackel, Pengyu Nie, Milos Gligoric, Junyi Jessy Li, and Raymond Mooney. 2020. Learning to Update Natural Language Comments Based on Code Changes. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, Online, 1853â1868. doi:10.18653/v1/2020.acl-main.168 [43]Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Chapter Six - Mutation Testing Advances: An Analysis and Survey. Advances in Computers, Vol. 112. Elsevier, 275â378. doi:10.1016/bs.adcom.2018.03.015 [44]Sanyogita Piya and Allison Sullivan. 2024. LLM4TDD: Best Practices for Test Driven Development Using Large Language Models. In Proceedings of the 1st International Workshop on Large Language Models for Code (Lisbon, Portugal) (LLM4Code â24). Association for Computing Machinery, New York, NY, USA, 14â21. doi:10.1145/3643795.3648382 [45] PMD Open Source Contributors. 2024. PMD. https://pmd.github.io Software for Static Code Analysis. [46]Dorin Pomian, Abhiram Bellur, Malinda Dilhara, Zarina Kurbatova, Egor Bogomolov, Andrey Sokolov, Timofey Bryksin, and Danny Dig. 2024. EM-Assist: Safe Automated ExtractMethod Refactoring with LLMs. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (Porto de Galinhas, Brazil) (FSE 2024). Association for Computing Machinery, New York, NY, USA, 582â586. doi:10.1145/3663529.3663803 [47]Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics 20 (1987), 53â65. [48]Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, and Xipeng Qiu. 2024. Balanced Data Sampling for Language Model Training with Clustering. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 14012â14023. doi:10.18653/v1/2024.findings-acl.833 [49]T. Sharma, P. Mishra, and R. Tiwari. 2016. Designite: A software design quality assessment tool. In Proceedings of the 1st International Workshop on Bringing Architectural Design Thinking into Developersâ Daily Activities. 1â4. [50]Atsushi Shirafuji, Yusuke Oda, Jun Suzuki, Makoto Morishita, and Yutaka Watanobe. 2023. Refactoring Programs Using Large Language Models with Few-Shot Examples. In 2023 30th Asia-Pacific Software Engineering Conference (APSEC). 151â160. doi:10.1109/APSEC60848.2023.00025 Manuscript submitted to ACM RefactorAssist: Agentic Refinement for Reliable Code Refactoring27 [51] SonarSource S.A. 2024. SonarQube. https://w.sonarsource.com/products/sonarqube Software for Static Code Analysis. [52] SpotBugs Team. 2024. SpotBugs. https://spotbugs.github.io Software for Static Code Analysis. [53]David Steinberg, Frank Budinsky, Marcelo Paternostro, and Ed Merks. 2009. EMF: Eclipse Modeling Framework 2.0 (2nd ed.). Addison-Wesley Professional. [54]Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. IntelliCode compose: code generation using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1433â1443. doi:10.1145/3368089.3417058 [55]Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang. 2021. On the Evaluation of Commit Message Generation Models: An Experimental Study. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). 126â136. doi:10.1109/ICSME52107.2021.00018 [56] Nikolaos Tsantalis, Ameya Ketkar, and Danny Dig. 2020. RefactoringMiner 2.0. IEEE Transactions on Software Engineering 48, 3 (2020), 930â950. [57]Michele Tufano, Shao Kun Deng, Neel Sundaresan, and Alexey Svyatkovskiy. 2022. Methods2Test: a dataset of focal methods mapped to test cases. In Proceedings of the 19th International Conference on Mining Software Repositories (Pittsburgh, Pennsylvania) (MSR â22). Association for Computing Machinery, New York, NY, USA, 299â303. doi:10.1145/3524842.3528009 [58]Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compres- sion of Pre-Trained Transformers. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 5776â5788. https://proceedings.neurips.c/paper_files/paper/2020/file/3f5e243547dee91fbd053c1c4a845a- Paper.pdf [59]Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. 2023. Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (San Francisco, CA, USA) (ESEC/FSE 2023). Association for Computing Machinery, New York, NY, USA, 172â184. doi:10.1145/ 3611643.3616271 [60]Chunqiu Steven Xia and Lingming Zhang. 2024. Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA â24). ACM, 819â831. doi:10.1145/3650212.3680323 [61] Cong Xu, Gayathri Saranathan, Mahammad Parwez Alam, Arpit Shah, James Lim, Soon Yee Wong, Foltin Martin, and Suparna Bhattacharya. 2024. Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Sampling. arXiv:2406.15527 [cs.LG] https: //arxiv.org/abs/2406.15527 [62]Chunli Yu, Shayan Noei, Haoxiang Zhang, and Ying Zou. 2026. An Empirical Study on the Characteristics of Reusable Code Clones. ACM Trans. Softw. Eng. Methodol. (Jan. 2026). doi:10.1145/3793251 Just Accepted. Manuscript submitted to ACM