Paper deep dive
Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency
Junchi Liu, Ali Bigdeli, Roya Daneshi, Atu Ambala, Sudipto Ghosh, Fabio Santos
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/17/2026, 4:56:13 AM
Summary
This study presents a comprehensive empirical analysis of Large Language Model (LLM)-based Automated Program Repair (APR) techniques, specifically ChatRepair and CodeCorrector, using models like GPT-5, DeepSeek, and Llama. The research evaluates performance across varying bug complexities (hunk-line and AST structural complexity), fault localization strategies (accurate, vague, none), and cost-efficiency. Key findings indicate that while complex bugs and imprecise localization increase difficulty, LLMs remain effective. GPT-5 outperforms DeepSeek variants in repairing complex bugs, but DeepSeek-V3.2 offers superior cost-efficiency. The study highlights a non-trivial trade-off between repair effectiveness and computational cost, suggesting low-cost models can handle over 50% of moderately complex bugs.
Entities (10)
Relation Signals (8)
DeepSeek-V3.2 → hasbestcostefficiency → Cost-Efficiency
confidence 95% · total repair cost of DeepSeek-V3.2 shows the best cost-efficiency performance.
Bug Complexity → impacts → Repair Difficulty
confidence 95% · structurally complex bugs and imprecise fault localization make repair more challenging
Fault Localization → impacts → Repair Effectiveness
confidence 95% · Imprecise fault localization can substantially enlarge the performance gap between APR techniques.
AtCoder → providesdatafor → APR Evaluation
confidence 95% · We construct a curated dataset by collecting algorithmic bugs from AtCoder...
CodeCorrector → uses → LLMs
confidence 95% · We evaluate two APR techniques (ChatRepair and CodeCorrector) using three LLMs...
ChatRepair → uses → LLMs
confidence 95% · We evaluate two APR techniques (ChatRepair and CodeCorrector) using three LLMs...
GPT-5 → outperforms → DeepSeek-V3.2
confidence 92% · GPT-5 repairs 7 and 39 more complex bugs than DeepSeek-V4-pro and DeepSeek-V3.2, respectively
GPT-5 → outperforms → DeepSeek-V4-Pro
confidence 92% · GPT-5 repairs 7 and 39 more complex bugs than DeepSeek-V4-pro...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Background: Software bugs remain a critical challenge in development, necessitating effective Automated Program Repair (APR) techniques. While Large Language Model (LLM)-based APR systems have shown promise, prior studies primarily focus on overall repair effectiveness. The effects of bug complexity, fault localization, reasoning settings, and repair cost-effectiveness remain insufficiently explored. Aims: This study presents a comprehensive empirical analysis of LLM-based APR, focusing on how repair performance is shaped by bug complexity, fault localization, reasoning settings, and costs. Method: We evaluate two APR techniques (ChatRepair and CodeCorrector) using three LLMs (DeepSeek, GPT, and Llama), and examine their performance across diverse levels of bug complexity and localization strategies through a multi-dimensional empirical framework and statistical analysis. Results: Although structurally complex bugs and imprecise fault localization make repair more challenging, LLM-based APR techniques still achieve competitive repair effectiveness. Imprecise fault localization can substantially enlarge the performance gap between APR techniques. Furthermore, higher-cost LLMs and stronger reasoning settings do not consistently yield better cost-efficiency, revealing a nontrivial trade-off between repair effectiveness and computational cost. Conclusions: Over 50% of moderately complex bugs can be repaired by low-cost LLM-based APR techniques. The repair effectiveness gap between APR techniques becomes larger as fault localization becomes less precise. GPT-5 repairs 7 and 39 more complex bugs than DeepSeek-V4-pro and DeepSeek-V3.2, respectively; whereas the total repair cost of DeepSeek-V3.2 shows the best cost-efficiency performance.
Tags
Links
- Source: https://arxiv.org/abs/2608.14065v1
- Canonical: https://arxiv.org/abs/2608.14065v1
Trouble viewing inline? Open PDF directly →
Full Text
65,042 characters extracted from source content.
Expand or collapse full text
Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency Junchi Liu# Colorado State University, United States Ali Bigdeli# Colorado State University, United States Roya Daneshi# Colorado State University, United States Atu Ambala# Colorado State University, United States Sudipto Ghosh# Colorado State University, United States Fabio Santos# Colorado State University, United States Abstract Background: Software bugs remain a critical challenge in development, necessitating effective Automated Program Repair (APR) techniques. While Large Language Model (LLM)-based APR systems have shown promise, prior studies primarily focus on overall repair effectiveness. The effects of bug complexity, fault localization, reasoning settings, and repair cost-effectiveness remain insufficiently explored. Aims: This study presents a comprehensive empirical analysis of LLM-based APR, focusing on how repair performance is shaped by bug complexity, fault localization, reasoning settings, and costs. Method: We construct a curated dataset by collecting algorithmic bugs from AtCoder, a com- petitive programming platform. We evaluate two APR techniques (ChatRepair and CodeCorrector) using three LLMs (DeepSeek, GPT, and Llama), with multiple model variants and reasoning settings, and examine their performance across diverse levels of bug complexity and localization strategies through a multi-dimensional empirical framework and statistical analysis. Results: Although structurally complex bugs and imprecise fault localization make repair more challenging, LLM-based APR techniques still achieve competitive repair effectiveness. Imprecise fault localization can substantially enlarge the performance gap between APR techniques. Furthermore, higher-cost LLMs and stronger reasoning settings do not consistently yield better cost-efficiency, revealing a nontrivial trade-off between repair effectiveness and computational cost. We further observe that the impact of reasoning strategies varies considerably across different LLM families, affecting both repair effectiveness and cost-efficiency. Conclusions: Over 50% of moderately complex bugs can be repaired by low-cost LLM-based APR techniques. The repair effectiveness gap between APR techniques becomes larger as fault localization becomes less precise. GPT-5 repairs 7 and 39 more complex bugs than DeepSeek-V4-pro and DeepSeek-V3.2, respectively; whereas the total repair cost of DeepSeek-V3.2 across the non- reasoning and reasoning stages is only approximately 4.6% and 7.7% of GPT-5 and DeepSeek-V4-pro, respectively. 2012 ACM Subject Classification Software and its engineering Keywords and phrases Automated Program Repair, Large Language Models, Fault Localization, Bug Complexity, Cost-efficiency Digital Object Identifier 10.4230/LIPIcs.ESEM.2026.40 © Junchi Liu, Ali Bigdeli, Roya Daneshi, Atu Ambala, Sudipto Ghosh, and Fabio Santos; licensed under Creative Commons License C-BY 4.0 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Editors: Robert Feldt, Maria Paasivaara, Daniel Mendez, Stefan Wagner, and Marvin Muñoz Barón; Article No. 40; p. 40:1–40:20 Leibniz International Proceedings in Informatics Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl Publishing, Germany arXiv:2608.14065v1 [cs.SE] 14 Aug 2026 40:2 Rethinking Automated Program Repair Category Technical Track Paper 1 Introduction In recent years, the rapid advancement of Generative AI, driven by intense competition among technology companies, has significantly accelerated the development of Large Language Models (LLMs) [30]. Current LLMs have shown substantial improvements in their ability to understand, reason, and generate programs [3,42,48]. As a result, they have been increasingly applied to a wide range of software engineering tasks, including code generation, code summarization, and bug detection [6, 19, 23]. In the context of Automated Program Repair (APR), LLM-based approaches have demon- strated significant advantages over traditional techniques, achieving strong performance across widely used benchmarks such as Defects4J [20], Bugs.jar [41], and QuixBugs [25]. Notably, many APR techniques achieve near-perfect performance on QuixBugs and In- troClass [13], small-scale algorithmic datasets primarily composed of simple, single-hunk bugs, suggesting that these datasets may be insufficiently challenging for evaluating the effectiveness of modern LLM-based APR approaches. Also, as LLMs exhibit strong per- formance in code understanding and reasoning, recent studies have increasingly explored end-to-end or self-directed APR approaches that reduce reliance on externally provided fault localization (FL) information [15,18,22]. Nevertheless, the impact of the absence of FL on repair effectiveness has not been systematically evaluated in the agent era. To address these unexplored gaps, our study introduces a more challenging algorithmic dataset, featuring bugs with varied complexity levels, and uses the tools of Git [44], srcML [7], and GumTreeDiff [12] to systematically and automatically categorize bugs based on their complexity, enabling a fine-grained empirical analysis of repair performance across different bug complexity levels as well as under accurate, vague, or no fault localization settings. Additionally, the reliance on commercial LLM APIs produces non-negligible computational costs [45]. APR tasks often require generating numerous candidate patches [24,52], resulting in substantial output token usage, particularly under reasoning-intensive settings [14,50]. Consequently, different LLM choices and reasoning settings can lead to significant cost discrepancies. Such costs are often overlooked in prior APR evaluations, despite being critical for practical deployment. In our study, we incorporate cost-efficiency metrics and standardize the unit to $1 for fair comparison. Specifically, we define Repaired Bugs/$ and Attempts/$ as the numbers of successfully repaired bugs and repair attempts divided by the total API cost, respectively, where the cost is calculated from input/output token usage and the corresponding API prices. We present a comprehensive evaluation of two LLM-based APR techniques across three LLMs, including multiple versions and reasoning settings. We employ multi-dimensional metrics (number of compilable patches, plausible patches, and repaired bugs,Top − k, consistency, and cost-efficiency metrics), and statistical tests to address the following research questions (RQs): RQ1. How does bug complexity affect the repair performance of LLM-based APR? RQ2. To what extent does fault localization influence the effectiveness of LLM-based APR? RQ3. How do LLM choices and reasoning settings impact the repair performance–cost trade-off in APR? The results of this study show that: J. Liu et al.40:3 Repair performance varies substantially across bug complexity, with the simplest complex- ity bugs being consistently the easiest to fix, as expected, but repair performance remains at a relatively considerable level as complexity increases, suggesting that LLM-based APR approaches maintain a stable reasoning ability even on structurally complex bugs. FL plays a critical role in improving both repair effectiveness and patch consistency. Al- though more precise localization generally leads to better repair performance, appropriate APR frameworks can still maintain relatively stable repair effectiveness in settings with vague or missing localization, suggesting potential for practical end-to-end APR scenarios. Also, the impact of FL precision on repair effectiveness decreases as bug complexity increases. More costly LLMs and reasoning strategies do not always produce proportional repair im- provements; the increased cost should be considered seriously, and reasoning strategies do not consistently yield ideal outcomes, revealing a nontrivial trade-off between performance and efficiency. Our contributions include: i- We empirically analyze the impact of bug complexity and fault localization on LLM-based APR performance, and incorporate unit-standardized cost-efficiency metrics to quantify the trade-off between repair effectiveness and computational cost. We also provide an extensible experimentation infrastructure to support future studies, replication efforts, and systematic evaluation of LLM-based APR techniques. i-Our study offers novel insights into how repair effectiveness varies with bug complexity and reveals the nontrivial cost implications of different LLM choices and reasoning settings. In particular, by systematically evaluating settings with accurate, vague, and no fault localization, we demonstrate the feasibility and potential of end-to-end LLM-based APR. i-Additionally, our results across multiple LLMs and reasoning strategies provide practical guidance for selecting appropriate LLMs under the trade-off between the repair perform- ance and economic cost, highlighting the importance of balancing repair performance and computational cost. 2 Background and Related Work This section provides a relevant review of the literature and highlights the limitations of existing studies. 2.1 Previous Evaluations of LLM-based APR Recent evaluations of APR are primarily based on the Defects4J Benchmark [20,27], which has become the dominant benchmark for APR evaluation. However, heavy reliance on a single benchmark can limit the diversity of bug types and weaken the evaluation of APR approaches’ generalizability. Nevertheless, several variants of Defects4J have been proposed, including DEFECTS4J-TRANS [21], and MuBench [34]. These benchmarks still primarily contain software-engineering–oriented bugs [40]. Meanwhile, algorithm-oriented datasets such as QuixBugs [25], IntroClass, and ManyBugs [13] have been widely utilized to evaluate the generalizability of APR techniques. However, the difficulty of these datasets is often insufficient to challenge recent LLM-based APR approaches [47], thereby leading to an over- estimation of their effectiveness. In addition, the continued reuse of long-standing benchmark datasets introduces data leakage issues [40], potentially leading to biased evaluation results. ESEM 2026 40:4 Rethinking Automated Program Repair Furthermore, Bean et al. [4] point out that the lack of statistical testing in the majority of benchmarks reduces the reliability of empirical findings. We collect recent real-world bugs from AtCoder [16] as an extension of algorithm-oriented datasets. Then, Git, srcML, and GumTreeDiff are utilized to categorize bugs based on complexity. Finally, we conduct an empirical evaluation of LLM-based APR under different bug complexity and FL settings, with a cost-efficiency and statistical analysis. 2.2 APR Techniques APR techniques have advanced significantly with the development of LLMs’ code-reasoning and code-generation capabilities [53]. With the success of LLMs in code reasoning and generation [17], recent studies have increasingly explored their applications in APR. Xia and Zhang [47] introduced ChatRepair, which iteratively generates and refines patches by leveraging feedback from both failed and successful repair attempts, and has been widely viewed as a representative baseline in subsequent studies. For instance, Yin et al. [49] proposed ThinkRepair, which uses Chain-of-Thought (CoT) prompting [46] to enable reasoning prior to patch generation. Bouzenia et al. [5] introduced RepairAgent, the first autonomous agent-based framework for APR. Li et al. [22] further proposed CodeCorrector, an end-to-end framework that first analyzes programs to infer repair directions and then generates patches, achieving the best performance on the Defects4J v1.2 dataset. In this study, we select ChatRepair as a representative LLM-based APR approach and CodeCorrector as a recent LLM-based method that demonstrates strong performance in prior studies. 2.3 Evaluation Metrics Existing APR evaluation metrics primarily focus on repair effectiveness, including the number of repaired bugs,Top− k, compilable patches, and plausible patches [11]. To further assess patch quality, Dai et al. [8] introduced the consistency metric, which measures the extent of changes between patches and buggy programs. However, these metrics fail to capture cost-efficiency, which has become increasingly important in the empirical study of LLM-based APR. Although recent studies have begun to report costs [5,22,47], they use inconsistent units, such as cost per bug, cost per successful repair, or cost per attempt, making it difficult to perform fair, direct comparisons across approaches. This lack of a unified cost metric limits the systematic evaluation of performance–cost trade-offs. Thus, we standardize costs to a common monetary unit, enabling a direct trade-off between repair performance and cost. 3 Research Design In this section, we discuss the research methodology. 3.1 Experimental Infrastructure The experimental infrastructure is deployed on computing servers with isolated execution environments to ensure controlled access and protected communication. Shared storage and distributed computing resources are utilized to support large-scale APR experiments across multiple machines, improving the overall experimental efficiency. The experimental infrastructure consists of five components: the dataset, APR techniques, LLMs, additional tools, and Python scripts. J. Liu et al.40:5 Figure 1 Flow of the research method. The dataset is constructed from AtCoder and includes problem descriptions, pairs of incorrect and correct C++ programs, and corresponding test cases (Section 3.3). The tools include the APR tools, along with Git, srcML, and GumTreeDiff. The AtCoder Library (ACL) is included as an external dependency to support compilation. Custom Python scripts are used for data analysis and dataset filtering. All the above components are maintained within our isolated environment. We utilize multiple LLMs in our study: gpt-5-2025-08-07 (GPT-5) [33], gpt-4o-mini- 2024-07-18 (GPT-4o-mini) [32], DeepSeek-V3.2 [9], DeepSeek-V4-pro [10], Llama-3.1-8B [2], and Llama-3.1-405B [1]. GPT-5 and GPT-4o-mini use the OpenAI API. DeepSeek-V3.2 and DeepSeek-V4-pro use the DeepSeek API. Llama-3.1 is deployed in our experimentation infrastructure. 3.2 Experimental Workflow Figure 1 illustrates the overall experimental infrastructure and workflow of our study. We first collect buggy and corrected programs from AtCoder and construct an automated benchmark and evaluation pipeline for LLM-based APR experiments (details are provided in subsection 3.3). Next, we analyze bug complexity from multiple perspectives. Git-based differencing is utilized to extract hunk-line complexity information, while srcML [7] and GumTreeDiff [12] are employed to measure structural complexity at the Abstract Syntax Tree (AST) level (details are provided in subsection 3.4). Based on the constructed benchmark and complexity analysis, we deploy APR frameworks and integrate multiple LLM APIs with different reasoning settings to conduct automated repair experiments under FL settings (details are provided in subsection 3.5). Finally, we perform a comprehensive empirical evaluation to investigate the impact of bug complexity, fault localization, and cost-efficiency trade-offs on the performance of LLM-based APR techniques. The corresponding findings and experimental analyses are presented in section 4. ESEM 2026 40:6 Rethinking Automated Program Repair 123456789 1011121314151617181920212223242526272829303132333435363738394041 42 43444546474849 50+ 0 100 200 Fixed Lines Number of Bugs Figure 2 Distribution of Fixed Size Across Raw Data 3.3 Dataset Existing APR benchmarks have been widely used, raising concerns about data contamin- ation in the evaluation of LLM-based APR. To mitigate this risk, we construct our dataset from the recent AtCoder submissions rather than relying on existing benchmarks. We focus on C++ programs because comparing FL settings and bug complexity across programming languages with different characteristics could introduce additional bias. This design enables a more controlled evaluation of LLM-based APR. In this study, we use AtCoder, a Japanese competitive programming platform, and collect 1,914 C++ source code submissions in October, 2025. Our data comes from regular contests, which typically include algorithmically challenging problems designed to assess intermediate-to-advanced programming skills. We pair every incorrect program and correct program from the same participant as a pair of a bug and a correct patch (bug-patch pair). For each bug-patch pair, we use thegit diffcommand to generate a diff file that captures the corresponding code changes. Before generating diff files using Git, we remove all comments and redundant blank lines. The diff files reflect line-level code changes, where ‘-‘ denotes deleted (buggy) linesN − , and ‘+‘ denotes added (patch) linesN + . For each hunk, the number of fixed lines is defined as Equation 1. N fixed = ( N + , if N + > N − , N − , otherwise (1) The distribution of bug-fix sizes across the original 1,914 C++ files is shown in Figure 2. Based on preliminary experiments comparing different fixed line thresholds on samples drawn from the full dataset, we observe that APR becomes considerably more difficult and computationally expensive as the number of fixed lines increases. Considering both repair feasibility and computational cost, we set a threshold of five fixed lines to focus on small-scope program repair. Finally, 640 bugs remain for our empirical study. 3.4 Bug Complexity Buggy hunk-line-level complexity: The diff files generated by Git not only identify buggy and patched lines but also organize code changes into hunks. We use Git to generate diff files for every bug-patch pair. Based on the number of hunks and the number of fixed lines within each hunk, we categorize bugs into four categories: Single-Hunk-Single-Line J. Liu et al.40:7 Figure 3 AST operation-type complexity. (SHSL), Single-Hunk-Multi-Line (SHML), Multi-Hunk-Single-Line (MHSL), and Multi-Hunk- Multi-Line (MHML). The categorization rule is defined in Equation 2. The number of hunks can be obtained directly from Git diff files. If any hunk contains more than one fixed line, the bug is classified as a multi-line bug. SHSL,H = 1∧∀h i ∈ H, L(h i ) = 1 SHML, H = 1∧∃h i ∈ H, L(h i ) > 1 MHSL, H > 1∧∀h i ∈ H, L(h i ) = 1 MHML, H > 1∧∃h i ∈ H, L(h i ) > 1 (2) whereHdenotes the total number of hunks in a bug,h i represents thei-th hunk (1≤ i≤ H), and L(h i ) denotes the number of fixed lines in hunk h i . AST operation-type complexity: We employ srcML to construct XML-based struc- tural representations of programs [7] and further utilize GumTreeDiff to extract AST-level edit operations between buggy and corrected program pairs [12]. Unlike hunk-line-level complexity, this metric measures structural differences at the AST node level, providing a finer-grained view of bug complexity. The AST edit distance [43] between a buggy and a correct program is defined by four edit operations: insert, delete, relabel, and move. In our study, bugs are further categorized based on the diversity of edit operations required for repair. Figure 3 illustrates how the AST edit-operation-type complexity is categorized. Table 1 and Table 2 present the number of bugs in each type under the hunk-line-level and AST operation-type complexity, respectively. We conduct empirical evaluations to analyze how bug complexity affects LLM-based APR performance. 3.5 Fault Localization And APR Settings We use three FL strategies to explore their impact on LLM-based APR: buggy line-level FL, buggy hunk-level FL, and no FL. Figure 4 presents the examples of line-level FL and hunk-level FL according to the diff file. ESEM 2026 40:8 Rethinking Automated Program Repair Table 1 Hunk-line complexity. Bug Type# Bugs Single Hunk Single Line251 Single Hunk Multi Line226 Multi Hunk Single Line67 Multi Hunk Multi Line96 Total640 Table 2 AST operation-type complexity Bug Type# Bugs Single Operation278 Two Operations158 Three Operations121 Four Operations83 Total640 @@ -58,7 +58,7 @@ int main () long long ans=0; for(int i=x;i<=k;i++) -ans=(ans+f[n%2][0][i])%mo; +ans=(ans+f[0][0][i])%mo; cout <<ans <<" "; return 0; (a) Diff file // ### Buggy Hunk Start ### long long ans=0; for(int i=x;i<=k;i++) ans=(ans+f[n%2][0][i])%mo; cout <<ans <<" "; return 0; // ### Buggy Hunk End ### (b) Hunk-level localization for(int i=x;i<=k;i++) // ### Buggy Line Start ### ans=(ans+f[n%2][0][i])%mo; // ### Buggy Line End ### (c) Line-level localization Figure 4 Fault Localization Strategy Examples Different LLMs provide different mechanisms for controlling reasoning behavior. (a) DeepSeek-V3.2 provides two model variants: DeepSeek-V3.2-chat and DeepSeek-V3.2- reasoner. For DeepSeek-V4-pro, the reasoning mode can be enabled or disabled through thethinking_typeparameter in the DeepSeek API. Additionally, thereasoning_effort parameter can be configured ashighormax. (b) GPT models do not provide separate chat or reasoner variants; instead, reasoning intensity can only be controlled through the reasoning.effortparameter, with settings such asmedium,high, andxhigh. For Llama models, there is currently no explicit mechanism to directly control reasoning settings. Furthermore, the temperature for all LLMs is set to 1.0 [39] for all experiments. For ChatRepair, we set the maximum number of conversation rounds to 3 [47]. For both ChatRepair and CodeCorrector, we provide the problem background, the buggy code, the error message, and at most 3 failed test cases. We retain the original prompts reported in the corresponding studies [22, 47], and we do not perform model-specific parameter tuning, allowing the comparison to focus on the influence of the APR frameworks, FL strategies, and LLM effectiveness. 3.6 Experimental Design To balance experimental comprehensiveness and computational cost, we adopt a three- stage experimental design. First, we evaluate two APR tools with three different low-cost LLMs on the complete set of 640 bugs under line-level FL: DeepSeek-V3.2-chat, GPT-4o-mini, and Llama-3.1-8B. The maximum number of repair rounds is set to 10 as in [35]. We conduct a comprehensive empirical analysis of various APR techniques and LLM combinations with respect to repair effectiveness and cost-efficiency. Next, we select the best-performing LLM from the first stage and further evaluate it on all 640 bugs at line-level, hunk-level, and no-FL. The results are then compared across three FL strategies to analyze the impact of different FL precision on two LLM-based APR techniques. For the first two stages, reasoning mode is disabled across all LLMs, allowing us to identify bugs that can already be repaired under non-reasoning configurations before J. Liu et al.40:9 Table 3 Comparison of repair effectiveness between CodeCorrector and ChatRepair frameworks using three base LLMs on the full benchmark. CodeCorrectorChatRepair DeepSeek- V3.2-chat GPT-4o- mini Llama-3.1- 8B DeepSeek- V3.2-chat GPT-4o- mini Llama-3.1- 8B Repaired Bugs295/640126/64011/640292/640140/64050/640 Attempts400353736305640064006400 Unique Patches3117(77.9%) 4425(82.4%) 5833(92.5%) 2623(41.0%) 3774(59.0%) 5111(80.0%) Compilable Patches 2830(70.7%) 3899(72.6%) 1466(23.3%) 2523(39.4%) 3511(54.9%) 3122(48.8%) Plausible Patches2951261142124776 Repaired Bugs / $47.6217.86-71.4320.41- Attempts / $646.17761.48-1565.56932.94- evaluating more computationally expensive reasoning-enabled models. To further investigate the repair effectiveness and cost-efficiency of advanced LLMs and reasoning settings under more challenging repair scenarios, we adopt a filtering strategy to fo- cus the evaluation on bugs with moderate repair difficulty while controlling the computational cost of evaluating advanced LLMs. Specifically, we exclude SHSL bugs, single-operation-type bugs, and four-operation-type bugs. In addition, bugs that can already be successfully re- paired by at least five APR techniques–LLM combinations in the first-stage experiments are removed. Consequently, the remaining benchmark focuses on bugs with moderate structural complexity and lower repairability under low-cost APR configurations. After filtering, 211 bugs remain for empirical analysis. We further select the better-performing APR technique from the previous stages as the representative APR method for subsequent experiments. In this stage, we evaluate four advanced LLMs: DeepSeek-V4-pro, DeepSeek-V3.2, GPT-5, and Llama-3.1-405B. The maximum number of repair rounds is increased to 13. During the first 10 rounds, reasoning mode is disabled. If a bug remains unfixed, reasoning mode is enabled for the final three rounds. For GPT-5, DeepSeek-V3.2, and DeepSeek-V4-pro, the reasoning effort is configured as high. Experimental Token Consumption. To facilitate the reproduction of our experiments, we report the total input and output tokens consumed by each evaluated LLM that incurred API costs. DeepSeek-V3.2 consumed 116.5M input tokens and 42.7M output tokens. GPT- 4o-mini used 49.5M input tokens and 14.4M output tokens. DeepSeek-V4-pro consumed 6.3M input tokens and 9.9M output tokens, GPT-5 consumed 3.4M input tokens and 7.7M output tokens, and Llama-3.1-405B consumed 6.3M input tokens and 1.7M output tokens. API Response Time. We also report the approximate response time per repair request. Without reasoning mode, all evaluated LLMs require approximately 1–3 minutes per request. With reasoning enabled, DeepSeek-V3.2 and DeepSeek-V4-pro required approximately 40–60 minutes per request, while GPT-5 required about 30 minutes. 4 Results and Empirical Analysis In this section, we show the results of our empirical experiment. 4.1 APR Performance With Different LLMs Table 3 presents the repair performance of CodeCorrector and ChatRepair with three base LLMs under the non-reasoning setting. In this experiment, line-level FL is provided ESEM 2026 40:10 Rethinking Automated Program Repair Table 4 Pairwise 2×2 contingency tables for chi-square tests. (a) Hunk-line Complexity Comparisons SHSL vs SHMLSHSL vs MHSLSHSL vs MHML TypeFixedUnfixed SHSL 197(78.5%)54 SHML 113(50.0%) 113 TypeFixedUnfixed SHSL 197(78.5%)54 MHSL 39(58.2%)28 TypeFixedUnfixed SHSL 197(78.5%)54 MHML 48(50.0%)48 SHML vs MHSLSHML vs MHMLMHSL vs MHML TypeFixedUnfixed SHML 113(50.0%) 113 MHSL 39(58.2%)28 TypeFixedUnfixed SHML 113(50.0%) 113 MHML 48(50.0%)48 TypeFixed Unfixed MHSL 39(58.2%)28 MHML 48(50.0%)48 (b) AST Operation Complexity Comparisons Single vs TwoSingle vs ThreeSingle vs Four TypeFixedUnfixed Single 209(75.2%)69 Two 90(57.0%)68 TypeFixedUnfixed Single 209(75.2%)69 Three 66(54.5%)55 TypeFixedUnfixed Single 209(75.2%)69 Four 32(38.6%)51 Two vs ThreeTwo vs FourThree vs Four TypeFixed Unfixed Two 90(57.0%)68 Three 66(54.5%)55 Type Fixed Unfixed Two 90(57.0%)68 Four 32(38.6%)51 TypeFixed Unfixed Three 66(54.5%)55 Four 32(38.6%)51 Table 5 Pairwise Pearson Chi-square test results across different bug complexity categories. SHSL SHMLMHSLMHML SHSL–p < 0.0001p = 0.0013p < 0.0001 SHML – p = 0.30 p = 1.000 MHSL –p = 0.38 MHML – (a) Hunk-line complexity 1-op2-op3-op4-op 1-op –p = 0.0001p < 0.0001p < 0.0001 2-op – p = 0.81p = 0.013 3-op –p = 0.041 4-op – (b) AST operation type complexity Darker cells indicate smaller p-values. within the incorrect programs, and the maximum repair round is set to 10. CodeCorrector stops after generating the first plausible patch, whereas ChatRepair continues generating candidate patches. Overall, DeepSeek-V3.2-chat achieves substantially stronger repair effectiveness than GPT-4o-mini across both APR frameworks. When integrated with DeepSeek-V3.2-chat, both CodeCorrector and ChatRepair repair more than twice as many bugs as they do with GPT-4o-mini (295, 292 vs. 126, 140), whereas Llama-3.1-8B demonstrates very limited repair behavior: only 11 bugs were corrected with CodeCorrector and 50 with ChatRepair. In addition, DeepSeek-V3.2-chat exhibits noticeably better cost-efficiency. DeepSeek- V3.2-chat repairs 47.62 bugs per dollar under CodeCorrector, compared with only 17.86 bugs per dollar for GPT-4o-mini, representing approximately 2.7×higher cost-efficiency. Similarly, under ChatRepair, DeepSeek achieves 71.43 repaired bugs per dollar, whereas GPT-4o-mini repairs only 20.41 bugs per dollar, corresponding to approximately 3.5×higher cost-efficiency. Finding 1: Among relatively low-cost LLMs, DeepSeek-V3.2-chat demonstrates the strongest overall APR effectiveness, achieving significantly better repair results and cost- efficiency than GPT-4o-mini, whereas Llama-3.1-8B exhibits limited repair effectiveness despite its negligible inference cost. J. Liu et al.40:11 Bug Complexity: We then analyze how bug complexity affects overall repairability, where a bug is considered repaired if any evaluated APR technique and LLM configuration can fix it. Table 4 and Table 5 present the pairwise 2×2 contingency tables and Pearson chi- square test results for the numbers of fixed and unfixed bugs across different bug complexity categories. Pairwise chi-square tests are conducted to assess whether differences in repair rates across complexity categories are statistically significant. Following [28], we consider differences statistically significant whenp <0.05, whilep <0.005 indicates sufficient evidence against the null hypothesis. SHSL and single-operation-type bugs exhibit the lowest structural complexity and con- sequently achieve the highest repair rates among all categories, reaching 78.5% and 75.2%, respectively. Furthermore, all pairwise comparisons between these two categories and the remaining complexity categories yieldp <0.005, providing strong statistical evidence that SHSL and single-operation-type bugs are substantially easier for APR techniques to repair. However, the SHML, MHSL, MHML, two-operation, and three-operation bugs also achieve repair rates above 50%. For hunk-line-level complexity, no statistically significant difference in repair is observed among multi-line bug categories (p >0.05). In contrast, although no statistically significant difference is observed between two-operation-type and three-operation-type bugs (p= 0.81), four-operation-type bugs exhibit significantly lower repair effectiveness than other operation- type categories (p <0.05). Thus, AST operation complexity further distinguishes highly complex bugs, with statistical analysis showing that four-operation bugs exhibit the most complex repair structure. Finding 2: SHSL and single-operation-type bugs are the easiest to repair, whereas four- operation-type bugs remain the most challenging. The repair rates of moderate-complexity categories exceed 50%, indicating that LLM-based APR techniques maintain strong repair effectiveness. Although four-operation-type bugs exhibit lower repair performance, their repair rates remain considerable. These results suggest that low-cost LLM-based APR techniques retain substantial repair effectiveness even for structurally complex bugs. RQ1 Summary. Bug complexity significantly affects repair effectiveness. Chi-square test results suggest three approximate levels of repair difficulty across different bug complexity categories. Overall, repair effectiveness decreases with increasing bug complexity, while the simplest bugs remain substantially easier to repair. Although four-operation bugs are the hardest to repair, APR techniques still maintain considerable repair capability for structurally complex bugs. Over 50% of moderate-complexity bugs can still be repaired successfully. These results demonstrate the potential of low-cost LLM-based APR techniques for handling challenging real-world repair scenarios. 4.2 Fault Localization Evaluation Given DeepSeek-V3.2-chat’s best performance in the previous experiment, we further evaluate the impact of different FL strategies on APR performance using this LLM with both APR techniques. In this experiment, the maximum repair round is still set to 10. Table 6 shows that CodeCorrector and ChatRepair achieve nearly identical repair effect- iveness under line-level FL, with CodeCorrector even repairing three more bugs. However, when only hunk-level FL or no FL is provided, ChatRepair demonstrates consistently greater stability and generalizability. Under hunk-level FL, ChatRepair repairs 58 more bugs than CodeCorrector, and this gap further increases to 93 bugs when FL is unavailable. ESEM 2026 40:12 Rethinking Automated Program Repair Table 6 Comparison of repair effectiveness between CodeCorrector and ChatRepair frameworks using three Fault Localization strategies on the full benchmark. CodeCorrectorChatRepair line-level hunk-level w/o FL line-level hunk-level w/o FL Repaired Bugs295/640179/640112/640292/640237/640205/640 Attempts400349865480640064006400 Unique Patches3117(77.9%) 4346(87.2%) 4012(73.2%) 2623(41.0%) 4741(74.1%) 3261(51.0%) Compilable Patches 2830(70.7%) 3503(70.3%) 3756(68.5%) 2523(39.4%) 4073(63.6%) 3174(49.6%) Plausible Patches295179112421542299 Repaired Bugs / $47.6216.6018.8771.4324.4147.62 Attempts / $646.17462.52923.181565.56659.131486.64 Table 7 Comparison of repair consistency under different FL strategies. FL StrategyCodeCorrectorChatRepair Line-level FL0.850.92 Hunk-level FL0.750.83 w/o FL0.540.85 Finding 3: The performance gap between APR techniques increases markedly as FL becomes less precise, indicating that repair stability and generalizability are critical for real- world APR scenarios. Different APR techniques may achieve similar repair rates under precise line-level FL, but their repair rates diverge substantially under imprecise FL settings. We further introduce the code consistency rate (CCR) to measure how well repaired programs preserve the original code’s structure. Equation 3 [8] defines the consistency metric, wherekdenotes the total number of code lines in the repaired program, andrdenotes the number of preserved code lines after repair. CCR (Code Consistency Rate) = r k (3) According to Table 7, line-level FL achieves the highest patch consistency across all FL settings for both APR techniques. ChatRepair generally generates patches with higher structural consistency than CodeCorrector, particularly without FL (0.85 vs. 0.54). Finding 4: FL not only improves repair effectiveness but also helps generate patches that remain more structurally consistent with the original programs. The design of APR techniques also significantly impacts the structural consistency of generated patches. We use logistic regression [29] to examine the extent to which reduced FL precision affects the likelihood of successful repair. We set line-level FL as the reference category and analyze each complexity level independently. The results are reported using odds ratios (ORs) and correspondingp-values. Smaller OR values reflect a greater reduction in the odds of successful repair. Figure 5 shows that repair rates consistently decrease across all complexity categories as FL becomes less precise (blue vs. orange vs. red). According to Table 8, both hunk-level FL and no-FL settings yield overallp-values smaller than 0.001, indicating that FL precision has a statistically significant effect on overall repair effectiveness. We further observe that the simplest bugs (SHSL and single-operation bugs) appear sensitive to reductions in FL precision, as both hunk-level FL and no-FL settings produce very J. Liu et al.40:13 SHSLSHMLMHSLMHML 0% 20% 40% 60% 80% 74.1 48.2 52.2 49 59 38.5 37.3 32.3 47.4 30.1 25.4 30.2 Hunk-line Complexity Repair Rate (%) Line-Level Hunk-Level w/o FL Single-oTwo-oThree-oFour-o 0% 20% 40% 60% 80% 70.9 53.8 53.7 36.1 54.3 44.3 38 28.9 45 32.3 30.6 24.1 AST-operation Type Complexity Repair Rate (%) Line-Level Hunk-Level w/o FL Figure 5 Repair rate under different bug complexity categories. Table 8 Logistic regression results of FL effects on repair success under different complexity categories. (a) Hunk-line complexity categories Scope Hunk-level FLNo FL ORpORp Overall 0.582 < .001 0.399 < .001 SHSL 0.502 < .001 0.315 < .001 SHML 0.672 0.037 0.462 < .001 MHSL 0.544 0.084 0.311 0.002 MHML 0.497 0.019 0.451 0.008 (b) AST-operation complexity categories Scope Hunk-level FLNo FL ORpORp Overall 0.582 < .001 0.399 < .001 single-op 0.489 < .001 0.336 < .001 two-op0.683 0.092 0.409 < .001 three-op 0.528 0.015 0.379 < .001 four-op 0.719 0.321 0.561 0.092 low odds ratios, also withp <0.001. In contrast, the highest complexity bugs (four-operation bugs) exhibit the least noticeable repair degradation under hunk-level FL and no-FL settings, as the odds ratios remain relatively higher and the FL degradation effects are not statistically significant (p > 0.05). Finding 5: Repair rates consistently decrease as FL becomes less precise across all bug complexity categories, indicating that FL precision remains important regardless of bug complexity. However, as bug complexity increases, repair effectiveness becomes less dependent on precise FL, suggesting it is more likely reliant on structural understanding and reasoning of program logic. RQ2 Summary. FL still plays a critical role in the bug repair process. More precise FL not only improves overall repair rates but also helps APR techniques generate more consistent patches. Furthermore, APR techniques with similar repair performance under the same FL strategy can exhibit substantially different effectiveness as FL becomes increasingly imprecise, resulting in progressively larger performance gaps among APR approaches. These results highlight the importance of the stability and generalizability of APR techniques for handling realistic bug-repair scenarios. Moreover, as bug structures become increasingly complex, the relative importance of FL precision decreases, indicating the growing significance of reasoning and program understanding. 4.3 APR With Advanced LLMs ChatRepair demonstrates overall better performance than CodeCorrector in sections 4.1 and 4.2, so this study chooses ChatRepair to evaluate performance and cost-efficiency for ESEM 2026 40:14 Rethinking Automated Program Repair Table 9 The Advanced LLMs Performance Repaired BugsAttemptsRepaired Bugs / $ Attempts / $ GPT-5175/2116662.9411.19 GPT-5-reasoning3/361060.155.24 DeepSeek-V3.2-chat75/211155075.751565.56 DeepSeek-V3.2-reasoner64/13631823.97119.10 DeepSeek-V4-pro90/21114754.1768.29 DeepSeek-V4-pro-reasoner81/1212323.169.05 Llama-3.1-405B49/2111801-- advanced LLM-based approaches. To reduce API cost, ChatRepair stops when generating the first plausible patch. Subsection 3.6 describes the experimental environment, and Table 9 presents the results of ChatRepair combined with four LLMs and two reasoning settings. Cost analysis shows that inference costs increase substantially when more advanced LLMs are adopted. DeepSeek-V3.2-chat supports approximately 1,565.56 repair attempts per dollar in non-reasoning mode and 119.10 in reasoning mode, whereas DeepSeek-V4-pro decreases to 68.29 in non-reasoning mode, and 9.05 in reasoning mode. GPT-5 supports only 11.19 attempts per dollar in non-reasoning mode and 5.24 attempts per dollar in reasoning mode. These results show that the cost increases considerably as larger and reasoning-oriented LLMs are used. Compared with the advanced LLMs, DeepSeek-V3.2 is much cheaper than GPT-5 and DeepSeek-V4-pro. Therefore, both the choice and reasoning settings of LLMs heavily impact the cost-efficiency. The performance of different LLMs varies substantially under non-reasoning and reasoning settings. Figure 6 (a) presents the number of additionally repaired bugs achieved by each LLM when reasoning mode is enabled. GPT-5 repairs 175 bugs under non-reasoning mode, achieving the highest number of repaired bugs among all evaluated models, while DeepSeek-V4-pro and DeepSeek-V3.2 repair 90 and 75 bugs, respectively. However, when the reasoning effort is increased to high, GPT-5 repairs only 3 additional bugs. In contrast, enabling high-reasoning-mode effort allows DeepSeek-V3.2 and DeepSeek-V4-pro to repair 64 and 81 additional bugs, respectively. Finding 6: Both LLM selection and reasoning settings have significant impacts on repair effectiveness and cost, while the effectiveness of reasoning strategies differs across LLM families. Enabling reasoning mode provides only limited repair improvement for GPT-5, whereas DeepSeek-V3.2 and DeepSeek-V4-pro benefit considerably more from reasoning-based settings, repairing many bugs that cannot be fixed under non-reasoning configurations. Figure 6 (b) presents the repair rates of different LLMs and reasoning settings under the cost metric of attempts per dollar (Attempts/$). Combined with Figure 6 (a), GPT-5 repairs 175 bugs under the non-reasoning setting, which is higher than all other evaluated LLMs under both reasoning and non-reasoning settings (171 for DeepSeek-V4-pro and 139 for DeepSeek-V3.2). Moreover, GPT-5 under the non-reasoning setting is even cheaper than DeepSeek-V4-pro under the reasoning setting in terms of attempts per dollar (11.19 vs. 9.05). Although GPT-5 under the non-reasoning setting is more expensive than DeepSeek-V3.2 under the reasoning setting, the repair rate is markedly higher. Figure 6 (b) shows that under the non-reasoning setting, DeepSeek-V3.2 achieves compet- itive repair performance compared with DeepSeek-V4-pro, while maintaining a substantially lower cost. Therefore, using DeepSeek-V3.2 in non-reasoning mode for overall bug repair, and subsequently applying GPT-5 in non-reasoning mode to repair the remaining unresolved J. Liu et al.40:15 GPT-5 DeepSeek-V4 DeepSeek-V3.2 Llama 0 50 100 150 49 64 75 81 90 3 175 Fixed Bugs Number GPT-5GPT-5-reasoning DeepSeek-V4-proDeepSeek-V4-pro-reasoner DeepSeek-V3.2-chatDeepSeek-V3.2-reasoner Llama-3.1-405B (a) Repair Effectiveness 0510151565.56 0 0.2 0.4 0.6 0.8 1 GPT5 GPT5re DSV3.2chat DSV3.2re DSV4Pro DSV4Prore Attempts / $ Repair Rate Note: re = reasoning mode. DS = DeepSeek. (b) Cost efficiency Figure 6 Repair effectiveness and cost efficiency of advanced LLMs. bugs, provides a more effective trade-off between repair performance and cost. Finding 7: GPT-5 achieves the strongest overall repair effectiveness, whereas DeepSeek- V3.2-chat demonstrates significantly better cost-efficiency. Combining cost-efficient LLMs with more advanced LLMs for unresolved bugs can therefore achieve a better balance between repair effectiveness and cost-efficiency in LLM-based APR techniques. Table 10 presents theTop− kmetric for both APR techniques across three LLM families and three different FL strategies. The first five repair rounds usually resolve the largest proportion of bugs across evaluated APR techniques, LLMs, and FL settings. However, the marginal repair gain decreases considerably as additional repair rounds are introduced, with later rounds contributing only a limited number of newly repaired bugs. Finding 8: The majority of bugs are repaired within the first five repair rounds, whereas the marginal repair effectiveness decreases significantly in later rounds. RQ3 Summary. Using advanced LLMs and reasoning modes improves repair rates and enables APR techniques to fix more structurally complex bugs. However, these improvements come with substantially higher costs, making the trade-off between repair performance and cost increasingly important. Moreover, the benefits of reasoning modes vary across different LLM families. In our experiments, GPT-5 shows the smallest gain from reasoning, whereas the DeepSeek series repairs many additional bugs that cannot be fixed in non-reasoning mode. We further find that applying cost-efficient LLMs first, and then using advanced LLMs to repair the remaining unresolved bugs, can achieve a better balance between repair effectiveness and cost. 5 Discussion We investigated the effectiveness of LLM-based APR under different bug complexity categories and fault localization strategies. The results reveal several insights regarding the repair effectiveness and cost-efficiency of the evaluated LLM-based APR techniques, providing useful guidance for future APR research and practical deployment. Impacts of bug complexity: Our results show that bug complexity substantially affects ESEM 2026 40:16 Rethinking Automated Program Repair Table 10 Repair performance (Top-k) across APR configurations. Values are reported as incremental fixes and cumulative total in parentheses. APRLLMsFLTop-1 Top-2 Top-3 Top-4 Top-5 Top-10 C L3.1-8Bll82 (10)1 (11)0 (11)0 (11)0 (11) 4o-minill8023 (103) 8 (111) 6 (117) 4 (121) 5 (126) DS-V3.2l191 49 (240) 18 (258) 14 (272) 6 (278) 17 (295) DS-V3.2hl106 32 (138) 12 (150) 12 (162) 2 (164) 15 (179) DS-V3.2w/o FL8015 (95) 6 (101) 1 (102) 2 (104) 8 (112) CR L3.1-8Bll155 (20)1 (21)9 (30)6 (36)14 (50) 4o-minill6716 (83)9 (92) 14 (106) 9 (115) 25 (140) DS-V3.2l194 30 (224) 8 (232) 19 (251) 8 (259) 33 (292) DS-V3.2hl9338 (131) 22 (153) 26 (179) 16 (195) 42 (237) DS-V3.2w/o FL 128 30 (158) 14 (172) 11 (183) 7 (190) 15 (205) CR adv.LLM L3.1-405Bll1512 (27)0 (27)5 (32)3 (35)14 (49) GPT-5l103 43 (146) 16 (162) 4 (166) 5 (171) 4 (175) DS-V3.2l458 (53)4 (57)3 (60)3 (63)12 (75) DS-V4-proll3126 (57)5 (62)5 (67)7 (74)16 (90) GPT-5(re)l10 (1)2 (3)– DS-V3.2(re)l3319 (52) 12 (64)– DS-V4-pro(re) l5815 (73)8 (81)– Note: C=CodeCorrector, CR=ChatRepair, L3.1 = Llama-3.1, 4o-mini = GPT-4o-mini, DS = DeepSeek, re = reasoning, w/o = without, l = line-level FL, hl = hunk-level FL. the effectiveness of LLM-based APR techniques. The simplest bugs achieve much higher repair rates, while structurally complex bugs containing multiple buggy lines or diverse AST edit operations remain more challenging. The repair performance across different bug complexity levels suggests that future APR evaluations could pay more attention to complex repair scenarios rather than to single-line or single-operation bugs. Structurally complex bugs may better reflect some of the challenges encountered in practical APR tasks. New benchmarks and standard performance metrics could consider evaluations across different bug categories, domains, and complexity levels to better characterize tool performance [11, 37, 40]. Impacts of fault localization: Precise FL not only improves repair effectiveness, but also helps APR techniques generate patches that are more consistent with the original program logic. Such consistency may facilitate the understanding, validation, and maintenance of repaired programs. Additionally, existing LLM-based APR methods are affected differently by less precise FL information. This difference highlights robustness to imprecise FL as an important requirement for practical APR techniques [11]. Since real-world FL is often noisy and incomplete, methods that maintain stable repair performance when the precision of FL decreases are more likely to be effective in practice. For complex bugs, the relatively smaller effect of FL precision indicates that FL may not be the only bottleneck. Repairing these bugs may also depend on reasoning ability and deeper program-understanding capabilities. The Trade-off Between Repair Effectiveness and Cost-efficiency: Using more advanced LLMs and reasoning modes substantially increases computational cost, while often improving repair effectiveness. Based on the results presented in section 4.3, GPT-5 achieves the highest repair effectiveness, while DeepSeek-V3.2-chat provides the most cost-efficient repair performance. Our results suggest that such a two-stage strategy could be a promising direction for future investigation. The marginal repair efficiency drops as the number of repair rounds increases [35,51], which further impacts overall cost, especially when using advanced LLMs such as GPT- 5 and DeepSeek-V4-pro in reasoning mode. Our results also suggest that selecting an J. Liu et al.40:17 appropriate maximum number of repair rounds is important for balancing repair effectiveness and cost-efficiency. 6 Threats to Validity External Validity: Our dataset is collected solely from the competitive programming platform AtCoder and focuses only on C++ programs, which are typically shorter but more algorithm-oriented [13,25] than industrial software systems. Thus, the findings of our study may not generalize fully to real-world software projects with larger codebases, complex dependencies, and practical software engineering constraints. Furthermore, we evaluate only two APR techniques and three LLM series, and thus, the best-performing approaches identified in this study should not be regarded as definitive state-of-the-art APR solutions. The evaluated LLMs have different knowledge cutoff dates, and we collect benchmark data after 2025-10-01, which is after the knowledge cutoff date of the GPT and Llama models we used. DeepSeek did not publish its knowledge cutoff date, so it remains difficult to completely rule out the possibility that some competitive programming solutions or related code patterns were exposed during model pretraining [31, 38]. Internal Validity: We set the temperature to 1.0 for all evaluated LLMs and used high reasoning-effort settings instead of the maximum settings for GPT-5, DeepSeek-V3.2-reasoner, and DeepSeek-V4-pro due to their substantial time costs. Different parameter configurations and reasoning settings may lead to variations in repair performance. In addition, API-based LLMs may produce nondeterministic outputs across different runs, which could also affect repair results. Furthermore, different prompting strategies, repair rounds, and stopping criteria may influence the effectiveness and cost-efficiency of APR. Construct Validity: We determine repaired bugs mainly based on whether the generated patches can pass all collected test cases. However, the test cases may not be sufficiently comprehensive to fully validate the semantic correctness of repaired programs. Therefore, some plausible patches may still contain hidden defects or overfitting issues. 7 Conclusion and Future Work This work conducts an empirical study on how bug complexity and FL affect LLM-based APR techniques. We utilize Git, srcML, and GumTreeDiff to categorize bug complexity from both hunk-line-level and AST-operation-level perspectives, and to evaluate APR effectiveness under accurate, vague, and no fault-localization settings. The experimental results show that structurally complex bugs and imprecise FL generally reduce repair effectiveness. Nevertheless, well-designed LLM-based APR techniques still maintain considerable repair effectiveness under these challenging conditions. Furthermore, different LLMs exhibit clear trade-offs between repair effectiveness and cost-efficiency, high- lighting the importance of balancing repair capability, robustness, and cost for practical APR techniques. Future work may develop a benchmark that can automatically collect, filter, and categorize bug complexity. Also, we will consider practical factors such as API response latency [26,36] and overall inference time when evaluating the efficiency of LLM-based APR systems. We plan to explore additional LLM families, such as Gemini and Claude, to further evaluate the effectiveness of LLM-based APR. Finally, the agent-based APR frameworks should be further investigated, as agent-based techniques have become increasingly popular across various software engineering and reasoning tasks in recent years. ESEM 2026 40:18 REFERENCES 8 Data Availability To support transparency and reproducibility, the de-identified data and analysis used in this study are available as a replication package here. Dataset: <https://zenodo.org/records/20257267>; Code: <https://github.com/Leo6-sys/Rethink-APR-Empirical-Study-2026> References 1 Meta AI. Llama-3.1-405b. https://huggingface.co/meta-llama/Meta-Llama-3.1-405B, 2024. 2 Meta AI. Llama-3.1-8b. https://huggingface.co/meta-llama/Llama-3.1-8B, 2024. 3Aram Bahrini, Mohammadsadra Khamoshifar, Hossein Abbasimehr, Robert J. Riggs, Maryam Esmaeili, Rastin Mastali Majdabadkohne, and Morteza Pasehvar. ChatGPT: Applications, opportunities, and threats. In 2023 Systems and Information Engineering Design Symposium (SIEDS), pages 274–279. IEEE, 2023. doi:10.1109/SIEDS58326.2023.10137850. 4Andrew M Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan Eghlidi, Chris Schmitz, Karolina Korgul, Hunar Batra, et al. Measuring what matters: Construct validity in large language model benchmarks. Advances in Neural Information Processing Systems, 38, 2026. doi:10.52202/085713-0590. 5Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. RepairAgent: An autonomous, llm-based agent for program repair. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering, ICSE ’25, pages 2188–2200. IEEE, 2025. doi:10.1109/ICSE55347.2025.00157. 6Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. arXiv:2107.03374. 7Michael L. Collard, Michael J. Decker, and Jonathan I. Maletic. srcML: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration. In Proceedings of the 29th IEEE International Conference on Software Maintenance, ICSM ’13, pages 516–519. IEEE, 2013. doi:10.1109/ICSM.2013.85. 8Zhenlong Dai, Bingrui Chen, Zhuoluo Zhao, Xiu Tang, Sai Wu, Chang Yao, Zhipeng Gao, and Jingyuan Chen. Less is more: Adaptive program repair with bug localization and preference learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 128–136. AAAI Press, 2025. doi:10.1609/aaai.v39i1.31988. 9 DeepSeek-AI. Deepseek-v3.2. https://api-docs.deepseek.com/news/news251201, 2025. 10 DeepSeek-AI. Deepseek-v4. https://api-docs.deepseek.com/news/news260424, 2026. 11Sena Dikici and Turgay Tugay Bilgin. Advancements in automated program repair: a comprehensive review. Knowledge and Information Systems, 67(6):4737–4783, 2025. 12Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin Monperrus. Fine- grained and accurate source code differencing. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, pages 313–324. ACM, 2014.doi:10.1145/ 2642937.2642982. 13Claire Le Goues, Neal Holtschulte, Edward K. Smith, Yuriy Brun, Premkumar Devanbu, Stephanie Forrest, and Westley Weimer. The ManyBugs and IntroClass benchmarks for automated repair of c programs. IEEE Transactions on Software Engineering, 41(12):1236–1256, 2015.doi:10.1109/TSE. 2015.2454513. 14Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware LLM reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24842–24855. Association for Computational Linguistics, 2025.doi:10.18653/v1/2025.f indings-acl.1274. 15Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Anh Nguyen, and Omer Tripp. A deep dive into large language models for automated bug localization and repair. Proceedings of the ACM on Software Engineering, 1(FSE):1471–1493, 2024.doi:10.1145/36 60773. 16 AtCoder Inc. Atcoder. https://atcoder.jp/. 17Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology, 35(2):1–72, 2026. doi:10.1145/3747588. 18 Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. InferFix: End-to-end program repair with LLMs. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE ’23, pages 1646–1656. ACM, 2023. doi:10.1145/3611643.3613892. 19Sathvik Joel, Jie JW Wu, and Fatemeh H. Fard. A survey on LLM-based code generation for low- resource and domain-specific programming languages. ACM Transactions on Software Engineering and Methodology, 2025. doi:10.1145/3770084. REFERENCES40:19 20 René Just, Darioush Jalali, and Michael D. Ernst. Defects4J: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA ’14, pages 437–440. ACM, 2014.doi:10.1145/2610384.2628055. 21Fengjie Li, Jiajun Jiang, Jiajun Sun, and Hongyu Zhang. Evaluating the generalizability of llms in automated program repair. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER ’25, pages 91–95. IEEE, 2025. doi:10.1109/ICSE-NIER66352.2025.00024. 22Yingling Li, Muxin Cai, Junjie Chen, Yang Xu, Lei Huang, and Jianping Li. Context-aware prompting for llm-based program repair. Automated Software Engineering, 32(2):42, 2025.doi:10.1007/s10515 -025-00512-w. 23Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode. Science, 378(6624):1092–1097, 2022. doi:10.1126/science.abq1158. 24Bo Lin, Shangwen Wang, Ming Wen, Liqian Chen, and Xiaoguang Mao. One size does not fit all: Multi-granularity patch generation for better automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’24, pages 1554–1566. ACM, 2024. doi:10.1145/3650212.3680381. 25Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. QuixBugs: A multi-lingual program repair benchmark set based on the quixey challenge. In Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, SPLASH Companion ’17, pages 55–56. ACM, 2017.doi:10.1145/3135932.31 35941. 26Fan Liu, Wen-Shuo Chao, Naiqiang Tan, and Hao Liu. Bag of tricks for inference-time computation of llm reasoning. Advances in Neural Information Processing Systems, 38, 2026. 27 Matias Martinez, Thomas Durieux, Romain Sommerard, Jifeng Xuan, and Martin Monperrus. Auto- matic repair of real bugs in java: A large-scale experiment on the Defects4J dataset. Empirical Software Engineering, 22(4):1936–1964, 2017. doi:10.1007/s10664-016-9470-4. 28 Mary L McHugh. The chi-square test of independence. Biochemia medica, 23(2):143–149, 2013. doi:10.11613/BM.2013.018. 29 Scott Menard. Applied logistic regression analysis. SAGE publications, 2001.doi:10.4135/97814129 83433. 30 Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. ACM Transactions on Intelligent Systems and Technology, 16(5):1–72, 2025. doi:10.1145/3744746. 31Shiwen Ni, Xiangtao Kong, Chengming Li, Xiping Hu, Ruifeng Xu, Jia Zhu, and Min Yang. Training on the benchmark is not all you need. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24948–24956. AAAI Press, 2025. doi:10.1609/aaai.v39i23.34678. 32 OpenAI. Gpt-4o mini. https://platform.openai.com/docs/models/gpt-4o-mini, 2024. 33 OpenAI. Gpt-5. https://platform.openai.com/docs/models/gpt-5, 2025. 34 Yicheng Ouyang, Jun Yang, and Lingming Zhang. Benchmarking automated program repair: An extensive study on both real-world and artificial bugs. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’24, pages 440–452. ACM, 2024. doi:10.1145/3650212.3652140. 35 Mayur Amarnath Palavalli and Mark Santolucito. Using a feedback loop for llm-based infrastructure as code generation. arXiv preprint arXiv:2411.19043, 2024. 36Gyeongmuk Park, Seonghyeon Lee, and Yeonsu Park. Minimizing response latency in LLM-based agent systems: A comprehensive survey. IEEE Access, 14:26140–26168, 2026.doi:10.1109/ACCESS.2 026.3664226. 37Meghana Puvvadi, Sai Kumar Arava, Adarsh Santoria, Sesha Sai Prasanna Chennupati, and Har- sha Vardhan Puvvadi. Coding agents: A comprehensive survey of automated bug fixing systems and benchmarks. In 2025 IEEE 14th International Conference on Communication Systems and Network Technologies (CSNT), pages 680–686. IEEE, 2025. 38Daniel Ramos, Claudia Mamede, Kush Jain, Paulo Canelas, Catarina Gamboa, and Claire Le Goues. Are large language models memorizing bug benchmarks? In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pages 1–8. IEEE, 2025.doi:10.1109/LLM4Code66 737.2025.00005. 39Matthew Renze. The effect of sampling temperature on problem solving in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7346–7356. Association for Computational Linguistics, 2024. doi:10.18653/v1/2024.findings-emnlp.432. 40Joseph Renzullo, Pemma Reiter, Westley Weimer, and Stephanie Forrest. Automated program repair: Emerging trends pose and expose problems for benchmarks. ACM Computing Surveys, 57(8):1–18, 2025. 41Ripon K. Saha, Yingjun Lyu, Wing Lam, Hiroaki Yoshida, and Mukul R. Prasad. Bugs.jar: A large-scale, diverse dataset of real-world java bugs. In Proceedings of the 15th International Conference on Mining Software Repositories, pages 10–13. ACM, 2018. doi:10.1145/3196398.3196473. ESEM 2026 40:20 REFERENCES 42 Sandeep Singh Sengar, Affan Bin Hasan, Sanjay Kumar, and Fiona Carroll. Generative artificial intelligence: A systematic review and applications. Multimedia Tools and Applications, 84(21):23661– 23700, 2025. doi:10.1007/s11042-024-20016-1. 43 Yewei Song, Cedric Lothritz, Xunzhu Tang, Tegawendé Bissyandé, and Jacques Klein. Revisiting code similarity evaluation with abstract syntax tree edit distance. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 38–46. Association for Computational Linguistics, 2024. doi:10.18653/v1/2024.acl-short.3. 44 Diomidis Spinellis. Git. IEEE Software, 29(3):100–101, 2012. doi:10.1109/MS.2012.61. 45 Chi Wang, Xueqing Liu, and Ahmed Hassan Awadallah. Cost-effective hyperparameter optimization for large language model generation inference. In Proceedings of the Second International Conference on Automated Machine Learning, volume 224 of Proceedings of Machine Learning Research, pages 21/1–17. PMLR, 2023. URL: https://proceedings.mlr.press/v224/wang23b.html. 46Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. URL:https://proceedings.neurips.c/p aper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html. 47 Chunqiu Steven Xia and Lingming Zhang. Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’24, pages 819–831. ACM, 2024. doi:10.1145/3650212.3680323. 48Fengli Xu, Qianyue Hao, Chenyang Shao, Zefang Zong, Yu Li, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, et al. Toward large reasoning models: A survey of reinforced reasoning with large language models. Patterns, 6(10):101370, 2025.doi:10.1016/j.patter.2025.10 1370. 49Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang. ThinkRepair: Self-directed automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’24, pages 1274–1286. ACM, 2024.doi: 10.1145/3650212.3680359. 50 Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In The Twelfth International Conference on Learning Representations, 2024. URL: https://openreview.net/forum?id=6okaSfANzh. 51Yuze Zhao, Zhenya Huang, Yixiao Ma, Rui Li, Kai Zhang, Hao Jiang, Qi Liu, Linbo Zhu, and Yu Su. Repair: Automated program repair with process-based feedback. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16415–16429, 2024. 52Xin Zhou, Bowen Xu, Kisub Kim, DongGyun Han, Hung Huu Nguyen, Thanh Le-Cong, Junda He, Bach Le, and David Lo. Leveraging large language model for automatic patch correctness assessment. IEEE Transactions on Software Engineering, 50(11):2865–2883, 2024. doi:10.1109/TSE.2024.3452252. 53Fida Zubair, Maryam Al-Hitmi, and Cagatay Catal. The use of large language models for program repair. Computer Standards & Interfaces, 93:103951, April 2025. doi:10.1016/j.csi.2024.103951.