Paper deep dive
Unlocking LLM Code Correction with Iterative Feedback Loops
Le Zhang, Suresh Kothari
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 6/20/2026, 10:00:31 AM
Summary
This study investigates the ability of Large Language Models (LLMs) to rectify code through iterative feedback loops using execution feedback (compiler errors, runtime errors, and testcase failures). The researchers evaluated four models (DeepSeek-R1, DeepSeek-V3, GPT-o4-mini, and GPT-4.1-mini) across Python and Java using three datasets (Core, Strain, and Challenge) derived from LeetCode. The results demonstrate that reasoning models (like DeepSeek-R1) and high-performing models (like GPT-o4-mini) significantly improve success rates through iteration. While syntactic and runtime errors are easily corrected, logical and algorithmic failures remain more challenging. The study introduces the Iterative Success Rate (ISR@k) and Median Iterations to Solve (MIS) metrics to better quantify iterative performance.
Entities (12)
Relation Signals (5)
Core Dataset â derivedfrom â LeetCode
confidence 100% · Core Dataset: 450 randomly selected LeetCode problems
DeepSeek-R1 â evaluatedon â Core Dataset
confidence 100% · Four LLMs were tasked with solving 450 problems from the Core Dataset
Strain Dataset â subsetof â Core Dataset
confidence 100% · Strain Dataset: A 200-problem subset of the Core Dataset
DeepSeek-V3 â isanonreasoningmodel â Non-Reasoning Model
confidence 90% · DeepSeek-V3 ... Reasoning Model Ă
DeepSeek-R1 â isareasoningmodel â Reasoning Model
confidence 90% · DeepSeek-R1 ... Reasoning Model â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models have shown remarkable capabilities in code generation. However, most existing evaluations focus only on single-attempt accuracy and overlook the iterative refinement process that is central to real-world programming. This study presents a systematic investigation of LLMs' ability to rectify their own code through execution feedback. Using real-world programming problems across four models and two major programming languages, this study evaluates performance using iterative refinement framework where LLMs receive compiler error messages and testcase feedback after each attempt. This study introduces metrics to evaluate code failures, analyze rectification patterns, and compare the effectiveness of reasoning and non-reasoning models, offering actionable insights into both the understanding and practical application of feedback loops in LLM-driven code generation systems. Results show that reasoning models consistently improve over iterations, substantially outperforming non-reasoning models in leveraging feedback, while syntactic and runtime errors are far more tractable than logical or algorithmic failures.
Tags
Links
- Source: https://arxiv.org/abs/2606.17514v1
- Canonical: https://arxiv.org/abs/2606.17514v1
Trouble viewing inline? Open PDF directly â
Full Text
47,391 characters extracted from source content.
Expand or collapse full text
Unlocking LLM Code Correction with Iterative Feedback Loops Le Zhang and Suresh Kothari Iowa State University, Ames Iowa 50010, USA lezhang, kothari@iastate.edu Abstract. Large Language Models have shown remarkable capabilities in code generation. However, most existing evaluations focus only on single-attempt accuracy and overlook the iterative refinement process that is central to real-world programming. This study presents a sys- tematic investigation of LLMsâ ability to rectify their own code through execution feedback. Using real-world programming problems across four models and two major programming languages, this study evaluates performance using iterative refinement framework where LLMs receive compiler error messages and testcase feedback after each attempt. This study introduces metrics to evaluate code failures, analyze rectification patterns, and compare the effectiveness of reasoning and non-reasoning models, offering actionable insights into both the understanding and practical application of feedback loops in LLM-driven code generation systems. Results show that reasoning models consistently improve over iterations, substantially outperforming non-reasoning models in leverag- ing feedback, while syntactic and runtime errors are far more tractable than logical or algorithmic failures. Keywords: Large Language Models (LLMs), Code Generation, Feed- back Loop, Iterative Refinement, Model Evaluation 1 Introduction Large Language Models (LLMs), such as DeepSeek and ChatGPT, are capable of generating code from natural language specifications and have the potential to significantly reduce human effort in software development. However, these ef- ficiency gains can quickly diminish if the generated code fails during testing, ne- cessitating human intervention for correction. Therefore, it is crucial that LLMs not only generate but also rectify the code. Analogous to human programmers, LLMs require feedback loops to iteratively improve their code. In such loops, testcase failures and error messages are fed back to the model, which in turn attempts to correct the code. Despite its importance, there is limited knowledge about the effectiveness of feedback loops in LLMs, and there are no established practical guidelines for their implementation. This study aims to systematically investigate LLMs with feedback loops by experimenting with state-of-the-art models. While previous work has extensively arXiv:2606.17514v1 [cs.SE] 16 Jun 2026 2Le Zhang et al. evaluated LLM code generation [6,5,7,15,9,20,13,8,10,11,18,34], these studies pri- marily focus on metrics such as pass@1 or pass@k and do not explore the role of feedback loops in code rectification. This study focus on aspects that are particularly relevant to real-world soft- ware development, where feedback loops typically operate automatically. In this study, researchers implement an automated feedback loop consisting of the fol- lowing steps: (i) execute the code generated by the LLM on a test case, (i) construct a prompt that incorporates any resulting failure messages, and (i) provide this prompt to the LLM as execution feedback. The loop initiates with code that has failed during testing. For each programming task, researchers em- ploy several hundred test cases designed to cover corner cases, logical errors, and violations of space and time constraints. The feedback loop iterates until the LLM either successfully rectifies the code or exhausts the opportunity for improvement. Conducting a systematic investigation of feedback in LLMs entails several challenges. First, it requires establishing a baseline understanding of LLM fail- ures across diverse scenarios, including logical errors and violations of computa- tional constraints. Second, realistic feedback loops must emulate practical soft- ware development, comprehensive evidence of failures while allowing for code rectification. Third, careful monitoring of the feedback process is necessary to track and characterize the types of improvements achieved and the failures that persist. Finally, the study must account for variability across LLMs, includ- ing differences between reasoning and non-reasoning models, to assess whether state-of-the-art reasoning models consistently outperform their non-reasoning counterparts. The main contributions of this research are threefold. First, a novel method- ology for systematically exploring and evaluating the impact of feedback in LLMs is proposed. Second, an extensive experimental setup that enables a rigorous in- vestigation of LLM code rectification in realistic conditions is developed. Third, actionable insights for software development, including metrics to assess code failures, patterns of rectification, and the comparative effectiveness of reason- ing versus non-reasoning models are provided. Collectively, these contributions advance both the understanding and practical application of feedback loops in LLM-driven code generation. The remainder of this paper is organized as follows. Section 2 details the research methodology. Section 3 presents the experimental results, followed by Section 4, which discusses notable cases and the limitations of this study. Sec- tion 5 reviews existing research. Finally, Section 6 summarizes the key findings and outlines directions for future research. Iterative Feedback Loops3 2 Research Methodology 2.1 Research Questions This study investigates how LLMs revise code using execution feedback. To explore this challenging problem, this study is framed through the following research questions. RQ1: How do LLMs perform across different problem difficulties using standard pass@1 evaluation metrics? This question establishes the baseline capabilities of LLMs, revealing how model performance scales with task complexity and where failures arise. Under- standing these baseline limitations, across functional errors, unmet constraints, and efficiency bottlenecks, is crucial for assessing the potential impact of feedback- driven code rectification. Detailed analysis and findings are presented in Sec- tion 3.1. RQ2: To what extent do LLMs respond to instructive prompts for algorithmic optimization? This question investigates whether explicit guidance improves the efficiency of generated code. Responsiveness to such prompts reflects an LLMâs ability to leverage execution feedback, a prerequisite for effective iterative refinement. Re- sults are presented in Section 3.2. RQ3: How effectively can LLMs use execution feedback to correct code with errors? This question examines the ability of LLMs to iteratively correct code that fails on the first attempt. Assessing this capability reveals the potential for au- tonomous code improvement and informs practical considerations for using iter- ative feedback loops. Detailed results are presented in Section 3.3. RQ4: How does iterative refinement performance differ between reasoning and non-reasoning LLMs? This question compares reasoning and non-reasoning models in their iter- ative refinement performance, highlighting factors that influence self-correction and guiding model selection for feedback-driven tasks. Results and analysis are presented in Section 3.4. RQ5: Which types of errors, syntactic, logical, or algorithmic, are most effec- tively corrected by iterative refinement? This question analyzes which error types are most amenable to correction, comparing recovery rates across models, programming languages, and error cate- gories. Understanding these patterns helps systematically map the strengths and limitations of iterative feedback-driven code generation. Results are presented in Section 3.5. 4Le Zhang et al. 2.2 Datasets and Models A controlled experiment is conducted to evaluate LLM performance in code generation, systematically assessing models across programming languages and establishing a baseline for subsequent analysis. Table 1: Problem distribution by difficulty tiers. DatasetEasy Medium Hard Total Core Dataset150 150 150450 Strain Dataset0101 99200 Challenge Dataset 062632 Data Collection Three datasets were used, each serving a distinct purpose: â Core Dataset: 450 randomly selected LeetCode problems evenly distributed across difficulty levels (150 easy, 150 medium, 150 hard), covering algorithms and data structures such as greedy algorithms, sorting, binary search, and tree-based problems. This dataset provides a representative benchmark for general code generation quality. â Strain Dataset: A 200-problem subset of the Core Dataset manually se- lected for algorithmic efficiency based on size of the inputs, emphasizing optimized implementations that meet strict time constraints. This dataset evaluates modelsâ ability to produce computationally efficient code. â Challenge Dataset: Comprising the 32 most frequently failed problems across all models and languages in the baseline evaluation. These problems are rarely solved on the first attempt, making this dataset ideal for assessing iterative refinement and feedback-driven improvement. LeetCode was chosen for its structured repository, rich test cases, clear cat- egorization, and emphasis on time and space efficiency, which aligns with the objectives of evaluating algorithmic performance and execution efficiency. Ta- ble 1 summarizes the key characteristics of these datasets. Table 2: Models used in our study. Model Size (B) Release Year Open Source Reasoning Model DeepSeek-R16712025 â â DeepSeek-V36712024 â Ă GPT-o4-mini8 (Est.) 2025 Ă â GPT-4.1-mini8 (Est.) 2025 Ă Ă Iterative Feedback Loops5 Selection of Models: Four state-of-the-art LLMs are evaluated: DeepSeek-R1 and DeepSeek-V3 (DeepSeek), and GPT-o4-mini and GPT-4.1-mini (OpenAI). These models were selected for accessibility, popularity, and representation of state-of-the-art code generation approaches (Table 2). By comparing DeepSeek- R1 vs. DeepSeek-V3 and GPT-o4-mini vs. GPT-4.1-mini, the influence of reason- ing capabilities on coding and iterative refinement performance can be isolated. 2.3 Experimental Design: Baseline Experiment While baseline pass@1 experiments measure single-attempt performance, real- world coding often involves iterative refinement based on compiler or runtime feedback. To evaluate LLMsâ ability to improve solutions over multiple attempts, iterative refinement experiments are conducted where models leveraged testcase results and execution feedback as a form of instructive guidance. Parameter Configuration To ensure consistent outputs, key decoding param- eters were carefully configured. GPT-o4-mini does not allow user-configurable parameters and was evaluated using its defaults. For all other models, we set top-p = 0.95, following prior studies that show this value improves pass@1 performance [4,3,30,1,29]. This setting broadens the sampling distribution, in- creasing the likelihood of generating correct solutions. Temperature controls token randomness. While higher values may help multi- attempt metrics like pass@k, they can reduce single-attempt (pass@1) perfor- mance [3]. We set temperature = 0.1 to balance output diversity and deter- minism, ensuring stable, reproducible results. Experimental Procedure Four LLMs were tasked with solving 450 problems from the Core Dataset in both Python and Java. Each model had a single at- tempt per problem per language. Generated code was submitted to LeetCode via a custom API, evaluated using official test cases, and results collected in JSON format. All experiments adhered to LeetCodeâs usage policies to ensure ethical compliance and minimal service disruption. Performance was assessed using the pass@1 metric (Section 2.5). Prompt Design A structured prompt was used to ensure consistent and high- quality outputs (Figure 1). The prompt instructs the model to act as âa software developer,â specifies the target programming language, includes the full prob- lem statement with examples and constraints, and provides a partially completed code snippet along with sample test cases. The prompt concludes by request- ing only the final code solution, without explanations, to facilitate automated evaluation. This design minimizes ambiguity, standardizes outputs across models, and promotes functional correctness and computational efficiency, aligning with real- world software development practices. 6Le Zhang et al. # Start of the Prompt You are a software developer. Implement a solution in Python for the following coding problem. Problem Description: Given an integernum, repeatedly add all its digits until the result has only one digit, and return it. Example: Input:num = 38 Output: 2 Explanation: The process is 38â 3 + 8â 11, then 11â 1 + 1â 2 Since 2 has only one digit, return it. Constraints: 0â€num†2 31 â 1 Code Snippet: class Solution: def addDigits(self, num: int) â int: Testcases: num = 38;num = 0 Additional Instructions: Follow the input constraints and write your code starting from the given code snippet. Ensure the code is well-formatted and adheres to best practices. Write the executable code only, avoid unnecessary explanations or comments. # End of the Prompt Fig. 1: Prompt Structure Example for Baseline Experiment 2.4 Experimental Design: Iterative Refinement Experiment While the baseline pass@1 experiments provide a measure of single-attempt performance, real-world coding often involves iterative refinement based on ex- ecution feedback. To evaluate LLMsâ ability to improve solutions over multiple attempts, iterative refinement experiments are conducted using test results and runtime feedback. Parameter Configuration GPT-o4-mini was evaluated using default param- eters, as they are not user-configurable. For the other three models, we set tem- perature = 0.9, following prior work showing higher temperatures improve multi-attempt code generation [3]. Since top-p guidance for iterative tasks is limited, a preliminary calibration is conducted using DeepSeek-R1 over 10 iterations on a subset of problems. Performance peaked at top-p = 0.3, which was applied to all models in iterative experiments (Table 3). Iterative Feedback Loops7 1234567891011121314151617181920 Iteration 0 1 2 3 4 5 Cumulative Solved Problems Fig. 2: Iteration limit calibration with DeepSeek-R1. Table 3: Metrics by different top-p values. Metric 0.10.30.50.70.9 ISR@10 65.6% 68.8% 68.8% 65.6% 62.5% MIS5465.54 Iteration Limit Setup To balance performance and computational cost, we determined an appropriate iteration limit via a preliminary experiment on 10 randomly sampled problems from the Challenge Dataset. Figure 2 showed that success rates plateaued after the 8th iteration. Accounting for marginal cases, we set a maximum of 10 iterations per problem. Experimental Procedure: As demonstrated in Figure 3, the iterative frame- work uses a multi-turn mechanism: each iteration involves (1) sending a prompt, (2) generating a solution, and (3) receiving execution feedback. The initial prompt follows the baseline structure (Section 2.3). Subsequent prompts include all pre- vious solutions and feedback, giving the model full context. Iterations continue until a correct solution is obtained or the 10th iteration is reached. All solutions and results were stored for analysis using metrics from Section 2.5. Execution Feedback LeetCode categorizes results as Accepted, Compile Er- ror, Runtime Error, Wrong Answer, Time Limit Exceeded, and Memory Limit Exceeded. Each error type can typically be corrected in specific ways (e.g., Time Limit Exceeded via algorithmic optimization). Execution feedback is provided to guide LLMs toward effective corrections, framing the feedback as actionable guidance in the form of instructive prompts (Figure 4). 8Le Zhang et al. Fig. 3: Iterative Experiment Procedure # Compile Error Feedback: Your code crashed at a compile error: [Error Message] Fix your code with above information. # Runtime Error Feedback: Your code crashed at a runtime error: [Error Message] Fix your code with above information. # Wrong Answer Feedback: Your code generated wrong outputs at testcase: [Testcase] Expected output: [Expected Output ] Actual output: [Actual Output ] # Time Limit Exceeded Feedback: Your code exceeded the maximum runtime allowance. Optimize the time complexity of your algorithm. # Memory Limit Exceeded Feedback: Your code exceeded the maximum memory allowance. Optimize the space complexity of your algorithm to reduce memory usage. Fig. 4: Examples of execution feedback [ role: âuserâ, content: âYou are a software developer. Implement ...â , role: âassistantâ, content: âSolution code #1â, role: âuserâ, content: âYour code crashed at a compile error: ...â, role: âassistantâ, content: âSolution code #2â, role: âuserâ, content: âYour code generated wrong output ...â, role: âassistantâ, content: âSolution code #3â, ... role: âuserâ, content: âYour code exceeded the maximum memory ...â, role: âassistantâ, content: âSolution code #10â ] Fig. 5: Example of multi-turn conversation with LLM Iterative Feedback Loops9 Prompt Design A standard multi-turn JSON-based prompt format is adopted (Figure 5). Prompts are labeled as âuserâ and model responses as âassistantâ. In each iteration, the LLM accesses the full conversation history, enabling it to leverage prior solutions and instructive feedback to generate improved outputs over successive iterations. 2.5 Evaluation Metrics pass@1 Metric In the baseline experiment, each model had a single attempt to solve every problem in each programming language. Generated solutions were submitted to LeetCode, and the results recorded. For a dataset of N problems, let S i = 1 if the model passes problem i on its first attempt, and S i = 0 otherwise. The pass@1 score is then computed as: pass@1 = 1 N N X i=1 S i (1) where pass@1 â [0, 1] represents the proportion of problems successfully solved on the first attempt. ISR@k Metric: In the iterative refinement framework, each LLM is allowed up to k iterations per problem. A problem is marked as a PASS if any iteration produces a correct solution; otherwise, it is marked as a FAIL. We define the Iterative Success Rate (ISR@k) as the percentage of problems solved within k iterations: ISR@k = 1 N N X i=1 1 max j=1,...,k S (j) i = 1 (2) where: â N is the total number of problems, â S (j) i = 1 if iteration j on problem i passes all test cases (otherwise 0), â 1(·) is the indicator function, returning 1 if the condition is true and 0 if false. MIS Metric: Consider a scenario in which two LLMs achieve the same ISR@k value. Model-A may solve most problems within three iterations, whereas Model- B requires more than eight iterations for the same problems. Clearly, Model-A is more efficient, but ISR@k alone cannot capture this distinction. To address this, we introduce the **Median Iterations to Solve (MIS)** met- ric, which quantifies the typical number of iterations a model requires to solve problems and highlights differences in iteration efficiency. 10Le Zhang et al. For a dataset of (N) problems, each problem (p) is assigned a score (S p ) as follows: â If problem (p) is solved at iteration (i) ((i†k)), set (S p = i). â If problem (p) remains unsolved after (k) iterations, set (S p = k + 1) to distinguish it from problems solved on the last iteration. The MIS is then defined as the median of these scores: MIS = median(S 1 , S 2 , . . . , S N )(3) Lower MIS values indicate higher efficiency, reflecting a modelâs ability to reach correct solutions in fewer iterations. 2.6 Replication Package All artifacts related to this study, including LeetCode APIs and tools, LeetCode problem datasets, experimental results, and LLM-generated code are available in this public repository: https://github.com/lezhangisu/LLM-Code-Correction. 3 Experimental Results and Data Analysis 3.1 RQ1: Baseline Performance Table 4: pass@1 score by difficulty tier and programming language. Difficulty Tier Programming Language DeepSeek V3 DeepSeek R1 GPT 4.1-mini GPT o4-mini Easy Python99.33100.0097.3398.67 Java98.6798.6797.33 98.67 Medium Python71.3386.6776.67 88.67 Java70.6786.0074.00 89.33 Hard Python46.6765.3354.67 80.00 Java45.3362.6754.00 74.00 Overall Python72.4484.0076.22 89.11 Java71.5682.4475.11 87.33 Following the experimental procedure outlined in Section 2.3, we evaluated the coding capabilities of four LLMs across two programming languages using the pass@1 metric. The results, summarized in Table 4, provide a baseline view of how well these models perform across languages and difficulty levels. Iterative Feedback Loops11 DeepSeek-V3DeepSeek-R1GPT-4.1-miniGPT-o4-mini 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Time Limit Exceeded Error Count 16.0 10.010.0 3.0 12.0 3.0 6.0 1.0 No Hint With Hint Fig. 6: Time Limit Exceeded error count with and without suggestive hint. Overall, all models achieved high accuracy on easy problems, but performance declined notably as problem difficulty increased. DeepSeek-V3 and GPT-4.1- mini exhibited the steepest drops on medium and hard tasks, whereas GPT-o4- mini maintained the most stable and highest overall performance. DeepSeek-R1 performed moderately across all levels, suggesting a balanced but less specialized capability profile. When comparing languages, Python consistently yielded higher pass@1 scores than Java across nearly all models and difficulty tiers. This trend indicates that LLMs are generally more proficient in Python-based code generation, likely due to Pythonâs simpler syntax[31], dynamic typing[32], and greater representation in public training data[33]. 3.2 RQ2: Prompt Sensitivity Analysis Building on the baseline results, we observed that while LLMs performed well on many problems, they still struggled to solve more complex tasksâparticularly those requiring algorithmic optimization. To examine whether instructive guid- ance could mitigate this limitation, we re-evaluated all models on 200 problems from the Strain Dataset (Section 2.2). In this experiment, we appended a one-line instruction to the prompt: âOptimize the time complexity of your algorithm.â As shown in Figure 6, this simple optimization-oriented prompt substan- tially reduced the number of unsuccessful Java solutions across all models. The improvement was most pronounced on medium and hard problems, indicating that explicit optimization cues can effectively guide LLMs toward generating more efficient and correct code. These findings suggest that LLMs can respond adaptively to concise, goal-directed instructions, highlighting the potential of prompt engineering as a lightweight yet effective strategy for enhancing LLM performance on challenging programming tasks. 12Le Zhang et al. DeepSeek-V3DeepSeek-R1GPT-4.1-miniGPT-o4-mini 0 20 40 60 80 100 Success Rate (%) Python 9.4 0.0 6.3 31.3 21.9 71.9 25.0 81.3 Baseline Iterative DeepSeek-V3DeepSeek-R1GPT-4.1-miniGPT-o4-mini 0 20 40 60 80 100 Success Rate (%) Java 12.5 0.0 9.4 28.1 15.6 62.5 18.8 87.5 Fig. 7: Success rate comparison, baseline vs iterative framework. 3.3 RQ3: Code Correction with Iterative Feedback Building on the previous finding that instructive prompts can guide model be- havior, we next investigated whether LLMs can autonomously improve through feedback-based iteration. Specifically, we examined how models respond when provided with informative feedback such as runtime error messages and failed test case details. To this end, we selected 32 of the most frequently failed prob- lems across all models to construct the Challenge Dataset (Section 2.2) and ap- plied the iterative refinement framework described in Section 2.4 to four LLMs across two programming languages. As shown in Figure 7, all models exhibited substantial gains in success rate compared to their single-attempt performance. Figures 8 and 9 further illustrate the cumulative number of successful solutions in Python and Java, respectively. DeepSeek-R1 and GPT-o4-mini, in particular, demonstrated consistent improve- ment across iterations, suggesting that these models can effectively interpret and act upon execution feedback. In contrast, DeepSeek-V3 and GPT-4.1-mini showed smaller but still noticeable gains, indicating limited but present feedback utilization capability. Overall, these results suggest that LLMs can indeed leverage execution feed- back to iteratively refine their code, though the degree of improvement varies by model and programming language. This finding highlights the potential of feedback-driven refinement as a powerful strategy for enhancing LLM problem- solving robustness beyond single-attempt code generation. 3.4 RQ4: Reasoning vs. Non-Reasoning Models Figures 8 and 9 reveal a clear distinction between reasoning models (DeepSeek- R1 and GPT-o4-mini) and non-reasoning models (DeepSeek-V3 and GPT-4.1- mini). The reasoning models consistently improved their success rates across multiple iterations, showing steady gains as they incorporated execution feed- back to refine their solutions. In contrast, the non-reasoning models exhibited only marginal improvements, often plateauing after one or two iterations. Iterative Feedback Loops13 12345678910 Iteration 0 5 10 15 20 Cumulative Solved Problems Models DeepSeek_R1 GPT-o4-mini DeepSeek_V3 GPT-4.1-mini Fig. 8: Cumulative ISR@10 for Python solutions. 12345678910 Iteration 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Cumulative Solved Problems Models DeepSeek_R1 GPT-o4-mini DeepSeek_V3 GPT-4.1-mini Fig. 9: Cumulative ISR@10 for Java solutions. These results indicate that models equipped with explicit reasoning capa- bilities, such as chain-of-thought (CoT) mechanisms, are more adept at inter- preting and utilizing feedback signals during iterative refinement. By contrast, non-reasoning models, despite comparable architectures and model sizes, lack the internal reasoning structure needed to translate feedback into meaningful code revisions. This finding highlights that iterative improvement depends not only on model scale or training data, but also on the modelâs underlying reasoning capacity to generalize from feedback and self-correct effectively. 14Le Zhang et al. Table 5: Statistic results for error analysis. ModelLanguage Compile Error Runtime Error Wrong Answer Time Limit Exceeded Memory Limit Exceeded total fixed % total fixed % total fixed % total fixed % total fixed % DeepSeek-R1 Python00â14 10 71.475 45 60.0 396 15.433 100.0 Java22 100.011 100.0 93 52 55.9 50 12 24.000â GPT-o4-mini Python00â12 11 91.765 34 52.3 146 42.800â Java98 88.932 66.759 32 54.281 12.500â DeepSeek-V3 Python00â11 100.0 210 60 28.6 426 14.340 0.0 Java54 80.031 33.3 237 73 30.8 289 32.110 0.0 GPT-4.1-mini Python00â22 100.0 222 53 23.9 234 17.410 0.0 Java11 100.0 100 0.0223 58 26.0 262 7.731 33.3 OverallPython00â29 24 82.8 572 192 33.6 118 22 18.683 37.5 Java17 15 88.2174 23.6 612 215 35.1 112 24 21.441 25.0 3.5 RQ5: Error Analysis In the iterative refinement experiments, we analyzed the distribution and cor- rection rates of different error types based on LeetCodeâs standard categories: Compile Error, Runtime Error, Wrong Answer, Time Limit Exceeded, and Mem- ory Limit Exceeded. The evaluation reports provided both the error type and the number of passed testcases for each iteration. To formally characterize when an error was considered fixed, we adopt the following definition: Definition 1. Let T i denote the number of testcases passed by the solution at iteration i. An error observed at iteration i is considered fixed if and only if T i+1 > T i .(4) In other words, a fix is recognized when the subsequent iteration yields an im- provement in the number of successfully passed testcases. Using this definition, we derived aggregate statistics across all models, as summarized in Table 5. The majority of failures, approximately 95%, belonged to the Wrong Answer and Time Limit Exceeded categories. These two types were also the most challenging to correct, with average fix rates of roughly 35% and 20%, respectively. The low fix rate for Time Limit Exceeded errors likely stems from their algorithmic nature, often involving inefficient logic, infinite loops, or suboptimal data structures, issues that require substantial code restructuring rather than minor syntax or logic edits. Interestingly, Compile Error appeared exclusively in Java solutions, suggest- ing that Pythonâs simpler and more flexible syntax makes it easier for LLMs to maintain syntactic correctness. Runtime Errors occurred more frequently in Python but were also among the easiest to correct, likely because they arise from direct execution failures that provide clear feedback signals. Iterative Feedback Loops15 Overall, these results indicate that syntactic and runtime errors are the most amenable to correction through iterative refinement, while algorithmic ineffi- ciencies and logical errors remain considerably harder to resolve. This pattern underscores the current strength of LLMs in local debugging and incremental code repair, contrasted with their more limited capacity for deep algorithmic reasoning and structural optimization. 4 Key Findings and Limitations This study provides a systematic analysis of LLMsâ capabilities for iterative code refinement, moving beyond static, single-attempt evaluations to highlight their feedback-driven potential. 4.1 Key Findings Iterative evaluation reveals latent capabilities: ISR@k demonstrates that many models can solve challenging problems through multiple attempts, high- lighting adaptive potential missed by pass@1. Reasoning capacity matters: Reasoning models (DeepSeek-R1, GPT-o4- mini) steadily improve across iterations, while non-reasoning models plateau quickly, indicating that chain-of-thought abilities enhance feedback utilization. Prompt guidance enhances efficiency: Optimization-oriented prompts re- duced inefficiency errors, showing that instructive guidanceâwhether via prompts or integrated feedbackâimproves refinement outcomes. Error-type fixability varies: Syntactic and runtime errors are easier to fix (often >80%), whereas logical and algorithmic errors (Wrong Answer, Time Limit Exceeded) remain challenging (<35%), revealing current limitations in deep algorithmic reasoning. Implications for Benchmarking: Relying solely on pass@1 risks misrepre- senting a modelâs practical utility. Metrics like ISR@k and MIS better capture robustness, iterative improvement, and efficiency, providing a more realistic as- sessment aligned with real-world coding workflows. 4.2 Limitations Dataset scope: Evaluation focused on LeetCode algorithmic problems, which may not generalize to domains like web development or systems programming. Structured feedback: Feedback from LeetCode evaluation system was ideal- ized and automated; real-world errors are often ambiguous or incomplete. Computational cost: Iterative refinement is resource-intensive, highlighting the need for smarter iteration strategies. Potential regressions: Repeated iterations may introduce new errors or vul- nerabilities, which current metrics do not capture. 16Le Zhang et al. 5 Related Work Generative AI models have rapidly advanced in synthesizing source code from natural language, reshaping software development. Early models like Codex [5] achieved syntactically and functionally correct programs on benchmarks such as HumanEval [5], while CodeGen [7] extended capabilities to multi-turn code synthesis. Production systems like GitHub Copilot demonstrate the real-world applicability of these models [8]. Standard evaluation metrics, including pass@1 and pass@k, assess single- attempt correctness but ignore iterative refinement, a key aspect of human pro- gramming [2,5,9,8,22,23,24,25,26,34]. Recent studies address this limitation: G- Pass@k [17] measures consistency across generations, and Pass@k Training [18] leverages pass-based metrics as optimization objectives. A growing body of work explores LLM self-correction via feedback. Frame- works such as ORPS [10], LDB [11], and CRITIC [12] show that incorporating verification and runtime feedback enhances performance. Systems like SelfE- volve [9], CodeIt [13], VeriMind [14], and GPU Kernel Scientist [16] demon- strate iterative refinement across domains, highlighting modelsâ ability to act as autonomous agents. Beyond correctness, interaction design [19] and prompt adaptation [27,28] improve alignment with user intent, as shown in ChatCoder [20] and Parsing Requirements for Automatic Prompting [21]. Surveys [8,15,22] emphasize holistic evaluation encompassing adaptability, reasoning depth, and self-correction. Despite these advances, gaps remain in systematically quantifying how LLMs leverage feedback to recover from errors across models, error types, and program- ming languages. This study addresses this challenge with a large-scale, multi- model investigation into iterative refinement framework, providing empirical in- sights into LLM reasoning and error recovery dynamics. 6 Conclusion and Future Work This study explored the iterative self-correction capabilities of LLMs in code generation through a novel iterative refinement framework that simulates real- world programming workflows. Experiments across three datasets, four models, and two languages showed that LLMs can improve success rates through itera- tive refinement, though gains vary with model architecture, reasoning ability, and error type. Reasoning models such as DeepSeek-R1 and GPT-o4-mini steadily improved across iterations, while non-reasoning models plateaued early. Syntac- tic and runtime errors were often resolved quickly, whereas logical and algorith- mic errors remained challenging, highlighting current limits in deep algorithmic reasoning. Future work should explore richer feedback mechanisms, like model-generated unit tests or symbolic execution traces, and extend iterative evaluation to other software engineering tasks such as vulnerability repair, code translation, and legacy system modernization. Additionally, fine-tuning models specifically for Iterative Feedback Loops17 iterative debugging, potentially via reinforcement learning or hindsight replay, may further enhance reliability and efficiency in multi-step reasoning tasks. By quantifying how LLMs learn from mistakes and adapt through feedback, this work not only advances the scientific understanding of the ability of LLMs but also offers practical metrics and methodologies for building next-generation, feedback-driven AI programming assistants. References 1. Coignion, T., Quinton, C. & Rouvoy, R. A performance study of llm-generated code on leetcode. Proceedings Of The 28th International Conference On Evaluation And Assessment In Software Engineering. p. 79-89 (2024) 2. D Ìoderlein, J., Kouadio, N., Acher, M., Khelladi, D. & Combemale, B. Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?. Journal Of Systems And Software. p. 112562 (2025) 3. Zheng, S., Zhang, Y., Zhu, Y., Xi, C., Gao, P., Xun, Z. & Chang, K. Gpt-fathom: Benchmarking large language models to decipher the evolutionary path towards gpt-4 and beyond. Findings Of The Association For Computational Linguistics: NAACL 2024. p. 1363-1382 (2024) 4. Holtzman, A., Buys, J., Du, L., Forbes, M. & Choi, Y. The Curious Case of Neural Text Degeneration. International Conference On Learning Representations. (2020) 5. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G. & Others Evaluating large language models trained on code. https://doi.org/10.48550/arXiv.2107.03374. (2021) 6. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q. & Others Program synthesis with large language models. https://doi.org/10.48550/arXiv.2108.07732. (2021) 7. Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S. & Xiong, C. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. The Eleventh International Conference On Learning Represen- tations . (2023) 8. Jiang, J., Wang, F., Shen, J., Kim, S. & Kim, S. A Survey on Large Lan- guage Models for Code Generation. ACM Trans. Softw. Eng. Methodol.. (2025,7), https://doi.org/10.1145/3747588 9. Jiang, S., Wang, Y. & Wang, Y. Selfevolve: A code evolution framework via large language models. https://doi.org/10.48550/arXiv.2306.02907. (2023) 10. Yu, Z., Gu, W., Wang, Y., Jiang, X., Zeng, Z., Wang, J., Ye, W. & Zhang, S. Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation. Forty-second International Conference On Machine Learning. (2025) 11. Zhong, L., Wang, Z. & Shang, J. Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step. Findings Of The Asso- ciation For Computational Linguistics ACL 2024. p. 851-870 (2024) 12. Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N. & Chen, W. Critic: Large language models can self-correct with tool-interactive critiquing. https://doi.org/10.48550/arXiv.2305.11738. (2023) 13. Butt, N., Manczak, B., Wiggers, A., Rainone, C., Zhang, D., Defferrard, M. & Cohen, T. CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay. ICML. (2024) 18Le Zhang et al. 14. Nadimi,B.,Boutaib,G.&Zheng,H.Verimind:Agenticllm forautomatedveriloggenerationwithanovelevaluationmetric. https://doi.org/10.48550/arXiv.2503.16514. (2025) 15. Wang, J. & Chen, Y. A review on code generation with llms: Application and evaluation. 2023 IEEE International Conference On Medical Artificial Intelligence (MedAI). p. 284-289 (2023) 16. Andrews, M. & Witteveen, S. GPU Kernel Scientist: An LLM-Driven Framework for Iterative Kernel Optimization. https://doi.org/10.48550/arXiv.2506.20807. (2025) 17. Liu, J., Liu, H., Xiao, L., Wang, Z., Liu, K., Gao, S., Zhang, W., Zhang, S. & Chen, K. Are Your LLMs Capable of Stable Reasoning?. Findings Of The Association For Computational Linguistics: ACL 2025. p. 17594-17632 (2025) 18. Chen, Z., Qin, X., Wu, Y., Ling, Y., Ye, Q., Zhao, W. & Shi, G. Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models. https://doi.org/10.48550/arXiv.2506.20807. (2025) 19. Shukla,S.,Joshi,H.&Syed,R.SecurityDegradationinItera- tiveAICodeGenerationâASystematicAnalysisoftheParadox. https://doi.org/10.48550/arXiv.2506.11022. (2025) 20. Wang, Z., Li, J., Li, G. & Jin, Z. ChatCoder: Chat-based refine requirement im- proves LLMsâ code generation. https://doi.org/10.48550/arXiv.2311.00272. (2023) 21. Davis, D. & Milanova, M. Parsing Requirements for Automatic Prompting of Large Language Models for Requirements Validation. Intelligent Computing-Proceedings Of The Computing Conference. p. 1-19 (2025) 22. Niu, C., Zhang, T., Li, C., Luo, B. & Ng, V. On evaluating the efficiency of source code generated by llms. Proceedings Of The 2024 IEEE/ACM First In- ternational Conference On AI Foundation Models And Software Engineering. p. 103-107 (2024) 23. Dougherty, Q. & Mehta, R. Proving the coding interview: A benchmark for for- mally verified code generation. 2025 IEEE/ACM International Workshop On Large Language Models For Code (LLM4Code). p. 72-79 (2025) 24. Guimaraes, E., Nascimento, N., Shivalingaiah, C. & Nelapati, A. Analyzing promi- nent llms: An empirical study of performance and complexity in solving leetcode problems. https://doi.org/10.48550/arXiv.2508.03931. (2025) 25. Walder, C. & Karkhanis, D. Pass@K Policy Optimization: Solving Harder Re- inforcement Learning Problems. The Thirty-ninth Annual Conference On Neural Information Processing Systems. (2025) 26. Masood, A. Code Generation with LLMs: Practical Challenges, Gotchas, and Nuances. (2025), https://medium.com/@adnanmasood/code-generation-with- llms-practical-challenges-gotchas-and-nuances-7b51d394f588, [Online; posted 28- Feburary-2025] 27. Pawar, V., Gawande, M., Kollu, A. & Bile, A. Exploring the Potential of Prompt Engineering: A Comprehensive Analysis of Interacting with Large Language Mod- els. 2024 8th International Conference On Computing, Communication, Control And Automation (ICCUBEA). p. 1-9 (2024) 28. Uusn Ìakki, J., Ihantola, P. & Mikkonen, T. Exploring Prompt Engineering with Large Language Models in the Context of Software Maintenance. Intelligent Computing-Proceedings Of The Computing Conference. p. 166-177 (2025) 29. Guo, D. & Others DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. https://doi.org/10.48550/arXiv.2501.12948. (2025) Iterative Feedback Loops19 30. Siml, J. Is there an optimal temperature and top-p for code generation with paid LLM APIs?. (2023), https://forem.julialang.org/svilupp/is-there-an-optimal- temperature-and-top-p-for-code-generation-with-paid-llm-apis-15am 31. Lappi, V., Tirronen, V. & Itkonen, J. A replication study on the intuitiveness of programming language syntax. Software Quality Journal. 31, 1211-1240 (2023) 32. Vitousek,M.&Siek,J.Gradualtypinginanopenworld. https://doi.org/10.48550/arXiv.1610.08476. (2016) 33. Cass, S. The Top Programming Languages 2025. IEEE Spectrum. (2025,10), https://spectrum.ieee.org/top-programming-languages-2025 34. Zhang, L. & Kothari, S. Holistic Evaluation of State-of-the-Art LLMs for Code Generation. ArXiv E-prints. (2025) A Tables Table 6: pass@1 scores across different models and datasets. Dataset Language DeepSeek-V3 DeepSeek-R1 GPT-4.1-mini GPT-o4-mini Core dataset Python72.484.076.289.1 Java71.682.475.187.3 Strain dataset Python55.579.067.087.0 Java55.576.566.584.0 Challenge dataset Python9.40.06.331.3 Java12.50.09.428.1 Table 7: Iterative metrics results for DeepSeek-R1 on various temperature and top-p settings. top-p=0.3top-p=0.6top-p=0.9 ISR@10 MIS ISR@10 MIS ISR@10 MIS T=0.1 62.59.062.57.556.29.0 T=0.5 62.57.062.54.062.58.0 T=0.9 68.84.068.86.562.54.0 20Le Zhang et al. B Case Study Two representative cases are presented from this iterative refinement experiment to illustrate when LLMs successfully improve their solutions and when their capability remains limited. Table 8: Example of successful iterative refinement. iteration123 Result Wrong Answer Wrong Answer Accepted Testcase Coverage 98.3%99.1%100.0% B.1 Case #1 GPT-4o-miniâs improvement on the LeetCode problem âMinimum Cost to Equal- ize Arrayâ is particularly noteworthy. As shown in Table 8, although its initial attempt failed due to a wrong answer, the model effectively incorporated feed- back over the following two iterations, progressively increasing testcase coverage and ultimately solving the problem by iteration #3. 1 # Python code at Iteration #1 2 ... 3 candidates = base 4 if T0 >= base: 5 candidates.add(T0) 6 candidates.add(T0 + 1) 7 candidates.add(base + 1) 1 # Python code at Iteration #2 2 ... 3 n = len(nums) 4 candidates = base , base + 1 5 if n >= 3: 6 candidates.add(T0i) 7 candidates.add(T0i - 1) Fig. 10: GPT-o4-mini solutions comparison, iteration #1 vs iteration #2. In the first attempt, the failed testcase was nums = [55, 52, 29, 11], cost1 = 18, cost2 = 2, where the expected output was 118 but the model pro- duced 122. This error likely resulted from an unhandled corner case. As illus- trated in Figure 10, the initial solution evaluated only two candidates, base and (base + 1), and considered T0 and (T0 + 1) only when T0 â„ base. In contrast, the revised solution evaluated all four candidates whenever len(nums) â„ 3. In this failed testcase, where len(nums) = 4, the expanded candidate set in the second solution enhanced corner-case coverage and successfully resolved the issue. Iterative Feedback Loops21 1 # Python code at Iteration #2 2 ... 3 n = len(nums) 4 candidates = base , base + 1 5 if n >= 3: 6 candidates.add(T0i) 7 candidates.add(T0i - 1) 1 # Python code at Iteration #3 2 ... 3 candidates = base , base + 1, 4 T0 - 1, T0, T0 + 1 Fig. 11: GPT-o4-mini solutions comparison, iteration #2 vs iteration #3. Although the second solution passed one previously failed testcase, it en- countered another failure on nums = [60, 19, 53, 31, 57], cost1 = 60, cost2 = 2, where the expected output was 90 but the model produced 144. This error likely stemmed from insufficient candidate coverage. As noted earlier, the second solu- tion considered four candidates, base, (base + 1), T0i, and (T0i - 1), which proved inadequate for this scenario. Consequently, GPT-4o-mini further refined its approach, as shown in Figure 11. In this version, the model removed condi- tional checks and directly evaluated all five possible candidates, thereby expand- ing its search space and improving robustness against corner cases. This revision enabled the model to pass all testcases successfully. This case exemplifies the LLMâs ability to iteratively refine its reasoning and self-correct functional errors, demonstrating clear progress in problem under- standing and solution generalization through iterative feedback. B.2 Case #2 Table 9: Example of unsuccessful iterative refinement. Iteration 12345678910 Result TLE 1 TLE TLE TLE TLE TLE TLE TLE TLE TLE Testcase Coverage 99.56% 99.56% 99.56% 99.56% 99.85% 99.56% 99.56% 99.56% 99.56% 99.56% In this case study, we analyze the iterative refinement behavior of DeepSeek-R1 on the LeetCode problem âLongest Special Path Iâ. As summarized in Table 9, all iterations consistently resulted in Time Limit Exceeded (TLE) outcomes, indicating that none of the revisions successfully optimized the algorithm to meet the efficiency constraints of the problem. 1 TLE stands for Time Limit Exceeded error. 22Le Zhang et al. 1 # Java code at Iteration #1 2 ... 3 for (int i = savedGlobalLeft; 4 i < globalLeft; i++) 5 int node = path.get(i); 6 int v = nums[node]; 7 freq[v]++; 8 ... 9 1 # Java code at Iteration #6 2 ... 3 for (int value : state.removals) 4 freq[value ]++; 5 ... 6 Fig. 12: DeepSeek-R1 solutions comparison, iteration #1 vs iteration #6. If we examine the Java solutions generated by DeepSeek-R1 for this problem, we can observe in Figure 12 a notable globalLeft restoration logic at the end of the DFS method (iteratoin #1). This design has clear performance implications: â The path can be O(n) long in the worst case (e.g., a linear chain tree). â globalLeft may move O(n) positions during a single DFS call. â The restoration loop runs O(n) times for each DFS call. â DFS itself makes O(n) calls in total. Consequently, the worst-case time complexity of this algorithm is O(n 2 ), which is inefficient. In subsequent iterations, the model attempted to address this issue by adopt- ing the approach shown on the right-hand side of Figure 12 (iteration #6). How- ever, in the worst case (a degenerate tree resembling a linked list), each node may accumulate up to O(n) entries in its state.removals list. Processing these entries requires O(n) operations per node. For such a degenerate tree: â There are n nodes along the path. â Each node processes up to n removals. This results in a total complexity of O(n 2 ), which is no better than the previous solution. Despite repeated refinement attempts, the modelâs testcase coverage remained nearly constant (around 99.56%) across iterations. This suggests that DeepSeek- R1 made only superficial changes without addressing the core performance bot- tleneck. This example highlights a limitation in the modelâs ability to iteratively improve solutions when fundamental algorithmic efficiency issues are involved.