Paper deep dive
Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping
Yiwen Zhang, Xiaodong Yan, Zhenyu Huang, Deng Zhao, Liang Jiang, Qing Cui, Zujie Wen, Zhiqiang Zhang, Jun Zhou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/28/2026, 3:15:11 AM
Summary
The paper introduces RobustTests, a framework for Reinforcement Learning from Verifiable Rewards (RLVR) that improves LLM code generation by synthesizing high-quality test cases using faulty code and applying a dense reward function. The method filters invalid tests via validator agents and clusters them for diversity, resulting in a 3% absolute performance gain on LiveCodeBench when fine-tuning Qwen3-32B.
Entities (7)
Relation Signals (6)
RobustTests → evaluatedon → LiveCodeBench
confidence 95% · achieves a 3% absolute gain on LiveCodeBench
RobustTests → improves → Qwen3-32b
confidence 95% · RL fine-tuning of Qwen3-32B via RobustTests achieves a 3% absolute gain
RobustTests → uses → faulty-code-driven test case synthesis
confidence 95% · RobustTests, a framework featuring a faulty-code-driven test case synthesis strategy.
RobustTests → uses → dense reward function
confidence 95% · a stepwise dense reward function based on pass rates is introduced
RobustTests → augments → code_contests
confidence 90% · construct an augmented version of the CodeContests+ dataset
RobustTests → evaluatedon → CodeForces
confidence 90% · evaluated its properties... on the standardized CodeForces competitive problem set
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement Learning from Verifiable Rewards (RLVR) is pivotal for enhancing LLM code generation, yet its efficacy is often hindered by insufficient test case coverage, leading to reward hacking and policy degradation. To address this, we propose RobustTests, a framework featuring a faulty-code-driven test case synthesis strategy. By leveraging "near-correct" faulty codes, RobustTests captures latent logical discrepancies and employs validator agents with behavioral feature clustering to filter invalid or redundant test cases. Additionally, a stepwise dense reward function based on pass rates is introduced to mitigate false negatives and enhance training robustness. Using this pipeline, we construct an augmented version of the CodeContests+ dataset with superior diagnostic utility. Experimental results show that RL fine-tuning of Qwen3-32B via RobustTests achieves a 3% absolute gain on LiveCodeBench, demonstrating its effectiveness in advancing LLM code generation proficiency. Codes and data are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.24135v2
- Canonical: https://arxiv.org/abs/2608.24135v2
Trouble viewing inline? Open PDF directly →
Full Text
63,977 characters extracted from source content.
Expand or collapse full text
Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping Yiwen Zhang Affiliation: Zhejiang University Affiliation: Ant Group Correspondence: jun.zhoujun@antgroup.com Xiaodong Yan Affiliation: Ant Group Correspondence: jun.zhoujun@antgroup.com Zhenyu Huang Affiliation: Ant Group Correspondence: jun.zhoujun@antgroup.com Deng Zhao Affiliation: Ant Group Correspondence: jun.zhoujun@antgroup.com Liang Jiang Affiliation: Ant Group Correspondence: jun.zhoujun@antgroup.com Qing Cui Affiliation: Ant Group Correspondence: jun.zhoujun@antgroup.com Zujie Wen Affiliation: Ant Group Correspondence: jun.zhoujun@antgroup.com Zhiqiang Zhang Affiliation: Ant Group Correspondence: jun.zhoujun@antgroup.com Jun Zhou †thanks: Corresponding authors Affiliation: Ant Group Correspondence: jun.zhoujun@antgroup.com Abstract Reinforcement Learning from Verifiable Rewards (RLVR) is pivotal for enhancing LLM code generation, yet its efficacy is often hindered by insufficient test case coverage, leading to reward hacking and policy degradation. To address this, we propose RobustTests, a framework featuring a faulty-code-driven test case synthesis strategy. By leveraging "near-correct" faulty codes, RobustTests captures latent logical discrepancies and employs validator agents with behavioral feature clustering to filter invalid or redundant test cases. Additionally, a stepwise dense reward function based on pass rates is introduced to mitigate false negatives and enhance training robustness. Using this pipeline, we construct an augmented version of the CodeContests+ dataset with superior diagnostic utility. Experimental results show that RL fine-tuning of Qwen3-32B via RobustTests achieves a 3% absolute gain on LiveCodeBench, demonstrating its effectiveness in advancing LLM code generation proficiency. Codes and data are available at https://huggingface.co/datasets/sid6/RobustTests. 1 Introduction Figure 1: Illustration of current methods for test cases synthesis by LLM.(a) Generate Test cases Directly, instructing the LLM to directly synthesize a suite of test cases compliant with the problem specifications.(b) Generate Test cases by Generator Program, where LLM is first instructed to generate generator programs designed to automate the synthesis of test case inputs. Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a pivotal technique for enhancing the code generation capabilities of Large Language Models (LLMs) El-Kishky et al. (2025); Jiang et al. (2026); Hou et al. (2024). However, its efficacy is fundamentally limited by the comprehensiveness of test cases. Insufficient coverage often causes false positives Le et al. (2022), where faulty code passes sparse test cases, leading to reward hacking and subsequent policy degradationGuo et al. (2025). Consequently, synthesizing high-coverage test cases is essential to refine RL feedback and ensure sustained performance gains Ma et al. (2025); Lin et al. (2025). Nevertheless, the automated generation of high-quality test cases and their subsequent integration into a performance-boosting RLVR paradigm remain formidable challenges. Firstly, the lack of precise metrics to characterize test case quality obscures which data distributions are truly conducive to the RLVR process Liu et al. (2023). Secondly, the optimal methodology for integrating even high-quality test cases into RLVR frameworks remains under-explored Gunjal et al. (2025). Although existing attempts leverage LLMs for direct test case augmentation (e.g., via few-shot prompting with problem descriptions) Zeng et al. (2025); Xu et al. (2025); Li and Yuan (2024), as shown in Figure 1(a), the resulting outputs often fail to regard boundary condition coverage. Furthermore, test cases synthesized by this approach frequently exhibit hallucination instances that violate the underlying problem constraints. Adopting such erroneous verification signals as reward feedback in reinforcement learning induces significant reward bias, which misguides the optimization trajectory and constrains further enhancement of the model’s proficiency. Kwa et al. (2024). To enhance synthetic quality, subsequent efforts have transitioned to a generator program paradigm He et al. (2025), using various logical hypotheses to improve the coverage of test cases, as shown in Figure 1(b). While this approach reduces the hallucination rate, it remains heavily reliant on manual annotation, which restricts its scalability. Building on this, Wang et al. (2025) introduced verification agents to automatically craft validator programs for constraint verification, further suppressing the hallucination rate. However, the completeness of automated validators is difficult to formally guarantee Olausson et al. (2023). Their intrinsic residual hallucinations, particularly under binary sparse rewards, frequently induce false negatives Liu et al. (2023). This results in correct code generating misleading gradient signals due to erroneous labeling Ouyang et al. (2022). Consequently, within the context of RL reward signaling, simply improving data quality has hit a bottleneck. It is imperative to refine the learning mechanisms or introduce robust reward modeling Casper et al. (2023) to mitigate the deleterious effects of residual hallucinations and ensure stable model evolution under noisy feedback. To address these challenges, we propose RobustTests, which at its core incorporates faulty-code-driven test case synthesis and a robust dense reward mechanism. During the synthesis stage, we first utilize LLMs to generate "near-correct" faulty programs refined via original test cases to guide the directed synthesis of test cases capable of triggering specific logical defects. Subsequently, we perform validity verification on these generated test cases to eliminate invalid ones. We further conduct clustering and filtering based on behavior feature vectors (i.e., the pass/fail status encodings of test cases over different faulty code snippets), which ensures that the test case suite can cover diverse error scenarios and alleviate false positives. During the training stage, to address the false negatives caused by unavoidable hallucinatory noise in synthetic test case suit, we introduce a stepwise dense reward function based on pass rates. This improves the robustness of the model to continuously learn from partially correct signals. Within the RobustTests framework, we augmented the test cases of the CodeContests+ dataset to construct a code high-quality dataset, and evaluated its properties using Test case Space Polarization (TSP) metric. The results demonstrate that RobustTests achieves a 7% improvement in TSP relative to the original CodeContests+, indicating its enhanced capacity to uncover a broader spectrum of failure modes. In our experiments, using problems of moderate difficulty from CodeContests+ as the training set, RL fine-tuning of Qwen3-32B via RobustTests yields an absolute 3% performance gain on the LiveCodeBench benchmark compared to the baseline methods. These findings not only confirm the effectiveness of the RobustTests framework in bolstering the code generation proficiency of LLMs but also establish a clear correlation between the TSP metric and model performance. In summary, the main contributions of this study are as follows: • We propose a novel approach for automated high-quality test case synthesis and RLVR reward modeling, utilizing faulty-code-driven synthesis and a robust dense reward mechanism to expand test case coverage and bolster resilience against synthetic noise during RL. • We construct a code dataset with more diverse test cases, significantly strengthens diagnostic utility across various faulty code scenarios. • An abosulte 3-percentage point performance gain on the LiveCodeBench benchmark when training Qwen3-32B compared to baselines, substantially advancing the code generation performance of LLMs. 2 Related work 2.1 Test case synthesis Method At present, the most accurate method for test case synthesis remains manual curation by human experts. This methodology underpins the test cases of numerous code evaluation benchmarks, including MBPP Austin et al. (2021), HumanEval Chen et al. (2021), and LiveCodeBench Jain et al. (2024). However, it is prohibitively expensive and lacks scalability, rendering it suitable only for small-scale benchmark and impractical for the construction of massive training corpora. Consequently, several LLM-based automated methods have emerged. CodeContests+ Cai et al. (2026) generates supplementary test cases by applying stochastic perturbations to harvested inputs. EvalPlus Liu et al. (2023) extends HumanEval by prompting LLMs to synthesize seed inputs guided by reference implementations. Furthermore, frameworks such as KodCode Xu et al. (2025) and AceCoder Zeng et al. (2025) have expanded this scope to include the joint synthesis of coding problems, test cases, and reference solutions. The HardTests He et al. (2025) method improves upon these strategies by utilizing generator programs for test case synthesis. Building upon these, CodeContests+ Wang et al. (2025) incorporates automated validator programs to verify input constraints, effectively mitigating LLM-induced hallucinations. Nevertheless, the challenge of synthesizing test cases that provide comprehensive diagnostic coverage for diverse faulty code remains an unresolved research problem. 2.2 RL for Enhancing LLM’s Code Ability Reinforcement learning (RL) has been increasingly integrated into the coding domain, leveraging code executability to provide objective feedback Jiang et al. (2026); Shojaee et al. (2023); Dou et al. (2024). Early studies, such as CodeRL Le et al. (2022) and AlphaCode Li et al. (2022), explored the use of compiler feedback or unit tests as reward signals to optimize models via algorithms like PPO Schulman et al. (2017). Building upon existing methodologies, DeepSeek-R1 Guo et al. (2025) introduced the GRPO as the current state-of-the-art (SOTA) algorithm. However, while DeepSeek-R1 relies on curated public datasets, training with synthetic test cases necessitates a different approach. Since synthetic test cases inevitably harbor errors, it is imperative to design a robust reward function that enhances resilience toward test case inaccuracies and provides more tolerant feedback signals. 3 Method Figure 2: Overview of the proposed RobustTests framework. The framework consists of three core components: (i) Automated Test case Generation in 3.1, aims to generate high-quality test cases inputs capable of effectively distinguishing correct code from faulty codes; (i) Test case Validation and Selection in 3.2, ensures that the generated test cases are semantically sound and capable of diagnosing a broad spectrum of faulty code while simultaneously maximizing set parsimony; and (i) Dense Reward Function Design in 3.3, accounts for the potential fallibility of the synthesized test case suite. To address the reward bias issues encountered when leveraging RLVR to enhance the code generation capabilities of LLMs, we introduce the RobustTests framework, illustrated in Figure 2. 3.1 Automated Test case Generation Automated test case generation aims to generate high-quality test cases inputs capable of effectively distinguishing correct code from faulty codes. This process comprises two pivotal stages: the generation of a diverse pool of faulty code and the subsequent generation of directed, faulty-code-driven test cases. Representative samples of faulty code and generated test cases are detailed in Appendix D.3. Faulty Code Generation To establish concrete targets for test case generation, we first generate a diverse ensemble of faulty implementations. For each problem PiP_i, inspired by the approach in Yang et al. (2025b), we leverage an LLM to generate stochastic faulty code, producing a broad spectrum of potential logical defects f∼LLM(⋅|Pi)f (·|P_i) via sampling. Prompts we used are shown in Appendix F.1. To ensure the non-triviality of the candidate implementations, we employ a rigorous dynamic filtering mechanism. Each candidate e is executed against an original test case suite TbaseT_base, producing a binary execution vector Vif∈0,1|Tbase|V^f_i∈\0,1\^|T_base|, where each dimension denotes the pass/fail status of the corresponding test case. We retain only the faulty code that satisfies the following criterion: ϕ(f)≜0<‖Vif‖1|Tbase|<1φ(f) 0< \|V^f_i\|_1|T_base|<1 (1) The strategy filters out both fully correct and completely incorrect code while preserving only nearly correct faulty code. This ensures that the generated test cases are capable of capturing potential logical defects. To further refine the faulty codes, we perform exact deduplication on implementations with the same execution vector VifV^f_i. By selecting a single representative sample for each unique failure mode, we construct the candidate pool FbaseF_base, thereby eliminating semantic redundancy and ensuring that the pool encompasses diverse logical discrepancies. Faulty-Code-Driven Test case Generation Given the qualified faulty code pool FbaseF_base, the primary objective is to generate test cases that probe the nuanced semantic boundaries between correct logic and specific implementation pitfalls. We adopt a failure-inducing prompting strategy Mu et al. (2024), where batches of faulty codes are provided as negative examples within the prompt. As shown in Appendix F.2, The model is instructed to generate constraint-compliant test cases tit_i that satisfy two primary criteria: (i) strict adherence to the input specifications of PiP_i, ensuring that the generated outputs correspond correctly to the inputs; and (i) the capability to induce a logical failure in at least one faulty code implementation while remaining consistent with the correct execution of the reference solution. All generated test cases and test cases in TbaseT_base are subsequently incorporated into the candidate pool TcandT_cand. 3.2 Test case Validation and Selection To ensure that the generated test cases are semantically sound and capable of diagnosing a broad spectrum of faulty code while simultaneously maximizing set parsimony, we employ a three-stage filtering pipeline comprising input verification, LLM instruction-compliance validation, and diversity-driven selection. Input Validation LLM-generated test case inputs frequently suffer from a "semantic-execution gap," wherein they violate implicit domain axioms or problem-specific constraints. To bridge this gap, following the methodology in Wang et al. (2025), we implement an executable validator AiA_i for each problem PiP_i, with generation details provided in Appendix F.3. The suite of valid test cases is formally defined as: Tvalid=t∈Tcand∣Ai(t)=TrueT_valid=\t∈ T_cand A_i(t)=True\ (2) This process prunes inputs that are syntactically correct but semantically invalid, ensuring that all subsequent evaluations are grounded in feasible execution scenarios. The input validator and its corresponding invalid test case detections are detailed in Appendix D.4. LLM Instruction-compliance Validation Following the validator-based pruning of invalid test cases from the candidate pool, we subject each remaining instance to LLM instruction-compliance validation. First, the ground-truth output for each test case is re-synthesized by executing the reference code, thereby ensuring semantic alignment between inputs and outputs. Subsequently, each test case is evaluated against the faulty code pool to determine whether it can successfully trigger a logical failure in at least one faulty implementation. Only test cases that simultaneously yield correct outputs and demonstrate the capacity to expose code defects are incorporated into the final test case suite TfinalT_final. Details are in Appendix D.5. Diversity-Driven Selection To maximize the coverage of heterogeneous failure modes while maintaining test case suite parsimony, we implement a clustering mechanism based on the failure profiles of test cases against faulty code. We first construct a binary execution vector Vit∈0,1|Fbase|V^t_i∈\0,1\^|F_base| for each test case in TfinalT_final, then employ the K-means algorithm Ahmed et al. (2020) to partition the test case space into K disjoint clusters C1,…,CK\C_1,…,C_K\. Euclidean distance Rudin (1976) is utilized as the dissimilarity metric to quantify differences between execution vectors: minC1,…,CK∑k=1K∑Vit∈Ck‖Vit−k‖22 _\C_1,…,C_K\ _k=1^K _V^t_i∈ C_k \|V^t_i- μ_k \|_2^2 (3) where k=1|Ck|∑Vjt∈CkVjt μ_k= 1|C_k| _V^t_j∈ C_kV^t_j denotes the centroid of cluster CkC_k. For each resulting cluster CkC_k, a round-robin selection strategy is implemented to extract representative medoids. This strategy ensures that the final refined test case suite Tsynth.=t1,…,tKT_synth.=\t_1,…,t_K\ spans the maximum range of scenarios across diverse faulty codes, thereby enhancing the overall diagnostic utility. The pseudo code is presented in Appendix E. 3.3 Dense Reward Function Design Despite rigorous two-stage test case pruning, residual flaws may still persist in Tsynth.T_synth., such as undetected validator loopholes Liu et al. (2023) or the insufficient robustness of reference solutions Li et al. (2022). Examples are in Appendix D.7. Consequently, we formulate a robust reward mechanism that explicitly accounts for the potential fallibility of the synthesized test case suite. First, the suite of test cases successfully executed by a candidate solution s within Tsynth.T_synth. is defined as: pass(s,Tsynth.)=t∈Tsynth.∣exec(s,t)=passpass(s,T_synth.)=\t∈ T_synth. (s,t)=pass\ (4) The corresponding reward function r(s)r(s) is formulated as follows: r(s)=1.1if |pass(s,Tsynth.)|=1−0.1if |pass(s,Tsynth.)|=0110⋅|pass(s,Tsynth.)||Tsynth.|otherwiser(s)= cases1.1&if |pass(s,T_synth.)|=1\\ -0.1&if |pass(s,T_synth.)|=0\\ 110· |pass(s,T_synth.)||T_synth.|&otherwise cases (5) This reward strategy is designed to bolster the stability and efficacy of the reinforcement learning process via granular feedback signals. It not only enhances the model’s resilience to potential noise within the test cases but also fosters stable curriculum learning by providing a progressive optimization trajectory. 4 Experiment Method Livecodebench Codeforces Score Score Rating Percentile Naive LLM Generation 65.75 35.41 83.36 91.70 HardTests 65.25 35.65 83.58 91.35 CodeContests+ 65.41 35.56 83.96 91.45 CodeContests-O 66.21 36.43 84.35 91.81 RobustTests(Ours) 68.39 38.50 85.99 94.67 Table 1: Performance of baselines and RobustTests on the LiveCodeBench and CodeForces benchmarks. The best result is bold and the second result is underline. 4.1 Experimental Settings Datasets We utilize CodeContests+ Wang et al. (2025), a comprehensive benchmark dataset that aggregates 11,636 programming problems from CodeForces Mirzayanov et al. (2020), AIZU AOJ Programming Challenge (2018); AOJ New Site (2018), and AtCoder AtCoder Inc. (2012). For each problem, approximately 100 test cases are generated via a "generator-validator" multi-agent framework. To ensure a sufficiently challenging training environment, we perform difficulty-based filtering using Qwen3-32B Yang et al. (2025a). Specifically, we assess the model’s performance on ten trials for each problem, as illustrated in Figure 3. Only problems with a pass@10 score between 0.2 and 0.9 were retained, resulting in a refined training subset named CodeConteststrain+CodeContests^+_train of approximately 3.3k problems. Although CodeContests+ provides a substantial volume of test cases, we observe significant homogeneity between the original instances, which constrains the efficacy of reinforcement learning from verifiable rewards (RLVR). To address the limitation, we introduce RobustTests, a refined training set built on CodeContests+, with approximately 200 highly diverse test cases capable of effectively uncovering latent logical defects. RobustTests significantly reduces the false positive rate, thereby enhancing the robustness and generalization capabilities of the RLVR-based training. Figure 3: Correctness distribution of Qwen3-32B over ten trials across all CodeContests+ problems, evaluated using the CodeContests+ test case suite. The x-axis represents the distribution of pass rates over ten trials(pass@10), while the y-axis denotes the number of problems within each pass rate interval. Training Setup We finetune Qwen3-32B Yang et al. (2025a) via GRPO Guo et al. (2025) with configuration: βclip=0.2 _clip=0.2, Adam (η=5×10−6η=5× 10^-6, β1=0.9 _1=0.9, β2=0.95 _2=0.95), batch size 256, and 10-step linear warmup. Answers are sampled at maximal entropy (T=1.0T=1.0, ptop=1.0p_top=1.0) with nsample=8n_sample=8 per prompt. Input truncation at 2,048 tokens and output extension to 38,912 tokens leverage the model’s 128K context window. The λKL=0.0 _KL=0.0 setting intentionally omits policy regularization, reserving KL-loss for future constrained exploration while prioritizing boundary-case discovery in high-capacity regimes. The details are in Appendix A. Baseline We evaluated the efficacy of the RobustTests in comparison with existing test case augmentation strategies within the RLVR framework. These comparative baselines encompass Naive LLM Generation Li and Yuan (2024), HardTests He et al. (2025), and CodeContests+ Wang et al. (2025), CodeContests-O Cai et al. (2026). The introduction of baselines are in Appendix C. Evaluation Benchmarks We set LiveCodeBench (2024.08–2025.01) and CodeForces as the primary benchmarks to evaluate the performance of RobustTests. For LiveCodeBench, a periodically updated benchmark, we utilize the pass rate (Score) as the evaluation metric. For the standardized CodeForces competitive problem set, we establish a multi-dimensional evaluation framework: (i) Score, which quantifies problem-solving accuracy; (i) Rating, representing the model’s competitive standing on the leaderboard determined via simulated contest participation; and (i) Percentile, denoting the proportion of historical human contestants outperformed by the model. The evaluation settings are detailed in Appendix B. 4.2 Main Results HardTestsCodeContests+Naive LLM GenerationCodeContests-ORobustTests0.80.80.850.850.90.9TSP6565666667676868LiveCodeBench Figure 4: Relationship between TSP values and LiveCodeBench performance across different test cases. Method Livecodebench Codeforces Score Score Rating Percentile RobustTests*(Ours) 67.84 38.47 85.16 93.97 w/o Test case Synthesis 66.91 36.41 84.4 92.12 w/o Diversity-Driven Selection 66.82 36.23 84.12 91.96 w/o Both Module 65.41 35.56 83.96 91.45 Table 2: Ablation study on the test case synthesis strategy of RobustTests* on LiveCodeBench and CodeForces benchmarks. RobustTests* denotes a variant of the RobustTests framework that excludes the dense reward mechanism. We investigate the performance impact of without Test case Generation (the second row), without Diversity-Driven Selection (the third row) and without Both Module(the forth row). The best results are in bold. Figure 5: The pass rate distribution of Qwen3-32B over ten trials across all CodeConteststrain+CodeContests^+_train problems, evaluated against the test cases provided by RobustTests and without Test case Generation and its configuration without Diversity-Driven Selection. Figure 6: Correlation between the TSP values of test cases employed during training and the coding performance of LLM on the LiveCodeBench benchmark. Dataset Method LiveCodeBench Codeforces Score Score Rating Percentile CodeConteststrain+CodeContests^+_train Sparse Reward 65.41 83.96 35.56 91.45 Dense Reward 66.30 84.24 37.74 92.05 RobustTests* Sparse Reward 67.84 85.16 38.47 93.97 Dense Reward 68.39 85.99 38.50 94.67 RobustTests* w/o validator Sparse Reward 66.02 83.82 35.91 91.28 Dense Reward 67.21 85.02 38.15 93.64 Table 3: Ablation study on the Reward Module of RobustTests on LiveCodeBench and CodeForces benchmarks, using both RobustTests* and CodeContests+ as the training datasets. The best results are in bold. 00.050.100.20646466666868ScaleScoreLiveCodeBench00.050.100.20828284848686ScaleScoreCodeforces00.050.100.2036363838ScaleRatingCodeforces00.050.100.20909092929494ScalePercentileCodeforces Figure 7: Ablation study on the reward scale in the dense reward function on LiveCodeBench and Codeforces benchmarks. The scale is varied from 00 to 0.200.20 using RobustTests∗ as the training dataset. Performance on LiveCodeBench and CodeForces We reimplement HardTests and Naive LLM Generation methods to synthesize test cases for each problem in CodeConteststrain+CodeContests^+_train. To ensure a fair comparison, the number of test cases per problem is capped at approximately 40 across all methods. For HardTests, Naive LLM Generation, and CodeContests+, CodeContests-O baselines, we conduct reinforcement learning on Qwen3-32B using binary (0-1) sparse rewards. In contrast, the RobustTests method employs dense rewards for reinforcement learning on the same model. The results are shown in Table 1, revealing that while the three baselines exhibited comparable performance on LiveCodeBench and CodeForces, RobustTests outperform them by approximately absolute 3% on both benchmarks. This performance gain is twofold: First, the high-quality test cases in our training set more effectively detect logical discrepancies in answers, thereby reducing false positives. Second, the stepwise dense reward function provides intermediate feedback even when hallucinations of test cases are unavoidable. The design not only mitigates false negatives but also facilitates a curriculum learning effect that guides the model toward absolute correctness. Analysis of Test case Diversity To better understand why RobustTests improves downstream training, we analyze the diversity of different test cases. Drawing on Jaeger (2000), we introduce Test case Space Polarization (TSP), a low-cost metric that quantifies the diagnostic coverage of a test case suite across diverse faulty codes: TSP=1M⋅∑s∈S(−nsN⋅log2nsN) TSP= 1M· _s∈ S (- n_sN· _2 n_sN ) (6) Here, M denotes the number of faulty codes in FbaseF_base, N is the total number of test cases in Tsynth.T_synth., and S represents the set of test cases in Tsynth.T_synth.. The term nsn_s indicates the frequency of a specific test case s in Tsynth.T_synth.. As shown in Figure 4, RobustTests achieves the highest TSP value among all compared test cases and obtains the best LiveCodeBench performance. In contrast, methods with lower TSP values, such as HardTests and CodeContests+, lead to weaker downstream performance. This trend suggests that TSP is positively correlated with the training utility of test cases and can serve as a low-cost proxy for estimating dataset potential before expensive RL training. 4.3 Ablation Study In this section, we conduct an extensive ablation study to assess the individual contribution of each component within the test case synthesis strategy to the aggregate performance. Furthermore, we introduce a novel metric, Test case Space Polarization (TSP), designed to quantify test case diversity by measuring their diagnostic coverage across faulty codes, and evaluate its subsequent influence on model performance. Finally, we investigate the impact of various reward functions used during the training stage on the efficacy of the model. Test case Synthesis Module Ablation We first investigated the individual contributions of the Diversity-Driven Selection and Test case Synthesis stages to model performance, with results summarized in Table 2. In all experimental configurations, we fix the test case budget at approximately 40 and trained all models using binary (0-1) sparse rewards. The findings indicate that Both Test Case Synthesis and Diversity-Driven Selection yield consistent gains of approximately 1.5% on LiveCodeBench when applied independently and the integrated approach yields the best performance for Qwen3-32B, demonstrating a clear complementarity between the two stages: Test case Synthesis introduces high-quality test cases into the suite that are capable of effectively distinguishing correct codes from faulty code, while Diversity-Driven Selection prunes the suite to retain test cases that maximize coverage across diverse faulty code. We further analyze the role of test case diversity in the ablation study using the TSP metric introduced in Section 4.2. As shown in Figure 6, under identical training configurations, training Qwen3-32B with test cases that have higher TSP values leads to stronger LiveCodeBench performance. This indicates that higher diagnostic diversity improves the quality of reward signals by reducing the likelihood that faulty solutions are mistakenly accepted as correct. Furthermore, we study the relationship between TSP and the pass rate of model outputs under different test cases. Figure 5 shows the pass-rate distribution over ten trials on 3.3k problems from CodeConteststrain+CodeContests^+_train using Qwen3-32B. After removing Test Case Synthesis and Diversity-Driven Selection from RobustTests, TSP decreases and the distribution shifts rightward, indicating that lower-diversity test cases are less effective at detecting faulty solutions and therefore introduce more false positives in reward assignment. In contrast, higher-TSP test cases provide stricter diagnostic signals and better distinguish correct solutions from faulty ones. Interestingly, as TSP increases, the pass@10 of some problems reaches 1. Further analysis suggests that this is caused by remaining spurious synthetic test cases, which can falsely reject semantically correct solutions and create persistent false negatives. This motivates both validator filtering and dense rewards: the former removes invalid test cases, while the latter lets the model learn from partial execution feedback instead of noisy binary rewards. Reward Module Ablation Table 3 shows that dense rewards outperform sparse rewards across different training test cases, including RobustTests, CodeContests+, and RobustTests∗ without validator filtering. Dense rewards consistently improve LiveCodeBench performance with both RobustTests and CodeContests+, demonstrating the effectiveness and generalizability of our stepwise dense reward function. Under the no-validator RobustTests∗ setting, dense rewards yield even larger gains on both LiveCodeBench and Codeforces, indicating stronger robustness to noisy or invalid test cases. To explain this result, we audit the validator filtering pipeline on approximately 3.3k problems from CodeContests+train^+_train. Each problem contains about 200 raw generated test cases, around 30% of which are rejected as invalid, with no reference-execution failures observed. However, manual post-hoc inspection shows that about 10% of the accepted test cases remain invalid, causing false-negative judgments for roughly 10% of all test cases. These findings suggest that validator filtering removes many invalid test cases but cannot fully eliminate test noise. Thus, when validator filtering is weakened or removed, dense rewards provide a more robust training signal by leveraging partial execution feedback and reducing the impact of false-negative sparse rewards. We further examine reward-scale sensitivity in the dense reward function. As shown in Figure 7, setting the scale to zero causes a clear performance drop, confirming the necessity of this component. Performance remains stable as the scale varies from 0.050.05 to 0.200.20, suggesting that this hyperparameter has limited influence once enabled. 5 Conclusion We propose RobustTests, integrates faulty-code-driven test case synthesis and a stepwise dense reward mechanism to establish a robust RL framework for code generation, significantly enhancing the diagnostic utility of the augmented CodeContests+ dataset. By leveraging "near-correct" faulty codes to expand diagnostic coverage, the framework effectively minimizes false positives, while the stepwise dense reward mitigates false negatives by enabling the model to learn from partially correct signals through a curriculum learning paradigm. Beyond code generation, these principles offer broad applicability: the synthesis strategy can be adapted for automated mutation testing in software engineering, and the dense reward mechanism is particularly suited for experimental planning and conclusion analysis in autonomous scientific discovery. By bridging these capabilities, RobustTests provides a versatile trajectory for enhancing reasoning and planning within complex scientific and engineering contexts. Limitations Although RobustTests successfully strengthens the coding generation abilities of LLMs, certain limitations persist that we aim to mitigate in subsequent work: • Dependency on solutions: The proposed approach is primarily applicable to programming tasks with available ground-truth solutions; consequently, its utility is constrained in real-world scenarios where reference implementations are absent. • Expansion of domain generalization: While the proposed framework is evaluated on competitive programming benchmarks such as LiveCodeBench and CodeForces, its generalizability to a broader range of software development tasks remains to be fully explored. Ethical Considerations The Use of AI Assistants We employed Gemini-3 to assist us in polishing our paper and coding. Acknowledgments This work was supported by Ant Group Research Intern Program. References Ahmed et al. (2020) M. Ahmed, R. Seraj, and S. M. S. Islam The k-means algorithm: a comprehensive survey and performance evaluation. Electronics 9 (8), p. 1295. Cited by: §3.2. AOJ New Site (2018) Aizu online judge (new site). Note: http://onlinejudge.u-aizu.ac.jp/homeAccessed: 23-Apr-2018 Cited by: §4.1. AOJ Programming Challenge (2018) Aizu online judge: programming challenge. Note: http://judge.u-aizu.ac.jp/onlinejudge/Accessed: 23 Apr. 2018 Cited by: §4.1. AtCoder Inc. (2012) AtCoder Inc. AtCoder: programming contest website. Note: https://atcoder.jp/ Cited by: §4.1. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §2.1. Cai et al. (2026) J. Cai, J. Zhu, R. Sun, K. Zhao, D. Xue, M. Feng, W. Zhou, and H. Li CodeContests-o: powering llms via feedback-driven iterative test case generation. arXiv preprint arXiv:2601.13682. Cited by: Appendix C, §2.1, §4.1. Casper et al. (2023) S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217. Cited by: §1. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §2.1. Dou et al. (2024) S. Dou, Y. Liu, H. Jia, E. Zhou, L. Xiong, J. Shan, C. Huang, X. Wang, X. Fan, Z. Xi, et al. Stepcoder: improving code generation with reinforcement learning from compiler feedback. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 4571–4585. Cited by: §2.2. El-Kishky et al. (2025) A. El-Kishky, A. Wei, A. Saraiva, B. Minaiev, D. Selsam, D. Dohan, F. Song, H. Lightman, I. Clavera, J. Pachocki, et al. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.06807. Cited by: §1. Gunjal et al. (2025) A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. Hendryx Rubrics as rewards: reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746. Cited by: §1. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §2.2, §4.1. He et al. (2025) Z. He, Y. M. Choi, K. Zhang, J. Ji, J. Zhou, D. Xu, I. Bercovich, A. Zhang, and L. Li Hardtests: synthesizing high-quality test cases for llm coding. arXiv preprint arXiv:2505.24098. Cited by: Appendix C, §1, §2.1, §4.1. Hou et al. (2024) X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang Large language models for software engineering: a systematic literature review. ACM Transactions on Software Engineering and Methodology 33 (8), p. 1–79. Cited by: §1. Jaeger (2000) J. A. Jaeger Landscape division, splitting index, and effective mesh size: new measures of landscape fragmentation. Landscape ecology 15 (2), p. 115–130. Cited by: §4.2. Jain et al. (2024) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica Livecodebench: holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974. Cited by: §2.1. Jiang et al. (2026) J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology 35 (2), p. 1–72. Cited by: §1, §2.2. Kwa et al. (2024) T. Kwa, D. Thomas, and A. Garriga-Alonso Catastrophic goodhart: regularizing rlhf with kl divergence does not mitigate heavy-tailed reward misspecification. Advances in Neural Information Processing Systems 37, p. 14608–14633. Cited by: §1. Le et al. (2022) H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi Coderl: mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems 35, p. 21314–21328. Cited by: §1, §2.2. Li and Yuan (2024) K. Li and Y. Yuan Large language models as test case generators: performance evaluation and enhancement. arXiv preprint arXiv:2404.13340. Cited by: Appendix C, §1, §4.1. Li et al. (2022) Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science 378 (6624), p. 1092–1097. Cited by: §2.2, §3.3. Lin et al. (2025) Z. Lin, S. Shen, J. Shang, J. Weston, and Y. Nie Learning to solve and verify: a self-play framework for code and test generation. arXiv preprint arXiv:2502.14948. Cited by: §1. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems 36, p. 21558–21572. Cited by: §1, §1, §2.1, §3.3. Ma et al. (2025) Z. Ma, X. Zhang, J. Zhang, J. Yu, S. Luo, and J. Tang Dynamic scaling of unit tests for code reward modeling. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6917–6935. Cited by: §1. Mirzayanov et al. (2020) M. Mirzayanov, O. Pavlova, P. Mavrin, R. Melnikov, A. Plotnikov, V. Parfenov, and A. Stankevich Codeforces as an educational platform for learning programming in digitalization. Olympiads in Informatics 14 (133-142), p. 14. Cited by: §4.1. Mu et al. (2024) L. Mu, W. Zhang, Y. Zhang, and P. Jin Ddprompt: differential diversity prompting in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 168–174. Cited by: §3.1. Olausson et al. (2023) T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama Is self-repair a silver bullet for code generation?. arXiv preprint arXiv:2306.09896. Cited by: §1. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1. Peng et al. (2023) B. Peng, J. Quesnelle, H. Fan, and E. Shippole Yarn: efficient context window extension of large language models. arXiv preprint arXiv:2309.00071. Cited by: Appendix B. Rudin (1976) W. Rudin Principles of mathematical analysis. McGraw-Hill, New York. Cited by: §3.2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2.2. Shojaee et al. (2023) P. Shojaee, A. Jain, S. Tipirneni, and C. K. Reddy Execution-based code generation using deep reinforcement learning. arXiv preprint arXiv:2301.13816. Cited by: §2.2. Team et al. (2025a) K. Team, Y. Bai, Y. Bao, G. Chen, J. Chen, N. Chen, R. Chen, Y. Chen, Y. Chen, Y. Chen, et al. Kimi k2: open agentic intelligence. arXiv preprint arXiv:2507.20534. Cited by: Appendix F. Team et al. (2025b) L. Team, A. Shen, B. Li, B. Hu, B. Jing, C. Chen, C. Huang, C. Zhang, C. Yang, C. Lin, et al. Every step evolves: scaling reinforcement learning for trillion-scale thinking model. arXiv preprint arXiv:2510.18855. Cited by: Appendix B. Wang et al. (2025) Z. Wang, S. Liu, Y. Sun, H. Li, and K. Shen Codecontests+: high-quality test case generation for competitive programming. arXiv preprint arXiv:2506.05817. Cited by: Appendix C, §D.4, §1, §2.1, §3.2, §4.1, §4.1. Xu et al. (2025) Z. Xu, Y. Liu, Y. Yin, M. Zhou, and R. Poovendran Kodcode: a diverse, challenging, and verifiable synthetic dataset for coding. In Findings of the Association for Computational Linguistics: ACL 2025, p. 6980–7008. Cited by: §1, §2.1. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.1, §4.1. Yang et al. (2025b) J. Yang, K. Lieret, C. E. Jimenez, A. Wettig, K. Khandpur, Y. Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang Swe-smith: scaling data for software engineering agents. arXiv preprint arXiv:2504.21798. Cited by: §3.1. Zeng et al. (2025) H. Zeng, D. Jiang, H. Wang, P. Nie, X. Chen, and W. Chen Acecoder: acing coder rl via automated test-case synthesis. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 12023–12040. Cited by: §1, §2.1. Appendix A Training Settings We conduct our experiments on 32 NVIDIA H200 GPUs, employing Qwen3-32B as the base model. The detailed experimental parameters are summarized in Table 4. Parameter Value Optimizer AdamW Learning Rate (η) 5×10−65× 10^-6 Adam β1,β2 _1, _2 0.9,0.950.9,0.95 Weight Decay 0.010.01 Gradient Clipping 1.01.0 Global Batch Size 256256 Warmup Steps 1010 Learning Rate Scheduler Linear nsamplen_sample 88 Clipping Range (βclip _clip) 0.20.2 KL Coefficient (λKL _KL) 0.00.0 Temperature (T) 1.01.0 Top-p 1.01.0 Max Input Tokens 2,0482,048 Max Output Tokens 38,91238,912 Numerical Precision BF16 Table 4: Hyperparameters for Experiment Appendix B Evaluation Settings Following the evaluation protocol established in Team et al. (2025b), we assess a standardized pipeline to ensure a rigorous and fair comparison. For benchmarks including LiveCodeBench(2024.08–2025.01) and CodeForces, evaluations are conducted using a 128K context window. For the Qwen3-32B model with limited native context length, we leverage YaRN Peng et al. (2023) for context extension and strictly adhere to the official hyperparameters specified for its open-weight release. Appendix C Introduction of Baselines Naive LLM Generation Naive LLM Generation Li and Yuan (2024) is an approach that directly generates test cases. Specifically, it first prompts the LLM to directly synthesize a test case suite that conforms to the problem specifications, and then validates and revises the generated test cases with the reference solution. HardTests HardTests He et al. (2025) is an approach that generates test cases via dedicated generator programs. Specifically, the LLM is first prompted to produce generator programs that can automatically synthesize test case inputs, and the corresponding outputs are then obtained using the reference solution. CodeContests+ CodeContests+ Wang et al. (2025) is an approach that generates test cases via a "generator-validator" multi-agent framework. It first produces test case inputs using programs generated by the LLM, and then validates these inputs with an input validator also generated by the LLM. The corresponding outputs are subsequently obtained using the reference solution. CodeContests-O CodeContests-O Cai et al. (2026) is a Feedback-Driven Iterative Framework that transforms test case synthesis from open-loop generation into a closed-loop process by utilizing execution feedback from both correct and incorrect solutions to refine test cases for high fidelity and discriminability. Appendix D Case Study In this section, we utilize a representative problem from CodeContests+ as a case study to provide a detailed exposition of the operational mechanics and the underlying necessity of each component in the test case synthesis process, while further elucidating the root causes behind the generation of invalid test cases. D.1 Problem We identify problem p03520 from the CodeContests+ dataset. The detailed problem statement is presented as follows: ⬇ Snuke found a record of a tree with N vertices in ancient ruins. The findings are as follows: * The vertices of the tree were numbered 1,2,...,N, and the edges were numbered 1,2,...,N-1. * Edge i connected Vertex a_i and b_i. * The length of each edge was an integer between 1 and 10^18 (inclusive). * The sum of the shortest distances from Vertex i to Vertex 1,...,N was s_i. From the information above, restore the length of each edge. The input guarantees that it is possible to determine the lengths of the edges consistently with the record. Furthermore, it can be proved that the length of each edge is uniquely determined in such a case. Constraints * 2 ≤ N ≤ 10^5 * 1 ≤ a_i,b_i ≤ N * 1 ≤ s_i ≤ 10^18 * The given graph is a tree. * All input values are integers. * It is possible to consistently restore the lengths of the edges. * In the restored graph, the length of each edge is an integer between 1 and 10^18 (inclusive). Input Input is given from Standard Input in the following format: N a_1 b_1 : a_N-1 b_N-1 s_1 s_2 ... s_N Output Print N-1 lines. The i-th line must contain the length of Edge i. D.2 Original Test cases The base test suite for this problem comprises 177 test cases, denoted as |Tbase|=177|T_base|=177. An illustrative example is provided below: ⬇ Input 5 1 2 1 3 1 4 1 5 10 13 16 19 22 Output 1 2 3 4 D.3 Faulty Codes and Generated Test cases As described in Section 3.1, we generate a collection of faulty codes to guide the synthesis of discriminative test cases. The total number of qualified faulty implementations for the problem is |Fbase|=12|F_base|=12, with an example provided as follows: ⬇ import sys input=sys.stdin.readline def int_lst(): return list(map(int,input().split())) N=int(input()) adj=[[]for _ in range(N)] edges=[] for idx in range(N-1): a,b=int_lst() a-=1;b-=1 edges.append((a,b,idx)) adj[a].append((b,idx)) adj[b].append((a,idx)) s=int_lst() sz=[0]*N def dfs(v,p): cnt=1 for u,idx in adj[v]: if u!=p: cnt+=dfs(u,v) sz[v]=cnt return cnt dfs(0,-1) ans=[0]*(N-1) for a,b,i in edges: if sz[a]<sz[b]: ans[i]=(s[b]-s[a])//(sz[a]-sz[b]) else: ans[i]=(s[a]-s[b])//(sz[b]-sz[a]) for val in ans:print(abs(val)) Subsequently, we leverage the synthesized faulty code to guide the LLM in generating test cases capable of discriminating between correct and faulty implementations. The process yields a total of 75 test cases, a representative example of which is provided below: ⬇ Input 6 1 2 2 3 3 4 4 5 5 6 15 12 9 9 12 15 Output 3 2 1 2 3 The outputs of these test cases are initially placeholder values and require calibration against the reference solution. Since TcandT_cand is constructed by merging the synthesized test cases with TbaseT_base, the resulting candidate set size is |Tcand|=252|T_cand|=252. D.4 Input Validator Drawing on the approach of Wang et al. (2025), an input validator is implemented for this problem as follows: ⬇ #include "testlib.h" #include <bits/stdc++.h> using namespace std; const int MAXN = 100000; const long long MAXSI = 1000000000000000000L; int parent[MAXN + 5]; int find(int x) if (parent[x] != x) parent[x] = find(parent[x]); return parent[x]; void unite(int x, int y) x = find(x); y = find(y); if (x != y) parent[x] = y; int main(int argc, char* argv[]) registerValidation(argc, argv); int n = inf.readInt(2, 100000); inf.readEoln(); for (int i = 1; i <= n; ++i) parent[i] = i; set<pair<int, int>> edges; for(int i = 0; i < n - 1; i++) int a = inf.readInt(1, n); inf.readSpace(); int b = inf.readInt(1, n); inf.readEoln(); ensuref(a != b, "Self-loop detected at edge %d", i+1); int u = min(a, b); int v = max(a, b); ensuref(edges.count(u, v) == 0, "Multiple edges between %d and %d", u, v); edges.insert(u, v); ensuref(find(a) != find(b), "Cycle detected while adding edge between %d and %d", a, b); unite(a, b); // Check connectedness int root = find(1); for(int i = 2; i <= n; i++) ensuref(find(i) == root, "Graph is not connected, node %d is in different component", i); // Read s_i vector<long long> s = inf.readLongs(n, 1, MAXSI); inf.readEoln(); inf.readEof(); return 0; The validator flags two test cases for failing to satisfy the problem constraints, one of which is illustrated as follows. ⬇ Input 8 1 2 2 3 3 4 4 5 5 6 6 7 7 8 28 24 20 16 12 8 4 0 Output 4 4 4 4 4 4 4 This test case includes an integer value of 0, thereby violating the problem’s defined input range of [1,1018][1,10^18]. D.5 LLM Instruction-compliance Validation To ensure the validity of the LLM-synthesized test cases, we implement a verification and refinement process. First, the test case outputs are calibrated using the reference solution; the updated version of the case mentioned in Appendix D.2 is illustrated as follows: ⬇ Input 6 1 2 2 3 3 4 4 5 5 6 15 12 9 9 12 15 Output 3 1 3 1 3 On the other hand, we prune non-discriminative test cases that are passed by the faulty implementations. In this instance, two test cases were filtered out, resulting in a final test suite of |Tfinal|=248|T_final|=248 after the two-stage refinement process. D.6 Diversity-Driven Selection In accordance with our experimental requirements, we select 40 representative test cases such that |Tsynth.|=40|T_synth.|=40. The formal selection process is outlined in Appendix E. D.7 Limitations of the Synthesized Test cases Due to the absence of a check for the existence of valid integer side lengths corresponding to array s, several test cases in Tsynth.T_synth. are falsely accepted by the input validator. An example of the hallucination test cases is illustrated as follows: ⬇ Input 8 1 2 2 3 3 4 4 5 5 6 6 7 7 8 32 28 22 16 16 22 28 32 Output 0 1 3 0 3 1 0 Algorithm 1 Diversity-Driven Test case Selection 0: Initial test case suite TfinalT_final, target cluster count K, maximum budget M; 0: Refined test case suite Tsynth.T_synth.; 1: for each test case ti∈Tfinalt_i∈ T_final do 2: Construct binary execution vector Vit∈0,1|Fbase|V^t_i∈\0,1\^|F_base| based on failure profiles; 3: end for 4: X←V1t,V2t,…,VntX←\V^t_1,V^t_2,…,V^t_n\; 5: X←Standardize(X)X (X); 6: K′←min(K,|Tfinal|)K ← (K,|T_final|); 7: ℒ,←K-Means(X,n_clusters=K′)\L, μ\ -Means(X,n\_clusters=K ); 8: Partition TfinalT_final into clusters C1,C2,…,CK′\C_1,C_2,…,C_K \ based on ℒL; 9: for each cluster k∈1,…,K′k∈\1,…,K \ do 10: for each test case tk,i∈Ckt_k,i∈ C_k do 11: Compute dissimilarity to centroid: δk,i=‖Vk,it−k‖2 _k,i=\|V^t_k,i- μ_k\|_2; 12: end for 13: Sort CkC_k in ascending order of δk,i _k,i (closest to centroid first); 14: end for 15: Tsynth.←∅T_synth.← , j←0j← 0; 16: while |Tsynth.|<M|T_synth.|<M and |Tsynth.|<|Tfinal||T_synth.|<|T_final| do 17: for k=1k=1 to K′K do 18: if j<|Ck|j<|C_k| then 19: Tsynth.←Tsynth.∪Ck[j]T_synth.← T_synth.∪\C_k[j]\; 20: if |Tsynth.|=M|T_synth.|=M then 21: break; 22: end if 23: end if 24: end for 25: j←j+1j← j+1; 26: end while 27: return Tsynth.T_synth. The rejection of correct answers from Qwen3-32B by this test case leads to false negatives. To address this, we employ a stepwise dense reward function to enhance the model’s robustness against imperfect test cases. Appendix E Algorithm definition In Section 3.2, we employ clustering techniques to filter test cases for enhanced diversity. A pseudo code of it is provided as follows in Algorithm 1. Appendix F Implementation Details This section details the specific prompt templates employed in three stages described in Section 3, including Faulty Code Generation, Faulty-Code-Driven Test Case Generation and Input Validation Generation, with Kimi-k2 Team et al. (2025a) serving as the underlying foundation model. F.1 Faulty Code Generation For each problem, we generate faulty code through a multi-prompt approach. As illustrated in Figures 8, 9 and 10, we incorporate three distinct prompts, each of which is executed for three independent sampling passes. This results in a total of nine model invocations to ensure high intra-class diversity among the generated faulty implementations. F.2 Faulty-Code-Driven Test Case Generation For each problem, we generate a set of discriminative test cases capable of differentiating between correct and faulty codes. The prompt is illustrated in Figure 11. F.3 Input Validator Generation For each problem, we construct input validators designed to ensure that the synthesized test case inputs strictly adhere to the problem specifications. The prompt is illustrated in Figure 12. Prompt: Generate faulty codes that fail on edge cases System message: Your input fields are: quesition: $title + $description solution: $solution-code Your output fields are: 1. ‘reasoning‘ (str): 2. ‘faulty codes‘ (str): Generate a **diverse and comprehensive** set of faulty codes to distinguish high-quality test cases. All interactions will be structured in the following way, with the appropriate values filled in. [[ ## question ## ]] [[ ## solution ## ]] [[ ## reasoning ## ]] [[ ## faulty codes ## ]] [[ ## completed ## ]] In adhering to this structure, your objective is: Given the fields ‘quesition‘ and ‘solution‘, produce the fields ‘faulty code‘ as many as possible, don’t produce the repeat ‘faulty code‘. faulty codes should satisfy: (i) The code language is Python 3. (i) The code should pass a subset of the given test cases (e.g., simple/typical cases) but fail on edge cases. (i) The faults should be subtle (e.g., integer overflow, boundary condition omission, incorrect loop termination, type confusion, unhandled exceptional cases), avoiding obvious syntax errors. (iv) The code structure should remain reasonable, avoiding "obviously wrong" patterns (e.g., deliberate division by zero). The format of faulty codes should satisfy: please provide the faulty codes in JSON-like format(a list of dictionaries): “‘json ["code": <string>] “‘ Ensure the faulty code match the types and structure expected for the question. Do not include any additional text or explanations, just the JSON-like object. User message: [[ ## question ## ]] $title + $description Respond with the corresponding output fields, starting with the field ‘[[ ## reasoning ## ]]‘, then ‘[[ ## faulty codes ## ]]‘, and then ending with the marker for ‘[[ ## completed ## ]]‘. Figure 8: One of prompts on Faulty Code Generation Stage Prompt: Generate faulty codes that fail on hidden logical paths System message: Your input fields are: quesition: $title + $description solution: $solution-code Your output fields are: 1. ‘reasoning‘ (str): 2. ‘faulty codes‘ (str): Generate a **diverse and comprehensive** set of faulty codes to distinguish high-quality test cases. All interactions will be structured in the following way, with the appropriate values filled in. [[ ## question ## ]] [[ ## solution ## ]] [[ ## reasoning ## ]] [[ ## faulty codes ## ]] [[ ## completed ## ]] In adhering to this structure, your objective is: Given the fields ‘quesition‘ and ‘solution‘, produce the fields ‘faulty code‘ as many as possible, don’t produce the repeat ‘faulty code‘. faulty codes should satisfy: (i) The code language is Python 3. (i) The code should pass a subset of the given test cases (e.g., simple/typical cases) but fail on hidden logical paths. (i) The faults should be subtle (e.g., integer overflow, boundary condition omission, incorrect loop termination, type confusion, unhandled exceptional cases), avoiding obvious syntax errors. (iv) The code structure should remain reasonable, avoiding "obviously wrong" patterns (e.g., deliberate division by zero). The format of faulty codes should satisfy: please provide the faulty codes in JSON-like format(a list of dictionaries): “‘json ["code": <string>] “‘ Ensure the faulty code match the types and structure expected for the question. Do not include any additional text or explanations, just the JSON-like object. User message: [[ ## question ## ]] $title + $description Respond with the corresponding output fields, starting with the field ‘[[ ## reasoning ## ]]‘, then ‘[[ ## faulty codes ## ]]‘, and then ending with the marker for ‘[[ ## completed ## ]]‘. Figure 9: One of prompts on Faulty Code Generation Stage Prompt: Generate faulty codes that fail on special inputs System message: Your input fields are: quesition: $title + $description solution: $solution-code Your output fields are: 1. ‘reasoning‘ (str): 2. ‘faulty codes‘ (str): Generate a **diverse and comprehensive** set of faulty codes to distinguish high-quality test cases. All interactions will be structured in the following way, with the appropriate values filled in. [[ ## question ## ]] [[ ## solution ## ]] [[ ## reasoning ## ]] [[ ## faulty codes ## ]] [[ ## completed ## ]] In adhering to this structure, your objective is: Given the fields ‘quesition‘ and ‘solution‘, produce the fields ‘faulty code‘ as many as possible, don’t produce the repeat ‘faulty code‘. faulty codes should satisfy: (i) The code language is Python 3. (i) The code should pass a subset of the given test cases (e.g., simple/typical cases) but fail on special inputs. (i) The faults should be subtle (e.g., integer overflow, boundary condition omission, incorrect loop termination, type confusion, unhandled exceptional cases), avoiding obvious syntax errors. (iv) The code structure should remain reasonable, avoiding "obviously wrong" patterns (e.g., deliberate division by zero). The format of faulty codes should satisfy: please provide the faulty codes in JSON-like format(a list of dictionaries): “‘json ["code": <string>] “‘ Ensure the faulty code match the types and structure expected for the question. Do not include any additional text or explanations, just the JSON-like object. User message: [[ ## question ## ]] $title + $description Respond with the corresponding output fields, starting with the field ‘[[ ## reasoning ## ]]‘, then ‘[[ ## faulty codes ## ]]‘, and then ending with the marker for ‘[[ ## completed ## ]]‘. Figure 10: One of prompts on Faulty Code Generation Stage Prompt: Generate test cases that can differentiate between correct code and faulty code System message: Your input fields are: quesition: $title + $description solution: $solution_code faulty_code: $faulty_code Your output fields are: 1. ‘reasoning‘ (str): 2. ‘test cases‘ (str): Generate a diverse and comprehensive set of test cases that can pass the solution but fail on the faulty code. All interactions will be structured in the following way, with the appropriate values filled in. [[ ## question ## ]] [[ ## solution ## ]] [[ ## faulty_code ## ]] [[ ## reasoning ## ]] [[ ## test cases ## ]] [[ ## completed ## ]] In adhering to this structure, your objective is: Given the fields ‘question‘, ‘solution‘ and ‘faulty_code‘, produce the fields ‘test cases‘ as many as possible. The ‘test cases‘ must satisfy the requirement that they can pass the solution but fail on the faulty code. This is a mandatory requirement. Please do not generate any ‘test cases‘ that do not meet this criterion. The format of test case should satisfy: please provide the input and output in JSON-like format(a list of dictionaries): “‘json ["input": <string>, "output": <string>] “‘ Ensure the input and output match the types and structure expected for the problem. Do not include any additional text or explanations, just the JSON-like object. User message: [[ ## question ## ]] $title + $description [[ ## solution ## ]] $solution_code [[ ## faulty_code ## ]] $faulty_code Respond with the corresponding output fields, starting with the field ‘[[ ## reasoning ## ]]‘, then ‘[[ ## test cases ## ]]‘, and then ending with the marker for ‘[[ ## completed ## ]]‘. Figure 11: Prompt on Faulty-Code-Driven Test Case Generation Stage Prompt: Generate the input validator System message: Your input fields are: question: $title + $description. Your output fields are: reasoning (str): Explain how to implement a robust validator using testlib.h, including the validation of basic types, ranges, whitespace, and complex geometric constraints (e.g., simple polygon check, no duplicate points). validator_cpp (str): The complete, production-ready C++ source code for the input validator. All interactions must be structured exactly as follows: [[ ## question ## ]] [[ ## reasoning ## ]] [[ ## validator_cpp ## ]] [[ ## completed ## ]] Objective: Given the question, produce the validator_cpp using the testlib.h library. The validator must satisfy the following mandatory requirements: Strict Formatting: Use registerValidation(argc, argv), inf.readInt(), inf.readSpace(), inf.readEoln(), and inf.readEof(). Constraint Verification: Ensure all variables (e.g., N, coordinates) are within the specified bounds. The format of test case should satisfy: please provide in JSON-like format(a list of dictionaries): “‘json ["validator_cpp": <string>] “‘ Ensure the validator_cpp match the types and structure expected for the problem. Do not include any additional text or explanations, just the JSON-like object. User Message: [[ ## question ## ]] $title + $description Instructions for Response: Respond with the corresponding output fields, starting with [[ ## reasoning ## ]], followed by [[ ## validator_cpp ## ]], and ending with the marker [[ ## completed ## ]]. Figure 12: Prompt on Input Validator Generation Stage