Paper deep dive
Questioning the Questions: Sustaining Self-Evolution in Reasoning Models
Jinyuan Li, Chengsong Huang, Langlin Huang, Donghong Cai, Shiping Gao, Yuyi Yang, Jiaxin Huang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/6/2026, 11:14:25 AM
Summary
The paper introduces R-Quest, a method to sustain self-evolution in reasoning models by addressing two key failure modes: invalid questions and repetitive mathematical tasks. R-Quest employs a validity-aware solver to reject invalid questions and a frozen base model to assess novelty via pairwise comparisons, preventing performance collapse over multiple training rounds. It outperforms baselines like R-Zero on 12 benchmarks across math, reasoning, and code generation.
Entities (15)
Relation Signals (9)
R-Quest â evaluatedon â Qwen3-4B-Base
confidence 95% ¡ Empirically, our method consistently achieves the highest average performance... across two model families... Qwen3-4B-Base
R-Quest â evaluatedon â OctoThinker-3B-Hybrid-Base
confidence 95% ¡ Empirically, our method consistently achieves the highest average performance... across two model families... OctoThinker-3B-Hybrid-Base
R-Quest â outperforms â R-Zero
confidence 95% ¡ R-Quest maintains stable performance gains over ten rounds of self-evolution... outperforming R-Zero by 17.32 points.
R-Quest â uses â validity feedback
confidence 95% ¡ R-Quest introduces two forms of feedback... (1) validity feedback from a solver initialized to recognize and reject invalid questions
R-Quest â uses â novelty feedback
confidence 95% ¡ (2) novelty feedback from the frozen base model, which compares sampled question pairs to identify repeated tasks.
BLEU clustering â failstodetect â mathematically equivalent questions
confidence 90% ¡ existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways
R-Quest â improvesupon â Self-evolution
confidence 90% ¡ sustaining self-evolution in reasoning models
R-Zero â â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. In this paper, we investigate why performance deteriorates over successive rounds and how to sustain self-evolution. Our analysis identifies two recurring quality problems in self-generated questions: invalid questions and repeated variants of the same mathematical questions. First, invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in training data. Second, existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways, which leads to question diversity collapse in later training rounds. Building on these findings, we introduce R-Quest, which uses question validity and novelty feedback to guide self-evolution. We first train the solver to recognize and reject invalid questions, then use its judgments to guide questioner rewards and filter solver training data. To avoid question repetition, we use a frozen base model to compare sampled question pairs and provide novelty feedback. Empirically, our method consistently achieves the highest average performance on 12 benchmarks in mathematical reasoning, general-domain reasoning, and code generation across two model families. Additionally, R-Quest maintains stable performance gains over ten rounds of self-evolution, peaking in the final round and outperforming R-Zero by 17.32 points.
Tags
Links
- Source: https://arxiv.org/abs/2610.04299v1
- Canonical: https://arxiv.org/abs/2610.04299v1
Trouble viewing inline? Open PDF directly â
Full Text
73,748 characters extracted from source content.
Expand or collapse full text
Published as a conference paper at ICLR 2027 QUESTIONING THE QUESTIONS: SUSTAINING SELF- EVOLUTION IN REASONING MODELS Jinyuan Li 1 , Chengsong Huang 1 , Langlin Huang 1 , Donghong Cai 1 , Shiping Gao 2 , Yuyi Yang 1 , Jiaxin Huang 1â 1 Washington University in St. Louis, 2 University of Michigan, Ann Arbor ljinyuan,jiaxinh@wustl.edu  Project Page § Code ABSTRACT Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. In this paper, we inves- tigate why performance deteriorates over successive rounds and how to sustain self-evolution. Our analysis identifies two recurring quality problems in self- generated questions: invalid questions and repeated variants of the same mathe- matical questions. First, invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in training data. Second, existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways, which leads to question diversity collapse in later training rounds. Building on these findings, we introduce R-Quest, which uses question validity and novelty feedback to guide self-evolution. We first train the solver to recognize and reject invalid questions, then use its judgments to guide questioner rewards and filter solver training data. To avoid question repetition, we use a frozen base model to compare sampled question pairs and provide novelty feedback. Empirically, our method consistently achieves the highest average performance on 12 benchmarks in mathematical rea- soning, general-domain reasoning, and code generation across two model families. Additionally, R-Quest maintains stable performance gains over ten rounds of self- evolution, peaking in the final round and outperforming R-Zero by 17.32 points. 1INTRODUCTION 0246810 Self-Evolution Round 35 40 45 50 Math Average (%) 46.17 Peak 49.10 Î = 17.32 R-Quest (Ours) R-Zero (Huang et al., 2025) 51.92 34.60 Figure 1: Average performance on seven math benchmarks. R-Quest exceeds the peak of R-Zero in every round and peaks at round ten, leading by 17.32 points. Self-evolving reasoning models learn by generating and solving their own training tasks (Huang et al., 2026a; 2023; Tao et al., 2024; Tan et al., 2024; Yu et al., 2025; Xiang et al., 2026), reducing their dependence on human- curated data. Recent frameworks organize this process around questioner and solver roles: the questioner learns to propose challenging problems, while the solver learns to solve them, allowing both roles to co-evolve through reinforcement learning (Zhao et al., 2026a; Chen et al., 2026; Liu et al., 2025; Shi et al., 2026). However, contin- ued self-evolution can reverse early gains and ultimately lead to performance collapse (Huang et al., 2025; Li et al., 2026a). In this paper, we study the practical bottlenecks to sustained self-evolution: Why does performance dete- riorate over successive rounds, and how can we address these bottlenecks to achieve sustained self-improvement? To understand why these gains fail to persist, we examine the self-generated questions by R-Zero (Huang et al., â Corresponding author. 1 arXiv:2610.04299v1 [cs.LG] 3 Oct 2026 Published as a conference paper at ICLR 2027 Let x, y â â , x+y=3, xy=3. Find x²+y². Unstable Answer Detects Invalid Question Rejected & Filtered Out Negative Reward Questioner Solver Only Valid Questions Earn Rewards! After Validity Initialization Same Problem, Different Costume Questioner Same Underlying Task 2 1 4 3 2 A BC DE Version A Find the shortest path from A to E. 2 1 4 3 2 A BC DE Version B A backpacker crosses a magic forest. Which route uses the least energy? 2 1 4 3 2 A BC DE R-ZeroR-Quest Version A Version B Low Lexical Similarity Weak Repetition Penalty Same task, over and over... Different Words, Less Penalty! Version A Version B Same Task: Shortest Path Frequent in Training Data Rejected & Filtered Out Only Novel Questions Earn Rewards! Select for Training Treated as Novel No real solutions? But I have to answer... Invalid Questions Rollout 1: x²+y²=3 Rollout 2: No real solution Rollout 3: x²+y²=3 High Uncertainty Treated as Hard Select for Training INVALID Wonât Answer R-Quest R-Zero (a) Rejecting Invalid Questions(b) Recognizing Repeated Tasks Figure 2: Two question-quality problems in self-evolution and how R-Quest addresses them: (a) R-Quest can recognize and reject invalid questions; (b) R-Quest can detect repeated mathematical tasks despite differences in expression. 2025). Our analysis reveals two recurring quality issues: (1) invalid questions and (2) repeated variants of the same mathematical tasks. First, some generated questions are invalid (e.g., due to ambiguous definitions, missing conditions or contradictory assumptions), yet the solver is not explicitly trained to reject such questions, as shown in Figure 2(a). The loop therefore lacks a mechanism to exclude or suppress invalid questions. We find that invalid questions become more prevalent across rounds and that answer-consistency filtering further increases their proportion in training data. Our comparison study further shows that repairing those invalid questions improves solver performance under matched training settings, indicating that invalid questions are a contributing factor to performance degradation and motivating explicit validity feedback during self-evolution. Second, as shown in Figure 2(b), the questioner can repeatedly generate variants of the same underlying mathematical task by changing the wording, notation, or numerical parameters, and easily bypass existing repetition penalties. We observe substantial concentration in a few mathematical task types in R-Zeroâs fifth round training data. Existing diversity controls could split questions with the same task structure into separate clusters. Because repetition penalties depend on cluster size, this fragmentation can weaken the penalty on frequently repeated tasks. These findings suggest that effective diversity control must be grounded in an understanding of what a question asks and how it is solved, so that different expressions of the same task are recognized as repeated training content. Building on these insights, we introduce R-Quest, which promotes sustained self-evolution by explic- itly assessing the validity of generated questions and the novelty of their underlying mathematical tasks. We first train the solver to recognize and reject invalid questions. During self-evolution, we incorporate its validity judgments into the questionerâs rewards to discourage invalid generation. We also assess task novelty through sampled pairwise comparisons, using the modelâs understanding of the questions to identify repeated mathematical task structures. Together, these judgments restrict the questionerâs difficulty rewards to questions assessed as valid and novel. Both feedback signals are generated during subsequent self-evolution without further calls to stronger external models. Across two model families, R-Quest achieves the highest domain-average scores among evaluated methods on 12 benchmarks spanning mathematics, code generation, and general reasoning. As shown in Figure 1, R-Quest sustains its gains over ten rounds of self-evolution, reaching its best mathematical performance in the final round and outperforming R-Zero by 17.32 points at the same iteration on Qwen3-4B-Base. After ten rounds of self-evolution, R-Quest is 5.75 points above the base model, whereas R-Zero is 11.57 points below it. These results demonstrate that assessing the validity and task novelty of self-generated questions provides a practical route to sustained self-improvement. 2PRELIMINARY STUDY 2.1BACKGROUND: THE QUESTIONERâSOLVER FRAMEWORK Self-evolution for mathematical reasoning commonly adopts a questionerâsolver co-training frame- work. Following the R-Zero paradigm (Huang et al., 2025), both models share the same initialization and are trained alternately using Group Relative Policy Optimization (GRPO) (Shao et al., 2024). 2 Published as a conference paper at ICLR 2027 12345 Self-evolution Round 20 40 60 80 Valid Questions (%) 76.5 65.0 59.0 58.0 39.5 72.63 55.65 51.20 44.35 27.64 Before FilteringAfter Filtering (a) 05101520 GRPO Step 46 47 48 49 50 Math Average (%) 49.91 46.54 +3.37 points Repaired Questions Original Questions (b) Figure 3: Question validity rate and solver performance on Qwen3-4B-Base. (a) Valid-question rates before and after R-Zeroâs answer-consistency filter. (b) Average math reasoning performance of the solver trained on matched original and repaired datasets. For each questionqgenerated by the questioner, the frozen solver samplesMresponses. Let Ěa(q) be the majority answer anda j the extracted answer of thej-th response; the solverâs consistency is defined ass(q) = 1 M P M j=1 1[a j = Ěa(q)]. The questioner is trained to generate novel and moderately challenging questions by rewarding uncertainty, with the reward maximized ats(q) = 1/2, while penalizing repetition. Questions are grouped by lexical similarity using pairwise BLEU distances, and each receives a repetition penalty of|C(q)|/B, whereC(q)is its cluster andBis the batch size. Subsequently, the frozen questioner proposes new questions, and only those satisfying intermediate consistency (|s(q)â 1/2|⤠δ) are retained to train the solver. The solver is then updated via GRPO using self-consistency pseudo-labels, assigning binary rewards 1[a(y) = Ěa(q)] to its responses. 2.2QUESTIONING THE QUESTIONS: DIAGNOSING FAILURES IN SELF-EVOLUTION 2.2.1INVALID QUESTIONS IN SOLVER TRAINING Self-generated questions are not always well posed. Our investigation into the collapse of R- Zero (Huang et al., 2025) reveals a growing proportion of invalid generated questions. We randomly sample 200 questions from the question pool of each of the five R-Zero rounds and use GPT-5.6-Sol to assess their validity. Figure 3(a) shows that the valid-question rate falls from 76.5% in round one to 39.5% in round five, leaving most questions invalid. In addition, R-Zero would filter questions based on majority-voted answer agreement s(q), retaining those between 0.3 and 0.8. We find that among the retained questions, the valid question rate is further reduced, reaching 27.64% in round five. Thus, answer-consistency filtering increases the proportion of invalid questions in solver training data. Pattern DistributionR-Zero BLEU Clustering Top 1 â 22 Subclusters Other 40.5% Top 3 11% Top 2 17% Top 1 31.5% Top 3 Combined: 59.5% Question A A sequence of numbersis defined by a 1 = 2, a 2 = 5, and for all nâĽ3,a n is the smallest positive integer greater than a nâ1 that is relatively prime to a nâ2 . What is the remainder whenthe sum of the first 20 terms of this sequence is divided by 17? Question B The sequence of numbers b 0 , b 1 , b 2 , ... is defined as follows:b 0 = 2, b 1 = 5, and for nâĽ2, b n is the smallest positive integer greater than b nâ1 that is relatively prime to b nâ2 . What is the remainder when the sum b 0 + b 1 + âŻ+ b 19 is divided by 17? Note: Two numbers are relatively prime if their greatest common divisor (GCD) is 1. Question A and B are equivalent mathematical tasks The same sequence, indexed from 1 in A and 0 in B. Both ask for the remainder when the sum of the first 20 terms is divided by 17. Figure 4: Repeated question distribution and BLEU fragmentation in the fifth round of R-Zero. BLEU separates the illustrated mathematically equivalent questions into multiple different clusters. To test whether invalid questions contribute to solver degradation, we compare GRPO training on matched original and repaired datasets using Qwen3-4B-Base. Each dataset contains 1,273 questions, including the same 766 valid questions. The remaining 507 ques- tions are invalid in the original dataset and replaced by their repaired counterparts in the other. We use GPT-5.6-Sol to repair invalid questions while preserving the domain and minimizing changes in difficulty. Figure 3(b) shows comparable early performance fol- lowed by diverging trajectories: training on the repaired dataset yields a final average score of 49.91 across seven mathematical benchmarks, compared with 46.54 on the original dataset. These results indicate that invalid training questions are a contributing factor to solver degradation in self-evolution. 3 Published as a conference paper at ICLR 2027 2.2.2REPETITION IN GENERATED QUESTIONS Generated questions can repeat the same mathematical task structure despite differences in wording, notation, or numerical parameters. Although R-Zero discourages repetition through BLEU-based clustering, lexical similarity may not capture repetition at the level of question types. In Figure 4, we show a pair of mathematically equivalent questions in two separate BLEU clusters: they share the same sequence construction and requested computation, despite differences in wording and indexing. To quantify the prevalence of repeated question types, we use GPT-5.6-Sol to group 200 questions randomly sampled from the filtered question pool generated in R-Zeroâs fifth round. Questions in each group share the same mathematical task structure, and thus should be considered as ârepeatedâ questions. The grouping protocol is detailed in Appendix B.3. As shown in Figure 4, the three largest question types account for 59.5% of the sample. Under the BLEU-based clustering procedure of R-Zero, the largest type of question is split into 22 clusters. Such fragmentation can weaken the repetition penalty (which is based on cluster size) on a frequently repeated type. These findings motivate us to assess repetition based on the underlying task structure. 3METHOD As shown in Figure 2, R-Quest introduces two forms of feedback to support sustained self-evolution: (1) validity feedback from a solver initialized to recognize and reject invalid questions; and (2) novelty feedback from the frozen base model, which compares sampled question pairs to identify repeated tasks. These signals guide questioner rewards during alternating questionerâsolver updates. 3.1VALIDITY-AWARE SOLVER INITIALIZATION Our preliminary analysis in Section 2.2 examines invalid questions as a contributor to performance collapse during self-evolution. We thus introduce validity-aware solver initialization to teach the solver to recognize invalid questions before self-evolution. We construct a dataset by deduplicating questions from the solver training sets of five R-Zero iterations and randomly sampling an equal number from each iteration. We use GPT-5.6-Sol to annotate these questions in two stages. First, the annotator attempts to solve each question and assesses whether it is valid. Second, it independently solves only the valid questions to produce reference answers. We further review these answers and discard examples whose answers cannot be reliably verified. The dataset statistics and annotation protocol are provided in Appendix A.1 and A.2, respectively. We then train the base model with GRPO to return a mathematical answer for a valid question or INVALIDfor an invalid one. The model is instructed to attempt a solution and useINVALIDonly when it identifies a flaw in the question itself. We use the reward R init (q,a) = 1[a = a â (q)]â Îť 1[a = INVALID⧠a â (q)̸= INVALID], wherea â (q)is the reference answer for a valid question andINVALIDotherwise, andÎť > 0controls the penalty for rejecting valid questions to discourage excessive refusal. The full reward specification and training settings are provided in Appendix A.3. We initialize the solver with this model, bringing its ability to recognize invalid questions into the self-evolution loop. 3.2QUESTION NOVELTY ASSESSMENT VIA SAMPLED PAIRWISE JUDGMENTS As discussed in Section 2.2, question diversity is another dimension along which self-evolution can collapse. We use the frozen base model to perform diversity judgment throughout self-evolution. Two questions are considered the same type when they share essentially the same mathematical setup and ask for the same kind of task. Changes in numerical values or wording alone do not establish a new type, while belonging to the same broad mathematical field is insufficient to establish repetition. The full task specification is provided in Appendix B.1. LetBdenote the number of candidate questions in a questioner rollout batch during GRPO training, and letp ij â0, 1indicate whether questionsiandjare judged to be of the same type. These binary pairwise judgments define an exhaustive novelty gate, Q j̸=i (1â p ij ), which equals one only when 4 Published as a conference paper at ICLR 2027 no repetition is detected: a single same-type match is sufficient for rejection. However, evaluating this gate for all candidates requires O(B 2 ) model judgments. To reduce this cost, we uniformly sampleKother questions per candidate without replacement and reject the candidate if any reference is judged to be of the same type. This reduces the total comparison cost toO(BK)and relaxes the exhaustive gate, withKcontrolling how strongly repeated task types are suppressed. Letpdenote the fraction of the remaining questions that the judge would classify as the same type. WhenKis small relative toB, the probability of passing isPr(pass| p)â (1â p) K . For example, withK = 8, a same-type fraction of10%yields a pass probability of approximately 43%, compared with only6%at a fraction of30%. Appendix B.2 provides the exact pass probability and a plot showing how it varies with the same-type fraction p and the comparison budget K. 3.3SELF-EVOLUTION WITH VALIDITY AND NOVELTY FEEDBACK Questioner Rewards. We initialize the solver with the validity-aware model from Section 3.1, bringing its ability to recognize and reject invalid questions into self-evolution. For a generated questionq, it is first instructed to assess validity while attempting a solution and returnINVALIDif it identifies a flaw in the question. We sample multiple responses and denote the fraction returning INVALIDbyu(q). We define the validity gate asg valid (q) = 1 2 â u(q) . A question is classified as invalid when g valid (q) < 0, meaning that more than half of the responses return INVALID. For questions that pass this check, we switch to a standard problem-solving instruction that asks for reasoning and a final mathematical answer, and sample a fresh set of responses. Lets(q)be the proportion of responses in the largest answer cluster. We defineg novel (q)to be1ifqpasses the sampled novelty check in Section 3.2, and 0 otherwise. The questioner reward is R question (q) = g valid (q),g valid (q) < 0, g novel (q)R uncertainty (q), g valid (q)⼠0, R uncertainty (q) = mins(q), 1â s(q). whereR uncertainty (q)is the uncertainty reward based on answer consistency. The questioner is updated with GRPO using this reward. The prompts for validity assessment and standard problem solving are provided in the Appendix Prompts A.5 and C.2, respectively. Solver Updates. The updated questioner generates candidate training questions. We first apply the same majority-vote validity check, then sample mathematical solutions for questions that pass. Consistent with R-Zero (Huang et al., 2025), we construct the training set using answer-consistency filtering and majority-vote pseudo-labels. The solver is updated with GRPO on this dataset, mixed with a small, fixed proportion of the solver initialization data to maintain its ability to recognize and reject invalid questions. Replay examples retain the validity-aware task and reward defined in Appendix A.3. The questioner and solver are then updated alternately. Sampling settings, filtering thresholds, and the replay proportion are provided in Appendix C.2. 4EXPERIMENTS 4.1EXPERIMENTAL SETUP Models and Baselines. We evaluate Qwen3-4B-Base (Yang et al., 2025) and OctoThinker-3B- Hybrid-Base (Wang et al., 2025), covering Qwen (Yang et al., 2025) and Llama (Grattafiori et al., 2024) model families. OctoThinker uses mid-training to address the limited gains from applying GRPO directly to Llama base models (Wang et al., 2025). We compare against the base models, R- Zero, and two recent methods for stabilizing self-evolution, OCNR (Lee et al., 2026) and R-Diverse (Li et al., 2026a). Baseline descriptions and implementation details are provided in Appendix D. Evaluation.We evaluate on seven mathematical reasoning benchmarks, three general-domain rea- soning benchmarks, and two code generation benchmarks. In line with baselines (Huang et al., 2025; Lee et al., 2026), we report mean@32 on AMC and both AIME benchmarks, and greedy-decoding ac- curacy on the remaining reasoning benchmarks. Code generation is evaluated on HumanEval+ (Chen et al., 2021) and MBPP+ (Austin et al., 2021) using EvalPlus (Liu et al., 2023), with pass@1 as the metric. For each domain, we report the checkpoint with the highest average benchmark score across evaluated rounds. Benchmark descriptions and evaluation settings are provided in Appendix E. 5 Published as a conference paper at ICLR 2027 Table 1: Mathematical reasoning results (%) across seven benchmarks. Avg. denotes the mean across benchmarks. For each backbone, the best score in each column is shown in bold. MethodAvg.AMCMinervaMATHGSM8KOlympiadAIME25AIME24 Qwen3-4B-Base Base model46.17 47.3451.8474.2088.6339.1111.3510.73 Base model (validity init.) 47.07 49.1453.6874.4089.7641.0410.7310.73 R-Zero49.10 54.8454.7876.2092.4242.3712.2910.83 OCNR 49.57 55.9458.0977.6092.1243.117.9212.19 R-Diverse49.82 55.3157.3577.2092.1943.709.4813.54 R-Quest (our method)51.64 57.8958.8277.6092.5746.6713.4414.48 OctoThinker-3B Base model24.90 25.0023.1641.8069.1411.850.003.33 Base model (validity init.) 25.22 19.8426.8443.4069.9013.630.002.92 R-Zero28.25 25.4729.7848.8075.8915.110.102.60 OCNR28.12 30.0028.6849.6073.1615.410.000.00 R-Diverse28.61 29.8427.2149.2074.5316.150.003.33 R-Quest (our method)30.69 32.2729.4155.8076.4217.190.313.44 Training Settings. All evaluated methods alternate GRPO updates of the questioner and solver, using a per-round budget of 10,000 candidate questions, with OCNR requiring twofold oversampling by design. The solver uses the validity-aware initialization in Section 3.1, with 10% replay from the initialization dataset each round. The validity gate uses majority voting over nine responses, while the novelty gate usesK = 8within-batch comparisons judged by a frozen copy of the corresponding base model. Further training details are provided in Appendices A.3 and C.2. 4.2MAIN RESULTS 012345678910 Self-Evolution Round 35 40 45 50 55 Math Average (7 Benchmarks, %) 34.60 Î = 17.32 49.22 37.84 R-Quest (Ours) R-Quest (w/o novelty) R-Quest (w/o validity) R-Zero Best: 51.92 ¡ Round 10 Peak: 49.10 Round 3 Figure 5: Average performance on seven benchmarks over ten rounds on Qwen3-4B-Base. Yellow and orange denote R-Quest without novelty or validity feedback. R-Quest outperforms self-evolution base- lines in mathematical reasoning. Con- sistent with evaluations in previous work (Huang et al., 2025; Li et al., 2026a), we compare all methods over five rounds of self-evolution, reporting the checkpoint with the highest mathematical average for each method. As shown in Table 1, R- Quest achieves the highest average perfor- mance across seven mathematical reason- ing benchmarks on both backbones, sur- passing R-Zero, OCNR, and R-Diverse. It improves over Qwen3-4B-Base by 5.47 points and over OctoThinker-3B by 5.79 points. Relative to R-Zero, R-Quest yields additional gains of 2.54 and 2.44 points. Across both backbones, validity-aware ini- tialization yields mathematical performance close to that of the base models. Most of R-Questâs gains emerge during subsequent self-evolution, suggesting that initialization supports later learning by enabling the solver to provide validity feedback. Measured from the same base models, R-Quest delivers 1.87 and 1.73 times the gains achieved by R-Zero, respectively, increasing the benefit of self-evolution across Qwen and Llama model families. R-Quest sustains gains over extended self-evolution. We extend self-evolution to ten rounds to assess sustained gains, as shown in Figure 5. R-Zero peaks at 49.10 in round three, then declines 6 Published as a conference paper at ICLR 2027 Table 2: Code generation (pass@1) and general-domain reasoning results (%). Avg. denotes the mean within each domain. Best scores within each backbone are in bold. Method Code generationGeneral-domain reasoning Avg. HumanEval+ MBPP+Avg. SuperGPQA MMLU-Pro BBEH Qwen3-4B-Base Base model58.7954.8862.7028.7926.2151.418.74 Base model (validity init.)59.6355.4963.7629.4026.2851.5810.35 R-Zero59.2756.1062.4331.2128.6654.0110.95 OCNR 60.2857.3263.2331.5629.0554.8210.82 R-Diverse60.5056.7164.2931.7628.9855.7910.51 R-Quest (our method)62.6959.7665.6132.7529.4057.3611.50 OctoThinker-3B Base model13.6714.6312.708.3010.7812.141.97 Base model (validity init.) 14.7715.2414.299.0411.4213.532.17 R-Zero14.1415.8512.4314.1013.9822.256.08 OCNR15.6015.8515.3413.8313.8921.865.73 R-Diverse 14.9816.4613.4914.5014.5622.916.04 R-Quest (our method)18.9017.6820.1115.5014.9625.006.55 to 34.60 by round ten, falling below the base modelâs score of 46.17. In contrast, R-Quest retains its gains despite minor fluctuations between rounds: every checkpoint from rounds one through ten exceeds R-Zeroâs best score. R-Quest ultimately reaches its highest score of 51.92 in round ten, exceeding R-Zero by 17.32 points at the same iteration. We further find that either validity or novelty feedback alone achieves a higher peak score than R-Zero. However, validity feedback is more critical for sustained gains: its removal leads to late-stage collapse, whereas the variant without novelty feedback shows a milder decline and remains above the base model. Table 3: Component ablations on Qwen3-4B-Base. Scores are averaged within each domain. VariantMathCodeGeneral R-Quest51.6462.6932.75 w/o novelty51.0261.9831.80 w/o validity50.2761.2831.53 w/o both (R-Zero)49.1059.2731.21 Frozen validity judge49.9361.8532.39 R-Quest improves code generation and general-domain reasoning. Prior work has shown that mathematical reasoning training can benefit performance beyond mathemat- ics (Huang et al., 2025). As shown in Table 2, R-Quest substantially improves code genera- tion across both Qwen and Llama backbones. Specifically, using a mathematics-focused self- evolution curriculum, it improves average code generation scores over the base models by 3.90 points on Qwen3-4B-Base and 5.23 points on OctoThinker-3B, whereas R-Zero, OCNR, and R-Diverse yield comparatively modest gains. R-Quest also achieves the highest average performance across the three general-domain reasoning benchmarks on both backbones. Validity initialization alone yields only modest gains in both domains; most of the improvement occurs during subsequent mathematics-focused self-evolution. Reducing training on invalid questions may limit the reinforcement of solutions based on unstated assumptions, encour- aging reasoning grounded in the information provided across domains. This is particularly relevant to code generation, where correct implementations must satisfy the constraints in the task. 4.3ANALYSIS 4.3.1COMPONENT ABLATIONS Validity and novelty feedback both improve performance.As shown in Table 3, removing either validity or novelty feedback lowers performance across all three domains. We also compare with 7 Published as a conference paper at ICLR 2027 Init.12345678910 45 50 55 60 65 70 75 80 Î = 18.64 R-Quest (Ours) R-Zero 64.80 72.93 54.65 54.29 12345678910 0 20 40 60 80 100 Î = 76.13 R-Quest (Ours) R-Zero 67.05 86.36 11.36 10.23 12345678910 20 40 60 80 100 Î = 45.5 R-Quest (Ours) R-Zero 88.0 94.5 76.5 49.0 12345678910 0 20 40 60 80 100 Î = 52.63 R-Quest (Ours) R-Zero 72.00 54.69 73.20 2.06 (a) Balanced Accuracy (%) Round (b) Invalid Recall (%) Round (c) Question Validity (%) Round (d) Majority Accuracy (%) Round Figure 6: Validity assessment, question and answer quality during self-evolution on Qwen3-4B-Base. Panels (a) and (b) show R-Questâs improved validity assessment on held-out questions. Panels (c) and (d) show that R-Quest maintains high question validity and better preserves majority-answer accuracy on valid questions. Bands in (c) show pointwise 95% Wilson confidence intervals. a frozen validity judge setting, with novelty feedback enabled. This variant underperforms the full R-Quest across all three domains, supporting that continuous validity-training through self-evolution is essential. 4.3.2VALIDITY ASSESSMENT DURING SELF-EVOLUTION Self-evolution strengthens validity assessment in R-Quest. We evaluate each roundâs solver validity assessment on the same 297 questions held out from the training process, as described in Appendix A.1. Using Prompt A.5, each solver is instructed to answer valid questions and reject invalid ones, with one greedy response generated per question. We report balanced accuracy as 1 2 (Recall valid + Recall invalid ), where the two terms are the fractions of valid questions accepted and invalid questions rejected, respectively. R-Questâs balanced accuracy increases from 64.80% after initialization to 72.93% in round ten (Figure 6(a)). Since a solver that accepts every question achieves 50% balanced accuracy without detecting any invalid questions, we additionally report invalid recall to assess this capability directly. Figure 6(b) shows that R-Quest achieves an invalid recall of 86.36% in round ten, whereas recall for R-Zero remains between 10.23% and 26.14% across all ten rounds. Thus, the validity assessment capability of R-Quest continues to improve through self-evolution. R-Quest generates more valid questions. We next examine whether stronger solver validity assessment is accompanied by improved question quality. For each method and round, we randomly sample 200 questions from the unfiltered pool generated by their questioner. We query GPT-5.6-Sol to assess each questionâs validity and compute the validity rate. Figure 6(c) shows that R-Quest increases this rate from 88.0% in round one to 94.5â97.5% throughout rounds four to ten. In contrast, the rate for R-Zero falls from 76.5% to 49.0%. This indicates that R-Quest steers question generation toward valid problems as self-evolution proceeds. R-Quest solver derives more reliable answers. Question generation and answer supervision are coupled across rounds: generated questions shape solver training, and the updated solver supplies pseudo-labels for subsequent rounds. Among the 200 questions sampled for each method and round, we evaluate majority-answer accuracy only on those judged valid, since invalid questions do not provide well-defined mathematical targets for solver training. We query GPT-5.6-Sol independently to derive and verify reference answers, and compare the majority-voted answers with these references. We report strict accuracy over questions with verified references, giving no credit to missing answers or unresolved judgments. Figure 6(d) shows that the two methods begin with comparable accuracy, but by round ten, R-Quest retains 54.69% accuracy, whereas R-Zero falls to just 2.06%. This shows that the deterioration in R-Zero extends from question generation to answer supervision. By maintaining a predominantly valid question distribution and more reliable pseudo-labels, R-Quest preserves the quality of self-generated supervision that supports sustainable self-evolution. 4.3.3QUESTION DIVERSITY DURING SELF-EVOLUTION R-Quest limits the concentration of recurring exercise patterns. Following the analysis in Section 2.2.2, we uniformly sample 200 solver training question records without replacement for each 8 Published as a conference paper at ICLR 2027 method and round over the first five rounds. GPT-5.6-Sol sequentially groups them by their underlying mathematical construction and requested task. We measure concentration using the top-five share, the fraction of questions in the five largest groups, averaged over three random orderings of the same sample. Appendix B.3 provides the grouping procedure and task specification. As shown in Figure 7, the top-five share for R-Zero rises from 17.0% to 69.0% between rounds one and five. R-Quest with the defaultK = 8remains less concentrated throughout, reaching 24.0% in round five while maintaining higher performance. 12345 Self-evolution Round 0 25 50 75 100 Cluster Share (%) 17 3.5 3.7 25 16 13.2 34.5 13.5 13.2 44.5 30 16.8 69 24 21.8 Top 1Top 2Top 3Top 4Top 5Other 42 44 46 48 50 52 Math Average (%) R-ZeroR-Quest (K=8)R-Quest (K=16) Figure 7: Question type clusters (bars, left axis) and mathe- matical performance (lines, right axis) on Qwen3-4B-Base. Numbers above the bars denote top-five group proportion. Overly strict diversity control can limit mathematical gains. We fur- ther examine the strength of novelty control in Figure 7 by increasingK from its default value of 8 to 16, with other settings unchanged. A larger Kincreases the probability that ques- tions from recurring exercise patterns are rejected by the novelty gate (Ap- pendix Figure 8). This further low- ers the top-five share, most notably from 30.0% to 16.8% in round four. However, performance remains below that ofK = 8throughout, suggest- ing that overly strict diversity control may suppress more challenging vari- ants within recurring exercise patterns. Effective novelty control should therefore balance exploration of new exercise patterns with progress toward harder problems within existing ones. 5RELATED WORK Self-improving Reasoning Models. Label-free reinforcement learning uses model confidence, answer agreement, and entropy as training signals (Zhao et al., 2026b; Zuo et al., 2026; Agarwal et al., 2026). Complementary work learns to generate training tasks through self-play. Absolute Zero validates tasks and solutions through code execution (Zhao et al., 2026a), while R-Zero (Huang et al., 2025) and Self-Questioning Language Models (Chen et al., 2026) train question generators and solvers using model-derived feedback. Related frameworks ground learning in games, document corpora, vision tasks, and tool environments (Liu et al., 2026; 2025; Yue et al., 2026; Huang et al., 2026c;b; Li et al., 2026b; He et al., 2025; Dai et al., 2026; Sun et al., 2026). Our work examines the conditions under which self-generated questions support sustained improvement, focusing on question validity and task novelty. Question Validity and Diversity in Self-evolution. For validity, prior work studies question validity in synthetic mathematical data (Liang et al., 2026; Shen et al., 2026; Yang et al., 2026). For self-evolution, VHG (Lai et al., 2026) relies on a stronger external model (GPT-5.4) to verify general mathematical problems during self-play. We teach the solver to reject invalid questions through mid- training, avoiding calls to stronger external models during subsequent self-evolution. For diversity, recent methods assess novelty using distances between solver representations or code-embedding similarity (Lee et al., 2026; Li et al., 2026a). R-Diverse targets repetition in underlying reasoning procedures, but relies on an additional code generator (Qwen2.5-Coder-7B (Hui et al., 2024)) and code encoder (Jina-Code-Embeddings-1.5B (Kryvosheieva et al., 2025)) during self-evolution. This departs from the goal of self-contained self-evolution, where the model generates its own training feedback. We instead reduce novelty assessment to an atomic judgment of whether two questions are variations of the same mathematical task, allowing a frozen copy of the same base model to provide novelty feedback with only a few random within-batch comparisons. 9 Published as a conference paper at ICLR 2027 6CONCLUSION We identified invalid questions and repeated mathematical tasks as two bottlenecks to sustained self-evolution. R-Quest addresses these bottlenecks by enabling the solver to recognize and reject invalid questions and using validity and task-level novelty feedback to guide question generation. Across two model families and 12 benchmarks, R-Quest achieves the highest domain-average scores among evaluated methods in mathematics, code generation, and general reasoning. It further sustains gains over ten rounds on Qwen3-4B-Base, reaching its best mathematical performance in the final round. Further analyses show that R-Quest strengthens validity assessment, limits task repetition, and maintains more reliable majority-vote supervision across rounds. These findings highlight the ability to assess self-generated questions as a key component of sustained self-improvement. ACKNOWLEDGMENTS This material is based upon work supported in part by the National Science Foundation under Award No. 2541822 and AI for Health Grants. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding agencies. AI USE STATEMENT Generative AI was used in the research workflow for synthetic data generation, model-based annota- tion and verification, and evaluation, and is also integral to the proposed self-evolution method as described in the paper. It was additionally used for limited language editing. AI-assisted research outputs were reviewed using the validation and consistency-checking procedures described in the paper. The authors reviewed the final manuscript and take responsibility for its contents. REPRODUCIBILITY STATEMENT We provide the information required to reproduce the main training and evaluation pipeline in the paper and appendices. Appendix A describes construction of the validity-aware initialization dataset, the annotation and reference-answer review procedures, the complete prompts, reward function, and mid-training hyperparameters. Appendix B specifies the semantic novelty judgment and sampling procedure. Appendix C provides the question-generation and problem-solving prompts, question- evaluation procedure, solver-data construction, replay strategy, sampling parameters, and GRPO training settings. Appendix D documents the implementations and configurations of the R-Zero, OCNR, and R-Diverse baselines, and Appendix E specifies the evaluation benchmarks, decoding procedures, answer judging, and reported metrics. REFERENCES Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effec- tiveness of entropy minimization in llm reasoning. Advances in Neural Information Processing Systems, 38:107150â107180, 2026. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Self-questioning language models. In Workshop on Multi-Agent Learning and Its Opportunities in the Era of Generative AI, 2026. URL https://openreview.net/forum?id=8OOlhDmpXL. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 10 Published as a conference paper at ICLR 2027 Runpeng Dai, Kaili Huang, Changsung Kang, and Ciya Liao. It takes two to match: Co-evolving generative retriever with reinforcement learning. arXiv preprint arXiv:2609.00638, 2026. Xeron Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. Advances in Neural Information Processing Systems, 38, 2026. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3828â3850, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.211. URLhttps://aclanthology.org/2024.acl-long. 211/. Yicheng He, Chengsong Huang, Zongxia Li, Jiaxin Huang, and Yonghui Yang. Visplay: Self-evolving vision-language models from images. arXiv preprint arXiv:2511.15661, 2025. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=7Bywt2mQsCe. Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-Zero: Self-evolving reasoning LLM from zero data. arXiv preprint arXiv:2508.05004, 2025. URL https://arxiv.org/abs/2508.05004. Chengsong Huang, Haolin Liu, Tong Zheng, Runpeng Dai, Langlin Huang, Jinyuan Li, Zongxia Li, Zhepei Wei, Yu Meng, and Jiaxin Huang. G-zero: Self-play for open-ended generation from zero data. arXiv preprint arXiv:2605.09959, 2026a. Chengsong Huang, Zifeng Wang, Rujun Han, Jun Yan, Yanfei Chen, Zoey CuiZhu, Ke Jiang, Peng Xia, Han Yu, Yufan Zhuang, et al. Envharness: Awakening static worlds for agent learning. arXiv preprint arXiv:2608.19880, 2026b. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 1051â1068, December 2023. doi: 10.18653/v1/ 2023.emnlp-main.67. URL https://aclanthology.org/2023.emnlp-main.67/. Siyuan Huang, Pengyu Cheng, Haotian Liu, Tao Chen, Yihao Liu, Jingwei Ni, Shijie Zhou, Ziyi Yang, Gangwei Jiang, Mengyu Zhou, et al. Skill self-play: Pushing the frontier of llm capability with co-evolving skills. arXiv preprint arXiv:2607.22529, 2026c. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024. Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, San- ket Vaibhav Mehta, Lalit K Jain, Virginia Aglietti, Disha Jindal, Yuanzhu Peter Chen, et al. Big-bench extra hard. In Proceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), p. 26473â26501, 2025. Daria Kryvosheieva, Saba Sturua, Michael GĂźnther, Scott Martens, and Han Xiao. Efficient code embeddings from code generation models. In NeurIPS 2025 Fourth Workshop on Deep Learning for Code, 2025. URL https://openreview.net/forum?id=smR7RPdWZs. Yuhang Lai, Jiazhan Feng, Yee Whye Teh, and Ning Miao. Verifier-backed hard problem generation for mathematical reasoning. arXiv preprint arXiv:2605.06660, 2026. 11 Published as a conference paper at ICLR 2027 Seungyoo Lee, Giung Nam, Hyungi Lee, and Juho Lee. OCNR: Stabilizing self-play by mitigating iteration-collapse with one-class novelty rewards. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=70lhVsTZ62. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843â3857, 2022. Gengsheng Li, Jinghan He, Shijie Wang, Ruiqi Liu, Renrui Zhang, Zijun Yao, Junfeng Fang, Haiyun Guo, Dan Zhang, and Jinqiao Wang. R-diverse: Mitigating diversity illusion in self- play LLM training. In Forty-third International Conference on Machine Learning, 2026a. URL https://openreview.net/forum?id=DZiuKVvrJW. Zongxia Li, Hongyang Du, Chengsong Huang, Xiyang Wu, Lantao Yu, Yicheng He, Jing Xie, Xiaomin Wu, Zhichao Liu, Jiarui Zhang, et al. Mm-zero: Self-evolving multi-model vision language models from zero data. arXiv preprint arXiv:2603.09206, 2026b. Hao Liang, Meiyi Qiang, Yuying Li, Zefeng He, Xiaochen Ma, Ruichuan An, Yongzhen Guo, Zhengzhou Zhu, Bin Cui, and Wentao Zhang. Mathdebugger: Detecting and diagnosing errors in synthetic mathematical data, 2026. URL https://arxiv.org/abs/2502.19058. Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, and Jason Weston. Spice: Self-play in corpus environments improves reasoning. arXiv preprint arXiv:2510.24684, 2025. Bo Liu, Simon Yu, Zichen Liu, Leon Guertler, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, et al. Spiral: Self-play on zero-sum games incentivizes reasoning via multi- agent multi-turn reinforcement learning. In International Conference on Learning Representations, volume 2026, p. 35407â35434, 2026. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems, 36:21558â21572, 2023. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Chengyu Shen, Zhen Hao Wong, Runming He, Hao Liang, Meiyi Qiang, Zimo Meng, Zhengyang Zhao, Bohan Zeng, Zhengzhou Zhu, Bin Cui, et al. Letâs verify math questions step by step. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, p. 2770â2781, 2026. Yucheng Shi, Zhenwen Liang, Kishan Panaganti, Dian Yu, Wenhao Yu, and Haitao Mi. Learning to build the environment: Self-evolving reasoning rl via verifiable environment synthesis. arXiv preprint arXiv:2605.14392, 2026. Jiashuo Sun, Pengcheng Jiang, Saizhuo Wang, Jiajun Fan, Heng Wang, Siru Ouyang, Ming Zhong, Yizhu Jiao, Chengsong Huang, Xueqiang Xu, et al. Rethinking the reranker: Boundary-aware evidence selection for robust retrieval-augmented generation. arXiv preprint arXiv:2602.03689, 2026. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 conference on empirical methods in natural language processing, p. 930â957, 2024. Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. A survey on self-evolution of large language models. arXiv preprint arXiv:2404.14387, 2024. 12 Published as a conference paper at ICLR 2027 Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 95266â95290, 2024. Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512, 2025. Zhishang Xiang, Chengyi Yang, Zerui Chen, Zhimin Wei, Yunbo Tang, Zongpei Teng, Zexi Peng, Zongxia Li, Chengsong Huang, Yicheng He, Chang Yang, Xinrun Wang, Xiao Huang, Qinggang Zhang, and Jinsong Su. A systematic survey of self-evolving agents: From model-centric to environment-driven co-evolution. TechRxiv, 2026(0227), 2026. doi: 10.36227/techrxiv.177203250. 05832634/v2. URLhttps://w.techrxiv.org/doi/abs/10.36227/techrxiv. 177203250.05832634/v2. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Chengyi Yang, Zhishang Xiang, Yunbo Tang, Zongpei Teng, Chengsong Huang, Fei Long, Yuhan Liu, and Jinsong Su. Ttcs: Test-time curriculum synthesis for self-evolving. arXiv preprint arXiv:2601.22628, 2026. Wenhao Yu, Zhenwen Liang, Chengsong Huang, Kishan Panaganti, Tianqing Fang, Haitao Mi, and Dong Yu. Guided self-evolving llms with minimal human supervision. arXiv preprint arXiv:2512.02472, 2025. Zhenrui Yue, Kartikeya Upasani, Xianjun Yang, Suyu Ge, Shaoliang Nie, Yuning Mao, Zhe Liu, and Dong Wang. Dr. zero: Self-evolving search agents without training data. arXiv preprint arXiv:2601.07055, 2026. Andrew Zhao, Yiran Wu, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. Advances in Neural Information Processing Systems, 38:105816â105879, 2026a. Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. In International Conference on Learning Representations, volume 2026, p. 2548â2581, 2026b. Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Xinwei Long, Ermo Hua, Biqing Qi, et al. Ttrl: Test-time reinforcement learning. Advances in Neural Information Processing Systems, 38:131459â131483, 2026. 13 Published as a conference paper at ICLR 2027 AVALIDITY-AWARE MID-TRAINING DETAILS A.1DATASET CONSTRUCTION We collect questions from the solver training sets of the first five R-Zero (Huang et al., 2025) rounds using Qwen3- 4B-Base (Yang et al., 2025). After deduplication within and across iterations, we randomly sample 460 questions per iteration, assigning 400 to training and 60 to valida- tion. Annotation and answer review yield a final dataset of 2,240 questions, with statistics reported in Table 4. Ta- ble 5 reports the distribution of error types among invalid questions. Table 4: Dataset statistics for validity- aware solver initialization. SplitValidInvalidTotal Training8021,1411,943 Validation121176297 Total9231,3172,240 Table 5: Error types among the 1,317 invalid questions in the training and validation splits. Error typeCountPercentage Contradiction38028.9% No solution36227.5% Ambiguity31523.9% Missing information18113.7% Poorly formed questions725.5% Other70.5% Total1,317100% A.2ANNOTATION AND ANSWER REVIEW We usegpt-5.6-Solto annotate the questions in two stages. Both stages use only the question, without the original R-Zero pseudo-label. First, the model attempts to solve each question and assesses whether it is valid using Prompt A.1. Category A is mapped to VALID, and BâF to INVALID. Prompt A.1: Validity annotation You are a careful reference judge for generated mathematical and reasoning questions. You see only the question. Restate its goal, inspect every definition and condition, check for contradictions and multiple reasonable interpretations, and genuinely attempt a solution. Then classify: A = self-contained, valid, solvable, and uniquely gradable; B = meaningful but open-ended or not precisely gradable; C = missing conditions, multiple answers, or key ambiguity; D = contradictory or has no solution; E = malformed, undefined, garbled, or unintelligible; F = you cannot judge reliably. Use A only when all strict requirements hold. A question can have multiple solutions and still be A when it explicitly asks for the complete solution set and that set is objectively gradable. Judge only the question as written. Keep the reasoning summary concise but concrete. For A,invalid_typemust benull; for B-F choose the single closest invalid_type. Second, a separate call independently solves the valid questions to produce reference answers using Prompt A.2. Prompt A.2: Reference answers You are the answer-verification pass for a mathematical reasoning dataset. The validity pass has already classified this question as A (valid). Solve it completely from scratch. Do not copy or trust an answer hint that may appear in the question. Check every stated condition, substitute or otherwise verify the result, and confirm that the final answer answers exactly what was asked. 14 Published as a conference paper at ICLR 2027 Prompt A.2: Reference answers (continued) canonical_final_answermust be a concise answer suitable for exact or mathematical-equivalence rewarding, without reasoning or a boxed wrapper. Setanswer_verified=trueonly when you have high confidence that the answer is correct and complete. If reliable verification is not possible, set itfalseand explain the uncertainty; never invent an answer to satisfy the schema. We then review the reference answers and exclude questions whose answers cannot be reliably verified. Prompts A.3 and A.4 provide the review instructions, with the latter used for questions requiring further review. Questions flagged forHUMAN_REVIEWwere manually reviewed and retained only when their validity and reference answers could be verified; unresolved cases were excluded. Prompt A.3: Answer review You are an independent mathematical answer consistency auditor. You receive an opaque questionID, the question, a Pass 1 derived answer (which may be missing), and a Pass 2 canonical answer. Neither answer is authoritative. Read the question carefully and identify exactly what it asks. Independently do the mathematical checks needed to decide whether the canonical answer is correct, complete, and responsive. Decide whether an ambiguity in the question could change the answer. Compare the two supplied answers by mathematical meaning, not merely by string equality: different strings can be equivalent, and equal strings are not proof of correctness. If reliable verification is not possible, say so explicitly. Keepreasoning_summary concise and concrete; do not provide a long proof. Relation meanings: - AGREE: same answer and meaning. - EQUIVALENT: different presentation but mathematically equivalent. - CONFLICT: incompatible answers. - NOT_COMPARABLE: their forms or scopes do not permit a reliable direct comparison. - MISSING: the derived answer is absent. Setneeds_deep_reviewwhenever there is a correctness concern, material ambiguity, conflict, or insufficient basis for a high-confidence screening decision. Prompt A.4: Further answer review You are the independent deep reviewer for a mathematical answer dataset. You receive only an opaque questionID, the question, a prior derived answer (possibly missing), and a canonical answer (possibly missing). None is authoritative. Solve the question independently from first principles. Verify what the question actually asks, whether it is clear and valid, and whether your independently derived answer is complete and responsive. Then assess the canonical answer and compare your answer separately with the derived and canonical answers by mathematical meaning, not string equality. Do not assume agreement means correctness. If ambiguity can change the answer, or reliable verification is not possible, say so. RecommendKEEP_CANONICALonly when the canonical answer is independently confirmed. Rec- ommendREPLACE_CANONICALonly when the canonical answer is wrong or missing and your re- placement is reliably verified. RecommendEXCLUDEfor a materially ambiguous or invalid question. RecommendHUMAN_REVIEWfor unresolved conflicts or uncertainty. Keepreasoning_summary concise but include the decisive mathematical check. When recommendingREPLACE_CANONICAL, setreplacement_answerto exactly theindependently_derived_answer; otherwise set replacement_answer to null. A.3REWARD AND TRAINING SETTINGS We train Qwen3-4B-Base (Yang et al., 2025) and OctoThinker-3B-Hybrid-Base (Wang et al., 2025) with GRPO to return a mathematical answer for a valid question orINVALIDfor an invalid one, using Prompt A.5. 15 Published as a conference paper at ICLR 2027 Table 6: Training settings for validity-aware solver initialization. SettingValue Training steps15 Questions per rollout batch512 Responses per question8 Actor global batch size128 Learning rate10 â6 KL coefficient10 â2 Sampling temperature1.0 Top-p0.99 Maximum prompt tokens2,048 Maximum response tokens4,096 Prompt A.5: Solver training Question: [Question] Solve the problem carefully and reason step by step. While solving, check whether the problem itself is well-defined, internally consistent, and solvable as stated. If the problem is valid, solve it normally. If, while solving, you determine that the problem itself is invalid because of missing information, contradiction, ambiguity, impossibility, malformed conditions, or lack of objective gradability, use INVALID as the final answer. Do not outputINVALIDmerely because the problem is difficult, because you are uncertain, or because you fail to solve it. First give your reasoning enclosed within<think>and</think>tags. Then put exactly one final answer inside . For a valid problem, that box must contain the mathematical answer; for an invalid problem, it must contain INVALID. Rewards are computed from the final output in . Rejection means returningINVALIDas the final answer. Mathematical correctness is evaluated against the reviewed reference answer using themathrulerequivalence grader used in R-Zero. For responses with a non-empty final answer box, we apply the reward in Section 3.1 with Îť = 0.5: r =    1,correct answer or correct rejection, â0.5, rejection of a valid question, 0,otherwise. (1) Responses with a missing or empty answer box receive a reward ofâ0.1. We sample eight responses per question and compute GRPO advantages within each group, with a KL penalty relative to the base model. The checkpoint at step 15 initializes the solver for self-evolution. Table 6 summarizes the training settings. 16 Published as a conference paper at ICLR 2027 BSEMANTIC NOVELTY ASSESSMENT B.1TASK SPECIFICATION The frozen base model compares a candidate question (Question A) with a sampled reference (Question B) using the task specification in Prompt B.1. Prompt B.1: Semantic comparison You are judging whether two generated math problems are repetitions of the same recurring exercise pattern. Choose SAME_TYPE when the two problems have essentially the same distinctive mathematical setup and ask the same kind of task, so that they feel like variations of the same exercise. Differences in constants, coefficients, variables, formulas, bounds, or other local details do not by them- selves make the problems different. Do not choose SAME_TYPE merely because the problems share a broad topic, similar wording, presenta- tion style. If the common pattern is not clear and specific, choose DIFFERENT. Briefly compare the exercise pattern of the two problems, then end with exactly SAME_TYPE or DIFFERENT. Question A: question_a Question B: question_b Analysis: B.2EFFECT OF RANDOM SAMPLING Consider a candidate among theBsuccessfully parsed questions in a Questioner rollout batch. Each batch initially contains 2,048 generated candidates. Among the otherB â 1questions, letmbe classified as the same exercise type by the judge, withp = m/(Bâ 1). The candidate passes the novelty check only if none of itsKreferences is classified as the same type. Uniform sampling without replacement gives P pass (B,m,K) = Bâ1âm K Bâ1 K , K ⤠Bâ 1,(2) where n K = 0forn < K. WhenKis small relative toB, the same-type fraction changes little across successive draws, yielding P pass (B,m,K)â (1â p) K .(3) 0255075100 Same-Type Fraction p (%) 0 25 50 75 100 Pass Probability (%) Same-Type Fraction: 5% â Pass Probability: 66.3% Same-Type Fraction: 5% â Pass Probability: 44.0% Same-Type Fraction: 20% â Pass Probability: 16.8% Same-Type Fraction: 20% â Pass Probability: 2.8% K=8 K=16 Figure 8: Approximate pass probability(1â p) K for the default comparison budgetK = 8and the additional setting K = 16. Figure 8 compares the defaultK = 8with the additionalK = 16setting. Increasing the comparison budget makes same-type matches more likely, raising the probability of rejection. For example, when a recurring exercise pattern accounts for 5% of the other candidates in the batch, increasingKfrom 8 to 16 lowers the approximate pass probability from 66.3% to 44.0%. For a more frequent pattern account- ing for 20% of the other candidates, the pass probability drops from 16.8% to just 2.8%. Thus,K = 8already provides substantial sup- pression: candidates from a pattern with a 20% share are rejected about five sixths of the time. Increasing K to 16 raises this rejection proba- bility to 97.2%, leaving very few candidates of that type eligible for the uncertainty reward. This 17 Published as a conference paper at ICLR 2027 helps interpret the results in Section 4.3.3, whereK = 16lowers the top-five share but yields worse mathematical performance thanK = 8. Overly strict novelty control may suppress not only repeated exercises but also more challenging variants within the same pattern, limiting opportunities for further learning. The default budget therefore balances suppression of recurring patterns with room for continued exploration within them. B.3EXERCISE-PATTERN CONCENTRATION ANALYSIS We analyze the sampled training records without additional filtering or deduplication. Only question text is supplied to the evaluator; method names, round indices, solver responses, and group sizes are withheld. All methods and rounds use the same evaluator and prompt. For each ordering, grouping starts with an empty inventory. Questions are processed one at a time using Prompt B.2. Each existing group is represented by a short pattern description and the question that established it. A question either joins an existing group or establishes a new group with its own description and representative. Existing descriptions and representatives remain fixed throughout the pass. Each question is assigned to exactly one group. We repeat this procedure from an empty inventory under three random orderings of the same 200 records. For each pass, we sum the sizes of the five largest groups and divide by 200, or sum all groups if fewer than five are formed. We report the mean top-five share across the three orderings. Prompt B.2: Incremental exercise-pattern grouping You are grouping mathematical questions by recurring exercise pattern. Given a new question and an inventory of existing groups, assign it to an existing group or establish a new one. Exercise-pattern definition Questions share a pattern when they have essentially the same distinctive mathematical setup and ask for the same kind of task, making them variations of the same exercise. Differences in wording, notation, numerical values, or other local details do not by themselves establish a new pattern. Changes to defining conditions or the requested task warrant a separate group when they change the underlying exercise. A shared broad topic or generic solution technique is not sufficient. Assignment Compare the new question with the description and representative question of each existing group. Choose the best-matching group only when the shared pattern is clear and specific, without broadening the groupâs definition. Otherwise, create a new group and briefly describe its mathematical setup and requested task. If the inventory is empty, create a new group. Classify the task as stated rather than repairing it. Apparent flaws alone are not a shared exercise pattern. Output Return a JSON object with the following fields: group_id: the matching group ID, or NEW. pattern: a concise description for a new group; otherwise, an empty string. reason: a brief explanation of the match or the need for a new group. Existing groups Each entry contains an ID, pattern description, and representative question. group_inventory New question question 18 Published as a conference paper at ICLR 2027 CSELF-EVOLUTION SETTINGS C.1TASK SPECIFICATIONS We use the question-generation and mathematical problem-solving instructions from R-Zero (Huang et al., 2025), provided in Prompts C.1 and C.2, respectively. Validity checks use Prompt A.5, and semantic comparisons use Prompt B.1. Prompt C.1: Question generation System You are an expert competition-math problem setter. FIRST, in your private scratch-pad, think step-by-step to design a brand-new, non-trivial problem. The problem could come from any field of mathematics, including but not limited to algebra, geometry, number theory, combinatorics, prealgebra, probability, statistics, and calculus. Aim for a difficulty such that fewer than 30 % of advanced high-school students could solve it. Avoid re-using textbook clichĂŠs or famous contest problems. THEN, without revealing any of your private thoughts, output **exactly** the following two blocks: <question> The full problem statement on one or more lines </question> final_answer Do NOT output anything elseâno explanations, no extra markup. User Generate one new, challenging reasoning question now. Remember to format the output exactly as instructed. Prompt C.2: Mathematical problem solving System Please reason step by step, and put your final answer within . User [Question] C.2SAMPLING AND TRAINING SETTINGS Questioner Rewards. To evaluate each generated question, the frozen solver first produces nine validity responses. A question is rejected if more than half of the outputs areINVALID. For questions that pass, we sample ten fresh mathematical solutions to compute answer consistency. These signals, together with the novelty judgment, determine the questioner reward defined in Section 3.3. Questions that fail the required output format receive a reward ofâ1. Solver Data Construction.After updating the questioner, we generate candidate training questions and apply the same evaluation procedure: nine validity responses, followed by ten mathematical solutions for questions that pass. We retain questions with answer consistencys(q)â [0.3, 0.8]and a non-empty majority-vote answer, which serves as the pseudo-label. Solver Training and Replay. Replay examples are sampled from the training split of the mid- training dataset to constitute10%of the mixed solver dataset. They retain the validity-aware task and reward from Appendix A.3. On newly generated questions, the solver uses the standard mathematical problem-solving task and the R-Zero training reward:0.9for agreement with the pseudo-label under mathematical equivalence checking, plus0.1for placing reasoning inside<think>and</think> tags followed by a boxed answer. Shared Settings. For question evaluation and GRPO rollouts, we use a sampling temperature of 1.0 and a maximum response length of 4,096 tokens. Both the questioner and solver are trained with a learning rate of10 â6 , a KL coefficient of10 â2 , and a maximum prompt length of 2,048 tokens. Table 7 reports question evaluation settings, while Table 8 reports GRPO training settings. Responses per input in the latter refer to training rollouts. 19 Published as a conference paper at ICLR 2027 Table 7: Question evaluation settings. SettingValue Validity responses per question9 Math solutions per accepted question10 Top-p1.0 Top-k40 Consistency filter for solver data[0.3, 0.8] Table 8: GRPO training settings. SettingQuestionerSolver Steps per iteration515 Inputs per rollout batch512512 Responses per input45 Actor global batch size4128 Top-p0.990.99 DBASELINE DETAILS Base Models.We evaluate Qwen3-4B-Base (Yang et al., 2025) and OctoThinker-3B (Wang et al., 2025) without additional training to establish their performance before self-evolution. R-Zero. We use the released implementation of R-Zero (Huang et al., 2025), which alternates reinforcement learning updates of a questioner and a solver. The questioner reward combines solver uncertainty with a BLEU-based penalty for within-batch repetition. Generated questions are assigned pseudo-labels through majority voting and used for subsequent solver training. OCNR.We implement OCNR (Lee et al., 2026) from its published description within the R-Zero training framework. OCNR augments the questioner reward with a novelty score from a one-class detector trained to distinguish questions used for solver training from questions withheld from training. Following OCNRâs twofold overgeneration scheme, we generate 20,000 candidate questions per round, twice our standard budget. The detector maintains one learnable center in the solver representation space. Using values reported in the paper, we set the novelty reward coefficient to0.3, the distance temperatureĎ = 0.5, and the EMA update weightÎą = 0.05. Question representations are obtained by mean-pooling the solverâs final-layer hidden states. The detector receives one full-batch SGD update per round with a learning rate of 7Ă 10 â3 . R-Diverse. We implement R-Diverse (Li et al., 2026a) from its published description within the R-Zero training framework. It penalizes repetition based on how questions are solved, both within the current batch and across rounds. Following the published configuration, we use Qwen2.5- Coder-7B (Hui et al., 2024) with greedy decoding to represent questions as solution programs and Jina-Code-Embeddings-1.5B (Kryvosheieva et al., 2025) to measure their similarity within the current batch and against earlier questions. For within-batch repetition, we retain R-Zeroâs average-linkage clustering with a cosine-distance threshold of0.5. Following the paper, we set the coefficients of the within-batch and cross-iteration repetition penalties to1. The memory penalty combines maximum and mean cosine similarities with equal weights and thresholds of0.5and0.25, respectively. From the second round onward, historical examples are uniformly sampled to constitute30%of the solver training data. 20 Published as a conference paper at ICLR 2027 EEVALUATION BENCHMARKS Although our self-evolution process focuses on mathematical questions, we evaluate both mathemati- cal reasoning and its generalization to other domains. Our evaluation covers mathematical reasoning, general-domain reasoning, and code generation. Mathematical Reasoning.We evaluate on seven benchmarks: AMC, Minerva (Lewkowycz et al., 2022), MATH-500 (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), OlympiadBench (He et al., 2024), AIME24, and AIME25. We use Qwen3-32B (Yang et al., 2025) to judge the equivalence of model answers and reference answers. Consistent with R-Zero (Huang et al., 2025), we report mean@32 on AMC and both AIME benchmarks, and accuracy under greedy decoding on the remaining benchmarks. General Domain Reasoning. We evaluate generalization beyond mathematics on three bench- marks: ⢠MMLU-Pro (Wang et al., 2024): Multiple-choice questions across 14 academic domains, requiring both subject knowledge and reasoning. ⢠SuperGPQA (Du et al., 2026): Graduate-level multiple-choice questions across 285 disci- plines, assessing specialized knowledge and reasoning. â˘BBEH (Kazemi et al., 2025): Challenging tasks covering logical deduction, spatial reason- ing, and multi-step computation. We follow the R-Zero evaluation protocol and report exact-match accuracy under greedy decoding. Code Generation. We evaluate code generation on two Python programming benchmarks: â˘HumanEval+ (Chen et al., 2021; Liu et al., 2023): Completing Python functions given their signatures and natural-language specifications. ⢠MBPP+ (Austin et al., 2021; Liu et al., 2023): Generating short Python programs from natural-language descriptions of basic programming tasks. We use greedy decoding and report pass@1 on the full EvalPlus test suites. 21 Published as a conference paper at ICLR 2027 Table 9: Per-benchmark results for the gate ablations on Qwen3-4B-Base. Variant Mathematical reasoning Avg. AMC Minerva MATH GSM8K Olympiad AIME25 AIME24 R-Quest51.64 57.8958.8277.6092.5746.6713.4414.48 w/o novelty51.02 55.9456.6278.0091.8145.6315.4213.75 w/o validity50.27 56.4858.8277.0091.7441.1913.5413.12 w/o both (R-Zero)49.10 54.8454.7876.2092.4242.3712.2910.83 Frozen validity judge49.93 58.1356.9978.2092.1243.118.0212.92 Variant Code generationGeneral-domain reasoning Avg. HumanEval+ MBPP+Avg. SuperGPQA MMLU-Pro BBEH R-Quest62.6959.7665.6132.7529.4057.3611.50 w/o novelty61.9859.1564.8131.8029.2155.9610.24 w/o validity61.2858.5464.0231.5328.4355.3910.77 w/o both (R-Zero)59.2756.1062.4331.2128.6654.0110.95 Frozen validity judge61.8559.1564.5532.3929.4156.2511.50 FADDITIONAL RESULTS F.1DETAILED ABLATION RESULTS Table 9 reports the individual benchmark scores underlying the domain averages in Table 3. Removing either validity or novelty feedback lowers performance on both code generation benchmarks and all three general reasoning benchmarks. The effects vary across mathematical benchmarks, with removing validity feedback producing the largest drop on OlympiadBench. Although individual ablations achieve higher scores on some benchmarks, the full R-Quest achieves the highest average performance in all three domains. 22