Paper deep dive
Direct Self-Evolving Optimization: Evolving LLMs without Challenger Training
Yuyang Deng, Yu Wang, Jiayun Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-evolving language models improve by generating tasks and learning from their own feedback, but adapting the task generator often requires a separate challenger-training loop. Can we generate tasks adapted to the current solver without explicitly training a challenger? We introduce \textbf{D}irect Self-\textbf{E}volving \textbf{O}ptimization (DEO), which replaces challenger parameter updates with solver-guided task sampling. The KL-regularized challenger objective defines an exponential tilt of a fixed base task distribution. DEO uses this distribution as a sampling target: a frozen LLM generates and mutates tasks, the solver scores them, and an approximate Metropolis selection rule refines the training pool. Only the solver is trained. Theoretically, for an idealized variant that samples exactly from the tilted distribution, and under regularity, local gradient-dominance, and initialization conditions, we show that DEO learns distributionally robust reasoning ability. In experiments, DEO achieves reasoning performance competitive with R-Zero while using over $50\%$ less wall-clock training time, and improves reasoning accuracy over a no-walk ablation. Replacing the task generator with a frozen API-only LLM further improves the local solver, illustrating a capability enabled by removing challenger training.
Tags
Links
- Source: https://arxiv.org/abs/2609.34279v1
- Canonical: https://arxiv.org/abs/2609.34279v1
Trouble viewing inline? Open PDF directly →
Full Text
99,402 characters extracted from source content.
Expand or collapse full text
Direct Self-Evolving Optimization: Evolving LLMs without Challenger Training Yuyang Deng Yu Wang Jiayun Wang Affiliation: Accenture, Center for Advanced AI Affiliation: Georgia Institute of Technology Email: matthewgo2009@gmail.comfeather1014@gmail.compjwang@gatech.edu Abstract Self-evolving language models improve by generating tasks and learning from their own feedback, but adapting the task generator often requires a separate challenger-training loop. Can we generate tasks adapted to the current solver without explicitly training a challenger? We introduce Direct Self-Evolving Optimization (DEO), which replaces challenger parameter updates with solver-guided task sampling. The KL-regularized challenger objective defines an exponential tilt of a fixed base task distribution. DEO uses this distribution as a sampling target: a frozen LLM generates and mutates tasks, the solver scores them, and an approximate Metropolis selection rule refines the training pool. Only the solver is trained. Theoretically, for an idealized variant that samples exactly from the tilted distribution, and under regularity, local gradient-dominance, and initialization conditions, we show that DEO learns distributionally robust reasoning ability. In experiments, DEO achieves reasoning performance competitive with R-Zero while using over 50%50\% less wall-clock training time, and improves reasoning accuracy over a no-walk ablation. Replacing the task generator with a frozen API-only LLM further improves the local solver, illustrating a capability enabled by removing challenger training. 1 Introduction Self-evolving training offers a way to improve large language models (LLMs) by generating their own tasks and learning from the resulting feedback. Recent methods, including R-Zero (Huang et al., 2025), Absolute Zero Reasoner (AZR) (Zhao et al., 2025), and Agent0 (Xia et al., 2025), couple task proposal with task solving to construct an adaptive curriculum. A challenger seeks tasks that challenge the current solver, and the solver learns from the generated tasks during this interaction. R-Zero and Agent0 train separate models for these roles; AZR trains one model to perform both (but still two-stage training). These approaches demonstrate that pretrained models can improve without an externally supplied task–answer dataset for this stage of training. Training the challenger, however, adds another optimization process to every self-evolving round. A separately trained challenger also requires its own training state and access to trainable model parameters. Yet the solver ultimately consumes the tasks produced by the challenger, rather than the challenger’s parameters. This motivates our central question: Can we directly generate tasks lying in the current solver’s capability frontier without explicitly training a challenger? Our starting point is to view the challenger as a distribution over tasks. For a fixed solver, maximizing challenger reward with a KL penalty to a base task distribution has an exponential-tilt solution. This is a standard property of KL-regularized optimization (Rafailov et al., 2023): tasks with higher challenger reward receive greater relative probability. The closed form specifies a target distribution; it does not by itself make sampling tractable. It nevertheless suggests an alternative to learning a parameterized challenger: adapt the task samples directly. An adaptive task distribution need not require updates to the generator’s parameters. Figure 1: Comparison between traditional challenger training self-evolving method and our DEO. (A) Conventional self-evolving method updates both a task proposer and a solver. (B) DEO keeps the generator frozen and uses solver feedback to select task mutations; only the solver receives parameter updates. DEO uses more than half less wall-clock time than R-Zero while yielding competitive performance. We develop this idea into Direct Self-Evolving Optimization (DEO). A frozen LLM generates an initial task pool and proposes mutations conditioned on the current tasks. The solver evaluates these proposals, and a Metropolis-style selection rule guides the pool toward the current challenger target: if the mutated task looks more challenging to the current solver, we tend to accept it. Through this LLM-based mutation, we produce tasks that lie near the frontier of the current solver’s capability. We then use the mutated tasks and solver-produced pseudo-labels to train the solver. Drawing on Metropolis–Hastings sampling (Metropolis et al., 1953; Hastings, 1970), this procedure uses LLM inference to adapt the curriculum while eliminating challenger parameter updates. Because the generator only needs inference access, DEO can also use a closed-source LLM as the task proposer and mutator. We then ask, what does DEO actually learn? We analyze an idealized variant of DEO that samples exactly from the optimal challenger distribution. The analysis shows that DEO learns a model whose worst-case answer uncertainty over a KL neighborhood of the base task distribution is small (Theorem 1). If the initial model’s modal answers are mostly correct, this worst-case answer concentration translates into a distributionally robust answer-accuracy bound (Corollary 1). These results explain the role of adversarial task sampling: it learns distributionally robust reasoning ability. They do not, however, imply uniform superiority on every evaluation domain. We conduct experiments on mathematical reasoning. DEO remains competitive with R-Zero, which trains a challenger, while using 52%52\%–61%61\% less wall-clock time. Because DEO replaces challenger parameter updates with LLM inference and selection, it can also use a closed-source model as generator and mutator: replacing the open-source generator with Claude Haiku 4.5 further improves the solver’s reasoning ability. Together, these results support solver-guided task sampling as a practical alternative to explicit challenger training. 2 Related Work Self-evolving training. Self-generated data and feedback provide several routes to improving LLM parameters. Self-Instruct (Wang et al., 2022) generates and filters instruction data for fine-tuning, while STaR (Zelikman et al., 2022) iteratively trains on generated rationales that produce known correct answers. Evol-Instruct (Xu et al., 2023) rewrites seed instructions into more complex training examples, and Self-Rewarding Language Models (Yuan et al., 2024) use model-generated judgments for iterative preference optimization. These methods reduce the need for manual supervision, with different requirements for seed tasks, labels, and feedback. An interacting-agent perspective on LLM training is discussed by Liu et al. (2024). Closest to our setting, R-Zero (Huang et al., 2025), AZR (Zhao et al., 2025), and Agent0 (Xia et al., 2025) adapt task generation to the evolving solver. R-Zero trains separate challenger and solver models; AZR trains a single model to propose and solve executor-verifiable tasks; and Agent0 co-evolves curriculum and executor agents with tool use. DEO shares their goal of constructing an adaptive curriculum, but replaces proposer parameter updates with solver-guided task sampling. LLM-based task rewriting itself has a precedent in Evol-Instruct; DEO instead grounds task selection in a regularized challenger objective and uses the current solver’s evaluations to guide Metropolis-style refinement toward its induced task distribution. Inference-time refinement and evolution. A complementary line of work improves outputs through repeated LLM inference. Self-Refine (Madaan et al., 2023) alternates answer generation, self-feedback, and revision, while Reflexion (Shinn et al., 2023) stores verbal reflections to guide subsequent attempts without updating model weights. Tree of Thoughts (Yao et al., 2023) searches over intermediate reasoning steps. Evolutionary approaches include Mind Evolution (Lee et al., 2025), which generates, recombines, and refines candidate responses, and FunSearch (Romera-Paredes et al., 2024), which pairs LLM-generated program variants with an automated evaluator. AlphaEvolve (Novikov et al., 2025) extends this approach to larger programs and algorithmic discovery, using LLM-proposed code edits, automated evaluators, and a population of candidate solutions. Its search optimizes programs against user-specified evaluation metrics. Related prompt optimization methods such as GEPA (Agrawal et al., 2025) use reflections on execution traces to evolve system prompts. DEO likewise uses a frozen LLM to propose revisions and feedback to select them, but the evolving objects are training tasks. Its walk constructs a curriculum for subsequent solver parameter updates, connecting inference-based search to self-evolving training. 3 Direct Self-Evolving Optimization Existing self-evolving methods typically train two roles: a challenger (proposer) that generates a pool of training tasks, and a solver that tries to solve them. The challenger is trained to generate tasks that are more challenging for the solver, while the solver is trained to solve these tasks as correctly as possible. 3.1 From challenger training to a task-sampling target Let πθ _θ be the solver policy parameterized by θ∈ℝdθ ^d, and let π0(x) _0(x) be a fixed base task distribution induced by a frozen generator and its prompting protocol. Let rc(x,θ)r_c(x,θ) score the challenging-ness of task x to the current solver πθ _θ. In the existing framework such as Huang et al. (2025), the challenger training stage’s objective is maxϕx∼πϕrc(x,θt)−βKL(πϕ∥π0), _φ \E_x _φr_c(x, _t)- ( _φ\| _0) \, (1) where ϕφ denotes the parameters of the challenger model (an LLM), θt _t is the solver at round t, and the KL term with coefficient β>0β>0 controls departure from the base task law. In words, the challenger is trained to place more probability on tasks with higher challenge scores. However, training this model can be costly. Inspired by DPO (Rafailov et al., 2023), we consider the following distributional formulation of (1): qt∈argmaxq≪π0x∼qrc(x,θt)−βKL(q∥π0),q_t∈ _q _0 \E_x qr_c(x, _t)- (q\| _0) \, (2) The key observation is that qtq_t admits the following closed form: qt(x)=π0(x)exp(rc(x,θt)/β)Zθt,Zθt=x∼π0exp(rc(x,θt)/β).q_t(x)= _0(x) (r_c(x, _t)/β)Z_ _t, Z_ _t=E_x _0 (r_c(x, _t)/β). (3) This follows from the Gibbs variational identity (Donsker and Varadhan, 1975). Thus qtq_t reweights the base distribution, giving exponentially more mass to tasks that are more challenging for the current solver θt _t. As the solver changes, the reward changes and hence qtq_t changes. If we can directly draw tasks from qtq_t, then we are able to obtain useful training samples without fitting a new generator at each round. 3.2 LLM-based Metropolis–Hastings sampling However, directly sampling from qtq_t is intractable. We thus propose a sampling step based on LLM-based mutation and Metropolis–Hastings-style selection, LLM-MH for short. A frozen LLM (the generator inducing π0 _0, or even a closed-source model) first generates a seed pool. It then proposes revisions conditioned on each current task, while the frozen current solver evaluates the proposals. A Metropolis–Hastings-style selection rule uses these evaluations to accept or reject each revision. In detail, each round starts with ntn_t seed tasks from the frozen LLM. For each task x, we prompt the same LLM to produce a structurally different task x′x by some invertible operation: x′∼π0(⋅|x,mutation prompt) x _0(·|x,mutation prompt) Denote this prompted proposal kernel by KLLM(x′∣x)K_LLM(x x). Reversible task edits can make reverse moves possible, but do not guarantee symmetric proposal probabilities or reversibility with respect to π0 _0. The current solver then draws m fresh responses to x′x . We accept x←x′x← x with probability α^(x,x′)=min1,exp(r^c(x′,θt)−r^c(x,θt)β). α(x,x )= \1, ( r_c(x , _t)- r_c(x, _t)β ) \. (4) where r^c r_c estimates rcr_c from m solver rollouts. This is an approximate selection rule. Exact Metropolis–Hastings targeting qtq_t would use exact rewards and also include the factor π0(x′)KLLM(x∣x′)π0(x)KLLM(x′∣x). _0(x )K_LLM(x x ) _0(x)K_LLM(x x). This factor cancels only for a π0 _0-reversible proposal. The practical walk omits this intractable factor and uses noisy scores and finitely many steps. The exact-sampling assumption in Section 4 is therefore an idealization, not a claimed consequence of running this walk longer. In our experiments, r^c r_c is a triangular agreement score with a batch-level BLEU repetition penalty, so acceptance is evaluated on the whole batch (Appendix E.3). 3.3 Solver learning and the self-evolving loop For each generated task, obtain m responses from the current solver and form a pseudo-label z^t,i=m(yt,i,1:m) z_t,i=A_m(y_t,i,1:m), where mA_m is an aggregation rule. The solver remains frozen during task generation and labeling. Supervision is derived from these responses, not from answers supplied by the task generator. Let rs(x,θ,z)r_s(x,θ;z) denote the solver loss against fixed supervision z. Throughout, rcr_c is maximized and rsr_s is minimized. Given t=(xt,i,z^t,i)i=1ntD_t=\(x_t,i, z_t,i)\_i=1^n_t, define L^t(θ) L_t(θ) :=1nt∑i=1ntrs(xt,i,θ,z^t,i),J^t(θ):=L^t(θ)+τtℛs(πθ,πref,t), := 1n_t _i=1^n_tr_s(x_t,i,θ; z_t,i), J_t(θ):= L_t(θ)+ _tR_s( _θ, _ref,t), (5) where the solver regularizer, reference, and coefficient schedule are independent choices from the challenger’s regularization. The goal of solver training is to optimize (5) and find θt+1 _t+1 that satisfies J^t(θt+1)≤infθJ^t(θ)+εopt,t. J_t( _t+1)≤ _θ J_t(θ)+ _opt,t. (6) where εopt,t _opt,t is the optimization error. Algorithm 3.3 gives the DEO loop. After each update, θt+1 _t+1 defines the target and supervision for the next round. The generator requires only inference access throughout. Algorithm 1: Direct Self-Evolving Optimization (DEO) Input: initial solver θ0 _0, frozen generator π0 _0, mutation kernel KLLMK_LLM, rounds T, pool sizes nt\n_t\, sweeps LwL_w, temperature β, rollouts m, aggregation mA_m. Output: evolved solver θT _T. Main routine 1: for t=0,…,T−1t=0,…,T-1 do 2: Draw seed task pool with ntn_t tasks from frozen base model: St∼π0ntS_t _0^n_t. 3: St←S_t← LLM-MH (St,πθt,KLLM,β,Lw,mS_t, _ _t,K_LLM,β,L_w,m). 4: ⊳ Approximate the challenger target qtq_t; 5: for i=1,…,nti=1,…,n_t do 6: Draw m responses yt,i,1:my_t,i,1:m from πθt(⋅∣xt,i) _ _t(· x_t,i). 7: z^t,i←m(yt,i,1:m) z_t,i _m(y_t,i,1:m). 8: t←(xt,i,z^t,i)i=1ntD_t←\(x_t,i, z_t,i)\_i=1^n_t. 9: Train θt+1 _t+1 by minimizing J^t J_t in (5). 10: return θT _T. Sampling subroutine 1: function LLM-MH(S,πθ,KLLM,β,Lw,mS, _θ,K_LLM,β,L_w,m) 2: Estimate r^c(xi,θ) r_c(x_i,θ) for each xi∈Sx_i∈ S using m solver responses. 3: for ℓ=1,…,Lw =1,…,L_w do 4: for i=1,…,|S|i=1,…,|S| do 5: x′∼KLLM(⋅∣xi)x K_LLM(· x_i). 6: ⊳ Prompt the frozen LLM to mutate the current task. 7: Draw m responses from πθ(⋅∣x′) _θ(· x ). 8: Compute Δ←r^c(x′,θ)−r^c(xi,θ) ← r_c(x ,θ)- r_c(x_i,θ) 9: ⊳ Finite roll-out estimation of the challenge score. 10: u∼Uniform(0,1)u (0,1). 11: if u≤min1,exp(Δ/β)u≤ \1, ( /β)\ then 12: Replace xix_i in S by x′x and cache its score. 13: return S. 4 Theoretical Analysis: Idealized DEO Learns Distributionally Robust Reasoning What does the model learn from DEO? We answer this question by deriving a generalization bound for DEO. For ease of analysis, we assume that the tasks are sampled independently from the exact optimal challenger qtq_t instead of approximately sampling from it using LLM mutation and Metropolis selection. That is, at iteration t, St∼qtntS_t q_t^n_t. The pseudo-labels are still generated by the current solver θt _t via finite rollout and majority voting. Let Pθ(a∣x)P_θ(a x) be the canonical-answer probability, pθ(x)=maxaPθ(a∣x)p_θ(x)= _aP_θ(a x), and at(x)a_t(x) the population modal answer of solver θt _t. Instantiate rc(x,θ)=1−pθ(x),rs(x,θ,z)=1−Pθ(z∣x).r_c(x,θ)=1-p_θ(x), r_s(x,θ;z)=1-P_θ(z x). (7) The empirical solver uses J^t(θ)=L^t(θ)+‖θ−θt‖222ηt. J_t(θ)= L_t(θ)+ \|θ- _t\|_2^22 _t. (8) Thus the challenger seeks uncertainty, while the empirical solver minimizes disagreement with the majority of m old-solver responses. The theoretical instance uses KL challenger regularization and a decreasing proximal step size; it does not change the practical GRPO configuration. Define Φβ(θ)=βlogπ0e[1−pθ(x)]/β,κT=KL(qT∥π0). _β(θ)=β E_ _0e^[1-p_θ(x)]/β, _T=KL(q_T\| _0). (9) Write Φ∗=infθΦβ(θ) _*= _θ _β(θ) and E0=Φβ(θ0)−Φ∗E_0= _β( _0)- _*. Initial answer separation. We assume that the initial solver satisfies, π0 _0-almost surely, Pθ0(a0(x)∣x)−maxa≠a0(x)Pθ0(a∣x)≥γ0>0.P_ _0(a_0(x) x)- _a≠ a_0(x)P_ _0(a x)≥ _0>0. (10) Our local result derives later-round separation by bounding total parameter movement, under the initialization and error-budget condition below. Theorem 1 (Worst-case uncertainty bound). Run the idealized variant of Algorithm 3.3 in which LLM-MH is replaced by exact sampling, with (7)–(8), ntn_t conditionally iid tasks per round from qtq_t at each iteration, m fresh labeling responses per task, and exact empirical minimization (εopt,t=0 _opt,t=0). Assume at most K≥2K≥ 2 canonical answers, (10), and globally L-smooth fixed-answer losses with gradient norm at most G>0G>0. Assume conditional loss-class Rademacher complexity at most Cℱ/ntC_F/ n_t and the local PL inequality ‖∇Φβ(θ)‖2≥2μ[Φβ(θ)−Φ∗]\|∇ _β(θ)\|^2≥ 2μ[ _β(θ)- _*] on ‖θ−θ0‖≤r0:=γ0/(4G)\|θ- _0\|≤ r_0:= _0/(4G). Fix δ∈(0,1)δ∈(0,1) and use ηt=18/[μ(t+s)] _t=18/[μ(t+s)], with s and the initialization/sampling conditions specified in Appendix B.2. With probability at least 1−δ1-δ, Φβ(θT)≤BT _β( _T)≤ B_T for BT:=Φ∗+(E0T2)⏟decaying optimization error+~(1T2∑t=0T−1(t+1)2Cℱnt+TKe−mγ02/8)⏟statistical error.B_T:= _*+ O\! ( E_0T^2 )_decaying optimization error+ O\! ( 1T^2 _t=0^T-1 (t+1)^2C_F n_t+TKe^-m _0^2/8 )_statistical error. (11) The notation hides constants and confidence logarithms. Consequently, supD:KL(D∥π0)≤κTD[1−pθT(x)]≤bT:=1−e−minBT,1/β1−e−1/β. _D:\,KL(D\| _0)≤ _TE_D[1-p_ _T(x)]≤ b_T:= 1-e^- \B_T,1\/β1-e^-1/β. (12) The exact bound, including inexact-optimization errors, is given in (26). DEO learns distributionally robust reasoning. The bound controls the final solver’s answer uncertainty over the induced KL neighborhood of the base distribution π0 _0. The bound decreases in terms of the number of generated tasks and rollout budgets. The residual Φ∗ _* accounts for the model class’s irreducible uncertainty. What drives the bound? At the current solver, disagreement with its old modal answer touches and upper-bounds uncertainty. Therefore qt∇θrs(x,θ,at(x))|θ=θt=∇Φβ(θt).E_q_t . _θr_s(x,θ;a_t(x)) |_θ= _t=∇ _β( _t). (13) The advantage of sampling from the current challenger is that it follows the tilted uncertainty potential, whereas base-law sampling (sampling from base task distribution π0 _0) generally follows mean uncertainty. This is why DEO can yield distributionally robust guarantee. The proof is deferred to Appendix B. An initial modal-answer correctness assumption further converts this concentration guarantee into a bound on sampled-answer error. Corollary 1 (From concentration to answer accuracy). Under Theorem 1, let a∗(x)a^*(x) be the correct answer, let g(y)g(y) extract the answer of response y, and suppose Prπ0[a0(x)≠a∗(x)]≤ε0 _ _0[a_0(x)≠ a^*(x)]≤ _0. Set ΔT=BT−Φ∗ _T=B_T- _* and b∗=(1−e−Φ∗/β)/(1−e−1/β)b_*=(1-e^- _*/β)/(1-e^-1/β). On the same probability-1−δ1-δ event, supD:KL(D∥π0)≤κTPrx∼D,y∼πθT(⋅∣x)[g(y)≠a∗(x)]≲min1,ε0+b∗+b∗2β+ΔT+ΔT. aligned & _D:\,KL(D\| _0)≤ _T _x D,\,y _ _T(· x)[g(y)≠ a^*(x)] \! \1,\; _0+b_*+ b_*2β+ _T+ _T \. aligned (14) The explicit remainder is given in (49). If Φ∗=0 _*=0, the bound is ε0+Oβ(T−1/2) _0+O_β(T^-1/2), capped at one. If ε0=0 _0=0, the sharper bound is bT≤b∗+Oβ(ΔT)b_T≤ b_*+O_β( _T). Because the modal answer is preserved, the limiting error may exceed the initial modal-answer error ε0 _0: the guarantee describes robust sharpening of single-sample accuracy toward the initial majority-vote accuracy, not the acquisition of answers the initial model could not produce. The non-vanishing terms are an error certificate, not the minimum robust error over the model class, and hold over the induced radius κT _T rather than a fixed neighborhood. Proofs appear in Appendices B–C; approximate sampling is discussed in Appendix D. 5 Experiments We instantiate DEO for mathematical reasoning and organize the evaluation around four questions: whether challenger training is necessary for competitive performance, how the methods compare in computational cost, whether task refinement adds value and whether DEO can use an API-only generator. Implementation details are in Appendix E. Models and comparisons. We evaluate Qwen3-4B-Base, Qwen3-8B-Base, and OctoThinker-3B-Hybrid-Base through three self-evolving rounds, following the setup in Huang et al. (2025); Xia et al. (2025). We compare with R-Zero (Huang et al., 2025), which trains a challenger, and no walk, which samples tasks independently from the same frozen base model without LLM based Metropolis walk. DEO and no walk each start from 2000 candidates per round and share the labeling, filtering, and nominal solver-update configuration (Appendix F). Octo-3BQwen-4BQwen-8B00151530304545606029.429.430.530.552.152.112.612.614.614.620.620.6Hours(a) Wall-clock time Octo-3BQwen-4BQwen-8B0010102020303040405050606025.9425.9445.9345.9352.8852.8828.3028.3047.5747.5753.6053.60Accuracy (%)(b) Math AVG7 Accuracy R-Zero DEO Figure 2: Wall-clock time and iteration-three AVG7 accuracy of DEO and R-Zero. DEO uses 57.1%, 52.1%, and 60.5% less wall-clock time on OctoThinker-3B, Qwen3-4B, and Qwen3-8B, respectively, while achieving competitive accuracy. The Qwen R-Zero runs use larger native question pools (Section 5.1). Evaluation. The primary comparison reports iteration-three accuracy on MATH-500 (Hendrycks et al., 2021; Lightman et al., 2023), GSM8K (Cobbe et al., 2021), AMC (MAA, n.d.a), Minerva (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), and the 2024 and 2025 editions of AIME (MAA, n.d.b). AVG7 is their unweighted mean; HARD5 averages the last five benchmarks. All methods use the same archived boxed-answer grading protocol. Additional Omni-MATH (Gao et al., 2024) results are reported in Appendix F.2. We additionally measure transfer to MMLU-Pro (Wang et al., 2024), SuperGPQA (M-A-P Team et al., 2025), and BBEH (Kazemi et al., 2025) at all three scales, with their mean denoted AVG3. Table entries are percentages. The evaluation protocol, repeated-question handling, and checkpoint conventions are detailed in Appendix F. 5.1 DEO competes without training a challenger, using half the training time Table 1 compares each pretrained checkpoint with the three methods after the same number of self-evolving rounds. DEO reaches AVG7 scores of 47.57 at 4B and 53.60 at 8B, exceeding R-Zero by 1.64 and 0.72 percentage points. The corresponding HARD5 gains are 1.80 and 0.99 points. On OctoThinker-3B, DEO reaches 28.30 AVG7 versus 25.94 for R-Zero, with MATH-500 improving from 50.20 to 52.60. Thus competitive aggregate performance does not require updating the challenger’s parameters in these experiments. The advantage is not uniform over rounds: at 4B, R-Zero leads in iterations one and two (Table 6). On the same eight-H100 node, DEO reduces wall-clock time by 52–61% (Figure 2). No walk takes 5.9, 4.9, and 6.2 hours through iteration three on OctoThinker-3B, Qwen3-4B, and Qwen3-8B, respectively. General-domain transfer. At 4B, DEO improves AVG3 from 30.13 to 32.17, above no walk’s 31.89 but below R-Zero’s 32.59. At 8B, DEO attains the highest AVG3, 35.04, versus 34.62 for no walk and 34.30 for R-Zero. DEO has the highest BBEH score at both Qwen scales, while R-Zero leads MMLU-Pro and, on OctoThinker-3B, AVG3. Relative transfer gains thus depend on the model and benchmark. Table 1: Base checkpoints and iteration-three results (accuracy, %). DEO and no walk use 2000 initial candidates; the Qwen R-Zero runs retain their native larger pools. Implementation settings are in Appendix E. Base denotes the pretrained checkpoint before self-evolution. Bold marks the best score in each column within each base-model block, including ties. Method MATH 500 GSM8K AMC Minerva Olympiad Bench AIME 2024 AIME 2025 AVG7 HARD5 MMLU-Pro SuperGPQA BBEH AVG3 OctoThinker-3B-Hybrid-Base Base 44.60 61.49 17.42 19.12 14.67 6.67 0.00 23.42 11.58 8.20 5.33 0.97 4.83 No walk 48.40 73.24 25.00 27.21 18.67 0.00 0.00 27.50 14.18 17.73 9.70 3.30 10.24 R-Zero 50.20 73.09 12.50 27.21 15.26 3.33 0.00 25.94 11.66 19.10 10.47 4.00 11.19 DEO 52.60 74.75 25.08 26.47 15.85 3.33 0.00 28.30 14.15 17.97 10.83 3.70 10.83 Qwen3-4B-Base Base 72.60 81.90 48.40 36.80 39.10 10.00 16.60 43.63 30.18 55.13 27.07 8.20 30.13 No walk 75.80 91.51 52.58 47.43 39.11 10.00 3.33 45.68 30.49 58.73 28.67 8.27 31.89 R-Zero 74.00 91.70 47.30 51.80 40.40 9.60 6.70 45.93 31.16 59.37 29.53 8.87 32.59 DEO 76.80 91.43 57.34 45.96 41.48 6.67 13.33 47.57 32.96 58.10 28.50 9.90 32.17 Qwen3-8B-Base Base 70.00 88.93 52.50 43.38 40.74 10.00 3.33 44.13 29.99 60.23 31.33 9.87 33.81 No walk 81.20 93.78 64.53 54.41 45.19 10.00 13.33 51.78 37.49 62.17 32.17 9.53 34.62 R-Zero 81.40 93.86 62.81 53.68 45.04 16.67 16.67 52.88 38.97 62.83 31.63 8.43 34.30 DEO 81.20 94.16 62.81 54.41 45.93 20.00 16.67 53.60 39.96 62.23 32.17 10.73 35.04 5.2 LLM mutation refines the question pool The no-walk comparison holds the candidate-pool and solver-update configurations fixed. In Table 1, DEO improves AVG7 over no walk by 1.89 points at 4B, 1.82 points at 8B, and 0.79 points on OctoThinker-3B. The Qwen HARD5 gains are 2.47 points at both scales, while no walk retains a slight OctoThinker-3B HARD5 edge (14.18 versus 14.15). The robustness guarantee of Section 4 does not imply dominance on every benchmark. The walk adds generation and solver scoring (up to Lw=5L_w=5 proposals per task), so this comparison is matched in pool size and solver-update configuration, not total compute. The trajectory of LLM mutation Figure 3 illustrates three archived trajectories from an instrumented Qwen3-4B run with the same fixed-temperature recipe. The frozen LLM introduces an additional trigonometric composition in a derivative problem (A), changes modular exponentiation into a multiplicative-order problem requiring a minimality argument (B), and replaces prime-divisibility conditions with congruence solution families before adding another constraint (C). The recorded uncertainty scores increase from 0.220.22 to 0.440.44, 00 to 0.890.89, and 0.220.22 to 0.890.89, respectively; the last accepted transition in C leaves the score at 0.890.89. These selected examples show how refinement can add reasoning requirements while increasing the solver’s measured uncertainty. Table 2: Question counts for Qwen3-4B, iterations 1–3. In-band counts precede the pseudo-label and format checks; retained counts follow them. DEO and no walk each start with 2000 candidates. In-band Retained Method Iter 1 Iter 2 Iter 3 Iter 1 Iter 2 Iter 3 No walk 1029 970 1044 977 913 1002 DEO 1386 1358 1393 1246 1208 1228 LLM mutation increases in-band questions. Table 2 reports the archived Qwen3-4B counts under the training agreement-band criterion defined in Appendix E.5. In short, in-band counts how many generated tasks fall in the target uncertainty range (the “sweet spot”), excluding extremely hard and extremely easy questions. Pooling rounds one through three, the in-band fraction is 50.72% for no walk and 68.95% for DEO, with 357, 388, and 349 additional in-band candidates, respectively. The advantage remains after filtering: 3682 retained tasks in total versus 2892 for no walk. This shows that solver-guided refinement increases the operational yield of the candidate pool with a frozen generator. These counts use the same cached responses that drove selection, so they measure yield under the walk’s own score, not independently verified difficulty or label correctness. Figure 3: Accepted LLM mutations add reasoning requirements (Qwen3-4B, iteration one). A and B show the seed and first accepted mutation, with the added reasoning explained on the right; C shows both accepted mutations. Each shown mutation raises the uncertainty score and is accepted with probability one. 5.3 DEO works with an API-only generator Removing challenger training permits a task generator whose parameters are inaccessible. We replace the local Qwen3-4B generator with Claude Haiku 4.5 for seed generation and mutation. The Qwen3-4B solver still supplies the scores and pseudo-labels; Claude’s answers are never training targets. The task budget and training recipe are unchanged (Appendix F). At iteration three, the Claude-generator run reaches 49.60 AVG7 and 35.77 HARD5, compared with 47.57 AVG7 and 32.96 HARD5 for the local-generator DEO run. Gains are concentrated in Minerva, OlympiadBench, and AIME 2024; MATH-500 is unchanged and AMC decreases. Thus a frozen API-only model can improve the task distribution used to train a local solver. Table 3: DEO with a Qwen3-4B versus Claude Haiku 4.5 generator/mutator; Qwen3-4B solver, iteration three (%). Bold: better score, including ties. For the Claude one, the pseudo-labels for training were still produced by last-round solver θt _t. The Claude generator+mutator can notably boost the performance, while the R-Zero type of method cannot leverage that. Generator MATH 500 GSM8K AMC Minerva Olympiad Bench AIME 2024 AIME 2025 AVG7 HARD5 Qwen3-4B 76.80 91.43 57.34 45.96 41.48 6.67 13.33 47.57 32.96 Claude 76.80 91.51 52.50 50.37 42.67 20.00 13.33 49.60 35.77 6 Conclusion DEO uses the challenger objective to define a sampling target and approximates it with LLM mutation and Metropolis selection, training only the solver. Idealized DEO provably learns distributionally robust reasoning; in practice DEO matches challenger training at about half the wall-clock time, and can use an API-only generator. Limitations: The guarantees cover idealized DEO only, while in practical algorithm we use LLM based MH walk to simulate the sampling process. Hence there is a gap between theory and the algorithm we actually use. In addition, the walk operates per question within a round and has no cross-iteration memory, whereas R-Zero-style challengers evolve continuously across iterations. Future work includes memory mechanisms for the walk and applying DEO to agent training. (See more discussion in Appendix A). References Agrawal et al. (2025) L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab GEPA: reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457. External Links: Link Cited by: §2. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. External Links: Link Cited by: §5. Donsker and Varadhan (1975) M. D. Donsker and S. S. Varadhan Asymptotic evaluation of certain markov process expectations for large time, i. Communications on pure and applied mathematics 28 (1), p. 1–47. Cited by: §3.1. Gao et al. (2024) B. Gao, F. Song, Z. Yang, Z. Cai, Y. Miao, Q. Dong, L. Li, C. Ma, L. Chen, R. Xu, Z. Tang, B. Wang, D. Zan, S. Quan, G. Zhang, L. Sha, Y. Zhang, X. Ren, T. Liu, and B. Chang Omni-MATH: a universal olympiad level mathematic benchmark for large language models. External Links: 2410.07985, Link Cited by: §F.2, §5. Hastings (1970) W. K. Hastings Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57 (1), p. 97–109. External Links: Document, Link Cited by: §1. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun OlympiadBench: a challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008. External Links: Link Cited by: §5. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874. External Links: Link Cited by: §5. Huang et al. (2025) C. Huang, W. Yu, X. Wang, H. Zhang, Z. Li, R. Li, J. Huang, H. Mi, and D. Yu R-Zero: self-evolving reasoning LLM from zero data. arXiv preprint arXiv:2508.05004. External Links: Link Cited by: §F.1, Appendix G, §1, §2, §3.1, §5. Kazemi et al. (2025) M. Kazemi, B. Fatemi, H. Bansal, J. Palowitch, C. Anastasiou, S. V. Mehta, L. K. Jain, V. Aglietti, D. Jindal, P. Chen, N. Dikkala, G. Tyen, X. Liu, U. Shalit, S. Chiappa, K. Olszewska, Y. Tay, V. Q. Tran, Q. V. Le, and O. Firat BIG-Bench extra hard. arXiv preprint arXiv:2502.19187. External Links: Link Cited by: §5. Lee et al. (2025) K. Lee, I. Fischer, Y. Wu, D. Marwood, S. Baluja, D. Schuurmans, and X. Chen Evolving deeper LLM thinking. arXiv preprint arXiv:2501.09891. External Links: Link Cited by: §2. Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858. External Links: Link Cited by: §5. Lightman et al. (2023) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. arXiv preprint arXiv:2305.20050. External Links: Link Cited by: §5. Liu et al. (2024) Y. Liu, P. Sun, and H. Li Large language models as agents in two-player games. arXiv preprint arXiv:2402.08078. Cited by: §2. M-A-P Team et al. (2025) M-A-P Team, X. Du, Y. Yao, K. Ma, et al. SuperGPQA: scaling LLM evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739. External Links: Link Cited by: §5. MAA (n.d.a) MAA American mathematics competitions (AMC). Note: Mathematical Association of AmericaOfficial competition information; accessed September 24, 2026 External Links: Link Cited by: §5. MAA (n.d.b) MAA MAA invitational competitions: american invitational mathematics examination (AIME). Note: Mathematical Association of AmericaOfficial competition information; accessed September 24, 2026 External Links: Link Cited by: §5. Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark Self-Refine: iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651. External Links: Link Cited by: §2. Metropolis et al. (1953) N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller Equation of state calculations by fast computing machines. The Journal of Chemical Physics 21 (6), p. 1087–1092. External Links: Document, Link Cited by: §1. Novikov et al. (2025) A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog AlphaEvolve: a coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131. External Links: Link Cited by: §2. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Link Cited by: §1, §3.1. Romera-Paredes et al. (2024) B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi Mathematical discoveries from program search with large language models. Nature 625 (7995), p. 468–475. External Links: Document, Link Cited by: §2. Shinn et al. (2023) N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366. External Links: Link Cited by: §2. Wang et al. (2022) Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi Self-Instruct: aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560. External Links: Link Cited by: §2. Wang et al. (2024) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574. External Links: Link Cited by: §5. Xia et al. (2025) P. Xia, K. Zeng, J. Liu, C. Qin, F. Wu, Y. Zhou, C. Xiong, and H. Yao Agent0: unleashing self-evolving agents from zero data via tool-integrated reasoning. arXiv preprint arXiv:2511.16043. External Links: Link Cited by: §1, §2, §5. Xu et al. (2023) C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, Q. Lin, and D. Jiang WizardLM: empowering large pre-trained language models to follow complex instructions. arXiv preprint arXiv:2304.12244. External Links: Link Cited by: §2. Yao et al. (2023) S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan Tree of thoughts: deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. External Links: Link Cited by: §2. Yuan et al. (2024) W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. Weston Self-rewarding language models. arXiv preprint arXiv:2401.10020. External Links: Link Cited by: §2. Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman STaR: bootstrapping reasoning with reasoning. arXiv preprint arXiv:2203.14465. External Links: Link Cited by: §2. Zhao et al. (2025) A. Zhao, Y. Wu, T. Wu, Q. Xu, Y. Yue, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang Absolute zero: reinforced self-play reasoning with zero data. In Advances in Neural Information Processing Systems, Vol. 38. External Links: Link Cited by: §1, §2. Appendix Contents Appendix A Limitations Theory versus implementation. Theorem 1 and Corollary 1 cover idealized DEO: exact iid sampling from qtq_t, the monotone reward 1−pθ1-p_θ, fresh labeling responses, and a proximal L2 solver update. The implemented walk omits the base-density/proposal ratio, uses noisy scores and finitely many sweeps, uses a triangular score with a repetition penalty and an agreement filter, reuses walk responses as labels, and trains with GRPO. Appendix D shows how a sampler error dtd_t would enter the bound, but we do not estimate dtd_t for the LLM walk. The theorem also relies on local PL and a localization budget that we cannot verify for LLMs. What the guarantee certifies. Supervision comes from the solver’s own majority vote, and the proof keeps the initial modal answer fixed. The accuracy bound therefore cannot fall below the initial modal-answer error ε0 _0. It describes robust sharpening of single-sample accuracy toward majority-vote accuracy, not the discovery of answers the initial model could not produce. The robustness radius κT _T is induced by the final model rather than fixed in advance. Label and task quality. Pseudo-labels are unverified majority answers. Higher solver uncertainty can reflect ill-posed or ambiguous tasks as well as harder ones. The in-band statistics in Section 5.2 reuse the responses that drove selection, so they are not an independent measure of difficulty or correctness. Design. The walk operates per question within a round and has no cross-iteration memory, whereas R-Zero-style challengers evolve continuously across iterations. Future work includes memory mechanisms for the walk and applying DEO to agent training. Appendix B Proof of Theorem 1 B.1 Definitions and objective compatibility Let g extract a valid canonical answer, with a fixed tie-breaking rule. The answer law and the empirical plurality rule used in the analysis are Pθ(a∣x) P_θ(a x) =Pry∼πθ(⋅∣x)[g(y)=a],at(x)∈argmaxaPθt(a∣x), = _y _θ(· x)[g(y)=a], a_t(x)∈ _aP_ _t(a x), (15) z^t,i z_t,i =m(yt,i,1:m)∈argmaxa∑j=1mg(yt,i,j)=a. =A_m(y_t,i,1:m)∈ _a _j=1^m1\g(y_t,i,j)=a\. (16) Labels are fixed when optimizing the candidate solver. Its loss against the old population mode is rs(x,θ,at(x))=1−Pθ(at(x)∣x),Lt(θ)=x∼qtrs(x,θ,at(x)).r_s(x,θ;a_t(x))=1-P_θ(a_t(x) x), L_t(θ)=E_x q_tr_s(x,θ;a_t(x)). (17) The empirical L2 instance sets ℛs(πθ,πθt)=12‖θ−θt‖22,τt=ηt−1,R_s( _θ, _ _t)= 12\|θ- _t\|_2^2, _t= _t^-1, (18) in a fixed parameterization. This is separate from the KL regularizer on the challenger’s question law. The KL-optimal challenger has the explicit exponential-tilt form qβθ(x)=π0(x)e[1−pθ(x)]/βZθ,Zθ=π0e[1−pθ(x)]/β,qt=qβθt.q_β^θ(x)= _0(x)e^[1-p_θ(x)]/βZ_θ, Z_θ=E_ _0e^[1-p_θ(x)]/β, q_t=q_β _t. (19) For later use, define worst-case uncertainty by Uncκ(θ):=supD:KL(D∥π0)≤κx∼D[1−pθ(x)].Unc_κ(θ):= _D:\,KL(D\| _0)≤κE_x D[1-p_θ(x)]. (20) The exact tilt induces the radius and achieves the risk κβ(θ) _β(θ) =qβθ[1−pθ(x)]−Φβ(θ)β, = E_q_β^θ[1-p_θ(x)]- _β(θ)β, (21) Uncκβ(θ)(θ) _ _β(θ)(θ) =qβθ[1−pθ(x)]=Φβ(θ)+βκβ(θ). =E_q_β^θ[1-p_θ(x)]= _β(θ)+β _β(θ). (22) We prove these identities below. The ball is induced by the final model; it need not be fixed over training, and qβθTq_β _T need not equal the last training distribution qT−1q_T-1. All iterates in this proof are the empirical iterates of the idealized (exact-sampling) variant of Algorithm 3.3. Write Φ=Φβ = _β, cθ(x)=rc(x,θ)c_θ(x)=r_c(x,θ) and gt=∇Φ(θt)g_t=∇ ( _t). The losses evaluated against population majority labels and their proximal minimizers below are comparison quantities in the proof, not additional updates performed by the algorithm. Since maxaPθ(a∣x)≥Pθ(at(x)∣x) _aP_θ(a x)≥ P_θ(a_t(x) x), the definitions in (7) give rc(x,θ) r_c(x,θ) ≤rs(x,θ,at(x)) ≤ r_s(x,θ;a_t(x)) for every θ, every $θ$, rc(x,θt) r_c(x, _t) =rs(x,θt,at(x)), =r_s(x, _t;a_t(x)), ∇θrc(x,θt) _θr_c(x, _t) =∇θrs(x,θ,at(x))|θ=θt. = . _θr_s(x,θ;a_t(x)) |_θ= _t. (23) The last equality holds when the old modal answer is locally unique and the answer probabilities are differentiable. Thus the touching upper bound and gradient alignment follow from the specified objectives. The upper bound is valid even outside the initial neighborhood; later we prove that the actual iterates preserve the initial modal answer. Regularity. Both rsr_s and rcr_c lie in [0,1][0,1]. Assume that each fixed-answer loss rs(x,⋅,z)r_s(x,·;z) has an L-Lipschitz gradient of norm at most G, uniformly in x,zx,z. It follows that rc(x,⋅)r_c(x,·) is G-Lipschitz; on the initial-margin neighborhood below it is also differentiable. All question-wise conditions hold π0 _0-almost surely. Work in ℝdR^d with β>0β>0 fixed and L+G2/β>0L+G^2/β>0. The current optimal challenger therefore gives ∇Lt(θt) ∇ L_t( _t) =qt[∇θrs(x,θ,at(x))|θ=θt] =E_q_t [ . _θr_s(x,θ;a_t(x)) |_θ= _t ] (24) =qt∇θrc(x,θt)=∇Φβ(θt). =E_q_t _θr_c(x, _t)=∇ _β( _t). Sampling from qtq_t aligns the expected solver-update gradient with the exponentially tilted uncertainty potential. Sampling from π0 _0 instead generally aligns only with mean uncertainty under the base law. B.2 Initial margin and the localization budget For completeness, we retain inexact empirical optimization in the proof; the two-term display in Theorem 1 specializes to εopt,t=0 _opt,t=0. Fix δ∈(0,1)δ∈(0,1) and define δm0 _m^0 :=min1,(K−1)e−mγ02/8, := \1,(K-1)e^-m _0^2/8\, (25) et e_t :=4Cℱnt+2log(2T/δ)2nt+2δm0+εopt,t. := 4C_F n_t+2 (2T/δ)2n_t+2 _m^0+ _opt,t. Choose H=L+G2/βH=L+G^2/β, s≥max3,36H/μs≥ \3,36H/μ\, and ηt=18/[μ(t+s)] _t=18/[μ(t+s)]. The sampling/optimization budget is et≤ζ/(t+s)2e_t≤ζ/(t+s)^2 for some ζ>0ζ>0. Set C0=(s−1)E0+3ζC_0=(s-1)E_0+3ζ and ΛT=(T+s−2)(T+s−1) _T=(T+s-2)(T+s-1). The exact finite-sample bound is BT:= B_T:= Φ∗+(s−2)(s−1)E0ΛT+3ΛT∑t=0T−1(t+s−1)(t+s)et _*+ (s-2)(s-1)E_0 _T+ 3 _T _t=0^T-1(t+s-1)(t+s)e_t (26) ≤Φ∗+C0T+s−1. ≤ _*+ C_0T+s-1. For fixed s, the first excess term is O(E0/T2)O(E_0/T^2). The weighted finite-task errors are O~(T−2∑t<T(t+1)2/nt) O(T^-2 _t<T(t+1)^2/ n_t), and the common finite-rollout error contributes O(TKe−mγ02/8)O(TKe^-m _0^2/8). These give (11) for exact empirical ERM. An inexact solver adds precisely 3ΛT−1∑t<T(t+s−1)(t+s)εopt,t3 _T^-1 _t<T(t+s-1)(t+s) _opt,t; it also consumes the same error and localization budgets. Write Φ=Φβ = _β, Et=Φ(θt)−Φ∗E_t= ( _t)- _*, and ℬ0=θ:‖θ−θ0‖≤r0B_0=\θ:\|θ- _0\|≤ r_0\, where r0=γ0/(4G)r_0= _0/(4G). The global fixed-answer gradient bound implies that, for θ∈ℬ0θ _0 and any a≠a0(x)a≠ a_0(x), Pθ(a0(x)∣x)−Pθ(a∣x) P_θ(a_0(x) x)-P_θ(a x) ≥γ0−2G‖θ−θ0‖ ≥ _0-2G\|θ- _0\| (27) ≥γ0/2. ≥ _0/2. This establishes unique, unchanged modal answers throughout a fixed neighborhood without assuming anything about future iterates. Since the tilt density relative to π0 _0 is strictly positive, the same statement holds almost surely under every ideal challenger distribution. Theorem 1 assumes the PL inequality on ℬ0B_0, not an a priori margin condition along the trajectory. With H=L+G2/βH=L+G^2/β, s≥max3,36H/μs≥ \3,36H/μ\, C0=(s−1)E0+3ζC_0=(s-1)E_0+3ζ, its explicit additional condition is 722HC0μs−2+418ζμ(s−1)≤γ04G. 72 2HC_0μ s-2+4 18ζμ(s-1)≤ _04G. (28) It requires sufficiently small initial excess potential and error budget relative to the initial answer separation. It is a sufficient, conservative local-basin condition, not a consequence of initial margin alone. Its constants are independent of the horizon when E0,ζ,sE_0,ζ,s are fixed. The empirical optimizer is still unconstrained in ℝdR^d; no projection, frozen-label training rule, or extra population update is added. Labels are regenerated from the current solver at every round. Finite sampling budgets. For a prescribed T, put AT,δ=2Cℱ+log(2T/δ)/2A_T,δ=2C_F+ (2T/δ)/2. For ζ>0ζ>0, the following sufficient choices ensure et≤ζ/(t+s)2e_t≤ζ/(t+s)^2: nt n_t ≥36AT,δ2(t+s)4ζ2, ≥ 36A_T,δ^2(t+s)^4ζ^2, (29) m m ≥8γ02log+(6(K−1)(T+s)2ζ), ≥ 8 _0^2 _+\! ( 6(K-1)(T+s)^2ζ ), εopt,t _opt,t ≤ζ3(t+s)2. ≤ ζ3(t+s)^2. Here log+(u)=max0,logu _+(u)= \0, u\, and positive integer sample sizes are rounded upward. A common n can use the largest displayed task budget. The common rollout count m remains fixed within a finite-horizon run; it must increase with the horizon in an asymptotic family of runs. These are sufficient theoretical budgets, not the settings used in the experiments. The finite-m term is an expected label-error contribution; the proof does not require all sampled labels to be correct simultaneously. Why decreasing steps alone are insufficient. The crude bound ‖θt+1−θt‖≤Gηt\| _t+1- _t\|≤ G _t for exact empirical proximal minimization preserves a neighborhood if ∑tηt _t _t is small, but then it does not yield a vanishing optimization term through the PL contraction. Our schedule has ∑tηt=∞ _t _t=∞. It is the decay of the update direction, together with decreasing statistical errors, that makes the actual path length finite. This is proved next rather than assumed. B.3 Uniform sampling error and finite-rollout labels Let ℱt,m=(x,z)↦rs(x,θ,z):θ∈ℝdF_t,m=\(x,z) r_s(x,θ;z):θ ^d\. Its sampling law is a fresh question from qtq_t together with its m-response pseudo-label. Conditional on the pre-round history, let ℜnt(ℱt,m) R_n_t(F_t,m) denote its expected Rademacher complexity. Under the stated capacity bound, use the deterministic envelope εn,t=2Cℱ/nt+log(2T/δ)2nt. _n,t=2C_F/ n_t+ (2T/δ)2n_t. (30) The theorem assumes ℜnt(ℱt,m)≤Cℱ/nt R_n_t(F_t,m)≤ C_F/ n_t uniformly over rounds and admissible histories. This is a capacity assumption; bounded loss alone does not imply it. For the following one-step estimates, condition on a history with θt∈ℬ0 _t _0. The frozen old solver and qtq_t are then fixed. Each pair (xt,i,z^t,i)(x_t,i, z_t,i) is sampled independently from the law specified in Theorem 1. Let L~t,m(θ):=x∼qt,z^∼Ht,m(⋅∣x)rs(x,θ;z^), L_t,m(θ):=E_x q_t,\, z H_t,m(· x)r_s(x,θ; z), where Ht,mH_t,m is the law of the m-rollout pseudo-label. A conditional uniform deviation bound for the bounded loss class gives supθ|L^t(θ)−L~t,m(θ)|≤εn,t _θ| L_t(θ)- L_t,m(θ)|≤ _n,t with conditional failure probability at most δ/Tδ/T. A union bound yields a simultaneous event for all pre-exit rounds with probability at least 1−δ1-δ. The stopping argument below proves that no exit occurs. Independence across rounds is not required. For the theoretical disagreement loss, let at(x)a_t(x) denote the population modal answer of the old solver. For any competing answer a, define the m independent variables Wj=g(yj)=a−g(yj)=at(x)W_j=1\g(y_j)=a\-1\g(y_j)=a_t(x)\. They lie in [−1,1][-1,1] and have mean at most −γ0/2- _0/2. A bounded-sum tail inequality gives Pr(∑jWj≥0)≤e−mγ02/8 ( _jW_j≥ 0)≤ e^-m _0^2/8. Taking a union bound over the K−1K-1 competitors, including ties as possible errors, gives Pr[z^≠at(x)]≤δm0:=min1,(K−1)e−mγ02/8. [ z≠ a_t(x)]≤ _m^0:= \1,(K-1)e^-m _0^2/8\. (31) Since rs∈[0,1]r_s∈[0,1], replacing the population modal label by z z changes its expected loss by at most δm0 _m^0, uniformly in x,θx,θ. Consequently, with Lt(θ)=qtrs(x,θ,at(x))L_t(θ)=E_q_tr_s(x,θ;a_t(x)), supθ|L^t(θ)−Lt(θ)|≤εn,t+δm0=:bt. _θ| L_t(θ)-L_t(θ)|≤ _n,t+ _m^0=:b_t. (32) This controls disagreement with the old solver’s modal answer, without assuming that this answer is correct. B.4 Relating empirical ERM to a comparison function Define the comparison function and its minimizer Ft(θ):=Lt(θ)+‖θ−θt‖222ηt,zt:=argminθFt(θ).F_t(θ):=L_t(θ)+ \|θ- _t\|_2^22 _t, z_t:= _θF_t(θ). (33) The loss LtL_t is L-smooth and hence L-weakly convex. Thus FtF_t is M-strongly convex with M=ηt−1−L≥1/(2ηt)>0M= _t^-1-L≥ 1/(2 _t)>0. The bounded loss and coercive quadratic imply that ztz_t exists and is unique. The empirical algorithm does not compute ztz_t. The same quadratic appears in FtF_t and J^t J_t. Comparing the actual empirical optimizer to ztz_t on the simultaneous deviation event gives Ft(θt+1) F_t( _t+1) ≤J^t(θt+1)+bt ≤ J_t( _t+1)+b_t ≤J^t(zt)+εopt,t+bt ≤ J_t(z_t)+ _opt,t+b_t ≤Ft(zt)+2bt+εopt,t. ≤ F_t(z_t)+2b_t+ _opt,t. Therefore Ft(θt+1)−Ft(zt)≤et:=2εn,t+2δm0+εopt,t.F_t( _t+1)-F_t(z_t)≤ e_t:=2 _n,t+2 _m^0+ _opt,t. (34) Strong convexity also yields ‖θt+1−zt‖22≤2et/M\| _t+1-z_t\|_2^2≤ 2e_t/M. B.5 Descent along the empirical iterates For qβθ=π0ecθ/β/Zθq_β^θ= _0e^c_θ/β/Z_θ, direct substitution gives the Gibbs identity Dcθ−βKL(D∥π0)=Φ(θ)−βKL(D∥qβθ).E_Dc_θ- (D\| _0)= (θ)- (D\|q_β^θ). (35) Boundedness and the gradient envelope justify differentiation under the expectation at each visited iterate. The derived identity (23) gives gt=qt∇cθt=qt[∇θrs(x,θ;at(x))|θ=θt]=∇Lt(θt).g_t=E_q_t∇ c_ _t=E_q_t [ . _θr_s(x,θ;a_t(x)) |_θ= _t ]=∇ L_t( _t). For any candidate θ′=θt+Δθ = _t+ , the tilt also gives Φ(θ′)−Φ(θt)=βlogqte[cθ′(x)−cθt(x)]/β. (θ )- ( _t)=β E_q_te^[c_θ (x)-c_ _t(x)]/β. Since cθ′−cθt∈[−G‖Δ‖2,G‖Δ‖2]c_θ -c_ _t∈[-G\| \|_2,G\| \|_2], the inequality loges(W−W)≤s2(b−a)2/8 Ee^s(W-EW)≤ s^2(b-a)^2/8 for W∈[a,b]W∈[a,b] implies Φ(θ′)−Φ(θt) (θ )- ( _t) ≤qt[cθ′−cθt]+G22β‖Δ‖22 ≤E_q_t[c_θ -c_ _t]+ G^22β\| \|_2^2 ≤Lt(θ′)−Lt(θt)+G22β‖Δ‖22. ≤ L_t(θ )-L_t( _t)+ G^22β\| \|_2^2. (36) The touching upper bound justifies the second line even when the modal answer changes between iterates. Write Δt=θt+1−θt _t= _t+1- _t and Δt∗=zt−θt _t^*=z_t- _t, and set A:=12ηt−G22β,B:=1ηt+L,M:=1ηt−L.A:= 12 _t- G^22β, B:= 1 _t+L, M:= 1 _t-L. The step-size condition gives A≥1/(4ηt)A≥ 1/(4 _t), B≤3/(2ηt)B≤ 3/(2 _t) and M≥1/(2ηt)M≥ 1/(2 _t). Since Ft(θt+1)≤Ft(θt)+etF_t( _t+1)≤ F_t( _t)+e_t, (36) yields Φ(θt+1)≤Φ(θt)−A‖Δt‖22+et. ( _t+1)≤ ( _t)-A\| _t\|_2^2+e_t. The first-order condition at the comparison point is ∇Lt(zt)=−Δt∗/ηt∇ L_t(z_t)=- _t^*/ _t. Smoothness then gives ‖gt‖22≤B2‖Δt∗‖22≤2B2‖Δt‖22+4B2et/M.\|g_t\|_2^2≤ B^2\| _t^*\|_2^2≤ 2B^2\| _t\|_2^2+4B^2e_t/M. Consequently, Φ(θt+1) ( _t+1) ≤Φ(θt)−A2B2‖gt‖22+(1+2A/M)et ≤ ( _t)- A2B^2\|g_t\|_2^2+(1+2A/M)e_t ≤Φ(θt)−ηt18‖gt‖22+3et, ≤ ( _t)- _t18\|g_t\|_2^2+3e_t, using A/(2B2)≥ηt/18A/(2B^2)≥ _t/18 and 2A/M≤22A/M≤ 2. Substituting (34) gives Φ(θt+1)≤ ( _t+1)≤ Φ(θt)−ηt18‖∇Φ(θt)‖22 ( _t)- _t18\|∇ ( _t)\|_2^2 (37) +6εn,t+6δm0+3εopt,t. +6 _n,t+6 _m^0+3 _opt,t. B.6 Closing the initial-margin induction The preceding one-step bounds apply whenever θt∈ℬ0 _t _0; they do not require θt+1 _t+1 to be in the ball. We now prove that exit cannot occur, which avoids assuming the desired future margins. Let τexit=inft:θt∉ℬ0 _ exit= \t: _t _0\. The conditional uniform-deviation argument applies on each pre-exit history. A union bound over these histories gives an event of probability at least 1−δ1-δ on which all the required pre-exit bounds hold. Work on this event. For t<τexitt< _ exit, local PL and (37), with ηt=18/[μ(t+s)] _t=18/[μ(t+s)], give Et+1≤(1−2t+s)Et+3et.E_t+1≤ (1- 2t+s )E_t+3e_t. (38) If et≤ζ/(t+s)2e_t≤ζ/(t+s)^2, elementary induction gives Et≤C0t+s−1,C0=(s−1)E0+3ζ,E_t≤ C_0t+s-1, C_0=(s-1)E_0+3ζ, (39) up to and including a potential first exit: indeed, for k=t+sk=t+s, (1−2k)C0k−1+3ζk2≤C0k, (1- 2k ) C_0k-1+ 3ζk^2≤ C_0k, because C0≥3ζC_0≥ 3ζ. We also need an upper bound on movement, not merely descent. Smoothness of LtL_t in (36) gives, for every h∈ℝdh ^d, Φ(θt+h)≤Φ(θt)+⟨gt,h⟩+H2‖h‖2. ( _t+h)≤ ( _t)+ g_t,h + H2\|h\|^2. Taking h=−gt/Hh=-g_t/H and using the global lower bound Φ∗ _* yields ‖gt‖≤2HEt\|g_t\|≤ 2HE_t. This argument uses a touching upper model; it does not assume that Φ is globally smooth across modal ties. The first-order condition for the comparison minimizer ztz_t implies ‖zt−θt‖≤ηt(‖gt‖+L‖zt−θt‖)≤ηt‖gt‖+12‖zt−θt‖.\|z_t- _t\|≤ _t (\|g_t\|+L\|z_t- _t\| )≤ _t\|g_t\|+ 12\|z_t- _t\|. Together with ‖θt+1−zt‖≤2ηtet\| _t+1-z_t\|≤ 2 _te_t, this gives ‖θt+1−θt‖≤2ηt2HEt+2ηtet.\| _t+1- _t\|≤ 2 _t 2HE_t+2 _te_t. (40) Substituting the error and potential envelopes and summing any prefix, ∑t<j‖θt+1−θt‖ _t<j\| _t+1- _t\| ≤362HC0μ∑t≥01(t+s)t+s−1 ≤ 36 2HC_0μ _t≥ 0 1(t+s) t+s-1 +218ζμ∑t≥0(t+s)−3/2 +2 18ζμ _t≥ 0(t+s)^-3/2 ≤722HC0μs−2+418ζμ(s−1)≤r0. ≤ 72 2HC_0μ s-2+4 18ζμ(s-1)≤ r_0. The integral bounds use s≥3s≥ 3. If a first exit j≤Tj≤ T existed, all terms in this prefix would obey the bounds, contradicting ‖θj−θ0‖>r0\| _j- _0\|>r_0. Hence every empirical iterate stays in ℬ0B_0. Equation (27) now proves mode preservation and gap at least γ0/2 _0/2 for all rounds. The statistical and descent bounds therefore hold throughout the run. B.7 Unrolling the recursion and proving the risk bound For ΛT=(T+s−2)(T+s−1) _T=(T+s-2)(T+s-1), the products in (38) telescope: ∏j=t+1T−1(1−2j+s)=(t+s−1)(t+s)ΛT. _j=t+1^T-1 (1- 2j+s )= (t+s-1)(t+s) _T. Consequently the fully explicit finite-sample bound is Φ(θT)≤ ( _T)≤ Φ∗+(s−2)(s−1)E0ΛT _*+ (s-2)(s-1)E_0 _T (41) +3ΛT∑t=0T−1(t+s−1)(t+s)[2εn,t+2δm0+εopt,t]. + 3 _T _t=0^T-1(t+s-1)(t+s)[2 _n,t+2 _m^0+ _opt,t]. This is bounded by BTB_T in (26) under the assumed complexity envelope. Equation (39) gives BT≤Φ∗+C0/(T+s−1)B_T≤ _*+C_0/(T+s-1) as well, by applying the same scalar recursion to its deterministic right-hand side. No population optimization step or output averaging is used. We next establish the induced adversarial set and its risk. The Gibbs identity shows that, for any D with KL(D∥π0)≤κβ(θ)KL(D\| _0)≤ _β(θ), Dcθ≤Φ(θ)+βKL(D∥π0)≤Φ(θ)+βκβ(θ)=qβθcθ.E_Dc_θ≤ (θ)+ (D\| _0)≤ (θ)+β _β(θ)=E_q_β^θc_θ. The last equality follows directly from log(qβθ/π0)=cθ/β−logZθ (q_β^θ/ _0)=c_θ/β- Z_θ. Since qβθq_β^θ itself is feasible, it attains the worst-case value. This proves (21) and (22). For r∈[0,1]r∈[0,1], concavity and the values at zero and one give 1−e−r/β≥r(1−e−1/β).1-e^-r/β≥ r(1-e^-1/β). Furthermore, qβθe−cθ/β=1Zθ=e−Φ(θ)/β.E_q_β^θe^-c_θ/β= 1Z_θ=e^- (θ)/β. Taking expectations in the preceding scalar inequality therefore proves Uncκβ(θ)(θ)≤1−e−Φ(θ)/β1−e−1/β.Unc_ _β(θ)(θ)≤ 1-e^- (θ)/β1-e^-1/β. (42) The right-hand side increases with Φ(θ) (θ). Since 0≤Φ(θT)≤minBT,10≤ ( _T)≤ \B_T,1\, substitution gives (12) on the same probability-1−δ1-δ event. This completes the proof of Theorem 1. □ B.8 Scope without PL Before a possible exit, the one-step argument without PL gives only ∑t<Tηt‖∇Φ(θt)‖2≤18[Φ(θ0)−Φ∗]+54∑t<Tet, _t<T _t\|∇ ( _t)\|^2≤ 18[ ( _0)- _*]+54 _t<Te_t, (43) if all these iterates are separately known to remain in the neighborhood. It does not establish last-iterate convergence of Φ or preservation from the initial margin alone. Theorem 1 explicitly retains local PL and the localization budget for both conclusions. Appendix C Proof of Corollary 1 Modal-answer preservation. Theorem 1, proved using only the initial gap and the explicit localization condition, already gives aT(x)=a0(x)a_T(x)=a_0(x), π0 _0-almost surely. No new small-step condition or per-round margin assumption is needed for this corollary. It also does not require every finite-rollout pseudo-label to equal the population mode. Transferring uncertainty to truth. Define the initial error set A0=x:a0(x)≠a∗(x)A_0=\x:a_0(x)≠ a^*(x)\ and the final sampling error under a question law D by ErrD(θT):=Prx∼D,y∼πθT(⋅∣x)[g(y)≠a∗(x)]=D[1−PθT(a∗(x)∣x)].Err_D( _T):= _x D,\,y _ _T(· x)[g(y)≠ a^*(x)]=E_D[1-P_ _T(a^*(x) x)]. Mode preservation yields the pointwise inequality 1−PθT(a∗(x)∣x)≤1−pθT(x)+x∈A0.1-P_ _T(a^*(x) x)≤ 1-p_ _T(x)+1\x∈ A_0\. (44) For every D with KL(D∥π0)≤κTKL(D\| _0)≤ _T, Pinsker’s inequality (with natural logarithms) gives D(A0)≤π0(A0)+TV(D,π0)≤ε0+κT/2.D(A_0)≤ _0(A_0)+TV(D, _0)≤ _0+ _T/2. Taking expectations in (44) and applying Theorem 1 on its simultaneous event proves supD:KL(D∥π0)≤κTErrD(θT)≤min1,ε0+bT+κT/2. _D:\,KL(D\| _0)≤ _TErr_D( _T)≤ \1, _0+b_T+ _T/2\. (45) This intermediate bound transfers uncertainty to truth. If ε0=0 _0=0, finite KL implies D≪π0D _0, hence D(A0)=0D(A_0)=0 for every feasible D. The sharper error bound is then bTb_T. An explicit radius bound. By (22), Φβ≥0 _β≥ 0, and the theorem, βκT=qβθT[1−pθT(x)]−Φβ(θT)≤bT.β _T=E_q_β _T[1-p_ _T(x)]- _β( _T)≤ b_T. (46) Consequently, a bound involving only BTB_T, β, and ε0 _0 is infD:KL(D∥π0)≤κTDPθT(a∗(x)∣x)≥[1−ε0−bT−bT/(2β)]+. _D:\,KL(D\| _0)≤ _TE_DP_ _T(a^*(x) x)≥ [1- _0-b_T- b_T/(2β) ]_+. (47) Separating the irreducible term from the vanishing error. For v≥0v≥ 0, define gβ(v):=1−e−minv,1/β1−e−1/β,Cβ:=1β(1−e−1/β).g_β(v):= 1-e^- \v,1\/β1-e^-1/β, C_β:= 1β(1-e^-1/β). Then bT=gβ(BT)b_T=g_β(B_T) and b∗=gβ(Φ∗)b_*=g_β( _*), since 0≤Φ∗≤10≤ _*≤ 1. For 0<v<10<v<1, gβ′(v)=e−v/β/[β(1−e−1/β)]≤Cβg_β (v)=e^-v/β/[β(1-e^-1/β)]≤ C_β, and gβg_β is constant above one. It is therefore nondecreasing and CβC_β-Lipschitz on [0,∞)[0,∞). With ΔT=BT−Φ∗≥0 _T=B_T- _*≥ 0, this gives bT≤b∗+CβΔT.b_T≤ b_*+C_β _T. (48) Combining (45), (46), and a+b≤a+b a+b≤ a+ b for a,b≥0a,b≥ 0 yields supD:KL(D∥π0)≤κTErrD(θT) _D:\,KL(D\| _0)≤ _TErr_D( _T) ≤min1,ε0+b∗+b∗2β+CβΔT+CβΔT2β. ≤ \! \1,\; _0+b_*+ b_*2β+C_β _T+ C_β _T2β \. (49) This proves (14). Its remainder is explicit in the task counts, labeling rollouts, and optimization errors through ΔT=BT−Φ∗ _T=B_T- _* in (26). Under the theorem’s budgets, ΔT≤C0/(T+s−1) _T≤ C_0/(T+s-1), so the remainder is at most CβC0T+s−1+CβC02β(T+s−1)=Oβ(T−1/2). C_βC_0T+s-1+ C_βC_02β(T+s-1)=O_β(T^-1/2). For Φ∗=0 _*=0, b∗=0b_*=0 and only ε0 _0 remains in the limiting upper bound. For ε0=0 _0=0, the earlier bound ErrD(θT)≤bTErr_D( _T)≤ b_T removes the distribution-shift term entirely; then (48) gives b∗+CβΔTb_*+C_β _T, with an Oβ(1/T)O_β(1/T) remainder under the same budgets. These are bounds on the induced KL set for each T, not on a fixed-radius set, and b∗+b∗/(2β)b_*+ b_*/(2β) is a consequence of the optimal uncertainty potential, not an optimal true-error value. Under the base question law itself, no distribution-shift term is needed: Errπ0(θT)≤min1,ε0+bTErr_ _0( _T)≤ \1, _0+b_T\. Here ε0 _0 bounds the initial population modal-answer error, not its single-response error. The corollary concerns sampling accuracy, not a change in modal predictions, and it does not certify the practical triangular-reward GRPO implementation. □ Appendix D Extension to Approximate Question Sampling Theorem 1 assumes the exact optimal question law while retaining finite-m labels and finite-ntn_t empirical training. However in practice we can only estimate pθp_θ with finite roll-outs. Here we examine the bound under this realization. For a sampler law q~t q_t with TV(q~t,qt)≤dtTV( q_t,q_t)≤ d_t, retain conditional iid questions, fresh independent labels, and q~t≪π0 q_t _0 so that the initial almost-sure margin applies to its support. Bounded losses add at most dtd_t to (32); replace ete_t by et+2dte_t+2d_t everywhere, including the error schedule and localization condition. Under these augmented conditions, BTapprox:=BT+6ΛT∑t=0T−1(t+s−1)(t+s)dt.B_T^approx:=B_T+ 6 _T _t=0^T-1(t+s-1)(t+s)d_t. (50) The complexity is evaluated under the actual question–label law. Both qTq_T and the induced radius still refer to the ideal challenger of the final model. Sampler error consumes the same localization budget; it cannot simply be added to the final bound while ignoring its effect on preservation. Constant nonzero sampler error is not covered by the vanishing-error asymptotic rate. Expected finite-rollout reward. For the theoretical uncertainty reward rc=1−pθr_c=1-p_θ in (7), define ρθ,m(x)=[1−maxaP^m(a∣x)] _θ,m(x)=E[1- _a P_m(a x)], where P^m P_m uses m independent valid answers. Convexity of the maximum and the multinomial frequency error give 0≤rc(x,θ)−ρθ,m(x) 0≤ r_c(x,θ)- _θ,m(x) ≤‖P^m−Pθ‖∞ ≤E\| P_m-P_θ\|_∞ ≤1−∑aPθ(a∣x)2m≤1m. ≤ 1- _aP_θ(a x)^2m≤ 1 m. If a sampler exactly targets qt(m)∝π0eρθt,m/βq_t^(m) _0e _ _t,m/β, the oscillation of its log-density ratio with qtq_t is at most h=1/(βm)h=1/(β m), implying dt≤tanh(14βm).d_t≤ \! ( 14β m ). (51) To see this, a density ratio w∈[a,b]w∈[a,b] of mean one with b/a≤ehb/a≤ e^h satisfies TV≤(b−1)(1−a)/(b−a)TV≤(b-1)(1-a)/(b-a) by convexity of |w−1||w-1|. Maximizing over a≤1≤ba≤ 1≤ b gives tanh(h/4) (h/4). This particular bound assumes the exponential of the expected finite-rollout reward and independent labeling responses. Exponentiating one noisy score, reusing selection responses as labels, filtering the question law, or using a diversity-coupled batch needs further analysis. These conditions are not an error certificate for the practical walk in Appendix E. Appendix E Mathematical-Reasoning Implementation This appendix instantiates the general algorithm of Section 3 for the mathematical-reasoning experiments. The concrete reward, LLM mutation instructions, finite-step acceptance rule, and solver training choices are collected here. This practical configuration differs from the ideal-sampling 1−p1-p and L2 instance analyzed in Section 4. Table 4: Configuration of the local-generator DEO runs. The agreement rollouts and GRPO training rollouts serve different purposes. Setting Value Base models Qwen3-4B/8B-Base; OctoThinker-3B-Hybrid-Base Candidate pool per round N=2000N=2000, freshly generated Mutation sweeps Lw=5L_w=5 MH temperature Fixed β=0.1β=0.1 Agreement rollouts per evaluated question m=9m=9 Agreement sampling Temperature 1.01.0, top-p=1.0p=1.0, top-k=40k=40 Solver response limit 4096 tokens BLEU distance threshold / repetition weight 0.50.5 / λrep=10 _rep=10 Training agreement filter p^∈[0.3,0.8] p∈[0.3,0.8] Solver training GRPO; 5 training responses per prompt Solver KL reference / coefficient Original base model / 0.010.01 Additional sampling interventions None Reported checkpoints Iterations 1, 2 and 3 E.1 Concrete challenger reward For each question x, draw m responses from the frozen current solver. Let p^(x) p(x) be the largest frequency of an extracted answer, divided by m. We use r^c(x,θ)=1−2|p^(x)−12|. r_c(x,θ)=1-2 | p(x)- 12 |. (52) For a pool X=(x1,…,xN)X=(x_1,…,x_N), the score used for selection is r^c(X,θ)=∑i=1N[r^c(xi,θ)−λreprrep(xi,X)]+,rrep(xi,X)=|i(X)|N, r_c(X,θ)= _i=1^N[ r_c(x_i,θ)- _repr_rep(x_i;X)]_+, r_rep(x_i;X)= |N_i(X)|N, (53) where i(X)N_i(X) contains the BLEU-based neighbors of xix_i, including itself. We use m=9m=9 and λrep=10 _rep=10. The same notation rcr_c applies to individual questions and complete pools; hats indicate cached finite-rollout estimates. The answer-extraction and neighbor-count definitions follow below. E.2 Answer statistics and repetition The frozen pretrained model πbase _base induces the fixed seed question law π0 _0 through its generation prompt, predefined topic sampling, and output parsing. During each walk, the current solver is frozen. For responses ξ=(y1,…,ym)ξ=(y_1,…,y_m), let g(y)=⊥g(y)= denote an invalid answer extraction and define, for a≠⊥a≠ , ca=∑j=1mg(yj)=a,p^=maxa≠⊥cam,a^∈argmaxa≠⊥ca.c_a= _j=1^m1\g(y_j)=a\, p= _a≠ c_am, a∈ _a≠ c_a. (54) If no answer is extracted, set p^=0 p=0 and a^=⊥ a= . Invalid responses remain in the denominator m. Answers are grouped by extracted string, without full mathematical-equivalence clustering. Thus p p is empirical answer agreement, not verified correctness. The stored pseudo-label comes from the current solver, not from the mutator’s proposed solution. The symmetric neighbor matrix thresholds smoothed sentence-BLEU distances at 0.50.5. Its row counts include self-neighbors and give rrep(xi,X)=|i(X)|/N.r_rep(x_i;X)=|N_i(X)|/N. (55) These are local neighbor counts, not transitive cluster sizes. Clipping in (53) occurs before the sum. A mutation changes the selected question’s neighbor row and column and the penalties of all affected neighbors. Unchanged questions retain their uncertainty estimates. The reported settings are N=2000N=2000, m=9m=9, Lw=5L_w=5, β=0.1β=0.1, and λrep=10 _rep=10. E.3 LLM mutation and the implemented walk Each round starts with N seed questions from the frozen base LLM. For each current question xkx_k, we prompt this LLM to produce a structurally different question xk′x_k by choosing one of five operations: generalize the setting, compose interacting conditions, invert the task, change the requested quantity, or exchange a mathematical concept for its dual. The prompt discourages mere number substitution. Thus mutation is conditional text generation by an LLM, rather than a learned challenger update. Denote this prompted proposal kernel by KLLM(xk′∣xk)K_LLM(x_k x_k). The current solver draws m fresh responses to xk′x_k ; these provide both its uncertainty score and its proposed pseudo-label. Let X′X replace xkx_k by xk′x_k . We accept with probability α^(X,X′)=min1,exp(r^c(X′,θt)−r^c(X,θt)β). α(X,X )= \1, ( r_c(X , _t)- r_c(X, _t)β ) \. (56) The score difference includes all affected repetition penalties. Acceptance replaces the question and its cached answer statistics; rejection retains the old state. Unchanged questions reuse their responses. We perform Lw=5L_w=5 randomly ordered sweeps at fixed β=0.1β=0.1. This rule is motivated by the KL-optimal batch tilt Qt(X)∝Π0(X)exp(rc(X,θt)/β)Q_t(X) _0(X) (r_c(X, _t)/β), where Π0=π0⊗N _0= _0 N and rc(X,θt)=r^c(X,θt)r_c(X, _t)=E r_c(X, _t). The implemented walk is approximate: it uses noisy scores, omits the base-density/proposal ratio, and runs finitely many steps. Appendix E.4 gives the exact target and acceptance rule. No challenger training, CD correction, or temperature adaptation is used. Algorithm 2: LLM-based mutation and Metropolis selection Input: initial pool X0X^0, frozen current solver πθ _θ, frozen LLM mutation kernel KLLMK_LLM, temperature β, sweeps LwL_w, rollout count m. Output: final pool X and cached (p^i,a^i)i=1N( p_i, a_i)_i=1^N. 1: Set X←X0X← X^0; draw m responses per question and cache (p^i,a^i,r^c(xi,θ))( p_i, a_i, r_c(x_i,θ)). 2: for ℓ=1,…,Lw =1,…,L_w do 3: Draw a random permutation σ of 1,…,N\1,…,N\. 4: for k=σ(1),…,σ(N)k=σ(1),…,σ(N) do 5: Prompt the frozen LLM with xkx_k to sample xk′∼KLLM(⋅∣xk)x_k K_LLM(· x_k). 6: if xk′x_k is parseable then 7: Draw m fresh solver responses to xk′x_k ; compute its answer statistics. 8: Form X′←X[k←xk′]X ← X[k← x_k ] and compute Δ←r^c(X′,θ)−r^c(X,θ) ← r_c(X ,θ)- r_c(X,θ). 9: Accept X′X and its statistics with probability min1,eΔ/β \1,e /β\; otherwise retain the cached state. 10: return X and its cached answer statistics. E.4 Batch target and approximation scope Write r^c(X,θ,ξ) r_c(X,θ;ξ) when the rollout randomness needs to be explicit. The ideal batch reward is its finite-rollout expectation, rc(X,θ)=ξr^c(X,θ,ξ).r_c(X,θ)=E_ξ r_c(X,θ;ξ). (57) It is not the triangular function applied to population probabilities. With Π0=π0⊗N _0= _0 N, the batch challenger objective and optimizer are t(Q) _t(Q) =Qrc(X,θt)−βKL(Q∥Π0), =E_Qr_c(X, _t)- (Q\| _0), (58) Qt∗(X) Q_t^*(X) =Π0(X)erc(X,θt)/βZt,Zt=Π0erc(X,θt)/β. = _0(X)e^r_c(X, _t)/βZ_t, Z_t=E_ _0e^r_c(X, _t)/β. (59) Indeed, t(Q)=βlogZt−βKL(Q∥Qt∗).J_t(Q)=β Z_t- (Q\|Q_t^*). (60) This optimizes over unrestricted batch laws. The repetition term couples questions, so the optimum generally does not factorize. Let K be the batch proposal kernel induced by coordinate selection and KLLMK_LLM. Exact Metropolis–Hastings acceptance would require α∗(X,X′)=min1,Π0(X′)(X∣X′)Π0(X)(X′∣X)e[rc(X′,θt)−rc(X,θt)]/β.α^*(X,X )= \1, _0(X )K(X X ) _0(X)K(X X)e^[r_c(X , _t)-r_c(X, _t)]/β \. (61) The implementation omits the base-density/proposal ratio and uses cached noisy scores. The omitted factor cancels if K is reversible with respect to Π0 _0; this is not established for the prompted LLM mutator. Symmetry of K alone is insufficient. Also, er^c/β≠er^c/βEe r_c/β≠ e^E r_c/β in general, and a finite walk need not mix. These distinctions prevent interpreting the implemented walk as exact sampling from (59). No CD penalty, U-statistic correction, adaptive temperature, memory, or mutation bandit is used in the reported configuration. The frozen LLM uses the standard V1 prompt: it chooses one of generalize, compose, invert, change-objective, and dualize, conditioned on the current question. Proposals are generated in batches for throughput, with their score changes processed against the current pool. The prompt requests a number, algebraic expression, or finite-set answer, disallows proof and yes/no questions, and rejects mere number swapping as a mutation strategy. The exact generation, mutation, solver, and evaluation prompts are given in Appendix G. E.5 Format filtering and GRPO The final dataset contains only pairs with p^∈[0.3,0.8] p∈[0.3,0.8] and a nonempty pseudo-label that pass the implemented format checks. The code applies parsing/regex checks and an LLM VALID/INVALID judge for template text, prompt leakage, corrupted output, non-math or unsupported question types, and missing answers. The judge is explicitly instructed not to solve the problem; it is not a correctness verifier. For m=9m=9, the band keeps modal counts 33 through 77. The budget N=2000N=2000 is measured before filtering, so the retained training size varies by round. The GRPO implementation maximizes RGRPO(x,y;a^)=0.9 1grade(g(y),a^)=1+0.1rformat(y),R_GRPO(x,y; a)=0.9\,1\grade(g(y), a)=1\+0.1\,r_format(y), (62) where the grader checks answer equivalence and rformatr_format checks the inherited reasoning/final-answer format. Equivalently, it minimizes the response-averaged loss 0.9rs(x,θ;a^)+0.1y∼πθ(⋅∣x)[1−rformat(y)],0.9\,r_s(x,θ; a)+0.1\,E_y _θ(· x)[1-r_format(y)], (63) with rs(x,θ,a^)r_s(x,θ; a) interpreted using the same answer-equivalence grader. Here rs(x,θ;a^)=1−Pry∼πθ(⋅∣x)[grade(g(y),a^)=1].r_s(x,θ; a)=1- _y _θ(· x)[grade(g(y), a)=1]. KL regularization has coefficient 0.010.01 and uses the original base model as reference; this differs from the previous-iterate L2 instance of the theorem. Training draws fresh responses against the stored pseudo-labels and returns the checkpoint specified in the experimental protocol. Appendix F Experimental Protocol and Supplementary Results F.1 Experimental Protocol Models and question generation. We use Qwen3-4B-Base, Qwen3-8B-Base and OctoThinker-3B-Hybrid-Base. For the local-generator runs, the generator and initial solver use the same checkpoint. DEO starts each round with N=2000N=2000 parsed candidate questions and applies five mutation sweeps at a fixed β=0.1β=0.1. Initial questions are sampled with temperature 1.01.0 and top-p=0.95p=0.95; the standard V1 mutator uses temperature 1.11.1. Both generation calls have a 1536-token output limit. The generator is frozen at the original pretrained checkpoint. The challenger reward and BLEU repetition penalty are defined in Appendix E, with no additional intervention. Table 4 gives the shared settings. Section 5.3 additionally replaces the 4B generator and mutator with Claude while retaining the Qwen solver. Solver training and comparisons. The native training jobs use the inherited verl GRPO configuration with 64 prompts per rollout batch, five responses per prompt, learning rate 10−610^-6 and a configured maximum of 20 training steps per round. The runner exports and evaluates the global_step_15 checkpoint, and uses that checkpoint to initialize the next round. Thus the configured 20-step job budget should not be confused with the checkpoint index. Experiments run on an eight-H100 AzureML node, with separate resources for generation, solver scoring, training and evaluation. The no-walk ablation also generates 2000 candidates per round and uses the same solver labeling, filtering and training recipe, but sets Lw=0L_w=0. This comparison holds the candidate-pool and solver-update configurations fixed; it does not equalize the additional generation and scoring cost of the walk. We also include the archived R-Zero [Huang et al., 2025] runs for all three base models. R-Zero trains a separate challenger with RL and uses solver-produced modal answers to supervise solver training. Its solver uses the same nominal GRPO update budget and exported checkpoint index; we compare iterations one through three under the same evaluation protocol. The 2000-candidate restriction applies to DEO and no walk: the Qwen R-Zero runs retain their native, larger question-generation pools. For example, its first three 4B training sets contain 3939, 4312 and 4376 post-filter questions. R-Zero is therefore a method-level baseline, whereas no walk is the matched-pool ablation. This comparison does not establish equal total compute or attribute every difference solely to the walk. Evaluation. For mathematical benchmarks, we report the first three iterations from the archived runs using the same Claude Haiku boxed-answer grading protocol. AVG7 is the unweighted mean over MATH-500, GSM8K, AMC, Minerva, OlympiadBench, AIME 2024, and AIME 2025; HARD5 averages the last five. All entries are percentages. Main-table means and the differences discussed in the text are calculated from the reported per-benchmark percentages and rounded after calculation. Omni-MATH is evaluated separately in Appendix F.2 and is excluded from AVG7 and HARD5. The evaluation loader repeats AMC and AIME questions 32 times; these repetitions are not additional independent problems. We therefore report the observed scores without treating repeated rows as independent samples for statistical significance. These mathematical comparisons use a common grading protocol without checkpoint selection beyond iteration three. For general-domain transfer, we evaluate all three base models on MMLU-Pro, SuperGPQA and BBEH using fixed stratified subsets (target size 3000 per benchmark; seed 0), shared across checkpoints. Generation is greedy with at most 4096 new tokens. Multiple-choice answers are graded by option letter, and BBEH uses normalized exact match; this evaluation does not use an LLM judge. AVG3 is the unweighted mean of the three accuracies. This is our project’s evaluation protocol, rather than a reproduction of R-Zero’s original general-domain evaluation harness. Closed-source generation. The Claude run uses Claude Haiku 4.5 for both seed generation and mutation, with the Qwen3-4B solver unchanged. The API uses a generation temperature capped at 1.0, whereas the local V1 mutator uses 1.1. Both runs retain the same candidate budget, solver-scoring and labeling roles, filtering, training, and evaluation conventions. Claude responses do not supply the solver’s target labels. Candidate pools and retained datasets. The 2000-task budget is measured before the final filter. The archived DEO datasets contain 1246, 1208, and 1228 retained training tasks in the first three 4B rounds, and 1520, 1508, and 1510 at 8B. These are dataset sizes, not counts of distinct tasks guaranteed to be consumed by the fixed-length solver-training job. F.2 Additional evaluation on Omni-MATH We additionally evaluate the base and iteration-three checkpoints on Omni-MATH [Gao et al., 2024], using a fixed 2000-question subset and the same harness across all models and methods. These scores are reported separately from the seven-benchmark comparison in the main text and do not enter AVG7 or HARD5. Table 5: Omni-MATH accuracy (%) on the fixed 2000-question subset. All trained methods use iteration three. Bold marks the best trained-method score in each row. Model Base No walk R-Zero DEO OctoThinker-3B 9.50 12.80 13.10 12.80 Qwen3-4B 22.00 25.50 24.75 24.80 Qwen3-8B 23.55 26.55 26.65 27.00 Table 5 shows similar accuracy across the trained methods, with model-dependent rankings: no walk leads at 4B, DEO leads at 8B, and R-Zero leads on OctoThinker-3B. F.3 First-three-round trajectories Table 6 gives the archived trajectories. The iteration-three rows agree with the primary comparison; intermediate rankings can differ. Table 6: DEO, R-Zero and the no-walk ablation through iteration three. DEO and no walk use 2000 initial candidates per round; R-Zero uses its native larger question pool. DEO uses fixed β=0.1β=0.1, five sweeps and plain noisy acceptance, without CD or auxiliary interventions. Bold marks the best score among the three methods at each scale and iteration. AVG7 (↑ ) HARD5 (↑ ) Model Method Iter 1 Iter 2 Iter 3 Iter 1 Iter 2 Iter 3 4B No walk 45.49 47.73 45.68 30.02 32.84 30.49 R-Zero 46.46 48.13 45.93 31.38 33.96 31.16 DEO 45.09 47.71 47.57 29.02 33.18 32.96 8B No walk 51.91 50.64 51.78 38.04 36.28 37.49 R-Zero 51.96 51.70 52.88 37.88 37.68 38.97 DEO 52.01 53.39 53.60 38.22 40.02 39.96 Appendix G Prompt Templates We give the prompt templates used in the reported experiments. The panels separate message roles, following the presentation of R-Zero [Huang et al., 2025]. Text is reproduced from the implementation, with line wrapping adjusted for readability. Placeholder meanings are given below each panel; chat messages are serialized with the corresponding model’s chat template. The seed-generation and V1 mutation templates are shared by the local and Claude generators. No walk uses the same seed-generation and solver prompts, without mutation. G.1 Initial task generation Seed-generation prompt System Message: You are an expert competition-math problem setter. FIRST, in your private scratch-pad, think step-by-step to design a brand-new, non-trivial problem. Aim for a medium-to-hard competition level. CRITICAL RULES: 1. The final answer MUST be a SPECIFIC NUMBER, ALGEBRAIC EXPRESSION, or FINITE SET. 2. DO NOT generate "Prove that", "Show that", "Justify", or "Explain why" questions. 3. DO NOT generate questions that ask for True/False or Yes/No answers. 4. Ensure the problem has exactly one unambiguous final answer. 5. LIMIT YOUR SCRATCH-PAD THINKING TO UNDER 50 WORDS! Do not write out the full proof. THEN, output the problem and answer exactly in this format: <question> [Write the full problem statement here on one or more lines] </question> final_answer User Message: Generate one new, challenging reasoning question now. YOU MUST STRICTLY FOCUS ON: **topic**. Placeholder. topic is sampled uniformly from: Algebra: Polynomial roots and coefficients; Geometry: Triangle centers and circles; Geometry: 3D spatial geometry and volume; Geometry: Coordinate geometry and loci; Number Theory: Diophantine equations; Number Theory: Modular arithmetic and congruences; Combinatorics: Probability and expected value; and Calculus: Limits and derivatives. The generator’s boxed answer is not used as the solver’s training label. G.2 Solver responses Shared solver prompt System Message: Please reason step by step, and put your final answer within . User Message: problem Use. problem is the question being scored, used for GRPO training, or evaluated on a mathematical benchmark. The same system message is used for the m=9m=9 scoring responses, modal-answer pseudo-labeling, fresh training responses, and benchmark evaluation. Training compares fresh answers with the stored pseudo-label; the pseudo-label is not inserted into this prompt. G.3 LLM-based task mutation The standard V1 template below is used throughout the reported DEO runs. It asks the frozen LLM to select one structural mutation. Task-mutation prompt (V1) System Message: You are an expert competition-math problem setter. I will provide a seed problem. Your task is to generate a STRUCTURALLY DIFFERENT problem by applying ONE of the following mutation strategies. MUTATION STRATEGIES (pick exactly ONE that is NOT already exemplified by the seed): [A] GENERALIZE — Lift the structure: replace a specific constant with a parameter, OR raise the dimension (2D → 3D, single-variable → multivariable, single equation → system of equations). [B] COMPOSE — Add a NEW non-trivial second condition that interacts with the existing structure (NOT just a range bound like "with x > 0"). The two conditions together must create a real interaction. [C] INVERT — Given the original answer or output, ask the reader to recover an input or precondition. [D] CHANGE_OBJECTIVE — Change WHAT is being asked: e.g. "find x" → "count integer solutions", "compute" → "find the smallest n such that …", "find value" → "find sum of all such values". [E] DUALIZE — Swap to a dual concept: sum↔ , max↔ , area↔ , gcd↔ , addition↔ , distance↔ . CRITICAL RULES: 1. DO NOT just swap numbers. If the only edit is a digit change, you have FAILED — restart with another strategy. 2. DO NOT pick the strategy already exemplified by the seed. (e.g. if seed is already 3D, don’t pick GENERALIZE→ dimension.) 3. The final answer MUST be a SPECIFIC NUMBER, ALGEBRAIC EXPRESSION, or FINITE SET. 4. NO "Prove that", "Show that", "Justify", "True/False", or "Yes/No" questions. 5. LIMIT scratch-pad reasoning to UNDER 50 WORDS. Output format (STRICT — all three tags required): <strategy>A|B|C|D|E</strategy> <question> [the NEW mutated problem statement] </question> final_answer User Message: Here is the seed problem: seed Pick ONE mutation strategy from A,B,C,D,E and apply it now. Remember: number-swapping is FAILURE. Placeholder. seed is the current accepted question at the selected pool coordinate. Each proposal receives this question alone: solver responses, numerical scores, and weakness-memory notes are not included. Solver scoring and the acceptance decision occur after the proposal is generated. On rejection, the current question is retained for the next mutation attempt. G.4 Training-data format check After parsing and band filtering, the implementation uses the following completion prompt to screen the question and its solver-produced pseudo-label. Format-check prompt Raw Completion Prompt: You are a strict validity filter for competition-math problems. Mark a problem as INVALID ONLY IF you are CONFIDENT one of these specific issues applies: - Contains placeholder/template text like "[the NEW mutated problem statement]", "final_answer", or "[Write the full problem statement]". - Contains visible prompt leakage, role tags (Assistant:, User:), HTML/XML tags (<question>, <strategy>), or markdown code fences (```). - Contains corrupted Unicode garbage, randomly-mixed unrelated languages, or web/chat artifacts. - Is clearly NOT a math problem. - Asks for a proof, justification, true/false, yes/no, or open-ended explanation. - The proposed boxed answer is missing or empty. When in doubt, the problem is VALID. Do NOT solve the problem. Do NOT reject for being hard, unusual, or having minor formatting quirks. Problem: <<< q >>> Proposed boxed answer: ans Verdict ("VALID" or "INVALID"): Placeholders and decoding. q is the parsed question and ans is the solver’s modal-answer pseudo-label. This is sent to the solver endpoint as a raw completion, with temperature 0 and a six-token output limit. It checks format and question type, not mathematical correctness. G.5 Mathematical-answer grading For the reported seven-benchmark results, the boxed-answer regrader uses Claude Haiku 4.5 (claude-haiku-4-5-20251001), with temperature 0.1 and a four-token output limit. No system message is supplied. Boxed-answer regrading prompt User Message: Hi, there is an answer: model_answer,and the ground truth answer is: reference_answer,please check whether the answer is correct or not, and return the **only**Yes or No. Placeholders and use. model_answer is the last boxed answer in the model response, or the literal string (NO BOXED ANSWER FOUND IN MODEL RESPONSE) if none is found. reference_answer is the last boxed reference answer, falling back to the reference text. Only responses scored incorrect by the initial grader are rechecked; an affirmative verdict promotes their score. The regrader receives these answer strings, not the question or the full model reasoning. This evaluation judge is separate from the training-data format check above. G.6 General-domain evaluation MMLU-Pro, SuperGPQA and BBEH use the shared solver system message from Appendix G.2. Their user messages are shown below. General-domain evaluation prompts User Message: MMLU-Pro / SuperGPQA: question Options: options Answer with the letter of the single correct option, and put that letter within . User Message: BBEH: question Put your final answer within . Placeholders and use. question is the benchmark question (the input field for BBEH). options expands to all supplied choices, one per line, in the form A. option text, B. option text, and so on. Generation is greedy, with a 4096-token output limit. Multiple-choice outputs are graded by option letter and BBEH by normalized exact match; neither uses the boxed-answer LLM regrader. Omni-MATH uses the shared solver prompt with its problem statement and no additional user-message suffix.