Paper deep dive
Balancing Reference Guidance and Free Generation in Trajectory Rollouts for Reasoning RL
Hanyu Wang, Nakul Agarwal, Hossein Nourkhiz Mahjoub, Ehsan Moradi Pari, Makoto Fukushima, Jinghui Chen, Vaishnav Tadiparthi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/9/2026, 8:18:49 AM
Summary
The paper introduces Adaptive Reference Guidance (ARG), a method for training reasoning models using Reinforcement Learning with Verifiable Rewards (RLVR). ARG balances reference guidance and free generation by using a learned prefix selector to choose optimal reference prefixes for model continuation. This approach maximizes the probability of generating correct, distinct trajectories while minimizing divergence from the model's ideal distribution, applied within Group Relative Policy Optimization (GRPO) to handle all-failure groups.
Entities (7)
Relation Signals (5)
Adaptive Reference Guidance → appliedto → Group Relative Policy Optimization
confidence 95% · we apply it to all-failure groups in Group Relative Policy Optimization (GRPO)
Adaptive Reference Guidance → uses → Prefix Selector
confidence 95% · From this analysis, we learn a prefix selector... The resulting Adaptive Reference Guidance (ARG) constructs correct trajectories
Adaptive Reference Guidance → evaluatedon → Qwen3-4B
confidence 92% · Experiments on Qwen3-4B and Qwen3-8B... show that ARG achieves the highest aggregate pass@12
Adaptive Reference Guidance → evaluatedon → Qwen3-8b
confidence 92% · Experiments on Qwen3-4B and Qwen3-8B... show that ARG achieves the highest aggregate pass@12
Prefix Selector → optimizes → Reference Surprisal
confidence 85% · The selector maximizes J(ϕ)... which averages dtSt/S0... where St is the reference surprisal
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A verified reference solution provides a correct trajectory for training a reasoning model. Alternatively, a prefix of the reference can guide the model in generating a trajectory of its own. How much reference guidance should we provide? We study this question through prefix continuation, where the model continues from a reference prefix and keeps the resulting trajectory if it passes verification, falling back to the reference otherwise. Since both procedures produce correct trajectories, we compare their distributions with the ideal distribution, the model's own distribution conditioned on successful verification. For one continuation, we derive the KL divergence in closed form, which, up to a bounded term, decreases with the product of the probability of generating a different correct trajectory and the reference surprisal, the negative log probability of the reference suffix given the prefix. Since a longer prefix tends to raise the former but lowers the latter, continuation success alone does not determine the preferred amount of guidance. From this analysis, we learn a prefix selector shared across training questions from continuation outcomes, without estimating success probabilities or additional generation. The resulting Adaptive Reference Guidance (ARG) constructs correct trajectories within a fixed generation budget, and we apply it to all-failure groups in Group Relative Policy Optimization (GRPO). Experiments on Qwen3-4B and Qwen3-8B across five mathematical reasoning benchmarks show that ARG achieves the highest aggregate pass@12 among the evaluated methods with competitive average sampled accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2610.11128v1
- Canonical: https://arxiv.org/abs/2610.11128v1
Trouble viewing inline? Open PDF directly →
Full Text
139,276 characters extracted from source content.
Expand or collapse full text
Balancing Reference Guidance and Free Generation in Trajectory Rollouts for Reasoning RL Hanyu Wang* †thanks: Work done during an internship at Honda Research Institute USA. Affiliation: The Pennsylvania State University Affiliation: Honda Research Institute USA Nakul Agarwal Affiliation: Honda Research Institute USA Hossein Nourkhiz Mahjoub Affiliation: Honda Research Institute USA Ehsan Moradi Pari Affiliation: Honda Research Institute USA Makoto Fukushima Affiliation: Honda Research Institute Japan Jinghui Chen Affiliation: The Pennsylvania State University Vaishnav Tadiparthi Affiliation: Honda Research Institute USA Abstract A verified reference solution provides a correct trajectory for training a reasoning model. Alternatively, a prefix of the reference can guide the model in generating a trajectory of its own. How much reference guidance should we provide? We study this question through prefix continuation, where the model continues from a reference prefix and keeps the resulting trajectory if it passes verification, falling back to the reference otherwise. Since both procedures produce correct trajectories, we compare their distributions with the ideal distribution, the model’s own distribution conditioned on successful verification. For one continuation, we derive the KL divergence in closed form, which, up to a bounded term, decreases with the product of the probability of generating a different correct trajectory and the reference surprisal, the negative log probability of the reference suffix given the prefix. Since a longer prefix tends to raise the former but lowers the latter, continuation success alone does not determine the preferred amount of guidance. From this analysis, we learn a prefix selector shared across training questions from continuation outcomes, without estimating success probabilities or additional generation. The resulting Adaptive Reference Guidance (ARG) constructs correct trajectories within a fixed generation budget, and we apply it to all-failure groups in Group Relative Policy Optimization (GRPO). Experiments on Qwen3-4B and Qwen3-8B across five mathematical reasoning benchmarks show that ARG achieves the highest aggregate pass@12 among the evaluated methods with competitive average sampled accuracy. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) (Lambert et al., 2024) trains Large Language Models (LLMs) using feedback on their own generated responses (Guo et al., 2025; Team et al., 2025; Jaech et al., 2024). On difficult questions, however, the model may rarely generate a response that reaches the correct answer, leaving little feedback to learn from. When such a question comes with a reference solution written by humans or a stronger model, a straightforward approach is to train on it directly (Yan et al., 2026; Fu et al., 2026; Ma et al., 2026). However, the reference may differ substantially from the trajectories the model would produce itself, which can make it a poor learning target (Setlur et al., 2026; Qu et al., 2026). Since the model learns more easily from familiar responses (Ren et al., 2024), another line of work uses the reference as guidance so that the model generates correct trajectories by itself. Some methods transform the reference into hints (Zhang et al., 2026c; Liao et al., 2026), which requires additional LLM generation and may introduce bias or errors. Other methods, in contrast, let the model generate a continuation from a reference prefix, which is an initial segment of the reference. This approach uses the reference without modification and controls how much guidance the model receives through how much of the reference is revealed in the prefix. However, when generating a trajectory following a prefix, deciding how much to reveal involves a trade-off. Specifically, using a longer reference prefix, which provides more guidance, makes the continuation more likely to be correct. Yet the resulting trajectory contains more content the model would not generate itself, making learning harder. Using a shorter prefix keeps the trajectory closer to the model’s own generation, but increases the chance of failure. At the two extremes, revealing the entire reference reduces to training on the reference directly, while revealing nothing reduces to standard RLVR, which provides no learning signal on questions the model cannot yet solve. Balancing the two is therefore essential. The prefix should reveal just enough for the model to succeed, and leave the rest for the model to generate on its own. To analyze this trade-off, we first ask which trajectories are ideal to learn from. Ideally, these trajectories should be correct and generated entirely by the model. We therefore define the distribution of the model’s own correct samples as the ideal distribution. We show that the Kullback-Leibler (KL) divergence from the distribution of generated trajectories used for training to this ideal distribution decreases as dtStd_tS_t increases, up to a small bounded term. Here dtd_t is the probability that the model follows the reference’s prefix to generate a successful trajectory, and StS_t is the negative log probability of the remaining reference given the prefix. Intuitively, dtd_t reflects how much the prefix helps the model succeed, while StS_t indicates how unfamiliar the reference suffix is. Moreover, dtStd_tS_t captures both sides of the trade-off. A longer prefix tends to increase dtd_t, since more guidance makes success more likely, but it decreases StS_t, since less of the reference is left. As a result, a good prefix should maximize dtStd_tS_t. However, dtd_t is unknown, and estimating it for every prefix of every question is impractical. We avoid this estimation by learning a prefix selector, a distribution over a few guidance levels shared across questions. Since the selector only needs to work well on average, the outcome of each continuation already provides the feedback it needs, without any additional generation. These analyses lead to Adaptive Reference Guidance (ARG). Given a question and its reference, ARG samples guidance levels from the prefix selector, generates continuations from the resulting prefixes within a fixed budget, and falls back to the reference when no continuation succeeds. The same continuations also update the prefix selector, which therefore requires no additional generation. To examine how the accepted trajectories affect policy learning, we apply ARG to Group Relative Policy Optimization (GRPO) (Shao et al., 2024) for groups in which every sampled response fails, resulting in no learning signals. Empirically, on Qwen3-4B and Qwen3-8B (Yang et al., 2025) across five mathematical reasoning benchmarks, ARG achieves the highest aggregate pass@12pass@12 of 67.33%67.33\% and 70.00%70.00\%, respectively. Moreover, its average sampled accuracy remains competitive, with a small gap to the strongest baseline. Our contributions are threefold. • We analyze reference-prefix continuation through the KL divergence to the model’s own correct trajectories, revealing a trade-off between reference guidance and surprisal. • We derive an unbiased update of the prefix selector from the same continuations, adapting the guidance level across questions and during training without additional generation. • We develop ARG from this analysis and apply it to all-failure groups in GRPO, achieving the highest aggregate pass@12pass@12 across five math benchmarks on two model scales. 2 Preliminaries Reasoning with verifiable rewards. Let πθ(y∣x) _θ(y x) be a language model that generates a trajectory y for a question x drawn from the training distribution D. The trajectory includes the reasoning, the final answer, and the termination token. A verifier assigns a binary reward Racc(x,y)∈0,1R_acc(x,y)∈\0,1\, and we call y correct when Racc(x,y)=1R_acc(x,y)=1. Let CxC_x be the set of correct trajectories and pθ(x)=πθ(Cx∣x)p_θ(x)= _θ(C_x x) be the success probability. During training, θold _old denotes the parameters of the current model, which generates the samples at each training step. We use natural logarithms throughout. Reference-prefix continuation with fallback. Suppose that the question comes with a correct reference r∈Cxr∈ C_x. For a prefix length 0≤t<|r|0≤ t<|r|, we write r=ht⊕atr=h_t a_t, where the reference prefix hth_t contains the first t tokens of r, the reference suffix ata_t contains the remaining tokens, and ⊕ denotes token concatenation. The model samples a continuation Z∼πθ(⋅∣x,ht)Z _θ(· x,h_t), which forms the candidate trajectory Y~t=ht⊕Z Y_t=h_t Z. At t=0t=0, the prefix is empty and the model generates under the original question alone. We keep the candidate trajectory if it is correct and fall back to the reference otherwise, which gives the accepted trajectory Y=Y~t,Y~t∈Cx,r,Y~t∉Cx,Y∼Qt.Y= cases Y_t,& Y_t∈ C_x,\\ r,& Y_t∉ C_x, cases Y Q_t. (1) Thus, QtQ_t assigns all probability to correct trajectories for every prefix length. In contrast, training on the reference corresponds to the distribution δr _r, which assigns all probability to r. Ideal distribution. As discussed in Section 1, we take the model’s own correct trajectories as the ideal distribution. Formally, for pθ(x)>0p_θ(x)>0, sampling from πθ(⋅∣x) _θ(· x) and keeping only correct trajectories gives νθ(y∣x)=πθ(y∣x)y∈Cxpθ(x). _θ(y x)= _θ(y x)1\y∈ C_x\p_θ(x). (2) This distribution also has a direct connection to likelihood learning (Phan et al., 2023; Singh et al., 2023). Specifically, for a finite trajectory space and a differentiable policy with positive probabilities, Y∼νθ(⋅∣x)[∇θlogπθ(Y∣x)]=∇θlogpθ(x).E_Y _θ(· x) [ _θ _θ(Y x) ]= _θ p_θ(x). (3) In other words, learning the likelihood of trajectories drawn from νθ _θ increases the success probability in expectation at the current model. Finally, we measure how far a distribution Q of correct trajectories is from the ideal distribution by the Kullback-Leibler (KL) divergence DKL(Q∥νθ)=Y∼Q[logQ(Y)νθ(Y∣x)].D_KL(Q\| _θ)=E_Y Q [ Q(Y) _θ(Y x) ]. Appendix A.2 derives Equation (3) and explains this direction of KL. 3 Analyzing and Selecting Reference Prefixes In this section, we analyze how the prefix length affects the accepted trajectory and turn the analysis into a practical way of choosing prefixes. Section 3.1 derives the distance from the accepted trajectory to the ideal distribution, which identifies the preferred prefix of a question. Section 3.2 explains why this prefix is difficult to find during training, and Section 3.3 addresses both challenges by learning a prefix selector that performs well on average across questions. 3.1 Distance to the Ideal Distribution Fix a question x with reference r and the current model πθold _ _old, and suppress the conditioning on x and the parameters by writing π, C, p=π(C)p=π(C), and ν. We assume a finite trajectory space with 0<π(r)<10<π(r)<1. For a prefix length t, we define the successful replacement probability dtd_t and the reference surprisal StS_t as dt=Pr(Y~t∈C,Y~t≠r),St=−logπ(at∣ht).d_t= ( Y_t∈ C,\ Y_t≠ r ), S_t=- π(a_t h_t). (4) Here dtd_t excludes an exact copy of the reference, since the accepted trajectory is then still r. In other words, dtd_t is the probability that the model replaces the reference suffix with a different correct continuation, and StS_t measures how unfamiliar the reference suffix is to the model after the prefix. Theorem 1 (Distance to the ideal distribution). Under the setup above, for every prefix length t, DKL(Qt∥ν)=logpπ(r)⏟DKL(δr∥ν)−dtSt+(1−dt)log(1−dt).D_KL(Q_t\|ν)= pπ(r)_D_KL( _r\|ν)-d_tS_t+(1-d_t) (1-d_t). (5) Moreover, every t∗∈argmaxtdtStt^*∈ _td_tS_t satisfies DKL(Qt∗∥ν)≤mintDKL(Qt∥ν)+1/eD_KL(Q_t^*\|ν)≤ _tD_KL(Q_t\|ν)+1/e. Theorem 1 expresses the distance between QtQ_t and the ideal distribution ν in closed form with three terms. The first term equals that of training on the reference, which does not depend on the prefix. The third term is bounded within [−1/e,0][-1/e,0]. Therefore, the prefix affects the distance mainly through the second term, dtStd_tS_t. Intuitively, this product formalizes the trade-off in choosing the prefix length. A longer prefix never increases StS_t, since it leaves less of the reference for the continuation to replace. In contrast, a longer prefix can increase dtd_t by making the continuation easier to succeed. Consequently, both a nearly complete prefix and an empty prefix on a hard question make dtStd_tS_t close to zero and leave the distance close to DKL(δr∥ν)D_KL( _r\|ν), indicating learning from the reference alone. Moreover, a prefix with a higher success probability of the continuation need not give a smaller distance, as illustrated in Appendix A.5. Instead, maximizing dtStd_tS_t attains the smallest distance up to 1/e1/e by the second statement in the theorem, and we call ht∗h_t^* the preferred prefix of the question under the current model. Appendix A gives the proof. 3.2 Challenges in Finding the Preferred Prefix Although Theorem 1 identifies the preferred prefix, finding it during training is difficult. The reference surprisal StS_t is available for every prefix length from a single forward pass of the model on the reference. However, the successful replacement probability dtd_t is unknown, which leads to the following two challenges. Challenge 1: Variation across questions and training steps. Theorem 1 concerns a fixed question and a fixed model, and dtd_t depends on both. Therefore, the estimates of dtd_t for one question do not apply to another question. Furthermore, since dtd_t changes as the model is updated, the estimates obtained at one training step may become outdated at the next. As a result, the search for t∗t^* would have to be repeated for every question and every model update. Challenge 2: Limited sampling budget. Moreover, each such search is expensive. Identifying t∗t^* compares dtStd_tS_t across prefix lengths and therefore requires an accurate estimate of dtd_t at every prefix length. Since dtd_t is often small on hard questions, such an estimate requires many continuations even for a single prefix length, and this cost multiplies across prefix lengths. 3.3 Learning a Prefix Selector across Questions Both challenges arise from seeking the preferred prefix of each question at each training step. Instead, we learn a way of choosing prefixes that performs well on average over the questions that receive guidance, which addresses both challenges. Sharing across questions and training steps. Averaging over questions requires describing prefixes in the same way for every question, which the prefix length does not achieve for references of different lengths. Since Theorem 1 measures the replaced suffix by its surprisal rather than its length, we describe a prefix through the reference surprisal. Specifically, for a question x with reference rxr_x, let S0,x=−logπθold(rx∣x)S_0,x=- _ _old(r_x x) be the surprisal of the entire reference. A guidance level α in a set ⊂(0,1]A⊂(0,1] selects a prefix length tx(α)t_x(α) whose reference surprisal is approximately αS0,xα S_0,x. Thus, α is the fraction of the reference surprisal left to the continuation, where α=1α=1 gives the empty prefix. As with prefix lengths, the levels cannot all be tried within the sampling budget. Therefore, A contains only a few levels, and Section 4.1 specifies the mapping from levels to prefix lengths. We then learn a prefix selector ρϕ _φ, which is a distribution over A with parameters ϕφ shared across questions. Let guideD_guide denote the distribution of questions that receive guidance at the current training step, which Section 4.3 specifies for our application. For X∼guideX _guide and α∼ρϕα _φ, we write dX,αd_X,α, SX,αS_X,α, and QX,αQ_X,α for dtd_t, StS_t, and QtQ_t at the prefix length tX(α)t_X(α), and νX _X for the ideal distribution of X. The selector maximizes J(ϕ)=X∼guide,α∼ρϕ[dX,αSX,αS0,X],J(φ)=E_X _guide,\,α _φ [ d_X,αS_X,αS_0,X ], (6) which averages dtStd_tS_t over the questions and the sampled levels in units of the surprisal of the entire reference. This normalization keeps questions with long or unfamiliar references from dominating the average, and it does not change which prefix maximizes dtStd_tS_t within any question. Since the selector is shared across questions and updated at every training step, continuations on all questions and at all steps inform the same choice, while the selected prefix still adapts to each question through its reference surprisal. This addresses Challenge 1, and Appendix B.3 bounds the sharing cost. Learning without estimating dtd_t. The average objective also avoids the accurate estimates of dtd_t required in Challenge 2. For a continuation from the prefix hth_t, we define the selector feedback Wt=Y~t∈C,Y~t≠rStS0,W_t=1 \ Y_t∈ C,\ Y_t≠ r \ S_tS_0, (7) and write WX,αW_X,α for the feedback at the prefix length tX(α)t_X(α). In other words, a successful replacement receives the fraction of the reference surprisal left to the continuation, while a failure or an exact copy of the reference receives zero. Since the indicator has expectation dtd_t, the feedback satisfies [Wt]=dtStS0.E[W_t]= d_tS_tS_0. (8) Thus, the feedback of a single continuation is an unbiased estimate of its term in J. A single estimate is noisy, and noisy estimates cannot reliably rank the prefix lengths of one question. However, since J is an average that is linear in dX,αd_X,α, the errors of single continuations cancel when pooled over many questions. Moreover, since A contains only a few levels and the selector keeps a positive probability on every level, every level receives feedback from many questions, which addresses Challenge 2. The following theorem shows that J still measures the distance in Theorem 1 and that the selector can be learned from this feedback. Theorem 2 (Learning the prefix selector from feedback). Fix the current model, the references, and guideD_guide, where every question satisfies the setup of Section 3.1. Let ℒ(ϕ)L(φ) and ℒrefL_ref be the expectations of DKL(QX,α∥νX)/S0,XD_KL(Q_X,α\| _X)/S_0,X and DKL(δrX∥νX)/S0,XD_KL( _r_X\| _X)/S_0,X over X∼guideX _guide and α∼ρϕα _φ. Then J(ϕ)≤ℒref−ℒ(ϕ)≤J(ϕ)+X[1eS0,X].J(φ) _ref-L(φ)≤ J(φ)+E_X [ 1eS_0,X ]. (9) Moreover, for a differentiable prefix selector with positive probabilities and a baseline b fixed before sampling, [(WX,α−b)∇ϕlogρϕ(α)]=∇ϕJ(ϕ).E [(W_X,α-b) _φ _φ(α) ]= _φJ(φ). (10) The first statement shows that, compared with training on the reference, the selector reduces the average normalized distance by at least J, where each distance is divided by the surprisal of the entire reference. The gap is at most X[1/(eS0,X)]E_X[1/(eS_0,X)], which is small when no reference is highly likely under the model. Therefore, maximizing J pursues the goal of Theorem 1 across questions. The second statement shows that raising the probability of the levels whose feedback exceeds the baseline b moves the selector toward a larger J on average, where b introduces no bias and serves to reduce the variance. Since these continuations also produce the accepted trajectories, the update requires no additional generation. Appendix B proves Theorem 2 and extends it to batches, changing models, and randomly sampled levels. 4 ARG: Adaptive Reference Guidance We now build ARG on Theorems 1 and 2. Section 4.1 describes how ARG generates accepted trajectories for a question, and Section 4.2 describes how the same continuations update the prefix selector. Neither step depends on a particular reinforcement learning algorithm. The algorithm only determines the questions that receive guidance and how the model learns from the accepted trajectories, and Section 4.3 specifies both for GRPO. 4.1 Generating Accepted Trajectories Consider a question x that receives guidance, with reference r. ARG first scores r once under the current model πθold _ _old, which gives the reference surprisal StS_t for every prefix length. Each guidance level α∈α then maps to a prefix length tx(α)t_x(α). Specifically, α=1α=1 gives the empty prefix. For α<1α<1, we take the latest prefix length whose reference surprisal is at least αS0α S_0 and move it to a nearby sentence or formula boundary, so that the prefix ends at a natural break. Appendix E.2 gives the boundary rules. The prefix selector assigns a logit ϕα _α to each level and samples from ρϕ(α)=(1−εsel)exp(ϕα)∑α′∈exp(ϕα′)+εsel||, _φ(α)=(1- _sel) ( _α) _α ( _α )+ _sel|A|, (11) where 0<εsel<10< _sel<1 keeps every level observed as the model changes. Within a fixed budget of K continuations per question, we independently sample αj∼ρϕ _j _φ for j=1,…,Kj=1,…,K. For the resulting prefix length tj=tx(αj)t_j=t_x( _j), we sample a continuation Zj∼πθold(⋅∣x,htj)Z_j _ _old(· x,h_t_j) and verify the candidate trajectory Y~j=htj⊕Zj Y_j=h_t_j Z_j. Every correct candidate trajectory different from r can serve as an accepted trajectory. If there is none, ARG falls back to the reference. Therefore, the question always receives a correct trajectory. When the number of accepted trajectories is limited, as in our application, we prefer candidate trajectories that keep less of the reference. Specifically, a correct candidate trajectory from the empty prefix keeps none of the reference. Indeed, it is generated entirely by the model and follows the ideal distribution restricted to trajectories other than r. Therefore, we accept these candidate trajectories first. Among candidate trajectories from nonempty prefixes, we prefer the shortest prefix, since the surprisal of the prefix, −logπθold(ht∣x)=S0−St- _ _old(h_t x)=S_0-S_t, never decreases with t. Finally, Theorem 1 concerns a single continuation. For accepted trajectories from several continuations, Appendix C.2 gives a sufficient condition under which the distance remains no larger than that of training on the reference. 4.2 Updating the Prefix Selector Every continuation provides the selector feedback Wj=WtjW_j=W_t_j in Equation (7), whether or not its candidate trajectory is accepted. Since the feedback does not depend on which trajectories are accepted, the second statement of Theorem 2 holds under any acceptance rule, while the first statement, like Theorem 1, concerns a single continuation. We collect continuations for all questions in a batch while keeping the current model and the prefix selector fixed. Let U index these continuations, with sampled levels αj _j and selector feedback WjW_j. For a nonempty U, we update the prefix selector by ϕ←ϕ+ηsel∑j∈(Wj−b)∇ϕlogρϕ(αj),φ←φ+ _sel _j (W_j-b) _φ _φ( _j), (12) where ηsel>0 _sel>0 is the step size and b is a running average of past feedback fixed before collection. By the second statement of Theorem 2, the update follows the gradient of J in expectation. Appendix B gives the baseline update and derives the expected update as the model and the questions change during training. 4.3 Applying ARG to GRPO GRPO (Shao et al., 2024) samples a group of G=8G=8 responses for each question and compares their rewards within the group. When every response in a group is incorrect, the group provides no signal for improving correctness, since all correctness rewards are identical. In such an all-failure group, adding a correct trajectory restores the learning signal. Therefore, we apply ARG to all-failure groups, using K=8K=8 continuations. These questions define guideD_guide. We insert at most four accepted trajectories into each group, which keeps at least four original failures and permits a balanced group, motivated by prior work on intermediate pass rates (Bae et al., 2026; Liu et al., 2026b). Following the preference in Section 4.1, we first accept correct candidate trajectories from the empty prefix. If fewer than four are accepted, we additionally accept at most one from a nonempty prefix, which limits repeated reference content in the same group. If no candidate trajectory is accepted, we insert the reference once as fallback. Each inserted trajectory replaces an original failure at a uniformly chosen position. We then recompute the group advantages AiA_i from the final correctness and format rewards, as specified in Appendix E.3. The model learns each inserted trajectory under the original question through its complete likelihood, including the prefix. This matches Theorem 1, which measures the complete accepted trajectory under the original question. Let ℐI index the inserted trajectories (xi,yi)(x_i,y_i) and let T be the total number of tokens in the loss. Their contribution to the gradient is ∇θℒinserted=−1T∑i∈ℐAi∇θlogπθ(yi∣xi), _θL_inserted=- 1T _i A_i _θ _θ(y_i x_i), (13) where the trajectories, advantages, and token count are fixed during differentiation. The remaining original responses use the clipped GRPO loss with the same denominator. Moreover, for binary rewards, Appendix D shows that inserting Y∼QY Q into an all-failure group gives an expected update proportional to the gradient of logpθ1−pθ−DKL(Q∥νθ) p_θ1-p_θ-D_KL(Q\| _θ) at the current model. The gradient of the second term vanishes when Q is the ideal distribution, in which case the update only increases the odds of success. This connects the distance in Theorem 1 to the GRPO update. After collecting continuations, we update the prefix selector by Equation (12) and take one policy update on the final groups. Algorithm 1 summarizes the training procedure. 5 Experiments 5.1 Experimental Setting Data and models. We use DeepMath-103k with offline reference solutions (He et al., 2026) as our training set. We use Qwen3-4B and Qwen3-8B with thinking mode disabled (Yang et al., 2025) as our base LLMs. For all methods, we evaluate on five mathematical reasoning benchmarks, AIME 2024, AIME 2025, AIME 2026, HMMT February 2025, and HMMT November 2025 (Balunovic et al., 2026; Dekoninck et al., 2026). Appendix E gives further details. Baselines and evaluation. We compare against the initial Qwen model, vanilla supervised fine-tuning (SFT), GRPO (Shao et al., 2024), OPSD (Zhao et al., 2026), SAGE (Liao et al., 2026), BREAD (Zhang et al., 2026d), and LUFFY (Yan et al., 2026). During evaluation, we draw 12 responses per question. pass@12pass@12 measures whether a question has at least one correct response, while avg@12avg@12 measures the fraction of correct responses. We use pass@12pass@12 as the primary metric. Model Method AIME24 AIME25 AIME26 HMMT_Feb HMMT_Nov Avg. Qwen3-4B Base 50.00 43.33 43.33 23.33 33.33 38.67 SFT 63.33 63.33 66.67 36.67 36.67 53.33 GRPO 76.67 70.00 73.33 53.33 46.67 64.00 OPSD 56.67 36.67 40.00 40.00 40.00 42.67 SAGE 73.33 70.00 70.00 43.33 56.67 62.67 BREAD 80.00 73.33 73.33 50.00 56.67 66.67 LUFFY 80.00 73.33 70.00 43.33 56.67 64.67 ARG (ours) 80.00 73.33 73.33 43.33 66.67 67.33 Qwen3-8B Base 46.67 43.33 53.33 16.67 46.67 41.33 SFT 80.00 70.00 73.33 46.67 56.67 65.33 GRPO 80.00 63.33 70.00 46.67 53.33 62.67 OPSD 66.67 60.00 50.00 36.67 43.33 51.33 SAGE 80.00 70.00 70.00 50.00 60.00 66.00 BREAD 80.00 66.67 66.67 53.33 70.00 67.33 LUFFY 76.67 76.67 73.33 46.67 63.33 67.33 ARG (ours) 83.33 73.33 73.33 56.67 63.33 70.00 Table 1: Main experimental results in pass@12pass@12 percentages. Avg. is the mean over the five benchmarks. Within each model, bold and underline mark the best and second-best values in each column. 5.2 Results Table 1 evaluates ARG through its application to GRPO. ARG achieves the highest aggregate pass@12pass@12 at both model scales, reaching 67.33%67.33\% on Qwen3-4B and 70.00%70.00\% on Qwen3-8B. Compared with GRPO, these results improve pass@12pass@12 by 3.333.33 and 7.337.33 percentage points, respectively. ARG also exceeds the reference-informed baselines in aggregate pass@12pass@12, which use reference solutions as off-policy responses in LUFFY, as prefixes in BREAD, and as hints in SAGE (Yan et al., 2026; Zhang et al., 2026d; Liao et al., 2026). Thus, ARG solves the largest fraction of evaluation questions under the 12-response budget. For avg@12avg@12, ARG improves over GRPO on both models and remains competitive with the strongest baseline, as discussed in Appendix F. Sections 5.3 and 5.4 examine the contributions of the continuations and the prefix selector. 5.3 Ablations We examine the contribution of the continuations, compare with unguided resampling, and evaluate the surprisal weighting in the selector feedback. Reference insertion. For each all-failure group, we replace a random failure in the original group with the reference without any extra generation. This ablation corresponds to training on the reference in Theorem 1. Table 2(a) shows that reference insertion achieves 56.67%56.67\% in pass@12pass@12 and 28.44%28.44\% in avg@12avg@12 on Qwen3-4B, compared with 67.33%67.33\% and 41.11%41.11\% for ARG. These results support using the reference to guide continuations and reserving the reference itself for fallback. Resampling. We allocate the same eight continuations to the empty prefix for each all-failure group. Correct candidate trajectories replace original failures, and groups with no correct candidate trajectory remain unchanged. Thus, this comparison evaluates ARG against additional sampling without reference guidance or fallback. Table 2(a) shows that ARG improves pass@12pass@12 from 60.00%60.00\% to 67.33%67.33\% and avg@12avg@12 from 34.00%34.00\% to 41.11%41.11\% on Qwen3-4B. During their respective training runs, candidate trajectories are accepted in 66.0%66.0\% of all-failure groups with ARG and only 32.5%32.5\% with resampling. More details are shown in Appendix G.2. Without surprisal weighting. We replace the selector feedback in Equation (7) with the successful replacement indicator. The guidance levels, surprisal-based prefix construction, selector update rule, and acceptance rule remain unchanged. Table 2(b) shows that removing the weighting lowers both evaluation metrics on both models, with pass@12pass@12 decreasing by 4.004.00 and 6.006.00 percentage points on Qwen3-4B and Qwen3-8B, respectively. These results support the surprisal weighting in the selector feedback. Section 5.4 examines the selector feedback and prefix selection under ARG, while Appendix G.3 compares the prefix selection with that of the ablation. Method pass@12pass@12 avg@12avg@12 Reference insertion 56.67 28.44 Resampling 60.00 34.00 ARG 67.33 41.11 (a) Sources of accepted trajectories on Qwen3-4B Method Qwen3-4B Qwen3-8B pass@12pass@12 avg@12avg@12 pass@12pass@12 avg@12avg@12 w/o weighting 63.33 34.67 64.00 31.89 ARG 67.33 41.11 70.00 42.28 (b) Surprisal weighting on both models Table 2: Ablation results averaged over the five benchmarks. Resampling uses eight continuations from the empty prefix without reference fallback. The weighting ablation removes St/S0S_t/S_0 from the selector feedback. 5.4 Training Dynamics Figure 1 examines Qwen3-8B training dynamics through successful replacement rates, selector feedback, and prefix selection. Successful replacement and selector feedback. Figure 1(a) reports the successful replacement rate for each guidance level. Each point averages all continuations at that level within a window of 50 training steps. The level α=1/4α=1/4 has a higher successful replacement rate than α=1/2α=1/2 in every window. However, Figure 1(b) shows that their ordering can change when the successful replacement indicator is weighted by the fraction St/S0S_t/S_0 of the reference surprisal left to the continuation. In the final window, for example, α=1/4α=1/4 has lower mean feedback than α=1/2α=1/2 (0.08020.0802 versus 0.09910.0991), although its successful replacement rate is higher (43.84%43.84\% versus 23.65%23.65\%). Thus, the feedback favors less reference guidance in this window, consistent with Section 3.1, where a higher success probability of the continuation need not give a smaller distance. (a) Successful replacement (b) Selector feedback (c) Prefix selection Figure 1: Training dynamics of a Qwen3-8B ARG run. Panel (a) reports successful replacement rates, panel (b) reports mean selector feedback WtW_t, and panel (c) shows the selection probabilities. In panels (a) and (b), a point at step s summarizes all continuations at that level across the past 50 steps, including failures and candidate trajectories not accepted for training. Panel (c) shows the state after each training step without smoothing. Prefix selection during training. The selector uses the feedback WtW_t from each continuation to update its selection probabilities. Figure 1(c) shows how these probabilities change during training. The selector favors α=1/4α=1/4 through much of training and increases its probability of selecting α=1/2α=1/2 later, assigning it the highest final probability of 49.0%49.0\%. Since a larger α leaves more of the reference surprisal to the continuation, this shift supplies less reference guidance. Appendix H provides additional training-dynamics results. 6 Related Work Learning signals beyond on-policy rollouts. LUFFY, SRFT, ReLIFT, and related methods combine reference or teacher trajectories with the model’s own rollouts, through mixed-policy groups or combined supervised and reinforcement updates (Yan et al., 2026; Fu et al., 2026; Ma et al., 2026; Zhang et al., 2026b; Liu et al., 2025a; Dong et al., 2026; Lv et al., 2025). OPSD and SDPO instead use the model conditioned on solution information or feedback as a teacher for token-level supervision (Zhao et al., 2026; Hübotter et al., 2026). Other methods recover learning signals from failed rollouts, either by repairing them into correct ones (Li et al., 2026b; Yang et al., 2026; Zhang et al., 2025; Ren et al., 2026) or by reshaping the rewards or advantages of groups with identical rewards (Chen et al., 2025; Nie et al., 2026; Nan et al., 2025; Le et al., 2026). Reference-guided generation. Closer to our work, many methods use the reference only to guide the model’s own generation, either by revealing a prefix of the reference for the model to continue (Liu et al., 2026a; Setlur et al., 2026; Qu et al., 2026; Zhang et al., 2026a; Su et al., 2026; Liu et al., 2025b) or by providing hints derived from the reference or its answer (Liao et al., 2026; Zhang et al., 2026c; Nath et al., 2025; Chen et al., 2026; Zhou et al., 2026a). The amount of guidance is fixed or scheduled (Xi et al., 2024; Huang et al., 2025; Li et al., 2026a), adapted to the observed success of the model (Zhang et al., 2026d; amani et al., 2026; Li et al., 2025b; Guo et al., 2026; Beliaev, 2026), or chosen by the gain of a prefix or the transfer of a hint (Zhou et al., 2026c; Xia et al., 2026). In contrast, we derive the amount of guidance from the distance between the accepted trajectory and the ideal distribution, which shows that success alone does not determine the preferred prefix, and we learn a prefix selector shared across questions from the continuations themselves. Posterior views of learning from correct trajectories. Training on the model’s own correct samples, as in STaR and ReSTEM, approximates learning from our ideal distribution (Zelikman et al., 2022; Singh et al., 2023), which is also the correct distribution closest to the model in KL and relates to the Bayesian view of KL-regularized RL (Dymetman, 2026; Korbak et al., 2022). Latent-variable methods approximate the posterior over rationales given the correct answer (Phan et al., 2023; Zhou et al., 2026b), and TRICE keeps a stored rationale until a new proposal is correct, similar to our fallback. Other work analyzes or modifies the function of the success probability that RL maximizes (Davis & Recht, 2025; Tajwar et al., 2026; Ramasubramanian et al., 2026), or shows that targets unlikely under the model are harder to learn from (Ren et al., 2024; Li et al., 2025a; Setlur et al., 2024). 7 Conclusion and Limitations We study reference-prefix continuation with fallback through the distance from the accepted trajectory to the ideal distribution. Up to a small bounded term, this distance decreases with dtStd_tS_t, which captures the trade-off between successful replacement and the reference surprisal. This analysis leads to ARG, which learns a prefix selector across questions from the continuations themselves. Applied to all-failure groups in GRPO, ARG achieves the highest aggregate pass@12pass@12 on Qwen3-4B and Qwen3-8B across five mathematical benchmarks, with competitive average sampled accuracy. Limitations. ARG is only applied to GRPO, although the method does not depend on a particular reinforcement learning algorithm. Moreover, the selector chooses among a predefined set of guidance levels, and our experiments cover mathematical reasoning with one model family. Future work may extend the method to other domains with verifiable rewards. References amani et al. (2026) mohammad hossein amani, Aryo Lotfi, Nicolas Baldwin, Samy Bengio, Mehrdad Farajtabar, Emmanuel Abbe, and Robert West. Rl for reasoning by adaptively revealing rationales. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (eds.), International Conference on Learning Representations, volume 2026, p. 118599–118620, 2026. URL https://proceedings.iclr.c/paper_files/paper/2026/file/c073a9d002f3dab99a7af18642d74bdb-Paper-Conference.pdf. Bae et al. (2026) Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 700–719, 2026. Balunovic et al. (2026) Mislav Balunovic, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions. Advances in Neural Information Processing Systems, 38, 2026. Beliaev (2026) Vladislav Beliaev. Max out grpo signal: Adaptive trace prefix control for hard reasoning problems. arXiv preprint arXiv:2607.07674, 2026. Chen et al. (2026) Justin Chen, Xiangyu Peng, Prafulla Kumar Choubey, Kung-Hsiang Huang, Jiaxin Zhang, Mohit Bansal, and Chien-Sheng Wu. Nudging the boundaries of llm reasoning. In International Conference on Learning Representations, volume 2026, p. 30315–30335, 2026. Chen et al. (2025) Peter Chen, Xiaopeng Li, Ziniu Li, Xi Chen, and Tianyi Lin. Stepwise guided policy optimization: Coloring your incorrect reasoning in grpo. arXiv preprint arXiv:2505.11595, 2025. Davis & Recht (2025) Damek Davis and Benjamin Recht. What is the objective of reasoning with reinforcement learning? arXiv preprint arXiv:2510.13651, 2025. Dekoninck et al. (2026) Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger, Kári Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms. arXiv preprint arXiv:2605.00674, 2026. Dong et al. (2026) Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei Ma, Jue Chen, Binhua Li, et al. Rl-plus: Countering capability boundary collapse of llms in reinforcement learning with hybrid-policy optimization. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 43029–43053, 2026. Dymetman (2026) Marc Dymetman. Binary rewards and reinforcement learning: Fundamental challenges. arXiv preprint arXiv:2605.02375, 2026. Fu et al. (2026) Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. In International Conference on Learning Representations, volume 2026, p. 102430–102459, 2026. Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Guo et al. (2026) Yongxin Guo, Wenbo Deng, Zhenglin Cheng, and Xiaoying Tang. G2rpo-a: Guided group relative policy optimization with adaptive guidance. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 4525–4539, 2026. He et al. (2026) Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. In International Conference on Learning Representations, volume 2026, p. 138306–138322, 2026. Huang et al. (2025) Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M Ponti, and Ivan Titov. Blending supervised and reinforcement fine-tuning with prefix sampling. arXiv preprint arXiv:2507.01679, 2025. Hübotter et al. (2026) Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026. Jaech et al. (2024) Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. Korbak et al. (2022) Tomasz Korbak, Ethan Perez, and Christopher Buckley. Rl with kl penalties is better viewed as bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, p. 1083–1091, 2022. Lambert et al. (2024) Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. Le et al. (2026) Thanh-Long V Le, Myeongho Jeon, Kim Vu, Viet Lai, and Eunho Yang. No prompt left behind: Exploiting zero-variance prompts in llm reinforcement learning via entropy-guided advantage shaping. In International Conference on Learning Representations, volume 2026, p. 121956–121982, 2026. Li et al. (2026a) Jiazheng Li, Hongzhou Lin, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Yi Wu, and Jingzhao Zhang. Questa: Expanding reasoning capacity in llms via question augmentation. In International Conference on Learning Representations, volume 2026, p. 144191–144215, 2026a. Li et al. (2026b) Shuozhe Li, Vaishnav Tadiparthi, Kwonjoon Lee, Nakul Agarwal, Hossein Nourkhiz Mahjoub, Ehsan Moradi Pari, Lizhang Chen, Amy Zhang, and Liu Leqi. Learning robust reasoning through guided adversarial self-play. arXiv preprint arXiv:2602.00173, 2026b. Li et al. (2025a) Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. In Findings of the Association for Computational Linguistics: ACL 2025, p. 25366–25394, 2025a. Li et al. (2025b) Ziheng Li, Zexu Sun, Jinman Zhao, Erxue Min, Yongcheng Zeng, Hui Wu, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Xu Chen, et al. Staying in the sweet spot: Responsive reasoning evolution via capability-adaptive hint scaffolding. arXiv preprint arXiv:2509.06923, 2025b. Liao et al. (2026) Baohao Liao, Hanze Dong, Xinxing Xu, Christof Monz, and Jiang Bian. Self-hinting language models enhance reinforcement learning. arXiv preprint arXiv:2602.03143, 2026. Liu et al. (2026a) Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. Uft: Unifying supervised and reinforcement fine-tuning. Advances in Neural Information Processing Systems, 38:101347–101383, 2026a. Liu et al. (2025a) Yihao Liu, Shuocheng Li, Lang Cao, Yuhang Xie, Mengyu Zhou, Haoyu Dong, Xiaojun Ma, Shi Han, and Dongmei Zhang. Superrl: Reinforcement learning with supervision to boost language model reasoning. arXiv preprint arXiv:2506.01096, 2025a. Liu et al. (2026b) Zehao Liu, Yuanpu Cao, Jinghui Chen, and Vasant G Honavar. Restoring the sweet spot: Pass-rate weighted self-distillation for llm reasoning. arXiv preprint arXiv:2605.27765, 2026b. Liu et al. (2025b) Ziru Liu, Cheng Gong, Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu. Ghpo: Adaptive guidance for stable and efficient llm reinforcement learning. arXiv preprint arXiv:2507.10628, 2025b. Lv et al. (2025) Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, Yuntian Wei, Zhekai Chen, Xuekai Zhu, Kaiyan Zhang, Bingning Wang, Ning Ding, et al. Towards a unified view of large language model post-training. arXiv preprint arXiv:2509.04419, 2025. Ma et al. (2026) Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Wong, Junbo Niu, Chengyu Shen, Runming He, Yanhao Li, et al. Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions. In International Conference on Learning Representations, volume 2026, p. 80802–80821, 2026. Nan et al. (2025) Gongrui Nan, Siye Chen, Jing Huang, Mengyu Lu, Dexun Wang, Chunmei Xie, Weiqi Xiong, Xianzhou Zeng, Qixuan Zhou, Yadong Li, et al. Ngrpo: Negative-enhanced group relative policy optimization. arXiv preprint arXiv:2509.18851, 2025. Nath et al. (2025) Vaskar Nath, Elaine Lau, Anisha Gunjal, Manasi Sharma, Nikhil Baharte, and Sean Hendryx. Adaptive guidance accelerates reinforcement learning of reasoning models. arXiv preprint arXiv:2506.13923, 2025. Nie et al. (2026) Wenhua Nie, Jianan Wu, Junlin Liu, Ziwei Li, Zheng Lin, Zhang Zijian, Yilong Fan, Haoran Zheng, and Jyh-Shing Roger Jang. Gradient starvation in binary-reward grpo: Why group-mean centering fails and why the simplest fix works. arXiv preprint arXiv:2605.07689, 2026. Phan et al. (2023) Du Phan, Matthew Douglas Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. Training chain-of-thought via latent-variable inference. Advances in Neural Information Processing Systems, 36:72819–72841, 2023. Qu et al. (2026) Yuxiao Qu, Amrith Setlur, Virginia Smith, Ruslan Salakhutdinov, and Aviral Kumar. Pope: Learning to reason on hard problems via privileged on-policy exploration. arXiv preprint arXiv:2601.18779, 2026. Ramasubramanian et al. (2026) Shrinivas Ramasubramanian, Daman Arora, Fahim Tajwar, Guanning Zeng, Qingyang Wu, Zhongzhu Zhou, Chenfeng Xu, Haiwen Feng, Yuda Song, Aarti Singh, et al. Tail-likelihood reinforcement learning. arXiv preprint arXiv:2609.02987, 2026. Ren et al. (2024) Xuan Ren, Biao Wu, and Lingqiao Liu. I learn better if you speak my language: Understanding the superior performance of fine-tuning large language models with llm-generated responses. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 10225–10245, 2024. Ren et al. (2026) Yanwei Ren, Haotian Zhang, Likang Xiao, Xikai Zhang, Jiaxing Huang, Jiayan Qiu, Baosheng Yu, Quan Chen, and Liu Liu. Recycling failures: Salvaging exploration in rlvr via fine-grained off-policy guidance. arXiv preprint arXiv:2602.24110, 2026. Setlur et al. (2024) Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold. Advances in Neural Information Processing Systems, 37:43000–43031, 2024. Setlur et al. (2026) Amrith Setlur, Zijian Wang, Andrew Cohen, Paria Rashidinejad, and Sang Michael Xie. Reuse your flops: Scaling rl on hard problems by conditioning on very off-policy prefixes. arXiv preprint arXiv:2601.18795, 2026. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Singh et al. (2023) Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023. Su et al. (2026) Mingyu Su, Jian Guan, Yuxian Gu, Minlie Huang, and Hongning Wang. Trust-region adaptive policy optimization. In International Conference on Learning Representations, volume 2026, p. 51806–51828, 2026. Tajwar et al. (2026) Fahim Tajwar, Guanning Zeng, Yueer Zhou, Yuda Song, Daman Arora, Yiding Jiang, Jeff Schneider, Ruslan Salakhutdinov, Haiwen Feng, and Andrea Zanette. Maximum likelihood reinforcement learning. arXiv preprint arXiv:2602.02710, 2026. Team et al. (2025) Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025. Xi et al. (2024) Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, et al. Training large language models for reasoning through reverse curriculum reinforcement learning. arXiv preprint arXiv:2402.05808, 2024. Xia et al. (2026) Yu Xia, Canwen Xu, Zhewei Yao, Julian McAuley, and Yuxiong He. Learning to hint for reinforcement learning. arXiv preprint arXiv:2604.00698, 2026. Yan et al. (2026) Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. Advances in Neural Information Processing Systems, 38:117157–117186, 2026. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Yang et al. (2026) Matthew Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, and Aviral Kumar. Int: Self-proposed interventions enable credit assignment in llm reasoning. In International Conference on Learning Representations, volume 2026, p. 85054–85091, 2026. Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022. Zhang et al. (2026a) Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan Hu, and Rui Yan. Stephint: Multi-level stepwise hints enhance reinforcement learning to reason. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 37846–37864, 2026a. Zhang et al. (2026b) Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. In International Conference on Learning Representations, volume 2026, p. 120693–120726, 2026b. Zhang et al. (2025) Xiaoying Zhang, Yipeng Zhang, Hao Sun, Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng. Critique-grpo: Advancing llm reasoning with natural language and numerical feedback. arXiv preprint arXiv:2506.03106, 2025. Zhang et al. (2026c) Xichen Zhang, Sitong Wu, Yinghao Zhu, Haoru Tan, Shaozuo Yu, Ziyi He, and Jiaya Jia. Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning. In International Conference on Learning Representations, volume 2026, p. 131946–131974, 2026c. Zhang et al. (2026d) Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak. Bread: Branched rollouts from expert anchors bridge sft & rl for reasoning. Advances in Neural Information Processing Systems, 38:96726–96752, 2026d. Zhao et al. (2026) Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026. Zhou et al. (2026a) Ruiyang Zhou, Shuozhe Li, Amy Zhang, and Liu Leqi. Expo: Unlocking hard reasoning with self-explanation-guided reinforcement learning. Advances in Neural Information Processing Systems, 38:90989–91016, 2026a. Zhou et al. (2026b) Xiangxin Zhou, Zichen Liu, Haonan Wang, Chao Du, Min Lin, Chongxuan Li, Liang Wang, and Tianyu Pang. Variational reasoning for language models. In International Conference on Learning Representations, volume 2026, p. 154370–154407, 2026b. Zhou et al. (2026c) Yuhang Zhou, Yixin Cao, and Guangnan Ye. From correctness to utility: Gain-based prefix evaluation for llm reasoning. arXiv preprint arXiv:2606.07190, 2026c. Appendix A Proofs for the Distance to the Ideal Distribution This appendix proves Theorem 1 and collects its consequences. Appendix A.1 fixes the setup. Appendix A.2 derives the gradient identity in Equation (3) and the distance of training on the reference. Appendix A.3 describes the accepted trajectory and the reference surprisal, and then proves Theorem 1. Appendix A.4 gives an equivalent reading of the closed form. Finally, Appendix A.5 illustrates the theorem with a finite example. A.1 Setup We use the setup of Section 3.1. Fix a question x, its correct reference r, and the current model, and suppress the conditioning on x. The trajectory space is finite, and π denotes the current policy πθold _ _old. Moreover, C is the set of correct trajectories, p=π(C)p=π(C) is the success probability, and ν=π(⋅∣C)ν=π(· C) is the ideal distribution in Equation (2). We assume 0<π(r)<10<π(r)<1. For a token sequence h, we write π(h)π(h) for the probability that a trajectory sampled from π begins with h, and π(z∣h)π(z h) for the probability that the model continues h with z. Since the continuation follows the same policy, including its termination rule, autoregressive factorization gives π(h⊕z)=π(h)π(z∣h)π(h z)=π(h)π(z h) for every complete trajectory h⊕zh z. For a prefix length 0≤t<|r|0≤ t<|r|, the reference splits as r=ht⊕atr=h_t a_t, and QtQ_t is the distribution of the accepted trajectory in Equation (1). Besides dtd_t and StS_t in Equation (4), the proofs use the probabilities of an exact copy of the reference and of a correct candidate trajectory, ut=π(at∣ht),vt=PrZ∼π(⋅∣ht)(ht⊕Z∈C).u_t=π(a_t h_t), v_t= _Z π(· h_t) (h_t Z∈ C ). Here an exact copy means equality of the complete token sequences, and different token sequences can still follow the same mathematical strategy. By definition, St=−logutS_t=- u_t, and in particular S0=−logπ(r)S_0=- π(r) is the reference surprisal of the empty prefix. Since autoregressive factorization gives ut=π(r)/π(ht)≥π(r)>0u_t=π(r)/π(h_t)≥π(r)>0, we have dt≤1−ut<1d_t≤ 1-u_t<1, and every StS_t is finite. A.2 The Ideal Distribution We first justify the choice of the ideal distribution as the learning target in Section 2. Lemma A.1 shows that learning the likelihood of its trajectories increases the success probability in expectation. Lemma A.2 then gives the distance of training on the reference, which is the first term in Theorem 1. Lemma A.1 (Gradient identity). Suppose that πθ _θ is differentiable in θ and assigns positive probability to every trajectory in a neighborhood of θold _old. Write pθ=πθ(C)p_θ= _θ(C). Then Y∼ν[∇θlogπθ(Y)|θ=θold]=∇θlogpθ|θ=θold.E_Y ν [ _θ _θ(Y) |_θ= _old ]= _θ p_θ |_θ= _old. Proof. Since the trajectory space is finite, differentiating pθ=∑y∈Cπθ(y)p_θ= _y∈ C _θ(y) term by term gives ∇θlogpθ=1pθ∑y∈C∇θπθ(y)=∑y∈Cπθ(y)pθ∇θlogπθ(y). _θ p_θ= 1p_θ _y∈ C _θ _θ(y)= _y∈ C _θ(y)p_θ _θ _θ(y). At θ=θoldθ= _old, the weight πθ(y)/pθ _θ(y)/p_θ of each y∈Cy∈ C equals ν(y)ν(y) by Equation (2), which proves the claim. ∎ Here ν is fixed at the current model, and differentiation applies only to the trajectory log likelihood. Lemma A.2 (Distance of training on the reference). Training on the reference has distance DKL(δr∥ν)=logpπ(r)=S0+logp∈[0,S0].D_KL( _r\|ν)= pπ(r)=S_0+ p∈[0,S_0]. In contrast, if ν assigns positive probability to a trajectory other than r, then DKL(ν∥δr)=∞D_KL(ν\| _r)=∞. Proof. Since δr _r assigns all probability to r and r∈Cr∈ C, Equation (2) gives DKL(δr∥ν)=−logν(r)=log(p/π(r))D_KL( _r\|ν)=- ν(r)= (p/π(r)). Substituting S0=−logπ(r)S_0=- π(r) gives S0+logpS_0+ p. This quantity is nonnegative since ν(r)≤1ν(r)≤ 1, and it is at most S0S_0 since p≤1p≤ 1. For the reverse direction, a trajectory y≠ry≠ r with ν(y)>0ν(y)>0 has δr(y)=0 _r(y)=0, which makes the term ν(y)log(ν(y)/δr(y))ν(y) (ν(y)/ _r(y)) infinite. ∎ Thus, the direction of KL in Section 2 permits a finite comparison with training on the reference. A.3 Proof of Theorem 1 The closed form in Theorem 1 combines two ingredients. Lemma A.3 describes the distribution of the accepted trajectory, and Lemma A.4 relates the reference surprisal to the probability of the prefix. Lemma A.3 (Distribution of the accepted trajectory). For every prefix length t, we have dt=vt−utd_t=v_t-u_t and Qt(r)=1−dtQ_t(r)=1-d_t. Moreover, every trajectory y≠ry≠ r with Qt(y)>0Q_t(y)>0 is correct and has the form y=ht⊕zy=h_t z, and it satisfies Qt(y)=π(z∣ht),Qt(y)ν(y)=pπ(ht).Q_t(y)=π(z h_t), Q_t(y)ν(y)= pπ(h_t). These trajectories have total probability dtd_t under QtQ_t. Proof. The candidate trajectory Y~t=ht⊕Z Y_t=h_t Z equals r exactly when Z=atZ=a_t, which has probability utu_t. Since r∈Cr∈ C, the event Y~t∈C\ Y_t∈ C\ is the disjoint union of Y~t=r\ Y_t=r\ and Y~t∈C,Y~t≠r\ Y_t∈ C,\ Y_t≠ r\. Therefore, vt=ut+dtv_t=u_t+d_t. By Equation (1), the accepted trajectory equals r when the candidate trajectory is an exact copy or is incorrect. These two events are disjoint and have total probability ut+(1−vt)=1−dtu_t+(1-v_t)=1-d_t. For y≠ry≠ r, the accepted trajectory equals y only when Y~t=y Y_t=y and y∈Cy∈ C. Hence, Qt(y)>0Q_t(y)>0 requires y∈Cy∈ C and y=ht⊕zy=h_t z for some z, in which case Qt(y)=π(z∣ht)Q_t(y)=π(z h_t). Autoregressive factorization gives π(z∣ht)=π(y)/π(ht)π(z h_t)=π(y)/π(h_t), where π(ht)≥π(r)>0π(h_t)≥π(r)>0. Since ν(y)=π(y)/pν(y)=π(y)/p for y∈Cy∈ C, the ratio Qt(y)/ν(y)Q_t(y)/ν(y) equals p/π(ht)p/π(h_t). Finally, these trajectories carry the remaining probability 1−Qt(r)=dt1-Q_t(r)=d_t. ∎ Lemma A.4 (Monotonicity of the reference surprisal). For every prefix length t, S0−St=−logπ(ht),0≤St≤S0.S_0-S_t=- π(h_t), 0≤ S_t≤ S_0. Moreover, StS_t is nonincreasing in t. Proof. Autoregressive factorization gives π(r)=π(ht)π(at∣ht)π(r)=π(h_t)π(a_t h_t). Taking negative logarithms gives S0=−logπ(ht)+StS_0=- π(h_t)+S_t. Since π(at∣ht)≤1π(a_t h_t)≤ 1 and π(ht)≤1π(h_t)≤ 1, both StS_t and S0−StS_0-S_t are nonnegative. For s≤ts≤ t, every trajectory that begins with hth_t also begins with hsh_s. Therefore, π(ht)≤π(hs)π(h_t)≤π(h_s), which gives St=S0+logπ(ht)≤SsS_t=S_0+ π(h_t)≤ S_s. ∎ With these two lemmas, the proof of Theorem 1 reduces to a direct computation. Theorem 1 (Distance to the ideal distribution, restated). Under the setup in Appendix A.1, for every prefix length t, DKL(Qt∥ν)=logpπ(r)⏟DKL(δr∥ν)−dtSt+(1−dt)log(1−dt).D_KL(Q_t\|ν)= pπ(r)_D_KL( _r\|ν)-d_tS_t+(1-d_t) (1-d_t). Moreover, every t∗∈argmaxtdtStt^*∈ _td_tS_t satisfies DKL(Qt∗∥ν)≤mintDKL(Qt∥ν)+1/eD_KL(Q_t^*\|ν)≤ _tD_KL(Q_t\|ν)+1/e. Proof. Write d=dtd=d_t. By Lemma A.3, QtQ_t assigns probability 1−d1-d to r, whose likelihood ratio is Qt(r)/ν(r)=(1−d)p/π(r)Q_t(r)/ν(r)=(1-d)p/π(r). It assigns the remaining probability d to trajectories whose likelihood ratio is p/π(ht)p/π(h_t). Therefore, DKL(Qt∥ν)=(1−d)log(1−d)pπ(r)+dlogpπ(ht).D_KL(Q_t\|ν)=(1-d) (1-d)pπ(r)+d pπ(h_t). Lemma A.4 gives logπ(ht)=logπ(r)+St π(h_t)= π(r)+S_t, and hence dlog(p/π(ht))=dlog(p/π(r))−dStd (p/π(h_t))=d (p/π(r))-dS_t. Collecting the two terms in log(p/π(r)) (p/π(r)) proves the closed form. Its first term equals DKL(δr∥ν)D_KL( _r\|ν) by Lemma A.2 and does not depend on t. For the second statement, write Dr=DKL(δr∥ν)D_r=D_KL( _r\|ν) and ct=(1−dt)log(1−dt)c_t=(1-d_t) (1-d_t). Since −aloga∈[0,1/e]-a a∈[0,1/e] for a∈[0,1]a∈[0,1], we have ct∈[−1/e,0]c_t∈[-1/e,0]. For every prefix length t, the maximality of dt∗St∗d_t^*S_t^* gives DKL(Qt∗∥ν) D_KL(Q_t^*\|ν) =Dr−dt∗St∗+ct∗≤Dr−dt∗St∗≤Dr−dtSt =D_r-d_t^*S_t^*+c_t^*≤ D_r-d_t^*S_t^*≤ D_r-d_tS_t =DKL(Qt∥ν)−ct≤DKL(Qt∥ν)+1e. =D_KL(Q_t\|ν)-c_t≤ D_KL(Q_t\|ν)+ 1e. Taking the minimum over the finitely many prefix lengths completes the proof. ∎ Since dtSt≥0d_tS_t≥ 0 and (1−dt)log(1−dt)≤0(1-d_t) (1-d_t)≤ 0, the closed form also gives DKL(Qt∥ν)≤DKL(δr∥ν)D_KL(Q_t\|ν)≤ D_KL( _r\|ν) for every prefix length t. Therefore, a continuation with fallback never moves the distribution of the accepted trajectory farther from the ideal distribution than training on the reference. A.4 Consequences of Theorem 1 We now give an equivalent reading of the closed form. Corollary A.5 rewrites the distance in terms of the reference content kept in the accepted trajectory. Corollary A.5 (Reference content kept in the accepted trajectory). Let h¯ h be the reference content kept in the accepted trajectory Y∼QtY Q_t, which is r when Y=rY=r and the prefix hth_t otherwise. Then, for every prefix length t, DKL(Qt∥ν)=logp+(1−dt)S0+dt(S0−St)⏟[−logπ(h¯)]+(1−dt)log(1−dt).D_KL(Q_t\|ν)= p+ (1-d_t)S_0+d_t(S_0-S_t)_E[- π( h)]+(1-d_t) (1-d_t). Proof. Lemma A.2 gives log(p/π(r))=logp+S0 (p/π(r))= p+S_0. Substituting this expression and S0=(1−dt)S0+dtS0S_0=(1-d_t)S_0+d_tS_0 into the closed form of Theorem 1 gives the display. By Lemma A.3, h¯=r h=r with probability 1−dt1-d_t and h¯=ht h=h_t with probability dtd_t. Since −logπ(r)=S0- π(r)=S_0 and −logπ(ht)=S0−St- π(h_t)=S_0-S_t by Lemma A.4, the middle two terms equal [−logπ(h¯)]E[- π( h)]. ∎ In other words, apart from logp p, which does not depend on the prefix, and a term in [−1/e,0][-1/e,0], the distance to the ideal distribution is the expected surprisal of the reference content that the model has not replaced. Both extremes of the prefix length keep nearly all of the reference surprisal S0S_0. A prefix covering almost the entire reference keeps it through the prefix, while an empty prefix on a hard question keeps it through fallback. A.5 A Finite Example This example shows that neither the success probability of the continuation nor the successful replacement probability alone determines which prefix gives the smaller distance. Consider five complete trajectories whose displayed strings omit a shared deterministic termination token, ⊕ H r, ⊕ H s, ⊕ H f, ⊕ O t, and ⊕ O g, with respective probabilities 0.080.08, 0.010.01, 0.010.01, 0.200.20, and 0.700.70. The correct trajectories are ⊕ H r, ⊕ H s, and ⊕ O t, and the reference is ⊕ H r. The first tokens are H and O, with probabilities 0.100.10 and 0.900.90, and the second-token probabilities are the corresponding conditional probabilities. These probabilities define a finite autoregressive distribution. Prefix vtv_t utu_t dtd_t StS_t dtStd_tS_t DKL(Qt∥ν)D_KL(Q_t\|ν) Empty 0.290.29 0.080.08 0.210.21 2.5257292.525729 0.5304030.530403 0.5712310.571231 H 0.900.90 0.800.80 0.100.10 0.2231440.223144 0.0223140.022314 1.1707151.170715 Training on the reference gives DKL(δr∥ν)=1.287854D_KL( _r\|ν)=1.287854. Supplying H increases vtv_t while making an exact copy more likely. In this setting, both the successful replacement probability dtd_t and the reference surprisal StS_t decrease. Therefore, the empty prefix gives the larger dtStd_tS_t and the smaller distance despite the lower success probability of its continuation. To isolate the role of the reference surprisal, keep the three probabilities under H unchanged and assign probabilities 0.040.04 and 0.860.86 to ⊕ O t and ⊕ O g, respectively. The resulting comparison is Prefix vtv_t utu_t dtd_t StS_t dtStd_tS_t DKL(Qt∥ν)D_KL(Q_t\|ν) Empty 0.130.13 0.080.08 0.050.05 2.5257292.525729 0.1262860.126286 0.3104930.310493 H 0.900.90 0.800.80 0.100.10 0.2231440.223144 0.0223140.022314 0.3683690.368369 Training on the reference now gives DKL(δr∥ν)=0.485508D_KL( _r\|ν)=0.485508. Here supplying H increases both vtv_t and dtd_t. However, the decrease in the reference surprisal outweighs the increase in dtd_t. Therefore, the empty prefix still gives the larger dtStd_tS_t and the smaller distance. Thus, even the successful replacement probability alone can favor a different prefix from the one preferred by Theorem 1. Both examples concern a fixed model and illustrate why the choice of prefix should account for dtd_t and StS_t together. Appendix B Proofs for Learning the Prefix Selector This appendix proves Theorem 2 and analyzes the prefix selector during training. Appendix B.1 fixes the setup and derives the properties of the selector feedback, and Appendix B.2 proves Theorem 2. Appendix B.3 examines the effects of sharing the selector across questions and of randomizing the guidance level. Finally, Appendix B.4 derives the expected update of the selector in the GRPO application as the model and the questions change, and Appendix B.5 bounds the accumulated sampling noise. B.1 Setup and Selector Feedback Fix the current model, the references, and guideD_guide as in Section 3.3. We assume that every question satisfies the setup of Appendix A.1, which includes 0<πθold(rx∣x)<10< _ _old(r_x x)<1 and hence 0<S0,x<∞0<S_0,x<∞. For a question x, we write νx _x for its ideal distribution. For a guidance level α, we write Qx,αQ_x,α, dx,αd_x,α, and Sx,αS_x,α for the distribution of the accepted trajectory, the successful replacement probability, and the reference surprisal at the prefix length tx(α)t_x(α). Here tx(α)t_x(α) is the actual prefix length, including the boundary adjustment in Section 4.1. Therefore, the nominal level α can differ from the realized ratio Sx,α/S0,xS_x,α/S_0,x. We first collect the properties of the selector feedback for a single question, for which we suppress x as in Appendix A. Lemma B.1 (Selector feedback). For every prefix length t, the selector feedback satisfies Wt∈[0,1]W_t∈[0,1] and [Wt]=dtSt/S0E[W_t]=d_tS_t/S_0. Moreover, with ct=(1−dt)log(1−dt)c_t=(1-d_t) (1-d_t), DKL(Qt∥ν)S0=DKL(δr∥ν)S0−[Wt]+ctS0,ctS0∈[−1eS0,0], D_KL(Q_t\|ν)S_0= D_KL( _r\|ν)S_0-E[W_t]+ c_tS_0, c_tS_0∈ [- 1eS_0,0 ], (14) and the normalized distances satisfy 0≤DKL(Qt∥ν)/S0≤DKL(δr∥ν)/S0≤10≤ D_KL(Q_t\|ν)/S_0≤ D_KL( _r\|ν)/S_0≤ 1. Proof. The indicator in Equation (7) has expectation dtd_t by Equation (4). Lemma A.4 gives 0≤St≤S00≤ S_t≤ S_0, and hence Wt∈[0,1]W_t∈[0,1] and [Wt]=dtSt/S0E[W_t]=d_tS_t/S_0, which proves Equation (8). Dividing the closed form of Theorem 1 by S0S_0 gives Equation (14), and the range of ctc_t follows from −aloga∈[0,1/e]-a a∈[0,1/e] for a∈[0,1]a∈[0,1]. Finally, since [Wt]≥0E[W_t]≥ 0 and ct≤0c_t≤ 0, Equation (14) gives DKL(Qt∥ν)≤DKL(δr∥ν)D_KL(Q_t\|ν)≤ D_KL( _r\|ν), and Lemma A.2 gives 0≤DKL(δr∥ν)≤S00≤ D_KL( _r\|ν)≤ S_0. ∎ Thus, a larger expected feedback corresponds to a smaller normalized distance, up to the bounded last term. Since normalization divides dtStd_tS_t at every prefix of the same reference by the same S0S_0, it preserves the ranking of dtStd_tS_t across these prefixes. Because of the bounded last term, this ranking need not equal the ranking of the exact distances. Across questions, normalization expresses each distance relative to the reference surprisal of the empty prefix. The ablation without surprisal weighting in Section 5.3 replaces WtW_t with the indicator of successful replacement, whose expectation is dtd_t. Since this indicator still excludes exact copies of the reference, comparing it with WtW_t isolates the effect of weighting a successful replacement by the reference surprisal. For the rest of this appendix, we write gx(α)=[Wx,α]=dx,αSx,αS0,x∈[0,1]g_x(α)=E[W_x,α]= d_x,αS_x,αS_0,x∈[0,1] for the expected selector feedback of question x at level α, where the equality follows from Lemma B.1. B.2 Proof of Theorem 2 Theorem 2 measures a selector by its average normalized distance and compares it with training on the reference. Written out, these distances are ℒ(ϕ)=X∼guide,α∼ρϕ[DKL(QX,α∥νX)S0,X],ℒref=X∼guide[DKL(δrX∥νX)S0,X].L(φ)=E_X _guide,\,α _φ [ D_KL(Q_X,α\| _X)S_0,X ], _ref=E_X _guide [ D_KL( _r_X\| _X)S_0,X ]. By Lemma B.1 and gx(α)∈[0,1]g_x(α)∈[0,1], the quantities ℒ(ϕ)L(φ), ℒrefL_ref, and J(ϕ)J(φ) lie in [0,1][0,1]. Theorem 2 (Learning the prefix selector from feedback, restated). Fix the current model, the references, and guideD_guide, where every question satisfies the setup of Section 3.1. Let ℒ(ϕ)L(φ) and ℒrefL_ref be the expectations of DKL(QX,α∥νX)/S0,XD_KL(Q_X,α\| _X)/S_0,X and DKL(δrX∥νX)/S0,XD_KL( _r_X\| _X)/S_0,X over X∼guideX _guide and α∼ρϕα _φ. Then J(ϕ)≤ℒref−ℒ(ϕ)≤J(ϕ)+X[1eS0,X].J(φ) _ref-L(φ)≤ J(φ)+E_X [ 1eS_0,X ]. Moreover, for a differentiable prefix selector with positive probabilities and a baseline b fixed before sampling, [(WX,α−b)∇ϕlogρϕ(α)]=∇ϕJ(ϕ).E [(W_X,α-b) _φ _φ(α) ]= _φJ(φ). Proof. For the first statement, averaging Equation (14) over X∼guideX _guide and α∼ρϕα _φ gives ℒ(ϕ)=ℒref−J(ϕ)+ϵ(ϕ),−X1eS0,X≤ϵ(ϕ)≤0,L(φ)=L_ref-J(φ)+ε(φ), -E_X 1eS_0,X≤ε(φ)≤ 0, (15) where ϵ(ϕ)=[cX,α/S0,X]ε(φ)=E[c_X,α/S_0,X] and cx,αc_x,α is the last term of Theorem 1 at the prefix length tx(α)t_x(α). Lemma B.1 bounds each cx,α/S0,xc_x,α/S_0,x, and rearranging Equation (15) gives the first statement. For the second statement, the level is sampled independently of the question. Condition on X=xX=x. Since α∼ρϕα _φ and [Wx,α∣α]=gx(α)E[W_x,α α]=g_x(α), [(Wx,α−b)∇ϕlogρϕ(α)∣X=x]=∑α∈(gx(α)−b)∇ϕρϕ(α)=∑α∈gx(α)∇ϕρϕ(α).E [(W_x,α-b) _φ _φ(α) X=x ]= _α (g_x(α)-b) _φ _φ(α)= _α g_x(α) _φ _φ(α). The baseline term vanishes since ∑α∇ϕρϕ(α)=∇ϕ1=0 _α _φ _φ(α)= _φ1=0. Define μα=X∼guide[gX(α)]∈[0,1] _α=E_X _guide[g_X(α)]∈[0,1], which does not depend on ϕφ. Then J(ϕ)=∑α∈ρϕ(α)μα,∇ϕJ(ϕ)=∑α∈μα∇ϕρϕ(α),J(φ)= _α _φ(α) _α, _φJ(φ)= _α _α _φ _φ(α), where differentiation acts on a finite sum with fixed bounded coefficients. Averaging the conditional identity over X∼guideX _guide gives the same expression, which proves the second statement. ∎ The first statement leads to the following guarantee for a selector that maximizes J. Corollary B.2 (Maximizing the objective). If J(ϕ∗)≥J(ϕ)J(φ^*)≥ J(φ), then ℒ(ϕ∗)≤ℒ(ϕ)+X[1/(eS0,X)]L(φ^*) (φ)+E_X[1/(eS_0,X)]. Proof. By Equation (15), J(ϕ∗)≥J(ϕ)J(φ^*)≥ J(φ) gives ℒ(ϕ∗)≤ℒref−J(ϕ∗)≤ℒref−J(ϕ)≤ℒ(ϕ)+X1eS0,X.∎L(φ^*) _ref-J(φ^*) _ref-J(φ) (φ)+E_X 1eS_0,X. Thus, a larger objective corresponds to a smaller average normalized distance, up to a bounded correction. Since the correction ϵ(ϕ)ε(φ) can change with the selector, J and ℒL can rank selectors differently within this bound. The bound is informative only when X[1/S0,X]E_X[1/S_0,X] is small, and it can be large when some reference surprisals are close to zero. In Section 4.2, the baseline is a running average of past feedback. For such a baseline, the same argument applies after conditioning on the history and the current question. Given this information, the baseline is fixed, the level still follows ρϕ _φ, and its feedback still has conditional mean gx(α)g_x(α). Moreover, the baseline update in Appendix B.4 keeps b∈[0,1]b∈[0,1], which ensures integrability. B.3 Sharing and Randomizing Guidance Levels The prefix selector shares one distribution over guidance levels across questions and samples a level for each continuation. We now examine both choices. Proposition B.3 bounds the cost of sharing within the objective J. Proposition B.4 then compares the distance after randomizing the level with the average over levels in J. For any distribution ρ over A, write J(ρ)=∑αρ(α)μαJ(ρ)= _αρ(α) _α with μα _α from the proof of Theorem 2, which gives J(ρϕ)=J(ϕ)J( _φ)=J(φ). The best shared distribution has value Jshared∗=maxαμαJ_shared^*= _α _α. In contrast, an oracle that knows gxg_x for every question and chooses a level for each question separately has value Jindividual∗=XmaxαgX(α)J_individual^*=E_X _αg_X(α). Proposition B.3 (Cost of sharing). For every distribution ρ over A, Jindividual∗−J(ρ)=(Jindividual∗−Jshared∗)+(Jshared∗−J(ρ)),J_individual^*-J(ρ)= (J_individual^*-J_shared^* )+ (J_shared^*-J(ρ) ), (16) where both terms are nonnegative. Moreover, the prefix selector in Equation (11) satisfies supϕJ(ϕ)=(1−εsel)maxαμα+εsel||∑αμα. _φJ(φ)=(1- _sel) _α _α+ _sel|A| _α _α. Proof. The decomposition is an identity. The first term is nonnegative since maxαX[gX(α)]≤XmaxαgX(α) _αE_X[g_X(α)] _X _αg_X(α). The second term is nonnegative since J(ρ)J(ρ) is a convex combination of the values μα _α. For the selector in Equation (11), write qϕq_φ for its softmax component. Then J(ϕ)=(1−εsel)∑αqϕ(α)μα+(εsel/||)∑αμαJ(φ)=(1- _sel) _αq_φ(α) _α+( _sel/|A|) _α _α. The first sum is at most maxαμα _α _α, and it approaches this value as qϕq_φ concentrates on a maximizing level. ∎ The first term in Equation (16) measures the cost of using the same distribution over levels for every question. It is zero if one level maximizes gxg_x for almost every question. The second term measures the gap between the current selector and the best shared distribution. If all μα _α are equal, every shared distribution has the same objective even when individual questions prefer different levels. Moreover, the uniform component of the selector has an explicit cost in the current objective, while it keeps every level observed as the model changes. Through Theorem 2, the objective J corresponds to the average of the normalized distance over the selected levels. However, sampling a level and then making one continuation produces a single distribution of the accepted trajectory, which mixes the distributions for different levels. For a fixed question x and a distribution ρ over A, this mixture is Qx,ρ=∑αρ(α)Qx,αQ_x,ρ= _αρ(α)Q_x,α. Let IxI_x be the mutual information between the sampled level and the accepted trajectory, and let ℋ(ρ)=−∑αρ(α)logρ(α)H(ρ)=- _αρ(α) ρ(α) be the entropy of ρ. Proposition B.4 (Randomizing the guidance level). For every question x and distribution ρ over A, DKL(Qx,ρ∥νx)=∑αρ(α)DKL(Qx,α∥νx)−Ix,D_KL(Q_x,ρ\| _x)= _αρ(α)D_KL(Q_x,α\| _x)-I_x, where 0≤Ix≤ℋ(ρ)0≤ I_x (ρ). Consequently, with jx(ρ)=∑αρ(α)gx(α)j_x(ρ)= _αρ(α)g_x(α), DKL(δrx∥νx)S0,x−jx(ρ)−1/e+ℋ(ρ)S0,x≤DKL(Qx,ρ∥νx)S0,x≤DKL(δrx∥νx)S0,x−jx(ρ). D_KL( _r_x\| _x)S_0,x-j_x(ρ)- 1/e+H(ρ)S_0,x≤ D_KL(Q_x,ρ\| _x)S_0,x≤ D_KL( _r_x\| _x)S_0,x-j_x(ρ). Proof. Every Qx,αQ_x,α has finite distance by Lemma B.1. Splitting the log probability ratio through Qx,ρQ_x,ρ and summing over trajectories with positive probability gives ∑αρ(α)DKL(Qx,α∥νx) _αρ(α)D_KL(Q_x,α\| _x) =∑α,yρ(α)Qx,α(y)[logQx,α(y)Qx,ρ(y)+logQx,ρ(y)νx(y)] = _α,yρ(α)Q_x,α(y) [ Q_x,α(y)Q_x,ρ(y)+ Q_x,ρ(y) _x(y) ] =Ix+DKL(Qx,ρ∥νx), =I_x+D_KL(Q_x,ρ\| _x), since ∑αρ(α)DKL(Qx,α∥Qx,ρ) _αρ(α)D_KL(Q_x,α\|Q_x,ρ) equals the mutual information IxI_x. The mutual information is nonnegative and at most the entropy of the level, which gives 0≤Ix≤ℋ(ρ)0≤ I_x (ρ). For the bounds, divide the identity by S0,xS_0,x and apply Equation (14) to each level. The upper bound uses Ix≥0I_x≥ 0 and cx,α≤0c_x,α≤ 0, and the lower bound uses Ix≤ℋ(ρ)I_x (ρ) and cx,α≥−1/ec_x,α≥-1/e. ∎ The mutual information is zero when all distributions with positive weight coincide. Otherwise, randomizing the level gives a smaller distance than the average over levels. This construction returns one accepted trajectory from one continuation, and Appendix C.2 analyzes acceptance after multiple continuations. B.4 Selector Updates in the GRPO Application Theorem 2 fixes the model, the references, and guideD_guide. We now specify guideD_guide and the baseline in the GRPO application of Section 4.3, and then derive the expected update of the selector for a batch and across training iterations. Questions selected for reference guidance. Fix the current model and a reference rxr_x for each question. Since the G responses of a group are sampled independently, the group is an all-failure group with probability fx=(1−pθold(x))Gf_x=(1-p_ _old(x))^G. For X∼X , define ζ=[fX]ζ=E_D[f_X]. When ζ>0ζ>0, the questions that receive reference guidance follow guide(x)=(x)fxζ.D_guide(x)= D(x)f_xζ. (17) Given the question, the guidance levels and continuations are sampled independently of the group. Therefore, restricting guidance to questions with an all-failure group changes their relative weights while preserving [Wx,α]=gx(α)E[W_x,α]=g_x(α) for each question and level. If ζ=0ζ=0, no question receives guidance and the selector update is zero almost surely. Baseline and update order. The selector starts with ϕ=0φ=0 and b=0b=0. Every term in Equation (12) uses the same selector parameters and baseline, which are fixed before the batch is collected. After a nonempty batch of continuations, the baseline is updated by b←(1−β)b+β||∑j∈Wj,b←(1-β)b+ β|U| _j W_j, (18) where 0<β≤10<β≤ 1. Since this update is a convex combination of values in [0,1][0,1], induction gives b∈[0,1]b∈[0,1] at every iteration. Centering the updated logits by subtracting their mean leaves the selection probabilities unchanged. An empty batch of continuations leaves ϕφ and b unchanged. Table 3 specifies the numerical hyperparameters. Reference surprisals, verification outcomes, and model parameters are fixed during the selector update. This update precedes the policy update, and both use the continuations generated by the same current model. All continuations supply feedback, including K zero feedback values when no continuation gives a successful replacement and ARG falls back to the reference. Expected batch update. Let Ψ denote the summed direction in Equation (12) before multiplication by the step size, Ψ=∑j∈(Wj−b)∇ϕlogρϕ(αj), = _j (W_j-b) _φ _φ( _j), with Ψ=0 =0 when U is empty. Proposition B.5 (Expected batch update). Consider a batch of n question positions, each with marginal distribution D. Suppose that the model, the selector, and the baseline are fixed during collection, and that every all-failure group receives K continuations whose levels and feedback follow the laws above. Then [Ψ]=nKζ∇ϕJ(ϕ),E[ ]=nKζ _φJ(φ), (19) where the right-hand side is zero when ζ=0ζ=0. Proof. Write Ψ as a sum over question positions i and continuations ℓ=1,…,K =1,…,K of the terms group i is all-failure(Wiℓ−b)∇ϕlogρϕ(αiℓ)1\group i is all-failure\(W_i -b) _φ _φ( _i ). Conditional on the question x at position i and on its all-failure event, each term has expectation ∑αgx(α)∇ϕρϕ(α) _αg_x(α) _φ _φ(α) by the proof of Theorem 2. Therefore, position i contributes ∑x(x)fxK∑αgx(α)∇ϕρϕ(α)=Kζ∑xguide(x)∑αgx(α)∇ϕρϕ(α)=Kζ∇ϕJ(ϕ), _xD(x)f_xK _αg_x(α) _φ _φ(α)=Kζ _xD_guide(x) _αg_x(α) _φ _φ(α)=Kζ _φJ(φ), where the first equality uses Equation (17). Summing over the n positions by linearity proves the claim. When ζ=0ζ=0, no group is an all-failure group almost surely, and hence Ψ=0 =0 almost surely. ∎ Since the proof uses only the marginal laws, it requires no independence between question positions or between continuations. Holding J fixed, the factors n, K, and ζ only scale the expected update at a fixed step size. However, changes in the all-failure probabilities also change the question weights in J and hence the update direction. Changing models and questions. Across training iterations, the model, the question distribution, and the selector all change. Let (ℱk)k(F_k)_k be the increasing filtration of the training history, where ℱkF_k contains the history before iteration k, including the current model, the selector parameters ϕk _k, and the baseline. For the question positions Xk,1,…,Xk,nX_k,1,…,X_k,n at iteration k, define their average conditional marginal distribution by k(x)=1n∑i=1nPr(Xk,i=x∣ℱk).D_k(x)= 1n _i=1^n (X_k,i=x _k). With the current all-failure probabilities fk,xf_k,x and expected feedback gk,x(α)g_k,x(α), define ζk=X∼k[fk,X],guide,k(x)=k(x)fk,xζk, _k=E_X _k[f_k,X], _guide,k(x)= D_k(x)f_k,x _k, Jk(ϕ)=X∼guide,k∑αρϕ(α)gk,X(α),J_k(φ)=E_X _guide,k _α _φ(α)g_k,X(α), where the last two are defined when ζk>0 _k>0. Let Ψk _k be the summed direction at iteration k. Proposition B.6 (Expected update under changing models and questions). Suppose that, conditional on ℱkF_k, the groups and continuations at iteration k follow the laws of Proposition B.5 under the current model, except that position i has the marginal distribution Pr(Xk,i=⋅∣ℱk) (X_k,i=· _k). Then [Ψk∣ℱk]=nKζk∇ϕJk(ϕk),E[ _k _k]=nK _k _φJ_k( _k), (20) where the right-hand side is zero when ζk=0 _k=0. Proof. Apply the proof of Proposition B.5 conditionally on ℱkF_k, with the marginal distribution of position i given by Pr(Xk,i=⋅∣ℱk) (X_k,i=· _k). Since the proof uses only marginal laws, the positions may have different marginal distributions. Summing their contributions replaces nnD with nknD_k, which gives Equation (20). ∎ The gradient in Equation (20) holds the history-dependent model, question distribution, and prefix mapping fixed. Thus, the update follows the current objective in expectation even as training changes these quantities. The setting includes sampling with replacement from a fixed distribution, for which k=D_k=D, and sampling without replacement from a remaining pool of questions. B.5 Accumulated Sampling Noise Proposition B.6 describes the expected update at each iteration. We now bound how far the accumulated updates deviate from these expectations, starting with the size of each update. Lemma B.7 (Bounded update direction). For the selector in Equation (11) and a baseline in [0,1][0,1], every term of Ψk _k has Euclidean norm at most 2 2. Consequently, ‖Ψk/(nK)‖2≤2\| _k/(nK)\|_2≤ 2. Proof. Let qϕq_φ be the softmax component of Equation (11) and αe_α the coordinate vector for level α. Then ∇ϕlogρϕ(α)=(1−εsel)qϕ(α)ρϕ(α)(α−qϕ). _φ _φ(α)= (1- _sel)q_φ(α) _φ(α) (e_α-q_φ ). The prefactor is at most one since ρϕ(α)≥(1−εsel)qϕ(α) _φ(α)≥(1- _sel)q_φ(α). Moreover, ‖α−qϕ‖22=(1−qϕ(α))2+∑α′≠αqϕ(α′)2≤2(1−qϕ(α))2≤2\|e_α-q_φ\|_2^2=(1-q_φ(α))^2+ _α ≠αq_φ(α )^2≤ 2(1-q_φ(α))^2≤ 2. Since WjW_j and b both lie in [0,1][0,1], we have |Wj−b|≤1|W_j-b|≤ 1, which bounds each term by 2 2. Finally, Ψk _k has at most nKnK terms, including the case of no continuations. ∎ Define the centered errors ξk=ΨknK−ζk∇ϕJk(ϕk). _k= _knK- _k _φJ_k( _k). Proposition B.8 (Accumulated sampling noise). Under the conditions of Proposition B.6 and a baseline in [0,1][0,1], for a deterministic number NiterN_iter of iterations, ‖1Niter∑k=1Niterξk‖22≤2Niter.E \| 1N_iter _k=1^N_iter _k \|_2^2≤ 2N_iter. (21) Proof. Proposition B.6 gives [ξk∣ℱk]=0E[ _k _k]=0. The conditional variance identity and Lemma B.7 then give [‖ξk‖22∣ℱk]=[‖ΨknK‖22|ℱk]−‖[ΨknK|ℱk]‖22≤2.E [\| _k\|_2^2 _k ]=E\! [ \| _knK \|_2^2 |F_k ]- \|E\! [ _knK |F_k ] \|_2^2≤ 2. For k<k′k<k , the error ξk _k is measurable with respect to ℱk′F_k . Therefore, ⟨ξk,ξk′⟩=⟨ξk,[ξk′∣ℱk′]⟩=0E _k, _k =E _k,E[ _k _k ] =0. Expanding the squared norm of the sum, the cross terms vanish and each of the NiterN_iter diagonal terms is at most 22, which proves the bound. ∎ The baseline update in Equation (18) satisfies the condition on the baseline. Proposition B.8 controls the noise around the changing expected directions. Appendix C Analysis of Multiple Continuations Theorem 1 concerns a single continuation, whereas ARG samples K continuations for each question and accepts candidate trajectories by the rule in Section 4.1. This appendix analyzes the distribution of the accepted trajectory after multiple continuations. Appendix C.1 fixes one prefix and studies how the distance to the ideal distribution changes with the number of continuations. Appendix C.2 gives a sufficient condition under which an arbitrary acceptance rule keeps the distance no larger than that of training on the reference. C.1 Repeated Continuations from a Fixed Prefix Fix a prefix length t and let HtH_t be the event that a trajectory begins with hth_t. Since the reference is correct and has positive probability, 0<ut≤vt≤10<u_t≤ v_t≤ 1. For a positive integer k, draw k independent continuations from hth_t. The first-success rule returns the first correct candidate trajectory, including an exact copy of r, and returns r if every candidate trajectory is incorrect. This rule treats an exact copy as a success. Let Qt(k)Q_t^(k) be the resulting distribution of the accepted trajectory, and let νht=π(⋅∣C∩Ht) _h_t=π(· C∩ H_t) be the ideal distribution restricted to trajectories that begin with hth_t. Lemma C.1 (Distribution after repeated continuations). Let wk=1−(1−vt)kw_k=1-(1-v_t)^k be the probability that at least one candidate trajectory is correct, and define λk=wk/vt _k=w_k/v_t and dt(k)=dtλkd_t^(k)=d_t _k. Then Qt(k)=(1−wk)δr+wkνht,Q_t^(k)=(1-w_k) _r+w_k _h_t, (22) and DKL(Qt(k)∥ν)=DKL(δr∥ν)−dt(k)St+(1−dt(k))log(1−dt(k))+dt(k)logλk.D_KL(Q_t^(k)\|ν)=D_KL( _r\|ν)-d_t^(k)S_t+ (1-d_t^(k) ) (1-d_t^(k) )+d_t^(k) _k. Proof. A correct trajectory y=ht⊕zy=h_t z is returned at continuation j with probability (1−vt)j−1π(z∣ht)(1-v_t)^j-1π(z h_t). Summing over j=1,…,kj=1,…,k gives Qt(k)(y)=λkπ(z∣ht)Q_t^(k)(y)= _kπ(z h_t) for every correct y≠ry≠ r. Since νht(y)=π(z∣ht)/vt _h_t(y)=π(z h_t)/v_t for these trajectories, this probability equals wkνht(y)w_k _h_t(y). The same calculation gives λkut _ku_t for an exact copy, and fallback adds (1−vt)k(1-v_t)^k. Therefore, Qt(k)(r)=(1−wk)+wkut/vt=1−dt(k)Q_t^(k)(r)=(1-w_k)+w_ku_t/v_t=1-d_t^(k), which proves Equation (22). For the distance, the trajectories other than r have total probability dt(k)d_t^(k) and likelihood ratio λkp/π(ht) _kp/π(h_t). Following the proof of Theorem 1 with this ratio gives the closed form. ∎ Compared with the closed form for one continuation, the additional term dt(k)logλkd_t^(k) _k accounts for the larger probability of each trajectory other than the reference after repeated continuations. For k=1k=1, we have λ1=1 _1=1, and the lemma recovers Theorem 1. The mixture form in Equation (22) shows that additional continuations move probability from the reference to νht _h_t, which leads to the following limit. Proposition C.2 (The limit of a fixed prefix). The distance DKL(Qt(k)∥ν)D_KL(Q_t^(k)\|ν) is nonincreasing in k and satisfies −logν(Ht)≤DKL(Qt(k)∥ν)≤−logν(Ht)+(1−vt)klogvtut.- ν(H_t)≤ D_KL(Q_t^(k)\|ν)≤- ν(H_t)+(1-v_t)^k v_tu_t. Consequently, it converges to −logν(Ht)- ν(H_t) as k→∞k→∞. Proof. For y∈C∩Hty∈ C∩ H_t, we have ν(y)=ν(Ht)νht(y)ν(y)=ν(H_t) _h_t(y). Hence, every distribution Q supported on C∩HtC∩ H_t satisfies DKL(Q∥ν)=DKL(Q∥νht)−logν(Ht)D_KL(Q\|ν)=D_KL(Q\| _h_t)- ν(H_t). By Lemma C.1, Qt(k)Q_t^(k) is such a distribution. Therefore, it suffices to study DKL(Qt(k)∥νht)D_KL(Q_t^(k)\| _h_t), and the lower bound follows from its nonnegativity. For the upper bound, convexity of KL applied to Equation (22) gives DKL(Qt(k)∥νht)≤(1−wk)DKL(δr∥νht)D_KL(Q_t^(k)\| _h_t)≤(1-w_k)D_KL( _r\| _h_t), where DKL(δr∥νht)=−logνht(r)=log(vt/ut)D_KL( _r\| _h_t)=- _h_t(r)= (v_t/u_t) and 1−wk=(1−vt)k1-w_k=(1-v_t)^k. For monotonicity, take ℓ>k >k. If wk=1w_k=1, then Qt(k)=Qt(ℓ)=νhtQ_t^(k)=Q_t^( )= _h_t. Otherwise, a=(1−wℓ)/(1−wk)∈[0,1]a=(1-w_ )/(1-w_k)∈[0,1] satisfies Qt(ℓ)=aQt(k)+(1−a)νhtQ_t^( )=aQ_t^(k)+(1-a) _h_t, and convexity gives DKL(Qt(ℓ)∥νht)≤aDKL(Qt(k)∥νht)≤DKL(Qt(k)∥νht).D_KL(Q_t^( )\| _h_t)≤ aD_KL(Q_t^(k)\| _h_t)≤ D_KL(Q_t^(k)\| _h_t). Finally, the upper bound converges to −logν(Ht)- ν(H_t) since vt≥ut>0v_t≥ u_t>0. ∎ For the empty prefix, HtH_t contains every trajectory. Therefore, the limiting distribution is ν and the limiting distance is zero. A nonempty prefix has the same limit exactly when HtH_t contains all correct trajectories with positive probability. Proposition C.2 orders the numbers of continuations for each fixed prefix. However, comparing different prefixes also requires their respective utu_t and vtv_t, which determine both the rate and the limit of the decrease. C.2 A Sufficient Condition for General Acceptance Rules To cover the acceptance rule of ARG, which excludes exact copies, mixes several prefixes, prefers some candidate trajectories over others, and may accept several of them, we consider a general procedure. Choose K≥1K≥ 1 reference prefixes ht1,…,htKh_t_1,…,h_t_K before generation. Conditional on this list, draw a continuation Zj∼π(⋅∣htj)Z_j π(· h_t_j) and form the candidate trajectory Y~j=htj⊕Zj Y_j=h_t_j Z_j for each j. The continuations may depend on each other, provided that each candidate trajectory has this conditional marginal law given the entire list. A procedure returns r or one of the correct candidate trajectories. If several trajectories are accepted, we consider the distribution obtained by selecting one of them with nonnegative weights that sum to one. Let Q denote the resulting distribution of the accepted trajectory, including fallback, and use the convention 0log0=00 0=0. Proposition C.3 (Multiple continuations and acceptance). Under the setup above, define dQ=1−Q(r)d_Q=1-Q(r) and Λ=∑j=1Kexp(−Stj) = _j=1^K (-S_t_j). Then DKL(Q∥ν)≤DKL(δr∥ν)+dQlogΛ+(1−dQ)log(1−dQ).D_KL(Q\|ν)≤ D_KL( _r\|ν)+d_Q +(1-d_Q) (1-d_Q). (23) In particular, Λ≤1 ≤ 1 guarantees DKL(Q∥ν)≤DKL(δr∥ν)D_KL(Q\|ν)≤ D_KL( _r\|ν). A sufficient condition is minjStj≥logK _jS_t_j≥ K. Proof. A trajectory y≠ry≠ r can be selected only if it appears among the candidate trajectories. Thus, Q(y)≤∑j=1KPr(Y~j=y)=∑j=1Ky begins with htjπ(y)π(htj)≤π(y)Λπ(r),Q(y)≤ _j=1^K ( Y_j=y)= _j=1^K1\y begins with h_t_j\ π(y)π(h_t_j)≤ π(y) π(r), where the last inequality uses π(r)/π(htj)=exp(−Stj)π(r)/π(h_t_j)= (-S_t_j) from Lemma A.4. Every y≠ry≠ r with Q(y)>0Q(y)>0 is correct and has π(y)>0π(y)>0. Hence, Q(y)/ν(y)≤pΛ/π(r)Q(y)/ν(y)≤ p /π(r). Since these trajectories have total probability dQd_Q and the reference has probability 1−dQ1-d_Q with Q(r)/ν(r)=(1−dQ)p/π(r)Q(r)/ν(r)=(1-d_Q)p/π(r), DKL(Q∥ν)≤logpπ(r)+(1−dQ)log(1−dQ)+dQlogΛ.D_KL(Q\|ν)≤ pπ(r)+(1-d_Q) (1-d_Q)+d_Q . Lemma A.2 gives log(p/π(r))=DKL(δr∥ν) (p/π(r))=D_KL( _r\|ν), which proves Equation (23). Since (1−dQ)log(1−dQ)≤0(1-d_Q) (1-d_Q)≤ 0, and dQlogΛ≤0d_Q ≤ 0 when Λ≤1 ≤ 1, the condition Λ≤1 ≤ 1 gives the comparison with training on the reference. Finally, minjStj≥logK _jS_t_j≥ K gives Λ≤Kexp(−logK)=1 ≤ K (- K)=1. ∎ The argument permits arbitrary selection among correct candidate trajectories, including the exclusion of exact copies and the priority rule of ARG. For one prefix used K times, Λ=Kexp(−St) =K (-S_t), and the bound becomes DKL(δr∥ν)−dQ(St−logK)+(1−dQ)log(1−dQ)D_KL( _r\|ν)-d_Q(S_t- K)+(1-d_Q) (1-d_Q). The term logK K accounts for the possible concentration from selecting among several candidate trajectories. For K=1K=1 and the acceptance rule in Equation (1), we have dQ=dtd_Q=d_t, and the bound becomes the equality in Theorem 1. In ARG, the prefixes come from guidance levels sampled before generation. The following corollary extends the comparison to this case. Corollary C.4 (Guidance levels sampled before generation). Under the setup of Proposition C.3, suppose that the K guidance levels are sampled before generation from any distribution over KA^K, and that minα∈Stx(α)≥logK _α S_t_x(α)≥ K. Then the distribution Q of the accepted trajectory, averaged over the sampled levels, satisfies DKL(Q∥ν)≤DKL(δr∥ν)D_KL(Q\|ν)≤ D_KL( _r\|ν). Proof. Conditional on each list of levels, every prefix satisfies Stj≥logKS_t_j≥ K. Therefore, Proposition C.3 bounds the conditional distance by DKL(δr∥ν)D_KL( _r\|ν). Since Q is the average of these conditional distributions, convexity of KL in its first argument gives the same bound for Q. ∎ The condition in Corollary C.4 uses the actual prefix lengths after the boundary adjustment and is available from the same reference surprisals used for prefix selection. Appendix D Analysis of Learning from Inserted Trajectories The preceding appendices compare distributions of accepted trajectories at a fixed model. This appendix examines how the inserted trajectories contribute to the policy update in Section 4.3. Appendix D.1 derives the gradient of inserted trajectories in Equation (13). Appendix D.2 derives the expected group update under binary rewards for one inserted trajectory, and Appendix D.3 extends it to multiple inserted trajectories. Finally, Appendix D.4 relates learning the complete trajectory and learning only the continuation to their respective success events. D.1 The Gradient of Inserted Trajectories Condition on the realized batch, its masks, and the recomputed group advantages. Let ℐI index the inserted trajectories, and let T>0T>0 be the total number of active response tokens in the batch loss. For token j of an inserted trajectory yiy_i, let ℓi,j(θ) _i,j(θ) be its log probability given the original question and all preceding tokens of the trajectory. Inserted trajectories use the clipped objective of Equation (29) in Appendix E.3 with the stopped current probability in the ratio denominator, ℒinserted(θ)=−1T∑i∈ℐ∑j=1|yi|minω~i,j(θ)Ai,clip(ω~i,j(θ),0.8,1.3)Ai,L_inserted(θ)=- 1T _i _j=1^|y_i| \ ω_i,j(θ)A_i,clip( ω_i,j(θ),0.8,1.3)A_i \, ω~i,j(θ)=exp(ℓi,j(θ)−stopgrad(ℓi,j(θ))), ω_i,j(θ)= ( _i,j(θ)-stopgrad( _i,j(θ)) ), where stopgradstopgrad preserves a value but assigns zero derivative to it. Lemma D.1 (Gradient of inserted trajectories). Suppose that the advantages AiA_i and the token count T are held fixed during differentiation and that every token of an inserted trajectory is active. Then the gradient of ℒinsertedL_inserted is given by Equation (13). Proof. By construction, ω~i,j=1 ω_i,j=1 and ∇θω~i,j=∇θℓi,j _θ ω_i,j= _θ _i,j. Since the clipping interval contains one in its interior, both branches of the clipped loss have the same derivative at this ratio. Therefore, for any fixed real advantage AiA_i, the loss of token j has derivative −Ai∇θℓi,j/T-A_i _θ _i,j/T. Summing over all positions, including any supplied prefix and the termination token, gives −Ai∇θlogπθ(yi∣xi)/T-A_i _θ _θ(y_i x_i)/T for each inserted trajectory. Summing over i∈ℐi proves Equation (13). ∎ Lemma D.1 identifies the gradient under the prescribed rule of automatic differentiation. For Ai>0A_i>0, minimizing ℒinsertedL_inserted gives a positively weighted supervised fine-tuning (SFT) contribution. The advantage specifies its learning weight, while candidate generation and acceptance determine the distribution of the trajectory. D.2 The Group Update with One Inserted Trajectory We now use binary correctness rewards to separate the contributions of correct and incorrect responses. This simplification ignores the format reward, which leaves the advantages unchanged when all responses in a group receive the same format reward. The calculation is at the current model and precedes division by the batch token count. Fix a question x with set of correct trajectories C, and suppress the conditioning on x unless shown explicitly. Assume a finite trajectory space, a differentiable policy with positive trajectory probabilities in a neighborhood of θold _old, and 0<p=πθold(C)<10<p= _ _old(C)<1. Draw G≥2G≥ 2 independent responses from πθold _ _old and condition on all of them failing verification. Then replace one uniformly chosen response by Y∼QY Q, where Q is supported on C, and where Y and the replaced position are independent of the original failures. Both Q and the collected responses are fixed during differentiation. Set pθ=πθ(C),νθ=πθ(⋅∣C),μθ=πθ(⋅∣Cc),p_θ= _θ(C), _θ= _θ(· C), _θ= _θ(· C^c), and write ν=νθoldν= _ _old for the ideal distribution at the current model. At the current model, the ratios of the original responses equal one and clipping is inactive. The inserted trajectory follows Lemma D.1, and all response tokens are included. We consider the group ascent direction, which is the sum of the reward-dependent gradients before division by the batch token count. With one inserted trajectory, the reward mean is 1/G1/G and the sample standard deviation is sR=1/Gs_R=1/ G. Therefore, the inserted trajectory has advantage (G−1)/[G(sR+δ)](G-1)/[G(s_R+δ)], and each failure has advantage −1/[G(sR+δ)]-1/[G(s_R+δ)], where δ>0δ>0 is the stabilizer in Equation (28). Define the positive coefficient cG=G−1G(1/G+δ).c_G= G-1G(1/ G+δ). Let F1,…,FG−1F_1,…,F_G-1 denote the original failures that remain after replacement. The realized group direction is g=cG[∇θlogπθ(Y∣x)−1G−1∑j=1G−1∇θlogπθ(Fj∣x)]θ=θold.g=c_G [ _θ _θ(Y x)- 1G-1 _j=1^G-1 _θ _θ(F_j x) ]_θ= _old. (24) Proposition D.2 (Inserted trajectories and the group update). Under the setup above, let g¯Q=[g] g_Q=E[g] be the expectation over the original failures, the replaced position, and the inserted trajectory. Then g¯Q=cG∇θ[logpθ1−pθ−DKL(Q∥νθ)]|θ=θold. g_Q=c_G . _θ [ p_θ1-p_θ-D_KL(Q\| _θ) ] |_θ= _old. (25) In particular, for Q=νQ=ν, g¯ν=cGp(1−p)∇θpθ|θ=θold. g_ν= c_Gp(1-p) . _θp_θ |_θ= _old. (26) Proof. Conditioning independent responses on all of them belonging to CcC^c gives independent draws from μθold _ _old, and uniform replacement preserves this distribution for the remaining failures. Therefore, with F∼μθoldF _ _old, g¯QcG=[Q∇θlogπθ(Y)−μθold∇θlogπθ(F)]|θ=θold. g_Qc_G= . [E_Q _θ _θ(Y)-E_ _ _old _θ _θ(F) ] |_θ= _old. Applying the argument of Lemma A.1 to the event CcC^c gives μθold∇θlogπθ(F)=∇θlog(1−pθ)E_ _ _old _θ _θ(F)= _θ (1-p_θ) at θold _old. For fixed Q, expanding logνθ(y)=logπθ(y)−logpθ _θ(y)= _θ(y)- p_θ gives ∇θDKL(Q∥νθ)=−Q∇θlogπθ(Y)+∇θlogpθ. _θD_KL(Q\| _θ)=-E_Q _θ _θ(Y)+ _θ p_θ. Substituting both expressions proves Equation (25). When Q=νQ=ν, the gradient of the KL term vanishes at θold _old by Lemma A.1. Since ∇θlog[pθ/(1−pθ)]=∇θpθ/[pθ(1−pθ)] _θ [p_θ/(1-p_θ)]= _θp_θ/[p_θ(1-p_θ)], this gives Equation (26). ∎ The ratio pθ/(1−pθ)p_θ/(1-p_θ) is the odds of passing verification. The first term in Equation (25) favors success over failure. The second term records which correct trajectories the distribution Q favors. For training on the reference, DKL(δr∥νθ)=−logνθ(r)D_KL( _r\| _θ)=- _θ(r), and this term favors the particular reference within the set of correct trajectories. In contrast, Equation (26) shows that inserting trajectories from the ideal distribution gives an expected group update proportional to the gradient of the success probability under the original question. For reference fallback, Y=rY=r makes the positive term in Equation (24) a weighted SFT gradient. For an accepted candidate trajectory, the positive term learns its complete sequence, including the supplied prefix. In both cases, the original failures supply the negative terms in the same group update. This combination of learning from correct trajectories with feedback on the model’s own failures connects our update to prior work on integrating supervised and reinforcement fine-tuning (Yan et al., 2026; Fu et al., 2026). D.3 Multiple Inserted Trajectories We keep the preceding setup but insert a random number N+N_+ of correct trajectories with 1≤N+<G1≤ N_+<G, where a single reference is inserted when no candidate trajectory is accepted. The inserted trajectories may depend on each other. However, they, their number N+N_+, and the replaced positions, which are uniformly chosen and distinct, are independent of the original failures. Conditional on N+=mN_+=m, let Q Q and M M be the empirical distributions of the inserted trajectories and of the remaining failures. Substituting the advantages (G−m)/[G(sm+δ)](G-m)/[G(s_m+δ)] and −m/[G(sm+δ)]-m/[G(s_m+δ)] of the inserted trajectories and the remaining failures gives the group direction g=κm[Q^∇θlogπθ(Y)−M^∇θlogπθ(F)]θ=θold,g= _m [E_ Q _θ _θ(Y)-E_ M _θ _θ(F) ]_θ= _old, where κm=m(G−m)G(sm+δ),sm=m(G−m)G(G−1). _m= m(G-m)G(s_m+δ), s_m= m(G-m)G(G-1). Define the average coefficient and the advantage-weighted distribution of inserted trajectories by κ¯=[κN+],Q+(y)=[κN+Q^(y)]κ¯. κ=E[ _N_+], Q_+(y)= E[ _N_+ Q(y)] κ. Since 1≤N+<G1≤ N_+<G, we have κ¯>0 κ>0, and Q+Q_+ is a probability distribution on C. Proposition D.3 (Multiple inserted trajectories). Under the setup above, [g]=κ¯∇θ[logpθ1−pθ−DKL(Q+∥νθ)]|θ=θold.E[g]= κ . _θ [ p_θ1-p_θ-D_KL(Q_+\| _θ) ] |_θ= _old. For one inserted trajectory, κ1=cG _1=c_G and Q+=Q_+=Q, which recovers Proposition D.2. Proof. Conditional on the inserted trajectories and their number, uniform replacement preserves the failure distribution μθold _ _old for the remaining failures. Therefore, [κN+M^(y)]=κ¯μθold(y)E[ _N_+ M(y)]= κ _ _old(y), and taking expectations in the group direction gives [g]=κ¯[Q+∇θlogπθ(Y)−μθold∇θlogπθ(F)]θ=θold.E[g]= κ [E_Q_+ _θ _θ(Y)-E_ _ _old _θ _θ(F) ]_θ= _old. The proof of Proposition D.2 then applies with cGc_G replaced by κ¯ κ and Q replaced by Q+Q_+. Finally, s1=1/Gs_1=1/ G gives κ1=cG _1=c_G. ∎ The distribution Q+Q_+ weights each realized set of inserted trajectories by its total positive advantage κN+ _N_+. Therefore, it accounts for both acceptance and the resulting composition of the group. D.4 Learning the Prefix and the Continuation The likelihood in Equation (13) includes every token of the complete accepted trajectory under the original question. For a candidate trajectory from a nonempty prefix, the chain rule gives logπθ(ht⊕z∣x)=logπθ(ht∣x)+logπθ(z∣x,ht). _θ(h_t z x)= _θ(h_t x)+ _θ(z x,h_t). (27) Hence, learning covers both producing the supplied prefix and completing the trajectory, and reference fallback trains the complete reference through Equation (13). We now connect the two terms in Equation (27) to different success events. Fix a prefix length t, let HtH_t be the event that a trajectory begins with hth_t, and let νht=πθold(⋅∣C∩Ht) _h_t= _ _old(· C∩ H_t) as in Appendix C.1. This distribution is well defined since the correct reference has positive probability. Let QZ,t(z)=νht(ht⊕z)Q_Z,t(z)= _h_t(h_t z) be the distribution of its continuations. Here πθ(C∣Ht) _θ(C H_t) is the probability that a continuation from hth_t gives a correct candidate trajectory, which equals vtv_t at θold _old. Proposition D.4 (Learning the prefix and the continuation). Under the assumptions of Lemma A.1, at θ=θoldθ= _old, Y∼νht∇θlogπθ(Y)=∇θlogπθ(C∩Ht),Z∼QZ,t∇θlogπθ(Z∣ht)=∇θlogπθ(C∣Ht).E_Y _h_t _θ _θ(Y)= _θ _θ(C∩ H_t), _Z Q_Z,t _θ _θ(Z h_t)= _θ _θ(C H_t). Moreover, the distribution Qt(k)Q_t^(k) of the accepted trajectory after k continuations in Lemma C.1 satisfies Y∼Qt(k)∇θlogπθ(Y)=(1−wk)∇θlogπθ(r)+wk∇θlogπθ(C∩Ht).E_Y Q_t^(k) _θ _θ(Y)=(1-w_k) _θ _θ(r)+w_k _θ _θ(C∩ H_t). Proof. The first identity follows from Lemma A.1 with C replaced by C∩HtC∩ H_t. For the second, Equation (27) gives ∇θlogπθ(ht⊕Z)=∇θlogπθ(ht)+∇θlogπθ(Z∣ht) _θ _θ(h_t Z)= _θ _θ(h_t)+ _θ _θ(Z h_t), and πθ(C∩Ht)=πθ(ht)πθ(C∣Ht) _θ(C∩ H_t)= _θ(h_t) _θ(C H_t). Subtracting ∇θlogπθ(ht) _θ _θ(h_t) from both sides of the first identity gives the second. Finally, the mixture form in Equation (22), whose weights are fixed at θold _old, and the first identity give the last statement. ∎ Under νht _h_t, the expected gradient of the complete trajectory addresses the joint event of producing the prefix and passing verification. In contrast, learning only the continuation addresses verification given an externally supplied prefix. The two coincide for the empty prefix. The joint event covers the correct trajectories that begin with this prefix, while the overall success event C also includes correct trajectories that begin differently. Finally, the mixture identity in Proposition D.4 shows how fallback and successful continuations contribute to the expected gradient of the accepted trajectory before any weighting. Appendix E Implementation and Evaluation Details We first specify the training procedure, reference boundaries, and policy loss. We then give the training configurations, question prompt, and evaluation protocol. Together, these details connect the single-continuation analysis in Theorem 1 to the complete training procedure. E.1 Training Procedure Training uses 11,068 screened DeepMath examples with matched offline DeepSeek-R1 reference solutions (Guo et al., 2025). We start from DeepMath-103K, keep problems with difficulty at least 5.0, and drop problems whose gold answer is not a single short expression, is a multiple-choice letter, or cannot be verified by our checker when written inside . From the remaining problems, we randomly sample 16,000 in proportion to difficulty and let the base Qwen3-4B attempt each one eight times under the training prompt and sampling settings. Problems solved in all eight attempts are removed. We also drop the few problems whose reference solution does not end in the gold answer. Table 3 gives the ARG configuration. Algorithm 1 separates collection under a fixed policy from updates of the selector and policy. All continuations, including failures and candidate trajectories not selected for the policy update, provide selector feedback. In the algorithm, xU_x contains K distinct batch-wide continuation indices for question x. Candidate quantities are computed for every j∈xj _x. The operation AcceptAccept returns the candidate indices selected by the rule in Section 4.3. Algorithm 1 One ARG training update with GRPO 1. Input Batch ℬB with references, policy θ, selector ϕφ, baseline b. 2. θold←θ _old←θ, ←∅U← 3. x=(yx,i)i=1Gy_x=(y_x,i)_i=1^G, yx,i∼iidπθold(⋅∣x)y_x,i iid _ _old(· x) for x∈ℬx 4. for x∈ℬx with ∑i=1GRacc(x,yx,i)=0 _i=1^GR_acc(x,y_x,i)=0 do 5. St←−logπθold(at∣x,ht)S_t←- _ _old(a_t x,h_t) for 0≤t<|r|0≤ t<|r|, x←∅J_x← 6. if 0<S0<∞0<S_0<∞ then 7. αj∼iidρϕ _j iid _φ, tj←tx(αj)t_j← t_x( _j) 8. Zj∼πθold(⋅∣x,htj)Z_j _ _old(· x,h_t_j), Y~j←htj⊕Zj Y_j← h_t_j Z_j 9. Wj←Y~j∈Cx,Y~j≠rStj/S0W_j 1\ Y_j∈ C_x,\ Y_j≠ r\\,S_t_j/S_0 10. ←∪xU _x, x←Accept((Y~j,tj)j∈x,r)J_x (( Y_j,t_j)_j _x,r) 11. end if 12. Replace rows in xy_x by (Y~j)j∈x( Y_j)_j _x, or by r if x=∅J_x= . 13. end for 14. Recompute final rewards RiR_i, then Ai←(Ri−R¯)/(sR+δ)A_i←(R_i- R)/(s_R+δ). 15. if ≠∅U≠ then 16. ϕ←ϕ+ηsel∑j∈(Wj−b)∇ϕlogρϕ(αj)φ←φ+ _sel _j (W_j-b) _φ _φ( _j) 17. b←(1−β)b+β||∑j∈Wjb←(1-β)b+ β|U| _j W_j 18. end if 19. Update θ using Equation (13) and the remaining-response loss in Appendix E.3. Setting ARG configuration Questions per update 32 Ordinary responses per question G 8 Continuations per all-failure group K 8 Guidance levels A 1,1/2,1/4\1,1/2,1/4\ Total acceptance cap in Algorithm 1 4 Maximum accepted trajectories with empty prefixes 4 Maximum accepted trajectories with nonempty prefixes 1 Selector exploration εsel _sel 0.20.2 Selector step size ηsel _sel 0.30.3 Baseline update rate β 0.10.1 Training budget 200 updates Learning rate 10−610^-6 Warmup updates 10 Weight decay 0.010.01 Gradient norm clipping 1 Policy optimization epochs per batch 1 Clipping ratio interval [0.8,1.3][0.8,1.3] Policy loss averaging Batch token mean Total reward Accuracy plus 0.20.2 format Advantage stabilizer δ 10−610^-6 Training temperature 1 Training top-p and top-k 1 and no truncation Question length limit 3072 tokens Trajectory length budget including the prefix 8192 tokens Additional KL loss None Entropy loss coefficient 0 Table 3: ARG configuration for both model sizes. E.2 Prefix Boundaries The level α=1α=1 always uses the empty prefix. For α<1α<1, we take the latest prefix length whose reference surprisal is at least αS0α S_0 and, if this length is positive, move it to the first valid sentence or formula boundary at or after it. If no such boundary exists, we use the latest earlier valid boundary, or the empty prefix if there is none. A valid boundary falls on a token boundary, closes all mathematical delimiters, and leaves a nontrivial suffix containing the final boxed answer. The selector feedback uses the reference surprisal StS_t at this adjusted prefix length, and the prefix counts toward the maximum trajectory length. E.3 Policy Loss Final rewards RiR_i combine binary verifier accuracy with 0.20.2 times a binary format reward indicating an extracted boxed answer. Thus, a correct response receives 1.21.2, while an incorrect response receives 0.20.2 or zero according to its format. We recompute advantages within each final group as Ai=Ri−R¯sR+δ,A_i= R_i- Rs_R+δ, (28) where R¯ R and sRs_R are the mean and sample standard deviation of the group’s final rewards, and δ=10−6δ=10^-6 ensures numerical stability. All-equal rewards give exactly zero advantages. For an ordinary response yiy_i, let yi,jy_i,j be token j and define ωi,j(θ)=πθ(yi,j∣xi,yi,<j)πθold(yi,j∣xi,yi,<j). _i,j(θ)= _θ(y_i,j x_i,y_i,<j) _ _old(y_i,j x_i,y_i,<j). Let O index the original responses that remain in their groups, and let T be the total number of active response tokens in the final batch. Their loss contribution uses the asymmetric clipping interval [0.8,1.3][0.8,1.3], ℒoriginal(θ)=−1T∑i∈∑j=1|yi|minωi,j(θ)Ai,clip(ωi,j(θ),0.8,1.3)Ai.L_original(θ)=- 1T _i _j=1^|y_i| \ _i,j(θ)A_i,clip( _i,j(θ),0.8,1.3)A_i \. (29) Inserted trajectories use the stopped current probability in the ratio denominator, as derived in Appendix D. With unit token weights over each complete inserted trajectory, this gives Equation (13). Loss-active tokens include supplied prefixes and termination tokens when present, and both contributions share the same batch token denominator. E.4 Training Configurations and Question Prompt The GRPO, SAGE, BREAD, LUFFY, and ARG runs share the optimizer and student rollout settings in Table 4. The resampling and weighting ablations use the same settings. SFT learns from the offline references. For OPSD, we follow Zhao et al. (2026) in using one student response per question, a training generation limit of 1024 tokens, and temperature 1.11.1. Table 4 specifies the remaining settings used in our experiments. All listed runs use 32 questions per update and one optimization epoch per collected batch. Their data and parameter-initialization seeds are zero, and the learning rate is constant after warmup. Setting GRPO-based runs SFT OPSD Learning rate 10−610^-6 10−610^-6 5×10−75× 10^-7 Warmup updates 10 10 0 Weight decay 0.010.01 0.010.01 0 Gradient norm clipping 1 1 0.10.1 Responses per question 8 Reference 1 Response length limit 8192 8192 1024 Generation temperature 1 N/A 1.11.1 Generation top-p 1 N/A 0.950.95 Generation top-k Unrestricted N/A 20 Table 4: Recorded configurations for both model sizes. GRPO-based runs here comprise GRPO, SAGE, BREAD, LUFFY, ARG, resampling, and the weighting ablation. The reference-insertion ablation is specified by its replacement rule in Section 5.3. The ordinary student question prompt and the evaluation prompt contain a single user message with the question, a blank line, and the instruction below. They use no system message or few-shot example. Auxiliary teacher or reference conditioning is specific to each training method. The Qwen3 chat template is applied with thinking mode disabled, which appends an empty thinking block before response generation. The rendered question prompt is shown schematically below. <|im_start|>user <question> Please reason step by step, and put your final answer within . <|im_end|> <|im_start|>assistant <think> </think> Here <question> denotes the question text, and the line break before <|im_end|> is added for display. For ARG generation, the selected reference prefix supplies the beginning of the candidate trajectory. Evaluation supplies no reference prefix. E.5 Checkpoint and Evaluation Records Table 5 lists the reported checkpoints. GRPO, SAGE, BREAD, LUFFY, ARG, resampling, and the weighting ablations use the fixed budget of 200 updates. Reference insertion also reports its step-200 checkpoint. For SFT and OPSD, we select the reported checkpoints by comparing mean pass@12pass@12 and avg@12avg@12 on the five evaluation benchmarks. Method 4B checkpoint 8B checkpoint Base 0 0 SFT 25 25 OPSD 60 100 GRPO, SAGE, BREAD, LUFFY, ARG 200 200 Resampling 200 N/A w/o weighting 200 200 Reference insertion 200 N/A Table 5: Reported checkpoint steps for both model sizes. Base uses the released parameters. SFT and OPSD use benchmark-selected checkpoints, and the remaining trained entries report step 200. All 20 reported model and method entries use the same 150 evaluation questions, comprising 30 questions from each of the five benchmarks. The benchmark files, answer extractor, and verifier implementation match across entries. Table 6 gives the common decoding settings. Each question contributes 12 scored responses. Setting Evaluation value Temperature 1 Top-p 0.950.95 Top-k 20 Maximum new tokens 16,384 Maximum context length 32,768 Scored responses per question 12 Random seed 1234 Parameter precision bfloat16 Generation engine vLLM 0.11.0 Thinking mode Disabled Table 6: Evaluation configuration shared by all reported entries. The scorer extracts the final balanced boxed answer. A response without a boxed answer is incorrect. Extracted answers are checked with Math-Verify 0.9.0 and a normalized string comparison. For each question, pass@12pass@12 records whether at least one of its 12 responses is correct, while avg@12avg@12 is the fraction of those responses that are correct. Both metrics are averaged over questions and reported as percentages. Since the benchmarks have equal sizes, the mean over benchmarks also equals the mean over all 150 questions. Appendix F Additional Main Results Table 7 reports avg@12avg@12 for the same checkpoints and the same 12 scored responses per question as Table 1, following the scoring protocol in Appendix E.5. Model Method AIME24 AIME25 AIME26 HMMT_Feb HMMT_Nov Avg. Qwen3-4B Base 22.78 22.22 13.33 11.94 9.17 15.89 SFT 19.44 20.56 16.94 9.72 9.17 15.17 GRPO 47.22 42.22 46.67 26.94 29.17 38.44 OPSD 22.22 18.89 20.83 14.44 13.06 17.89 SAGE 44.17 39.17 40.83 25.00 28.06 35.44 BREAD 56.39 47.78 47.78 29.17 35.56 43.33 LUFFY 50.28 43.61 41.67 25.00 33.06 38.72 ARG (ours) 52.50 46.67 46.39 26.94 33.06 41.11 Qwen3-8B Base 25.28 21.67 18.61 9.72 9.44 16.94 SFT 33.06 27.78 28.06 12.22 17.50 23.72 GRPO 48.89 35.83 36.67 22.22 30.56 34.83 OPSD 33.89 29.44 26.67 14.17 19.44 24.72 SAGE 54.72 40.00 44.17 27.22 35.28 40.28 BREAD 60.56 42.50 43.06 27.78 39.44 42.67 LUFFY 50.83 42.22 43.89 25.28 31.94 38.83 ARG (ours) 55.28 46.94 46.39 27.22 35.56 42.28 Table 7: Main experimental results in avg@12avg@12 percentages. Avg. is the mean over the five benchmarks. Within each model, bold and underline mark the best and second-best values in each column. ARG improves aggregate avg@12avg@12 over GRPO by 2.672.67 and 7.457.45 percentage points on Qwen3-4B and Qwen3-8B, respectively, and achieves the second-highest aggregate avg@12avg@12 on both models. The difference between the two metrics is consistent with where ARG intervenes. Specifically, ARG modifies the GRPO update only for all-failure groups, where all G responses to a training question are incorrect, and leaves the update for all other questions unchanged. Therefore, the additional learning signal concentrates on questions that the model currently fails to solve. When a question changes from unsolved to solved by a few of the 12 responses, pass@12pass@12 counts the entire question, whereas avg@12avg@12 counts only the fraction of correct responses. As a result, the effect of ARG appears most clearly in pass@12pass@12, which measures the fraction of questions that the model solves within the sampling budget. Appendix G Additional Ablation Results We provide benchmark-level results, then examine how the ablations change the accepted trajectories and the prefix selector during training. Reference insertion and resampling use Qwen3-4B, while the surprisal-weighting comparison covers both model sizes. G.1 Evaluation Results Tables 8 and 9 report pass@12pass@12 and avg@12avg@12 separately for all five benchmarks. All entries use the checkpoint after 200 training steps and the same evaluation protocol as the main results. Each question contributes 12 scored responses, as specified in Appendix E.5. Model Method AIME24 AIME25 AIME26 HMMT_Feb HMMT_Nov Avg. Qwen3-4B Reference insertion 70.00 60.00 63.33 43.33 46.67 56.67 Resampling 76.67 63.33 63.33 43.33 53.33 60.00 w/o weighting 76.67 70.00 66.67 50.00 53.33 63.33 ARG 80.00 73.33 73.33 43.33 66.67 67.33 Qwen3-8B w/o weighting 83.33 70.00 66.67 50.00 50.00 64.00 ARG 83.33 73.33 73.33 56.67 63.33 70.00 Table 8: Benchmark-level ablation results in pass@12pass@12 percentages. Avg. is the mean over benchmarks. Here w/o weighting denotes ARG without surprisal weighting. Model Method AIME24 AIME25 AIME26 HMMT_Feb HMMT_Nov Avg. Qwen3-4B Reference insertion 36.67 34.17 34.17 17.78 19.44 28.44 Resampling 44.44 35.56 40.00 21.67 28.33 34.00 w/o weighting 43.61 40.83 39.17 21.39 28.33 34.67 ARG 52.50 46.67 46.39 26.94 33.06 41.11 Qwen3-8B w/o weighting 45.28 33.89 38.33 18.33 23.61 31.89 ARG 55.28 46.94 46.39 27.22 35.56 42.28 Table 9: Benchmark-level ablation results in avg@12avg@12 percentages, computed from the same scored responses as Table 8. Avg. is the mean over benchmarks. For the ablations in Table 9, ARG has higher average sampled accuracy on every benchmark. G.2 Accepted Candidate Trajectories during Training To measure how often continuations supply the trajectories used for training, we count all-failure groups that accept at least one candidate trajectory. Reference fallback is excluded from this count. Figure 2 reports the proportion in consecutive windows of 50 training steps, using the total group counts within each window. Each curve describes the all-failure questions encountered in its own training run. (a) Qwen3-4B resampling (b) Qwen3-8B surprisal weighting Figure 2: Proportion of all-failure groups that accept candidate trajectories. A point at step s aggregates steps s−49s-49 through s. Reference fallback is excluded. The curves come from their respective training runs. On Qwen3-4B, the proportion increases from 59.4%59.4\% to 71.8%71.8\% for ARG across the first and last windows, while resampling decreases from 37.6%37.6\% to 26.4%26.4\%. Over the full runs, ARG accepts candidate trajectories in 318 of 482 all-failure groups and uses reference fallback in the remaining 164. Resampling accepts candidate trajectories in 145 of 446 all-failure groups, leaving the remaining 301 unchanged. Resampling retains all correct candidate trajectories. Both methods use eight continuations per all-failure group, and ARG also uses reference guidance and fallback. The weighting ablation gives a different comparison. On Qwen3-8B, it accepts candidate trajectories in 64.8%64.8\% of all-failure groups, compared with 60.2%60.2\% for ARG, despite its lower evaluation scores. On Qwen3-4B, these proportions are similar at 65.9%65.9\% and 66.0%66.0\%. Thus, the proportion of groups that accept candidate trajectories alone does not explain the evaluation results. G.3 Surprisal Weighting and Prefix Selection The weighting ablation uses the successful replacement indicator as feedback, while retaining the same guidance levels, prefix construction, selector hyperparameters, and acceptance rule. Figure 3 compares the resulting selection probabilities. On Qwen3-4B, both selectors favor α=1/4α=1/4 for most of training. On Qwen3-8B, the unweighted selector favors α=1/4α=1/4 throughout training, whereas the probabilities under ARG change more over time. The final probabilities under ARG are 49.0%49.0\% for α=1/2α=1/2 and 34.6%34.6\% for α=1/4α=1/4, compared with 6.8%6.8\% and 86.4%86.4\% without weighting. (a) Qwen3-4B ARG (b) Qwen3-4B w/o weighting (c) Qwen3-8B ARG (d) Qwen3-8B w/o weighting Figure 3: Selection probabilities of the prefix selector with and without surprisal weighting. Curves show the state after each training step without smoothing. Steps with no selector update retain the previous probabilities. The level α=1α=1 uses an empty prefix, while smaller levels leave less of the reference surprisal to the continuation. Appendix H Training Dynamics We supplement Section 5.4 with the corresponding analysis on Qwen3-4B and the sources of accepted trajectories for both models. Comparisons with the ablated methods appear in Appendix G. H.1 Successful Replacement, Feedback, and Prefix Selection Figure 4 reports successful replacement rates, mean selector feedback, and prefix selection on Qwen3-4B. As in the Qwen3-8B plots, each point in the first two panels uses all continuations sampled at that level within a window of 50 training steps. Successful replacement requires a correct candidate trajectory different from the reference. We compute WtW_t with the actual St/S0S_t/S_0 after prefix-boundary adjustment, including zero feedback from failures and exact reference copies. The third panel shows the selection probabilities after each training step without smoothing. (a) Successful replacement (b) Selector feedback (c) Prefix selection Figure 4: Training dynamics of a Qwen3-4B ARG run. In panels (a) and (b), a point at step s averages the continuations at each level from steps s−49s-49 through s. Panel (c) shows the selection probabilities after each training step without smoothing. The colors and corresponding axis ranges match Figure 1. The level α=1/4α=1/4 has the highest successful replacement rate in each window. Weighting reduces its advantage over α=1/2α=1/2, and their mean-feedback ordering varies across windows. Over the full run, the mean feedback is 0.07540.0754 for α=1/4α=1/4 and 0.05880.0588 for α=1/2α=1/2. These observations accompany the persistent preference for α=1/4α=1/4 shown in Figure 4(c). The windowed statistics summarize continuations collected as the policy, the all-failure questions, and the selection probabilities change. Since the selector updates sequentially from these continuations, its state reflects their order and the preceding updates. Steps with no selector update retain the previous probabilities. This gives a view of the feedback observed during training alongside the resulting selection history. H.2 Sources of Accepted Trajectories Figure 5 records which trajectories enter policy learning. We call a candidate trajectory guided if it comes from a nonempty prefix and unguided if it comes from the empty prefix. Each all-failure group belongs to one of four categories, according to whether it accepts only guided candidate trajectories, only unguided ones, both types, or the reference as fallback. Thus, the figure describes groups, with each group counted once even when it accepts multiple trajectories. (a) Qwen3-4B (b) Qwen3-8B Figure 5: Sources of accepted trajectories among all-failure groups in consecutive windows of 50 training steps. The mixed category contains groups that accept both guided and unguided candidate trajectories. Fallback supplies the complete reference when no candidate trajectory is accepted. Counts below the windows give the number of all-failure groups. Candidate trajectories are accepted in 66.0%66.0\% and 60.2%60.2\% of all-failure groups for Qwen3-4B and Qwen3-8B, respectively. The remaining groups use reference fallback. Most groups that accept candidate trajectories use guided ones, while a smaller fraction accept unguided ones. Fallback remains present throughout training, and its proportion varies across windows as the all-failure questions change. These records complement the selection curves by showing the trajectories used for the policy updates.