Paper deep dive
Branching On-Policy Distillation
Dongyang Chen, Shudong Liu, Shubo Li, Duoxun Tang, Jinwei Liang, Zehuan Wu, Lewei Lu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 10/12/2026, 7:25:48 AM
Summary
The paper introduces Branching On-Policy Distillation (BranchOPD), a method that enhances on-policy distillation by selecting informative decision points based on student uncertainty and teacher-student disagreement, then resampling continuations from shared prefixes. This approach increases supervision coverage while reducing token costs. The authors also propose Protocol-Boundary Repair (PBR) to mitigate repetition and wasted generation. BranchOPD outperforms Vanilla OPD and Multi-Rollout OPD on mathematics and agent benchmarks.
Entities (13)
Relation Signals (11)
BranchOPD → evaluatedon → AIME-2024
confidence 98% · For mathematics, we report avg@16... on MATH-500, AIME 2024... BranchOPD gives the best trained-student results
BranchOPD → evaluatedon → Search-QA
confidence 98% · On WebShop... and on Search-Qa... BranchOPD gives the best trained-student results
BranchOPD → evaluatedon → AIME 2025
confidence 98% · For mathematics, we report avg@16... on MATH-500, AIME 2024, and AIME 2025... BranchOPD gives the best trained-student results
BranchOPD → evaluatedon → ALFWorld
confidence 98% · Shared-prefix reuse reduces student-generated token costs by approximately 19% on ALFWorld... BranchOPD outperforms... across all evaluated benchmarks
BranchOPD → evaluatedon → WebShop
confidence 98% · Shared-prefix reuse reduces student-generated token costs by approximately 31% on WebShop... BranchOPD outperforms... across all evaluated benchmarks
BranchOPD → evaluatedon → MATH-500
confidence 98% · For mathematics, we report avg@16... on MATH-500... BranchOPD gives the best trained-student results
Qwen3-4B → isstudentfor → Qwen3-30B-A3B
confidence 95% · For the agent tasks, the student is Qwen3-4B; the teacher is Qwen3-30B-A3B on ALFWorld
Qwen3-8B → isteacherfor →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy distillation (OPD) provides teacher supervision on student-generated trajectories, but a single trajectory realizes only one continuation at each decision, limiting supervision coverage under a finite sampling budget. We propose BranchOPD (Branching On-Policy Distillation), which selects informative decision points using student uncertainty and teacher-student disagreement, resamples student-policy continuations from shared prefixes, and applies token-level teacher supervision to both main trajectories and branches. BranchOPD operates at token-level decisions in mathematics and within-turn decisions in agent tasks. Shared-prefix reuse reduces student-generated token costs by approximately 19% on ALFWorld and 31% on WebShop relative to seven independent trajectories. We also introduce protocol-boundary repair (PBR), which uses targeted supervision under protocol constraints to mitigate repetition and wasted generation. BranchOPD outperforms Vanilla OPD and Multi-Rollout OPD (n=7) across all evaluated benchmarks, with gains over Vanilla OPD of 18.55 percentage points in mean mathematics accuracy over 16 sampled answers (avg@16) and 13.8 points in mean ALFWorld ID/OOD success. Mechanistic analyses and selection ablations show that branching at uncertain decisions expands subsequent behavior coverage and outperforms random selection and turn-initial branching.
Tags
Links
- Source: https://arxiv.org/abs/2610.13481v1
- Canonical: https://arxiv.org/abs/2610.13481v1
Trouble viewing inline? Open PDF directly →
Full Text
92,561 characters extracted from source content.
Expand or collapse full text
Branching On-Policy Distillation Dongyang Chen Affiliation: Tsinghua University Affiliation: SenseTime Research github.com/chendy25/BranchOPD huggingface.co/BranchOPD Shudong Liu Affiliation: Peking University Affiliation: SenseTime Research github.com/chendy25/BranchOPD huggingface.co/BranchOPD Shubo Li Affiliation: SenseTime Research github.com/chendy25/BranchOPD huggingface.co/BranchOPD Duoxun Tang Affiliation: Tsinghua University Jinwei Liang Affiliation: SenseTime Research github.com/chendy25/BranchOPD huggingface.co/BranchOPD Zehuan Wu Affiliation: SenseTime Research github.com/chendy25/BranchOPD huggingface.co/BranchOPD Lewei Lu †thanks: Corresponding author. Affiliation: SenseTime Research github.com/chendy25/BranchOPD huggingface.co/BranchOPD Abstract On-policy distillation (OPD) provides teacher supervision on student-generated trajectories, but a single trajectory realizes only one continuation at each decision, limiting supervision coverage under a finite sampling budget. We propose BranchOPD (Branching On-Policy Distillation), which selects informative decision points using student uncertainty and teacher–student disagreement, resamples student-policy continuations from shared prefixes, and applies token-level teacher supervision to both main trajectories and branches. BranchOPD operates at token-level decisions in mathematics and within-turn decisions in agent tasks. Shared-prefix reuse reduces student-generated token costs by approximately 19% on ALFWorld and 31% on WebShop relative to seven independent trajectories. We also introduce protocol-boundary repair (PBR), which uses targeted supervision under protocol constraints to mitigate repetition and wasted generation. BranchOPD outperforms Vanilla OPD and Multi-Rollout OPD (n=7n=7) across all evaluated benchmarks, with gains over Vanilla OPD of 18.55 percentage points in mean mathematics accuracy over 16 sampled answers (avg@16) and 13.8 points in mean ALFWorld ID/OOD success. Mechanistic analyses and selection ablations show that branching at uncertain decisions expands subsequent behavior coverage and outperforms random selection and turn-initial branching. 1 Introduction Distillation is a standard way to transfer the reasoning and interaction abilities of a large language model to a smaller one. Training on fixed teacher outputs (Hinton et al., 2015; Kim and Rush, 2016) exposes the student only to prefixes the teacher would write, whereas at inference the student must continue from its own. On-policy distillation (OPD) closes this gap: the student generates trajectories and the teacher scores every generated token (Agarwal et al., 2023; Ross et al., 2010; Bengio et al., 2015; Gu et al., 2023; Ko et al., 2024). Each trajectory, however, follows only one of the continuations available at every decision. Even with the teacher’s full next-token distribution at a prefix, the student receives no supervision on the multi-step histories that the unchosen tokens would have led to. This matters in mathematical reasoning, where an early step constrains the rest of the derivation, and in agent tasks, where an action changes all later observations (Yao et al., 2022b; Schick et al., 2023; Wang et al., 2022b; Wei et al., 2022; Qin et al., 2023). Sampling more independent trajectories widens coverage, but most of the extra tokens regenerate identical or near-identical prefixes. As Figure 1 shows, resampling from a single prefix can reach quite different paths, sometimes even when the first resampled token matches the original. Recent OPD methods decide where to spend supervision on trajectories that have already been sampled: they reweight tokens, reasoning steps, or turns (Xu et al., 2026; Zhong et al., 2026; Tan et al., 2026; Sheng et al., 2026), order trajectories by difficulty (Wang et al., 2026b), or reuse teacher-scored samples (Zheng et al., 2026). We ask instead which additional trajectories to sample. Tree search and branching reinforcement learning already share prefixes to explore locally (Li et al., 2025; Dong et al., 2025; Zong et al., 2026; Ji et al., 2025; Yao et al., 2023; Zhou et al., 2023a; Hao et al., 2023; Khomich et al., 2026; Yu et al., 2026c; He et al., 2026); we bring this idea to distillation, where every branch token receives dense teacher supervision and no task reward is needed. Figure 1: Resampling from one prefix on a mathematics problem and a logged ALFWorld trajectory. Scores mark the fork; q is the student probability of each continuation’s first token. We propose BranchOPD. For each input, the student first generates a main trajectory. We then pick a few fork points using student uncertainty and teacher–student disagreement, and sample new continuations from the frozen student policy at each fork. In mathematics tasks, we select token positions directly; in agent tasks we first select turns and then positions within them. The teacher scores all generated tokens on the main trajectory and the branches, and a shared prefix enters the loss only once. Because branches come from the student rather than from the teacher or a refined trajectory (Jiang et al., 2026; Zhou et al., 2026), they stay on the student’s own distribution. A commonly encountered failure mode in OPD is improper termination: the student may continue generating after producing the final answer, often emitting redundant content or stray formatting tokens (Yang et al., 2026b; Luo et al., 2026). We handle this problem by protocol-boundary repair (PBR), an auxiliary loss on detected protocol events, and apply PBR to the baselines so that comparisons isolate the effect of branching. Our contributions are: • Branching on-policy distillation. A shared-prefix sampling scheme that selects informative decisions and resamples student continuations from them for token-level teacher supervision, in both mathematical reasoning and multi-turn agent tasks. • Protocol-boundary repair. An auxiliary loss that reinforces beneficial and suppresses harmful protocol events in student outputs, reducing repetition and wasted generation. • Experiments and analysis. BranchOPD outperforms Vanilla OPD and Multi-Rollout OPD on all evaluated benchmarks, including when every method uses PBR, while generating about 19% and 31% fewer student tokens than seven independent trajectories on ALFWorld and WebShop. Ablations and diagnostics show that the choice of fork points matters. 2 Related Work On-policy distillation. GKD, MiniLLM, and DistiLLM train students on their own samples and study which divergence to minimize (Agarwal et al., 2023; Gu et al., 2023; Ko et al., 2024). Reasoning supervision can also come from solution selection, process feedback, or explanations (Lightman et al., 2023; Zelikman et al., 2022; Hsieh et al., 2023). More recent OPD methods reweight supervision across tokens (Xu et al., 2026; Sheng et al., 2026), reasoning steps (Zhong et al., 2026), or interaction turns (Tan et al., 2026), or schedule trajectories by difficulty (Wang et al., 2026b). EOPD adds a forward-KL term at tokens where the teacher has high entropy (Jin et al., 2026), and other work reformulates the target or constrains the update (Jang et al., 2026; Ko et al., 2026). Closest to ours, SPOT forces teacher-proposed tokens at selected positions, scores the resulting student continuations with an outcome verifier, and distills a calibrated local target (Qu et al., 2026). In BranchOPD, branches are sampled from the student itself and supervised token by token, without task rewards. Quality of distillation supervision. Probability mismatch and degenerate agreement motivate studies of what makes OPD supervision useful (Yu et al., 2026d; Xin et al., 2026; Armandpour et al., 2026). Trajectory refinement and expert corrections improve the examples shown to the student (Jiang et al., 2026; Lauffer et al., 2025), and teacher continuations from student anchors extend supervision beyond sampled rollouts (Zhou et al., 2026); our continuations come from the student. Our objective is token-local and complements work that separates token-level updates from sequence-level credit assignment (Liu et al., 2026). PBR follows unlikelihood training (Welleck et al., 2019) in supervising specific output events. Tree search and reinforcement learning. Local and tree-structured exploration reuses prefixes to sample alternative decisions (Li et al., 2025; Dong et al., 2025; Zong et al., 2026; Ji et al., 2025). Recent methods branch at high-entropy tokens with sibling-return baselines (He et al., 2026), adapt rollout budgets (Khomich et al., 2026; Yu et al., 2026c), or roll back the environment and reflect (Yu et al., 2026b). These methods compare branches through task rewards; BranchOPD uses shared-prefix exploration to collect more tokens for teacher supervision. Appendix E.6 covers further learning and systems work. 3 Preliminaries 3.1 Student trajectories Let x∼x be an input, V the vocabulary, πθ _θ the student policy, and πT _T a frozen teacher. We write at∈a_t for a generated token and sts_t for its prefix. In mathematical reasoning, a trajectory is a single response τ=(x,a1,…,a|τ|)τ=(x,a_1,…,a_|τ|) that ends at a stop token or the length limit, with st=(x,a<t)s_t=(x,a_<t) and πθ(τ∣x)=∏tπθ(at∣st) _θ(τ x)= _t _θ(a_t s_t). In multi-turn agent tasks, following Tan et al. (2026), the student interacts with an environment ℰE for up to M turns: τ τ =(x,y1,o1,…,yM,oM), =(x,y_1,o_1,…,y_M,o_M), πθ(τ∣x) _θ(τ x) =∏t∈(τ)πθ(at∣st)∏m=1MPℰ(om∣x,y≤m,o<m). = _t (τ) _θ(a_t s_t) _m=1^MP_E(o_m x,y_≤ m,o_<m). (1) Each response ymy_m is a token sequence that may contain reasoning and an environment action, and omo_m is the resulting observation. (τ)T(τ) indexes the student-generated tokens, whose prefixes sts_t include earlier responses and observations; observations are never supervised. Mathematics is the special case M=1M=1 without observations. For t∈(τ)t (τ), let τ≥t _≥ t denote the continuation from sts_t, including later environment transitions, and πθ(τ≥t∣st) _θ( _≥ t s_t) its distribution. To resume from sts_t in an agent task, the environment must be restored to its state at sts_t, by snapshot or action replay, so that the resumed continuation has the same distribution as an uninterrupted rollout. 3.2 On-policy distillation OPD trains the student on its own trajectories under teacher supervision (Agarwal et al., 2023; Gu et al., 2023; Ko et al., 2024). Trajectories are sampled from πθold _ _ old, a frozen copy of the student; we sample at temperature one without truncation, so πθold _ _ old is exactly the sampling distribution. OPD minimizes the reverse KL divergence on student-visited prefixes: ℒOPD(θ)=x∼,τ∼πθold(⋅∣x)[1|(τ)|∑t∈(τ)KL(πθ(⋅∣st)∥πT(⋅∣st))].L_ OPD(θ)=E_x ,\,τ _ _ old(· x) [ 1|T(τ)| _t (τ)KL ( _θ(· s_t)\,\|\, _T(· s_t) ) ]. (2) In practice, the teacher scores only the sampled tokens, which gives the token-level distillation signal Δlogpt=logπT(at∣st)−logπθ(at∣st). p_t= _T(a_t s_t)- _θ(a_t s_t). (3) If the teacher’s support covers the student’s, at∼πθ(⋅∣st)[−Δlogpt]E_a_t _θ(· s_t)[- p_t] equals the reverse KL at sts_t. Δlogpt p_t is positive when the teacher assigns the sampled token higher probability than the student does. With the importance ratio rt(θ)=πθ(at∣st)/πθold(at∣st)r_t(θ)= _θ(a_t s_t)/ _ _ old(a_t s_t) and πθ≪πθold _θ _ _ old, the gradient at each prefix is ∇θKL(πθ(⋅∣st)∥πT(⋅∣st))=−at∼πθold(⋅∣st)[rt(θ)Δlogpt∇θlogπθ(at∣st)]. _θKL ( _θ(· s_t)\,\|\, _T(· s_t) )=-E_a_t _ _ old(· s_t) [r_t(θ)\, p_t\, _θ _θ(a_t s_t) ]. (4) OPD is thus a policy-gradient update with the detached token advantage AtOPD=sg[Δlogpt]A OPD_t=sg[ p_t], usually optimized with the PPO clipped surrogate (Schulman et al., 2017; Tan et al., 2026). With n trajectories τii=1n\ _i\_i=1^n per input and N=∑i|(τi)|N= _i|T( _i)|, ℒclip(θ)=−1N∑i=1n∑t∈(τi)minri,t(θ)Ai,tOPD,clip(ri,t(θ),1−ϵ,1+ϵ)Ai,tOPD.L_ clip(θ)=- 1N _i=1^n _t ( _i) \r_i,t(θ)A OPD_i,t,\,clip(r_i,t(θ),1-ε,1+ε)A OPD_i,t \. (5) Vanilla OPD uses n=1n=1; Multi-Rollout OPD uses n>1n>1 independent trajectories. As in the local distillation update of Liu et al. (2026), gradients do not flow through the sampled prefixes (derivations in Appendix A). 4 Method As Figure 2 shows, BranchOPD selects fork points on a student trajectory, resamples continuations from them with the student policy, and trains on all generated tokens with the OPD loss and PBR. Figure 2: Overview of BranchOPD. Fork points are selected on the main trajectory, suffixes are resampled from the student, and the teacher scores every generated token; shared prefixes are counted once. The bottom examples show concrete PBR constraints used in Section 5.5. 4.1 Selecting fork points For each input x, we sample a main trajectory τ0∼πθold(⋅∣x) _0 _ _ old(· x) and select Beff≤B_ eff≤ B fork positions t1,…,tBeff∈(τ0)t_1,…,t_B_ eff ( _0) with prefixes sb=stbs_b=s_t_b. From each prefix we independently sample k new suffixes from πθold(⋅∣sb) _ _ old(· s_b), so each input yields at most 1+Bk1+Bk trajectories. In agent tasks the environment is first restored to its state at sbs_b; in mathematics, generation simply continues from sbs_b. We keep every sampled suffix, including those whose first token repeats the main trajectory’s, since they can still diverge later. Candidate positions are scored with two signals. The uncertainty score UtU_t is the student’s entropy, computed from its top-20 token probabilities. The disagreement score DtD_t aggregates −Δlogpt- p_t over a local window of 128 tokens, so it is large where the student prefers tokens that the teacher does not. Both scores are rank-normalized within the trajectory (denoted R) and mixed: vt=αR(Ut)+(1−α)R(Dt),vturn(m)=mean(TopFracηvt:t∈ym),v_t=α R(U_t)+(1-α)R(D_t), v_ turn(m)=mean (TopFrac_η\v_t:t∈ y_m\ ), (6) where TopFracηTopFrac_η keeps the top η fraction of scores within turn m. In mathematics, we select up to B forks among the highest-scoring eligible positions. In agent tasks, we select up to B turns with the highest vturnv_ turn, subject to a minimum gap between selected turns, and place each fork at a position drawn uniformly from the three highest-scoring eligible positions of its turn. By default, α=0.5α=0.5, η=0.2η=0.2, B=3B=3, and k=2k=2; the scores are not used in the loss. Appendix D lists the eligibility rules and selection settings. 4.2 Conditional resampling and coverage Let ℱ0F_0 collect the main trajectory and the selected forks, and let τb,j _b,j be the j-th suffix sampled from sbs_b. We fix πθold _ _ old and θ before sampling the branches. For an integrable suffix statistic fθf_θ, define Jθ(sb)=τ≥tb∼πθold(⋅∣sb)fθ(τ≥tb),J^RS(sb)=1k∑j=1kfθ(τb,j).J_θ(s_b)=E_ _≥ t_b _ _ old(· s_b)f_θ( _≥ t_b), J_ RS(s_b)= 1k _j=1^kf_θ( _b,j). (7) Proposition 1 (Conditional unbiasedness). If restored contexts have the correct continuation distribution and each branch satisfies τb,j∣ℱ0∼πθold(⋅∣sb) _b,j _0 _ _ old(· s_b), then [J^RS(sb)∣ℱ0]=Jθ(sb)E[ J_ RS(s_b) _0]=J_θ(s_b). The result follows from linearity of conditional expectation (Appendix A). If the branches are also conditionally independent with finite variance, Var(J^RS∣ℱ0)=Var(fθ∣ℱ0)/kVar( J_ RS _0)=Var(f_θ _0)/k. Fork selection may depend on the main trajectory, but it is contained in ℱ0F_0 and therefore does not bias this estimate. Branching also widens coverage at the fork. Let a0a_0 be the main-trajectory token at sbs_b and ab,ja_b,j the first token of τb,j _b,j. Then Pr(a visited∣ℱ0) (a visited _0) =1−[1−πθold(a∣sb)]k, =1-[1- _ _ old(a s_b)]^k, Pr(∃j:ab,j≠a0∣ℱ0) (∃ j:a_b,j≠ a_0 _0) =1−πθold(a0∣sb)k. =1- _ _ old(a_0 s_b)^k. (8) Both probabilities grow with k but with diminishing increments, which motivates spreading the budget over several forks. 4.3 Training objective The training batch contains the main trajectory and the BeffkB_ effk branches, indexed by i. The mask Mi,tM_i,t keeps all student-generated tokens of the main trajectory and, on each branch, only the tokens from the fork onward; observations and padding are masked out. We compute Δlogpi,t p_i,t (Equation 3) with the current parameters, apply a log-probability floor, and detach the result to form the advantage. With signal threshold δ and PPO clipping ϵε (Schulman et al., 2017), Equation 5 becomes Ai,t A_i,t =clip(sg[Δlogpi,t],−δ,δ),ri,t(θ)=πθ(ai,t∣si,t)πθold(ai,t∣si,t), =clip\! (sg[ p_i,t],-δ,δ ), r_i,t(θ)= _θ(a_i,t s_i,t) _ _ old(a_i,t s_i,t), (9) gi,t g_i,t =−minri,tAi,t,clip(ri,t,1−ϵ,1+ϵ)Ai,t, =- \r_i,tA_i,t,clip(r_i,t,1-ε,1+ε)A_i,t\, (10) Ltree L_ tree =∑i,tMi,tgi,t∑i,tMi,t. = _i,tM_i,tg_i,t _i,tM_i,t. (11) Because the loss averages over valid tokens, longer suffixes contribute more terms, and each shared prefix is counted once. 4.4 Protocol-boundary repair Some errors are defined by the task protocol rather than by the teacher: a mathematical response should end after its final answer, and an agent response should not begin with a stray closing-thought tag. OPD does not reliably remove them, because a teacher conditioned on a repetitive student prefix may continue the repetition itself (Appendix B.5). PBR supervises such events directly. Fixed detection rules scan each generated trajectory and emit constraints (s,C,b)(s,C,b): at prefix s, the next token should (b=1b=1) or should not (b=0b=0) lie in a nonempty token set C⊊C ; detection is not differentiated. We define pθ(C∣s) p_θ(C s) =∑v∈Cπθ(v∣s), = _v∈ C _θ(v s), ℓproto(s,C,b) _ proto(s,C,b) =−blogpθ(C∣s)−(1−b)log[1−pθ(C∣s)], =-b p_θ(C s)-(1-b) [1-p_θ(C s)], (12) Ltrain L_ train =Ltree+Ψ(λmean(s,C,b)∈batchℓproto), =L_ tree+ \! ( _(s,C,b) _ batch _ proto ), (13) where batchC_ batch is the set of constraints detected in the batch and λ is the PBR weight; the auxiliary term is zero when batchC_ batch is empty. Ψ is the identity, or Ψ(u)=min(u,cPBR) (u)= (u,c_ PBR) when a loss cap is used. Positive constraints raise the probability of beneficial events and negative constraints lower that of harmful ones, as in unlikelihood training (Welleck et al., 2019). PBR applies equally to main and branch trajectories. Appendix F gives pseudocode. 5 Experiments 5.1 Experimental setup We evaluate on mathematical reasoning and three agent benchmarks. For mathematics, we distill Qwen3-8B into Qwen3-4B-Base on DAPO-Math-17K (Yang et al., 2025; Yu et al., 2025). For the agent tasks, the student is Qwen3-4B; the teacher is Qwen3-30B-A3B on ALFWorld and a GRPO-trained checkpoint of the same model on WebShop and Search-QA. To test other student sizes and model families, we also distill Qwen3-4B into Qwen3-1.7B, and Qwen2.5-7B-RL into Qwen2.5-3B on ALFWorld. Benchmarks, prompts, and hyperparameters are described in Appendices E, G, and D. Evaluation. For mathematics, we report avg@16, the accuracy averaged over 16 sampled answers per problem, on MATH-500, AIME 2024, and AIME 2025. On ALFWorld (Shridhar et al., 2020), we report success rate (SR) on 140 seen (ID) and 134 unseen (OOD) tasks and their mean. On WebShop (Yao et al., 2022a), we report SR on 128 held-out tasks, and on Search-QA, exact-match SR on the same mixed validation set for all models. Agents are evaluated with one trajectory per task; no branching is used at evaluation time. Baselines. Our main baselines are Vanilla OPD, Multi-Rollout OPD with seven independent trajectories per input (matching the 1+Bk=71+Bk=7 trajectories of BranchOPD), and Entropy-Weighted OPD, which reweights the token losses of a single trajectory. Table 1 also includes offline KD, GRPO, EOPD (Jin et al., 2026), and SPOT (Qu et al., 2026) (Appendix D.2); Table 2 includes KD, GRPO, and the curriculum method TCOD (Wang et al., 2026b). Baselines trained with PBR are marked (+PBR). Table 1: Mathematics avg@16 (%). Avg is the macro-average across the three evaluations. Baseline variants using protocol-boundary repair are marked (+PBR). Method MATH-500 AIME24 AIME25 Avg Qwen3-8B teacher 89.13 58.54 46.25 64.64 Qwen3-4B-Base student 48.91 8.75 7.50 21.72 KD 74.73 12.50 12.08 33.10 GRPO 79.18 14.17 13.33 35.56 Vanilla OPD 64.33 7.08 6.04 25.82 Multi-Rollout OPD (n=7n=7) 66.64 10.21 6.67 27.84 Entropy-Weighted OPD 67.94 12.29 8.96 29.73 EOPD 79.73 17.50 16.67 37.97 SPOT 79.88 17.50 17.92 38.43 Vanilla OPD (+PBR) 79.13 19.58 18.13 38.94 Multi-Rollout OPD (n=7n=7, +PBR) 78.94 19.79 18.54 39.09 Entropy-Weighted OPD (+PBR) 80.24 21.88 20.83 40.98 BranchOPD 81.24 26.04 25.83 44.37 Table 2: Performance on ALFWorld, Search-QA, and WebShop. We report success rate (SR, %); ALFWorld reports ID SR and OOD SR. Avg. SR is the mean success rate over the three datasets, averaging ALFWorld ID and OOD first. † : the WebShop and Search-QA teacher checkpoints are GRPO-trained. Method ALFWorld WebShop Search-QA Avg. SR ID SR OOD SR SR SR Qwen3-30B-A3B 53.57 50.00 31.25† 51.12† 44.72 Qwen3-4B student 35.71 35.82 2.34 28.71 22.27 KD 38.57 38.06 18.75 43.50 33.52 GRPO 51.43 50.75 31.25 49.00 43.78 Vanilla OPD 42.14 43.28 25.78 48.24 38.91 Multi-Rollout OPD (n=7n=7) 43.57 42.54 26.56 48.26 39.29 Entropy-Weighted OPD 46.43 46.27 27.34 48.56 40.75 TCOD 52.14 50.75 31.25 48.70 43.80 Vanilla OPD (+PBR) 49.29 49.25 30.47 48.58 42.77 Multi-Rollout OPD (n=7n=7, +PBR) 50.00 47.76 31.25 48.60 42.91 Entropy-Weighted OPD (+PBR) 52.86 51.49 32.03 48.90 44.37 BranchOPD 57.86 55.22 33.59 49.05 46.39 5.2 Main Results BranchOPD gives the best trained-student results in both tables, improving over Vanilla OPD by 18.55 points in mathematics Avg and 13.83 points in mean ALFWorld success (Tables 1 and 2). With PBR on every method, it reaches 44.37 mathematics Avg, 5.43 and 5.28 points above Vanilla OPD and Multi-Rollout OPD (n = 7), and above Entropy-Weighted OPD. Mathematical reasoning. BranchOPD outperforms KD, GRPO, EOPD, and SPOT by 11.27, 8.81, 6.40, and 5.94 Avg points (Table 1). Gains are larger on AIME than on MATH-500: over SPOT, BranchOPD improves by 1.36 points on MATH-500 but by 8.54 and 7.91 points on AIME24 and AIME25. GRPO and SPOT use answer verification, whereas BranchOPD uses only teacher supervision. Agent tasks. Averaged over the three benchmarks, BranchOPD reaches 46.39 SR, outperforming KD, GRPO, and TCOD by 12.87, 2.62, and 2.60 points, respectively (Table 2). On ALFWorld, it raises mean success from 49.27% for Vanilla OPD (+PBR) to 56.54%, ahead of the other PBR baselines and 4.75 points above its teacher (51.79%), without any task reward. On WebShop, it raises SR from 30.47% for Vanilla OPD (+PBR) to 33.59%, above the teacher (31.25%). On Search-QA, the gain over Vanilla OPD (+PBR) is 0.47 points. Tables 4 and 5 report results for a smaller student and another model family. 5.3 Ablations: Where and How Much to Branch Table 3: Ablations of fork selection and branch budget, all with PBR (ALFWorld SR and mathematics Avg, %). Fork selection rows share the eligible positions, budget, and resampling, and differ only in how forks are chosen. Random: uniform over eligible positions, without scoring (agent tasks keep the minimum turn gap). Entropy only / Disagreement only: vtv_t in Equation 6 with α=1α=1 / α=0α=0. Turn-initial (agent tasks only): turns chosen by the same vturnv_ turn, but each fork is placed at the turn’s first generated token. Branch budget rows vary the number of forks B and continuations per fork k (1+Bk1+Bk trajectories per input). Setting ALFWorld Math Avg ID SR OOD SR Fork selection (B=3,k=2B=3,k=2) Random 50.71 46.27 40.11 Entropy only (α=1α=1) 55.71 52.99 43.24 Disagreement only (α=0α=0) 55.00 52.24 42.97 Turn-initial 55.71 53.73 – Entropy + disagreement (α=0.5α=0.5) 57.86 55.22 44.37 Branch budget (entropy + disagreement) B=3,k=1B=3,k=1 54.29 50.75 42.18 B=3,k=3B=3,k=3 57.14 54.48 43.91 B=2,k=2B=2,k=2 57.14 54.48 43.94 B=3,k=2B=3,k=2 (default) 57.86 55.22 44.37 Where to branch. The caption of Table 3 defines each selection rule. Mixing entropy and disagreement works best. Compared with random forks, it improves ALFWorld ID/OOD SR by 7.14/8.96 points and raises mathematics Avg from 40.11 to 44.37. Either signal alone recovers most, but not all, of this gain. Turn-initial branching trails the default by 2.14/1.49 points on ALFWorld, so where the fork falls within a turn also matters. Table 6 reports further selection and filtering controls. How much to branch. With B=3B=3, increasing the continuations per fork from k=1k=1 to k=2k=2 improves all three metrics, while k=3k=3 brings no further gain. With k=2k=2, going from two to three forks adds 0.43 mathematics points and 0.73 ALFWorld points, so it pays to cover several decisions with a few continuations each (Table 7). 5.4 Behavior Coverage and Sampling Efficiency Behavior coverage is the range of alternative behaviors the student explores from the same decision state. We measure both the probability mass left to alternatives at a fork, 1−πθold(a0∣s)1- _ _ old(a_0 s) with a0a_0 the main-trajectory token, and the behavioral differences that branches actually produce. At the selected forks (step 100), this mass is about 0.63 on ALFWorld and 0.69 on WebShop, and branches do produce different behavior: 74.2% and 47.7% take a different action sequence from the main trajectory, and 55.5% and 37.5% contain an action absent from it (Table 14). Branching thus turns mass that a single rollout leaves unrealized into additional histories with teacher feedback. Branches can also recover from failures: when the main trajectory fails, at least one branch succeeds in 5.0% (WebShop) to 42.7% (mathematics) of cases (Table 15). Appendix C tracks these statistics over training. Figure 3: ALFWorld generation cost and success with PBR, for three (left) and seven (right) trajectory slots. OPD, Independent, and Reweighting denote Vanilla OPD, Multi-Rollout OPD, and Entropy-Weighted OPD. While achieving broader behavior coverage, BranchOPD requires fewer student-generated tokens by reusing the prefix shared with the main trajectory. On ALFWorld, one trajectory requires 2,813 student tokens on average; seven independent trajectories would therefore require 19,691 tokens, whereas our seven-trajectory branching setting uses only 15,924 tokens, saving 19.1%. On WebShop, the corresponding cost is 4,824 tokens versus 7,014 for seven independent trajectories, saving 31.2% (Figure 3; Table 11). These counts exclude teacher scoring, backpropagation, and environment replay. 5.5 Protocol-Boundary Repair Figure 4: Training dynamics with PBR: mathematics response length and truncation rate (fraction of responses stopped at the length limit), and ALFWorld turn-level protocol violation (fraction of turns with a violation). Repetition penalties delay budget exhaustion but leave generation near the cap; PBR reduces both truncation and violations. PBR explicitly controls protocol events that should either be encouraged or suppressed. For example, it encourages termination after a complete mathematical answer by setting (C=EOSEOS) and (b=1), while suppressing extra leading closing-thought tags in agent responses with (b=0). On mathematics, PBR alone lifts Vanilla OPD from 25.82 to 38.94 Avg (Table 1) and leaves more room below the length limit (Figure 4). On ALFWorld, it improves ID/OOD SR by 7.14/5.97 points and removes the extra leading tags entirely. A teacher that does not repeat in its own rollouts can still reinforce repetition when conditioned on a repetitive student prefix, so the repair must act on student prefixes (Appendix B.5). Why targeted repair? We also compare PBR with generic remedies that broadly discourage repetition or truncate problematic samples, which are consistently less effective than PBR (Table 13). Repetition penalties achieve only 28.70–30.40 Avg, filtering overlong samples reaches 33.20, and early stopping reaches 22.05, compared with 38.94 for PBR. Moreover, repetition penalties mainly slow the increase in response length rather than preventing it from eventually approaching the generation limit (Figure 4). 6 Conclusion We presented BranchOPD, which resamples student continuations at selected decision points to broaden on-policy distillation, together with protocol-boundary repair. It outperforms Vanilla OPD and Multi-Rollout OPD on every evaluated benchmark while generating fewer student tokens than independent rollouts. What a student learns from OPD depends not only on the teacher signal but also on which of its own alternative histories it gets to see. AI Use Statement We used large language models to assist with literature retrieval and synthesis, paper drafting and translation, refinement of theoretical formulations and proofs, and interpretation of results. The authors are responsible for the accuracy, originality, and final content of the submission, including AI-assisted text, claims, and artifacts. Ethics Statement Our research uses public benchmarks and controlled task environments, and does not involve human-subject experiments or collection of sensitive personal data. References Agarwal et al. (2023) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. External Links: Link, 2306.13649 Cited by: §1, §2, §3.2. Armandpour et al. (2026) M. Armandpour, F. Ilhan, D. Harrison, A. Jaiswal, D. N. M. Hoang, F. Faghri, Y. Zhang, M. Cho, and M. Farajtabar Unmasking on-policy distillation: where it helps, where it hurts, and why. External Links: Link, 2605.10889 Cited by: §2. Bengio et al. (2015) S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer Scheduled sampling for sequence prediction with recurrent neural networks. External Links: Link, 1506.03099 Cited by: §1. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. External Links: Link, 2110.14168 Cited by: §E.5. Dao et al. (2022) T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré FlashAttention: fast and memory-efficient exact attention with io-awareness. External Links: Link, 2205.14135 Cited by: §E.6. DeepSeek-AI et al. (2025) DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, S. Ye, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Zhao, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Xu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. External Links: Link, 2501.12948 Cited by: §E.6. Dong et al. (2025) G. Dong, H. Mao, K. Ma, L. Bao, Y. Chen, Z. Wang, Z. Chen, J. Du, H. Wang, F. Zhang, G. Zhou, Y. Zhu, J. Wen, and Z. Dou Agentic reinforced policy optimization. External Links: Link, 2507.19849 Cited by: §1, §2. Feng et al. (2025) L. Feng, Z. Xue, T. Liu, and B. An Group-in-group policy optimization for llm agent training. External Links: Link, 2505.10978 Cited by: §E.6. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. De Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma The llama 3 herd of models. External Links: Link, 2407.21783 Cited by: §E.6. Gu et al. (2023) Y. Gu, L. Dong, F. Wei, and M. Huang MiniLLM: on-policy distillation of large language models. External Links: Link, 2306.08543 Cited by: §1, §2, §3.2. Hao et al. (2023) S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu Reasoning with language model is planning with world model. External Links: Link, 2305.14992 Cited by: §1. He et al. (2026) B. He, Y. Chen, X. Zhang, and X. Liu Branching policy optimization: sandbox-native language agent reinforcement learning. External Links: Link, 2607.14171 Cited by: §1, §2. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset. External Links: Link, 2103.03874 Cited by: §E.1. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. External Links: Link, 1503.02531 Cited by: §1. Horvitz and Thompson (1952) D. G. Horvitz and D. J. Thompson A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, p. 663–685. External Links: Link, Document Cited by: §A.5. Hsieh et al. (2023) C. Hsieh, C. Li, C. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C. Lee, and T. Pfister Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. External Links: Link, 2305.02301 Cited by: §2. Jang et al. (2026) I. Jang, J. Yeom, J. Yeo, H. Lim, and T. Kim Stable on-policy distillation through adaptive target reformulation. External Links: Link, 2601.07155 Cited by: §2. Ji et al. (2025) Y. Ji, Z. Ma, Y. Wang, G. Chen, X. Chu, and L. Wu Tree search for llm agent reinforcement learning. External Links: Link, 2509.21240 Cited by: §1, §2. Jiang et al. (2026) L. Jiang, H. Xu, Y. Ding, and A. Zhang Trajectory-refined distillation. External Links: Link, 2606.08432 Cited by: §1, §2. Jin et al. (2025) B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han Search-r1: training llms to reason and leverage search engines with reinforcement learning. External Links: Link, 2503.09516 Cited by: §E.6. Jin et al. (2026) W. Jin, T. Min, Y. Yang, D. Wei, Y. Zhou, S. R. Kadhe, N. Baracaldo, and K. Lee Entropy-aware on-policy distillation of language models. External Links: Link, 2603.07079 Cited by: §D.2, §2, §5.1. Joshi et al. (2017) M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. External Links: Link, 1705.03551 Cited by: §E.5. Khomich et al. (2026) N. Khomich, L. Hermansson, and I. Hakimi EPIG-tree: compute-optimal branching for gradient-efficient reinforcement learning. External Links: Link, 2609.20004 Cited by: §1, §2. Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation. External Links: Link, 1606.07947 Cited by: §1. Ko et al. (2026) J. Ko, S. Abdali, Y. J. Kim, T. Chen, and P. Cameron Scaling reasoning efficiently via relaxed on-policy distillation. External Links: Link, 2603.11137 Cited by: §2. Ko et al. (2024) J. Ko, S. Kim, T. Chen, and S. Yun DistiLLM: towards streamlined distillation for large language models. External Links: Link, 2402.03898 Cited by: §1, §2, §3.2. Kwiatkowski et al. (2019) T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, p. 452–466. External Links: Link, Document Cited by: §E.4. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with pagedattention. External Links: Link, 2309.06180 Cited by: §E.6. Lauffer et al. (2025) N. Lauffer, X. Deng, S. Kundurthy, B. Kenstler, and J. Da Imitation learning for multi-turn lm agents via on-policy expert corrections. External Links: Link, 2512.14895 Cited by: §2. Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra Solving quantitative reasoning problems with language models. External Links: Link, 2206.14858 Cited by: §E.5. Li et al. (2025) Q. Li, R. Xue, J. Wang, M. Zhou, Z. Li, X. Ji, Y. Wang, M. Liu, Z. Yang, M. Qiu, and J. Yang CURE: critical-token-guided re-concatenation for entropy-collapse prevention. External Links: Link, 2508.11016 Cited by: §1, §2. Lightman et al. (2023) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. External Links: Link, 2305.20050 Cited by: §E.1, §2. Liu et al. (2026) S. Liu, Z. He, L. Tao, G. Zhan, J. Gao, F. Zhang, J. Duan, W. Xiong, K. Sheng, B. Zhang, Y. Guan, and S. E. Li Beyond token-local imitation: reward-compatible temporal credit assignment for on-policy distillation. External Links: Link, 2609.16937 Cited by: §A.1, §2, §3.2. Lu et al. (2026) Z. Lu, Z. Yao, Z. Han, Z. Wang, J. Wu, Q. Gu, X. Cai, W. Lu, J. Xiao, Y. Zhuang, and Y. Shen Self-distilled agentic reinforcement learning. External Links: Link, 2605.15155 Cited by: §E.6. Luo et al. (2026) F. Luo, Y. Chuang, G. Wang, Z. Xu, X. Han, T. Zhang, and V. Braverman Demystifying opd: length inflation and stabilization strategies for large language models. arXiv preprint arXiv:2604.08527. Cited by: §1. Luo et al. (2025) X. Luo, Y. Zhang, Z. He, Z. Wang, S. Zhao, D. Li, L. K. Qiu, and Y. Yang Agent lightning: train any ai agents with reinforcement learning. External Links: Link, 2508.03680 Cited by: §E.6. Penaloza et al. (2026) E. Penaloza, D. Vattikonda, N. Gontier, A. Lacoste, L. Charlin, and M. Caccia Privileged information distillation for language models. External Links: Link, 2602.04942 Cited by: §E.6. Qi et al. (2024) Z. Qi, X. Liu, I. L. Iong, H. Lai, X. Sun, W. Zhao, Y. Yang, X. Yang, J. Sun, S. Yao, T. Zhang, W. Xu, J. Tang, and Y. Dong WebRL: training llm web agents via self-evolving online curriculum reinforcement learning. External Links: Link, 2411.02337 Cited by: §E.6. Qin et al. (2023) Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun ToolLLM: facilitating large language models to master 16000+ real-world apis. External Links: Link, 2307.16789 Cited by: §1. Qu et al. (2026) Z. Qu, M. Zhang, M. Kong, Z. Shang, Z. Chen, Y. Ban, S. Qiu, and Z. Dai SPOT: sparse probing and outcome calibration for on-policy distillation. External Links: 2608.04419, Link Cited by: §D.2, §2, §5.1. Qwen et al. (2024) Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. External Links: Link, 2412.15115 Cited by: §E.6. Ross et al. (2010) S. Ross, G. J. Gordon, and J. A. Bagnell A reduction of imitation learning and structured prediction to no-regret online learning. External Links: Link, 1011.0686 Cited by: §1. Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools. External Links: Link, 2302.04761 Cited by: §1. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. External Links: Link, 1707.06347 Cited by: §E.6, §3.2, §4.3. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: Link, 2402.03300 Cited by: §E.6. Sheng et al. (2024) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu HybridFlow: a flexible and efficient rlhf framework. External Links: Link, 2409.19256 Cited by: §E.6. Sheng et al. (2026) H. Sheng, Z. Ye, H. Wang, J. Wang, J. Gu, and J. Kang 1% of tokens can be enough: on gradient estimation in on-policy distillation. External Links: Link, 2609.24432 Cited by: §1, §2. Shridhar et al. (2020) M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht ALFWorld: aligning text and embodied environments for interactive learning. External Links: Link, 2010.03768 Cited by: §E.2, §5.1. Tan et al. (2026) Q. Tan, Z. Zong, M. Li, Y. Shi, Y. Li, and P. Chen ATOD: annealed turn-aware on-policy distillation for multi-turn agentic tasks. External Links: Link, 2606.27814v6 Cited by: §1, §2, §3.1, §3.2. Trivedi et al. (2021) H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal MuSiQue: multihop questions via single-hop question composition. External Links: Link, 2108.00573 Cited by: §E.5. Wang et al. (2026a) H. Wang, G. Wang, H. Xiao, Y. Zhou, Y. Pan, J. Wang, K. Xu, Y. Wen, X. Ruan, X. Chen, and H. Qi Skill-sd: skill-conditioned self-distillation for multi-turn llm agents. External Links: Link, 2604.10674 Cited by: §E.6. Wang et al. (2026b) J. Wang, W. Zhang, W. Shi, Y. Li, and J. Cheng TCOD: exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents. External Links: Link, 2604.24005v1 Cited by: §1, §2, §5.1. Wang et al. (2022a) R. Wang, P. Jansen, M. Côté, and P. Ammanabrolu ScienceWorld: is your agent smarter than a 5th grader?. External Links: Link, 2203.07540 Cited by: §E.5. Wang et al. (2022b) X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. External Links: Link, 2203.11171 Cited by: §1. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. External Links: Link, 2201.11903 Cited by: §1. Welleck et al. (2019) S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston Neural text generation with unlikelihood training. External Links: Link, 1908.04319 Cited by: §2, §4.4. Xi et al. (2024) Z. Xi, Y. Ding, W. Chen, B. Hong, H. Guo, J. Wang, D. Yang, C. Liao, X. Guo, W. He, S. Gao, L. Chen, R. Zheng, Y. Zou, T. Gui, Q. Zhang, X. Qiu, X. Huang, Z. Wu, and Y. Jiang AgentGym: evolving large language model-based agents across diverse environments. External Links: Link, 2406.04151 Cited by: §E.5. Xin et al. (2026) H. Xin, A. Zhao, Y. Sun, J. Li, X. Shen, and H. Xiong Escaping the kl agreement trap in on-policy distillation. External Links: Link, 2606.09471 Cited by: §2. Xu et al. (2026) Y. Xu, H. Sang, Z. Zhou, R. He, Z. Wang, and A. Geramifard TIP: token importance in on-policy distillation. External Links: Link, 2604.14084 Cited by: §1, §2. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: Link, 2505.09388 Cited by: §E.6, §5.1. Yang et al. (2026a) W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin Learning beyond teacher: generalized on-policy distillation with reward extrapolation. External Links: Link, 2602.12125 Cited by: §E.6. Yang et al. (2026b) Y. Yang, T. Yu, S. Li, K. Zhao, X. Zhang, C. Bansal, H. Yao, T. W. Killian, and W. Zhang When eos tokens disagree: understanding length inflation in on-policy distillation. arXiv preprint arXiv:2609.20511. Cited by: §1. Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning HotpotQA: a dataset for diverse, explainable multi-hop question answering. External Links: Link, 1809.09600 Cited by: §E.5. Yao et al. (2022a) S. Yao, H. Chen, J. Yang, and K. Narasimhan WebShop: towards scalable real-world web interaction with grounded language agents. External Links: Link, 2207.01206 Cited by: §E.3, §5.1. Yao et al. (2023) S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan Tree of thoughts: deliberate problem solving with large language models. External Links: Link, 2305.10601 Cited by: §1. Yao et al. (2022b) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. External Links: Link, 2210.03629 Cited by: §1. Ye et al. (2026) T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei On-policy context distillation for language models. External Links: Link, 2602.12275 Cited by: §E.6. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang DAPO: an open-source llm reinforcement learning system at scale. External Links: Link, 2503.14476 Cited by: §E.1, §5.1. Yu et al. (2026a) Y. Yu, Z. Lu, Y. Liu, Y. Pan, A. Wang, Q. Chen, H. Yang, W. Zhang, W. Lu, Q. Chen, and Y. Shen RetireOPD: self-retiring on-policy distillation for agentic reinforcement learning. External Links: Link, 2609.20784 Cited by: §E.6. Yu et al. (2026b) Y. Yu, L. Yao, Y. Li, E. Wang, and L. Wu Rollback the world, keep the reflection: rollback-induced reflection for long-horizon llm agents. External Links: Link, 2609.18304 Cited by: §2. Yu et al. (2026c) Y. Yu, S. Jang, and H. Yu Difficulty-adaptive tree-structured policy optimization for expanding reasoning coverage in rlvr. External Links: Link, 2609.08650 Cited by: §1, §2. Yu et al. (2026d) Z. Yu, C. Yu, S. Xu, Y. Han, B. Jiang, X. Wang, and D. Zou Mismatch matters: on-policy distillation beyond token agreement. External Links: Link, 2608.09836 Cited by: §2. Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman STaR: bootstrapping reasoning with reasoning. External Links: Link, 2203.14465 Cited by: §2. Zhao et al. (2026) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. External Links: Link, 2601.18734 Cited by: §E.6. Zheng et al. (2026) K. Zheng, H. Li, Z. Cheng, and Y. Gao CLOOPD: closing the learner loop in on-policy distillation. External Links: Link, 2609.24141 Cited by: §E.6, §1. Zhong et al. (2026) Q. Zhong, M. Zheng, M. Song, X. Lin, J. Sun, H. Jiang, X. Wang, and J. Fang SOD: step-wise on-policy distillation for small language model agents. External Links: Link, 2605.07725 Cited by: §1, §2. Zhou et al. (2023a) A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y. Wang Language agent tree search unifies reasoning acting and planning in language models. External Links: Link, 2310.04406 Cited by: §1. Zhou et al. (2026) J. Zhou, Y. Zhou, H. Guo, X. Wang, S. Feng, S. Gao, J. Xu, Q. Guo, and X. Zhang Teacher should think ahead: adaptive continuations for reliable on-policy distillation. External Links: Link, 2609.22254 Cited by: §1, §2. Zhou et al. (2023b) S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig WebArena: a realistic web environment for building autonomous agents. External Links: Link, 2307.13854 Cited by: §E.5. Zhou et al. (2024) Y. Zhou, A. Zanette, J. Pan, S. Levine, and A. Kumar ArCHer: training language model agents via hierarchical multi-turn rl. External Links: Link, 2402.19446 Cited by: §E.6. Zong et al. (2026) Z. Zong, D. Chen, Y. Li, Q. Yi, B. Zhou, C. Li, B. Qian, P. Chen, and J. Jiang AT²po: agentic turn-based policy optimization via tree search. External Links: Link, 2601.04767 Cited by: §1, §2. Appendix A Theoretical Details Throughout, the rollout policy πθold _ _ old is fixed within each sampling group, s denotes a prefix sts_t, and Δlogp(s,a)=logπT(a∣s)−logπθ(a∣s) p(s,a)= _T(a s)- _θ(a s) (14) is the distillation signal of Equation 3 at token a. We assume πT(a∣s)>0 _T(a s)>0 whenever πθ(a∣s)>0 _θ(a s)>0, and πθ≪πθold _θ _ _ old at every supervised prefix. A.1 OPD gradient and token surrogate At a fixed prefix s, the reverse KL is KL(πθ(⋅∣s)∥πT(⋅∣s))=∑aπθ(a∣s)[−Δlogp(s,a)]KL( _θ(· s)\| _T(· s))= _a _θ(a s)[- p(s,a)]. Using ∇θπθ=πθ∇θlogπθ _θ _θ= _θ _θ _θ and ∇θ[−Δlogp(s,a)]=∇θlogπθ(a∣s) _θ[- p(s,a)]= _θ _θ(a s), ∇θKL(πθ(⋅∣s)∥πT(⋅∣s)) _θKL( _θ(· s)\| _T(· s)) =∑aπθ(a∣s)(1−Δlogp(s,a))∇θlogπθ(a∣s) = _a _θ(a s) (1- p(s,a) ) _θ _θ(a s) (15) =−∑aπθ(a∣s)Δlogp(s,a)∇θlogπθ(a∣s), =- _a _θ(a s)\, p(s,a)\, _θ _θ(a s), (16) since ∑aπθ∇θlogπθ=∇θ∑aπθ=0 _a _θ _θ _θ= _θ _a _θ=0. Under πθ≪πθold _θ _ _ old, rewriting the sum as an expectation over a∼πθold(⋅∣s)a _ _ old(· s) with weight r(θ)=πθ(a∣s)/πθold(a∣s)r(θ)= _θ(a s)/ _ _ old(a s) gives Equation 4. With AOPD=sg[Δlogp(s,a)]A OPD=sg[ p(s,a)], the unclipped surrogate −r(θ)AOPD-r(θ)A OPD has gradient −AOPDr(θ)∇θlogπθ(a∣s)-A OPDr(θ) _θ _θ(a s), whose expectation under πθold _ _ old equals the KL gradient when the detached signal is evaluated at the differentiation point. PPO and signal clipping then define the training surrogates in Equations 5 and 11. Summing over t∈(τ)t (τ) and averaging over τ∼πθoldτ _ _ old yields the gradient of Equation 2 with the prefix distribution held fixed; the update does not differentiate through the rollout distribution (Liu et al., 2026). Equation 2 normalizes within each trajectory, whereas the token-mean surrogate normalizes over the batch and thus weights trajectories by length; the two coincide for a single trajectory. A.2 Conditional resampling Let ℱ0F_0 contain the main trajectory and the selected fork prefixes. At a fork prefix sb=stbs_b=s_t_b, write a fresh continuation as τ≥tb=(a,τ>tb) _≥ t_b=(a, _>t_b), where a is its first token and τ>tb _>t_b includes subsequent tokens and environment transitions. For an integrable statistic fθf_θ fixed before drawing the continuations, πθold(τ≥tb∣sb) _ _ old( _≥ t_b s_b) =πθold(a∣sb)πθold(τ>tb∣sb,a), = _ _ old(a s_b)\, _ _ old( _>t_b s_b,a), (17) μθ(sb,a) _θ(s_b,a) =τ>tb∼πθold(⋅∣sb,a)fθ(a,τ>tb), =E_ _>t_b _ _ old(· s_b,a)f_θ(a, _>t_b), (18) Jθ(sb) J_θ(s_b) =∑aπθold(a∣sb)μθ(sb,a). = _a _ _ old(a s_b)\, _θ(s_b,a). (19) Restoring the fork state and drawing each continuation from πθold(⋅∣sb) _ _ old(· s_b) gives [J^RS(sb)∣ℱ0]=1k∑j=1k[fθ(τb,j)∣ℱ0]=Jθ(sb).E[ J_ RS(s_b) _0]= 1k _j=1^kE[f_θ( _b,j) _0]=J_θ(s_b). (20) This proves Proposition 1 using only the conditional marginal law of each fresh branch. For conditionally independent branches with finite variance, Var(J^RS(sb)∣ℱ0)=Var(fθ(τ≥tb)∣ℱ0)k.Var( J_ RS(s_b) _0)= Var(f_θ( _≥ t_b) _0)k. (21) Fork selection depends only on the main trajectory and is therefore contained in ℱ0F_0; the expectation above is taken over the fresh branch draws alone. A.3 Coverage and diminishing returns A token a is absent from k independent branch starts at sbs_b with probability (1−πθold(a∣sb))k(1- _ _ old(a s_b))^k. Therefore, for the number NkN_k of distinct first tokens among those branches, [Nk∣ℱ0] [N_k _0] =∑a∈[1−(1−πθold(a∣sb))k], = _a [1-(1- _ _ old(a s_b))^k ], (22) [Nk+1−Nk∣ℱ0] [N_k+1-N_k _0] =∑a∈πθold(a∣sb)(1−πθold(a∣sb))k. = _a _ _ old(a s_b)(1- _ _ old(a s_b))^k. (23) The marginal increase is nonnegative and decreases with k. For the realized main token a0a_0, at least one branch starts differently with probability 1−πθold(a0∣sb)k1- _ _ old(a_0 s_b)^k. Branches that share the first token can still diverge later, so we measure both first-token alternatives and later behavioral differences (Appendix C). A.4 Token aggregation and protocol repair Proposition 1 concerns the average of a fixed suffix statistic, whereas the training loss averages over valid tokens in the main trajectory and all branches. A suffix with L generated tokens contributes L loss terms, and shared prefix tokens enter once. Fork selection thus determines which prefixes are supervised, continuation length determines how much each contributes, and the mask Mi,tM_i,t sets the normalization. For PBR, a positive event contributes −logpθ(C∣s)- p_θ(C s) and a negative event −log(1−pθ(C∣s))- (1-p_θ(C s)); their gradients increase and decrease the event probability, respectively. Detection is held fixed during differentiation. With the cap Ψ(u)=min(u,cPBR) (u)= (u,c_ PBR), the auxiliary gradient is unchanged below the cap and zero above it; a batch without constraints contributes zero. A.5 Weighted forced-token comparator The forced-token controls in Table 8 use a natural token plus k alternatives. At a fixed prefix s, write pold(a)=πθold(a∣s)p_ old(a)= _ _ old(a s), sample A0∼poldA_0 p_ old, and let S(A0)=A0∪TopKa≠A0pold(a).S(A_0)=\A_0\ _a≠ A_0p_ old(a). (24) With deterministic tie-breaking and at least k+1k+1 tokens in the support, order tokens as a1,a2,…a_1,a_2,… by decreasing poldp_ old and write Qk+1=∑i=1k+1pold(ai)Q_k+1= _i=1^k+1p_ old(a_i). Their inclusion probabilities are ρ(ai)=1,i≤k,Qk+1,i=k+1,pold(ai),i>k+1.ρ(a_i)= cases1,&i≤ k,\\ Q_k+1,&i=k+1,\\ p_ old(a_i),&i>k+1. cases (25) The first k tokens are always included; token k+1k+1 enters when A0A_0 is among the first k+1k+1 tokens; all others enter through the natural draw. After forcing token a, the continuation τ>t(a)∼πθold(⋅∣s,a)τ^(a)_>t _ _ old(· s,a) has conditional mean μθ(s,a) _θ(s,a). Horvitz–Thompson weights (Horvitz and Thompson, 1952) then give J^HT(s) J_ HT(s) =∑a∈S(A0)pold(a)ρ(a)fθ(a,τ>t(a)), = _a∈ S(A_0) p_ old(a)ρ(a)f_θ(a,τ^(a)_>t), (26) [J^HT(s)∣s] [ J_ HT(s) s] =∑apold(a)ρ(a)ρ(a)μθ(s,a)=Jθ(s). = _a p_ old(a)ρ(a)ρ(a) _θ(s,a)=J_θ(s). (27) Every token with positive probability has positive inclusion probability. The scaled weights wscaled(a)=(k+1)pold(a)/ρ(a)w_ scaled(a)=(k+1)\,p_ old(a)/ρ(a) satisfy J^HT=(k+1)−1∑a∈Swscaled(a)fθ(a,τ>t(a)) J_ HT=(k+1)^-1 _a∈ Sw_ scaled(a)f_θ(a,τ^(a)_>t) and ∑a∈Swscaled(a)=k+1E _a∈ Sw_ scaled(a)=k+1. Table 8 uses scaled HT weights in a pooled token loss, with the main-slot weight averaged across its forks. RB instead self-normalizes the weights within each realized group; because its denominator is random, it is a different estimator. The HT identity above holds when the natural draw and the continuations follow the fixed-prefix sampling distribution. Appendix B Additional Results and Controlled Ablations This section reports results for other students, sampling rules, generation budgets, and protocol repair, using the metrics of Section 5. B.1 Student size and model family Table 4: Qwen3-4B to Qwen3-1.7B distillation. Teachers are task-specific GRPO checkpoints. Values are task SR or mathematics mean avg@16 (%); (+PBR) identifies repaired baselines. Method ALFWorld WebShop Search-QA Math Avg ID SR OOD SR SR SR Qwen3-4B GRPO teacher 46.43 41.04 27.34 47.20 61.60 Qwen3-1.7B student 7.86 6.72 3.91 26.40 11.08 Vanilla OPD (+PBR) 44.29 42.54 26.56 45.10 30.18 Multi-Rollout OPD (n=7n=7, +PBR) 45.00 40.30 26.56 45.40 30.05 Entropy-Weighted OPD (+PBR) 47.14 45.52 27.34 45.90 31.60 BranchOPD 51.43 48.51 29.69 46.80 34.86 Table 5: Qwen2.5-7B-RL to Qwen2.5-3B on ALFWorld. SR (%) uses 140 ID and 134 OOD tasks; trained baselines use PBR. Method ALFWorld ID SR OOD SR Qwen2.5-7B-RL teacher 64.29 55.22 Qwen2.5-3B zero-shot 7.14 2.99 Vanilla OPD (+PBR) 47.14 41.79 Multi-Rollout OPD (n=7n=7, +PBR) 47.86 40.30 Entropy-Weighted OPD (+PBR) 50.71 44.03 BranchOPD 55.71 50.75 With Qwen3-1.7B as the student, BranchOPD improves over Vanilla OPD (+PBR) by 6.56 points on ALFWorld and 4.68 points on mathematics, and exceeds the teacher on ALFWorld and WebShop without task rewards (Table 4). With Qwen2.5-3B, the ALFWorld gain is 8.76 points (Table 5). B.2 Selection and branching budgets Table 6: Selection and eligibility ablations with B=3,k=2B=3,k=2 and PBR. ALFWorld SR and mathematics mean avg@16 are percentages. α weights entropy in the mixed selector; – denotes an unevaluated setting. Setting ALFWorld Math Avg ID SR OOD SR Random sites 50.71 46.27 40.11 Student entropy only 55.71 52.99 43.24 Disagreement only 55.00 52.24 42.97 Entropy + disagreement (α=0.3α=0.3) 57.14 54.48 43.88 Entropy + disagreement (α=0.5α=0.5) 57.86 55.22 44.37 Entropy + disagreement (α=0.7α=0.7) 56.43 53.73 43.65 No mathematics-aware eligibility – – 42.03 Turn-initial 55.71 53.73 – Failed mains only 55.00 50.75 36.82 Table 7: Branch budgets with entropy + disagreement selection and PBR. Performance is in percent; slots equal 1+Bk1+Bk. Generation is relative to Vanilla OPD on ALFWorld; † marks interpolated or extrapolated costs. Setting ALFWorld Math Avg Slots Generation ID SR OOD SR B=3,k=1 54.29 50.75 42.18 4 3.33× † B=3,k=2 57.86 55.22 44.37 7 5.66× B=3,k=3 57.14 54.48 43.91 10 7.99× † B=1,k=2 54.29 51.49 42.55 3 2.55× B=2,k=2 57.14 54.48 43.94 5 4.11× † Entropy + disagreement is the best of the tested selection rules (Table 6). Disabling the mathematics-aware eligibility filter lowers mathematics Avg to 42.03, and branching only from failed main trajectories lowers it to 36.82, so branches from successful trajectories are useful as well. The two budgets play different roles (Table 7). Going from k=1k=1 to k=2k=2 helps, and k=3k=3 adds nothing. At k=2k=2, increasing B from one to three raises mathematics Avg from 42.55 to 44.37. The five-trajectory setting (B=2,k=2B=2,k=2) keeps most of the gain at a lower generation cost. B.3 Sampling and perturbation controls Table 8: Expansion rules with PBR. Performance is in percent. HT uses scaled inclusion weights; RB normalizes weights within each sampled group (Appendix A). Expansion Weights ALFWorld Math Avg WebShop Search-QA ID SR OOD SR SR SR forced-topk Uniform 52.86 48.51 40.90 31.25 48.76 forced-topk RB 57.14 51.49 41.77 31.25 49.22 forced-topk HT (scaled) 55.71 50.75 42.11 32.03 48.96 resample Uniform 57.86 55.22 44.37 33.59 49.05 Table 9: ALFWorld continuation diagnostics. The first five metrics use step 100; low-likelihood rates summarize 64-token windows. Rates are percentages and new actions is a count. Metric Forced + HT Resample Main-token probability 37.19% 36.67% Alternative mass 62.81% 63.33% Action disagreement 75.8% 74.2% New-action coverage (NSR) 58.6% 55.5% Mean new actions 2.59 2.44 Low-likelihood suffixes 19.2% 5.8% Among the expansion rules, resampling gives the highest ALFWorld, mathematics, and WebShop scores, and RB the highest Search-QA SR (Table 8). Both rules change actions at similar rates, but resampling produces far fewer low-likelihood suffixes, 5.8% versus 19.2% (Table 9). A suffix counts as low-likelihood when its mean student log probability over 64 tokens falls below the fifth percentile of on-policy windows of the same length. Useful coverage can therefore be obtained from continuations that the student is likely to generate. Table 10: Noise-token branching on ALFWorld with PBR. ID/OOD SR and their mean are percentages. Each branch starts with the forced token !. Setting ALFWorld ALF Avg ID SR OOD SR Vanilla OPD (+PBR) 49.29 49.25 49.27 Two-stage + noise token 50.71 50.00 50.36 Turn-initial + noise 50.00 50.00 50.00 Forcing a noise token (!) at the start of each branch yields small ALFWorld gains of 1.09 and 0.73 points over Vanilla OPD (+PBR), depending on where the forks are placed (Table 10). Perturbed continuations still add some supervision, but the much larger gains of informative branching show that fork selection matters. B.4 Generation and environment costs Table 11: Generation costs and performance with PBR. Scores are mathematics Avg or task SR (%); ALFWorld averages ID/OOD. Costs are relative to Vanilla OPD, and savings use the 7×7× independent-rollout reference. Task Vanilla OPD Multi-Rollout OPD (n=7n=7) BranchOPD Generation Saving vs. 7×7× Cost basis Mathematics 38.94 39.09 44.37 5.68× 18.9% Position estimate ALFWorld 49.27 48.88 56.54 5.66× 19.1% Measured tokens WebShop 30.47 31.25 33.59 4.81× 31.2% Measured tokens Search-QA 48.58 48.60 49.05 4.24× 39.4% Position estimate Table 12: Three-slot ALFWorld sampling with PBR. SR is the ID/OOD mean (%); generation and environment steps are relative to Vanilla OPD. Setting Slots Generation Env. steps ALF Avg Vanilla OPD (+PBR) 1 1.00× 1.00× 49.27 Resample (B=1,k=2B=1,k=2) 3 2.55× 3.02× 52.89 Forced, turn-initial + RB (B=1,k=2B=1,k=2) 3 2.49× 3.07× 54.48 Relative to seven independent rollouts, BranchOPD saves 19.1% and 31.2% of student generation on ALFWorld and WebShop while improving SR; estimates from fork positions give savings of 18.9% on mathematics and 39.4% on Search-QA (Table 11). With two added continuations, ALFWorld uses 2.55×2.55× the generation and 3.02×3.02× the environment steps of Vanilla OPD, because state replay adds environment cost on top of suffix generation (Table 12). B.5 Protocol repair and repetition Table 13: Repetition treatments for Vanilla OPD on mathematics. Values are mean avg@16 (%). Recipe Math Avg No PBR 25.82 Repetition penalty 28.70 n-gram / line penalty in advantage 30.40 Overlong-sample filtering 33.20 Early stopping during training 22.05 Full PBR 38.94 Without PBR, mathematics responses reach the length cap around step 30. Repetition penalties delay this but do not prevent persistent budget exhaustion (Figure 4). PBR reaches 38.94 mathematics Avg, compared with 33.20 for filtering and 28.70–30.40 for penalties; after step 50, its median response length is about 8,600 tokens and its median truncation rate is 0.28. A teacher whose own rollouts are clean can still reinforce repetition when queried on a repetitive student prefix; PBR targets these prefix-conditioned errors. On ALFWorld, teacher responses contain on average 1.64/1.50 excess leading </think> markers on ID/OOD. Vanilla OPD raises this to 1.94/1.91, and PBR reduces it to zero. A single response can contain several such markers. Figure 5: Trajectory-level protocol violation (PVP): the fraction of the 192 main and branch trajectories that contain a violation. All methods shown use PBR; BranchOPD-F denotes forced-token expansion (Table 8). Trajectory-level protocol violation drops to zero by step 25 and stays near zero (Figure 5). Together with the behavior coverage in Appendix C, this indicates that protocol repair does not suppress exploration. Appendix C Behavior Coverage and Recovery These diagnostics follow the chain from selecting uncertain decisions, to generating alternative behavior, to recovering successful outcomes. C.1 Probability available at selected decisions Figure 6: Probability mass available for alternative decisions. (a) “Mass kept by the main path” is the probability of the sampled main token, πθold(a0∣s) _ _ old(a_0 s). (b) “Mass discarded” and MKM_K denote the alternative-token mass 1−πθold(a0∣s)1- _ _ old(a_0 s), i.e., the mass left unrealized by the main draw. (c) All supervised positions compared with selected forks. ALFWorld and WebShop use training dumps; mathematics and Search-QA use reconstructed summaries. In the plot, q denotes πθold _ _ old and turn-open-RS denotes Turn-initial branching. At selected forks, the alternative-token mass 1−πθold(a0∣s)1- _ _ old(a_0 s) is about 0.63 on ALFWorld and 0.69 on WebShop (Figure 6). Ordinary supervised positions are far more concentrated: 80.4% and 59.1% of them have alternative mass below 0.05 (Figure 7). The selected forks are thus decisions with several plausible continuations. Figure 7: Empirical CDF of alternative mass over supervised positions (ALFWorld step 200, WebShop step 210), accumulated over 20 equal-width bins. BranchOPD-F denotes forced-token expansion (Table 8), and “uniform” denotes uniform weights on its branches. C.2 Alternative actions across training Table 14: Behavior coverage from 64 main trajectories and 128 branches per checkpoint. Action and observation disagreement and NSR are percentages; new actions is a count per branch. Task / step Action (%) Observation (%) NSR (%) New actions ALFWorld @100 74.2 74.2 55.5 2.44 ALFWorld @200 74.2 73.4 57.8 3.02 WebShop @100 47.7 38.3 37.5 0.68 WebShop @200 52.3 40.6 32.0 0.60 Action disagreement (EBR) is the fraction of branches whose action sequence differs from that of the main trajectory, and observation disagreement is the analogous fraction for observations. New-action coverage (NSR) is the fraction of branches that contain an action absent from the main trajectory, and mean new actions counts such actions per branch. At step 100, EBR is 74.2% on ALFWorld and 47.7% on WebShop, and NSR is 55.5% and 37.5% (Table 14). Figure 8: Behavior coverage of resampled branches over training. “New state” in the plot denotes new-action coverage (NSR). High disagreement and NSR mean that branches reach behavior not present in the main trajectory. This coverage persists throughout training (Figure 8): on WebShop it stabilizes after an initial drop, and on ALFWorld it stays high. C.3 Recovery opportunities We define the rescue rate R↑=Pr(some branch succeeds∣main fails)R_ = (some branch succeeds fails) and the harm rate R↓=Pr(all branches fail∣main succeeds)R_ = (all branches fail succeeds). In the latter case, the successful main trajectory is still used for training. Table 15: Recovery outcomes (%). R↑R_ : at least one successful branch given a failed main; R↓R_ : all branches fail given a successful main. The main trajectory is retained. Task / setting Main success (%) R↑R_ (%) R↓R_ (%) Math B=3,k=2 @200 41.4 42.7 0.0 ALFWorld resample @100 48.4 21.2 19.4 ALFWorld resample @200 45.3 14.3 6.9 WebShop resample @200 37.5 5.0 4.2 Search-QA B=3, pooled dumps 51.9 15.8 15.2 Figure 9: Rescue and harm rates during training, each conditioned on the outcome of the main trajectory. “Mean net” is R↑−R↓R_ -R_ averaged over training steps. “Spoil” means that all branches fail although the main trajectory succeeds; the main trajectory itself stays in the training batch. The rescue rate ranges from 5.0% on WebShop to 42.7% on mathematics (Table 15), and successful alternatives keep appearing throughout training (Figure 9). Such recovered continuations provide corrective supervision and may contribute to the task gains. C.4 Where behavior begins to diverge Figure 10: Fork location and distance to the first changed action. (a) Fork position relative to the full trajectory and (b) relative to the selected turn, where 0 and 1 mark the start and end. (c) Number of generated student tokens from the fork to the first changed action. Fork locations depend on the task and often fall inside a turn (Figure 10). The first changed action appears roughly 12–18 generated tokens after the fork on ALFWorld, 1.5–2 on WebShop, and 30 on Search-QA, so a token-level fork can change later actions while reusing the preceding context. Appendix D Training and Evaluation Settings D.1 Shared training configuration The default configuration uses B=3B=3 forks and k=2k=2 continuations per fork. All continuations are kept, including unsuccessful ones, and resampled branches receive no importance weights. Prefix deduplication masks the tokens strictly before the fork and keeps the fork token; environment observations are excluded from the distillation loss. Fork scores mix entropy and teacher–student disagreement with α=0.5α=0.5, using a 128-token disagreement window and the top-20 student log probabilities. In agent tasks, each turn is scored by the mean of its top 20% eligible-token scores; high-scoring turns are selected with a minimum gap of one turn, and the fork is drawn uniformly from the three highest-scoring eligible tokens of each selected turn. Turn-opening eligibility covers the first 16 tokens of a turn, with no initial-token skip, and is not restricted to turns that follow a tool call. The minimum entropy for eligibility is 0 for agent tasks and 0.5 for mathematics. Training uses the sampled-token (k1k_1) estimator of the reverse KL, i.e., the detached advantage sg[Δlogpt]sg[ p_t], with PPO clipping ratio ϵ=0.2ε=0.2 and token-mean reduction. The teacher is queried for its top-64 log probabilities, and the loss uses the sampled-token log ratio. Recorded loss values are clamped to [−10,10][-10,10], and log probabilities are floored at −10-10. Task rewards, KL rewards, and a separate actor KL penalty are disabled. Rollouts and branches use temperature 1, top-p=1p=1, and no top-k truncation. The learning rate is 10−610^-6. We train with FSDP, dynamic batching, offloading, gradient checkpointing, and asynchronous vLLM rollout that records log probabilities. For agent tasks, PBR uses an unlikelihood weight λ=0.01λ=0.01 and a cap of cPBR=0.02c_ PBR=0.02 on the weighted loss, and λ=0λ=0 for the −-PBR variants; the forbidden boundary token for Qwen models is 151668. Table 16: Training settings. Setting ALFWorld WebShop Search-QA Mathematics Training examples 6,400 6,782 NR 17,398 Train/PPO batch size 64/64 64/64 128/128 128/128 Prompt limit 2,048 4,096 2,048 2,048 Response limit 24,576 16,384 8,192 16,384 Per-turn tokens 512 512 512 N/A Environment step limit 50 15 4 N/A Environment pool 16 8 NR N/A Training duration 2 epochs 4 epochs 600 steps 2 epochs Rollout/teacher TP 2/8 2/8 2/4 1/2 Rollout/teacher GPUs 8/8 8/8 4/4 8/8 Memory fractions 0.45/0.75 0.45/0.75 0.40/0.70 0.45/0.65 PPO token limit 32,768 24,576 16,384 49,152 Table 16 lists the per-task settings. Prompt, response, per-turn, and PPO limits are in tokens; NR marks an unreported setting and N/A a setting that does not apply. Mathematics training runs for two epochs over 17,398 examples, i.e., ⌊17,398/128⌋=135 17,398/128 =135 steps per epoch and 270 steps in total, with checkpointing and evaluation every 50 steps. For mathematics, rollout and teacher inference each run on a separate eight-GPU machine, with tensor-parallel degrees 1 and 2 and GPU memory fractions 0.45 and 0.65. D.2 Additional mathematics baselines Offline knowledge distillation (KD). The teacher samples one response per training problem to form a fixed dataset. The student is trained on these sequences with an equally weighted (0.5 each) sum of token cross-entropy and the forward KL from teacher to student, computed over the teacher’s top-16 tokens. Outcome-based GRPO. The student generates eight responses per problem and receives a binary answer-correctness reward; group-relative advantages computed from these rewards drive the GRPO updates. No teacher is used. KD and GRPO are trained without PBR in Table 1. Entropy-aware OPD (EOPD). EOPD (Jin et al., 2026) adds a forward-KL term to reverse-KL OPD at tokens where the teacher entropy exceeds a threshold h0h_0: ℓEOPD(s)=ℓOPD(s)+β 1H(πT(⋅∣s))>h0KL(π~T∥πθ), _EOPD(s)= _OPD(s)+β\,1\H( _T(· s))>h_0\KL( π_T\| _θ), (28) where ℓOPD _OPD is the reverse-KL surrogate, π~T π_T is the teacher distribution renormalized over its top-k tokens, and β weights the added term. EOPD changes the local target but samples no additional suffixes. It also differs from Entropy-Weighted OPD, which only reweights the token losses of a single trajectory. Sparse probing and outcome calibration (SPOT). SPOT (Qu et al., 2026) selects positions by normalized teacher entropy, teacher top-k probability mass, and teacher–student mismatch. At each selected position it forces teacher-proposed candidate tokens, lets the student continue, and scores the continuations with an outcome verifier. For positions with a positively valued candidate, it constructs the target pcal(v∣s)=π~T(v∣s)exp(γV^(s,v))∑u∈(s)π~T(u∣s)exp(γV^(s,u)),p_cal(v s)= π_T(v s) (γ V(s,v)) _u (s) π_T(u s) (γ V(s,u)), (29) where (s)K(s) is the teacher candidate set, V V the verifier-based continuation value, and γ the calibration strength. A cross-entropy loss toward this target is added to OPD. BranchOPD instead resamples from the student and applies dense teacher supervision along the new suffixes. All four additional baselines are trained without PBR in Table 1. D.3 EOS alignment for the base student and instruction teacher The official Qwen3-4B-Base configuration uses EOS token ID 151643 (<|endoftext|>), whereas Qwen3-8B uses ID 151645 (<|im_end|>) as its primary EOS and accepts both 151645 and 151643 in its generation configuration. In our runs, the student adopts the teacher’s EOS configuration and stops on either token (eos_token_id = [151645, 151643]). This overrides the base student’s single-token default, so teacher and student share the same end-of-response convention for both teacher probabilities and terminal supervision. D.4 Evaluation protocol For mathematics, we sample 16 answers per problem with temperature 0.6, top-p=0.95p=0.95, and top-k=20k=20, with prompt and response limits of 2,048 and 16,384 tokens. For agent tasks, we sample one trajectory per task at temperature 0.4. The model’s optional thinking mode is disabled in the generation configuration; protocol tags written in the prompt are handled separately. WebShop uses the small environment with human interaction disabled and seed 42, and holds out 128 of its 6,910 tasks. The WebShop and Search-QA teachers are taken after 125 and 500 GRPO steps, respectively. Search returns three results per call with a 30-second timeout. ALFWorld SR is computed separately on the 140 ID and 134 OOD tasks, and the ALFWorld aggregate is their unweighted mean. WebShop SR is the fraction of the 128 tasks solved, and Search-QA SR is exact-match accuracy on the same mixed validation set for every model. The agent Avg. SR is the unweighted mean of the ALFWorld aggregate, WebShop SR, and Search-QA SR; mathematics Avg is the unweighted mean of the three benchmark accuracies. Appendix E Benchmarks and Additional Background E.1 Mathematical reasoning MATH contains competition problems in algebra, geometry, number theory, and related subjects (Hendrycks et al., 2021). We evaluate on MATH-500, the 500-problem test subset used in Let’s Verify Step by Step (Lightman et al., 2023), and on AIME 2024 and 2025. Each AIME year has 30 problems from two 15-problem examinations, with integer answers from 000 to 999. Each problem requires a free-form solution and a final answer. avg@16 averages correctness over 16 independent solutions per problem and then over problems. Training problems come from DAPO (Yu et al., 2025). E.2 ALFWorld ALFWorld (Shridhar et al., 2020) is a text-based household environment that grounds language actions in embodied tasks. The agent reads textual observations and issues commands to navigate, inspect, pick up, place, and transform objects, and succeeds if it completes the goal within the step limit. We report success on 140 in-distribution (ID) and 134 out-of-distribution (OOD) tasks, and their unweighted mean. During branch training, the selected interaction state is restored, or the preceding actions are replayed, before generation continues from a token-level fork; observations serve as context and are not supervised. E.3 WebShop WebShop (Yao et al., 2022a) is a simulated online-shopping benchmark. Given a purchase instruction, the agent searches a catalog, browses product pages, selects options, and buys an item. We report success rate (SR) on 128 held-out tasks. E.4 Search-QA Retrieval-augmented question answering answers information-seeking questions from retrieved evidence (Kwiatkowski et al., 2019). In our Search-QA setting, a trajectory interleaves search queries, retrieved results, and a final answer. SR is the fraction of questions whose normalized final answer exactly matches an accepted answer. Each search returns up to three results, and at most four interaction steps are allowed. The teacher, the untrained student, and all trained models are evaluated on the same mixed validation set. E.5 Benchmarks discussed as background GSM8K (Cobbe et al., 2021) contains grade-school math word problems, and Minerva (Lewkowycz et al., 2022) studies quantitative reasoning with language models. ScienceWorld (Wang et al., 2022a) tests science-oriented agent reasoning, WebArena (Zhou et al., 2023b) evaluates agents in reproducible web environments, and AgentGym (Xi et al., 2024) covers a range of agent tasks. HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), and MuSiQue (Trivedi et al., 2021) are question-answering benchmarks, some requiring multi-hop evidence. E.6 Additional learning and systems context Context distillation, self-distillation, and privileged supervision are other ways to transfer information into model parameters (Ye et al., 2026; Zhao et al., 2026; Penaloza et al., 2026). Recent work also studies reward extrapolation, skill transfer, hybrid supervision, and teacher retirement (Yang et al., 2026a; Wang et al., 2026a; Lu et al., 2026; Yu et al., 2026a). PPO and group-relative optimization are the underlying optimizers (Schulman et al., 2017; Shao et al., 2024; DeepSeek-AI et al., 2025), and agent training systems such as GiGPO, ArCHer, Agent Lightning, WebRL, and Search-R1 optimize task rewards (Feng et al., 2025; Zhou et al., 2024; Luo et al., 2025; Jin et al., 2025; Qi et al., 2024), unlike the protocol-event loss used here. Our experiments use the Qwen3 and Qwen2.5 families, and Llama 3 is a related open model family (Yang et al., 2025; Grattafiori et al., 2024; Qwen et al., 2024). HybridFlow (Sheng et al., 2024), FlashAttention (Dao et al., 2022), and PagedAttention (Kwon et al., 2023) improve execution and memory efficiency, complementing our reduction in generated student tokens. Continual on-policy distillation studies supervision and training distributions that change over time (Zheng et al., 2026); our analysis instead fixes the rollout policy within each sampling group. Appendix F Training Algorithm Algorithm 1 Branching on-policy distillation 1: Student πθ _θ, teacher πT _T, task source, fork budget B, continuation budget k 2: for each training iteration do 3: Freeze the rollout policy πθold←πθ _ _ old← _θ for this sampling group 4: Sample a task and generate one main trajectory under πθold _ _ old 5: Compute eligible-token entropy and teacher–student disagreement scores 6: if mathematical reasoning then 7: Select up to B token-level fork positions 8: else 9: Select turns, then select eligible fork tokens within those turns 10: end if 11: Initialize training tokens with the main trajectory’s generated tokens 12: for each selected fork prefix sbs_b do 13: for j=1,…,kj=1,…,k do 14: Restore the fork state or replay the prefix actions for this branch, if needed 15: Independently sample a continuation from πθold(⋅∣sb) _ _ old(· s_b) 16: Keep its generated tokens from the fork onward, whatever its outcome 17: end for 18: end for 19: Query teacher log probabilities for retained sampled tokens 20: Exclude observations and duplicate branch-prefix tokens from the loss 21: Form stopped-gradient sampled-token advantages and the clipped token-mean surrogate 22: Construct applicable protocol-boundary events and add the PBR loss 23: Update θ using the combined objective 24: end for If no eligible fork is found, the iteration trains on the main trajectory alone. All sampled branches contribute regardless of task success. At evaluation, agents generate one trajectory per task, and mathematics uses 16 independent responses per problem. Appendix G Prompt Templates The figures below give the prompts used in training and evaluation. Mathematics uses a single-turn template with a boxed final answer. ALFWorld and WebShop separate the initial task instruction from later observations and require structured action outputs. Search-QA additionally specifies the format of retrieval results and the feedback for invalid actions. These templates define the output boundaries that PBR checks. Figure 11: Mathematical reasoning prompt templates. Figure 12: ALFWorld prompt templates. Figure 13: WebShop prompt templates. Figure 14: Search-QA prompt templates.