Paper deep dive

Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu, Feng Xiao, Lu Liu, Yanlong Du, Zifeng Cheng, Ziwei Jiang, Qing Gu

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 42

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/23/2026, 1:51:09 AM

Summary

The paper introduces Step-Level On-Policy Distillation (SOPD), a method that combines the long-horizon correction of Supervised Fine-Tuning (SFT) with the on-policy advantage of On-Policy Distillation (OPD). SOPD provides step-level supervision over complete student-generated trajectories, addressing the fragmented correction issue of standard token-level OPD. The method interpolates between SFT and OPD based on step length and demonstrates superior performance in both reasoning and agent tasks, such as ALFWorld, where it significantly improves success rates and reduces interaction rounds compared to baseline methods.

Entities (7)

SOPD · method · 98%OPD · method · 95%SFT · method · 95%ALFWorld · datasettask · 92%Qwen2.5-3B-Instruct · model · 90%Qwen2.5-7B · model · 90%TCOD · method · 85%

Relation Signals (6)

SOPD → combines → SFT

confidence 95% · SOPD, which combines the long-horizon correction of supervised fine-tuning (SFT) with the on-policy advantage of OPD

SOPD → combines → OPD

confidence 95% · SOPD, which combines the long-horizon correction of supervised fine-tuning (SFT) with the on-policy advantage of OPD

SOPD → improves → ALFWorld

confidence 92% · on ALFWorld, SOPD improves the average success rate by 13.4 points over Vanilla OPD.

SOPD → outperforms → SFT

confidence 90% · SOPD substantially outperforms conventional SFT and OPD.

SOPD → outperforms → OPD

confidence 90% · SOPD substantially outperforms conventional SFT and OPD.

Qwen2.5-7B → servesasteacherfor → Qwen2.5-3B-Instruct

confidence 90% · we use a domain-RL-trained Qwen2.5-7B teacher to distill a Qwen2.5-3B-Instruct student.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conventional off-policy distillation with substantially less data. However, standard token-level OPD can provide only fragmented corrections along an erroneous student trajectory and cannot unfold a complete and correct repair path. Motivated by this limitation, we propose \emph{Step-Level On-Policy Distillation} (SOPD), which combines the long-horizon correction of supervised fine-tuning (SFT) with the on-policy advantage of OPD to provide step-level supervision over complete student-generated trajectories. We show that, at different limits of step length, SOPD reduces to SFT or approximates OPD. Compared with SFT, the teacher responses in SOPD are conditioned on student trajectories and therefore align more closely with student-visited states; compared with OPD, SOPD provides longer-horizon corrections rather than fragmented token-level guidance. Across both reasoning and agent tasks, SOPD substantially outperforms conventional SFT and OPD. For example, on ALFWorld, SOPD improves the average success rate by 13.4 points over Vanilla OPD. We hope this work offers a new perspective for future research on distillation methods.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

38,660 characters extracted from source content.

Expand or collapse full text

Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling Affiliation: State Key Laboratory for Novel Software Technology, Nanjing University, China Affiliation: XingYun Lab, HUJING Digital Media & Entertainment Group Affiliation: XingYun Lab, HUJING Digital Media & Entertainment Group Affiliation: XingYun Lab, HUJING Digital Media & Entertainment Group Affiliation: XingYun Lab, HUJING Digital Media & Entertainment Group Affiliation: University of Chinese Academy of Sciences, Beijing, China Jiahang Xie Affiliation: XingYun Lab, HUJING Digital Media & Entertainment Group Affiliation: School of Data Science, Fudan University, Shanghai, China Zhiyong Zheng Affiliation: XingYun Lab, HUJING Digital Media & Entertainment Group Yujia Wang Affiliation: XingYun Lab, HUJING Digital Media & Entertainment Group Hao Liu Affiliation: XingYun Lab, HUJING Digital Media & Entertainment Group Feng Xiao Affiliation: XingYun Lab, HUJING Digital Media & Entertainment Group Lu Liu Affiliation: XingYun Lab, HUJING Digital Media & Entertainment Group Yanlong Du Affiliation: XingYun Lab, HUJING Digital Media & Entertainment Group Zifeng Cheng Ziwei Jiang Qing Gu Affiliation: State Key Laboratory for Novel Software Technology, Nanjing University, China Abstract On-policy distillation (OPD) aligns a student model with a teacher’s logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conventional off-policy distillation with substantially less data. However, standard token-level OPD can provide only fragmented corrections along an erroneous student trajectory and cannot unfold a complete and correct repair path. Motivated by this limitation, we propose Step-Level On-Policy Distillation (SOPD), which combines the long-horizon correction of supervised fine-tuning (SFT) with the on-policy advantage of OPD to provide step-level supervision over complete student-generated trajectories. We show that, at different limits of step length, SOPD reduces to SFT or approximates OPD. Compared with SFT, the teacher responses in SOPD are conditioned on student trajectories and therefore align more closely with student-visited states; compared with OPD, SOPD provides longer-horizon corrections rather than fragmented token-level guidance. Across both reasoning and agent tasks, SOPD substantially outperforms conventional SFT and OPD. For example, on ALFWorld, SOPD improves the average success rate by 13.4 points over Vanilla OPD. We hope this work offers a new perspective for future research on distillation methods. 22footnotetext: Corresponding authors. 1 Introduction Recently, on-policy distillation (OPD) (1; 21; 15) has emerged as an effective post-training paradigm for improving the capabilities of large language models (LLMs). Unlike earlier off-policy distillation methods (19; 7), which train the student on teacher-generated trajectories, OPD allows the student to learn from teacher supervision—namely, predicted logits—on student-generated tokens. Standard OPD queries a teacher distribution at every token generated by the student. Although this provides relatively dense supervision, it also fragments the teacher’s correction signal: in a single OPD rollout, the student receives only one-token guidance at each position rather than a continuous and correct segment-level correction (10). Recent work has revisited the potential of DAgger-style methods for training LLM agents, seeking to preserve SFT’s long-horizon supervision while generating more training trajectories from student states (12). DAgger, however, inherently inserts teacher responses into the student-generated trajectory, substantially changing the trajectory the student would otherwise have produced. We combine the on-policy property of OPD with the long-horizon supervision of SFT and propose Step-Level On-Policy Distillation (SOPD). SOPD provides step-level supervision over a complete student-generated trajectory. After the student completes the full trajectory, SOPD asks the teacher to generate one step from the student prefix at every student step. Our method reduces to SFT when the number of steps approaches one and approximates forward-KL OPD when the step length approaches one token. We show that SOPD outperforms both SFT and OPD across different tasks. For agent tasks, we simply define a step as one actual environment-interaction turn. SOPD substantially improves over both SFT and OPD while reducing the number of interaction rounds. For mathematical reasoning, we define steps using natural reasoning boundaries, and SOPD improves over OPD by an average of 10 percentage points. These results demonstrate the general effectiveness of SOPD for both reasoning and agent tasks. Moreover, standard OPD requires access to teacher logits, whereas SOPD requires only teacher-generated responses and can therefore be applied more directly to black-box distillation. The total amount of teacher generation per sample is close to the length of one complete response, making the practical cost of black-box calls comparable to standard SFT. In agent tasks, each teacher call generates only one step, so distillation avoids the additional waiting time caused by teacher–environment interaction and tool execution. Our contributions are: • We combine the respective advantages of OPD and SFT and propose Step-Level On-Policy Distillation (SOPD). Our method is also better suited than standard OPD to black-box distillation. • We report results on both agent and reasoning tasks, demonstrating the advantages of SOPD over OPD and SFT. • In two limiting cases, SOPD reduces to standard SFT or approximate forward-KL OPD. This connection between SFT and OPD offers a new perspective on both distillation paradigms. 2 Related Work Knowledge distillation. Knowledge distillation transfers a teacher distribution into a smaller student (9). Sequence-level distillation trains on teacher-generated sequences and reduces the train–test mismatch created by token-level targets (11). MiniLLM instead minimizes reverse KL and samples from the student during optimization (6). These methods establish sequence generation and student sampling as distinct design choices. SOPD combines student-induced prefixes with teacher-generated local targets. On-policy language-model distillation. Generalized Knowledge Distillation trains the student on its own generations and queries teacher logits at those states (1). Recent work has expanded OPD in several directions. On-policy self-distillation (OPSD) lets the same model serve as student and teacher under ordinary and privileged contexts, achieving stronger reasoning performance and higher training efficiency than off-policy distillation without requiring a separate teacher model (25). Extrapolative OPD (ExOPD) uses reward extrapolation to relax direct imitation of the teacher, allowing a student that fuses multiple experts to surpass the teachers’ performance boundary (22). SOD, StepOPSD, and SAF-OPD combine step-level distillation signals with GRPO or RLVR to alleviate sparse trajectory rewards and local credit-assignment difficulties (26; 24; 5). GAD realizes black-box on-policy distillation through a generator–discriminator game and outperforms sequence-level distillation while accessing only teacher-generated text (23). Together, these advances show that OPD has expanded from standard teacher–student logit alignment to self-distillation, surpassing the teacher, integration with RL, and black-box distillation. At the same time, recent work has identified an inherent limitation of standard OPD. Trajectory-Refined Distillation (TRD) observes that supervision from a single rollout does not contain a complete counterfactual correction trajectory: even with a perfect teacher, token-wise OPD can provide only fragmented corrections along an erroneous student trajectory rather than unfold a complete, correct repair path (10). Later targets remain conditioned on the original erroneous student prefixes instead of prefixes that incorporate earlier teacher corrections. TRD constructs a repair path by rewriting the complete student trajectory; SOPD instead preserves the complete student trajectory while allowing the teacher to generate one coherent local correction within each natural step. Distillation and guidance for multi-turn agents. Action errors compound in interactive agents because every action changes the next observation. ALFWorld instantiates this dependence through textual actions in embodied household tasks (18). Existing approaches differ in who generates the training states and whether teacher outputs are executed in the environment. On-policy Expert Correction (OEC) begins with a student rollout, switches control to an expert partway through the trajectory, and applies SFT to reward-filtered expert suffixes (12). Agent-RLVR lets the student attempt a task, augments failed attempts with plans, error feedback, or interaction information from an external model, and then applies RL with verifiable environment rewards; the guidance is used only during training (4). These methods help the learner enter successful regions through expert suffixes or guidance-enhanced reward optimization, rather than producing an independent target at every fixed student state. TCOD uses a temporal curriculum to control which interval of a multi-turn trajectory is assigned to the student: forward-to-backward expands the student’s prefix of control, whereas backward-to-forward begins from a teacher trajectory prefix and moves student control toward the beginning (20). Guided-OPD chooses a teacher or student action at each environment turn according to a curriculum and anneals the probability of teacher intervention; selected teacher actions are executed, so training visits states induced by a teacher–student mixture (13). ReOPD avoids repeated environment interaction during training by replaying prefixes from pre-collected teacher trajectories, letting the student act at selected steps, and requesting dense teacher supervision (14). These methods largely depend on teacher intervention to adjust the generated trajectory, which weakens the on-policy nature of training. SOPD instead provides guidance over a complete student trajectory. At every student-generated step, the teacher produces one independent fragment conditioned on the corresponding student prefix, yielding supervision that remains more faithfully on-policy. Because each teacher call generates only one step, teacher generation does not wait for environment interaction or returned tool results; after the student trajectory is complete, the step-level teacher queries can also be issued in parallel. 3 Preliminaries: SFT and OPD We first establish notation for the two training paradigms that delimit SOPD. Let x∼x be a training prompt, πθ _θ the student to be optimized, and πT _T a fixed teacher. Supervised fine-tuning (SFT). SFT applies teacher forcing to a complete teacher or expert sequence yT=(y1T,…,yLT)y^T=(y^T_1,…,y^T_L): ℒSFT=(x,yT)∼T[−1L∑t=1Llogπθ(ytT∣x,y<tT)].L_SFT=E_(x,y^T) _T [- 1L _t=1^L _θ(y^T_t x,y^T_<t) ]. (1) Supervision within a target sequence unfolds along the teacher’s own preceding tokens, providing a coherent long-range target. Its contexts, however, come from teacher data rather than the current student, creating an off-policy state-distribution mismatch. On-policy distillation (OPD). OPD first samples yS∼πθ(⋅∣x)y^S _θ(· x) and then queries the teacher distribution at every student prefix htS=(x,y<tS)h_t^S=(x,y^S_<t). A common reverse-KL objective is ℒOPD=x,yS∼πθ[1|yS|∑tKL(πθ(⋅∣htS)∥πT(⋅∣htS))].L_OPD=E_x,\,y^S _θ [ 1|y^S| _tKL\! ( _θ(· h_t^S)\,\|\, _T(· h_t^S) ) ]. (2) This objective aligns training states with those encountered by the student at inference and supplies dense token-level feedback, but its standard implementation requires teacher logits. Moreover, after the student has entered an erroneous trajectory, the teacher distribution at the next position is still conditioned on an even longer erroneous student prefix. The resulting position-wise targets need not form a correction path that can be executed coherently (10). 4 Step-Level On-Policy Distillation SOPD preserves the student state occupancy of OPD while retaining coherent multi-token supervision within each local target. It first partitions a complete student trajectory into natural steps and then asks the teacher to generate one target from the beginning of every step. The student trajectory defines the states between steps; a teacher target unfolds autoregressively only within its own step. Figure 1 contrasts OEC, token-wise OPD, and SOPD and illustrates the packed attention mask. Figure 1: Overview of Step-Level On-Policy Distillation (SOPD). The left and center panels compare On-Policy Distillation (OPD), On-Policy Expert Correction (OEC), and SOPD. The right panel illustrates the packed attention mask used for efficient training. We pack the input prompt, the complete student trajectory, and its multiple step-level teacher targets into a single sequence, enabling these step-level supervision instances to be processed in parallel within a single forward pass to ensure training efficiency. Each teacher target can attend only to the input prompt, its corresponding student prefix, and its own preceding causal teacher tokens, while future student steps and other teacher targets remain masked. 4.1 Student Rollouts Fix the Visited States Let x denote a prompt or initial task state. A rollout student πθr _ _r generates a complete trajectory τS=(s1,s2,…,sK),sk∼πθr(⋅∣hk), _S=(s_1,s_2,…,s_K), s_k _ _r(· h_k), (3) where sks_k is one natural step and hkh_k is the student-visited prefix before that step. For static reasoning, hk=(x,s<k)h_k=(x,s_<k). In an interactive environment, hkh_k also contains the observations returned after the environment executes the student actions in s<ks_<k. The partition follows the structure of each domain. In mathematics, a paragraph boundary or an explicit heading such as “Step” or “Solution” closes a reasoning step. A 1,024-token safety cap closes a span only when the model emits no natural boundary. In ALFWorld, one assistant response and its parsed action form a step. The environment executes that student action and returns the observation that defines the next prefix. SOPD records every hkh_k before teacher generation. It therefore preserves the state distribution induced by the rollout student: hk∼dkπθr.h_k d _ _r_k. (4) Because the rollout has already fixed the complete trajectory, teacher outputs cannot alter dkπθrd _ _r_k. Thus, every step begins at a state actually visited by the student rather than at one produced by executing a teacher target. 4.2 Teacher Targets Do Not Alter the Trajectory Let qT(⋅∣hk)q_T(· h_k) denote the teacher distribution over the next natural step from student prefix hkh_k. For every recorded prefix, the teacher generates one target step: s~k∼qT(⋅∣hk). s_k q_T(· h_k). (5) Each query contains the original prompt and the preceding student steps, but no earlier teacher target. This aligns every target with a state the student actually visited and prevents teacher-written text from defining the context of a later target. The teacher supplies generated text only; SOPD requires neither teacher logits nor gradients and therefore supports black-box teachers. For mathematics, the teacher stops at the same family of natural boundaries used by the partitioner. For agents, all recorded turn states are submitted after the episode. Because no target appears in another target’s context, these teacher queries are conditionally independent, can be generated in parallel, and require neither tool execution nor new environment observations. 4.3 Step Balancing Prevents Long Targets from Dominating SOPD trains the student to generate each teacher target from its corresponding student prefix. Let BS_B contain every teacher step target collected from rollout batch B. We average token cross-entropy within each target and then average over the pooled steps: ℒSOPD(B)=1|B|∑(hk,s~k)∈B1|s~k|∑j=1|s~k|−logπθ(s~k,j∣hk,s~k,<j).L_SOPD(B)= 1|S_B| _(h_k, s_k) _B 1| s_k| _j=1^| s_k|- _θ ( s_k,j h_k, s_k,<j ). (6) The inner average prevents a long teacher step from dominating shorter decisions, and the outer average assigns equal weight to every collected step. A trajectory with more retained steps consequently contributes more step terms. Under stochastic teacher decoding, one target is a Monte Carlo sample of the length-normalized risk s~∼qT(⋅∣h)[−1|s~|logπθ(s~∣h)].E_ s q_T(· h) [- 1| s| _θ( s h) ]. (7) Under greedy decoding, qT=δgT(hk)q_T= _g_T(h_k), and the step term equals |gT(hk)|−1KL(δgT(hk)∥πθ)|g_T(h_k)|^-1KL( _g_T(h_k)\| _θ). This length normalization distinguishes Equation 6 from unnormalized sequence-level forward KL. The packed implementation shares repeated prefixes without changing Equation 6. It stores the prompt, the student steps needed by every prefix, and all teacher targets in one packed example. Its attention mask lets target s~k s_k attend to x, s<ks_<k, and its own causal target prefix, while blocking s≥ks_≥ k and every other teacher target. Loss labels cover teacher tokens only. 4.4 SOPD Connects Two Granularity Limits The step granularity locates SOPD between sequence SFT and token-level OPD. One-step limit: sequence SFT. If an entire response is the sole step, then K=1K=1, h1=xh_1=x, and the teacher generates a complete target y~∼qT(⋅∣x) y q_T(· x). Equation 6 becomes ℒK=1=−1|y~|∑jlogπθ(y~j∣x,y~<j),L_K=1=- 1| y| _j _θ( y_j x, y_<j), (8) which is the sequence teacher-forcing loss in Equation 1. Caching the teacher target yields ordinary teacher-data SFT; online generation changes how the data are acquired, not the optimization objective. One-token limit: a sampled forward-KL OPD objective. If every step contains exactly one token, the step at position t begins at student prefix htSh_t^S, and the teacher samples y~t∼qT(⋅∣htS) y_t q_T(· h_t^S). Taking expectation over this sample gives y~t∼qT[−log⁡πθ(y~t∣htS)] _ y_t q_T[- _θ( y_t h_t^S)] =H(qT(⋅∣htS),πθ(⋅∣htS)) =H\! (q_T(· h_t^S), _θ(· h_t^S) ) =H(qT(⋅∣htS))+KL(qT(⋅∣htS)∥πθ(⋅∣htS)). =H\! (q_T(· h_t^S) )+KL\! (q_T(· h_t^S)\| _θ(· h_t^S) ). (9) The teacher entropy is constant with respect to θ, so minimizing expected cross-entropy is equivalent to minimizing forward KL on student-visited prefixes. A single sampled teacher token is a Monte Carlo estimate of that objective. A greedy teacher instead yields hard imitation of the teacher’s argmax action, not an unbiased estimate of the full-distribution forward KL. A natural step lies between these limits. Its start remains student-selected, retaining OPD’s student-state coverage; its multiple target tokens unfold along the teacher’s own prefix, retaining the locally coherent supervision of SFT. The teacher can therefore express an executable correction within a step without changing the state at which the next student step was observed. 4.5 Relation to Other Distillation Procedures Off-policy sequence distillation trains on teacher-generated trajectories (11). Standard OPD queries teacher probabilities along student outputs and minimizes a token-level divergence (1). SOPD retains student-induced states but replaces logit access with local teacher generation. ExOPD uses reward extrapolation to enable a student to exceed a single teacher (22); SOPD uses no reward in its objective. TCOD assigns control intervals in a multi-turn trajectory through a temporal curriculum (20); Guided-OPD allows teacher actions to enter the environment according to a curriculum (13); and ReOPD replays offline teacher prefixes instead of interacting with the environment throughout training (14). SOPD instead keeps the natural temporal order, queries every student-visited step, and never executes a teacher target. Step granularity together with student-state anchoring defines the method. 5 Experiments We compare SOPD and OPD in two settings with long student-induced trajectories. ALFWorld partitions an interactive trajectory into environment turns, whereas mathematical reasoning partitions one response into reasoning steps. Both evaluations measure end-to-end outcomes and use the corresponding natural step structure during training. 5.1 ALFWorld Agent Interaction Setup. Following TCOD (20), we use a domain-RL-trained Qwen2.5-7B teacher to distill a Qwen2.5-3B-Instruct student. The Explorer samples batches of 16 tasks, each rollout contains at most 30 environment turns, and the Trainer consumes 64 experiences per batch with maximum staleness two. We evaluate on all 140 Valid Seen, 134 Valid Unseen, and 121 Hard tasks at temperature 0.4 with a 4,096-token response cap. Success rate measures task completion, and Rounds counts environment interactions. Table 1: ALFWorld results with a 3B student and 7B RL teacher. SR denotes success rate (%). TCOD supplies all rows except SOPD (20). Arrows show the absolute change from Vanilla OPD; bold marks the best student result in each column. Valid Seen Valid Unseen Hard Method SR ↑ Rounds ↓ SR ↑ Rounds ↓ SR ↑ Rounds ↓ Qwen2.5-7B-RL (Teacher) 85.71 10.61 76.87 13.06 6.61 27.31 Qwen2.5-3B Zero-Shot 7.86 28.73 2.24 29.63 0.83 29.88 Supervised fine-tuning (SFT) 32.14 22.85 25.37 24.16 4.96 29.12 Vanilla OPD 65.72 14.73 60.45 16.21 10.74 28.64 TCOD (B2F) 77.86 ↑ 12.14 12.57 ↓ 2.16 70.90 ↑ 10.45 14.56 ↓ 1.65 13.22 ↑ 2.48 28.16 ↓ 0.48 TCOD (F2B) 81.43 ↑ 15.71 11.76 ↓ 2.97 79.19 ↑ 18.74 12.47 ↓ 3.74 9.92 ↓ 0.82 28.57 ↓ 0.07 SOPD 84.29 ↑ 18.57 11.20 ↓ 3.53 82.09 ↑ 21.64 11.88 ↓ 4.33 10.74 ↔ 0.00 28.15 ↓ 0.49 The SOPD run leads on Seen and Unseen interaction. Relative to Vanilla OPD, SOPD raises Seen success from 65.72 to 84.29 and Unseen success from 60.45 to 82.09 while reducing mean rounds by 3.53 and 4.33, respectively. Among the student methods, SOPD records both the highest success and the fewest rounds on these two splits. Its Unseen success exceeds TCOD-F2B by 2.90 points. Performance across difficulty splits. On Hard, SOPD reaches 10.74% success while recording 28.15 mean rounds, the shortest interaction trajectory among all student methods. Together with the substantial gains on Seen and Unseen, these results show that SOPD learns common and unseen interaction states more effectively while maintaining stable behavior on difficult tasks. Figure 2: ALFWorld full-split evaluation over normalized Explorer progress. Seen and Unseen success improve over training while interaction length decreases overall. The curves summarize full-split evaluations at the saved checkpoints. Figure 2 shows the full-split training trajectory corresponding to Table 1. Seen and Unseen success improve overall during training as their interaction lengths decrease, while the Hard split remains comparatively stable. This trend agrees with the main table and shows that SOPD improves task completion while learning more efficient interaction policies. Following TCOD’s multi-panel analysis (20), Figure 3 reports SOPD-specific online signals. The generation-only objective computes no KL quantity. Explorer curves report the current 16-task rollout batches, whereas the optimization panel follows the Trainer. These online signals complement the full-split evaluations in Figure 2. Figure 3: ALFWorld training diagnostics following TCOD’s multi-panel analysis. (a) Explorer-batch task success; (b) mean environment rounds; (c) mean tokens per generated teacher turn target; and (d) step-balanced SFT loss in the Trainer. The horizontal axis normalizes each signal to the endpoint of its corresponding component. 5.2 Mathematical Reasoning Setup. We primarily use the Qwen3-4B non-thinking model (21). The student is initialized from Qwen3-4B non-thinking, and the domain teacher is obtained by applying reinforcement learning to the same base model on domain-specific data. Following G-OPD (22), we filter DeepMath (8) to retain 57,000 examples with difficulty level at least 6 and use this set as the mathematical RL data. We then apply Group Relative Policy Optimization (GRPO) (17) to obtain Qwen3-4B-Non-Thinking-RL-Math. The teacher receives reward 1.0 when the final answer is correct and 0.0 otherwise. Appendix A reports the complete teacher-training configuration. For distillation, SOPD partitions responses at natural reasoning boundaries, whereas standard OPD optimizes sampled-token reverse KL. Both methods use the same filtered DeepMath prompt pool and sample one student response for each prompt. Detailed SOPD and OPD hyperparameters, including the step-partition rule, are given in Appendix A. We evaluate mathematical reasoning on four competition-level benchmarks: AIME24 (2), AIME25 (16), HMMT25-Feb, and HMMT25-Nov (3). For every evaluation, we set temperature and top-p to 1.0 and the maximum generation length to 32,768 tokens. We sample 32 solutions for each problem and report mean accuracy on each benchmark. Answer correctness is determined by the rule-based Math-Verify11 1 https://github.com/huggingface/Math-Verify verifier. Table 2: Comparison of OPD and SOPD using the same-size teacher–student pair. Teacher denotes Qwen3-4B-Non-Thinking-RL-Math, and Student denotes the initial Qwen3-4B non-thinking model. The first three rows are reference results reported by G-OPD (22). Values are mean accuracy (%); Avg4Avg_4 averages the four benchmarks, and bold marks the best student method in each column. Method AIME24 AIME25 HMMT25 (Feb.) HMMT25 (Nov.) Avg4Avg_4 Teacher 58.0 54.6 32.5 38.9 46.0 ++ continued RL (100 updates) 60.9 55.6 32.8 38.4 46.9 Student 21.5 21.9 10.0 8.0 15.4 OPD 61.9 57.0 32.5 39.6 47.7 SOPD 71.8 67.4 38.9 52.7 57.7 The SOPD run leads on all four benchmarks. SOPD raises mean accuracy over OPD by 9.9 points on AIME24, 10.4 points on AIME25, 6.4 points on HMMT25-Feb, and 13.1 points on HMMT25-Nov. Its four-benchmark average reaches 57.7, compared with 47.7 for OPD. Each cell aggregates 32 decoding samples for each of 30 problems, providing a comprehensive evaluation of the final checkpoint. Figure 4: SOPD step structure over mathematical training. (a) Mean natural steps per AIME24 evaluation response; (b) mean student tokens per natural step; and (c) mean supervised teacher tokens per retained step during training. All boundaries use the semantic-step rule and 1,024-token safety cap. The learned response structure changes during training. Figure 4 tracks the quantities that define SOPD’s supervision units. As training progresses, the number of natural reasoning steps generated by the student increases substantially, while the length of each step remains broadly stable. This indicates that the model extends its solution process mainly by organizing more structured reasoning steps. Mean teacher supervision grows from 29.0 tokens at update 1 to 35.0 at update 8, then increases to 41.9 and 45.2 at the final two updates. Together, the three statistics show that later checkpoints both organize more reasoning steps and receive richer teacher targets at each local state. 6 Conclusion SOPD turns student-visited intermediate states into black-box teacher queries. It preserves the complete student trajectory, generates one teacher target at every natural step, and optimizes a step-balanced cross-entropy loss. At response-level granularity, the method recovers sequence SFT; at token-level granularity, it corresponds to a sampled forward-KL OPD objective. Results in mathematical reasoning and ALFWorld show that SOPD provides coherent, dense local supervision without allowing the teacher to control later states, yielding consistent gains across four mathematical benchmarks and the ALFWorld Seen and Unseen splits. Reproducibility Statement Section 5 specifies the model pairs, training data, rollout protocol, optimization settings, and evaluation suites used in the main experiments. Appendix A gives the per-update batch sizes, learning rates, decoding parameters, response limits, evaluation sample counts, seeds, and ALFWorld interaction limits. The submission source includes the figure-generation scripts and the aggregated records used to produce the reported tables and figures. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §4.5. AI-MO (2024) AI-MO AIME 2024. Cited by: §5.2. Balunović et al. (2025) M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev MathArena: evaluating LLMs on uncontaminated math competitions. SRI Lab, ETH Zurich. External Links: Link Cited by: §5.2. Da et al. (2025) J. Da, C. Wang, X. Deng, Y. Ma, N. Barhate, and S. Hendryx Agent-RLVR: training software engineering agents via guidance and environment rewards. arXiv preprint arXiv:2506.11425. External Links: Link Cited by: §2. Ding et al. (2026) Y. Ding, X. Wei, Y. Y. Li, Z. Li, Y. Lu, S. Zhang, D. Ma, R. Weng, X. Cai, and Y. Chen SAF-OPD: stable advantage fusion for on-policy distillation. arXiv preprint arXiv:2607.29209. External Links: Link Cited by: §2. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang MiniLLM: knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2. Guha et al. (2025) E. Guha, R. Marten, S. Keh, N. Raoof, G. Smyrnis, H. Bansal, M. Nezhurina, J. Mercat, T. Vu, Z. Sprague, et al. OpenThoughts: data recipes for reasoning models. arXiv preprint arXiv:2506.04178. External Links: Link Cited by: §1. He et al. (2025) Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, Z. Zhang, R. Wang, Z. Tu, H. Mi, and D. Yu DeepMath-103K: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456. External Links: Link Cited by: §5.2. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: §2. Jiang et al. (2026) L. Jiang, H. Xu, Y. Ding, and A. Zhang Trajectory-refined distillation. arXiv preprint arXiv:2606.08432. External Links: Link Cited by: §1, §2, §3. Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, p. 1317–1327. Cited by: §2, §4.5. Lauffer et al. (2025) N. Lauffer, X. Deng, S. Kundurthy, B. Kenstler, and J. Da Imitation learning for multi-turn LM agents via on-policy expert corrections. arXiv preprint arXiv:2512.14895. External Links: Link Cited by: §1, §2. Li et al. (2026) G. Li, M. Zheng, M. Song, R. Liu, T. Yang, J. Sun, Q. Zhong, H. Guo, J. Fang, D. Zhang, and J. Wang On-policy distillation with curriculum turn-level guidance for multi-turn agents. arXiv preprint arXiv:2606.15912. External Links: Link Cited by: §2, §4.5. Liao et al. (2026) B. Liao, H. Dong, C. Monz, X. Xu, L. Dong, and F. Wei Multi-turn on-policy distillation with prefix replay. arXiv preprint arXiv:2607.04763. External Links: Link Cited by: §2, §4.5. Lu and Lab (2025) K. Lu and T. M. Lab On-policy distillation. Thinking Machines Lab: Connectionism. External Links: Document Cited by: §1. OpenCompass (2025) OpenCompass AIME 2025. Cited by: §5.2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. External Links: Link Cited by: §5.2. Shridhar et al. (2020) M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht ALFWorld: aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768. Cited by: §2. Taori et al. (2023) R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto Alpaca: a strong, replicable instruction-following model. Note: Stanford Center for Research on Foundation Models External Links: Link Cited by: §1. Wang et al. (2026) J. Wang, W. Zhang, W. Shi, Y. Li, and J. Cheng TCOD: exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents. arXiv preprint arXiv:2604.24005. External Links: Link Cited by: §2, §4.5, §5.1, §5.1, Table 1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. External Links: Link Cited by: §1, §5.2. Yang et al. (2026) W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin Learning beyond teacher: generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125. External Links: Link Cited by: §A.1, Table 3, §2, §4.5, §5.2, Table 2. Ye et al. (2025) T. Ye, L. Dong, Z. Chi, X. Wu, S. Huang, and F. Wei Black-box on-policy distillation of large language models. arXiv preprint arXiv:2511.10643. Cited by: §2. Zhang et al. (2026) Y. Zhang, X. Lin, and C. Wu StepOPSD: step-aware online preference distillation for agent reinforcement learning. arXiv preprint arXiv:2605.27140. External Links: Link Cited by: §2. Zhao et al. (2026) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. Cited by: §2. Zhong et al. (2026) Q. Zhong, M. Zheng, M. Song, X. Lin, J. Sun, H. Jiang, X. Wang, and J. Fang SOD: step-wise on-policy distillation for small language model agents. arXiv preprint arXiv:2605.07725. External Links: Link Cited by: §2. Appendix A Training and Evaluation Details A.1 Mathematical Reasoning The mathematical runs use Qwen3-4B non-thinking as the base student and Qwen3-4B-Non-Thinking-RL-Math-Step500 as the teacher. We construct the teacher following the mathematical GRPO configuration of G-OPD (22). Specifically, we retain 57,000 DeepMath examples with difficulty level at least 6, assign reward 1.0 when the final answer is correct and 0.0 otherwise, and train for 500 optimizer updates. Table 3 lists the complete teacher-training configuration used in our experiments. Table 3: GRPO hyperparameters used to train the mathematical domain teacher, following G-OPD (22). Hyperparameter Value Training batch size 128 Micro-batch size 128 Rollouts per prompt 8 Maximum prompt length 2,048 Maximum response length 16,384 Temperature 1.0 Top-p 1.0 Learning rate 1×10−61× 10^-6 Optimizer updates 500 KL coefficient 0.0 SOPD and OPD then consume the same fixed order of prompts from the filtered DeepMath pool. Each optimizer update uses 1,024 prompts and one student rollout per prompt. Both methods run for ten updates, so each processes 10,240 prompt instances and 10,240 student rollouts. The maximum prompt and response lengths are 2,048 and 32,768 tokens, respectively; student rollouts use temperature 1.0 and top-p 1.0. SOPD uses learning rate 2×10−62× 10^-6 and generates each teacher step greedily at temperature 0. Standard OPD uses learning rate 10−510^-5 and sampled-token reverse KL. For SOPD, a blank-line paragraph boundary or an explicit reasoning heading such as “Step” or “Solution” terminates a natural step. If no such boundary appears, a 1,024-token safety cap closes the span. Every retained student prefix receives one independently generated teacher step, and the loss is averaged first over tokens within each target and then over retained steps. The terminal evaluation sends 32 independent generations for each of 30 problems in AIME24, AIME25, HMMT25-Feb, and HMMT25-Nov. Each method therefore contributes 960 responses per benchmark. Evaluation uses temperature 1.0, top-p 1.0, and a 32,768-token response cap. Every response is a separate request with seed 42+32i+j42+32i+j for problem i and sample j. The evaluator extracts the final boxed answer and applies rule-based symbolic verification with Math-Verify. The result extractor checks all numerators and denominators against the durable summaries. A.2 ALFWorld We use the Qwen2.5-3B-Instruct student and ‘langfeng01/GiGPO-Qwen2.5-7B-Instruct-ALFWorld’ teacher from the TCOD model pairing. The Explorer samples 16 tasks with one rollout each. The Trainer consumes 64 experiences per batch with learning rate 10−610^-6, temperature 1.0, staleness two, a 512-token training response cap, a 10,240-token prompt cap, and at most 30 environment turns. The released asynchronous launcher lets Explorer and Trainer counters advance independently. Full evaluation covers Valid Seen-140, Valid Unseen-134, and Hard-121. It samples at temperature 0.4 with a 4,096-token response cap. We report the task-level success mean and mean environment rounds. Appendix B LLM Usage Generative AI tools were used to assist language editing, translation, figure-layout iteration, and LaTeX preparation. The authors determined the research claims, methods, experiments, and conclusions, and manually verified the final manuscript and reported results.

Instant research discovery