Paper deep dive
Interactive-Policy Distillation with Bidirectional Propose-and-Verify
Shutong Wu, Xiwen Chen, Brendan Rappazzo, Daiheng Zhang, Anderson Schneider, Yuriy Nevmyvaka, Jiawei Zhang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD may suffer from teacher unanchoring, where the student's reasoning trajectory drifts far from the teacher, causing the teacher to be queried on states it would hardly visit and thus provide unreliable supervision. We propose Interactive-Policy Distillation (IPD), which applies adaptive teacher intervention to the student rollout. Under a bidirectional propose-and-verify state machine, the student and teacher alternately exchange their roles as proposer and verifier, and collaboratively generate mixed-source trajectories. Then different supervisions are applied according to the source of each token. This bidirectional propose-and-verify mechanism and the source-split loss make IPD not only a more performant distillation method, but also a unified bridge between on-policy and off-policy paradigms. To make the interleaved dual-model rollouts more efficient, we also design a dedicated fused inference engine that co-hosts both models in one serving instance with separate KV caches and instantiates the state machine model to distribute, collect, and process requests. On math reasoning tasks and across multiple teacher-student model pairs, student models trained with IPD not only outperform those trained with OPD, but also demonstrate higher data efficiency. Specifically, when distilling Qwen3-30B-A3B into Qwen3-1.7B-Base, IPD brings a +3.28 mean@8 and a +3.28 best@8 benchmark-averaged accuracy improvement compared with OPD. Besides, IPD only consumes about 1/4 of the training examples and steps to outperform OPD trained on the whole training dataset for one epoch. We also investigate the impact of different loss variants and takeover / handback configurations, and demonstrate the robustness of IPD on different training data.
Tags
Links
- Source: https://arxiv.org/abs/2609.36546v1
- Canonical: https://arxiv.org/abs/2609.36546v1
Trouble viewing inline? Open PDF directly →
Full Text
62,821 characters extracted from source content.
Expand or collapse full text
Interactive-Policy Distillation with Bidirectional Propose-and-Verify Shutong Wu1*, Xiwen Chen2, Brendan Rappazzo2, Daiheng Zhang3, Anderson Schneider2, Yuriy Nevmyvaka2, Jiawei Zhang1 1Department of Computer Sciences, University of Wisconsin–Madison 2Machine Learning Research, Morgan Stanley 3Department of Electrical and Computer Engineering, Rutgers University–New Brunswick Abstract On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD may suffer from teacher unanchoring, where the student’s reasoning trajectory drifts far from the teacher, causing the teacher to be queried on states it would hardly visit and thus provide unreliable supervision. We propose Interactive-Policy Distillation (IPD), which applies adaptive teacher intervention to the student rollout. Under a bidirectional propose-and-verify state machine, the student and teacher alternately exchange their roles as proposer and verifier, and collaboratively generate mixed-source trajectories. Then different supervisions are applied according to the source of each token. This bidirectional propose-and-verify mechanism and the source-split loss make IPD not only a more performant distillation method, but also a unified bridge between on-policy and off-policy paradigms. To make the interleaved dual-model rollouts more efficient, we also design a dedicated fused inference engine that co-hosts both models in one serving instance with separate KV caches and instantiates the state machine model to distribute, collect, and process requests. On math reasoning tasks and across multiple teacher-student model pairs, student models trained with IPD not only outperform those trained with OPD, but also demonstrate higher data efficiency. Specifically, when distilling Qwen3-30B-A3B into Qwen3-1.7B-Base, IPD brings a +3.28+3.28 mean@8 and a +3.28+3.28 best@8 benchmark-averaged accuracy improvement compared with OPD. Besides, IPD only consumes about 1/41/4 of the training examples and steps to outperform OPD trained on the whole training dataset for one epoch. We also investigate the impact of different loss variants and takeover / handback configurations, and demonstrate the robustness of IPD on different training data. 11footnotetext: Work done during the internship at Morgan Stanley. Correspondence to Shutong Wu <swu494@wisc.edu> and Jiawei Zhang <jzhang2924@wisc.edu>. Code available at https://github.com/cychomatica/Interactive-Policy-Distillation. 1 Introduction Large language models (LLMs) have advanced at a remarkable pace over the past few years, and current frontier models attain expert-level performance in mathematics, programming, and scientific problems, and drive increasingly capable agents (OpenAI, 2026; Anthropic, 2026; Z.ai, 2026; Xu et al., 2026). These extraordinary capabilities, however, are usually obtained by the very largest models, which pair over hundreds of billions of parameters and long chains of thought. The high memory and computation cost makes them a suboptimal fit for latency-sensitive, on-device, or high-throughput deployments. Knowledge distillation (Hinton et al., 2015) offers a way out by transferring as much of the large model’s capability as possible into a more compact and cheaper student model. In the LLM era, the dominant paradigm used to be off-policy distillation via supervised fine-tuning (SFT), where the teacher generates high-quality reasoning traces, and the student learns to imitate them token by token (Kim and Rush, 2016; Ho et al., 2023; Magister et al., 2023; Hsieh et al., 2023). The distilled variants of DeepSeek-R1, trained by SFT on roughly 800K teacher-generated samples, are a prominent recent example of how far this simple recipe scales (Guo et al., 2025). Despite its scalability, off-policy distillation faces a structural challenge: the student is trained under the teacher’s demonstrations, and the imitation is only ever supervised on teacher-anchored states that the student is not likely to visit. Therefore, the student never learns to continue from its own partial generations, and small early deviations can compound at inference time, as discussed in the classic exposure-bias problem (Bengio et al., 2015; Ranzato et al., 2015; Arora et al., 2022). Long and stylized teacher traces can exceed what a small student is able to absorb (Mirzadeh et al., 2020; Shing et al., 2025; Li et al., 2025), and the forward KL pushes the student to spread probability mass over every teacher mode rather than reliably on modes it can actually realize (Gu et al., 2024). On-policy distillation (OPD) overcomes these challenges by moving the training distribution to the student. The student generates its own responses, and the teacher supervises every generated token with dense probabilistic feedback (Agarwal et al., 2024; Gu et al., 2024; Lu and Lab, 2025). Training on self-generated states removes the training-inference mismatch by construction. The student learns from states it actually visits (Ross et al., 2011), while the teacher’s per-token distributions provide denser learning signals than scalar outcome rewards of reinforcement learning with verifiable rewards (RLVR). With those advantages, OPD can match RLVR-based training with much less compute (Lu and Lab, 2025), and has quickly moved into production practice as a core stage of post-training pipelines for building reasoning models (Yang et al., 2025; Ma et al., 2026), accompanied by a growing line of refinements on divergence choice and data mixing (Ko et al., 2024). Figure 1: The rollout and training demonstration of IPD. The student and teacher collaboratively generate the rollout under a bidirectional propose-and-verify state machine. Then tokens are grouped by their sources, and the IPD source-split loss is applied to update the student weights. However, student-generated trajectories can deviate far from the teacher when there is a large reasoning gap between them, and then the teacher is queried by states it would hardly visit and may provide unreliable learning signals (Ko et al., 2024; Xu et al., 2025). We call this failure mode teacher unanchoring. To overcome this challenge, we introduce Interactive-Policy Distillation (IPD), which couples a bidirectional propose-and-verify state machine with different supervisions on tokens from different sources. As shown in Figure 1, IPD allows the teacher to dynamically correct a drifting rollout and returns control once the corrected continuation falls within the student’s acceptance region where student can actually resume. As the resulting trajectory interleaves the student’s own decisions with teacher-provided corrective demonstrations, IPD assigns different supervisions by token sources, i.e., reverse KL at student-generated positions and forward KL at teacher-generated positions. Our contributions are summarized as follows: • We introduce IPD, a novel distillation algorithm with adaptive teacher intervention through a bidirectional propose-and-verify state machine, in which student and teacher dynamically exchange proposer and verifier roles. Its source-split objective applies reverse KL to student-generated positions and forward KL to teacher-generated positions, bridging on-policy and off-policy distillation paradigms. • We design a fused inference engine, which co-hosts both models in one serving instance, maintains separate KV caches, and schedules requests according to their current proposer and verifier roles, to achieve more efficient dual-model scheduling during the rollout stage. • Empirically, across four teacher-student pairs on mathematical reasoning, IPD achieves the highest benchmark-averaged mean@8 and best@8 accuracy and leads on most individual benchmark metrics. Besides, IPD shows higher data efficiency. When distilling Qwen3-30B-A3B to Qwen3-1.7B-Base, IPD surpasses the mean@8 of OPD trained for a full epoch (279 steps) after only 60 training steps. We also demonstrate the robustness of IPD across different training datasets. 2 Related Works On-policy distillation for LLMs. LLM distillation used to be dominated by off-policy distillation on teacher-generated traces (Hinton et al., 2015; Kim and Rush, 2016; Guo et al., 2025). The main weakness of off-policy distillation is that the student trains on teacher-authored prefixes but deploys on its own, and is vulnerable to exposure bias (Bengio et al., 2015; Arora et al., 2022). GKD (Agarwal et al., 2024) applies purely on-policy student sequences for training, and MiniLLM (Gu et al., 2024) studies sequence-level reverse KL on those sequences. DistiLLM (Ko et al., 2024) introduces skew divergences with an adaptive off-policy scheduler, and DistiLLM-2 (Ko et al., 2025) contrasts teacher and student responses with skew forward and reverse KL. Recently, OPD has already been incorporated as a core stage into reasoning-model post-training pipelines (Yang et al., 2025; Lu and Lab, 2025; Xiao et al., 2026). Challenges in on-policy distillation. Recent analyses characterize unreliable teacher feedback on student rollouts (Fu et al., 2026; Armandpour et al., 2026; Li et al., 2026). EOPD (Jin et al., 2026) adjusts the loss while retaining student-generated rollouts. TrOPD (Xing et al., 2026) uses a teacher-generated prefix followed by a student continuation and applies region-specific losses, but without dynamic teacher intervention during the continuation. SKD (Xu et al., 2025) adapts speculative propose-and-verify (Leviathan et al., 2023; Chen et al., 2023) to rollout and implements dynamic teacher corrections. The student and teacher remain fixed roles as proposer and verifier. When a student token is rejected, the teacher resamples that position, and the student resumes generation from the corrected prefix. Interactive imitation learning studies intervention and return of control through human judgment in HG-DAgger (Kelly et al., 2019) or uncertainty-based gating in EnsembleDAgger (Menda et al., 2019). In contrast, IPD build a state machine model that dynamically swaps proposer and verifier roles through teacher takeover on rejected student tokens and handback on accepted teacher tokens, and then trains the resulting mixed-source trajectory with a source-split objective. 3 Methodology 3.1 Preliminaries Let y∼y be a prompt from a dataset D, x=(x1,…,xT)x=(x_1,…,x_T) a response trajectory, and ct=(y,x<t)c_t=(y,x_<t) the prefix context at step t. The student and teacher next-token distributions are denoted as πθ(⋅∣ct) _θ(· c_t) and q(⋅∣ct)q(· c_t), and their complete-response probabilities factorize as πθ(x∣y)=∏t=1|x|πθ(xt∣ct) _θ(x y)= _t=1^|x| _θ(x_t c_t) and q(x∣y)=∏t=1|x|q(xt∣ct)q(x y)= _t=1^|x|q(x_t c_t). Off-Policy Distillation. Traditional knowledge distillation trains on prefixes sampled from a distribution μ, which is typically a dataset of fixed reference responses or trajectories sampled from a teacher model (Hinton et al., 2015; Kim and Rush, 2016). Its standard objective is minimizing the forward KL, ℒoff(θ)=y∼,x∼μ(⋅∣y)[1|x|∑t=1|x|DKL(q(⋅∣ct)∥πθ(⋅∣ct))].L_off(θ)=E_ subarraycy ,x μ(· y) subarray [ 1|x| _t=1^|x|D_KL\! (q(· c_t)\,\|\, _θ(· c_t) ) ]. (1) The fixed prefixes make optimization stable and permit dense supervision over the vocabulary, but they also induce a state-distribution mismatch: at inference time, the student must condition on its own previous tokens, including mistakes that are absent from teacher-generated trajectories. On-Policy Distillation. Compared with off-policy distillation, on-policy distillation (OPD) evaluates teacher feedback on responses sampled from the student itself (Agarwal et al., 2024; Gu et al., 2024). Under the notations above, its theoretical objective is the sequence-level reverse KL ℒonseq(θ)=y∼[DKL(πθ(⋅∣y)∥q(⋅∣y))]=y∼,x∼πθ(⋅∣y)[∑t=1|x|DKL(πθ(⋅∣ct)∥q(⋅∣ct))],L^seq_on(θ)=E_y [D_KL\! ( _θ(· y)\,\|\,q(· y) ) ]=E_ subarraycy ,x _θ(· y) subarray [ _t=1^|x|D_KL\! ( _θ(· c_t)\,\|\,q(· c_t) ) ], (2) The outer expectation samples prefixes from the student-induced state distribution, while each summand is a token-level conditional reverse KL at one such prefix. At a fixed prefix ctc_t, the exact token-level conditional reverse KL is computed over the entire vocabulary V as ℓtfull(θ)=DKL(πθ(⋅∣ct)∥q(⋅∣ct))=∑v∈πθ(v∣ct)logπθ(v∣ct)q(v∣ct). ^full_t(θ)=D_KL\! ( _θ(· c_t)\,\|\,q(· c_t) )= _v _θ(v c_t) _θ(v c_t)q(v c_t). (3) Optimizing Eq. (3) requires forward and backward passes over all |||V| indices at every position, which is computationally expensive (Li et al., 2026; Oh et al., 2026). Sampled-Token and Top-K OPD. As a less expensive alternative to the full-vocabulary KL in Eq. (3), sampled-token OPD evaluates only the student-sampled token xt∼πθ(⋅∣ct)x_t _θ(· c_t). The resulting single-sample Monte Carlo estimate of the token-level KL value at the fixed prefix ctc_t is ℓ^tsample=logπθ(xt∣ct)−logq(xt∣ct). ^sample_t= _θ(x_t c_t)- q(x_t c_t). (4) Although Eq. (4) is an unbiased estimate for the full-vocabulary KL, the variance introduced by sampling a single token can make updates noisy and unstable (Oh et al., 2026). Top-K OPD reaches a sweet spot between full-vocabulary and sampled-token OPD. It retains multi-token supervision on a small high-probability support while avoiding a full-vocabulary computation (Li et al., 2026; Fu et al., 2026). Let Stπ=TopK(πθ(⋅∣ct)),Stq=TopK(q(⋅∣ct)),S_t^π=Top_K\! ( _θ(· c_t) ), S_t^q=Top_K\! (q(· c_t) ), (5) denote the student and teacher top-K supports, respectively, where TopKTop_K returns the K highest-probability indices, and denote St⊆S_t as the support for distillation, which can usually be StπS_t^π, StqS_t^q, Stπ∪StqS_t^π∪ S_t^q, or Stπ∩StqS_t^π∩ S_t^q. As the two models generally assign different total probability mass to StS_t, each distribution should be normalized by π¯θSt(v∣ct)=πθ(v∣ct)[v∈St]∑u∈Stπθ(u∣ct),q¯St(v∣ct)=q(v∣ct)[v∈St]∑u∈Stq(u∣ct). π^S_t_θ(v c_t)= _θ(v c_t) 1[v∈ S_t] _u∈ S_t _θ(u c_t), q^S_t(v c_t)= q(v c_t) 1[v∈ S_t] _u∈ S_tq(u c_t). (6) Then, the token-level top-K KL and its rollout-level aggregated loss are given by ℓttop-K(θ;St)=DKL(π¯θSt(⋅∣ct)∥q¯St(⋅∣ct))=∑v∈Stπ¯θSt(v∣ct)logπ¯θSt(v∣ct)q¯St(v∣ct), ^top-K_t(θ;S_t)=D_KL\! ( π^S_t_θ(· c_t)\,\|\, q^S_t(· c_t) )= _v∈ S_t π^S_t_θ(v c_t) π^S_t_θ(v c_t) q^S_t(v c_t), (7) ℒontop-K(θ)=y∼,x∼πθ(⋅∣y)[∑t=1|x|ℓttop-K(θ,St)]. ^top-K_on(θ)=E_ subarraycy ,x _θ(· y) subarray [ _t=1^|x| ^top-K_t(θ;S_t) ]. (8) Although top-K OPD is a biased approximation to full-vocabulary OPD as it discards probability mass outside StS_t, it replaces the high-variance single-token estimate of sampled-token OPD with deterministic multi-token supervision over the selected high-probability support for each fixed prefix ctc_t, and therefore provides a practical middle ground with denser and more stable supervision than sampled-token OPD (Fu et al., 2026), while at the same time reducing the per-token computation and backward-pass cost of full-vocabulary OPD (Li et al., 2026; Oh et al., 2026). 3.2 Interactive-Policy Distillation Vanilla OPD samples trajectories from the student, and the teacher feedback is evaluated on states that may be far from where it would actually visit. We name it as teacher unanchoring. Figure 2 illustrates the average entropy and reverse KL over position intervals on the student (Qwen3-1.7B-Base) responses for 512 randomly selected DAPO-MATH-17K (Yu et al., 2025) prompts, split by the final answer correctness. The teacher (Qwen3-30B-A3B non-thinking) queried by incorrect trajectories yields high uncertainty, even as the reverse KL decays along the position. We propose Interactive-Policy Distillation (IPD), which formulates generation as a state machine in which the student and teacher alternate as proposer and verifier. to address this challenge by dynamic adaptive teacher intervention during rollout. Figure 2: Average entropy and reverse KL over position intervals on the student responses split by the final answer correctness. 3.2.1 Adaptive Teacher Intervention with Bidirectional Propose-and-Verify Let mt∈,m_t∈\ S, T\ denote the driver model state before generating token at position t, where S and T indicate student driving and teacher driving, respectively, and we initialize every rollout with m1=m_1= S. Let zt∈,z_t∈\ S, T\ record the source of the token actually emitted at position t. We use top-K coverage as the takeover and handback rules, with StπS_t^π and StqS_t^q defined in Eq. (5). Student driving. When mt=m_t= S, the student proposes x~tπ∼πθ(⋅∣ct) x_t^π _θ(· c_t) and the teacher verifies whether the student-generated token is covered by the teacher’s top-K support, i.e., x~tπ∈Stq x_t^π∈ S_t^q. If covered, which indicates that the student’s proposal within the teacher’s acceptance region, IPD accepts the student proposal, assigns zt=z_t= S, and remains in student driving state. Otherwise, the proposal is discarded, and the teacher samples a replacement x~tq∼q(⋅∣ct) x_t^q q(· c_t). IPD emits this teacher-sampled token with zt=z_t= T, and the rollout control switches to the teacher from position t+1t+1. Teacher driving. When mt=m_t= T, the teacher proposes x~tq∼q(⋅∣ct) x_t^q q(· c_t) and the student verifies whether the teacher-sampled token satisfies x~tq∈Stπ x_t^q∈ S_t^π. If it is not covered by the student’s top-K support, the teacher retains control. Otherwise, as the teacher’s proposal is back to the student’s acceptance region, which indicates that the student is possible to generate is on its own, it is kept and serves as the handback trigger and the rollout control is returned to student rom position t+1t+1. The complete per-token transition can be written as (xt,zt,mt+1)=(x~tπ,,),mt=,x~tπ∈Stq,(x~tq,,),mt=,x~tπ∉Stq,(x~tq,,),mt=,x~tq∈Stπ,(x~tq,,),mt=,x~tq∉Stπ.(x_t,z_t,m_t+1)= cases( x_t^π, S, S),&m_t= S,\ x_t^π∈ S_t^q,\\ ( x_t^q, T, T),&m_t= S,\ x_t^π∉ S_t^q,\\ ( x_t^q, T, S),&m_t= T,\ x_t^q∈ S_t^π,\\ ( x_t^q, T, T),&m_t= T,\ x_t^q∉ S_t^π. cases (9) The state machine iterates until an EOS token is emitted or the maximum rollout length is reached. Eq. (9) is written token by token for clarity. Following Leviathan et al. (2023) and Xu et al. (2025), for a faster rollout, the implementation usually consecutively proposes a chunk of γ tokens and verify them in parallel, instead of token-by-token verification immediately after decoded. Note that in this paper we use the top-K coverage as the takeover and handback rules. As a unified framework, IPD is also compatible with other rules, and the takeover and handback rules can also be asymmetric. For example, we can instead use a threshold on the entropy of the student or the teacher distributions to determine whether to take over or hand back control. SKD (Xu et al., 2025) can be included as a special case of IPD with a top-K coverage takeover rule and a fixed-one-token handback rule, and in our experiments, SKD is also implemented under our customized fused engine for more efficient rollout. 3.2.2 Source-Split KL Objective The state machine model formulated by Eq. (9) produces a response trajectory x together consisting of student-generated and teacher-generated tokens with recorded source labels z=(z1,…,zT)z=(z_1,…,z_T). Denote Ion(z)=t:zt=I_on(z)=\t:z_t= S\ and Ioff(z)=t:zt=I_off(z)=\t:z_t= T\ as the on-policy (student-generated) and off-policy (teacher-generated) position sets, respectively. Since the trajectory x is a mixure of student and teacher tokens, it is natural provide the two groups of positions with different supervisions. We therefore design a source-split KL objective that applies different KL divergences to student-generated and teacher-generated positions. Specifically, the token-level KL divergences applied to student-generated and teacher-generated positions are respectively denoted as ℓton(θ)=DKL(π¯θSt(⋅∣ct)∥q¯St(⋅∣ct)),ℓtoff(θ)=DKL(q¯St(⋅∣ct)∥π¯θSt(⋅∣ct)). _t^on(θ)=D_KL\! ( π^S_t_θ(· c_t)\,\|\, q^S_t(· c_t) ), _t^off(θ)=D_KL\! ( q^S_t(· c_t)\,\|\, π^S_t_θ(· c_t) ). (10) Student-generated positions therefore receive the token-level reverse KL used by on-policy distillation, while teacher-generated tokens receive the token-level forward KL used by off-policy distillation. Both losses are evaluated at the realized mixed-source prefix ctc_t, regardless of which model generated earlier tokens. Let ρθIPD(x,z∣y) _θ^IPD(x,z y) denote the joint distribution over emitted responses and source labels induced by the state machine in Eq. (9). The final length-normalized objective is minimizing the following source-split KL ℒIPD(θ)=y∼,(x,z)∼ρθIPD(⋅,⋅∣y)[1|x|(∑t∈Ion(z)ℓton(θ)+∑t∈Ioff(z)ℓtoff(θ))].L_IPD(θ)=E_ subarraycy ,\,(x,z) _θ^IPD(·,· y) subarray\! [ 1|x| ( _t∈ I_on(z) _t^on(θ)+ _t∈ I_off(z) _t^off(θ) ) ]. (11) With the bidirectional propose-and-verify mechanism and the source-split KL, IPD naturally becomes a unified bridge between on-policy and off-policy distillations. Specifically, never transferring control to the teacher yields length-normalized on-policy distillation with reverse KL supervision on pure student-generated trajectories, while always rejecting the first student proposal and never handing control back yields off-policy distillation with forward KL supervision on pure teacher-generated trajectories. 3.2.3 Fused Inference Engine for Efficient Dual-Model Scheduling Figure 3: IPD rollout throughput of fused engine and naive two-engine implementation. To integrate IPD with verl (Sheng et al., 2025), built on vLLM (Kwon et al., 2023), we design a fused inference engine for more efficient dual-model scheduling. A naive implementation that hosts two models on separate vLLM serving instances will have to re-encode the whole prefix for every verification round, as an off-the-shelf vLLM engine bypasses its prefix cache for requests that return prompt log-probabilities and cannot append externally generated tokens to the KV cache of a live request. Besides, there are also communication overheads between two engines. As shown in Figure 3, with Qwen3-4B-Base (non-thinking) as the teacher and Qwen3-1.7B-Base as the student, executing the IPD rollout with bidirectional propose-and-verify on the same 512 randomly selected DAPO-MATH-17K prompts under identical settings and memory budget, fused engine is 21.4× faster than the naive implementation on the initial checkpoint and 71.4× faster on the final one. Specifically, as illustrated in Figure 4, our fused engine co-hosts the student and teacher in one serving instance, and each active request maintains a verified prefix, a chunk of unverified draft tokens, the identity of its current driver model, and separate student and teacher KV-cache states. For each request, the two models serve as symmetrical roles. Each model maintains its own cache and may act as either the proposer or the verifier, with the verifier always being the counterpart of the current driver model. One fused engine step consists of a verification, a verdict, and a proposal stage. In the verification stage, requests are grouped by their verifier, and each verifier evaluates all unverified draft tokens of its sub-batch in parallel through a single prefill pass. Only the newly appended tokens require computation as long as the cache of the verified prefix remains resident. If vLLM preempts a request under resource contention and evicts its cached blocks, the affected model, whether verifier or proposer, will instead recompute the KV states of the whole prefix. In the verdict stage, the engine determines the acceptance or rejection of each draft, detects takeover and handback transitions, updates the driver model, and aligns both KV caches with the resulting verified prefix. If no transition occurs, all drafted tokens are kept and both caches are preserved. If a student token triggers takeover, the token is rejected and will be regenerated by the teacher, so both caches are rolled back to the preceding token. If a teacher token triggers handback, the token is retained, so both caches are rolled back only to it and the subsequent draft tokens are discarded. Either way, the two caches end at the same accepted-prefix boundary. In the proposal stage, requests are regrouped by their updated driver model, and each proposer autoregressively decodes a new chunk of γ draft tokens for its sub-batch while extending only its own KV cache, leaving the counterpart model to catch up through the prefill of the next verification stage. Consequently, apart from draft tokens discarded by a transition and cache blocks evicted by preemption, every token is decoded once by its proposer and prefilled once by its verifier. The engine repeats these stages until every request has emitted an accepted EOS token or reached the maximum generation length. A student-generated EOS token terminates the request only if it passes the teacher’s takeover test and otherwise triggers a teacher takeover at that position, whereas a teacher-generated EOS token is accepted directly. Figure 4: Example of one IPD fused engine step for a request batch. The engine distributes requests to their respective verifier models for verification, collects verification results, updates their driver models, manages their KV cache on each model, re-distributes requests to their respective proposer models for new unverified draft tokens, and return the updated requests for next step or termination. 4 Experiments 4.1 Experimental Settings We train four Qwen3 (Yang et al., 2025) teacher-student pairs on DAPO-Math-17K (Yu et al., 2025) for one epoch, with thinking mode disabled for all teachers. We set K=16K=16: all methods apply distillation over the union support St=Stπ∪StqS_t=S_t^π∪ S_t^q, and IPD and SKD use top-K coverage for takeover, with IPD also using it for handback. We use the codebase from Li et al. (2026) for OPD, and additionally build our fused inference engine and implement SKD and IPD training pipelines on top of that. Both IPD and SKD use a proposal chunk length of γ=5γ=5. We evaluate the final checkpoints on six mathematical reasoning benchmarks and report mean@8 and best@8 accuracy. Appendix A provides more detailed model, training, hardware, and evaluation configurations. Table 1: Comparison of different distillation methods across various student and teacher model combinations. We report mean@8 and best@8 accuracy (%) on different math reasoning benchmarks, as well as the benchmark-wise average. Teacher/ Student Method MATH-500 Minerva Olympiad AMC23 AIME24 AIME25 Avg. mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 Qwen3-4B/ Qwen3-1.7B-Base OPD 66.68 84.06 26.42 44.04 31.00 48.45 35.69 57.29 7.92 21.77 2.92 8.70 28.44 44.05 SKD 66.70 84.50 27.94 43.69 29.75 49.52 35.09 52.90 8.75 17.41 4.58 10.73 28.80 43.13 IPD (ours) 69.28 84.59 29.60 46.26 31.80 49.63 36.14 57.25 9.17 21.69 6.67 15.06 30.44 45.75 Qwen3-30B-A3B/ Qwen3-1.7B-Base OPD 63.35 83.54 23.25 41.43 28.14 46.88 32.68 54.39 10.42 18.84 5.83 13.12 27.28 43.03 SKD 67.38 84.34 26.70 44.69 30.39 49.73 33.89 54.67 8.75 22.36 5.83 14.92 28.82 45.12 IPD (ours) 69.35 84.61 29.60 46.44 32.56 51.29 34.79 59.60 11.67 22.79 5.42 13.10 30.56 46.31 Qwen3-8B/ Qwen3-4B-Base OPD 79.63 90.83 36.53 52.22 44.80 60.49 44.28 66.31 15.83 25.40 13.75 29.09 39.14 54.06 SKD 80.43 91.49 37.91 51.15 44.71 61.32 51.51 72.01 15.42 24.76 15.00 29.68 40.83 55.07 IPD (ours) 81.45 92.21 37.87 52.66 46.43 62.19 48.34 67.83 17.08 28.86 17.08 28.66 41.38 55.40 Qwen3-30B-A3B/ Qwen3-4B-Base OPD 81.38 90.00 39.11 53.41 46.66 61.18 48.80 67.88 15.42 24.78 12.92 26.36 40.71 53.93 SKD 81.03 91.96 40.40 55.39 47.42 63.95 49.85 72.82 16.25 24.85 16.67 27.60 41.93 56.09 IPD (ours) 82.28 92.40 41.18 55.52 47.66 63.45 49.55 70.64 19.58 28.39 13.75 26.22 42.33 56.10 4.2 Main Results We compare IPD with OPD (Lu and Lab, 2025; Li et al., 2026) and SKD (Xu et al., 2025) on multiple student and teacher model combinations across different sizes. Thinking mode is disabled for all non-base teachers. In Table 1, our algorithm demonstrates consistent improvements over the baselines across all benchmarks and model combinations. Specifically, when distilling Qwen3-4B to Qwen3-1.7B-Base, IPD achieves an average mean@8 accuracy of 30.4430.44 across all benchmarks, outperforming OPD and SKD by +2.00+2.00 and +1.64+1.64, respectively. Besides, IPD also allows the student to robustly learn from a larger teacher. According to the discussion in Li et al. (2026), a larger teacher cannot always distill a better student, and when the teacher-student reasoning gap is too large, the distillation performance can be suboptimal. In our experiments, when distilling Qwen3-30B-A3B to Qwen3-1.7B-Base, we also find this phenomenon that the distilled student achieves an average mean@8 accuracy of 27.2827.28, which is lower than the 28.4428.44 achieved by distilling a Qwen3-4B teacher. Meanwhile, with the bidirectional propose-and-verify mechanism, IPD can effectively bridge the reasoning gap between the teacher and student, achieving an average mean@8 accuracy of 30.5630.56, which is 3.283.28 higher than OPD and 1.741.74 higher than SKD. Besides, IPD is also a more data-efficient distillation algorithm. As shown in Figure 5, IPD shows a much faster convergence than other baselines. Specifically, trained with only fewer than 1/4 training examples for 60 training steps, IPD can outperform OPD and matches SKD trained for a whole epoch with 279 training steps. Besides, IPD shows a stably increasing top-K overlap ratio (defined by |Stπ∩Stq|/K|S^π_t∩ S^q_t|/K), indicating a healthy training under large a teacher-student gap (Li et al., 2026). Figure 5: The mean@8 accuracy over math reasoning benchmarks and top-K overlap ratio during training on DAPO-MATH-17K. In addition to better final checkpoint performance, IPD also shows a much higher data efficiency than other baselines. 4.3 Ablation Study KL direction. To investigate the impact of different KL directions for on-policy and off-policy positions, and the resonability of our source-split KL design, we conduct an ablation study on the IPD KL objective. We compare the following configurations: (i) Forward KL for all positions (Pure Forward KL), (i) Reverse KL for all positions (Pure Reverse KL), (i) Forward KL for on-policy and Reverse KL for off-policy positions (Inverse-IPD-Split KL), and (iv) Reverse KL for on-policy and Forward KL for off-policy positions (IPD-Split KL, default). As shown in Figure 6, we observe that the IPD Default configuration, which uses Reverse KL for on-policy positions and Forward KL for off-policy positions, achieves the best mean@8 accuracy. Meanwhile, Pure Reverse KL turns out to be the runner-up, with the second-best mean@8 accuracy and a 0.120.12 higher best@8 accuracy than the default source-split KL. The teacher intervention ratio, which is calculated by the proportion of teacher-generated tokens among all tokens, gradually decreases during training, indicating that the student is learning to generate more tokens that are covered by the teacher’s top-K support, and leaning more towards on-policy paradigm. Figure 6: The mean@8 and best@8 average accuracy over benchmarks and the teacher intervention ratio with different losses during IPD training. Coverage width. In our main experiments, we set K=16K=16 for the top-K coverage takeover and handback rules in the IPD framework. Under a smaller K, there will be more teacher takeovers and less frequent teacher handbacks, which leans more towards off-policy training, results in a higher teacher invervention ratio during the rollout stage, and requires more computational resources. While under a larger K, the teacher will invervene less, leaning more towards an on-policy training. To study the effect of different coverage widths, we further conduct experiments with K=4K=4, K=8K=8, and K=32K=32. As shown in Figure 7, IPD under K=4K=4 achieves the best performance with 30.4630.46 mean@8 and 45.8045.80 best@8. The default K=16K=16 appears to be a close runner-up with only −0.02-0.02 mean@8 and −0.05-0.05 best@8 gap, while getting much lower teacher intervention ratio, which provides a good balance between performance and computational efficiency. Figure 7: The mean@8 and best@8 average accuracy over benchmarks and the teacher intervention ratio with different coverage widths K during IPD training. A smaller K leads to more frequent teacher takeovers and less frequent teacher handbacks, resulting in a higher intervention ratio. Figure 8: Comparison of different distillation algorithms on DAPO-MATH-17K and DeepMath-GE8-17K. Intervention-based distillation algorithms including IPD and SKD demonstrate consistent robustness, while OPD apprears to be more sensitive to the change of training data. 4.4 Robustness on Different Training Data We further explore the robustness of IPD and other distillation algorithms under different training data. In addition to DAPO-MATH-17K, we select examples from DeepMath-103K (He et al., 2026) with difficulty levels greater than or equal to 8 and get 17,085 training examples, denoted as DeepMath-GE8-17K. As shown in Figure 8, we compare different algorithms on these two training datasets. Specifically, we find that algorithms with teacher intervention during rollout, including IPD and SKD, demonstrate a good resistance to the change of training data. Meanwhile, OPD shows a non-negligible mean@8 accuracy drop, which indicates that OPD is more sensitive to the training data. On the Qwen3-30B-A3B to Qwen3-4B-Base group, the mean@8 accuracy of OPD even gets a significant drop of −6.75-6.75. This indicates that without teacher intervention, OPD is more sensitive to the training data, and teacher intervention applied by IPD and SKD can alleviate this sensitivity, which requires less effort on data preparation and makes them more practical paradigms. 5 Conclusion and Discussion In this paper, we propose Interactive-Policy Distillation (IPD), which addresses the teacher unanchoring issue and improve the distillation performance through bidirectional takeover and handback during rollout, together with source-split supervision based on token provenance. This design dynamically corrects trajectories that leave the teacher’s acceptance region, and also provides a unified bridge between on-policy and off-policy distillations. Across diverse teacher-student pairs, IPD demonstrates better distillation performance and higher data efficiency, as well as better robustness to training data. We also design a fused inference engine to support our algorithm with efficient dual-model scheduling during the rollout stage. Currently, like most works in the OPD field, we only consider same-tokenization distillation here. Extending our method to cross-tokenization scenarios is an important problem. Besides, more different rules beyond top-K coverage can be further explored. On the infrastructure side, we only consider synchrnous scheduling for verification and proposal stages. Optimizations like asynchronous scheduling can further improve the throughput of dual-model rollout. We leave these directions for future exploration. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Vol. 2024, p. 21246–21263. Cited by: §B.3, §1, §2, §3.1. Anthropic (2026) Anthropic Introducing Claude Fable 5.1 and Claude Mythos 5.1. External Links: Link Cited by: §1. Armandpour et al. (2026) M. Armandpour, F. Ilhan, D. Harrison, A. Jaiswal, D. N. Hoang, F. Faghri, Y. Zhang, M. Cho, and M. Farajtabar Unmasking on-policy distillation: where it helps, where it hurts, and why. arXiv preprint arXiv:2605.10889. Cited by: §2. Arora et al. (2022) K. Arora, L. El Asri, H. Bahuleyan, and J. C. K. Cheung Why exposure bias matters: an imitation learning perspective of error accumulation in language generation. In Findings of the Association for Computational Linguistics: ACL 2022, p. 700–710. Cited by: §1, §2. Bengio et al. (2015) S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems 28. Cited by: §1, §2. Chen et al. (2023) C. Chen, S. Borgeaud, G. Irving, J. Lespiau, L. Sifre, and J. Jumper Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. Cited by: §2. Fu et al. (2026) Y. Fu, H. Huang, K. Jiang, J. Liu, Z. Jiang, Y. Zhu, and D. Zhao Revisiting on-policy distillation: empirical failure modes and simple fixes. In Third Conference on Language Modeling, Cited by: §B.1, §2, §3.1, §3.1. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang Minillm: knowledge distillation of large language models. In International Conference on Learning Representations, Vol. 2024, p. 32694–32717. Cited by: §1, §2, §3.1. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §2. He et al. (2025) B. He, Z. Qu, Z. Liu, Y. Chen, Y. Zuo, C. Qian, K. Zhang, W. Chen, C. Xiao, G. Cui, et al. Justrl: scaling a 1.5 b llm with a simple rl recipe. arXiv preprint arXiv:2512.16649. Cited by: Appendix D. He et al. (2026) Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, et al. Deepmath-103k: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. In International Conference on Learning Representations, Vol. 2026, p. 138306–138322. Cited by: §4.4. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: §1, §2, §3.1. Ho et al. (2023) N. Ho, L. Schmid, and S. Yun Large language models are reasoning teachers. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), p. 14852–14882. Cited by: §1. Hsieh et al. (2023) C. Hsieh, C. Li, C. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C. Lee, and T. Pfister Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the association for computational linguistics: ACL 2023, p. 8003–8017. Cited by: §1. Jin et al. (2026) W. Jin, T. Min, Y. Yang, D. Wei, Y. Zhou, S. R. Kadhe, N. Baracaldo, and K. Lee Entropy-aware on-policy distillation of language models. In Forty-third International Conference on Machine Learning, Cited by: §2. Kelly et al. (2019) M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer Hg-dagger: interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), p. 8077–8083. Cited by: §2. Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, p. 1317–1327. Cited by: §1, §2, §3.1. Ko et al. (2025) J. Ko, T. Chen, S. Kim, T. Ding, L. Liang, I. Zharkov, and S. Yun DistiLLM-2: a contrastive approach boosts the distillation of LLMs. In Forty-second International Conference on Machine Learning, Cited by: §2. Ko et al. (2024) J. Ko, S. Kim, T. Chen, and S. Yun DistiLLM: towards streamlined distillation for large language models. In International Conference on Machine Learning, p. 24872–24895. Cited by: §1, §1, §2. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, p. 611–626. Cited by: §3.2.3. Leviathan et al. (2023) Y. Leviathan, M. Kalman, and Y. Matias Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, p. 19274–19286. Cited by: §2, §3.2.1. Li et al. (2026) Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, and N. Ding Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. In ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning, Cited by: Appendix A, §B.2, Appendix D, §2, §3.1, §3.1, §3.1, §4.1, §4.2, §4.2. Li et al. (2025) Y. Li, X. Yue, Z. Xu, F. Jiang, L. Niu, B. Y. Lin, B. Ramasubramanian, and R. Poovendran Small models struggle to learn from strong reasoners. In Findings of the Association for Computational Linguistics: ACL 2025, p. 25366–25394. Cited by: §1. Lu and Lab (2025) K. Lu and T. M. Lab On-policy distillation. Thinking Machines Lab: Connectionism. Note: https://thinkingmachines.ai/blog/on-policy-distillation External Links: Document Cited by: §B.1, §1, §2, §4.2. Ma et al. (2026) W. Ma, J. Wei, L. Zhao, H. Zhang, B. Xiao, L. Li, Q. Yang, B. Gao, Y. Wang, R. Li, et al. Mopd: multi-teacher on-policy distillation for capability integration in llm post-training. arXiv preprint arXiv:2606.30406. Cited by: §1. Magister et al. (2023) L. C. Magister, J. Mallinson, J. Adamek, E. Malmi, and A. Severyn Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 1773–1781. Cited by: §1. Menda et al. (2019) K. Menda, K. Driggs-Campbell, and M. J. Kochenderfer Ensembledagger: a bayesian approach to safe imitation learning. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 5041–5048. Cited by: §2. Mirzadeh et al. (2020) S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, and H. Ghasemzadeh Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, p. 5191–5198. Cited by: §1. Oh et al. (2026) M. Oh, S. Song, G. Choi, Y. Choi, and Y. Jo KL for a KL: on-policy distillation with control variate baseline. In 3rd AI for Math Workshop: Toward Self-Evolving Scientific Agents, Cited by: §B.1, §B.1, §3.1, §3.1, §3.1. OpenAI (2026) OpenAI GPT-6 Astra: a new generation of intelligence. External Links: Link Cited by: §1. Ranzato et al. (2015) M. Ranzato, S. Chopra, M. Auli, and W. Zaremba Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732. Cited by: §1. Ross et al. (2011) S. Ross, G. Gordon, and D. Bagnell A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, p. 627–635. Cited by: §1. Schulman (2020) J. Schulman Approximating KL divergence. Note: http://joschu.net/blog/kl-approx.html Cited by: §B.1. Sheng et al. (2025) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: Appendix A, §3.2.3. Shing et al. (2025) M. Shing, K. Misaki, H. Bao, S. Yokoi, and T. Akiba TAID: temporally adaptive interpolated distillation for efficient knowledge transfer in language models. In The Thirteenth International Conference on Learning Representations, Cited by: §1. Xiao et al. (2026) B. Xiao, B. Xia, B. Yang, B. Gao, B. Shen, C. Zhang, C. He, C. Lou, F. Luo, G. Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780. Cited by: §2. Xing et al. (2026) X. Xing, H. Wang, B. Gao, Z. Li, and Y. Tang Trust region on-policy distillation. arXiv preprint arXiv:2606.01249. Cited by: §2. Xu et al. (2026) A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. Deepseek-v4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: §1. Xu et al. (2025) W. Xu, R. Han, Z. Wang, L. Le, D. Madeka, L. Li, W. Wang, R. Agarwal, C. Lee, and T. Pfister Speculative knowledge distillation: bridging the teacher-student gap through interleaved sampling. In International Conference on Learning Representations, Vol. 2025, p. 64616–64646. Cited by: §B.3, §1, §2, §3.2.1, §3.2.1, §4.2. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §1, §2, §4.1. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems 38, p. 113222–113244. Cited by: Appendix A, §3.2, §4.1. Z.ai (2026) Z.ai GLM-5.3: Frontier Coding with Emergent Cyber Capabilities. External Links: Link Cited by: §1. Appendix A Detailed Experimental Settings Data and models. We conduct our main experiments on DAPO-Math-17K (Yu et al., 2025) with four teacher–student pairs: Qwen3-4B / Qwen3-1.7B-Base, Qwen3-30B-A3B / Qwen3-1.7B-Base, Qwen3-8B / Qwen3-4B-Base, and Qwen3-30B-A3B / Qwen3-4B-Base. Thinking mode is disabled for all teacher models. Training configuration. We use the codebase from Li et al. (2026) for OPD, which is under the verl (Sheng et al., 2025) framework. On top of their codebase, we build our fused inference engine and implement SKD and IPD training pipelines. Specifically, as a special case of IPD, SKD also uses our fused inference engine for rollout. Following Li et al. (2026), unless otherwise specified, we train each model for one epoch (279 steps) with a learning rate of 1×10−61× 10^-6. The maximum rollout sequence length is 8192, comprising a maximum prompt length of 1024 and a maximum generation length of 7168. Each prompt produces four training rollouts sampled with temperature 1.0 and top-p=1.0p=1.0. All experiments are conducted on two compute nodes, each equipped with 4 NVIDIA H100 SXM 80GB GPUs. Distillation configuration. All methods use top-K distillation with K=16K=16. Each KL term is computed after separately renormalizing the student and teacher distributions over the union support St=Stπ∪StqS_t=S_t^π∪ S_t^q, as defined in Eq. (6). IPD and SKD both use K=16K=16 for the takeover coverage rule and a proposal length of γ=5γ=5. IPD additionally applies the symmetric top-K coverage rule for handback. Evaluation. We evaluate the final checkpoint of every run on MATH-500, Minerva, Olympiad-Bench, AMC23, AIME24, and AIME25. For each problem, we independently sample eight responses with temperature 0.7 and top-p=0.95p=0.95. Same as training, we use a maximum generation length of 7168. We report mean@8 and best@8 accuracy for each benchmark and their benchmark-wise averages. Appendix B Further Discussions B.1 Sampled-Token OPD Since the sampled token follows xt∼πθ(⋅∣ct)x_t _θ(· c_t), taking the expectation of Eq. (4) over xtx_t recovers the full-vocabulary reverse KL in Eq. (3), xt∼πθ(⋅∣ct)[ℓ^tsample]=∑v∈πθ(v∣ct)logπθ(v∣ct)q(v∣ct)=ℓtfull.E_x_t _θ(· c_t) [ ^sample_t ]= _v _θ(v c_t) _θ(v c_t)q(v c_t)= ^full_t. (12) According to Schulman (2020), ℓ^tsample ^sample_t is therefore the k1k_1 estimator of the reverse KL, which is unbiased even though its individual realizations can be negative while ℓtfull≥0 ^full_t≥ 0 is guaranteed. Given the identity ∇θπθ(v∣ct)=πθ(v∣ct)∇θlogπθ(v∣ct) _θ _θ(v c_t)= _θ(v c_t) _θ _θ(v c_t), the product rule applied to Eq. (3) gives ∇θℓtfull _θ ^full_t =∑v∈∇θπθ(v∣ct)logπθ(v∣ct)q(v∣ct)+∑v∈πθ(v∣ct)∇θlogπθ(v∣ct)q(v∣ct) = _v _θ _θ(v c_t)\, _θ(v c_t)q(v c_t)+ _v _θ(v c_t) _θ _θ(v c_t)q(v c_t) (13) =∑v∈∇θπθ(v∣ct)logπθ(v∣ct)q(v∣ct)+∑v∈∇θπθ(v∣ct) = _v _θ _θ(v c_t)\, _θ(v c_t)q(v c_t)+ _v _θ _θ(v c_t) =∑v∈∇θπθ(v∣ct)logπθ(v∣ct)q(v∣ct)+∇θ∑v∈πθ(v∣ct) = _v _θ _θ(v c_t)\, _θ(v c_t)q(v c_t)+ _θ _v _θ(v c_t) =∑v∈πθ(v∣ct)logπθ(v∣ct)q(v∣ct)∇θlogπθ(v∣ct)+∇θ1 = _v _θ(v c_t) _θ(v c_t)q(v c_t) _θ _θ(v c_t)+ _θ1 =xt∼πθ(⋅∣ct)[ℓ^samplet∇θlogπθ(xt∣ct)]. =E_x_t _θ(· c_t) [ ^sample_t\, _θ _θ(x_t c_t) ]. We denote the gradient estimator as g^t=ℓ^tsample∇θlogπθ(xt∣ct). g_t= ^sample_t\, _θ _θ(x_t c_t). (14) Under the context of gradient ascent in reinforcement learning, −g^t- g_t is the policy gradient obtained when OPD is implemented as maximizing t=xt∼πθ(⋅∣ct)[sg[−ℓ^samplet]], _t=E_x_t _θ(· c_t) [sg [- ^sample_t ] ], (15) where the per-token reward is the stop-gradient operated −ℓ^tsample=logq(xt∣ct)−logπθ(xt∣ct)- ^sample_t= q(x_t c_t)- _θ(x_t c_t) (Lu and Lab, 2025; Oh et al., 2026). Given a prefix ctc_t, variances of the sampled-token loss and gradient are Var[ℓ^tsample∣ct] [ ^sample_t c_t ] =∑v∈πθ(v∣ct)(logπθ(v∣ct)q(v∣ct))2−(ℓtfull)2 = _v _θ(v c_t) ( _θ(v c_t)q(v c_t) )^2- ( ^full_t )^2 =∑v∈πθ(v∣ct)(logπθ(v∣ct)q(v∣ct)−ℓtfull)2, = _v _θ(v c_t) ( _θ(v c_t)q(v c_t)- ^full_t )^2, (16) Var[g^t∣ct] [ g_t c_t ] =[‖g^t−∇θℓtfull‖2∣ct] =E [\| g_t- _θ ^full_t\|^2 c_t ] =[(ℓ^tsample)2‖∇θlogπθ(xt∣ct)‖2∣ct]−‖∇θℓtfull‖2 =E [( ^sample_t)^2\| _θ _θ(x_t c_t)\|^2 c_t ]-\| _θ ^full_t\|^2 =∑v∈πθ(v∣ct)‖logπθ(v∣ct)q(v∣ct)∇θlogπθ(v∣ct)−∇θℓtfull‖2. = _v _θ(v c_t) \| _θ(v c_t)q(v c_t) _θ _θ(v c_t)- _θ ^full_t \|^2. (17) Eq. (16) and Eq. (17) reveal a common source of variance in sampled-token OPD. When the teacher and student disagree strongly, the student may assign much higher probability to tokens with q(v∣ct)≪πθ(v∣ct)q(v c_t) _θ(v c_t), yielding much larger log-ratios than on tokens where the models agree (Fu et al., 2026). Sampling among these tokens makes the loss and gradient fluctuate, leading to noisy and potentially unstable updates (Oh et al., 2026). B.2 Distillation Support According to the GitHub issue discussion from the official repository of Li et al. (2026), different choices of support (including StπS_t^π, StqS_t^q, Stπ∪StqS_t^π∪ S_t^q, and Stπ∩StqS_t^π∩ S_t^q) do not bring significant differences for same-size distillation. From our observation, for different-size distillation, using the union of the two top-K sets Stπ∪StqS_t^π∪ S_t^q as the support for top-K OPD is slightly better than using others. Unless specified, we use the union of the two top-K sets by default in this paper. B.3 Loss Aggregation Following GKD (Agarwal et al., 2024) and SKD (Xu et al., 2025), Eq. (11) uses a sequence-mean-token-mean aggregation, which assigns equal weight to each response and prevents longer reasoning traces from dominating the minibatch objective. Another aggregation commonly used in OPD implementations is the global token mean, which assigns equal weight to every valid token and consequently weights each rollout in proportion to its length. This aggregation choice is orthogonal to the rollout policy, the token source, and the direction of the source-split KL. From our practice, global token-mean aggregation and sequence-mean-token-mean aggregation produce comparable performances, while the latter yields slightly more stable optimization. We therefore use sequence-mean-token-mean aggregation by default in this paper. B.4 Student Models In this paper, we use Qwen3 Base models as student for all the experiments. Non-base Qwen3 models are trained to switch between thinking mode and non-thinking mode within single model, and this switch is contorlled by special ⟨think⟩ and ⟨/think⟩ /think tokens. Specifically, when disabling thinking mode, the tokenizer will append an empty ⟨think⟩⟨/think⟩ /think block to the prompt before decoding. If using non-base Qwen3 models with thinking mode disabledas student, it can be difficult to distinguish whether the student performance gain is from successfully aligning with the teacher, or it just somehow manages to bypass the template-controlled thinking mode switch and silently activates its thinking capability obtained from previous training. Therefore, if using non-base Qwen3 models with thinking mode disabled as teacher, we think using Qwen3 Base models as student is a cleaner setting for distillation experiments. Another clean setting is to use non-base Qwen3 models with thinking mode enabled as both teacher and student, but this setting would require a much longer response length and consume more computational resources. We leave this setting for future exploration. Appendix C Additional Experimental Results C.1 Additional Main Results In addition to the main results in Table 1, we also report the mean@8 accuracy and top-K overlap ratio curves during training, as shown in Figure 9. Figure 9: The mean@8 accuracy and top-K overlap ratio curve for four groups of models. C.2 Additional Ablation Results KL direction. Table 2 shows that the default source-split KL obtains the highest benchmark-averaged mean@8 accuracy (30.4430.44), while pure reverse KL has a slightly higher best@8 (45.8745.87) than default (45.7545.75). Table 2: Benchmark-wise mean@8 and best@8 accuracy for the KL-direction ablation. MATH-500 Minerva Olympiad AMC23 AIME24 AIME25 Avg. Loss Variant mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 Pure Reverse KL 68.23 84.05 29.18 46.21 32.13 49.70 35.39 56.46 10.42 27.24 4.17 11.56 29.92 45.87 Pure Forward KL 66.80 83.64 28.77 45.15 30.46 49.69 34.19 58.58 6.67 18.41 4.17 11.60 28.51 44.51 Inverse-IPD-Split KL 67.35 84.84 27.16 44.06 31.37 50.31 33.58 56.63 9.17 22.66 6.67 15.09 29.22 45.60 IPD-Split KL (default) 69.28 84.59 29.60 46.26 31.80 49.63 36.14 57.25 9.17 21.69 6.67 15.06 30.44 45.75 Coverage width. Table 3 shows that K=4K=4 yields the highest average mean@8 and best@8, but only slightly exceeds the default K=16K=16 by +0.02+0.02 and +0.05+0.05, respectively. Table 3: Benchmark-wise mean@8 and best@8 accuracy for the top-K coverage ablation. MATH-500 Minerva Olympiad AMC23 AIME24 AIME25 Avg. Coverage width mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 K=4K=4 68.30 85.36 28.49 43.59 31.97 49.76 35.24 53.75 10.00 24.21 8.75 18.17 30.46 45.80 K=8K=8 69.20 84.94 28.95 45.00 32.45 50.97 33.43 54.60 8.75 17.68 6.25 14.10 29.84 44.55 K=16K=16 (default) 69.28 84.59 29.60 46.26 31.80 49.63 36.14 57.25 9.17 21.69 6.67 15.06 30.44 45.75 K=32K=32 68.03 83.81 28.77 44.66 32.50 50.50 34.64 53.96 8.75 17.58 5.42 14.30 29.68 44.13 Appendix D Same-origin Distillation In addition to different models from the Qwen3 family, we also conduct a same-origin distillation experiment with DeepSeek-R1-Distill-Qwen-1.5B as the student and JustRL-DeepSeek-1.5B as the teacher. Specifically, JustRL-DeepSeek-1.5B is the further post-trained version of DeepSeek-R1-Distill-Qwen-1.5B via simple GRPO recipe (He et al., 2025). The results are shown in Table 4 and Figure 10. We find that intervention methods including IPD and SKD do not bring as statistically significant improvement over OPD compared with our main experiments with different-size Qwen3 models. From Figure 10, we can observe that the mean@8 accuracy and top-K overlap ratio curves of three methods are very close to each other’s, which is expected as the student and teacher are same-origin models. The same-origin experiment indicates that OPD can already achieve a good distillation performance when the reasoning gap between the student and teacher are not significant, which is consistent with the discussion from Li et al. (2026), and the main advantage of IPD remains in large-gap distillation scenarios. Table 4: Same-origin distillation experiment with DeepSeek-R1-Distill-Qwen-1.5B as the student and JustRL-DeepSeek-1.5B as the teacher on DAPO-Math-17K-Processed. Intervention methods including IPD and SKD do not bring as statistically significant improvement over OPD compared with our main experiments with different-size Qwen3 models. Teacher/ Student Method MATH-500 Minerva Olympiad AMC23 AIME24 AIME25 Avg. mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 mean@8 best@8 JustRL-DeepSeek-1.5B/ DeepSeek-R1-Distill- Qwen-1.5B OPD 85.15 92.79 38.01 49.41 47.29 60.63 64.16 82.92 27.50 52.85 20.83 30.59 47.16 61.53 SKD 85.55 93.36 37.13 47.80 48.74 61.70 63.55 79.67 27.92 50.43 23.33 32.58 47.70 60.92 IPD (ours) 85.75 93.29 37.78 49.99 48.11 61.36 62.80 79.53 30.42 47.13 22.92 29.80 47.96 60.18 Figure 10: The mean@8 accuracy and top-K overlap ratio curve for same-origin distillation experiment with DeepSeek-R1-Distill-Qwen-1.5B as the student and JustRL-1.5B as the teacher. Appendix E Distillation on Limited Trainig Examples We also run distillation trainings on limited training examples to evaluate the impact of data scarcity on the distillation process. Specifically, we randomly select 1,024 training examples from DAPO-MATH-17K and conduct distillation experiments using Qwen3-4B as teacher and Qwen3-1.7B-Base as student. The student model is trained for 15 epochs (with 240 steps in total), and the other training configurations remains the same as the main experiments. The results are shown in Figure 11. We find that IPD can still outperform OPD and SKD and shows a higher convergence under limited training data. Another interesting observation is that, even if the training examples only count for about 1/17 of the original training data, all the three methods can matches the performance of the student model trained on the full training dataset. The limited training data experiment indicates that although with limited unique training prompts, as long as they can be replayed to generate diverse rollouts, the distillation can still be effective. Figure 11: The mean@8 accuracy and top-K overlap ratio curve for distillation on 1,024 randomly selected DAPO-MATH-17K training examples. As long as the limited unique training prompts can be replayed to generate diverse rollouts, the distillation can still be effective.