Paper deep dive
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
Zhenyu Wang, Tianze Wang, Linjun Zhang, Yifan Hu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OPD treats all teacher signals equally, assuming that the teacher's supervision is equally important for every token. However, teacher signals at different tokens may have very different effects on the student's performance: some correct important reasoning errors, while others have little effect on the final answer. Motivated by this observation, we introduce Dr. OPD (OPD Done Right), which defines the optimal weighted OPD to maximize the student's performance. We formulate Dr. OPD as a bilevel optimization problem in which the student learns from weighted teacher supervision, while the weights are selected to maximize the expected reward of the resulting student. To solve Dr. OPD, we develop an efficient iterative solver that updates the token weights and student policy alternatively. At each round, it updates weights in closed form and then takes one gradient step on the resulting weighted OPD objective. Under regularity conditions, we show that this weighted update achieves a higher expected reward than a vanilla OPD update. Empirically, across strong-to-weak and same-size distillation on math and code, Dr. OPD consistently outperforms all evaluated baselines. In particular, in the strong-to-weak distillation setting, Dr. OPD improves average math performance by $9.7$ points over vanilla OPD, and enables the smaller student to surpass its larger teacher.
Tags
Links
- Source: https://arxiv.org/abs/2609.38025v1
- Canonical: https://arxiv.org/abs/2609.38025v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
79,010 characters extracted from source content.
Expand or collapse full text
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models Zhenyu Wang †thanks: Equal contribution. Tianze Wang11footnotemark: 1 Linjun Zhang †thanks: Co-corresponding authors. Yifan Hu22footnotemark: 2 Affiliation: Department of Statistics, Rutgers University Email: zw425,tw522@stat.rutgers.edu Email: zlj11112222@gmail.com yifan.hu@rutgers.edu Abstract On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OPD treats all teacher signals equally, assuming that the teacher’s supervision is equally important for every token. However, teacher signals at different tokens may have very different effects on the student’s performance: some correct important reasoning errors, while others have little effect on the final answer. Motivated by this observation, we introduce Dr. OPD (OPD Done Right), which defines the optimal weighted OPD to maximize the student’s performance. We formulate Dr. OPD as a bilevel optimization problem in which the student learns from weighted teacher supervision, while the weights are selected to maximize the expected reward of the resulting student. To solve Dr. OPD, we develop an efficient iterative solver that updates the token weights and student policy alternatively. At each round, it updates weights in closed form and then takes one gradient step on the resulting weighted OPD objective. Under regularity conditions, we show that this weighted update achieves a higher expected reward than a vanilla OPD update. Empirically, across strong-to-weak and same-size distillation on math and code, Dr. OPD consistently outperforms all evaluated baselines. In particular, in the strong-to-weak distillation setting, Dr. OPD improves average math performance by 9.79.7 points over vanilla OPD, and enables the smaller student to surpass its larger teacher. †footnotetext: Code: https://github.com/zywang0701/Dr-OPD. 1 Introduction On-policy distillation (OPD) has become an important post-training approach for transferring capabilities from a stronger teacher to a student model that is cheaper to deploy or has undergone less training (Agarwal et al., 2024; Xu et al., 2026; Team et al., 2026). Unlike outcome-reward reinforcement learning methods like RLOO and GRPO (Ahmadian et al., 2024; Shao et al., 2024), which update the model based on a scalar reward for an entire generated trajectory, OPD leverages the teacher model to provide dense, token-level supervision along with the student’s generated responses (Gu et al., 2024). Specifically, OPD trains the student by minimizing the divergence between the next-token distributions of the student and the teacher at each token visited by the student. Therefore, it provides a learning signal at every generated token rather than only at the end of a response. Vanilla OPD assigns equal importance to the teacher’s supervisions across tokens. However, given a long response generated by the student, should every teacher signal be treated equally? Consider a student solving a math problem. The teacher could either correct a crucial reasoning step or simply prefer different wording under the same mathematical meaning. These two signals are supposed to enter the distillation loss with different weights, as their effects on the final answer can be very different. Indeed, recent work has likewise shown that not every signal from the teacher across each token is equally useful to the student (Armandpour et al., 2026; Xie et al., 2026; Xing et al., 2026). To study how the student should trust teacher signals across tokens, we revisit the goal of OPD. In particular, note that the purpose of distillation is ultimately to improve the student’s task performance. Minimizing the divergence towards the stronger teacher policy is only an approach used to achieve this goal. Indeed, the teacher signal only tells us what the teacher prefers at each token, but not whether following the preference improves the student’s performance. This motivates us to adaptively choose the weight of each teacher signal according to its usefulness for the student. In this paper, we therefore study how to weight token-level teacher signals so that the student achieves the best performance. Following this principle, we introduce Dr. OPD (OPD Done Right), which assigns weights w(s,v)w(s,v) to each teacher signal on the tokens v and context s=(x,y)s=(x,y) according to its usefulness for improving the student, as illustrated in Figure 1. A weight wt>1w_t>1 places more trust in the teacher at token t, while 0<wt<10<w_t<1 places less trust. Vanilla OPD sets equal weights of wt≡1w_t≡ 1 across all tokens. We note that for any positive weight w, the corresponding w-weighted OPD remains a valid distillation objective that pushes the student toward the teacher policy, just as vanilla OPD does. Dr. OPD then selects, among these valid weights, the one that gives the best resulting student: maxwStudent’s performance after training with w-weighted OPD. _w \ Student's performance after training with $w$-weighted OPD \. (1) The performance is measured by an outcome reward based on its generated responses (Ouyang et al., 2022; Shao et al., 2024). We formally formulate Dr. OPD as a bilevel optimization in Section 3. Figure 1: Illustration of Dr. OPD. Left: vanilla OPD assigns the same weight to every teacher signal, while Dr. OPD adapts the weights according to their benefit to the student. Right: student rollouts receive teacher and outcome-reward signals, whose gradient alignment determines token credits. These credits are converted into weights for distillation, where λ controls the weighting strength. Solving (1) exactly would require repeatedly training the student under different weight functions and then selecting the best weight, which is computationally infeasible. We therefore develop an iterative solver, illustrated in Figure 1. At each round, we build a surrogate objective of (1) to update the weights, leading to a simple closed-form update instead of learning a complex weight function. Then the student performs a weighted OPD update. Specifically, the student generates its own responses, from which we obtain the teacher’s token-level supervision and the outcome reward signal. We use these two signals to assess whether following the teacher more closely at each token would improve the student’s performance. Under regularity conditions, we show that the student, after the weighted update, achieves a higher reward than the corresponding vanilla OPD update. To summarize, we make the following contributions: • Bilevel Formulation of Distillation. We introduce the concept of learning useful signals from the teacher via weighted OPD and propose Dr. OPD, a bilevel optimization formulation that defines the optimal weighted OPD to maximize the student’s performance. The lower-level problem trains the student under a given weight function, while the upper-level problem selects the weights according to the performance of the resulting student. • Efficient Algorithm. Despite the challenge of solving Dr. OPD directly, we develop an efficient iterative solver with a closed-form weight update followed by a gradient update of the weighted OPD. To incorporate outcome rewards into the weight updates, it requires computing an inner product between two parameter-dimensional gradients for every token, which is prohibitively expensive for large models. We instead express each inner product as the product of two scalars, and then these products for all tokens can be computed altogether using a single Jacobian–vector product (JVP); see Section 4 for details. • Significant Empirical Gains. We evaluate Dr. OPD across strong-to-weak and same-size distillation on math and code tasks, where it achieves the best performance across all evaluated settings and consistently outperforms all baselines. As shown in Figure 2, Dr. OPD improves Qwen3-1.7B over vanilla OPD by +9.7+9.7 points on math and even surpasses the larger Qwen3-4B teacher (35.0 vs. 34.1). In same-size distillation, it also surpasses both RL-trained teachers. Figure 2: Empirical gains. Across both distillation setups, Dr. OPD consistently outperforms existing baselines and even produces a student that surpasses its teacher (Full results in Section 5.2). 2 Weighted On-Policy Distillation OPD trains a student model on its generated responses under the supervision from a stronger frozen teacher model. Let x be a prompt, and y=(y1,…,yT)y=(y_1,…,y_T) a response generated by the student. At step t, the context st=(x,y<t)s_t=(x,y_<t) contains the prompt and all the preceding tokens generated before t. We denote pθ(v∣st)p_θ(v s_t) as the student’s probability of generating the next token v∈v , and p∗(v∣st)p^*(v s_t) as the teacher’s probability in the same context, where V is the vocabulary of tokens. Vanilla OPD aims to train the student policy to mimic the teacher policy using the reverse KL divergence: minθD(θ):=[∑tKL(pθ(⋅∣st)∥p∗(⋅∣st))], _θD(θ):=E [ _t (p_θ(· s_t)\,\|\,p^*(· s_t) ) ], (2) where KL(pθ(⋅∣s)∥p∗(⋅∣s))=∑vpθ(v∣s)logpθ(v∣s)p∗(v∣s), (p_θ(· s)\,\|\,p^*(· s) )= _vp_θ(v s) p_θ(v s)p^*(v s), and the expectation is taken over the prompts and responses generated by the student policy. Following prior works (Agarwal et al., 2024; Shao et al., 2026), the generated contexts are treated as fixed during each student update and are refreshed as the student changes. As shown in (2), vanilla OPD assigns equal weights to teacher signals at every token. However, not all teacher signals are equally informative (Armandpour et al., 2026; Xie et al., 2026). We introduce a weight function w(s,v)>0w(s,v)>0, which controls the importance of matching the teacher’s probability of token v at context s. For a fixed context s, we use w(s,⋅)w(s,·) to denote the corresponding vector of token weights. We then define a weighted KL divergence that generalizes the ordinary KL divergence. Definition 1 (Weighted KL divergence). At a context s, given the positive weights w(s,⋅)w(s,·), we define KLw(s,⋅)(pθ(⋅∣s)∥p∗(⋅∣s)) _w(s,·)\! (p_θ(· s)\,\|\,p^*(· s) ) :=∑vw(s,v)[pθ(v∣s)logpθ(v∣s)p∗(v∣s)−pθ(v∣s)+p∗(v∣s)]. := _vw(s,v) [p_θ(v s) p_θ(v s)p^*(v s)-p_θ(v s)+p^*(v s) ]. (3) where the additional term −pθ(v∣s)+p∗(v∣s)-p_θ(v s)+p^*(v s) ensures nonnegativity under token-dependent weighting, following the generalized KL divergence (also known as I-divergence) with nonnegative measures (Csiszár, 1975; Banerjee et al., 2005). This weighted KL divergence equals zero if and only if pθ(v∣s)=p∗(v∣s)p_θ(v s)=p^*(v s) for every token v. Moreover, when w(s,v)≡1w(s,v)≡ 1 for all v, it holds ∑vw(s,v)(p∗(v∣s)−pθ(v∣s))=0 _vw(s,v) (p^*(v s)-p_θ(v s) )=0, so (3) reduces to the ordinary KL divergence. With the weighted KL divergence, we define the weighted OPD as follows. Definition 2 (Weighted OPD). Given a weight function w(s,⋅)>0w(s,·)>0 for every context s, we define weighted OPD by minθDw(θ):=[∑tKLw(st,⋅)(pθ(⋅∣st)∥p∗(⋅∣st))]. _θD_w(θ):=E\! [ _t _w(s_t,·)\! (p_θ(· s_t)\,\|\,p^*(· s_t) ) ]. (4) We next show that this remains a valid distillation objective. In particular, minimizing Dw(θ)D_w(θ), for any positive w, still pushes the student to match the teacher. Theorem 3 (Validity of weighted OPD). For any positive weight function w, we have Dw(θ)≥0.D_w(θ)≥ 0. Equality holds if and only if pθ(v∣s)=p∗(v∣s)p_θ(v s)=p^*(v s) for every token v at almost every context under the reference context distribution. When w(s,v)≡1w(s,v)≡ 1, weighted OPD in (4) reduces to vanilla OPD. 3 OPD Done Right: Find the Best Weight for OPD We now study how to choose the weight function w. Recall that the ultimate goal of distillation is to improve the student’s task performance, while minimizing the divergence to a stronger teacher is merely a means to achieve this goal. This principle motivates us to select the weight function according to the performance of the resulting student, formulating a bilevel optimization problem. 3.1 Formulation of Dr. OPD We evaluate the student’s performance using an outcome reward V(x,y)∈[0,1]V(x,y)∈[0,1] for each prompt–response pair (x,y)(x,y). The student’s expected reward is then R(θ):=x,y∼pθ(⋅∣x)[V(x,y)].R(θ):=E_x,\;y p_θ(· x) [V(x,y) ]. We now introduce DR. OPD (OPD done right), the optimal weighted OPD to maximize the student’s performance: maxwmin≤w≤wmaxR(θ∗(w))subject toθ∗(w)∈argminθDw(θ). _w_ ≤ w≤ w_ R\! (θ^*(w) ) to θ^*(w)∈ _θD_w(θ). (5) Dr. OPD is a bilevel optimization problem, where the upper level chooses the weight function w to maximize the resulting student’s reward, and the lower level trains the student θ by minimizing Dw(θ)D_w(θ), under the given w. Here, the constraint w∈[wmin,wmax]w∈[w_ ,w_ ] means pointwise that w(s,v)∈[wmin,wmax]w(s,v)∈[w_ ,w_ ] for every (s,v)(s,v), where 0<wmin<1<wmax0<w_ <1<w_ are prespecified bounds. For simplicity, we write θ∗(w)θ^*(w) as if the lower-level solution were unique. When multiple lower-level solutions exist, we choose the one that maximizes the upper-level reward; see the formal definition in Appendix C.3. 3.2 Iterative Solver for Dr. OPD Solving the bilevel Dr. OPD problem in (5) exactly is computationally intractable, since evaluating each candidate weight function w would require fully optimizing the lower-level student and then evaluating its reward. In addition, the weight function w(s,v)w(s,v) is defined over all possible context–token pairs, making the upper-level optimization infinite-dimensional. We therefore design an iterative solver that alternates between student and weight updates. At each round, starting from the current student θ¯ θ, we take one gradient update of weighted OPD with the weights fixed. We then update the weights according to the reward of the resulting updated student, restricting the weights update to the sampled tokens in responses. This reduces the original infinite-dimensional functional optimization into a finite-dimensional problem at each step. Lower-level update. For a fixed weight function w, we take one gradient descent step from the current student θ¯ θ. We generate a response y=(y1,…,yT)y=(y_1,…,y_T) from pθ¯p_ θ and hold the sampled trajectory fixed during this round. Here, yty_t is the realized token at step t, whereas the previous v∈v in (2) indexes all possible tokens. We write wt:=w(st,yt),pt(θ¯):=pθ¯(yt∣st),pt∗:=p∗(yt∣st).w_t:=w(s_t,y_t), p_t( θ):=p_ θ(y_t s_t), p_t^*:=p^*(y_t s_t). Differentiating the weighted OPD objective at θ¯ θ gives −∇θDw(θ¯)=[∑twtgt],wheregt:=logpt∗pt(θ¯)∇θlogpθ(yt∣st)|θ=θ¯.- _θD_w( θ)=E\! [ _tw_tg_t ], where g_t:= p_t^*p_t( θ) . _θ p_θ(y_t s_t) |_θ= θ. (6) The derivation is provided in Appendix C.2.2. The corresponding one-step update on the lower-level solution is therefore θ+(w):=θ¯−η∇Dw(θ¯)=θ¯+η[∑twtgt],θ^+(w):= θ-η∇ D_w( θ)= θ+η\,E\! [ _tw_tg_t ], (7) where η>0η>0 is the student step size. The expectation in (6)–(7) is taken over the current policy pθ¯p_ θ. We call gtg_t in (6) the teacher correction at token yty_t, which is interpreted as follows. If the teacher assigns token yty_t a higher probability than the student, then gtg_t increases the student’s probability of generating yty_t. On the other hand, if the teacher assigns it a lower probability, then gtg_t decreases its probability. The weight wtw_t controls the strength of this teacher correction to the student update. Upper-level update. We next update the weight function w according to the reward of the updated student θ+(w)θ^+(w). Since the weight function is defined over all possible context–token pairs and we do not impose a parametric form on it, a gradient update like in (7) does not apply here. Instead, we construct a tractable lower bound approximation on R(θ+(w))R(θ^+(w)) and optimize this bound over the weight function. As shown below, this leads to a simple pointwise update for the weights. We denote the one-step vanilla OPD update from the current student θ¯ θ as follows, which sets w≡1w≡ 1, θvani+:=θ+(1). _vani^+:=θ^+(1). To determine how the weights should be updated, we ask how changing the weight wtw_t of the teacher signal at token t affects the resulting student’s reward. We define the credit at token t as ct:=1η∂R(θ+(w))∂wt|w≡1=gt⊤∇R(θvani+),c_t:= 1η . ∂ R\! (θ^+(w) )∂ w_t |_w≡ 1=g_t ∇ R\! ( _vani^+ ), (8) where the second equality follows from the chain rule and (7). The derivative is understood per unit reference measure; Appendix C.4 gives its precise functional interpretation. The credit ctc_t measures whether following the teacher more strongly at token t would locally improve the student’s reward. Recall that the teacher correction gtg_t in (6) locally moves the sampled token’s probability toward the teacher’s. However, moving closer to the teacher does not necessarily improve the student’s task performance. The credit ctc_t captures this distinction. If ct>0c_t>0, placing more weight on the teacher correction is beneficial to the student’s reward, suggesting that we should trust this signal more and choose wt>1w_t>1. On the other hand, if ct<0c_t<0, following the teacher more strongly is harmful, suggesting that the signal should be downweighted with wt<1w_t<1. Thus, gtg_t determines the direction of the token-level teacher correction, while ctc_t determines how strongly the student should follow the teacher signal. We next derive a lower bound surrogate on the reward R(θ+(w))R(θ^+(w)). Proposition 4 (Lower bound on reward). Suppose that the reward gradient ∇R(θ)∇ R(θ) is locally LRL_R-Lipschitz in θ and that the regularity conditions in Appendix C.1 hold. Then, for any weight function w∈[wmin,wmax]w∈[w_ ,w_ ] and any λ>0λ>0 satisfying ληLR[∑t‖gt‖2]≤1.λη L_RE\! [ _t\|g_t\|^2 ]≤ 1. R(θ+(w))≥R(θvani+)+η[∑t((wt−1)ct−(wt−1)22λ)].R\! (θ^+(w) )≥ R\! ( _vani^+ )+η\,E\! [ _t ((w_t-1)c_t- (w_t-1)^22λ ) ]. (9) The right-hand side provides a tractable lower-bound surrogate for R(θ+(w))R(θ^+(w)). The parameter λ>0λ>0 is introduced to simplify the lower bound; see Appendix C.1 for details. Maximizing this surrogate with respect to w yields a closed-form update wt+=clip(1+λct,wmin,wmax).w_t^+=clip (1+λ c_t,\,w_ ,\,w_ ). (10) The update implies that λ can also be interpreted as the hyperparameter to control the weighting strength, with larger values allowing stronger adjustments to token-level weights. Importantly, this closed-form update removes the need to parameterize or train a separate model for the weight function. Instead, we directly compute the weight wt+w_t^+ for each sampled token, which substantially reduces the computational cost. Note that the updated w+w^+ attains a higher reward than vanilla OPD, as long as wt+≢1w_t^+ ≡ 1. Theorem 5 (Reward improvement over vanilla OPD). Under the conditions ensuring (9), R(θ+(w+))−R(θvani+)≥η2λ[∑t(wt+−1)2]≥0.R\! (θ^+(w^+) )-R\! ( _vani^+ )≥ η2λE\! [ _t(w_t^+-1)^2 ]≥ 0. (11) 4 Algorithm In this section, we propose a practical algorithm to solve Dr. OPD. At each round, starting from the current student θ¯ θ, we sample multiple responses per prompt and obtain the teacher signals and outcome-rewards. Such outcome rewards are readily available for verifiable tasks such as math and code through answer checking and test execution (Guo et al., 2025), and can also be provided by learned reward models for more general tasks (Ouyang et al., 2022). Given the signals from both the teacher and the outcome reward, we estimate the token credits ctc_t in (8) and update the weights as in (10). Lastly, we update the student with one gradient step on the resulting weighted OPD. Computation of credits. Recall that ct=gt⊤∇R(θvani+),c_t=g_t ∇ R( _vani^+), where the teacher correction gtg_t is directly computed as in (6) but estimating ∇R(θvani+)∇ R( _vani^+) can be computationally expensive. In particular, estimating ∇R(θvani+)∇ R( _vani^+) would require first applying a vanilla OPD update from the current θ¯ θ to obtain θvani+ _vani^+, then generating a new set of responses from this updated student and using them to estimate its gradient of reward. This would introduce an additional round of model update and response generation at every training iteration, substantially increasing the computational cost. To avoid the cost, we approximate ∇R(θvani+)∇ R( _vani^+) by ∇R(θ¯)∇ R( θ) evaluated at the current student θ¯ θ, which can be estimated using the responses already sampled. For a general outcome reward function, ∇R(θ¯)∇ R( θ) can be estimated by weighting each response’s log-probability gradient by its reward. For the verifiable math and code tasks, we can follow GRPO (Shao et al., 2024) to estimate ∇R(θ¯)∇ R( θ) by r^:=1|ℬ|∑(x,y)∈ℬA(x,y)∑t∇θlogpθ(yt∣st)|θ=θ¯, r:= 1|B| _(x,y) A(x,y) _t . _θ p_θ(y_t s_t) |_θ= θ, where A(x,y)A(x,y) denotes the group-relative advantage with the explicit expression in Appendix B, and ℬB denotes the batch of prompt-response pairs. We then approximate the token credit by gt⊤r^g_t r. Efficient computation with Jacobian–vector product (JVP). For a response with T tokens, we shall compute credits c^t=gt⊤r c_t=g_t r for each token t=1,…,Tt=1,…,T. A naive implementation would require computing T parameter gradients g1,…,gT∈ℝ|θ¯|,g_1,…,g_T ^| θ|, one for each token. Since the number of model parameters |θ¯|| θ| is very large, explicitly forming all gtt=1T\g_t\_t=1^T is prohibitively expensive. It turns out that we do not need to explicitly compute or store the parameter gradient gtg_t. Recall that gt=dt∇θlogpθ(yt∣st)|θ=θ¯,dt:=logp∗(yt∣st)pθ¯(yt∣st)∈ℝ.g_t=d_t . _θ p_θ(y_t s_t) |_θ= θ, d_t:= p^*(y_t s_t)p_ θ(y_t s_t) . (12) Substituting this expression into c^t=gt⊤r c_t=g_t r gives, for each token t=1,…,Tt=1,…,T, c^t=dt⟨∇θlogpθ(yt∣st)|θ=θ¯,r^⟩=dtdϵlogpθ¯+ϵr^(yt∣st)|ϵ=0, c_t=d_t . _θ p_θ(y_t s_t) |_θ= θ, r =d_t . ddε p_ θ+ε r(y_t s_t) |_ε=0, (13) where the second equality follows from the directional-derivative identity. Thus, instead of explicitly constructing the T parameter-dimensional gradients gtt=1T\g_t\_t=1^T, we only need the T scalar directional derivatives dϵlogpθ¯+ϵr^(yt∣st)|ϵ=0t=1T \ . ddε p_ θ+ε r(y_t s_t) |_ε=0 \_t=1^T, which can be computed altogether with a single JVP. Updating weights. Following the closed-form update in (10), we update the token weights: w^t=clip(1+λc^tmax(σ,ϵ),wmin,wmax), w_t= \! (1+λ c_t (σ,ε),\,w_ ,\,w_ ), (14) where ϵ>0ε>0 prevents division by zero, and σ is the root-mean-squared (RMS) scale of the estimated credits, whose explicit expression is provided in Appendix B.3. This RMS normalization helps stabilize the weight updates. We then perform one gradient step on the w^t\ w_t\-weighted OPD, as in (7), but with the population expectation replaced by an empirical average. The complete procedure is in Algorithm 1. Compared with vanilla OPD, the only additional steps are computing the gradient of the outcome rewards, the token credits, and the corresponding weights. Since the credits can be computed with a single JVP and the weights are obtained in closed form, these computations add little overhead. Algorithm 1 Solver for DR-OPD 1: Student pθp_θ, teacher p∗p^*, outcome reward function V, weighting strength λ, bounds wmin,wmaxw_ ,w_ 2: Set current student policy θ¯←θ θ←θ 3: for each training round do 4: Sample multiple responses per prompt from θ¯ θ. 5: Evaluate responses’ outcome reward and compute the estimated gradient of reward r r. 6: Compute the token discrepancies dtd_t in (12). 7: Compute credits c^t c_t by JVP using (13). 8: Update weights w^t w_t using (14). 9: Update θ¯ θ by gradient update of w^t\ w_t\-weighted OPD in (7). 10: end for 5 Experimental Results We evaluate Dr. OPD across four student–teacher configurations: (Base,Instruct),(Instruct,Instruct),(Base,RL),(Instruct,RL),(Base,Instruct), (Instruct,Instruct), (Base,RL), (Instruct,RL), where the first entry denotes the type of student model and the second denotes the type of teacher. These terms refer to different stages of post-training. A Base model is the pretrained model before instruction tuning, while an Instruct model has been post-trained to better follow user instructions and solve downstream tasks. An RL model is obtained through further reinforcement learning using outcome rewards and represents a more strongly post-trained model. These configurations form two complementary distillation regimes. In strong-to-weak regime, a larger Instruct model serves as the teacher for either a Base or an Instruct student. In same-size regime, we use an RL-trained teacher and a base or Instruct model of the same size as the student. 5.1 Experimental setup Models. We instantiate the four student–teacher configurations above using Qwen3 models, where models without the “Base” or “RL” suffix refer to their Instruct versions. For strong-to-weak distillation, we use Qwen3-4B (Yang et al., 2025) as the teacher and consider two students, Qwen3-1.7B-Base and Qwen3-1.7B. For same-size distillation, we consider Qwen3-4B-Base with an RL-trained Qwen3-4B-Base teacher, and Qwen3-4B with an RL-trained Qwen3-4B teacher (Yang et al., 2026). All teachers are frozen during distillation. Training data and protocol. For mathematical reasoning, we train on DeepMath-103K (He et al., 2026). For code generation, we train on Eurus-RL-Code (Cui et al., 2025). Following prior OPD work (Yang et al., 2026; Sun et al., 2026b), we use a small number of post-training steps: 50 steps in all settings. Full training details are provided in Appendix A. Evaluation. For math, we evaluate on AIME24, AIME25, AMC, Minerva Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024), and report accuracy averaged over 16 sampled responses per problem (avg@16). For code, we evaluate on HumanEval+ and MBPP+ from EvalPlus (Liu et al., 2023), together with LiveCodeBench (Jain et al., 2025), and report avg@4. We evaluate checkpoints every 10 optimization steps on the evaluated datasets and report the best checkpoint for each method, as adopted in works (Ding and Zhang, 2026; Zhao et al., 2026). Baselines. We compare Dr. OPD with four OPD-based baselines. OPD (Gu et al., 2024) is the standard on-policy distillation baseline with uniform token weighting. ExOPD (Yang et al., 2026) extrapolates the teacher’s distillation signal with the goal of moving the student beyond the teacher. With outcome reward signals, GRPD (Lin et al., 2026) considers reward-gated distillation, while OPD+GRPO (Xiaomi LLM-Core Team et al., 2026) adds an additional GRPO objective alongside the OPD objective. Implementation details are provided in Appendix A. 5.2 Distillation results Strong-to-weak distillation. We distill the Qwen3-4B teacher into the Qwen3-1.7B and Qwen3-1.7B-Base students. As shown in Table 1, Dr. OPD achieves the best average performance on both math and code for both students. Notably, vanilla OPD still leaves a substantial gap between the student and the stronger teacher, suggesting that simply mimicking a stronger, larger teacher is insufficient to bring the student close to the teacher’s performance. A smaller student can outperform its larger teacher. For the 1.7B student, Dr. OPD even surpasses the larger 4B teacher in average math performance (35.0 vs. 34.1). This improvement reflects the design of Dr. OPD, which does not merely mimic the teacher but also uses the student’s outcome reward to determine which teacher signals to follow more closely. However, simply adding outcome rewards is not sufficient, since both OPD+GRPO and GRPD also use such rewards but yield only marginal improvements. This suggests that how the outcome reward is incorporated into distillation is important. For the 1.7B-Base student, Dr. OPD still improves over vanilla OPD by 1.8 points on math, while the gain is much larger on code, reaching 6.4 points. Compared with the 1.7B student, the 1.7B-Base student shows a smaller gain on math. We attribute this mainly to its much lower starting accuracy. Since the Base student produces fewer successful rollouts, the outcome-reward signal used to estimate token credits is less informative for identifying useful teacher supervision. Despite this, Dr. OPD still achieves the best average performance among all compared methods. Math Code Method AIME24 AIME25 AMC Minerva Olym. Avg. HE+ MBPP+ LCB Avg. Teacher (Qwen3-4B) 22.5 16.9 60.5 27.1 43.7 34.1 77.1 64.8 16.1 52.7 Student: Qwen3-1.7B Initial 12.3 9.0 39.7 19.1 32.9 22.6 59.8 52.5 11.9 41.4 OPD 15.4 12.5 43.7 19.4 35.8 25.3 60.4 54.9 15.0 43.4 ExOPD 17.5 10.0 44.0 20.2 36.6 25.7 61.1 56.5 15.4 44.3 OPD+GRPO 16.0 13.1 42.9 21.0 35.1 25.6 60.7 55.4 14.9 43.7 GRPD 15.8 9.2 43.5 19.4 35.4 24.7 64.0 52.0 14.9 43.6 Dr. OPD 25.6+10.2 23.5+11.0 56.9+13.2 24.6+5.2 44.4+8.6 35.0+9.7 64.2+3.8 60.7+5.8 20.4+5.4 48.4+5.0 Student: Qwen3-1.7B-Base Initial 1.7 1.5 10.8 4.4 7.8 5.2 5.5 4.2 1.3 3.7 OPD 6.9 5.0 28.8 20.1 23.3 16.8 54.0 55.0 11.7 40.2 ExOPD 6.7 5.6 27.5 20.6 23.8 16.8 61.1 54.7 14.3 43.4 OPD+GRPO 8.1 5.2 29.3 21.0 23.7 17.5 63.1 55.1 15.0 44.4 GRPD 6.9 5.6 28.8 20.0 23.0 16.9 59.3 50.3 13.0 40.9 Dr. OPD 9.2+2.3 4.8-0.2 31.3+2.5 21.3+1.2 26.5+3.2 18.6+1.8 64.3+10.3 58.5+3.5 16.9+5.2 46.6+6.4 Table 1: Strong-to-weak distillation. We distill Qwen3-4B into Instruct and Base Qwen3-1.7B students on math and code generation. Colored subscripts report the absolute change over vanilla OPD. Bold denotes the best distillation result within each student setting. Same-size distillation. We next consider the setting where the student and teacher have the same model size, with the teacher obtained through further RL training. As shown in Table 2, vanilla OPD already brings the student close to or even beyond the teacher, leaving limited room for further gains by simply mimicking the teacher. Despite this, Dr. OPD still achieves the best average performance for both setups, improving over vanilla OPD by 2.3 and 3.0 points, respectively. Surpassing the RL-trained teacher. Notably, Dr. OPD surpasses the RL-trained teacher in both settings. For the Instruct student, it exceeds the teacher on all five benchmarks, a level of consistent improvement not achieved by any competing method. For the Base student, Dr. OPD exceeds the teacher on four of the five benchmarks and improves the average performance from 34.1 to 36.0. Method AIME24 AIME25 AMC Minerva Olym. Avg. Student: Qwen3-4B Teacher: Qwen3-4B-RL Teacher 54.0 46.5 81.0 36.9 57.7 55.2 Initial 22.5 16.9 60.5 27.1 43.7 34.1 OPD 55.8 45.8 81.0 37.3 57.9 55.6 ExOPD 56.9 43.3 82.3 37.6 56.6 55.3 OPD+GRPO 55.6 46.3 82.0 37.5 57.7 55.8 GRPD 55.8 49.2 81.6 37.6 57.0 56.2 Dr. OPD 58.3+2.5 51.7+5.8 82.9+1.9 37.8+0.5 58.5+0.6 57.8+2.3 Student: Qwen3-4B-Base Teacher: Qwen3-4B-Base-RL Teacher 23.1 22.1 55.4 26.4 43.5 34.1 Initial 11.9 8.5 36.8 9.7 27.6 18.9 OPD 21.0 19.2 56.0 27.4 41.5 33.0 ExOPD 26.9 19.2 56.7 25.6 43.1 34.3 OPD+GRPO 21.9 21.0 57.5 27.2 42.3 34.0 GRPD 21.5 18.8 56.8 27.6 43.1 33.5 Dr. OPD 25.4+4.4 20.0+0.8 58.4+2.4 31.9+4.5 44.1+2.6 36.0+3.0 Table 2: Same-size distillation. We evaluate Instruct and Base students distilled from their corresponding RL-trained teachers on mathematical reasoning. Colored subscripts in the Dr. OPD rows report the absolute improvement over vanilla OPD. Teacher and Initial are reference points; bold denotes the best distillation result within each student–teacher setting. 5.3 Ablation Study: Effect of λ The weighting strength λ is the main hyperparameter in Algorithm 1, controlling how strongly the token weights can deviate from the vanilla OPD value of 11. When λ=0λ=0, Dr. OPD reduces to vanilla OPD. As shown in Table 3, every positive value of λ improves over OPD on all five benchmarks and in average performance, suggesting that Dr. OPD does not require precise tuning of λ. Among the values tested, λ=0.4λ=0.4 performs best overall and is used in our main experiments. λ AIME24 AIME25 AMC Minerva Olym. Avg. 0 (OPD) 15.4 12.5 43.7 19.4 35.8 25.3 0.2 18.5 13.5 47.8 22.8 37.7 28.1 0.3 19.4 14.0 45.0 23.2 37.5 27.8 0.4 25.6 23.5 56.9 24.6 44.4 35.0 0.5 24.0 19.2 54.7 25.0 42.6 33.1 0.6 23.8 19.8 54.1 23.3 44.0 33.0 Table 3: Effect of the weighting strength λ on mathematical reasoning. We vary λ for the Qwen3-1.7B student distilled from Qwen3-4B. λ=0λ=0 reduces to vanilla OPD, while larger λ allows stronger token-level weight adjustments. Bold denotes the best result in each column. 6 Related Works On-policy distillation. Learning from teacher feedback on student-generated sequences addresses the mismatch between training trajectories and the contexts encountered by the student at inference. GKD studies this principle with a family of divergence objectives (Agarwal et al., 2024), while MiniLLM develops reverse-KL optimization for language-model distillation (Gu et al., 2024). Subsequent work demonstrates its effectiveness for reasoning post-training (Yang et al., 2025). Multi-teacher OPD also consolidates specialist capabilities in MiMo-V2-Flash, DeepSeek-V4, and Kimi K3 (Xiaomi LLM-Core Team et al., 2026; Xu et al., 2026; Team et al., 2026). Beyond direct teacher matching, G-OPD and ExOPD introduce a reference policy and reward extrapolation (Yang et al., 2026), and OPD2 uses the difference between a reasoning-tuned teacher and its base model as the distillation signal (Heo et al., 2026). Analyses further identify teacher–student compatibility as a factor in successful transfer (Li et al., 2026). Trust and weighting in OPD. Several methods already adapt the strength or location of teacher supervision. REOPOLD stabilizes learning through reward clipping and entropy-based token sampling (Ko et al., 2026). TrOPD identifies trustworthy regions through teacher–student agreement and treats outlier tokens with alternative objectives (Xing et al., 2026). IW-OPD weights tokens using accumulated distribution discrepancy to address deteriorating supervision along long student trajectories (Xie et al., 2026). REOPD combines token-level compatibility weights with a batch-level budget to adapt extrapolation beyond the teacher (Sun et al., 2026b). For multiple teachers, TrustMOPD allocates token-level supervision using each specialist’s displacement from a shared pre-RL reference (Sun et al., 2026a). Outcome feedback also supports selective distillation: OPDVR gates sampled-token signals according to response correctness (Lin et al., 2026), while VG-OPD verifies an expert’s advantage on individual answer criteria and localizes its supervision through token disagreement (Xu and Zhang, 2026). Concurrent work SparseOPD (Liu et al., 2026) studies extremely sparse supervision for OPD, showing that retaining only one or two token-level teacher signals per trajectory can match or outperform dense vanilla OPD. SparseOPD can be viewed as a binary-weighted OPD that masks most token-level signals, while Dr. OPD, instead, adaptively assigns continuous weights according to how each teacher signal affects the student’s performance. Thus, SparseOPD studies how little supervision is sufficient, while Dr. OPD studies how teacher supervision should be weighted to maximize student performance. Closely related to our motivation, Unmasking OPD diagnoses teacher signals through alignment with an estimated success-improving gradient in token-logit space (Armandpour et al., 2026). Dr. OPD uses the sensitivity of the updated student’s reward to each supervision weight as an optimization criterion. This yields parameter-space credits for teacher corrections and a clipped weighting rule derived from a reward lower bound relative to vanilla OPD. Bilevel optimization, reweighting, and influence. Selecting supervision through the performance of the learner connects to bilevel optimization and differentiation through learning dynamics (Franceschi et al., 2018; Shaban et al., 2019). A common strategy in this literature is to evaluate the outer objective through one or a few gradient updates of the inner learner, as in MAML (Finn et al., 2017). This idea is also used for data reweighting, where Ren et al. (2018) selects example weights through a one-step learner update evaluated on clean validation data, leading to a gradient-alignment criterion. Influence-function methods study how upweighting training examples affects the learned model, typically through an inverse-Hessian correction (Koh and Liang, 2017), while TracIn uses gradient inner products along optimization trajectories (Pruthi et al., 2020). Our method brings these ideas to token-level teacher supervision: we measure the reward sensitivity of a one-step OPD update, use outcome reward as the outer objective, and derive finite weight updates from a lower bound on reward improvement. 7 Conclusion We proposed Dr. OPD, a bilevel optimization formulation that finds the optimal weighted OPD objective for improving students’ task performance. The lower-level problem trains the student under a given weight function, while the upper-level problem selects the weights according to the reward of the resulting student. To solve Dr. OPD efficiently, we developed an iterative solver with closed-form weight updates followed by a standard gradient update on the weighted OPD. Across all evaluated experimental setups, Dr. OPD consistently outperforms existing baselines and can even enable students to surpass their teachers. AI Use Statement In this work, we used generative AI tools as follows. Among tasks requiring disclosure, AI tools assisted with implementing and debugging code, preparing scripts for experiments, and checking and refining mathematical derivations and proofs. All AI-assisted code was inspected and executed by the authors, all reported experimental results were obtained from our own training and evaluation runs, and all AI-assisted mathematical arguments were independently verified by the authors. We did not use generative AI tools for research ideation, developing the proposed method, designing the experiments, selecting hyperparameters, or providing methodological feedback. Generative AI tools did not autonomously run experiments or generate experimental results. Among the tasks for which disclosure is recommended, we used generative AI tools to assist with producing scientific figures and plotting code, as well as drafting and editing the manuscript for clarity and readability. All AI-assisted text were reviewed and revised by the authors. We take full responsibility for the final content of this work, including all text, claims, results, and artifacts produced with the aid of generative AI. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In International Conference on Learning Representations, p. 21246–21263. Cited by: §1, §2, §6. Ahmadian et al. (2024) A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker Back to basics: revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 12248–12267. Cited by: §1. Armandpour et al. (2026) M. Armandpour, F. Ilhan, D. Harrison, A. Jaiswal, D. N. Hoang, F. Faghri, Y. Zhang, M. Cho, and M. Farajtabar Unmasking on-policy distillation: where it helps, where it hurts, and why. arXiv preprint arXiv:2605.10889. Cited by: §1, §2, §6. Banerjee et al. (2005) A. Banerjee, S. Merugu, I. S. Dhillon, and J. Ghosh Clustering with bregman divergences. Journal of machine learning research 6 (Oct), p. 1705–1749. Cited by: §2. Csiszár (1975) I. Csiszár I-divergence geometry of probability distributions and minimization problems. The annals of probability, p. 146–158. Cited by: §2. Cui et al. (2025) G. Cui, L. Yuan, Z. Wang, H. Wang, Y. Zhang, J. Chen, W. Li, B. He, Y. Fan, T. Yu, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. Cited by: §A.2, §5.1. Ding and Zhang (2026) Y. Ding and R. Zhang Does on-policy distillation really distill? from noisy teacher to self-improvement. Note: arXiv preprint arXiv:2608.31046 Cited by: §5.1. Finn et al. (2017) C. Finn, P. Abbeel, and S. Levine Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 70, p. 1126–1135. Cited by: §6. Franceschi et al. (2018) L. Franceschi, P. Frasconi, S. Salzo, R. Grazzi, and M. Pontil Bilevel programming for hyperparameter optimization and meta-learning. In International conference on machine learning, p. 1568–1577. Cited by: §6. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang Minillm: knowledge distillation of large language models. In International Conference on Learning Representations, Vol. 2024, p. 32694–32717. Cited by: §1, §5.1, §6. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), p. 633–638. Cited by: §4. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. Olympiadbench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3828–3850. Cited by: §5.1. He et al. (2026) Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, et al. Deepmath-103k: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. In International Conference on Learning Representations, Vol. 2026, p. 138306–138322. Cited by: §A.2, §5.1. Hendrycks et al. (2021) D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938. Cited by: §A.2. Heo et al. (2026) B. Heo, J. Hwang, S. Yun, and D. Han On-policy delta distillation for multilingual math reasoning. arXiv preprint arXiv:2608.05802. Cited by: §6. Jain et al. (2025) N. Jain, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica Livecodebench: holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, Vol. 2025, p. 58791–58831. Cited by: §5.1. Ko et al. (2026) J. Ko, S. Abdali, Y. J. Kim, T. Chen, and P. Cameron Scaling reasoning efficiently via relaxed on-policy distillation. arXiv preprint arXiv:2603.11137. Cited by: §6. Koh and Liang (2017) P. W. Koh and P. Liang Understanding black-box predictions via influence functions. In International conference on machine learning, p. 1885–1894. Cited by: §6. Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems 35, p. 3843–3857. Cited by: §5.1. Li et al. (2023) R. Li, J. Fu, B. Zhang, T. Huang, Z. Sun, C. Lyu, G. Liu, Z. Jin, and G. Li Taco: topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852. Cited by: §A.2. Li et al. (2026) Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, et al. Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016. Cited by: §A.1, §6. Lin et al. (2026) W. Lin, J. Zhao, X. Jiang, S. Rao, Y. Li, S. Wang, B. He, and G. Huang On-policy distillation with verifiable reward. arXiv preprint arXiv:2608.24696. Cited by: §5.1, §6. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems 36, p. 21558–21572. Cited by: §5.1. Liu et al. (2026) Z. Liu, X. Xu, M. S. Seyfioglu, P. Xu, and K. Bouyarmane Extremely Sparse Supervision Incentivizes Reasoning Ability. Note: arXiv preprint arXiv:2609.04565 External Links: 2609.04565, Document, Link Cited by: §6. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1, §4. Pruthi et al. (2020) G. Pruthi, F. Liu, S. Kale, and M. Sundararajan Estimating training data influence by tracing gradient descent. Advances in neural information processing systems 33, p. 19920–19930. Cited by: §6. Ren et al. (2018) M. Ren, W. Zeng, B. Yang, and R. Urtasun Learning to reweight examples for robust deep learning. In International conference on machine learning, p. 4334–4343. Cited by: §6. Shaban et al. (2019) A. Shaban, C. Cheng, N. Hatch, and B. Boots Truncated back-propagation for bilevel optimization. In The 22nd international conference on artificial intelligence and statistics, p. 1723–1732. Cited by: §6. Shao et al. (2026) B. Shao, J. Zhang, L. Ma, Y. Shen, S. Jin, X. Guo, Y. Yang, M. Chai, Z. Xi, B. Liu, J. Shang, T. Gui, Q. Zhang, and X. Huang A token-level analysis of sampled-token reverse-kl on-policy distillation. Note: arXiv preprint arXiv:2608.25643 External Links: 2608.25643, Link Cited by: §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §1, §4. Sheng et al. (2025) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: §A.1. Sun et al. (2026a) J. Sun, M. Zheng, M. Song, Z. Liu, G. Li, H. Jiang, Y. Cheng, B. Feng, Y. Cai, J. Fang, and X. Wang Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation. Note: arXiv preprint arXiv:2609.23697 External Links: 2609.23697 Cited by: §6. Sun et al. (2026b) Y. Sun, L. Ma, H. Qin, Y. Liu, H. Lu, Y. Zhu, P. Cai, and G. Yan REOPD: reliability-adaptive reward extrapolation for on-policy distillation. arXiv preprint arXiv:2608.11698. Cited by: §5.1, §6. Team et al. (2026) K. Team, T. Bai, Y. Bai, Y. Bao, J. Cai, X. Cai, P. Cao, Y. Cao, Z. Chai, Y. Charles, et al. Kimi k3: open frontier intelligence. arXiv preprint arXiv:2607.24653. Cited by: §1, §6. Xiaomi LLM-Core Team et al. (2026) Xiaomi LLM-Core Team, B. Xiao, B. Xia, B. Yang, B. Gao, B. Shen, C. Zhang, C. He, C. Lou, F. Luo, et al. MiMo-V2-Flash Technical Report. Note: arXiv preprint arXiv:2601.02780 External Links: 2601.02780 Cited by: §5.1, §6. Xie et al. (2026) Y. Xie, S. Zhu, T. Wen, B. Chen, and Y. Wang On the Position Bias of On-Policy Distillation. Note: arXiv preprint arXiv:2606.22600 External Links: 2606.22600 Cited by: §1, §2, §6. Xing et al. (2026) X. Xing, H. Wang, B. Gao, Z. Li, and Y. Tang Trust Region On-Policy Distillation. Note: arXiv preprint arXiv:2606.01249 External Links: 2606.01249 Cited by: §1, §6. Xu et al. (2026) A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. Deepseek-v4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: §1, §6. Xu and Zhang (2026) X. Xu and Z. Zhang Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning. Note: arXiv preprint arXiv:2609.15404 External Links: 2609.15404 Cited by: §6. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 Technical Report. Note: arXiv preprint arXiv:2505.09388 External Links: 2505.09388 Cited by: §5.1, §6. Yang et al. (2026) W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. Note: arXiv preprint arXiv:2602.12125 External Links: 2602.12125 Cited by: §A.1, §5.1, §5.1, §5.1, §6. Zhao et al. (2026) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. Cited by: §5.1. Appendix A Hyperparameters and Experimental Setup A.1 Shared training configuration All methods are implemented on top of verl (Sheng et al., 2025) and share the same trainer, rollout engine, and outcome verifier; they differ only in how the per-token learning signal is constructed. Each run uses a single node with 8×H100 GPUs. Both RL teachers are public checkpoints that we keep frozen, without any further training: the math-RL Qwen3-4B of Yang et al. (2026) and the GRPO-trained Qwen3-4B-Base of Li et al. (2026). Within each student–teacher setting, all methods use the same frozen teacher, so their comparison isolates the choice of training objective. Table 4 lists the hyperparameters shared by all methods and both tasks. The learning rate, batch sizes, rollout settings, and sequence limits are identical for every method. Table 5 lists the method-specific settings. Dr. OPD reweights all valid response tokens, using a separate credit RMS for each prompt and the bounds wmin=0.001w_ =0.001 and wmax=3w_ =3; Appendix B gives the full implementation. Table 4: Hyperparameters shared by all methods and both tasks. Hyperparameter Value Learning rate 1×10−51× 10^-5 LR schedule constant Gradient clipping 1.0 Batch / Mini-batch size 128 Rollouts per prompt K 8 Rollouts per step 1024 Max prompt length 1024 tokens Max response length 12288 tokens Temperature 1.0 Table 5: Method-specific hyperparameters. All other settings follow Table 4. Method Hyperparameters OPD None ExOPD λ=1.25λ=1.25; the initial student as reference policy OPD+GRPO Equal weighting (1:1) of the distillation and outcome terms GRPD None (default settings of the released method) Dr. OPD λ=0.4λ=0.4; wmin=0.001w_ =0.001; wmax=3w_ =3 A.2 Training Data For mathematical reasoning, we train on DeepMath-103K (He et al., 2026). The dataset provides difficulty labels. We train the Qwen3-1.7B-Base student on the [3,4)[3,4) band and every other student on the [6,10][6,10] band. For code generation, we train on Eurus-RL-Code (Cui et al., 2025), again with the Qwen3-1.7B-Base student as the exception: it is trained on an easier corpus drawn from APPS (introductory) (Hendrycks et al., 2021) and TACO (EASY) (Li et al., 2023). These easier subsets increase the frequency of groups containing both correct and incorrect responses for the Base student. Such groups provide nonzero group-relative advantages; groups with identical rewards have zero advantages. A.3 Prompt Templates and Stopping Rules Chat template For all students except Qwen3-1.7B-Base, we use the model’s own chat template with thinking disabled (enable_thinking=False). The problem statement forms a single user message, followed by the suffix supplied with the training set. The math template is shown below: ⬇ question Please reason step by step, and put your final answer within . Few-shot template Qwen3-1.7B-Base has no chat template and does not reliably follow the zero-shot instruction format. For this student, we use the fixed four-shot math prompt below. ⬇ Solve the following math problems. Reason step by step, and put your final answer within . Problem: What is $1+2+3+·s+10$? Solution: The sum of the first $n$ positive integers is $ n(n+1)2$. For $n=10$ this is $ 10· 112=55$. The final answer is $ 55$. Problem: Solve for $x$: $2x+3=11$. Solution: Subtracting $3$ from both sides gives $2x=8$. Dividing both sides by $2$ gives $x=4$. The final answer is $ 4$. Problem: A rectangle has length $8$ and width $5$. What is its area? Solution: The area of a rectangle is length times width, so it is $8× 5=40$. The final answer is $ 40$. Problem: A fair coin is flipped twice. What is the probability that both flips are heads? Solution: The flips are independent, each heads with probability $ 12$. So both are heads with probability $ 12· 12= 14$. The final answer is $ 14$. Problem: question Solution: Stopping rule. With the few-shot prompt, the student may continue by generating additional problems. We therefore truncate each response at the first completed ... or at the next-problem delimiter " :", whichever comes first; everywhere else generation ends at the end-of-sequence token or at the 12,28812,288-token limit. Truncation is applied to the sampled token IDs after generation. We use the same stopping rule for every method during both training and evaluation. Appendix B Implementation of Dr. OPD For simplicity, Algorithm 1 uses the raw GRPO estimate of the reward gradient r r to compute the token credits ctc_t. In our implementation, we additionally smooth r r and apply the coordinate-wise scaling induced by the actor optimizer, matching the transformation applied to the reward gradient in standard GRPO policy optimization. B.1 Group-relative reward signal Index training rounds by k, prompts by b, and the K responses to a prompt by i. We suppress k on rollout-level quantities when the round is clear. Let rbi=V(xb,ybi)r_bi=V(x_b,y_bi), and let TbiT_bi be the number of valid response tokens. The group-relative advantages are r¯b=1K∑i=1Krbi,sb=[1K−1∑i=1K(rbi−r¯b)2]1/2,Abi=rbi−r¯bsb+ϵA. r_b= 1K _i=1^Kr_bi, s_b= [ 1K-1 _i=1^K(r_bi- r_b)^2 ]^1/2, A_bi= r_bi- r_bs_b+ _A. Here K=8K=8 and ϵA>0 _A>0 stabilizes the denominator. If all rewards in a group agree, its advantages are zero. With advantages and response lengths held fixed, the length-normalized reward direction is Gk=∑b1K∑i=1KAbiTbi∑t=1Tbi∇θlogpθ(ybi,t∣sbi,t)|θ=θ¯k,G_k= _b 1K _i=1^K A_biT_bi _t=1^T_bi . _θ p_θ(y_bi,t s_bi,t) |_θ= θ_k, up to the common distributed averaging factor. It is computed by differentiating a negative log-probability surrogate with detached coefficients Abi/(KTbi)A_bi/(KT_bi) and reversing the gradient’s sign. The factor 1/Tbi1/T_bi normalizes each response’s contribution; a response with no valid tokens contributes zero. B.2 Optimizer-scaled reward direction We maintain a reward first moment, separate from the actor optimizer’s first moment: mkR=β1mk−1R+(1−β1)Gk,m0R=0.m_k^R= _1m_k-1^R+(1- _1)G_k, m_0^R=0. The direction used in round k is constructed from the state available before the current actor update. For k>1k>1, let m^k−1R=mk−1R/(1−β1k−1) m_k-1^R=m_k-1^R/(1- _1^k-1). Let vactorv^actor be the actor AdamW optimizer’s stored second moment, j its number of completed optimizer steps, and v^actor=vactor/(1−β2j) v^actor=v^actor/(1- _2^j). The credit direction is uk=m^k−1Rv^actor+ϵAdam,u_k= m_k-1^R v^actor+ _Adam, where the operations are elementwise. Thus the numerator uses reward momentum, while the denominator reuses the actor’s second moment. The current GkG_k updates the reward momentum for subsequent rounds. If the required optimizer or momentum state is unavailable, all weights are set to one. The decoupled weight-decay term is excluded from uku_k. B.3 Token credits, RMS scale, and weight clipping For each sampled token, define the teacher log-ratio and credit by dbi,t d_bi,t =logp∗(ybi,t∣sbi,t)pθ¯k(ybi,t∣sbi,t), = p^*(y_bi,t s_bi,t)p_ θ_k(y_bi,t s_bi,t), c^bi,t c_bi,t =dbi,tdαlogpθ¯k+αuk(ybi,t∣sbi,t)|α=0=gbi,t⊤uk. =d_bi,t . ddα p_ θ_k+α u_k(y_bi,t s_bi,t) |_α=0=g_bi,t u_k. A forward-mode JVP computes these directional derivatives without materializing a parameter gradient for each token. The student replica used for the JVP is synchronized with the actor, and dropout is disabled. The derivative is evaluated by automatic differentiation, without a finite-difference perturbation. Let ℐkbI_kb contain all valid response-token positions across the K rollouts for prompt xbx_b in round k: ℐkb=(i,t):1≤i≤K, 1≤t≤Tbi,Nkb=|ℐkb|=∑i=1KTbi.I_kb=\(i,t):1≤ i≤ K,\ 1≤ t≤ T_bi\, N_kb=|I_kb|= _i=1^KT_bi. The explicit RMS scale used in (14) is σkb=(1Nkb∑i=1K∑t=1Tbic^bi,t 2)1/2(Nkb>0). _kb= ( 1N_kb _i=1^K _t=1^T_bi c_bi,t^\,2 )^1/2 (N_kb>0). This is a token-weighted RMS across the responses to one prompt, without subtracting the mean credit. Prompt and padding tokens are excluded; valid response tokens with zero credit are included. No probability-based token selection is applied. For each (i,t)∈ℐkb(i,t) _kb, we set w^bi,t=clip(1+λc^bi,tmax(σkb,ϵσ), 0.001, 3),λ=0.4 w_bi,t= \! (1+λ c_bi,t ( _kb, _σ),\,0.001,\,3 ), λ=0.4 in the main experiments; the ablation varies λ. Here ϵσ>0 _σ>0 is the stability constant denoted by ϵε in (14). If all credits for a prompt are zero, its valid tokens receive weight one. If Nkb=0N_kb=0, we set σkb=0 _kb=0 and that prompt contributes no response tokens to the actor loss. The positive lower bound preserves strictly positive weights, as required by the weighted-distillation formulation. B.4 Actor update The detached products w^bi,tdbi,t w_bi,td_bi,t are passed as token advantages to the actor loss used by vanilla OPD. The actor uses a PPO-style probability-ratio objective, including ratio clipping and dual clipping for negative advantages, token-mean aggregation, and AdamW. At the reference policy, where the probability ratio equals one, the ascent gradient is the token average of w^bi,tgbi,t w_bi,tg_bi,t. Gradients used to construct the reward direction are cleared before the actor update. The weights and teacher signals remain detached during that update. Appendix C Proofs and Theoretical Details This section collects the assumptions, derivations, and proofs used in Sections 2–3.2. We first specify the reference distribution for one optimization round, then establish the weighted-distillation properties and the reward-improvement guarantee. C.1 Notation and regularity conditions Fix the current student θ¯ θ. We assume a finite vocabulary and strictly positive student and teacher probabilities. A feasible weight function is a measurable function in =w:wmin≤w(s,v)≤wmax,0<wmin<1<wmax<∞.W=\w:w_ ≤ w(s,v)≤ w_ \, 0<w_ <1<w_ <∞. The reference context distribution and w are held fixed when we differentiate the distillation objective. In particular, the loss for this round is Dwθ¯(θ):=x,y∼pθ¯(⋅∣x)[∑tKLw(st,⋅)(pθ(⋅∣st)∥p∗(⋅∣st))].D_w θ(θ):=E_x,\,y p_ θ(· x) [ _t _w(s_t,·) (p_θ(· s_t)\|p^*(· s_t) ) ]. (C.1) We suppress the superscript θ¯ θ below and in the main text. The reference distribution is refreshed between rounds. Define the reference measure on context–token pairs by ∫f(s,v)dμ(s,v):=x,y∼pθ¯(⋅∣x)[∑t=1Tf(st,yt)] f(s,v)\,dμ(s,v):=E_x,\,y p_ θ(· x) [ _t=1^Tf(s_t,y_t) ] for nonnegative measurable f. Its total mass is [T]E[T]; thus μ is an unnormalized token-occupancy measure. It includes both context visitation and the conditional probability of the sampled token. We use ⟨f,h⟩μ=∫fhμ f,h _μ= fh\,dμ and ‖f‖μ2=∫f2μ\|f\|_μ^2= f^2\,dμ for scalar functions, and write g(st,yt)=gtg(s_t,y_t)=g_t for the teacher correction in (6). Assumption 6 (Local regularity and integrability). Let μ denote the fixed reference distribution over sampled trajectories, and let =w:wmin≤w≤wmaxW=\w:w_ ≤ w≤ w_ \. Assume the following conditions hold. (i) Integrability. The expected response length is finite, μ[T]<∞.E_μ[T]<∞. Moreover, the weighted OPD objective is finite in a neighborhood of θ¯ θ, and differentiation can be interchanged with expectation. In particular, ∇θDw(θ)=μ[∇θℓw(θ,Z)],w∈, _θD_w(θ)=E_μ\! [ _θ _w(θ;Z) ], w , for all θ in this neighborhood. (i) Finite second moment. The token-level teacher correction satisfies M:=μ[∑t=1T‖gt‖2]<∞.M:=E_μ\! [ _t=1^T\|g_t\|^2 ]<∞. (i) Local smoothness of the reward. There exists an open convex set U such that θ¯∪θ+(w):w∈⊂U,\ θ\∪\θ^+(w):w \⊂ U, and R is continuously differentiable on U with LRL_R-Lipschitz gradient: ‖∇R(θ)−∇R(θ′)‖≤LR‖θ−θ′‖,∀θ,θ′∈U.\|∇ R(θ)-∇ R(θ )\|≤ L_R\|θ-θ \|, ∀\,θ,θ ∈ U. All parameter norms are Euclidean. Cauchy–Schwarz gives ∫‖g‖μ≤[T]M \|g\|\,dμ≤ E[T]M, so the response in (7) satisfies ‖θ+(w)−θ¯‖≤ηwmax[T]M.\|θ^+(w)- θ\|≤η w_ E[T]M. Thus a sufficiently small student step size keeps all responses in a fixed neighborhood where the required smoothness holds. The condition ληLRM≤1λη L_RM≤ 1 is equivalent to λ≤1/(ηLRM)λ≤ 1/(η L_RM) when LRM>0L_RM>0; when LRM=0L_RM=0, it imposes no further restriction on positive λ. Unlike the fixed-reference distillation loss, the reward R(θ)R(θ) is evaluated under the student’s own response distribution. Under the corresponding differentiation and summability conditions, ∇R(θ)=x,y∼pθ[V(x,y)∑t∇θlogpθ(yt∣st)].∇ R(θ)=E_x,\,y p_θ [V(x,y) _t _θ p_θ(y_t s_t) ]. This follows by differentiating pθ(y∣x)=∏tpθ(yt∣st)p_θ(y x)= _tp_θ(y_t s_t), with termination included in the response convention. C.2 Properties of weighted OPD C.2.1 Proof of Theorem 3 Proof. For positive a,ba,b, define d(a,b)=alog(a/b)−a+b=bϕ(a/b),ϕ(u)=ulogu−u+1.d(a,b)=a (a/b)-a+b=bφ(a/b), φ(u)=u u-u+1. Since ϕ′(u)=loguφ (u)= u and ϕ′(u)=1/u>0φ (u)=1/u>0, its unique minimum is ϕ(1)=0φ(1)=0. Hence d(a,b)≥0d(a,b)≥ 0, with equality if and only if a=ba=b. At each context s, KLw(s,⋅)(pθ∥p∗)=∑vw(s,v)d(pθ(v∣s),p∗(v∣s))≥0. _w(s,·)(p_θ\|p^*)= _vw(s,v)d (p_θ(v s),p^*(v s) )≥ 0. Because every weight is strictly positive, equality holds precisely when the student and teacher probabilities agree for every token. Taking the reference expectation proves Dw(θ)≥0D_w(θ)≥ 0. The expected loss is zero if and only if the distributions agree at almost every context under the reference context measure. In particular, the claim holds at every context of positive reference mass; the objective places no restriction on contexts of zero measure. Finally, when w≡1w≡ 1, the correction terms sum to zero by normalization, leaving the ordinary reverse KL and the vanilla OPD loss. ∎ The linear correction is needed for token-dependent weights. For example, if p=(1/4,3/4)p=(1/4,3/4), p∗=(1/2,1/2)p^*=(1/2,1/2), and w=(4,1)w=(4,1), then ∑vwvpvlog(pv/pv∗)=−log2+(3/4)log(3/2)<0 _vw_vp_v (p_v/p_v^*)=- 2+(3/4) (3/2)<0. The corrected divergence remains nonnegative term by term. When w(s,v)=a(s)w(s,v)=a(s) is independent of the token, the correction cancels and we recover a(s)KL(pθ(⋅∣s)∥p∗(⋅∣s))a(s) (p_θ(· s)\|p^*(· s)). C.2.2 Derivation of the sampled-token gradient Derivation of (6). At a fixed context, let pv=pθ(v∣s)p_v=p_θ(v s) and qv=p∗(v∣s)q_v=p^*(v s). Differentiating with w fixed gives ∇θKLw(s,⋅)(pθ∥p∗) _θ _w(s,·)(p_θ\|p^*) =∑vw(s,v)[log(pv/qv)+1−1]∇θpv = _vw(s,v) [ (p_v/q_v)+1-1 ] _θp_v =∑vpvw(s,v)log(pv/qv)∇θlogpv. = _vp_vw(s,v) (p_v/q_v) _θ p_v. The derivative of −pv-p_v cancels the additional 11 separately for each token. At θ=θ¯θ= θ, the last sum is a conditional expectation under the reference student’s next-token distribution. Taking the reference expectation over contexts therefore yields −∇Dw(θ¯)=[∑twtlogpt∗pt(θ¯)∇θlogpθ(yt∣st)|θ=θ¯]=[∑twtgt].-∇ D_w( θ)=E [ _tw_t p_t^*p_t( θ) . _θ p_θ(y_t s_t) |_θ= θ ]=E [ _tw_tg_t ]. The interchange is justified by Assumption 6. ∎ The identity allows token-dependent w because the full conditional expectation is retained. Differentiating the reference sampling law, or recomputing w as a function of the optimization variable, would introduce additional terms. An explicit unbiased sampled loss is D^w(θ,x,y):=∑twtpt(θ¯)[pt(θ)logpt(θ)pt∗−pt(θ)+pt∗], D_w(θ;x,y):= _t w_tp_t( θ) [p_t(θ) p_t(θ)p_t^*-p_t(θ)+p_t^* ], where (x,y)(x,y) is sampled from the reference policy and the denominator and weights are fixed. Conditional expectation over yty_t recovers the vocabulary sum in (C.1). At θ¯ θ, the factor pt(θ)/pt(θ¯)p_t(θ)/p_t( θ) in its gradient is one, so −∇D^w(θ¯,x,y)=∑twtgt-∇ D_w( θ;x,y)= _tw_tg_t. This explains why (6) contains no explicit importance denominator. At another parameter value, the probability ratio must be retained when using the same reference rollouts. For the interpretation of gtg_t, let zt=∇θlogpθ(yt∣st)|θ¯z_t= . _θ p_θ(y_t s_t) |_ θ. Then dϵlogpθ¯+ϵgt(yt∣st)|ϵ=0=logpt∗pt(θ¯)‖zt‖2. . ddε p_ θ+ε g_t(y_t s_t) |_ε=0= p_t^*p_t( θ)\,\|z_t\|^2. Thus the infinitesimal correction increases or decreases the sampled token’s log-probability according to the teacher–student log-ratio, unless zt=0z_t=0. This is a local statement about one token; descent of the full distillation loss follows from the aggregate gradient. C.2.3 Descent of the lower-level objective If, in addition, DwD_w has an LDL_D-Lipschitz gradient on U, uniformly over feasible w, its one-step response decreases the fixed-reference loss whenever ηLD≤1η L_D≤ 1. Proof. The smoothness inequality and θ+(w)=θ¯−η∇Dw(θ¯)θ^+(w)= θ-η∇ D_w( θ) give Dw(θ+(w)) D_w(θ^+(w)) ≤Dw(θ¯)−η(1−ηLD2)‖∇Dw(θ¯)‖2 ≤ D_w( θ)-η (1- η L_D2 )\|∇ D_w( θ)\|^2 ≤Dw(θ¯)−η2‖∇Dw(θ¯)‖2. ≤ D_w( θ)- η2\|∇ D_w( θ)\|^2. ∎ This additional loss-smoothness assumption is only needed for the descent statement, not for the reward bounds below. C.3 Bilevel formulation with multiple inner solutions Let P denote the student parameter space. For a fixed reference distribution, define Θ(w):=argminθ∈Dw(θ). (w):= _θ D_w(θ). The optimistic convention used in (5) selects the highest-reward member of this set: maxw∈maxθ∈Θ(w)R(θ). _w \; _θ∈ (w)R(θ). (C.2) We assume the inner solution sets are nonempty and that the displayed maxima are attained. If a reward maximum is not attained, the corresponding optimal value is written as a supremum; an empty inner solution set instead requires a different formulation. If exact teacher matching is attainable, all feasible positive weightings share the same zero-loss minimizers by Theorem 3. Weighting can change the best attainable compromise when exact matching is infeasible and the learning trajectory before convergence. Our local analysis replaces Θ(w) (w) by the specified response θ+(w)θ^+(w). It therefore does not require an attained or unique global inner optimum and does not establish convergence to (C.2). C.4 Credit as sensitivity to the weight function Let F(w)=R(θ+(w))F(w)=R(θ^+(w)) and define c(s,v)=g(s,v)⊤∇R(θvani+)c(s,v)=g(s,v) ∇ R( _vani^+). The next calculation makes the derivative notation in (8) precise. Derivation of the credit. For a bounded measurable perturbation h, the function 1+ϵh1+ε h is feasible for sufficiently small |ϵ||ε|. Linearity of the response gives θ+(1+ϵh)=θvani++ηϵ∫hgμ.θ^+(1+ε h)= _vani^++ηε hg\,dμ. Applying the chain rule, dϵF(1+ϵh)|ϵ=0=η∫hg⊤∇R(θvani+)dμ=η⟨h,c⟩μ. . ddεF(1+ε h) |_ε=0=η h\,g ∇ R( _vani^+)\,dμ=η h,c _μ. (C.3) Moreover, ‖c‖μ2≤M‖∇R(θvani+)‖2<∞\|c\|_μ^2≤ M\|∇ R( _vani^+)\|^2<∞. Thus the L2(μ)L^2(μ) gradient of F at w=1w=1 is ηcη c. ∎ Accordingly, ∂R(θ+(w))/∂wt∂ R(θ^+(w))/∂ w_t in the main text denotes sensitivity per unit reference measure, evaluated at (st,yt)(s_t,y_t). For an atom (s,v)(s,v), the ordinary coordinate derivative instead includes its mass: ∂F(w)∂w(s,v)|w=1=ημ((s,v))c(s,v). . ∂ F(w)∂ w(s,v) |_w=1=ημ(\(s,v)\)c(s,v). A derivative with respect to a weight in a finite sample average likewise includes that average’s coefficient. Changing weights only on a set of zero reference measure has no effect on F. C.5 Proof of Proposition 4 Proof. The argument has two steps: bound the change in the student response, then apply reward smoothness. Step 1: Control the response change. Let h=w−1h=w-1 and Δ(w)=θ+(w)−θvani+=η∫hgμ (w)=θ^+(w)- _vani^+=η hg\,dμ. By Cauchy–Schwarz, ‖Δ(w)‖2≤η2(∫h2μ)(∫‖g‖2μ)=η2M‖h‖μ2.\| (w)\|^2≤η^2 ( h^2\,dμ ) ( \|g\|^2\,dμ )=η^2M\|h\|_μ^2. All terms are finite because h is bounded, μ has finite mass, and M<∞M<∞. The trajectory-length contribution is already included in μ and M. Step 2: Lower-bound the reward change. Both responses and the segment joining them lie in U. Therefore, R(θ+(w))−R(θvani+) R(θ^+(w))-R( _vani^+) ≥∇R(θvani+)⊤Δ(w)−LR2∥Δ(w)∥2 ≥∇ R( _vani^+) (w)- L_R2\| (w)\|^2 ≥η⟨h,c⟩μ−LRη2M2‖h‖μ2. ≥η h,c _μ- L_Rη^2M2\|h\|_μ^2. Using ληLRM≤1λη L_RM≤ 1 and the definition of μ gives R(θ+(w))−R(θvani+)≥η∫(hc−h22λ)μ,R(θ^+(w))-R( _vani^+)≥η (hc- h^22λ )\,dμ, which is (9). The bound is exact at w=1w=1, and its first variation there agrees with (C.3). ∎ C.6 Clipped weights and proof of Theorem 5 Proof. We first optimize the lower bound pointwise, then quantify its gain. Step 1: Maximize the surrogate. Completing the square yields (w−1)c−(w−1)22λ=λc22−(w−1−λc)22λ.(w-1)c- (w-1)^22λ= λ c^22- (w-1-λ c)^22λ. Its unique maximizer on [wmin,wmax][w_ ,w_ ] is w+=clip(1+λc,wmin,wmax)w^+= (1+λ c,w_ ,w_ ). This function is measurable and feasible. Since the surrogate is integrable, pointwise maximization also maximizes its expectation, uniquely up to sets of zero μ-measure. Step 2: Bound the improvement. Write h+=w+−1h^+=w^+-1. The first-order optimality condition, evaluated against the feasible choice w=1w=1, gives (c−h+λ)(1−w+)≤0,h+c≥(h+)2λ. (c- h^+λ )(1-w^+)≤ 0, h^+c≥ (h^+)^2λ. Integrating and applying Proposition 4, R(θ+(w+))−R(θvani+)≥η2λ‖w+−1‖μ2≥0.R(θ^+(w^+))-R( _vani^+)≥ η2λ\|w^+-1\|_μ^2≥ 0. This is (11). The gain is strictly positive when w+≠1w^+≠ 1 on a set of positive μ-measure. Because 11 lies strictly inside the feasible interval, this is equivalent to nonzero credit on a set of positive measure. If the credit vanishes almost everywhere, the update recovers vanilla OPD. ∎ Appendix D Supplementary Experimental Results This section provides additional experimental results and training dynamics. All quantities below are measured on the student’s own training rollouts. Figure 3 shows the training-reward dynamics for the experiments in Table 1. Instruct student: Qwen3-1.7B Base student: Qwen3-1.7B-Base (a) Math (b) Code (c) Math (d) Code Figure 3: Dynamics of training rewards across students and tasks, computed by verifier accuracy of the student’s own rollouts during training. Lines are exponential moving averages (α=0.95α=0.95). Figure 4 shows the training dynamics for the Qwen3-4B-Base student distilled from an RL-trained teacher of the same model size. The training reward is the fraction of rollouts receiving a positive outcome reward, and the response length is the average number of generated tokens per rollout. Although the student and teacher differ only in post-training, Dr. OPD maintains a clear advantage in training reward. (a) Training reward (b) Response length Figure 4: Training dynamics for the Qwen3-4B-Base student distilled from the Qwen3-4B-Base-RL teacher on math. Thick lines show exponential moving averages (α=0.95α=0.95). Figure 5 further examines the effect of the weighting strength λ, with λ=0λ=0 corresponding to vanilla OPD. All tested positive values of λ achieve higher training rewards than vanilla OPD and exhibit broadly similar reward curves. In contrast, response length can vary substantially for larger values of λ, with differences of thousands of tokens late in training. (a) Training reward (b) Response length (c) Entropy Figure 5: Training dynamics under different values of λ. (a) Training reward, (b) response length, and (c) entropy over training steps for vanilla OPD (λ=0λ=0) and Dr. OPD with λ∈0.2,0.3,0.4,0.5,0.6λ∈\0.2,0.3,0.4,0.5,0.6\, using the Qwen3-1.7B student and Qwen3-4B teacher. Thick lines show exponential moving averages (α=0.95α=0.95).