Paper deep dive

REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation

Yang Sun, Lichao Ma, Houyuan Qin, Yuxin Liu, Hanyang Lu, Yao Zhu, Pinlong Cai, Guohang Yan

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 40

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:On-policy distillation (OPD) trains a student on its own trajectories under dense token-level supervision from a teacher. Reward-extrapolation methods such as ExOPD amplify the teacher-reference log-likelihood ratio to move beyond direct imitation, but apply a single global coefficient $\lambda$ to every token. This can drive the student to fit extreme peaks in the implicit reward, causing reward hacking and unstable training, and the optimal $\lambda$ varies across domains, requiring costly sweeps. We propose REOPD, a reliability-adaptive reward extrapolation framework for OPD. REOPD combines a token-level compatibility weight with a batch-level adaptive budget, yielding a token-wise coefficient $\lambda_{b,t}=1+\gamma_b q_t$ that preserves teacher alignment while selectively extrapolating along reliable teacher-reference directions. It requires no verifier, reward model, value model, or extra rollout beyond standard OPD. REOPD outperforms G-OPD on single-teacher mathematics and on both domains in the multi-teacher setting, while matching G-OPD on single-teacher code, demonstrating effective fine-grained reliability adaptation across domains and teacher configurations.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

38,608 characters extracted from source content.

Expand or collapse full text

REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation Yang Sun Lichao Ma Affiliation: Peking University Houyuan Qin Affiliation: Southwest Jiaotong University [0.25em] Yuxin Liu Affiliation: School of Computer Science and Technology,University of Science and Technology of China Hanyang Lu Affiliation: School of Automation Engineering,University of Electronic Science and Technology of China Yao Zhu Affiliation: Renmin University of China [0.25em] Pinlong Cai Affiliation: Frontier Discovery Center, Shanghai Artificial Intelligence Laboratory Guohang Yan Affiliation: Autonomous Driving, Shanghai Artificial Intelligence Laboratory [0.6em] Shanghai Artificial Intelligence Laboratory [0.3em] Equal contribution.Email (Yang Sun): 25213080092@m.fudan.edu.cn Abstract On-policy distillation (OPD) trains a student on its own generated trajectories under dense token-level supervision from a teacher, providing an effective post-training paradigm for large language models. Reward-extrapolation methods such as ExOPD further amplify the teacher–reference log-likelihood ratio to move beyond direct imitation. However, ExOPD uses a single global scalar λ to apply the same extrapolation strength indiscriminately to every token. This can drive the student to aggressively fit extreme peaks in the teacher–reference log-ratio that defines the implicit reward, resulting in reward hacking and unstable training. Moreover, the optimal λ varies across domains, requiring costly domain-specific sweeps that may still fail to identify an appropriate extrapolation strength. We propose REOPD, a Reliability-Adaptive Reward Extrapolation framework for On-Policy Distillation. REOPD combines a token-level compatibility weight with a batch-level adaptive budget. The former modulates token-wise residuals according to the student–teacher discrepancy, while the latter dynamically adjusts the overall extrapolation strength according to the reliability and scale of residual signals in each batch. Together, they yield a token-wise effective coefficient λb,t=1+γbqt _b,t=1+ _bq_t, which preserves the original teacher-alignment term while selectively extrapolating along reliable teacher–reference directions. REOPD requires no additional verifier, reward model, value model, or rollout beyond the standard OPD pipeline. Evaluations show that REOPD outperforms G-OPD on single-teacher mathematics and on both domains in the multi-teacher setting, while matching G-OPD on single-teacher code. These results demonstrate the effectiveness of fine-grained reliability adaptation for reward extrapolation in on-policy distillation across task domains and teacher configurations. Introduction On-policy distillation (OPD) trains a student on its own trajectories and queries the teacher on the same prefixes (1). Unlike distillation on fixed teacher-generated data, OPD supervises states that the current student actually visits and provides a dense learning signal at every sampled token. This makes it an effective post-training approach for transferring reasoning capabilities from specialized teachers. Yet dense supervision is heterogeneous: its usefulness depends not only on teacher quality, but also on the local compatibility between student and teacher (10; 16). G-OPD interprets OPD as dense KL-regularized reinforcement learning, in which the teacher–reference log-ratio defines an implicit token-level reward (18). Standard OPD corresponds to a reward coefficient λ=1λ=1, whereas ExOPD sets λ>1λ>1 to extrapolate beyond direct teacher matching. ExOPD, however, applies one global coefficient to every token throughout training. Uniform scaling can overemphasize extreme teacher–reference log-ratios together with useful residuals, allowing a small number of peaks to dominate policy updates and increasing the risk of reward hacking or unstable optimization. Moreover, the preferred coefficient differs across domains, so each new setting requires multiple full training and evaluation runs to select λ. Figure 1: Fixed-coefficient ExOPD applies the same residual gain λ−1λ-1 to every token and may amplify teacher–reference log-ratio peaks. REOPD preserves teacher alignment, gates the beyond-teacher residual with compatibility qb,i,tq_b,i,t, and derives a bounded micro-batch budget γb _b from synchronized statistics. Our key observation is that teacher alignment and extrapolation beyond the teacher should be controlled separately. A token with a large student–teacher discrepancy may still provide a useful alignment signal, yet be unsuitable for amplified extrapolation. We therefore propose REOPD, which preserves the standard OPD alignment term and adapts only the additional teacher–reference residual. As shown in Figure 1, REOPD combines a token-level compatibility weight with a bounded micro-batch budget. Their product defines a token-wise effective coefficient, allowing reliable residuals to be emphasized without uniformly increasing every update. The method reuses the student, teacher, and reference log-probabilities already available in G-OPD and requires no verifier, reward model, value model, or additional rollout. We evaluate REOPD on mathematical reasoning, code generation, and mixed-domain multi-teacher distillation. Across repeated experiments, REOPD outperforms G-OPD on single-teacher mathematics and on both mathematics and code in the multi-teacher setting, while achieving comparable performance to G-OPD on single-teacher code. These results indicate that online residual control can reduce reliance on setting-specific coefficient sweeps while maintaining competitive performance across task domains and teacher configurations. Our contributions are summarized as follows: • We formulate the indiscriminate amplification and domain-specific tuning of fixed-coefficient reward extrapolation as a residual-control problem. • We propose REOPD, which combines token-level compatibility with a bounded micro-batch budget while preserving the standard OPD alignment term and requiring no external outcome supervision. • We evaluate REOPD against OPD and a fixed-coefficient sweep on mathematics, code, and multi-teacher distillation, showing competitive performance without selecting a task-specific global coefficient. Related Work Knowledge Distillation and On-Policy Distillation. Knowledge distillation transfers teacher knowledge through output distributions, representations, or generated sequences (7; 9). Sequence-level distillation for autoregressive models is usually off-policy, creating a mismatch between teacher trajectories used for training and student prefixes encountered at inference. MiniLLM reduces this mismatch by optimizing reverse KL on student samples (5), while generalized knowledge distillation formalizes supervision on student-generated trajectories and includes OPD as an on-policy instance (1). Recent studies show that successful OPD also depends on student–teacher compatibility and token-level utility, rather than teacher strength or discrepancy alone (10; 16). REOPD builds on this on-policy setting but targets the extrapolation residual rather than the complete distillation loss. Reward Extrapolation for On-Policy Distillation. G-OPD casts OPD as dense KL-regularized reinforcement learning (18): the teacher–reference log-ratio serves as an implicit reward, and the student–reference KL constrains policy deviation. The formulation recovers OPD at λ=1λ=1 and defines ExOPD by λ>1λ>1, where an additional teacher–reference residual is scaled by λ−1λ-1. This objective-level extrapolation differs from weight-space methods such as ExPO, which extrapolate model parameters after preference optimization (23). Although a suitable λ can improve distillation, fixed global scaling is sensitive to both log-ratio peaks and the task domain. REOPD directly extends ExOPD by replacing its global residual multiplier with online token- and micro-batch-level control. Adaptive and Reliability-Aware Distillation. Adaptive distillation adjusts supervision according to training state or sample quality. AdaKD adapts token selection and temperature (14), while ASKD conditions a KL objective on externally provided sample quality (21). Within OPD, TIP selects informative alignment tokens (16), and Prune-OPD attenuates teacher supervision or truncates rollouts after local support drift (19). These methods modify teacher alignment or the rollout process; REOPD instead preserves alignment and controls only the beyond-teacher residual. Reliability can also be estimated from task-level feedback. SCOPE routes verifier-labeled trajectories between self-reinforcement and OPD (22); SG-OPD uses agreement between outcome and teacher signals and verified teacher rollouts (15); and reward-gated OPD uses verifier feedback to regulate teacher guidance (2). REOPD addresses the complementary setting without outcome labels or verifiers: it derives residual control solely from student, teacher, and reference log-probabilities already computed in white-box OPD. REOPD: Reliability-Adaptive Reward Extrapolation Problem Setup Let D denote the prompt distribution. Given a prompt x∼x , the student policy πθ _θ generates an on-policy response y=(y1,…,yT)y=(y_1,…,y_T), where y∼πθ(⋅∣x)y _θ(· x). We denote the prefix state at token t by st=(x,y<t)s_t=(x,y_<t). The teacher and reference policies are denoted by πT _T and πref _ref, respectively. In the multi-teacher setting, πT _T denotes the domain teacher routed to the current example. Let mt∈0,1m_t∈\0,1\ indicate whether yty_t is a valid response token. For a sampled token yty_t, we define the student–teacher alignment cost as at=log⁡πθ(yt∣st)−log⁡πT(yt∣st).a_t= _θ(y_t s_t)- _T(y_t s_t). (1) Its expectation under the student distribution corresponds to the reverse KL divergence between the student and teacher at sts_t. Thus, ata_t is an alignment cost term; the PPO advantage is defined later as the negative total token cost. G-OPD further defines the teacher–reference log-ratio as a dense implicit reward (18): rt=log⁡πT(yt∣st)−log⁡πref(yt∣st).r_t= _T(y_t s_t)- _ref(y_t s_t). (2) With a global reward coefficient λ, the sampled-token ExOPD cost can be written as CtExOPD=at−(λ−1)rt.C_t^ExOPD=a_t-(λ-1)r_t. (3) When λ=1λ=1, Equation (3) reduces to standard OPD. When λ>1λ>1, the residual term encourages the student to move beyond the teacher along the teacher–reference direction. However, the same multiplier λ−1λ-1 is applied to every token. This uniform scaling can amplify extreme implicit-reward peaks and requires λ to be selected separately for different training settings. We therefore preserve ata_t and replace only the residual multiplier with a bounded, data-dependent coefficient. Method Overview Let b denote a synchronized micro-batch whose sufficient statistics are aggregated across data-parallel ranks, and let i index a response in b. REOPD constructs the effective extrapolation coefficient λb,i,t=1+γbqb,i,t, _b,i,t=1+ _bq_b,i,t, (4) where qb,i,t∈(0,1]q_b,i,t∈(0,1] is a token-level compatibility weight and γb∈[0,γmax] _b∈[0, _ ] is a shared micro-batch extrapolation budget. The resulting token cost is Cb,i,tREOPD=ab,i,t−γbqb,i,trb,i,t.C_b,i,t^REOPD=a_b,i,t- _bq_b,i,tr_b,i,t. (5) Figure 1(b) separates REOPD into a token path and a controller path. The token path retains the local residual qb,i,trb,i,tq_b,i,tr_b,i,t, whereas the controller path all-reduces detached sufficient statistics to obtain the shared budget γb _b. Their product determines how much of the extra residual is applied to each token. Neither path modifies the teacher-alignment cost ab,i,ta_b,i,t. This formulation contains several useful special cases. Setting γb=0 _b=0 recovers OPD. Setting qb,i,t=1q_b,i,t=1 and γb=λ−1 _b=λ-1 recovers fixed-λ ExOPD. A fixed γ with adaptive qb,i,tq_b,i,t gives token-only control, while qb,i,t=1q_b,i,t=1 with adaptive γb _b gives micro-batch-only control. Token-Level Compatibility Weight REOPD estimates local student–teacher compatibility from log-probabilities already computed on sampled response tokens. We first construct the low-variance k3k_3 discrepancy proxy xb,i,t x_b,i,t =log⁡πT(yi,t∣si,t)−log⁡πθ(yi,t∣si,t), = _T(y_i,t s_i,t)- _θ(y_i,t s_i,t), (6) δ^b,i,t δ_b,i,t =exp⁡(xb,i,t)−xb,i,t−1. = (x_b,i,t)-x_b,i,t-1. The compatibility weight is then defined as qb,i,t=exp⁡(−δ^b,i,tτ),τ>0.q_b,i,t= (- δ_b,i,tτ ), τ>0. (7) Because δ^b,i,t≥0 δ_b,i,t≥ 0, the weight lies in (0,1](0,1] in exact arithmetic. A small sampled discrepancy gives qb,i,t≈1q_b,i,t≈ 1 and retains most of the extrapolation residual, whereas a large discrepancy yields a smaller weight. The temperature τ controls how rapidly this attenuation occurs. We emphasize that qb,i,tq_b,i,t measures local compatibility rather than task-level correctness. A small discrepancy does not guarantee that the teacher is correct, and a large discrepancy does not make its alignment signal useless. This is why REOPD applies qb,i,tq_b,i,t only to the additional reward residual. The compatibility proxy and its resulting weight are detached from the computation graph. Intermediate log-ratios are numerically bounded in implementation, but no full-vocabulary KL or additional teacher forward pass is required. Micro-Batch Reliable Residual Statistics Token compatibility alone does not determine how much extrapolation the current micro-batch can support. REOPD therefore aggregates two statistics over valid response tokens. For compactness, let ∑b _b denote summation over all sequence–token pairs (i,t)(i,t) in synchronized micro-batch b. We define the compatibility-weighted residual proportion as ρb=∑bmi,t|rb,i,t|qb,i,t∑bmi,t|rb,i,t|+ϵ. _b= _bm_i,t|r_b,i,t|q_b,i,t _bm_i,t|r_b,i,t|+ε. (8) The statistic ρb∈[0,1] _b∈[0,1] measures the fraction of residual magnitude retained after compatibility weighting. We further define the reliable residual scale sb=(∑bmi,t(qb,i,trb,i,t)2∑bmi,t+ϵ)1/2.s_b= ( _bm_i,t(q_b,i,tr_b,i,t)^2 _bm_i,t+ε )^1/2. (9) Thus, ρb _b captures the relative amount of compatible residual, whereas sbs_b captures its absolute RMS scale. All sufficient statistics are summed across data-parallel ranks before the ratios are evaluated, so each rank uses the same controller output. To reduce micro-batch noise, REOPD maintains exponential moving averages: z¯b=βz¯b−1+(1−β)zb,z∈ρ,s. z_b=β z_b-1+(1-β)z_b, z∈\ρ,s\. (10) Both statistics and their moving averages are computed without gradient tracking. Bounded Micro-Batch Extrapolation Budget REOPD converts the smoothed statistics into a target extrapolation budget: γ~b=clip⁡(B0ρ¯bs¯b+ϵ,0,γmax). γ_b=clip ( B_0 ρ_b s_b+ε,0, _ ). (11) A larger ρ¯b ρ_b permits stronger extrapolation when a larger fraction of the residual remains compatible. In contrast, a larger s¯b s_b reduces the coefficient so that a micro-batch with large residual scale does not dominate the update. The upper bound γmax _ provides an explicit limit on extrapolation. The clipped target is further smoothed before being applied: γb=βγγb−1+(1−βγ)γ~b. _b= _γ _b-1+(1- _γ) γ_b. (12) Since γ~b∈[0,γmax] γ_b∈[0, _ ], the smoothed budget remains in the same interval when initialized within it. When Equation (11) is not clipped and ϵε is negligible, it approximately satisfies γ~bs¯b≈B0ρ¯b γ_b s_b≈ B_0 ρ_b. This relation should be interpreted as an adaptive target rather than a strict constraint after clipping and smoothing. The scale B0B_0 can be specified directly. In automatic mode, REOPD initializes it during the first K0K_0 controller updates from a scaled moving average of the alignment RMS, κRMSb(a)κ\,RMS_b(a), and then keeps it fixed. An optional warm-up can hold γb _b at a prescribed value during the initial training steps. The corresponding initialization length, smoothing coefficients, and warm-up configuration are reported with the training hyperparameters. Final Objective and Optimization The PPO-style actor update (13) uses the negative token cost as its advantage: Ab,i,tREOPD=−Cb,i,tREOPD.A_b,i,t^REOPD=-C_b,i,t^REOPD. (13) In exact arithmetic, Equations (4)–(12) imply 1≤λb,i,t≤1+γmax.1≤ _b,i,t≤ 1+ _ . (14) The advantage is masked to valid response tokens and passed to the same PPO-style policy surrogate used by the OPD baseline. The compatibility weights, sufficient statistics, moving averages, and extrapolation budget are all treated as stop-gradient control signals. REOPD therefore changes only the construction of the token advantage and reuses the student, teacher, and reference evaluations already required by G-OPD. Algorithm 1 REOPD Training Input: Prompt distribution D; student πθ _θ; teacher πT _T; reference πref _ref; τ, γmax _ , β, βγ _γ; optional B0B_0 or auto-calibration parameters κ,K0κ,K_0 Output: Updated student policy πθ _θ 1: for each synchronized micro-batch b do 2: Sample responses y∼πθ(⋅∣x)y _θ(· x), x∼x 3: Compute sampled-token log-probabilities under πθ _θ, πT _T, and πref _ref 4: Compute a, r, δ δ, and detached q using Eqs. (1)–(7) 5: All-reduce masked sufficient statistics across data-parallel ranks 6: Compute ρb _b, sbs_b, and alignment RMS 7: Update B0B_0 during the first K0K_0 controller calls if auto-calibration is enabled 8: Update ρ¯b ρ_b and s¯b s_b using Eq. (10) 9: Compute γ~b γ_b and γb _b using Eqs. (11)–(12) 10: Construct AREOPDA^REOPD using Eqs. (5) and (13) 11: Update θ with the PPO-style policy objective 12: end for Experiments and Analysis We evaluate REOPD in single-teacher mathematics, single-teacher code generation, and mixed-domain multi-teacher distillation. Our experiments are designed to answer four questions: (i) whether REOPD improves over standard on-policy distillation; (i) whether it improves over fixed-coefficient ExOPD at λ=1.25λ=1.25; (i) how token-level compatibility and micro-batch-level budgeting affect performance; and (iv) how the adaptive controller evolves during training. Experimental Setup Models and data. We initialize both the student and the reference policy from Qwen3-4B (17). We use task-specialized Qwen3-4B non-thinking policies trained with reinforcement learning as the mathematics and code teachers. The mathematics run uses 57,046 level-6 examples from the filtered DeepMath-103K training set (6), while the code run uses 25,276 examples from the Eurus code training split (20). The multi-teacher set contains 25,276 examples from each domain. Each mixed-domain example carries a domain label that routes its teacher log-probability to the corresponding mathematics or code teacher. Baselines. We compare REOPD with standard OPD (λ=1λ=1) and fixed-coefficient ExOPD at λ=1.25λ=1.25. Standard OPD performs teacher alignment without the additional beyond-teacher residual, whereas ExOPD applies the same extrapolation strength to every sampled token. We use the same λ=1.25λ=1.25 baseline for mathematics and code in both the single- and multi-teacher settings, without post-hoc coefficient selection. Sensitivity results for other fixed coefficients are reported in Fig. 2. Mathematical Reasoning Code Generation Method AIME24 AIME25 HMMT25-F HMMT25-N Avg. HE+ MBPP+ LCB Avg. Reference Models Student 23.65 22.50 12.50 9.27 16.98 80.34 64.88 17.43 56.83 Teachers 58.02 54.58 32.50 38.85 45.99 86.59 70.11 27.57 63.49 Single-Teacher Distillation OPD 60.10 55.52 32.19 37.29 46.28 84.15 69.31 27.71 62.55 ExOPD (λ=1.25λ=1.25) 61.88 56.25 32.81 38.96 47.47 83.84 67.53 28.43 61.72 REOPD 61.98 55.31 34.69 38.65 47.66 85.67 70.57 27.29 63.45 Multi-Teacher Distillation OPD 58.85 56.35 31.46 39.06 46.43 84.76 68.65 26.29 61.99 ExOPD (λ=1.25λ=1.25) 62.92 56.04 31.98 36.98 46.98 86.43 68.78 28.14 62.90 REOPD 61.35 55.10 33.75 37.81 47.01 85.82 70.11 27.57 63.32 Table 1: Benchmark-level accuracy (%). HMMT25-F/N denote the February/November 2025 contests, and LCB denotes the test6 split of LiveCodeBench v6. The mathematics average pools 3,840 completions; the code average is weighted by task count over 2,868 completions. ExOPD uses λ=1.25λ=1.25; bold denotes the best result within each distillation setting. Figure 2: Aggregate performance across fixed ExOPD coefficients. Gray bars denote OPD, red hatched bars denote fixed ExOPD, and green dashed lines denote REOPD. Axes are truncated to expose setting-dependent differences. Training details. All main comparisons use the actor checkpoint at global step 50. Each step draws 1,024 prompts and one on-policy response per prompt. We optimize the student with AdamW (12) using a learning rate of 10−510^-5, weight decay 0.010.01, gradient clipping at 1.01.0, and one PPO epoch. The maximum prompt and response lengths are 2,048 and 16,384 tokens, respectively. Training uses BF16 mixed precision with FSDP on eight NVIDIA A100 80GB GPUs, and rollout generation uses tensor parallelism of size four. For REOPD, we set τ=0.007τ=0.007, γmax=1 _ =1, κ=0.5κ=0.5, β=0.95β=0.95, and βγ=0.9 _γ=0.9. The initial budget scale B0B_0 is automatically calibrated over the first ten controller calls. The mathematics run uses a five-call warm-up with γ=0.25γ=0.25; code and multi-teacher runs use no warm-up. The multi-teacher setting maintains one shared controller across the two domains. Unless otherwise stated, all methods use the same student initialization, teacher, training file, prompt budget, and step-50 checkpoint. Equal prompt budgets do not imply equal generated-token budgets because response lengths differ across methods. Ablation protocol. Component ablations are conducted in the single-teacher mathematics setting. All variants use the same mathematics training set, step-50 checkpoint, and evaluation protocol with seed 42. The no_q variant sets qt=1q_t=1 while retaining the adaptive micro-batch budget and its bound. The no_bound variant retains qtq_t and the adaptive controller but removes only the explicit upper bound on γb _b, while preserving the controller statistics, synchronization, EMA, and update frequency. The no_batch variant retains qtq_t but replaces γb _b with a fixed γ=λ0−1γ= _0-1. To measure its dependence on this fixed choice, we evaluate λ0∈1.0,1.25,1.5,1.75 _0∈\1.0,1.25,1.5,1.75\. Evaluation. For mathematics, we evaluate on AIME 2024, AIME 2025, HMMT February 2025, and HMMT November 2025. Each benchmark contains 30 problems, and we sample 32 responses per problem with temperature 1 and top-p=1p=1. We report pooled sample accuracy over all 3,840 completions. For code, we evaluate HumanEval+ (164 tasks) and MBPP+ (378 tasks) using EvalPlus (4; 3; 11), together with the 175-task test6 split of LiveCodeBench v6 (8). We draw four responses per problem using the same sampling temperature and report official pass@1. Our aggregate code score is the task-count-weighted accuracy over all 2,868 completions, rather than the unweighted mean of the three benchmark percentages. No external judge is used. Main Results Single-teacher distillation. As shown in Table 1, REOPD reaches 47.66% pooled sample accuracy on mathematics, exceeding both OPD at 46.28% and ExOPD at λ=1.25λ=1.25 at 47.47%. On code generation, REOPD obtains 63.45% weighted accuracy and remains comparable to G-OPD; under the common λ=1.25λ=1.25 baseline, it exceeds ExOPD at 61.72% and OPD at 62.55%. Multi-teacher distillation. When mathematics and code examples share one student and one REOPD controller, REOPD reaches 47.01% mathematics sample accuracy and 63.32% code weighted accuracy. Both results exceed OPD at 46.43% and 61.99%, as well as ExOPD at λ=1.25λ=1.25 at 46.98% and 62.90%, respectively. These results show that a shared adaptive controller can improve both domains without selecting separate fixed coefficients for mathematics and code. Sensitivity to a Global Extrapolation Coefficient Figure 2 exposes substantial variation in the preferred fixed coefficient. Single-teacher mathematics peaks at λ=1.25λ=1.25 with 47.47%, whereas single-teacher code peaks at λ=1.5λ=1.5 with 63.60%. In multi-teacher distillation, λ=1.25λ=1.25 and 1.751.75 tie on mathematics at 46.98%, while λ=1.25λ=1.25 is best on code at 62.90%. Performance is non-monotonic: increasing λ beyond its setting-specific optimum can reduce accuracy. To test whether REOPD’s adaptation can recover the effect of tuning a fixed coefficient for each setting, we compare it directly with the best ExOPD result from each G-OPD coefficient sweep. REOPD changes accuracy by +0.19+0.19, −0.15-0.15, +0.03+0.03, and +0.42+0.42 percentage points on single-teacher mathematics, single-teacher code, multi-teacher mathematics, and multi-teacher code, respectively. It therefore exceeds the corresponding best fixed-coefficient result in three of the four settings and trails it by only 0.15 points on single-teacher code. Overall, REOPD reaches comparable or better performance than the task-specific best fixed coefficient, showing that token-level compatibility and micro-batch budgeting can adapt the extrapolation strength effectively without a separate λ sweep for each setting. Taken together, the best fixed coefficient varies across task domains and teacher configurations, whereas REOPD replaces per-setting selection of a global λ with online adaptation. REOPD is not hyperparameter free: the controller retains hyperparameters shared across settings, and the mathematics run uses the warm-up schedule described in the Experimental Setup. Ablation Studies Variant Token q Adapt. γb _b λ0 _0 Acc. Full REOPD Yes Yes – 47.66 no_q – Yes – 43.39 no_bound Yes Yes – 47.16 no_batch Yes – 1.00 46.48 no_batch Yes – 1.25 47.66 no_batch Yes – 1.50 46.81 no_batch Yes – 1.75 46.68 Table 2: Component ablations on single-teacher mathematics only. no_q sets qt=1q_t=1; no_bound removes the upper bound on γb _b; and no_batch fixes γ=λ0−1γ= _0-1. Accuracy pools 3,840 completions. Bold marks the best no_batch result. Figure 3: Raw controller trajectories. Panel (a) reports the micro-batch budget γb _b; panel (b) reports mean compatibility qtq_t and effective coefficient λb,i,t=1+γbqb,i,t _b,i,t=1+ _bq_b,i,t. Table 2 isolates the two adaptation levels in the single-teacher mathematics setting. Removing token compatibility (no_q) reduces pooled accuracy from 47.66% to 43.39%, a drop of 4.27 percentage points. The decrease occurs on all four benchmarks (2.40–7.19 points). Thus, applying a shared micro-batch budget uniformly to all tokens is insufficient; token compatibility is the central component of REOPD. The no_bound variant isolates the explicit budget cap while preserving both levels of adaptation. It obtains 47.16% pooled accuracy (1,811/3,840), 0.50 percentage points below Full REOPD. Its benchmark accuracies are 61.04% on AIME 2024, 53.96% on AIME 2025, 34.58% on HMMT February 2025, and 39.06% on HMMT November 2025. Relative to Full REOPD, these scores are lower by 0.94, 1.35, and 0.11 points on the first three benchmarks, respectively, but higher by 0.41 points on HMMT November 2025. The cross-benchmark effect is therefore mixed, but the lower pooled accuracy suggests that the explicit bound provides a modest safeguard; its contribution is substantially smaller than that of token compatibility. The no_batch sweep tests the complementary question by retaining qtq_t while using a fixed base coefficient λ0 _0. Its pooled accuracy ranges from 46.48% to 47.66%. The best configuration uses λ0=1.25 _0=1.25 and reaches 47.66%, matching Full REOPD. Thus, adaptive micro-batch budgeting matches the best tuned token-only variant while avoiding a setting-specific λ0 _0 sweep. Together with the no_q and no_bound results, the ablation identifies token-level compatibility as the key component and the explicit bound as a modest safeguard, while the online micro-batch budget provides adaptive control without fixed-coefficient selection. Controller Dynamics The logged trajectories confirm that REOPD does not use a constant effective coefficient. Over the first ten steps, the mean budget γ is 0.608, 0.953, and 0.957 for mathematics, code, and multi-teacher training, respectively; over the final ten steps, these values increase to 1.000, 0.986, and 0.990. Meanwhile, mean token compatibility increases from 0.774/0.745/0.754 to 0.850/0.827/0.834, and the mean effective coefficient increases from 1.477/1.710/1.720 to 1.850/1.815/1.826. The controller therefore adapts most strongly early in training. Because the micro-batch budget approaches its upper bound later, the remaining late-stage variation is primarily supplied by the token-level compatibility weight. Conclusion and Limitations We studied reward extrapolation for on-policy distillation, where a fixed global coefficient applies the same beyond-teacher residual gain to every sampled token and must be selected separately for different training settings. REOPD replaces this global multiplier with a token-level compatibility weight and a bounded micro-batch extrapolation budget. It preserves standard teacher alignment, adapts only the additional teacher–reference residual, and requires no verifier or extra rollout. Across repeated experiments, REOPD outperforms G-OPD on single-teacher mathematics and on both multi-teacher domains, while achieving comparable performance on single-teacher code. In the mathematics ablation, removing qtq_t lowers accuracy by 4.27 points, removing the explicit budget bound lowers accuracy by 0.50 points, and the best token-only variant at λ0=1.25 _0=1.25 matches Full REOPD. These results identify token-wise residual filtering as the principal component, with the explicit bound providing a modest safeguard and online micro-batch budgeting preserving tuned performance without a setting-specific coefficient sweep. Several limitations remain. First, the compatibility weight is a sampled student–teacher discrepancy proxy rather than a correctness estimator; it cannot identify trajectories on which the student and teacher agree but are both wrong. Second, REOPD retains controller choices such as τ, γmax _ , and the calibration of B0B_0; its statistics depend on micro-batch composition, and the budget approaches its upper bound late in training. Future work should evaluate additional model families, scales, and teacher configurations, and compare shared versus per-teacher controllers to further test generality. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Introduction, Knowledge Distillation and On-Policy Distillation.. Akhondzadeh et al. (2026) M. S. Akhondzadeh, V. Lingam, A. Tejaswi, C. Ekbote, S. Sanghavi, and A. Bojchevski Reward-gated on-policy distillation. arXiv preprint arXiv:2607.04037. External Links: 2607.04037, Link Cited by: Adaptive and Reliability-Aware Distillation.. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. V. Le, and C. Sutton Program synthesis with large language models. arXiv preprint arXiv:2108.07732. External Links: 2108.07732, Link Cited by: Evaluation.. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. External Links: 2107.03374, Link Cited by: Evaluation.. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang MiniLLM: knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Knowledge Distillation and On-Policy Distillation.. He et al. (2025) Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, Z. Zhang, R. Wang, Z. Tu, H. Mi, and D. Yu DeepMath-103K: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456. External Links: 2504.11456, Link Cited by: Models and data.. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. External Links: 1503.02531, Link Cited by: Knowledge Distillation and On-Policy Distillation.. Jain et al. (2024) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974. External Links: 2403.07974, Link Cited by: Evaluation.. Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, p. 1317–1327. External Links: Document, Link Cited by: Knowledge Distillation and On-Policy Distillation.. Li et al. (2026) Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, and N. Ding Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016. External Links: 2604.13016, Link Cited by: Introduction, Knowledge Distillation and On-Policy Distillation.. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210. External Links: 2305.01210, Link Cited by: Evaluation.. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: Training details.. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. External Links: 1707.06347, Link Cited by: Final Objective and Optimization. Xie et al. (2026) X. Xie, Z. Xue, J. Wu, J. Li, Y. Wang, X. Hu, Y. Liu, and J. Zhang LLM-oriented token-adaptive knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence 40 (40), p. 34070–34078. External Links: Document, Link Cited by: Adaptive and Reliability-Aware Distillation.. Xu et al. (2026a) H. Xu, H. Wang, Y. Gao, J. Li, X. Zhang, and X. Yuan SG-OPD: sign-gated on-policy distillation via sign-consistency gating and phased teacher sampling. arXiv preprint arXiv:2606.09304. External Links: 2606.09304, Link Cited by: Adaptive and Reliability-Aware Distillation.. Xu et al. (2026b) Y. Xu, H. Sang, Z. Zhou, R. He, Z. Wang, and A. Geramifard TIP: token importance in on-policy distillation. arXiv preprint arXiv:2604.14084. External Links: 2604.14084, Link Cited by: Introduction, Knowledge Distillation and On-Policy Distillation., Adaptive and Reliability-Aware Distillation.. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. External Links: 2505.09388, Link Cited by: Models and data.. Yang et al. (2026a) W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin Learning beyond teacher: generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125. External Links: 2602.12125, Link Cited by: Introduction, Reward Extrapolation for On-Policy Distillation., Problem Setup. Yang et al. (2026b) Z. Yang, Z. Guo, Y. Song, M. Xu, Y. Wang, Y. Wang, X. Liang, and J. Tang Prune-OPD: efficient and reliable on-policy distillation for long-horizon reasoning. arXiv preprint arXiv:2605.07804. External Links: 2605.07804, Link Cited by: Adaptive and Reliability-Aware Distillation.. Yuan et al. (2024) L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, Z. Liu, B. Zhou, H. Peng, Z. Liu, and M. Sun Advancing LLM reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078. External Links: 2404.02078, Link Cited by: Models and data.. Zhang et al. (2026) M. Zhang, X. Zhou, Y. Luo, Y. Liu, S. Zhang, and W. Ye ASKD: reinforcement learning-style knowledge distillation with quality-adaptive skewness. Proceedings of the AAAI Conference on Artificial Intelligence 40 (41), p. 34781–34789. External Links: Document, Link Cited by: Adaptive and Reliability-Aware Distillation.. Zheng et al. (2026) B. Zheng, X. Ma, Y. Liang, J. Ruan, X. Fu, K. Lin, B. Zhu, K. Zeng, and X. Cai SCOPE: signal-calibrated on-policy distillation enhancement with dual-path adaptive weighting. arXiv preprint arXiv:2604.10688. External Links: 2604.10688, Link Cited by: Adaptive and Reliability-Aware Distillation.. Zheng et al. (2025) C. Zheng, Z. Wang, H. Ji, M. Huang, and N. Peng Model extrapolation expedites alignment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 1025–1041. External Links: Document, Link Cited by: Reward Extrapolation for On-Policy Distillation..

Instant research discovery