Paper deep dive

REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation

Yang Sun, Lichao Ma, Houyuan Qin, Yuxin Liu, Hanyang Lu, Yao Zhu, Pinlong Cai, Guohang Yan

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 39

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 8/14/2026, 4:14:32 AM

Summary

The paper introduces REOPD (Reliability-Adaptive Reward Extrapolation for On-Policy Distillation), a framework that improves upon standard On-Policy Distillation (OPD) and ExOPD by dynamically adjusting the reward extrapolation coefficient. Instead of using a single global scalar, REOPD employs a token-level compatibility weight and a batch-level adaptive budget to create a token-wise coefficient. This approach preserves teacher alignment while selectively extrapolating along reliable teacher-reference directions, mitigating reward hacking and training instability. Evaluations show REOPD outperforms G-OPD in mathematics and multi-teacher settings, and matches it in code generation, without requiring external verifiers or reward models.

Entities (11)

ReOPD · method · 98%On-Policy Distillation · method · 95%Qwen3-4B · model · 95%ExOPD · method · 92%AIME-2024 · benchmark · 90%DEEPMATH-103K · dataset · 90%Eurus · dataset · 90%G-OPD · method · 90%HumanEval · benchmark · 90%Reward Hacking · problem · 85%Teacher-Reference Log-Ratio · concept · 85%

Relation Signals (8)

ReOPD → usescomponent → Token-Level Compatibility Weight

confidence 95% · REOPD combines a token-level compatibility weight with a batch-level adaptive budget

ReOPD → usescomponent → Batch-Level Adaptive Budget

confidence 95% · REOPD combines a token-level compatibility weight with a batch-level adaptive budget

ReOPD → improves → G-OPD

confidence 90% · REOPD outperforms G-OPD on single-teacher mathematics and on both domains in the multi-teacher setting

ReOPD → mitigates → Reward Hacking

confidence 90% · yielding a token-wise coefficient ... that preserves teacher alignment while selectively extrapolating along reliable teacher-reference directions

ExOPD → suffersfrom → Reward Hacking

confidence 90% · This can drive the student to fit extreme peaks in the implicit reward, causing reward hacking and unstable training

Qwen3-4B → usedin → ReOPD

confidence 90% · We initialize both the student and the reference policy from Qwen3-4B

AIME 2024 → usedforevaluation → ReOPD

confidence 85% · For mathematics, we evaluate on AIME 2024

DEEPMATH-103K → usedfortraining → ReOPD

confidence 85% · The mathematics run uses 57,046 level-6 examples from the filtered DeepMath-103K training set

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:On-policy distillation (OPD) trains a student on its own trajectories under dense token-level supervision from a teacher. Reward-extrapolation methods such as ExOPD amplify the teacher-reference log-likelihood ratio to move beyond direct imitation, but apply a single global coefficient $\lambda$ to every token. This can drive the student to fit extreme peaks in the implicit reward, causing reward hacking and unstable training, and the optimal $\lambda$ varies across domains, requiring costly sweeps. We propose REOPD, a reliability-adaptive reward extrapolation framework for OPD. REOPD combines a token-level compatibility weight with a batch-level adaptive budget, yielding a token-wise coefficient $\lambda_{b,t}=1+\gamma_b q_t$ that preserves teacher alignment while selectively extrapolating along reliable teacher-reference directions. It requires no verifier, reward model, value model, or extra rollout beyond standard OPD. REOPD outperforms G-OPD on single-teacher mathematics and on both domains in the multi-teacher setting, while matching G-OPD on single-teacher code, demonstrating effective fine-grained reliability adaptation across domains and teacher configurations.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

37,633 characters extracted from source content.

Expand or collapse full text

REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation Yang Sun 1,* Lichao Ma 2,* Houyuan Qin 3,* Yuxin Liu 4 Hanyang Lu 5 Yao Zhu 6 Pinlong Cai 7 Guohang Yan 8 1 Shanghai Artificial Intelligence Laboratory 2 Peking University 3 Southwest Jiaotong University 4 School of Computer Science and Technology, University of Science and Technology of China 5 School of Automation Engineering, University of Electronic Science and Technology of China 6 Renmin University of China 7 Frontier Discovery Center, Shanghai Artificial Intelligence Laboratory 8 Autonomous Driving, Shanghai Artificial Intelligence Laboratory * Equal contribution. Email (Yang Sun): 25213080092@m.fudan.edu.cn Abstract On-policy distillation (OPD) trains a student on its own generated trajectories under dense token-level supervision from a teacher, providing an effective post-training paradigm for large language models. Reward-extrapolation methods such as ExOPD further amplify the teacher–reference log-likelihood ratio to move beyond direct imitation. However, ExOPD uses a single global scalarλto apply the same extrapolation strength indiscriminately to every token. This can drive the student to aggressively fit extreme peaks in the teacher–reference log-ratio that defines the implicit reward, resulting in reward hacking and unstable training. Moreover, the optimalλ varies across domains, requiring costly domain-specific sweeps that may still fail to identify an appropriate extrapolation strength. We propose REOPD, a Reliability-Adaptive Reward Extrapolation framework for On- Policy Distillation. REOPD combines a token-level compatibility weight with a batch-level adaptive budget. The former modulates token-wise residuals according to the student–teacher discrepancy, while the latter dynamically adjusts the overall extrapolation strength according to the reliability and scale of residual signals in each batch. Together, they yield a token-wise effective coefficientλ b,t = 1 +γ b q t , which preserves the original teacher-alignment term while selectively extrapolating along reliable teacher–reference directions. REOPD requires no additional verifier, reward model, value model, or rollout beyond the standard OPD pipeline. Evaluations show that REOPD outperforms G-OPD on single-teacher mathematics and on both domains in the multi-teacher setting, while matching G-OPD on single-teacher code. These results demonstrate the effectiveness of fine-grained reliability adaptation for reward extrapolation in on-policy distillation across task domains and teacher configurations. 1 Introduction On-policy distillation (OPD) trains a student on its own trajectories and queries the teacher on the same prefixes [1]. Unlike distillation on fixed teacher-generated data, OPD super- vises states that the current student actually visits and provides a dense learning signal at every sampled token. This makes it an effective post-training approach for transferring reasoning capabilities from specialized teachers. Yet dense supervision is heterogeneous: its usefulness depends not only on teacher quality, but also on the local compatibility between student and teacher [2, 3]. G-OPD interprets OPD as dense KL-regularized reinforce- ment learning, in which the teacher–reference log-ratio defines an implicit token-level reward [4]. Standard OPD corresponds to a reward coefficientλ= 1, whereas ExOPD setsλ >1 to extrapolate beyond direct teacher matching. ExOPD, however, applies one global coefficient to every token throughout training. Uniform scaling can overemphasize extreme teacher–reference log-ratios together with useful residuals, allowing a small num- ber of peaks to dominate policy updates and increasing the risk of reward hacking or unstable optimization. Moreover, the pre- ferred coefficient differs across domains, so each new setting requires multiple full training and evaluation runs to select λ. arXiv:2608.11698v2 [cs.LG] 13 Aug 2026 Preprint – REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation2 (a) Fixed-lambda ExOPD(b) REOPD: Reliability-Adaptive Extrapolation Beyond-teacher residual |r t | log-ratio peak Uniform gain (λ − 1) Scaled residual dominant peak Reward hacking Training instability Math · Code · Multi-teacher Task-specific λ sweeps Synchronized budget Compatibility proxy δ̂ t q 1 q 2 q 3 q 4 q 5 q 6 δ̂ t ↑ ⇒ q t ↓ detached stats Masked sufficient sums All-Reduce ρ b , s b → EMA Shared budget γ b ρ̄ b ↑ ⇒ γ b ↑ ; s̄ b ↑ ⇒ γ b ↓ 0 ≤ γ b ≤ γ max Residual |r t | × q t Token gate |q t r t | q t r t stays rank-local × γ b λ b,t = 1 + γ b q t REOPD cost C b,t = a b,t − γ b q t r t Teacher alignment a t remains unchanged Figure 1: Fixed-coefficient ExOPD applies the same residual gainλ−1 to every token and may amplify teacher–reference log-ratio peaks. REOPD preserves teacher alignment, gates the beyond-teacher residual with compatibilityq b,i,t , and derives a bounded micro-batch budget γ b from synchronized statistics. Our key observation is that teacher alignment and extrap- olation beyond the teacher should be controlled separately. A token with a large student–teacher discrepancy may still pro- vide a useful alignment signal, yet be unsuitable for amplified extrapolation. We therefore propose REOPD, which preserves the standard OPD alignment term and adapts only the additional teacher–reference residual. As shown in Figure 1, REOPD combines a token-level compatibility weight with a bounded micro-batch budget. Their product defines a token-wise effective coefficient, allowing reliable residuals to be emphasized without uniformly increasing every update. The method reuses the stu- dent, teacher, and reference log-probabilities already available in G-OPD and requires no verifier, reward model, value model, or additional rollout. We evaluate REOPD on mathematical reasoning, code gen- eration, and mixed-domain multi-teacher distillation. Across repeated experiments, REOPD outperforms G-OPD on single- teacher mathematics and on both mathematics and code in the multi-teacher setting, while achieving comparable performance to G-OPD on single-teacher code. These results indicate that online residual control can reduce reliance on setting-specific coefficient sweeps while maintaining competitive performance across task domains and teacher configurations. Our contributions are summarized as follows: • We formulate the indiscriminate amplification and domain-specific tuning of fixed-coefficient reward ex- trapolation as a residual-control problem. • We propose REOPD, which combines token-level com- patibility with a bounded micro-batch budget while preserving the standard OPD alignment term and re- quiring no external outcome supervision. •We evaluate REOPD against OPD and a fixed- coefficient sweep on mathematics, code, and multi- teacher distillation, showing competitive performance without selecting a task-specific global coefficient. 2 Related Work Knowledge Distillation and On-Policy Distillation.Knowl- edge distillation transfers teacher knowledge through output distributions, representations, or generated sequences [5,6]. Sequence-level distillation for autoregressive models is usually off-policy, creating a mismatch between teacher trajectories used for training and student prefixes encountered at inference. MiniLLM reduces this mismatch by optimizing reverse KL on student samples [7], while generalized knowledge distillation formalizes supervision on student-generated trajectories and in- cludes OPD as an on-policy instance [1]. Recent studies show that successful OPD also depends on student–teacher compati- bility and token-level utility, rather than teacher strength or dis- crepancy alone [2,3]. REOPD builds on this on-policy setting but targets the extrapolation residual rather than the complete distillation loss. Reward Extrapolation for On-Policy Distillation. G-OPD casts OPD as dense KL-regularized reinforcement learning [4]: the teacher–reference log-ratio serves as an implicit reward, and the student–reference KL constrains policy deviation. The formulation recovers OPD atλ= 1 and defines ExOPD byλ >1, where an additional teacher–reference residual is scaled by λ− 1. This objective-level extrapolation differs from weight-space methods such as ExPO, which extrapolate model parameters after preference optimization [8]. Although a suitableλcan improve distillation, fixed global scaling is sensitive to both Preprint – REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation3 log-ratio peaks and the task domain. REOPD directly extends ExOPD by replacing its global residual multiplier with online token- and micro-batch-level control. Adaptive and Reliability-Aware Distillation. Adaptive dis- tillation adjusts supervision according to training state or sample quality. AdaKD adapts token selection and temperature [9], while ASKD conditions a KL objective on externally provided sample quality [10]. Within OPD, TIP selects informative align- ment tokens [3], and Prune-OPD attenuates teacher supervision or truncates rollouts after local support drift [11]. These methods modify teacher alignment or the rollout process; REOPD instead preserves alignment and controls only the beyond-teacher resid- ual. Reliability can also be estimated from task-level feed- back. SCOPE routes verifier-labeled trajectories between self- reinforcement and OPD [12]; SG-OPD uses agreement between outcome and teacher signals and verified teacher rollouts [13]; and reward-gated OPD uses verifier feedback to regulate teacher guidance [14]. REOPD addresses the complementary setting without outcome labels or verifiers: it derives residual control solely from student, teacher, and reference log-probabilities already computed in white-box OPD. 3 REOPD: Reliability-Adaptive Reward Extrapolation 3.1 Problem Setup LetDdenote the prompt distribution. Given a prompt x ∼ D, the student policyπ θ generates an on-policy response y= (y 1 ,..., y T ), wherey∼ π θ (·| x). We denote the prefix state at tokentbys t = (x, y <t ). The teacher and reference policies are denoted byπ T andπ ref , respectively. In the multi-teacher setting, π T denotes the domain teacher routed to the current example. Let m t ∈0, 1 indicate whether y t is a valid response token. For a sampled tokeny t , we define the student–teacher align- ment cost as a t = logπ θ (y t | s t )− logπ T (y t | s t ).(1) Its expectation under the student distribution corresponds to the reverse KL divergence between the student and teacher ats t . Thus,a t is an alignment cost term; the PPO advantage is defined later as the negative total token cost. G-OPD further defines the teacher–reference log-ratio as a dense implicit reward [4]: r t = logπ T (y t | s t )− logπ ref (y t | s t ).(2) With a global reward coefficientλ, the sampled-token Ex- OPD cost can be written as C ExOPD t = a t − (λ− 1)r t .(3) Whenλ= 1, Equation(3)reduces to standard OPD. When λ >1, the residual term encourages the student to move beyond the teacher along the teacher–reference direction. However, the same multiplierλ−1 is applied to every token. This uniform scaling can amplify extreme implicit-reward peaks and requires λto be selected separately for different training settings. We therefore preservea t and replace only the residual multiplier with a bounded, data-dependent coefficient. 3.2 Method Overview Letbdenote a synchronized micro-batch whose sufficient statistics are aggregated across data-parallel ranks, and letiindex a response inb. REOPD constructs the effective extrapolation coefficient λ b,i,t = 1 + γ b q b,i,t ,(4) whereq b,i,t ∈(0,1] is a token-level compatibility weight and γ b ∈ [0,γ max ] is a shared micro-batch extrapolation budget. The resulting token cost is C REOPD b,i,t = a b,i,t − γ b q b,i,t r b,i,t .(5) Figure 1(b) separates REOPD into a token path and a con- troller path. The token path retains the local residualq b,i,t r b,i,t , whereas the controller path all-reduces detached sufficient statis- tics to obtain the shared budgetγ b . Their product determines how much of the extra residual is applied to each token. Neither path modifies the teacher-alignment cost a b,i,t . This formulation contains several useful special cases. Set- tingγ b = 0 recovers OPD. Settingq b,i,t = 1 andγ b =λ−1 recovers fixed-λExOPD. A fixedγwith adaptiveq b,i,t gives token-only control, whileq b,i,t = 1 with adaptiveγ b gives micro- batch-only control. 3.3 Token-Level Compatibility Weight REOPD estimates local student–teacher compatibility from log-probabilities already computed on sampled response tokens. We first construct the low-variance k 3 discrepancy proxy x b,i,t = logπ T (y i,t | s i,t )− logπ θ (y i,t | s i,t ), b δ b,i,t = exp(x b,i,t )− x b,i,t − 1. (6) The compatibility weight is then defined as q b,i,t = exp       − b δ b,i,t τ       , τ > 0.(7) Because b δ b,i,t ≥ 0, the weight lies in (0,1] in exact arith- metic. A small sampled discrepancy givesq b,i,t ≈1 and retains most of the extrapolation residual, whereas a large discrepancy yields a smaller weight. The temperatureτcontrols how rapidly this attenuation occurs. We emphasize thatq b,i,t measures local compatibility rather than task-level correctness. A small discrepancy does not guar- antee that the teacher is correct, and a large discrepancy does not makeits alignment signal useless. This is why REOPD applies q b,i,t only to the additional reward residual. The compatibility proxy and its resulting weight are detached from the computa- tion graph. Intermediate log-ratios are numerically bounded in Preprint – REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation4 implementation, but no full-vocabulary KL or additional teacher forward pass is required. 3.4 Micro-Batch Reliable Residual Statistics Token compatibility alone does not determine how much extrapolation the current micro-batch can support. REOPD therefore aggregates two statistics over valid response tokens. For compactness, let P b denote summation over all sequence– token pairs (i, t) in synchronized micro-batchb. We define the compatibility-weighted residual proportion as ρ b = P b m i,t |r b,i,t |q b,i,t P b m i,t |r b,i,t | + ε .(8) The statisticρ b ∈[0,1] measures the fraction of residual magnitude retained after compatibility weighting. We further define the reliable residual scale s b = P b m i,t (q b,i,t r b,i,t ) 2 P b m i,t + ε ! 1/2 .(9) Thus,ρ b captures the relative amount of compatible resid- ual, whereass b captures its absolute RMS scale. All sufficient statistics are summed across data-parallel ranks before the ratios are evaluated, so each rank uses the same controller output. To reduce micro-batch noise, REOPD maintains exponen- tial moving averages: ̄z b = β ̄z b−1 + (1− β)z b ,z∈ρ, s.(10) Both statistics and their moving averages are computed without gradient tracking. 3.5 Bounded Micro-Batch Extrapolation Budget REOPD converts the smoothed statistics into a target ex- trapolation budget: e γ b = clip B 0 ̄ρ b ̄s b + ε , 0,γ max ! .(11) A larger ̄ρ b permits stronger extrapolation when a larger fraction of the residual remains compatible. In contrast, a larger ̄s b reduces the coefficient so that a micro-batch with large resid- ual scale does not dominate the update. The upper boundγ max provides an explicit limit on extrapolation. The clipped target is further smoothed before being applied: γ b = β γ γ b−1 + (1− β γ ) e γ b .(12) Since e γ b ∈[0,γ max ], the smoothed budget remains in the same interval when initialized within it. When Equation(11) is not clipped andεis negligible, it approximately satisfies e γ b ̄s b ≈ B 0 ̄ρ b . This relation should be interpreted as an adaptive target rather than a strict constraint after clipping and smoothing. The scale B 0 can be specified directly. In automatic mode, REOPD initializes it during the firstK 0 controller updates from Algorithm 1 REOPD Training Input: Prompt distributionD; studentπ θ ; teacherπ T ; reference π ref ;τ,γ max ,β,β γ ; optionalB 0 or auto-calibration parameters κ, K 0 Output: Updated student policyπ θ 1: for each synchronized micro-batch b do 2:Sample responses y∼ π θ (·| x), x∼D 3: Compute sampled-token log-probabilities underπ θ ,π T , and π ref 4:Compute a, r, b δ, and detached q using Eqs. (1)–(7) 5:All-reduce masked sufficient statistics across data-parallel ranks 6:Compute ρ b , s b , and alignment RMS 7:UpdateB 0 during the firstK 0 controller calls if auto- calibration is enabled 8:Update ̄ρ b and ̄s b using Eq. (10) 9:Compute e γ b and γ b using Eqs. (11)–(12) 10:Construct A REOPD using Eqs. (5) and (13) 11:Update θ with the PPO-style policy objective 12: end for a scaled moving average of the alignment RMS,κ RMS b (a), and then keeps it fixed. An optional warm-up can holdγ b at a pre- scribed value during the initial training steps. The corresponding initialization length, smoothing coefficients, and warm-up con- figuration are reported with the training hyperparameters. 3.6 Final Objective and Optimization The PPO-style actor update [15] uses the negative token cost as its advantage: A REOPD b,i,t =−C REOPD b,i,t .(13) In exact arithmetic, Equations (4)–(12) imply 1≤ λ b,i,t ≤ 1 + γ max .(14) The advantage is masked to valid response tokens and passed to the same PPO-style policy surrogate used by the OPD baseline. The compatibility weights, sufficient statistics, mov- ing averages, and extrapolation budget are all treated as stop- gradient control signals. REOPD therefore changes only the con- struction of the token advantage and reuses the student, teacher, and reference evaluations already required by G-OPD. 4 Experiments and Analysis We evaluate REOPD in single-teacher mathematics, single- teacher code generation, and mixed-domain multi-teacher distil- lation. Our experiments are designed to answer four questions: (i) whether REOPD improves over standard on-policy distilla- tion; (i) whether it improves over fixed-coefficient ExOPD at λ= 1.25; (i) how token-level compatibility and micro-batch- level budgeting affect performance; and (iv) how the adaptive controller evolves during training. Preprint – REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation5 4.1 Experimental Setup Models and data. We initialize both the student and the ref- erence policy from Qwen3-4B [16]. We use task-specialized Qwen3-4B non-thinking policies trained with reinforcement learning as the mathematics and code teachers. The mathematics run uses 57,046 level-6 examples from the filtered DeepMath- 103K training set [17], while the code run uses 25,276 exam- ples from the Eurus code training split [18]. The multi-teacher set contains 25,276 examples from each domain. Each mixed- domain example carries a domain label that routes its teacher log- probability to the corresponding mathematics or code teacher. Baselines. We compare REOPD with standard OPD (λ= 1) and fixed-coefficient ExOPD atλ= 1.25. Standard OPD performs teacher alignment without the additional beyond- teacher residual, whereas ExOPD applies the same extrapolation strength to every sampled token. We use the sameλ= 1.25 baseline for mathematics and code in both the single- and multi- teacher settings, without post-hoc coefficient selection. Sensitiv- ity results for other fixed coefficients are reported in Fig. 2. Training details. All main comparisons use the actor check- point at global step 50. Each step draws 1,024 prompts and one on-policy response per prompt. We optimize the student with AdamW [19] using a learning rate of 10 −5 , weight decay 0.01, gradient clipping at 1.0, and one PPO epoch. The maximum prompt and response lengths are 2,048 and 16,384 tokens, re- spectively. Training uses BF16 mixed precision with FSDP on eight NVIDIA A100 80GB GPUs, and rollout generation uses tensor parallelism of size four. For REOPD, we setτ= 0.007,γ max = 1,κ= 0.5,β= 0.95, andβ γ = 0.9. The initial budget scaleB 0 is automatically calibrated over the first ten controller calls. The mathematics run uses a five-call warm-up withγ= 0.25; code and multi-teacher runs use no warm-up. The multi-teacher setting maintains one shared controller across the two domains. Unless otherwise stated, all methods use the same student initialization, teacher, training file, prompt budget, and step-50 checkpoint. Equal prompt budgets do not imply equal generated-token budgets because response lengths differ across methods. Ablation protocol.Component ablations are conducted in the single-teacher mathematics setting. All variants use the same mathematics training set, step-50 checkpoint, and evaluation protocol with seed 42. Theno_qvariant setsq t = 1 while retaining the adaptive micro-batch budget and its bound. The no_boundvariant retainsq t and the adaptive controller but re- moves only the explicit upper bound onγ b , while preserving the controller statistics, synchronization, EMA, and update fre- quency. Theno_batchvariant retainsq t but replacesγ b with a fixedγ=λ 0 −1. To measure its dependence on this fixed choice, we evaluate λ 0 ∈1.0, 1.25, 1.5, 1.75. Evaluation. For mathematics, we evaluate on AIME 2024, AIME 2025, HMMT February 2025, and HMMT November 2025. Each benchmark contains 30 problems, and we sample 32 responses per problem with temperature 1 and top-p= 1. We report pooled sample accuracy over all 3,840 completions. For code, we evaluate HumanEval+ (164 tasks) and MBPP+ (378 tasks) using EvalPlus [20,21,22], together with the 175- tasktest6split of LiveCodeBench v6 [23]. We draw four responses per problem using the same sampling temperature and report official pass@1. Our aggregate code score is the task-count-weighted accuracy over all 2,868 completions, rather than the unweighted mean of the three benchmark percentages. No external judge is used. 4.2 Main Results Single-teacher distillation. As shown in Table 1, REOPD reaches 47.66% pooled sample accuracy on mathematics, ex- ceeding both OPD at 46.28% and ExOPD atλ= 1.25 at 47.47%. On code generation, REOPD obtains 63.45% weighted accuracy and remains comparable to G-OPD; under the commonλ= 1.25 baseline, it exceeds ExOPD at 61.72% and OPD at 62.55%. Multi-teacher distillation. When mathematics and code ex- amples share one student and one REOPD controller, REOPD reaches 47.01% mathematics sample accuracy and 63.32% code weighted accuracy. Both results exceed OPD at 46.43% and 61.99%, as well as ExOPD atλ= 1.25 at 46.98% and 62.90%, respectively. These results show that a shared adaptive con- troller can improve both domains without selecting separate fixed coefficients for mathematics and code. 4.3 Sensitivity to a Global Extrapolation Coefficient Figure 2 exposes substantial variation in the preferred fixed coefficient. Single-teacher mathematics peaks atλ= 1.25 with 47.47%, whereas single-teacher code peaks atλ= 1.5 with 63.60%. In multi-teacher distillation,λ= 1.25 and 1.75 tie on mathematics at 46.98%, whileλ= 1.25 is best on code at 62.90%. Performance is non-monotonic: increasingλbeyond its setting-specific optimum can reduce accuracy. To test whether REOPD’s adaptation can recover the effect of tuning a fixed coefficient for each setting, we compare it di- rectly with the best ExOPD result from each G-OPD coefficient sweep. REOPD changes accuracy by +0.19,−0.15, +0.03, and +0.42 percentage points on single-teacher mathematics, single- teacher code, multi-teacher mathematics, and multi-teacher code, respectively. It therefore exceeds the corresponding best fixed- coefficient result in three of the four settings and trails it by only 0.15 points on single-teacher code. Overall, REOPD reaches comparable or better performance than the task-specific best fixed coefficient, showing that token-level compatibility and micro-batch budgeting can adapt the extrapolation strength ef- fectively without a separate λ sweep for each setting. Taken together, the best fixed coefficient varies across task domains and teacher configurations, whereas REOPD replaces per-setting selection of a globalλwith online adaptation. RE- OPD is not hyperparameter free: the controller retains hyperpa- rameters shared across settings, and the mathematics run uses the warm-up schedule described in Section 4.1. 4.4 Ablation Studies Table 2 isolates the two adaptation levels in the single- teacher mathematics setting. Removing token compatibility (no_q) reduces pooled accuracy from 47.66% to 43.39%, a Preprint – REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation6 Mathematical ReasoningCode Generation MethodAIME24 AIME25 HMMT25-F HMMT25-N Avg. HE+ MBPP+ LCB Avg. Reference Models Student23.6522.5012.509.2716.98 80.3464.8817.43 56.83 Teachers58.0254.5832.5038.8545.99 86.5970.1127.57 63.49 Single-Teacher Distillation OPD60.1055.5232.1937.2946.28 84.1569.3127.71 62.55 ExOPD (λ = 1.25)61.88 56.2532.8138.9647.47 83.8467.53 28.43 61.72 REOPD61.9855.3134.6938.6547.66 85.67 70.5727.29 63.45 Multi-Teacher Distillation OPD58.85 56.3531.4639.0646.43 84.7668.6526.29 61.99 ExOPD (λ = 1.25) 62.9256.0431.9836.9846.98 86.4368.78 28.14 62.90 REOPD61.3555.1033.7537.8147.01 85.82 70.1127.57 63.32 Table 1: Benchmark-level accuracy (%). HMMT25-F/N denote the February/November 2025 contests, and LCB denotes the test6split of LiveCodeBench v6. The mathematics average pools 3,840 completions; the code average is weighted by task count over 2,868 completions. ExOPD uses λ = 1.25; bold denotes the best result within each distillation setting. OPD1.251.501.75 45.5 46.0 46.5 47.0 47.5 48.0 Sample Accuracy (%) (a) Single-Teacher Math 46.28 47.47 46.61 45.99 47.66 OPD1.251.501.75 61.5 62.0 62.5 63.0 63.5 64.0 Weighted Accuracy (%) (b) Single-Teacher Code 62.55 61.72 63.60 62.83 63.45 OPD1.251.501.75 46.2 46.4 46.6 46.8 47.0 47.2 Sample Accuracy (%) (c) Multi-Teacher Math 46.43 46.98 46.48 46.98 47.01 OPD1.251.501.75 61.75 62.00 62.25 62.50 62.75 63.00 63.25 63.50 Weighted Accuracy (%) (d) Multi-Teacher Code 61.99 62.90 62.55 62.52 63.32 OPDExOPDREOPD Figure 2: Aggregate performance across fixed ExOPD coefficients. Gray bars denote OPD, red hatched bars denote fixed ExOPD, and green dashed lines denote REOPD. Axes are truncated to expose setting-dependent differences. drop of 4.27 percentage points. The decrease occurs on all four benchmarks (2.40–7.19 points). Thus, applying a shared micro-batch budget uniformly to all tokens is insufficient; token compatibility is the central component of REOPD. Theno_boundvariant isolates the explicit budget cap while preserving both levels of adaptation. It obtains 47.16% pooled accuracy (1,811/3,840), 0.50 percentage points below Full REOPD. Its benchmark accuracies are 61.04% on AIME Preprint – REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation7 Figure 3: Raw controller trajectories. Panel (a) reports the micro-batch budgetγ b ; panel (b) reports mean compatibilityq t and effective coefficient λ b,i,t = 1 + γ b q b,i,t . VariantToken q Adapt. γ b λ 0 Acc. Full REOPDYesYes–47.66 no_q–Yes–43.39 no_boundYesYes–47.16 no_batchYes–1.00 46.48 no_batchYes–1.25 47.66 no_batchYes–1.50 46.81 no_batchYes–1.75 46.68 Table 2: Component ablations on single-teacher mathematics only.no_qsetsq t = 1;no_boundremoves the upper bound onγ b ; andno_batchfixesγ=λ 0 −1. Accuracy pools 3,840 completions. Bold marks the best no_batch result. 2024, 53.96% on AIME 2025, 34.58% on HMMT February 2025, and 39.06% on HMMT November 2025. Relative to Full REOPD, these scores are lower by 0.94, 1.35, and 0.11 points on the first three benchmarks, respectively, but higher by 0.41 points on HMMT November 2025. The cross-benchmark effect is therefore mixed, but the lower pooled accuracy suggests that the explicit bound provides a modest safeguard; its contribution is substantially smaller than that of token compatibility. Theno_batchsweep tests the complementary question by retainingq t while using a fixed base coefficientλ 0 . Its pooled accuracy ranges from 46.48% to 47.66%. The best configuration usesλ 0 = 1.25 and reaches 47.66%, matching Full REOPD. Thus, adaptive micro-batch budgeting matches the best tuned token-only variant while avoiding a setting-specificλ 0 sweep. Together with theno_qandno_boundresults, the ablation iden- tifies token-level compatibility as the key component and the explicit bound as a modest safeguard, while the online micro- batch budget provides adaptive control without fixed-coefficient selection. 4.5 Controller Dynamics The logged trajectories confirm that REOPD does not use a constant effective coefficient. Over the first ten steps, the mean budgetγis 0.608, 0.953, and 0.957 for mathematics, code, and multi-teacher training, respectively; over the final ten steps, these values increase to 1.000, 0.986, and 0.990. Meanwhile, mean token compatibility increases from 0.774/0.745/0.754 to 0.850/0.827/0.834, and the mean effective coefficient increases from 1.477/1.710/1.720 to 1.850/1.815/1.826. The controller therefore adapts most strongly early in training. Because the micro-batch budget approaches its upper bound later, the remain- ing late-stage variation is primarily supplied by the token-level compatibility weight. 5 Conclusion and Limitations We studied reward extrapolation for on-policy distillation, where a fixed global coefficient applies the same beyond-teacher residual gain to every sampled token and must be selected sepa- rately for different training settings. REOPD replaces this global multiplier with a token-level compatibility weight and a bounded micro-batch extrapolation budget. It preserves standard teacher alignment, adapts only the additional teacher–reference residual, and requires no verifier or extra rollout. Across repeated exper- iments, REOPD outperforms G-OPD on single-teacher math- ematics and on both multi-teacher domains, while achieving comparable performance on single-teacher code. In the math- ematics ablation, removingq t lowers accuracy by 4.27 points, removing the explicit budget bound lowers accuracy by 0.50 points, and the best token-only variant atλ 0 = 1.25 matches Full REOPD. These results identify token-wise residual filtering as the principal component, with the explicit bound providing a modest safeguard and online micro-batch budgeting preserving tuned performance without a setting-specific coefficient sweep. Several limitations remain. First, the compatibility weight is a sampled student–teacher discrepancy proxy rather than a Preprint – REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation8 correctness estimator; it cannot identify trajectories on which the student and teacher agree but are both wrong. Second, REOPD retains controller choices such asτ,γ max , and the calibration ofB 0 ; its statistics depend on micro-batch composition, and the budget approaches its upper bound late in training. Future work should evaluate additional model families, scales, and teacher configurations, and compare shared versus per-teacher controllers to further test generality. References [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Pi- otr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language mod- els: Learning from self-generated mistakes.In The Twelfth International Conference on Learning Representa- tions, 2024. URLhttps://openreview.net/forum? id=3zKtaqxLhW. [2]Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethink- ing on-policy distillation of large language models: Phe- nomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026. URLhttps://arxiv.org/ abs/2604.13016. [3] Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. TIP: Token importance in on-policy distillation. arXiv preprint arXiv:2604.14084, 2026. URL https://arxiv.org/abs/2604.14084. [4]Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiy- ong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapo- lation. arXiv preprint arXiv:2602.12125, 2026. URL https://arxiv.org/abs/2602.12125. [5]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. URLhttps://arxiv.org/ abs/1503.02531. [6]Yoon Kim and Alexander M. Rush. Sequence-level knowl- edge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1139. URL https://aclanthology.org/D16-1139/. [7] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language mod- els. In The Twelfth International Conference on Learn- ing Representations, 2024. URLhttps://openreview. net/forum?id=5h0qf7IBZZ. [8] Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Model extrapolation expedites alignment. In Proceedings of the 63rd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pages 1025–1041, Vienna, Austria, 2025. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/ 2025.acl-long.51. URLhttps://aclanthology.org/ 2025.acl-long.51/. [9]Xurong Xie, Zhucun Xue, Jiafu Wu, Jian Li, Yabiao Wang, Xiaobin Hu, Yong Liu, and Jiangning Zhang. LLM-oriented token-adaptive knowledge distillation. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 40(40):34070–34078, 2026. doi: 10.1609/aaai. v40i40.40701. URLhttps://ojs.aaai.org/index. php/AAAI/article/view/40701. [10]Mingjie Zhang, Xiaoling Zhou, Yuxiao Luo, Yiyu Liu, Shikun Zhang, and Wei Ye.ASKD: Reinforcement learning-style knowledge distillation with quality-adaptive skewness. Proceedings of the AAAI Conference on Ar- tificial Intelligence, 40(41):34781–34789, 2026.doi: 10.1609/aaai.v40i41.40780. URLhttps://ojs.aaai. org/index.php/AAAI/article/view/40780. [11] Zhicheng Yang, Zhijiang Guo, Yifan Song, Minrui Xu, Yongxin Wang, Yiwei Wang, Xiaodan Liang, and Jing Tang. Prune-OPD: Efficient and reliable on-policy dis- tillation for long-horizon reasoning.arXiv preprint arXiv:2605.07804, 2026. URLhttps://arxiv.org/ abs/2605.07804. [12]Binbin Zheng, Xing Ma, Yiheng Liang, Jingqing Ruan, Xiaoliang Fu, Kepeng Lin, Benchang Zhu, Ke Zeng, and Xunliang Cai. SCOPE: Signal-calibrated on-policy dis- tillation enhancement with dual-path adaptive weighting. arXiv preprint arXiv:2604.10688, 2026. URLhttps: //arxiv.org/abs/2604.10688. [13]Haoran Xu, Hongyu Wang, Yifei Gao, Jiaze Li, Xiaofeng Zhang, and Xiaosong Yuan. SG-OPD: Sign-gated on- policy distillation via sign-consistency gating and phased teacher sampling. arXiv preprint arXiv:2606.09304, 2026. URL https://arxiv.org/abs/2606.09304. [14] Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, and Alek- sandar Bojchevski. Reward-gated on-policy distillation. arXiv preprint arXiv:2607.04037, 2026. URLhttps: //arxiv.org/abs/2607.04037. [15]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347. [16]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, et al. Qwen3 Technical Re- port.arXiv preprint arXiv:2505.09388, 2025.URL https://arxiv.org/abs/2505.09388. [17] Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. DeepMath-103K: A large- scale, challenging, decontaminated, and verifiable math- ematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456, 2025. URLhttps://arxiv.org/ abs/2504.11456. [18]Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruob- ing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees. arXiv preprint Preprint – REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation9 arXiv:2404.02078, 2024. URLhttps://arxiv.org/ abs/2404.02078. [19]Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learn- ing Representations, 2019. URLhttps://openreview. net/forum?id=Bkg6RiCqY7. [20]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. URLhttps://arxiv. org/abs/2107.03374. [21]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. URLhttps://arxiv. org/abs/2108.07732. [22] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Ling- ming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210, 2023. URL https://arxiv.org/abs/2305.01210. [23]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language mod- els for code. arXiv preprint arXiv:2403.07974, 2024. URL https://arxiv.org/abs/2403.07974.

Instant research discovery