Paper deep dive
On-policy Distillation with Verifiable Reward
Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, Shenzhi Wang, Bingxiang He, Gao Huang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.24696v1
- Canonical: https://arxiv.org/abs/2608.24696v1
Trouble viewing inline? Open PDF directly →
Full Text
52,651 characters extracted from source content.
Expand or collapse full text
marginparsep has been altered. topmargin has been altered. marginparpush has been altered. The page layout violates the ICML style. Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you. We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again. On-policy Distillation with Verifiable Reward Wenze Lin 1∗†^\,1\,*\, , Jiale Zhao 1,2∗^\,1,2\,*, Xitai Jiang 1∗^\,1\,*, Songde Rao 3, Yining Li 1, Shenzhi Wang 1, Bingxiang He 4, and Gao Huang 1🖂^\,1\, 1^1\,LeapLab, Tsinghua University 2^2\,Beihang University 3^3\,SMS, Peking University 4^4\,NLPLab, Tsinghua University ∗ Equal Contribution † Project Lead 🖂 Corresponding Author †footnotetext: Correspond to: linwz25@mails.tsinghua.edu.cn, gaohuang@tsinghua.edu.cn. Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards—thereby aligning the distillation signal with task success while preserving the teacher’s distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at https://github.com/LeapLabTHU/OPDVR. 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR) has become a prevalent and effective post-training paradigm for reasoning tasks (Guo et al., 2025; Shao et al., 2024; Jaech et al., 2024; Trinh et al., 2024; Yang et al., 2024). RLVR delivers explicit reward signals based on task outcomes, such as correct mathematical final answers or compilable code, which directly align model optimization with core task objectives. However, these rewards are typically sparse, providing little supervision for intermediate steps and making credit assignment challenging. Recently, On-policy Distillation (OPD) has emerged as a promising post-training paradigm (Xu et al., 2026; Xiao et al., 2026; Yang et al., 2025a; Zeng et al., 2026). Unlike RLVR, which relies on sparse outcome-level rewards, OPD leverages a teacher model to provide dense token-level guidance. However, OPD’s objective is purely distributional—it drives the student to mimic the teacher’s output distribution without considering whether the generated response is correct or incorrect, limiting the student’s performance to that of the teacher. The complementary limitations of RLVR and OPD make combining them a natural and promising direction, where OPD provides fine-grained dense guidance and RLVR offers reliable task-level correctness supervision. Existing approaches typically treat OPD and RLVR as two separate methods, either combining them explicitly or selectively applying one based on heuristic criteria. Despite their empirical effectiveness, such designs often introduce extra hyperparameters and heuristic trade-offs (Cai et al., 2026; Yang et al., 2026a; Li et al., 2026a; Wang et al., 2026). In this work, we propose On-policy Distillation with Verifiable Reward (OPDVR). We apply an extremely simple ReLU gating mechanism to sampled-token OPD, seamlessly combining OPD and RLVR without introducing any hyperparameters. We first revisit sampled-token OPD from an RLVR perspective and provide a mathematical reformulation of sample-token OPD’s reward. From the RLVR view, if we separate by trajectory correctness, sampled-token OPD can be viewed as applying token-level supervision by weighting binary task correctness rewards with teacher-student distribution discrepancies. Specifically, for sampled tokens that constitute correct reasoning trajectories, the model is assigned a token-level reward +1+1 weighted by log(πT/πθ) ( _T/ _θ); for tokens leading to incorrect trajectories, the model receives a reward −1-1 weighted by log(πθ/πT) ( _θ/ _T). However, we identify a critical limitation of this inherent reward design: both log(πT/πθ) ( _T/ _θ) and log(πθ/πT) ( _θ/ _T) are unbounded and can be either positive or negative. However, a key empirical principle shared by mainstream RLVR algorithms (Schulman et al., 2017; Guo et al., 2025; Shao et al., 2024) is that all tokens leading to a correct final outcome should receive non-negative advantages, while tokens leading to an incorrect outcome should receive non-positive advantages. This principle ensures that every token in a correct trajectory is treated as a valid prediction and reinforced accordingly, while every token in an incorrect trajectory is treated as an erroneous prediction and suppressed accordingly. Sampled-token OPD violates this principle: on correct trajectories, a negative value of log(πT/πθ) ( _T/ _θ) penalizes valid token behaviors, while on incorrect trajectories, a negative value of log(πθ/πT) ( _θ/ _T) encourages erroneous token predictions. Motivated by this gap, we use a ReLU gating mechanism to standardize the reward signs according to trajectory correctness: it enforces non-negative log(πT/πθ) ( _T/ _θ) values for all verified correct trajectories and ensures non-negative log(πθ/πT) ( _θ/ _T) values for all incorrect trajectories. This minimal correction directly converts conventional sampled-token OPD into a valid RLVR method while preserving the guidance from the teacher model. Specifically, for correct trajectories, larger log(πT/πθ) ( _T/ _θ) values—indicating accurate student predictions where the teacher is more confident than the student—yield stronger rewards. For incorrect trajectories, larger log(πθ/πT) ( _θ/ _T) values—corresponding to wrong student predictions where the student is more confident than the teacher—incur heavier penalties. This learning paradigm aligns well with human learning intuition, which reinforces reliable correct behaviors and suppresses overconfident erroneous predictions. Furthermore, by reformulating sampled-token OPD into an RLVR formulation, our OPDVR supports seamless integration with existing prominent RL algorithms, such as GRPO, DAPO and PPO. To instantiate this, we combine OPDVR with GRPO to obtain a new variant, which we call Group Relative Policy Distillation (GRPD). Overall, our method endows the model with dual supervisory signals: the rigorous task-level verifiable correctness from RLVR and the dense token-level distributional guidance from teacher distillation, leading to more robust and effective post-training optimization for reasoning tasks. Our experiments show that both OPDVR and GRPD consistently outperform OPD across all benchmarks. Figure 1: Left: Overview of OPDVR. The ReLU gating mechanism ensures correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards, while preserving the teacher’s distributional guidance. Right: Results on AIME24, AIME25, and AMC under the same-architecture setting (Qwen3-4B ← Qwen3-4B-RL), reported as avg@16 accuracy. 2 Related Work Reinforcement Learning with Verifiable Rewards (RLVR) Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a highly effective paradigm for post-training large language models (Schulman et al., 2017; Jaech et al., 2024; Trinh et al., 2024; Yang et al., 2024; Guo et al., 2025; Shao et al., 2024). RLVR leverages rule-based verifiers to provide objective, deterministic signals such as answer correctness or code compilation. Various extensions have been proposed to improve RLVR, such as controlling length (Liu et al., 2025; Sui et al., 2025; Xiang et al., 2025; Hammoud et al., 2025; Hou et al., 2025; Huang et al., 2026) and stabilizing entropy (Petrenko et al., 2026; Yang et al., 2025b; Jiang et al., 2025; Su et al., 2025). However, a fundamental limitation persists: the reward is sparse, leading to a severe credit assignment problem. On-policy Distillation On-policy Distillation (OPD) has recently emerged as an efficient post-training paradigm that provides dense, token-level supervisory signals by distilling a teacher model’s output distribution into the student (Xu et al., 2026; Xiao et al., 2026; Yang et al., 2025a; Zeng et al., 2026). Unlike RLVR’s sparse outcome-based rewards, OPD offers fine-grained guidance at every generation step. A growing line of work has explored various aspects of OPD, including its failure modes (Fu et al., 2026; Li et al., 2026b), how to determine which tokens are worth training (Xing et al., 2026; Jin et al., 2026), and its training stability issues in practice (Oh et al., 2026). However, OPD’s objective is purely distributional: it drives the student to mimic the teacher’s output distribution without considering whether the generated response is correct or incorrect. This observation naturally raises the question of whether OPD can be combined with RLVR, where trajectory correctness provides the ultimate optimization signal. Combining the dense supervision of OPD with the task-level verifiability of RLVR thus represents a promising and increasingly relevant direction. Combining OPD with RLVR Given the complementary strengths of OPD (dense token-level guidance) and RLVR (task-level correctness), several recent works have attempted to combine them. These include directly weighting the two objectives (Hubotter et al., 2026), applying OPD and RLVR separately depending on the sign of the advantage (Wang et al., 2026; Cai et al., 2026), aligning outcome rewards with process-level distillation signals (Hou et al., 2026), and leveraging teacher-student probability ratios for to weight the advantage in GRPO (Yang et al., 2026a). Despite their empirical gains, these approaches often treat OPD and RLVR as distinct objectives or losses that must be either explicitly combined or selectively applied based on heuristic criteria, often introducing additional hyperparameters for balancing the two terms, or relies on heuristic trade-offs. Instead, our method implicitly transforms sampled-token OPD into a proper RLVR method using a simple gated mechanism, while preserving the teacher’s distributional guidance—achieving the benefits of both paradigms without explicit multi-objective balancing. 3 Preliminaries 3.1 Reinforcement Learning with Verifiable Rewards (RLVR) Reinforcement Learning with Verifiable Rewards (RLVR) optimizes a policy using reward signals that can be automatically verified against ground truth. In mathematical reasoning tasks, the reward indicates whether the final answer is correct. A common choice for REINFORCE is R=+1R=+1 for a correct trajectory and R=−1R=-1 for an incorrect one, while many other RLVR methods (including GRPO) use R=+1R=+1 for correct and R=0R=0 for incorrect. Consider a policy πθ _θ that generates a response o=o1,…,o|o|o=\o_1,…,o_|o|\ given a prompt q. The RLVR objective is to maximize the expected reward: (θ)=o∼πθ(⋅∣q)[R],J(θ)=E_o _θ(· q) [R ], where R is the trajectory-level reward. To optimize this objective via gradient ascent, we compute the gradient: ∇θ(θ)=∇θo∼πθ(⋅∣q)[R]. _θJ(θ)= _θE_o _θ(· q) [R ]. Using the REINFORCE (log-derivative) trick, we rewrite this as: ∇θ(θ)=o∼πθ(⋅∣q)[R⋅∇θlogπθ(o∣q)], _θJ(θ)=E_o _θ(· q) [R· _θ _θ(o q) ], where logπθ(o∣q)=∑t=1|o|logπθ(ot∣q,o<t) _θ(o q)= _t=1^|o| _θ(o_t q,o_<t). We sample a single response o∼πθ(⋅∣q)o _θ(· q) to obtain a Monte Carlo estimate: ∇θ(θ)≈R⋅∑t=1|o|∇θlogπθ(ot∣q,o<t). _θJ(θ)≈ R· _t=1^|o| _θ _θ(o_t q,o_<t). This gradient can be implemented by minimizing the following loss over the sequence: ℒRLVR(θ)=−R⋅∑t=1|o|logπθ(ot∣q,o<t).L_RLVR(θ)=-R· _t=1^|o| _θ(o_t q,o_<t). When R=+1R=+1, minimizing this loss increases the probability of the entire response (reward). When R=−1R=-1, it decreases the probability (penalty). This formulation embodies the core RL principle: correct actions are rewarded, incorrect ones are penalized. The REINFORCE formulation above suffers from high variance, as it relies on a single trajectory-level reward. A more stable alternative is Group Relative Policy Optimization (GRPO) (Guo et al., 2025; Shao et al., 2024). Specifically, for a group of G responses o1,…,oG\o_1,…,o_G\ sampled from the policy, GRPO normalizes each response’s reward against the group mean and standard deviation: A^i,t=Ri−mean(Rjj=1G)std(Rjj=1G). A_i,t= R_i-mean(\R_j\_j=1^G)std(\R_j\_j=1^G). Note that GRPO typically uses R∈0,1R∈\0,1\ (1 for correct, 0 for incorrect) rather than −1,+1\-1,+1\, since the group-relative normalization naturally centers the advantages around zero. This group-relative advantage reduces variance and serves as a stable, computationally efficient alternative to direct binary rewards. For simplicity, we present the on-policy version of GRPO without clipping or KL penalty; the loss is: ℒGRPO(θ)=−1G∑i=1G1|oi|∑t=1|oi|A^i,tlogπθ(oi,t∣q,oi,<t).L_GRPO(θ)=- 1G _i=1^G 1|o_i| _t=1^|o_i| A_i,t _θ(o_i,t q,o_i,<t). 3.2 On-policy Distillation On-policy Distillation (OPD) optimizes a student policy πθ _θ toward a teacher policy πT _T by minimizing their reverse KL divergence: minθo∼πθ(⋅∣q)[∑t=1|o|DKL(πθ(⋅∣q,o<t)∥πT(⋅∣q,o<t))]. _θ\ E_o _θ(· q) [ _t=1^|o|D_KL( _θ(· q,o_<t) _T(· q,o_<t)) ]. The full-vocabulary OPD computes this exact KL at every token position: ℒOPDfull(θ)=o∼πθ(⋅∣q)[∑t=1|o|∑v∈πθ(v∣q,o<t)logπθ(v∣q,o<t)πT(v∣q,o<t)].L_OPD^full(θ)=E_o _θ(· q) [ _t=1^|o| _v _θ(v q,o_<t) _θ(v q,o_<t) _T(v q,o_<t) ]. This loss is directly minimized. In practice, the log-ratio log(πθ/πT) ( _θ/ _T) is treated as a constant reward with stop-gradient when computing gradients, which is equivalent to the exact gradient of the reverse KL via the log-derivative trick. However, computing the reverse KL over the full vocabulary at every token position is computationally expensive, especially for large language models with very large vocabularies. A practical compromise is Top-k OPD, which restricts the KL computation to the k tokens with the highest student probabilities: ℒOPDtop-k(θ)=o∼πθ(⋅∣q)[∑t=1|o|DKL(π¯θ(St)∥π¯T(St))],L_OPD^top-k(θ)=E_o _θ(· q) [ _t=1^|o|D_KL ( π_θ^(S_t) π_T^(S_t) ) ], where StS_t is the set of top-k tokens. The most widely adopted variant is sampled-token OPD. It uses a single Monte Carlo sample ot∼πθ(⋅∣q,o<t)o_t _θ(· q,o_<t) to approximate the reverse KL gradient: ℒOPDsample(θ)=o∼πθ(⋅∣q)[∑t=1|o|logπθ(ot∣q,o<t)πT(ot∣q,o<t)].L_OPD^sample(θ)=E_o _θ(· q) [ _t=1^|o| _θ(o_t q,o_<t) _T(o_t q,o_<t) ]. The log-ratio log(πθ/πT) ( _θ/ _T) is treated as a constant reward with stop-gradient. 4 Method 4.1 Revisiting Sampled-Token OPD from an RLVR Perspective We begin with the sampled-token OPD loss. Recall that sampled-token OPD approximates the reverse KL gradient using a single Monte Carlo sample. The loss over the entire response o=o1,…,o|o|o=\o_1,…,o_|o|\ is defined as: ℒOPDsample(θ)=o∼πθ(⋅∣q)[∑t=1|o|logπθ(ot∣q,o<t)πT(ot∣q,o<t)].L_OPD^sample(θ)=E_o _θ(· q) [ _t=1^|o| _θ(o_t q,o_<t) _T(o_t q,o_<t) ]. For a sampled response o∼πθ(⋅∣q)o _θ(· q), the per-token loss is: ℓt=logπθ(ot∣q,o<t)πT(ot∣q,o<t). _t= _θ(o_t q,o_<t) _T(o_t q,o_<t). The gradient of the sequence-level loss with respect to θ is: ∇θℒOPDsample(θ)=∑t=1|o|logπθ(ot∣q,o<t)πT(ot∣q,o<t)⋅∇θlogπθ(ot∣q,o<t). _θL_OPD^sample(θ)= _t=1^|o| _θ(o_t q,o_<t) _T(o_t q,o_<t)· _θ _θ(o_t q,o_<t). We now compare this with the standard RLVR gradient. Recall from Section 3 that the RLVR loss for a trajectory with reward R is: ℒRLVR(θ)=−R⋅∑t=1|o|logπθ(ot∣q,o<t).L_RLVR(θ)=-R· _t=1^|o| _θ(o_t q,o_<t). Its gradient is: ∇θℒRLVR(θ)=−R⋅∑t=1|o|∇θlogπθ(ot∣q,o<t). _θL_RLVR(θ)=-R· _t=1^|o| _θ _θ(o_t q,o_<t). We observe that the sampled-token OPD gradient shares the same form as the RLVR gradient, where the log-ratio term corresponds to the reward coefficient. By matching the coefficients of ∇θlogπθ(ot|q,o<t) _θ _θ(o_t|q,o_<t), we obtain: −R=logπθ(ot∣q,o<t)πT(ot∣q,o<t).-R= _θ(o_t q,o_<t) _T(o_t q,o_<t). this observation allows us to reinterpret the log-ratio term as an implicit token-level reward ROPD(ot)R_OPD(o_t) ROPD(ot)=−logπθ(ot∣q,o<t)πT(ot∣q,o<t)=logπT(ot∣q,o<t)πθ(ot∣q,o<t).R_OPD(o_t)=- _θ(o_t q,o_<t) _T(o_t q,o_<t)= _T(o_t q,o_<t) _θ(o_t q,o_<t). However, unlike RLVR, where the reward sign is determined by the verifier outcome, the sign of Ropd is solely determined by the teacher-student probability ratio. To analyze their alignment, we consider two cases based on trajectory correctness: ROPD(ot)=logπT(ot∣q,o<t)πθ(ot∣q,o<t)⋅(+1),trajectory correct,logπθ(ot∣q,o<t)πT(ot∣q,o<t)⋅(−1),trajectory incorrect.R_OPD(o_t)= cases _T(o_t q,o_<t) _θ(o_t q,o_<t)·(+1),&trajectory correct,\\[10.0pt] _θ(o_t q,o_<t) _T(o_t q,o_<t)·(-1),&trajectory incorrect. cases In other words, on correct trajectories, the gradient is weighted by log(πT/πθ) ( _T/ _θ); on incorrect trajectories, it is weighted by log(πθ/πT) ( _θ/ _T) with a negative sign. The empirical RLVR principle requires that correct trajectories receive positive rewards and incorrect trajectories receive negative rewards. However, the sign of log(πT/πθ) ( _T/ _θ) is determined by whether πT>πθ _T> _θ or πT<πθ _T< _θ, which is independent of trajectory correctness. This creates two failure cases: correct trajectories can receive negative rewards when πT<πθ _T< _θ, and incorrect trajectories can receive positive rewards when πT>πθ _T> _θ. Both violate the RL principle. 4.2 OPDVR: A Simple Gated Mechanism Figure 2: OPDVR method We propose On-policy Distillation with Verifiable Reward (OPDVR) to resolve this issue. The solution is extremely simple: apply a ReLU gate to enforce RLVR compliance on the sampled token while preserving the teacher’s distributional guidance. The reward can be written as: ROPDVR(ot)=max(0,logπT(ot∣q,o<t)πθ(ot∣q,o<t))⋅(+1),trajectory correct,max(0,logπθ(ot∣q,o<t)πT(ot∣q,o<t))⋅(−1),trajectory incorrect.R_OPDVR(o_t)= cases (0, _T(o_t q,o_<t) _θ(o_t q,o_<t) )·(+1),&trajectory correct,\\[12.0pt] (0, _θ(o_t q,o_<t) _T(o_t q,o_<t) )·(-1),&trajectory incorrect. cases The corresponding loss is: ℒOPDVR(θ)=−∑t=1|o|ROPDVR(ot)⋅logπθ(ot∣q,o<t).L_OPDVR(θ)=- _t=1^|o|R_OPDVR(o_t)· _θ(o_t q,o_<t). This gated mechanism ensures that: • Correct trajectories receive non-negative rewards, and incorrect trajectories receive non-positive rewards, aligning the reward sign with trajectory correctness. • For tokens on correct trajectories, the reward magnitude is larger when the teacher is more confident than the student, i.e., log(πT/πθ) ( _T/ _θ) is larger. This encourages the model to reinforce choices that are both correct and reliable according to the teacher. • For tokens on incorrect trajectories, the penalty magnitude is larger when the student is more confident than the teacher, i.e., log(πθ/πT) ( _θ/ _T) is larger. This forces the model to suppress overconfident mistakes. 4.3 Interpreting the ReLU Gating Mechanism as a Conditional Mask In this subsection, we interpret the ReLU gating mechanism in OPDVR as a conditional token mask: it selectively zeroes out gradient updates on tokens whose learning direction conflicts with the verifier signal. Standard sampled-token OPD lacks this mask, so it inevitably updates all tokens based solely on the teacher-student confidence ratio, regardless of trajectory correctness. To see this, recall the standard OPD gradient for a sampled token a: gOPD=[logπTπθ]+∇θlogπθ(a)⏟Term A: push up (teacher more confident)−[logπθπT]+∇θlogπθ(a)⏟Term B: pull down (student more confident).g_OPD= [ _T _θ ]_+ _θ _θ(a)_Term A: push up (teacher more confident)- [ _θ _T ]_+ _θ _θ(a)_Term B: pull down (student more confident). (1) The OPDVR gradient instead applies the ReLU gate conditioned on the verifier reward R∈+1,−1R∈\+1,-1\: gOPDVR=Term A,R=+1,−Term B,R=−1.g_OPDVR= casesTerm A,&R=+1,\\ -Term B,&R=-1. cases (2) Thus, compared to standard OPD, OPDVR removes exactly two types of tokens whose updates conflict with the verifier: • Type I Conflicting Token (Correct Trajectory, πθ>πT _θ> _T): When the trajectory is correct (R=+1R=+1), the student has already produced the right answer. If the student is more confident than the teacher on a specific token (πθ>πT _θ> _T), then Term B is activated. Standard OPD applies a negative gradient, pulling down this correct token’s probability to match the teacher’s lower confidence. This update works against the verifier signal in two ways: 1. The student’s higher confidence is consistent with the correct outcome. 2. Reducing the student’s confidence on a correct answer introduces a distributional distortion that is not driven by task performance. • Type I Conflicting Token (Incorrect Trajectory, πT>πθ _T> _θ): When the trajectory is incorrect (R=−1R=-1), the student has produced a wrong answer. If the teacher is more confident than the student on a specific token (πT>πθ _T> _θ), then Term A is activated. Standard OPD applies a positive gradient, pushing up this incorrect token’s probability to align with the teacher. This update works against the verifier signal in two ways: 1. The teacher’s high confidence does not accompany a correct trajectory here. 2. Increasing the probability of tokens on an incorrect trajectory reinforces a reasoning pattern that the verifier has marked as wrong. By masking out these two token types, OPDVR preserves the student’s correct high-confidence predictions and withholds the teacher’s guidance on wrong answers. The teacher still controls the magnitude of the update via the log-ratio, but the verifier now determines the direction—reinforce or suppress—so that the distillation process never works against the task reward. We provide a further analysis in Appendix A. 4.4 Group Relative Policy Distillation (GRPD) Having established that OPDVR transforms sampled-token OPD into a proper RLVR method with a binary verifier reward, a natural extension is to replace this coarse binary signal with a more nuanced, group-relative advantage estimate like GRPO (Guo et al., 2025; Shao et al., 2024). Recall from Section 3 that GRPO computes a group-relative advantage A^i,t A_i,t for each token position t in response oio_i: A^i,t=Ri−mean(Rjj=1G)std(Rjj=1G), A_i,t= R_i-mean(\R_j\_j=1^G)std(\R_j\_j=1^G), where Ri∈0,1R_i∈\0,1\ is the verifier reward for response oio_i, and G is the group size. The key property of A^i,t A_i,t is that it is positive for responses that are better than the group average, and negative for those that are worse—capturing relative performance within the sampled batch. We now apply the same ReLU gating logic, but with the binary correctness sign R replaced by the group-relative advantage A^i,t A_i,t. The GRPD reward becomes: RGRPD(oi,t)=max(0,logπT(oi,t∣q,oi,<t)πθ(oi,t∣q,oi,<t))⋅(+1),A^i,t>0,max(0,logπθ(oi,t∣q,oi,<t)πT(oi,t∣q,oi,<t))⋅(−1),A^i,t<0.R_GRPD(o_i,t)= cases \! (0, _T(o_i,t q,o_i,<t) _θ(o_i,t q,o_i,<t) )·(+1),& A_i,t>0,\\[10.0pt] \! (0, _θ(o_i,t q,o_i,<t) _T(o_i,t q,o_i,<t) )·(-1),& A_i,t<0. cases Equivalently, this can be written compactly as: RGRPD(oi,t)=sign(A^i,t)⋅ReLU(sign(A^i,t)⋅logπT(oi,t∣q,oi,<t)πθ(oi,t∣q,oi,<t)),R_GRPD(o_i,t)=sign( A_i,t)·ReLU\! (sign( A_i,t)· _T(o_i,t q,o_i,<t) _θ(o_i,t q,o_i,<t) ), where sign(A^i,t)sign( A_i,t) ensures the direction of the update is governed by the advantage sign. The corresponding loss is: ℒGRPD(θ)=−1G∑i=1G1|oi|∑t=1|oi|RGRPD(oi,t)⋅logπθ(oi,t∣q,oi,<t).L_GRPD(θ)=- 1G _i=1^G 1|o_i| _t=1^|o_i|R_GRPD(o_i,t)· _θ(o_i,t q,o_i,<t). 5 Experiments 5.1 Main Experimental Settings We conduct experiments on both same-architecture and cross-architecture distillation settings to evaluate the effectiveness of our method. Same-architecture setting. We use Qwen3-4B-nonthinking as the student model and distill it from a teacher model of the same architecture, which is obtained by training Qwen3-4B with GRPO on the DeepMath dataset (He et al., 2026). Following Yang et al. (2026b), we use the filtered subset of the DeepMath dataset consisting of 57k samples with difficulty level ≥6≥ 6. Cross-architecture setting. We use the DAPO-Math-17k dataset (Yu et al., 2026). The teacher model is Qwen3-4B-base, fine-tuned with GRPO for 3 epochs on the same dataset. The student model is Qwen3-1.7B-base. The distillation training runs for 3 epochs. Benchmarks. We evaluate all models on six reasoning benchmarks: AIME24, AIME25, AMC, MATH500, Minerva, and OlympiadBench. 5.2 Main Results We compare our method OPDVR against standard sampled-token OPD and top-64 OPD on both same-architecture and cross-architecture distillation settings. Tables 1 and 2 report the performance on six reasoning benchmarks. Table 1: Results on same-architecture distillation (Qwen3-4B ← Qwen3-4B-RL). All models are evaluated with avg@16 accuracy. Method AIME24 AIME25 AMC MATH500 Minerva OlympiadBench Avg. Student (Qwen3-4B) 24.0 15.8 60.8 80.9 27.6 42.9 42.0 Teacher (Qwen3-4B-RL) 36.0 29.0 65.9 87.0 35.4 49.3 50.4 Sampled-Token OPD 34.2 26.0 63.1 85.5 31.6 46.5 47.8 Top-64 OPD 34.6 23.5 62.0 85.0 32.2 46.8 47.4 OPDVR (Ours) 36.9 28.1 64.8 84.7 33.2 47.0 49.1 Table 2: Results on cross-architecture distillation (Qwen3-1.7B-Base ← Qwen3-4B-Base-RL). All models are evaluated with avg@16 accuracy. Method AIME24 AIME25 AMC MATH500 Minerva OlympiadBench Avg. Student (Qwen3-1.7B-Base) 4.1 1.7 23.2 48.9 8.9 17.1 17.3 Teacher (Qwen3-4B-Base-RL) 10.6 13.1 40.3 74.2 17.2 30.0 30.9 Sampled-Token OPD 6.5 2.1 24.8 59.1 11.5 21.6 20.9 Top-64 OPD 8.5 3.3 26.4 60.1 10.7 21.4 21.7 OPDVR (Ours) 8.5 3.3 30.3 60.8 11.6 22.0 22.8 As shown in both tables, OPDVR consistently outperforms standard sampled-token OPD and top-64 OPD across all six benchmarks in both distillation settings. In the same-architecture setting, OPDVR achieves gains of 2.7 points on AIME24 and 2.1 points on AIME25 over sampled-token OPD, and even surpasses the teacher model on AIME24. In the cross-architecture setting, OPDVR delivers substantial improvements over the baselines, with gains of 5.5 points on AMC and 1.7 points on MATH500 over sampled-token OPD, highlighting its robustness to architectural differences. 5.3 Group Relative Policy Distillation To further validate the effectiveness of replacing the binary verifier reward with group-relative advantages, we instantiate OPDVR with GRPO-style advantage estimation (describe in Section 4.4) and evaluate it under the same-architecture setting. The teacher model is the same Qwen3-4B-Nonthinking model trained with GRPO on DeepMath. The student model is Qwen3-4B-nonthinking. To more cleanly evaluate the effectiveness of replacing binary rewards with group-relative advantages, we conduct this experiment on DAPO-Math-17K, distinct from the teacher’s DeepMath training data. The group size is set to G=8G=8. We compare GRPD against GRPO and standard sampled-token OPD. Table 3 reports the results. GRPD consistently outperforms both GRPO and OPD across all six benchmarks, with notable gains of 6.5 points on AIME24 and 10.9 points on AIME25 over GRPO. It also surpasses OPD on five out of six benchmarks, achieving a 2.8-point improvement on AIME24. These results demonstrate that combining group-relative advantage estimation with the ReLU-gated distillation signal yields stronger and more stable improvements than either method alone. Table 3: Results on Group Relative Policy Distillation (Qwen3-4B ← Qwen3-4B-RL). All models are evaluated with avg@16 accuracy. Method AIME24 AIME25 AMC MATH500 Minerva OlympiadBench Avg. Student (Qwen3-4B) 24.0 15.8 60.8 80.9 27.6 42.9 42.0 Teacher (Qwen3-4B-RL) 36.0 29.0 65.9 87.0 35.4 49.3 50.4 GRPO 28.3 20.8 62.3 83.9 28.9 44.6 44.8 OPD 32.0 31.7 65.6 85.4 28.9 46.6 48.4 GRPD (Ours) 34.8 31.7 67.0 85.6 30.5 47.0 49.4 5.4 Ablation Study: Inverse-Gated Experiment Figure 3: Ablation study on the gating mechanism. Left: training-time accuracy reward of OPD, OPDVR, and the inverse-gated variant. Right: average accuracy over six benchmarks. Table 4: Ablation study on the gating mechanism. We compare OPD, OPDVR, and an inverse-gated variant where the ReLU gate is applied in the opposite direction. Method AIME24 AIME25 AMC MATH500 Minerva OlympiadBench Avg. Student (Qwen3-4B) 24.0 15.8 60.8 80.9 27.6 42.9 42.0 Teacher (Qwen3-4B-RL) 36.0 29.0 65.9 87.0 35.4 49.3 50.4 OPD 34.2 26.0 63.1 85.5 31.6 46.5 47.8 OPDVR (Ours) 36.9 28.1 64.8 84.7 33.2 47.0 49.1 Inverse-Gated 30.3 21.2 62.3 83.6 27.7 42.8 44.6 To examine the effectiveness of the gating mechanism in OPDVR, we conduct an inverse-gated ablation under the same-architecture setting (Qwen3-4B ← Qwen3-4B-RL) on DeepMath, following the same experimental setup as the main experiments.. Recall that OPDVR keeps a token only when its update direction agrees with the verifier signal: on a correct trajectory (R=+1R=+1), tokens where the teacher is more confident than the student (πT>πθ _T> _θ) are kept and tokens where the student is already more confident (πθ>πT _θ> _T) are gated out; on an incorrect trajectory (R=−1R=-1), the roles are reversed – tokens where the student is more confident than the teacher are kept, and tokens where the teacher is more confident are gated out. The inverse-gated variant swaps these two sets exactly: it keeps the tokens that OPDVR gates out (the student-more-confident tokens on correct trajectories and the teacher-more-confident tokens on incorrect trajectories), and gates out the tokens that OPDVR keeps, while keeping the masking ratio and everything else unchanged. Table 4 shows the final benchmark results, and Figure 3 tracks the accuracy reward on the training set. The three variants start from the same point and separate monotonically throughout training, yielding a consistent ordering OPDVR >> OPD >> Inverse-Gated in both the training curves and the final benchmarks. Reversing the gate falls below vanilla OPD on all six benchmarks, confirming the effectiveness of the gating mechanism. Notably, even the inverse-gated variant still improves over the initial model, indicating that the teacher’s distributional guidance remains useful—though its benefits are substantially hindered when the sign is misaligned with trajectory correctness (a) Training dynamic of same-architecture distillation (Qwen3-4B ← Qwen3-4B-RL). (b) Training dynamic of cross-architecture distillation (Qwen3-1.7B-Base ← Qwen3-4B-Base-RL). Figure 4: Training dynamics 5.5 Training Dynamics To understand how token-level gating shapes the distillation process, we track three quantities throughout training: the student policy entropy, the average response length, and the zero-gated token ratio, i.e., the fraction of tokens whose distillation loss is masked by the gate. Figure 4 shows the dynamics of the same-architecture setting (Qwen3-4B ← Qwen3-4B-RL) and the cross-architecture setting (Qwen3-1.7B-Base ← Qwen3-4B-Base-RL), respectively. The entropy and response-length trajectories of the sampled-token OPD baseline show no consistent pattern across settings; instead, they depend strongly on the teacher–student pair. In the same-architecture setting, the entropy of both methods drifts mildly upward (from ∼ 0.33 to ∼ 0.40) while the response length inflates by more than four times, from roughly 1.6k to over 6.7k tokens. In the cross-architecture setting, the picture inverts: the student entropy collapses rapidly from ∼ 2.0 at the start of training and the response length stays nearly flat throughout. These quantities therefore do not follow a setting-independent trend – their evolution is dictated by the specific teacher and student models rather than by the distillation objective itself. In contrast, the zero-gated token ratio is consistent across both settings. It remains in a band around fifty percent during the entire course of training (≈ 0.48–0.50 for the 4B student and ≈ 0.40–0.44 for the 1.7B student), never degenerating toward the trivial extremes of gating all or no tokens. This means that roughly half of the sampled tokens are zeroed by the ReLU gate at any point of training. Recalling the failure cases identified in Section 4, these are exactly the tokens whose implicit OPD reward pushes against the RLVR direction: on correct trajectories where the student already assigns higher probability than the teacher (πθ>πT _θ> _T), and on incorrect trajectories where the teacher still outranks the student (πT>πθ _T> _θ). The stable ratio indicates that such RLVR-violating tokens constitute a persistent fraction of the data throughout training – the gate consistently filters them out while distilling the remaining tokens with their full teacher–student log-ratio magnitude. 6 Conclusion We revisited sampled-token OPD from an RLVR perspective and observed that its implicit reward is governed by the teacher-student probability ratio rather than trajectory correctness, which can lead to updates that can penalize tokens in correct trajectories or reward tokens in incorrect ones. To address this, we proposed OPDVR, which adds a simple ReLU gate on the sampled token to enforce RLVR compliance. This minimal modification combines OPD and RLVR without any hyperparameter or heuristic trade-off. Experiments across mathematical reasoning benchmarks demonstrate that OPDVR consistently outperforms standard OPD. More importantly, by reformulating OPD as an RLVR method, OPDVR becomes a general framework that can be readily integrated with any policy gradient algorithm, including REINFORCE, GRPO, and DAPO. We instantiate this with GRPO and present Group Relative Policy Distillation (GRPD), which further demonstrates the versatility and strength of our approach. The empirical success of both OPDVR and GRPD across multiple benchmarks confirms that aligning distillation signals with trajectory correctness is not only effective but also broadly applicable, offering a simple yet powerful recipe for future post-training methods. References Cai et al. (2026) Q. Cai, Y. Ma, L. Li, P. Li, Y. Chen, Q. Guo, Y. Zou, T. Gui, X. Feng, and B. Qin H 2 sd: hybrid hindsight self-distillation. arXiv preprint arXiv:2607.18955. Cited by: §1, §2. Fu et al. (2026) Y. Fu, H. Huang, K. Jiang, J. Liu, Z. Jiang, Y. Zhu, and D. Zhao Revisiting on-policy distillation: empirical failure modes and simple fixes. arXiv preprint arXiv:2603.25562. Cited by: §2. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §1, §2, §3.1, §4.4. Hammoud et al. (2025) H. A. A. K. Hammoud, K. Alhamoud, A. Hammoud, E. Bou-Zeid, M. Ghassemi, and B. Ghanem Train long, think short: curriculum learning for efficient reasoning. arXiv preprint arXiv:2508.08940. Cited by: §2. He et al. (2026) Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, et al. Deepmath-103k: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. In International Conference on Learning Representations, Vol. 2026, p. 138306–138322. Cited by: §5.1. Hou et al. (2025) B. Hou, Y. Zhang, J. Ji, Y. Liu, K. Qian, J. Andreas, and S. Chang Thinkprune: pruning long chain-of-thought of llms via reinforcement learning. arXiv preprint arXiv:2504.01296. Cited by: §2. Hou et al. (2026) W. Hou, S. Peng, W. Wang, Z. Ruan, Y. Zhang, Z. Zhou, M. Gao, Y. Chen, K. Wang, H. Yang, et al. Uni-opd: unifying on-policy distillation with a dual-perspective recipe. arXiv preprint arXiv:2605.03677. Cited by: §2. Huang et al. (2026) C. Huang, Z. Zhang, and C. Cardie Hapo: training language models to reason concisely via history-aware policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 31122–31130. Cited by: §2. Hubotter et al. (2026) J. Hubotter, F. Lubeck, L. D. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. K. Buening, C. Guestrin, and A. Krause Reinforcement learning via self-distillation. ArXiv abs/2601.20802. External Links: Link Cited by: §2. Jaech et al. (2024) A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720. Cited by: §1, §2. Jiang et al. (2025) Y. Jiang, Y. Li, G. Chen, D. Liu, Y. Cheng, and J. Shao Rethinking entropy regularization in large reasoning models. arXiv preprint arXiv:2509.25133. Cited by: §2. Jin et al. (2026) W. Jin, T. Min, Y. Yang, D. Wei, Y. Zhou, S. R. Kadhe, N. Baracaldo, and K. Lee Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079. Cited by: §2. Li et al. (2026a) G. Li, T. Yang, J. Fang, M. Song, M. Zheng, H. Guo, D. Zhang, J. Wang, and T. Chua Unifying group-relative and self-distillation policy optimization via sample routing. arXiv preprint arXiv:2604.02288. Cited by: §1. Li et al. (2026b) Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, et al. Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016. Cited by: §2. Liu et al. (2025) W. Liu, R. Zhou, Y. Deng, Y. Huang, J. Liu, Y. Deng, Y. Zhang, and J. He Learn to reason efficiently with adaptive length-based reward shaping. arXiv preprint arXiv:2505.15612. Cited by: §2. Oh et al. (2026) M. Oh, S. Song, G. Choi, Y. Choi, and Y. Jo KL for a kl: on-policy distillation with control variate baseline. arXiv preprint arXiv:2605.07865. Cited by: §2. Petrenko et al. (2026) A. Petrenko, B. Lipkin, K. Chen, E. Wijmans, M. Cusumano-Towner, R. Giryes, and P. Krähenbühl Entropy-preserving reinforcement learning. arXiv preprint arXiv:2603.11682. Cited by: §2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1, §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §1, §2, §3.1, §4.4. Sheng et al. (2025) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: Appendix B. Su et al. (2025) Z. Su, L. Pan, M. Lv, Y. Li, W. Hu, F. Zhang, K. Gai, and G. Zhou CE-gppo: coordinating entropy via gradient-preserving clipping policy optimization in reinforcement learning. ArXiv abs/2509.20712. External Links: Link Cited by: §2. Sui et al. (2025) Y. Sui, Y. Chuang, G. Wang, J. Zhang, T. Zhang, J. Yuan, H. Liu, A. Wen, S. Zhong, N. Zou, et al. Stop overthinking: a survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419. Cited by: §2. Trinh et al. (2024) T. H. Trinh, Y. Wu, Q. V. Le, H. He, and T. Luong Solving olympiad geometry without human demonstrations. Nature 625 (7995), p. 476–482. Cited by: §1, §2. Wang et al. (2026) C. Wang, Z. Li, J. Bai, Y. Zhang, H. Deng, G. Lan, and Y. Wang Distilled reinforcement learning for llm post-training. arXiv preprint arXiv:2607.17247. Cited by: §1, §2. Xiang et al. (2025) V. Xiang, C. Blagden, R. Rafailov, N. Lile, S. Truong, C. Finn, and N. Haber Just enough thinking: efficient reasoning with adaptive length penalties reinforcement learning. arXiv preprint arXiv:2506.05256. Cited by: §2. Xiao et al. (2026) B. Xiao, B. Xia, B. Yang, B. Gao, B. Shen, C. Zhang, C. He, C. Lou, F. Luo, G. Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780. Cited by: §1, §2. Xing et al. (2026) X. Xing, H. Wang, B. Gao, Z. Li, and Y. Tang Trust region on-policy distillation. arXiv preprint arXiv:2606.01249. Cited by: §2. Xu et al. (2026) A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. Deepseek-v4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: §1, §2. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §1, §2. Yang et al. (2024) A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, et al. Qwen2. 5-math technical report: toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. Cited by: §1, §2. Yang et al. (2026a) C. Yang, C. Qin, Q. Si, M. Chen, N. Gu, D. Yao, Z. Lin, W. Wang, J. Wang, and N. Duan Self-distilled rlvr. arXiv preprint arXiv:2604.03128. Cited by: §1, §2. Yang et al. (2025b) K. Yang, X. Xu, Y. Chen, W. Liu, J. Lyu, Z. Lin, D. Ye, and S. Yang EntroPIC: towards stable long-term training of llms via entropy stabilization with proportional-integral control. arXiv preprint arXiv:2511.15248. Cited by: §2. Yang et al. (2026b) W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin Learning beyond teacher: generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125. Cited by: §5.1. Yu et al. (2026) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems 38, p. 113222–113244. Cited by: §5.1. Zeng et al. (2026) A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763. Cited by: §1, §2. Appendix A Theoretical Analysis: Why OPDVR Improves over Sampled-Token OPD We now provide a formal characterization of the advantage of OPDVR over the sampled-token OPD objective. Throughout this section, let st=(q,o<t)s_t=(q,o_<t) denote the state at step t, at=ota_t=o_t the sampled action, and define the token-level teacher-student log-ratio as rt=logπT(at∣st)πθ(at∣st).r_t= _T(a_t s_t) _θ(a_t s_t). (3) As before, R∈+1,−1R∈\+1,-1\ denotes the trajectory-level verifier reward. A.1 Directional Alignment with the Verifier Gradient Let ut=∇θlogπθ(at∣st)u_t= _θ _θ(a_t s_t) denote the policy gradient direction of the sampled token. The token-level update direction of sampled-token OPD is ΔOPD,t=rtut, _OPD,t=r_tu_t, (4) whereas the OPDVR update direction is ΔOPDVR,t=RReLU(Rrt)ut, _OPDVR,t=R\,ReLU(Rr_t)\,u_t, (5) with ReLU(x)=max(0,x)ReLU(x)= (0,x). Recall that the pure verifier-driven RLVR update direction is ΔRLVR,t=Rut _RLVR,t=Ru_t. The following proposition shows that OPDVR is always aligned with this verifier direction, while OPD may be anti-aligned. Proposition A.1 (Verifier alignment). For any token with ut≠0u_t≠ 0, the OPDVR update is never in the opposite direction of the verifier gradient. Specifically, ⟨ΔOPDVR,t,ΔRLVR,t⟩≥0. _OPDVR,t, _RLVR,t ≥ 0. (6) In contrast, sampled-token OPD satisfies ⟨ΔOPD,t,ΔRLVR,t⟩=rtR‖ut‖2, _OPD,t, _RLVR,t =r_tR\|u_t\|^2, (7) which becomes negative whenever rtr_t and R have opposite signs. Proof. Using (5) and ΔRLVR,t=Rut _RLVR,t=Ru_t, we have ⟨ΔOPDVR,t,ΔRLVR,t⟩ _OPDVR,t, _RLVR,t =⟨RReLU(Rrt)ut,Rut⟩ = R\,ReLU(Rr_t)u_t,Ru_t =R2ReLU(Rrt)‖ut‖2 =R^2\,ReLU(Rr_t)\,\|u_t\|^2 =ReLU(Rrt)‖ut‖2≥0. =ReLU(Rr_t)\,\|u_t\|^2≥ 0. (8) For sampled-token OPD, using (4) gives ⟨ΔOPD,t,ΔRLVR,t⟩=⟨rtut,Rut⟩=rtR‖ut‖2, _OPD,t, _RLVR,t = r_tu_t,Ru_t =r_tR\|u_t\|^2, (9) which is negative when rtR<0r_tR<0. ∎ The condition rtR<0r_tR<0 corresponds exactly to the two conflict cases identified earlier: (i) a correct trajectory with πθ>πT _θ> _T, and (i) an incorrect trajectory with πT>πθ _T> _θ. In both cases, OPD pushes the policy in a direction that reduces the verifier reward. A.2 Decomposition of the OPD Gradient The gated mechanism can be interpreted as explicitly removing the verifier-conflicting component from the OPD gradient. Proposition A.2 (Conflict removal). The sampled-token OPD update can be decomposed as ΔOPD,t=ΔOPDVR,t+Δconflict,t, _OPD,t= _OPDVR,t+ _conflict,t, (10) where Δconflict,t=rt 1(rtR<0)ut. _conflict,t=r_t\,1(r_tR<0)\,u_t. (11) Moreover, the conflict term always has a non-positive projection onto the verifier gradient: ⟨Δconflict,t,ΔRLVR,t⟩=−|rt| 1(rtR<0)‖ut‖2≤0. _conflict,t, _RLVR,t =-|r_t|\,1(r_tR<0)\,\|u_t\|^2≤ 0. (12) Proof. When rtR≥0r_tR≥ 0, we have RReLU(Rrt)=rtR\,ReLU(Rr_t)=r_t, so ΔOPDVR,t=rtut _OPDVR,t=r_tu_t and the conflict term vanishes. When rtR<0r_tR<0, we have ReLU(Rrt)=0ReLU(Rr_t)=0, so ΔOPDVR,t=0 _OPDVR,t=0 and the conflict term equals ΔOPD,t _OPD,t. The projection onto ΔRLVR,t _RLVR,t follows directly. ∎ Thus OPDVR is not an arbitrary modification of OPD; it is precisely OPD with the harmful verifier-opposing component removed. A.3 A Simplified Token-Level Analysis: OPDVR Can Strictly Outperform the Teacher To illustrate the mechanism while retaining the token-level structure of LLM generation, we consider a simplified single-token decision problem. Let s be a fixed prefix. At this prefix, the model must choose between two candidate next tokens: a1a_1 denotes a correct key token that leads to a correct final answer, and a2a_2 denotes an incorrect key token that leads to a wrong final answer. The trajectory-level verifier reward is therefore R(a1)=+1R(a_1)=+1 and R(a2)=−1R(a_2)=-1. Let πθ(a1∣s)=q _θ(a_1 s)=q and πT(a1∣s)=p _T(a_1 s)=p. Assume the teacher is suboptimal, i.e., p<12p< 12, and that the initial student policy is already better than the teacher, i.e., q0>pq_0>p. Under sampled-token OPD, the expected loss reduces to the reverse KL divergence ℒOPD=DKL(πθ(⋅∣s)∥πT(⋅∣s)),L_OPD=D_KL( _θ(· s)\,\|\, _T(· s)), (13) whose global minimizer is πθ(⋅∣s)=πT(⋅∣s) _θ(· s)= _T(· s). Hence the OPD optimum satisfies q=pq=p, with expected verifier reward JOPD=2p−1.J_OPD=2p-1. (14) Under OPDVR, the expected update for token a1a_1 is R(a1)ReLU(R(a1)logpq0)=ReLU(logpq0)=0,R(a_1)\,ReLU\! (R(a_1) pq_0 )=ReLU\! ( pq_0 )=0, (15) because log(p/q0)<0 (p/q_0)<0 since q0>pq_0>p. Similarly, for token a2a_2 we have R(a2)=−1R(a_2)=-1 and log1−p1−q0>0 1-p1-q_0>0 (16) since q0>pq_0>p, so the OPDVR update for a2a_2 is R(a2)ReLU(R(a2)log1−p1−q0)=−ReLU(−log1−p1−q0)=0.R(a_2)\,ReLU\! (R(a_2) 1-p1-q_0 )=-ReLU\! (- 1-p1-q_0 )=0. (17) Thus the expected OPDVR gradient vanishes, and the policy remains at q0q_0. Consequently, JOPDVR=2q0−1>2p−1=JOPD=Jteacher.J_OPDVR=2q_0-1>2p-1=J_OPD=J_teacher. (18) Therefore, OPDVR strictly outperforms both standard sampled-token OPD and the teacher policy. Appendix B Hyperparameters We provide the detailed hyperparameter configurations used in our experiments in Table 5. All models are trained using the Verl Sheng et al. (2025) framework with the settings specified below. Table 5: Hyperparameter settings. Hyperparameter Value Learning Rate 1e-6 Train Batch Size 256 PPO Mini-Batch Size 256 Max Response Length 8192 Max Prompt Length 1024 Rollout Temperature 1.0 Evaluation Temperature 0.7 Evaluation Top-p 0.95 Appendix C Hardware Setup All experiments in this paper are conducted on NVIDIA GeForce RTX 5090 GPUs.