Paper deep dive
E$^2$-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation
Yifei Liu, Minghao Fang, Xinyu Gu, Chengkai Yao, Mengdi Liu, Tengfei Ma, Jiangbin Zheng, Chang Yu, Zhangyang Gao
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy self-distillation (OPSD) provides dense token-level supervision without a second model: one network acts as teacher with the reference solution and as student with only the problem. We identify a specific failure mode of this recipe. During training, student token entropy rises past the teacher's and remains elevated, a pattern we call entropy overshoot. We trace it to both sides of distillation. The reference-conditioned teacher is confident along its answer-directed reasoning path, but this confidence transfers poorly to student-generated prefixes, making its supervision overly tied to answer-specific cues rather than reusable reasoning patterns; meanwhile, the forward KL used by OPSD continually diffuses the student's predictive distribution without pulling it back. We introduce E$^2$-OPSD to address both causes. Exemplar-guided teaching replaces the current answer with a retrieved solved neighboring problem, providing transferable reasoning guidance without revealing the destination and better matching student-reachable states. Entropy-aware distillation uses the student-teacher entropy gap to determine the direction and strength of each token's correction. E$^2$-OPSD improves math reasoning by up to 4.3 points in mean@16 over OPSD, while out-of-domain evaluations show gains over the corresponding base models of up to 4.9 points in mean@16 and 5.5 points in pass@8. Despite these gains, E$^2$-OPSD remains simple, requiring no additional forward passes or networks.
Tags
Links
- Source: https://arxiv.org/abs/2610.05048v1
- Canonical: https://arxiv.org/abs/2610.05048v1
Trouble viewing inline? Open PDF directly →
Full Text
81,756 characters extracted from source content.
Expand or collapse full text
E2-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation Yifei Liu Minghao Fang Xinyu Gu Affiliation: Shanghai Artificial Intelligence Laboratory Chengkai Yao Affiliation: Shanghai Jiaotong University Mengdi Liu Affiliation: Visual Information Processing and Learning, ICT, CAS Tengfei Ma Affiliation: Hunan University Jiangbin Zheng Affiliation: Westlake University Chang Yu Affiliation: Nanjing University Zhangyang Gao Affiliation: The Chinese University of Hong Kong Affiliation: Zhejiang University Abstract On-policy self-distillation (OPSD) provides dense token-level supervision without a second model: one network acts as teacher with the reference solution and as student with only the problem. We identify a specific failure mode of this recipe. During training, student token entropy rises past the teacher’s and remains elevated, a pattern we call entropy overshoot. We trace it to both sides of distillation. The reference-conditioned teacher is confident along its answer-directed reasoning path, but this confidence transfers poorly to student-generated prefixes, making its supervision overly tied to answer-specific cues rather than reusable reasoning patterns; meanwhile, the forward KL used by OPSD continually diffuses the student’s predictive distribution without pulling it back. We introduce E2-OPSD to address both causes. Exemplar-guided teaching replaces the current answer with a retrieved solved neighboring problem, providing transferable reasoning guidance without revealing the destination and better matching student-reachable states. Entropy-aware distillation uses the student–teacher entropy gap to determine the direction and strength of each token’s correction. E2-OPSD improves math reasoning by up to 4.3 points in mean@16 over OPSD, while out-of-domain evaluations show gains over the corresponding base models of up to 4.9 points in mean@16 and 5.5 points in pass@8. Despite these gains, E2-OPSD remains simple, requiring no additional forward passes or networks. 11footnotetext: Equal contribution. † author. 11155263220@link.cuhk.edu.hk 2minghao.22@intl.zju.edu.cn 1 Introduction Figure 1: Left: Entropy overshoot in Qwen3-1.7B. Right: The reference-conditioned teacher is less certain on student prefixes, suggesting a reasoning-path mismatch. Reinforcement learning with verifiable rewards (RLVR) is now the standard recipe for improving LLM reasoning, yet its supervision remains coarse: an entire rollout collapses into a single scalar reward (Shao et al., 2024; Guo et al., 2025). On-policy distillation (OPD) instead offers dense, token-level supervision by having a teacher score the student’s own rollouts (Agarwal et al., 2024; Gu et al., 2023), and is now a standard post-training component in Qwen3 (Yang et al., 2025), GLM-5 (GLM-5-Team et al., 2026), and DeepSeek-V4 (DeepSeek-AI et al., 2026). Despite this fine granularity, it still requires a stronger open-weight teacher that shares the student’s vocabulary. On-policy self-distillation (OPSD) further removes the need to serve a second model: a copy of the student acts as the teacher, with access to the reference solution, while the student sees only the problem. Teacher and student are evaluated on the same student-generated prefixes and matched with forward KL (Zhao et al., 2026). This makes OPSD an appealing way to turn privileged solution information into dense supervision. In practice, however, its gains are unstable and can deteriorate as training proceeds. (a) Exemplar-Guided Teaching (b) Forward/Reverse KL Figure 2: (a) Exemplar-Guided Teaching. Standard OPSD gives the teacher the current reference solution, letting it jump from the student prefix toward the answer. E2-OPSD instead uses a solved neighbor as a strategy guide, forcing the teacher to follow the student’s intermediate step and solve the current problem itself, turning answer leakage into strategy transfer. (b) Forward and reverse KL play complementary roles. Forward KL promotes exploration over teacher-supported alternatives, while reverse KL removes unsupported probability mass to concentrate the policy. Entropy-aware distillation switches between them. We identify a characteristic failure mode, as shown in Figure 1 (left), entropy overshoot: instead of settling toward the teacher, the student becomes progressively more uncertain, eventually surpassing the teacher’s entropy and staying there. Entropy overshoot therefore describes a regime where useful exploration spills into persistent and excessive uncertainty, gradually eroding the student’s decisiveness and amplifying teacher-side uncertainty into noisy student behavior. We examine this failure from both teacher and student perspectives. First, OPSD produces supervision mismatched to the student’s reasoning path. The teacher is confident on its own trajectories but becomes substantially more uncertain on student-generated prefixes (Figure 1, right), suggesting that its guidance does not transfer cleanly to the intermediate states the student actually reaches. This mismatch stems from a deeper issue: seeing the current solution lets the teacher narrate toward a known answer rather than reason from scratch, making its supervision answer-specific rather than transferable (Figure 2(a)). Second, OPSD lacks a mechanism to regulate policy spread. Forward KL is beneficial when the student is too concentrated as it expands probability toward teacher-supported alternatives. But the same pressure persists even after the policy becomes overly diffuse, which makes exploration a one-way ratchet: uncertainty keeps spreading, with no force pulling the policy back into focus (Figure 2(b)). These observations motivate E2-OPSD, a simple framework that repairs both sides of the learning process with Exemplar-guided teaching and Entropy-aware distillation, as illustrated in Figure 3. More specifically, E2-OPSD makes three contributions: • What to learn — Exemplar-guided teaching. We replace the current problem’s reference solution in the teacher’s context with a retrieved solved neighboring problem. The exemplar provides a transferable problem-solving reference without revealing the current answer. Conditioned on this reference and the student’s generated prefix, the teacher’s token-level guidance is meant to be more compatible with the student’s ongoing reasoning. • How to learn — Entropy-aware distillation. We use the student–teacher entropy gap as a feedback signal: when the student is more concentrated than the teacher, forward KL encourages exploration; when it becomes more diffuse, reverse KL suppresses unsupported mass and encourages concentration. The gap magnitude downweights uncertain routing decisions and concentrates learning on tokens with larger entropy mismatches. This turns self-distillation from a fixed one-way objective into closed-loop self-balancing control of exploration. • Simple and Generalized. E2-OPSD adds no extra forward passes during distillation and minimum overhead. Across various model families, sizes from 0.6B to 7B, and both thinking and instruction-tuned models, it mitigates persistent entropy overshoot and improves math reasoning by up to 4.3 points over OPSD, with gains transferring to out-of-domain tasks by up to 4.9 mean@16 points over the corresponding base model. These results suggest that effective self-distillation depends not on maximizing imitation, but on aligning teacher guidance with the student’s reasoning path and regulating when the student should explore versus concentrate. Figure 3: Overview of E2-OPSD. 2 Preliminaries and Related Work 2.1 KL Divergence The Kullback–Leibler (KL) divergence is asymmetric, and its two directions induce distinct approximation biases (Minka, 2005; Jin et al., 2026). For a target distribution P and model distribution Q, KL(P∥Q)=v∼P[logP(v)Q(v)],KL(Q∥P)=v∼Q[logQ(v)P(v)].KL(P\|Q)=E_v P [ P(v)Q(v) ], (Q\|P)=E_v Q [ Q(v)P(v) ]. (1) Forward KL is mode covering: if P(v)>0P(v)>0 while Q(v)→0Q(v)→ 0, its contribution diverges, strongly penalizing target-supported modes that the model fails to cover. Reverse KL has the complementary mode-seeking bias: if Q(v)>0Q(v)>0 while P(v)→0P(v)→ 0, its contribution diverges, strongly penalizing model mass unsupported by the target (Minka, 2005). Thus, forward KL favors coverage, whereas reverse KL favors concentration. 2.2 On-Policy Distillation On-policy distillation (OPD) trains a student πθ _θ to match a teacher πT _T on the student’s own generations (Agarwal et al., 2024; Gu et al., 2023; Lu and Lab, 2025). Given a problem x∼x , the student samples y^=(y^1,…,y^|y^|) y=( y_1,…, y_| y|), and both models score the same student-generated prefix st=(x,y^<t)s_t=(x, y_<t): ℒ(θ)=x∼y^∼πθ(⋅∣x)[1|y^|∑t=1|y^|D(πT(⋅∣st)∥πθ(⋅∣st))],L(θ)=E_x E_ y _θ(· x) [ 1| y| _t=1^| y|D\! ( _T(· s_t)\, \|\, _θ(· s_t) ) ], (2) where D denotes a per-token divergence between the teacher and student distributions, instantiated as forward KL KL(πT∥πθ)KL( _T\| _θ) or reverse KL KL(πθ∥πT)KL( _θ\| _T), and gradients flow only through the student. Reverse KL is a common choice (Gu et al., 2023; Lu and Lab, 2025), while prior work also explores mixed or skewed divergences as alternative distillation objectives (Agarwal et al., 2024; Ko et al., 2024; Shing et al., 2025; Wu et al., 2025; Jin et al., 2026). OPSD computes D over the full vocabulary and applies per-token pointwise clipping, capping each vocabulary entry’s contribution at τ so that a few stylistic tokens do not dominate the signal (Zhao et al., 2026). We keep this pointwise clipping and, following recent sparse-support distillation practice, approximate D on the teacher’s top-KvK_v tokens together with a tail bucket carrying the remaining probability mass (Li et al., 2026b). 2.3 Related Work On-policy self-distillation. On-policy self-distillation (OPSD) instantiates Eq. 2 with a copy of the student as teacher, πT=πθ0 _T= _ _0, whose context includes the reference solution y∗y^* unseen by the student and is matched via forward KL (Zhao et al., 2026). This privileged teacher creates an irreducible objective gap (Yang et al., 2026), overweights stylistic and answer-implied tokens over genuine decisions (Pan et al., 2026; Harne et al., 2026), and suppresses uncertainty important to thinking models (Kaur et al., 2026; Kim et al., 2026). Existing remedies largely patch these effects downstream. Verifier-directed methods outsource update control to outcome rewards (Yang et al., 2026; Ke et al., 2026; He et al., 2026), effectively reintroducing RLVR. Signal-purification methods cancel privileged bias with an additional teacher pass (Pan et al., 2026; Shen et al., 2026), paying extra compute after the bias is already introduced. Context-curation methods expose less of the solution or replace it with a successful sibling rollout (Han et al., 2026; Hübotter et al., 2026), but still rely on solution-specific supervision and auxiliary control. These approaches mitigate symptoms without removing their source: teacher guidance tied to the current problem’s solution. E2-OPSD removes this mismatch at the source, restores the concentrating correction absent from forward KL, and retains a single frozen-teacher pass with no reward. Entropy-aware post-training. Entropy has become a common control signal in post-training, but serves different roles across RLVR and distillation. In RLVR, it is primarily used to regulate exploration, from preventing entropy collapse to concentrating updates on high-entropy or decision-critical tokens (Cui et al., 2025; Yu et al., 2025; Wang et al., 2025; Hao et al., 2025). In distillation, entropy has been used to guide supervision selection and adaptation: EOPD activates additional forward-KL supervision at high-teacher-entropy tokens, while subsequent work further adapts supervision across uncertainty regimes (Jin et al., 2026; Zhang et al., 2026; Lee et al., 2026). E2-OPSD instead targets the relative entropy dynamics of the student and teacher, using their signed token-level entropy gap to correct entropy overshoot during on-policy self-distillation. Training-time contextual guidance. A broad line of work improves post-training by providing auxiliary context that is available during training but absent at inference. Context distillation internalizes prompts, exemplars, or scratchpads into model weights (Askell et al., 2021; Snell et al., 2022; Shin et al., 2025; Kalwar et al., 2026). More recent methods diversify the form of this guidance: OPCD distills experiential knowledge from historical solution traces, SDFT learns on-policy from demonstrations, SKALD conditions a self-teacher on abstract skill cards, and PMD accumulates reusable procedural memories across training episodes (Ye et al., 2026; Shenfeld et al., 2026; Jiang et al., 2026; Liu et al., 2026). Other approaches inject more instance-specific scaffolds directly into optimization, such as partial solutions in QuestA, off-policy reasoning traces in LUFFY, in-context exemplars in XRPO, or retrieved analogous demonstrations in RA-RFT (Li et al., 2026a; Yan et al., 2025; Bamba et al., 2025; Xiao et al., 2026). OPSD conditions its self-teacher on the current problem’s reference solution; E2-OPSD instead uses a retrieved solved neighbor, transferring reusable strategy across instances without revealing the current answer. 3 Method E2-OPSD repairs both sides of the entropy-overshoot failure identified in Section 1: exemplar-guided teaching changes what the teacher is conditioned on so its guidance is grounded in transferable reasoning rather than answer-specific cues, while entropy-aware distillation changes how the student matches it, using the per-token entropy gap to set both the direction and strength of the correction. Both run inside the standard OPSD loop with one frozen teacher and no additional forward passes or extra network. Setup. The student πθ _θ and frozen teacher πϕ _φ, initialized from the same model (ϕ=θ0φ= _0), score the same student-generated prefix y^<t y_<t but under different contexts: pt(v)=πθ(v∣x,y^<t),qt(v,r)=πϕ(v∣x,r(x),y^<t),p_t(v)= _θ(v x, y_<t), q_t(v;r)= _φ(v x,r(x), y_<t), (3) where r(x)r(x) denotes teacher-only privileged context. Standard OPSD sets r(x)=y∗r(x)=y^*, giving the teacher the reference solution to the current problem. 3.1 Exemplar-Guided Teaching Conditioning on y∗y^* lets the teacher narrate toward a known answer, producing guidance that depends on information absent from the student’s context. Exemplar-Guided Teaching (EGT) instead replaces the current solution with a solved neighboring problem, so the teacher must still solve x itself while using the exemplar only as transferable strategy (Figure 2(a)). We embed all training problems and compute pairwise cosine similarity offline, keeping each problem’s K nearest neighbors (self-excluded) as K(x)N_K(x). For the g-th of G rollouts per problem, we draw one neighbor uniformly, rg(x)=(xg′,yg′∗),xg′∼Uniform(K(x)),xg′≠x,r_g(x)=(x _g,\,y *_g), x _g \! (N_K(x) ), x _g≠ x, (4) seeded deterministically by problem and rollout index, so different rollouts of the same problem see different exemplars at no extra retrieval cost. Thus the teacher is conditioned on (x,xg′,yg′∗,y^<t)(x,x _g,y_g *, y_<t), while the student still sees only (x,y^<t)(x, y_<t). EGT therefore removes the current answer from teacher conditioning and replaces answer-specific privilege with cross-problem strategy guidance. Prompts and offline preprocessing cost are in Appendix A.4. 3.2 Entropy-Aware Distillation EGT improves what the teacher provides; Entropy-Aware Distillation (EAD) controls how the student follows it. Building on the complementary mode-covering and mode-seeking biases of forward and reverse KL (Section 2.1), EAD has two components: Bidirectional Entropy Routing (BER) selects the KL direction from the student–teacher entropy gap, while Adaptive Gap Damping (AGD) uses its magnitude to scale the update. Bidirectional entropy routing. We use the entropy gap Δt=H(pt)−H(qt) _t=H(p_t)-H(q_t) to determine which correction the student needs. When Δt≤0 _t≤ 0, the student is relatively more concentrated than the teacher, so BER applies forward KL to broaden coverage. When Δt>0 _t>0, the student is relatively more diffuse, so BER switches to reverse KL to refocus the policy (Figure 2(b)). Using ~t V_t, the teacher’s top-KvK_v tokens plus a tail bucket, and the per-element clip τ inherited from OPSD (Section 2.2), DtBER=∑v∈~tmin(qt(v)logqt(v)pt(v),τ),Δt≤0,∑v∈~tmin(pt(v)logpt(v)qt(v),τ),Δt>0.D_t^BER= cases _v∈ V_t \! (q_t(v) q_t(v)p_t(v),\,τ ),& _t≤ 0,\\[6.0pt] _v∈ V_t \! (p_t(v) p_t(v)q_t(v),\,τ ),& _t>0. cases (5) BER thus replaces OPSD’s fixed one-way pressure with a state-dependent correction that can either broaden or refocus the student policy. Adaptive gap damping. The sign of Δt _t selects the correction direction, while its magnitude indicates the need for correction: near-zero gaps make routing sensitive to small perturbations, whereas larger gaps indicate more pronounced student–teacher entropy mismatches that call for stronger correction. We therefore use the gap magnitude as a confidence weight to downweight uncertain routing decisions and concentrate learning on tokens with larger entropy mismatches. Over the valid response tokens T in the current mini-batch, we define the typical gap scale σ2=1||∑(i,t)∈Δi,t2σ^2= 1|T| _(i,t) _i,t^2 and weight each routed update by ct=Δt2Δt2+max(σ2,ϵ2)∈[0,1),c_t= _t^2 _t^2+ (σ^2,ε^2)∈[0,1), (6) where ϵε is the FP32 machine epsilon, with ct=0c_t=0 when Δt=0 _t=0. Gaps small relative to the batch scale are therefore suppressed, while those much larger than the scale retain most of their update. Because σ2σ^2 is recomputed every mini-batch, the damping adapts to the training dynamics without a tunable threshold. Training objective. Combining exemplar-guided teaching, bidirectional entropy routing, and adaptive gap damping, E2-OPSD minimizes ℒ(θ)=1||∑(i,t)∈sg(ci,t)Di,tBER,L_T(θ)= 1|T| _(i,t) sg(c_i,t)\,D_i,t^BER, (7) where T contains the valid response tokens in the current loss computation, indexed by response i and position t. Sampled responses and exemplars are held fixed during optimization. The teacher conditions on the sampled solved neighbor (x′,y′∗)(x ,y *) through qtq_t, while the student context remains unchanged. Both the routing decision and ctc_t are treated as stop-gradient quantities, so optimization acts only through the student probabilities in DtBERD_t^BER. Compared with OPSD, E2-OPSD changes only the teacher context and per-token divergence, requiring the same rollouts and frozen-teacher scoring with no additional forward pass, verifier, or reward during distillation. See Appendix A.4 for more overhead analysis. At inference, all privileged context is removed. 4 Experiments 4.1 Experimental Settings Models & Datasets. We evaluate four configurations across the Qwen3 (Yang et al., 2025) and Olmo3 (Olmo et al., 2026) families: Qwen3-0.6B and Qwen3-1.7B with thinking enabled, Qwen3-4B-Instruct-2507, and Olmo-3-7B-Instruct. The reasoning mode is consistent across the student, teacher (where applicable), and evaluation. Following Pan et al. (2026), we use DeepMath-103K (He et al., 2025) for training, with a difficulty band assigned to each configuration (Appendix A.1). We evaluate in-domain mathematical reasoning on AMC23 (math-ai, 2023), AIME’24 (Zhang and Math-AI, 2024), AIME’25 (Zhang and Math-AI, 2025), and HMMT’25 Feb (Harvard–MIT Mathematics Tournament, 2025). Out-of-domain evaluation covers logical reasoning on a fixed, randomly sampled 100-puzzle subset of ZebraLogic (Lin et al., 2025)for computational tractability, and code generation on HumanEval (Chen et al., 2021). Baselines. We compare against (1) Base, the untrained checkpoints; (2) OPSD (Zhao et al., 2026), which distills a frozen teacher conditioned on the current problem’s reference solution using forward KL; (3) SDPO (Hübotter et al., 2026), which uses an EMA teacher conditioned on a successful sibling rollout and a generalized JSD loss; (4) GRPO (Shao et al., 2024), which optimizes a clipped policy objective using group-normalized advantages from verifiable rewards, with KL regularization against a frozen reference policy; (5) RLSD (Yang et al., 2026), which uses verifiable rewards to determine update directions and self-distillation signals to modulate token-level update magnitudes; and (6) OPSD + EOPD, our self-distillation adaptation of EOPD (Jin et al., 2026), which combines reverse KL with forward KL at high-teacher-entropy positions. We retain this entropy-gated divergence but replace EOPD’s external stronger teacher with OPSD’s reference-conditioned self-teacher for a controlled comparison. Implementation Details. We use veRL (Sheng et al., 2025) with vLLM (Kwon et al., 2023) on 8×H200 GPUs and a learning rate of 10−610^-6. We faithfully implement each baseline following the algorithmic design in its original paper, preserving its method-specific objectives and update rules, including teacher construction for self-distillation methods. All trained methods and ablations share the same training settings and a budget of 360 optimizer steps. Detailed baseline configurations, including the explicit adaptation of EOPD to the OPSD setting, are provided in Appendix A.4 for reproducibility. The baseline results in Table 1 are obtained from these implementations under the shared protocol. For self-distillation methods, the teacher is frozen at its initial weights except in SDPO and RLSD. We use K=15K=15 for a diverse exemplar selection, and supply retrieved exemplars only to the teacher. We evaluate each model with three evaluation seeds to reduce noise. We provide the model with only the original questions during evaluation inference. See Appendix A for detailed training and evaluation settings. Table 1: Main results across model scales and types on math reasoning, reported as mean@16 over three seeds. Green subscripts show our gains over Base, computed from the displayed values. Best values are bold and second-best distinct values underlined. Full evaluation results are in Appendix B. Math Reasoning (mean@16) Model Method AMC23 AIME24 AIME25 HMMT25 Avg. Qwen3-0.6B (Thinking) Base 45.845.8 9.89.8 14.214.2 8.38.3 19.519.5 OPSD 45.945.9 10.9¯ 10.9 14.214.2 9.29.2 20.020.0 SDPO 45.545.5 9.49.4 14.614.6 7.87.8 19.319.3 GRPO 46.146.1 9.89.8 16.416.4 8.48.4 20.2¯ 20.2 RLSD 46.846.8 11.011.0 15.115.1 7.87.8 20.2¯ 20.2 OPSD + EOPD 46.446.4 9.89.8 15.915.9 8.58.5 20.120.1 E2-OPSD (Ours) 46.6¯+0.8 46.6_ [rgb]0,0.5,0+0.8 11.0+1.211.0_ [rgb]0,0.5,0+1.2 16.2¯+2.0 16.2_ [rgb]0,0.5,0+2.0 8.9¯+0.6 8.9_ [rgb]0,0.5,0+0.6 20.7+1.220.7_ [rgb]0,0.5,0+1.2 Qwen3-1.7B (Thinking) Base 75.575.5 46.046.0 34.634.6 21.921.9 44.544.5 OPSD 74.974.9 46.046.0 35.035.0 22.7¯ 22.7 44.644.6 SDPO 76.1¯ 76.1 48.048.0 35.3¯ 35.3 21.921.9 45.3¯ 45.3 GRPO 74.674.6 47.047.0 35.235.2 21.921.9 44.744.7 RLSD 75.975.9 48.3¯ 48.3 34.034.0 21.821.8 45.045.0 OPSD + EOPD 74.374.3 47.447.4 34.234.2 21.521.5 44.344.3 E2-OPSD (Ours) 79.0+3.579.0_ [rgb]0,0.5,0+3.5 52.3+6.352.3_ [rgb]0,0.5,0+6.3 39.2+4.639.2_ [rgb]0,0.5,0+4.6 25.0+3.125.0_ [rgb]0,0.5,0+3.1 48.9+4.448.9_ [rgb]0,0.5,0+4.4 Qwen3-4B- Instruct-2507 Base 85.485.4 61.761.7 46.546.5 29.029.0 55.7¯ 55.7 OPSD 84.884.8 62.4¯ 62.4 45.645.6 29.5¯ 29.5 55.655.6 SDPO 83.983.9 61.361.3 45.845.8 27.527.5 54.654.6 GRPO 84.484.4 61.761.7 45.645.6 29.129.1 55.255.2 RLSD 85.185.1 62.262.2 46.9¯ 46.9 28.428.4 55.7¯ 55.7 OPSD + EOPD 85.5¯ 85.5 62.062.0 46.546.5 28.428.4 55.655.6 E2-OPSD (Ours) 87.1+1.787.1_ [rgb]0,0.5,0+1.7 62.8+1.162.8_ [rgb]0,0.5,0+1.1 51.0+4.551.0_ [rgb]0,0.5,0+4.5 30.2+1.230.2_ [rgb]0,0.5,0+1.2 57.8+2.157.8_ [rgb]0,0.5,0+2.1 Olmo-3-7B- Instruct Base 80.580.5 52.852.8 37.837.8 23.323.3 48.648.6 OPSD 80.580.5 52.852.8 38.338.3 22.422.4 48.548.5 SDPO 80.380.3 51.351.3 39.9¯ 39.9 24.524.5 49.049.0 GRPO 80.9¯ 80.9 52.652.6 38.638.6 23.523.5 48.948.9 RLSD 80.480.4 53.6¯ 53.6 38.838.8 23.823.8 49.2¯ 49.2 OPSD + EOPD 79.679.6 53.553.5 39.239.2 24.0¯ 24.0 49.149.1 E2-OPSD (Ours) 81.2+0.781.2_ [rgb]0,0.5,0+0.7 56.3+3.556.3_ [rgb]0,0.5,0+3.5 40.0+2.240.0_ [rgb]0,0.5,0+2.2 23.9+0.623.9_ [rgb]0,0.5,0+0.6 50.3+1.750.3_ [rgb]0,0.5,0+1.7 Table 2: Out-of-domain evaluation on logical reasoning and code generation. E2-OPSD consistently improves both mean@16 and pass@8 across ZebraLogic and HumanEval, showing that its gains extend beyond in-domain mathematical reasoning. Logic: ZebraLogic Coding: HumanEval Model Method mean@16 pass@8 mean@16 pass@8 Base 28.828.8 56.556.5 52.652.6 78.478.4 Qwen3-0.6B (Thinking) E2-OPSD 29.029.0+0.2 56.556.5+0.0 53.153.1+0.5 79.079.0+0.6 Base 62.562.5 78.378.3 83.883.8 93.293.2 Qwen3-1.7B (Thinking) E2-OPSD 67.467.4+4.9 83.883.8+5.5 84.584.5+0.7 93.893.8+0.6 Base 79.179.1 92.492.4 84.084.0 89.989.9 Qwen3-4B Instruct-2507 E2-OPSD 82.282.2+3.1 92.892.8+0.4 84.884.8+0.8 90.790.7+0.8 Base 32.732.7 56.656.6 78.278.2 92.192.1 OLMo-3-7B Instruct E2-OPSD 33.933.9+1.2 57.957.9+1.3 78.778.7+0.5 92.592.5+0.4 4.2 Main Results E2-OPSD delivers robust and transferable gains. As shown in Tables 1 and 2, E2-OPSD achieves the highest average math mean@16 across all four model configurations, spanning two model families and both thinking and instruct modes. In particular, E2-OPSD improves over Base by 4.4 and 2.1 percentage points on Qwen3-1.7B Thinking and Qwen3-4B-Instruct-2507, respectively, while leading on all four math benchmarks for both models. Its consistent effectiveness across distinct reasoning modes suggests that E2-OPSD addresses a fundamental failure mode of OPSD, rather than relying on a model- or reasoning-format-specific recipe. More importantly, the gains extend beyond math reasoning: E2-OPSD improves over Base in both logical reasoning and code generation across all four configurations, with logical reasoning gains of 4.9 and 3.1 points on Qwen3-1.7B Thinking and Qwen3-4B-Instruct-2507, respectively, alongside consistent gains in code generation as well. Together, these gains support our central premise that effective self-distillation should transfer reasoning strategies and regulate distillation dynamics, rather than merely imitate solutions. (a) Entropy dynamics on Olmo-3-7B-Instruct. (b) Entropy dynamics on Qwen3-4B-Instruct-2507. (c) Emergent KL-routing dynamics. Figure 4: Training dynamics. (a,b) Student and teacher token entropy under OPSD (dashed) and E2-OPSD (solid); the dotted line marks the first student–teacher entropy crossing under OPSD. (c) On Qwen3-1.7B Thinking, routing share denotes the fraction of response tokens assigned to forward or reverse KL by BER. The routing naturally shifts from forward-dominated exploration toward increased reverse-KL concentration as training proceeds. E2-OPSD mitigates persistent entropy overshoot. Figures 4(a) and 4(b) revisit the failure identified in the introduction. Under OPSD, student entropy crosses the teacher early and continues to drift upward, creating a persistent entropy gap. In contrast, E2-OPSD keeps student entropy tightly regulated around the teacher throughout training, preventing runaway diffusion while preserving sufficient exploration. This curbs entropy overshoot and stabilizes distillation. E2-OPSD exhibits an emergent exploration-to-concentration transition. As shown in Figure 4(c), forward KL dominates early, promoting exploration toward teacher-supported alternatives, while reverse KL becomes increasingly prevalent later, suppressing excess probability mass and concentrating the student distribution. This shift emerges without an explicit schedule and naturally tracks the evolving student state. As both routing curves flatten, E2-OPSD further exhibits a self-equilibrating diffusion–concentration dynamic. (a) Component contributions. (b) Teacher-context probe. (c) AGD gradient probe. Figure 5: Component ablation and mechanism probes. (a) Performance contribution of each design component; “+” denotes the component added on top of OPSD. (b) EGT improves teacher–student compatibility. The x-axis denotes teacher-to-student forward KL evaluated on student prefixes; the y-axis shows response length from each setting’s own rollouts (log scale). (c) AGD sharpens entropy-aware routing. HL denotes high-student/low-teacher entropy and LH the reverse. 4.3 Ablation Study Performance breakdown by design components. Figure 5(a) isolates the contribution of each design choice on Qwen3-1.7B Thinking. Starting from OPSD at 44.6 macro mean@16, exemplar-guided teaching raises performance to 46.0, while bidirectional entropy routing reaches 47.6. Adding adaptive gap damping further improves the score to 48.4, and the full combination achieves 48.9. The steady gains suggest that the components provide complementary benefits, improving both the supervision the student receives and how that supervision is applied. For detailed results, please refer to Table 11. State-dependent KL routing outperforms fixed directions. With the reference-answer teacher fixed and adaptive gap damping disabled, our entropy-aware routing clearly outperforms all other methods that apply fixed KL directions (see Figure 6). To be more precise, forward KL keeps expanding the student and eventually drives entropy overshoot, making the student overly diffuse. Reverse KL suppresses excess mass but also restricts exploration when the student has not yet covered enough plausible alternatives. Their fixed mixture softens both extremes, yet applies expansion and contraction simultaneously regardless of the student’s current state, causing the two corrections to partially counteract each other rather than selecting the one that is needed. In contrast, E2-OPSD uses the student–teacher entropy gap to select the needed correction, expanding when the student is too concentrated and contracting when it is too diffuse, yielding more stable entropy dynamics and stronger validation performance. Figure 6: State-dependent KL routing avoids fixed directional bias. KL-direction dynamics on Qwen3-1.7B Thinking with the teacher context fixed and AGD disabled. The x-axis shows optimizer steps; the left y-axis reports student and teacher token entropy, while the right y-axis of panel (d) reports validation mean@8. Panels compare (a) forward KL, (b) reverse KL, (c) an equal-weight forward/reverse KL mixture, and (d) BER. The trajectories reveal distinct distributional dynamics under fixed versus state-dependent KL routing. Exemplar guidance makes the teacher locally learnable. Figure 5(b) reveals why replacing the current reference solution matters. When conditioned on the reference answer, the teacher produces much shorter and more confident trajectories than the student, with an entropy of 0.190 nats versus the student’s 0.268 nats. EGT instead keeps the teacher closer to the student’s reasoning regime, at 2,090 tokens and 0.261 nats. Moreover, EGT reduces teacher–student forward KL by 25.9% (0.1549 to 0.1148 nats) on the same student prefixes. Together, these observations suggest that EGT brings the teacher’s native uncertainty closer to the student’s reasoning regime while providing more closely aligned token-level guidance on student-visited states. AGD sharpens entropy-aware routing. Figure 5(c) isolates how AGD redistributes token-level learning signals. Here, high entropy means H≥ln2H≥ 2, corresponding to at least two effective next-token choices. Entropy-discordant positions—HL and LH, where student and teacher fall into opposite high/low entropy regimes—account for only 7.67% of all response tokens, yet contribute 20.18% of the relative gradient mass without AGD. Enabling AGD further raises this share to 28.50%, with both HL and LH increasing consistently. This shows that AGD does not uniformly rescale the distillation objective; instead, it makes BER more selective by shifting learning toward positions with clearer student–teacher disagreement in policy spread. To rule out that small entropy gaps merely hide distributional mismatch, we measure probability-mass overlap: strongly damped positions have 0.985 mean overlap (Appendix C), supporting the entropy gap as a simple yet principled proxy for local distributional mismatch. In this sense, BER determines which direction to correct, while AGD determines which routing decisions deserve stronger updates. 5 Conclusion We identify entropy overshoot as a key failure mode of on-policy self-distillation: a teacher that has read the answer supplies guidance the student cannot fully learn, and OPSD’s forward-KL objective keeps pushing the student outward, turning useful exploration into persistent distributional mismatch. We introduce E2-OPSD, which controls this process by constructing more locally learnable teacher targets and adapting both the direction and the strength of distillation to the student’s training state. Across both thinking and instruct models, E2-OPSD consistently improves model’s reasoning capability, with gains transferring to out-of-domain tasks. Our analyses show that effective self-distillation requires matching teacher guidance to student-reachable states and uncertainty structure, rather than simply increasing or suppressing entropy. These results suggest that controlling where exploration occurs is as important as encouraging exploration itself. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Vol. 2024, p. 21246–21263. Cited by: §1, §2.2, §2.2. Askell et al. (2021) A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. Cited by: §2.3. Bamba et al. (2025) U. Bamba, M. Fang, Y. Yu, H. Zheng, and F. Lai XRPO: pushing the limits of grpo with targeted exploration and exploitation. ArXiv abs/2510.06672. External Links: Link Cited by: §2.3. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. S. Winter, P. Tillet, F. P. Such, D. W. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating large language models trained on code. ArXiv abs/2107.03374. External Links: Link Cited by: §4.1. Cui et al. (2025) G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. Cited by: §2.3. DeepSeek-AI et al. (2026) DeepSeek-AI, A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, C. Lu, C. Zhao, C. Deng, C. Hou, C. Xu, C. Shao, C. Ruan, C. Sun, D. Dai, D. Guo, D. Yang, D. Chen, D. Li, D. Ji, E. Li, F. Wei, F. Lin, F. Yuan, F. Xia, F. Dai, G. Hao, G. Chen, G. Cao, G. Meng, G. Li, H. Yu, H. Zhang, H. Xu, H. Li, H. Liang, H. Zhang, H. Luo, H. Wei, H. Yuan, H. Zhang, H. Luo, H. Chen, H. Ji, H. Zhang, H. Ding, H. Tang, H. Cao, H. Gao, H. Qu, H. Zeng, J. Yang, J. Zhu, J. Luo, J. Song, J. Yu, J. Huang, J. Cai, J. Liang, J. Zhou, J. Ye, J. Li, J. Xu, J. Hu, J. Yang, J. Chen, J. Yan, J. Chen, J. Zhou, J. Xiang, J. Yuan, J. Cheng, J. Zhou, J. Zhu, J. Yu, J. Sun, J. Ran, J. Jiang, J. Qiu, J. Li, J. Zheng, J. Song, K. Dong, K. Gao, K. Guan, K. Zhou, K. Huang, K. Yu, L. Wang, L. Zhang, L. Wang, L. Xia, L. Zhang, L. Zhao, L. Guo, L. Luo, L. Ma, L. Zhu, L. Wang, L. Cai, L. Zhang, L. Chen, M. Di, M. Xu, M. Mei, M. Wang, M. Zhang, M. Zhang, M. Tang, M. Li, M. Zhou, M. Han, N. Wang, P. Huang, P. Wang, P. Cong, P. Wang, P. Zhang, Q. Wang, Q. Zhu, Q. Li, Q. Chen, Q. Du, Q. Jiang, R. Tian, R. Xu, R. Lu, R. Xu, R. Ge, R. Zhang, R. Pan, R. Wang, R. Chen, R. Yin, R. Xu, R. Shen, R. Zhang, R. Chen, S. Liu, S. Lu, S. Sun, S. Zhou, S. Chen, S. Cai, S. Nie, S. Wu, S. Chen, S. Hu, S. Liu, S. Hu, S. Ma, S. Wang, S. Yu, S. Zhou, S. Pan, S. Yu, S. Zhou, T. Ni, T. Yun, T. Jin, T. Pei, T. Ye, T. Lin, T. Ji, T. Cui, T. Yue, T. Yu, T. Wang, W. Zhang, W. Xiao, W. Zeng, W. An, W. Zhao, W. Liu, W. Liang, W. Pang, W. Luo, W. Yao, W. Gao, W. Yang, W. Huang, W. Hou, W. Zhang, W. Ma, X. Gao, X. He, X. Wang, X. Wang, X. Bi, X. Liu, X. Wang, X. Chen, X. Zhang, X. Nie, X. Sun, X. Wang, X. Cheng, X. Liu, X. Xie, X. Liu, X. Liu, X. Yu, X. Li, X. Yang, X. Zhang, X. Chen, X. Wang, X. Su, X. Chen, X. Lin, X. Fu, Y. Yan, Y. Wang, Y. Ma, Y. Luo, Y. Zhang, Y. Xu, Y. Ma, Y. Huang, Y. Li, Y. Li, Y. Xu, Y. Zhao, Y. Sun, Y. Wang, Y. Qian, Y. Shao, Y. Yu, Y. Zhang, Y. Ding, Y. Shi, Y. Wu, Y. Xiong, Y. Ma, Y. He, Y. Tang, Y. Zhou, Y. Luo, Y. Zhong, Y. Piao, Y. Wang, Y. Zhang, Y. Chen, Y. Tan, Y. Wei, Y. Ma, Y. Liu, Y. Yang, Y. Guo, Y. Wu, Y. Wu, Y. Li, Y. Cheng, Y. Ou, Y. Xu, Y. Li, Y. Wang, Y. Yang, Y. Xu, Y. Wu, Y. Meng, Y. Zou, Y. Zha, Y. Xiong, Y. Chen, Y. Lin, Y. Cao, Y. Wang, Y. Zhang, Y. Yan, Y. Lin, Y. Gu, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. Zhou, Y. Huang, Z. Wu, Z. Wang, Z. Zhao, Z. Ren, Z. Zhang, Z. Sha, Z. Fu, Z. Ju, Z. Xu, Z. Xie, Z. Zhang, Z. Gao, Z. Hao, Z. Gou, Z. Ma, Z. Yan, Z. Shao, Z. Huang, Z. Chen, Z. Wu, Z. Ren, Z. Wu, Z. Li, Z. Zhang, Z. Xu, Z. Wang, Z. Qu, Z. Gu, Z. Zhu, Z. Li, Z. Zhang, Z. Xie, Z. Gao, Z. Wan, Z. Pan, and Z. Yao DeepSeek-v4: towards highly efficient million-token context intelligence. External Links: 2606.19348, Link Cited by: §1. GLM-5-Team et al. (2026) GLM-5-Team, :, A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, C. Zhu, C. Yin, C. Wang, G. Pan, H. Zeng, H. Zhang, H. Wang, H. Chen, J. Zhang, J. Jiao, J. Guo, J. Wang, J. Du, J. Wu, K. Wang, L. Li, L. Fan, L. Zhong, M. Liu, M. Zhao, P. Du, Q. Dong, R. Lu, Shuang-Li, S. Cao, S. Liu, T. Jiang, X. Chen, X. Zhang, X. Huang, X. Dong, Y. Xu, Y. Wei, Y. An, Y. Niu, Y. Zhu, Y. Wen, Y. Cen, Y. Bai, Z. Qiao, Z. Wang, Z. Wang, Z. Zhu, Z. Liu, Z. Li, B. Wang, B. Wen, C. Huang, C. Cai, C. Yu, C. Li, C. Hu, C. Zhang, D. Zhang, D. Lin, D. Yang, D. Wang, D. Ai, E. Zhu, F. Yi, F. Chen, G. Wen, H. Sun, H. Zhao, H. Hu, H. Zhang, H. Liu, H. Zhang, H. Peng, H. Tai, H. Zhang, H. Liu, H. Wang, H. Yan, H. Ge, H. Liu, H. Chu, J. Zhao, J. Wang, J. Zhao, J. Ren, J. Wang, J. Zhang, J. Gui, J. Zhao, J. Li, J. An, J. Li, J. Yuan, J. Du, J. Liu, J. Zhi, J. Duan, K. Zhou, K. Wei, K. Wang, K. Luo, L. Zhang, L. Sha, L. Xu, L. Wu, L. Ding, L. Chen, M. Li, N. Lin, P. Ta, Q. Zou, R. Song, R. Yang, S. Tu, S. Yang, S. Wu, S. Zhang, S. Li, S. Li, S. Fan, W. Qin, W. Tian, W. Zhang, W. Yu, W. Liang, X. Kuang, X. Cheng, X. Li, X. Yan, X. Hu, X. Ling, X. Fan, X. Xia, X. Zhang, X. Zhang, X. Pan, X. Zou, X. Zhang, Y. Liu, Y. Wu, Y. Li, Y. Wang, Y. Zhu, Y. Tan, Y. Zhou, Y. Pan, Y. Zhang, Y. Su, Y. Geng, Y. Yan, Y. Tan, Y. Bi, Y. Shen, Y. Yang, Y. Li, Y. Liu, Y. Wang, Y. Li, Y. Wu, Y. Zhang, Y. Duan, Y. Zhang, Z. Liu, Z. Jiang, Z. Yan, Z. Zhang, Z. Wei, Z. Chen, Z. Feng, Z. Yao, Z. Chai, Z. Wang, Z. Zhang, B. Xu, M. Huang, H. Wang, J. Li, Y. Dong, and J. Tang GLM-5: from vibe coding to agentic engineering. External Links: 2602.15763, Link Cited by: §1. Gu et al. (2023) Y. Gu, L. Dong, F. Wei, and M. Huang Minillm: on-policy distillation of large language models. arXiv preprint arXiv:2306.08543. Cited by: §1, §2.2, §2.2. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1. Han et al. (2026) Z. Han, T. Zhang, H. Wang, and Y. Sun Adaptive teacher exposure for self-distillation in llm reasoning. arXiv preprint arXiv:2605.11458. Cited by: §2.3. Hao et al. (2025) Z. Hao, H. Wang, H. Liu, J. Luo, J. Yu, H. Dong, Q. Lin, C. Wang, and J. Chen Rethinking entropy interventions in rlvr: an entropy change perspective. ArXiv abs/2510.10150. External Links: Link Cited by: §2.3. Harne et al. (2026) S. Harne, C. Karkar, Y. Pandya, A. Awadallah, and A. Nambi Privileged, but biased: how pi-conditioned teachers break self-distillation. arXiv preprint arXiv:2608.04794. Cited by: §2.3. Harvard–MIT Mathematics Tournament (2025) Harvard–MIT Mathematics Tournament HMMT February 2025: problems and solutions. External Links: Link Cited by: §4.1. He et al. (2026) Y. He, S. Kaur, A. Bhaskar, Y. Yang, J. Liu, N. Ri, L. Fowl, A. Panigrahi, D. Chen, and S. Arora Self-distillation zero: self-revision turns binary rewards into dense supervision. arXiv preprint arXiv:2604.12002. Cited by: §2.3. He et al. (2025) Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, Z. Zhang, R. Wang, Z. Tu, H. Mi, and D. Yu DeepMath-103k: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. ArXiv abs/2504.11456. External Links: Link Cited by: §A.1, §4.1. Hübotter et al. (2026) J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. K. Buening, C. Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802. Cited by: §2.3, §4.1. Jiang et al. (2026) Y. Jiang, F. Xie, Z. Jiang, and H. Zhang Distill skills into weights, not prompts: abstract skills as privileged signals for on-policy self-distillation. arXiv preprint arXiv:2608.09826. Cited by: §2.3. Jin et al. (2026) W. Jin, T. Min, Y. Yang, D. Wei, Y. Zhou, S. R. Kadhe, N. Baracaldo, and K. Lee Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079. Cited by: §A.4, §2.1, §2.2, §2.3, §4.1. Kalwar et al. (2026) D. Kalwar, V. Palod, and S. Kambhampati Masked distillation: internalizing the chain-of-thought in language models. arXiv preprint arXiv:2607.22629. Cited by: §2.3. Kaur et al. (2026) S. Kaur, N. Ri, Y. He, L. Fowl, and S. Arora Rethinking on-policy self-distillation for thinking models. arXiv preprint arXiv:2607.05184. Cited by: §2.3. Ke et al. (2026) J. Ke, Z. Wen, W. Li, C. He, and L. Zhang Respecting self-uncertainty in on-policy self-distillation for efficient llm reasoning. arXiv preprint arXiv:2605.13255. Cited by: §2.3. Kim et al. (2026) J. Kim, X. Luo, M. Kim, S. Lee, D. Kim, J. Jeon, D. Li, and Y. Yang Why does self-distillation (sometimes) degrade the reasoning capability of llms?. arXiv preprint arXiv:2603.24472. Cited by: §2.3. Ko et al. (2024) J. Ko, S. Kim, T. Chen, and S. Yun Distillm: towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898. Cited by: §2.2. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with pagedattention. External Links: 2309.06180, Link Cited by: §4.1. Lee et al. (2026) C. Lee, Z. Cheng, Y. Wang, R. Ni, S. Sahu, S. Zhang, and W. Campbell SEAD: competence-aware on-policy distillation via entropy-guided supervision. arXiv preprint arXiv:2606.28562. Cited by: §2.3. Li et al. (2026a) J. Li, H. Lin, H. Lu, K. Wen, Z. Yang, J. Gao, Y. Wu, and J. Zhang QuestA: expanding reasoning capacity in llms via question augmentation. External Links: 2507.13266, Link Cited by: §2.3. Li et al. (2026b) Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, and N. Ding Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. External Links: 2604.13016, Link Cited by: §2.2. Lin et al. (2025) B. Y. Lin, R. L. Bras, K. Richardson, A. Sabharwal, R. Poovendran, P. Clark, and Y. Choi ZebraLogic: on the scaling limits of llms for logical reasoning. ArXiv abs/2502.01100. External Links: Link Cited by: §4.1. Liu et al. (2026) Y. Liu, S. Bansal, B. Pang, Y. Li, Z. L. Liu, Y. Ming, Z. Ke, S. Joty, and S. Yavuz Procedural memory distillation: online reflection for self-improving language models. arXiv preprint arXiv:2607.01480. Cited by: §2.3. Lu and Lab (2025) K. Lu and T. M. Lab On-policy distillation. Thinking Machines Lab: Connectionism. Note: https://thinkingmachines.ai/blog/on-policy-distillation External Links: Document Cited by: §2.2, §2.2. math-ai (2023) math-ai AMC 2023. Note: Dataset compiled from the 2023 American Mathematics Competitions External Links: Link Cited by: §4.1. Minka (2005) T. P. Minka Divergence measures and message passing. External Links: Link Cited by: §2.1, §2.1. Olmo et al. (2026) T. Olmo, :, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. J. V. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. L. Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi Olmo 3. External Links: 2512.13961, Link Cited by: §4.1. Pan et al. (2026) L. Pan, S. Tao, Y. Zhai, L. Zhang, Z. Liu, B. Ding, A. Liu, and L. Wen RLCSD: reinforcement learning with contrastive on-policy self-distillation. arXiv preprint arXiv:2606.11709. Cited by: §2.3, §4.1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §4.1. Shen et al. (2026) Z. Shen, J. Tong, S. Yan, C. Shen, H. Chen, W. Ye, X. Hu, R. Miao, H. Wang, J. Zhao, et al. Purified opsd: on-policy self-distillation without losing how to think. arXiv preprint arXiv:2607.02234. Cited by: §2.3. Shenfeld et al. (2026) I. Shenfeld, M. Damani, J. Hübotter, and P. Agrawal Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897. Cited by: §2.3. Sheng et al. (2025) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu HybridFlow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, p. 1279–1297. External Links: Link, Document Cited by: §4.1. Shin et al. (2025) H. Shin, L. Ji, Y. Gong, S. Kim, E. Choi, and M. Seo Generative prompt internalization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 7338–7363. Cited by: §2.3. Shing et al. (2025) M. Shing, K. Misaki, H. Bao, S. Yokoi, and T. Akiba TAID: temporally adaptive interpolated distillation for efficient knowledge transfer in language models. External Links: 2501.16937, Link Cited by: §2.2. Snell et al. (2022) C. Snell, D. Klein, and R. Zhong Learning by distilling context. arXiv preprint arXiv:2209.15189. Cited by: §2.3. Wang et al. (2025) S. Wang, L. Yu, C. Gao, C. Zheng, S. Liu, R. Lu, K. Dang, X. Chen, J. Yang, Z. Zhang, Y. Liu, A. Yang, A. Zhao, Y. Yue, S. Song, B. Yu, G. Huang, and J. Lin Beyond the 80/20 rule: high-entropy minority tokens drive effective reinforcement learning for llm reasoning. External Links: 2506.01939, Link Cited by: §2.3. Wu et al. (2025) T. Wu, C. Tao, J. Wang, R. Yang, Z. Zhao, and N. Wong Rethinking kullback-leibler divergence in knowledge distillation for large language models. In Proceedings of the 31st International Conference on Computational Linguistics, p. 5737–5755. Cited by: §2.2. Xiao et al. (2026) Z. Xiao, Q. Ma, C. J. Chen, X. Chen, A. Atreya, H. Chen, and V. Ordonez Learning to reason by analogy via retrieval-augmented reinforcement fine-tuning. External Links: 2606.13680, Link Cited by: §2.3. Yan et al. (2025) J. Yan, Y. Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y. Cheng, and Y. Zhang Learning to reason under off-policy guidance. External Links: 2504.14945, Link Cited by: §2.3. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Link Cited by: §1, §4.1. Yang et al. (2026) C. Yang, C. Qin, Q. Si, M. Chen, N. Gu, D. Yao, Z. Lin, W. Wang, J. Wang, and N. Duan Self-distilled rlvr. arXiv preprint arXiv:2604.03128. Cited by: §2.3, §4.1. Ye et al. (2026) T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei On-policy context distillation for language models. arXiv preprint arXiv:2602.12275. Cited by: §2.3. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang DAPO: an open-source llm reinforcement learning system at scale. External Links: 2503.14476, Link Cited by: §2.3. Zhang et al. (2026) H. Zhang, C. Wang, K. Chen, Y. Pan, Y. Xiang, J. Wang, and M. Zhang Tailoring teaching to aptitude: direction-adaptive self-distillation for llm reasoning. arXiv preprint arXiv:2605.22263. Cited by: §2.3. Zhang et al. (2025) Y. Zhang, M. Li, D. Long, X. Zhang, H. Lin, B. Yang, P. Xie, A. Yang, D. Liu, J. Lin, F. Huang, and J. Zhou Qwen3 embedding: advancing text embedding and reranking through foundation models. ArXiv abs/2506.05176. External Links: Link Cited by: §A.4. Zhang and Math-AI (2024) Y. Zhang and T. Math-AI American invitational mathematics examination (aime) 2024. Cited by: §4.1. Zhang and Math-AI (2025) Y. Zhang and T. Math-AI American invitational mathematics examination (aime) 2025. Cited by: §4.1. Zhao et al. (2026) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. Cited by: §1, §2.2, §2.3, §4.1. Appendix A Experimental Details This section expands the experimental protocol in Section 4.1. Complete results and standard deviations are given in Appendix B. A.1 Data and Model Configurations Training prompts come from DeepMath-103K (He et al., 2025), filtered to the difficulty bands in Table 3. Within each configuration, the same reasoning mode is used for the student, teacher (where applicable), and evaluation. The Instruct configurations use Qwen3-4B-Instruct-2507 and Olmo-3-7B-Instruct. Table 3: Backbone configurations and DeepMath training difficulty bands. Backbone Reasoning mode Difficulty levels Qwen3-0.6B Thinking 1–3 Qwen3-1.7B Thinking 5–7 Qwen3-4B-Instruct-2507 Instruct 6–8 Olmo-3-7B-Instruct Instruct 7–10 For in-domain mathematical reasoning, we evaluate on AMC23, AIME’24, AIME’25, and HMMT’25 Feb. For out-of-domain evaluation, we use ZebraLogic for logical reasoning and HumanEval for code generation. We construct a fixed subset of 100 ZebraLogic puzzles by sampling four puzzles per grid size, using the same subset for all configurations. A.2 Shared Training Parameters and Budgets All methods and ablations use veRL with vLLM rollouts on a single 8×H200 node. They share the training settings in Table 4 and a budget of 360 optimizer steps. Table 4: Shared training parameters. Parameter Value Optimizer AdamW Learning rate 1×10−61× 10^-6 Warmup 100 steps Weight decay 0.010.01 Gradient-norm clipping 0.10.1 Prompts per rollout round 256 Responses per prompt G=8G=8 Mini-batch size 32 prompt groups Optimizer updates per round 8 Rollout temperature 1.01.0 Rollout top-p / top-k 0.950.95 / 20 Maximum rollout response length 16,384 tokens A.3 Evaluation Protocol We evaluate models with the settings in Table 5. We report mean@16 averaged over three evaluation seeds. Evaluation uses the student alone with the original question and the backbone’s corresponding reasoning mode. Table 5: Evaluation parameters. Batch sizes count prompts. Group Parameter Value Evaluation Responses per problem 16 Evaluation seeds 17, 42, 1234 Temperature 0.60.6 Top-p / top-k 0.950.95 / 20 Maximum prompt length 2,048 tokens Maximum response length Math reasoning 38,912 tokens Logic (ZebraLogic) 38,912 tokens Coding (HumanEval) 16,384 tokens Evaluation batch size Math reasoning 4 prompts Logic (ZebraLogic) 16 prompts Coding (HumanEval) 8 prompts A.4 Method-Specific Implementation and Exemplars Table 6 summarizes the settings that differ across methods. OPSD + EOPD adapts EOPD (Jin et al., 2026) by using OPSD’s reference-conditioned self-teacher. Distillation uses gradient-free teacher passes; EMA and snapshot updates follow the schedules below. Entropy is computed over the full vocabulary. Table 6: Method-specific objectives and training settings. Method Setting Value Description OPSD Teacher Frozen initial weights Current reference solution as context. Objective Forward KL Teacher-to-student distillation. Support / clipping Top-100 + tail; 0.050.05 Teacher-selected support; per-element cap. SDPO Teacher / EMA decay EMA; 0.950.95 Updated after each optimizer step. Teacher context Successful sibling Excludes the rollout being scored. JSD mixing α 0.50.5 Equal student–teacher mixture. Support Top-100 + tail Student-selected support. Importance-ratio cap 2.02.0 Upper cap on the distillation ratio. GRPO Reward / advantage Binary; group-normalized Verifiable correctness; normalized within each response group. Policy-ratio clip ϵ=0.2ε=0.2 Clipped policy-gradient objective. Reference / KL weight Frozen initial weights; 0.0010.001 KL regularization in the actor loss. RLSD Teacher / refresh Snapshot; every 10 steps Copies student weights every 10 optimizer steps. PPO / evidence clips ϵ=0.2ε=0.2; ϵw=0.2 _w=0.2 Policy-ratio and evidence-weight clipping. Modulation λ 0.5→00.5→ 0 Linear decay over 50 training steps. OPSD + EOPD Teacher Frozen initial weights Current reference solution as context. Entropy gate HT≥0.8H_T≥ 0.8 nats Adds forward KL to sampled reverse KL. Forward-KL coefficient 1.01.0 Weight of the entropy-gated term. Forward-KL support Top-16; no tail Renormalized teacher, full-vocabulary student; no per-element cap. E2-OPSD Teacher Frozen initial weights One retrieved solved neighbor as context. Objective BER + AGD Entropy gap sets direction and weight. Support / clipping Top-100 + tail; 0.050.05 Teacher-selected support; per-element cap. Exemplar candidates 15 neighbors Self-excluded; one draw per rollout. Exemplar construction and teacher prompt. Using Qwen3-Embedding-8B (Zhang et al., 2025), we build a self-excluded 15-neighbor index within each difficulty band. Each rollout draws one neighbor and its dataset reference solution, with sampling seeded by the training seed, step, problem, and rollout slot. The box reproduces the fixed user message with variable fields in braces. After applying the backbone’s native chat template and reasoning mode, the teacher scores the student’s continuation conditioned on student_prefix. Exemplars are used only during training. For Qwen3-1.7B, one-time embedding and neighbor-index construction took 219 seconds on eight H200 GPUs, accounting for 0.41% of the GPU time spent on rollout generation and actor updates over 360 optimizer steps. EGT teacher prompt Problem: current_problem Here is a reference solution to a DIFFERENT but related problem. It is not the problem you must solve, and its final answer is NOT the answer to your problem -- use it only as a guide to the strategies and techniques that transfer: === Reference Solution Begin === Related problem: related_problem Solution to the related problem: related_solution === Reference Solution End === After reading the reference solution above, make sure you understand the reasoning behind each step, then solve the original problem with whatever transfers from it. Please reason step by step, and put your final answer within . Assistant continuation: student_prefix The teacher scores the next token after this prefix; this line is not part of the user message. Appendix B Full Evaluation Results B.1 Evaluation Metrics For N problems, let ci(m)c_i(m) be the number of correct answers among m independently sampled responses to problem i. We compute mean@k=1N∑i=1Nci(k)k,pass@k^=1N∑i=1N[1−(n−ci(n)k)(nk)].mean@k= 1N _i=1^N c_i(k)k, pass@k= 1N _i=1^N [1- n-c_i(n)k nk ]. Here k≤n=16k≤ n=16, with (ak)=0 ak=0 for a<ka<k. Estimated pass@1 equals mean@16, so its duplicate table is omitted. Entries are mean ± population standard deviation (%) over three seeds; bold/underlined means indicate best/second-best distinct results. B.2 In-domain Mathematical Reasoning Table 7: Mathematical reasoning mean@16 (%). Avg. equally weights the four benchmarks. Math Reasoning (mean@16) Model Method AMC23 AIME24 AIME25 HMMT25 Avg. Qwen3-0.6B (Thinking) Base 45.81±0.9845.81±0.98 9.79±0.299.79±0.29 14.17±1.1214.17±1.12 8.26±0.438.26±0.43 19.51±0.1919.51±0.19 OPSD 45.88±0.8645.88±0.86 10.90±0.8610.90±0.86 14.17±0.5914.17±0.59 9.17±0.519.17±0.51 20.03±0.4920.03±0.49 SDPO 45.48±0.2745.48±0.27 9.38±0.349.38±0.34 14.58±1.3914.58±1.39 7.78±1.707.78±1.70 19.30±0.6219.30±0.62 GRPO 46.11±0.9846.11±0.98 9.79±0.179.79±0.17 16.39±0.2616.39±0.26 8.40±0.778.40±0.77 20.17±0.0320.17±0.03 RLSD 46.84±1.9546.84±1.95 10.97¯±0.49 10.97±0.49 15.14±0.6415.14±0.64 7.78±0.267.78±0.26 20.18¯±0.53 20.18±0.53 OPSD + EOPD 46.36±0.6246.36±0.62 9.79±0.959.79±0.95 15.90±0.3515.90±0.35 8.54±1.038.54±1.03 20.15±0.2420.15±0.24 E2-OPSD (Ours) 46.56¯±1.37 46.56±1.37 11.04±0.2911.04±0.29 16.18¯±0.35 16.18±0.35 8.89¯±1.39 8.89±1.39 20.67±0.4220.67±0.42 Qwen3-1.7B (Thinking) Base 75.50±0.2875.50±0.28 46.04±1.7846.04±1.78 34.58±0.2934.58±0.29 21.88±0.7821.88±0.78 44.50±0.2744.50±0.27 OPSD 74.87±0.4674.87±0.46 45.97±1.5845.97±1.58 35.00±1.3635.00±1.36 22.71¯±0.61 22.71±0.61 44.64±0.4144.64±0.41 SDPO 76.05¯±0.98 76.05±0.98 47.99±2.0447.99±2.04 35.28¯±1.72 35.28±1.72 21.88±1.2821.88±1.28 45.30¯±0.29 45.30±0.29 GRPO 74.65±0.0974.65±0.09 47.01±1.3947.01±1.39 35.21±0.9535.21±0.95 21.88±0.9021.88±0.90 44.69±0.4244.69±0.42 RLSD 75.85±0.2075.85±0.20 48.26¯±0.10 48.26±0.10 33.96±1.3933.96±1.39 21.81±0.5521.81±0.55 44.97±0.5244.97±0.52 OPSD + EOPD 74.27±0.1574.27±0.15 47.43±1.2847.43±1.28 34.24±1.2134.24±1.21 21.46±0.7821.46±0.78 44.35±0.0944.35±0.09 E2-OPSD (Ours) 79.02±0.4179.02±0.41 52.29±0.4552.29±0.45 39.24±0.2639.24±0.26 25.00±0.9025.00±0.90 48.89±0.3248.89±0.32 Qwen3-4B- Instruct-2507 Base 85.39±0.4885.39±0.48 61.74±0.4961.74±0.49 46.46±2.0446.46±2.04 29.03±0.7129.03±0.71 55.65±0.7955.65±0.79 OPSD 84.81±0.0984.81±0.09 62.36¯±1.39 62.36±1.39 45.63±1.4745.63±1.47 29.51¯±0.84 29.51±0.84 55.58±0.5255.58±0.52 SDPO 83.86±0.6983.86±0.69 61.32±1.3261.32±1.32 45.83±1.4545.83±1.45 27.50±1.0627.50±1.06 54.63±0.6654.63±0.66 GRPO 84.44±0.7784.44±0.77 61.67±0.4561.67±0.45 45.56±1.8345.56±1.83 29.10±1.7029.10±1.70 55.19±0.5555.19±0.55 RLSD 85.12±0.8485.12±0.84 62.22±0.9462.22±0.94 46.94¯±0.84 46.94±0.84 28.40±0.2628.40±0.26 55.67¯±0.30 55.67±0.30 OPSD + EOPD 85.47¯±0.49 85.47±0.49 62.01±2.2162.01±2.21 46.46±0.2946.46±0.29 28.40±0.2628.40±0.26 55.59±0.7155.59±0.71 E2-OPSD (Ours) 87.10±0.6387.10±0.63 62.85±1.0962.85±1.09 50.97±1.1350.97±1.13 30.21±0.6130.21±0.61 57.78±0.3457.78±0.34 Olmo-3-7B- Instruct Base 80.52±0.2980.52±0.29 52.85±1.2152.85±1.21 37.78±0.2037.78±0.20 23.26±0.3523.26±0.35 48.60±0.3948.60±0.39 OPSD 80.52±0.4080.52±0.40 52.78±1.3252.78±1.32 38.26±1.7938.26±1.79 22.36±0.4922.36±0.49 48.48±0.6948.48±0.69 SDPO 80.27±0.1280.27±0.12 51.32±0.7951.32±0.79 39.93¯±1.74 39.93±1.74 24.51±1.4324.51±1.43 49.01±0.1449.01±0.14 GRPO 80.92¯±1.08 80.92±1.08 52.64±0.3952.64±0.39 38.61±0.8038.61±0.80 23.47±0.5223.47±0.52 48.91±0.1848.91±0.18 RLSD 80.37±0.5180.37±0.51 53.61¯±1.09 53.61±1.09 38.82±0.1038.82±0.10 23.82±0.2623.82±0.26 49.16¯±0.31 49.16±0.31 OPSD + EOPD 79.62±0.4779.62±0.47 53.47±1.1953.47±1.19 39.17±2.1739.17±2.17 23.96¯±1.47 23.96±1.47 49.05±0.3749.05±0.37 E2-OPSD (Ours) 81.22±1.0581.22±1.05 56.25±1.3556.25±1.35 40.00±2.1340.00±2.13 23.89±0.3523.89±0.35 50.34±0.7550.34±0.75 Table 8: Mathematical reasoning unbiased pass@2 (%). Math Reasoning (pass@2) Model Method AMC23 AIME24 AIME25 HMMT25 Avg. Qwen3-0.6B (Thinking) Base 56.41±0.9756.41±0.97 14.05±1.1014.05±1.10 19.74±1.6319.74±1.63 11.48±0.1711.48±0.17 25.42±0.3525.42±0.35 OPSD 56.63±1.4656.63±1.46 16.39±1.2316.39±1.23 19.95±1.0519.95±1.05 12.79±0.8012.79±0.80 26.44±0.7626.44±0.76 SDPO 56.11±0.1056.11±0.10 14.44±0.8014.44±0.80 20.19±1.8820.19±1.88 10.79±2.1710.79±2.17 25.38±0.7025.38±0.70 GRPO 56.75±0.9656.75±0.96 14.69±0.9314.69±0.93 22.94±0.1622.94±0.16 11.56±1.0711.56±1.07 26.49±0.2226.49±0.22 RLSD 57.57±2.4357.57±2.43 16.79¯±0.73 16.79±0.73 21.13±0.3621.13±0.36 10.72±0.4010.72±0.40 26.55¯±0.39 26.55±0.39 OPSD + EOPD 56.52±0.5656.52±0.56 14.85±2.1414.85±2.14 21.94±0.2421.94±0.24 11.03±1.7511.03±1.75 26.08±0.3126.08±0.31 E2-OPSD (Ours) 56.81¯±0.77 56.81±0.77 17.17±0.8217.17±0.82 22.28¯±0.54 22.28±0.54 12.20¯±1.39 12.20±1.39 27.11±0.3727.11±0.37 Qwen3-1.7B (Thinking) Base 82.44±0.3682.44±0.36 59.14±1.9059.14±1.90 41.86±0.1941.86±0.19 25.96±1.2425.96±1.24 52.35±0.2452.35±0.24 OPSD 82.34±0.4882.34±0.48 58.31±1.9458.31±1.94 43.56¯±1.89 43.56±1.89 27.72¯±1.12 27.72±1.12 52.99±0.7952.99±0.79 SDPO 82.85±0.7282.85±0.72 59.87±2.2159.87±2.21 42.85±2.4842.85±2.48 26.75±2.2126.75±2.21 53.08¯±0.70 53.08±0.70 GRPO 81.76±0.2381.76±0.23 59.20±1.0259.20±1.02 42.74±1.0942.74±1.09 26.76±1.3926.76±1.39 52.62±0.5752.62±0.57 RLSD 83.14¯±0.39 83.14±0.39 61.35¯±0.21 61.35±0.21 41.59±2.1741.59±2.17 26.10±1.0126.10±1.01 53.05±0.6653.05±0.66 OPSD + EOPD 81.14±0.8581.14±0.85 60.64±1.7160.64±1.71 41.79±1.0041.79±1.00 25.77±1.2125.77±1.21 52.33±0.1452.33±0.14 E2-OPSD (Ours) 86.10±0.2886.10±0.28 62.86±0.6362.86±0.63 49.32±0.9849.32±0.98 29.95±1.0829.95±1.08 57.06±0.5257.06±0.52 Qwen3-4B- Instruct-2507 Base 89.44±0.2089.44±0.20 71.11±0.1771.11±0.17 56.45±2.3856.45±2.38 33.00±0.9333.00±0.93 62.50±0.8262.50±0.82 OPSD 89.14±0.4189.14±0.41 71.94¯±1.83 71.94±1.83 56.09±1.2956.09±1.29 34.26±1.4234.26±1.42 62.86¯±0.68 62.86±0.68 SDPO 88.79±0.1988.79±0.19 71.28±0.3371.28±0.33 55.66±2.0455.66±2.04 31.97±1.2131.97±1.21 61.92±0.7161.92±0.71 GRPO 89.26±0.5589.26±0.55 71.44±0.5071.44±0.50 54.86±2.0654.86±2.06 33.25±2.1433.25±2.14 62.20±0.1962.20±0.19 RLSD 89.54±0.8389.54±0.83 72.16±0.1972.16±0.19 56.64¯±0.95 56.64±0.95 32.80±0.2132.80±0.21 62.78±0.1762.78±0.17 OPSD + EOPD 89.60¯±0.40 89.60±0.40 71.25±2.7971.25±2.79 55.30±1.0055.30±1.00 32.44±0.4832.44±0.48 62.14±0.6462.14±0.64 E2-OPSD (Ours) 90.92±0.3990.92±0.39 70.65±0.7670.65±0.76 60.89±1.4860.89±1.48 34.25¯±0.65 34.25±0.65 64.18±0.3364.18±0.33 Olmo-3-7B- Instruct Base 87.40±0.3087.40±0.30 63.84±0.6563.84±0.65 47.07±0.7347.07±0.73 28.08±0.9228.08±0.92 56.60±0.5056.60±0.50 OPSD 87.37±0.4187.37±0.41 63.90¯±1.77 63.90±1.77 46.72±1.6446.72±1.64 26.38±0.3126.38±0.31 56.09±0.9356.09±0.93 SDPO 87.38±0.1387.38±0.13 62.15±1.2762.15±1.27 49.31±1.5449.31±1.54 29.88±0.9429.88±0.94 57.18¯±0.65 57.18±0.65 GRPO 88.14±0.7488.14±0.74 62.26±0.3562.26±0.35 46.99±0.7646.99±0.76 28.76±1.3528.76±1.35 56.54±0.2256.54±0.22 RLSD 87.53±0.4587.53±0.45 63.67±1.7663.67±1.76 48.24±0.8148.24±0.81 29.07¯±0.34 29.07±0.34 57.13±0.4057.13±0.40 OPSD + EOPD 87.00±0.2887.00±0.28 63.10±0.7763.10±0.77 48.56±2.6748.56±2.67 28.94±1.5228.94±1.52 56.90±0.5456.90±0.54 E2-OPSD (Ours) 87.92¯±0.81 87.92±0.81 67.39±1.3567.39±1.35 48.86¯±1.97 48.86±1.97 28.86±0.8628.86±0.86 58.26±0.5958.26±0.59 Table 9: Mathematical reasoning unbiased pass@4 (%). Math Reasoning (pass@4) Model Method AMC23 AIME24 AIME25 HMMT25 Avg. Qwen3-0.6B (Thinking) Base 66.30±0.6766.30±0.67 19.70±2.1519.70±2.15 24.67±1.9824.67±1.98 15.02±0.3115.02±0.31 31.42±0.7131.42±0.71 OPSD 66.28±1.9466.28±1.94 23.39±2.3323.39±2.33 25.01±1.6125.01±1.61 16.84±1.1916.84±1.19 32.88±1.1232.88±1.12 SDPO 65.01±0.5765.01±0.57 21.34±1.4621.34±1.46 25.35±2.8525.35±2.85 14.25±2.3014.25±2.30 31.49±0.9131.49±0.91 GRPO 66.38¯±0.94 66.38±0.94 20.94±2.3120.94±2.31 29.06±0.4829.06±0.48 14.94±1.5814.94±1.58 32.83±0.5832.83±0.58 RLSD 66.82±2.2666.82±2.26 23.79¯±1.18 23.79±1.18 27.00±0.9027.00±0.90 14.42±0.7714.42±0.77 33.01¯±0.01 33.01±0.01 OPSD + EOPD 65.42±0.6765.42±0.67 21.59±3.9021.59±3.90 27.59±0.9227.59±0.92 13.86±2.4713.86±2.47 32.11±0.7532.11±0.75 E2-OPSD (Ours) 65.85±0.2865.85±0.28 25.02±1.3325.02±1.33 27.65¯±0.67 27.65±0.67 16.09¯±1.30 16.09±1.30 33.65±0.1633.65±0.16 Qwen3-1.7B (Thinking) Base 87.14±0.4387.14±0.43 68.65±2.2868.65±2.28 48.73±0.6448.73±0.64 31.31±2.0831.31±2.08 58.96±0.7558.96±0.75 OPSD 87.65±0.3187.65±0.31 67.16±2.8967.16±2.89 51.35¯±2.24 51.35±2.24 33.75¯±1.32 33.75±1.32 59.97¯±1.17 59.97±1.17 SDPO 87.72±0.8787.72±0.87 69.17±1.6269.17±1.62 49.76±2.7549.76±2.75 32.14±2.9032.14±2.90 59.69±0.8759.69±0.87 GRPO 86.83±0.3586.83±0.35 69.06±1.0369.06±1.03 49.83±0.4949.83±0.49 33.23±2.2733.23±2.27 59.74±0.7559.74±0.75 RLSD 88.02¯±0.72 88.02±0.72 70.87±0.6570.87±0.65 49.51±1.8449.51±1.84 31.29±1.5431.29±1.54 59.92±0.4859.92±0.48 OPSD + EOPD 85.68±1.8185.68±1.81 70.87±2.2370.87±2.23 49.64±1.0049.64±1.00 31.13±2.2531.13±2.25 59.33±0.4059.33±0.40 E2-OPSD (Ours) 90.91±0.7390.91±0.73 70.54¯±1.37 70.54±1.37 57.71±1.9857.71±1.98 34.83±1.2634.83±1.26 63.50±0.9963.50±0.99 Qwen3-4B- Instruct-2507 Base 91.67±0.1291.67±0.12 77.81±0.2477.81±0.24 64.22±2.1664.22±2.16 36.89±1.1336.89±1.13 67.65±0.6467.65±0.64 OPSD 91.36±0.7791.36±0.77 78.85±1.3978.85±1.39 64.51±1.0564.51±1.05 38.20±2.2738.20±2.27 68.23±0.7268.23±0.72 SDPO 91.75±0.4891.75±0.48 78.45±1.1278.45±1.12 62.88±2.1162.88±2.11 36.11±1.1036.11±1.10 67.30±0.7167.30±0.71 GRPO 91.93±0.5691.93±0.56 79.07¯±0.99 79.07±0.99 61.84±2.3461.84±2.34 37.09±3.1337.09±3.13 67.48±0.4567.48±0.45 RLSD 91.99¯±0.94 91.99±0.94 79.73±0.6579.73±0.65 64.67¯±1.17 64.67±1.17 36.79±0.6936.79±0.69 68.30¯±0.27 68.30±0.27 OPSD + EOPD 91.93±0.3391.93±0.33 77.64±3.0577.64±3.05 61.66±1.8961.66±1.89 36.41±1.1536.41±1.15 66.91±0.3466.91±0.34 E2-OPSD (Ours) 93.00±0.5293.00±0.52 76.29±0.3276.29±0.32 67.94±1.3867.94±1.38 38.04¯±0.97 38.04±0.97 68.82±0.3668.82±0.36 Olmo-3-7B- Instruct Base 90.69±0.3390.69±0.33 73.01±1.0273.01±1.02 55.42±1.9055.42±1.90 32.72±1.3032.72±1.30 62.96±0.7462.96±0.74 OPSD 90.76±0.7290.76±0.72 73.33¯±1.87 73.33±1.87 54.37±1.3654.37±1.36 29.51±0.5429.51±0.54 61.99±1.0761.99±1.07 SDPO 90.77±0.2590.77±0.25 72.20±1.2872.20±1.28 57.58±1.4257.58±1.42 34.62±1.9034.62±1.90 63.79¯±0.86 63.79±0.86 GRPO 91.90±0.5291.90±0.52 70.90±0.9870.90±0.98 53.90±0.3553.90±0.35 33.33±2.4533.33±2.45 62.51±0.3662.51±0.36 RLSD 91.04±0.5491.04±0.54 72.45±1.6672.45±1.66 56.75¯±0.87 56.75±0.87 33.39±1.1233.39±1.12 63.41±0.6363.41±0.63 OPSD + EOPD 91.02±0.1391.02±0.13 71.88±0.1871.88±0.18 56.71±2.5456.71±2.54 33.55¯±0.92 33.55±0.92 63.29±0.6363.29±0.63 E2-OPSD (Ours) 91.25¯±0.66 91.25±0.66 76.34±1.4476.34±1.44 56.06±1.7856.06±1.78 33.49±1.7333.49±1.73 64.28±0.7464.28±0.74 B.3 Out-of-domain Evaluation Table 10: Out-of-domain evaluation on logical reasoning and code generation. ZebraLogic uses whole-puzzle correctness; best means are bold within each model and dataset. Model Method mean@16 pass@2 pass@4 pass@8 Logic: ZebraLogic Base 28.83±0.3028.83±0.30 38.93±0.3538.93±0.35 48.54±0.2348.54±0.23 56.47±0.6356.47±0.63 Qwen3-0.6B (Thinking) E2-OPSD (Ours) 28.96±0.2128.96±0.21 39.27±0.3239.27±0.32 48.86±0.5348.86±0.53 56.54±0.7156.54±0.71 Base 62.54±1.0562.54±1.05 69.14±0.9969.14±0.99 74.19±1.2174.19±1.21 78.26±1.3278.26±1.32 Qwen3-1.7B (Thinking) E2-OPSD (Ours) 67.35±0.4367.35±0.43 73.57±0.6273.57±0.62 78.75±0.8478.75±0.84 83.77±0.9883.77±0.98 Base 79.15±0.1679.15±0.16 85.65±0.2385.65±0.23 89.61±0.4189.61±0.41 92.35±0.6492.35±0.64 Qwen3-4B-Instruct-2507 E2-OPSD (Ours) 82.23±0.2182.23±0.21 87.57±0.1087.57±0.10 90.65±0.3590.65±0.35 92.83±0.8292.83±0.82 Base 32.69±0.3932.69±0.39 42.38±0.3742.38±0.37 50.61±0.2550.61±0.25 56.56±0.0556.56±0.05 Olmo-3-7B-Instruct E2-OPSD (Ours) 33.90±0.4333.90±0.43 43.96±0.5343.96±0.53 52.16±0.7152.16±0.71 57.94±0.9557.94±0.95 Coding: HumanEval Base 52.55±0.3152.55±0.31 63.96±0.4563.96±0.45 72.60±0.3472.60±0.34 78.42±0.4878.42±0.48 Qwen3-0.6B (Thinking) E2-OPSD (Ours) 53.09±0.7053.09±0.70 64.26±0.7664.26±0.76 72.84±0.7172.84±0.71 79.01±0.7279.01±0.72 Base 83.75±0.2183.75±0.21 88.91±0.2288.91±0.22 91.55±0.1791.55±0.17 93.17±0.4893.17±0.48 Qwen3-1.7B (Thinking) E2-OPSD (Ours) 84.50±0.1384.50±0.13 89.12±0.1989.12±0.19 91.79±0.3491.79±0.34 93.77±0.3493.77±0.34 Base 83.96±0.1683.96±0.16 86.41±0.2486.41±0.24 88.37±0.3688.37±0.36 89.90±0.5189.90±0.51 Qwen3-4B-Instruct-2507 E2-OPSD (Ours) 84.76±0.2284.76±0.22 87.13±0.0387.13±0.03 88.92±0.1288.92±0.12 90.67±0.2090.67±0.20 Base 78.18±0.5178.18±0.51 85.33±0.4785.33±0.47 89.68±0.4489.68±0.44 92.05±0.2592.05±0.25 Olmo-3-7B-Instruct E2-OPSD (Ours) 78.66±0.1978.66±0.19 85.78±0.3485.78±0.34 90.14±0.5290.14±0.52 92.52±0.7492.52±0.74 B.4 Ablation Results Table 11: Per-benchmark mean@16 with standard deviations for the component ablation on Qwen3-1.7B Thinking (Figure 5(a)). Avg. equally weights the four benchmarks. Best means are bold and second-best distinct means underlined. AGD denotes adaptive gap damping. Method AMC23 AIME24 AIME25 HMMT25 Avg. Base 75.5075.50 ±0.28± 0.28 46.0446.04 ±1.78± 1.78 34.5834.58 ±0.29± 0.29 21.8821.88 ±0.78± 0.78 44.5044.50 ±0.27± 0.27 OPSD 74.8774.87 ±0.46± 0.46 45.9745.97 ±1.58± 1.58 35.0035.00 ±1.36± 1.36 22.7122.71 ±0.61± 0.61 44.6444.64 ±0.41± 0.41 + EGT 76.4676.46 ±0.99± 0.99 48.2648.26 ±1.21± 1.21 36.8136.81 ±1.49± 1.49 22.5722.57 ±0.20± 0.20 46.0246.02 ±0.40± 0.40 + BER 77.2677.26 ±0.38± 0.38 52.0152.01 ±1.13± 1.13 37.1537.15 ±0.94± 0.94 24.0324.03 ±0.35± 0.35 47.6147.61 ±0.64± 0.64 + BER + AGD 77.66¯ 77.66 ±0.68± 0.68 52.15¯ 52.15 ±1.58± 1.58 39.4439.44 ±0.69± 0.69 24.31¯ 24.31 ±1.83± 1.83 48.39¯ 48.39 ±0.55± 0.55 E2-OPSD (Ours) 79.0279.02 ±0.41± 0.41 52.2952.29 ±0.45± 0.45 39.24¯ 39.24 ±0.26± 0.26 25.0025.00 ±0.90± 0.90 48.8948.89 ±0.32± 0.32 Appendix C Mechanism Probe Settings and Results C.1 Rollout and Probe Settings Figures 5(b,c) use the same diagnostic batch of student responses. We sample 256 distinct problems from the DeepMath level 5–7 training pool with seed 42 and generate eight responses per problem. All three conditions use the same frozen Qwen3-1.7B base weights in Thinking mode, with no parameter updates. The student receives only the original problem; the reference-conditioned teacher additionally receives its reference solution and answer; the EGT teacher instead receives one retrieved solved neighbor. We retain the same problems and exemplar assignments when generating the teachers’ own responses. Table 12: Diagnostic rollout settings. Indices i and g start at zero. The two teacher conditions share sampling seeds. Teacher generation budgets are also limited by the remaining context window. Parameter Value Problems / responses per problem 256 / 8 Sampling temperature 1.0 Top-p / top-k 0.95 / 20 Maximum response length 16,384 tokens Maximum total context length 40,960 tokens Student prompt length limit 2,048 tokens Student sampling seed 42+i42+i for problem index i Teacher sampling seed 42+8i+g42+8i+g for response slot g Entropy and KL scoring temperature 1.0 Teacher-context probe. For Figure 5(b), we score each teacher both on its own generated response and on the corresponding student response. Entropy and teacher-to-student forward KL are computed over the full vocabulary, without top-k truncation or clipping. We average over response tokens within each trajectory, then equally over 2,047 matched response slots; one slot is excluded because the EGT prompt and student response exceed the context window. Response lengths use all 2,048 generations per condition and include generated end-of-sequence tokens. Relative entropy increase is the difference between the two mean entropies divided by the own-prefix mean, rather than an average of per-response ratios. AGD gradient probe. For Figure 5(c), we reuse all 2,048 student responses with the reference-conditioned teacher. At each response position, we compute the ℓ2 _2 norm of the routed loss gradient with respect to the full student logit vector, using teacher top-100 support plus a tail bucket and a per-element cap of 0.05. We compare the same fixed distributions with AGD off and on, stopping gradients through the routing decision and damping weight. The gap scale is computed over valid tokens in fixed batches of 32 responses, ordered by problem and response slot. Gradient shares are normalized within each response, averaged over responses within a problem, and then over problems. High entropy means H≥ln2H≥ 2; the first and second letters in H, HL, LH, and L refer to student and teacher entropy, respectively. This is an offline logit-gradient probe, not a replay of parameter updates during training. C.2 Detailed Probe Results Table 13 reports the teacher-context comparison. EGT reduces forward KL on the same student prefixes by 25.9%. Own-prefix entropy and response length describe each condition’s separate trajectories; the forward-KL comparison uses shared student prefixes. Table 13: Teacher-context probe underlying Figure 5(b). Lengths use 2,048 responses per condition; entropy and KL use 2,047 matched slots. Student zeros denote identical-prefix or self comparisons. Metric Student Reference teacher EGT teacher Mean response length (tokens) 9,139.7 835.6 2,089.5 Own-prefix entropy (nats) 0.2679 0.1900 0.2605 Forward KL to student (nats) 0 0.1549 0.1148 Table 14 reports all four entropy buckets for Figure 5(c). HL and LH together contain 1,436,048 of 18,718,121 response tokens (7.67%). Their gradient shares increase by 3.43 and 4.89 percentage points, respectively. Table 14: AGD gradient probe on 2,048 fixed student responses. Token shares pool all response tokens; gradient shares use the response- and problem-level normalization described in Appendix C.1. Changes are computed from the displayed gradient shares. Bucket Token share (%) AGD off (%) AGD on (%) Change (p) H 14.40 45.57 43.26 −2.31-2.31 HL 2.33 8.99 12.42 +3.43+3.43 LH 5.34 11.19 16.08 +4.89+4.89 L 77.93 34.25 28.24 −6.01-6.01 Distributional agreement at strongly damped positions. Using the same reference-teacher gradient probe, we compute full-vocabulary probability-mass overlap Ot=∑vminpt(v),qt(v)O_t= _v \p_t(v),q_t(v)\ on student prefixes. Table 15 reports token-pooled statistics for strongly damped positions (ct≤0.1c_t≤ 0.1). Table 15: Overlap at strongly damped positions. Percentages are computed within each row. Subset Positions Mean overlap Ot<0.5O_t<0.5 (%) All strongly damped positions 12,708,962 0.9848 0.380 with H(pt)≥ln2H(p_t)≥ 2 766,682 0.8343 3.662 Positions with both ct≤0.1c_t≤ 0.1 and Ot<0.5O_t<0.5 account for 0.464% of ∑tgt _tg_t pooled over all response tokens, where gt=‖∇ztDtBER‖2g_t=\| _z_tD_t^BER\|_2 is the logit-gradient norm before damping.