Paper deep dive
SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation
Zikun Qu, Min Zhang, Mingze Kong, Zhiwei Shang, Yikun Ban, Shuang Qiu, Zhongxiang Dai
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/9/2026, 3:42:41 AM
Summary
The paper introduces SPOT (Sparse Probing and Outcome Calibration for On-Policy Distillation), a method to improve on-policy distillation (OPD) by addressing the limitations of standard reverse-KL training. SPOT uses a three-stage acquisition-exploration-exploitation procedure to select positions for probing based on teacher entropy, candidate mass, and student-teacher mismatch, then evaluates candidate continuations via a verifier, and finally constructs outcome-calibrated targets that balance teacher priors with downstream success probabilities.
Entities (9)
Relation Signals (6)
SPOT → improves → On-Policy Distillation
confidence 95% · SPOT addresses two coupled decisions... through an acquisition–exploration–exploitation procedure.
SPOT → usesstudent → Qwen3-0.6B-Base
confidence 95% · Qwen3-0.6B-Base... as students
SPOT → usesteacher → Qwen3-8b
confidence 95% · We use Qwen3-8B... as the common teacher for all distillation methods
SPOT → outperforms → EOPD
confidence 92% · Relative to EOPD, the closest uncertainty-aware OPD baseline, SPOT improves macro Avg@8 by 0.29–0.68 points...
SPOT → evaluateson → MATH
confidence 90% · Extensive experiments across multiple student models and reasoning benchmarks demonstrate the effectiveness of SPOT...
SPOT → uses → Reverse KL
confidence 90% · SPOT applies this additional local loss only at positions where at least one tested candidate receives positive verifier reward... standard reverse-KL training...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy distillation (OPD) provides dense teacher supervision on student-generated trajectories, but standard reverse-KL training can assign insufficient probability to other plausible continuations. Teacher entropy alone does not reveal whether uncertainty is concentrated among a few plausible next tokens or dispersed over a long probability tail, nor whether the student already represents those candidates well. Moreover, local teacher probabilities may not predict downstream success. We introduce Sparse Probing and Outcome-calibrated Targets OPD (SPOT), which addresses two coupled decisions, where to probe and what to distill, through an acquisition--exploration--exploitation procedure. During acquisition, a position-level score combines normalized teacher entropy, the probability mass captured by a small top-$k$ candidate set, and student--teacher mismatch to allocate a limited probing budget. During exploration, SPOT evaluates teacher-proposed candidates through verifier-scored student continuations. During exploitation, these outcomes produce a closed-form, KL-regularized target that favors candidates with better downstream outcomes while remaining anchored to the teacher distribution. Extensive experiments across multiple student models and reasoning benchmarks demonstrate the effectiveness of SPOT in improving reasoning performance while balancing solution quality and coverage.
Tags
Links
- Source: https://arxiv.org/abs/2608.04419v1
- Canonical: https://arxiv.org/abs/2608.04419v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
67,621 characters extracted from source content.
Expand or collapse full text
SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation Zikun Qu1, Min Zhang2 , Mingze Kong1, Zhiwei Shang1, Yikun Ban3, Shuang Qiu4, Zhongxiang Dai1∗ 1The Chinese University of Hong Kong, Shenzhen, 2East China Normal University, 3Beihang University, 4City University of Hong Kong Corresponding authors Abstract On-policy distillation (OPD) provides dense teacher supervision on student-generated trajectories, but standard reverse-KL training can assign insufficient probability to other plausible continuations. Teacher entropy alone does not reveal whether uncertainty is concentrated among a few plausible next tokens or dispersed over a long probability tail, nor whether the student already represents those candidates well. Moreover, local teacher probabilities may not predict downstream success. We introduce Sparse Probing and Outcome-calibrated Targets OPD (SPOT), which addresses two coupled decisions, where to probe and what to distill, through an acquisition–exploration–exploitation procedure. During acquisition, a position-level score combines normalized teacher entropy, the probability mass captured by a small top-k candidate set, and student–teacher mismatch to allocate a limited probing budget. During exploration, SPOT evaluates teacher-proposed candidates through verifier-scored student continuations. During exploitation, these outcomes produce a closed-form, KL-regularized target that favors candidates with better downstream outcomes while remaining anchored to the teacher distribution. Extensive experiments across multiple student models and reasoning benchmarks demonstrate the effectiveness of SPOT in improving reasoning performance while balancing solution quality and coverage. 1 Introduction Transferring the reasoning capabilities of large language models to smaller student models is an important goal in LLM post-training. Supervised fine-tuning and off-policy distillation leverage expert- or teacher-generated trajectories, but train the student on contexts that differ from those induced by its own predictions, leading to exposure bias and compounding errors at inference (Bengio et al., 2015; Ranzato et al., 2016). On-policy reinforcement learning (RL) instead optimizes student-generated rollouts, but reasoning-oriented RL typically relies on sequence-level or terminal verifier rewards, providing limited fine-grained credit assignment (Shao et al., 2024; Guo et al., 2025). On-policy distillation (OPD) combines these advantages: it trains on student-generated prefixes while retaining dense token-level teacher feedback (Agarwal et al., 2024; Lu and Lab, 2025). Yet standard OPD typically minimizes reverse KL, whose mode-seeking behavior favors the teacher’s dominant continuation but can assign insufficient probability to other plausible continuations, potentially limiting solution coverage. EOPD addresses this limitation by using teacher entropy as a trigger: at high-entropy positions, it augments reverse-KL training with a top-k approximation of forward KL to preserve plausible local alternatives (Minka, 2005; Jin et al., 2026). However, high entropy alone does not reveal whether the teacher’s uncertainty is concentrated among a few plausible next tokens or dispersed over a long probability tail. Nor does it reveal whether the student already assigns sufficient probability to and similarly ranks those plausible tokens. Evaluating an alternative next token requires rolling out a student continuation and checking its final outcome, making such probing costly. An entropy threshold can flag uncertain positions, but by itself cannot prioritize where a limited probing budget will be most useful. The first challenge is therefore to allocate that budget to positions where the teacher assigns substantial probability to a small set of alternatives that the student does not yet represent well. Selecting a candidate position does not yet determine what to distill. On student-generated prefixes, the teacher’s local next-token probabilities need not predict downstream success: a token assigned higher probability by the teacher may yield an unsuccessful continuation under the current student policy, whereas a lower-probability alternative may yield a successful one (Li et al., 2026; Hou et al., 2026). We therefore treat the teacher distribution as a proposal prior over candidate branches, rather than as definitive evidence of downstream success, and calibrate it using the outcomes of student continuations. Thus, teacher uncertainty and student mismatch help prioritize where to acquire additional evidence, while verified downstream outcomes determine how that evidence should modify the supervision target. These observations motivate Sparse Probing and Outcome-calibrated Targets for on-policy distillation (SPOT), which reframes uncertainty-aware distillation around two coupled decisions: where to probe and what to distill. SPOT addresses them through a three-stage acquisition–exploration–exploitation procedure. During acquisition, a lightweight position-level score sts_t prioritizes positions that satisfy three conditions: the teacher assigns meaningful probability to multiple next tokens, most of the teacher’s probability mass lies within a small top-k candidate set, and the student either underweights or differently ranks those candidates. Because these factors are multiplied, a low value on any one condition lowers the position’s overall probing priority. The score therefore estimates where additional evidence may be useful; it does not identify which candidates yield successful student continuations, which is assessed only after the candidates are rolled out and verified. During exploration, at each selected position, SPOT appends each candidate from the teacher’s top-k set in turn, rolls out a continuation under the student policy, and evaluates the completed continuation with a verifier. During exploitation, the verified continuation values produce a closed-form, KL-regularized target: candidates with better downstream outcomes receive more probability, while the target remains anchored to the teacher distribution. SPOT applies this additional local loss only at positions where at least one tested candidate receives positive verifier reward. This allows downstream evidence to modify the local token-level target rather than serving only as a trajectory-level score. In summary, our contributions are threefold: • A two-decision formulation. We formulate selective supervision in OPD as determining where additional outcome evidence is worth acquiring and how that evidence should be converted into a supervision target, separating position selection from target construction. • Sparse probing and outcome-calibrated targets. We propose SPOT, which probes positions via teacher uncertainty, top-k mass, and student mismatch, then derives a closed-form target by reward-tilting the teacher distribution with verifier-scored student continuations. • Empirical validation. Across three evaluated Qwen student scales and six mathematical reasoning benchmarks, SPOT achieves the highest macro Pass@8 in all three settings and the highest or second-highest macro Avg@8 among the compared methods. These results indicate stronger multi-sample solution coverage while maintaining competitive average accuracy. 2 Preliminaries On-Policy Distillation. Let D denote the prompt distribution, and let πθ _θ and πT _T be the student and teacher policies over a shared vocabulary V. Given q∼q , OPD samples a student trajectory x=(x1,…,xT)∼πθ(⋅∣q)x=(x_1,…,x_T) _θ(· q) and queries the teacher on each student-induced prefix ct=(q,x<t)c_t=(q,x_<t). At each prefix, it uses the reverse-KL loss ℒtOPD:=DKL(πθ(⋅∣ct)∥πT(⋅∣ct))L^OPD_t:=D_KL( _θ(· c_t)\| _T(· c_t)). Averaging over the student trajectory gives ℒOPD(θ)=q∼,x∼πθ(⋅∣q)[1T∑t=1TℒtOPD].L_OPD(θ)=E_q ,\,x _θ(· q)\! [ 1T _t=1^TL^OPD_t ]. (1) In practice, this objective is estimated from tokens sampled by a frozen behavior policy πθold _ _old and optimized with PPO-style clipping. Querying the teacher on student-visited prefixes reduces state-distribution mismatch and provides dense token-level feedback. However, the mode-seeking reverse KL can under-cover plausible alternatives when the teacher is uncertain. Entropy-Aware On-Policy Distillation. EOPD (Jin et al., 2026) augments OPD with mode-covering supervision at positions of high teacher entropy. The raw entropy and its vocabulary-normalized counterpart are HT(ct):=−∑v∈πT(v∣ct)logπT(v∣ct),H¯T(ct):=HT(ct)log||∈[0,1].H_T(c_t):=- _v _T(v c_t) _T(v c_t), H_T(c_t):= H_T(c_t) |V|∈[0,1]. (2) EOPD gates supervision using HTH_T, whereas our acquisition score later uses the bounded, vocabulary-normalized entropy H¯T H_T. For efficiency, let Stk:=TopKk(πT(⋅∣ct))S_t^k:=TopK_k( _T(· c_t)) denote the teacher’s top-k candidate set. For j∈T,θj∈\T,θ\, write π¯jk(v∣ct):=πj(v∣ct)/∑u∈Stkπj(u∣ct) π_j^k(v c_t):= _j(v c_t)/ _u∈ S_t^k _j(u c_t) for the restriction of πj _j renormalized on this candidate set. EOPD approximates forward KL as ℒtFKL:=∑v∈Stkπ¯Tk(v∣ct)logπ¯Tk(v∣ct)πθ(v∣ct).L^FKL_t:= _v∈ S_t^k π_T^k(v c_t) π_T^k(v c_t) _θ(v c_t). (3) Only the teacher target is truncated and renormalized; the student retains its full-vocabulary probabilities. With entropy threshold τ and forward-KL weight α, the per-token objective is ℒtEOPD:=ℒtOPD+α[HT(ct)>τ]ℒtFKL.L^EOPD_t:=L^OPD_t+α\,I\! [H_T(c_t)>τ ]L^FKL_t. (4) Here, πj _j always denotes a full-vocabulary policy, while π¯jk π_j^k denotes its top-k-renormalized shape—a distinction that separates probability mass from relative shape in our method. EOPD promotes coverage, but still uses a scalar entropy criterion to decide where to intervene and the uncalibrated teacher prior to determine what to distill; SPOT revisits both decisions. 3 Methodology 3.1 Overview Figure 1: An overview of our SPOT framework for on-policy distillation. SPOT follows a three-stage acquisition–exploration–exploitation procedure that addresses two coupled decisions: where to probe and what to distill. As illustrated in Figure 1, acquisition uses a lightweight position-level score to prioritize where to acquire additional evidence; exploration estimates the downstream values of teacher-proposed next-token candidates through verifier-scored student continuations; and exploitation combines these values with the teacher probabilities to construct outcome-calibrated targets. In this way, acquisition allocates the probing budget, while exploration and exploitation determine how the acquired evidence modifies local supervision. Algorithm 1 SPOT Training 1:Student πθ _θ, teacher πT _T, verifier R, prompt set D, learning rate η 2:Position budget M, top-k sizes ksk_s (scoring) and kpk_p (probing), inverse temperature γ, branch-loss weight β 3:for each training iteration do 4: πθold←πθ _ _old← _θ 5: Rollout buffer ℛ←∅R← 6: for each prompt q in a batch sampled from D do 7: Roll out x∼πθold(⋅∣q)x _ _old(· q) 8: Query the teacher πT(⋅∣ct) _T(· c_t) at every prefix ct=(q,x<t)c_t=(q,x_<t) 9: Score every valid position by its position-level acquisition score st=H¯T(ct)⋅Ctks⋅Gtkss_t= H_T(c_t)· C_t^k_s· G_t^k_s 10: using Eqs. (5)–(7) 11: Select sparse probing positions ℬ=Top-M(stt=1|x|)B=Top -M (\s_t\_t=1^|x| ) 12: for each position t∈ℬt and candidate v∈Stkpv∈ S_t^k_p do 13: Sample y∼πθold(⋅∣ct,v)y _ _old(· c_t,v) and estimate V^t(v) V_t(v) using R(q,x<t,v,y)R(q,x_<t,v,y) 14: Retain positions with at least one positive-reward candidate as ℬ+⊆ℬB^+ 15: Form reward-tilted targets π~T(⋅∣ct) π_T(· c_t) for t∈ℬ+t ^+ using Eq. (10) 16: Add the rollout and its supervision to ℛR 17: for each gradient step on a mini-batch sampled from ℛR do 18: Update θ←θ−η∇θℒθ←θ-η _θL by minimizing ℒ=1T∑t=1TℒtOPD+βmax1,|ℬ+|∑t∈ℬ+ℒtBranchL= 1T _t=1^TL^OPD_t+ β \1,|B^+|\ _t ^+L^Branch_t 19: using Eq. (12) 3.2 SPOT: Sparse Probing and Outcome-Calibrated Targets Position Acquisition: Where to Probe. Because probing next-token candidates requires extra rollouts, acquisition prioritizes positions where several teacher candidates are plausible, a small candidate set captures most teacher mass, and the student does not already represent those candidates well. Entropy alone cannot distinguish this pattern from uncertainty spread over a long tail. Recent analyses also suggest that OPD gains depend on student–teacher compatibility and genuinely new teacher information (Li et al., 2026). We therefore use student–teacher mismatch to signal potential correction needed, while verifier-scored rollouts test the utility of the selected candidates. Starting from the normalized teacher entropy H¯T(ct) H_T(c_t), let ksk_s denote the number of teacher candidates used to compute the position-level acquisition score. We first measure the teacher mass captured by its top-ksk_s candidate set: Ctks=∑v∈StksπT(v∣ct).C_t^k_s= _v∈ S_t^k_s _T(v c_t). (5) The quantity CtksC_t^k_s is the teacher probability mass captured by its top-ksk_s candidates. A large value means that a small candidate set represents most of the teacher distribution. Combined with high entropy, this favors positions with multiple plausible candidates in that set over positions whose uncertainty is dispersed across a long probability tail. Teacher-side structure, however, does not imply that the resulting supervision is useful to a student that may already represent the same alternatives. We thus measure the student–teacher gap on the same candidate set: Gtks=λmass(1−Atks)+λshapeDJS(π¯Tks∥π¯θoldks),G_t^k_s= _mass(1-A_t^k_s)+ _shapeD_JS( π_T^k_s\,\|\, π_ _old^k_s), (6) where Atks=∑v∈Stksπθold(v∣ct)A_t^k_s= _v∈ S_t^k_s _ _old(v c_t) and λmass+λshape=1 _mass+ _shape=1. Here DJSD_JS denotes the Jensen–Shannon divergence normalized to [0,1][0,1]; the two terms capture mass under-coverage and relative-shape mismatch, respectively. The final acquisition score sts_t acts as a soft conjunction: a position receives a high score only when the teacher presents multiple plausible candidates, the top-ksk_s set captures most teacher probability mass, and the student either underweights or differently ranks those candidates: st=H¯T(ct)⋅Ctks⋅Gtks.s_t= H_T(c_t)· C_t^k_s· G_t^k_s. (7) After masking special, padding, whitespace-only, and punctuation-only tokens, we allocate probing to ℬ=Top-M(stt=1|x|)B=Top -M(\s_t\_t=1^|x|). Consequently, the rollout budget is reserved for compact teacher alternatives that the current student neither adequately covers nor matches. Sparse Probing and Outcome Calibration: What to Distill. Acquisition prioritizes where to probe, but a teacher’s local next-token probability does not by itself predict downstream success. Prior work suggests that it can become a weaker guide on deeper student-generated prefixes (Li et al., 2026); moreover, a teacher-preferred token may lead to an incorrect solution, while a less-preferred candidate may still enable a correct one (Hou et al., 2026). We therefore treat the teacher distribution as a proposal prior (an initial preference over candidates) and calibrate it using downstream outcomes. To obtain outcome-grounded evidence, we perform sparse probing at each selected position t∈ℬt . Let kpk_p denote the number of teacher candidates probed at each selected position and NpN_p the number of continuations sampled per candidate. For each v∈Stkpv∈ S_t^k_p, we append v to the prefix ctc_t and independently sample NpN_p continuations y(n)∼πθold(⋅∣ct,v)y^(n) _ _old(· c_t,v), n=1,…,Npn=1,…,N_p, from the frozen behavior policy. These continuations probe the candidate-conditioned branch induced by v, whose student-executable value is V^t(v)=y∼πθold(⋅∣ct,v)[R(q,x<t,v,y)], V_t(v)=E_y _ _old(· c_t,v)\! [R(q,x_<t,v,y) ], (8) which measures whether the current student can turn a teacher-proposed local alternative into a high-reward trajectory. To avoid imposing additional supervision where exploration finds no viable alternative, we retain only positions for which at least one candidate has a positive estimated continuation value; the resulting set is denoted by ℬ+⊆ℬB^+ . Because probing evaluates at most M×kp×NpM× k_p× N_p candidate continuations per trajectory, its overhead is explicitly controlled by the position, candidate, and per-candidate sampling budgets. To convert the probed outcomes into actionable supervision, we construct outcome-calibrated targets at each retained position t∈ℬ+t ^+. Let Δ(Stkp)=ρt:ρt(v)≥0,∑v∈Stkpρt(v)=1 (S_t^k_p)=\ _t: _t(v)≥ 0,\, _v∈ S_t^k_p _t(v)=1\ denote the probability simplex over the teacher candidate set. Rather than collapsing onto the empirically best candidate, we maximize expected downstream value within a KL trust region around the renormalized teacher prior: maxρt∈Δ(Stkp)∑v∈Stkpρt(v)V^t(v)s.t.DKL(ρt∥π¯Tkp(⋅∣ct))≤ϵ. _ _t∈ (S_t^k_p)\; _v∈ S_t^k_p _t(v) V_t(v) .t. D_KL\! ( _t\,\|\, π_T^k_p(· c_t) )≤ε. (9) The derivation is deferred to Appendix A. For the corresponding dual parameter γ>0γ>0, which acts as an inverse temperature, the problem admits the closed-form optimizer π~T(v∣ct)=π¯Tkp(v∣ct)exp(γV^t(v))∑u∈Stkpπ¯Tkp(u∣ct)exp(γV^t(u)). π_T(v c_t)= π_T^k_p(v c_t) \! (γ V_t(v) ) _u∈ S_t^k_p π_T^k_p(u c_t) \! (γ V_t(u) ). (10) In log space, equation 10 reads logπ~T(v∣ct)=logπ¯Tkp(v∣ct)+γV^t(v)−logZt π_T(v c_t)= π_T^k_p(v c_t)+γ V_t(v)- Z_t, exposing each target log-probability as teacher log-probability plus an outcome-grounded continuation-value bonus, up to the normalizer ZtZ_t. Subtracting this expression for any u,v∈Stkpu,v∈ S_t^k_p yields the pairwise log-odds decomposition logπ~T(v∣ct)π~T(u∣ct)=logπ¯Tkp(v∣ct)π¯Tkp(u∣ct)+γ(V^t(v)−V^t(u)). π_T(v c_t) π_T(u c_t)= π_T^k_p(v c_t) π_T^k_p(u c_t)+γ\! ( V_t(v)- V_t(u) ). (11) This identity makes the division of roles explicit: the teacher supplies the prior odds, while the verifier adds a correction proportional to relative downstream value. Conditional on t∈ℬ+t ^+, only value differences affect the target, making it invariant to uniform reward shifts. Together with the ℬ+B^+ gate, outcome feedback first tests absolute viability and then corrects relative preference. As γ→0γ→ 0, the target recovers the teacher prior; larger γ increasingly favors candidates with higher estimated continuation values. Finally, we augment OPD with outcome-calibrated local supervision at verifier-supported positions: ℒ=1T∑t=1TℒtOPD+βmax1,|ℬ+|∑t∈ℬ+ℒtBranch,L= 1T _t=1^TL_t^OPD+ β \1,|B^+|\ _t ^+L_t^Branch, (12) where ℒtBranch=−∑v∈Stkpπ~T(v∣ct)logπθ(v∣ct)L_t^Branch=- _v∈ S_t^k_p π_T(v c_t) _θ(v c_t). The denominator averages this auxiliary loss over retained positions, and β controls its strength relative to trajectory-wide OPD. When ℬ+=∅B^+= , the branch term vanishes and the objective reduces to standard OPD. 4 EXPERIMENTS For a rigorous, controlled comparison of SPOT, we follow the setup of recent OPD work (Jin et al., 2026). Further details appear in Appendix B. 4.1 Experimental Settings Models and Training Data. We use Qwen3-8B, with thinking mode disabled, as the common teacher for all distillation methods, and Qwen3-0.6B-Base, Qwen3-1.7B-Base, and Qwen3-4B-Base as students (Yang et al., 2025). The 0.6B and 1.7B students are trained on MATH (Hendrycks et al., 2021); the 4B student uses the more challenging DAPO dataset (3). Baselines. We compare SPOT with four baselines: • KD (Hinton et al., 2015; Kim and Rush, 2016): off-policy distillation using forward KL and cross-entropy on teacher-generated data. • OPD (Agarwal et al., 2024; Li et al., 2026): on-policy training on student rollouts with per-token reverse-KL supervision. • GRPO (Shao et al., 2024): outcome-based RL using group-relative advantages from verifiable rewards, without teacher supervision. • EOPD (Jin et al., 2026): OPD augmented with a top-k forward-KL term at positions selected by a fixed teacher-entropy threshold. Evaluation. We evaluate all models zero-shot with the same prompt template and answer verifier on MATH-500 (Hendrycks et al., 2021; Lightman et al., 2023), AIME 2024 (Zhang and Math-AI, 2024), AIME 2025 (Zhang and Math-AI, 2025), AMC 2023 (Yang et al., 2024), Minerva Math (Lewkowycz et al., 2022), and HMMT 2025 (balunović2026matharenaevaluatingllmsuncontaminated). We sample eight responses per problem with temperature 1.0, top-p sampling with p=0.8p=0.8, and a maximum response length of 8192 tokens. Avg@8 is the mean response accuracy, and Pass@8 is the fraction of problems solved at least once. We report per-benchmark scores and their unweighted macro-average across the six benchmarks. 4.2 Main Results Consistent gains across student scales and training sets. As shown in Table 1, SPOT achieves the best macro Pass@8 at all three evaluated student scales and the best or second-best macro Avg@8. Relative to standard OPD, SPOT improves macro Avg@8 by 0.47–1.48 points and macro Pass@8 by 4.55–5.28 points. Relative to EOPD, the closest uncertainty-aware OPD baseline, SPOT improves macro Avg@8 by 0.29–0.68 points and macro Pass@8 by 2.49–3.19 points. These gains persist from 0.6B to 4B students and across MATH and DAPO-Math-14k training, demonstrating that the benefit is not tied to a particular capacity or training regime. Broader multi-sample coverage without sacrificing average accuracy. Avg@8 estimates per-sample correctness, whereas Pass@8 measures whether eight samples reach at least one correct solution. The markedly larger Pass@8 gains, together with preserved or improved Avg@8, reveal a favorable coverage–quality profile: SPOT places meaningful probability on viable alternatives without diluting an individual attempt. This directly matches SPOT’s goal of improving coverage without sacrificing average accuracy. The comparison with EOPD is consistent with our motivating distinction: teacher entropy alone cannot determine whether uncertainty is concentrated in a compact candidate set, whether the student already represents those candidates, or which candidates lead to successful student continuations. SPOT addresses these gaps by using normalized teacher entropy, top-ksk_s probability mass, and student–teacher mismatch to allocate the probing budget, then using verifier-scored continuation values to construct outcome-calibrated local targets. SPOT therefore selectively expands coverage toward useful alternative reasoning modes rather than stylistic or erroneous variation. Table 1: Main results on six mathematical reasoning benchmarks. Avg. is the unweighted mean over the six benchmarks, computed before rounding. Bold and underlined entries denote the best and second-best results within each student setting, respectively. Method Avg@8 Pass@8 MATH500 AMC23 Minerva HMMT AIME24 AIME25 Avg. MATH500 AMC23 Minerva HMMT AIME24 AIME25 Avg. Student Model: Qwen3-0.6B-Base (Training Data: MATH) KD 47.80 23.43 14.34 0.21 2.08 0.83 14.78 69.60 52.50 30.88 1.67 6.67 6.67 28.00 GRPO 53.33 28.13 16.41 1.04 4.58 0.83 17.39 74.40 55.00 32.72 3.33 10.00 10.00 30.91 OPD 50.10 24.69 16.04 0.42 2.50 1.25 15.83 73.20 57.50 31.25 1.67 10.00 6.67 30.05 EOPD 50.50 27.81 15.95 1.46 4.17 1.25 16.86 75.00 55.00 33.46 5.00 13.33 6.67 31.41 SPOT (Ours) 50.50 28.44 15.99 1.88 5.00 2.08 17.31 76.40 60.00 34.56 6.67 13.33 16.67 34.60 Student Model: Qwen3-1.7B-Base (Training Data: MATH) KD 62.85 37.81 27.07 1.25 10.42 3.33 23.79 84.20 70.00 44.12 3.33 20.00 16.67 39.72 GRPO 67.00 38.13 28.17 1.88 9.17 5.42 24.96 84.00 72.50 48.16 6.67 20.00 16.67 41.33 OPD 67.03 39.06 27.62 1.46 8.33 6.25 24.96 84.80 70.00 47.06 3.33 20.00 16.67 40.31 EOPD 67.68 40.31 27.90 1.67 10.83 7.50 25.98 83.60 75.00 44.49 6.67 26.67 20.00 42.74 SPOT (Ours) 67.43 39.69 28.45 2.08 12.50 7.50 26.27 87.80 75.00 47.43 6.67 33.33 23.33 45.59 Student Model: Qwen3-4B-Base (Training Data: DAPO-Math-14k) KD 74.73 48.13 34.65 3.13 12.50 12.08 30.87 92.20 80.00 51.47 13.33 26.67 23.33 47.83 GRPO 79.18 53.13 39.84 2.71 14.17 13.33 33.73 90.00 80.00 52.21 11.67 26.67 26.67 47.87 OPD 80.38 55.94 37.64 5.83 17.92 16.25 35.66 93.00 80.00 51.10 13.33 30.00 30.00 49.57 EOPD 79.73 55.63 36.08 7.08 17.50 16.67 35.45 93.20 82.50 51.84 16.67 33.33 33.33 51.81 SPOT (Ours) 79.88 56.25 37.73 7.50 17.50 17.92 36.13 93.40 85.00 54.04 23.33 33.33 36.67 54.30 4.3 Ablation Study 4.3.1 Effect of the Branch-Acquisition Score Because verifier-based candidate probing is costly, SPOT restricts it to the top-M positions. Here, H denotes normalized teacher entropy, C denotes the teacher mass captured by its top-ksk_s candidates, and G denotes the student–teacher gap combining mass undercoverage and JS shape mismatch. We compare five variants: H uses entropy alone; HC adds C; HG adds G but omits C; HC-Mass uses H, C, and only the mass-undercoverage term; and Full uses all components. All runs share the teacher, data, schedule, probing budget, and evaluation protocol. Table 2: Branch-acquisition score ablation. Avg@8 / Pass@8 (%) for Qwen3-0.6B-Base trained on MATH. Macro denotes the unweighted mean; best results are bold and Full is shaded. Avg@8 / Pass@8 Acquisition score MATH500 AMC23 AIME24 AIME25 Macro H: H¯T H_T 50.50 / 76.20 25.00 / 60.00 1.67 / 3.33 0.42 / 3.33 19.40 / 35.72 HC: H¯TCtks H_TC_t^k_s 50.30 / 73.20 27.50 / 57.50 1.67 / 3.33 0.00 / 0.00 19.87 / 33.51 HG: H¯TGtks H_TG_t^k_s 50.00 / 74.20 28.13 / 55.00 2.08 / 10.00 1.25 / 10.00 20.37 / 37.30 HC-Mass: H¯TCtks(1−Atks) H_TC_t^k_s(1-A_t^k_s) 50.75 / 75.80 27.81 / 62.50 2.08 / 6.67 1.25 / 6.67 20.47 / 37.91 Full: H¯TCtksGtks H_TC_t^k_sG_t^k_s 50.50 / 76.40 28.44 / 60.00 5.00 / 13.33 2.08 / 16.67 21.51 / 41.60 Table 2 shows that Full achieves the best macro Avg@8/Pass@8 (21.51/41.60), with its larger advantage in Pass@8 suggesting improved solution coverage. HG underperforms Full on both metrics across all benchmarks, supporting CtksC_t^k_s as a reliable filter against diffuse teacher uncertainty. HC, which omits the student–teacher gap, also consistently underperforms Full, supporting the inclusion of GtksG_t^k_s. HC-Mass is the strongest partial variant but remains below Full, suggesting that mass undercoverage and JS shape mismatch capture complementary student deficits. Overall, the results support the multiplicative score as a soft conjunction of teacher ambiguity, top-ksk_s mass. 4.3.2 Effect of Verifier-Guided Calibration Verifier-guided calibration converts the teacher prior into an outcome-aware branch target. To isolate this component, we remove reward tilting and positive-reward gating while retaining Full acquisition and the branch-distillation objective. This ablation therefore uses the uncalibrated teacher proposal prior at all probed positions. All other training and evaluation settings match the Qwen3-1.7B-Base main experiment. Table 3 shows that the full configuration improves macro Avg@8/Pass@8 by 3.21/7.38 points. Pass@8 increases on all four benchmarks, with the largest gain on AIME24 (+13.33+13.33). On MATH500, Avg@8 changes by only +0.13+0.13 while Pass@8 rises by +2.00+2.00, suggesting that verifier-guided calibration primarily improves solution coverage in this setting. Table 3: Verifier-guidance ablation. Avg@8 / Pass@8 (%) for Qwen3-1.7B-Base trained on MATH. Macro denotes the unweighted mean; best results are bold and the full model is shaded. Avg@8 / Pass@8 Variant MATH500 AMC23 AIME24 AIME25 Macro w/o verifier guidance 67.30 / 85.80 34.06 / 67.50 8.33 / 20.00 4.58 / 16.67 28.57 / 47.49 SPOT (Full) 67.43 / 87.80 39.69 / 75.00 12.50 / 33.33 7.50 / 23.33 31.78 / 54.87 4.3.3 Scaling with the Evaluation-Time Sampling Budget To test whether SPOT improves coverage beyond a fixed Pass@8 protocol, we vary k∈4,8,16,32,64k∈\4,8,16,32,64\ on AIME24, AIME25, and AMC23, holding decoding fixed. Here, k denotes responses per problem. Figure 2: Pass@k and its gain over OPD on AIME 2024, AIME 2025, and AMC 2023. Each point is read from the corresponding summary of a separately generated k-sample run. Figure 2 reveals a budget-dependent separation. The methods are close at k=4k=4, with EOPD leading on AIME25, whereas SPOT ranks first or ties from k=8k=8 onward. At k=64k=64, SPOT retains 12.50–16.67-point gains over OPD, showing that its coverage advantage extends beyond Pass@8 and remains visible as the sampling budget grows. 4.3.4 Effect of Branch-Distillation Weight The coefficient β trades off trajectory-wide OPD against outcome-calibrated branch supervision. Figure 3: Sensitivity to the branch-distillation weight. Pass@8 of Qwen3-1.7B-Base for β∈0.05,0.1,0.5,1.0β∈\0.05,0.1,0.5,1.0\. The main configuration β=0.1β=0.1 is marked as Ours. Among the tested values in Figure 3, β=0.1β=0.1 leads the three competition benchmarks and ties β=0.5β=0.5 on MATH500. The competition benchmarks show greater observed sensitivity, with their mean Pass@8 lowest at β=1.0β=1.0. This pattern is consistent with branch feedback serving as a calibrated correction to trajectory-wide supervision: weaker weighting may underuse local outcome signals, whereas stronger weighting may overemphasize sparse branch targets. We therefore use β=0.1β=0.1 as a practical default. 4.3.5 Out-of-Domain Generalization Although trained exclusively on MATH, SPOT generalizes beyond the training distribution, with its clearest gains on deliberative reasoning while remaining competitive on broader knowledge and instruction following (Appendix C.1, Table 9). This pattern suggests that outcome-calibrated branch supervision transfers reusable decision behavior rather than merely fitting math-specific solution templates. The non-uniform gains further indicate that branch calibration complements, rather than replaces, token-level uncertainty transfer and domain-aligned supervision. 4.3.6 Consistency Across Model Families The Llama results (Appendix C.2, Table 10) preserve the main Qwen pattern: SPOT leads both macro metrics, with a clearer advantage in Pass@8 than Avg@8. Together with the Qwen results across model scales and training sets, this supports the portability of outcome-calibrated branch supervision beyond a single architecture and suggests that its primary benefit is consistently broader coverage of viable reasoning paths. 5 Related Work Knowledge Distillation. Knowledge distillation transfers capabilities from a high-capacity teacher to a compact student by matching teacher predictive distributions or imitating teacher-generated sequences (Hinton et al., 2015; Kim and Rush, 2016). Recent work improves this paradigm through alternative divergence objectives, contrastive learning, and selective supervision based on sample difficulty or token importance (Ko et al., 2024; 2025; He et al., 2025; Guo et al., 2025). Despite their effectiveness, these methods train on contexts that are not sampled from the current student policy, creating exposure bias and compounding errors when the student conditions on its own predictions at inference time (Bengio et al., 2015; Ranzato et al., 2016). On-Policy Distillation for Reasoning Models. On-policy distillation trains students on self-generated rollouts with dense token-level teacher feedback, mitigating exposure bias and providing finer credit assignment than outcome-only RL (Agarwal et al., 2024; Gu et al., 2026; Lu and Lab, 2025). Recent work has broadened OPD along four directions: teacher access and transfer scope, through reward extrapolation, black-box teachers, and multi-teacher distillation (Yang et al., 2026b; Ye et al., 2026a; Hou et al., 2026); adaptive supervision, through uncertainty- or disagreement-aware token selection, teachability modeling, and trajectory-aware guidance (Jin et al., 2026; Xu et al., 2026; Wang et al., 2026; Jiang and Ferraro, 2026); representation and training efficiency, through hidden-state alignment and offline teacher scoring (Yang et al., 2026a; Wu et al., 2026; Ziheng et al., 2026); and self-distillation, where privileged or auxiliary context induces internal teachers for knowledge internalization (Zhao et al., 2026; Ye et al., 2026b). Together, these advances have established OPD as a general post-training paradigm, with adoption in large-scale systems such as Qwen3 and DeepSeek-V4 (Yang et al., 2025; DeepSeek-AI et al., 2026). 6 Conclusion Standard reverse-KL OPD can under-cover plausible alternatives, while teacher entropy alone cannot determine where limited probing is most useful or which candidates lead to successful student continuations. We introduced SPOT, which uses normalized teacher entropy, top-ksk_s probability mass, and student–teacher mismatch to allocate a sparse probing budget, then uses verifier-scored continuation values to construct KL-regularized, outcome-calibrated targets anchored to the teacher proposal prior. Across the evaluated student scales, training sets, reasoning benchmarks, and model families, SPOT achieves the highest macro Pass@8 in every setting and the highest or second-highest macro Avg@8, with controlled ablations supporting both selective acquisition and verifier guidance. Overall, our results support separating where to probe from what to distill: uncertainty and student mismatch guide probing, while downstream outcomes calibrate supervision. References R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. External Links: 2306.13649, Link Cited by: §1, 2nd item, §5. S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer (2015) Scheduled sampling for sequence prediction with recurrent neural networks. External Links: 1506.03099, Link Cited by: §1, §5. [3] (2025) DAPO: An Open-Source LLM Reinforcement Learning System at Scale, author=Qiying Yu and Zheng Zhang and Ruofei Zhu and Yufeng Yuan and Xiaochen Zuo and Yu Yue and Weinan Dai and Tiantian Fan and Gaohong Liu and Lingjun Liu and Xin Liu and Haibin Lin and Zhiqi Lin and Bole Ma and Guangming Sheng and Yuxuan Tong and Chi Zhang and Mofan Zhang and Wang Zhang and Hang Zhu and Jinhua Zhu and Jiaze Chen and Jiangjie Chen and Chengyi Wang and Hongli Yu and Yuxuan Song and Xiangpeng Wei and Hao Zhou and Jingjing Liu and Wei-Ying Ma and Ya-Qin Zhang and Lin Yan and Mu Qiao and Yonghui Wu and Mingxuan Wang. External Links: 2503.14476, Link Cited by: §4.1. DeepSeek-AI, A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, C. Lu, C. Zhao, C. Deng, C. Hou, C. Xu, C. Shao, C. Ruan, C. Sun, D. Dai, D. Guo, D. Yang, D. Chen, D. Li, D. Ji, E. Li, F. Wei, F. Lin, F. Yuan, F. Xia, F. Dai, G. Hao, G. Chen, G. Cao, G. Meng, G. Li, H. Yu, H. Zhang, H. Xu, H. Li, H. Liang, H. Zhang, H. Luo, H. Wei, H. Yuan, H. Zhang, H. Luo, H. Chen, H. Ji, H. Zhang, H. Ding, H. Tang, H. Cao, H. Gao, H. Qu, H. Zeng, J. Yang, J. Zhu, J. Luo, J. Song, J. Yu, J. Huang, J. Cai, J. Liang, J. Zhou, J. Ye, J. Li, J. Xu, J. Hu, J. Yang, J. Chen, J. Yan, J. Chen, J. Zhou, J. Xiang, J. Yuan, J. Cheng, J. Zhou, J. Zhu, J. Yu, J. Sun, J. Ran, J. Jiang, J. Qiu, J. Li, J. Zheng, J. Song, K. Dong, K. Gao, K. Guan, K. Zhou, K. Huang, K. Yu, L. Wang, L. Zhang, L. Wang, L. Xia, L. Zhang, L. Zhao, L. Guo, L. Luo, L. Ma, L. Zhu, L. Wang, L. Cai, L. Zhang, L. Chen, M. Di, M. Xu, M. Mei, M. Wang, M. Zhang, M. Zhang, M. Tang, M. Li, M. Zhou, M. Han, N. Wang, P. Huang, P. Wang, P. Cong, P. Wang, P. Zhang, Q. Wang, Q. Zhu, Q. Li, Q. Chen, Q. Du, Q. Jiang, R. Tian, R. Xu, R. Lu, R. Xu, R. Ge, R. Zhang, R. Pan, R. Wang, R. Chen, R. Yin, R. Xu, R. Shen, R. Zhang, R. Chen, S. Liu, S. Lu, S. Sun, S. Zhou, S. Chen, S. Cai, S. Nie, S. Wu, S. Chen, S. Hu, S. Liu, S. Hu, S. Ma, S. Wang, S. Yu, S. Zhou, S. Pan, S. Yu, S. Zhou, T. Ni, T. Yun, T. Jin, T. Pei, T. Ye, T. Lin, T. Ji, T. Cui, T. Yue, T. Yu, T. Wang, W. Zhang, W. Xiao, W. Zeng, W. An, W. Zhao, W. Liu, W. Liang, W. Pang, W. Luo, W. Yao, W. Gao, W. Yang, W. Huang, W. Hou, W. Zhang, W. Ma, X. Gao, X. He, X. Wang, X. Wang, X. Bi, X. Liu, X. Wang, X. Chen, X. Zhang, X. Nie, X. Sun, X. Wang, X. Cheng, X. Liu, X. Xie, X. Liu, X. Liu, X. Yu, X. Li, X. Yang, X. Zhang, X. Chen, X. Wang, X. Su, X. Chen, X. Lin, X. Fu, Y. Yan, Y. Wang, Y. Ma, Y. Luo, Y. Zhang, Y. Xu, Y. Ma, Y. Huang, Y. Li, Y. Li, Y. Xu, Y. Zhao, Y. Sun, Y. Wang, Y. Qian, Y. Shao, Y. Yu, Y. Zhang, Y. Ding, Y. Shi, Y. Wu, Y. Xiong, Y. Ma, Y. He, Y. Tang, Y. Zhou, Y. Luo, Y. Zhong, Y. Piao, Y. Wang, Y. Zhang, Y. Chen, Y. Tan, Y. Wei, Y. Ma, Y. Liu, Y. Yang, Y. Guo, Y. Wu, Y. Wu, Y. Li, Y. Cheng, Y. Ou, Y. Xu, Y. Li, Y. Wang, Y. Yang, Y. Xu, Y. Wu, Y. Meng, Y. Zou, Y. Zha, Y. Xiong, Y. Chen, Y. Lin, Y. Cao, Y. Wang, Y. Zhang, Y. Yan, Y. Lin, Y. Gu, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. Zhou, Y. Huang, Z. Wu, Z. Wang, Z. Zhao, Z. Ren, Z. Zhang, Z. Sha, Z. Fu, Z. Ju, Z. Xu, Z. Xie, Z. Zhang, Z. Gao, Z. Hao, Z. Gou, Z. Ma, Z. Yan, Z. Shao, Z. Huang, Z. Chen, Z. Wu, Z. Ren, Z. Wu, Z. Li, Z. Zhang, Z. Xu, Z. Wang, Z. Qu, Z. Gu, Z. Zhu, Z. Li, Z. Zhang, Z. Xie, Z. Gao, Z. Wan, Z. Pan, and Z. Yao (2026) DeepSeek-v4: towards highly efficient million-token context intelligence. External Links: 2606.19348, Link Cited by: §5. C. Goddard and L. Atkins (2024) DistillKit: flexible knowledge distillation for large language models External Links: Link Cited by: Appendix B. Y. Gu, L. Dong, F. Wei, and M. Huang (2026) MiniLLM: on-policy distillation of large language models. External Links: 2306.08543, Link Cited by: §5. Y. Guo, W. Yang, Z. Sun, N. Ding, Z. Liu, and Y. Lin (2025) Learning to focus: causal attention distillation via gradient-guided token pruning. External Links: 2506.07851, Link Cited by: §1, §5. C. He, Y. Ding, J. Guo, R. Gong, H. Qin, and X. Liu (2025) DA-KD: Difficulty-aware knowledge distillation for efficient large language models. In Forty-second International Conference on Machine Learning, Cited by: §5. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. External Links: 2103.03874, Link Cited by: §4.1, §4.1. G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. External Links: 1503.02531, Link Cited by: 1st item, §5. W. Hou, S. Peng, W. Wang, Z. Ruan, Y. Zhang, Z. Zhou, M. Gao, Y. Chen, K. Wang, H. Yang, C. Zhang, Z. Tian, H. Hu, Y. Yang, F. Wu, and H. Fan (2026) Uni-opd: unifying on-policy distillation with a dual-perspective recipe. External Links: 2605.03677, Link Cited by: §1, §3.2, §5. Y. Jiang and F. Ferraro (2026) Bridging reasoning trajectories in on-policy distillation via near-future guidance. External Links: 2606.00305, Link Cited by: §5. W. Jin, T. Min, Y. Yang, D. Wei, Y. Zhou, S. R. Kadhe, N. Baracaldo, and K. Lee (2026) Entropy-aware on-policy distillation of language models. External Links: 2603.07079, Link Cited by: Appendix B, §1, §2, 4th item, §4, §5. Y. Kim and A. M. Rush (2016) Sequence-level knowledge distillation. External Links: 1606.07947, Link Cited by: 1st item, §5. J. Ko, T. Chen, S. Kim, T. Ding, L. Liang, I. Zharkov, and S. Yun (2025) DistiLLM-2: a contrastive approach boosts the distillation of llms. External Links: 2503.07067, Link Cited by: §5. J. Ko, S. Kim, T. Chen, and S. Yun (2024) DistiLLM: towards streamlined distillation for large language models. External Links: 2402.03898, Link Cited by: §5. [17] Math-Verify: Math Verification Library External Links: Link Cited by: Appendix B. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra (2022) Solving quantitative reasoning problems with language models. External Links: 2206.14858, Link Cited by: §4.1. Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, and N. Ding (2026) Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. External Links: 2604.13016, Link Cited by: §1, §3.2, §3.2, 2nd item. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023) Let’s verify step by step. External Links: 2305.20050, Link Cited by: §4.1. K. Lu and T. M. Lab (2025) On-policy distillation. Thinking Machines Lab: Connectionism. Note: https://thinkingmachines.ai/blog/on-policy-distillation External Links: Document Cited by: §1, §5. T. P. Minka (2005) Divergence measures and message passing. External Links: Link Cited by: §1. M. Ranzato, S. Chopra, M. Auli, and W. Zaremba (2016) Sequence level training with recurrent neural networks. External Links: 1511.06732, Link Cited by: §1, §5. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §1, 3rd item. G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) HybridFlow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, p. 1279–1297. External Links: Link, Document Cited by: Appendix B. Y. Wang, S. Lu, Y. Gu, P. Wang, Y. Yang, Z. Yan, C. Xie, J. Wu, and H. Yang (2026) Not all disagreement is learnable: token teachability in on-policy distillation. External Links: 2605.26844, Link Cited by: §5. Y. Wu, S. Han, and H. Cai (2026) Lightning opd: efficient post-training for large reasoning models with offline on-policy distillation. External Links: 2604.13010, Link Cited by: §5. Y. Xu, H. Sang, Z. Zhou, R. He, Z. Wang, and A. Geramifard (2026) TIP: token importance in on-policy distillation. External Links: 2604.14084, Link Cited by: §5. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu (2025) Qwen3 technical report. External Links: 2505.09388, Link Cited by: §4.1, §5. A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, et al. (2024) Qwen2 technical report. arXiv preprint arXiv:2407.10671. Cited by: §4.1. S. Yang, G. Zhu, B. Song, H. Wang, M. Xia, X. Zheng, Y. Ma, Z. Chen, W. Wang, J. Zhao, and G. Chen (2026a) OPRD: on-policy representation distillation. External Links: 2606.06021, Link Cited by: §5. W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin (2026b) Learning beyond teacher: generalized on-policy distillation with reward extrapolation. External Links: 2602.12125, Link Cited by: §5. T. Ye, L. Dong, Z. Chi, X. Wu, S. Huang, and F. Wei (2026a) Black-box on-policy distillation of large language models. External Links: 2511.10643, Link Cited by: §5. T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei (2026b) On-policy context distillation for language models. External Links: 2602.12275, Link Cited by: §5. Y. Zhang and T. Math-AI (2024) American invitational mathematics examination (aime) 2024. Cited by: §4.1. Y. Zhang and T. Math-AI (2025) American invitational mathematics examination (aime) 2025. Cited by: §4.1. S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026) Self-distilled reasoner: on-policy self-distillation for large language models. External Links: 2601.18734, Link Cited by: §5. L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng (2024) SGLang: efficient execution of structured language model programs. External Links: 2312.07104, Link Cited by: Appendix B. Z. Ziheng, J. Li, H. Tang, Y. N. Wu, and D. Terzopoulos (2026) Less is more: early stopping rollout for on-policy distillation. External Links: 2605.27028, Link Cited by: §5. Appendix A Derivation and Properties of Outcome-Calibrated Targets This appendix derives the outcome-calibrated target and establishes several properties of its value–prior tradeoff. Fix a retained position t∈ℬ+t ^+, and abbreviate =Stkp,p(v)=π¯Tkp(v∣ct),V(v)=V^t(v).S=S_t^k_p, p(v)= π_T^k_p(v c_t), V(v)= V_t(v). Here, V is treated as the branch-value estimate obtained after probing. The candidate set S is finite, and p(v)>0p(v)>0 for every v∈v . We write Δ() (S) for the probability simplex over S. A.1 Closed-Form Derivation and Trust-Region Equivalence For any inverse-temperature parameter γ>0γ>0, consider the KL-regularized problem maxρ∈Δ()v∼ρ[V(v)]−1γDKL(ρ∥p). _ρ∈ (S) \E_v ρ[V(v)]- 1γD_KL(ρ\|p) \. (13) The first term favors branches with high estimated downstream value, while the second anchors the target to the teacher prior. Proposition A.1 (Closed-form optimizer). The objective in Eq. (13) has the unique optimizer ργ(v)=p(v)exp(γV(v))∑u∈p(u)exp(γV(u)). _γ(v)= p(v) (γ V(v) ) _u p(u) (γ V(u) ). (14) Proof. Let Z(γ)=∑u∈p(u)exp(γV(u))Z(γ)= _u p(u) (γ V(u)). For any ρ∈Δ()ρ∈ (S), DKL(ρ∥ργ) D_KL(ρ\| _γ) =∑v∈ρ(v)logρ(v)Z(γ)p(v)exp(γV(v)) = _v ρ(v) ρ(v)Z(γ)p(v) (γ V(v)) =DKL(ρ∥p)−γρ[V]+logZ(γ). =D_KL(ρ\|p)- _ρ[V]+ Z(γ). (15) Rearranging gives the Gibbs variational identity ρ[V]−1γDKL(ρ∥p)=logZ(γ)γ−1γDKL(ρ∥ργ).E_ρ[V]- 1γD_KL(ρ\|p)= Z(γ)γ- 1γD_KL(ρ\| _γ). (16) The right-hand side is maximized if and only if DKL(ρ∥ργ)=0D_KL(ρ\| _γ)=0, or equivalently ρ=ργρ= _γ. ∎ The regularized and trust-region views describe the same solution path. Define the radius induced by γ as ϵ(γ)=DKL(ργ∥p).ε(γ)=D_KL( _γ\|p). (17) Proposition A.2 (Trust-region equivalence). For every γ>0γ>0, ργ _γ is the unique solution of maxρ∈Δ()ρ[V]s.t.DKL(ρ∥p)≤ϵ(γ). _ρ∈ (S)E_ρ[V] s.t. D_KL(ρ\|p)≤ε(γ). (18) Conversely, if the KL constraint is active and its KKT multiplier is η>0η>0, the solution has the form in Eq. (14) with γ=1/ηγ=1/η. Proof. Suppose a feasible ρ attained a larger expected value than ργ _γ. Since DKL(ρ∥p)≤DKL(ργ∥p)D_KL(ρ\|p)≤ D_KL( _γ\|p), it would also attain a strictly larger objective in Eq. (13), contradicting Proposition A.1. The same argument, together with the uniqueness of the regularized optimizer, excludes any distinct feasible optimizer with equal expected value. For the converse direction, introduce multipliers η≥0η≥ 0 and λ∈ℝλ . The Lagrangian for Eq. (18) is (ρ,η,λ)=ρ[V]−η(DKL(ρ∥p)−ϵ)+λ(∑v∈ρ(v)−1).J(ρ,η,λ)=E_ρ[V]-η (D_KL(ρ\|p)-ε )+λ ( _v ρ(v)-1 ). When the constraint is active, stationarity with respect to ρ(v)ρ(v) yields V(v)−η(logρ(v)p(v)+1)+λ=0.V(v)-η ( ρ(v)p(v)+1 )+λ=0. Normalizing the resulting probabilities gives ρ(v)∝p(v)exp(V(v)/η)ρ(v) p(v) (V(v)/η), which is Eq. (14) with γ=1/ηγ=1/η. ∎ Thus, a shared γ parameterizes the regularized solution directly and induces a position-dependent trust-region radius ϵt(γ)=DKL(ργ,t∥π¯Tkp(⋅∣ct)) _t(γ)=D_KL ( _γ,t \| π_T^k_p(· c_t) ). For a prescribed active radius, the corresponding inverse temperature is the reciprocal of the KL multiplier rather than the multiplier itself. A.2 Geometry of the Calibration Path The log-partition function provides a compact characterization of how γ trades off estimated branch value against deviation from the teacher. Let F(γ)=logZ(γ)=log∑v∈p(v)exp(γV(v)).F(γ)= Z(γ)= _v p(v) (γ V(v)). Proposition A.3 (Monotone value–prior tradeoff). Along the calibration path ργ≥0\ _γ\_γ≥ 0, with ρ0=p _0=p defined by continuity, dγργ[V] ddγE_ _γ[V] =Varργ(V)≥0, =Var_ _γ(V)≥ 0, (19) dγDKL(ργ∥p) ddγD_KL( _γ\|p) =γVarργ(V)≥0. = _ _γ(V)≥ 0. (20) If V is nonconstant on S, both quantities are strictly increasing for γ>0γ>0. Proof. Differentiating F gives F′(γ)=ργ[V],F′(γ)=Varργ(V).F (γ)=E_ _γ[V], F (γ)=Var_ _γ(V). Moreover, Eq. (14) implies DKL(ργ∥p)=γF′(γ)−F(γ).D_KL( _γ\|p)=γ F (γ)-F(γ). Differentiating this identity proves Eqs. (19)–(20). Because p and hence ργ _γ have full support on S, the variance is positive whenever V is nonconstant. ∎ This monotonicity makes the role of γ precise: increasing it yields a target with higher estimated downstream value while moving it farther from the teacher prior. It also yields a one-to-one correspondence between γ and the induced radius over the nondegenerate portion of the path. Specifically, let ⋆=argmaxv∈V(v)A = _v V(v) and P⋆=∑v∈⋆p(v)P = _v p(v). Then limγ→0ργ=p,limγ→∞ργ(v)=p(v)/P⋆,v∈⋆,0,v∉⋆, _γ→ 0 _γ=p, _γ→∞ _γ(v)= casesp(v)/P ,&v ,\\ 0,&v , cases (21) and the induced radius increases from 0 to −logP⋆- P . Notably, when several candidates share the maximum value, the large-γ limit preserves their relative teacher probabilities rather than selecting an arbitrary one. Corollary A.4 (Diminishing returns of KL budget). Assume that V is nonconstant, and let G(ϵ)=maxρ:DKL(ρ∥p)≤ϵρ[V]G(ε)= _ρ:\,D_KL(ρ\|p)≤εE_ρ[V] for 0<ϵ<−logP⋆0<ε<- P . If γ(ϵ)γ(ε) denotes the unique inverse temperature inducing radius ϵε, then G′(ϵ)=1γ(ϵ),G′(ϵ)=−1γ(ϵ)3Varργ(ϵ)(V)<0.G (ε)= 1γ(ε), G (ε)=- 1γ(ε)^3Var_ _γ(ε)(V)<0. (22) Thus, relaxing the teacher-centered KL budget improves the optimal estimated branch value, but with strictly diminishing marginal returns. Proof. Along the calibration path, G(ϵ(γ))=ργ[V]G(ε(γ))=E_ _γ[V]. Proposition A.3 gives dG/dγ=Varργ(V)dG/dγ=Var_ _γ(V) and dϵ/dγ=γVarργ(V)dε/dγ= _ _γ(V). Applying the chain rule once gives G′(ϵ)=1/γG (ε)=1/γ; differentiating once more yields Eq. (22). ∎ Corollary A.5 (Finite-temperature teacher anchoring). Let ΔV=maxvV(v)−minvV(v) _V= _vV(v)- _vV(v). For every finite γ>0γ>0 and v∈v , exp(−γΔV)≤ργ(v)p(v)≤exp(γΔV). (-γ _V)≤ _γ(v)p(v)≤ (γ _V). (23) Consequently, calibration preserves the teacher top-kpk_p support at every finite temperature. Proof. Since exp(γminvV(v))≤Z(γ)≤exp(γmaxvV(v)) (γ _vV(v))≤ Z(γ)≤ (γ _vV(v)), the result follows directly from ργ(v)/p(v)=exp(γV(v))/Z(γ) _γ(v)/p(v)= (γ V(v))/Z(γ). ∎ A.3 Estimated-Value Improvement under Teacher Anchoring Proposition A.6 (Improvement–deviation bounds). The outcome-calibrated target satisfies ργ[V]−p[V]≥1γDKL(ργ∥p)≥0.E_ _γ[V]-E_p[V]≥ 1γD_KL( _γ\|p)≥ 0. (24) Moreover, with ϵ(γ)=DKL(ργ∥p)ε(γ)=D_KL( _γ\|p), 0≤ργ[V]−p[V]≤ΔVϵ(γ)2.0 _ _γ[V]-E_p[V]≤ _V ε(γ)2. (25) Proof. Optimality of ργ _γ in Eq. (13), using the teacher prior p as a feasible comparator, gives ργ[V]−1γDKL(ργ∥p)≥p[V],E_ _γ[V]- 1γD_KL( _γ\|p) _p[V], which proves Eq. (24). For the upper bound, the expectation difference is at most ΔVTV(ργ,p) _V\,TV( _γ,p). Pinsker inequality gives TV(ργ,p)≤DKL(ργ∥p)/2TV( _γ,p)≤ D_KL( _γ\|p)/2, yielding Eq. (25). ∎ Equation (24) formalizes the benefit of the exponential tilt, whereas Eq. (25) formalizes its conservatism. The guarantee concerns the probed estimate V=V^tV= V_t; it does not by itself assert improvement in the unknown population branch value. A.4 Preference Correction and the Binary-Verifier Case The target changes a teacher preference only when the outcome advantage is large enough to overcome the prior log-odds. In particular, for candidates u,v∈u,v , ργ(v)>ργ(u)⟺γ(V(v)−V(u))>logp(u)p(v). _γ(v)> _γ(u) γ (V(v)-V(u) )> p(u)p(v). (26) Thus, γ sets an explicit evidence threshold for reversing a teacher ranking. Candidates with equal values preserve their teacher-relative odds, and adding the same constant to all branch values leaves the target unchanged. The current training verifier is binary and uses one probe rollout per candidate, so the realized branch-value estimates lie in 0,1\0,1\. This case admits a particularly direct interpretation. Proposition A.7 (Binary-verifier odds update). Let +=v∈:V(v)=1S^+=\v :V(v)=1\, and define the prior and calibrated probability masses on successful branches as P+=∑v∈+p(v),Q+=∑v∈+ργ(v).P_+= _v ^+p(v), Q_+= _v ^+ _γ(v). Then Q+=eγP+1−P++eγP+,Q+1−Q+=eγP+1−P+,Q_+= e^γP_+1-P_++e^γP_+, Q_+1-Q_+=e^γ P_+1-P_+, (27) Moreover, DKL(ργ∥p)=DKL(Bern(Q+)∥Bern(P+)).D_KL( _γ\|p)=D_KL (Bern(Q_+)\,\|\,Bern(P_+) ). (28) These identities hold whenever 0<P+<10<P_+<1. Within either outcome group, the target preserves the teacher-relative probabilities. Proof. For successful branches the exponential factor in Eq. (14) is eγe^γ, whereas for unsuccessful branches it is 11. Hence Z(γ)=eγP++(1−P+),Z(γ)=e^γP_++(1-P_+), and summing Eq. (14) over +S^+ gives Eq. (27). If two candidates have the same binary value, their exponential factors cancel in their probability ratio, leaving the corresponding teacher odds unchanged. Hence the conditional distributions within +S^+ and its complement are unchanged. Applying the chain rule for KL divergence over this binary partition yields Eq. (28). ∎ Equation (28) shows that calibration spends its entire KL deviation on reallocating probability mass between the two outcome groups, introducing no within-group distortion. This proposition also clarifies the interaction between probing, gating, and calibration. If P+=0P_+=0, the position is removed by the ℬ+B^+ gate. If P+=1P_+=1, all candidates have equal value and ργ=p _γ=p. In the informative mixed case, outcome calibration multiplies the aggregate prior odds of successful versus unsuccessful branches by exactly eγe^γ, while retaining the finer teacher preferences within each group. Appendix B Implementation Details Off-policy Training. Following (Jin et al., 2026), KD is implemented with DistillKit (Goddard and Atkins, 2024), with one teacher response sampled per training problem. Table 4 summarizes the off-policy configuration. Table 4: Hyperparameters used for off-policy distillation. Hyperparameter Value Learning rate 1×10−51× 10^-5 LR scheduler type cosine Optimizer AdamW CE loss weight 0.5 Forward-KL loss weight 0.5 Training batch size 128 Training epoch 3 Cutoff length 4,096 Top-k (for FKL) 16 On-policy Training. We implement OPD, EOPD, GRPO, and SPOT with verl (Sheng et al., 2025) and generate rollouts asynchronously with SGLang (Zheng et al., 2024). All main runs use 4 × A800 GPUs, FSDP2, and bfloat16. Table 5 summarizes the principal training hyperparameters. Table 5: Hyperparameters used for on-policy distillation and GRPO. Hyperparameter OPD, EOPD, SPOT GRPO Learning rate 3×10−63× 10^-6 3×10−63× 10^-6 LR scheduler type cosine cosine Optimizer AdamW AdamW Training batch size 128 128 Mini-batch size 32 32 Samples per prompt 1 8 Temperature 1.0 1.0 Top-p 1.0 (Qwen), 0.8 (Llama) 1.0 (Qwen), 0.8 (Llama) Max prompt length 1,024 1,024 Max response length 4,096 4,096 Clipping ratio 0.2 0.2 All methods use AdamW, weight decay 0.010.01, gradient clipping at 1.01.0, and four mini-batch updates per rollout iteration; the distillation methods use no learning-rate warmup. EOPD applies a unit-weight top-16 forward-KL loss at teacher-entropy values of at least 0.80.8, while GRPO uses KL and entropy coefficients of 10−310^-3 and 0, respectively. Training Schedule. Table 6 records the scale-specific data and effective schedules of the main on-policy runs. Table 6: Training data and schedules. Iterations denote rollout iterations. Student Training set Examples Epochs Iterations Qwen3-0.6B-Base MATH 7,500 3 174 Qwen3-1.7B-Base MATH 7,500 3 174 Qwen3-4B-Base DAPO-Math 14,116 2 220 Training examples are shuffled, no validation set is used, and prompts exceeding the listed length limit are filtered. Chat Template. Following EOPD, the Qwen runs of OPD, EOPD, and SPOT use the Qwen3-8B teacher’s non-thinking chat format. With thinking disabled, the Qwen3-Base tokenizer renders the same prompt: <|im_start|>user query<|im_end|> <|im_start|>assistant <think> </think> . GRPO instead uses the default Qwen3-Base generation prompt, which omits the empty thinking block: <|im_start|>user query<|im_end|> <|im_start|>assistant . SPOT-specific Training. Table 7 summarizes the hyperparameters specific to SPOT; the acquisition, probing, gating, and loss are defined in Algorithm 1 and Section 3.2. Table 7: Hyperparameters used for SPOT. Hyperparameter Value Selected positions M 2 Scoring top-k ksk_s 16 Probing top-k kpk_p 4 λmass,λshape _mass, _shape 0.5,0.50.5,0.5 Reward-tilt coefficient γ 1.0 Branch-loss weight β 0.1 Probe rollouts per candidate 1 Training verifier binary rule-based boxed-answer match Evaluation Details. We use vLLM with bfloat16 precision and random seed 42, render one user message with the model’s native chat template and thinking disabled, and append the instruction Please reason step by step, and put your final answer within . We extract the final balanced expression and use Math-Verify (Kydlíček, ) for symbolic equivalence; missing or unparsable answers are counted as incorrect. Table 8: Sizes of the evaluation benchmarks. Benchmark Problems MATH-500 500 AMC 2023 40 Minerva Math 272 HMMT 2025 (Feb.+Nov.) 60 AIME 2024 30 AIME 2025 30 Appendix C More Ablation Experiment Results C.1 Out-of-Domain Generalization Table 9 reports the complete out-of-domain results analyzed in Section 4.3.5. Together, these benchmarks separate deliberative reasoning, broad knowledge, and instruction following, providing a more diagnostic view of transfer than any single aggregate score. Table 9: Out-of-domain results for Qwen3-1.7B-Base trained on MATH (all values in %). GPQA-Diamond uses eight samples per question; MMLU-Pro uses category-matched five-shot CoT demonstrations and reports Pass@1; and AlpacaEval 2.0 reports win rate (WR) and length-controlled win rate (LC-WR). Bold and underlined entries denote the best and second-best results, respectively. The SPOT column is shaded. AlpacaEval scores marked † use 803 (EOPD) and 800 (SPOT) valid judgments out of 805, respectively. Benchmark Metric KD GRPO OPD EOPD SPOT (Ours) GPQA-Diamond Avg@8 21.10 19.26 25.00 27.21 29.42 Pass@8 62.18 55.05 67.17 69.19 80.81 MMLU-Pro Pass@1 38.13 41.86 41.23 42.90 42.26 AlpacaEval 2.0 LC-WR 24.10 21.86 27.08 28.13† 27.59† WR 27.58 25.71 31.18 33.23† 34.63† The comparison reveals a structured transfer profile rather than uniform dominance. Relative to the strongest baseline on GPQA-Diamond, SPOT improves Avg@8 by 2.21 points and Pass@8 by 11.62 points, and it leads AlpacaEval WR by 1.40 points. On MMLU-Pro, SPOT ranks second at 42.26: it trails EOPD by only 0.64 points, while exceeding GRPO and OPD by 0.40 and 1.03 points, respectively. This near-best result indicates that branch calibration preserves broad subject-matter reasoning despite training only on MATH, while EOPD’s edge suggests that token-level uncertainty matching remains especially useful for heterogeneous knowledge questions. On AlpacaEval, length control reverses the raw-WR ordering, indicating that part of the preference gain is associated with response length. Overall, outcome calibration transfers most clearly to deliberative reasoning, with method-dependent tradeoffs on broader forms of generalization. C.2 Consistency Across Model Families To evaluate the generality of our method beyond Qwen3, we repeat the comparison with a Llama-3.2-3B-Instruct student trained on MATH for three epochs and a Llama-3.1-8B-Instruct teacher for the distillation methods. All methods retain their native optimization protocols and are evaluated with eight samples under identical decoding. Because this setting differs from the Qwen experiments in architecture, checkpoint type, and student–teacher scale, Table 10 compares methods only within Llama; the experiment tests transfer of the relative method pattern, not absolute performance across families. Table 10: Results on the Llama model family. Macro avg. is the unweighted mean over the four benchmarks, computed before rounding. Bold and underlined entries denote the best and second-best results, respectively. SPOT cells are shaded in blue for Avg@8 and green for Pass@8. Benchmark Metric GRPO OPD EOPD SPOT MATH500 Avg@8 43.65 35.90 37.75 43.78 Pass@8 61.60 63.60 65.20 66.20 AMC23 Avg@8 24.06 13.13 19.06 23.44 Pass@8 47.50 45.00 47.50 50.00 AIME24 Avg@8 2.08 0.83 1.67 2.50 Pass@8 16.67 6.67 13.33 16.67 AIME25 Avg@8 0.00 0.42 0.00 0.42 Pass@8 0.00 3.33 0.00 3.33 Macro avg. Avg@8 17.45 12.57 14.62 17.53 Pass@8 31.44 29.65 31.51 34.05 Within-family comparison. Within the Llama setting, SPOT raises macro Avg@8 from 12.57 for OPD to 17.53 and macro Pass@8 from 29.65 to 34.05. Relative to the strongest baseline for each metric, however, the distinction is asymmetric: Avg@8 is essentially tied with GRPO (17.53 vs. 17.45), whereas Pass@8 exceeds EOPD by 2.54 points. The benchmark-level results show the same pattern. SPOT is best or tied-best in Pass@8 on all four benchmarks, while GRPO remains stronger in AMC23 Avg@8 and OPD ties SPOT on AIME25. Thus, the evidence supports a more consistent improvement in multi-sample solution coverage than in average per-sample accuracy, rather than uniform dominance on every benchmark. Cross-family interpretation. This asymmetry mirrors the Qwen results: across the 0.6B and 1.7B students trained on MATH and the 4B student trained on DAPO, SPOT leads macro Pass@8 in every setting, while macro Avg@8 is best or nearly tied with the strongest baseline. Its recurrence across Base and Instruct checkpoints, two model families, multiple capacities, and two training-data regimes makes a Qwen-specific explanation less plausible. The aggregates are not directly comparable because the Llama panel contains four benchmarks whereas the Qwen panel contains six; moreover, architecture, capacity, initialization, teacher pairing, and training data are not independently controlled. We therefore view these results as evidence of portability across the evaluated configurations, not architecture invariance or a causal scaling law.