Paper deep dive
ResOPD: Tail Residualization for Sparse On-Policy Distillation
Penghui Yang, Long Xing, Xuanlang Dai, Ziyu Liu, Kai Chen, Yuhang Zang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy distillation (OPD) trains a student model on self-generated trajectories, but transmitting dense teacher distributions across long reasoning traces creates prohibitive communication and memory bottlenecks. Practical systems therefore rely on sparse teacher interfaces, typically transmitting either the sampled-token score or a small Top-$k$ distribution. However, this sparse setting faces a fundamental dilemma: sampled-token estimators are unbiased but suffer from severe gradient variance, whereas directly optimizing Top-$k$ objectives introduces systematic bias. To improve this trade-off, we propose ResOPD (On-Policy Distillation with Tail Residualization), which provides unbiased full-vocabulary reverse KL gradient estimation under on-policy sampling, with substantial variance reduction under the same sparse payload in the evaluated settings. ResOPD aggregates the unobserved vocabulary into an observable coarse tail event, computes its exact aggregate gradient, and samples only the fine-grained within-tail residual, which requires no additional teacher queries or forward passes. Extensive experiments demonstrate that ResOPD substantially reduces gradient variance, stabilizes online training dynamics, and improves downstream performance across the evaluated settings. These results establish ResOPD as an efficient, plug-and-play variance reduction primitive for sparse on-policy distillation.
Tags
Links
- Source: https://arxiv.org/abs/2610.04882v1
- Canonical: https://arxiv.org/abs/2610.04882v1
Trouble viewing inline? Open PDF directly →
Full Text
85,048 characters extracted from source content.
Expand or collapse full text
logotrue † † Corresponding author: Yuhang Zang (zangyuhang@pjlab.org.cn).† Code is available at https://github.com/InternLM/ResOPD. ResOPD: Tail Residualization for Sparse On-Policy Distillation Penghui Yang Affiliation: Shanghai Jiao Tong University Affiliation: Shanghai Artificial Intelligence Laboratory Long Xing Affiliation: University of Science and Technology of China Xuanlang Dai Affiliation: Shanghai Artificial Intelligence Laboratory Affiliation: Fudan University Ziyu Liu Affiliation: Shanghai Jiao Tong University Affiliation: Shanghai Artificial Intelligence Laboratory Kai Chen Affiliation: Shanghai Artificial Intelligence Laboratory Yuhang Zang† Affiliation: Shanghai Artificial Intelligence Laboratory Abstract On-policy distillation (OPD) trains a student model on self-generated trajectories, but transmitting dense teacher distributions across long reasoning traces creates prohibitive communication and memory bottlenecks. Practical systems therefore rely on sparse teacher interfaces, typically transmitting either the sampled-token score or a small Top-k distribution. However, this sparse setting faces a fundamental dilemma: sampled-token estimators are unbiased but suffer from severe gradient variance, whereas directly optimizing Top-k objectives introduces systematic bias. To improve this trade-off, we propose ResOPD (On-Policy Distillation with Tail Residualization), which provides unbiased full-vocabulary reverse KL gradient estimation under on-policy sampling, with substantial variance reduction under the same sparse payload in the evaluated settings. ResOPD aggregates the unobserved vocabulary into an observable coarse tail event, computes its exact aggregate gradient, and samples only the fine-grained within-tail residual, which requires no additional teacher queries or forward passes. Extensive experiments demonstrate that ResOPD substantially reduces gradient variance, stabilizes online training dynamics, and improves downstream performance across the evaluated settings. These results establish ResOPD as an efficient, plug-and-play variance reduction primitive for sparse on-policy distillation. keywordsLarge Language Models, On-Policy Distillation, Variance Reduction, Sparse Teacher Feedback 1 Introduction On-policy distillation (OPD) [1, 9] transfers reasoning and generation ability from a strong teacher to a smaller student. Unlike offline distillation [17] or supervised fine-tuning, it trains the student on self-generated trajectories with token-level teacher supervision, which reduces exposure bias over long sequences. OPD is now a standard step in Large Language Model (LLM) post-training at frontier labs, including Thinking Machines [20], GLM [29], and DeepSeek [27]. Standard OPD [1] needs the teacher’s full distribution at every generated token, which is prohibitively costly for modern LLMs whose vocabularies can reach 250K tokens, so the payload grows with vocabulary size per token and with rollout length per sequence. Practical training systems therefore expose a sparse teacher interface that, at each step, returns only the teacher’s Top-k tokens with their probabilities, along with the probability of the sampled token [30, 12]. We refer to OPD trained through such an interface as sparse OPD. The difficulty is that the omitted tail is never observed: objectives defined directly on the Top-k support, whether by truncation or renormalization [32] or by aggregating the omitted mass into a ghost token [3, 12], are surrogates of the full-vocabulary reverse Kullback-Leibler (rKL) objective and carry a systematic gradient bias (Fig. 1(b)). Bias can be traded for variance: rather than shrinking the objective to the observed support, a second family of methods retains the full-vocabulary rKL objective and estimates its gradient from the sparse payload, remaining unbiased but noisy. However, existing estimators fail to fully exploit this payload. At one extreme, sampled-token OPD [9, 20] queries only the rollout token’s score, suffering from severe gradient variance due to single-sample Monte Carlo estimation (Fig. 1(b)). At the other, methods with Top-k access underutilize it: scalar-baseline variants such as vOPD [23] reduce the head to a detached baseline rather than evaluating its token-wise gradient analytically (Tab. 2), while hybrid schemes such as Zhang and Ba [30], which we refer to as Head-Exact Tail-Sampled (HETS), evaluate the head exactly but treat the tail correction as entirely unknown despite knowing its aggregate mass. Consequently, each tail hit must compensate for the entire omitted mass, injecting large gradient spikes; even at 98.4% head coverage, a 30K-token reasoning trace contains 460–470 such spikes, thousands per optimizer step (Sec. 4.3.1). Practitioners are thus left choosing between a biased objective and an unstable gradient, which causes sampled-token OPD and HETS to peak early and degrade later in training (Sec. 4.3). Figure 1: (a) Sparse OPD: the teacher model only sends the Top-k head probabilities and the sampled token score, leaving the tail distribution unobserved. (b) Bias vs. variance trade-off: while deterministic Top-k surrogates suffer from systematic gradient bias, ResOPD provides an unbiased gradient estimator and substantially reduces gradient variance. (c) Downstream evaluation: the radar chart shows benchmark scores across mathematics, coding, and science; while the relative ranking varies by task, ResOPD achieves the best overall performance. To resolve this trade-off, we analyze sparse OPD at the level of the gradient estimator. We first show that any estimator evaluating the head exactly, adding a prefix-dependent control vector weighted by the student’s tail mass, and correcting on tail hits is unbiased for the full-vocabulary rKL gradient, with HETS as the special case of a zero control (Sec. 3.2). Our covariance-trace decomposition reveals that variance splits into an invariant within-tail sampling term and a between-event term proportional to the control’s distance from the true tail mean. This explains HETS’s gradient spikes: its zero control pays for the full tail mean on every hit, whereas any control that better predicts this mean strictly reduces variance. Crucially, the residual masses outside the head define a coarse pseudo-token [3, 12] whose analytical gradient serves as a computable tail-mean predictor. Building on this, we propose On-Policy Distillation with Tail Residualization (ResOPD): at every step it computes the exact gradient of the (k+1)(k+1)-way coarse distribution, adding only a small residual correction on tail hits (Fig. 2(a)). The resulting estimator is provably unbiased while retaining the same sparse teacher payload. We evaluate ResOPD in two settings. In exact frozen-prefix audits that enumerate every next-token action, ResOPD lowers the gradient covariance trace by 52–75% relative to sampled-token OPD under Top-44 feedback, below both HETS and a payload-matched vOPD baseline (Tab. 2). In end-to-end distillation from Qwen3.5-27B to Qwen3.5-4B, it attains the best average score among the evaluated sparse OPD baselines on four mathematical reasoning benchmarks, 4.40 points above HETS under the same payload (Tab. 3), and its average advantage over HETS consistently holds across every tested support size k∈1,4,16,32k∈\1,4,16,32\, with small margins (Tab. 6). Our contributions are as follows: (1) Bias and variance analysis of sparse OPD. We show that truncated, renormalized, and stopped-coefficient Top-k rKL objectives are all biased surrogates of the full-vocabulary objective, and we derive a covariance-trace decomposition that separates within-tail sampling variance from between-event variance. This yields the exact condition under which a coarse tail control lowers variance relative to HETS, and a two-control family in which HETS, ResOPD, and the concurrent SC-TA [12] appear as special cases. (2) ResOPD estimator. We propose an estimator that splits the full-vocabulary rKL gradient into an exact (k+1)(k+1)-way coarse term, computed at every step from the Top-k payload and the residual tail masses, and a within-tail residual correction applied only when the rollout token lands in the tail. The estimator is unbiased under on-policy sampling and requires no additional teacher queries or forward passes. (3) Empirical validation. Using exact frozen-prefix gradient audits and end-to-end distillation settings, we show that ResOPD reduces gradient covariance, lowers online residual moments during training, and improves downstream accuracy over the evaluated sparse OPD baselines. 2 Related Work On-Policy Language-Model Distillation. On-policy distillation (OPD) mitigates exposure bias by training students on self-generated rollouts under dense teacher supervision [9, 1]. Subsequent work explores skew divergences [18], adaptive targets [15, 11, 16], and diagnostic behaviors in complex reasoning regimes [8, 32]. However, full-vocabulary OPD incurs prohibitive communication and memory footprints during distributed rollout generation, severely hindering scalability. Sparse Divergences and Truncation Biases. To alleviate communication bottlenecks, distributed training frequently adopts sparse Teacher Top-k payloads [32]. Nonetheless, naively renormalizing or truncating to the Top-k support optimizes a biased surrogate that misallocates tail probability mass. While coarse-support approaches like TA-OPD [12] incorporate a ghost token [3] to aggregate omitted mass, they still optimize a surrogate (k+1)(k+1)-way divergence that discards the teacher’s fine-grained within-tail geometry. Stochastic Estimators and Variance Reduction. An alternative paradigm constructs stochastic estimators to recover the true full-vocabulary rKL gradient unbiasedly. Standard sampled-token estimators [9, 20] suffer from extreme Monte Carlo variance, motivating variance-reduction baselines like vOPD [23]. HETS, a hybrid scheme proposed in EMA-PG [30], computes the Top-k head analytically and samples only within the tail, yet still incurs severe gradient bursts upon tail hits, which destabilizes training dynamics. 3 Methodology In this section, we present the theoretical foundations and algorithmic formulation of On-Policy Distillation with Tail Residualization (ResOPD). We begin in Sec. 3.1 by establishing the full-vocabulary reverse-KL gradient and analyzing the trade-off of sparse teacher payloads, exposing how naive Top-k heuristics induce objective bias and why hybrid estimators like HETS suffer from tail-sampling variance. In Sec. 3.2, we formulate ResOPD (illustrated in Fig. 2), demonstrating how aggregate tail masses serve as an analytic control variate to achieve an unbiased, variance-reduced estimator. Sec. 3.3 describes the sampling assumptions and connects theoretical covariance bounds to observable online diagnostics and downstream task performance. Finally, Sec. 3.4 extends the analysis to a two-control family. 3.1 Preliminaries: From sampled gradients to an exact sparse head At one fixed trajectory prefix, let p(i)>0p(i)>0 and q(i)>0q(i)>0 denote Student and Teacher next-token probabilities over a shared vocabulary V. Define ri r_i ≔logpiqi, p_iq_i, i s_i ≔∇θlogpi, _θ p_i, i G_i ≔rii. r_i s_i. (1) Because ∑ipii= _ip_i s_i= 0, the full-vocabulary reverse-KL gradient is =∑ipii=A∼p[A]. g= _ip_i G_i=E_A p\! [ G_A ]. (2) Here, A G_A denotes the stopped-coefficient sampled-token OPD gradient. Equation (2) demonstrates that this sampled-token gradient is an unbiased estimator of the full-vocabulary reverse-KL gradient. Why direct Teacher Top-k rKL is biased. Let S be the fixed Teacher Top-k support, with 0<k<||0<k<|V|, P=∑i∈SpiP= _i∈ Sp_i and Q=∑i∈SqiQ= _i∈ Sq_i. Two common scalar shortcuts have different biases: Lrenormalized=DKL(pSP∥qSQ),Ltruncated=∑i∈Spilogpiqi.L_renormalized=D_KL\! ( p_SP \| q_SQ ), L_truncated= _i∈ Sp_i p_iq_i. (3) Renormalization depends only on the conditional distribution within S. And for every omitted logit zjz_j, ∂Lrenormalized/∂zj=0∂ L_renormalized/∂ z_j=0. Truncation retains the unnormalized head but introduces a boundary term: ∂Ltruncated∂zj=−pj∑i∈Spi(ri+1)⏟CS,j∉S. ∂ L_truncated∂ z_j=-p_j _i∈ Sp_i(r_i+1)_C_S, j∉ S. (4) When CS>0C_S>0, gradient descent increases every omitted logit. We adopt the stopped-coefficient alternative [32] =∑i∈Spirii h= _i∈ Sp_ir_i s_i in our experiments, which avoids that scalar boundary term but is still biased by the omitted tail. Appendix A.1 proves all three statements. Figure 2: Overview of ResOPD. (a) When the student sampled-token falls in the head, ResOPD uses only the exact coarse gradient; upon a tail hit, it adds a residual correction. (b) ResOPD suppresses between-event variance compared to HETS by centering tail samples around B μ_B, achieving superior alignment with the teacher distribution across rollout steps. The interface returns full-normalized qiq_i for i∈Si∈ S and qAq_A for the sampled action. Write τp≔1−P,τq≔1−Q,≔P[A∣A∈S]=∑i∈Spii,tail≔[A∣A∉S]=1τp∑i∉Spii. gathered _p 1-P, _q 1-Q,\\[-1.0pt] h P\,E[ G_A A∈ S]= _i∈ Sp_i G_i, μ_tail \! [ G_A A∉ S ]= 1 _p _i∉ Sp_i G_i. gathered (5) The head h is exactly known, while the fine-tail mean gradient is unavailable. With J=A∉SJ=I\A∉ S\, the prior Head-Exact Tail-Sampled estimator is ^HETS=+JA,[^HETS]=+τptail=. g^HETS= h+J G_A, \! [ g^HETS ]= h+ _p μ_tail= g. (6) This HETS baseline [30] equals stopped Teacher Top-k rKL on head draws and adds the entire correction only on tail draws: ^HETS=,A∈S,+A,A∉S,Pr(A∈S)=P. g^HETS= cases h,&A∈ S,\\ h+ G_A,&A∉ S, cases (A∈ S)=P. (7) Large P therefore creates a rare-correction regime: most tokens use the biased head-only update, while long minibatches still accumulate many sampled tail corrections. 3.2 Tail residualization Top-k also reveals aggregate tail masses. Following the tail-bucketing approach in Hübotter et al. [13] and Huang and Wei [12], collapsing the tail into a single pseudo-token yields b b ≔logτpτq, _p _q, ¯tail s_tail ≔[A∣J=1]=∇θlogτp, [ s_A J=1]= _θ _p, B μ_B ≔b¯tail. b\, s_tail. (8) The exact gradient of the resulting complete coarse distribution is coarse=+τpB. g_coarse= h+ _p μ_B. (9) Here B μ_B predicts, but need not equal, the unavailable fine-tail mean. In fact, this formulation can be generalized to arbitrary baseline vectors: for A∼pA p and any prefix-dependent baseline vector B, ^()=+τp+J(A−), g( B)= h+ _p B+J( G_A- B), (10) is unbiased, i.e., A∼p[^()]=E_A p\! [ g( B) ]= g. Fig. 2 summarizes the construction. HETS is the special case = B= 0. The expectation is explicit: [^()]=+τp+τp(tail−)=.E[ g( B)]= h+ _p B+ _p( μ_tail- B)= g. (11) The control may depend on the fixed prefix and current model probabilities, but must be fixed with respect to the sampled action and stopped before backward. Furthermore, defining the total variance of any random vector X as ()≔tr(Cov())V( X) (Cov( X)), applying the law of total covariance yields the explicit variance of ^() g( B): (^())=τptail(A)+Pτp∥tail−∥22,V( g( B))= _pV_tail( G_A)+P _p μ_tail- B _2^2, (12) where tail(A)≔tr(Cov(A∣A∈tail))V_tail( G_A) (Cov( G_A A )). Here, the first term τptail(A) _pV_tail( G_A) captures the baseline-invariant within-tail sampling variance (weighted by the tail probability τp _p), while the second term accounts for the between-event variance induced by the residual approximation error ∥tail−∥22 μ_tail- B _2^2, scaled by PτpP _p, the variance of the binary head-versus-tail indicator. To see this directly, condition on J: the head branch is deterministic, the tail branch retains Covtail(A)Cov_tail( G_A), and the two conditional means differ by tail− μ_tail- B. The law of total covariance then gives Equation (12). Equation (12) also shows that the variance of the estimator ^() g( B) is minimized when the control vector is set to the true tail mean, i.e., =tail B= μ_tail. However, since tail μ_tail is unavailable in practice, ResOPD substitutes it with the computable tail-bucket surrogate B μ_B: ^ResOPD=coarse+J(A−B). g^ResOPD= g_coarse+J( G_A- μ_B). (13) The predictable coarse correction now appears on every draw; only the fine-tail residual remains rare. ResOPD achieves strictly lower variance than HETS whenever ∥tail−B∥2<∥tail∥2 μ_tail- μ_B _2< μ_tail _2, i.e., whenever the tail-bucket vector B μ_B predicts the true tail mean better than zero, as observed in aggregate in our frozen-prefix audits. The complete derivation is given in Appendix A.2. 3.3 Sampling assumptions and evidence boundaries The unbiasedness identities above assume A∼pθA p_θ at the fixed prefix. Rollout filters change this sampling law. Our primary mathematics experiments (Sec. 4.3) use temperature 1.01.0, rollout top-p=0.95p=0.95, and rollout top-k=20k=20, while the full-support audits (Sec. 4.2) and cross-domain runs (Sec. 4.4) use the ancestral condition. Sec. 4.1 specifies both, and Sec. 4.6 evaluates the effect of removing the filters. The coarse-to-fine variance decomposition can also be characterized under an arbitrary realized rollout distribution p~ p. Let τ~=Prp~(A∉S) τ= _ p(A∉ S) and ~tail=p~[A∣A∉S] μ_tail=E_ p[ G_A A∉ S]. For ^()=+τp+A∉S(A−) g( B)= h+ _p B+I\A∉ S\( G_A- B), the estimator still admits the exact decomposition p~(^())=τ~p~,tail(A)+(1−τ~)τ~‖~tail−‖22.V_ p( g( B))= τV_ p,tail( G_A)+(1- τ) τ\| μ_tail- B\|_2^2. (14) Hence, for 0<τ~<10< τ<1, ResOPD has strictly lower sampling variance than HETS whenever ‖~tail−B‖2<‖~tail‖2\| μ_tail- μ_B\|_2<\| μ_tail\|_2. The expected update under this realized sampling law is p~[^()]=+τp+τ~(~tail−).E_ p[ g( B)]= h+ _p B+ τ( μ_tail- B). (15) The variance identity remains exact under p~ p, but it does not turn a truncated rollout into an unbiased draw from pθp_θ. Crucially, truncation is confined strictly to sequence generation. All student quantities (pip_i, rir_i, sis_i, τp _p, and the coarse-tail terms) are faithfully recomputed via a fresh, untruncated forward pass during training. Equation (12) characterizes estimator quality at fixed prefixes. We test it by exact action enumeration (Sec. 4.2). During online OPD, both policies and visited prefixes change, while sparse Teacher feedback hides the fine tail. We instead monitor residual second moments M2M_2 as cheap mechanism-aligned online diagnostics. Their decline is consistent with reduced head-versus-tail event noise, but does not by itself estimate full gradient covariance. Appendix D.1 gives the exact definitions. Nevertheless, variance reduction does not imply monotonic downstream accuracy gains. Lower covariance trace reduces stochastic optimization noise, but final performance is still shaped by training dynamics and decoding randomness, both of which are discretized by benchmark scoring. Thus, ResOPD can reliably suppress gradient noise without strictly outperforming HETS at every checkpoint. We therefore view theoretical covariance, online residual moments, and downstream accuracy as complementary signals across different stages of the learning pipeline. 3.4 A Unified View of Sparse Gradient Estimators ResOPD changes the between-event component of the HETS variance while preserving its conditional within-tail randomness. This distinction provides a useful way to compare sparse unbiased estimators: a second, conditionally zero-mean control can act inside the tail. We present the family, its oracle optimum, and the exact conditions governing the ResOPD–SC-TA comparison here. The corresponding audits and downstream results appear in Sec. 4.8. 3.4.1 Separating between-event and within-tail controls The single-control family acts only on the head-versus-tail event component. A second control can reduce the conditional variance that remains inside the tail without changing the sparse Teacher payload. Define the conditionally zero-mean score fluctuation A=b(A−¯tail) C_A=b( s_A- s_tail), so that [A∣J=1]=E[ C_A J=1]= 0. For any finite prefix-measurable B and scalar γ, both stopped before drawing A, define ^(,γ)=+τp+J(A−γA). g( B,γ)= h+ _p B+J\! ( G_A- B-γ C_A ). (16) Equation (16) satisfies [^(,γ)] \! [ g( B,γ) ] =+τp+τp(tail−)−γτp[A∣J=1] = h+ _p B+ _p( μ_tail- B)-γ _pE[ C_A J=1] (17) =+τptail=. = h+ _p μ_tail= g. Conditioned on the head event its value is +τp h+ _p B, while conditioned on the tail event its mean is +τp+tail− h+ _p B+ μ_tail- B. Their difference is tail− μ_tail- B, while only the tail branch retains conditional randomness. The law of total covariance therefore gives (^(,γ))=τptail(A−γA)+Pτp∥tail−∥22.V\! ( g( B,γ) )= _pV_tail( G_A-γ C_A)+P _p μ_tail- B _2^2. (18) This factorization proves that the two controls are orthogonal in trace covariance: B changes only the between-event term and γ only the within-tail term. We call Equation (16) the Unified family: the first coordinate generalizes the single control in Equation (10), while the second subtracts a known-zero-mean within-tail fluctuation. If tail(A)>0V_tail( C_A)>0, their joint oracle is ⋆=tail,γ⋆=tr(Covtail(A,A))tail(A), B = μ_tail, γ = tr\! (Cov_tail( G_A, C_A) )V_tail( C_A), (19) and completing the square in both coordinates yields (^(,γ))−(^(⋆,γ⋆)) \! ( g( B,γ) )-V\! ( g( B ,γ ) ) (20) =Pτp∥−tail∥22+τptail(A)(γ−γ⋆)2. =P _p B- μ_tail _2^2+ _pV_tail( C_A)(γ-γ )^2. If tail(A)=0V_tail( C_A)=0, γ is immaterial and the first term remains exact. Importantly, neither oracle coordinate is available online: ⋆ B requires the full Teacher tail, while the per-prefix γ⋆γ requires its conditional covariance. ResOPD instead makes the simple but effective choice (B,0)( μ_B,0), and the global fitted coefficient is a separate offline diagnostic. Table 1: Coordinates in the two-control family. Unbiased estimator B γ Role HETS 0 00 structural baseline ResOPD B μ_B 00 our estimator SC-TA B μ_B 11 boundary estimator Joint oracle tail μ_tail γ⋆γ unavailable optimum 3.4.2 ResOPD and SC-TA as endpoints We now specialize the two-level family to =B B= μ_B and connect it to sample-corrected TA-OPD (SC-TA) from the concurrent work of Huang and Wei [12]. Their primary method, TA-OPD, augments the Teacher Top-k support with a tail pseudo-token and optimizes the resulting coarse (k+1)(k+1)-way reverse-KL objective. SC-TA is subsequently derived as a sample-corrected variant that restores full-vocabulary unbiasedness. We complement this construction with a gradient-level characterization within our two-control family. In particular, we derive the exact pointwise relation between ResOPD and SC-TA, characterize their covariance ordering as a function of the within-tail control coefficient γ, identify the trace-optimal γ⋆γ , and evaluate these predictions across natural prompt distributions. Exact fine-tail decomposition. On a tail draw, rA=logpAqA=logτpτq⏟b+logpA/τpqA/τq.r_A= p_Aq_A= _p _q_b+ p_A/ _pq_A/ _q. (21) Using B=b¯tail μ_B=b s_tail therefore gives A=B+b(A−¯tail)⏟A+(rA−b)A⏟A,[A∣J=1]=. G_A= μ_B+ b( s_A- s_tail)_ C_A+ (r_A-b) s_A_ R_A, [ C_A J=1]= 0. (22) The terms have different roles. B μ_B is the coarse tail-bucket prediction; A C_A is a known-zero-mean fluctuation around the conditional Student score; and A R_A records the unresolved Student–Teacher conditional tail-shape mismatch. With coarse=+τpB g_coarse= h+ _p μ_B, the specialization of Equation (16) is ^γ=coarse+J[A+(1−γ)A]. g_γ= g_coarse+J\! [ R_A+(1-γ) C_A ]. (23) Equation (17) already proves that every fixed γ targets g. Its two named endpoints are ^ResOPD g^ResOPD =^γ=0=coarse+J(A+A), = g_γ=0= g_coarse+J( R_A+ C_A), (24) ^SC g^SC =^γ=1=coarse+JA. = g_γ=1= g_coarse+J R_A. Hence their signed pointwise relationship is ^SC=^ResOPD−JA g^SC= g^ResOPD-J C_A (25) The difference has zero expectation but is generally nonzero on tail actions. Variance optimum and endpoint certificate. Let 0=A+A=A−B Z_0= R_A+ C_A= G_A- μ_B. Decomposing the total variance into the common between-event component and the γ-dependent within-tail component yields (^γ) ( g_γ) =τptail(0−γA)+Pτp∥tail−B∥22, = _pV_tail( Z_0-γ C_A)+P _p μ_tail- μ_B _2^2, (26) tail(0−γA) _tail( Z_0-γ C_A) =tail(0)−2γtr(Covtail(0,A))+γ2tail(A). =V_tail( Z_0)-2 \! (Cov_tail( Z_0, C_A) )+γ^2V_tail( C_A). When tail(A)>0V_tail( C_A)>0, completing the square with respect to γ identifies the trace optimum: γ⋆ γ =tr(Covtail(0,A))tail(A), = tr\! (Cov_tail( Z_0, C_A) )V_tail( C_A), (27) (^γ)−(^γ⋆) ( g_γ)-V( g_γ ) =τptail(A)(γ−γ⋆)2. = _pV_tail( C_A)(γ-γ )^2. Evaluating the quadratic regret at the boundary points γ=1γ=1 (SC-TA) and γ=0γ=0 (ResOPD) reveals that SC-TA achieves strictly lower trace covariance than ResOPD if and only if γ⋆>1/2γ >1/2. Substituting 0=A+A Z_0= R_A+ C_A expands the optimal coefficient as γ⋆=1+tr(Covtail(A,A))tail(A).γ =1+ tr\! (Cov_tail( R_A, C_A) )V_tail( C_A). (28) Consequently, SC-TA outperforms ResOPD exactly when tr(Covtail(A,A))>−tail(A)/2tr\! (Cov_tail( R_A, C_A) )>-V_tail( C_A)/2. By the Cauchy–Schwarz inequality, a sufficient condition is tail(A)<tail(A)/4.V_tail( R_A)<V_tail( C_A)/4. (29) In particular, when the conditional tail shapes match, the residual vanishes (A= R_A= 0), yielding γ⋆=1γ =1. These conditions serve as distribution-dependent certificates rather than guaranteeing a universal variance ordering. 4 Experiments We evaluate three complementary questions: whether residualization reduces exact conditional gradient covariance, whether its observable effects persist during online training, and whether the resulting policies improve downstream performance. We first specify the protocols, then present frozen-prefix audits, mathematics results and diagnostics, cross-domain transfer, and controlled ablations of support size, rollout sampling, learning rate, and the within-tail control. 4.1 Experimental Setup Offline mechanism audit. We freeze Qwen3.5 [24] models and sample one 64-token trajectory per prompt at selected frozen Math prefixes, measuring covariance trace and Noise-to-Signal Ratio: NSR=(^)‖full-rKL‖22 V( g)\| g_full-rKL\|_2^2. Experiments cover five settings: GSM8K [4] (32 prompts) across 0.8B/4B, 4B/27B and 9B/27B pairs; all 33 HMMT26 prompts [5] (4B/27B); and 64 prompts sampled from DeepMath-103K [10] (4B/27B). Online OPD training for mathematical reasoning. We distill the thinking-enabled Qwen3.5-4B student from the Qwen3.5-27B teacher. Starting from DeepMath-103K, we filter for challenging competition problems after deduplication and benchmark contamination removal. To ensure an appropriate teacher-student capability gap, we retain prompts where the teacher succeeds twice while the student fails at least once across sampled rollouts. This yields 1,276 unique problems, partitioned into 1,024 for training and 252 for conditional held-out evaluation. Training runs for 3 epochs (192 steps), saving intermediate checkpoints at steps 6, 32, 64, 128, and 192. Evaluation. Our evaluation spans four benchmarks: the DeepMath held-out set, AIME26 (30 problems) [31], HMMT26 (33 problems) [5], and the English split of OlymMATH (200 problems) [26]. Given the limited problem counts in AIME26 and HMMT26, we report the average over 16 sampled responses per problem (avg@16) to mitigate evaluation variance, while DeepMath held-out and OlymMATH use avg@2. In addition to terminal performance, we evaluate all intermediate saved checkpoints to compute the Area Under the Curve (AUC), quantifying learning dynamics and training efficiency. We compute normalized trapezoidal AUC over the saved checkpoint trajectory, and average the benchmark-wise scores and AUCs for the aggregate columns. Full configurations for both training and evaluation are collected in Appendix B. Realized rollout distributions. Following the Qwen3.5 official recipe, the primary mathematics rollouts and evaluations use temperature 1.01.0, top-p=0.95p=0.95, and top-k=20k=20. Truncating low-probability tokens helps avoid degenerate prefixes that can destabilize OPD and Teacher feedback [8]. The Student probabilities used in training are recomputed by a fresh, untruncated forward pass. As explained in Sec. 3.3, this setup retains a well-defined sampling-variance comparison but does not satisfy the idealized full-rKL unbiasedness assumption. We therefore also report a matched full-support mathematics ablation (Sec. 4.6) and cross-domain training with top-p=1.0p=1.0, top-k=−1k=-1 (Sec. 4.4). 4.2 Exact frozen-prefix covariance audits Table 2: Exact frozen-prefix covariance trace and NSR. Covariance traces of student logit-gradients are computed via full action enumeration and averaged across rollouts; NSR normalizes each trace by the squared L2L_2 norm of the full-rKL gradient. Sparse estimators (vOPDT-Top-kvOPD_T-Top-k, HETS, ResOPD) share an identical teacher budget of k=4k=4, with vOPDT-Top-kvOPD_T-Top-k adopting a detached conditional-KL baseline over this support. Best results per row are in bold, and second-best are underlined. Dataset Models Covariance Trace ↓ NSR ↓ Sampled vOPDT-Top-kvOPD_T-Top-k HETS ResOPD Sampled vOPDT-Top-kvOPD_T-Top-k HETS ResOPD GSM8K 0.8B / 4B 2.146 1.730 1.559 1.018 5.422 4.370 3.940 2.573 GSM8K 9B / 27B 0.215 0.192 0.107 0.068 6.202 5.560 3.101 1.955 GSM8K 4B / 27B 0.333 0.297 0.183 0.108 6.058 5.400 3.335 1.964 HMMT26 4B / 27B 0.208 0.189 0.092 0.052 6.589 5.977 2.914 1.630 DeepMath 4B / 27B 0.183 0.169 0.087 0.047 7.352 6.785 3.500 1.907 Tab. 2 reports the exact covariance trace and NSR across five frozen-prefix panels. By analytically enumerating every next-token action at each Student-visited prefix, we isolate the conditional sampling variance of each estimator without decoding noise or optimizer feedback. Under the panel-level aggregation used in the table, both metrics exhibit a consistent ordering across every panel: ResOPD<HETS<vOPD<sampled-token OPDResOPD<HETS<vOPD<sampled-token OPD (lower is better). Relative to sampled-token OPD, the aggregate covariance trace is reduced by 7.7%–19.4%7.7\%--19.4\% for payload-matched Teacher-Top-k vOPD, 27.3%–55.8%27.3\%--55.8\% for HETS, and 52.5%–75.0%52.5\%--75.0\% for ResOPD. While Teacher-Top-k vOPD serves as a competitive baseline in our end-to-end experiments (Tab. 3), its detached scalar control fails to exploit the structural head–tail decomposition, leaving the entire action vector sampled at every step. Consequently, its variance reduction remains modest compared to HETS and ResOPD. Because HETS and ResOPD share the identical payload budget, exact Top-4 head integration, and conditional target structure, their contrast directly isolates the gain from coarse-tail residualization without baseline confounds. We therefore focus our primary ablations on the HETS-to-ResOPD progression. As the support size k increases, the exact Top-k head absorbs more probability mass, causing the residual tail opportunity PτpP _p to contract. Accordingly, the marginal covariance reduction of ResOPD over HETS gradually narrows. The full support-size sweep in Appendix D.3 tests this communication–variance trade-off directly. 4.3 End-to-end mathematical reasoning Table 3: Qwen3.5-27B-to-4B OPD results. DeepMath held-out and OlymMATH are evaluated using avg@2, while AIME26 and HMMT26 adopt avg@16 due to their smaller problem sets. Method Benchmark Accuracy (%) Aggregate (%) DeepMath held-out (avg@2) AIME26 (avg@16) HMMT26 (avg@16) OlymMATH (avg@2) Avg. Score Avg. AUC ▼ Original Models Qwen3.5-4B Student 48.41 56.25 35.23 31.50 42.85 – Qwen3.5-27B Teacher 89.29 80.62 62.88 63.25 74.01 – ▼ Alternative Sparse OPD (Biased) Teacher Top-k fKL 50.20 54.58 35.04 34.50 43.58 42.80 Teacher Top-k rKL 51.79 55.83 34.66 36.75 44.76 44.63 TA-OPD 58.13 58.96 39.58 36.00 48.17 47.94 ▼ Unbiased rKL Estimators Sampled-token OPD 48.21 57.50 36.17 34.25 44.03 46.69 vOPDT-Top-k_T-Top-k 58.93 60.00 39.20 37.25 48.85 47.49 HETS 54.76 56.25 37.88 34.00 45.72 46.18 ResOPD (ours) 60.91 59.38 39.96 40.25 50.12 47.96 Tab. 3 reports the primary distillation results: ResOPD achieves the strongest aggregate performance among all evaluated OPD baselines, attaining both the highest terminal average score (50.12%) and learning-trajectory AUC (47.96%). This favorable training trajectory also extends to external mathematical competitions. Across AIME26, HMMT26, and OlymMATH, competing baselines exhibit pronounced post-peak degradation during later stages of training, whereas ResOPD avoids severe late-stage degradation and recovers to its best aggregate performance at the terminal checkpoint. Step-wise evaluations over intermediate checkpoints are provided and analyzed in Appendix C.1. 4.3.1 Training dynamics Table 4: Aggregated online diagnostics across training (lower is better). Values represent the average over all 192 optimization steps. Δ denotes the relative percentage reduction of ResOPD compared to HETS. Diagnostic Metric Top-k rKL vOPDT-Top-kvOPD_T-Top-k HETS ResOPD (ours) Δ (%)↓ Support-mass gap (×10−3× 10^-3) 11.17 3.53 5.17 3.50 −32.3%-32.3\% ||Tail log-mass mismatch|| 0.715 0.858 0.723 0.322 −55.5%-55.5\% Token residual M2M_2 (×10−3× 10^-3) – 10.55 11.73 10.67 −9.0%-9.0\% Trajectory residual M2M_2 (×10−5× 10^-5) – 2.76 3.50 2.65 −24.3%-24.3\% Figure 3: Training-time diagnostics (64 steps/epoch; lower is better). Panels (a) and (b) illustrate the student–teacher mismatch in Top-k support mass and tail log-mass across vOPD, HETS, ResOPD, and direct Teacher Top-k rKL. Panels (c) and (d) compare vOPD, HETS and ResOPD via the squared residual between the binary Top-k indicator and its student-predicted probability, evaluated at the token level and trajectory level, respectively. Detailed formulations are provided in Appendix D.1. As illustrated in Tab. 4 and Fig. 3, ResOPD consistently suppresses both student–teacher mass discrepancies and stochastic residual fluctuations relative to HETS on average over 192 training steps. Across the matched HETS–ResOPD runs, ResOPD reduces the support-mass gap by 32.3%32.3\% and the tail log-mass mismatch by 55.5%55.5\%. Although Teacher-Top-k vOPD exhibits a comparable support-event M2M_2, this scalar residual on self-generated trajectories omits centered log-ratio and score geometry, and exact gradient covariance comparisons on shared frozen prefixes (Tab. 2) confirm ResOPD’s structural advantage. This provides a mechanistic explanation for the optimization behavior of ResOPD. Under HETS, the full fine-tail gradient correction is concentrated on rare tail-hit events. Despite approximately 98.4%98.4\% head coverage, a 30K30K-token reasoning trace still contains 460–470 such corrections, accumulating to thousands per optimizer step. By subtracting the payload-available tail-bucket component B μ_B, ResOPD redistributes the predictable between-event component while preserving the full-rKL expectation under the on-policy condition in Sec. 3.3. Downstream accuracy remains noisier because it depends on discrete answer correctness, but the completion-only Teacher NLL audit below shows consistently higher teacher likelihood of student-generated trajectories for ResOPD. Together with the matched endpoint and curve-AUC results, these diagnostics connect the estimator-level covariance reduction to a smoother optimization path and improved downstream performance. 4.3.2 Behavioral fidelity Beyond task-specific accuracy metrics (e.g., avg@2 and avg@16 on downstream math benchmarks), monitoring the student’s negative log-likelihood under the teacher model provides an intrinsic, task-agnostic readout of distillation fidelity. Specifically, a lower Teacher NLL indicates that the student samples receive higher likelihood under the teacher. Fig. 4 tracks this metric across the matched Math run. As illustrated in Panel A, while both estimators begin around 0.460.46 nats/token, ResOPD achieves a steeper drop by step 6 and sustains a monotonic decline down to 0.360.36 nats/token by step 192. In contrast, HETS plateaus around 0.3850.385 nats/token and displays slight drift in later stages. The difference ΔTeacher-NLL (ResOPD−HETS) -NLL (ResOPD-HETS) in Panel B remains strictly negative across all optimizer steps, widening to over −0.027-0.027 nats/token. This demonstrates that ResOPD consistently preserves higher behavioral fidelity to the teacher than HETS, offering evidence of higher teacher-likeness, complementary to accuracy. 4.4 Cross-domain transfer to science and coding Figure 4: Teacher negative log-likelihood (NLL) across training in the Math OPD setup. Evaluations are computed over the four core benchmark families with identical weights to the main suite: AIME26, HMMT26, DeepMath held-out, and OlymMATH. Lower values indicate higher fidelity to the Teacher distribution. Table 5: Distillation results from Qwen3-30B-A3B-Instruct to Qwen3-4B. Evaluations use avg@10 for GPQA-D and avg@3 for SuperGPQA, LCB-v5, and LCB-v6. AUC integrates performance across the fixed training trajectory (Base/10/25/50/100). Method Science Benchmarks (%) Code Benchmarks (%) Aggregate (%) GPQA-D (avg@10) SuperGPQA (avg@3) LCB-v5 (avg@3) LCB-v6 (avg@3) Avg. Score Avg. AUC ▼ Original Models Student 45.25 32.46 21.76 24.38 30.96 – Teacher 56.06 48.87 45.51 41.71 48.04 – ▼ Unbiased rKL Estimators Sampled-token OPD 50.30 36.87 43.31 39.43 42.48 41.21 HETS 48.89 36.10 46.31 43.24 43.63 42.13 ResOPD (ours) 47.12 37.54 48.90 43.05 44.15 42.65 To further evaluate the generalization and cross-domain effectiveness of ResOPD, we follow Heo et al. [11] and extend our investigation to science and coding tasks. Our experimental protocol largely mirrors that of Heo et al. [11]: we distill from a Qwen3-30B-A3B-Instruct [28] teacher into a Qwen3-4B student, training on a 30k-sample mixture compiled from OpenMathReasoning [21], OpenScienceReasoning-2 [22], and OpenCodeReasoning [2]. The run uses full-support ancestral rollouts with temperature 1.01.0, top-p=1.0p=1.0, and top-k=−1k=-1, without thinking mode. For evaluation, we benchmark the code domain on LiveCodeBench [14, LCB;] v5 (167 problems) and v6 (175 problems). For the science domain, we evaluate on GPQA-Diamond [25, GPQA-D, 198 problems;] and SuperGPQA [6], adopting the exact 650-problem subset used in Heo et al. [11]. Tab. 5 reports the evaluation results. Overall, ResOPD achieves the strongest aggregate results, leading in both terminal average score (44.15%44.15\%) and trajectory AUC (42.65%42.65\%). Although ResOPD and HETS nominally surpass the teacher on code tasks (e.g., ResOPD’s 48.90%48.90\% vs. 45.51%45.51\% on LCB-v5), paired bootstrap 95% CIs cross zero (e.g., ResOPD−-Teacher spans [−2.48,+5.33][-2.48,+5.33] on LCB-v6), indicating on-par performance rather than statistically significant super-teacher generalization. Across training scales, ResOPD maintains consistent optimization advantages (e.g., reducing trajectory residual M2M_2 by 9.5%9.5\%–12.6%12.6\% and gradient norms by 14.6%14.6\%–23.7%23.7\%) and exhibits directional AUC gains over HETS on code (+0.48+0.48 and +0.73+0.73 points), even though Sampled-token OPD yields the strongest performance on science tasks (detailed training checkpoint curves are provided in Appendix C.2). 4.5 Online support-size ablation Table 6: Performance under different Top-k support sizes (matched one-epoch budget). Evaluation setups follow Tab. 3 (avg@16 for AIME26/HMMT26 and avg@2 for OlymMATH). Δ denotes the average improvement of ResOPD over HETS. k Method AIME26 HMMT26 OlymMATH Average Δ 1 HETS 60.62 37.69 34.75 44.35 +0.11+0.11 ResOPD 56.88 39.02 37.50 44.47 4 HETS 59.17 36.93 34.50 43.53 +1.01+1.01 ResOPD 60.00 37.88 35.75 44.54 16 HETS 57.92 39.39 35.75 44.35 +0.70+0.70 ResOPD 59.38 39.02 36.75 45.05 32 HETS 59.79 39.02 36.75 45.19 +1.56+1.56 ResOPD 61.67 38.07 40.50 46.75 Figure 5: Online diagnostic metrics across teacher Top-k support sizes. (a) Between-event variance factor PτpP _p for HETS and ResOPD, characterizing the scaling coefficient of the head-versus-tail partition. (b) Tail log-mass mismatch reduction (HETS−ResOPDHETS-ResOPD, measured via Δ|log(τp/τq)| | ( _p/ _q)|). Appendix D.3 examines the impact of the support size k with frozen prefixes, reporting the exact covariance trace and NSR. In contrast, here we provide an end-to-end online ablation of OPD. Specifically, Tab. 6 and Fig. 5 show the results across teacher support sizes k∈1,4,16,32k∈\1,4,16,32\. As shown in Tab. 6, ResOPD consistently outperforms HETS across all tested support sizes in average downstream accuracy, with improvements ranging from +0.11+0.11 percentage points at k=1k=1 up to +1.56+1.56 points at k=32k=32. However, while downstream task scores consistently favor ResOPD, the intrinsic variance reduction benefit offered by ResOPD is expected to diminish as k increases, since a larger teacher support naturally captures more probability mass and renders tail corrections less frequent. Fig. 5 validates this mechanism on the runs from Tab. 6. In Fig. 5(a), HETS and ResOPD exhibit nearly identical variance factors PτpP _p, both dropping by over three orders of magnitude from k=1k=1 to 3232. This directly scales the theoretical ResOPD–HETS variance contrast alongside tail-mean approximation quality. Meanwhile, Fig. 5(b) shows that ResOPD consistently lowers tail log-mass mismatch across all supports, with the largest margin in tight-support regimes. 4.6 Rollout sampling: default filters versus full support While the idealized ancestral sampling condition A∼pθA p_θ is strictly enforced in both our exact enumeration audits (Sec. 4.2) and the cross-domain transfer experiments (Sec. 4.4), we also verify for completeness that the primary math results do not depend on rollout truncation. Specifically, we evaluate the first training epoch under full-support sampling (top-p=1.0top-p=1.0, top-k=−1top-k=-1) while keeping all other training hyperparameters identical. As shown in Tab. 7, removing the rollout filters lowers AUC for all three evaluated methods. Nevertheless, ResOPD consistently maintains its advantage over HETS and the naive baseline across both benchmark accuracy and trajectory AUC. In addition, we observe that removing rollout filters yields longer reasoning trajectories and a higher frequency of hitting the response token limit during training. 4.7 Learning-rate sensitivity Table 7: First-epoch Math performance under default vs. full-support rollouts. Benchmark scores and Avg. denote step-64 accuracy (%); AUC (%) measures the normalized learning curve area over steps 0,6,32,64\0,6,32,64\. Default: top-p=0.95p=0.95, top-k=20k=20; Full: top-p=1.0p=1.0, top-k=−1k=-1. Rollout Method DeepMath AIME26 HMMT26 OlymMATH Avg. AUC Default Naive Top-k rKL 53.17 56.04 35.42 35.00 44.91 45.45 HETS 53.17 59.17 36.93 34.50 45.94 46.12 ResOPD 54.76 60.00 37.88 35.75 47.10 47.57 Full Naive Top-k rKL 45.44 55.42 35.80 34.25 42.72 44.96 HETS 49.80 57.50 37.88 35.50 45.17 45.68 ResOPD 57.14 60.42 37.88 36.00 47.86 46.79 Experimental setup and evaluation protocol. To evaluate optimization robustness under aggressive step sizes, we conduct a controlled learning-rate sweep (k=4k=4) identical to the primary Math experimental setup, except that training is compressed to 1 epoch (64 optimizer steps). All runs share a batch size of 16, a 32K response budget, and identical sampling configurations with the primary experiments. Checkpoints at steps 0, 6, 32, and 64 are evaluated under the Macro-4 protocol (averaging 16 responses per problem for AIME26 and HMMT26, and 2 responses for DeepMath held-out and OlymMATH). Table 8: Learning-rate sensitivity of HETS vs. ResOPD. Δ denotes the improvement of ResOPD over HETS. Learning Rate Final Score (%) Trajectory AUC (%) HETS ResOPD Δ HETS ResOPD Δ 1×10−61× 10^-6 45.94 47.10 +1.16+1.16 46.12 47.57 +1.45+1.45 2×10−62× 10^-6 45.94 46.61 +0.67+0.67 45.69 46.17 +0.48+0.48 5×10−65× 10^-6 42.54 46.35 +3.81 43.88 47.77 +3.89 Performance resilience across learning rates. As shown in Tab. 8, ResOPD demonstrates substantially stronger resilience at elevated learning rates. When scaling the learning rate to 5×10−65× 10^-6, HETS suffers severe degradation, with its final score dropping from 45.94%45.94\% to 42.54%42.54\% (−3.40%-3.40\%) and its trajectory AUC declining by 2.24%2.24\%. In contrast, ResOPD retains consistent performance (46.35%46.35\% final score, 47.77%47.77\% AUC), widening its lead over HETS to +3.81%+3.81\% in final score and +3.89%+3.89\% in AUC. Examining the sub-benchmark breakdown at this aggressive rate, ResOPD’s advantage is consistently positive across all four constituent benchmarks at step 64: +3.96%+3.96\% on AIME26, +2.08%+2.08\% on HMMT26, +6.94%+6.94\% on DeepMath held-out, and +2.25%+2.25\% on OlymMATH. We provide detailed training dynamics across different learning rates in Appendix D.4. Stability boundaries and failure modes. When the learning rate is scaled further to 1×10−51× 10^-5, both methods diverge rapidly, triggering heavy gradient clipping (4/64/6 steps for HETS and 5/65/6 steps for ResOPD) and severe response-length collapse within six optimizer steps. While ResOPD does not shift the ultimate divergence boundary (1×10−51× 10^-5) under our default training recipe, its coarse-tail residualization provides a substantially wider stability margin in moderately aggressive regimes. Scaling beyond this threshold may require coordinating higher learning rates with complementary stabilization techniques (such as larger batch sizes or tighter clipping thresholds), which we leave to future work. 4.8 Comparing the two-control endpoints The analysis in Sec. 3.4 identifies a distinct opportunity to reduce within-tail variance through γ. We evaluate that prediction using SC-TA (γ=1γ=1) and ResOPD (γ=0γ=0), keeping the observable coarse control =B B= μ_B fixed. 4.8.1 Cross-distribution fixed-prefix audit While the two-control framework indicates that SC-TA (γ=1γ=1) can mitigate within-tail fluctuations beyond ResOPD (γ=0γ=0), Equation (29) implies this ordering is distribution-dependent rather than universal. To evaluate stability across domains, we freeze the Qwen3.5-4B/27B pair (Top-4 support, 64-token horizon) across three natural prompt distributions (DeepMath [10], MegaScience [7] and SynLogic [19]), evaluating 128 trajectories (8,192 prefixes) with 20,000 bootstrap resamples. We track the normalized trace variance reduction: Δ≔SC−ResOPDsampled, V_SC-V_ResOPDV_sampled, (30) where negative values favor SC-TA. As reported in Tab. 9, the aggregate trace variance consistently favors SC-TA over ResOPD across all three evaluated domains (Δ∈[−0.207,−0.126] ∈[-0.207,-0.126]). Simultaneously, prefix-level crossover remains non-trivial (up to 18.0%18.0\% of prefixes favor ResOPD in SynLogic). This empirically corroborates the qualified prediction of our two-control formulation: while setting γ=1γ=1 may yield lower variance in expectation, the local variance ordering remains strictly prefix-dependent rather than globally uniform. In principle, one could pre-select a subset of prompts from the training distribution to empirically calibrate an optimal coefficient γ⋆γ , potentially unlocking additional variance reduction during optimization (even though γ⋆γ inherently evolves alongside model weights). In practice, however, our experiments indicate that the incremental gains from such calibration over the fixed heuristic γ=1γ=1 are marginal. We therefore omit further discussion in favor of simplicity. Table 9: SC-TA vs. ResOPD covariance trace ordering across natural prompt distributions. Dataset SC/ResOPDV_SC/V_ResOPD Δ ResOPD-Lower Prefixes DeepMath 0.512 −0.126-0.126 06.8% MegaScience 0.532 −0.207-0.207 11.3% SynLogic 0.558 −0.161-0.161 18.0% Table 10: Matched evaluation curves across training updates (%). Evaluated on Qwen3.5-27B-to-4B distillation across AIME26/HMMT26 (avg@16) and held-out DeepMath/OlymMATH (avg@2). Method Base (Step 0) Step 6 Step 32 Step 64 Step 128 Step 192 HETS 42.85 48.86 45.04 45.94 46.60 45.72 ResOPD 42.85 48.23 47.95 47.10 47.71 50.12 SC-TA 42.85 47.66 46.69 47.26 48.27 46.33 4.8.2 Downstream evaluation comparison While SC-TA halves the fixed-prefix residual covariance trace of ResOPD, this advantage does not necessarily translate into superior downstream accuracy. As shown in Tab. 10, SC-TA does not exhibit a consistent downstream advantage over ResOPD: it is slightly stronger at several intermediate checkpoints, whereas ResOPD achieves the higher terminal score, with trajectory-level AUC remaining similar. This decoupling between gradient covariance trace and downstream performance is informative rather than contradictory. While lower estimator variance validates the two-control residualization mechanism locally, downstream benchmark accuracy reflects the global trajectory of policy updates. Consequently, while SC-TA exhibits lower overall variance across the evaluated audit sets, ResOPD retains strong aggregate performance in this evaluated setting without explicitly subtracting the additional within-tail score fluctuation. 5 Conclusion We introduced ResOPD, a principled estimator for full-vocabulary reverse-KL distillation under sparse teacher feedback. By integrating the observable coarse distribution analytically and sampling only the unresolved fine-tail residual, ResOPD reduces variance in the evaluated settings while preserving exact unbiasedness under ancestral on-policy sampling, without expanding the teacher payload or adding forward passes. Supported by theoretical analysis, exact covariance audits, and empirical evaluations across math, science, and coding tasks, ResOPD demonstrates higher aggregate downstream scores in the evaluated settings. References [1] R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Vol. 2024, p. 21246–21263. Cited by: §1, §1, §2. [2] W. U. Ahmad, S. Narenthiran, S. Majumdar, A. Ficek, S. Jain, J. Huang, V. Noroozi, and B. Ginsburg (2025) Opencodereasoning: advancing data distillation for competitive coding. arXiv preprint arXiv:2504.01943. Cited by: §4.4. [3] Anshumann, M. A. Zaidi, A. Kedia, J. Ahn, T. Kwon, K. Lee, H. Lee, and J. Lee (2025) Sparse logit sampling: accelerating knowledge distillation in LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 18085–18108. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1, §1, §2. [4] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §4.1. [5] J. Dekoninck, N. Jovanović, T. Gehrunger, K. Rögnvaldsson, I. Petrov, C. Sun, and M. Vechev (2026) Beyond benchmarks: matharena as an evaluation platform for mathematics with llms. External Links: 2605.00674, Link Cited by: §4.1, §4.1. [6] X. Du, Y. Yao, K. Ma, B. Wang, T. Zheng, M. Liu, Y. Liang, X. Jin, Z. Wei, C. Zheng, et al. (2026) SuperGPQA: scaling llm evaluation across 285 graduate disciplines. Advances in Neural Information Processing Systems 38. Cited by: §4.4. [7] R. Fan, Z. Wang, and P. Liu (2025) Megascience: pushing the frontiers of post-training datasets for science reasoning. arXiv preprint arXiv:2507.16812. Cited by: §4.8.1. [8] Y. Fu, H. Huang, K. Jiang, J. Liu, Z. Jiang, Y. Zhu, and D. Zhao (2026) Revisiting on-policy distillation: empirical failure modes and simple fixes. arXiv preprint arXiv:2603.25562. Cited by: §2, §4.1. [9] Y. Gu, L. Dong, F. Wei, and M. Huang (2024) Minillm: knowledge distillation of large language models. In International Conference on Learning Representations, Vol. 2024, p. 32694–32717. Cited by: §1, §1, §2, §2. [10] Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, Z. Zhang, R. Wang, Z. Tu, H. Mi, and D. Yu (2025) DeepMath-103k: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. External Links: 2504.11456, Link Cited by: §4.1, §4.8.1. [11] B. Heo, J. Hwang, S. Yun, and D. Han (2026) On-policy delta distillation. arXiv preprint arXiv:2607.15161. Cited by: §2, §4.4. [12] H. Huang and H. Wei (2026) Tail-aware top-k on-policy distillation. arXiv preprint arXiv:2608.14728. Cited by: §1, §1, §1, §2, §3.2, §3.4.2. [13] J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. K. Buening, C. Guestrin, et al. (2026) Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802. Cited by: §3.2. [14] N. Jain, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica (2025) Livecodebench: holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, Vol. 2025, p. 58791–58831. Cited by: §4.4. [15] I. Jang, J. Yeom, J. Yeo, H. Lim, and T. Kim (2026) Stable on-policy distillation through adaptive target reformulation. In Findings of the Association for Computational Linguistics: ACL 2026, p. 42217–42227. Cited by: §2. [16] W. Jin, T. Min, Y. Yang, D. Wei, Y. Zhou, S. R. Kadhe, N. Baracaldo, and K. Lee (2026) Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079. Cited by: §2. [17] Y. Kim and A. M. Rush (2016) Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras (Eds.), Austin, Texas, p. 1317–1327. External Links: Link, Document Cited by: §1. [18] J. Ko, S. Kim, T. Chen, and S. Yun (2024) Distillm: towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898. Cited by: §2. [19] J. Liu, Y. Fan, J. Zhuo, H. Ding, Y. Hu, C. Zhang, Y. Shi, S. Weng, A. Chen, S. Chen, et al. (2026) Synlogic: synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond. Advances in Neural Information Processing Systems 38, p. 100976–100997. Cited by: §4.8.1. [20] K. Lu and T. M. Lab (2025) On-policy distillation. Thinking Machines Lab: Connectionism. Note: https://thinkingmachines.ai/blog/on-policy-distillation External Links: Document Cited by: §1, §1, §2. [21] I. Moshkov, D. Hanley, I. Sorokin, S. Toshniwal, C. Henkel, B. Schifferer, W. Du, and I. Gitman (2025) Aimo-2 winning solution: building state-of-the-art mathematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.16891. Cited by: §4.4. [22] NVIDIA Corporation (2025) OpenScienceReasoning-2. Note: Hugging Face dataset External Links: Link Cited by: §4.4. [23] M. Oh, S. Song, G. Choi, Y. Choi, and Y. Jo (2026) KL for a kl: on-policy distillation with control variate baseline. arXiv preprint arXiv:2605.07865. Cited by: §1, §2. [24] Qwen Team (2026) Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §4.1. [25] D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman (2023) Gpqa: a graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022. Cited by: §4.4. [26] H. Sun, Y. Min, Z. Chen, W. X. Zhao, Z. Liu, Z. Wang, L. Fang, and J. Wen (2025) Challenging the boundaries of reasoning: an olympiad-level math benchmark for large language models. External Links: 2503.21380, Link Cited by: §4.1. [27] A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. (2026) Deepseek-v4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: §1. [28] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.4. [29] A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. (2026) Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763. Cited by: §1. [30] L. Zhang and J. Ba (2026) Ema policy gradient: taming reinforcement learning for llms with ema anchor and top-k kl. arXiv preprint arXiv:2602.04417. Cited by: §1, §1, §2, §3.1. [31] Y. Zhang and T. Math-AI (2026) American invitational mathematics examination (aime) 2026. Cited by: §4.1. [32] S. Zhu, X. Ye, H. Lu, W. Shi, and G. Liu (2026) The many faces of on-policy distillation: pitfalls, mechanisms, and fixes. arXiv preprint arXiv:2605.11182. Cited by: §1, §2, §2, §3.1. Appendix A Detailed estimator derivations The main text gives the estimator, its unbiasedness identity, and the unified variance analysis. This appendix supplies the detailed Top-k bias derivations and the single-control oracle calculation, including the boundary-term cancellation for the complete coarse distribution. A.1 Three biased forms of Teacher Top-k reverse KL This subsection distinguishes three operations that are often conflated: renormalizing both distributions on the head, truncating the scalar reverse-KL objective, and truncating only the stopped-coefficient gradient. Each is computable from Teacher Top-k, but none generally estimates correctly the gradient of full-vocabulary reverse KL. Here “bias” is relative to that target gradient, not Monte Carlo sampling bias. Fix the support S and Teacher probabilities, let z be unrestricted Student logits, and define R≔∑ipiri=DKL(p∥q),RS≔∑i∈Spiri,P≔∑i∈Spi,Q≔∑i∈Sqi. gatheredR _ip_ir_i=D_KL(p\|q), R_S _i∈ Sp_ir_i,\\ P _i∈ Sp_i, Q _i∈ Sq_i. gathered (31) The exact full-rKL logit gradient, which provides the reference below, is [∇zR]j=pj(rj−R).[ _zR]_j=p_j(r_j-R). (32) Renormalized Top-k reverse KL. Define p~i=pi/P p_i=p_i/P, q~i=qi/Q q_i=q_i/Q and r~i=log(p~i/q~i) r_i= ( p_i/ q_i) for i∈Si∈ S, with R~=∑i∈Sp~ir~i R= _i∈ S p_i r_i. Since p~i=exp(zi)/∑ℓ∈Sexp(zℓ) p_i= (z_i)/ _ ∈ S (z_ ), the objective and its logit gradient are Lrenormalized L_renormalized =∑i∈Sp~ilogp~iq~i, = _i∈ S p_i p_i q_i, (33) [∇zLrenormalized]j [ _zL_renormalized]_j =j∈Sp~j(r~j−R~). =I\j∈ S\ p_j( r_j- R). Every tail-logit component is exactly zero, whereas Equation (32) is generally nonzero there. The global minimum is the set p~i=q~i(i∈S)⟺pi=PQqi(i∈S), p_i= q_i (i∈ S) p_i= PQq_i (i∈ S), (34) with arbitrary P and arbitrary tail shape. Renormalization therefore learns only the conditional head distribution; it identifies neither head mass nor the omitted conditional distribution. Scalar-truncated Top-k reverse KL. The unnormalized scalar shortcut is Ltruncated=RS=∑i∈SpiriL_truncated=R_S= _i∈ Sp_ir_i. Writing CS≔∑i∈Spi(ri+1)=RS+PC_S _i∈ Sp_i(r_i+1)=R_S+P, direct differentiation gives ∇θLtruncated _θL_truncated =+∇θP, = h+ _θP, (35) [∇zLtruncated]j [ _zL_truncated]_j =j∈Spj(rj+1)−pjCS. =I\j∈ S\p_j(r_j+1)-p_jC_S. Thus its bias relative to full rKL is ∇θP−τptail _θP- _p μ_tail. In particular, for j∉Sj∉ S, [∇zLtr]j=−pjCS[ _zL_tr]_j=-p_jC_S; when CS>0C_S>0, gradient descent increases every omitted logit. The corresponding global optimum makes the failure more explicit. For each head coordinate, xlogxqi≥−qie,with equality iff x=qie.x xq_i≥- q_ie, equality iff x= q_ie. (36) Because ∑i∈Sqi/e=Q/e<1 _i∈ Sq_i/e=Q/e<1, all coordinatewise minima are jointly feasible, and therefore pi⋆=qie(i∈S),P⋆=Qe,∑j∉Spj⋆=1−Qe,p_i = q_ie (i∈ S), P = Qe, _j∉ Sp_j =1- Qe, (37) with arbitrary allocation inside the tail. Even if Q is near one, this objective prefers roughly 1−1/e1-1/e Student mass outside the Teacher head; the tail-push is thus an objective-level property, not merely a transient gradient sign. Stopped-coefficient Top-k reverse-KL gradient. Stopping the coefficients removes the ∇P∇ P boundary term, yet leaves the tail expectation uncorrected: Top-k≔∑i∈Sstopgrad(piri)i=,[Top-k]j=j∈Spjrj−pjRS,Top-k−=−τptail. gathered g_Top-k _i∈ Sstopgrad(p_ir_i) s_i= h,\\ [ g_Top-k]_j=I\j∈ S\p_jr_j-p_jR_S, g_Top-k- g=- _p μ_tail. gathered (38) In unrestricted categorical-logit space with a non-empty tail, vanishing gradients (Top-k= g_Top-k=0) force RS=0R_S=0 via the tail components (j∉Sj∉ S), which subsequently implies ri=0r_i=0 for all head components (i∈Si∈ S). Consequently, pi=qip_i=q_i on the head, correctly matching the aggregate tail mass (1−P=1−Q1-P=1-Q); however, the conditional distribution over the tail remains unconstrained and arbitrary. While this stopped-gradient formulation alters the underlying target the least among the three heuristics, it remains inherently biased whenever the tail vector is non-zero. HETS and ResOPD resolve this residual bias by introducing a sampled tail correction to restore the missing expectation. A.2 Single-control residualization and its variance optimum Use the head/tail notation of Equation (5), let J=A∉SJ=I\A∉ S\, and let B be any finite prefix-measurable vector stopped before sampling A∼pA p. Consider the one-control estimator ^()≔+τp+J(A−). g( B) h+ _p B+J( G_A- B). (39) It is unbiased for every such B: [^()]=+τp+τp(tail−)=.E[ g( B)]= h+ _p B+ _p( μ_tail- B)= g. (40) Conditioned on J=0J=0, its value is the deterministic vector +τp h+ _p B. Conditioned on J=1J=1, its mean is +τp+tail− h+ _p B+ μ_tail- B and its covariance is Covtail(A)Cov_tail( G_A). The two conditional means differ by tail− μ_tail- B. Hence the law of total covariance gives (^())=τptail(A)+Pτp∥tail−∥22.V( g( B))= _pV_tail( G_A)+P _p μ_tail- B _2^2. (41) The first term is independent of B. Therefore, whenever 0<P<10<P<1, the unique trace-covariance minimizer and its exact regret are ⋆ B =tail, = μ_tail, (42) (^())−(^(tail)) ( g( B))-V( g( μ_tail)) =Pτp∥−tail∥22. =P _p B- μ_tail _2^2. The oracle removes all head-versus-tail event variance, but it is unavailable under a sparse Teacher interface because it requires the full conditional fine-tail mean. HETS and ResOPD are the two payload-compatible choices ^HETS=^(),^ResOPD=^(B). g^HETS= g( 0), g^ResOPD= g( μ_B). (43) Their regrets relative to the single-control oracle are respectively Pτp∥tail∥22P _p μ_tail _2^2 and Pτp∥tail−B∥22P _p μ_tail- μ_B _2^2. Subtraction yields (^HETS)−(^ResOPD)=Pτp(2⟨tail,B⟩−∥B∥22).V( g^HETS)-V( g^ResOPD)=P _p\! (2 μ_tail, μ_B - μ_B _2^2 ). (44) Thus ResOPD has lower trace covariance exactly when B μ_B is closer to tail μ_tail than the HETS control 0 is; no unconditional ordering is asserted. It remains to verify why B μ_B is available without the fine Teacher tail. The reverse KL of the complete coarse distribution over the k head tokens and one tail bucket is Lcoarse≔∑i∈Spilogpiqi+τplogτpτq.L_coarse _i∈ Sp_i p_iq_i+ _p _p _q. (45) Since ∇τp=τp¯tail=−∇P∇ _p= _p s_tail=-∇ P, direct differentiation gives ∇Lcoarse ∇ L_coarse =+∇P+(b+1)∇τp = h+∇ P+(b+1)∇ _p (46) =+b∇τp=+τpB=coarse. = h+b∇ _p= h+ _p μ_B= g_coarse. The boundary terms cancel because the tail pseudo-token completes the coarse support. This is why coarse g_coarse is an exact gradient (but biased), even though B μ_B need not equal the true conditional fine-tail mean. Appendix B Detailed training and evaluation configurations Tabs. 11 and 12 collect the complete settings for the primary mathematics and cross-domain runs. Their sampling assumptions are discussed in Secs. 3.3 and 4.1; the matched full-support mathematics ablation is reported in Sec. 4.6. Table 11: Primary Math OPD experimental configurations and hyperparameters. Category Hyperparameter Configuration / Value Models Student model Qwen3.5-4B Teacher model Qwen3.5-27B Data Training dataset 1,024 prompts sampled from DeepMath-103K Max prompt length 2,048 Sampling Enable thinking True Max response length 32,768 (training and evaluation) temperature 1.0 top-p 0.95 top-k 20 Training Training epochs 3 Global batch size 16 Optimizer AdamW Gradient clipping 1.0 Learning rate 1×10−61× 10^-6 System & Hardware Compute infrastructure 16×16× NVIDIA H200 GPUs Resource allocation 8 GPUs for Student rollout, 4 for Student training, 4 for hosting the Teacher inference service Software frameworks Training: verl Inference: vLLM Appendix C Step-wise evaluation results C.1 Math OPD experiments Table 12: Cross-domain Qwen3 OPD experimental configurations and hyperparameters. Category Hyperparameter Configuration / Value Models Student model Qwen3-4B Teacher model Qwen3-30B-A3B-Instruct-2507 Data Training dataset 30,000 deduplicated prompts; Math/Code/Science: 10,000 each Max prompt length 1,024 Sampling Enable thinking False Max response length 8192 (training) and 32768 (evaluation) temperature 1.0 top-p 1.0 top-k −1-1 Training Training steps 100 Global batch size 288 Optimizer AdamW Gradient clipping 1.0 Peak learning rate 5×10−65× 10^-6 LR schedule 10-step warmup; cosine decay to 0.1×0.1× peak System & Hardware Compute infrastructure 16×16× NVIDIA H200 GPUs Resource allocation 8 GPUs for student rollout, 4 for student training, 4 for hosting the teacher inference service Software frameworks Training: verl Inference: vLLM Fig. 6 illustrates the training trajectories across evaluation suites. On the DeepMath held-out set, ResOPD displays consistent late-stage gains, achieving the highest terminal accuracy among all estimators. While competitive baselines such as TA-OPD and Teacher Top-k vOPD show steady improvements, they fall short of ResOPD at the terminal checkpoint. Meanwhile, Sampled-token OPD deteriorates severely after an early peak, and HETS suffers a late-stage drop. A similar trend holds on the external math competitions: although TA-OPD maintains strong intermediate trajectory performance and vOPD remains competitive, ResOPD exhibits superior late-stage resilience and decisively surpasses all alternatives at the final step, avoiding the post-peak degradation observed in estimators like Sampled-token OPD and HETS. Together with the lower fixed-prefix covariance trace and improved online training diagnostics reported elsewhere in the paper, these trajectories are consistent with the proposed coarse-tail residualization reducing stochastic gradient noise and improving late-stage optimization stability. C.2 Cross-Domain OPD Experiments Figure 6: Step-wise learning curves corresponding to Tab. 3 on DeepMath held-out (left) and external competitions (right) across 192 optimizer steps. Fig. 7 illustrates the checkpoint evaluation trajectories along the fixed training progression (steps 0, 10, 25, 50, and 100) across all four benchmarks. On code benchmarks (LiveCodeBench v5 and v6), ResOPD and HETS rapidly converge and consistently outperform Sampled-token OPD by a substantial margin throughout training, with ResOPD achieving the highest overall trajectory AUC. In contrast, science benchmarks display mixed domain dynamics: Sampled-token OPD demonstrates superior retention and final accuracy on GPQA-D, whereas ResOPD remains competitive across training steps and achieves the best terminal performance on SuperGPQA. These trajectories corroborate that while Sampled-token OPD remains strong in specialized scientific reasoning, ResOPD provides transfer performance on code tasks without sacrificing overall multi-domain balance. Figure 7: Evaluation trajectories during cross-domain transfer distillation from Qwen3-30B-A3B-Instruct to Qwen3-4B across optimizer steps (0, 10, 25, 50, and 100). Appendix D Details of diagnostics D.1 What the online residual moments measure At optimizer update u, let i∈1,…,Bui∈\1,…,B_u\ index the non-empty sampled responses and t∈1,…,Tuit∈\1,…,T_ui\ index their valid tokens. We define the scalar support residual and its components as Puit≔pθ(Suit∣xuit),Huit≔Auit∈Suit,Zuit≔Huit−Puit.P_uit p_θ(S_uit x_uit), H_uit \A_uit∈ S_uit\, Z_uit H_uit-P_uit. (47) Here, PuitP_uit denotes the Student probability mass allocated to the transmitted Teacher Top-k support SuitS_uit, while HuitH_uit indicates whether the sampled Student token lands within this set. Under an exact on-policy draw, the residual is conditionally zero-mean, [Zuit∣xuit]=0E[Z_uit x_uit]=0, with conditional variance [Zuit2∣xuit]=Puit(1−Puit)E[Z_uit^2 x_uit]=P_uit(1-P_uit). More generally, its uncentered second moment captures both intrinsic sampling variability and any realized calibration drift. Fig. 3 reports two diagnostic aggregations of this quantity. Panel C displays the masked token-level second moment: M2,tok(u)=1∑i=1BuTui∑i=1Bu∑t=1TuiZuit2,M_2,tok^(u)= 1 _i=1^B_uT_ui _i=1^B_u _t=1^T_uiZ_uit^2, (48) whereas Panel D evaluates the trajectory-level variance by first averaging within each response and then weighting responses uniformly: Z¯ui=1Tui∑t=1TuiZuit,M2,traj(u)=1Bu∑i=1BuZ¯ui2. Z_ui= 1T_ui _t=1^T_uiZ_uit, M_2,traj^(u)= 1B_u _i=1^B_u Z_ui^2. (49) Intuitively, M2,tokM_2,tok captures the instantaneous magnitude of the unresolved head-versus-tail events (which is leveraged in Fig. 5 to approximate the between-event variance factor PτpP _p), whereas M2,trajM_2,traj measures the persistent, signed event imbalance surviving along sequence rollouts. Defining the batch mean Z¯u=Bu−1∑i=1BuZ¯ui Z_u=B_u^-1 _i=1^B_u Z_ui, the empirical standard error of the trajectory mean is given by SE^(Z¯u)=maxM2,traj(u)−Z¯u2,0Bu−1. SE( Z_u)= \M_2,traj^(u)- Z_u^2,0\B_u-1. (50) The connection to ResOPD is exact at the scalar-coefficient level. Noting that tail indicators and tail masses satisfy Juit=1−HuitJ_uit=1-H_uit and τp,uit=1−Puit _p,uit=1-P_uit, we obtain Zuit=τp,uit−JuitZ_uit= _p,uit-J_uit. Substituting this into Equation (10) yields ^uitResOPD−^uitHETS=ZuitB,uit. g_uit^ResOPD- g_uit^HETS=Z_uit μ_B,uit. (51) Hence, the two M2M_2 metrics directly track the magnitude and sequence-level persistence of the random scalar modulating ResOPD’s bucket control. Because they intentionally abstract away ∥B,uit∥ μ_B,uit , its alignment with the fine-tail mean, and the orthogonal within-tail fluctuations, they serve as computationally lightweight, mechanism-aligned online diagnostics rather than full covariance estimators or replacements for fixed-prefix NSR. Finally, while Z can trivially be recorded for a sparse Top-k objective, direct Top-k rKL lacks a sampled tail correction and remains inherently biased with respect to the true rKL target. Its zero action-sampling variance is therefore not comparable to the unbiased vOPD, HETS and ResOPD estimators in Panels C and D. D.2 Finite gradient-enumeration check We also verify the unbiasedness identities without Monte Carlo error on a four-action softmax. The Student and Teacher probabilities are respectively (0.35,0.30,0.20,0.15)(0.35,0.30,0.20,0.15) and (0.60,0.25,0.10,0.05)(0.60,0.25,0.10,0.05), and the first two actions form the Teacher Top-2 support. We differentiate the full reverse KL directly, enumerate each possible Student action, and weight its estimator gradient by the exact Student action probability. The resulting HETS expectation matches the dense full-rKL gradient to maximum absolute error 5.6×10−175.6× 10^-17. ResOPD changes the per-action vector by (τp−J)B( _p-J) μ_B relative to HETS. Because the same enumeration has [J]=τpE[J]= _p, this correction sums exactly to zero and ResOPD recovers the same dense gradient. This finite check complements the symbolic identity in Equation (10): it validates the stopped-coefficient and support-mask implementation semantics, but it is not empirical evidence about covariance or downstream accuracy. D.3 Complete frozen-prefix support diagnostics Table 13: Support-size (k) sweep under fixed Teacher Top-k payloads. “Var. Removed” measures the empirical covariance-trace reduction relative to sampled-token OPD. Payload accounts for k Top-k entries plus the sampled token (INT32 ID and FP32 log-probability each). Student mass and Head Hit denote the predicted and empirical probabilities of landing in the Top-k support, respectively. Best variance reduction and NSR per setting are in bold. Dataset Student/Teacher k Payload (B/token) Student mass Head Hit Var. Removed (%) ↑ NSR ↓ HETS ResOPD HETS ResOPD GSM8K 0.8B/4B 1 16 75.0%75.0\% 74.8%74.8\% −0.5-0.5 20.520.5 5.446 4.3104.310 4 40 93.1%93.1\% 92.9%92.9\% 27.3 52.552.5 3.940 2.5732.573 16 136 97.7%97.7\% 97.9%97.9\% 62.6 75.375.3 2.027 1.3371.337 64 520 99.3%99.3\% 99.7%99.7\% 85.1 89.589.5 0.807 0.5700.570 HMMT26 4B/27B 1 16 89.1%89.1\% 89.3%89.3\% −1.0-1.0 32.032.0 6.656 4.4794.479 4 40 99.1%99.1\% 98.8%98.8\% 55.8 75.375.3 2.914 1.6301.630 16 136 99.9%99.9\% 100.0%100.0\% 89.8 94.294.2 0.675 0.3820.382 Figure 8: Comprehensive frozen-prefix support-size (k) diagnostics across benchmarks. ResOPD consistently reduces the dimensionless NSR across all evaluated support sizes on GSM8K and HMMT (log scale). Axis annotations detail the transmitted Teacher support size k, the corresponding communication payload (bytes/token), and the empirical head-hit probability. Error bars denote 95% trajectory-level bootstrap confidence intervals. Across both suites in Tab. 13, ResOPD consistently outperforms HETS at every matched support size, achieving higher covariance-trace elimination and lower NSR. Expanding k from 44 to 1616 and 6464 scales transmission costs by 3.4×3.4× and 13×13× (40→136→52040→ 136→ 520 B/token) for diminishing marginal variance gains. Crucially, the head hit rates demonstrate why high support coverage alone is insufficient: even when head mass reaches 98.8%98.8\% (k=4k=4 on HMMT26), the rare 1.2%1.2\% tail draws still induce a substantial HETS–ResOPD variance gap (55.8%55.8\% vs. 75.3%75.3\% removed). Moreover, at k=1k=1 where HETS exhibits negative reduction (−1.0%-1.0\%), ResOPD reliably eliminates 32.0%32.0\% of the gradient variance. This firmly corroborates the theoretical condition in Equation (44), confirming that uncorrected tail draws dominate residual gradient noise despite their low nominal probability mass. D.4 Training dynamics across learning rates Underlying training dynamics. The training trajectories in Fig. 9 reveal two clear mechanisms behind this divergence. First, in terms of gradient magnitude, ResOPD maintains substantially lower pre-clipping gradient norms at 5×10−65× 10^-6 (mean 0.430.43 vs. 0.480.48; p99 0.970.97 vs. 1.841.84) and completely avoids gradient clipping (0/640/64 steps vs. 2/642/64 steps for HETS). Second, regarding residual stabilization, Panels (B)–(D) show that ResOPD consistently curbs the support-mass gap (Q−PQ-P) and suppresses both token- and trajectory-level residual second moments (M2M_2), consistent with the more stable learning trajectory observed at this step size. Figure 9: Training dynamics and learning-rate sensitivity of HETS (orange) vs. ResOPD (blue) on DeepMath across three learning rates (1×10−61× 10^-6 solid, 2×10−62× 10^-6 dashed, 5×10−65× 10^-6 dotted). (A) Pre-clipping global gradient norm (inset displays full-scale unclipped peaks over early steps). (B) Support-mass gap (Q−PQ-P) on the transmitted Teacher Top-4 tokens. (C, D) Token-level and trajectory-level second residual moments (M2M_2). While HETS exhibits large gradient-norm excursions and growing mass/residual discrepancies as the learning rate scales up to 5×10−65× 10^-6, ResOPD maintains stable dynamics and stays close to its low-rate trajectories throughout optimization.