Paper deep dive
SR-OPSD: Self-Referenced On-Policy Self-Distillation
Zhuo Sun, Entong Li, Yanlong Zhao, Xiaoyuan Cheng, Wenxuan Yuan, Kaiyu Li, Che Liu, Huihang Liu, Harrison Bo Hua Zhu, Li Zeng
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards. However, the self-teacher policy used in OPSD is typically a stop-gradient or exponential-moving-average copy of the policy conditioned on additional context information, and thus co-evolves with both the student policy and its on-policy context distribution. Directly matching such a moving target with a fixed projection objective can lead to unstable optimization or excessive distributional concentration. This nature of OPSD motivates the proposed \emph{Self-Referenced On-Policy Self-Distillation (SR-OPSD)}. At fixed student-generated contexts, a token-level variational characterization identifies the effective distillation target as a geometric interpolation between the self-teacher policy and a reference policy. Meanwhile, we use the Rényi divergence family to generalize the projection geometry. This formulation separates \emph{where} the adaptive target is placed from \emph{how} the student is projected toward it: the interpolation coefficient controls underlying target, while the Rényi order controls the projection geometry and its sensitivity to token-level density ratios. Extensive experiments across scientific evaluation, mathematical reasoning, and coding generation tasks with multiple large language models show that SR-OPSD achieves the state-of-the-art or competitive performance across various settings.
Tags
Links
- Source: https://arxiv.org/abs/2608.09745v1
- Canonical: https://arxiv.org/abs/2608.09745v1
Trouble viewing inline? Open PDF directly →
Full Text
78,886 characters extracted from source content.
Expand or collapse full text
SR-OPSD: Self-Referenced On-Policy Self-Distillation Zhuo Sun1,2 Entong Li3,∗ Yanlong Zhao4,∗ Xiaoyuan Cheng5,∗ Wenxuan Yuan6 Kaiyu Li5 Che Liu2 Huihang Liu1 Harrison Bo Hua Zhu7,2,8,† Li Zeng9,† 1Shanghai University of Finance and Economics, 2Imperial College London, 3Independent Researcher, 4University of Science and Technology of China, 5University College London, 6Nanyang Technological University, 7Technical University of Denmark, 8University of Copenhagen, 9Peking University Equal Contribution.Corresponding Author. Correspondence to zhuosunreid@outlook.com, habhuz@dtu.dk, li.zeng@pku.edu.cn. Abstract On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards. However, the self-teacher policy used in OPSD is typically a stop-gradient or exponential-moving-average copy of the policy conditioned on additional context information, and thus co-evolves with both the student policy and its on-policy context distribution. Directly matching such a moving target with a fixed projection objective can lead to unstable optimization or excessive distributional concentration. This nature of OPSD motivates the proposed Self-Referenced On-Policy Self-Distillation (SR-OPSD). At fixed student-generated contexts, a token-level variational characterization identifies the effective distillation target as a geometric interpolation between the self-teacher policy and a reference policy. Meanwhile, we use the Rényi divergence family to generalize the projection geometry. This formulation separates where the adaptive target is placed from how the student is projected toward it: the interpolation coefficient controls underlying target, while the Rényi order controls the projection geometry and its sensitivity to token-level density ratios. Extensive experiments across scientific evaluation, mathematical reasoning, and coding generation tasks with multiple large language models show that SR-OPSD achieves the state-of-the-art or competitive performance across various settings. 1 Introduction Post-training of large language models (LLMs) encompasses supervised instruction tuning, preference optimization, and reinforcement learning (Wei et al., 2022; Sanh et al., 2022; Ouyang et al., 2022; Rafailov et al., 2023). Among reinforcement-learning methods, RLHF methods commonly optimize a learned reward model using proximal policy optimization (PPO) (Schulman et al., 2017; Ouyang et al., 2022), whereas recent reasoning systems increasingly employ reinforcement learning with verifiable rewards (RLVR), using algorithms such as Group Relative Policy Optimization (GRPO) (Shao et al., 2024) and its variants (Yu et al., 2025). Methods such as Direct Preference Optimization (DPO) (Rafailov et al., 2023) optimize preference data without explicit reward modeling. In terms of reasoning, RLVR is particularly attractive because the correctness of the outcome can often be assessed automatically through external checks, unit tests, or rule-based verifiers. However, these methods typically assigns a scalar-valued reward to an entire response. Although this reward induces policy-gradient updates over the generated tokens, it does not directly identify which intermediate tokens or reasoning steps caused success or failure, resulting in coarse and sparse credit assignment over long sequences. On-policy distillation (OPD) offers an alternative approach by utilizing the dense token-level supervision on trajectories sampled from the student policy (Gu et al., 2023; Agarwal et al., 2024; Zhu et al., 2026; Jin et al., 2026). A teacher policy evaluates student-generated prefixes, reducing the state-distribution mismatch associated with offline distillation. Existing on-policy distillation methods can be viewed through three interacting design choices including: the divergence objective, the source of supervision, and the mechanism used to stabilize policy optimization. These choices are often designed largely in isolation, which is relatively benign in conventional offline distillation where both the teacher and the training distribution are fixed. This benefit, however, generally requires continued access to an extra teacher policy. On-policy self-distillation (OPSD) instead uses two views of the same evolving model: the policy we aim to optimize is conditioned on the original prompt, whereas the so-called self-teacher policy is the same policy conditioned on feedback, privileged information, or richer contextual signals (Zhao et al., 2026a; Hübotter et al., 2026). OPSD can therefore convert feedback, reflections, verified traces, or successful responses into dense token-level supervision. Figure 1: Illustration of SR-OPSD. Motivation However, in on-policy self-distillation, the policy to be optimized generates its own contexts while a self-teacher policy evolves together with it. Consequently, the target, the context distribution, and the policy being optimized all change throughout training, making these three design choices intrinsically coupled. It is sensitive to the construction of the self-teacher policy, the choice of divergence, the extra context information provided to the self-teacher, and top-K approximation as discussed in (Zhu et al., 2026). For instance, self-distillation with reverse Kullback–Leibler (KL) divergence may amplify mode-seeking behavior and reduce output diversity (Jin et al., 2026); as demonstrated in Figure˜2 and in Appendix B. This coupling is already evident in the OPSD methods based on the two predominant KL objectives. Forward KL (Zhao et al., 2026a) is mode-covering and takes its action expectation under the self-teacher distribution, whereas reverse KL (Zhao et al., 2026a; Hübotter et al., 2026) is naturally aligned with student sampling but is generally more mode-seeking. Hence, in on-policy self-distillation, forward KL may suffer from an action-level expectation mismatch when estimated from student-sampled tokens, while reverse KL can weakly correct modes already underrepresented by the student, potentially reinforcing such biases as the self-teacher co-evolves. Existing approaches mitigate parts of these issues through exponential-moving-average (EMA) self-teachers, symmetric divergences (Hübotter et al., 2026), entropy-aware switching (Jin et al., 2026), and regularization-based stabilization (Luo et al., 2026; Yang et al., 2026; Yu et al., 2026). These motivate an on-policy self-distillation learning objective beyond vanilla forward-KL, reverse-KL. We instead seek a formulation that separately controls where the adaptive target is placed and how the student is projected toward it. We therefore propose Self-Referenced On-Policy Self-Distillation (SR-OPSD), illustrated in Figure˜1, which offers a new framework with Rényi projection and reference-policy anchoring for on-policy self-distillation. Main Contributions Our main contributions are threefold. (i) We adapt the reference-regularized on-policy self-distillation formulation to frozen-context self-distillation and construct a normalized target from a self-teacher policy and a reference policy. (i) We optimize this target using Rényi divergence and derive the resulting logit gradient, showing that Rényi order controls the power weighting of self-teacher-to-student probability ratios without changing the unconstrained target distribution. (i) We evaluate the resulting method across scientific reasoning, mathematical reasoning, and code generation, and open-source the code which supports multiple hardware platforms, including NVIDIA GPUs. 2 Background Notation We denote the policy to be optimized by πθ _θ, the self-teacher policy by πθ¯ _ θ, and the reference policy by πref _ref. Given a prompt x∼x and an on-policy response y=(y1,…,yL)y=(y_1,…,y_L), the student prefix at position t is ht=(x,y<t)h_t=(x,y_<t). We write htsfh_t^sf for the corresponding self-feedback or privileged context supplied to the self-teacher and htrefh_t^ref for the context supplied to the reference policy. When a statement is token-level and the contexts are fixed, we suppress the position index and write h, hsfh_sf, and hrefh_ref. Thus, h, hsfh_sf, and hrefh_ref always denote three possibly different fixed contexts. All token-level distributions are defined over the same next-token vocabulary V. If a candidate set t⊆C_t is used, all compared distributions are normalized on the same candidate space, preferably with a tail bucket retaining the omitted probability mass. The operator sg[⋅]sg[·] denotes stop-gradient. Rényi Divergence For probability distributions p and q with common positive support, the Rényi divergence (Van Erven and Harremos, 2014) of order ρ>0ρ>0, ρ≠1ρ≠ 1, is Dρ(p∥q)=1ρ−1log∑a∈p(a)ρq(a)1−ρ.D_ρ(p\|q)= 1ρ-1 _a p(a)^ρq(a)^1-ρ. The divergence direction is determined solely by the argument order. We call Dρ(q∥p)D_ρ(q\|p) the reverse projection when q is the student and p is the self-teacher target, and Dρ(p∥q)D_ρ(p\|q) the forward projection. Both converge to the corresponding KL as ρ→1ρ→ 1. On-Policy Self-Distillation Given a teacher policy πT _T, on-policy distillation trains the student on its own sampled prefixes, for example through the reverse-KL loss x∼y∼πθold(⋅∣x)[∑t=1LDKL(πθ(⋅∣ht)∥sg[πT(⋅∣ht)])].E_ subarraycx \\ y _ _old(· x) subarray [ _t=1^LD_KL\! ( _θ(· h_t)\, \|\,sg\! [ _T(· h_t) ] ) ]. Here πθold _ _old denotes the rollout policy used to collect the current on-policy batch; it coincides with the current student before the optimization step. In on-policy self-distillation, the teacher policy is often replaced by an exponential-moving-average of πθ _θ with additional context information, sg[πθ¯(⋅∣htsf)]sg[ _ θ(· h_t^sf)]. The additional context htsfh_t^sf can contain textual feedback, verified solutions, reflections, or successful sibling responses that are unavailable in the ordinary student context hth_t. Self-Teacher in OPSD The self-teacher policy πθ¯ _ θ may be a stopped copy, a lagged snapshot, or an exponential-moving-average version of the policy to be optimized πθ _θ. We use β∈[0,1)β∈[0,1) as the exponential moving average coefficient and adopt the convention θ¯k+1=βθ¯k+(1−β)θk+1. θ_k+1=β θ_k+(1-β) _k+1. Thus, larger β produces a slower-moving self-teacher policy. During each update, θ¯k θ_k and the resulting self-teacher policy are held fixed (Hübotter et al., 2026). 3 Method Revisiting On-Policy Self-Distillation At each outer iteration, OPSD first samples a batch from the rollout policy πθold _ _old and then treats the resulting contexts as fixed while updating the student. A standard reverse-KL frozen-rollout objective is maxθx∼y∼πθold(⋅∣x)[∑t=1L−DKL(πθ(⋅∣ht)∥sg[πθ¯(⋅∣htsf)])]. _θ\;E_ subarraycx \\ y _ _old(· x) subarray [ _t=1^L-D_KL\! ( _θ(· h_t)\, \|\,sg\! [ _ θ(· h_t^sf) ] ) ]. The distinction between the sequence-level contexts (ht,htsf,htref)(h_t,h_t^sf,h_t^ref) and the fixed-context shorthand (h,hsf,href)(h,h_sf,h_ref) is important: the following characterization is pointwise in fixed contexts and therefore applies directly to the frozen-rollout inner update. Proposition 3.1 (Token-Level Variational Characterization of OPSD). Fix h, hsfh_sf, and hrefh_ref, and let α∈[0,1]α∈[0,1]. Assume that, for every token a∈a , πθ(a∣h)>0,sg[πθ¯(a∣hsf)]>0,πref(a∣href)>0 _θ(a h)>0, \! [ _ θ(a h_sf) ]>0, _ref(a h_ref)>0. Define Zα(hsf,href):=∑a∈sg[πθ¯(a∣hsf)]απref(a∣href)1−αZ_α(h_sf,h_ref):= _a sg\! [ _ θ(a h_sf) ]^α _ref(a h_ref)^1-α and πα⋆(a∣hsf,href):=sg[πθ¯(a∣hsf)]απref(a∣href)1−αZα(hsf,href) _α (a h_sf,h_ref):= sg\! [ _ θ(a h_sf) ]^α _ref(a h_ref)^1-αZ_α(h_sf,h_ref). Then the conditional token-level objective α(θ;h,hsf,href):=a∼πθ(⋅∣h)[αlogsg[πθ¯(a∣hsf)]πref(a∣href)]−DKL(πθ(⋅∣h)∥πref(⋅∣href)) _α(θ;h,h_sf,h_ref):=E_a _θ(· h) [α sg\! [ _ θ(a h_sf) ] _ref(a h_ref) ]-D_KL\! ( _θ(· h)\, \|\, _ref(· h_ref) ) satisfies α(θ;h,hsf,href)=−DKL(πθ(⋅∣h)∥πα⋆(⋅∣hsf,href))+logZα(hsf,href)J_α(θ;h,h_sf,h_ref)=-D_KL\! ( _θ(· h)\, \|\, _α (· h_sf,h_ref) )+ Z_α(h_sf,h_ref). Consequently, for fixed contexts and frozen target components, maximizing αJ_α with respect to θ is equivalent to minimizing the reverse KL from the student to πα⋆ _α . When α=1α=1, Z1=1Z_1=1 and π1⋆(⋅∣hsf,href)=sg[πθ¯(⋅∣hsf)] _1 (· h_sf,h_ref)=sg[ _ θ(· h_sf)], recovering the token-level objective. The proposition establishes a conditional variational equivalence, rather than an unconditional equivalence between complete moving-distribution objectives. Although Zα(hsf,href)Z_α(h_sf,h_ref) is constant in a fixed inner update, its expectation under a context distribution generated by a changing policy need not be constant. The result therefore provides a reference-regularized, trust-region-style interpretation of the practical frozen-rollout on-policy self-distillation update. Self-Referenced On-Policy Self-Distillation Empirically, the combination of a moving target and a mode-seeking reverse KL divergence projection can produce unstable long-budget behavior, as observed in Figure˜3, while the associated entropy dynamics are shown in Figure˜2. Forward KL and Jensen–Shannon divergence (JSD) have also been considered as alternative divergences for on-policy self-distillation (Zhao et al., 2026a; Hübotter et al., 2026). At a fixed student-generated prefix, full-vocabulary forward KL, reverse KL, JSD, and Rényi divergence are all well-defined and can be evaluated directly when the corresponding logits are available. The practical difficulty arises when the token objective is approximated using actions sampled from the student or a restricted candidate set. We use the Rényi divergence family to generalize the projection geometry of OPSD. Rényi OPSD does not eliminate expectation computation; instead, it gives a student-expectation density-ratio form whose power is controlled continuously by ρ. The reverse Rényi projection can be written as maxθ∑t−Dρ(πθ(a∣ht)∥sg[πθ¯(a∣htsf)])=∑t11−ρloga∼πθ(a∣ht)[(sg[πθ¯(a∣htsf)]πθ(a∣ht))1−ρ]. _θ _t-D_ρ\!( _θ(a h_t)\|sg\![ _ θ(a h_t^sf)])= _t 11-ρ _a _θ(a h_t)[( sg\! [ _ θ(a h_t^sf) ] _θ(a h_t))^1-ρ]. The forward Rényi projection is maxθ∑t−Dρ(sg[πθ¯(a∣htsf)]∥πθ(a∣ht))=∑t11−ρloga∼πθ(a∣ht)[(sg[πθ¯(a∣htsf)]πθ(a∣ht))ρ]. _θ _t-D_ρ\!(sg\![ _ θ(a h_t^sf)]\| _θ(a h_t))= _t 11-ρ _a _θ(a h_t)[( sg\! [ _ θ(a h_t^sf) ] _θ(a h_t))^ρ]. Thus, the two directions share a common likelihood-ratio form but apply different powers to the teacher-to-student ratio. For 0<ρ<10<ρ<1, these powers temper extreme ratios and provide a tunable projection geometry. This reformulation concerns the token distribution at an already visited prefix. Algorithm 1 SR-OPSD 1:πθ,πθ¯,πref _θ, _ θ, _ref 2:for k=0,1,…k=0,1,… do 3: Sample prompts x∼x and responses y∼πθk(⋅∣x)y _ _k(· x) 4: for t=1,…,|y|t=1,…,|y| do 5: Construct hth_t, htsfh_t^sf, and htrefh_t^ref 6: Form the frozen target πα,k⋆(⋅∣htsf,htref) _α,k (· h_t^sf,h_t^ref) 7: end for 8: Update the policy θk+1←Update(θk,−η∇θ∑tDρ(πα,k⋆(⋅∣htsf,htref)∥πθ(⋅∣ht))|θ=θk) _k+1 ~( _k,~-η _θ _tD_ρ\! ( _α,k (· h_t^sf,h_t^ref)\, \|\, _θ(· h_t) ) |_θ= _k) 9: Update the self-teacher: θ¯k+1←βθ¯k+(1−β)θk+1 θ_k+1←β θ_k+(1-β) _k+1 10:end for 11:return πθ _θ Proposition 3.2 (Token-Level Variational Characterization of SR-OPSD). Fix h, hsfh_sf, and hrefh_ref. Let α∈[0,1]α∈[0,1] and ρ>0ρ>0, ρ≠1ρ≠ 1, and assume that all token probabilities are strictly positive. Define the token-level pseudo-reward rα(a;hsf,href):=αlogsg[πθ¯(a∣hsf)]πref(a∣href)r_α(a;h_sf,h_ref):=α sg\! [ _ θ(a h_sf) ] _ref(a h_ref). Define the Rényi-aggregated token functional ℛα,ρ(θ;h,hsf,href):=−1ρ−1log∑a∈πref(a∣href)ρπθ(a∣h)1−ρexp(ρrα(a;hsf,href))∑a∈πθ(a∣h)ρπref(a∣href)1−ρR_α,ρ(θ;h,h_sf,h_ref):=- 1ρ-1 _a _ref(a h_ref)^ρ _θ(a h)^1-ρ \! (ρ r_α(a;h_sf,h_ref) ) _a _θ(a h)^ρ _ref(a h_ref)^1-ρ. Let α,ρ(θ;h,hsf,href):=ℛα,ρ(θ;h,hsf,href)−Dρ(πθ(⋅∣h)∥πref(⋅∣href))J_α,ρ(θ;h,h_sf,h_ref):=R_α,ρ(θ;h,h_sf,h_ref)-D_ρ\! ( _θ(· h)\, \|\, _ref(· h_ref) ). Then α,ρ(θ;h,hsf,href)=−Dρ(πα⋆(⋅∣hsf,href)∥πθ(⋅∣h))−ρ−1logZα(hsf,href). _α,ρ(θ;h,h_sf,h_ref)=-D_ρ\! ( _α (· h_sf,h_ref)\, \|\, _θ(· h) )- ρ-1 Z_α(h_sf,h_ref). (1) Therefore, for fixed contexts and frozen target components, maximizing α,ρJ_α,ρ with respect to θ is equivalent to minimizing the forward Rényi projection from πα⋆(⋅∣hsf,href) _α (· h_sf,h_ref) to πθ(⋅∣h) _θ(· h). See Appendix C.2 for proof. The functional ℛα,ρR_α,ρ depends on the policy to be optimized and should be interpreted as a conditional risk-sensitive variational representation, rather than as an environment reward. When α=1α=1, Z1=1Z_1=1 and the proposition reduces to Rényi OPSD with the self-teacher as its target. The variational target suggests explicitly combining the self-teacher policy with a reference policy. At position t, define πα⋆(a∣htsf,htref)∝sg[πθ¯(a∣htsf)]απref(a∣htref)1−α _α (a h_t^sf,h_t^ref) \! [ _ θ(a h_t^sf) ]^α _ref(a h_t^ref)^1-α. A natural fixed reference is πref:=sg[πθ0] _ref:=sg[ _ _0], where θ0 _0 denotes the parameters of the policy before optimization. More generally, the reference policy should provide a useful anchor. The corresponding token-level objective is: maxθ _θ x∼y∼πθold(⋅∣x)∑t=1L11−ρ[log∑a∈exp((1−ρ)logπθ(a∣ht) _ subarraycx \\ y _ _old(· x) subarray _t=1^L 11-ρ[ _a ((1-ρ) _θ(a h_t) (2) +ραlogsg[πθ¯(a∣htsf)]+ρ(1−α)logπref(a∣htref))]. +ρα \! [ _ θ(a h_t^sf) ]+ρ(1-α) _ref(a h_t^ref))]. See Algorithm˜1 for the pseudo-code, where Update(⋅)Update(·) denotes an optimizer such as stochastic gradient descent or Adam. When a candidate-space approximation is used, the same objective is evaluated on a common normalized candidate set, ideally augmented with a tail bucket. The coefficient α controls the location of the self-referenced target: α=1α=1 recovers the self-teacher, whereas α=0α=0 recovers the reference policy. In contrast, ρ controls the Rényi projection toward this target. As ρ→1ρ→ 1, the projection recovers the corresponding KL limit. Proposition 3.3 further shows that ρ power-tempers the target-to-student density ratio. Although α and ρ parameterize different components of the objective, their effects interact in the resulting gradient through the product αραρ. As shown in Figure˜3, the proposed method achieves effective performance throughout training, while Figure˜2 shows stable policy-entropy dynamics. Qualitative examples are provided in Appendix B. Proposition 3.3 (Logit Gradient of SR-OPSD). Fix h, hsfh_sf, and hrefh_ref, and treat πα⋆(⋅∣hsf,href) _α (· h_sf,h_ref) as frozen during the student update. Let zθ(a∣h)z_θ(a h) denote the student logit such that πθ(a∣h)=exp(zθ(a∣h))∑b∈exp(zθ(b∣h)) _θ(a h)= (z_θ(a h)) _b (z_θ(b h)). For ρ>0ρ>0, ρ≠1ρ≠ 1, define π~α,ρ,θ(a∣h,hsf,href)∝πθ(a∣h)(πα⋆(a∣hsf,href)/πθ(a∣h))ρ π_α,ρ,θ(a h,h_sf,h_ref) _θ(a h)~( _α (a h_sf,h_ref) _θ(a h))^ρ. Then ∂zθ(a∣h)Dρ(πα⋆(⋅∣hsf,href)∥πθ(⋅∣h))=πθ(a∣h)−π~α,ρ,θ(a∣h,hsf,href) ∂ z_θ(a h)D_ρ\! ( _α (· h_sf,h_ref)\, \|\, _θ(· h) )= _θ(a h)- π_α,ρ,θ(a h,h_sf,h_ref). 4 Experiments We evaluate the proposed method across various representative scenarios that capture different feedback regimes and levels of supervision, including: (i) Science Q&A benchmarks without rich environment feedback; (i) mathematical reasoning on challenging problems with access to ground-truth solutions; and (i) scaling behavior on coding tasks with rich execution-based feedback; and (iv) ablation study on the effectiveness of each component of the proposed method. All experiments are conducted on NVIDIA H200 and A800 GPUs. 4.1 Performance with Bootstrapping on Generated Responses as Feedback Setting We compare our method against SDPO, on-policy GRPO, which samples trajectories from the current policy, and GRPO, which reweights trajectories from previous policies using importance sampling. For all baselines, we use the same hyperparameter settings as SDPO. We evaluate scientific reasoning on undergraduate-level problems in chemistry, physics, biology, and materials science using the L3 reasoning subset of SciKnowEval (Feng et al., 2024), with a train–test split to assess in-domain generalization. We initialize the policies from Qwen3-8B (Yang et al., 2025) and OLMo-3-7B-Instruct (Olmo et al., 2025), and report Avg@16 in Table˜1 in terms of wall-clock training time. We also report Maj@16 accuracy and Best@16 accuracy in Figure˜4 and Figure˜5. All experiments are conducted using the verl framework on NVIDIA H200 GPUs to ensure fair comparisons with Hübotter et al. (2026). See Table˜6 for detailed experimental configurations. Table 1: Performance on SciKnowEval benchmark. We report Avg@16 accuracy. The best and second-best results are shown in bold and underlined, respectively. Chemistry Physics Biology Materials Tool use 5h 10h 15h 5h 10h 15h 5h 10h 15h 5h 10h 15h 5h 10h 15h Qwen3-8B 41.2 59.2 30.8 58.9 57.5 + GRPO 63.163.1 68.368.3 72.672.6 62.762.7 72.072.0 74.1 35.235.2 52.552.5 53.553.5 74.374.3 75.975.9 79.3 62.662.6 66.366.3 66.3 + GRPO (on-policy) 51.851.8 62.962.9 68.168.1 60.860.8 62.262.2 62.662.6 33.833.8 35.535.5 36.436.4 70.470.4 73.173.1 74.874.8 60.160.1 62.562.5 65.565.5 + SDPO 80.580.5 80.5 80.5 71.1 73.4 73.473.4 66.066.0 66.066.0 66.066.0 76.7 78.3 78.378.3 63.863.8 63.863.8 63.863.8 + SR-OPSD (ours) 80.2 82.882.8 82.882.8 80.180.1 81.281.2 82.382.3 58.5 58.5 62.5 77.777.7 79.679.6 80.980.9 63.6 64.4 67.567.5 Olmo3-7B-Instruct 22.8 37.7 16.2 36.7 39.3 + GRPO 39.139.1 45.445.4 53.453.4 58.958.9 61.461.4 62.862.8 33.133.1 38.538.5 42.642.6 71.471.4 73.373.3 74.274.2 56.956.9 61.0 61.0 + GRPO (on-policy) 37.537.5 40.340.3 44.444.4 54.354.3 59.159.1 60.960.9 29.829.8 31.631.6 34.934.9 66.466.4 70.770.7 73.373.3 55.355.3 57.057.0 59.659.6 + SDPO 79.679.6 79.6 79.6 67.6 70.8 70.8 51.951.9 52.052.0 52.052.0 77.477.4 77.477.4 77.477.4 58.858.8 60.460.4 60.460.4 + SR-OPSD (ours) 78.7 79.979.9 79.979.9 68.168.1 72.372.3 74.874.8 50.1 51.1 51.1 76.2 76.2 76.2 58.1 61.461.4 62.762.7 Figure 2: Investigation of Entropy of the Policy on Science Q&A (Chemistry). Left: Avg@16 accuracy. Right: Training entropy of the policy. Results Analysis As shown in Table˜1, SR-OPSD achieves the best 1515-hour Avg@16 accuracy in seven of the ten cases and outperforms both GRPO variants in all ten settings. For Qwen3-8B, SR-OPSD surpasses the strongest competing baseline by 2.32.3, 8.28.2, 1.61.6, and 1.21.2 percentage points on Chemistry, Physics, Materials, and Tool Use, respectively, while SDPO remains stronger on Biology by 3.53.5 points. Averaged across the five domains, SR-OPSD reaches 75.275.2 Avg@16 at 15 hours, compared with 72.472.4 for SDPO and 69.269.2 for GRPO. On OLMo3-7B-Instruct, SR-OPSD obtains the best results on Chemistry, Physics, and Tool Use, improving over the strongest baseline by 0.30.3, 4.04.0, and 1.71.7 points, respectively. It remains competitive on Biology and Materials, trailing SDPO by only 0.90.9 and 1.21.2 points. Its five-domain average reaches 68.968.9, exceeding SDPO and GRPO by 0.90.9 and 10.110.1 points, respectively. Meanwhile, we also investigate the entropy of the policy and corresponding generation quality in Figure˜2 and Appendix B. It is found that SR-OPSD also stabilizes the entropy of the policy and has better generation quality than the associated baselines. Figure 3: SR-OPSD improves long-budget training stability. In Figure˜3, we report Avg@16 validation accuracy over training steps on five reasoning-related benchmarks for Qwen3-8B and Olmo3-7B-Instruct. Compared with the JSD-based SDPO baseline, SR-OPSD generally maintains more stable long-budget performance, whereas SDPO often reaches an early peak and then degrades. These results suggest that reference anchoring and Rényi projection provide a more favorable performance and training stability. 4.2 Performance on Mathematical Reasoning Benchmarks Setting We then evaluate the performance in a setting where we have access to ground-truth solutions as feedback on very hard mathematical reasoning problems. Following the data construction of Agrawal et al. (2026), we use a training collection of 738 particularly challenging mathematics problems. We benchmark our approach against OPSD (forward KL), SDPO (reverse KL), and GRPO on AIME 2024, AIME 2025, HMMT 2025, AMC 2023, and Minerva, with all methods initialized from Qwen3-4B-Instruct-2507. Due to the high computational cost of evaluation, we train all methods for 200 steps and report performance at the final. See Table˜6 for detailed experimental configurations. Table 2: Performance on mathematical reasoning benchmarks. The best results are shown in bold, and the second-best results are underlined. Model Method AIME24 AIME25 HMMT25 AMC23 Minerva Avg@64 Pass@64 Avg@64 Pass@64 Avg@64 Pass@64 Avg@64 Pass@64 Avg@64 Pass@64 Qwen3-4B Base 61.4 82.0 50.3 69.5 30.3 48.8 93.8 98.8 43.2 48.6 GRPO 58.8 83.3 49.2 73.3 32.5 60.0 88.8 100.0 45.9 59.6 SDPO 41.3 80.0 31.9 66.7 19.6 50.0 74.5 97.5 42.9 57.7 OPSD 48.6 76.7 37.0 56.7 22.3 46.7 81.1 97.5 45.0 57.0 SR-OPSD (Ours) 59.3 83.3 51.8 80.0 32.5 70.0 91.0 100.0 46.2 58.8 Results Analysis As shown in Table˜2, SR-OPSD achieves the strongest overall performance across the five mathematical-reasoning benchmarks. It obtains the best Avg accuracy on AIME25 and Minerva, ties for the best Avg accuracy on HMMT25, and achieves the best or tied-best Pass accuracy on AIME24, AIME25, HMMT25, and AMC23. In particular, compared with GRPO, SR-OPSD improves Pass accuracy by 6.76.7 points on AIME25 and 10.010.0 points on HMMT25, while increasing Avg accuracy by 2.62.6, 2.22.2, and 0.30.3 points on AIME25, AMC23, and Minerva, respectively. Averaged uniformly across the five benchmarks, SR-OPSD reaches 56.256.2 Avg accuracy and 78.478.4 Pass accuracy, compared with 55.055.0 and 75.275.2 for GRPO. The advantage over the self-distillation baselines is more pronounced. SR-OPSD exceeds forward-KL OPSD by 9.49.4 points in mean Avg accuracy and 11.511.5 points in mean Pass accuracy, and exceeds reverse-KL SDPO by 14.114.1 and 8.08.0 points, respectively. These results suggest that directly matching a self-teacher with KL projection can substantially degrade the performance of the base models in this challenging regime, whereas reference anchoring and Rényi projection better preserve its initial capabilities while incorporating solution feedback. SR-OPSD show the promising performance across these benchmarks, substantially improving AIME25, HMMT25, and Minerva without the severe degradation observed for OPSD and SDPO. 4.3 Performance on Code generation under Model Scaling Setting We next assess our method on code-generation tasks. Programming provides a representative setting with informative feedback signals, including execution errors and unsuccessful unit tests. Success in this domain depends on effective credit assignment, since the model must locate the precise source of an error and use that information to avoid making the same mistake in subsequent attempts. Following the protocol of (Hübotter et al., 2026), we focus our evaluation on the LCBv6 subset. We use the Qwen3 model family for this investigation and train all methods for 120 iterations. See Table˜7 for detailed experimental configurations. Table 3: Performance on LiveCodeBench v6 across Qwen3 model scales. The best result in each column is shown in bold, and the second-best result is underlined. Method Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B Base 9.2 14.7 23.1 27.9 GRPO 18.5 29.3 39.9 41.2 SDPO 18.7 30.7 44.7 48.8 SR-OPSD 18.0 32.0 46.2 50.1 Results Analysis As shown in Table 3, all methods improve over their corresponding base models across the four model scales. SR-OPSD achieves the best performance on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, reaching 32.032.0, 46.246.2, and 50.150.1, respectively. Compared with GRPO, these results correspond to improvements of 2.72.7, 6.36.3, and 8.98.9 percentage points, while the gains over SDPO are 1.31.3, 1.51.5, and 1.31.3 points. On Qwen3-0.6B, SR-OPSD remains competitive but trails GRPO and SDPO by 0.50.5 and 0.70.7 points, respectively. One potential reason for the slightly degraded performance on the 0.6B model is that we did not tune the hyperparameters due to the expensive computational cost. Nevertheless, it still nearly doubles the base-model pass-all success rate, improving it from 9.29.2 to 18.018.0. Moreover, except at the smallest scale, SR-OPSD outperforms SDPO as model scale increases. 4.4 Ablation Study Table 4: Ablation studies on SciKnowEval Physics. The best result in each column is shown in bold. Optimization Objective Self-reference Rényi Avg@16 JSD × × 79.4 JSD ✓ × 77.2 Forward KL × × 79.1 Forward KL ✓ × 76.3 Forward Rényi (ρ=0.95ρ=0.95) × ✓ 77.7 SR-OPSD (ρ=0.50ρ=0.50) ✓ ✓ 79.9 SR-OPSD (ρ=0.70ρ=0.70) ✓ ✓ 80.6 SR-OPSD (ρ=0.95ρ=0.95) ✓ ✓ 81.1 Setting We conduct a controlled ablation to examine the effects of self-reference, divergence choice, and the Rényi order ρ. All variants use the same model initialization, training data, optimization budget, and evaluation protocol as the main experiment, and performance is measured using Avg@16 accuracy. We first compare two commonly used on-policy self-distillation objectives, JSD (Hübotter et al., 2026) and forward KL (Zhao et al., 2026a), both with and without the reference-anchored target. We then evaluate the complete SR-OPSD objective, which combines self-reference with Rényi projection, tuning ρ∈0.50,0.70,0.95ρ∈\0.50,0.70,0.95\. Results Analysis Table 4 shows that self-reference does not improve performance when it is combined with the standard JSD or forward-KL objectives. Adding self-reference reduces Avg@16 from 79.479.4 to 77.277.2 for JSD and from 79.179.1 to 76.376.3 for forward KL, suggesting that target anchoring alone is insufficient and may even be detrimental when paired with an unsuitable projection geometry. By contrast, SR-OPSD, which combines self-reference with Rényi projection, achieves the strongest performance. Its Avg@16 score increases from 79.979.9 at ρ=0.50ρ=0.50 to 80.680.6 at ρ=0.70ρ=0.70 and 81.181.1 at ρ=0.95ρ=0.95, outperforming the best non-Rényi variant by 1.91.9 points. These results indicate that the benefit of self-reference depends critically on the divergence used to approach the anchored target, and that tuning ρ provides an effective mechanism for controlling this projection. 5 Related Work On-Policy Distillation On-policy distillation (OPD) uses student-generated trajectories for training, allowing the teacher to provide distributional supervision on prefixes visited by the student Agarwal et al. (2024); Wen et al. (2023); Gu et al. (2023); Lu and Lab (2025); Jin et al. (2026); Luo et al. (2026); Yang et al. (2026); Stein et al. (2026); Oh et al. (2026); Kim et al. (2026). Gu et al. (2023) optimizes sequence-level reverse KL through policy-gradient estimators, while Agarwal et al. (2024) evaluates forward KL, reverse KL, and JSD on student rollouts. Later work studies entropy-dependent divergence selection, reference-policy constraints, and rollout mixing to improve stability (Jin et al., 2026; Luo et al., 2026). Yang et al. (2026) further formulates external-teacher OPD as KL-regularized policy optimization, where a teacher-to-reference log-probability ratio defines the dense reward and controls interpolation or extrapolation between the two policies. See (Song and Zheng, 2026) for a detailed review. On-Policy Self-Distillation OPSD replaces the external teacher with a self-teacher derived from the same underlying model and conditioned on training-only privileged or auxiliary context. Existing methods use ground-truth solutions, environment feedback, successful sibling responses, or document context to construct the self-teacher (Zhao et al., 2026a; Hübotter et al., 2026; Penaloza et al., 2026; Ye et al., 2026; Sang et al., 2026; Zhao et al., 2026b; Wang et al., 2026). Subsequent work controls the amount of additional information exposed to the teacher, restricts distillation to selected spans, or gates token-level supervision according to teacher reliability and uncertainty (Han et al., 2026; Wang et al., 2026; Liu et al., 2026; Ke et al., 2026). Other studies analyze the effects of additional context information, Top-K approximation, and update schemes of self-teacher policies (Zhu et al., 2026; Guo et al., 2026). Existing objectives mainly use forward KL, reverse KL, or JSD for direct distribution matching (Zhao et al., 2026a; Hübotter et al., 2026), while preference-based self-distillation replaces direct matching with reward-regularized preference optimization (Yu et al., 2026). In this work, we regard the additional context information and the construction of the self-teacher policies as given, and focus on the target distribution and divergence geometry used for matching at fixed on-policy contexts. Specifically, we combine the self-teacher policy and the reference policy into the normalized geometric target. This log-linear teacher–reference target is closely related to the target induced by reward-scaled KL-regularized OPD (Yang et al., 2026). The coefficient α controls interpolation, or extrapolation when permitted, along the log-density path between the reference policy and the self-teacher. We then project the student toward πα⋆ _α using a Rényi divergence of order ρ. 6 Conclusion In this work, we propose SR-OPSD, which constructs a reference-anchored self-teacher target and optimizes it using a R’enyi projection. Our results highlight that the effectiveness of self-reference depends critically on the objective through which it is imposed. Reference anchoring alone is not sufficient, and self-reference can even degrade performance under standard forward-KL or JSD objectives. In contrast, coupling the self-referenced target with an appropriate Rényi projection improves performance and training stability across scientific reasoning, mathematical reasoning, and code generation. These results suggest that feedback in on-policy self-distillation is most effective when both its influence on the target distribution and the policy’s response to that target are explicitly controlled, rather than treating feedback as an unconstrained imitation signal. Several limitations remain. Our theoretical characterization concerns token-level optimization at fixed rollout contexts and does not establish global convergence of the fully coupled student–self-teacher dynamics. In particular, context-distribution shift remains an additional source of error that must be controlled separately. Moreover, the target-interpolation coefficient, Rényi order, and teacher update rate are fixed and selected empirically. Future work could study adaptive choices of these quantities based on teacher reliability or policy drift, develop tighter guarantees for the evolving on-policy dynamics, and evaluate SR-OPSD under noisier feedback and at larger model scales. References R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Cited by: §1, §5. R. Agrawal, J. Fein-Ashley, and P. Rashidinejad (2026) Reinforcement learning from rich feedback with distributional DAgger. In Continual Reinforcement Learning Workshop at RLC 2026, External Links: Link Cited by: §4.2. K. Feng, X. Shen, W. Wang, X. Zhuang, Y. Tang, Q. Zhang, and K. Ding (2024) Sciknoweval: evaluating multi-level scientific knowledge of large language models. arXiv preprint arXiv:2406.09098. Cited by: §4.1. Y. Gu, L. Dong, F. Wei, and M. Huang (2023) Minillm: on-policy distillation of large language models. arXiv preprint arXiv:2306.08543. Cited by: §1, §5. H. Guo, B. Bi, R. Zhang, B. Sun, and W. Zhang (2026) When should the teacher move? temporal coupling and stability in self on-policy distillation. arXiv preprint arXiv:2606.03532. Cited by: §5. Z. Han, T. Zhang, H. Wang, and Y. Sun (2026) Adaptive teacher exposure for self-distillation in llm reasoning. arXiv preprint arXiv:2605.11458. External Links: 2605.11458, Link Cited by: §5. J. Hübotter, F. Lübeck, L. D. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. K. Buening, C. Guestrin, and A. Krause (2026) Reinforcement learning via self-distillation. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §1, §1, §2, §3, §4.1, §4.3, §4.4, §5. W. Jin, T. Min, Y. Yang, D. Wei, Y. Zhou, S. R. Kadhe, N. Baracaldo, and K. Lee (2026) Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079. Cited by: §1, §1, §5. J. Ke, Z. Wen, W. Li, C. He, and L. Zhang (2026) Respecting self-uncertainty in on-policy self-distillation for efficient llm reasoning. arXiv preprint arXiv:2605.13255. External Links: 2605.13255, Link Cited by: §5. Y. Kim, D. Shin, M. Kang, B. Na, and I. Moon (2026) Distillation of large language models via concrete score matching. In International Conference on Learning Representations, Vol. 2026, p. 16649–16682. Cited by: §5. X. Liu, X. Wang, Y. Ma, Y. Zhang, and C. Xiao (2026) When are teacher tokens reliable? position-weighted on-policy self-distillation for reasoning. arXiv preprint arXiv:2605.21606. Cited by: §5. K. Lu and T. M. Lab (2025) On-policy distillation. Thinking Machines Lab: Connectionism. Note: https://thinkingmachines.ai/blog/on-policy-distillation External Links: Document Cited by: §5. F. Luo, Y. Chuang, G. Wang, Z. Xu, X. Han, T. Zhang, and V. Braverman (2026) Demystifying opd: length inflation and stabilization strategies for large language models. arXiv preprint arXiv:2604.08527. Cited by: §1, §5. M. Oh, S. Song, G. Choi, Y. Choi, and Y. Jo (2026) KL for a kl: on-policy distillation with control variate baseline. arXiv preprint arXiv:2605.07865. Cited by: §5. T. Olmo, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, et al. (2025) Olmo 3. arXiv preprint arXiv:2512.13961. Cited by: §4.1. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 27730–27744. External Links: Document, Link Cited by: §1. E. Penaloza, D. Vattikonda, N. Gontier, A. Lacoste, L. Charlin, and M. Caccia (2026) Privileged information distillation for language models. arXiv preprint arXiv:2602.04942. External Links: 2602.04942, Link Cited by: §5. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §1. H. Sang, Y. Xu, Z. Zhou, R. He, Z. Wang, and J. Sun (2026) Crisp: compressed reasoning via iterative self-policy distillation. arXiv preprint arXiv:2603.05433. Cited by: §5. V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. Nayak, D. Datta, J. Chang, M. T. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Teehan, T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush (2022) Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, External Links: Link Cited by: §1. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1. M. Song and M. Zheng (2026) A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626. Cited by: §5. A. Stein, F. Huang, and T. Goldstein (2026) GATES: self-distillation under privileged context with consensus gating. arXiv preprint arXiv:2602.20574. External Links: 2602.20574, Link Cited by: §5. T. Van Erven and P. Harremos (2014) Rényi divergence and kullback-leibler divergence. IEEE transactions on information theory 60 (7), p. 3797–3820. Cited by: §2. J. Wang, X. Ouyang, Z. Chen, Y. Hu, Z. Pan, X. Li, and L. Guo (2026) TRACE: distilling where it matters via token-routed self on-policy alignment. arXiv preprint arXiv:2605.10194. External Links: 2605.10194, Link Cited by: §5. J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le (2022) Finetuned language models are zero-shot learners. In International Conference on Learning Representations, External Links: Link Cited by: §1. Y. Wen, Z. Li, W. Du, and L. Mou (2023) F-divergence minimization for sequence-level knowledge distillation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 10817–10834. External Links: Link, Document Cited by: §5. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.1. W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin (2026) Learning beyond teacher: generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125. Cited by: §1, §5, §5. T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei (2026) On-policy context distillation for language models. arXiv preprint arXiv:2602.12275. Cited by: §5. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, YuYue, W. Dai, T. Fan, G. Liu, J. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, R. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, Y. Wu, and M. Wang (2025) DAPO: an open-source LLM reinforcement learning system at scale. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1. X. Yu, L. Liao, Y. Zhang, Y. Yu, L. Xue, and Q. Guo (2026) Preference-based self-distillation: beyond kl matching via reward regularization. arXiv preprint arXiv:2605.05040. Cited by: §1, §5. S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026a) Self-distilled reasoner: on-policy self-distillation for large language models. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §1, §1, §3, §4.4, §5. Z. Zhao, L. Ma, and W. Zhang (2026b) Training with harnesses: on-policy harness self-distillation for complex reasoning. arXiv preprint arXiv:2605.08741. Cited by: §5. S. Zhu, X. Ye, H. Lu, W. Shi, and G. Liu (2026) The many faces of on-policy distillation: pitfalls, mechanisms, and fixes. arXiv preprint arXiv:2605.11182. External Links: 2605.11182, Link Cited by: §1, §1, §5. Appendix A Notation Table Table 5: Summary of the notation. Symbol Meaning x,x,D Prompt and prompt distribution, with x∼x . y,yt,y<ty,y_t,y_<t On-policy response, token at position t, and prefix before t. a,,ta,V,C_t Generic next-token action, full vocabulary, and candidate set at position t. hth_t Student prefix context, ht=(x,y<t)h_t=(x,y_<t). htsf,htrefh_t^sf,h_t^ref Context information for the self-teacher policy and reference policy at position t, respectively. h,hsf,hrefh,h_sf,h_ref Fixed-context shorthand obtained by suppressing the position index t. πθ,θ _θ,θ Policy to be optimized (student policy) and its parameters. πθ¯,θ¯ _ θ, θ Frozen or EMA self-teacher policy and its parameters. πθold _ _old Rollout policy used to sample the current on-policy batch. πref _ref Reference policy. θ0 _0 Parameters of the student policy before optimization. πT _T External teacher policy, when on-policy distillation is discussed. sg[⋅]sg[·] Stop-gradient operator. πα⋆,Zα _α ,Z_α Reference-anchored self-teacher target and its normalizing constant. α,ρα,ρ Target-interpolation coefficient and Rényi order. DKL,DρD_KL,D_ρ KL divergence and Rényi divergence. β EMA coefficient. Appendix B Qualitative Comparison of Generations We select representative cases from SciKnowEval that expose recurrent differences in reasoning stability rather than estimate their population frequency. All compared models are Qwen3-8B checkpoints post-trained on SciKnowEval using the corresponding methods. The model post-trained with SR-OPSD generally gives shorter, more direct explanations while preserving the connection between intermediate reasoning and the selected option. Models post-trained with the other methods more often extend an initially valid argument until they introduce a contradiction, factual error, or answer-option drift. The model post-trained with SDPO (RKL) exhibits a different failure mode characterized by terse, repetitive, or generic reasoning. In the examples below, red text marks the first substantive departure from a valid derivation or from the correct answer-option mapping. CASE 1 Answer-Binding Failure After Correct Calculation Math Prompt. An item costs $89. It is discounted by 30%, then a 12% sales tax is applied. What is the final price? Choices. A. $99.68; B. $69.78; C. $62.30; D. $71. Gold. B Method Response Excerpt Quality SR-OPSD (w. ref.) A 30% discount reduces $89 to $62.30. Applying 12% tax gives $62.30 + $7.48 = $69.78. Answer: B ✓ Correct SR-OPSD (wo. ref.) The final price is approximately $69.78. However, Option B is not sufficiently aligned with the calculation, and only Option D accounts for both discount and tax. Answer: D × Answer drift SDPO (RKL) The calculation gives $62.3 * (1 + 0.12) = $69.78, but the final option changes. Answer: D × Binding failure Finding. Both models reached the correct value but failed to bind it to the correct option; model from SR-OPSD (w. ref.) preserved calculation–answer alignment. CASE 2 Nuclear-Decay Reasoning Stability Physics Prompt. An element emits one alpha particle, followed by two beta particles. How does its atomic number change? Choices. A. zero; B. −1-1; C. −2-2; D. +1+1. Gold. A Method Response Excerpt Quality SR-OPSD (w. ref.) Alpha emission reduces the atomic number by 2; two beta emissions increase it by 2. The net change is −2+2=0-2+2=0. Answer: A ✓ Faithful SR-OPSD (wo. ref.) The response states the correct local rules but claims the total effect is not zero and selects Option D. Answer: D × Conclusion drift SDPO (RKL) It incorrectly states that alpha emission decreases the atomic number by 1 and retains the wrong conclusion. Answer: D × Rule collapse Finding. Without self-reference, the conclusion drifts despite correct local rules; reverse KL additionally corrupts the underlying decay rule. CASE 3 Self-Contradiction in Anatomical Knowledge Anatomy Prompt. In the spinal cord, motor neuron cell bodies are located in which lamina? Choices. A. I; B. I; C. V; D. IX. Gold. D Method Response Excerpt Quality SR-OPSD (w. ref.) Motor neuron cell bodies are located in lamina IX, within the ventral horn responsible for voluntary movement. Answer: D ✓ Stable SR-OPSD (wo. ref.) The response identifies the ventral horn but then claims lamina IX is not part of the motor system and selects lamina I. Answer: A × Contradiction SDPO (RKL) It gives a generic statement about spinal laminae and claims motor neurons are located in lamina I. Answer: B × Hallucination Finding. SR-OPSD (w. ref.) preserves the correct anatomical entity; the alternatives contradict it or replace it with a generic claim. CASE 4 Repetition and Incorrect Option Selection Nutrition Prompt. How many kcal are there in one gram of ethanol? Choices. A. 5.8; B. 4.1; C. 7.1; D. 8.1 kcal/g. Gold. C Method Response Excerpt Quality SR-OPSD (w. ref.) Ethanol provides approximately 7.1 kcal per gram, matching Option C. Answer: C ✓ Correct SR-OPSD (wo. ref.) It claims Option A (5.8 kcal/g) is the most scientifically accurate choice and that Option C overestimates the energy. Answer: A × Plausible error SDPO (RKL) It repeats the clue 29.7 kJ per gram but does not convert it consistently. Answer: B × Repetitive mismatch Finding. The alternatives either prefer a plausible wrong value or repeat the correct clue without binding it to the final answer. CASE 5 Template Collapse on a Complex Scientific Question Chemistry Prompt. What is the molar weight of a complex peptide-like molecule specified by a long IUPAC name? Choices. A. 1517.700; B. 1589.900; C. 1536.800; D. 1490.700. Gold. C Method Response Excerpt Quality SR-OPSD (w. ref.) Accounting for all atoms gives the molar weight represented by Option C (1536.800). Answer: C ✓ Correct SR-OPSD (wo. ref.) It discusses the functional groups but concludes that Option A is most accurate and Option C is too high. Answer: A × Option drift SDPO (RKL) It begins with atomic-weight summation but drifts to the irrelevant claim that the groups indicate a hydrophilic molecule. Answer: D × Template collapse Finding. SR-OPSD (w. ref.) remains aligned with molar-weight estimation; the alternatives drift to a nearby option or a generic chemistry template. Appendix C Theoretical Results and Proofs C.1 Proof of Proposition 3.1 Proof. Fix h, hsfh_sf, and hrefh_ref, and abbreviate q(a):=πθ(a∣h),p(a):=sg[πθ¯(a∣hsf)],r(a):=πref(a∣href).q(a):= _θ(a h), p(a):=sg\! [ _ θ(a h_sf) ], r(a):= _ref(a h_ref). The conditional objective expands as α _α =∑aq(a)[αlogp(a)r(a)−logq(a)r(a)] = _aq(a) [α p(a)r(a)- q(a)r(a) ] =∑aq(a)[−logq(a)+αlogp(a)+(1−α)logr(a)]. = _aq(a) [- q(a)+α p(a)+(1-α) r(a) ]. By definition, logπα⋆(a∣hsf,href)=αlogp(a)+(1−α)logr(a)−logZα(hsf,href). _α (a h_sf,h_ref)=α p(a)+(1-α) r(a)- Z_α(h_sf,h_ref). Substituting this identity yields α _α =∑aq(a)[−logq(a)+logπα⋆(a∣hsf,href)+logZα(hsf,href)] = _aq(a) [- q(a)+ _α (a h_sf,h_ref)+ Z_α(h_sf,h_ref) ] =−DKL(πθ(⋅∣h)∥πα⋆(⋅∣hsf,href))+logZα(hsf,href), =-D_KL\! ( _θ(· h)\, \|\, _α (· h_sf,h_ref) )+ Z_α(h_sf,h_ref), where the last step uses ∑aq(a)=1 _aq(a)=1. Since the contexts and target components are fixed, ZαZ_α is constant with respect to the inner optimization variable θ. When α=1α=1, Z1(hsf,href)=∑ap(a)=1Z_1(h_sf,h_ref)= _ap(a)=1 and π1⋆(a∣hsf,href)=p(a)=sg[πθ¯(a∣hsf)]. _1 (a h_sf,h_ref)=p(a)=sg\! [ _ θ(a h_sf) ]. The objective therefore reduces to the reverse-KL OPSD objective. The constancy statement is conditional on the fixed contexts: after averaging over a context distribution that itself changes with the policy, the expected normalizer need not be constant. ∎ C.2 Proof of Proposition 3.2 Proof. Fix h, hsfh_sf, and hrefh_ref. To simplify the notation, define q(a) q(a) :=πθ(a∣h), := _θ(a h), p(a) p(a) :=sg[πθ¯(a∣hsf)], :=sg\! [ _ θ(a h_sf) ], r(a) r(a) :=πref(a∣href). := _ref(a h_ref). Because all token probabilities are strictly positive, all logarithms, probability ratios, and real-valued powers appearing below are well-defined. Under these abbreviations, the normalizing constant is Zα:=Zα(hsf,href)=∑a∈p(a)αr(a)1−α. Z_α:=Z_α(h_sf,h_ref)= _a p(a)^αr(a)^1-α. The normalized self-referenced target satisfies πα⋆(a∣hsf,href)=p(a)αr(a)1−αZα. _α (a h_sf,h_ref)= p(a)^αr(a)^1-αZ_α. Multiplying both sides of above equation by ZαZ_α gives p(a)αr(a)1−α=Zαπα⋆(a∣hsf,href). p(a)^αr(a)^1-α=Z_α _α (a h_sf,h_ref). (3) We first simplify the numerator of ℛα,ρR_α,ρ. By the definition of the pseudo-reward rαr_α, rα(a;hsf,href)=αlogp(a)r(a). r_α(a;h_sf,h_ref)=α p(a)r(a). Multiplying both sides by ρ yields ρrα(a;hsf,href)=ραlogp(a)r(a). ρ r_α(a;h_sf,h_ref)=ρα p(a)r(a). (4) Exponentiating both sides of Equation˜4 gives exp(ρrα(a;hsf,href)) \! (ρ r_α(a;h_sf,h_ref) ) =exp(ραlogp(a)r(a)) = \! (ρα p(a)r(a) ) =(p(a)r(a))ρα = ( p(a)r(a) )^ρα =p(a)ραr(a)−ρα. =p(a)^ραr(a)^-ρα. (5) Consider one summand in the numerator of ℛα,ρR_α,ρ. Using Equation˜5, we obtain r(a)ρq(a)1−ρexp(ρrα(a;hsf,href)) r(a)^ρq(a)^1-ρ \! (ρ r_α(a;h_sf,h_ref) ) =r(a)ρq(a)1−ρp(a)ραr(a)−ρα. =r(a)^ρq(a)^1-ρp(a)^ραr(a)^-ρα. (6) Combining the two powers of r(a)r(a) gives r(a)ρr(a)−ρα r(a)^ρr(a)^-ρα =r(a)ρ−ρα =r(a)^ρ-ρα =r(a)ρ(1−α). =r(a)^ρ(1-α). (7) Substituting Equation˜7 into Equation˜6 yields r(a)ρq(a)1−ρexp(ρrα(a;hsf,href)) r(a)^ρq(a)^1-ρ \! (ρ r_α(a;h_sf,h_ref) ) =q(a)1−ρp(a)ραr(a)ρ(1−α). =q(a)^1-ρp(a)^ραr(a)^ρ(1-α). (8) Therefore, Equation˜8 becomes r(a)ρq(a)1−ρexp(ρrα(a;hsf,href)) r(a)^ρq(a)^1-ρ \! (ρ r_α(a;h_sf,h_ref) ) =q(a)1−ρ[p(a)αr(a)1−α]ρ. =q(a)^1-ρ [p(a)^αr(a)^1-α ]^ρ. (9) Applying Equation˜3 to Equation˜9 gives r(a)ρq(a)1−ρexp(ρrα(a;hsf,href)) r(a)^ρq(a)^1-ρ \! (ρ r_α(a;h_sf,h_ref) ) =q(a)1−ρ[Zαπα⋆(a∣hsf,href)]ρ =q(a)^1-ρ [Z_α _α (a h_sf,h_ref) ]^ρ =Zαρπα⋆(a∣hsf,href)ρq(a)1−ρ. =Z_α^ρ _α (a h_sf,h_ref)^ρq(a)^1-ρ. (10) Summing both sides of Equation˜10 over a∈a gives ∑a∈r(a)ρq(a)1−ρexp(ρrα(a;hsf,href)) _a r(a)^ρq(a)^1-ρ \! (ρ r_α(a;h_sf,h_ref) ) =∑a∈Zαρπα⋆(a∣hsf,href)ρq(a)1−ρ. = _a Z_α^ρ _α (a h_sf,h_ref)^ρq(a)^1-ρ. (11) Because ZαZ_α does not depend on the summation variable a, it can be taken outside the sum: ∑a∈r(a)ρq(a)1−ρexp(ρrα(a;hsf,href)) _a r(a)^ρq(a)^1-ρ \! (ρ r_α(a;h_sf,h_ref) ) =Zαρ∑a∈πα⋆(a∣hsf,href)ρq(a)1−ρ. =Z_α^ρ _a _α (a h_sf,h_ref)^ρq(a)^1-ρ. (12) By the definition of Rényi divergence, Dρ(πα⋆(⋅∣hsf,href)∥q) D_ρ\! ( _α (· h_sf,h_ref)\, \|\,q ) =1ρ−1log∑a∈πα⋆(a∣hsf,href)ρq(a)1−ρ. = 1ρ-1 _a _α (a h_sf,h_ref)^ρq(a)^1-ρ. (13) Multiplying both sides of Equation˜13 by ρ−1ρ-1 gives (ρ−1)Dρ(πα⋆(⋅∣hsf,href)∥q) (ρ-1)D_ρ\! ( _α (· h_sf,h_ref)\, \|\,q ) =log∑a∈πα⋆(a∣hsf,href)ρq(a)1−ρ. = _a _α (a h_sf,h_ref)^ρq(a)^1-ρ. (14) Exponentiating both sides of Equation˜14 gives exp((ρ−1)Dρ(πα⋆(⋅∣hsf,href)∥q)) \! ((ρ-1)D_ρ\! ( _α (· h_sf,h_ref)\, \|\,q ) ) =∑a∈πα⋆(a∣hsf,href)ρq(a)1−ρ. = _a _α (a h_sf,h_ref)^ρq(a)^1-ρ. (15) Substituting Equation˜15 into Equation˜12 yields ∑a∈r(a)ρq(a)1−ρexp(ρrα(a;hsf,href)) _a r(a)^ρq(a)^1-ρ \! (ρ r_α(a;h_sf,h_ref) ) =Zαρexp((ρ−1)Dρ(πα⋆(⋅∣hsf,href)∥q)). =Z_α^ρ \! ((ρ-1)D_ρ\! ( _α (· h_sf,h_ref)\, \|\,q ) ). (16) We next simplify the denominator of ℛα,ρR_α,ρ. By the definition of Rényi divergence, Dρ(q∥r)=1ρ−1log∑a∈q(a)ρr(a)1−ρ. D_ρ(q\|r)= 1ρ-1 _a q(a)^ρr(a)^1-ρ. (17) Multiplying both sides of Equation˜17 by ρ−1ρ-1 gives (ρ−1)Dρ(q∥r)=log∑a∈q(a)ρr(a)1−ρ. (ρ-1)D_ρ(q\|r)= _a q(a)^ρr(a)^1-ρ. (18) Exponentiating both sides of Equation˜18 gives exp((ρ−1)Dρ(q∥r))=∑a∈q(a)ρr(a)1−ρ. \! ((ρ-1)D_ρ(q\|r) )= _a q(a)^ρr(a)^1-ρ. (19) Using Equations˜16 and 19 in the definition of ℛα,ρR_α,ρ gives ℛα,ρ(θ;h,hsf,href) _α,ρ(θ;h,h_sf,h_ref) =−1ρ−1logZαρexp((ρ−1)Dρ(πα⋆(⋅∣hsf,href)∥q))exp((ρ−1)Dρ(q∥r)). =- 1ρ-1 Z_α^ρ \! ((ρ-1)D_ρ\! ( _α (· h_sf,h_ref)\, \|\,q ) ) \! ((ρ-1)D_ρ(q\|r) ). (20) We then obtain ℛα,ρ(θ;h,hsf,href) _α,ρ(θ;h,h_sf,h_ref) =−1ρ−1[log(Zαρexp((ρ−1)Dρ(πα⋆(⋅∣hsf,href)∥q))) =- 1ρ-1 [ \! (Z_α^ρ \! ((ρ-1)D_ρ\! ( _α (· h_sf,h_ref)\, \|\,q ) ) ) −log(exp((ρ−1)Dρ(q∥r)))]. 136.5733pt- \! ( \! ((ρ-1)D_ρ(q\|r) ) ) ]. (21) We then have ℛα,ρ(θ;h,hsf,href)=−1ρ−1[logZαρ+log(exp((ρ−1)Dρ(πα⋆(⋅∣hsf,href)∥q)))−log(exp((ρ−1)Dρ(q∥r)))]. _α,ρ(θ;h,h_sf,h_ref)=- 1ρ-1 [ Z_α^ρ+ \! ( \! ((ρ-1)D_ρ\! ( _α (· h_sf,h_ref)\, \|\,q ) ) )- \! ( \! ((ρ-1)D_ρ(q\|r) ) ) ]. We then have ℛα,ρ(θ;h,hsf,href)=−1ρ−1[ρlogZα+(ρ−1)Dρ(πα⋆(⋅∣hsf,href)∥q)−(ρ−1)Dρ(q∥r)]. _α,ρ(θ;h,h_sf,h_ref)=- 1ρ-1 [ρ Z_α+(ρ-1)D_ρ\! ( _α (· h_sf,h_ref)\, \|\,q )-(ρ-1)D_ρ(q\|r) ]. Then, we obtain ℛα,ρ(θ;h,hsf,href)=−ρ−1logZα−Dρ(πα⋆(⋅∣hsf,href)∥q)+Dρ(q∥r). _α,ρ(θ;h,h_sf,h_ref)=- ρ-1 Z_α-D_ρ\! ( _α (· h_sf,h_ref)\, \|\,q )+D_ρ(q\|r). By definition, α,ρ(θ;h,hsf,href)=ℛα,ρ(θ;h,hsf,href)−Dρ(q∥r). _α,ρ(θ;h,h_sf,h_ref)=R_α,ρ(θ;h,h_sf,h_ref)-D_ρ(q\|r). The Dρ(q∥r)D_ρ(q\|r) terms cancel, so we have α,ρ(θ;h,hsf,href)=−Dρ(πα⋆(⋅∣hsf,href)∥q)−ρ−1logZα. _α,ρ(θ;h,h_sf,h_ref)=-D_ρ\! ( _α (· h_sf,h_ref)\, \|\,q )- ρ-1 Z_α. Finally, restoring q=πθ(⋅∣h)q= _θ(· h) and Zα=Zα(hsf,href),Z_α=Z_α(h_sf,h_ref), we obtain α,ρ(θ;h,hsf,href)=−Dρ(πα⋆(⋅∣hsf,href)∥πθ(⋅∣h))−ρ−1logZα(hsf,href). _α,ρ(θ;h,h_sf,h_ref)=-D_ρ\! ( _α (· h_sf,h_ref)\, \|\, _θ(· h) )- ρ-1 Z_α(h_sf,h_ref). For fixed h, hsfh_sf, and hrefh_ref, both p(a)p(a) and r(a)r(a) are frozen during the inner student update. Therefore, Zα(hsf,href)=∑a∈p(a)αr(a)1−α Z_α(h_sf,h_ref)= _a p(a)^αr(a)^1-α does not depend on the inner optimization variable θ. Consequently, −ρ−1logZα(hsf,href)- ρ-1 Z_α(h_sf,h_ref) is constant with respect to θ. It follows that argmaxθα,ρ(θ;h,hsf,href) *argmax_θ\,J_α,ρ(θ;h,h_sf,h_ref) =argmaxθ[−Dρ(πα⋆(⋅∣hsf,href)∥πθ(⋅∣h))] = *argmax_θ [-D_ρ\! ( _α (· h_sf,h_ref)\, \|\, _θ(· h) ) ] =argminθDρ(πα⋆(⋅∣hsf,href)∥πθ(⋅∣h)). = *argmin_θD_ρ\! ( _α (· h_sf,h_ref)\, \|\, _θ(· h) ). This proves the stated conditional variational equivalence. When α=1α=1, Z1(hsf,href) Z_1(h_sf,h_ref) =∑a∈p(a)=1, = _a p(a)=1, and π1⋆(a∣hsf,href)=sg[πθ¯(a∣hsf)]. _1 (a h_sf,h_ref)=sg\! [ _ θ(a h_sf) ]. Hence, 1,ρ(θ;h,hsf,href) _1,ρ(θ;h,h_sf,h_ref) =−Dρ(sg[πθ¯(⋅∣hsf)]∥πθ(⋅∣h)). =-D_ρ\! (sg\! [ _ θ(· h_sf) ]\, \|\, _θ(· h) ). ∎ C.3 Proof of Proposition 3.3 Proof. By the definition of Rényi divergence, Dρ(πα⋆(⋅∣hsf,href)∥πθ(⋅∣h)) D_ρ\! ( _α (· h_sf,h_ref)\, \|\, _θ(· h) ) =1ρ−1log∑b∈πα⋆(b∣hsf,href)ρπθ(b∣h)1−ρ. = 1ρ-1 _b _α (b h_sf,h_ref)^ρ _θ(b h)^1-ρ. (22) Define Sα,ρ,θ:=∑b∈πα⋆(b∣hsf,href)ρπθ(b∣h)1−ρ. S_α,ρ,θ:= _b _α (b h_sf,h_ref)^ρ _θ(b h)^1-ρ. (23) Then Dρ=1ρ−1logSα,ρ,θ. D_ρ= 1ρ-1 S_α,ρ,θ. We first compute the derivative of the student probability with respect to the student logit. By the softmax definition, πθ(b∣h)=exp(zθ(b∣h))∑c∈exp(zθ(c∣h)). _θ(b h)= (z_θ(b h)) _c (z_θ(c h)). Therefore, ∂πθ(b∣h)∂zθ(a∣h) ∂ _θ(b h)∂ z_θ(a h) =a=bexp(zθ(b∣h))∑cexp(zθ(c∣h)) = 1\a=b\ (z_θ(b h)) _c (z_θ(c h)) −exp(zθ(b∣h))exp(zθ(a∣h))(∑cexp(zθ(c∣h)))2 - (z_θ(b h)) (z_θ(a h)) ( _c (z_θ(c h)) )^2 =πθ(b∣h)[a=b−πθ(a∣h)]. = _θ(b h) [1\a=b\- _θ(a h) ]. (24) Since πα⋆ _α is frozen during the student update, only πθ _θ depends on zθz_θ. Hence, ∂Sα,ρ,θ∂zθ(a∣h) ∂ S_α,ρ,θ∂ z_θ(a h) =∑b∈πα⋆(b∣hsf,href)ρ∂πθ(b∣h)1−ρ∂zθ(a∣h) = _b _α (b h_sf,h_ref)^ρ ∂ _θ(b h)^1-ρ∂ z_θ(a h) =(1−ρ)∑b∈πα⋆(b∣hsf,href)ρπθ(b∣h)−ρ∂πθ(b∣h)∂zθ(a∣h). =(1-ρ) _b _α (b h_sf,h_ref)^ρ _θ(b h)^-ρ ∂ _θ(b h)∂ z_θ(a h). Substituting Equation˜24 gives ∂Sα,ρ,θ∂zθ(a∣h) ∂ S_α,ρ,θ∂ z_θ(a h) =(1−ρ)∑b∈πα⋆(b∣hsf,href)ρπθ(b∣h)1−ρ =(1-ρ) _b _α (b h_sf,h_ref)^ρ _θ(b h)^1-ρ ×[a=b−πθ(a∣h)]. × [1\a=b\- _θ(a h) ]. Separating the two terms, ∂Sα,ρ,θ∂zθ(a∣h) ∂ S_α,ρ,θ∂ z_θ(a h) =(1−ρ)πα⋆(a∣hsf,href)ρπθ(a∣h)1−ρ =(1-ρ) _α (a h_sf,h_ref)^ρ _θ(a h)^1-ρ −(1−ρ)πθ(a∣h)∑b∈πα⋆(b∣hsf,href)ρπθ(b∣h)1−ρ -(1-ρ) _θ(a h) _b _α (b h_sf,h_ref)^ρ _θ(b h)^1-ρ =(1−ρ)[πα⋆(a∣hsf,href)ρπθ(a∣h)1−ρ−πθ(a∣h)Sα,ρ,θ]. =(1-ρ) [ _α (a h_sf,h_ref)^ρ _θ(a h)^1-ρ- _θ(a h)S_α,ρ,θ ]. (25) Next, differentiating the Rényi divergence gives ∂Dρ∂zθ(a∣h) ∂ D_ρ∂ z_θ(a h) =1ρ−11Sα,ρ,θ∂Sα,ρ,θ∂zθ(a∣h). = 1ρ-1 1S_α,ρ,θ ∂ S_α,ρ,θ∂ z_θ(a h). Substituting Equation˜25, ∂Dρ∂zθ(a∣h) ∂ D_ρ∂ z_θ(a h) =1−ρ−1[πα⋆(a∣hsf,href)ρπθ(a∣h)1−ρSα,ρ,θ−πθ(a∣h)]. = 1-ρ-1 [ _α (a h_sf,h_ref)^ρ _θ(a h)^1-ρS_α,ρ,θ- _θ(a h) ]. Since 1−ρ−1=−1, 1-ρ-1=-1, we obtain ∂Dρ∂zθ(a∣h) ∂ D_ρ∂ z_θ(a h) =πθ(a∣h)−πα⋆(a∣hsf,href)ρπθ(a∣h)1−ρSα,ρ,θ = _θ(a h)- _α (a h_sf,h_ref)^ρ _θ(a h)^1-ρS_α,ρ,θ =πθ(a∣h)−π~α,ρ,θ(a∣h,hsf,href). = _θ(a h)- π_α,ρ,θ(a h,h_sf,h_ref). Finally, by definition, πα⋆(a∣hsf,href)=sg[πθ¯(a∣hsf)]απref(a∣href)1−αZα(hsf,href). _α (a h_sf,h_ref)= sg\! [ _ θ(a h_sf) ]^α _ref(a h_ref)^1-αZ_α(h_sf,h_ref). Raising both sides to the power ρ gives πα⋆(a∣hsf,href)ρ=sg[πθ¯(a∣hsf)]αρπref(a∣href)(1−α)ρZα(hsf,href)ρ. _α (a h_sf,h_ref)^ρ= sg\! [ _ θ(a h_sf) ]^αρ _ref(a h_ref)^(1-α)ρZ_α(h_sf,h_ref)^ρ. Then, the common factor Zα(hsf,href)−ρZ_α(h_sf,h_ref)^-ρ appears in both the numerator and denominator and therefore cancels. Hence, π~α,ρ,θ(a∣h,hsf,href)∝ π_α,ρ,θ(a h,h_sf,h_ref) sg[πθ¯(a∣hsf)]αρ \! [ _ θ(a h_sf) ]^αρ ×πref(a∣href)(1−α)ρπθ(a∣h)1−ρ. × _ref(a h_ref)^(1-α)ρ _θ(a h)^1-ρ. ∎ Appendix D Extra Experimental Details D.1 Extra Results for Science Q&A Figure 4: Maj@16 validation accuracy over training. We report Maj@16 validation accuracy across training steps on five reasoning-related benchmarks for Qwen3-8B and Olmo3-7B-Instruct. Figure 5: Best@16 validation accuracy over training. We report Best@16 validation accuracy across training steps on five reasoning-related benchmarks for Qwen3-8B and Olmo3-7B-Instruct. D.2 Experimental Configurations Table 6: Hyperparameters for the Science Q&A and mathematical-reasoning experiments. Parameter SDPO (JSD) Science Q&A SR-OPSD (Ours) Science Q&A SDPO (RKL) Mathematics SR-OPSD (Ours) Mathematics General Benchmark SciKnowEval SciKnowEval AIME24, AIME25, HMMT25, AMC23, Minerva Model Qwen3-8B Qwen3-8B Qwen3-4B-Instruct-2507 Qwen3-4B-Instruct-2507 Primary evaluation metrics Avg@16 Avg@16 Avg@64 and Pass@64 Avg@64 and Pass@64 Evaluation thinking mode Disabled Disabled Enabled Enabled Data, rollout, and evaluation Maximum prompt length 2048 2048 – – Maximum training response length 8192 8192 16384 16384 Question batch / PPO mini-batch size 32 / 32 32 / 32 – – Per-device batch / gradient accumulation – – 1 / 1 1 / 1 Training rollouts per question 8 8 1 1 Training temperature / top-p / top-k 1.0 / 1.0 / −1-1 1.0 / 1.0 / −1-1 0.7 / 0.95 / 20 0.7 / 0.95 / 20 Evaluation samples per question 16 16 64 64 Evaluation temperature / top-p / top-k 0.6 / 0.95 / −1-1 0.6 / 0.95 / −1-1 0.7 / 0.95 / 20 0.7 / 0.95 / 20 Self-distillation objective Projection objective Jensen–Shannon Forward Rényi Reverse KL Forward Rényi Rényi order – 0.95 – 0.95 Self-reference coefficient – 0.90 – 0.90 Frozen-reference anchoring No Yes No Yes Reference policy – Frozen initial policy – Frozen initial policy Top-K distillation / tail bucket 100 / Yes 100 / Yes 100 / No 100 / No Teacher EMA update rate 0.05 0.05 0.05 0.05 Distillation IS clip 2.0 2.0 – – Token-loss clip – – 0.05 0.05 Optimization Optimizer AdamW AdamW AdamW AdamW Learning rate 1×10−51× 10^-5 1×10−51× 10^-5 5×10−65× 10^-6 5×10−65× 10^-6 Learning-rate schedule / warmup steps constant / 10 constant / 10 linear / 0 linear / 0 Weight decay 0.01 0.01 0 0 Gradient clipping norm 1.0 1.0 0.1 0.1 Table 7: Hyperparameters for the Qwen3 model-scaling experiments on LiveCodeBench v6. Training uses eight rollouts per prompt, whereas validation uses 16 sampled solutions per problem. Parameter GRPO SDPO SR-OPSD (Ours) General Models Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B Benchmark LiveCodeBench v6 Training and evaluation thinking mode Disabled Disabled Disabled Data, rollout, and validation Maximum prompt / response length 2048 / 8192 2048 / 8192 2048 / 8192 Question batch / PPO mini-batch size 32 / 32 32 / 32 32 / 32 Training rollouts per question 8 8 8 Training temperature / top-p / top-k 1.0 / 1.0 / −1-1 1.0 / 1.0 / −1-1 1.0 / 1.0 / −1-1 Validation frequency / samples per question 5 / 16 5 / 16 5 / 16 Validation temperature / top-p / top-k 0.6 / 0.95 / −1-1 0.6 / 0.95 / −1-1 0.6 / 0.95 / −1-1 Method-specific objective Policy/distillation objective GRPO Reverse KL Forward Rényi Environment feedback in reprompting – Yes Yes Rényi order – – 0.95 Self-reference coefficient – – 0.90 Frozen-reference anchoring – No Yes Reference policy – – Frozen initial policy Top-K distillation / tail bucket – 100 / Yes 100 / Yes Teacher EMA update rate – 0.05 0.05 Rollout IS correction / distillation IS clip token / – token / 2.0 token / 2.0 Optimization Optimizer AdamW AdamW AdamW Learning rate / schedule 1×10−51× 10^-5 / constant 1×10−51× 10^-5 / constant 1×10−51× 10^-5 / constant Warmup steps 10 10 10 Weight decay / gradient clipping norm 0.01 / 1.0 0.01 / 1.0 0.01 / 1.0 PPO epochs / loss aggregation 1 / token mean 1 / token mean 1 / token mean D.3 Extra Details of Ablation Study Table 8: Ablation Study on Science Q&A Physics. Model Method Reference ρ Physics Avg@16 Pass@16 Majority@16 Format Qwen3-8B JSD No – 79.4 88.3 81.7 99.8 JSD Yes – 77.2 84.6 79.4 99.8 FKL No – 79.1 91.9 81.8 100.0 FKL Yes – 76.3 82.3 78.3 100.0 SR-OPSD (Ours) Yes 0.50 79.9 88.9 80.0 100.0 SR-OPSD (Ours) Yes 0.70 80.6 82.4 81.5 100.0 SR-OPSD (Ours) Yes 0.95 81.3 86.8 82.4 100.0