Paper deep dive
Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization
Jiaqi Tian, Ting Long
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/12/2026, 6:27:45 AM
Summary
The paper identifies a capability degradation phenomenon in On-Policy Self-Distillation (OPSD) methods for Large Language Models (LLMs), where previously correct responses become incorrect during training despite overall performance gains. This degradation is linked to the deterioration of hidden representation geometry, specifically reduced intra-cluster compactness and inter-cluster separation. The authors propose Representation Geometry Regularization (RGR), a method that preserves representation geometry by encouraging compactness within clusters and separation across clusters, effectively mitigating capability degradation while retaining self-improvement benefits.
Entities (11)
Relation Signals (7)
Jiaqi Tian → authored → Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization
confidence 98% · Jiaqi Tian Affiliation: School of Artificial Intelligence Affiliation: Jilin University
Ting Long → authored → Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization
confidence 98% · Ting Long ††thanks: Corresponding author. Affiliation: School of Artificial Intelligence Affiliation: Jilin University
Representation Geometry Regularization → mitigates → Capability degradation
confidence 96% · RGR generally alleviates capability degradation while retaining capability improvements.
On-Policy Self-Distillation → causes → Capability degradation
confidence 95% · we find that OPSD methods can progressively degrade previously acquired capabilities
Capability degradation → ischaracterizedby → Deterioration of Representation Geometry
confidence 93% · this degradation coincides with a deterioration of the geometry of hidden representations, characterized by reduced intra-cluster compactness and diminished inter-cluster separation.
Representation Geometry Regularization → preserves → Intra-cluster Compactness
confidence 90% · RGR ... preserving representation geometry during post-training by encouraging compactness within cluster
Representation Geometry Regularization → preserves → Inter-cluster Separation
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-Policy Self-distillation is a self-improvement paradigm that enables large language models (LLMs) to improve their capabilities by leveraging the same model as both teacher and student under different conditioning contexts. However, we find that OPSD methods can progressively degrade previously acquired capabilities: queries that are correctly responded at earlier training stages may become incorrectly responded as optimization proceeds, even the overall performance continues to improve. Further analysis reveals that this degradation coincides with a deterioration of the geometry of hidden representations, characterized by reduced intra-cluster compactness and diminished inter-cluster separation. Motivated by this observation, we propose Representation Geometry Regularization (RGR), which preserves representation geometry during post-training by encouraging compactness within cluster and separation across cluster. RGR can be seamlessly integrated into existing OPSD methods without altering their core optimization procedures. Extensive experiments on multiple OPSD methods and benchmarks show that RGR generally alleviates capability degradation while retaining capability improvements. Our code is available in this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.12855v1
- Canonical: https://arxiv.org/abs/2610.12855v1
Trouble viewing inline? Open PDF directly →
Full Text
64,837 characters extracted from source content.
Expand or collapse full text
Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization Jiaqi Tian Affiliation: School of Artificial Intelligence Affiliation: Jilin University Affiliation: China Email: tjq26@mails.jlu.edu.cn Ting Long †thanks: Corresponding author. Affiliation: School of Artificial Intelligence Affiliation: Jilin University Affiliation: China Email: longting@jlu.edu.cn Abstract On-Policy Self-distillation is a self-improvement paradigm that enables large language models (LLMs) to improve their capabilities by leveraging the same model as both teacher and student under different conditioning contexts. However, we find that OPSD methods can progressively degrade previously acquired capabilities: queries that are correctly responded at earlier training stages may become incorrectly responded as optimization proceeds, even the overall performance continues to improve. Further analysis reveals that this degradation coincides with a deterioration of the geometry of hidden representations, characterized by reduced intra-cluster compactness and diminished inter-cluster separation. Motivated by this observation, we propose Representation Geometry Regularization (RGR), which preserves representation geometry during post-training by encouraging compactness within cluster and separation across cluster. RGR can be seamlessly integrated into existing OPSD methods without altering their core optimization procedures. Extensive experiments on multiple OPSD methods and benchmarks show that RGR generally alleviates capability degradation while retaining capability improvements. Our code is available in https://github.com/supermachinelearner/RGR 1 Introduction Figure 1: (a) The capability degradation ratio of existing methods; (b) Although the overall performance improves during training, some problems that were previously responded correctly become incorrectly answered as optimization progresses. On-Policy Self-Distillation (OPSD) (Zhao et al., 2026; Hübotter et al., 2026; Yang et al., 2026; Li et al., 2026), is a self-improvement paradigm for the post-training of large language models (LLMs) (Brown et al., 2020; Vaswani et al., 2017; Yang et al., 2025; Zeng et al., 2026; Xiao et al., 2026; Xu et al., 2026), which enables LLMs to improve their capabilities through self-knowledge distillation. Unlike conventional knowledge distillation(Hinton et al., 2015; Sanh et al., 2019), which relies on a separately trained teacher, OPSD uses the same LLM as both teacher and student under different conditioning contexts. Specifically, the student model generates responses to the original query, while teacher model, which share the same parameters with the student model, conditioned on additional privileged information (e.g., verified reasoning traces) to provide supervision for these generated responses. Although recent studies (Zhao et al., 2026; Hübotter et al., 2026; Yang et al., 2026; Li et al., 2026; Wang et al., 2026) have empirically demonstrated that optimizing the LLM with OPSD can outperform conventional post-training methods such as SFT (Wei et al., 2021) and GRPO (Shao et al., 2024), benefiting from the dense and fine-grained token-level supervision of OPSD. We observe a progressive reasoning capability degradation phenomenon during self-improvement in existing OPSD methods. As shown in Figure 1(b), a query that is responded correctly at earlier training stages gradually becomes incorrectly responded as training proceeds, even as overall performance continues to improve. Furthermore, we examine several representative OPSD methods and find that this phenomenon consistently occurs across these methods, albeit to varying degrees, as it is shown in Figure 1 (a). Such degradation may undermine the stability of self-improvement. To better understand this phenomenon, we conduct further analysis, and the results reveal that this capability degradation is accompanied by a progressive deterioration in the geometry of hidden representations. When a query is answered correctly at earlier stages of training, the hidden representations of its generated responses exhibit a relatively well-structured clustering pattern, characterized by larger inter-cluster distances and clearer cluster boundaries. As training proceeds and the response to the same query gradually becomes incorrect, the relative distances between clusters shrink and the cluster boundaries become increasingly indistinct. This observation suggests that preserving the geometric structure of the representation space may be important for maintaining previously acquired capabilities during self-improvement. Based on this finding, we theoretically show that preserving intra-cluster compactness and inter-cluster separation in the hidden representation space helps maintain the capabilities that the LLM has already acquired. Based on this theoretical insight, we develop Representation Geometry Regularization (RGR), which constrains the evolution of hidden representations during self-improvement. RGR incorporates two geometry-preserving constraints called intra-cluster compactness and inter-cluster separation into the OPSD methods, enabling the model to improve targeted capabilities while mitigating the degradation of previously acquired ones. Experiments across four existing SD models and three widely used benchmarks show that RGR substantially reduces correct-to-incorrect transitions during training while retaining its self-improvement gains. In summary, our contributions are summarized as follows: • We uncover an overlooked failure mode, where OPSD methods can degrade previously acquired capabilities, causing previously correct predictions to become incorrect. • We identify representation geometry distortion as a key factor underlying this degradation, and theoretically characterize the importance of hidden representation geometry for prediction stability. • We propose RGR, which incorporates geometry-preserving constraints into OPSD methods to enable self-improvement while reducing capability degradation. 2 Related works On-policy self-distillation (OPSD) enables LLMs to improve their capabilities without requiring an external model as a teacher. Recently, it has attracted significant attention as a promising paradigm for scalable and efficient LLM self-improvement. Based on how self-teacher signals are constructed and how external feedback is incorporated, we organize existing OPSD methods into privileged-information-driven methods and external-feedback-driven methods.The privileged-information-driven methods construct self-teacher signals using additional privileged information that is unavailable to the student policy at inference time. As a representative method, OPSD (Zhao et al., 2026) uses the same model to construct self-teacher signals and optimize the student policy under different conditioning contexts: the self-teacher signals are constructed from both the question and a verified reasoning trace, whereas the student policy is conditioned only on the question. The external-feedback-driven methods retain the self-teacher structure while incorporating environmental feedback, verification results, or outcome rewards to construct the supervision for policy updating. Within this category, feedback can be used to construct distillation targets, select training objectives, or modulate token-level advantages. SDPO (Hübotter et al., 2026) conditions the current model on textual or reward-derived feedback to form a self-teacher, whose next-token predictions serve as distillation targets for the student policy. SRPO (Li et al., 2026)routes successful trajectories to reward-driven GRPO updates and failed trajectories to entropy-weighted, token-level self-distillation against a feedback-conditioned teacher distribution. RLSD (Yang et al., 2026) uses verifiable outcomes to determine the direction of policy updates and token-level teacher–student discrepancies to modulate their magnitude, integrating self-distillation into reward-driven learning. 3 Preliminaries Given a query q, an LLM πθ _θ typically generates a response y=(y1,…,yT)y=(y_1,…,y_T) according to yt∼πθ(⋅∣q,y<t)y_t _θ(· q,y_<t), where y<t=(y1,…,yt−1)y_<t=(y_1,…,y_t-1) denotes the previously generated tokens. OPSD methods improve the LLM by using the same model under different conditioning contexts as the teacher and student to conduct knowledge distillation. Specifically, the student is designed to generate an on-policy response based only on the original query, like the LLM, i.e., y∼πθ(⋅∣q)y _θ(· q). While the teacher is designed to generate the response with additional privileged information z, such as a verified reasoning trace, πθ(⋅∣q,z,y<t) _θ(· q,z,y_<t). The OPSD methods optimize the model by minimizing the divergence between the teacher and student token distributions along the student’s trajectory: ℒOPSD(θ)=q∼,y∼πθ(⋅∣q)[∑t=1TD(πθ(⋅∣q,z,y<t)∥πθ(⋅∣q,y<t))].L_OPSD(θ)=E_q ,\,y _θ(· q) [ _t=1^TD ( _θ(· q,z,y_<t)\, \|\, _θ(· q,y_<t) ) ]. (1) where D(⋅|⋅)D(·|·) denotes a divergence measure between the teacher and student token distributions. D can be instantiated by forward KL divergence (Kullback and Leibler, 1951), reverse KL divergence (Minka et al., 2005), Jensen–Shannon divergence (Lin, 1991), or other suitable distribution-matching objectives. Since the teacher and student share the underlying model parameters, optimizing ℒOPSDL_OPSD updates the LLM itself. Consequently, the model iteratively improves its responses by distilling the additional information available to the teacher into its ordinary, query-conditioned behavior. 4 The Geometry of Capability Degradation 4.1 Progressive Capability Degradation Figure 2: Hidden-state representations and response correctness across training epochs. The query is answered correctly at epochs 25 and 50, but incorrectly at epoch 75. As discussed previously, although OPSD methods continuously improve the overall performance in the training, some queries that are correctly responded at earlier training stages become incorrectly responded as training proceeds. To better understand this phenomenon, for a fixed query, we generate a response and visualize the hidden representations corresponding to the generated response every 25 training epochs. We observe that changes in the geometry of the hidden representations consistently coincide with changes in response correctness. As illustrated in Figure. 2, when the model generates a correct response at an earlier training stage (Epoch 25), the corresponding hidden representations exhibit a relatively structured geometric organization, with clearly distinguishable clusters and relatively large distances between clusters. As training proceeds and the response becomes incorrect, the relative distances between clusters gradually decrease, while the boundaries between clusters become increasingly indistinct (Epoch 75). We hypothesize that as a query transitions from being correctly to incorrectly responded, the reasoning process in the generated response gradually becomes less coherent. When the query is answered correctly, the reasoning tends to exhibit a clear and consistent logical structure, which is reflected in the hidden representation space as distinct and well-separated clusters corresponding to different reasoning patterns. As the reasoning becomes increasingly disorganized, these patterns become less distinguishable, causing the corresponding representations to become more entangled and eventually leading to an incorrect response. Interestingly, this phenomenon is consistently observed in existing OPSD methods, such as OPSD(Zhao et al., 2026), SDPO(Hübotter et al., 2026), RLSD(Yang et al., 2026), SRPO(Li et al., 2026). These observations suggest a strong connection between the geometry of the hidden representation and the stability of the model’s acquired capabilities during self-improvement. 4.2 Theoretical Analysis for Capability Degradation To further substantiate the relationship between hidden representation geometry and response correctness, we conduct a theoretical analysis. Suppose the LLM undergoing on-policy self-distillation is denoted by πθ(⋅∣q,y<t) _θ(· q,y_<t), where πθ∈ΠOPSD _θ∈ _OPSD, and ΠOPSD _OPSD denote the policy set trained using OPSD. Let πθ∗(⋅∣q,z,y<t) _θ*(· q,z,y_<t) denote the optimal teacher distribution conditioned on the privileged information z, and let πθ∗(⋅∣q,y<t) _θ*(· q,y_<t) denote the corresponding optimal student distribution without access to z. We define the excess distillation risk as follows. Definition 4.1 (The Excess Distillation Risk). The excess distillation risk of the student model πθ(⋅∣q,y<t) _θ(· q,y_<t) is defined as the difference between its distillation risk with respect to the optimal teacher and that achieved by the optimal student, i.e., ℰKD(πθ)=q∼,y∼πθ(⋅∣q)[∑t=1TD(πθ∗(⋅∣q,z,y<t)∥πθ(⋅∣q,y<t)) _KD( _θ)=E_q ,y _θ(· q) [ _t=1^TD ( _θ^*(· q,z,y_<t)\, \|\, _θ(· q,y_<t) ) (2) −D(πθ∗(⋅∣q,z,y<t)∥πθ∗(⋅∣q,y<t))]. -D ( _θ^*(· q,z,y_<t)\, \|\, _θ^*(· q,y_<t) ) ]. This quantity measures the gap between the distillation risk of the current student and that of the optimal student, with a smaller value indicating a closer approximation to the optimal. Let ℋ^=^ii=1T H=\ h_i\_i=1^T and ℋ∗=i∗i=1TH^*=\h_i^*\_i=1^T denote the hidden representations produced by the current teacher πθ(⋅∣q,z,y<t) _θ(· q,z,y_<t) and the optimal teacher πθ∗(⋅∣q,z,y<t) _θ^*(· q,z,y_<t), respectively. Their midpoint representations are denoted by ℋ†=i†i=1TH =\h_i \_i=1^T, where i†=(^i+i∗)2h_i = ( h_i+h_i^*)2. Suppose that ℋ†H is partitioned into K clusters, denoted by †=C1†,…,CK†C =\C_1 ,…,C_K \. The center of is cluster Ck†C_k is μk†=1|Ck†|∑i†∈Ck†i† _k = 1|C_k | _h_i ∈ C_k h_i . Let ℋ=ii=1TH=\h_i\_i=1^T denote the student hidden representations corresponding to the current-teacher representations ^ii=1T\ h_i\_i=1^T. For each cluster Ck†C_k , we define k=i∣i†∈Ck†X_k=\h_i _i ∈ C_k \. That is, if the midpoint i†h_i associated with a current-teacher representation ^i h_i belongs to Ck†C_k , its corresponding student representation ih_i is assigned to kX_k. The center of is cluster kX_k is μk=1|k|∑hi∈ki _k^X= 1|X_k| _h_i _kh_i. Let μ=1T∑iTiμ= 1T _i^Th_i denote the mean of hidden representations. Based on the definition, we have Theorem 4.1 (Excess Risk Upper Bound). ℰKD(πθ)≤2‖M‖2N||∑Nsupπθ∈ΠOPSD|Δ|[Td2+12∑|†|∑i∈k‖i−μk‖22−12∑|†||k†|‖μk−μ‖22+ _KD( _θ)≤ 2||M||_2N|V| _N _ _θ∈ _OPSD| |[ Td2+ 12 _|C | _h_i _k||h_i- _k^X||_2^2- 12 _|C ||C _k||| _k^X-μ||^2_2+ (3) ∑|†||k†|‖μk−μk†‖22+∑|†|∑i†∈k‖i†−μk†‖22] _|C ||C_k ||| _k^X- _k ||^2_2+ _|C | _h_i _k||h_i - _k ||_2^2] where N denote the number of samples, |Δ|=| |= ∑i‖i^−i∗‖22 _i|| h_i-h_i^*||_2^2, M=WT(I||−1||||||T)WM=W^T(I_|V|- 1|V|1_|V|1_|V|^T)W. The detail proof could be found in Appendix D From Theorem 4.1, we observe that the excess risk upper bound is affected by both intra-cluster compactness and inter-cluster separation of the learned representations. Specifically, reducing the intra-cluster variance term ‖i−μk‖22\|h_i- _k^X\|_2^2 and increasing the inter-cluster dispersion term ‖μk−μ‖22\| _k^X-μ\|_2^2 can tighten the excess risk upper bound. Meanwhile, the term ‖μk−μk†‖22\| _k^X- _k \|_2^2 reflects the deviation between the learned student representation and the teacher and optimal teacher representation. Therefore, preventing capability degradation during OPSD requires not only maintaining the alignment with the teacher representation, but also preserving compact intra-cluster compactness and clear inter-cluster separation in the representation space. 5 Methodology As discussed previously, we observe that increasingly blurred boundaries between hidden representation clusters coincide with the progressive degradation of model capabilities during OPSD training. Motivated by this observation, we propose Representation Geometry Regularization (RGR) to preserve clear boundaries between hidden representation clusters. To achieve this goal, we consider two complementary geometric properties: (1) intra-cluster compactness, where representations within each cluster remain sufficiently compact, and (2) inter-cluster separation, where different clusters maintain sufficient separation from each other. In the following, we discuss how to preserve the two properties in detail and derive the corresponding regularization objectives. 5.1 Intra-cluster compactness According to previous works , the intra-cluster compactness of hidden representations can be evaluated by the Within-Cluster Sum of Squares (WCSS)(Lloyd, 1982), defined as the sum of squared distances between each hidden representation and the center of its corresponding cluster. However, directly minimizing WCSS may lead to excessive compactness within each cluster. In the extreme case, hidden representations within the same cluster may become overly similar, reducing the discriminative information encoded in the representation space. Therefore, we introduce the teacher model as a reference and encourage the WCSS of the student representations within each cluster to remain close to that of the teacher representations. Given the query q, the hidden representations of the response generated by the student model are denoted as 1,…,T\h_1,…,h_T\, while the corresponding hidden representations of the reference model are denoted as 1ref,…,Tref\h_1^ref,…,h_T^ref\. Suppose that a cluster CkrefC^ref_k in the reference model’s representation space corresponds to a cluster CkalignC_k^align in the student model’s representation space, with cluster centers krefu_k^ref and kalignu_k^align, respectively.To align the intra-cluster compactness of the student model with that of the reference model, the learning target for intra-cluster compactness is defined as: ℒintra=∑|ref||∑i∈Ckalign,‖i−kalign‖22−∑iref∈Ckref‖iref−kref‖22|,L_intra= _|C_ref| | _h_i∈ C_k^align, \|h_i- μ_k^align \|_2^2- _h_i^ref∈ C_k^ref \|h_i^ref- μ_k^ref \|_2^2 |, (4) where |ref||C_ref| denotes the number of clusters. However, directly optimizing Eq. 17 is difficult. To address this, we stack the hidden representations of the student model as =[1,…,T]∈ℝT×dH=[h_1,…,h_T] ^T× d, where each hidden representation is the row vector of H. Similarly, we stack the hidden representations of the student model as ref=[1ref,…,Tref]H_ref=[h_1^ref,…,h_T^ref], where each hidden representation is the row vector of refH_ref. Under the above definitions and assumptions, we have: Theorem 5.1 (Intra-cluster Compactness Upper Bound). The upper bound of the intra-cluster compactness learning target is ℒintra≤(T−|ref|)12‖⊤−refrefT‖F,L_intra≤(T-|C_ref|) 12||HH -H_refH_ref^T||_F, (5) where |ref||C_ref| denotes the number of clusters of the reference model. The details of the proof can be found in Appendix E. 5.2 Inter-cluster separation We construct a Laplacian matrix(Von Luxburg, 2007) from the pairwise relationships among all hidden representations and use its low-end spectrum, corresponding to the smallest eigenvalues of the Laplacian matrix, to evaluate the separability between clusters. However, directly maximizing these low-end eigenvalues is undesirable, as excessively large inter-cluster separation can over-expand the representation space and distort the relative geometric relationships among clusters. Therefore, we use the teacher model as a reference model to encourage the student model to preserve its inter-cluster separation, thereby preventing undesirable geometric distortion during optimization. Specifically, given the query q, the hidden representations of the response generated by the student model are denoted as 1,…,T\h_1,…,h_T\, while the corresponding hidden representations of the reference model are denoted as 1ref,…,Tref\h_1^ref,…,h_T^ref\. We define the Gaussian kernel matrices of the hidden representations of the student model and the reference model as: Aij A_ij =exp(−‖i−j‖222σ2),i≠j,0,i=j,Aijref = cases (- \|h_i-h_j \|_2^22σ^2 ),&i≠ j,\\ 0,&i=j, cases A_ij^ref =exp(−‖iref−jref‖222σ2),i≠j,0,i=j, = cases (- \|h_i^ref-h_j^ref \|_2^22σ^2 ),&i≠ j,\\ 0,&i=j, cases (6) where σ denotes the standard deviation of the Gaussian kernel. The corresponding degree matrices of the student and reference Gaussian kernel matrices are denoted by D and refD^ref, respectively, where Dii=∑j=1TAijD_i= _j=1^TA_ij and Diiref=∑j=1TAijrefD_i^ref= _j=1^TA_ij^ref. The unnormalized Laplacian matrices of the student and reference hidden representations are then defined as =−L=D-A and ref=ref−refL^ref=D^ref-A^ref, respectively. The corresponding eigenvalue vectors are λ =[λ1(),…,λT()]⊤, = [ _1(L),…, _T(L) ] , (7) ref λ^ref =[λ1(ref),…,λT(ref)]⊤, = [ _1(L^ref),…, _T(L^ref) ] , where the eigenvalues are arranged in nondecreasing order. The learning target for inter-cluster separation could be represented by the difference between the eigenvalue vectors of the student and the reference, that is ℒinter=‖−ref‖2.L_inter= \| λ- λ^ref \|_2. (8) Then, by the Hoffman–Wielandt inequality (Hoffman et al., 1953), we obtain ℒinter≤(1+T)‖−‖FL_inter≤(1+ T)||L-L^ref||_F (9) Under the above definitions and assumptions, we have Theorem 5.2 (Inter-cluster Separation Upper Bound). The upper bound of the inter-cluster separation could be induced by applying Taylor expansion and the Newton–Schwarz inequality: ℒinter≤(1+Tσ)2‖⊤−refref⊤‖F,L_inter≤ ( 1+ Tσ )^2 \|HH -H_refH_ref \|_F, (10) The proof is provided in Appendix F. 5.3 Optimization Building upon the previous analysis, preserving the intra-cluster compactness and inter-cluster separation of the hidden representation space requires minimizing ℒintraL_intra and ℒinterL_inter. From the above theoretical analysis, we show that both geometric discrepancies can be controlled by the discrepancy between the Gram matrices of the student and reference hidden representations. Specifically, we have ℒintra≤Cintra‖⊤−refref⊤‖FL_intra≤ C_intra \|HH -H_refH_ref \|_F and ℒinter≤Cinter‖⊤−refref⊤‖F,L_inter≤ C_inter \|HH -H_refH_ref \|_F, where CintraC_intra and CinterC_inter are constants independent of the student representations. Therefore, minimizing the Gram matrix discrepancy provides a unified objective for preserving both intra-cluster compactness and inter-cluster separation. We thus define the geometry-preserving objective as ℒgeo=‖⊤−refref⊤‖F2.L_geo= \|HH -H_refH_ref \|_F^2. (11) In addition to representation geometry, recent studies (Jin et al., 2026) on on-policy distillation have highlighted the importance of controlling the entropy of the model output distribution during optimization,which motivates us to explicitly incorporate an entropy-related objective into the optimization process. Accordingly, we jointly preserve the hidden representation geometry and regulate the output distribution entropy during OPSD optimization. Combining the original OPSD objective with the geometry-preserving and entropy-related objectives, we formulate the overall training objective as ℒ=ℒOPSD+αℒgeo+βℒent,L=L_OPSD+ _geo+ _ent, (12) where ℒOPSDL_OPSD denotes the original on-policy self-distillation objective, ℒgeoL_geo denotes the proposed geometry-preserving regularization, and ℒent=‖⊤−ref⊤ref‖F2L_ent= \|H H-H_ref H_ref \|_F^2 denotes the entropy-related objective. The coefficients α and β control the contributions of the geometry-preserving and entropy-related terms, respectively. 6 EXEPERIMENT In this section, we conduct experiments to evaluate the performance of RGR. We aim to answer the following research questions: (1) Is RGR effective for improving the performance of self distillation (RQ1)? (2) How do different components of RGR contribute to the performance gains (RQ2)? (3) Can RGR alleviate the degradation of previously acquired capabilities during OPSD optimization, and how does it affect the evolution of hidden representations (RQ3)? (4) Can RGR generalize to task-transfer settings and alleviate the degradation of previously acquired capabilities when learning new tasks (RQ4)? 6.1 Experiment setup Datasets. Following the dataset setup of OPSD (Zhao et al., 2026), we sample up to 30K problem-solution pairs containing chain-of-thought reasoning from the mathematical reasoning subset of OpenThoughts(Guha et al., )for model training.. We conduct evaluation on three representative benchmarks: AIME 2024 (Zhang and Math-AI, 2024), AIME 2025 (Zhang and Math-AI, 2025), and HMMT 2025 (Balunovic et al., 2026). AIME 2024 consists of problems from the 2024 American Invitational Mathematics Examination, covering diverse areas of high-school mathematics. AIME 2025 consists of problems from the 2025 American Invitational Mathematics Examination. HMMT 2025 consists of problems from the Harvard-MIT Mathematics Tournament, covering algebra, combinatorics, geometry, and number theory. Baseline. To assess the effectiveness of RGR, we integrate it into four representative self-distillation methods, namely, OPSD (Zhao et al., 2026), SDPO (Hübotter et al., 2026), SRPO (Li et al., 2026), and RLSD (Yang et al., 2026), and compare each RGR-enhanced method with its original counterpart. These methods differ in how they optimize the self-distillation objective. Specifically, OPSD employs forward KL divergence (Kullback and Leibler, 1951), with the policy update constrained by token-level KL clipping; SDPO uses Jensen–Shannon divergence (Lin, 1991) to align the student with the teacher; SRPO routes successful responses to reward-based reinforcement and failed responses to self-distillation; and RLSD uses environment-derived rewards to determine the direction of policy updates, while reweighting token-level advantages according to the teacher–student log-probability gap to modulate the update magnitude. ImplementationDetails. Our experiments are conducted on two foundation LLMs: Qwen3-1.7B and Qwen3-4B(Yang et al., 2025), with thinking mode enabled throughout training and validation. All experiments are performed on four NVIDIA A40 GPUs. All methods use LoRA (Hu et al., 2021) with a rank of 64 and a scaling factor of 128, with the learning rate set to 5 × 10−610^-6. The teacher model remains frozen throughout training. For our method, we fix the parameter α in Eq.(12) at 2.5×10−22.5× 10^-2 and the parameter β in Eq.(12) at 5×10−25× 10^-2 across all benchmarks. 6.2 Main results (RQ1) Table 1: Performance gains from RGR. Best results in bold; RGR improvements in green. Model Method AIME24 AIME25 HMMT25 Avg. Qwen3-1.7B OPSD 56.67 43.06 28.89 42.87 OPSD + RGR 59.89 43.60 30.00 44.50+1.63 SDPO 38.61 28.61 21.39 29.54 SDPO + RGR 41.94 31.39 21.94 31.76+2.22 RLSD 50.27 39.70 25.83 38.60 RLSD + RGR 51.39 39.17 25.56 38.70+0.10 SRPO 37.78 30.00 20.00 29.26 SRPO + RGR 39.44 29.72 21.67 30.28+1.02 Qwen3-4B OPSD 74.17 68.06 45.00 62.41 OPSD + RGR 75.56 70.56 47.50 64.54+2.13 SDPO 67.22 61.39 38.61 55.74 SDPO + RGR 72.50 63.61 41.11 59.07+3.33 RLSD 75.00 65.00 44.44 61.48 RLSD + RGR 74.72 66.11 44.72 61.85+0.37 SRPO 72.22 64.16 39.17 58.52 SRPO + RGR 73.33 66.67 41.67 60.56+2.04 Following OPSD (Zhao et al., 2026), we evaluate each model by generating 12 responses for each problem and report the average accuracy over the 12 responses (Avg@12). The results are reported in Table 1. As it is shown in Table 1, we can observe that: (1) RGR generally improves different methods across both Qwen3-1.7B and Qwen3-4B. OPSD, SDPO, and SRPO consistently benefit from RGR, demonstrating that representation geometry preservation provides complementary benefits to different optimization objectives. Although a few settings show slight decreases, RGR improves the majority of cases, supporting its general effectiveness; (2) The gains vary across different methods, with larger improvements generally observed for OPSD and SDPO but limited gains for RLSD. This can be attributed to their different optimization mechanisms: OPSD and SDPO directly rely on self-distillation signals for dense token-level supervision, while RLSD mainly uses self-distillation to determine the update magnitude and obtains the update direction from RLVR. Consequently, RGR has more room to benefit methods whose optimization induces stronger changes in the representation space; (3) The improvements also vary across AIME24, AIME25, and HMMT25. This variation may be attributed to the different difficulty levels and reasoning characteristics of these benchmarks, which result in different distributions of correct and incorrect responses during self-improvement. Overall, RGR remains beneficial across diverse mathematical reasoning benchmarks, while its improvement varies with the characteristics of the target dataset. 6.3 Ablation Study (RQ2) To investigate the contribution of each component to RGR, we conduct an ablation study with the following variants: • w/o Ent: We remove the ℒentL_ent in Eq.(12), removing the constraint on output distribution. • w/o Geo: We remove the ℒgeoL_geo in Eq.(12), eliminating the constraint on intra-cluster compactness and inter-cluster separation. Table 2: Ablation study of RGR. Method AIME24 AIME25 HMMT25 OPSD 56.67 43.06 28.89 w/o Ent 57.22 41.39 28.33 w/o Geo 58.61 42.22 29.17 RGR 59.89 43.60 30.00 The results are shown in Table 2 that RGR achieves the best performance across all three benchmarks. From Table 2, we can also observe that: (1) Removing the geometry constraint (w/o Geo) consistently degrades the performance across all benchmarks, compared with the full RGR. This indicates that explicitly preserving intra-cluster compactness and inter-cluster center separation is crucial for maintaining a structured representation space during OPSD optimization. Without this constraint, the hidden representations may become less organized, leading to limited capability improvement. Figure 3: Comparison of the capability degradation ratio between the baseline and the baseline+RGR model on AIME24 (%). Figure 4: Visualization of hidden state representations across different models and training steps, with response correctness indicated. (2) Removing the entropy constraint (w/o Ent) also leads to performance drops on one benchmarks. This suggests that entropy regularization provides complementary optimization benefits by further stabilizing the output distribution during self-distillation. However, the performance gap between RGR and w/o Geo is larger than that between RGR and w/o Ent, indicating that representation geometry preservation serves as the primary contributor to the improvements. 6.4 Training Dynamics Visualization (RQ3) Effect of RGR on mitigating progressive capability degradation. To evaluate whether RGR alleviates progressive capability degradation, we compare the ratio of queries that transition from correct to incorrect responses before and after incorporating RGR into different OPSD variants. As shown in Figure 3, adding RGR consistently reduces the degradation ratio across all evaluated methods. Specifically, RGR eliminates the degradation in OPSD and reduces the degradation ratios of SDPO, RLSD, and SRPO by 14.28%, 7.15%, and 21.43%, respectively. These results demonstrate that RGR not only improves self-improvement performance but also helps preserve previously acquired capabilities by maintaining a more stable hidden representation geometry during optimization. Representation geometry analysis. To further understand how RGR alleviates progressive capability degradation, we visualize the hidden representations of responses corresponding to the degraded queries. Specifically, we select queries that are initially responded correctly but become incorrectly responded during optimization, and compare the representation geometry before and after incorporating RGR. As shown in Figure 4, without RGR, the hidden representations gradually lose their geometric structure, exhibiting reduced inter-cluster center separation and less distinguishable cluster boundaries when the response changes from correct to incorrect. In contrast, RGR maintains more compact clusters with clearer boundaries in most cases. These results are consistent with our theoretical analysis and demonstrate that preserving representation geometry helps stabilize the evolution of hidden representations during OPSD training. Table 3: Forgetting evaluation on AIME24 (%). Before denotes the performance before training on the tool-use dataset, while After denotes the performance after tool-use training. Method Before After Forgetting ↓ OPSD 56.67 54.72 1.95 OPSD + RGR 59.89 58.06 1.83 SDPO 38.61 33.33 5.28 SDPO + RGR 41.94 40.00 1.94 RLSD 50.27 47.22 3.05 RLSD + RGR 51.39 48.00 3.39 SRPO 37.78 34.72 3.06 SRPO + RGR 39.44 38.33 1.11 Overall, RGR achieves consistent reductions in capability degradation across OPSD, SDPO, RLSD, and SRPO, despite these methods adopting different self-distillation objectives. This suggests that RGR serves as a general geometry-preserving regularization mechanism that can be integrated into different OPSD methods. 6.5 Cross-Task Forgetting Study (RQ4) To further investigate whether RGR can mitigate capability degradation beyond the original task, we conduct a cross-task forgetting experiment. We first post-train a model on OpenThoughts to acquire mathematical problem-solving capabilities, and evaluate the model on AIME24 to measure performance. Then we continue post-training it on a tool-use dataset(Tang et al., 2023), after convergence, we evaluate the model again on AIME24 to measure the performance after tool-use training. We apply this setting to OPSD, SDPO, RLSD, and SRPO, and compare their performance before and after training on the tool-use dataset, as shown in Table 3. From Table 3, we can observe that: (1) Sequential training on a new task leads to forgetting of previously acquired capabilities. After tool-use training, all four methods exhibit performance degradation on AIME24, consistent with previous studies of catastrophic forgetting in continual fine-tuning of LLMs (Luo et al., 2025; Li et al., 2024). (2) RGR generally alleviates cross-task forgetting. RGR reduces the forgetting gap for OPSD, SDPO, and SRPO, with particularly notable improvements for SDPO (5.28 → 1.94) and SRPO (3.06 → 1.11). This suggests that preserving representation geometry during subsequent optimization can help retain previously acquired capabilities. (3) The effectiveness of RGR depends on the underlying post-training method. For RLSD, the forgetting gap slightly increases from 3.05 to 3.39 after introducing RGR, suggesting that the effect of representation geometry preservation may vary across different optimization dynamics. Nevertheless, RGR reduces forgetting for three of the four methods, supporting its broader applicability to sequential post-training. 7 Conclusion In this work, we identify and investigate the progressive capability degradation phenomenon in on-policy self-distillation, where LLM may lose previously acquired capabilities during self-improvement despite overall performance improvements. Through empirical analysis and theoretical investigation, we reveal that this degradation is associated with the deterioration of hidden representation geometry, characterized by reduced intra-cluster compactness and inter-cluster center separation. Based on these findings, we propose Representation Geometry Regularization (RGR), a general regularization framework that preserves intra-cluster compactness and inter-cluster center separation during OPSD optimization. Extensive experiments across multiple OPSD variants and mathematical reasoning benchmarks demonstrate that RGR effectively reduces correct-to-incorrect transitions while maintaining the self-improvement capability of OPSD. Future work will explore more adaptive strategies for preserving representation geometry and extending the proposed framework to broader model optimization paradigms beyond self-distillation. References Balunovic et al. (2026) Mislav Balunovic, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions. Advances in Neural Information Processing Systems, 38, 2026. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. (3) Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models, 2025. URL https://arxiv. org/abs/2506.04178. Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. Hoffman et al. (1953) Alan J Hoffman, Helmut W Wielandt, et al. The variation of the spectrum of a normal matrix. Duke Math. J, 20(1):37–39, 1953. Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. Hübotter et al. (2026) Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026. Jin et al. (2026) Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079, 2026. Kullback and Leibler (1951) Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86, 1951. Li et al. (2026) Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. Unifying group-relative and self-distillation policy optimization via sample routing. arXiv preprint arXiv:2604.02288, 2026. Li et al. (2024) Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. Revisiting catastrophic forgetting in large language model tuning. In Findings of the association for computational linguistics: EMNLP 2024, pages 4297–4308, 2024. Lin (1991) Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145–151, 1991. Lloyd (1982) Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28(2):129–137, 1982. Luo et al. (2025) Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing, 2025. Minka et al. (2005) Tom Minka et al. Divergence measures and message passing. 2005. Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Tang et al. (2023) Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023. Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. Von Luxburg (2007) Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17(4):395–416, 2007. Wang et al. (2026) Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Zheng Pan, Xin Li, and Lan-Zhe Guo. Trace: Distilling where it matters via token-routed self on-policy alignment. arXiv preprint arXiv:2605.10194, 2026. Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. Xiao et al. (2026) Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026. Xu et al. (2026) Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Yang et al. (2026) Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr. arXiv preprint arXiv:2604.03128, 2026. Zeng et al. (2026) Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Zhang and Math-AI (2024) Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2024, 2024. Zhang and Math-AI (2025) Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2025, 2025. Zhao et al. (2026) Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026. Appendix A Sensitivity Analysis Hyperparameter sensitivity. We investigate the sensitivity of our method to the hyperparameters α and β in Eq. (12). We vary one hyperparameter at a time while fixing the other at its selected value. Specifically, we vary β over 2.5,4.0,5.0,6.0×10−2\2.5,4.0,5.0,6.0\× 10^-2 while fixing α=2.5×10−2α=2.5× 10^-2. Similarly, we vary α over 0.5,2.5,3.5,4.0,5.0×10−2\0.5,2.5,3.5,4.0,5.0\× 10^-2 while fixing β=5×10−2β=5× 10^-2. As shown in Figure 5, As shown in the figure, the performance of our model follows a smooth increasing-then-decreasing trend with respect to the hyperparameter values. This observation suggests that the proposed method does not require highly precise hyperparameter tuning and remains stable change across a relatively broad range of parameter settings. Figure 5: Sensitivity analysis of the hyperparameters α and β on AIME24. Left: performance under different values of β, with α fixed at 2.5×10−22.5× 10^-2. Right: performance under different values of α, with β fixed at 5×10−25× 10^-2. The stars indicate the selected setting: α=2.5×10−2α=2.5× 10^-2 and β=5×10−2β=5× 10^-2. Appendix B Implementation Details Table 4 lists the training hyperparameters that are shared across all experiments. All runs use LoRA for parameter-efficient fine-tuning with rank r=64r=64 and α=128α=128, and are optimized with a learning rate of 5×10−65× 10^-6 under a maximum gradient norm of 0.10.1. The maximum sequence length is 20,00020,000 tokens, while completions are capped at 1,0241,024 tokens. Rollout sampling uses a temperature of 1.11.1, top-p of 0.950.95, and top-k of 2020. Training is performed in BF16 precision. Table 4: Hyperparameters used across all experiments. Hyperparameter Value Learning rate 5×10−65× 10^-6 Max gradient norm 0.1 LoRA rank r 64 LoRA α 128 Max sequence length 20,000 Max completion length 1,024 Temperature 1.1 Top-p 0.95 Top-k 20 BF16 Enabled Appendix C Notation and Additional Preliminaries Table 5: Summary of the main notation used in the paper. Symbol Description q a query πθ _θ LLM policy parameterized by θ and trained within the OPSD framework. θ Parameters of the LLM policy. yty_t Token generated at decoding step t by πθ _θ. =(y1,…,yT)y=(y_1,…,y_T) Response consisting of T tokens generated by πθ _θ. <t=(y1,…,yt−1)y_<t=(y_1,…,y_t-1) Sequence of tokens generated before step t. πθ(⋅∣q,<t) _θ(· q,y_<t) Conditional distribution over the next token given the query and previously generated tokens. T Length of the generated response. ℋ=1,…,TH=\h_1,…,h_T\ the hidden representations corresponding to the generated response. ℋ^=^1,…,^T H=\ h_1,…, h_T\ the hidden representations generated by the teacher model. ℋ∗=1∗,…,T∗H^*=\h_1^*,…,h_T^*\ the hidden representations generated by the optimal teacher model. ℋ†=1∗+^12+…,T∗+^T2H =\ h_1^*+ h_12+…, h_T^*+ h_T2\ the mixed hidden representations of ℋH and ℋ H. ℋref=1ref,…,TrefH^ref=\h_1^ref,…,h_T^ref\ the hidden representations of the reference model. ℋ=1,…,TH=\h_1,…,h_T\ the hidden representations corresponding to the generated response. ℋ=1,…,TH=\h_1,…,h_T\ the hidden representations corresponding to the generated response. =C1,…,CKC=\C_1,…,C_K\ ℋH into K representation clusters. †=C1†,…,CK†C =\C_1 ,…,C_K \, ℋ†H into K representation clusters. ref=C1ref,…,CKrefC^ref=\C_1^ref,…,C_K^ref\, ℋrefH^ref into K representation clusters. kX_k := i∣i∈ℐk†\h_i i _k \ the associated index set of for each cluster Ck†C_k |||C| the number of clusters of C. |†||C | the number of clusters of †C . |ref||C_ref| the number of clusters of refC_ref. =[1,…,T]∈ℝT×dH=[h_1,…,h_T] ^T× d the hidden representation matrix of the student model. ref=[1ref,…,Tref]∈ℝT×dH^ref=[h_1^ref,…,h_T^ref] ^T× d the hidden representation matrix of the reference model. μk _k 1|Ck|∑i∈Cki 1|C_k| _h_i∈ C_kh_i. μk† _k 1|Ck†|∑i†∈Ck†i† 1|C_k | _h_i ∈ C_k h_i . μkref _k^ref 1|Ckref|∑iref∈Ckrefiref 1|C_k^ref| _h_i^ref∈ C_k^refh_i^ref. L The Laplacian matrix computed from the hidden representation matrix of the student model with a Gaussian kernel. refL^ref The Laplacian matrix computed from the hidden representation matrix of the reference model with a Gaussian kernel. =[λ1(),…,λn()]⊤ λ= [ _1(L),…, _n(L) ] the eigenvalues of the Laplacian matrix L. ref=[λ1(ref),…,λn(ref)]⊤ λ^ref= [ _1(L^ref),…, _n(L^ref) ] the eigenvalues of the Laplacian matrix refL^ref. μkref _k^ref 1|Ckref|∑iref∈Ckrefiref 1|C_k^ref| _h_i^ref∈ C_k^refh_i^ref. V the vocabulary of the LLM. |||V| the size of the vocabulary. W∈ℝ||×dW ^|V|× d the weight matrix of the final layer for the trained model. I||∈ℝ||×||I_|V| ^|V|×|V| the identity matrix . ||∈ℝ||1_|V| ^|V| the all-ones column vector. |Δ|| | ∑i‖i^−i∗‖2 _i|| h_i-h_i^*||_2. D(⋅|⋅)D(·|·) a divergence measure between the teacher and student token distributions. ℰKD(πθ)E_KD( _θ) a divergence measure between the teacher and student token distributions. Appendix D Proof of Theorem 4.1 By definition 4.1, we obtain ℰKD(πθ)=q∼,y∼πθ(⋅∣q)[∑t=1TD(πθ∗(⋅∣q,z,y<t)∥πθ(⋅∣q,y<t))−D(πθ∗(⋅∣q,z,y<t)∥πθ∗(⋅∣q,y<t))]E_KD( _θ)=E_q ,y _θ(· q) [ _t=1^TD ( _θ^*(· q,z,y_<t)\, \|\, _θ(· q,y_<t) )\\ -D ( _θ^*(· q,z,y_<t)\, \|\, _θ^*(· q,y_<t) ) ] (13) To simplify the notation, we define qt∗:=πθ∗(⋅∣q,z,y<t)q^*_t:= _θ^*(· q,z,y_<t) pt:=πθ(⋅∣q,y<t)p_t:= _θ(· q,y_<t) qt:=πθ(⋅∣q,z,y<t)q_t:= _θ(· q,z,y_<t) pt∗:=πθ∗(⋅∣q,y<t)p^*_t:= _θ^*(· q,y_<t) Let ztz_t denote the corresponding logits of ptp_t, zt∗z_t^* denote the corresponding logits of qt∗q_t^* and zt z_t denotes the corresponding logit of qtq_t. For each f∈ℱOPSDf _OPSD, let pfp_f denote the policy induced by f. For one response and q∼,y∼πθq ,y _θ, let ℰq,θ=∑t(KL(qt∗||pt))−E(KL(qt∗||pt∗)E_q,θ= _t(KL(q^*_t||p_t))-E(KL(q^*_t||p^*_t), we have ℰKD(πθ)=q∼,y∼πθℰq,θE_KD( _θ)=E_q ,y _θE_q,θ ℰq,θ=∑t(KL(qt∗||pt))−KL(qt∗||pt∗))E_q,θ= _t(KL(q^*_t||p_t))-KL(q^*_t||p^*_t)) ℰq,θ=∑t[KL(qt∗||pt)−KL(qt||pt)+KL(qt||pt)−KL(qt∗||qt)+KL(qt∗||qt)−KL(qt∗||pt∗)]E_q,θ= _t[KL(q^*_t||p_t)-KL(q_t||p_t)+KL(q_t||p_t)-KL(q_t^*||q_t)+KL(q_t^*||q_t)-KL(q^*_t||p^*_t)] ∑t[KL(qt∗||pt)−KL(qt||pt)]≤supπθ∈ΠOPSD|∑t[KL(qt∗||pf)−KL(qt||pf)]| _t[KL(q^*_t||p_t)-KL(q_t||p_t)]≤ sup_ _θ∈ _OPSD| _t[KL(q^*_t||p_f)-KL(q_t||p_f)]| ∑t(KL(qt||pt)−KL(qt∗||qt))≤0 _t(KL(q_t||p_t)-KL(q_t^*||q_t))≤ 0 ∑t(KL(qt∗||qt)−KL(qt∗||pt∗))≤supπθ∈ΠOPSD|∑t[KL(qt∗||pf)−KL(qt||pf)]| _t(KL(q_t^*||q_t)-KL(q^*_t||p^*_t))≤ _ _θ∈ _OPSD| _t[KL(q^*_t||p_f)-KL(q_t||p_f)]| so we have ℰq,θ≤2∗supπθ∈ΠOPSD|∑t[KL(qt∗||pf)−KL(qt||pf)]|E_q,θ≤ 2* _ _θ∈ _OPSD| _t[KL(q^*_t||p_f)-KL(q_t||p_f)]| ℰq,θ≤2∗supπθ∈ΠOPSD|∑t[∑iqt,i∗logqt,i∗−qt,i∗logpt,i−q^t,ilogqt,i∗+q^t,ilogpt,i]|E_q,θ≤ 2* _ _θ∈ _OPSD| _t[ _iq^*_t,ilogq^*_t,i-q^*_t,ilogp_t,i- q_t,ilogq^*_t,i+ q_t,ilogp_t,i]| (14) Consider an arbitrary LLM and fix an input context. Let ~=(z~1,…,z~V)⊤∈ℝV z=( z_1,…, z_V) ^V denote the logit vector produced by the model, where V is the vocabulary size. The corresponding next-token probability vector =(p1,…,pV)⊤p=(p_1,…,p_V) is given by pi:=exp(z~i)∑j=1Vexp(z~j),i∈[V].p_i:= ( z_i) _j=1^V ( z_j), i∈[V]. Let ∈ℝV1 ^V denote the all-ones vector. Since softmax is invariant to a constant shift, for any c∈ℝc , softmax(~+c)=softmax(~).softmax( z+c1)=softmax( z). Define the centering matrix A:=IV−1V⊤A:=I_V- 1V11 and the centered logit vector :=A~=~−(1V⊤~).z:=A z= z- ( 1V1 z )1. Then ⊤=0,A=,1 z=0, Az=z, and softmax()=softmax(~).softmax(z)=softmax( z). Therefore, without loss of generality, we may work with centered logits and assume throughout that =A,or equivalently∑j=1Vzj=0.z=Az, equivalently _j=1^Vz_j=0. For centered logits satisfying ∥2→0 _2→ 0, Taylor expansion gives ezi=1+zi+12zi2+O(∥23).e^z_i=1+z_i+ 12z_i^2+O\! ( _2^3 ). Consequently, ∑j=1Vezj _j=1^Ve^z_j =V+∑j=1Vzj+12∑j=1Vzj2+O(∥23) =V+ _j=1^Vz_j+ 12 _j=1^Vz_j^2+O\! ( _2^3 ) =V+12∥22+O(∥23), =V+ 12 _2^2+O\! ( _2^3 ), where the second equality follows from ∑j=1Vzj=0 _j=1^Vz_j=0. Therefore, log(∑j=1Vezj) ( _j=1^Ve^z_j ) =logV+log(1+12V∥22+O(∥23)) = V+ (1+ 12V _2^2+O\! ( _2^3 ) ) =logV+12V∥22+O(∥23). = V+ 12V _2^2+O\! ( _2^3 ). In particular, pi=1V(1+zi+O(∥22)).p_i= 1V (1+z_i+O( _2^2) ). For Qwen3, let ∈ℝdh ^d denote the final hidden representation at the current token position and let W∈ℝV×dW ^V× d denote the weight matrix of its bias-free language-modeling head. The original logits are ~=W z=Wh. Defining A:=IV−1V⊤,A:=I_V- 1V11 , the centered logits can be written as :=A~=AW.z:=A z=AWh. Since ⊤=01 z=0 and softmax is invariant to constant shifts, z induces the same next-token distribution as ~ z. Since OPSD employs LoRA without applying it to the output layer, the output-layer weights remain unchanged. Therefore, we obtain: So we can simlify 14: ℰq,θ≤2||∗supπθ∈ΠOPSD|∑t[(ht−ht∗+ht^2)TWT(I||−1||1||1||T)W(ht∗−h^t)]|E_q,θ≤ 2|V|* _ _θ∈ _OPSD| _t[(h_t- h_t^*+ h_t2)^TW^T(I_|V|- 1|V|1_|V|1_|V|^T)W(h^*_t- h_t)]| (15) Throughout our analysis, we assume that the hidden representations are normalized using RMSNorm without learnable scaling parameters. By the Cauchy–Schwarz inequality, we have ℰKD(πθ)≤2N||∑Nsupπθ∈ΠOPSD|Δ|||M||2[Td2+12∑|†|∑i∈k‖i−μk‖22−12∑|†||k†|‖μk−μ‖22+ _KD( _θ)≤ 2N|V| _N _ _θ∈ _OPSD| |||M||_2[ Td2+ 12 _|C | _h_i _k||h_i- _k^X||_2^2- 12 _|C ||C _k||| _k^X-μ||^2_2+ (16) ∑|†||k†|‖μk−μk†‖22+∑|†|∑i†∈k‖i†−μk†‖22] _|C ||C_k ||| _k^X- _k ||^2_2+ _|C | _h_i _k||h_i - _k ||_2^2] Appendix E Proof of Theorem 5.1 Proof. For each cluster, we stack all hi,ref∈krefh_i,ref _k^ref to form Hkref=[(h1ref)⊤,…,(h|Ckref|ref)⊤]⊤H_k^ref=[(h_1^ref) ,…,(h_|C_k^ref|^ref) ] , then we define Gkref=HkrefHkref⊤G_k^ref=H_k^refH_k^ref . Accordingly, we stack all hi,align∈kalignh_i,align _k^align to form Hkalign=[(h1align)⊤,…,(h|Ckalign|align)⊤]⊤H_k^align=[(h_1^align) ,…,(h_|C_k^align|^align) ] , then we define Gkalign=HkalignHkalign⊤G_k^align=H_k^alignH_k^align . and |Ckref|∈ℝ|Ckref|1_|C_k^ref| ^|C_k^ref| denotes the all-ones column vector.|ref||C_ref| denotes the number of clusters in the reference model. First, we can simplify the optimization objective of intra-cluster compactness for each cluster: |∑i∈Ckalign,‖i−kalign‖22−∑iref∈Ckref‖iref−kref‖22|,≤‖I|Cref|−|Ckref|⊤|Ckref|‖F‖Gkref−Gk‖F | _h_i∈ C_k^align, \|h_i- μ_k^align \|_2^2- _h_i^ref∈ C_k^ref \|h_i^ref- μ_k^ref \|_2^2 |,≤||I_|C_ref|-1_|C^ref_k| 1_|C^ref_k|||_F||G^ref_k-G_k||_F (17) Second, GkalignG_k^align is a principal submatrix of HHTHH^T and GkrefG_k^ref is a principal submatrix of HHTHH^T. Because the squared Frobenius norm is additive over pairwise disjoint principal submatrices, Therefore, for all clusters, we can obtain this through the Cauchy-Schwarz inequality: ∑k‖I|Ckref|−|Ckref|⊤|Ckref|‖F‖Gkref−Gkalign‖F≤(T−|Cref|)12||∑k(Gkref−Gkalign)||F _k||I_|C_k^ref|-1_|C_k^ref| 1_|C_k^ref|||_F||G_k^ref-G_k^align||_F≤(T-|C_ref|) 12|| _k(G_k^ref-G_k^align)||_F Third, through the inclusion of the pairwise disjoint principal submatrices, we can get the upper bound of ℒintraL_intra: ≤(T−|ref|)12‖HHT−HrefHref⊤‖F ≤(T-|C_ref|) 12||H^T-H_refH_ref ||_F (18) ∎ Appendix F Proof of Proposition 5.2 Proof. Through the definition of the Laplacian matrix and the Cauchy-Schwarz inequality, we can obtain: Let T∈ℝ1_T ^T denote the all-ones column vector. ‖L−Lref‖F=‖diag(AT)−diag(ArefT)+A−Aref‖F ||L-L^ref||_F=||diag(A1_T)-diag(A^ref1_T)+A-A^ref||_F ≤‖diag(A−Aref)T‖F+||A−Aref||F, ≤||diag(A-A^ref)1_T||_F+||A-A^ref||_F, ‖diag(A−Aref)T‖F=‖A−Aref‖2≤T||A−Aref||F||diag(A-A^ref)1_T||_F=||A-A^ref||_2≤ T||A-A^ref||_F ‖diag(A−Aref)T‖F+‖A−Aref‖F≤(T+1)||A−Aref||F||diag(A-A^ref)1_T||_F+||A-A^ref||_F≤( T+1)||A-A^ref||_F (T+1)‖A−Aref‖F≤1+T2σ2‖D−Dref‖F( T+1)||A-A^ref||_F≤ 1+ T2σ^2||D-D^ref||_F ‖D−Dref‖F=‖diagHHTT+TdiagHHT−2HHT−diagHrefHrefT1T−1diagHrefHrefT+2HrefHrefT‖F||D-D^ref||_F=||diag\H^T\1_T^T+1_Tdiag\H^T\-2H^T-diag\H_refH_ref^T\1^T-1diag\H_refH_ref^T\+2H_refH_ref^T||_F Through Cauchy-Schwarz inequality, we have ∑i|λ(L)−λ(Lref)|2≤(1+Tσ)2‖HHT−HrefHrefT‖F _i|λ(L)-λ(L^ref)|^2≤( 1+ Tσ)^2||H^T-H_refH_ref^T||_F ∎