Paper deep dive
OPSDL: On-Policy Self-Distillation for Long-Context Language Models
Xinsen Zhang, Zhenkai Ding, Tianjun Pan, Run Yang, Chun Kang, Xue Xiong, Jingnan Gu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/27/2026, 8:10:09 AM
Summary
OPSDL (On-Policy Self-Distillation for Long-Context Language Models) is a post-training method designed to extend the effective context length of LLMs. It leverages the model's own short-context capability as a 'self-teacher' to supervise its generation in long-context scenarios using point-wise reverse KL divergence. This approach provides dense, token-level supervision signals that mitigate hallucinations and improve sample efficiency compared to sequence-level methods like DPO or SFT. The method was evaluated on Qwen2.5 models (7B to 32B) and showed significant improvements on RULER and LongBench V2 benchmarks, effectively closing the gap with models explicitly trained for million-token contexts without degrading short-context performance.
Entities (7)
Relation Signals (5)
OPSDL → evaluatedon → RULER
confidence 100% · We empirically validate OPSDL on the RULER(Hsieh et al. (2024))
OPSDL → evaluatedon → LongBench V2
confidence 100% · We evaluate our method on three representative long-context benchmarks: RULER (Hsieh et al. (2024)), and LongBench V2(Bai et al. (2025)).
OPSDL → improves → LLM
confidence 100% · We propose OPSDL, an On-Policy Self-Distillation method for enhancing the Long-context capabilities of LLMs.
Qwen2.5-7B-Instruct → isabackbonefor → OPSDL
confidence 100% · We adopt the Qwen2.5-Instruct series as our backbone models
OPSDL → uses → Reverse KL-divergence
confidence 100% · the self-teacher provides per-token supervision signals via point-wise reverse KL divergence
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Extending the effective context length of large language models (LLMs) remains a central challenge for real-world applications. While recent post-training methods have made progress in long-context scaling, they either rely on high-quality supervision data or sparse sequence-level rewards, leading to unstable and inefficient optimization. We propose OPSDL, an On-Policy Self-Distillation method for enhancing the Long-context capabilities of LLMs. Unlike other recent self-distillation methods that inject privileged information and rely on the model's in-context learning ability to act as a teacher, OPSDL leverages the model's own inherently strong short-context capability as a self-teacher to supervise its own generation in long-context scenarios. The model first generates responses conditioned on the full long-context, then the self-teacher provides per-token supervision signals via point-wise reverse KL divergence under the relevant extracted short-context. This dense token-level signal encourages faithful use of relevant evidence and mitigates hallucinations induced by irrelevant context. We evaluate OPSDL on long-context benchmarks across a range of models from 7B to 32B parameters. Results show consistent and substantial improvements across varying context lengths, outperforming standard post-training approaches such as SFT and DPO with higher sample efficiency. Notably, these gains are achieved without degrading general short-context performance. These findings highlight the effectiveness of OPSDL as a scalable and stable approach for long-context learning.
Tags
Links
- Source: https://arxiv.org/abs/2604.17535v1
- Canonical: https://arxiv.org/abs/2604.17535v1
Trouble viewing inline? Open PDF directly →
Full Text
29,954 characters extracted from source content.
Expand or collapse full text
OPSDL: On-Policy Self-Distillation for Long-Context Language Models Xinsen Zhang1, Zhenkai Ding1, Tianjun Pan1, Run Yang1, Chun Kang1, Xue Xiong1, Jingnan Gu1 1Baidu Inc Abstract Extending the effective context length of large language models (LLMs) remains a central challenge for real-world applications. While recent post-training methods have made progress in long-context scaling, they either rely on high-quality supervision data or sparse sequence-level rewards, leading to unstable and inefficient optimization. We propose OPSDL, an On-Policy Self-Distillation method for enhancing the Long-context capabilities of LLMs. Unlike other recent self-distillation methods that inject privileged information and rely on the model’s in-context learning ability to act as a teacher, OPSDL leverages the model’s own inherently strong short-context capability as a self-teacher to supervise its own generation in long-context scenarios. The model first generates responses conditioned on the full long-context, then the self-teacher provides per-token supervision signals via point-wise reverse KL divergence under the relevant extracted short-context. This dense token-level signal encourages faithful use of relevant evidence and mitigates hallucinations induced by irrelevant context. We evaluate OPSDL on long-context benchmarks across a range of models from 7B to 32B parameters. Results show consistent and substantial improvements across varying context lengths, outperforming standard post-training approaches such as SFT and DPO with higher sample efficiency. Notably, these gains are achieved without degrading general short-context performance. These findings highlight the effectiveness of OPSDL as a scalable and stable approach for long-context learning. 1 Introduction Extending the effective context length of large language models (LLMs) is a central challenge in enabling real-world applications such as long-document comprehension, repository-level code understanding, and multi-hop reasoning over scattered evidence. While architectural innovations in positional encoding and attention mechanisms have made it possible to accept much longer inputs (Peng et al. (2023); Su et al. (2024); Team et al. (2026)), a growing body of empirical evidence reveals a persistent gap between the maximum context window and the effective context capacity of a model (Paulsen (2025); Bai et al. (2025)). These findings suggest that closing the gap between maximum and effective context length requires not only architectural advances but also innovations in post-training paradigms. Current training approaches for long-context LLM performance optimization are numerous and effective(Shen et al. (2025); Zhang et al. (2025); Chen et al. (2025)). While these methods have shown effectiveness, they suffer from either the need for high-quality training data or sparse sequence-level reward signals that limit sample efficiency. Under long contexts, such sparse, sequence-level signals make optimization extremely difficult, leading to unstable and sample-inefficient training. Furthermore, these methods typically rely on auxiliary components such as frozen verifier models or learned reward models, which increases system complexity and couples the training pipeline to the availability and quality of external resources. To address this, we propose OPSDL, a simple yet effective method that overcomes these limitations. The key observation is that, given a query that derived from the short context, a model that given this short context naturally serves as a teacher compared to the model that given a longer contex. OPSDL leverages this asymmetry by distilling the model’s own short-context behavior into its long-context behavior via token-level reverse KL divergence, computed on-policy during training. Rather than relying on external supervision or reward signals, the model itself serves as both student and teacher, with the short-context ability providing a natural and readily available training signal. This design eliminates the need for human-annotated data, reward models, or elaborate reward engineering, while the on-policy distillation framework ensures that the training signal remains aligned with the model’s current behavior, yielding strong sample efficiency. We empirically validate OPSDL on the RULER(Hsieh et al. (2024)) across multiple backbone models and context lengths. Our results show that OPSDL consistently outperforms standard instruction tuned models on both Qwen2.5-7B family(Yang et al. (2024)), demonstrating strong generalization across model families rather than reliance on a specific architecture. Notably, the performance gains become increasingly pronounced as the context length grows, indicating that our method effectively mitigates the degradation commonly observed under long horizon settings. Moreover, despite not relying on specialized long context pretraining, OPSDL achieves performance comparable to, and in some cases exceeding, Qwen2.5-7B-Instruct-1M(Yang et al. (2025)), a model explicitly trained for million-token contexts. These results suggest that replacing sparse sequence-level preference optimization with on-policy, token-level distillation enables stable policy-level self-evolution, unlocking robust long-context reasoning while preserving short-context capabilities. In summary, our contributions are as follows: • We propose OPSDL, an on-policy self-distillation method that leverages a model’s inherent short-context capability as a teacher to supervise its own long-context generation. By computing token-level supervision signals via point-wise reverse KL divergence, OPSDL provides dense training signals that encourage faithful use of relevant evidence and mitigate context-induced hallucinations. • Unlike existing long-context training methods that rely on human-annotated data, reward models, or sparse sequence-level rewards, OPSDL eliminates the need for external supervision or auxiliary components. This design yields superior sample efficiency and training stability while remaining model-agnostic across different architectures. • We conduct comprehensive evaluations on long-context benchmarks across models from 7B to 32B parameters. Results demonstrate consistent improvements across varying context lengths, with performance gains becoming increasingly pronounced as the context grows. Notably, OPSDL achieves these gains without degrading short-context performance. 2 Related Work Post-Training for Long-Context Modeling A straightforward approach to improving long-context capability is supervised fine-tuning (SFT) on long-context training data, though it is constrained by data quality and prone to distribution shift. To overcome these limitations, preference optimization methods have been explored. LongReward(Zhang et al. (2025)) constructs reward signals via multidimensional LLM feedback and applies DPO to boost long-context performance. LongPO(Chen et al. (2025)) proposes a self-evolution framework that treats short-context generations as positive samples and long-context counterparts as negative ones, iteratively optimizing the model via DPO. SoLoPO(Sun et al. (2025)) further decomposes alignment into short-context optimization and a short-to-long consistency constraint. From a training strategy perspective, QwenLong-L1(Wan et al. (2025)) treat long-context reasoning as a long-horizon decision-making problem, utilizing warm-up SFT and progressive context scaling to stabilize RL training. However, these methods rely on externally constructed preference pairs and coarse-grained sequence-level objectives. In contrast, our approach adopts an on-policy self-distillation method with token-level fine-grained optimization signals, enabling more scalable and stable long-context learning. On-Policy Distillation and Self-Distillation On-Policy Distillation (OPD) has recently attracted considerable attention due to its on-policy nature and token-level supervision. Pioneered by Generalized Knowledge Distillation (GKD) (Agarwal et al., 2024), this line of work minimizes reverse KL divergence on student-generated trajectories under teacher supervision. To bypass external teacher dependency, on-policy self-distillation has emerged as a key paradigm. OPSD (Zhao et al., 2026) and SDPO (Hübotter et al., 2026) enable the model itself to serve as the teacher by incorporating more privileged context, such as verified reasoning traces or environmental feedback. OPCD (Ye et al., 2026) internalizes contextual knowledge and prevents forgetting by distilling from teacher-enriched contexts. While these methods typically construct a stronger teacher by enriching the input context, heavy reliance on externally provided correct trajectories may compromise the model’s intrinsic reasoning capability. In contrast, our approach takes the opposite direction: rather than enriching the context, we extract key information from long contexts to reduce noise and utilize the model’s superior short-context ability, constructing a token-level self-teacher to supervise its outputs derived from long contexts. 3 Method Figure 1: Overview of the OPSDL framework. Given a long context CLC_L and its extracted short context CSC_S, the model generates responses on-policy conditioned on CLC_L. The same model under CSC_S serves as a self-teacher, providing token-level supervision via point-wise reverse KL divergence to align the long-context generation with the short-context behavior. We propose OPSDL, which leverages the short-context capability of the current policy to supervise its own long-context generation. 3.1 Problem Formulation Let CLC_L denote the long-context input, and CSC_S denote the corresponding short context extracted from CLC_L, which contains the core information. Based on CSC_S, an instruction or query Q is formulated. Given the query Q and the context, the large language model, parameterized by θ and denoted as πθ _θ, auto-regressively generates a response sequence y=y1,y2,…,yTy=\y_1,y_2,…,y_T\. The key insight of our approach is that the same policy πθ _θ, when prompted with the short-context input CSC_S, produces a more calibrated and accurate next-token distribution, since CSC_S avoids distracting noise and fits perfectly within the model’s well-trained context window. This allows the model to leverage its well-aligned short-context capabilities to self-align in long-context scenarios. Therefore, instead of relying on external reward models, we formulate the long-context optimization problem as a self-alignment process: we aim to optimize θ such that the model’s generation distribution under the long context, πθ(y∣CL,Q) _θ(y C_L,Q), closely approximates its own high-quality distribution under the short context, πθ(y∣CS,Q) _θ(y C_S,Q). 3.2 Data Construction We constructs (CL,CS,Q)(C_L,C_S,Q) triplets from raw long documents without human annotation. We sample a long document CLC_L from the training corpus, then extract a contiguous segment CS⊂CLC_S⊂ C_L with |Cs|≤|CL||C_s|≤|C_L| that preserves the core evidence while fitting within the model’s well-trained context window. Finally, we generate a question Q∼πθ(⋅∣CS)Q _θ(· C_S) conditioned on CSC_S, ensuring that Q targets evidence present in both contexts. This yields triplets where the question Q is answerable from both CLC_L and CSC_S, enabling meaningful comparison between the short-context teacher and the long-context student. 3.3 Short-to-Long Self-Distillation via Policy Gradient To mitigate the distraction from irrelevant information in the long context, we utilize the model itself as a dynamic teacher, using its predictions under the short context CSC_S as a robust anchor. Formally, for a response y=(y1,…,yT)y=(y_1,…,y_T) sampled from the long-context student policy πθ(⋅∣CL,Q) _θ(· C_L,Q), the teacher probability at each token position is: P(yt)=πTeacher(yt∣CS,Q,y<t),P(y_t)= _Teacher(y_t C_S,Q,y_<t), (1) where the same model parameters θ are used under the short context, forming a self-referential teacher-student structure that evolves jointly during training. Token-Level Advantage. Instead of relying on an external reward model or human annotations, we define a token-level advantage At(yt)A_t(y_t) as the log-probability ratio between the short-context (teacher) and long-context (student) distributions: At(yt)=logπTeacher(yt∣CS,Q,y<t)πθ(yt∣CL,Q,y<t),A_t(y_t)= _Teacher(y_t C_S,Q,y_<t) _θ(y_t C_L,Q,y_<t), (2) This quantity serves as an unbiased estimator for the gradient of the point-wise reverse KL divergence between the two distributions, and admits a natural interpretation: • Positive At(yt)>0A_t(y_t)>0: the teacher assigns higher probability than the student, indicating that the student under-weights evidence clearly present in the short context—a signal of context utilization degradation under long input. • Negative At(yt)<0A_t(y_t)<0: the student assigns higher probability than the teacher, suggesting that the model hallucinates or attends to irrelevant content introduced by the long context. • Near-zero At(yt)≈0A_t(y_t)≈ 0: both distributions agree, indicating the token is unaffected by context length and receives negligible gradient signal. This design enables that only tokens where the long-context policy deviates from the short-context anchor receive non-trivial gradient signal, avoiding the noise introduced by uniform supervision. Policy Gradient Objective. We optimize the model using a policy gradient objective over the long-context distribution πθ(⋅∣CL,Q) _θ(· C_L,Q): ℒPG(θ)=−y∼πθ(⋅∣CL,Q)[∑t=1TAt(yt)⋅logπθ(yt∣CL,Q,y<t)],L_PG(θ)=-E_y _θ(· C_L,Q) [ _t=1^TA_t(y_t)· _θ(y_t C_L,Q,y_<t) ], (3) where the advantage At(yt)A_t(y_t) is defined in Eq. (2). Intuitively, this objective encourages the long-context policy to increase the probability of tokens that the short-context teacher considers likely, while suppressing tokens that arise from irrelevant long-context distractions. This reveals that OPSDL performs on-policy self-distillation at the token level, where the short-context distribution acts as a dynamically evolving teacher signal. Unlike standard knowledge distillation with a frozen teacher, the teacher here co-evolves with the student, ensuring that the anchor remains calibrated throughout training. 3.4 Overall Training Procedure Algorithm 1 summarizes the complete training procedure of OPSDL, consisting of a data construction phase that builds training triples ⟨CL,CS,Q⟩ C_L,C_S,Q , and a training phase that iteratively optimizes the policy via token-level advantage estimation. Algorithm 1 OPSDL Algorithm 0: Long-document corpus D; Initial LLM πθ _θ to be optimized; Short-context-aligned LLM πTeacher _Teacher. 0: Optimized LLM πθ _θ with longer effective context length 1: // Data Construction Phase 2: Sample a long context CL∼C_L 3: Extract short chunk CS⊂CLC_S⊂ C_L and generate query Q via Self-QA based on CSC_S 4: Construct the training triple ⟨CL,CS,Q⟩ C_L,C_S,Q 5: // Training Phase 6: for each training iteration do 7: // On-policy Rollout 8: Generate response y=y1,y2,…,yT∼πθ(⋅∣CL,Q)y=\y_1,y_2,…,y_T\ _θ(· C_L,Q) 9: // Token-level Reverse-KL Advantage Estimation 10: for each token yty_t do 11: Compute student log-prob: st=logπθ(yt∣CL,Q,y<t)s_t= _θ(y_t C_L,Q,y_<t) 12: Compute teacher log-prob: t=logπTeacher(yt∣CS,Q,y<t)t_t= _Teacher(y_t C_S,Q,y_<t) 13: At=t−stA_t=t_t-s_t = logPteacherPstudent P_teacherP_student 14: end for 15: // Policy Optimization 16: Update πθ _θ using token-level advantages At\A_t\ 17: end for 18: return πθ _θ 4 Experiment 4.1 Experiment Setup Data Construction Following the data preparation pipeline in LongPO( Chen et al. (2025)), we construct training data from raw long documents without human annotation. We adopt a reverse construction strategy: for each long document CLC_L, we first randomly sample a shortened chunk CSC_S and then prompt the model to generate instructions based on the CSC_S via Self-Instruct( Wang et al. (2023)). To ensure diversity, the model first generates an instruction pool and then randomly samples an instruction I from it. In contrast to LongPO, our method only requires the (CL,CS,I)(C_L,C_S,I) triplets and does not require any preference responses. Specifically, our method trains on responses sampled on-policy from the model itself, which not only simplifies data preparation but also mitigates the distribution mismatch issues associated with off-policy data. Methods RULER LongBench V2 Total Avg. 4K 8K 16K 32K 64K 128K Avg. Easy Hard Short Medium Long Overall 7B Models Qwen2.5-7B-Instruct 94.93 92.87 92.03 89.20 68.77 25.14 77.16 29.2±0.0 24.4±0.0 33.3±0.0 23.3±0.0 20.4±0.0 26.2±0.0 51.68 + Long-SFT 92.16 89.13 87.59 82.58 76.01 63.78 81.88 28.2±0.9 24.9±0.1 32.6±0.7 22.7±0.2 22.2±0.0 26.1±0.3 53.99 + LongPO† 95.04 93.33 89.81 86.40 74.18 61.28 83.34 29.2±0.0 26.5±0.6 38.4±0.6 23.9±0.3 16.7±0.0 27.5±0.3 55.42 + Ours 94.30 92.30 90.04 87.07 80.39 73.84 86.32 34.4±0.0 31.6±0.3 36.1±0.0 32.7±0.4 26.9±0.0 32.6±0.2 56.61 Qwen2.5-7B-Instruct-1M 94.28 93.14 93.39 90.76 88.15 81.84 90.26 36.0±0.6 27.3±0.0 39.3±0.5 25.6±0.0 26.1±0.5 30.6±0.2 60.43 14B Models Qwen2.5-14B-Instruct 96.56 95.26 93.78 92.20 77.13 46.69 83.61 35.0±0.2 29.5±0.3 37.9±0.7 28.4±0.5 27.6±0.5 31.6±0.2 57.61 + Long-SFT 95.96 94.98 92.92 90.47 84.62 73.69 88.77 34.0±0.2 31.5±0.5 40.6±0.5 28.5±0.4 26.9±1.3 32.5±0.4 60.64 + LongPO† – – – – – – – – – – – – – – + Ours 96.15 94.81 93.48 91.93 88.08 80.94 90.90 34.4±0.5 33.2±0.5 41.5±0.6 27.1±0.2 33.6±0.6 33.7±0.3 62.30 Qwen2.5-14B-Instruct-1M 96.81 96.29 95.46 94.08 93.01 89.42 94.18 38.8±0.7 34.9±0.3 45.6±0.5 32.1±0.7 29.6±0.8 36.4±0.3 65.29 32B Models Qwen2.5-32B-Instruct 96.53 96.23 95.98 94.09 81.62 54.02 86.41 36.2±1.0 30.9±0.8 39.9±0.6 28.4±0.9 30.4±0.5 32.9±0.6 59.65 + Long-SFT 96.05 95.62 95.21 94.10 89.72 80.70 91.90 34.8±0.8 33.9±0.6 40.3±0.7 30.6±0.8 31.3±0.5 34.2±0.6 63.05 + LongPO† – – – – – – – – – – – – – – + Ours 96.56 96.29 96.17 95.81 91.04 84.31 93.36 36.1±0.5 36.6±0.5 39.1±0.7 34.9±0.4 35.0±0.5 36.5±0.4 64.93 Table 1: Comprehensive results on RULER and LongBench V2 benchmarks across different model sizes. For LongBench V2, each cell reports the mean over 4 independent runs at LLM sampling temperature 0.1, with ± denoting the standard deviation across runs. †For 7B model, LongPO results are reproduced using our curated 5K data; for 14B and 32B models, we found that LongPO failed to converge during our reproduction, and thus we do not report its results. Best results among trainable methods (excluding -1M variants) are in bold. Baselines We compare our method with several representative long-context post-training methods. Long-SFT performs supervised fine-tuning on long-context training data by directly minimizing the negative log-likelihood of the target responses. LongPO minimizes the DPO loss on self-generated preference pairs, where responses produced from short contexts are preferred over those from long contexts, to reduce the performance gap between short- and long-context scenarios. Models We adopt the Qwen2.5-Instruct series as our backbone models, spanning three scales: Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, and Qwen2.5-32B-Instruct. For each backbone, we apply Long-SFT and LongPO as baseline training methods. In addition, to better assess the effectiveness of our approach in long-context settings, we compare against Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M (Yang et al., 2025), which are specifically trained to handle contexts of up to 1M tokens. Evaluation benchmarks We evaluate our method on three representative long-context benchmarks: RULER (Hsieh et al. (2024)), and LongBench V2(Bai et al. (2025)). RULER is a synthetic long-context evaluation suite designed to quantify a model’s long-context modeling capacity. LongBench v2 is a challenging multiple-choice benchmark that evaluates deep understanding and reasoning over long documents across diverse domains and context lengths. 4.2 Main Result OPSDL achieves the best overall performance among trainable methods. Table 1 presents comprehensive results on RULER and LongBench V2 across three model scales. Our method consistently achieves the largest performance improvement over the base instruct model at every scale, demonstrating strong generalizability. At the 7B scale, OPSDL raises the performance from 51.68 to 56.61, surpassing both Long-SFT and LongPO. The advantages become more pronounced at larger scales. Substantial improvements on long contexts. The most striking gains appear at extended context lengths on RULER. Across all three scales, the base instruct models suffer significant performance degradation beyond 64K tokens, whereas OPSDL maintains robust performance. For example, at 128K tokens, OPSDL improves over the base instruct model by +48.70, +34.25, and +30.29 points at the 7B, 14B, and 32B scales respectively. These improvements are consistently larger than those achieved by Long-SFT and LongPO, confirming that our method, which concentrates optimization on tokens where the model’s long-context behavior deviates from its short-context anchor, yields more targeted and efficient learning than uniform sequence-level supervision. OPSDL narrows the gap with officially long-context-optimized models. The Qwen2.5-Instruct-1M variants are officially released models that undergo dedicated long-context training—including multi-stage supervised fine-tuning on sequences up to 256K tokens and length extrapolation techniques—to support context lengths up to 1M tokens. These models achieve the highest absolute scores. Nevertheless, OPSDL substantially closes the gap without relying on such extensive long-context training pipelines. On RULER average, OPSDL narrows the gap to the 1M variant from 13.10 to 3.94 at 7B and from 10.57 to 3.28 at 14B. These results highlight that on-policy token-level self-distillation provides a lightweight yet effective alternative to dedicated long-context training. Improvements span both synthetic and realistic benchmarks. A key strength of OPSDL is that it improves performance on both RULER, which evaluates fundamental long-context capabilities such as retrieval and tracking, and LongBench V2, which tests realistic long-document reasoning. Across all three scales, OPSDL consistently improves LongBench V2 overall scores alongside RULER gains, confirming that our approach enhances genuine long-context reasoning rather than overfitting to synthetic patterns. Notably, LongPO failed to converge at the 14B and 32B scales during our reproduction, whereas OPSDL trains stably across all model sizes, underscoring the robustness of our training method. Methods MMLU ARC-C Hellaswag Winogrande MT-Bench Qwen2.5-7B-Instruct 74.34% 67.41% 81.37% 75.53% 7.70 + Long-SFT 71.66% 63.40% 77.09% 71.51% 7.17 + LongPO-128K 73.40% 66.55% 80.38% 74.03% 7.73 + Ours 73.13% 66.30% 80.01% 73.95% 7.71 Table 2: Performance on Short-Context and General Benchmarks after Long-Context Training. OPSDL preserves short-context capabilities. As shown in Table 2, OPSDL incurs only marginal degradation on short-context and general benchmarks: across MMLU, ARC-C, Hellaswag, and Winogrande, the average drop is about 1.3 percentage points relative to the base Qwen2.5-7B-Instruct model, while MT-Bench remains virtually unchanged. In contrast, Long-SFT suffers notably larger degradation (3–4 percentage points on average). These results confirm that OPSDL effectively enhances long-context performance without sacrificing the model’s general capabilities. 5 Conclusion We propose OPSDL, an on-policy self-distillation framework that leverages a model’s own short-context capability as a dynamically co-evolving teacher to supervise its long-context generation. By computing token-level advantages via point-wise reverse KL divergence, OPSDL provides dense and targeted training signals that focus optimization on tokens where the model’s long-context behavior deviates from its short-context anchor, eliminating the need for external reward models, human-annotated data, or offline preference pairs. Comprehensive experiments on RULER and LongBench V2 across the Qwen2.5-Instruct series at 7B, 14B, and 32B scales demonstrate that OPSDL consistently outperforms both Long-SFT and LongPO, with particularly striking gains at extended context lengths. Moreover, OPSDL substantially narrows the performance gap with the officially long-context-optimized Qwen2.5-Instruct-1M variants, while preserving short-context capabilities with minimal degradation. These results establish OPSDL as a simple, scalable, and effective paradigm for long-context post-training. References R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. In The twelfth international conference on learning representations, Cited by: §2. Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, et al. (2025) Longbench v2: towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3639–3664. Cited by: §1, §4.1. G. Chen, X. Li, M. Q. Shieh, and L. Bing (2025) LongPO: long context self-evolution of large language models through short-to-long preference optimization. arXiv preprint arXiv:2502.13922. Cited by: §1, §2, §4.1. C. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg (2024) RULER: what’s the real context size of your long-context language models?. arXiv preprint arXiv:2404.06654. Cited by: §1, §4.1. J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. K. Buening, C. Guestrin, et al. (2026) Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802. Cited by: §2. N. Paulsen (2025) Context is what you need: the maximum effective context window for real world limits of llms. arXiv preprint arXiv:2509.21361. Cited by: §1. B. Peng, J. Quesnelle, H. Fan, and E. Shippole (2023) Yarn: efficient context window extension of large language models. arXiv preprint arXiv:2309.00071. Cited by: §1. W. Shen, Z. Yang, C. Li, Z. Lu, M. Peng, H. Sun, Y. Shi, S. Liao, S. Lai, B. Zhang, et al. (2025) QwenLong-l1. 5: post-training recipe for long-context reasoning and memory management. arXiv preprint arXiv:2512.12967. Cited by: §1. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu (2024) Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. Cited by: §1. H. Sun, S. Liao, Y. Han, Y. Bai, Y. Gao, C. Fu, W. Shen, F. Wan, M. Yan, J. Zhang, et al. (2025) SoLoPO: unlocking long-context capabilities in llms via short-to-long preference optimization. arXiv preprint arXiv:2505.11166. Cited by: §2. M. Team, W. An, Y. Chen, Y. Fang, J. Li, X. Li, Y. Li, Y. Li, Y. Li, B. Lin, et al. (2026) MiniCPM-sala: hybridizing sparse and linear attention for efficient long-context modeling. arXiv preprint arXiv:2602.11761. Cited by: §1. F. Wan, W. Shen, S. Liao, Y. Shi, C. Li, Z. Yang, J. Zhang, F. Huang, J. Zhou, and M. Yan (2025) Qwenlong-l1: towards long-context large reasoning models with reinforcement learning. arXiv preprint arXiv:2505.17667. Cited by: §2. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi (2023) Self-instruct: aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), p. 13484–13508. Cited by: §4.1. A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, X. Liu, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, Z. Guo, and Z. Fan (2024) Qwen2 technical report. External Links: 2407.10671, Link Cited by: §1. A. Yang, B. Yu, C. Li, D. Liu, F. Huang, H. Huang, J. Jiang, J. Tu, J. Zhang, J. Zhou, et al. (2025) Qwen2. 5-1m technical report. arXiv preprint arXiv:2501.15383. Cited by: §1, §4.1. T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei (2026) On-policy context distillation for language models. arXiv preprint arXiv:2602.12275. Cited by: §2. J. Zhang, Z. Hou, X. Lv, S. Cao, Z. Hou, Y. Niu, L. Hou, Y. Dong, L. Feng, and J. Li (2025) LongReward: improving long-context large language models with AI feedback. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 3718–3739. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1, §2. S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026) Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. Cited by: §2.