Paper deep dive
Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning
Chen Henry Wu, Thomas Zhang, Aditi Raghunathan
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A long-standing goal of AI is a model that can continually learn and improve itself. On post-trained models, supervised finetuning (SFT) on new data often causes poor generalization and catastrophic forgetting. As such, the conventional wisdom is that on-policy training is a prerequisite for continual learning. In practice, however, data containing new knowledge or capabilities are often off-policy. While methods such as on-policy self-distillation (OPSD) try to bridge this gap by converting off-policy data into on-policy signal, they have been shown to cause reasoning collapse. In this paper, we show that off-policy merging beats OPSD for continual learning. We first show that SFT learns a useful signal from new data, but naively applying its update interferes with existing capabilities. We reduce this interference with a simple recipe we term grafting, which changes where the update is learned and how it is applied: (1) learning the update on an earlier donor checkpoint, ideally even before the end of pretraining, and applying the weight update to the post-trained model; (2) scaling the weight update, equivalent to a form of model merging; and (3) optionally, masking the most sensitive update directions when the new data distribution is far from the post-trained model. Across continual learning settings including (1) distilling from expert traces, (2) self-improvement with STaR and Pedagogical RL, and (3) injecting knowledge after pretraining cutoff, grafting Pareto-dominates both SFT and OPSD in new-task and old-task performance, while avoiding expensive on-policy sampling. Therefore, our work challenges on-policy training as a necessity for continual learning on RL-trained models.
Tags
Links
- Source: https://arxiv.org/abs/2610.05872v1
- Canonical: https://arxiv.org/abs/2610.05872v1
Trouble viewing inline? Open PDF directly →
Full Text
69,139 characters extracted from source content.
Expand or collapse full text
Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning Abstract A long-standing goal of AI is a model that can continually learn and improve itself. On post-trained models, supervised finetuning (SFT) on new data often causes poor generalization and catastrophic forgetting. As such, the conventional wisdom is that on-policy training is a prerequisite for continual learning. In practice, however, data containing new knowledge or capabilities are often off-policy. While methods such as on-policy self-distillation (OPSD) try to bridge this gap by converting off-policy data into on-policy signal, they have been shown to cause reasoning collapse. In this paper, we show that off-policy merging beats OPSD for continual learning. We first show that SFT learns a useful signal from new data, but naively applying its update interferes with existing capabilities. We reduce this interference with a simple recipe we term grafting, which changes where the update is learned and how it is applied: (1) learning the update on an earlier donor checkpoint, ideally even before the end of pretraining, and applying the weight update to the post-trained model; (2) scaling the weight update, equivalent to a form of model merging; and (3) optionally, masking the most sensitive update directions when the new data distribution is far from the post-trained model. Across continual learning settings including (1) distilling from expert traces, (2) self-improvement with STaR and Pedagogical RL, and (3) injecting knowledge after pretraining cutoff, grafting Pareto-dominates both SFT and OPSD in new-task and old-task performance, while avoiding expensive on-policy sampling. Therefore, our work challenges on-policy training as a necessity for continual learning on RL-trained models. Chen Henry Wu, Thomas Zhang, Aditi Raghunathan Carnegie Mellon University chenwu2,thomasz4,raditi@cs.cmu.edu https://github.com/ar-forum/grafting https://ar-forum.github.io/grafting-web/ 1 Introduction A long-standing goal of AI is a model that keeps acquiring new capabilities without forgetting what it can already do (McCloskey and Cohen, 1989; Kirkpatrick et al., 2017). This is particularly challenging for models that have undergone extensive post-training. In particular, direct supervised finetuning (SFT) on new data can cause forgetting and fail to generalize (Chu et al., 2025). This has motivated the view that continual learning should remain on-policy, i.e. the model learns from behavior produced by its current policy (Chen et al., 2025). However, extended on-policy RL can plateau and lose coverage after exhausting non-zero reward behaviors to amplify (Yue et al., 2025; Zhang et al., 2026). Furthermore, data containing new knowledge or capabilities are often off-policy, such as new knowledge after the pretraining cutoff Jang et al. (2022), expert demonstrations Taori et al. (2023), delayed feedback from environments Trajectory Labs (2026), and synthetic data for self-improvement Chakraborty et al. (2026). Recent work on on-policy self-distillation (OPSD) tries to bridge this gap by converting off-policy data into on-policy updates, distilling from a solution-conditioned copy of the model (Zhao et al., 2026; Hubotter et al., 2026; Shenfeld et al., 2026). However, conditioning on the solution can suppress exploration, leak privileged information into the teacher’s reasoning, and degrade out-of-domain performance or collapse reasoning length (Kim et al., 2026; Kaur et al., 2026). More fundamentally, this raises a simple question: Is on-policy training necessary for continual learning, or are we using off-policy updates poorly? We premise with a surprisingly simple finding: model merging, i.e., by interpolating the SFT-ed weights back toward the original post-trained model Wortsman et al. (2022b), θmerge=(1−λ)θpost+λθpost′ _merge=(1-λ)\, _post+λ\, _post where θpost′ _post is the model after SFT on a task, SFT is elevated to a strong baseline (Figure 1, left), achieving higher performance on the new task and comparable performance on old tasks. Since merging only scales the SFT update down without changing its direction, this demonstrates SFT learns a useful signal from the new data. However, naive integration of SFT updates clearly interferes with the post-trained model’s capabilities. This leads to a tangible key objective: Figure 1: (Left) Model merging alone greatly improve generalization; Grafting dominates SFT and OPSD. Model: Qwen3-1.7B (thinking). (Right) The best donor model for grafting is even before the end of pretraining. Model: OLMo-3-7B (thinking). Only frontier points are plotted. How can we better minimize the interference induced by off-policy updates? We first rewrite merging as adding a scaled update to the post-trained model, θmerge=θpost+λ(θpost′−θpost) _merge= _post+λ\,( [rgb]0.1602,0.5,0.7266 _post - [rgb]0.1602,0.5,0.7266 _post). Written this way, merging decouples two design choices: where the update is learned (here, naively on θpost _post itself) and how it is applied (here, naively scaled by λ). Task arithmetic (Ilharco et al., 2023) shows that an update need not be derived from the model it is applied to. We argue that this separation is important for continual learning. This is because training on a narrow distribution skews which task a model infers its inputs to come from (Kotha et al., 2024), so finetuning the post-trained model routes much of the new learning signal through the parameters supporting its existing capabilities, while a checkpoint from before post-training routes less of it there (§3). We therefore learn the update on an earlier donor model θdonor [rgb]0.1602,0.5,0.7266 _donor, and apply it to the post-trained model: θpost+λ(θdonor-ft−θdonor) _post+λ\,( [rgb]0.1602,0.5,0.7266 _donor-ft- [rgb]0.1602,0.5,0.7266 _donor), where θdonor-ft _donor-ft is the donor after finetuning. Along the OLMo-3 training trajectory (OLMo Team, 2026), we see that the best donors are checkpoints before the end of pretraining (Figure 1, right), which is perhaps not surprising since later stages of modern pretraining are high-quality data that resembles post-training. Donor-trained updates with naive scaling write less to the parameters on which the post-trained model is most sensitive, but when the new distribution is far from the post-trained model, they can still damage them. We therefore propose to optionally protect the most sensitive directions, estimated with the Fisher information (Kirkpatrick et al., 2017; Ikram et al., 2026). This matters most for knowledge injection (§6.2), where retaining unrelated capabilities is hard. We call the full recipe grafting: θpost+λ⋅ℳρ(θdonor-ft−θdonor) _post+λ· [rgb]0.1523,0.6836,0.375M_ρ( [rgb]0.1602,0.5,0.7266 _donor-ft- [rgb]0.1602,0.5,0.7266 _donor), where the mask ℳρ [rgb]0.1523,0.6836,0.375M_ρ projects away the top-ρ fraction of Fisher-weighted directions from the update. We study three settings where off-policy data arises naturally: (1) distillation from expert traces on hard problems the model cannot solve; (2) self-improvement on hard problems, where models bootstrap themselves without stronger supervision, e.g., via their own rejection-sampled successes (Zelikman et al., 2022) or with indirect supervision, e.g., from thinking traces written by a copy of themselves with privileged information (Chakraborty et al., 2026, Pedagogical RL;), and (3) knowledge injection of new facts absent from prior training. In all three, we assume access only to the data (with our without thinking traces depending on the specific setting), not to the weights or logits of a stronger teacher model, so on-policy distillation (OPD) from a teacher (Agarwal et al., 2023) is not possible. In all three settings, grafting dominates OPSD in both new-task and old-task performance while avoiding expensive on-policy sampling. In self-improvement, grafting roughly doubles new-task performance over the post-trained model while also improving old-task performance. Together, our results suggest that on-policy updates are not a necessary condition for continual learning. Since continual learning applications often involve off-policy experiences, we hope our work inspires more effort on making off-policy training generalize. 2 Preliminaries We consider teaching a post-trained model πθpost _ _post a new target task, formally a set of inputs x\x\ which we want the model to solve. A task may be a skill (e.g., a class of hard math problems) or some knowledge (e.g., new facts after the knowledge cutoff). Two objectives should be optimized jointly: • New task: performance on held-out IID instances of the target task. • Old task: performance on tasks outside the target task – for unrelated tasks, we want less forgetting; for related tasks, we want the new skill or knowledge to generalize to these tasks. Reinforcement learning (RL) is a standard way to learn a task without using off-policy data. It maximizes y∼πθ(⋅∣x)R(x,y)E_y _θ(· x)\,R(x,y) over outputs it samples itself. However, given its mode-seeking nature, RL is known to mostly amplify and compose behaviors the model already possesses (Zhang et al., 2026). For genuinely new knowledge or very hard problems beyond the model’s capability, the model places almost no probability on correct outputs and, therefore, receives no training signal to continue improving. These are the scenarios where off-policy data can be extremely useful. Off-policy data can be formalized as a corpus =(x,y)D=\(x,y)\ of inputs x and outputs y, where x are not generated under πθpost _ _post. Such data is abundant in practice, e.g., new knowledge acquired after the model’s pretraining cutoff, expert demonstrations for a hard task, feedback from new environments, and synthetic data created by the model for self-improvement. Supervised fine-tuning (SFT) has been the standard way to learn from off-policy data, optimizing the model likelihood −(x,y)∼logπθ(y∣x)-E_(x,y) _θ(y x) directly on the outputs y given by the data D. However, SFT is known to memorize rather than generalize (Chu et al., 2025) and to cause catastrophic forgetting of capabilities outside D (McCloskey and Cohen, 1989; Kirkpatrick et al., 2017; Biderman et al., 2024). On-policy self-distillation (OPSD) addresses the shortcomings of SFT by converting the off-policy corpus into on-policy updates (Zhao et al., 2026; Shenfeld et al., 2026; Hubotter et al., 2026). The model samples responses and is supervised at each token by a solution-conditioned version of itself: (x,y)∼y~∼πθ(⋅∣x,y)∑tDα(πθ¯(⋅∣x,y,y~<t)∥πθ(⋅∣x,y~<t)),E_(x,y) \;E_ y _θ(· x,\,y)\, _tD_α\! ( _ θ(· x,y, y_<t)\, \|\, _θ(· x, y_<t) ), where the teacher θ¯ θ is a stop-grad copy of the student , and DαD_α is a family of f-divergences (Agarwal et al., 2023) whose α interpolates from forward KL (α=0α=0) to reverse KL (α=1α=1). 3 Why does SFT interfere with post-trained capabilities? Why does SFT interfere with post-trained capabilities? We argue that this is because models trained on narrow distributions (e.g., post-training and late stages of pretraining) explain new data differently, also known as implicit task inference Kotha et al. (2024). We demonstrate this intuition with a minimal model. Consider two linear regression tasks with data distributions AD_A and BD_B. For each task T∈A,BT∈\A,B\, x∼T,y=βT⊤x,x _T, y= _T x, Task A is a post-trained capability we want to retain, while task B is a capability we want to continually learn. We model the two tasks with a mixture of linear experts, y^(x)=gϕ(x)wA⊤x+(1−gϕ(x))wB⊤x,gϕ(x)=σ(ϕ⊤x). y(x)=g_φ(x)w_A x+(1-g_φ(x))w_B x, g_φ(x)=σ(φ x). We assume pretraining learns both tasks imperfectly but learns their assignment relatively well. On task-B inputs, assume for simplicity that gpre(x)=εD,x∼Bg_pre(x)= _D,x _B. After post-training on task A, the gate becomes more biased toward expert A on the same task-B inputs gpost(x)=εP,x∼Bg_post(x)= _P,x _B, with 0≤εD<εP<10≤ _D< _P<1, which is shown by Kotha et al. (2024). Thus ε measures how much task-B data is routed through expert A. We now analyze fine-tuning on task B. For a checkpoint with task-B assignment ε , under squared loss, LB L_B =12rB⊤ΣBrB,where = 12r_B _Br_B, rB r_B =εwA+(1−ε)wB−βB, = w_A+(1- )w_B- _B, ΣB _B =x∼B[xx⊤]. =E_x _B[x ]. The gradients with respect to the two experts are ∇wALB=εΣBrB,∇wBLB=(1−ε)ΣBrB. _w_AL_B= _Br_B, _w_BL_B=(1- ) _Br_B. Therefore the same task-B learning signal is routed to the two experts in the ratio ‖∇wALB‖∇wBLB‖=ε1−ε. \| _w_AL_B\|\| _w_BL_B\|= 1- . (1) Hence task assignment directly controls how much fine-tuning on task B modifies the expert supporting task A. Since we assume εD<εP _D< _P, finetuning the pretrained model routes less of the task-B learning signal through task A than directly fine-tuning the post-trained model. This is aligned with what we show in §7. 4 Method Algorithm 1 Grafting Input: post-trained model θpost _post, injection data D, scale λ∈(0,1]λ∈(0,1]. 1. θdonor-ft← _donor-ft← finetune θdonor _donor on D 2. Δ←θdonor-ft−θdonor ← _donor-ft- _donor 3. ℳρ←M_ρ← compute projection from Eq. 5 4. θ(λ,ρ)←θpost+λ⋅ℳρ(Δ)θ(λ,ρ)← _post+λ·M_ρ( ) return θ(λ,ρ)θ(λ,ρ) Our method grafting consists of two main components: a scaled parameter delta derived from training on a donor model (i.e., a checkpoint from earlier in training), and an optional projection mask to protect sensitive weight directions when the new task data is far from the post-trained distribution. We display the core recipe in Algorithm 1. Extract updates from a donor model Task arithmetic (Ilharco et al., 2023) shows that an update need not be derived from the model it is applied to. We argue that this separation is important for continual learning. This is because training on a narrow distribution skews which task a model infers its inputs to come from (Kotha et al., 2024), so finetuning the post-trained model routes much of the new learning signal through the parameters supporting its existing capabilities, while a checkpoint from before post-training routes less of it there (§3). Therefore, instead of directly finetuning the post-trained model θpost _post, we finetune an earlier checkpoint which we call a “donor model” θdonor _donor. Specifically, we finetune θdonor _donor to obtain θdonor-ft _donor-ft, and isolate the difference Δ=θdonor-ft−θdonor = _donor-ft- _donor. We then apply the update scaled by λ to θpost _post: θ(λ)=θpost+λ(θdonor-ft−θdonor).θ(λ)\;=\; _post\;+\;λ\, ( _donor-ft- _donor ). (2) While λ∈(0,1]λ∈(0,1] controls the scale of the graft, we remark that λ admits an interpretation that goes beyond interpolating between old and new task performance. This arises from the model merging literature (see e.g., WiSE-FT, Wortsman et al., 2022b), where linearly interpolating a later checkpoint with an earlier one is known to benefit generalization. This suggests that moderate λ can be beneficial, which is reflected by our experiments on grafting from hard math data. Protect sensitive weights via projection As we will show, applying a donor-derived update directly may not be enough when the injected distribution is far from the post-trained model (e.g., injecting new facts into a reasoning model). However, precisely characterizing weight interference is complex. We take inspiration from on-policy methods, and consider a surrogate goal of shaping the update direction such that the updated model modifies the behavior on the old domain as little as possible. Concretely, we want an update Δ that keeps the conditional KL divergence z∼old[DKL(pθpost(⋅|z)∥pθpost+Δ(⋅|z))]E_z _old [D_KL(p_ _post(·\,|\,z) p_ _post+ (·\,|\,z)) ] over the old domain small. To efficiently translate this to the directions of Δ to mask, we recall the standard quadratic approximation of KL (Cover et al., 1991): DKL(pθpost(⋅|z)∥pθpost+Δ(⋅|z)) D_KL(p_ _post(·\,|\,z) p_ _post+ (·\,|\,z)) ≈12Δ⊤Fz(θpost)Δ, ≈ 12 F_z( _post) , (3) where Fz(θpost)=y∼pθpost(⋅|z)[(∇θlogpθpost(y|z))⊗2]F_z( _post)=E_y p_ _post(·\,|\,z)\! [ ( _θ p_ _post(y\,|\,z) ) 2 ] is the conditional Fisher Information evaluated at θpost _post. This can be estimated with a retention set ret=z1,…,zND_ret=\z_1,…,z_N\ of inputs from, e.g., an old task: Fret:=1N∑n=1Ny∼pθpost(⋅|zn)[(∇θlogpθpost(y|zn))⊗2].F_ret:=\; 1N _n=1^NE_y p_ _post(·\,|\,z_n)\! [ ( _θ p_ _post(y\,|\,z_n) ) 2 ]. (4) However, the full Fisher is infeasible to compute. We consider existing approaches that take a diagonal (i.e., entrywise) approximation (Kirkpatrick et al., 2017). We also propose another efficient approximation informed by the Kronecker-Factored Approximate Curvature (KFAC) (Martens and Grosse, 2015), similar to approaches in model-editing literature (Ikram et al., 2026), which we can have achieve better performance with less masking than the diagonal approximation. For concreteness, we walk through deriving the mask from the diagonal approximation. Measuring the Fisher quadratic form along the diagonal: Δ⊤diag(Fret)Δ=∑j=1dθ(Fret)jjΔj2 (F_ret) = _j=1^d_θ(F_ret)_j _j^2, we observe the entry-wise score sj=(Fret)jjΔj2s_j=(F_ret)_j\, _j^2 approximates how much entry j in the update Δ contributes to the model change (Eq. 3). Therefore, a natural approach to reducing the model change induced by Δ is to mask the top ρ-fraction of entries SρS_ρ ranked by sjs_j: θ(λ,ρ)=θpost+λ(ℳρ⊙Δ),(ℳρ)j=0j∈Sρ,1otherwise,θ(λ,ρ)\;=\; _post\;+\;λ\, (M_ρ ), (M_ρ)_j\;=\; cases0&j∈ S_ρ,\\[2.0pt] 1&otherwise, cases (5) where ⊙ is the element-wise product. We defer analogous derivation of the KFAC-based projection and other implementation details in this section to Appendix F. In our experiments, we show that retention-based projections derived from the Diagonal or KFAC Fisher work remarkably well in removing interference from donor updates by just masking a small number of directions (ρ≤0.01ρ≤ 0.01), outperforming prior Fisher-based merging (Matena and Raffel, 2022), and pushing the old/new-domain performance frontier compared to un-projected updates. 5 Experiments on Learning Hard Math Problems 5.1 Experimental setup Models We experiment with two model pairs: Qwen3-1.7B (thinking) / Qwen3-1.7B-Base Yang et al. (2025) and Qwen3-8B (thinking) / Qwen3-8B-Base as the post-trained model θpost _post and the donor model θdonor _donor. Our experiments include both thinking and instruct modes. Unless stated otherwise the donor model is the officially released base model of the same family. In §5.3 we additionally use OLMo-3-7B (thinking) (OLMo Team, 2026) as θpost _post, because it releases intermediate checkpoints along both its pretraining and post-training trajectories, which lets us vary θdonor _donor rather than fix it to the official base model. Math data We consider three settings for where the off-policy data comes from (see §A): • Distillation with teacher traces: we get teacher traces from the AoPS subset of Nemotron-SFT-Math-v4 (NVIDIA, 2026; Du et al., 2025), whose solutions are generated by DeepSeek-V4-Pro (DeepSeek-AI, 2026). At both scales we mine hard problems from this pool that the post-trained model cannot solve. For each model, we sample 1616 attempts per problem and keep those with 00 success for 1.7B and ≤1≤ 1 success for 8B, leaving 1,5331,533 training samples for 1.7B and 985985 for 8B. • Self-improvement with rejection sampling (STaR; Zelikman et al., 2022): we apply rejection sampling to the post-trained model to get supervision of both thinking and solution tokens. We sample up to 3232 attempts per problem and keep one verified trace for each solved problem. • Self-improvement with Pedagogical RL (Chakraborty et al., 2026): STaR requires that the model can solve the problem with a large budget. Pedagogical RL removes this requirement by synthesizing reasoning traces from gold solutions and do SFT on the synthesized traces (Appendix B). We find replacing the SFT step of Pedagogical RL with grafting largely improves performance. Evaluation For math, new-task performance is measured on around 100100 held-out hard math problems from the training set. Old-task performance is the mean over three benchmarks (AIME 2024, AIME 2025, HMMT 2025). All metrics are estimated over 1616 samples at temperature 1.01.0 and top-p 1.01.0, decoding up to 3232K tokens in thinking mode, unless otherwise specified. Appendix C gives the training details. 5.2 Model merging alone improves both old and new tasks We first show that model merging already improves both old and new tasks. We consider three methods: SFT, OPSD, and grafting, with full details given in Appendix C. In Figure 2, we sweep all methods with 99 learning rates (5×10−75×10^-7 to 5×10−55×10^-5) and train 55 epochs. Figure 2 (left) shows that, without model merging, neither SFT nor OPSD outperforms the post-trained model, while grafting with the best hyperparameters does. We then sweep the merging scale for all methods in Figure 2 (right). We see that merging improves generalization and erases OPSD’s original edge over SFT, despite its slow and expensive on-policy sampling. Given its uniform benefit, we apply merging across all methods in the rest of the paper. Figure 2: Model merging improves both old and new task performance. 5.3 Optimal θdonor _donor is before the end of pretraining OLMo-3-7B (thinking) (OLMo Team, 2026) has many checkpoints for us to move θdonor _donor freely. On the post-training side, we have base model →SFT SFT SFT checkpoint →DPO DPO DPO checkpoint →RL RL OLMo-3-7B (thinking); on the pretraining side, we move θdonor _donor up to 1212k steps before the base model. We first investigate how each post-training stage makes finetuning harder. We see in Figure 3 (left) that grafting (base ckpt) outperforms grafting (SFT ckpt), grafting (SFT ckpt) outperforms grafting (DPO ckpt), and grafting (DPO ckpt) outperforms direct SFT on the final post-trained model. This suggests that each stage of post-training (SFT, DPO, RL) makes OLMo-3 harder to finetune. The question that remains is which pretraining checkpoint to use as the donor model. There is a tension in this choice. The end of pretraining anneals on higher-quality data and already resembles an early stage of SFT, so an earlier θdonor _donor should be easier to finetune, but it also sits farther from θpost _post, which makes it less likely to stay linearly connected (Frankle et al., 2020; Neyshabur et al., 2020) with θpost _post. Figure 3 (right) moves θdonor _donor up to 1212k steps before the final base checkpoint. We see that earlier pretrained checkpoints clearly outperforms the final pretrained checkpoint, while the gain is not strictly monotonic. This suggests that for grafting we should search for an early pretrained checkpoint that balances the two forces. Qwen3 releases only its final base model and no earlier pretraining checkpoints, so in the rest of the paper we use the official base model as θdonor _donor. Figure 3: Grafting from different θdonor _donor with OLMo-3-7B (thinking). (Left) θdonor _donor is varied along the post-training trajectory: base model →SFT SFT SFT checkpoint →DPO DPO DPO checkpoint →RL RL OLMo-3-7B (thinking). (Right) θdonor _donor is moved back along pretraining, up to 1212k steps before the base model. (a) Qwen3-1.7B (thinking) (b) Qwen3-8B (thinking) (c) Qwen3-8B (instruct) Figure 4: Distillation from teacher traces across different models. 5.4 Results for distillation with teacher traces Thinking mode Figure 4(a) and Figure 4(b) show results with teacher traces as off-policy data, on Qwen3-1.7B (thinking) and Qwen3-8B (thinking), respectively. The same merging sweep (§5.2) is applied for all methods. We see that, for 1.7B, grafting dominates both SFT and OPSD and is the only method that meaningfully dominates the post-trained model. The same ordering holds at pass@4 and pass@8, suggesting that the gain reflects expanded capability and not simply a “sharpening effect”. For 8B, both grafting and SFT dominate OPSD. For both sizes, OPSD fails to largely outperform the original post-trained model for generalization, which echoes the findings in Kim et al. (2026); Kaur et al. (2026). Non-thinking mode (aka. instruct mode) We repeat the experiment at 8B in non-thinking mode on DAPO (Yu et al., 2025), an easier data for the non-thinking mode to learn, and we follow the thinking mode experiments to mine hard problems (see Appendix A for details). The same merging sweep (§5.2) is applied for all methods. We see from Figure 4(c) that grafting again dominates SFT and OPSD at pass@11. 5.5 Self-improvement with rejection sampling (STaR) Figure 5: Self-improvement with rejection sampling (STaR) on Qwen3-1.7B (thinking). We now turn to the second setting, where the off-policy data come from the post-trained model itself via rejection sampling (§5.1), also known as STaR (Zelikman et al., 2022). Specifically, we sample up to 3232 attempts per problem and keep one verified trace for each solved problem. Unlike distillation from teacher traces, STaR lets the model self-improve by training on its own data. Figure 5 shows that new-task performance is roughly the same compared to the strongest baseline (i.e., SFT, which stands for the original STaR) but generalization to old task rises about eight points above it. Compared to distillation from teacher traces (Figure 4(a)), we see that STaR is generally worse in terms of new-task performance but similar for generalization to the old task. 5.6 Self-improvement with Pedagogical RL Figure 6: Self-improvement with Pedagogical RL on Qwen3-1.7B (thinking). Pedagogical RL (Chakraborty et al., 2026) takes a different route to self-improvement, where a copy of the model that sees the reference solution (without seeing thinking traces) is trained with RL to write thinking traces the model can learn from (Appendix B). With these self-generated thinking traces, the original method then learns with SFT (Pedagogical RL + SFT in Figure 6). Compared to direct distillation from teacher traces, Pedagogical RL does not need access to thinking traces, which is more practical since solutions from humans or expert systems do not expose thinking traces. We keep the RL training and thinking trace generation steps intact and propose to learn the same traces with grafting. Since Pedagogical RL is designed to generate thinking traces and OPSD does not use them, we reuse the OPSD results from STaR (§5.5). Figure 6 shows that grafting again dominates SFT and OPSD. With Pedagogical RL, grafting learns the new task better and generalizes to the old task, while SFT loses in both directions. 6 Experiments on Knowledge Injection 6.1 Experimental setup Models We experiment with Qwen3-1.7B (thinking) as the post-trained model θpost _post and Qwen3-1.7B-Base Yang et al. (2025) as the donor model θdonor _donor. Knowledge data Our data for knowledge acquisition mainly follows Shenfeld et al. (2026), but with the key exception that we use purely synthetic knowledge to avoid any possible leakage. We construct a fictional world with 200200 invented entities and 1,1251,125 atomic facts. Each fact is written as question-answer pairs in 88 distinct phrasings, and the model is trained on those; evaluation then uses held-out paraphrases the model has never seen. For OPSD, we write wiki-style articles that state the fictional facts for conditioning. Since the knowledge data’s QA form is very different from a reasoning model’s, we mix knowledge QA with AoPS teacher traces we use in the math experiments. Together, we have 9,0009,000 QA samples and 1,5331,533 AoPS traces. All methods are trained on this same mixture. Evaluation For knowledge, new-task performance is the exact-match accuracy on held-out paraphrases the model never saw, decoded greedily. Old-task performance is the mean over three benchmarks (AIME 2024, AIME 2025, HMMT 2025). Appendix C gives the training details. Figure 7: (Left) Knowledge injection results. (Right) Disentangling the contribution of grafting’s components. The protection mask (ρ sweep) dominates no mask (λ sweep). 6.2 Knowledge injection results Figure 7 (left) shows the knowledge injection result with Qwen3-1.7B (thinking), where the same model merging sweep (§5.2) is applied to all methods. We see that grafting achieves nearly perfect injection with retention at or above the post-trained model. SFT injects just as well but suffers from forgetting. OPSD achieves near-perfect retention but also learns less. Within OPSD, forward KL (α=0α=0) learns much more than JSD (α=0.5α=0.5) and reverse KL (α=1α=1). This shows that learning a new fact requires a mode-covering objective rather than mode-seeking. 7 Analysis Ablation studies Here we ablate the mask step that zeroes the most sensitive coordinates. Figure 7 (right) compares the protection mask (ρ sweep, in red) with plain model merging without the mask (λ sweep, in gray). We see that the protection mask Pareto-dominates plain model merging. Figure 7 (right) also shows Fisher merging (Matena and Raffel, 2022), which weights each coordinate by its Fisher ratio between the two tasks. We see that Fisher merging does not inject at all in this setting. We find that this is because the knowledge injection Fisher is usually dominated by the retention Fisher. Finally, the KFAC mask (§4, in navy) reaches the same old-task performance as the diagonal mask while zeroing far fewer directions: at ρ=0.001ρ=0.001 it keeps new-task performance at 0.9950.995 with old-task performance 0.3900.390, which the diagonal mask needs ρ=0.01ρ=0.01 to match. We note that KFAC being a better approximant to the retention Fisher does not directly imply interference must be more concentrated to few directions. The fact that retention of old tasks and injected knowledge can be achieved by zero-ing very few directions is in line with parallel work demonstrating the top few directions of curvature contain mostly interference (Zhu et al., 2026), and can be removed with little effect on task performance (Song et al., 2025). Figure 8: (Left) Grafting is much faster to train than OPSD. We show the per-step sampling and weight update time for Qwen3-8B (thinking). (Right) Grafting writes less to sensitive coordinates than SFT. We show the weighted delta ∑jFret(j)Δj2 _jF_ret(j) _j^2 summed across the 1%1\% of coordinates with the largest FretF_ret. Training latency Off-policy training also gets rid of the latency cost of on-policy sampling. We show in Figure 8 (left) the per-step wall clock time for both sampling and weight update for grafting and OPSD, both with Qwen3-8B (thinking). We see that sampling is roughly 90%90\% of the step. In other words, grafting is 10×10× faster to train than OPSD, while achieving much better performance. Training on donor model writes less to sensitive weights We show in Figure 8 (right) the Fisher-weighted weight delta ∑jFret(j)Δj2 _jF_ret(j) _j^2 that falls inside the top 1%1\% of coordinates with the largest FretF_ret, normalized by that of all coordinates. In other words, this number shows how much a weight update writes into the most sensitive coordinates of the post-trained weights. We first see that the 1%1\% coordinates explain nearly 75%75\% of the weight changes, which empirically justifies the protection mask. More importantly, we see that grafting writes less to the sensitive weights than SFT, for both math and knowledge learning tasks, which helps explain why the protection mask alone cannot push SFT to the same frontier as grafting, as we see above. In Section 3, we show how this effect can be predicted by a minimal linear model. 8 Related Work Continual learning for post-trained models. Continually trained networks can overwrite earlier capabilities (McCloskey and Cohen, 1989). Classic remedies constrain the update written into the parameters, e.g., elastic weight consolidation (Kirkpatrick et al., 2017). For language models, it has been shown that low-rank updates (Hu et al., 2022) forget less but also learn less (Biderman et al., 2024). SFT and RL learn in different ways (Chu et al., 2025; Matsutani et al., 2025), and SFT in particular memorizes at the cost of generalization. This echoes the common belief is that continual training should be on-policy, and has been demonstrated by the success of continual training models using RL with verifiable rewards (RLVR) (DeepSeek-AI, 2025). Chen et al. (2025) attribute RL’s smaller forgetting specifically to its use of on-policy data, rather than to other algorithmic choices such as KL regularization or advantage estimation. In practice, however, most data for continual learning is off-policy, such as knowledge after the pretraining cutoff (Jang et al., 2022), expert demonstrations (Taori et al., 2023), delayed feedback from environments (Trajectory Labs, 2026), and synthetic data for self-improvement Zelikman et al. (2022). On-policy self-distillation and its failure modes. OPSD converts off-policy data into an on-policy learning signal, by distilling a solution-conditioned copy of the model into itself on its own samples (Zhao et al., 2026; Shenfeld et al., 2026; Hubotter et al., 2026; Penaloza et al., 2026). The approach is being scaled in practice (Trajectory Labs, 2026). Its benefit, however, appears to hold only within a limited scope of tasks. Reported failure modes include suppressed exploration, degeneration in reasoning length, and failure to generalize (Kim et al., 2026; Kaur et al., 2026). Recent work responds to these failures by filtering the part of the teacher signal that does not transfer to the student (Shen et al., 2026) or by routing token-level supervision between privileged teacher and student policies (Yu et al., 2026). Model merging and weight-space operations. Weight-space operations are promised by loss-landscape geometry (Frankle et al., 2020; Neyshabur et al., 2020; Ainsworth et al., 2023), and a large literature does weight interpolation with models trained or finetuned with the same initialization (Wortsman et al., 2022a; Matena and Raffel, 2022; Li et al., 2022; Wortsman et al., 2022b; Ilharco et al., 2022), with refinements that resolve interference, linearize fine-tuning, or localize a skill in a sparse subset of the delta (Yadav et al., 2023; Yu et al., 2024; Ortiz-Jimenez et al., 2023; Panigrahi et al., 2023). Model merging has been shown to improve robustness (Wortsman et al., 2022b) and generation diversity (Dang et al., 2025). Most similar to our work, task arithmetic Ilharco et al. (2023), also known as Shadow-FT (Wu et al., 2025) or chat vector Huang et al. (2024), composes weight deltas trained on the base model. We improve over them with the proposed Fisher masking and donor model selection. Fisher merging (Matena and Raffel, 2022) derives a soft weighting between multiple tasks that train the model, while we derive a sparse mask using only the retention data and show that retention-only sparse masking is substantially more effective. Models become harder to finetune. It has been known that training changes how well a model takes a further update, or plasticity (Ash and Adams, 2020; Dohare et al., 2024). Recent work demonstrates this for large language models. The longer a base model is pretrained beyond a certain point, the more sensitive its parameters become and the worse it finetunes (Springer et al., 2025). To mitigate such overtraining effect, it has been proposed to increase learning rate (Rofin et al., 2026), or more broadly, to bias pretraining toward flatter minima (Watts et al., 2026). Our experiments on varying the donor model for grafting point the same way: an earlier pretraining checkpoint can be a better θdonor _donor than the final base model and later post-trained intermediate checkpoints. Different from previous work, we show that the harder-to-finetune effect can be mitigated not only by training an easier-to-finetune model, but also by better continual update rules. 9 Conclusion and Future Work We asked whether continual learning for RL-trained models has to be on-policy, and our results suggest that it does not. Model merging alone shows that SFT learns a useful signal from new data, but naively applying its update interferes with existing capabilities. Building on that, grafting reduces this interference by learning the update on an earlier donor checkpoint, scaling it, and optionally masking the most sensitive update directions. Across three settings, grafting Pareto-dominates both SFT and OPSD in new-task and old-task performance, while avoiding expensive on-policy sampling. We show that w donor model to use for grafting turns out to be crucial too. Earlier pretraining checkpoints are usually a better choice, though future work can study more principled approaches to select the optimal pretraining checkpoint for grafting. Another direction to optimize for future work is to predict optimal λ and ρ instead of a sweep. More broadly, since off-policy data is what continual learning mostly has, we hope this work encourages more effort on making off-policy training generalize. Acknowledgments We wish to thank Jacob Springer, Gaurav Ghosal, Ruslan Salakhutdinov, Weichen Yu, Neil Kale, Jingchu Gai, Christina Baek, Ziqian Zhong, Kevin Wang for their discussion and feedback. This research is partially supported by the DARPA expMath program through the DARPA CMO contract number HR0011262E028. We gratefully acknowledge support from Schmidt Sciences, NSF, Apple, Open Philanthropy, Google. We would like to thank CMU FLAME Cluster for providing generous compute. References Agarwal et al. (2023) R. Agarwal, N. Vieillard, Y. Zhou, P. Stańczyk, S. Ramos, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Cited by: §1, §2. Ainsworth et al. (2023) S. K. Ainsworth, J. Hayase, and S. Srinivasa Git re-basin: merging models modulo permutation symmetries. In International Conference on Learning Representations (ICLR), Cited by: §8. Ash and Adams (2020) J. T. Ash and R. P. Adams On warm-starting neural network training. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §8. Biderman et al. (2024) D. Biderman, J. Portes, J. J. G. Ortiz, M. Paul, P. Greengard, C. Jennings, D. King, S. Havens, V. Chiley, J. Frankle, C. Blakeney, and J. P. Cunningham LoRA learns less and forgets less. Transactions on Machine Learning Research. Cited by: §2, §8. Chakraborty et al. (2026) S. Chakraborty, N. Ziems, F. Huang, M. Jiang, A. S. Bedi, and O. Khattab Pedagogical RL: teaching models to teach themselves from privileged information. Note: https://noahziems.com/pedagogical-rl Cited by: Appendix B, Appendix B, §1, §1, 3rd item, §5.6. Chen et al. (2025) H. Chen, N. Razin, K. Narasimhan, and D. Chen Retaining by doing: the role of on-policy data in mitigating forgetting. arXiv preprint arXiv:2510.18874. Cited by: §1, §8. Chu et al. (2025) T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S. Levine, and Y. Ma SFT memorizes, rl generalizes: a comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161. Cited by: §1, §2, §8. Cover et al. (1991) T. M. Cover, J. A. Thomas, and J. Kieffer Elements of information theory. Vol. 2, wiley New York. Cited by: §4. Dang et al. (2025) X. Dang, C. Baek, K. Wen, J. Z. Kolter, and A. Raghunathan Weight ensembling improves reasoning in language models. arXiv preprint arXiv:2504.10478. Cited by: §8. DeepSeek-AI (2025) DeepSeek-AI DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645, p. 633–638. Cited by: §8. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-v4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: 1st item. Dohare et al. (2024) S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood, and R. S. Sutton Loss of plasticity in deep continual learning. Nature 632, p. 768–774. Cited by: §8. Du et al. (2025) W. Du, S. Toshniwal, B. Kisacanin, S. Mahdavi, I. Moshkov, G. Armstrong, S. Ge, E. Minasyan, F. Chen, and I. Gitman Nemotron-math: efficient long-context distillation of mathematical reasoning from multi-mode supervision. arXiv preprint arXiv:2512.15489. Cited by: 1st item. Frankle et al. (2020) J. Frankle, G. K. Dziugaite, D. M. Roy, and M. Carbin Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning (ICML), Cited by: §5.3, §8. George et al. (2018) T. George, C. Laurent, X. Bouthillier, N. Ballas, and P. Vincent Fast approximate natural gradient descent in a kronecker factored eigenbasis. Advances in neural information processing systems 31. Cited by: Appendix F. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: §8. Huang et al. (2024) S. Huang, P. Li, Y. Hsu, K. Chen, Y. T. Lin, S. Hsiao, R. T. Tsai, and H. Lee Chat vector: a simple approach to equip llms with instruction following and model alignment in new languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §8. Hubotter et al. (2026) J. Hubotter, F. Lubeck, L. D. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. K. Buening, C. Guestrin, and A. Krause Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802. Cited by: §1, §2, §8. Ikram et al. (2026) Z. Ikram, A. Firouzkouhi, S. Tu, M. Soltanolkotabi, and P. Rashidinejad CrispEdit: low-curvature projections for scalable non-destructive LLM editing. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §1, §4. Ilharco et al. (2023) G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi Editing models with task arithmetic. In International Conference on Learning Representations (ICLR), Cited by: §1, §4, §8. Ilharco et al. (2022) G. Ilharco, M. Wortsman, S. Y. Gadre, S. Song, H. Hajishirzi, S. Kornblith, A. Farhadi, and L. Schmidt Patching open-vocabulary models by interpolating weights. In Advances in Neural Information Processing Systems, Cited by: §8. Jang et al. (2022) J. Jang, S. Ye, S. Yang, J. Shin, J. Han, G. Kim, S. J. Choi, and M. Seo Towards continual knowledge learning of language models. In International Conference on Learning Representations (ICLR), Note: arXiv:2110.03215 Cited by: §1, §8. Kaur et al. (2026) S. Kaur, N. Ri, Y. He, L. Fowl, and S. Arora Rethinking on-policy self-distillation for thinking models. arXiv preprint arXiv:2607.05184. Cited by: §1, §5.4, §8. Kim et al. (2026) J. Kim, X. Luo, M. Kim, S. Lee, D. Kim, J. Jeon, D. Li, and Y. Yang Why does self-distillation (sometimes) degrade the reasoning capability of llms?. arXiv preprint arXiv:2603.24472. Cited by: §1, §5.4, §8. Kirkpatrick et al. (2017) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13), p. 3521–3526. Cited by: §1, §1, §2, §4, §8. Kotha et al. (2024) S. Kotha, J. M. Springer, and A. Raghunathan Understanding catastrophic forgetting in language models via implicit inference. In International Conference on Learning Representations, Cited by: §1, §3, §3, §4. Li et al. (2022) M. Li, S. Gururangan, T. Dettmers, M. Lewis, T. Althoff, N. A. Smith, and L. Zettlemoyer Branch-train-merge: embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03306. Cited by: §8. Martens and Grosse (2015) J. Martens and R. Grosse Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, p. 2408–2417. Cited by: Appendix F, §4. Matena and Raffel (2022) M. Matena and C. Raffel Merging models with fisher-weighted averaging. In Advances in Neural Information Processing Systems, Cited by: §4, §7, §8. Matsutani et al. (2025) K. Matsutani, S. Takashiro, G. Minegishi, T. Kojima, Y. Iwasawa, and Y. Matsuo RL squeezes, sft expands: a comparative study of reasoning llms. arXiv preprint arXiv:2509.21128. Cited by: §8. McCloskey and Cohen (1989) M. McCloskey and N. J. Cohen Catastrophic interference in connectionist networks: the sequential learning problem. In Psychology of Learning and Motivation, Vol. 24, p. 109–165. Cited by: §1, §2, §8. Neyshabur et al. (2020) B. Neyshabur, H. Sedghi, and C. Zhang What is being transferred in transfer learning?. In Advances in Neural Information Processing Systems, Cited by: §5.3, §8. NVIDIA (2026) NVIDIA Nemotron-sft-math-v4. Note: https://huggingface.co/datasets/nvidia/Nemotron-SFT-Math-v4Hugging Face dataset Cited by: Appendix A, 1st item. OLMo Team (2026) OLMo Team OLMo 3: charting a path through the model flow. arXiv preprint. Cited by: Appendix E, Appendix E, §1, §5.1, §5.3. Ortiz-Jimenez et al. (2023) G. Ortiz-Jimenez, A. Favero, and P. Frossard Task arithmetic in the tangent space: improved editing of pre-trained models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §8. Panigrahi et al. (2023) A. Panigrahi, N. Saunshi, H. Zhao, and S. Arora Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning (ICML), Cited by: §8. Penaloza et al. (2026) E. Penaloza, D. Vattikonda, N. Gontier, A. Lacoste, L. Charlin, and M. Caccia Privileged information distillation for language models. arXiv preprint arXiv:2602.04942. Cited by: §8. Rofin et al. (2026) M. Rofin, A. Varre, and N. Flammarion (How) learning rates regulate catastrophic overtraining. arXiv preprint arXiv:2604.13627. Cited by: §8. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: Appendix B. Shen et al. (2026) Z. Shen, J. Tong, S. Yan, C. Shen, H. Chen, W. Ye, X. Hu, R. Miao, H. Wang, J. Zhao, G. Chen, and J. Ye Purified opsd: on-policy self-distillation without losing how to think. arXiv preprint arXiv:2607.02234. Cited by: §8. Shenfeld et al. (2026) I. Shenfeld, M. Damani, J. Hübotter, and P. Agrawal Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897. Cited by: §1, §2, §6.1, §8. Song et al. (2025) M. Song, K. Ahn, and C. Yun Does sgd really happen in tiny subspaces?. In International Conference on Learning Representations, Vol. 2025, p. 8086–8120. Cited by: §7. Springer et al. (2025) J. M. Springer, S. Goyal, K. Wen, T. Kumar, X. Yue, S. Malladi, G. Neubig, and A. Raghunathan Overtrained language models are harder to fine-tune. In International Conference on Machine Learning (ICML), Note: arXiv:2503.19206 Cited by: §8. Taori et al. (2023) R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto Stanford alpaca: an instruction-following llama model. GitHub. Note: https://github.com/tatsu-lab/stanford_alpaca Cited by: §1, §8. Trajectory Labs (2026) Trajectory Labs Scaling sdpo. Note: https://trajectory.ai/field-notes/scaling-sdpoField notes; cited for delayed environment feedback as an off-policy data source Cited by: §1, §8, §8. Watts et al. (2026) I. Watts, C. Li, S. Goyal, J. M. Springer, and A. Raghunathan Sharpness-aware pretraining mitigates catastrophic forgetting. arXiv preprint arXiv:2605.02105. Cited by: §8. Wortsman et al. (2022a) M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the 39th International Conference on Machine Learning (ICML), Cited by: §8. Wortsman et al. (2022b) M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. Gontijo-Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong, and L. Schmidt Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §4, §8. Wu et al. (2025) T. Wu, R. Yang, J. Li, P. Hu, Y. Wu, N. Wong, and Y. Yang Shadow-ft: tuning instruct model via training on paired base model. arXiv preprint arXiv:2505.12716. Cited by: §8. Yadav et al. (2023) P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal TIES-merging: resolving interference when merging models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §8. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §5.1, §6.1. Yu et al. (2024) L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li Language models are super mario: absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning (ICML), Cited by: §8. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang DAPO: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: Appendix A, §5.4. Yu et al. (2026) X. Yu, G. Li, Q. Si, G. Zhang, Y. Xu, C. Wang, S. Dong, K. Tuo, X. Zeng, K. Feng, Q. Wang, Y. Shi, X. Hu, X. Yue, J. Wang, and S. Yan DOPD: dual on-policy distillation. arXiv preprint arXiv:2606.30626. Cited by: §8. Yue et al. (2025) Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, and G. Huang Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1. Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman STaR: bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), p. 15476–15488. Cited by: §1, 2nd item, §5.5, §8. Zhang et al. (2026) C. Zhang, G. Neubig, and X. Yue On the interplay of pre-training, mid-training, and rl on reasoning language models. In Forty-third International Conference on Machine Learning, Cited by: §1, §2. Zhao et al. (2026) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. Cited by: §1, §2, §8. Zhu et al. (2026) K. Zhu, Z. Jin, S. Tu, H. Yuan, Y. Bai, K. Liu, J. Li, and J. Zhao SFT conflicts, rl coexists: a theoretical and empirical analysis of multi-task learning for llms. arXiv preprint arXiv:2608.03573. Cited by: §7. Appendix A Data Knowledge (synthetic closed world). To guarantee the facts are new, we build a closed fictional world rather than reuse public corpora: 200200 invented entities and 1,1251,125 atomic facts about them. Each fact is written as question-answer pairs in 88 distinct phrasings, giving 9,0009,000 QA pairs, and new-task performance is measured on held-out phrasings never seen in training, so recall cannot be inflated by memorized surface form. For OPSD we additionally write 200200 wiki-style articles that state the facts, used only for conditioning. The facts are absent from the post-trained model: closed-book exact-match is ≈0%≈\!0\% (instruct 0.09%0.09\%, base 0.13%0.13\%). Answers are scored by rule-based <answer> exact-match. All methods train on the same mixture of the 9,0009,000 QA pairs and the 1,5331,533 AoPS traces of the math corpus. Math reasoning, Qwen3-1.7B (thinking). Training traces are competition math solutions sampled from the AoPS split of Nemotron-SFT-Math [NVIDIA, 2026]. We mine the hard subset by sampling Qwen3-1.7B (thinking) 1616 times per problem at temperature 1.01.0 and keeping only problems it fails on all 1616 attempts, so the target skill is absent. After length-filtering and deduplication we keep 1,5331,533 training traces, and hold out a disjoint set of 111111 mined-hard problems to measure new-task performance. Math reasoning, Qwen3-8B (thinking). At 8B we mine the same pool with a looser criterion (so we have a meaningful number of samples), keeping problems the post-trained model solves at most once in 1616 attempts, which leaves 985985 training samples. A disjoint set of 100100 problems is held out to measure new-task performance. Math reasoning, Qwen3-8B (non-thinking). The non-thinking experiment uses problems from DAPO Yu et al. [2025]. Hard problems are mined the same way as above, by sampling Qwen3-8B in non-thinking mode at temperature 1.01.0 and keeping only the problems it never solves, and the worked solutions for those problems are the off-policy data. This leaves 1,1631,163 training samples, with a disjoint set of 200200 problems held out to measure new-task performance. Appendix B Pedagogical RL Pedagogical RL [Chakraborty et al., 2026] improves a model in two stages: (1) it synthesizes reasoning traces from problems and their reference solutions, and (2) it finetunes the model on the synthesized traces with SFT. We keep stage (1) unchanged and replace SFT in stage (2) with grafting. Notation Following Chakraborty et al. [2026], let x denote a problem, c its reference solution without reasoning, and τ the reasoning trace to be synthesized. Two copies of the model play different roles: a privileged teacher πθT(τ∣x,c), _ _T(τ x,c), whose prompt contains the reference solution and instructs it to solve the problem in its own way rather than copy c, and a student πθS(τ∣x), _ _S(τ x), which sees only the problem. Teacher training The teacher is trained to write traces that are both correct and plausible under the student. For each position t of a trace, let the student’s most likely next token be atmax=argmaxaπθS(a∣x,τ<t),a_t = _a _ _S (a x, _<t ), and define the surprise gap dt=logπθS(atmax∣x,τ<t)πθS(τt∣x,τ<t),d_t= _ _S (a_t x, _<t ) _ _S ( _t x, _<t ), which is zero when the trace follows the student’s top choice and grows as the trace takes tokens the student finds unlikely. The spike-aware learnability score aggregates the gaps over the T tokens of the trace, GspikeθS(τ∣x)=exp[−λβlog(1T∑t=1Texp(βdt))],G_spike _S(τ x)= [- λβ ( 1T _t=1^T (β d_t) ) ], where β interpolates between penalizing the mean gap (β→0β→ 0) and the largest gap (β→∞β→∞), and λ sets the strength of the penalty. The pedagogical reward multiplies the learnability score with the correctness reward R, rped(x,c,τ)=R(x,c,τ)GspikeθS(τ∣x),r_ped(x,c,τ)=R(x,c,τ)\,G_spike _S(τ x), and the teacher is trained with GRPO [Shao et al., 2024] to maximize maxθTτ∼πθT(⋅∣x,c)[rped(x,c,τ)]. _ _T\;E_τ _ _T(· x,c) [r_ped(x,c,τ) ]. The student πθS _ _S stays fixed during teacher training and only scores how learnable the teacher’s traces are. We denote the trained teacher by π~T π_T. Trace synthesis For each problem and its reference solution, we sample reasoning traces τ∼π~T(⋅∣x,c)τ π_T(· x,c) from the trained teacher. These traces are the training data for stage (2). Problems and solutions The pairs (x,c)\(x,c)\ are the 1,5331,533 training samples of the 1.7B teacher-trace setting (§5.1), covering 628628 unique problems: x is the problem and c is the written solution of the teacher trace with its reasoning removed. Each pair also carries the problem’s final answer, which the correctness reward R checks. The synthesized pairs (x,τ)\(x,τ)\ are the off-policy data for stage (2), where grafting replaces SFT. Hyperparameters The teacher and the student are both initialized from Qwen3-1.7B in thinking mode, and the maximum sequence length is 1818K tokens. Appendix C Training Grafting and direct SFT (math). Both fine-tune with identical optimization and differ only in the checkpoint updated (the earlier checkpoint for grafting, the post-trained model for direct SFT). The grid spans nine learning rates (5×10−75×10^-7 to 5×10−55×10^-5) and five epochs, snapshotting after each epoch, at global batch 3232 and sequence length 1818K. We optimize with AdamW (β=(0.9,0.95)β=(0.9,0.95), weight decay 0.010.01) under a cosine schedule with warmup ratio 0.10.1 and gradient clipping at 1.01.0. Grafting and direct SFT (knowledge). Full fine-tuning at learning rate 2×10−52×10^-5 with a cosine schedule, warmup ratio 0.10.1, β=(0.9,0.95)β=(0.9,0.95), weight decay 0.010.01, gradient clip 1.01.0, global batch 3232, and sequence length 1818K. The knowledge QA samples receive an empty-think prefix (<think> </think>), while the replay samples receive the standard <think>reasoning_content</think>. On-policy self-distillation (OPSD). OPSD follows grafting and SFT in terms of training hyperparameters. Here are OPSD-specific details: the model samples its own response conditioned on a demonstration and is trained to match a frozen, demonstration-conditioned teacher under the divergence DαD_α, with rollout n=1n=1 and top-k 100100 for the teacher logits. We report α∈0,0.5,1α∈\0,0.5,1\ for OPSD. Appendix D Evaluation Decoding. All math scores are averaged over 1616 rollouts at temperature 1.01.0 and top-p 1.01.0, decoding up to 3232K tokens in thinking mode and 1616K in non-thinking mode. Knowledge injection is decoded greedily. New task. New-task performance is measured on a held-out set the model was never trained on, as defined in §A. Old task. Old-task performance is the mean over the same three out-of-domain competition-math suites, AIME 2024, AIME 2025 and HMMT 2025. The quantity measures generalization when the continual learning task is math and retention when it is knowledge. Appendix E OLMo-3 experiments Data. We mine hard problems for OLMo-3-7B (thinking) the same way as for Qwen3-1.7B (thinking): sampling OLMo-3-7B (thinking) 1616 times per problem at temperature 1.01.0 and keeping only problems it fails on all 1616 attempts. We hold out 100100 problems i.i.d. to measure new-task performance, and train on every trace of the rest. Checkpoints. On the post-training side we use the four checkpoints OLMo-3 releases: the base model, the SFT checkpoint, the DPO checkpoint, and OLMo-3-7B (thinking) itself. On the pretraining side we use stage-3 (see OLMo Team [2026] for details) steps 11,92111,921, which is the released base model, 9,0009,000, 6,0006,000 and 3,0003,000, together with stage-2 step 47,68447,684, the last stage-2 checkpoint. The gaps from the released base model are 2,9212,921, 5,9215,921, 8,9218,921 and 11,92111,921 steps, which we write as 33k, 66k, 99k and 1212k. Training. We sweep over five learning rates (5×10−65×10^-6 to 5×10−55×10^-5), for five epochs at global batch 3232 and maximum length 18,43218,432, in thinking mode. Training stage-2 checkpoints on long outputs. Stage-2 base checkpoints OLMo Team [2026] set the max lengths as 8,1928,192 with no RoPE scaling, so we copy stage-3’s YaRN block onto them to finetune at 18,43218,432 tokens to enable training stage-2 base checkpoints on long reasoning traces. Appendix F Fisher estimation We discuss the various derivation and implementation details of the retention Fisher equation 4. Recall that the Diagonal Fisher approximation leads to scores that weight the donor update by entry, and the resulting projection equation 5 reduces to a 0,1\0,1\ mask corresponding to the entries with the highest scores. In general, let us consider an orthonormal basis U∈ℝdθ×dθU ^d_θ× d_θ. As examples, the Diagonal Fisher approximation corresponds to the standard coordinate basis U=[ei]U=[e_i]. On the other hand, if U is the eigenbasis of Fret=UΣU⊤F_ret=U U , Σ=diag(σi) = ( _i) then we note the corresponding scores and projection: Δ⊤FretΔ=(U⊤Δ)⊤Σ(U⊤Δ)=∑i=1dθσi⟨ui,Δ⟩2. F_ret =(U ) (U )= _i=1^d_θ _i u_i, ^2. Thus, defining the score accordingly si:=σi⟨ui,Δ⟩2s_i:= _i u_i, ^2 and the set Sρ:=ui:si in top-ρ fractionS_ρ:=\u_i:s_i in top-ρ fraction\, a mask in the diagonal case turns into a projection onto the orthogonal complement of SρS_ρ: ℳρ:=ProjSρ⟂(⋅) [rgb]0.1523,0.6836,0.375M_ρ:=Proj _S_ρ(·). However, we note that the projection based on SρS_ρ is exact only when U diagonalizes FretF_ret, as otherwise projecting based on scores sis_i ignores the contribution from off-diagonal elements (Fret)ijΔiΔj(F_ret)_ij _i _j, which can be greatly suboptimal if Δi _i and Δj _j are tightly coupled, e.g. parameters in the same weight matrix. The coordinate basis is often a suboptimal choice for this purpose, and an exact eigendecomposition of the full-dimensional Fisher is infeasible. This is a similar motivation as prior optimization literature that seeks to approximate Natural Gradient Descent for parametric maximum likelihood estimation, which uses the Fisher Information matrix as a preconditioner. A seminal work on this front is the KFAC optimizer [Martens and Grosse, 2015]. The essential observation is that the Fisher Information matrix can be approximated by a block-diagonal matrix, where each block corresponds to a layer’s parameters; then, for matrix-shaped weights W∈ℝdout×dinW ^d_out× d_in this can further be approximated by a Kronecker factor between two symmetric matrices of shape P∈ℝdout×doutP ^d_out× d_out, Q∈ℝdin×dinQ ^d_in× d_in. For example, given a feedforward weight matrix h=Wah=Wa, we have P=zy∼pθ(⋅|z)[∂L(θ,y,z)∂h∂L(θ,y,z)∂h⊤],Q=z[a⊤], P=E_zE_y p_θ(·\,|\,z) [ ∂ L(θ;y,z)∂ h ∂ L(θ;y,z)∂ h ], Q=E_z[a ], (6) where L(θ,y,z)=−logpθ(y|z)L(θ;y,z)=- p_θ(y|z) is the negative log-likelihood of the model’s own generation y given input z. We note that for <2<2-D parameters the original Diagonal Fisher approximation is made and scores computed as described in Section 4. Consider the eigendecompositions P=UΣU⊤P=U U , Q=VDV⊤Q=VDV , and for a given update Δ , consider the (appropriately re-shaped) subset corresponding to the current layer ΔW ^W. Observe by the Kronecker mixed-product property we have: vec(ΔW)⊤(Q⊗P)vec(ΔW) ( ^W) (Q P)vec( ^W) =Tr(PΔWQΔW) = (P ^WQ ^W) =Tr(UΣU⊤ΔWVDV⊤ΔW) = (U U ^WVDV ^W) =Tr(ΣΔ~WD(Δ~W)⊤),Δ~W:=U⊤ΔWV = ( ^WD( ^W) ), ^W:=U ^WV =∑i,j=1dout,dinσidj(Δ~i,jW)2. = _i,j=1^d_out,d_in _id_j( ^W_i,j)^2. Therefore, for this dout⋅dind_out· d_in parameter subset of Δ , the scores become s(i,j)=σidj(ui⊤ΔWvj)2s_(i,j)= _id_j(u_i ^Wv_j)^2. Tallying the scores up across all scalar/vector parameters and tensor parameters, the KFAC-based projection mask thus becomes a projection onto the orthogonal complement of ek∪ui⊗vj\e_k\∪\u_i v_j\, where sk,s(i,j)s_k,s_(i,j) belong to the top-ρ of scores across scalar/vector and matrix parameters. Though a KFAC decomposition does not in general perfectly diagonalize the full Fisher Information, as it does not capture block-to-block dependencies, it is known empirically to be a significantly better decomposition than the diagonal [George et al., 2018]. Implementation. To estimate the retention Fisher matrix, we compute the per-token-normalized from 18581858 rollouts of the Qwen3-1.7B thinking model θpost _post at temperature 1.01.0 prompted with AoPS-style math problems. The ρ score threshold is estimated by taking a ρ-quantile of a subsample of the list of scores and using that as the selection threshold. The Diagonal Fisher projection is implemented efficiently via a mask. The KFAC projection is slightly more arithmetic-intensive to compute due to the layer-wise eigendecompositions and resulting matrix multiplications, but remains efficient due to the layer-wise factorization.