Paper deep dive
No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation
Seonghyeon Kim, Chaeyun Jang, Noah Lee, Boseop Kim, Juho Lee
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.
Tags
Links
- Source: https://arxiv.org/abs/2609.34745v1
- Canonical: https://arxiv.org/abs/2609.34745v1
Trouble viewing inline? Open PDF directly →
Full Text
51,242 characters extracted from source content.
Expand or collapse full text
No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation Seonghyeon Kim 1,* Chaeyun Jang 1,* Noah Lee 2 Boseop Kim 2 Juho Lee 1 1 KAIST AI, 2 Kakao *Equal Contribution Abstract: Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training. 1. Introduction 0.00.51.0 No init SFT λ= 1.0 λ= 2.1 IM-MOPD 5-domain avg. 55 13 34 22 31 0.00.51.0 Tool-Use 3→5 1→3 44 22 41 0.00.51.0 Medical 5→5 2→3 3→4 11 32 0.00.51.0 Finance 52 1→5 21 44 23 Norm. score No initSFT warm-up initTask-vector merge initIM-MOPD Figure 1: Student initialization matters for MOPD, but stronger initial performance does not necessarily lead to better outcomes. On Qwen3-4B in the 5-domain setting, MOPD initialized from the shared reference model struggles especially with the SFT-trained Tool Use and Medical teachers, while successfully recovering the RL-trained Finance teacher. Merge initialization can start below SFT warm-up yet finish higher after MOPD. Here,휆denotes the global merge scale, defined as the sum of the merge coefficients. IM-MOPD further improves performance in a single run, without an additional training stage or a separate coefficient search. Open and filled circles denote initial and final normalized scores, respectively (reference: 0; teacher: 1). Experimental details are provided in Appendix B.1. Building a unified language model requires integrating capabilities acquired through different data and training procedures. Different domains often benefit from different post-training paradigms and recipes, in- cluding supervised fine-tuning (SFT), reinforcement learning (RL), or their combinations, with domain-specific choices of data and optimization (Zhang et al., 2024; Grattafiori et al., 2024; Chu et al., 2025; Blakeman et al., 2025). These experts may be developed across different timelines, motivating a modular workflow of independent specialization and subsequent integration. Multi-teacher on-policy distillation (MOPD) supports Correspondence to: shkim0824, jcy9911, juholee@kaist.ac.kr. Code: https://github.com/shkim0824/immopd. arXiv:2609.34745v1 [cs.LG] 28 Sep 2026 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation this workflow by transferring the capabilities of domain teachers into a single student. Each teacher provides token-level supervision at the prefixes generated by the student. Recent work demonstrates the practical value of this approach for combining independently developed capabilities (Ma et al., 2026; Blakeman et al., 2026). However, it remains unclear whether MOPD can effectively integrate teachers with different post-training histories. Recent work suggests that MOPD can struggle to transfer teacher capabilities when the teacher and student have substantially different SFT histories even though they share the same reference model (Li et al., 2026; Blakeman et al., 2026; Zhu et al., 2026). This is particularly important for MOPD, where teacher supervision is provided at student-generated prefixes. Because MOPD distills on student-generated prefixes, the initialization determines the states on which distillation begins and can therefore affect subsequent learning. One way to mitigate this issue is to apply a light SFT warm-up before MOPD (Blakeman et al., 2025). Such warm-up can bring the student closer to the teachers. However, it still requires designing mixed-domain SFT, including the data mixture and training schedule. This motivates a way to prepare the student for MOPD while preserving the benefits of independent teacher development. We show that merge initialization provides such an alternative. Following the notation of task arithmetic (Il- harco et al., 2023), we represent the weight difference between each teacher and the shared reference model as a task vector and initialize the student by adding a weighted combination of these vectors to the reference model. SFT warm-up incurs additional training cost and requires balancing multiple domain objectives again, weakening the benefit of developing multi-teachers independently. Merge initialization reduces this preparation to selecting and combining teacher task vectors, improving subsequent MOPD without an additional stage of mixed-domain training. Interestingly, initial benchmark performance does not reliably identify a good MOPD initialization. In Figure 1, SFT warm-up starts above merge initialization but finishes below it after MOPD. We further find that effective merging depends on both domain-wise ratios and the global merge scale휆 = ∑︀ 푖 훼 푖 . In some settings,휆 > 1improves post-MOPD recovery, placing effective initializations outside the simplex of con- vex parameter averaging. Selecting a good merge initialization therefore requires considering both domain-wise ratios and the global scale, while its quality cannot be reliably assessed from initial performance alone. Rather than directly searching over continuous merge coefficients, we adapt teacher contributions during training. Starting from a uniform merge initialization, we alternate MOPD with small task-vector corrections for under-recovered domains. This replaces a domain-wise continuous coefficient search with repeated merge- or-not decisions using a shared correction size, allowing both the relative domain weights and total task-vector contribution to evolve with the student. Teacher-continuation analysis (Section 5) shows that teachers perform better with prefixes generated by the merged student than with those from the SFT warm-up student, even when the merged student itself performs similarly or worse. This suggests that the merged student can benefit more from teacher supervision during MOPD. Empirically, our method achieves normalized recovery scores of 86.9 on 4B and 76.0 on 1.7B, compared with 59.0 and 46.5 for SFT warm-up, respectively. Contributions. In summary, our key contributions in this paper are as follows. • Initialization quality beyond initial performance. We show that higher initial performance does not neces- sarily lead to higher post-MOPD performance, and provide additional prefix-level evidence that benchmark scores do not fully characterize a good MOPD initialization (Figure 1, Figure 5). • Merge initialization without additional SFT. We show that merging teacher task vectors improves capability recovery after MOPD over base initialization and can outperform the evaluated SFT warm-up baseline. This initialization requires neither an additional SFT stage nor access to the experts’ training examples (Figure 1, Table 1). •Iterative merging during MOPD. We propose IM-MOPD, which periodically adds fixed task-vector incre- ments for domains whose recovery remains below a threshold. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD baselines using SFT warm-up or uniform merge initialization (Table 1, Figure 4). 2. Preliminaries We consider a same-origin multi-teacher setting with퐾domain teacher policies휋 휑 푖 퐾 푖=1 , parameterized by 휑 푖 퐾 푖=1 , and their corresponding prompt distributions풟 푖 퐾 푖=1 . The student policy휋 휃 , parameterized by휃, and 2 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation 휃 ! =휃 "#$ Medical 휃 "#$ +휏 %#& Tool-call 휃 "#$ +휏 '(() IF 휃 "#$ +휏 *+ 휃 ! 휃 " # MOPD MOPD δτ $%& 훿휏 '(() + 휃 * # 훿휏 +, TokenSupervision Merge 1 + Merge 2 Medical Tool-call IF Medical Tool-call IF Medical Tool-call IF IM-MOPD Medical 휃 "#$ +휏 %#& Tool-call 휃 "#$ +휏 '(() IF 휃 "#$ +휏 *+ 휃 ! 휃 " MOPD MOPD휃 * TokenSupervision MOPD Frozen Teachers Frozen Teachers Task-vector Addition InitialStudent InitialStudent Figure 2: Overview of IM-MOPD. From a general SFT model휃 ref and퐾domain teachers, IM-MOPD starts from a uniform merge of task vectors휏 푖 = 휑 푖 − 휃 ref and alternates MOPD with selective merging: domains below the recovery threshold 훾receive a fixed increment훿 ∑︀ 푖∈ℳ 푡 휏 푖 before the next round of distillation as a direct weight-space correction for an enhanced initialization. For specific details, see Algorithm 1. all domain teachers share the same architecture and originate from a common reference checkpoint휃 ref , after which the teachers are subsequently specialized through different domain-specific post-training procedures. Our goal is to integrate the capabilities of these independently specialized teachers into a single student policy휋 휃 . Multi-Teacher On-Policy Distillation (MOPD). On-policy distillation trains a student on its own generated re- sponses, with a teacher providing supervision at the prefixes visited by the student (Gu et al., 2024; Agarwal et al., 2024). MOPD extends this procedure to multiple domain teachers by routing each prompt to its corresponding teacher (Ma et al., 2026). For a prompt푥∼풟 푖 , the student generates a response푦 = (푦 1 ,...,푦 푇 )∼ 휋 휃 (·| 푥). At token position푡, the student-generated prefix is푠 푡 = (푥,푦 <푡 ). MOPD minimizes the token-averaged reverse KL between the student and the corresponding teacher over these student-generated trajectories: ℒ MOPD (휃) = 1 퐾 퐾 ∑︁ 푖=1 E 푥∼풟 푖 푦∼휋 휃 (·|푥) [︃ 1 푇 푇 ∑︁ 푡=1 퐷 KL (휋 휃 (·| 푠 푡 )‖휋 휑 푖 (·| 푠 푡 )) ]︃ .(2.1) In practice, we optimize this objective using the policy-gradient formulation of on-policy distillation for com- putational efficiency; implementation details are provided in Appendix A.2. Because supervision is conditioned on student-generated prefixes, initialization shapes the prefixes encountered during distillation and thus the subsequent MOPD trajectory. Model Merging. Model merging combines compatible model checkpoints directly in parameter space without additional training (Izmailov et al., 2018; Wortsman et al., 2022). In our setting, all teachers share the reference checkpoint휃 ref , allowing each teacher to be represented by a task vector휏 푖 = 휑 푖 −휃 ref (Ilharco et al., 2023). We parameterize a merged model as휃 merge (훼) = 휃 ref + ∑︀ 퐾 푖=1 훼 푖 휏 푖 , where훼 푖 ≥ 0controls the contribution of teacher 푖. We define the global merge scale as휆 = ∑︀ 푖 훼 푖 . Standard weighted parameter averaging corresponds to휆 = 1, with uniform averaging given by훼 푖 = 1/퐾. We do not otherwise constrain휆to one, allowing configurations with larger total teacher contribution. A merged model can be used as the initial MOPD student,휃 0 = 휃 merge (훼). 3 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation −10 Mean Tool Med IF Law Fin -0.38 -0.10 +0.07 -0.58 +0.02 -1.29 01 +0.45 +0.20 -0.06 +0.85 +0.06 +1.20 ■With merge■No merge Merge+50 MOPD steps (a) 5-domain merge effect. 74767880 MedQA acc. (%) 50 60 70 80 IFEval acc. (%) 0.20.4 0.6 0.8 InitializationAfter MOPD IM-MOPD (b) 2-domain grid search. 0255075100 MOPD step 0 20 40 60 80 Norm. score (%) Non-uniform, λ= 2.1Uniform, λ= 2.1 Base Non-uniform, λ= 1Uniform, λ= 1 (c) Merge ratios and scale. Figure 3: Selecting a fixed merge initialization requires accounting for subsequent learning, ratios, and scale. (a) From the same 5-domain MOPD checkpoint, we continue training with or without a merge intervention. Although merging can reduce performance immediately, the intervened branch achieves higher recovery after 50 additional MOPD updates. (b) 2-domain Medical-IF coefficient grid vs. IM-MOPD. Open/filled circles denote performance before/after MOPD for fixed merge initializations; numbers indicate the Medical coefficient, with the remainder assigned to IF, and the star denotes IM-MOPD. (c) 5-domain MOPD under different merge initialization. Both ratios and scale affect post-MOPD recovery. 3. Method Overview. We study how student initialization influences the integration of domain teachers with hetero- geneous post-training histories under MOPD. All teachers originate from a shared reference model휃 ref but are independently specialized through SFT or RLVR. Details of teacher construction and the experimental setup are provided in Appendix A.1 and Appendix A.2. Following Ma et al. (2026), let푠 푖 (휃)denote the evaluation performance of model휃on domain푖. We define the normalized domain score as ̃푠 푖 (휃) = 푠 푖 (휃)−푠 푖 (휃 ref ) 푠 푖 (휑 푖 )−푠 푖 (휃 ref ) , such that the shared reference model has score0and the corresponding domain expert has score1. We also report the average normalized score ̃푠(휃) = 1 퐾 ∑︀ 퐾 푖=1 ̃푠 푖 (휃). 3.1. Observations Merge initialization provides an effective starting point for MOPD. As shown in Figure 1, MOPD from the shared reference model struggles to recover the capabilities of SFT-trained teachers such as Medical and Tool-Use, while approaching teacher-level performance on Finance. This uneven recovery suggests that the choice of student initialization can affect how effectively different teacher capabilities are transferred. SFT warm-up partially alleviates this issue, but introduces an additional training cost and requires separate choices of data mixture and training schedule. We find that initializing the student by merging teacher task vectors substantially improves subsequent MOPD recovery without any additional training. This suggests that model merging can serve as a simple and low-cost way to prepare students for MOPD. A stronger student is not necessarily a better MOPD initialization. In Figure 1, SFT warm-up yields higher initial performance than the scaled merge initialization (휆 = 2.1) on Tool-Use, yet the ordering reverses after MOPD. IM-MOPD further improves recovery despite not starting from the strongest benchmark performance. A similar pattern appears for merge interventions during training. In Figure 3a, we compare continued MOPD with and without intermediate merge interventions from the same checkpoint. Although a merge intervention can temporarily reduce normalized performance, the intervened branch achieves higher performance after 50 additional MOPD updates. These results indicate that immediate benchmark performance alone is insufficient for selecting states for subsequent MOPD. Selecting good merge coefficients becomes increasingly difficult as the number of domains grows. Even with two domains, the coefficient sweep in Figure 3b yields different post-MOPD trade-offs, and the best initialization by average normalized score is not the best after MOPD. Evaluating a candidate therefore requires considering the learning it enables. With 5-domains, Figure 3c shows that both merge ratios and scale matter: non-uniform weighting improves recovery at the larger scale, while normalizing the coefficients to sum to one weakens the result. Increasing the uniform scale helps but does not match the stronger non-uniform config- 4 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation uration. Good coefficients thus require jointly choosing domain-wise ratios and total contribution, with each candidate assessed through MOPD. As the teacher set grows, this makes a fixed coefficient search increasingly costly, motivating progressive merge-or-not decisions during training. 3.2. Iterative Merging for MOPD (IM-MOPD) Algorithm 1 IM-MOPD Require:휃 ref reference model;휑 푖 퐾 푖=1 teachers;휏 푖 = 휑 푖 − 휃 ref task vectors;푛MOPD iterations;퐻correction interval; 훿 fixed correction scale; 훾 threshold 휃 ← 휃 ref + 1 퐾 ∑︀ 퐾 푖=1 휏 푖 for 푡 = 1,...,푛 do 휃 ← MOPD(휃;휑 푖 퐾 푖=1 ) if 푡 mod 퐻 = 0 and 푡 < 푛 then ℳ←푖| ̃푟 푖 (휃)≤ 훾 휃 ← 휃 + 훿 ∑︀ 푖∈ℳ 휏 푖 end if end for return 휃 Motivated by the observations above, we replace one-shot merge coefficient search with an iterative procedure that in- terleaves MOPD updates with small, fixed-scale additions of selected teacher task vectors (Figure 2). IM-MOPD initializes the student with a uniform merge,휃 0 = 휃 ref + 1 퐾 ∑︀ 퐾 푖=1 휏 푖 , and alternates distillation with selective task-vector correc- tions for domains whose capabilities remain insufficiently recovered. This allows the effective merge weights to adapt throughout training without determining the full coefficient set in advance. At intervals of퐻MOPD updates, we assess domain re- covery on a validation set. Let ̃푟 푖 (휃)denote the normalized score evaluated on validation data for domain푖, with the reference model and corresponding teacher anchoring zero and one. We select domains whose recovery is at or below a threshold훾, formingℳ =푖| ̃푟 푖 (휃)≤ 훾, and update the student as휃 ← 휃 +훿 ∑︀ 푖∈ℳ 휏 푖 . Here,훿 > 0is a fixed correction scale shared across domains and intervention points. Algorithm 1 summarizes the procedure. MOPD resumes from the corrected student, and subsequent merge decisions reflect the recovery achieved through further learning. A domain may receive additional corrections if its recovery remains below the thresh- old. These task-vector additions allow both the relative merge ratios and the total merge scale to evolve during training, without constraining the cumulative coefficients to sum to one. By interleaving these corrections with distillation, IM-MOPD progressively shapes the student from which further teacher-guided learning proceeds. 4. Experiments 4.1. Experimental Setup Reference model and teachers. We first fine-tune Qwen3-4B-Base and Qwen3-1.7B-Base (Yang et al., 2025) on OpenThoughts3 (Guha et al., 2026) to obtain a common reference checkpoint for the student and domain teachers at each model size. We then train 5 domain teachers separately from this reference: Medical, Law, and Tool Use through SFT, and Finance and Instruction Following (IF) through reinforcement learning with verifiable rewards (RLVR). For detailed training and data configurations, please refer to Appendix A.1. Baselines. We compare our method with three representative initialization baselines. MOPD starts from the common reference model (Ma et al., 2026). Uniform Merge starts from the equal-weight average of the five teachers and serves as the static merge initialization baseline, isolating the effect of iterative adaptation from the choice of a more sophisticated one-shot merging rule. SFT Warm-up first trains the student on mixed teacher-generated samples before MOPD. More details on the baselines are provided in Appendix A.2–A.4. Iterative-merging configurations. Both model sizes start from uniform merging and use the same validation- based recovery rule. We set the recovery threshold to훾 = 0.6for 4B and훾 = 0.4for 1.7B, the fixed correction scale to훿 = 0.3for both, and the correction interval to퐻 = 25. Every퐻MOPD updates, we add the corre- sponding teacher task vector for domains whose recovery is at or below훾. Appendix A.4 provides the validation protocol and correction schedules. Evaluations. We evaluate Medical on MedQA (Jin et al., 2021), Law on CaseHOLD (Zheng et al., 2021), Finance on FinQA (Chen et al., 2021), Instruction Following on IFBench (Pyatkin et al., 2025), and Tool Use on휏 2 -Telecom (Barres et al., 2026). We report answer accuracy for MedQA and CaseHOLD, numeric-answer accuracy for FinQA, prompt-level loose accuracy for IFBench, and task success for telecom tasks. For the 5 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation Table 1: MOPD results across 5-domains at Qwen3-4B and Qwen3-1.7B. IM-MOPD achieves the highest average normalized score among the compared student initialization methods at both model sizes, without an additional SFT stage. Domain Teacher reports the performance of the domain specialist for each domain. The best result in each column within each model size is shown in bold. MethodMedQA CaseHOLD FinQA IFBench휏 2 Norm. Qwen3-4B Qwen3-4B-OT369.561.859.124.95.30.0 Domain Teacher80.773.475.059.4 66.7 100.0 MOPD69.164.774.356.97.942.8 SFT Warm-up + MOPD73.968.271.153.4 31.659.0 Uniform Merge + MOPD72.767.674.753.6 11.453.8 IM-MOPD76.670.974.057.869.386.9 Qwen3-1.7B Qwen3-1.7B-OT344.144.642.418.47.90.0 Domain Teacher58.569.662.446.4 28.1 100.0 MOPD49.052.259.136.80.034.9 SFT Warm-up + MOPD52.756.654.832.6 10.546.5 Uniform Merge + MOPD51.259.958.634.64.446.3 IM-MOPD53.759.455.942.928.176.0 main results, we report pass@1 for Tool Use and avg@3 for the other benchmarks, together with the average normalized score across the five domains. Evaluation protocols are provided in Appendix A.5. 4.2. Main Results Merge initialization improves recovery, but leaves uneven domain transfer. Table 1 shows that starting MOPD from a uniform merge substantially improves aggregate recovery over starting from the reference model, without requiring an additional SFT stage. SFT warm-up provides a further improvement in aggregate recovery, but requires an additional training stage. Despite these gains, both approaches leave some domains substantially under-recovered, most notably Tool Use. Thus, a better initialization improves MOPD overall, but does not by itself ensure balanced capability integration across domains. Iterative merging improves aggregate capability integration. For Qwen3-4B, iterative merging increases the average normalized recovery from 53.8% with uniform initialization to 86.9%. For Qwen3-1.7B, IM-MOPD reaches 76.0%, compared with 46.5% for SFT warm-up, and recovers Tool Use to the teacher level. Across both model sizes, IM-MOPD achieves higher scores than SFT warm-up on all five benchmarks. Relative to uniform initialization, the largest improvements are observed in Medical and Tool Use, while performance on Finance and Instruction Following remains broadly comparable. These results show that iterative corrections can improve recovery in domains that remain under-recovered after initialization without substantially degrading domains already transferred effectively by MOPD. 4.3. Ablations Iterative merging versus merging only at initialization. We compare IM-MOPD with an initialization-only variant that merges teacher task vectors using the same cumulative coefficients, followed by the same number of MOPD updates (Table 2). Iterative merging reaches 86.9% normalized recovery, versus 79.0% for the initialization-only variant, with the largest gain in Tool Use. With the cumulative coefficients held fixed, distributing task-vector additions across distillation is more effective than applying them all at the start. Merging and MOPD reinforce each other. After MOPD from the reference model with no additional ini- tialization, we apply the final cumulative merge coefficients obtained by IM-MOPD to the trained student. As shown in Table 2, this late merging achieves only 23.3% normalized recovery, below Base MOPD and far below 6 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation 050100 0.0 0.5 1.0 Norm. score Medical 050100 Tool-Use 050100 Finance 050100 Law 050100 IF MOPD step Base-initSFT warm-upUniform MergeIM-MOPDBefore mergeAfter merge Figure 4: Iterative merging improves recovery in domains left behind by fixed initialization. On Qwen3-4B, IM-MOPD starts from Uniform Merge and progressively improves under-recovered domains during MOPD. The largest gains are observed in Medical, Tool Use, and Law, while performance in already well-recovered domains remains broadly stable. Table 2: Merge-timing ablation on Qwen3-4B. We take the final cumulative merge coefficients produced by IM-MOPD and apply them either entirely at initialization or entirely after MOPD. Both controls underperform the original IM-MOPD sched- ule, showing that when the same final merge coefficients are applied matters. The best result in each column is shown in bold. Merge timingMedQACaseHOLDFinQAIFBench휏 2 Norm. No merge (Base MOPD)69.164.774.356.97.942.8 All at initialization76.370.874.052.457.079.0 All after MOPD68.171.059.638.210.523.3 Iterative (IM-MOPD)76.670.974.057.869.386.9 IM-MOPD. Together with the initialization-only control, this result shows that the gains of IM-MOPD cannot be explained simply by the final task-vector displacement induced by merging. The same merge coefficients are substantially more effective when interleaved with MOPD, indicating that subsequent teacher-guided updates after each correction are important for the final recovery. Table 3: IM-MOPD sensitivity to훾and훿. Average normalized recovery (%) across 5 domains. Model훾훿 = .2훿 = .3훿 = .4 Qwen3-4B0.680.386.986.1 Model훾훿 = .1훿 = .2훿 = .3 Qwen3-1.7B0.462.257.276.0 0.657.765.069.8 Hyperparameter sensitivity. We evaluate sensi- tivity to the recovery threshold훾 ∈0.4, 0.6and correction size훿 ∈ 0.1, 0.2, 0.3, 0.4in Table 3. Qwen3-4B maintains over 80% normalized recov- ery across tested correction sizes, while Qwen3- 1.7B performs best with훾 = 0.4and훿 = 0.3. Ef- fective settings can thus be identified from a small candidate set. More detailed results are provided in Appendix B.2. 5. Analysis Teacher continuation from student-generated prefixes. Prior work evaluates the reliability of teacher supervision in OPD by continuing from student-generated prefixes (Li et al., 2026). We adopt this diagnostic to assess teacher–student compatibility for merged and SFT warm-up students. Each student generates an initial prefix, after which the corresponding domain teacher completes the response. We vary the fraction generated by the student and evaluate task performance (Figure 5). Higher continuation performance indicates greater compatibility between the student-generated prefix and the teacher’s subsequent generation. Protocol details are provided in Appendix B.3. Across Medical and Tool Use, teacher continuation performance drops even with short prefixes from the SFT warm-up student, whereas prefixes from the merged student better preserve it over the early portion of 7 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation 010305070100 Student prefix (%) 70 72 74 76 78 80 82 Acc. (%) Teacher only Merge initialization SFT warm-up initialization (a) Medical (MedQA). 010305070100 Student prefix (%) 20 30 40 50 60 70 80 Task success (%) Teacher only Iterative merging + MOPD SFT warm-up + MOPD (b) Tool Use (휏 2 -Telecom). Figure 5: Teacher continuation is stronger from merge-based student prefixes on Qwen3-4B. In Medical, teacher continuation is consistently stronger from merge-initialized prefixes than from SFT-warm-up prefixes despite similar student-only performance. In Tool Use, IM-MOPD yields higher continuation success from short prefixes despite lower student-only performance at the evaluated checkpoint. The 0% and 100% endpoints denote teacher-only and student-only performance, respectively. the trajectory (Figure 5). These results are consistent with prior findings that teacher–student compatibility matters for effective OPD (Li et al., 2026). This advantage persists even when the merged student’s standalone performance is similar to or lower than that of the SFT warm-up student. Together, these results suggest that merging helps the student generate prefixes from which teachers can more successfully continue, allowing the student to benefit more from teacher supervision during MOPD. 6. Related Work On-Policy Distillation for Large Language Models (LLMs). Knowledge distillation (KD) traditionally trans- fers teacher knowledge by matching predictive distributions on a fixed training distribution (Hinton et al., 2015). Kim and Rush, 2016 extended the paradigm to apply sequence-level KD from teacher-generated sequences. However, such offline supervision exposes the student primarily to teacher- or data-induced prefixes, creating a mismatch between train and inference setups. On-policy distillation (OPD) addresses this mismatch by instead querying the teacher on student-generated trajectories (Agarwal et al., 2024), and is commonly paired with reverse-KL objectives that naturally optimize the student distribution over its own visited states (Gu et al., 2024). This formulation has become increasingly relevant for reasoning LLMs, where the context has scaled with the reasoning length (Xu et al., 2026; Blakeman et al., 2026; Ma et al., 2026; Team et al., 2026). Yet, the benefit of on-policy supervision depends on the teacher remaining informative on states visited by the student: as teacher–student distributions diverge, teacher feedback can become less effective or unreliable (Xu et al., 2025; Li et al., 2026; Blakeman et al., 2026). Recent methods therefore selectively constrain or adapt distillation according to teacher–student alignment (Xing et al., 2026), highlighting distributional compatibility as a central challenge in effective OPD. Consolidating Domain Specialists. As post-training increasingly targets heterogeneous capabilities often through reinforcement learning (RL), consolidating these divergent capabilities into a single model has become a central challenge. Existing approaches differ in how capabilities are acquired and integrated. Mixed RL jointly optimizes a shared policy over a mixture of domains, enabling direct capability integration but coupling domains with potentially heterogeneous data, rewards, rollout lengths, and optimization dynamics (Blakeman et al., 2025). Cascade RL instead applies domain-specific RL stages sequentially, allowing each stage to use domain-specific environments and optimization configurations (Wang et al., 2025). While carefully designed 8 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation cascades can largely preserve previously acquired capabilities, capability drift can accumulate across increas- ingly specialized stages, motivating intermediate consolidation or stabilization (Yang et al., 2026). Alternatively, domain specialists can be trained independently and subsequently consolidated through parameter merging (Team et al., 2025) or multi-teacher on-policy distillation (MOPD) (Ma et al., 2026). Our work builds on this specialist-consolidation paradigm and investigates whether iterative merging can progressively integrate heterogeneous capabilities while maintaining effective teacher–student alignment. 7. Conclusion We study MOPD when domain teachers share a common reference model but undergo different post-training procedures. We find that initial benchmark performance is an unreliable criterion for selecting MOPD ini- tializations, and that effective merge initialization depends on both relative teacher weights and the overall merge scale. Motivated by these observations, we introduce IM-MOPD, which starts from a uniform merge and progressively adds task-vector corrections for domains that remain under-recovered during distillation. Across five domains at both 4B and 1.7B scales, IM-MOPD achieves higher aggregate capability recovery than uniform merging, and SFT warm-up, without requiring an additional SFT stage. Merge-timing ablations further show that applying the same final cumulative merge coefficients entirely before or after MOPD is less effective than interleaving the corresponding corrections with distillation. Together, these results suggest that teacher contributions are more effectively determined progressively during MOPD than fixed entirely before training. Acknowledgements This work was conducted in collaboration with and supported by Kakao Corp. Reproducibility Statement Algorithm 1 specifies IM-MOPD. Appendix A.1 documents teacher training and data sources, while Appen- dices A.2–A.3 describe MOPD and SFT warm-up settings. Validation data and merge schedules are given in Appendix A.4, and evaluation protocols in Appendix A.5. Appendices B.1–B.4 provide additional ablation results, the student-prefix diagnostic procedure, and two-domain results. 9 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, pages 21246–21263, 2024. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik R Narasimhan.휏 2 -Bench: Evaluating Conver- sational Agents in a Dual-Control Environment. In International Conference on Machine Learning, 2026. Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khattar, Adi Renduchintala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, et al. Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning. arXiv preprint arXiv:2512.20848, 2025. Aaron Blakeman, Aaron Thomas, Aastha Jhunjhunwala, Abhibha Gupta, Abhinav Khattar, Adam Rajfer, Adi Renduchintala, Adil Asif, Aditya Vavre, Adriana Flores Miranda, et al. Nemotron 3 ultra: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning. arXiv preprint arXiv:2606.15007, 2026. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. FinQA: A Dataset of Numerical Reasoning over Financial Data. In Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, 2021. Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-training. In International Conference on Machine Learning, 2025. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In International Conference on Learning Representations, volume 2024, pages 32694–32717, 2024. Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. In International Conference on Learning Representations, volume 2026, pages 108059–108130, 2026. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams. Applied Sciences, 11(14):6421, 2021. Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, 2016. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016, 2026. Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, et al. Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training. arXiv preprint arXiv:2606.30406, 2026. 10 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing Verifiable Instruction Following. In Conference on Neural Information Processing Systems, 2025. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y Charles, et al. Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. Meituan LongCat Team, Anchun Gui, Bei Li, Bingyang Tao, Bole Zhou, Borun Chen, Chao Zhang, Chengcheng Han, Chenhui Yang, Chi Zhang, et al. Introducing longcat-flash-thinking: A technical report. arXiv preprint arXiv:2509.18883, 2025. Boxin Wang, Chankyu Lee, Nayeon Lee, Sheng-Chieh Lin, Wenliang Dai, Yang Chen, Yangyi Chen, Zhuolin Yang, Zihan Liu, Mohammad Shoeybi, et al. Nemotron-cascade: Scaling cascaded reinforcement learning for general-purpose reasoning models. arXiv preprint arXiv:2512.13607, 2025. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965–23998, 2022. Xingrun Xing, Haoqing Wang, Boyan Gao, Ziheng Li, and Yehui Tang. Trust Region On-Policy Distillation. arXiv preprint arXiv:2606.01249, 2026. Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. Wenda Xu, Rujun Han, Zifeng Wang, Long Le, Dhruv Madeka, Lei Li, William Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling. In International Conference on Learning Representations, volume 2025, pages 64616–64646, 2025. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, et al. Nemotron-cascade 2: Post-training llms with cascade rl and multi-domain on-policy distillation. arXiv preprint arXiv:2603.19220, 2026. Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The effect of data, model and finetuning method. In International Conference on Learning Representations, pages 44694–44713, 2024. Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In International Conference on Artificial Intelligence and Law, pages 159–168, 2021. Siqi Zhu, Xuyan Ye, Hongyu Lu, Weiye Shi, and Ge Liu. The many faces of on-policy distillation: Pitfalls, mechanisms, and fixes. arXiv preprint arXiv:2605.11182, 2026. 11 Contents 1 Introduction1 2 Preliminaries2 3 Method4 3.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.2 Iterative Merging for MOPD (IM-MOPD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4 Experiments5 4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4.2 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.3 Ablations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 5 Analysis7 6 Related Work8 7 Conclusion9 A Experimental Details13 A.1 Teacher Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 A.2 MOPD Training Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 A.3 SFT Warm-up Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 A.4 Main Experiment Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 A.5 Evaluation Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 B Additional Experiments16 B.1 Initialization Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B.2 Ablation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B.3 Student-prefix Teacher-continuation Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B.4 Two-domain Initialization and Merge Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 12 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation A. Experimental Details Training was performed on NVIDIA A100 80GB GPUs. A.1. Teacher Training We first fine-tune Qwen3-4B-Base 1 and Qwen3-1.7B-Base 2 on OpenThoughts3 3 to obtain a common reference at each model size. Domain teachers are initialized from the corresponding reference. Table 4 and Table 5 summarize their training configurations. For GRPO, batch size is given as prompts× completions. Table 4: Training configurations of the 4B domain teachers. Teacher Training Batch Length Peak LR Tokens (M) Medical SFT16 8,19210 −5 426 LawSFT128 20,48010 −5 269 Tool Use SFT64 32,768 5· 10 −6 796 Finance GRPO 32× 8 16,384 3· 10 −6 315 IFGRPO 64× 8 32,768 3· 10 −6 483 Table 5: Training configurations of the 1.7B domain teachers. Teacher Training Batch Length Peak LR Tokens (M) Medical SFT16 8,19210 −5 426 LawSFT64 20,48010 −5 269 Tool Use SFT64 32,768 5· 10 −6 796 Finance GRPO 32× 8 16,384 3· 10 −6 275 IFGRPO 64× 8 32,768 3· 10 −6 424 SFT data. Medical uses 40,853 correct, deduplicated samples generated by Qwen3.6-35B-A3B from MedQA training prompts. Law uses 56,123 correct, deduplicated samples generated by the same model from CaseHOLD and bar-exam prompts. Tool Use uses 32,758 retail, airline, and telecom samples frominclusionAI/AReaL-tau2-data 4 , after removing overlength samples and telecom task-ID overlap. Tool schemas are preserved, and supervision covers the final assistant turn with earlier thinking removed. SFT optimization. SFT uses the Trainer implementation in Hugging Face Transformers 5 . All three domains use packing, zero weight decay, gradient-norm clipping at 1, and cosine decay with 5% warm-up. Tool Use additionally uses a 10% learning-rate floor. RL data. Finance uses 7,565 numeric-answer prompts from the training splits of FinQA and TAT-QA. 6 IF uses 11,728 verifiable instruction-following prompts from therlvr1blend of Nemotron-RL-Ultra-Training-Blends. 7 GRPO. Finance and IF use NeMo-RL 8 GRPO with numeric-answer and instruction-constraint rewards, re- spectively. Both use AdamW with zero weight decay, ten learning-rate warm-up updates, and a constant rate thereafter. Training uses clipped policy updates and overlength penalties, without a reference KL penalty. Sampling uses temperature 1 and top-푝 1. 1 https://huggingface.co/Qwen/Qwen3-4B-Base 2 https://huggingface.co/Qwen/Qwen3-1.7B-Base 3 https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M 4 https://huggingface.co/datasets/inclusionAI/AReaL-tau2-data 5 https://github.com/huggingface/transformers 6 https://github.com/NExTplusplus/TAT-QA 7 https://huggingface.co/datasets/nvidia/ Nemotron-RL-Ultra-Training-Blends 8 https://github.com/NVIDIA-NeMo/RL 13 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation A.2. MOPD Training Configuration MOPD data. Medical uses 8,384 four- and five-option MedQA training prompts. Law uses 31,563 CaseHOLD and bar-exam prompts. 9 Finance and IF reuse the RL prompt pools described in Appendix A.1. Tool Use uses 31,718 dialogue contexts from the retail, airline, and telecom portions of AReaL-tau2-data, with each context serving as a prompt for the next assistant turn. MOPD optimization. All student methods use 100 MOPD updates, a batch of 128 prompts, uniform domain sampling, and one completion per prompt. Policy-gradient distillation clips advantage magnitude at 5 and masks truncated completions, without task-reward or reference-KL terms. The learning rate is10 −6 , and completions are capped at 16,384 tokens. Sampling uses temperature 1 and top-푝 1. A.3. SFT Warm-up Baseline The warm-up baseline trains on mixed samples generated by the five teachers at the corresponding model size until the student reaches approximately 30% normalized recovery. MOPD then starts from this warm-up checkpoint (Table 6). Table 6: SFT warm-up configurations. Student Training Batch Length Peak LR Tokens (M) 4BSFT32 32,76810 −5 167 1.7BSFT32 32,76810 −5 171 Warm-up uses packed sequences and cosine learning-rate decay with 5% warm-up. A.4. Main Experiment Settings Main results use훾 = 0.6for 4B and훾 = 0.4for 1.7B, with훿 = 0.3for both. Corrections are applied after updates 25, 50, and 75 according to the recovery rule in Algorithm 1. Table 7 lists the selected domains for each configuration. Validation data. For Tool Use, we sample 128 tasks from the full dataset. For each other domain, we sample 256 prompts from a validation set held out from its training pool. Table 7: Realized merge schedules. T/M/L/F/I denote Tool Use, Medical, Law, Finance, and IF. Size 훾 훿 Update 25 Update 50 Update 75 4B 0.6 0.2T,M,L,IT,MT 4B 0.6 0.3T,M,L,ITT 4B 0.6 0.4T,M,L,IT∅ 1.7B 0.4 0.1T,M,L,IT,MT 1.7B 0.4 0.2T,M,F,ITT 1.7B 0.4 0.3T,M,ITT,I 1.7B 0.6 0.1 T,M,L,F,IT,M,IT,M,I 1.7B 0.6 0.2 T,M,L,F,IT,M,IT 1.7B 0.6 0.3 T,M,L,F,IT,M,IT,F,I A.5. Evaluation Protocol Evaluation uses temperature 1, top-푝1, and disabled top-푘. Single-turn tasks use zero presence penalty: MedQA/CaseHOLD/FinQA have 1,273/5,314/1,147 examples and a 26,624-token completion cap; IFBench has 300 prompts and a 32,768-token cap. We use MedQA, CaseHOLD, FinQA, IFBench, and IFEval for the non-tool 9 Bar-exam data: https://huggingface.co/datasets/reglab/barexam_qa. 14 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation domains. 10 Telecom evaluation uses the official v1.0.1 harness 11 on 114 base-split tasks, with a 32,768-token context and a 200-step limit. The user simulator isgpt-4.1-2025-04-14at temperature 0; empty responses and execution or context-limit errors count as failures. The split includes 20 task IDs flagged for upstream training overlap, so teacher-data filtering does not exclude upstream exposure. 10 https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options https://huggingface.co/datasets/casehold/casehold https://github.com/czyssrs/FinQA https://huggingface.co/datasets/allenai/IFBench_test https://huggingface.co/datasets/google/IFEval 11 https://github.com/sierra-research/tau2-bench 15 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation B. Additional Experiments B.1. Initialization Comparison Table 8 gives the five-domain results underlying Figure 1. The휆 = 1.0merge assigns coefficient0.2to each teacher; the휆 = 2.1merge uses(0.5, 0.3, 0.2, 0.3, 0.8)in the order Medical, Law, Finance, IF, and Tool Use. SFT warm-up and IM-MOPD follow Appendices A.3 and A.4, respectively. All post-MOPD scores use 100 updates. Table 8: Five-domain performance for the initializations in Figure 1. Each entry shows before / after MOPD. IF is evaluated with IFEval prompt-level strict accuracy. InitializationMedQACaseHOLDFinQAIFEval휏 2 No init69.5 / 69.161.8 / 64.759.1 / 74.337.5 / 74.95.3 / 7.9 SFT warm-up72.6 / 73.965.4 / 68.269.1 / 71.156.6 / 71.030.7 / 31.6 Merge (휆 = 1.0)71.7 / 72.766.5 / 67.669.0 / 74.752.5 / 74.70.9 / 11.4 Merge (휆 = 2.1)73.7 / 77.567.9 / 68.966.0 / 73.552.7 / 66.213.2 / 50.9 IM-MOPD71.7 / 76.666.5 / 70.969.0 / 74.052.5 / 75.00.9 / 69.3 B.2. Ablation Details Table 9–10 report the complete benchmark results underlying the main-text ablations. Both scales are analyzed using the validation recovery rule described in Appendix A.4. Table 9: 4B correction-size sweep with 훾 = 0.6. Correction MedQA CaseHOLD FinQA IFBench 휏 2 Norm. 훿 = 0.279.170.3 73.958.7 36.8 80.3 훿 = 0.376.670.9 74.057.8 69.3 86.9 훿 = 0.478.971.8 73.054.2 59.6 86.1 Table 10: 1.7B recovery-threshold and correction-size sweep. (훾,훿)MedQA CaseHOLD FinQA IFBench 휏 2 Norm. (0.4, 0.1)53.762.9 59.034.8 14.0 62.2 (0.4, 0.2)52.259.5 59.334.0 14.0 57.2 (0.4, 0.3)53.759.4 55.942.9 28.1 76.0 (0.6, 0.1)54.662.1 58.635.2 8.8 57.7 (0.6, 0.2)55.063.9 58.238.4 12.3 65.0 (0.6, 0.3)56.964.7 50.541.8 19.3 69.8 B.3. Student-prefix Teacher-continuation Diagnostics We vary the student-generated fraction훼 ∈ 0.1, 0.3, 0.5, 0.7; the 100% endpoints are evaluated using the student alone. Medical. For each recorded MedQA response of length퐿, the teacher continues after the first⌊훼퐿⌋student- generated tokens. The prefix length is therefore determined separately for each sample. Tool Use. In휏 2 -Telecom, we instead use a per-turn budget푁 = round(훼 ̄ 퐿), where ̄ 퐿 is the student’s mean tokens per assistant turn in a standalone evaluation. The teacher continues when the student reaches this budget. We use the mean because changing one turn in a multi-turn simulation changes subsequent interactions, so later turns cannot be matched to their original sample-specific lengths. Because the budget uses the mean turn length, some shorter turns may be generated entirely by the student. 16 No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation B.4. Two-domain Initialization and Merge Timing Table 11 summarizes the MedQA and IFEval results. IFEval reports prompt-level strict accuracy over 541 prompts. Norm. averages normalized recovery over these two benchmarks. Table 11: Two-domain performance before and after MOPD. Simplex coefficients are ordered as Medical and IF. The last row uses uniform initialization and mid-training merging. InitializationAfter MOPD Initialization / method MedQA IFEval Norm. MedQA IFEval Norm. (0.2, 0.8)73.4 79.7 76.776.0 78.4 86.4 (0.4, 0.6)75.6 79.7 86.776.7 79.5 91.3 (0.6, 0.4)77.4 64.5 75.479.4 79.1 102.5 (0.8, 0.2)80.0 49.9 68.980.3 78.4 105.4 IM-MOPD78.3 68.8 84.880.0 79.3 105.2 17