Paper deep dive
D$^3$-MOPD: Adaptive Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
Zechen Sun, Zhiwei Zhang, Fei Zhao, Juntao Li, Mu Chuan, Huayu Deng, Guojian Zhan, Wenliang Chen, Yao Hu, Min Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/27/2026, 4:24:50 AM
Summary
The paper introduces D3-MOPD, a dynamic domain scheduling method for Multi-Teacher On-Policy Distillation (MOPD). It addresses the inefficiency of fixed domain mixtures by using an off-process watcher to monitor per-domain reverse-KL divergence signals. This signal is used to dynamically adjust domain sampling ratios based on remaining performance gap and descent velocity, improving training efficiency and final model performance compared to vanilla MOPD.
Entities (12)
Relation Signals (11)
D3-MOPD → appliedto → Qwen3.6-35B-A3B
confidence 95% · On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D3-MOPD closes 97% of the average student-to-teacher performance gap
D3-MOPD → extends → MOPD
confidence 95% · we propose D3-MOPD (Dynamic Domain ScheDuling for MOPD)... To address this, we propose D3-MOPD
D3-MOPD → outperforms → MOPD
confidence 95% · D3-MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD
D3-MOPD → uses → Reverse KL Divergence
confidence 95% · repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online
D3-MOPD → evaluatedon → AIME 2025
confidence 90% · We evaluate on seven benchmarks... AIME 2025... D3-MOPD... surpasses the specialist teachers on three of seven benchmarks
D3-MOPD → evaluatedon → LiveCodeBench
confidence 90% · We evaluate on seven benchmarks... LiveCodeBench Code Generation
D3-MOPD → evaluatedon → IFBench
confidence 90% · We evaluate on seven benchmarks... IFBench
Qwen3.6-35B-A3B → trainedondomains → Math
confidence 90% · student distilled from four domain-expert teachers spanning Math, Code, Instruction Following, and Tool-use
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollouts. Existing approaches typically fix the per-domain data mixture before training, overlooking the fact that different domains converge at substantially different rates: some plateau early while others continue to improve throughout the training budget. A fixed mixture therefore wastes compute on fast-converging domains and undertrains slower-converging ones. To address this, we propose D$^3$-MOPD (Dynamic Domain ScheDuling for MOPD), a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online. Running asynchronously outside the training process, an off-process watcher periodically tracks each domain's KL trajectory, estimates remaining headroom and current improvement rate, and accordingly adjusts the domain sampling ratios without altering the core training loop. Our D$^3$-MOPD scales naturally to arbitrary numbers of domains, and the expected benefit grows as more domains introduce more diverse convergence patterns for the scheduler to exploit. On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D$^3$-MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD, reaches the same peak performance with an approximately 3$\times$ reduction in rollout steps, and surpasses the specialist teachers on three of seven benchmarks.
Tags
Links
- Source: https://arxiv.org/abs/2608.24987v1
- Canonical: https://arxiv.org/abs/2608.24987v1
Trouble viewing inline? Open PDF directly →
Full Text
60,673 characters extracted from source content.
Expand or collapse full text
D 3 -MOPD: Adaptive Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation AllSpark Team Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student’s own rollouts. Existing approaches typically fix the per-domain data mixture before training, overlooking the fact that different domains converge at substantially different rates: some plateau early while others con- tinue to improve throughout the training budget. A fixed mixture therefore wastes compute on fast-converging domains and undertrains slower-converging ones. To address this, we propose D 3 -MOPD (DynamicDomain ScheDuling for MOPD), a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online. Running asynchronously outside the training process, an off-process watcher periodically tracks each domain’s KL trajectory, estimates remaining headroom and current improvement rate, and accordingly adjusts the domain sampling ratios without altering the core training loop. Our D 3 -MOPD scales naturally to arbitrary numbers of domains, and the expected benefit grows as more domains introduce more diverse convergence patterns for the scheduler to exploit. On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D 3 -MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD, reaches the same peak performance with an approximately 3×reduction in rollout steps, and surpasses the specialist teachers on three of seven benchmarks. Date:August 22, 2026 AllSpark 1 Introduction D 3 MOPD (ours) Vanilla MOPD Math: mediumIF: slowestCode: fastest Initial course schedule (unchanged) Early Math Code IF 33% 33% 33% Warm-up (train for a while) Math Code IF student-teacher gap Training Later ... ... ... ⚠ Fast domains: Over-trained ⚠ Slow domains: Insufficient trained 63% Final Proximity to Teachers Gap & Descent Velocity (via per-domain r-KL) r-KL Math r-KL Code r-KL IF 33% 33% 33% 30% 10% 60% ✅ Allocate more to hard-but-learnable domains ✅ Reduce for easy-and-saturated domains 97% Final Proximity to Teachers Figure 1:Comparison of vanilla MOPD and D 3 -MOPD withK=3 domains. Multi-teacher on-policy distillation (MOPD) has recently emerged as an effective framework for combining the capa- bilities of several domain-expert teacher models into a single student [ 15,4]. At each rollout step, the student generates on-policy trajectories from aK-domain prompt set [28,31,25], and each domain teacher prefills on the corresponding trajec- tories to produce its per-token distribution. The student then minimizes the per-token reverse-KL divergence [1,7] between its own distribution and the teacher’s along the trajectory. The result is a unified model that inherits the strengths of all teach- ers, without requiring all of them to be served at inference. Current MOPD implementations typically rely on a static do- main mixturep k , fixed before training to match the prompt pool sizes. However, the student rarely learns each domain at the same rate [3,16]: some domains plateau early as the student quickly closes the performance gap to the teacher, whereas others remain far from convergence throughout the entire training budget. Consequently, a fixed mixture wastes significant compute on converged domains while limiting the training budget for actively learning domains (Figure 1). Since the per-domain reverse-KL is already com- puted by the loss objective at every rollout step, a natural and promising solution is to repurpose it as a readily available signal for identifying and correcting this training imbalance. In this paper, we proposeDynamicDomain ScheDuling for MOPD(D 3 -MOPD), a lightweight scheduler that repurposes the per-domain reverse-KL signal already produced by the training loop to update the mixture p k online. An asynchronous watcher periodically converts each domain’s KL history into acomposite sig- arXiv:2608.24987v1 [cs.LG] 25 Aug 2026 nalthat combines its remaining gap to the teacher with its recent descent velocity, and maps this signal to a valid mixture via a temperature-controlled softmax with a per-domain floor. It then passes the updated mix- ture to the stratified data source, ensuring the core training loop remains strictly unmodified. By confining modifications to the data loader and running the watcher detached, D 3 -MOPD remains compatible with advances in loss functions [27,32,10] and teacher training [24,6], without additional training overhead. We validate D 3 -MOPD on a Qwen3.6-35B-A3B [21] student distilled from four domain-expert teachers span- ning Math, Code, Instruction Following, and Tool-use. Experimental results highlight that D 3 -MOPD out- performs vanilla MOPD, closing 97% of the average student-to-teacher performance gap (compared to 63% for the baseline) while achieving a higher peak accuracy across all tasks, surpassing the specialist teachers on three of seven benchmarks. It also improves training efficiency by reaching the baseline’s optimal per- formance in just 47 rollout steps, approximately 3×faster than vanilla MOPD (143 steps). Furthermore, we argue that the expected benefits of D 3 -MOPD will naturally scale withK, as more domains introduce more diverse convergence patterns for the scheduler to exploit. Overall, this paper first identifies a systematic mismatch between MOPD’s static mixture and the student’s asynchronous per-domain convergence, and resolves it with D 3 -MOPD, a zero-overhead scheduler that turns the reverse-KL signal already computed by the training loop into a dynamic data mixture to improve both peak quality and training efficiency. 2 Motivation: Limitations of Fixed Domain Mixtures In this section, we begin by formalizing vanilla MOPD and its fixed mixture (§2.1), then analyze per-domain learning dynamics in isolation (§2.2), and finally demonstrate how their disparate convergence rates waste compute under a static mixture (§2.3). 2.1 Vanilla MOPD and Fixed Mixtures On-policy distillation (OPD) trains a student policyπ θ on its own generations, supervising them with a teacherπ T under a reverse Kullback-Leibler (KL) objective. Multi-teacher OPD (MOPD) extends this paradigm toKdomains, where each domain is associated with a prompt distributionD k and an expert teacherπ T k . MOPD minimizes a weighted sum of per-domain losses: J MOPD (θ)= K ∑ k=1 p k E x∼D k ,y∼π θ (·|x) h D KL π θ (·|x,y)∥π T k (·|x,y) i ,(1) where the KL divergence is summed over the generated tokens ofy, and the mixture weightp k satisfies p k ≥0 with ∑ k p k =1. In vanilla MOPD, the per-domain datasetsD k are pooled and shuffled, resulting in an expected per-batch share for domainkofp k =N k / ∑ i N i , whereN k =|D k |. Crucially, this expected share remains strictly defined prior to training and is held constant throughout the run. Pool-and-shuffle sampling fixes only this expectation, and each batch’s realized share fluctuates aroundp k . 2.2 Domain-Specific Learning Dynamics To understand the inefficiency of a staticp k , we analyze each domain. Specifically, we train the student separately on Math, Code, and Instruction Following (IF) using single-domain OPD, with identical initial- izations and the corresponding expert teacher for each domain. Figure2reports the per-step reverse-KL on a held-out evaluation set. Math drops sharply during the first quarter of the training budget before plateauing near a floor. Code declines steadily at a slower rate, showing no signs of convergence within the budget. Meanwhile, IF continues to reduce its loss throughout, yet its absolute reverse-KL remains one to two orders of magnitude above the other domains at every step. In conclusion, these domains reach their plateau phases at different stages, with their absolute KL signals varying by orders of magnitude. 2.3 Computational Inefficiency of Fixed Mixtures When these same three domains are trained jointly under vanilla MOPD with a uniform mixture (p math = p code =p if =1/3), the varying convergence rates documented in § 2.2translate into a substantial waste of training compute. As illustrated in Figure3, the three domains enter their low-KL phases at markedly different points: Code by roughly step 48, Math by step 96, and IF only around step 144. Yet the fixed mixture allocates one-third of every batch to each domain for the remainder of training. Consequently, 2 050 Rollout step 0.000 0.005 0.010 Reverse-KL Math raw EMA (= 0.1) 050 Rollout step 0.00 0.01 0.02 0.03 Code raw EMA (= 0.1) 050 Rollout step 0.2 0.4 0.6 0.8 Instruction Following raw EMA (= 0.1) 050 Rollout step 0.0 0.2 0.4 0.6 0.8 All three (shared axis) Math Code Instruction Following Figure 2:Per-domain OPD reverse-KL on Math, Code, and IF. The three domains converge at widely differ- ent rates, with the absolute KL of IF remaining one to two orders of magnitude above the others throughout the training process. 0100200 Rollout step 0.001 0.002 0.003 0.004 Reverse-KL plateau step 96 Math raw EMA (= 0.1) 0100200 Rollout step 0.004 0.008 0.012 plateau step 48 Code raw EMA (= 0.1) 0100200 Rollout step 0.08 0.16 0.24 plateau step 144 Instruction Following raw EMA (= 0.1) 0100200 Rollout step 0.08 0.16 0.24 Average raw EMA (= 0.1) Figure 3:Vanilla MOPD with a fixed uniform mixture (p k =1/3). The three domains reach their low- KL phases at markedly different points, yet the static mixture allocates one-third of every batch to each throughout the remaining training (shaded regions). valuable training compute is wasted on domains long past the point where additional samples yield further KL reductions. This waste stems not from the data or the teachers themselves, but from holdingp k fixed. Takeaway.Under vanilla MOPD, distinct domains enter low-KL plateau phases at entirely different train- ing stages (§ 2.2). A fixed mixture inevitably wastes compute on domains that have ceased to improve (§2.3). This highlights the necessity for a dynamic sampling ratiop k that smoothly adapts to native training signals, motivating the design of our method in §3. 3 Method Based on these observations, we proposeDynamicDomain ScheDuling for MOPD (D 3 -MOPD), a lightweight scheduler that adapts the per-domain sampling ratiop k using the reverse-KL signal. §3.1 describes the framework, which consists of an off-process watcher and a stratified data source. §3.2then details how the per-domain KL history is converted into a dynamic mixture. 3.1 Framework D 3 -MOPD introduces two components to a vanilla MOPD training loop, as illustrated in Figure4: anoff- process watcherthat computes the per-domain mixturep k from the reverse-KL signals already produced by the trainer, and astratified data sourcethat readsp k to assemble each batch with the target mixture. This pipeline generalizes to any number of domainsK. The trainer follows vanilla MOPD: at each rollout step, the student generates responses from prompts drawn by the data source. Each response is dispatched to its corresponding domain teacher, which prefills on the trajectory to produce its per-token distribution. The resulting per-domain reverse-KL is appended to a shared log. Since vanilla MOPD typically logs only the domain-averaged reverse-KL, the watcher groups per-sample KL values by domain to recover per-domain signals. It runs as a separate process that periodically computes a new mixturep k following § 3.2and writes it to a status file that the data source reads between updates so that the trainer never blocks. We replace the conventional pool-and-shuffle loader with thestratified data source. Given a batch sizeBand per-batch mixture ̃ p k (obtained by applying the batch-level jitter below to the watcher-suppliedp k ), 3 Trainer (main process) A.Datapools D Math D Code D Tool ... domain-partitioned prompt pools Ratio-drivendata source 푝 푘 with푗푖푡푒푟η no batch-level sampling noise mixture푝 푘 from watcher (updated every n steps) AnyDomain 1 static 푝 ! = 푁 ! ∑ 푁 " " % same 푝 푘 , 푗푖푡푒푟η =0.3 Dynamic mini-batch composition 2 Uniformmixture(vanillaMOPD) B.Dynamic BatchComposer D IF Tool 43% IF 15% Code 15% Math 24% Tool 25% IF 25% Code 25% Math 25% shown with K=4; 퐷 # 푀푂푃퐷is defined for any K Watcher(off-process) Input Output rollout step per-domainreverse-KL history math plateau mixture푝 푘 휖 푝 $%&' 푝 )*+, 푝 -. 푝 /**0 4 domains descend asynchronously ComputeModule max-norm across ksoftmax+ floor 휖 each 푝 ! ≥ 휖 8푝 ! =1 ! status.json 5 Feedback:new mixture for the next batch three consecutive batches 43%16%13%28% 43%16%20%21% 47%12%16%25% 6 Remaining gap 푔 ! = KL ! 푡 KL ! 0 normalized gap Descent velocity 푣 ! = recent descent rate Score 푠 ! =푔 ! ×푣 ! × minimum floor 휖 prevents domain starvation C.Rollout student 휋 ! 휋 1 !"#$ trajectoryy=푦 2 ,푦 3 ,...,푦 1 푦 2 푦 3 푦 1 ... per-domain reverse-KL 푦 ! 푦 " 푦 # ... ℓ $456 78 푦 2:1 휋 1 %&&' 푦 ! 푦 " 푦 # ... (푛표푡 푟푒푞푢푒푟푖푒푑) ... ... ... dispatch yto its domain teacher (only that teacher is queried) OPD reverse-KL loss(token-level) ℓ OPD (y)=D KL ( 휋 # ! (·|y <t ), π θ (·|y <t )) for t=1,...,T Student update minimize OPD loss on y on-policy rollout 4 3 Figure 4:The framework of D 3 -MOPD. At each rollout step, the student generates responses that the teach- ers prefill on to produce per-domain reverse-KL signals. A watcher periodically maps these signals into a continuous per-domain mixturep k that shapes subsequent batches. it first assigns⌊B· ̃ p k ⌋samples to each domain, then distributes the remainingB− ∑ k ⌊B· ̃ p k ⌋samples to the domains with the largest fractional parts. This strictly aligns each domain’s per-batch share with the target mixture, replacing the uncontrolled variance of pooled shuffling with the purposeful batch-level jitter described below. Batch-level jitter.Because the watcher updates the mixture only everynrollout steps, consecutive batches would otherwise share identical domain proportions. The data source therefore perturbs each batch’s pro- portions via ̃ p k =p k (1+u k )/ ∑ j p j (1+u j )withu k ∼Uniform(−η,η), restoring batch-to-batch variation while keeping the long-run mean close top k (E[u k ]=0). Integration with existing MOPD.All modifications are strictly confined to the data path, leaving the rollout, teacher prefill, and student update operations untouched. An existing MOPD implementation can adopt D 3 -MOPD by replacing the standard loader with the stratified data source and launching a watcher process alongside the trainer. 3.2 Composite Ratio from Remaining Gap and Descent Velocity The watcher translates the per-domain reverse-KL history into a mixturep k at each update. Our goal is to allocate more samples to domains exhibiting ongoing improvement and fewer to those that have converged. We quantify this improvement along two axes: theremaining gap f KL k (the proportion of the initial KL left to close) and thedescent velocity v k (the recent rate of KL reduction). Relying on either axis alone is insufficient. A domain might exhibit a large remaining gap but plateau, rendering further samples ineffective. Conversely, a nearly converged domain might descend rapidly within the current window, yet its minimal room for improvement makes additional samples redundant. Their product ensures that a domain receives a large share only when it has room to improve and is actively doing so. Remaining gap.We define the remaining gap by normalizing the current smoothed KL by its initial value, f KL k (t)= KL k (t) KL (0) k ,KL (0) k = 1 S 0 S 0 ∑ t=1 KL k (t),(2) where KL k (t)is the exponential moving average (EMA) of domain-kreverse-KL up to rollout stept, and KL (0) k is the mean of the firstS 0 observations seeded once at the start of training. Under normal training, 4 f KL k (t)∈(0, 1]and approaches zero as the domain converges. As documented in §2.2, this normalization enables fair comparison across domains whose absolute KL values may differ by orders of magnitude. Descent velocity.To determine whether a domain’s KL continues to decrease, we measure its recent rate of change. LetWbe a fixed window length in rollout steps andRthe number of non-overlapping windows used for averaging. We define the single-window relative change as δ (i) k (t)= KL k (t−iW)−KL k (t−(i+1)W) KL k (t−(i+1)W) ,i=0, . . . ,R−1,(3) whereδ (0) k represents the most recent window. The descent velocity takes the negative average of theseR changes and clips the result at zero, v k (t)=max 0,− 1 R R−1 ∑ i=0 δ (i) k (t) ,(4) meaningv k (t)>0 strictly requires the domain’s KL to decrease on average over the lastRwindows. Composite signal.We compute the composite signal via the product of these two metrics, s k (t)= f KL k (t)·v k (t), ̃ s k (t)=s k (t) max j s j (t),(5) ensurings k remains large only when a domain is both far from convergence and steadily descending. The max-normalization brings ̃ s k into[0, 1], making the softmax temperatureTindependent of the raw signal magnitude. In the degenerate case where all domains plateau and max j s j (t)=0, we set ̃ s k (t)=0 for allk, and the mapping defaults to a uniform mixture. Appendix Eprovides a theoretical analysis showing that under an exponential decay model,s k approximates the normalized marginal KL reduction, justifying the gap-velocity product as a greedy allocation rule. Mapping to a valid mixture.We map ̃ s k to a probability mixture through a softmax function parameter- ized by temperatureTand a per-domain floorε, p k (t)=ε+ 1−Kε · exp ̃ s k (t)/T ∑ K j=1 exp ̃ s j (t)/T ,(6) guaranteeing thatp k (t)≥εfor every domain and ∑ k p k (t)=1. The watcher recomputes this mixture at every update and delivers it to the data source via the status file described in § 3.1. Why average overRwindows.WithR=1, the velocity estimate is highly sensitive to random noise in the KL signal. Because Eq.4clips the sum at zero, a short-term KL spike in an actively learning do- main would wrongly pushv k to zero, cutting off its sample allocation. Averaging the changes overR non-overlapping windows prevents these brief spikes from accidentally triggering the cutoff. As a result, v k becomes zero only when the domain remains flat or rises over a longer period, making it a reliable sign of actual convergence. Warmup.The composite signal becomes available once 2Wobservations ofKL k are recorded. At this point, f KL k relies on the seeded KL (0) k , and computing a singleδ (0) k requires the two window endpoints at t−Wandt. Eq.4represents the standard formulation with a full history ofRwindows. During the warmup phase 2W≤t<(R+1)W, the algorithm replacesRwith the available window countR ′ (t)=⌊t/W⌋−1. Fromt=(R+1)Wonward,R ′ (t)is capped atR, and the standard form of Eq. 4takes over. Prior tot=2W, the watcher provides no mixture updates, causing the data source to default to a uniform mixturep k =1/K. The batch-level jitter (§ 3.1) remains active throughout this initial period. Design rationale.The temperatureTmodulates how sharply the sampling share is redirected: asT→∞, the mixture becomes uniform, whereas asT→0, the mixture concentrates on the argmax of ̃ s. Finally, the floorεprevents any domain from being completely discarded, ensuring already-converged domains retain a minimal presence to mitigate catastrophic forgetting. Algorithm 1summarizes the scheduling process. 5 Algorithm 1 DynamicDomain ScheDuling for MOPD (D 3 -MOPD). Require:Domain poolsD k K k=1 ; hyperparametersT,ε,S 0 ,R,W,n,η,B; shared KL log and status file. 1:Initializeat stept=S 0 : KL (0) k ← 1 S 0 ∑ S 0 τ=1 KL k (τ)for allk▷Eq.2; seeded once 2:functionWATCHERTICK(t)▷invoked at rollout stept, everynsteps 3:ift<2Wthen return▷initial warmup; data source uses uniform fallback 4:end if 5:R ′ ←min R,⌊t/W⌋−1 ▷R ′ =Roncet≥(R+1)W 6:fork=1, . . . ,Kdo 7: f KL k ←KL k (t)/KL (0) k ▷Eq.2 8:δ (i) k ← KL k (t−iW)−KL k (t−(i+1)W) /KL k (t−(i+1)W)fori=0, . . . ,R ′ −1 9:v k ←max 0,− 1 R ′ ∑ i δ (i) k ▷Eq.4 10:s k ← f KL k ·v k 11:end for 12:ifmax j s j =0then▷degenerate case: all domains plateaued 13: ̃ s k ←0 for allk 14:else 15: ̃ s k ←s k / max j s j for allk▷Eq.5 16:end if 17:p k ←ε+(1−Kε) exp( ̃ s k /T) ∑ j exp( ̃ s j /T) for allk▷Eq.6 18:Atomically writep k to the status file 19:end function 20:functionGETBATCH(B)▷invoked at every rollout step 21:p k ←status file (fallbackp k ←1/K) 22:u k ∼Uniform(−η,η); ̃ p k ←p k (1+u k ) ∑ j p j (1+u j )▷jitter + renormalize 23:Assignn k ←⌊B ̃ p k ⌋; distribute the remainingB− ∑ k n k samples to the domains with the largest fractional parts so ∑ k n k = B 24:returnn k prompts sampled fromD k for eachk 25:end function 4 Experiments 4.1 Training Setup We validate D 3 -MOPD using Qwen3.6-35B-A3B [21] as the student model andK=4 domain-expert teachers that share the same backbone but are individually fine-tuned via GRPO [22] on a single domain: Math, Code, Instruction Following, or Tool-use, extending the three-domain analysis of §2with a fourth do- main. We optimize the student using the vanilla MOPD [16] per-token reverse-KL (r-KL) objective on the slime [35] framework, which employs an asynchronous rollout-update loop that overlaps teacher prefill with student generation. The composite signal watcher operates as an off-process job alongside the trainer following §3.1. It recomputes the mixturep k everyn=10 rollout steps and delivers it to the stratified data source via a status file. All main runs use a rollout batch size of 128 over 256 rollout steps. Other training details are provided in AppendixB. Training data.We construct a prompt set of∼4k prompts per domain, formatted as single-turn message lists in JSONL with per-sample domain tags. The Math and Code splits come from DAPO-Math-17K [ 30] and CodeI/O [12], respectively. The Instruction Following split is filtered from Nemotron-Cascade 2 [28], and the Tool-use split uses the Fission-GRPO [33] training set. 4.2 Evaluation Setup We evaluate on seven benchmarks covering the four training domains: AIME 2025 [ 17] (avg@64) and HMMT November 2025 [ 5] (avg@32) for mathematics; LiveCodeBench Code Generation [9] (avg@6) and OJBench C++ [26] (default protocol) for code; IFBench [19] and IFEval [34] for instruction following; and BFCL v3 Multi-Turn (Base) [ 18] for tool-use. We generate responses in non-thinking mode with temperature 0.7, top-p0.8, top-k20, and presence penalty 1.5, following the official Qwen3.6-35B-A3B recommendations. 6 063127191255 70 71 72 73 74 Percentage (%) AIME 2025 (Avg@64) Vanilla MOPD D 3 -MOPD (Ours) 063127191255 71 72 73 HMMT Nov 2025 (Avg@32) Vanilla MOPD D 3 -MOPD (Ours) 063127191255 30 33 36 39 42 45 48 IFBench Vanilla MOPD D 3 -MOPD (Ours) +2.8 063127191255 87 88 89 90 91 92 IFEval Vanilla MOPD D 3 -MOPD (Ours) 063127191255 Rollout Step 60 61 62 63 Percentage (%) LiveCodeBench v6 (Avg@6) Vanilla MOPD D 3 -MOPD (Ours) +1.6 063127191255 Rollout Step 25 26 27 28 29 OJBench C++ Vanilla MOPD D 3 -MOPD (Ours) +2.1 063127191255 Rollout Step 58 59 60 61 62 BFCL v3 Multi-Turn (Base) Vanilla MOPD D 3 -MOPD (Ours) 063127191255 Rollout Step 57 58 59 60 61 62 Average Vanilla MOPD D 3 -MOPD (Ours) +1.0 Figure 5:Per-benchmark accuracy of vanilla MOPD and D 3 -MOPD across the 16 evaluated rollout check- points, spanning seven benchmarks plus their unweighted average. D 3 -MOPD matches or outperforms the vanilla baseline on every benchmark at almost every checkpoint. We evaluate every 16 rollout steps starting from step 15. D 3 -MOPD and vanilla MOPD are trained under identical settings except for the mixture policy: fixedp k vs. online-updatedp k . Since raw student-to- teacher performance gaps differ across domains, averaging standard accuracies over-weights domains with larger margins. We therefore report a per-benchmark normalized score ˆ s b =(s b −s stu b )/(s tea b −s stu b ), which scales the initial student to 0 and the domain-expert teacher to 1, and compute its uniform average across benchmarks. Values above 1.0 indicate improvement beyond the respective specialist teacher. 4.3 Main Results Figure5shows the per-benchmark accuracy of D 3 -MOPD and vanilla MOPD over the training budget. As shown in the average panel, both methods improve rapidly during the first∼50 steps before plateauing, with D 3 -MOPD consistently maintaining a higher overall trajectory. The composite signal first activates at step 20 (remaining-gap component only) and integrates the velocity component from step 40. Consequently, the initial evaluation at step 15 shows minimal performance differences. Overall, D 3 -MOPD outperforms vanilla MOPD in both peak performance and training efficiency. We highlight three primary findings: •D 3 -MOPD peaks higher on all benchmarks.The vanilla baseline reaches its optimal average of 61.4 at step 143. In contrast, D 3 -MOPD attains a higher peak of 62.3 as early as step 95, demonstrating individ- ual benchmark gains ranging from+0.2 on AIME 2025 (74.1 vs 73.9) to+2.8 on IFBench (48.9 vs 46.1). The early peak in average accuracy occurs because code benchmarks (LiveCodeBench and OJBench) con- verge and begin degrading after∼80 steps, consistent with the domain over-optimization observed in § 2 (Figure3). By reducing the share of these converging domains, the scheduler redirects the rollout budget toward domains with active growth, raising the overall peak accuracy. •D 3 -MOPD reaches the vanilla peak with3×fewer steps.While vanilla MOPD requires 143 steps to reach its average score of 61.4, D 3 -MOPD surpasses this threshold by step 47 (scoring 62.1). Furthermore, vanilla MOPD fails to establish its per-benchmark peaks until as late as step 207 (AIME 2025). Notably, D 3 -MOPD matches or exceeds every one of these individual peaks within the first 79 steps. Because the scheduler continuously shifts prompt sampling from converging domains to those still improving, each rollout step yields a larger marginal gain, accelerating overall convergence. •Domain convergence order aligns with r-KL.Code benchmarks peak earliest (LiveCodeBench by step 79, OJBench by step 95), whereas instruction-following benchmarks peak latest (IFBench at step 175, 7 Table 1:Ablation results. Best-S denotes the rollout step whose checkpoint achieves the highest average across the evaluated benchmarks over the full 256-step run. All variants differ from D 3 -MOPD in only the ablated variable; other settings are identical. MathIFCodeTool MethodBest-S AIME25 HMMT(N) IFBench IFEval LCB v6 OJB C++ BFCL(B)Avg Vanilla MOPD143 73.1771.0445.74 91.04 60.95 27.59 60.0061.36 w/o velocity127 72.6671.1545.17 90.91 59.52 28.4562.0061.41(+0.05) w/o gap255 71.7271.4649.0190.83 60.10 24.57 62.5061.46(+0.10) w/o jitter (η=0)95 72.6671.1546.58 89.90 60.95 27.16 63.0061.63(+0.27) w/o smoothing (R=1) 95 73.0771.4646.9492.1461.8126.2963.5062.17(+0.81) D 3 -MOPD (η=0.30,R=3) 95 74.01 71.3545.07 91.1662.52 29.7462.5062.34(+0.98) IFEval at step 191), while mathematics falls in between. This progression aligns with the r-KL conver- gence pattern in Figure3, where code r-KL plateaus first and IF r-KL last. This consistent alignment validates r-KL as an effective monitoring signal: a domain with a plateaued r-KL has likely approached its peak accuracy, and further budget allocation risks computational waste and model degradation. 4.4 Ablation Studies Table 1evaluates the individual contributions of three design choices in D 3 -MOPD. The per-domain ratio trajectories of each variant are provided in AppendixG. Composite signal vs. single-signal variants.Although both single-signal variants outperform vanilla MOPD (w/o velocity 61.41, w/o gap 61.46) and validate r-KL-driven dynamic scheduling, neither matches the composite formulation (62.34). The two signals differ in convergence speed. The w/o velocity variant peaks early at step 127 because it uses the current normalized r-KL levels to identify plateauing domains. In contrast, the w/o gap variant peaks at step 255. Since all domains’ KL drops rapidly during early train- ing, the velocity signal cannot differentiate them until descent rates diverge. Notably, the w/o gap variant records the highest IFBench score (49.01), consistent with IF converging last (§4.3). A velocity-driven sched- uler allocates budget to IF since its descent rate remains high, enabling prolonged capability growth. Effectiveness of batch jitter.Removing jitter (η=0) does not alter the peak step (both at 95) but reduces the peak average by 0.71 (61.63 vs 62.34), with the most severe drops on code benchmarks (OJBench−2.6, LiveCodeBench−1.6). We hypothesize that the proportion variation from jitter acts as a form of exploration: for domains nearing their r-KL plateau, a fixed sampling ratio yields only weak gradients, and the random- ized bursts from jitter may deliver stronger updates. This would explain why the rapidly plateauing code benchmarks benefit the most. The identical peak step confirms that jitter improves overall training quality without altering convergence speed. Effectiveness of velocity smoothing.The hyperparameterRcontrols how many non-overlapping obser- vation windows are averaged to estimate descent velocity:R=1 relies on a single window (reactive but noisy), whereas D 3 -MOPD adoptsR=3 for smoother estimates. Both variants peak at step 95, andR=1 already achieves 62.17 (+0.81 over vanilla), indicating that even a single-window velocity captures useful convergence information.R=3 further improves the average to 62.34 by mitigating random r-KL variance. This smoothing prevents a single noise-induced r-KL fluctuation from temporarily zeroing the velocity sig- nal and cutting off budget to actively learning domains. Interestingly,R=1 surpasses D 3 -MOPD on BFCL (63.50 vs 62.50), likely because tool-use exhibits high cross-checkpoint variance, where a noisier estimate encourages beneficial exploration. 4.5 Supplementary Analysis D 3 -MOPD closes 97% of the student-teacher gap.Table2compares peak accuracy against the student baseline and domain-expert teachers. Vanilla MOPD closes 63% of the average student-to-teacher gap ( ̄ ˆ s= 8 Table 2:Per-benchmark peak accuracy (%) and normalized score (§4.2) over 256 rollout steps. Each value represents the maximum across all 16 checkpoints.∆rows show gaps to the domain-expert teacher. Bold marks the column best, underline the runner-up. MathInstruction FollowingCodeTool-use MethodAIME25 HMMT(N) IFBench IFEval LCB-v6 OJB-C++ BFCL(B)Norm. Teacher76.272.352.191.964.828.567.01.00 Student69.870.730.086.860.025.957.50.00 ∆(Student−Teacher)−6.4−1.6−22.1−5.1−4.8−2.6−9.5−1.00 Vanilla MOPD73.972.046.191.161.527.661.50.63 ∆(Vanilla−Teacher)−2.3−0.3−6.0−0.8−3.3−0.9−5.5−0.37 D 3 -MOPD (Ours)74.173.248.992.363.229.762.50.97 ∆(D 3 -MOPD−Teacher)−2.1+0.9−3.2+0.4−1.6+1.2−4.5−0.03 0.63), whereas D 3 -MOPD closes 97% of this gap ( ̄ ˆ s=0.97). Consistent with findings by Ma et al. [ 16], both methods generally remain below the maximum performance of the domain-expert teachers, indicating that distillation primarily closes existing capability gaps rather than generating novel skills. Notably, D 3 -MOPD surpasses the teacher on three benchmarks (HMMT(N), IFEval, and OJBench C++, ˆ s b >1.0), whereas vanilla MOPD remains below the teacher on all tasks. Because these three benchmarks exhibit small initial student-to-teacher absolute accuracy gaps (∆≤5.1), even modest absolute improvements translate into normalized scores exceeding 1.0. Appendix Cfurther compares the two methods at their respective best- average checkpoints, confirming that D 3 -MOPD’s improvement persists under this setting. 063127191255 0.0 0.1 0.2 0.3 0.4 0.5 Domain Ratio Math Vanilla MOPD D 3 -MOPD 063127191255 0.0 0.1 0.2 0.3 0.4 0.5 Code Vanilla MOPD D 3 -MOPD 063127191255 Rollout Step 0.0 0.1 0.2 0.3 0.4 0.5 Domain Ratio Instruction Following Vanilla MOPD D 3 -MOPD 063127191255 Rollout Step 0.0 0.1 0.2 0.3 0.4 0.5 Tool-use Vanilla MOPD D 3 -MOPD Figure 6:Per-domain sampling ratio of vanilla MOPD and D 3 -MOPD. D 3 -MOPD dynamically reallocates budget as domains converge. D 3 -MOPD shifts budget from converging to ac- tive domains.Figure6shows the per-domain sampling ratio throughout training. While vanilla MOPD maintains all four domains near 0.25 with only batch-level noise, D 3 -MOPD dynamically adapts this allocation, closely tracking domain con- vergence. Code converges fastest (its r-KL plateaus first in Figure3) and is steadily downsampled from 0.25 to∼0.15 throughout training. The freed bud- get flows initially to Math, whose ratio rises to ∼0.50 during steps 60–127 as the composite signal identifies Math as the domain with the largest re- maining gap and active descent. Once Math con- verges after step 127, its ratio recedes, and the scheduler redirects budget toward Instruction Fol- lowing and Tool-use, pushing their ratios to∼0.55 and∼0.50 after step 150. Because these two do- mains exhibit the slowest r-KL convergence, this late-stage budget increase enables the student to sustain progress where capacity for improvement remains high. Ultimately, this dynamic reallocation func- tions as an implicit curriculum: each domain receives concentrated training during the window when it can still improve, rather than having the budget split uniformly regardless of learning state. 5 Related Work 5.1 On-Policy Distillation On-policy distillation (OPD) trains a student on its own generations, supervised by the teacher’s distribu- tion under a reverse Kullback-Leibler objective [7,1]. Recent work refines this supervision at the token or sample level: Jin et al. [ 11] switch between forward and reverse KL based on the teacher’s entropy, Li et al. [14] filter and reweight tokens to suppress noisy gradients, and Hou et al. [8] correct the mismatch between per-token KL and sequence-level outcomes. Other directions stabilize the learning target [ 10], dis- till selectively from reasoning prefixes [32], or extend beyond the teacher via reward signals [27]. Li et al. 9 [13] provide a systematic analysis of these design choices. These methods operate on the local supervision signal, improving which tokens or samples the student learns from. In this paper, we address a comple- mentary dimension: instead of refining per-token supervision, we dynamically adjust the per-domain data allocation when multiple teachers are involved. 5.2 Multi-Teacher On-Policy Distillation Multi-teacher OPD (MOPD) extends the single-teacher setting by distilling several domain-expert teachers into one student [16]. Recent large-scale post-training pipelines adopt MOPD-style distillation: MiMo- V2-Flash [15] and DeepSeek-V4 [4] use it for capability integration, while Baichuan-M3 [25], Nemotron- Cascade 2 [28], and GLM-5 [31] apply it in vertical-domain or staged pipelines. Chen et al. [3] further observe that domain teachers can pull the student in conflicting directions, and mitigate this through alter- nating per-domain training; Shen et al. [23] and Yin et al. [29] address failure modes such as tool-call bound- ary drift and heterogeneous teacher confidence. However, all existing methods keep the per-domain data ratio fixed throughout training and therefore cannot react to the asynchronous convergence documented in §2. In this paper, we revisit MOPD from a scheduling perspective: the per-domain data ratio is treated as a continuous variable, updated online from the reverse-KL signal the training loop already produces. 6 Conclusion This paper identifies and addresses a systematic inefficiency in multi-teacher on-policy distillation (MOPD) arising from the mismatch between its static domain mixture and the student’s asynchronous per-domain convergence rates. D 3 -MOPD resolves this by combining each domain’s remaining KL gap with its descent velocity into a composite signal and mapping the result to an updated sampling ratio through a softmax- floor formulation, all without modifying the core training loop. Experiments across four domains with a Qwen3.6-35B-A3B student show that D 3 -MOPD closes 97% of the student-teacher performance gap (vs. 63% for vanilla MOPD) while reaching the baseline’s peak accuracy in 3×fewer rollout steps. Ablation studies confirm that each design choice, namely the composite signal, batch jitter, and velocity smoothing, contributes to both peak accuracy and convergence speed. Trajectory analysis reveals an emergent implicit curriculum in which the scheduler concentrates budget on each domain during its active learning window. References [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. InInternational Conference on Learning Representations, volume 2024, p. 21246–21263, 2024. [2]Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025. [3]Tianlei Chen, Jiao Ou, Ziyuan Liu, Ruiming Tang, Jian Liang, and Han Li. Counteraction-aware multi- teacher on-policy distillation for general capability recovery with domain preservation.arXiv preprint arXiv:2605.27115, 2026. [4]DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026. URL https://arxiv.org/abs/2606.19348. [5]Jasper Dekoninck, Nikola Jovanovi ́c, Tim Gehrunger, Kári Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms.arXiv preprint arXiv:2605.00674, 2026. [6]Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao, Shuang Chen, Kaituo Feng, Yunlong Lin, Lin Chen, Zehui Chen, Shaosheng Cao, and Feng Zhao. Flow-opd: On-policy distillation for flow match- ing models, 2026. URL https://arxiv.org/abs/2605.08063. [7]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations, volume 2024, p. 32694–32717, 2024. 10 [8]Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, et al. Uni-opd: Unifying on-policy distillation with a dual- perspective recipe.arXiv preprint arXiv:2605.03677, 2026. [9]Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InInternational Conference on Learning Representations, volume 2025, p. 58791–58831, 2025. [10]Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggyu Lim, and Taesup Kim. Stable on-policy distillation through adaptive target reformulation. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (eds.),Findings of the Association for Computational Linguistics: ACL 2026, p. 42217–42227, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-395-1. doi: 10.18653/v1/2026.findings-acl.2094. URLhttps://aclanthology.org/2026. findings-acl.2094/. [11]Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079, 2026. [12]Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codeio: Condensing rea- soning patterns via code input-output prediction. InInternational Conference on Machine Learning, p. 34471–34489. PMLR, 2025. [13]Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe, 2026. URLhttps://arxiv.org/abs/2604.13016. [14]Yuying Li, Leqi Zheng, Yongzi Yu, Wenrui Zhou, Xuchang Zhong, Xing Hu, Jing Jin, Huangjie Yuan, and Tao Feng. Filter, then reweight: Rethinking optimization granularity in on-policy distillation. arXiv preprint arXiv:2606.02684, 2026. [15]LLM-Core Xiaomi. Mimo-v2-flash technical report, 2026. URLhttps://arxiv.org/abs/2601.02780. [16]Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, et al. Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training.arXiv preprint arXiv:2606.30406, 2026. [17]Mathematical Association of America. 2025 American Invitational Mathematics Examination I.https: //artofproblemsolving.com/wiki/index.php/2025_AIME_I, 2025. Held on February 6, 2025. [18]Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. InForty-second International Conference on Machine Learning, 2025. [19]Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hanna Hajishirzi. Generalizing verifiable instruction following.Advances in Neural Information Processing Systems, 38, 2026. [20]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URLhttps://qwen.ai/ blog?id=qwen3.5 . [21]Qwen Team. Qwen3.6-35B-A3B: Agentic coding power, now open to all, April 2026. URLhttps: //qwen.ai/blog?id=qwen3.6-35b-a3b. [22]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. [23]Jiabin Shen, Guang Chen, and Chengjun Mao. Diagnosing and calibrating tool-call boundary drift in multi-teacher on-policy distillation, 2026. URL https://arxiv.org/abs/2607.07050. [24]Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models, 2026. URLhttps://arxiv.org/abs/2604.00626. 11 [25]M3 Team, Chengfeng Dou, Fan Yang, Fei Li, Jiyuan Jia, Qiang Ju, Shuai Wang, Tianpeng Li, Xiangrong Zeng, Yijie Zhou, Hongda Zhang, Jinyang Tai, Linzhuang Sun, Peidong Guo, Yichuan Mo, Xiaochuan Wang, Hengfu Cui, and Zhishou Zhang. Baichuan-m3: Modeling clinical inquiry for reliable medical decision-making, 2026. URLhttps://arxiv.org/abs/2602.06570. [26]Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofei Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, Zhilin Yang, et al. Ojbench: A competition level code benchmark for large language models.arXiv preprint arXiv:2506.16395, 2025. [27]Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation, 2026. URLhttps://arxiv.org/ abs/2602.12125. [28]Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, et al. Nemotron-cascade 2: Post-training llms with cascade rl and multi-domain on-policy distillation.arXiv preprint arXiv:2603.19220, 2026. [29]Qixiang Yin, Huanjin Yao, Yuchen Cai, Jianghao Chen, Ziyi Wang, Min Yang, Fei Su, and Zhicheng Zhao. H-opd: Confidence aware heterogeneous multi-teacher multimodal on-policy distillation, 2026. URLhttps://arxiv.org/abs/2607.02592. [30]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2025. [31]Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026. [32]Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler I, Qian Qian, Gregory D Lyng, Sanjit Singh Batra, and Robert E. Tillman. Fast and effective on-policy distillation from reasoning prefixes. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (eds.),Findings of the Association for Computational Linguistics: ACL 2026, p. 25553–25569, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-395-1. doi: 10.18653/ v1/2026.findings-acl.1276. URL https://aclanthology.org/2026.findings-acl.1276/. [33]Zhiwei Zhang, Fei Zhao, Rui Wang, Zezhong Wang, Bin Liang, Jiakang Wang, Yao Hu, Shaosheng Cao, and Kam-Fai Wong. Robust tool use via fission-grpo: Learning to recover from execution errors.arXiv preprint arXiv:2601.15625, 2026. [34]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023. [35]Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An llm post-training framework for rl scaling.https://github.com/THUDM/slime, 2025. GitHub repository. Corresponding author: Xin Lv. 12 A Contributors Zechen Sun, Zhiwei Zhang, Fei Zhao, Juntao Li, Mu Chuan, Huayu Deng, Guojian Zhan, Wenliang Chen, Yao Hu, Min Zhang. B Training Hyperparameters and Infrastructure All main and ablation runs share the hyperparameters below; only the mixture-scheduling components (§3.2) differ across variants. B.1 D 3 -MOPD Scheduler Table3lists the scheduler settings used for the D 3 -MOPD composite-signal method. Table 3:D 3 -MOPD scheduler hyperparameters. Symbol DescriptionValue nWatcher update cadence (rollout steps)10 WWindow length in the descent-velocity estimate (rollout steps) 10 R Number of non-overlapping windows averaged in v k 3 S 0 Seed length for the initial-KL normalizer KL (0) k 5 –EMA window applied to raw per-step KL10 –Numerical KL floor (ε KL ) in denominators0.15 TSoftmax temperature in Eq.60.5 εPer-domain mixture floor in Eq. 60.10 ηBatch-level jitter amplitude0.30 –File-poll interval of the watcher process (seconds)300 B.2 Student Training The student is trained with slime [ 35] on top of the Megatron backend; rollout inference is handled by an SGLang engine group co-located with the trainer. All runs use 7 nodes of GPUs: 4 trainer nodes, 2 SGLang rollout nodes, and 1 teacher node shared byK=4 teachers, connected via RDMA InfiniBand. Table4lists the training-loop hyperparameters, and Table5lists parallelism and performance settings. Table 4:Training hyperparameters. SettingValue Student initializationQwen3.6-35B-A3B Number of teachersK4 Total rollout steps256 Rollout batch sizeB(prompts per step)128 Responses sampled per prompt4 Global mini-batch size128 Gradient-accumulation micro-steps per rollout step 4 OptimizerAdam Learning rate1×10 −6 Learning-rate scheduleconstant Weight decay0.1 Adam(β 1 ,β 2 )(0.9, 0.98) Policy-loss variant (advantage estimator)CISPO [ 2] Clip ratio (low, high)(0.2, 0.2) OPD reverse-KL coefficient1.0 Max prompt length16,384 tokens Max response length8,192 tokens Rollout sampling temperature1.0 Chat-template thinking modedisabled 13 Table 5:Parallelism and performance settings (Megatron backend). SettingValue Tensor-model parallel size2 Context-parallel size4 Expert-model parallel size8 Max tokens per GPU per micro-batch 6,144 SGLang rollout: GPUs per engine4 Throughput overhead.Because the watcher runs as a standalone process that reads the training log and writes updated ratios to a status file without requiring the trainer to pause, it adds no synchronization overhead. Over the full 256-step run the mean throughputs of D 3 -MOPD (520 tokens/GPU/s) and vanilla MOPD (531 tokens/GPU/s) differ by only 2.1%, a gap attributable to the shift in sequence-length distribu- tion as the scheduler redirects budget toward longer-response domains rather than to any computational overhead from the watcher. C Best-Average Checkpoint Comparison Table2reports the per-benchmarkpeakaccuracy, where each benchmark’s best value may come from a different checkpoint. Table6instead reports the accuracy at the single checkpoint with the highest average score, reflecting the performance of a single deployable model. Table 6:Per-benchmark accuracy (%) and normalized score at the best-average checkpoint (vanilla MOPD: step 143; D 3 -MOPD: step 95). The Teacher row reports each domain’s dedicated expert model (one spe- cialized model per domain, not a single model). Unlike Table2, all values in each row come from a single checkpoint. Bold marks the column best, underline the runner-up. MathInstruction FollowingCodeTool-use MethodAIME25 HMMT(N) IFBench IFEval LCB-v6 OJB-C++ BFCL(B)Norm. Teacher76.272.352.191.964.828.567.01.00 Student69.870.730.086.860.025.957.50.00 Vanilla MOPD(step 143)73.271.045.791.061.027.660.00.48 ∆(Vanilla−Teacher)−3.0−1.3−6.4−0.9−3.8−0.9−7.0−0.52 D 3 -MOPD(step 95)74.071.445.191.262.529.762.50.73 ∆(D 3 -MOPD−Teacher)−2.2−0.9−7.0−0.7−2.3+1.2−4.5−0.27 ∆(D 3 -MOPD−Vanilla)+0.8+0.4−0.6+0.2+1.5+2.1+2.5+0.25 Comparison with per-benchmark peaks.Compared with Table2, the normalized scores drop for both methods (vanilla MOPD: 0.63→0.48; D 3 -MOPD: 0.97→0.73), since no single checkpoint can maximize every benchmark at once. The drop is concentrated in benchmarks whose peaks occur far from the best- average step: for example, D 3 -MOPD’s HMMT(N) falls from 73.2 (per-benchmark peak) to 71.4 at step 95, indicating that HMMT(N) peaks at a later checkpoint. Nevertheless, D 3 -MOPD at step 95 still outperforms vanilla MOPD at step 143 on six of seven benchmarks and achieves a notably higher normalized score (0.73 vs. 0.48), confirming that the improvement holds when deploying a single checkpoint. D Generalization to 4B Student To verify that D 3 -MOPD generalizes across model scales, we replicate the training pipeline with Qwen3.5- 4B [20] as the student. The four domain-specific teachers are obtained by GRPO-training the same model on the corresponding data used for the Qwen3.6-35B-A3B experiments. Training hyperparameters (Table 4) and scheduler settings (Table3) follow the same configuration with minor scale-related adjustments. 14 Table 7:Per-benchmark accuracy (%) and normalized score with Qwen3.5-4B student (vanilla MOPD: step 159; D 3 -MOPD: step 119). The Teacher row reports each domain’s dedicated GRPO-trained 4B ex- pert. Bold marks the column best, underline the runner-up. MathInstruction FollowingCodeTool-use MethodAIME25 HMMT(N) IFBench IFEval LCB-v6 OJB-C++ BFCL(B)Norm. Teacher63.466.455.985.353.318.163.01.00 Student47.851.735.985.339.413.851.00.00 Vanilla MOPD (step 159) 59.363.445.4 82.7 53.718.563.00.86 ∆(Vanilla−Teacher)−4.1−3.0−10.5−2.6+0.4+0.40.0−0.14 D 3 -MOPD(step 119)61.864.449.584.554.619.864.51.01 ∆(D 3 -MOPD−Teacher)−1.6−2.0−6.4−0.8+1.3+1.7+1.5+0.01 ∆(D 3 -MOPD−Vanilla)+2.5+1.0+4.1+1.8+0.9+1.3+1.5+0.15 Results.Table7shows that the improvement pattern observed at the 35B-A3B scale carries over to the 4B student. D 3 -MOPD outperforms vanilla MOPD on all seven benchmarks, with per-benchmark gains rang- ing from+0.9 (LCB-v6) to+4.1 (IFBench). The normalized score of D 3 -MOPD (1.01) slightly surpasses the composite teacher ceiling (1.00), whereas vanilla MOPD reaches 0.86. Both methods surpass the domain- expert teachers on the code benchmarks (LCB-v6 and OJB-C++), and D 3 -MOPD additionally exceeds the tool-use teacher on BFCL(B) by+1.5. The remaining gap to the teacher is largest on IFBench (−6.4), consis- tent with instruction following being the hardest domain for distillation at both model scales. Convergence speed.D 3 -MOPD reaches its best-average checkpoint 40 steps earlier than vanilla MOPD (step 119 vs. step 159), a relative reduction of 25% in the number of training steps needed to achieve peak average performance. This faster convergence aligns with the 35B-A3B result (step 95 vs. step 143, Table6), indicating that dynamic domain scheduling accelerates convergence across model scales. E Theoretical Analysis of Composite-Signal Allocation We provide a simplified analysis showing that under an exponential decay model, the composite signal s k = f KL k ·v k is proportional to the marginal reduction in the normalized gap, justifying its use as a greedy- optimal allocation criterion. Exponential decay model.We model each domain’s reverse-KL as decaying exponentially in its cumula- tive training exposure: KL k (n k )=KL k (0)·e −λ k n k ,(7) wheren k is the cumulative sample count allocated to domainkandλ k >0 is a domain-specific learning rate. This captures the empirical observation (§2.2) that per-domain KL decreases roughly exponentially at domain-specific rates. Proposition 1Under the exponential decay model (Eq.7), with batch size B, window length W, uniform allocation p k =1/K, and small per-window changes (λ k B W/K≪1), the composite signal satisfies s k = f KL k ·v k ≈ B W K · ∂ f KL k ∂n k .(8) Allocating samples toarg max k s k thus maximizes the instantaneous reduction in the average normalized gap ̄ g= 1 K ∑ k f KL k . Proof.The normalized gap is f KL k =KL k (n k )/KL k (0)=e −λ k n k , with marginal reduction ∂ f KL k ∂n k =λ k e −λ k n k =λ k f KL k . 15 Under uniform allocationp k =1/K, each window ofWrollout steps deliversB W/Ksamples to domaink. Substituting into Eq.3, the single-window relative change isδ (i) k =e −λ k B W/K −1, constant across windows i. The descent velocity (Eq.4) therefore reduces tov k =1−e −λ k B W/K . A first-order expansion yields v k ≈λ k B W/K, giving s k = f KL k ·v k ≈ B W K ·λ k f KL k = B W K · ∂ f KL k ∂n k . SinceB W/Kis constant across domains, arg max k s k =arg max k |∂ f KL k /∂n k |.□ Interpretation.Proposition1shows that the composite signal ranks domains by the magnitude of their normalized marginal KL reduction. The initial-KL normalization in f KL k (Eq.2) is essential: without it, domains with large absolute KL (e.g., IF, whose KL is∼65×that of Math) would dominate the allocation regardless of their relative improvement rate. The first-order conditionλ k B W/K≪1 requires per-window KL changes to be small, which is satisfied in practice by the short window length (W=10 steps). Practical robustness.The softmax-floor mapping (Eq.6) smooths the greedy arg max k s k allocation: the temperatureTdistributes samples across domains in proportion to their exponential signal strength rather than concentrating on a single domain, while the floorεguarantees every domain retains a minimum sam- ple rate, preventing catastrophic forgetting of domains that have temporarily plateaued but may resume descent later in training. Remark on the exponential decay assumption.The exponential decay model is a simplifying assump- tion that we adopt to derive the proportionality betweens k and the marginal KL reduction in closed form. The method itself does not require this assumption to hold exactly, because the composite signal is self- correcting by construction: if a domain deviates from exponential decay and enters an unexpected plateau, its descent velocityv k drops toward zero and automatically reduces its allocation regardless of the un- derlying functional form. The theoretical analysis therefore serves as a design rationale for choosing the gap-velocity product as the allocation criterion rather than a necessary condition for the method to produce effective schedules. F Composite Signal Trajectories Figure7visualizes how the composite signal ̃ s k and the normalized KL f KL k evolve for each domain over the full 256-step training run. Panel (e) shows the corresponding allocation ratiop k produced by the scheduler. The scheduler concentrates budget on whichever domain currently exhibits both a large remaining gap and rapid descent, producing a sequence of distinct allocation phases. In the first phase (steps 20–55), IF receives the highest signal because its initial KL gap is the largest among all domains and it descends rapidly once training begins, causing the allocation to reach approximately 38% while the other three domains receive near-floor shares. As IF’s descent velocity slows around step 55, Math takes over as the dominant domain and holds ̃ s math =1.0 for roughly 60 consecutive steps, during which its normalized KL drops steadily from 0.42 to 0.25. The scheduler then redirects budget to Tool-use (steps 120–160) and finally returns to IF (steps 180–250) as IF’s accumulated gap reopens after an extended period of reduced allocation. The self-correcting property of the composite signal is visible in the interplay between the shaded signal regions and the KL curves within each panel. When a domain’s signal drops to zero and it receives only the floor allocationε=0.10, its normalized KL tends to plateau or rise (for example, IF’s normalized KL climbs from 0.29 to 0.65 during steps 65–120), which eventually increases the remaining gap component and restores a nonzero signal in a later update. This feedback loop ensures that no domain is permanently neglected even without explicit revisitation logic in the scheduler. G Ablation Domain Ratio Trajectories Figure8shows the per-domain sampling ratio of each ablation variant alongside vanilla MOPD and D 3 - MOPD over 256 rollout steps. 16 50100150200250 0.0 0.5 1.0 Signal s k (a) Math 0.0 0.2 0.4 0.6 Normalized KL 50100150200250 0.0 0.5 1.0 (b) Code 0.0 0.2 0.4 0.6 Normalized KL 50100150200250 Rollout step 0.0 0.5 1.0 Signal s k (c) IF 0.0 0.2 0.4 0.6 Normalized KL 50100150200250 Rollout step 0.0 0.5 1.0 (d) Tool-use 0.0 0.2 0.4 0.6 Normalized KL 50100150200250 Rollout step 0.2 0.4 Target ratio p k uniform (e) Allocation ratio (all domains) MathCodeIFTool-use Figure 7:Per-domain composite signal and normalized KL over 256 rollout steps, with the resulting alloca- tion ratio in panel (e). The scheduler produces a sequence of phase transitions that track the domain with the largest remaining gap and fastest descent at each point in training. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 (a) Domain Ratio Math 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Code 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Instruction Following 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Tool-use 0.0 0.1 0.2 0.3 0.4 0.5 0.6 (b) Domain Ratio 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.0 0.1 0.2 0.3 0.4 0.5 0.6 (c) Domain Ratio 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.0 0.1 0.2 0.3 0.4 0.5 0.6 063127191255 Rollout Step 0.0 0.1 0.2 0.3 0.4 0.5 0.6 (d) Domain Ratio 063127191255 Rollout Step 0.0 0.1 0.2 0.3 0.4 0.5 0.6 063127191255 Rollout Step 0.0 0.1 0.2 0.3 0.4 0.5 0.6 063127191255 Rollout Step 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Vanilla MOPD D 3 -MOPDD 3 -MOPD w/o velocityD 3 -MOPD w/o gapD 3 -MOPD w/o jitterD 3 -MOPD w/o smoothing Figure 8:Per-domain sampling ratio of each ablation variant vs. D 3 -MOPD and vanilla MOPD over 256 rollout steps. (a) w/o velocity, (b) w/o gap, (c) w/o jitter, (d) w/o smoothing. •w/o velocity (row a).Without descent velocity, the scheduler relies solely on the KL gap. The ratios are flatter than D 3 -MOPD: Math rises to∼0.30 but never reaches the∼0.50 peak of D 3 -MOPD, and the late- 17 stage shift toward IF and Tool-use is weaker. Because the gap signal alone cannot distinguish an actively descending domain from one plateaued at a high KL, the scheduler spreads budget uniformly. •w/o gap (row b).Without the remaining gap, the scheduler uses only descent velocity. Early in training all domains descend rapidly, so the velocity signal cannot differentiate them, producing large oscillations before step 63. Later, IF is identified as the fastest-descending domain and heavily upweighted, but the lack of gap information causes abrupt ratio swings (e.g., Math drops to∼0.10 around step 100 before rebounding). This instability explains why the w/o gap variant peaks latest at step 255 (Table1). •w/o jitter (row c).The overall trajectory shape is similar to D 3 -MOPD, confirming that jitter does not alter the scheduler’s long-run allocation. However, the ratio curves show sharper step-like transitions between watcher updates, most visible in Code near step 200 and Tool-use near step 240. The corre- sponding accuracy drops on code benchmarks (Table 1) suggest that this reduced per-batch variation hurts domains near their r-KL plateau. •w/o smoothing (row d).A single observation window (R=1) makes the velocity estimate more reactive but noisier. The ratio curves oscillate visibly more than D 3 -MOPD (R=3), with frequent spikes in Math and Code around steps 60–130. The overall allocation pattern nonetheless remains close to D 3 -MOPD, which explains why this variant still achieves 62.17 (+0.81 in Table1). 18