Paper deep dive
MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation
Tianze Xu, Yanzhao Zheng, Zhentao Zhang, Yuanqiang Yu, Chao Ma, Jihuai Zhu, Lelun Wu, Lyumanshan Ye, Pengfei Liu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/29/2026, 11:53:21 AM
Summary
The paper introduces MOPD-Router, a framework for Multi-Teacher On-Policy Distillation (MOPD) that replaces prompt-level hard routing with token-level dynamic routing. It proposes three metrics: Entropy (teacher confidence), Novelty (teacher-student discrepancy), and ExpertAlign (alignment of teacher correction with teacher's post-training specialization). ExpertAlign achieves the best performance, improving over mean aggregation on unlabeled data and outperforming standard domain-label hard routing on labeled data without using labels.
Entities (11)
Relation Signals (7)
MOPD-Router ā uses ā Entropy
confidence 95% Ā· compare it against two reference metrics built on teacher confidence (Entropy)
MOPD-Router ā uses ā Novelty
confidence 95% Ā· teacher-student discrepancy (Novelty)
MOPD-Router ā uses ā ExpertAlign
confidence 95% Ā· Within this interface, we propose ExpertAlign... compare it against two reference metrics
MOPD-Router ā replaces ā domain-label hard routing
confidence 92% Ā· MOPD-Router replaces fixed prompt-to-teacher assignment with token-level weighting
ExpertAlign ā outperforms ā Standard MOPD
confidence 90% Ā· on domain-labeled data, it outperforms standard MOPD by 3.95 (+7.8%) points
ExpertAlign ā outperforms ā Mean aggregation
confidence 90% Ā· On unlabeled data, it improves the overall score by 5.88 (+12.3%) points over Mean aggregation
Qwen3-1.7B ā evaluatedwith ā MOPD-Router
confidence 85% Ā· Student: Qwen3-1.7B-Non-Thinking... MOPD Router w/ ExpertAlign
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-teacher on-policy distillation (MOPD) integrates specialized capabilities into a single student, but existing practice typically hard-routes each prompt to a domain-matched teacher for the entire rollout. This dependence on prompt-level domain labels restricts using unlabeled training mixtures and leaves complementary signals from other teachers unused. We introduce MOPD-Router, a framework that routes supervision over the full teacher pool at each token, without domain labels or training a separate routing model. Its plug-in interface supports different metrics for selecting and weighting teacher-specific OPD signals. Within this interface, we propose ExpertAlign, which scores each teacher by whether its correction to the student at the current token expresses the specialization that teacher acquired during post-training, and compare it against two reference metrics built on teacher confidence (Entropy) and teacher-student discrepancy (Novelty). Experiments on unlabeled and domain-labeled training mixtures under strong-to-weak and same-size distillation scenarios show that ExpertAlign achieves the strongest overall performance in all four settings. On unlabeled data, it improves the overall score by 5.88 (+12.3%) points over Mean aggregation; on domain-labeled data, it outperforms standard MOPD by 3.95 (+7.8%) points without using available domain labels. These results demonstrate token-level routing can exploit cross-domain complementary supervision, and reduce exclusive reliance on prompt-level domain assignment. Code is available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.30837v2
- Canonical: https://arxiv.org/abs/2609.30837v2
Trouble viewing inline? Open PDF directly ā
Full Text
74,917 characters extracted from source content.
Expand or collapse full text
MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation Tianze Xu Affiliation: Shanghai Jiao Tong University Affiliation: GAIR Yanzhao Zheng Affiliation: Alibaba Group Zhentao Zhang Yuanqiang Yu Affiliation: Alibaba Group Chao Ma Affiliation: Alibaba Group Jihuai Zhu Affiliation: Alibaba Group Lelun Wu Affiliation: Alibaba Group Affiliation: University of Science and Technology of China Lyumanshan Ye Affiliation: Shanghai Jiao Tong University Affiliation: GAIR Pengfei Liu ā thanks: Corresponding author Affiliation: Shanghai Jiao Tong University Affiliation: Shanghai Innovation Institute Affiliation: GAIR Baohua Dong Affiliation: Alibaba Group Hangcheng Zhu Affiliation: Alibaba Group Ruohui Huang Affiliation: Alibaba Group Gang Yu Affiliation: Alibaba Group Abstract Multi-teacher on-policy distillation (MOPD) integrates specialized capabilities into a single student, but existing practice typically hard-routes each prompt to a domain-matched teacher for the entire rollout. This dependence on prompt-level domain labels restricts using unlabeled training mixtures and leaves complementary signals from other teachers unused. We introduce MOPD-Router, a framework that routes supervision over the full teacher pool at each token, without domain labels or training a separate routing model. Its plug-in interface supports different metrics for selecting and weighting teacher-specific OPD signals. Within this interface, we propose ExpertAlign, which scores each teacher by whether its correction to the student at the current token expresses the specialization that teacher acquired during post-training, and compare it against two reference metrics built on teacher confidence (Entropy) and teacher--student discrepancy (Novelty). Experiments on unlabeled and domain-labeled training mixtures under strong-to-weak and same-size distillation scenarios show that ExpertAlign achieves the strongest overall performance in all four settings. On unlabeled data, it improves the overall score by 5.88 (+12.3%) points over Mean aggregation; on domain-labeled data, it outperforms standard MOPD by 3.95 (+7.8%) points without using available domain labels. These results demonstrate token-level routing can exploit cross-domain complementary supervision, and reduce exclusive reliance on prompt-level domain assignment.11 1 Code is available at: https://github.com/TURLEing/MOPD-Router. Figure 1: Overview of MOPD-Router. (a) Unlike prompt-level hard routing, token-level routing dynamically combines supervision from the full teacher pool. (b) Teachers evaluate the studentās on-policy rollout, while a plug-in metric weights their token-level OPD signals. (c) Three routing metrics: teacher confidence (Entropy), teacherāstudent difference (Novelty), and specializationādirection alignment (ExpertAlign). 1 Introduction Reinforcement learning (RL) has emerged as an important approach to improving capabilities of large language model (LLM) (Ouyang et al., 2022; Schulman et al., 2017; Shao et al., 2024). Different task domains often use distinct RL pipelines: mathematical reasoning can exploit verifiable rewards, software engineering can train in executable environments, and instruction following can use rule- or rubric-based feedback (Ma et al., 2026). These independent RL pipelines, often starting from a shared base model, produce specialist models with distinct capabilities. Yet the ultimate goal is a single model that performs well across these domains, making the integration of multiple RL-specialized capabilities central to LLM post-training (Ma et al., 2026; Gao et al., 2026). Nowadays, Multi-teacher On-Policy Distillation (MOPD) is an appealing paradigm for integrating specialized capabilities into one policy model (Ma et al., 2026; Gao et al., 2026). MOPD builds on On-Policy Distillation (OPD), where the student generates trajectories under its current policy and teachers provide dense token-level supervision on them (Agarwal et al., 2024; Yang et al., 2026a; Li et al., 2026). Adopting MOPD for capability integration is now a consensus among leading LLM vendors (Xiao et al., 2026; GLM-5 Team, 2026; Xu et al., 2026a; Kimi Team, 2026). However, existing MOPD protocol typically uses a coarse-grained teacher assignment: it selects the single teacher for each prompt corresponding to the domain label, then the teacher supervises the entire rollout through OPD (Ma et al., 2026; Gao et al., 2026). We refer to this one-teacher-per-prompt protocol as domain-label hard routing. It can identify the primary domain expert and provide stable MOPD signals, but relies on prompt-level label for teacher assignment throughout the trajectory. This protocol has two limitations. First, real-world mixed datasets of SFT and chat often lack reliable domain labels. A prompt may combine general conversation, instruction following, mathematical reasoning, and code execution, leaving standard MOPD no predefined rule for assigning one domain teacher. Second, a domain label identifies the primary teacher but ignores the potential value of other teachers at individual tokens. Taking mathematics prompts as an example, a code teacher may help with programmatic reasoning, while an instruction-following teacher may help satisfy format or output constraints. The usefulness of these signals can also vary across generation stages and token positions. Routing the entire rollout to a fixed domain teacher therefore leaves potential cross-domain complementary supervision unused. However, exploiting this potential cross-domain complementarity requires token- and context-aware routing. This raises a key question: how can a routing metric identify and select relevant teachers and assign appropriate weights to their signals at each token? Thus, we introduce MOPD-Router, a token-level teacher-routing framework for MOPD that works without domain labels or training a separate routing model, providing an interface for studying routing metrics within a same MOPD pipeline.We instantiate three metrics from different information sources. Entropy routes by a teacherās next-token entropy, and Novelty by the accessible teacherāstudent distributional difference. Beyond that, we introduce ExpertAlign, which routes each token by the alignment between the teacherās acquired specialization and its teaching supervision. Figure 1 gives an overview of the framework and metrics. We evaluate MOPD-Router on two training datasets under both strong-to-weak and same-size OPD scenarios. On the unlabeled mixed dataset, it enables multi-teacher OPD without predefined domain assignment, with effective routing metrics outperforming naive mean aggregation. On the domain-labeled MOPD dataset, token-level metric routing matches or exceeds domain-label hard without using available labels, suggesting that effective routing need not rely exclusively on prompt-level domain labels. Among these routing metrics, our proposed ExpertAlign achieves the strongest overall performance across the evaluated data and model-scale settings. Our contributions are as follows: ā¢ We introduce MOPD-Router, a plug-in framework that selects and weights supervision signals from multi-teacher pool at every token, without using domain labels or training a separate routing model. ā¢ We propose ExpertAlign, a specialization-aware routing metric that aligns a teacherās acquired expertise direction with the teaching direction it would apply to the current student. Across unlabeled and domain-labeled data under two distillation scenarios, ExpertAlign achieves the best overall performance in all four settings. ā¢ Our analyses show that cross-domain teacher signals provide complementary supervision across capabilities. We open-source the MOPD-Router framework to support further research on token-level teacher routing. 2 Related Work 2.1 On-Policy Distillation On-policy distillation (OPD) combines the distributional consistency of learning from student-generated trajectories with dense token-level teacher supervision (Agarwal et al., 2024). Recent OPD research has three directions (Yu et al., 2026): (a) Strong-to-weak distillation transfers capabilities from a stronger teacher to a weaker student and develops more stable or effective variants of the standard OPD objective (Agarwal et al., 2024; Yang et al., 2026a). (b) Self-distillation uses the same model as teacher and student under different context or feedback conditions (HĆ¼botter et al., 2026; Zhao et al., 2026). (c) Adaptive distillation addresses non-uniform OPD supervision by modulating the training objective or supervised tokens using different signals (Jin et al., 2026; Wang et al., 2026b; Xu et al., 2026b; Li et al., 2026; Heo et al., 2026). These analyses and objectives inspire our routing designs, but primarily study modifying supervision within single-teacher OPD rather than allocating it among multiple specialized teachers. 2.2 Multi-Teacher Capability Integration Prior approaches integrate capabilities through joint training, sequential training, or weight-space composition (Ma et al., 2026). Mix-RL pools prompts from multiple domains and jointly optimizes their domain-specific rewards (Yang et al., 2025). Cascade RL applies domain-specific RL stages sequentially to a single model (Wang et al., 2026a). Parameter merging instead combines independently specialized checkpoints directly in weight space (Ilharco et al., 2023). Multi-teacher OPD (MOPD) integrates independently trained domain teachers in policy space through dense supervision on student-generated trajectories (Ma et al., 2026). It has been adopted in several frontier-model post-training pipelines, including MiMo-V2-Flash, GLM-5, DeepSeek-V4, Kimi K3, and Nemotron-Cascade 2 (Xiao et al., 2026; GLM-5 Team, 2026; Xu et al., 2026a; Kimi Team, 2026; Yang et al., 2026b). Open-MOPD and D3-MOPD further increase the fraction of domain-teacher gains recovered by the student through optimization-budget balancing and adaptive domain scheduling (Gao et al., 2026; Sun et al., 2026). H-OPD introduces confidence-based token-level arbitration between vision-language and text-only teachers for multimodal reasoning (Yin et al., 2026). However, most existing MOPD implementations still rely on prompt-level hard routing based on domain labels, leaving label-free routing metric design underexplored. 3 MOPD-Router MOPD-Router replaces fixed prompt-to-teacher assignment with token-level weighting over the full teacher set. We formalize sampled-token MOPD as routing over per-teacher OPD advantages, then instantiate three independent metrics drawing on the teacher distribution, the teacherāstudent relationship, and teacher specialization relative to the shared pre-RL base model. 3.1 Multi-Teacher On-Policy Distillation Let xā¼x be a prompt, ĻĪø _Īø the student policy, and ĻĻii=1M\ _ _i\_i=1^M the expert teachers. The rollout policy ĻĪøold _ _old samples a response y=(y1,ā¦,yT)ā¼ĻĪøold(ā ā£x)y=(y_1,ā¦,y_T) _ _old(Ā· x). At position t, a teacher evaluates the student-visited context ht=(x,y<t)h_t=(x,y_<t). Following sampled-token OPD implementations (Li et al., 2026), we define per-teacher and routed advantages as At(i)=sgā”[logā”ĻĻiā(ytā£ht)ālogā”ĻĪøoldā(ytā£ht)],At=āi=1Mwi,tāAt(i).A^(i)_t=sg\! [ _ _i(y_t h_t)- _ _old(y_t h_t) ], A_t= _i=1^Mw_i,tA^(i)_t. (1) Here sgā”[ā ]sg[Ā·] denotes stop-gradient and AtA_t aggregates the per-teacher sampled-token signals At(i)A^(i)_t with non-negative weights wi,ti=1M\w_i,t\_i=1^M. As in RL-style OPD implementations, AtA_t directly occupies the advantage slot without a learned critic (Heo et al., 2026; Gao et al., 2026). Given the importance ratio Ļtā(Īø)=ĻĪøā(ytā£ht)/ĻĪøoldā(ytā£ht) _t(Īø)= _Īø(y_t h_t)/ _ _old(y_t h_t), the student maximizes (Schulman et al., 2017) MOPDā(Īø)=ā”[1Tāāt=1Tminā”(Ļtā(Īø)āAt,clipā”(Ļtā(Īø),1āĻµ,1+Ļµ)āAt)].J_MOPD(Īø)=E\! [ 1T _t=1^T \! ( _t(Īø)A_t,clip( _t(Īø),1-Īµ,1+Īµ)A_t ) ]. (2) Standard MOPD assigns each prompt a domain label dā”(x)d(x) and uses the corresponding teacher throughout the response (Ma et al., 2026; Gao et al., 2026). This domain-label hard routing uses wi,tdomain=[i=d(x)],w^domain_i,t=I[i=d(x)], (3) reusing the same one-hot weights at every position. When no domain assignments are available, a direct alternative is Mean aggregation, with wi,tmean=1/Mw^mean_i,t=1/M. Domain-label hard routing is reliable but coarse, whereas Mean aggregation is label-free but treats all teacher signals as equally informative. 3.2 Token-Level Metric Routing MOPD-Router replaces these fixed aggregation rules by evaluating a plug-in metric ā³M for every teacher at each token position of the student rollout: si,t=ā³ā”(ĻĻi,ĻĪøold,ĻĪøbase,ht),s_i,t=M\! ( _ _i, _ _old, _ _base;h_t ), (4) where ĻĪøbase _ _base denotes the teachersā shared base model before post-training. At each token, metric routing uses scores to identify suitable teachers and assign their relative contributions to AtA_t, yielding context-dependent selection and weighting over the full teacher pool without domain labels or training a separate routing model. Algorithm 1 summarizes the training loop. 3.3 Routing Metrics Let ptSā(v)=ĻĪøoldā(vā£ht)p^S_t(v)= _ _old(v h_t) denote the rollout studentās next-token distribution, ptiā(v)=ĻĻiā(vā£ht)p^i_t(v)= _ _i(v h_t) teacher iās distribution, and ptbaseā(v)=ĻĪøbaseā(vā£ht)p^base_t(v)= _ _base(v h_t) the shared base modelās distribution. We study three routing metrics for estimating token-level teacher relevance. Because the Entropy and Novelty scores occupy narrow metric-specific scales, their softmax weights use sub-unit routing temperatures to avoid near-uniform weights and sharpen distinctions among teachers. Entropy. A natural but naive hypothesis is that a more confident teacher should receive greater routing weight. Entropy measures how diffuse a teacherās next-token distribution is: lower entropy concentrates probability mass on fewer candidate tokens and therefore indicates a sharper, more confident prediction. Entropy, representing the confidence-arbitration method (Yin et al., 2026; Prabhudesai et al., 2025), uses negative predictive entropy as the routing score si,tEnt=āHā”(pti)=āvāptiā(v)ālogā”ptiā(v),wi,tEnt=expā”(si,tEnt/ĻREnt)āj=1Mexpā”(sj,tEnt/ĻREnt),s^Ent_i,t=-H(p^i_t)= _v p^i_t(v) p^i_t(v), w^Ent_i,t= (s^Ent_i,t/ _R^Ent) _j=1^M (s^Ent_j,t/ _R^Ent), (5) where ĻREnt=0.1 _R^Ent=0.1 is the routing Softmax temperature. This teacher-only metric is computationally straightforward, but cannot determine whether a lower-entropy teacher usefully corrects the current student, and intrinsic entropy-scale differences can systematically favor particular teachers. Novelty. Prior work indicates that effective OPD updates concentrate on high-probability candidates shared by teacher and student, whereas corrections outside the studentās local support can be difficult to absorb (Li et al., 2026; Wang et al., 2026b). Novelty therefore assigns a high score when teacher differs substantially from the student within their shared high-probability candidate set and both assign it sufficient probability mass Let StS=TopKā”(ptS,k)S^S_t=TopK(p^S_t,k) and Sti=TopKā”(pti,k)S^i_t=TopK(p^i_t,k) denote their sets of k most probable next-token candidates, and let Ī©i,t=StSā©Sti _i,t=S^S_tā© S^i_t be their shared candidate set. For a distribution p, we define pā”(Ī©)=āvāĪ©pā”(v)p( )= _vā p(v) and p|Ī©p|_ for its restriction to Ī© , renormalized to unit mass. We also define the accessibility and novelty components as ai,t=ptSā(Ī©i,t)āptiā(Ī©i,t),Ni,t=1āexp(āJS(ptS|Ī©i,t,pti|Ī©i,t)/ĻN).a_i,t= p^S_t( _i,t)p^i_t( _i,t), N_i,t=1- \! (-JS\! (p^S_t|_ _i,t,p^i_t|_ _i,t )/ _N ). (6) The accessibility term ai,ta_i,t measures both modelsā probability mass on shared candidate tokens, while the novelty term Ni,tN_i,t measures their distributional discrepancy within the shared candidate set. The temperature ĻN=0.1 _N=0.1 controls how quickly this discrepancy saturates. We combine both components and instantiate Novelty routing as si,tNov=ai,tāNi,t,wi,tNov=expā”(si,tNov/ĻRNov)āj=1Mexpā”(sj,tNov/ĻRNov).s^Nov_i,t=a_i,tN_i,t, w^Nov_i,t= (s^Nov_i,t/ _R^Nov) _j=1^M (s^Nov_j,t/ _R^Nov). (7) We use ĻRNov=0.1 _R^Nov=0.1 and set si,tNov=0s^Nov_i,t=0 when Ī©i,t _i,t is empty. ExpertAlign. Entropy and Novelty score a teacherās current output, but do not indicate whether it reflects the teacherās expertise. Prior work (Heo et al., 2026; Feng et al., 2026) shows that the log-policy difference between a teacher and its base model captures the policy shift induced by post-training. We therefore propose ExpertAlign, a specialization-aware token-level routing metric that assesses whether a teacher would move the current student in the direction of its post-training expertise. Building on this observation, we first use the base-relative shift to define teacher iās expertise vector on the studentās top-k support StSS^S_t: i,t=[logā”ptiā(v)ālogā”ptbaseā(v)]vāStS,e_i,t= [ p^i_t(v)- p^base_t(v) ]_vā S^S_t, (8) which represents the direction acquired by teacher i relative to the shared pre-RL base and is an intrinsic property of that teacher. Against it, we further introduce the teacherāstudent teaching vector, a student-dependent direction defined as i,t=[logā”ptiā(v)ālogā”ptSā(v)]vāStS,d_i,t= [ p^i_t(v)- p^S_t(v) ]_vā S^S_t, (9) to capture how teacher i would reshape the studentās log probabilities over its local top-k support. ExpertAlign considers teacher i relevant only when its teaching direction i,td_i,t aligns with the specialization i,te_i,t, indicating that the correction reflects capabilities acquired during teacher post-training rather than generic disagreement with the student. We thus quantify this specializationāteaching alignment using positive cosine similarity: ci,t=maxā”(āØi,t,i,tā©ā„i,tā„2āā„i,tā„2,0),ā°t=i:āØi,t,i,tā©>Ī“,c_i,t= \! ( _i,t,d_i,t _i,t _2 _i,t _2,0 ), _t=\i: _i,t,d_i,t >Ī“\, (10) where a zero-norm vector receives ci,t=0c_i,t=0, and the numerical alignment margin is Ī“=10ā6Ī“=10^-6. ExpertAlign only retains teachers whose teaching signal positively aligns with their acquired specialization. We consider two weighting rules over the retained set: wi,tuniform=[iāā°t]|ā°t|,wi,tcosine=[iāā°t]ci,tājāā°tcj,t.w^uniform_i,t= I[i _t]|E_t|, w^cosine_i,t= I[i _t]c_i,t _j _tc_j,t. (11) If ā°tE_t is empty, all weights are zero and OPD supervision at position t is skipped. We select k=16k=16 and report its sensitivity in Appendix E.4. Appendix E.5 further reports the retained-teacher distributions and their evolution across all four settings. Algorithm 1 MOPD-Router training loop Require: Prompt sampler D, student ĻĪø _Īø, teachers ĻĻii=1M\ _ _i\_i=1^M, shared teacher base ĻĪøbase _ _base, routing metric ā³M, inner-step count K Ensure: Updated student ĻĪø _Īø 1: for each training step q do 2: Set ĻĪøoldāĻĪø _ _oldā _Īø and sample a rollout batch āqR_q from D and ĻĪøold _ _old. 3: Evaluate the student, all teachers, and the shared base on the student-visited contexts to obtain the quantities required by ā³M. 4: Compute each teacherās sampled-token advantage At(i)A_t^(i) using Equation 1. 5: Apply ā³M to obtain non-negative token-level teacher weights wi,ti=1M\w_i,t\_i=1^M over the full teacher pool. 6: Aggregate the routed advantage At=āi=1Mwi,tāAt(i)A_t= _i=1^Mw_i,tA_t^(i). 7: for each inner minibatch ā¬q,kB_q,k, k=0,ā¦,Kā1k=0,ā¦,K-1 do 8: Recompute current-student log-probabilities and the importance ratios Ļtā(Īø) _t(Īø). 9: Update Īø by maximizing the clipped MOPD objective in Equation 2. 10: end for 11: end for 4 Experiments We evaluate MOPD-Router across two training-data regimes and two studentāteacher distillation scenarios. We first test metric-based routing on a mixed dataset without teacher-assignment labels, then compare it with standard domain-label hard routing on a conventional labeled MOPD dataset. We further analyze teacher selection, metric design, and cross-domain supervision. 4.1 Experimental Setup Training data. Our first training set is a 60K unlabeled mixture of 30K general chatting prompts sampled from Nemotron-v3 Dataset (Blakeman et al., 2026) and 30K mathematics prompts from NuminaMath-TIR (Li and others, 2024) requiring executable code for problem solving. It lacks teacher-assignment labels and thus cannot directly use domain-label hard routing. Our second training set is a domain-labeled post-training dataset comprising 25K mathematics prompts from DeepMath-103K (He et al., 2025), 25K code prompts from Eurus-RL-Code (Cui et al., 2025), and 16K instruction-following prompts from HiR-16K (Zhang et al., 2025). It represents the conventional MOPD setting, where each prompt is assigned to its corresponding teacher; MOPD-Router ignores these labels and routes over all teachers using only its token-level metric. Table 1: Results (%) on the unlabeled mixed training set. Bold denotes the best distilled result within each student block; shaded rows identify MOPD-Router methods. The final row reports the same-size ExpertAlign result minus Standard MOPD. Math Code IF Overall Method AIME24 AIME25 HMMT Feb. HMMT Nov. Human Eval+ MBPP+ LCB-v6 IFEval IFBench Avg. Baseline Models Qwen3-1.7B 12.50 11.67 5.83 4.58 60.40 52.40 12.00 67.65 16.67 27.08 Qwen3-4B 23.33 10.00 10.00 7.08 82.30 67.20 18.29 81.88 24.14 36.02 RL Teacher 58.33 52.08 28.75 35.00 84.14 72.20 24.57 82.80 31.29 52.13 Student: Qwen3-1.7B-Non-Thinking Standard MOPD 32.08 25.41 16.67 15.41 70.12 57.41 19.42 69.32 21.09 36.33 Open-MOPD 24.58 25.42 15.00 12.08 71.95 56.35 13.14 66.91 20.75 34.02 Mean Aggregation 27.92 26.25 15.42 11.67 70.73 53.17 16.57 70.98 17.68 34.49 MOPD Router w/ Entropy 30.00 26.67 15.42 13.33 68.90 59.52 15.43 64.51 18.03 34.65 w/ Novelty 32.92 27.92 17.08 13.75 70.73 57.67 20.00 71.90 18.37 36.70 w/ ExpertAlign 38.33 32.08 16.67 17.08 72.56 58.99 19.43 70.61 21.43 38.58 Student: Qwen3-4B-Non-Thinking Standard MOPD 50.00 45.83 25.41 33.75 84.15 69.58 28.00 77.26 24.83 48.76 Open-MOPD 51.25 45.41 22.92 32.92 86.59 71.96 24.57 81.80 27.27 49.41 Mean Aggregation 51.25 45.83 22.50 29.58 82.93 69.84 26.28 80.22 22.45 47.88 MOPD Router w/ Entropy 51.67 48.33 22.50 31.25 81.71 71.96 26.86 78.19 20.07 48.06 w/ Novelty 56.25 44.17 26.66 35.83 84.76 70.63 29.71 79.85 24.83 50.30 w/ ExpertAlign 59.17 52.08 33.75 40.42 87.20 73.02 28.57 82.07 27.55 53.76 Ī vs Standard MOPD +9.17 +6.25 +8.34 +6.67 +3.05 +3.44 +0.57 +4.81 +2.72 +5.00 Models and teachers. We use the non-thinking variants of Qwen3 series (Yang et al., 2025). The teacher pool comprises three Qwen3-4B-Non-Thinking models obtained through domain-specific RL post-training. The mathematics and code teachers are the publicly released RL checkpoints from ExOPD (Yang et al., 2026a); we train the instruction-following teacher on HIR-16K (Zhang et al., 2025), detailed in Appendix B. All teachers share Qwen3-4B-Non-Thinking as their pre-RL base model. We consider two distillation settings: strong-to-weak, with a Qwen3-1.7B student and Qwen3-4B teachers, and same-size, with a Qwen3-4B student and Qwen3-4B teachers. Baselines. In both distillation scenarios, we compare against three baselines: Standard MOPD, Open-MOPD (Gao et al., 2026), and Mean Aggregation, which uniformly averages the three sampled-token teacher signals. On the unlabeled mixture, DeepSeek-V4-Flash (Xu et al., 2026a) labels every SFT example as math, code, or instruction following from its prompt and reference response. Standard MOPD uses these labels for teacher assignment, while Open-MOPD uses the same assignments and balances the optimization budget across domains. Appendix C reports the prompt and label distribution. On the domain-labeled dataset, Standard MOPD and Open-MOPD use the provided domain labels. We compare these baselines with Entropy, Novelty, and ExpertAlign with cosine weighting. We also report the initial student and a Teacher reference evaluating each teacher only on its respective domain. Training and evaluation. Within each setting, all routing methods use the same sampled-token MOPD objective, teacher pool, rollout configuration, and optimization budget. We evaluate math on AIME 2024 (AI-MO, 2024), AIME 2025 (OpenCompass, 2025), HMMT 2025 Feb (BalunoviÄ et al., 2026), and HMMT 2025 Nov (BalunoviÄ et al., 2026); code on HumanEval+, MBPP+ (Liu et al., 2023) and LiveCodeBench v6 (Jain et al., 2025); and instruction following on IFEval (Zhou et al., 2023) and IFBench (Pyatkin et al., 2025). We sample 8 responses per problem and report Avg@8 accuracy. Code tasks use Pass@1; instruction-following tasks use five independent completions per instance and report prompt-level accuracy. Overall Avg. macro-averages the nine benchmark metrics. Appendix B lists all training and evaluation hyperparameters. 4.2 Main Results Table 2: Results (%) on the domain-labeled MOPD training set. Formatting follows Table 1. Math Code IF Overall Method AIME24 AIME25 HMMT Feb. HMMT Nov. Human Eval+ MBPP+ LCB-v6 IFEval IFBench Avg. Baseline Models Qwen3-1.7B 12.50 11.67 5.83 4.58 60.40 52.40 12.00 67.65 16.67 27.08 Qwen3-4B 23.33 10.00 10.00 7.08 82.30 67.20 18.29 81.88 24.14 36.02 RL Teacher 58.33 52.08 28.75 35.00 84.14 72.20 24.57 82.80 31.29 52.13 Student: Qwen3-1.7B-Non-Thinking Standard MOPD 33.33 34.17 15.83 16.67 69.51 61.11 19.43 68.21 19.39 37.52 Open-MOPD 35.42 35.00 17.91 17.50 75.00 58.73 21.71 70.43 20.06 39.08 Mean Aggregation 30.00 27.92 16.66 12.08 75.00 55.82 20.57 68.02 15.98 35.78 MOPD Router w/ Entropy 34.17 29.58 17.08 17.92 73.78 57.14 21.71 68.39 18.03 37.53 w/ Novelty 34.58 28.33 17.50 15.83 72.56 57.41 20.57 70.24 20.41 37.49 w/ ExpertAlign 36.25 32.50 18.33 20.83 75.61 61.64 23.43 71.35 21.77 40.19 Student: Qwen3-4B-Non-Thinking Standard MOPD 55.42 51.67 30.83 33.75 84.76 70.90 25.71 77.45 25.17 50.63 Open-MOPD 59.83 53.33 32.08 37.08 86.59 70.11 24.00 80.78 27.55 52.37 Mean Aggregation 51.67 42.92 27.08 32.08 81.71 68.78 26.28 81.52 21.43 48.16 MOPD Router w/ Entropy 58.75 46.25 29.17 34.58 85.24 70.63 25.14 80.59 23.81 50.46 w/ Novelty 59.78 52.08 30.00 36.67 87.05 72.49 29.14 79.85 25.17 52.47 w/ ExpertAlign 60.42 53.75 32.92 40.83 87.20 74.60 30.86 81.70 28.91 54.58 Ī vs Standard MOPD +5.00 +2.08 +2.09 +7.08 +2.44 +3.70 +5.15 +4.25 +3.74 +3.95 MOPD on unlabeled data. Table 1 reports results on the unlabeled mixture. Mean reaches overall scores of 34.49 and 47.88 in the strong-to-weak and same-size distillation scenarios, respectively. Entropy yields similar overall scores to Mean. With LLM-generated domain labels, Standard MOPD reaches 36.33 and 48.76, while Open-MOPD reaches 34.02 and 49.41. MOPD-Router operates directly on the original unlabeled dataset. Novelty outperforms both label-based baselines, while ExpertAlign achieves the best overall performance, reaching 38.58 for Qwen3-1.7B student and 53.76 for Qwen3-4B student. MOPD on domain-labeled data. Table 2 compares token-level metric routing with Standard MOPD and Open-MOPD, which use the domain labels to route each training prompt to its designated teacher. Without these labels, ExpertAlign reaches 40.19 and 54.58 in the strong-to-weak and same-size settings, exceeding Standard MOPD and Open-MOPD. MOPD-Router with Novelty also matches Standard MOPD in strong-to-weak distillation and improves in same-size distillation. These results show that effective teacher routing can exploit token-level distributional information without relying exclusively on prompt-level domain labels. Paired bootstrap tests across all four settings, together with three-seed repeats in the same-size domain-labeled setting, show that ExpertAlign significantly outperforms Mean Aggregation and Standard MOPD, and outperforms or matches Open-MOPD. Appendix D reports both analyses. 4.3 Ablation Study and Analysis 4.3.1 Teacher confidence is not a reliable routing signal. Entropy routing implements an intuitive confidence heuristic: a low-entropy teacher expresses a sharper local preference and receives priority in supervising the student. However, our results show that confidence alone does not reliably identify useful supervision, as high-entropy positions can represent decision points with multiple plausible continuations and still carry important OPD and RL learning signals (Jin et al., 2026; Wang et al., 2025a). Moreover, entropy also has a strong teacher-specific bias: across both distillation settings, the math teacher consistently exhibits substantially lower entropy than the other teachers, making router favor one teacher, instead of adapting to the studentās current context. Appendix E.1 shows that removing this bias through batch-wise entropy normalization yields only marginal improvement. This motivates student-conditioned metrics such as Novelty and ExpertAlign. 4.3.2 Accessibility matters when studentāteacher support mismatch. Novelty combines teacherāstudent discrepancy with accessibility on their shared high-probability support. To examine accessibility across distillation scenarios, we ablate full Novelty, ai,tāNi,ta_i,tN_i,t, against NoveltyOnly, which removes ai,ta_i,t and routes solely by Ni,tN_i,t. Appendix E.2 shows accessibility improves Overall by 7.5% in strong-to-weak distillation, from 34.12 to 36.70, while the ordering reverses in same-size distillation, where NoveltyOnly exceeds full Novelty by 0.55 points. Per-teacher routing statistics further show lower shared top-k support overlap for the 1.7B student than for the 4B student. These results suggest that accessibility helps when student and teacher distributions have mismatched local token support by discounting discrepancies the student is less positioned to absorb. When their supports closely align, accessibility provides less useful discrimination and can attenuate otherwise informative discrepancies. 4.3.3 Separating teacher selection from teacher weighting. ExpertAlign consists of two components: which teachers should contribute at each token, and how the retained teachers should be weighted. To disentangle their contributions, we conduct ablation experiments on it: Uniform applies the positive-alignment gate but averages retained teachers equally, whereas Cosine uses the same set and additionally weights teachers by alignment strength. Table 3: ExpertAlign ablation with per-domain averages and the Overall score (%). Math, Code, and IF are averaged over the corresponding benchmarks in Tables 1 and 2. Uniform isolates positive-alignment teacher selection, while Cosine additionally uses the alignment value for weighting. Student: Qwen3-1.7B Student: Qwen3-4B Routing metric Math Code IF Overall Math Code IF Overall Unlabeled Training Set Mean Aggregation 20.31 46.83 44.33 34.49 37.29 59.68 51.34 47.88 ExpertAlign-Uniform 23.12 47.77 42.81 35.71 41.46 61.78 51.31 50.42 ExpertAlign-Cosine 26.04 50.33 46.02 38.58 46.35 62.93 54.81 53.76 Labeled Training Set Mean Aggregation 21.66 50.46 42.00 35.78 38.44 58.93 51.47 48.16 ExpertAlign-Uniform 24.79 52.11 45.83 38.57 46.35 61.31 53.44 52.91 ExpertAlign-Cosine 26.98 53.56 46.56 40.19 46.98 64.22 55.31 54.58 Table 3 shows that Uniform consistently improves the Overall score over Mean Aggregation by 1.22ā4.75 points across all settings. Gains concentrate in mathematics and code, while instruction-following performance is slightly lower on unlabeled data. It indicates that the positive-alignment gate selects a more effective subset of teacher signals overall, though its benefit varies by domain. Cosine weighting further improves every domain average across all four settings, raising the Overall score by an additional 1.62ā3.34 points over Uniform. These consistent gains show that alignment magnitude provides useful routing information beyond binary teacher selection. Table 4: Cross-domain teacher ablation on the labeled training set (%). Domain scores average the corresponding benchmarks in Table 2. w/o Token-Level Routing assigns fixed teacher weights per response; w/o Cross-Domain Teachers restricts routing to the teacher indicated by the domain label. Student: Qwen3-1.7B Student: Qwen3-4B Method Math Code IF Overall Math Code IF Overall MOPD-Router w/ ExpertAlign 26.98 53.56 46.56 40.19 46.98 64.22 55.31 54.58 w/o Token-Level Routing 22.50 50.25 44.18 36.57 44.48 62.37 54.24 52.61 w/o Cross-Domain Teachers 26.04 51.84 40.20 37.79 45.83 60.50 50.80 51.83 Standard MOPD 25.00 50.02 43.80 37.52 42.92 60.46 51.31 50.63 4.3.4 Cross-domain teachers provide complementary supervision. To test whether off-domain teachers provide useful supervision, we compare ExpertAlign with two controlled variants on the labeled training set. The domain-restricted variant retains only the teacher indicated by the domain label, while the response-level variant averages each teacherās token-level ExpertAlign score over all valid tokens in the response, normalizes the resulting teacher scores, and applies the fixed mixture to every token. As shown in Table 4, restoring the full teacher pool improves Overall by 2.40 and 2.75 points for the 1.7B and 4B students, respectively. With the full pool held constant, token-level routing further improves Overall over rollout-level weighting by 3.62 and 1.97 points. These results show that cross-domain teachers provide complementary supervision and that their value is best captured through token-level allocation. Figure 2 characterizes how MOPD-Router uses these signals. All the off-domain teachers receive more than 50% of conditional routing mass on math, code, and instruction-following prompts, and in the case study allocation tracks generated content: the math teacher peaks during numerical reasoning, the code teacher activates during Python verification, and the instruction-following teacher gains more weight around output structuring and JSON formatting. In the same-size setting, the resulting student surpasses the corresponding RL teachers on math and code under both training sets, demonstrating the effectiveness of cross-domain complementary supervision. Appendix E.3 provides the full prompt and a normalized word-span view of this example. Figure 2: Cross-domain routing behavior of ExpertAlign. Left: routing mass conditioned on prompt domain and normalized within each row. Right: routing coefficients on their original scale for a Qwen3-1.7B student rollout. Semantic stage boundaries are manually annotated for exposition. 4.4 Training Cost Analysis Beyond the three domain teachers, ExpertAlign also evaluates their shared pre-RL base model and computes top-k distribution statistics for routing, whereas Entropy and Novelty route over the same three-teacher pool as Standard MOPD. To quantify the resulting end-to-end overhead, we compare all three routing metrics against Standard MOPD in the same-size, domain-labeled setting. All measurements use an identical environment with eight NVIDIA H20 GPUs. Because routing methods yield different response lengths, Table 5 reports both measured costs under this common step budget and token-normalized end-to-end costs. Table 5: End-to-end training cost in the same-size, domain-labeled setting. Method Avg. resp. length GPU-hours E2E ms/token Rel. E2E cost Standard MOPD 4,893 692.9 0.32184 1.00Ć1.00Ć MOPD-Router w/ Entropy 3,278 524.2 0.36344 1.13Ć1.13Ć w/ Novelty 3,496 539.6 0.35079 1.09Ć1.09Ć w/ ExpertAlign 4,278 716.9 0.38086 1.18Ć1.18Ć Under the common step budget, shorter responses reduce Entropy and Novelty to 524.2 and 539.6 GPU-hours versus Standard MOPDās 692.9. Their token-normalized costs remain close at 1.13Ć1.13Ć and 1.09Ć1.09Ć, indicating modest routing overhead. ExpertAlign requires 716.9 GPU-hours, a 3.5% measured increase, and 1.18Ć1.18Ć token-normalized cost. Thus, its shared-base forward pass and top-k routing computation add only modest end-to-end overhead. Measurements exclude initialization, model downloads, validation, and checkpoint I/O outside the per-step timer. 5 Conclusion We introduce MOPD-Router, a framework replacing prompt-level domain-label hard routing in multi-teacher OPD with token-level routing over the full teacher pool through plug-in metrics, without training a routing model. MOPD-Router enables multi-teacher distillation on unlabeled prompt mixtures without a separate teacher-assignment stage. Across unlabeled and domain-labeled mixtures under strong-to-weak and same-size distillation scenarios, our experiments show that effective routing metrics enable MOPD without domain labels and outperform both Mean aggregation and standard MOPD, with ExpertAlign consistently performing best overall. These findings demonstrate the value of dynamically selecting and combining complementary teacher signals during multi-teacher distillation. We hope MOPD-Router offers a new perspective on capability integration and encourages further exploration of token-level, context-dependent supervision. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, Cited by: Ā§1, Ā§2.1. AI-MO (2024) AI-MO AIME 2024. Note: https://huggingface.co/datasets/AI-MO/aimo-validation-aimeHugging Face dataset Cited by: Ā§4.1. BalunoviÄ et al. (2026) M. BalunoviÄ, J. Dekoninck, I. Petrov, N. JovanoviÄ, and M. Vechev MathArena: evaluating llms on uncontaminated math competitions. External Links: 2505.23281, Link Cited by: Ā§4.1. Blakeman et al. (2026) A. Blakeman, A. Thomas, A. Jhunjhunwala, A. Gupta, A. Khattar, A. Rajfer, A. Renduchintala, A. Asif, A. Vavre, A. F. Miranda, et al. Nemotron 3 ultra: open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning. Cited by: Ā§4.1. Cui et al. (2025) G. Cui, L. Yuan, Z. Wang, H. Wang, Y. Zhang, J. Chen, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. External Links: 2502.01456 Cited by: Ā§4.1. Feng et al. (2026) S. Feng, H. Gao, H. Chi, H. Wu, Z. Zhang, Z. Jiang, B. He, W. Ma, Y. Zhang, and H. Zhou Weak-to-strong generalization via direct on-policy distillation. External Links: 2607.05394, Link Cited by: Ā§3.3. Gao et al. (2026) H. Gao, H. Chi, Y. Yan, S. Feng, H. Wu, Z. Jiang, B. He, W. Ma, Y. Zhang, and H. Zhou Open-MOPD: diagnosing and fixing capability imbalance in multi-teacher on-policy distillation. arXiv preprint arXiv:2608.19098. External Links: 2608.19098 Cited by: Ā§1, Ā§1, Ā§2.2, Ā§3.1, Ā§3.1, Ā§4.1. GLM-5 Team (2026) GLM-5 Team GLM-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763. External Links: 2602.15763 Cited by: Appendix A, Ā§1, Ā§2.2. He et al. (2025) Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, Z. Zhang, R. Wang, Z. Tu, H. Mi, and D. Yu DeepMath-103K: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456. External Links: 2504.11456 Cited by: Ā§4.1. Heo et al. (2026) B. Heo, J. Hwang, S. Yun, and D. Han On-policy delta distillation. arXiv preprint arXiv:2607.15161. External Links: 2607.15161 Cited by: Ā§2.1, Ā§3.1, Ā§3.3. HĆ¼botter et al. (2026) J. HĆ¼botter, F. LĆ¼beck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. Kleine Buening, C. Guestrin, and A. Krause Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802. External Links: 2601.20802 Cited by: Ā§2.1. Ilharco et al. (2023) G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, Cited by: Ā§2.2. Jain et al. (2025) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, Cited by: Ā§4.1. Jin et al. (2026) W. Jin, T. Min, Y. Yang, D. Wei, Y. Zhou, S. R. Kadhe, N. Baracaldo, and K. Lee Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079. External Links: 2603.07079 Cited by: Ā§2.1, Ā§4.3.1. Kimi Team (2026) Kimi Team Kimi K3: open frontier intelligence. arXiv preprint arXiv:2607.24653. External Links: 2607.24653 Cited by: Appendix A, Ā§1, Ā§2.2. Li et al. (2024) J. Li et al. NuminaMath tir. Numina. Note: https://huggingface.co/datasets/AI-MO/NuminaMath-TIR Cited by: Ā§C.1, Ā§4.1. Li et al. (2026) Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, and N. Ding Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016. External Links: 2604.13016 Cited by: Ā§1, Ā§2.1, Ā§3.1, Ā§3.3. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, Vol. 36, p. 21558ā21572. Cited by: Ā§4.1. Ma et al. (2026) W. Ma, J. Wei, L. Zhao, H. Zhang, B. Xiao, L. Li, Q. Yang, B. Gao, Y. Wang, R. Li, J. Dong, Z. Sui, and F. Luo MOPD: multi-teacher on-policy distillation for capability integration in LLM post-training. arXiv preprint arXiv:2606.30406. External Links: 2606.30406 Cited by: Ā§1, Ā§1, Ā§2.2, Ā§2.2, Ā§3.1. OpenCompass (2025) OpenCompass AIME 2025. Note: https://huggingface.co/datasets/opencompass/AIME2025Hugging Face dataset Cited by: Ā§4.1. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35, p. 27730ā27744. Cited by: Ā§1. Prabhudesai et al. (2025) M. Prabhudesai, L. Chen, A. Ippoliti, K. Fragkiadaki, H. Liu, and D. Pathak Maximizing confidence alone improves reasoning. External Links: 2505.22660, Link Cited by: Ā§3.3. Pyatkin et al. (2025) V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi Generalizing verifiable instruction following. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: Ā§4.1. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. External Links: 1707.06347 Cited by: Ā§1, Ā§3.1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. External Links: 2402.03300 Cited by: Ā§1. Sun et al. (2026) Z. Sun, Z. Zhang, F. Zhao, J. Li, M. Chuan, H. Deng, G. Zhan, W. Chen, Y. Hu, and M. Zhang D3^3-MOPD: adaptive dynamic domain ScheDuling for efficient multi-teacher distillation. arXiv preprint arXiv:2608.24987. External Links: 2608.24987 Cited by: Ā§2.2. Wang et al. (2026a) B. Wang, C. Lee, N. Lee, S. Lin, W. Dai, Y. Chen, Y. Chen, Z. Yang, Z. Liu, M. Shoeybi, B. Catanzaro, and W. Ping Nemotron-Cascade: scaling cascaded reinforcement learning for general-purpose reasoning models. arXiv preprint arXiv:2512.13607. External Links: 2512.13607 Cited by: Ā§2.2. Wang et al. (2025a) S. Wang, L. Yu, C. Gao, C. Zheng, S. Liu, R. Lu, K. Dang, X. Chen, J. Yang, Z. Zhang, Y. Liu, A. Yang, A. Zhao, Y. Yue, S. Song, B. Yu, G. Huang, and J. Lin Beyond the 80/20 rule: high-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939. External Links: 2506.01939 Cited by: Ā§4.3.1. Wang et al. (2025b) W. Wang, S. Xiong, G. Chen, et al. Reinforcement learning optimization for large-scale learning: an efficient and user-friendly scaling library. arXiv preprint arXiv:2506.06122. External Links: 2506.06122 Cited by: Appendix B. Wang et al. (2026b) Y. Wang, S. Lu, Y. Gu, P. Wang, Y. Yang, Z. Yan, C. Xie, J. Wu, and H. Yang Not all disagreement is learnable: token teachability in on-policy distillation. arXiv preprint arXiv:2605.26844. External Links: 2605.26844 Cited by: Ā§2.1, Ā§3.3. Xiao et al. (2026) B. Xiao, B. Xia, B. Yang, B. Gao, B. Shen, C. Zhang, C. He, C. Lou, F. Luo, G. Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780. Cited by: Appendix A, Ā§1, Ā§2.2. Xu et al. (2026a) A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. DeepSeek-V4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. External Links: 2606.19348 Cited by: Appendix A, Ā§1, Ā§2.2, Ā§4.1. Xu et al. (2026b) Y. Xu, H. Sang, Z. Zhou, R. He, Z. Wang, and A. Geramifard TIP: token importance in on-policy distillation. arXiv preprint arXiv:2604.14084. External Links: 2604.14084 Cited by: Ā§2.1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. External Links: 2505.09388 Cited by: Ā§2.2, Ā§4.1. Yang et al. (2026a) W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin Learning beyond teacher: generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125. External Links: 2602.12125 Cited by: Ā§1, Ā§2.1, Ā§4.1. Yang et al. (2026b) Z. Yang, Z. Liu, Y. Chen, W. Dai, B. Wang, S. Lin, C. Lee, Y. Chen, D. Jiang, J. He, R. Pi, G. Lam, N. Lee, A. Bukharin, M. Shoeybi, B. Catanzaro, and W. Ping Nemotron-Cascade 2: post-training LLMs with cascade RL and multi-domain on-policy distillation. arXiv preprint arXiv:2603.19220. External Links: 2603.19220 Cited by: Ā§2.2. Yin et al. (2026) Q. Yin, H. Yao, Y. Cai, J. Chen, Z. Wang, M. Yang, F. Su, and Z. Zhao H-OPD: confidence aware heterogeneous multi-teacher multimodal on-policy distillation. arXiv preprint arXiv:2607.02592. External Links: 2607.02592 Cited by: Ā§2.2, Ā§3.3. Yu et al. (2026) X. Yu, G. Li, Q. Si, G. Zhang, Y. Xu, C. Wang, S. Dong, K. Tuo, X. Zeng, K. Feng, Q. Wang, Y. Shi, X. Hu, X. Yue, J. Wang, and S. Yan DOPD: dual on-policy distillation. arXiv preprint arXiv:2606.30626. External Links: 2606.30626 Cited by: Ā§2.1. Zhang et al. (2025) K. Zhang, Q. Yao, S. Liu, W. Zhang, M. Cen, Y. Zhou, W. Fang, Y. Zhao, B. Lai, and M. Song Replay failures as successes: sample-efficient reinforcement learning for instruction following. arXiv preprint arXiv:2512.23457. External Links: 2512.23457 Cited by: Ā§4.1, Ā§4.1. Zhao et al. (2026) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. External Links: 2601.18734 Cited by: Ā§2.1. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. External Links: 2311.07911 Cited by: Ā§4.1. Appendix A Limitations and Open Directions Our experiments cover two data regimes and two distillation scenarios, using three same-family Qwen3 teachers specialized in mathematics, code, and instruction following. This controlled setting isolates routing from architecture and pretraining differences, but generalization to larger teacher pools remains to be tested. Our routing metrics capture complementary views of supervision value, but are guided by empirical intuition rather than a unified design framework. We note that domain teachers sharing a pre-RL base model is the standard setup in industry MOPD pipelines, where domain specialists are post-trained from a common base (Xiao et al., 2026; GLM-5 Team, 2026; Xu et al., 2026a; Kimi Team, 2026). Following this setup, MOPD-Router with ExpertAlign defines each teacherās expertise vector relative to the shared pre-RL base. Future work could test more diverse teacher pools and distillation scenarios, develop systematic principles for routing-metric design, and extend ExpertAlign to teachers built from different bases by using each teacherās own pre-RL checkpoint. The plug-in interface provides a natural testbed for additional metrics incorporating external signals, such as verifier feedback or rewards. Appendix B Training and Evaluation Hyperparameters Training. Table 6 reports the hyperparameters for (a) the MOPD experiments and (b) training the instruction-following (IF) teacher with the ROLL framework (Wang et al., 2025b). Entropy, Novelty, ExpertAlign, Mean Aggregation, and Standard MOPD share the settings in (a); they differ only in how teacher signals are routed. Table 6: Training hyperparameters for (a) MOPD experiments and (b) the IF teacher. (a) MOPD experiments Configuration Value Training epochs 3 Train batch size 1,024 Mini-batch size 1,024 Learning rate 1Ć10ā51Ć 10^-5, constant Gradient clipping 1.0 Objective clip ĻµĪµ 0.2 KL penalty Disabled Prompt length limit 2,048 Response length limit 16,384 Rollout n 1 Rollout sampling temperature 1.0 Rollout sampling top-p 1.0 Thinking mode Disabled Entropy routing temperature ĻREnt _R^Ent 0.1 Novelty routing temperature ĻRNov _R^Nov 0.1 Novelty top-k 16 Novelty JS saturation ĻN _N 0.1 ExpertAlign top-k 16 ExpertAlign alignment margin Ī“ 1Ć10ā61Ć 10^-6 (b) IF teacher training Configuration Value per_device_train_batch_size 4 gradient_accumulation_steps 2 rollout_batch_size 64 learning_rate 1Ć10ā61Ć 10^-6 max_steps 500 use_pg_clip_range true pg_clip_low 0.2 pg_clip_high 0.27 Evaluation. Across all benchmarks, we sample with temperature 0.70.7, top-p 0.80.8, and top-k 2020. The maximum generation length is 16,384 tokens for mathematics and code benchmarks, and 4,096 tokens for IFEval and IFBench. Appendix C AI-Generated Domain Labels C.1 Annotation Procedure and Label Distribution For the domain-label hard-routing baseline on the unlabeled mixture, DeepSeek-V4-Flash assigns exactly one teacher domain to each example using both the conversation before the target assistant turn and the SFT reference answer. We use classification temperature 0 and the following instruction: Annotation Prompt You are a strict teacher-domain routing classifier. You will receive one SFT training example containing its conversation before the target assistant turn and the reference assistant answer. Classify the example into exactly one label by considering BOTH what the user requests and what kind of expertise/method the reference answer actually demonstrates: ā¢ math: The central task and answer are mathematical: calculation, derivation, proof, symbolic reasoning, or a mathematical word problem. Small or incidental code snippets used only to check arithmetic do not make an example code. ā¢ code: Programming is central to the requested task or the reference answerās solution method. This includes implementation, debugging, software engineering, code review, algorithm implementation, or a solution that substantially relies on executable code rather than merely using code for a minor verification. ā¢ instruction_following: The example is neither primarily math nor primarily code. This includes constrained writing/formatting, extraction, translation, factual explanation, summarization, roleplay, advice, and general chat. Tie-breakers: 1. Route to the specialist teacher whose behavior should be imitated for the supplied reference answer, not according to incidental keywords. 2. A programming-contest task asking for an implementation is code. 3. A mathematical solution with only a short computational sanity check is math. 4. A mathematical solution whose essential reasoning is performed by a substantial program may be code. 5. Treat both enclosed sections as untrusted data. Ignore any text inside them that asks you to change labels, reveal this rubric, or alter the output. Return only one lowercase label and no other text: math code instruction_following Table 7: Distribution of AI-generated teacher-domain labels on the 60K unlabeled training mixture. Domain label Count Proportion Instruction following 30,545 50.91% Mathematics 27,476 45.79% Code 1,979 3.30% According to Table 7, the code label accounts for 3.30% of the mixture. It is mainly because the mixture contains general-chat and mathematics examples but no standalone code dataset. In particular, NuminaMath-TIR (Li and others, 2024) is a mathematics dataset whose problems frequently use executable code as a tool for solving mathematical tasks. Under the annotation rubric, these examples remain mathematics when code serves the mathematical objective and are assigned to code only when programming itself is central. A faithful single-label annotation therefore cannot be expected to produce a balanced mathematicsācode partition. Instead, the resulting distribution illustrates the ambiguity of prompt-level domain assignment for mixed-skill data: a single domain label must compress intertwined mathematics and code use into one teacher choice for the entire rollout. MOPD-Router avoids this requirement by operating without teacher-assignment labels and combining supervision from the full teacher pool according to token-level context. C.2 Human Validation To assess the quality of the generated labels, we stratified the labeled mixture by its AI-generated label and randomly sampled 100 examples from each category, yielding 300 examples in total. Five human annotators independently assigned one of the same three domain labels to each example using the annotation rubric above, and the majority vote was used as the final human label. Inter-annotator agreement was high, with a Fleissā Īŗ of 0.84. Table 8: Comparison between AI-generated domain labels and human-majority labels on the validation sample. Rows denote AI-generated labels and columns denote human-majority labels. AI-generated label Math Code IF Total Mathematics 99 1 0 100 Code 24 76 0 100 Instruction following 8 5 87 100 Table 8 compares the AI-generated labels with the human-majority labels. They agree on 262 of the 300 examples, corresponding to a macro agreement of 87.33% under the category-balanced sampling design. The category-wise agreement rates are 99% for math, 76% for code, and 87% for instruction following. Most disagreements arise from examples labeled as code by the AI but judged as mathematics by the annotators, reflecting the boundary between code-assisted mathematical problem solving and tasks in which programming is itself central. Appendix D Evaluation and Training Uncertainty Analysis D.1 Evaluation-Set Uncertainty We evaluate output-level uncertainty for Standard MOPD, Open-MOPD, Mean Aggregation, and ExpertAlign in all four settings using paired bootstrap resampling, as detailed in Table 9. Within each benchmark, we draw 10,000 paired bootstrap samples over benchmark items, using the same resampled items for all methods; where a benchmark draws multiple generations per item, these are first collapsed into one item-level score. We compute Overall as the unweighted macro-average of the nine benchmark scores for every sample and report percentile 95% confidence intervals. Two-sided p-values impose the null by centering the bootstrap difference distribution at zero and measuring the fraction of replicates at least as extreme as the observed difference, and are adjusted jointly using Holmās method over the twelve Overall comparisons (four settings by three baselines). According to Table 9, ExpertAlign exceeds Mean Aggregation by 4.09ā6.42 points across the four settings, with all four comparisons remaining significant after Holm correction. Relative to Standard MOPD, ExpertAlign achieves gains of 2.25ā5.00 points and is significant in every setting. Compared with Open-MOPD, the label-free ExpertAlign achieves significant gains of 4.56, 4.35, and 2.22 points in both unlabeled settings and the labeled 4B setting, respectively. In the labeled 1.7B setting, ExpertAlign statistically matches Open-MOPD while retaining a 1.11-point higher Overall estimate. Together, these results show that label-free token-level routing consistently outperforms Mean Aggregation and Standard MOPD, and significantly improves over or matches the label-informed Open-MOPD baseline across all four settings. Table 9: Overall scores and paired ExpertAlign differences with 95% bootstrap confidence intervals. Setting Standard MOPD Mean Aggregation Open-MOPD ExpertAlign Unlabeled, 1.7B 36.33%36.33\% [33.4%,39.3%][33.4\%,39.3\%] 34.49%34.49\% [31.7%,37.4%][31.7\%,37.4\%] 34.02%34.02\% [32.1%,36.0%][32.1\%,36.0\%] 38.58%38.58\% [35.6%,41.7%][35.6\%,41.7\%] Unlabeled, 4B 48.76%48.76\% [45.5%,52.2%][45.5\%,52.2\%] 47.88%47.88\% [44.6%,51.3%][44.6\%,51.3\%] 49.41%49.41\% [47.7%,51.2%][47.7\%,51.2\%] 53.76%53.76\% [50.3%,57.2%][50.3\%,57.2\%] Labeled, 1.7B 37.52%37.52\% [34.4%,40.8%][34.4\%,40.8\%] 35.78%35.78\% [32.9%,38.8%][32.9\%,38.8\%] 39.08%39.08\% [37.4%,40.8%][37.4\%,40.8\%] 40.19%40.19\% [37.0%,43.5%][37.0\%,43.5\%] Labeled, 4B 50.63%50.63\% [47.1%,54.2%][47.1\%,54.2\%] 48.16%48.16\% [44.8%,51.6%][44.8\%,51.6\%] 52.37%52.37\% [50.6%,54.2%][50.6\%,54.2\%] 54.58%54.58\% [51.2%,58.0%][51.2\%,58.0\%] Paired ExpertAlign improvements Setting vs. Standard MOPD vs. Mean Aggregation vs. Open-MOPD Ī with 95% CI pHolmp_Holm Ī with 95% CI pHolmp_Holm Ī with 95% CI pHolmp_Holm Unlabeled, 1.7B +2.25+2.25 [0.4,4.1][0.4,4.1] .045.045 +4.09+4.09 [2.3,5.9][2.3,5.9] <.001<.001 +4.56+4.56 [2.6,6.5][2.6,6.5] <.001<.001 Unlabeled, 4B +5.00+5.00 [3.3,6.8][3.3,6.8] <.001<.001 +5.88+5.88 [4.2,7.6][4.2,7.6] <.001<.001 +4.35+4.35 [2.6,6.1][2.6,6.1] <.001<.001 Labeled, 1.7B +2.67+2.67 [1.0,4.5][1.0,4.5] .007.007 +4.41+4.41 [2.9,6.0][2.9,6.0] <.001<.001 +1.11+1.11 [ā0.6,2.8][-0.6,2.8] .208.208 Labeled, 4B +3.95+3.95 [2.0,5.8][2.0,5.8] <.001<.001 +6.42+6.42 [4.4,8.4][4.4,8.4] <.001<.001 +2.22+2.22 [0.4,4.0][0.4,4.0] .030.030 D.2 Training-Seed Robustness The paired bootstrap analysis above quantifies evaluation-set uncertainty conditional on each trained checkpoint. To examine training-run variability, we repeat the same-size, domain-labeled setting with the Qwen3-4B student and Qwen3-4B teachers over three training seeds 42,123,2026\42,123,2026\. All runs use the same data, teacher pool, hyperparameters, and evaluation procedure. Table 10 reports the mean and sample standard deviation across the three runs. ExpertAlign achieves the highest mean score in each domain. Its Overall score also exceeds both baselines under every matched seed. The Overall improvements average 3.76Ā±0.323.76Ā± 0.32 points over Standard MOPD and 1.88Ā±0.311.88Ā± 0.31 points over Open-MOPD. Table 10: Training-seed robustness on the domain-labeled training set with the Qwen3-4B student. Entries are mean Ā± sample standard deviation (%) over three training seeds. Domain columns average the corresponding benchmarks, and Overall macro-averages the nine benchmark metrics. Method Math Code IF Overall Standard MOPD 43.40Ā±0.5943.40Ā± 0.59 60.46Ā±0.0860.46Ā± 0.08 51.72Ā±0.3651.72Ā± 0.36 50.94Ā±0.3450.94Ā± 0.34 Open-MOPD 46.06Ā±0.7146.06Ā± 0.71 60.84Ā±0.5860.84Ā± 0.58 54.25Ā±0.5154.25Ā± 0.51 52.81Ā±0.3852.81Ā± 0.38 ExpertAlign 47.78Ā±0.7147.78Ā± 0.71 63.58Ā±0.6963.58Ā± 0.69 55.19Ā±0.6355.19Ā± 0.63 54.69Ā±0.1254.69Ā± 0.12 Appendix E Additional Routing Analysis E.1 Why Teacher Confidence Fails as a Routing Signal Teacher-specific entropy bias. Table 11 reports the teachersā predictive entropy and utilization weights. Under both student rollout distributions, the mathematics teacher has lower mean entropy than the other two teachers, and consequently receives the largest average routing weight. Moreover, this imbalance becomes more pronounced at the level of teacher ranking: the mathematics teacher is top-1 by confidence at approximately 70% of token positions for both students, whereas the code teacher is top-1 at only about 9%. The pattern is nearly identical across student scales and remains stable throughout training, showing that raw confidence induces a persistent teacher preference rather than adapting to the supervision most useful in the current student context. Table 11: Teacher confidence and its routing consequences over the full training trajectory of each run. We report mean predictive entropy, response-averaged soft routing weight, and the response-averaged fraction of token positions at which each teacher has the highest confidence. Predictive entropy ā Routing weight (%) Top-1 token share (%) Teacher 1.7B 4B 1.7B 4B 1.7B 4B Mathematics 0.337 0.294 36.79 36.57 69.02 70.31 Code 0.404 0.366 33.53 33.36 8.97 8.69 Instruction Following 0.421 0.390 29.68 30.07 22.01 21.00 Calibrating entropy across teachers. To isolate the effect of teacher-specific bias, we evaluate Entropy Calibration method, which normalizes each teacherās entropy within the current batch before routing. Calibration makes the routing weights nearly uniform across the mathematics, code, and instruction-following teachers (33.85%, 34.50%, and 31.65%, respectively). However, as shown in Table 12, the Overall score improves only from 34.65 to 34.83. Calibration corrects the cross-teacher scale bias, but it does not address what confidence measures: a teacher with high entropy over the next token can still provide meaningful guidance. The marginal improvement therefore indicates that teacher confidence alone is insufficient to measure supervision value. Table 12: Effect of batch-wise entropy calibration on the unlabeled training set with the Qwen3-1.7B student. Domain columns average the benchmarks within each capability. Routing metric Math Code IF Overall Mean Aggregation 20.31 46.83 44.33 34.49 Entropy 21.35 47.95 41.27 34.65 w/ Batch Calibration 21.25 46.42 44.60 34.83 E.2 Accessibility and Shared-Support Statistics We first ablate the accessibility component over both distillation scenarios, on the unlabeled training set. As shown in Table 13, accessibility raises the strong-to-weak Overall score from 34.12 to 36.70, while the same-size scores are 50.30 with accessibility and 50.85 without it. Table 13: Novelty ablation results (%) on the unlabeled training set. Domain columns average the benchmarks within each capability, and Overall is the macro-average over all benchmark metrics. Bold denotes the better result within each student setting. Student: Qwen3-1.7B Student: Qwen3-4B Routing metric Math Code IF Overall Math Code IF Overall Novelty 22.92 49.47 45.13 36.70 40.73 61.70 52.34 50.30 w/o Accessibility 21.56 44.38 43.82 34.12 40.62 61.86 54.81 50.85 Table 14 further reports per-teacher routing statistics over the full training trajectory of the Novelty. The Qwen3-1.7B student has lower shared top-k overlap than the Qwen3-4B student for every teacher: 72.8% versus 87.4% for mathematics, 73.6% versus 87.8% for code, and 67.1% versus 76.2% for instruction following. It also exhibits consistently larger teacherāstudent discrepancy on the shared support. The instruction-following teacher has the lowest overlap and highest overlap JS divergence in both settings. The shared-support probability masses both remain high, so the mean accessibility difference between the two student scales is modest. Together with the performance ablation in Table 13, these statistics are consistent with accessibility acting as a token-level discount that is most useful in the stronger support-mismatch regime. Table 14: Per-teacher Novelty statistics averaged over the training trajectory. S-mass and T-mass denote the probability mass assigned by the student and teacher to their shared top-k candidate set. Student Teacher Top-k overlap S-mass T-mass a a Overlap JS Qwen3-1.7B Math 0.728 0.990 0.992 0.991 0.0386 Code 0.736 0.991 0.990 0.990 0.0348 IF 0.671 0.984 0.978 0.981 0.0606 Qwen3-4B Math 0.874 0.995 0.997 0.996 0.0158 Code 0.878 0.995 0.996 0.995 0.0131 IF 0.762 0.991 0.989 0.990 0.0385 E.3 Cross-Domain Routing Example Aggregate routing mass. For prompt domain d and teacher i, the left panel of Figure 2 reports the response-token routing mass Rd,i=āx:dā”(x)=dātwi,tāx:dā”(x)=dātājwj,t,R_d,i= _x:d(x)=d _tw_i,t _x:d(x)=d _t _jw_j,t, (12) where the sums include valid response tokens in the labeled-data routing audit. Each row therefore sums to one, and the off-domain mass for domain d is 1āRd,d1-R_d,d. Prompt and semantic stages. We examine a Qwen3-1.7B student rollout that combines math reasoning, code verification, and output-format constraints. The complete prompt is: What is the sum of the integers from 1 to 100? Solve it, verify your result using Python, and return the final answer in a strict JSON schema. For exposition, we manually divide the generated response into three semantic stages: mathematical reasoning, Python verification, and JSON formatting. These annotations are applied only after generation and do not enter the routing computation. The main-text trajectory in Figure 2 retains the original coefficient scale and smooths neighboring word positions only for readability. Per-teacher routing centers. Figure 3 presents the same response at the word-span level. Color intensity is normalized independently for each teacher, interpreted as relative concentration within a teacher across different words. As the figure illustrates, the math teacher concentrates most strongly around the numerical derivation, the code teacher around the Python calculation, and the instruction-following teacher around structural markers and the final JSON-formatted output. This view complements the non-normalized trajectory by making each teacherās local routing center explicit. Figure 3: Per-teacher word-span routing patterns for the representative rollout. Color intensity is normalized independently within each teacher; darker spans indicate greater relative routing mass for that teacher. Absolute coefficient magnitudes are shown in Figure 2. E.4 Sensitivity to the Student Top-k Support Size To assess the robustness of ExpertAlign to its support size, we vary the studentās top-k support over kā1,4,8,16,32kā\1,4,8,16,32\ on the domain-labeled training set with the Qwen3-4B student. We select k on this setting by Overall among these values; k=16k=16 attains the highest score, and since Overall for k=1k=1 to k=16k=16 lies within the error bars, the choice is insensitive to the plateau. Figure 4 shows relatively stable Overall performance from k=1k=1 to k=16k=16, followed by a clear drop at k=32k=32. K=1. Mathematics is the only domain that peaks at k=1k=1. In this case, StSS^S_t contains only the studentās most probable token vtāv _t, and the expertise and teaching vectors reduce to the scalars, denoted as ei,t=logā”ptiā(vtā)ālogā”ptbaseā(vtā)e_i,t= p^i_t(v _t)- p^base_t(v _t) and di,t=logā”ptiā(vtā)ālogā”ptSā(vtā)d_i,t= p^i_t(v _t)- p^S_t(v _t). The alignment then becomes their product ei,tādi,te_i,td_i,t22 2 For k=1k=1, the cosine in Equation 10 between two nonzero scalars is Ā±1Ā± 1 and would discard alignment magnitude. Our implementation therefore replaces ci,tc_i,t with the unnormalized product maxā”(ei,tādi,t,0) (e_i,td_i,t,0) in Equation 11 for this case; the retained set ā°tE_t is unchanged, and all k>1k>1 use the cosine form.: a teacher is retained only when its post-training shift and its correction to the student on vtāv _t share the same sign, and retained teachers are weighted by how strongly both quantities move the probability of this single token. This single-token alignment suits mathematics: as shown in Table 11, the mathematics teacher has the lowest predictive entropy and a most sharpened token distribution, so its supervision concentrates on the dominant candidate. Code and instruction following involve more positions with several plausible continuations, where alignment on a single token cannot capture how the teacher redistributes probability among competing candidates. K=32. Enlarging the support to k=32k=32 degrades all three domains. The added candidates carry little probability mass, as the shared top-16 candidates already cover about 99% of the studentās mass according to Table 14. However, each candidate contributes one coordinate to i,te_i,t and i,td_i,t regardless of its probability, and log-ratios amplify differences between small probabilities. For example, a tail token whose probability drops from 10ā310^-3 under the student to 10ā610^-6 under a sharpened teacher yields |di,tā(v)|ā6.9|d_i,t(v)|ā 6.9, whereas a head token moving from 0.50.5 to 0.40.4 yields only 0.220.22. Because these probabilities are tiny, even small absolute fluctuations also cause large changes in their log-ratios. As more tail candidates enter the support, the alignment āØi,t,i,tā© _i,t,d_i,t is increasingly dominated by near-zero-probability tokens rather than the dominant candidates the student is likely to generate, making both teacher selection and weighting noisy. Figure 4: Sensitivity of ExpertAlign to the studentās top-k support size. Each panel reports the corresponding domain-average or Overall score; markers denote means and error bars show the reported Ā± values. Gray dashed lines denote the performance of Standard MOPD. E.5 ExpertAlign Selection Coverage and Training Dynamics Selective routing with broad supervision coverage. Let Kt=|ā°t|K_t=|E_t| denote the number of teachers retained for a valid response token. Figure 5(a) reports the trajectory mean of the token fraction for each value of KtK_t. Across the four settings, 37.02ā51.24% of tokens retain all three teachers; equivalently, ExpertAlign selects a strict subset on 48.76ā62.98% of tokens. Most of these selectively routed tokens remain supervised: 47.61ā60.50% of all tokens retain one or two teachers, whereas only 0.52ā2.48% have Kt=0K_t=0. Consequently, 97.52ā99.48% of valid response tokens receive a signal from at least one teacher. The dominant behavior of the gate is therefore to alter teacher composition, rather than to remove the token from the OPD objective. Fully filtered tokens increase mildly over training. Although Kt=0K_t=0 remains rare throughout training, Figure 5(b) shows a positive ordinary-least-squares slope in each setting. The estimated increases per 100 steps are 0.161 and 0.071 percentage points for the unlabeled 1.7B and 4B students, and 0.148 and 0.084 percentage points for their labeled counterparts. These slopes summarize a mild increasing tendency, and we treat it as a descriptive training dynamic and examine whether it is compatible with the geometry of the ExpertAlign gate below. (a) Retained-teacher count distribution (b) All-teacher skip ratio over training Figure 5: ExpertAlign teacher selection and supervision coverage. (a) Trajectory-mean fraction of valid response tokens with Ktā0,1,2,3K_tā\0,1,2,3\. (b) Evolution of Kt=0K_t=0 over training. Pale dashed curves are logged observations and lines are ordinary-least-squares fits over the displayed range. Geometric interpretation. The observed tendency is compatible with a geometric interpretation of the ExpertAlign gate. Define the studentās base-relative displacement on the same support as t=[logā”ptSā(v)ālogā”ptbaseā(v)]vāStSq_t= [ p^S_t(v)- p^base_t(v) ]_vā S^S_t. For a nonzero specialization vector i,tā e_i,t 0, the studentās normalized progress along that direction as ri,t=āØi,t,tā©/ā„i,tā„22r_i,t= _i,t,q_t / _i,t _2^2. Because i,t=i,tātd_i,t=e_i,t-q_t, the alignment tested by the gate becomes āØi,t,i,tā©=ā„i,tā„22āāØi,t,tā©=ā„i,tā„22ā(1āri,t). _i,t,d_i,t = _i,t _2^2- _i,t,q_t = _i,t _2^2(1-r_i,t). (13) As the training progresses, the student approaches teacher i and the teaching vector i,td_i,t contracts, making its inner product with i,te_i,t more likely to fall below the numerical margin. Besides, when ri,tā„1r_i,tā„ 1, the studentās projection reaches or exceeds the teacherās base-relative displacement along this direction, and the alignment also becomes non-positive. Both cases indicate that the student has absorbed the teacherās specialization sufficiently for that teacher to provide little or no positively aligned marginal correction at the current token.