Paper deep dive

Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation

Shi-Yu Tian, Yong Liu, Ximan Liu, Tianrui Zhu, Sule Bai, Xinwei Yang, Ruiqi Liu, Kun-Yang Yu, Ming Yang, Zhi Zhou, Yu-Feng Li, Yipeng Sun, Gu Simiu

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 100

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 10/13/2026, 2:44:27 AM

Summary

The paper introduces ROAD MOPD (Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation), a method designed to mitigate gradient conflicts in multi-teacher on-policy distillation (MOPD). By using non-routed teachers as auxiliary critics, ROAD MOPD employs token-level credit assignment based on teacher agreement and preventive multi-teacher gradient projection to resolve conflicting updates in shared parameters, thereby improving performance in high-conflict domain settings.

Entities (11)

ROAD MOPD · method · 98%MOPD · method · 95%OPD · method · 95%Preventive Gradient Projection · technique · 93%Token-Level Credit Assignment · technique · 93%Gradient Conflict · problem · 92%Code · domain · 90%Math · domain · 90%Medical · domain · 90%Qwen3-1.7B · model · 90%Qwen3-4B · model · 90%

Relation Signals (6)

ROAD MOPD → mitigates → Gradient Conflict

confidence 95% · ROAD MOPD mitigates degradation in the disadvantaged domain, gains over 5 percentage points under high conflict

ROAD MOPD → uses → Token-Level Credit Assignment

confidence 95% · cross-teacher-guided credit assignment uses teacher agreement to strengthen token-level corrections

ROAD MOPD → uses → Preventive Gradient Projection

confidence 95% · preventive multi-teacher gradient projection uses auxiliary gradients from the same trajectories to constrain potentially conflicting updates

MOPD → suffersfrom → Gradient Conflict

confidence 90% · supervision that is effective in isolation may still lead to persistent gradient conflicts and capability degradation during joint optimization

Qwen3-4B → usedas → Student

confidence 90% · We use Qwen3-4B and Qwen3-1.7B students

ROAD MOPD → improves → Medical

confidence 85% · In high-conflict settings, ROAD MOPD improves Medical accuracy by 5.6–8.8 percentage points over Vanilla MOPD

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) by routing domain-specific data to specialized teachers, integrating their capabilities through dense token-level supervision on student-generated trajectories. However, domain routing only distinguishes the supervision sources for different domains; updates from all domains still act on shared parameters. When teachers differ substantially in response style and domain expertise, supervision that is effective in isolation may still lead to persistent gradient conflicts and capability degradation during joint optimization. To address this challenge, we propose Cross-Teacher Guarding for MOPD (Road MOPD), which introduces non-routed teachers as auxiliary cross-domain critics on the same student trajectories and coordinates multi-teacher supervision through token-level credit assignment and gradient updates. Specifically, cross-teacher-guided credit assignment uses teacher agreement to strengthen token-level corrections jointly supported by the teachers and attenuate conflicting signals; preventive multi-teacher gradient projection uses auxiliary gradients from the same trajectories to constrain potentially conflicting updates and adaptively controls projection strength to retain target-domain learning signals. In experiments, Road MOPD mitigates degradation in the disadvantaged domain, gains over 5 percentage points under high conflict, and improves overall performance in low-conflict and three-domain settings.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

96,621 characters extracted from source content.

Expand or collapse full text

Paper Under Review BEYOND DOMAIN-ROUTED SUPERVISION: CROSS-TEACHER GUARDING FOR MULTI-TEACHER ON-POLICY DISTILLATION Shi-Yu Tian 1,2,∗ , Yong Liu 2,∗ , Ximan Liu 2,† , Tianrui Zhu 2 , Sule Bai 2 , Xinwei Yang 2 , Ruiqi Liu 2 , Kun-Yang Yu 1 , Ming Yang 1 , Zhi Zhou 1,‡ , Yu-Feng Li 1,‡ , Yipeng Sun 2 , Gu Simiu 2 , 1 Nanjing University 2 JD.com ∗ Equal contribution † Project leader ‡ Corresponding authors ABSTRACT Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) by routing domain-specific data to specialized teachers, integrating their capa- bilities through dense token-level supervision on student-generated trajectories. However, domain routing only distinguishes the supervision sources for different domains; updates from all domains still act on shared parameters. When teach- ers differ substantially in response style and domain expertise, supervision that is effective in isolation may still lead to persistent gradient conflicts and capa- bility degradation during joint optimization. To address this challenge, we pro- pose Cross-Teacher Guarding for MOPD (ROAD MOPD), which introduces non- routed teachers as auxiliary cross-domain critics on the same student trajectories and coordinates multi-teacher supervision through token-level credit assignment and gradient updates. Specifically, cross-teacher-guided credit assignment uses teacher agreement to strengthen token-level corrections jointly supported by the teachers and attenuate conflicting signals; preventive multi-teacher gradient pro- jection uses auxiliary gradients from the same trajectories to constrain potentially conflicting updates and adaptively controls projection strength to retain target- domain learning signals. In experiments, ROAD MOPD mitigates degradation in the disadvantaged domain, gains over 5 percentage points under high conflict, and improves overall performance in low-conflict and three-domain settings. 1INTRODUCTION Reinforcement learning (RL) has emerged as an important paradigm for post-training large language models (OpenAI, 2023; DeepSeek-AI, 2025). However, training with outcome-level rewards lacks direct feedback on intermediate reasoning steps, making it difficult to attribute a response’s success or failure to specific generation decisions. Optimization therefore remains largely dependent on policy exploration, posing challenges for credit assignment and stable optimization. On-policy dis- tillation (OPD) (Gu et al., 2024; Agarwal et al., 2024; Ko et al., 2025) offers a promising alternative for addressing this limitation: it leverages a strong teacher to provide dense token-level supervision on trajectories generated by the student’s current policy, combining on-policy sampling aligned with the student’s current generation distribution with fine-grained teacher feedback. Prior work has improved OPD capability transfer through better distillation objectives and optimiza- tion strategies (Li et al., 2026a; Lin et al., 2026; Yang et al., 2026a; Zhang et al., 2026). A natu- ral extension is to consolidate specialized teachers’ complementary capabilities in a single student across domains. Multi-teacher on-policy distillation (MOPD) (Ma et al., 2026; DeepSeek-AI et al., 2026) pursues this goal, typically through direct domain routing: a sample from domain d receives supervision only from its routed teacher T d , while updates from all domains act on the same stu- dent model. This design exploits each teacher’s domain expertise but does not explicitly coordinate updates from different domains to shared parameters. Consequently, supervision that is effective in 1 arXiv:2610.13521v1 [cs.AI] 9 Oct 2026 Paper Under Review 11020304049 Optimizer update 0 25 50 75 100 Conflicting layers (%) Math × Code Avg. 2.2% Medical × Code Avg. 36.6% Medical × Math Avg. 45.2% a Gradient conflicts AIME24AIME25 HMMT-Feb HMMT-Nov HumanEval+ MBPP+ LCB v6 0 +2 +4 Score change (Vanilla MOPD − single-domain OPD) b Math + Code +2.08 +3.54 +0.42 +1.35 +0.76 -0.07 +0.29 MathCode AIME24AIME25 HMMT-Feb HMMT-Nov MedQA MedXpert MedMCQA GPQA-Med -15 -10 -5 0 +5 c Math + Medical +0.62 +3.54 +0.42 +2.19 -14.45 -3.07 -7.05 -9.26 Math Medical HumanEval+ MBPP+ LCB v6 MedQA MedXpert MedMCQA GPQA-Med -12 -8 -4 0 d Code + Medical +0.46 -1.33 -0.29 -12.10 -2.53 -6.81 -7.19 Code Medical Figure 1: Update conflicts reveal a failure mode of domain-routed MOPD. (a) Fraction of con- flicting layers during training; labels show training averages. (b–d) Per-benchmark score changes of Vanilla MOPD relative to single-domain OPD. Math–Code remains compatible, whereas both Medical pairings consistently degrade Medical performance. single-domain training may induce interfering update directions during joint optimization, limiting multi-domain capability integration. Prior MOPD studies have demonstrated the potential of joint learning on domain combinations such as math and code (Yang et al., 2026b; Hou et al., 2026), but these gains may not generalize to other combinations. As shown in Figure 1, joint training on math and code generally preserves or improves upon the corresponding single-domain OPD performance, with relatively few gradient conflicts. In contrast, jointly training medicine with either math or code consistently yields a high proportion of layers with conflicting gradients, and this proportion increases further in later stages of training. In both pairings, medical performance falls below the single-domain OPD baseline, show- ing that the joint-learning benefits observed for math and code do not extend to these combinations. This contrast highlights compatibility between domain updates as an important factor in MOPD and suggests that allocating supervision through domain routing alone is insufficient to coordinate capability transfer through shared parameters. To address this limitation, we reconsider teachers’ roles in MOPD: can non-routed teachers help anticipate and mitigate update interference while preserving the routed teacher’s domain-specific supervision? We retain this supervision and introduce non-routed teachers as cross-domain critics on the same student-generated trajectories. Their feedback on the same behavior provides local cues about potentially incompatible learning signals. We aim to reduce joint-optimization interference through this feedback while retaining target-domain learning signals. Specifically, we propose Cross-Teacher Guarding for MOPD (ROAD MOPD), to exploit cross- teacher feedback at both the token and gradient levels. At the token level, ROAD MOPD com- pares evidence from the routed and auxiliary teachers relative to a shared frozen reference model, strengthening supervision when their implied correction directions agree and downweighting the corresponding tokens when they conflict. At the gradient level, auxiliary supervision on the same trajectory defines a cross-domain reference direction, and preventive gradient projection suppresses components of the target-domain update that conflict with this direction. In high-conflict settings, ROAD MOPD improves Medical accuracy by 5.6–8.8 percentage points over Vanilla MOPD. It also improves overall performance in low-conflict and three-domain settings. Our contributions are as follows: • We identify a failure mode of domain-routed MOPD: when domain updates are poorly compatible, separately assigned supervision can induce persistent gradient conflicts in shared parameters. These conflicts accompany asymmetric performance degradation, high- lighting the importance of update compatibility for joint learning. • We propose ROAD MOPD, using non-routed teachers as cross-domain critics on the same student trajectories. Cross-teacher agreement guides token weighting, while preventive gradient projection adapts its strength under a norm-retention constraint to protect target- domain learning signals. 2 Paper Under Review • Across domain combinations and teacher–student configurations, ROAD MOPD improves adversely affected domains under high conflict, preserves low-conflict gains, and gener- alizes to three domains, outperforming standard MOPD and gradient-conflict correction baselines in overall performance. 2RELATED WORK 2.1ON-POLICY DISTILLATION Classical knowledge distillation trains a student to match a teacher’s predictive distribution (Hinton et al., 2015). Sequence-level distillation applies this idea to autoregressive models using teacher- generated responses, but creates a mismatch between training and inference prefixes (Kim & Rush, 2016). On-policy distillation (OPD) instead queries the teacher on student-generated trajectories, using reverse-KL, general on-policy, or contrastive objectives (Gu et al., 2024; Agarwal et al., 2024; Ko et al., 2025). Subsequent work analyzes sampled-token estimators (Shao et al., 2026); makes supervision more selective through token filtering, reweighting, verifiable rewards, or reasoning progress (Li et al., 2026a; Lin et al., 2026; Yang et al., 2026a; Zhang et al., 2026); and extends OPD across tokenizers or intermediate generation steps (He et al., 2026; Sun et al., 2026a). Reference- based extrapolation further scales teacher corrections against a frozen student initialization (Yang et al., 2026b). Despite these advances, OPD can generalize unevenly across capabilities (Li et al., 2026b). These methods refine how evidence from one assigned teacher is estimated, selected, or scaled; they do not test its compatibility with capabilities represented by other teachers. We instead study how domain-specialized teachers can jointly assess the same student-generated prefixes. 2.2MULTI-TEACHER ON-POLICY DISTILLATION Multi-teacher on-policy distillation (MOPD) transfers capabilities from domain specialists to a shared student, typically routing each rollout to its corresponding teacher for dense supervision (Ma et al., 2026). Domain routing is also used in large-scale and staged multi-domain post-training systems (Xiaomi LLM-Core Team et al., 2026; Yang et al., 2026c). Related methods adjust do- main proportions during pretraining or multi-domain reinforcement learning (Xie et al., 2023; Liang et al., 2025), but do not coordinate teacher feedback on the same trajectory. Among recent MOPD methods, Open-MOPD balances token shares and teacher–student gaps while refreshing stale re- wards (Gao et al., 2026), and D 3 -MOPD schedules domains by their remaining distillation gaps and learning progress (Sun et al., 2026b). Other methods separate capability-recovery and domain- preservation updates (Chen et al., 2026), or calibrate confidence across heterogeneous teachers (Yin et al., 2026). Gradient-conflict methods such as PCGrad and conflict-averse gradient descent instead modify already-computed task gradients (Yu et al., 2020; Liu et al., 2021). None uses non-routed teachers to evaluate the current target-domain trajectory. ROAD MOPD lets routed and auxiliary teachers assess the same prefixes: their agreement guides token-level credit assignment, while aux- iliary supervision defines preventive gradient constraints before domain updates are aggregated. 3METHOD 3.1PROBLEM SETUP AND ON-POLICY DISTILLATION Let π θ be the student policy, π T a teacher, and D a prompt distribution. For x ∼ D, the student samples y = (y 1 ,...,y T ) with prefixes h t = (x,y <t ). OPD minimizes the sequence-level reverse- KL objective: L OPD (θ;π T ,D) =E x∼D D KL (π θ (·| x)∥π T (·| x)).(1) Following prior work (Yang et al., 2026b), we express OPD as relative-reward maximization with a KL penalty to a frozen reference π ref . With reward scale λ > 0, we maximize J λ (θ) =E θ λ log π T (y | x) π ref (y | x) − D KL (π θ (·| x)∥π ref (·| x)) .(2) 3 Paper Under Review Vanilla MOPD: Domain-Routed Supervision Domain A data D A Domain B data D B · Σ · e.g. medical domaine.g. math domain Router hard routing Teacher A T A domain-specific Teacher B T B domain-specific no interaction gradient conflict g A T g B < 0 g A g B >90° shared parameters θ Routed supervision each domain → its teacher Coupled updates shared params, conflicting grads Student π θ rollout Rollout on domain d data x ∼D d y 1 y 2 y 3 y t · Shared prefixes Routed Teacher T routed assessment Auxiliary Teacher T auxiliary assessment Frozen Reference π ref shared baseline (A) Cross-Te a c h e r-Guided Credit Assignment evidence auxiliary y 1 y 2 y 3 y 4 y 5 · y t · gate Agreement-modulated loss L d g d upweight downweight neutral (B) Preventive Multi-Te a c h e r G r a d i e n t P r o j e c t i o n Auxiliary loss L′ d→ projected gradient minimum norm retention g d −g′ d g′ d g ̃ d projected gradient θ Target- domain loss L d minimum norm retention agr agr Road MOPD: CrossTeacher Guarding ra routed a Figure 2: Vanilla MOPD and ROAD MOPD. Left: Vanilla MOPD’s domain-routed supervision can produce conflicting updates to shared parameters. Right: ROAD MOPD uses same-trajectory auxiliary-teacher feedback for (A) agreement-guided token credit assignment and (B) preventive gradient projection under a target-gradient norm-retention constraint. whereE θ averages over x∼D and y ∼ π θ (·| x). Maximizing this objective recovers standard OPD at λ = 1; 0 < λ < 1 attenuates the teacher’s relative reward, while λ > 1 amplifies it (ExOPD). The reference provides a shared baseline, not an additional teacher. For K domain–teacher pairs(D d ,π T d ) K d=1 , MOPD routes each prompt to its domain teacher and minimizes the equal-weight objective: L MOPD (θ) = 1 K K X d=1 L OPD (θ;π T d ,D d ).(3) For a fixed target domain d ∈ 1,...,K, denote its routed teacher by T r = T d and a teacher from an auxiliary domain by T a . The upright labels r and a denote teacher roles, while d indexes domains. Both teachers and the reference score the same student-generated tokens under teacher forcing, without additional rollouts. We write ℓ r t , ℓ a t , and ℓ ref t for their respective log-probabilities of y t given h t . 3.2CROSS-TEACHER-GUIDED CREDIT ASSIGNMENT A fixed λ in Eq. (2) applies the same reward scale to every token, although a trajectory may con- tain both transferable reasoning patterns and domain-specific corrections. The routed teacher alone provides no cross-domain check on these corrections. We therefore query an auxiliary teacher on the same student-generated prefixes and use its feedback to adjust the routed teacher’s token-level reward. This strengthens supervision supported by both teachers and attenuates conflicting signals. Importantly, the auxiliary teacher is not treated as a supervisor for solving the target-domain task. Its relative preference is used only as a compatibility signal for the capability represented by that teacher. To compare teacher preferences, we measure each teacher’s log-probability relative to the shared frozen reference. Their token-wise agreement is s t = tanh ℓ r t − ℓ ref t τ |z routed-teacher evidence × tanh ℓ a t − ℓ ref t τ |z auxiliary-teacher evidence , s t ∈ [−1, 1].(4) 4 Paper Under Review Positive s t means that both teachers favor the same direction of correction relative to the refer- ence. Negative s t indicates opposing preferences, with values near −1 corresponding to strong cross-teacher disagreement. The score approaches zero when either teacher provides little evidence beyond the reference. We use this agreement to replace the global reward scale in Eq. (2) with a token-wise coefficient: λ t = clip(1 + ρ g s t ,λ min ,λ max ).(5) Here, ρ g > 0 controls the modulation strength; the subscript g denotes the agreement gate. When s t > 0, λ t > 1 strengthens the routed teacher’s relative reward on tokens supported by both teach- ers. When s t < 0, λ t < 1 attenuates the reward, with stronger disagreement producing greater attenuation. When s t = 0, λ t = 1 recovers standard OPD. Clipping limits both amplification and attenuation. For a sampled token y t and a comparison policy q, the k 1 log-ratio estimator is k 1,t (π θ ∥q) = logπ θ (y t | h t )− logq(y t | h t ). The modulated signal λ t (ℓ r t − ℓ ref t )− k 1,t (π θ ∥π ref ) is clipped and detached as the advantage in the PPO-style surrogate ℓ agr d,i,t (θ). Gradients flow through the policy importance ratio, not the advantage (Appendix A.1). With multiple auxiliary teachers, we average their token-wise agreement with the routed teacher before applying the same reward modulation (Appendix A.1). 3.3PREVENTIVE MULTI-TEACHER GRADIENT PROJECTION Token-level agreement does not guarantee compatible parameter updates, since token supervision signals are aggregated through shared parameters. We therefore formulate gradient correction as a constrained optimization problem: minimize conflict with an auxiliary teacher on the same trajecto- ries while preserving a minimum fraction of the target-gradient norm. We first compute the target-domain gradient g d = ∇ θ L agr d from the agreement-modulated loss. To construct the auxiliary gradient, we replace the routed teacher with the auxiliary teacher and disable agreement modulation. Let ℓ OPD d→a,i,t denote the resulting plain token-level OPD loss. Applying the same response-level reduction gives the auxiliary loss and gradient, which we distinguish from their target-domain counterparts using primes: L ′ d→a = 1 |B d | X i∈B d 1 T i T i X t=1 ℓ OPD d→a,i,t , g ′ d =∇ θ L ′ d→a .(6) Both losses use the same stored domain-d trajectories. Thus, g ′ d reflects the auxiliary teacher’s pref- erences on the current behavior, rather than its training gradient on auxiliary-domain data. Differ- ences in knowledge, response style, or confidence can shape this signal; its usefulness as a constraint is assessed through joint-domain performance. We optimize the corrected gradient separately for each trainable parameter tensor. Let g and g ′ be nonzero slices of g d and g ′ d , and let v denote a candidate corrected gradient. For an intended tensor- level step−ηv with η > 0, the first-order change in the same-trajectory auxiliary loss is−η⟨v,g ′ ⟩. Thus,⟨g,g ′ ⟩ < 0 indicates local conflict; otherwise, we retain g unchanged. For a conflicting pair, we restrict v to the line segment between g and its full projection onto the hyperplane orthogonal to g ′ (Yu et al., 2020): g full = g− ⟨g,g ′ ⟩ ∥g ′ ∥ 2 2 g ′ , S(g,g ′ ) = convg,g full .(7) Here, conv denotes the convex hull. This restriction preserves the component orthogonal to g ′ and only removes the conflicting component, up to full projection. Full projection eliminates local conflict but can nearly erase g when the gradients oppose each other. We therefore choose the gradient with the least residual conflict subject to a norm-retention floor m∈ (0, 1]: eg = arg min v∈S(g,g ′ ) −⟨v,g ′ ⟩ subject to ∥v∥ 2 2 ≥ m 2 ∥g∥ 2 2 .(8) On this segment, −⟨v,g ′ ⟩ ≥ 0 measures the predicted auxiliary-loss increase per unit step size, while the constraint preserves the target-gradient norm. The solution can be written as a partial 5 Paper Under Review projection: eg = g− α ⋆ ⟨g,g ′ ⟩ ∥g ′ ∥ 2 2 g ′ , α ⋆ ∈ [0, 1].(9) Here, α ⋆ is the fraction of the conflicting component removed. We use full projection (α ⋆ = 1) whenever it satisfies the norm floor; otherwise, we remove as much conflict as the floor permits, with∥eg∥ 2 = m∥g∥ 2 . For non-conflicting pairs, α ⋆ = 0. The closed-form coefficient and its deriva- tion are given in Appendix A.2. These are local statements about the same-trajectory auxiliary loss, not guarantees of auxiliary-domain performance after an optimizer step. Tensors with negligible gra- dient norm are left unchanged; auxiliary gradients are never applied as separate optimizer updates. The projected tensors form eg d . With multiple auxiliary teachers, we apply the projections sequentially under a shared norm- retention floor relative to the original target gradient (Appendix A.2). Method-specific hyperpa- rameters are summarized in Appendix B.3. A related approach in multi-task learning is PCGrad (Yu et al., 2020), which performs post-hoc gra- dient surgery by subtracting conflicting components from independently computed task gradients. Our projection is preventive: the auxiliary teacher assesses the current target-domain trajectory to construct a gradient constraint specifically for that update, using its feedback to constrain potentially incompatible corrections while retaining target-domain learning signals. We empirically compare these two projection strategies in Section 4.3. 3.4OVERALL OBJECTIVE AND TRAINING We use sequence-mean aggregation, averaging token losses within each response and then across re- sponses and domains. Auxiliary losses use the same reduction but guide only the gradient correction, so the overall distillation objective includes only target-domain losses: L overall (θ) = 1 K K X d=1 1 |B d | X i∈B d 1 T i T i X t=1 ℓ agr d,i,t (θ).(10) After applying the preventive projection in Section 3.3, we average only the projected target-domain gradients, using the same domain weights as Eq. (10). The resulting g update is globally clipped for one AdamW step: g update = 1 K K X d=1 eg d .(11) 4EXPERIMENTS 4.1EXPERIMENTAL SETUP Evaluation datasets. We evaluate 11 benchmarks across Code (HumanEval+, MBPP+, and Live- CodeBench) (Liu et al., 2023; Jain et al., 2024), Math (AIME 2024/2025 and HMMT Febru- ary/November 2025) (MAA, 2024; 2025; Harvard–MIT Mathematics Tournament, 2025a;b), and Medical (MedQA, MedXpertQA, MedMCQA, and SuperGPQA-Med) (Jin et al., 2020; Zuo et al., 2025; Pal et al., 2022; M-A-P Team et al., 2025). We report Avg@4 accuracy for Code, Avg@32 for Math, and Avg@1 for Medical. Avg@n averages correctness over n completions per problem and then over problems, equivalent to the same-sample pass@1 estimate for Math and Code. Baselines. We compare with four representative approaches: (1) Vanilla MOPD routes each on- policy rollout only to its corresponding domain teacher and applies token-level reverse-KL supervi- sion (Ma et al., 2026); (2) ExOPD constructs an extrapolated teacher target using the frozen student initialization as a reference (Yang et al., 2026b); (3) CaMOPD separates capability-recovery and domain-preservation updates and prioritizes samples with larger teacher–student gaps (Chen et al., 2026); and (4) Open-MOPD mitigates cross-domain imbalance through token-share balancing, dy- namic domain loss weighting, and reward refresh (Gao et al., 2026). Implementation details. We use Qwen3-4B and Qwen3-1.7B students with thinking disabled (Yang et al., 2025). Math and Code teachers use specialized Qwen3-4B checkpoints (Yang et al., 6 Paper Under Review Table 1: Main results across paired-domain settings. Domain Avg. averages benchmarks within each domain; Avg. averages the two domains. Bold/underlined scores mark the best/second-best multi-domain methods per student model. (a) Math× Medical Student Model: Qwen3-4B Method MathMedicalAll AIME24AIME25 HMMT25 Feb. HMMT25 Nov. Domain Avg.MedQA MedXpert QAMedMCQA SuperGPQA -Med Domain Avg. Avg. Student21.4621.7710.008.2315.3764.7313.8055.6329.8741.0128.19 Single OPD58.1351.6731.5635.7344.2777.0615.3162.4436.8147.9046.09 Vanilla MOPD58.7555.2131.9837.9245.9762.6112.2455.3927.5539.4542.71 ExOPD61.6755.6233.5437.9247.1969.9913.7659.5032.2343.8745.53 Open-MOPD56.5655.1032.6037.0845.3470.3113.0660.9433.7644.5244.93 CaMOPD61.1554.2733.1238.02 46.6477.4516.6161.8935.0347.7547.20 Ours63.8553.9633.6538.9647.6077.0615.6762.8337.3548.2347.92 Student Model: Qwen3-1.7B Method MathMedicalAll AIME24AIME25 HMMT25 Feb. HMMT25 Nov. Domain Avg.MedQA MedXpert QAMedMCQA SuperGPQA -Med Domain Avg. Avg. Student11.9810.216.043.968.0546.0310.3342.5522.5830.3719.21 Single OPD39.9034.4818.7517.2927.6053.8113.3950.9924.5035.6731.64 Vanilla MOPD41.4635.1018.8518.0228.3647.3711.3945.2520.6931.1829.77 ExOPD41.5635.4218.1318.6528.4449.889.8049.4922.8733.0130.72 Open-MOPD37.4031.9818.9618.3326.6746.5811.3944.2020.0430.5528.61 CaMOPD36.3533.1319.0618.2326.6947.8410.1644.6321.4931.0328.86 Ours40.1031.8817.8119.0627.2158.1311.0652.3526.0336.8932.05 (b) Code× Medical Student Model: Qwen3-4B Method CodeMedical All HumanEval+MBPP+LCB Domain Avg.MedQA MedXpert QAMedMCQA SuperGPQA -Med Domain Avg. Avg. Student76.6865.0817.1452.9764.7313.8055.6329.8741.0146.99 Single OPD84.3070.4428.0060.9177.0615.3162.4436.8147.9054.41 Vanilla MOPD84.7669.1127.7160.5364.9612.7855.6329.6240.7550.64 ExOPD85.9868.9830.0061.6572.8213.6360.4632.8144.9353.29 Open-MOPD84.4570.7727.2960.8376.8315.5961.6835.2547.3454.09 CaMOPD84.9169.5827.4360.6476.1215.3962.4935.1047.2853.96 Ours87.6572.6928.7163.0275.7315.9662.9736.3347.7555.38 Student Model: Qwen3-1.7B Method CodeMedicalAll HumanEval+MBPP+LCB v6 Domain Avg.MedQA MedXpert QAMedMCQA SuperGPQA -Med Domain Avg. Avg. Student60.3753.4412.5742.1346.0310.3342.5522.5830.3736.25 Single OPD73.4859.1322.4351.6853.8113.3950.9924.5035.6743.67 Vanilla MOPD71.9558.2723.0051.0746.1111.0245.6820.9130.9341.00 ExOPD75.7662.3024.43 54.1644.6210.3744.8519.2429.7741.97 Open-MOPD71.9558.6020.4350.3345.8010.7843.4619.6429.9240.12 CaMOPD71.6555.2923.0049.9847.5310.3345.8021.7131.3440.66 Ours73.7861.4424.4353.2256.5612.2451.3525.8836.5144.86 2026b). The Medical teacher is I-Medical-8B, a Qwen3-8B RL checkpoint (Intelligent Internet, 2025). Each paired setting uses 10,000 sampled prompts per domain from DeepMath, Eurus-RL- Code, and Medical sources including the MedMCQA training split (Table 5). Full configurations are in Appendix B. 4.2MAIN RESULTS Best overall performance across domain pairs and student sizes. Across Math × Medical and Code × Medical with both student sizes, ROAD MOPD achieves the highest two-domain average among the compared multi-domain methods in all four settings (Table 1), exceeding the strongest overall baseline in each by 0.72–2.89 points. 7 Paper Under Review Table 2: Low-conflict generalization: Math× Code. Qwen3-4B domain averages (%) and their mean (All). Table 3: Three-domain generalization: Math× Code× Medical. Qwen3-4B domain averages (%) and their mean (All). MethodMathCodeAll Student15.3752.9734.17 Single OPD44.2760.9152.59 Vanilla MOPD46.1261.2453.68 ExOPD48.2861.8255.05 Open-MOPD44.3259.8252.07 CaMOPD46.3060.37 53.34 Ours47.7363.0755.40 MethodMathCodeMedicalAll Student15.3752.9741.0136.45 Single OPD44.2760.9147.9051.03 Vanilla MOPD46.9861.2641.0349.76 ExOPD47.0961.5245.6251.41 Open-MOPD44.0660.5638.8347.82 CaMOPD45.9660.9748.20 51.71 Ours47.2962.9448.4552.89 1 Lowest 2345 Highest Teacher disagreement (S − quintile) 0 25 50 75 100 Raw gradient conflict (%) Medical 76% Math 40% 46 48 52 60 24 20 34 26 Math domain Medical auxiliary ...204minutestotal.Butletmejustcheckagain,maybeImadeamistakewith theequations?Wait,perhapsIshouldmakesurethatthestepsarecorrect. Alternatively,let'sseeifthere'sanotherway.Letmecheckifthevalue... Medical domain Math auxiliary ...,eveniftheattendingsaysnotto.OrperhapsthecorrectanswerisB,the residenthastotelltheattendingthattheycan'tfailtodisclose. Thequestionasks whichisthecorrectnextaction.MaybetheanswerisA? Wait,... λ t 0.5 attenuate1.0 neutral1.5 amplify Figure 3: Teacher disagreement and local supervision. Left: raw gradient conflict by disagree- ment quintile (Math: blue; Medical: orange). Right: Math (top) and Medical (bottom) response windows; blue attenuates (λ t < 1) and orange amplifies (λ t > 1) token supervision. 4B students: Medical recovery accompanies gains in the other domain. In both Math× Med- ical and Code × Medical, Vanilla MOPD lowers Medical accuracy below the initial student’s 41.01, while ExOPD, Open-MOPD, and CaMOPD mitigate this degradation to varying degrees. ROAD MOPD reaches Medical averages of 48.23 and 47.75, respectively, recovering to at or near the Single OPD reference of 47.90 and exceeding the strongest multi-domain Medical baseline in each setting by 0.48 and 0.41 points. It also achieves the highest Math and Code averages in the re- spective settings, indicating that Medical recovery accompanies gains in the other domain’s average performance. 1.7B students: Larger Medical gains improve the cross-domain trade-off. With the smaller student, all multi-domain baselines fall below the Single OPD Medical average of 35.67 in both pairs. ROAD MOPD instead reaches 36.89 and 36.51, respectively, exceeding this reference in both settings. Its margins over the strongest multi-domain Medical baseline in each setting increase to 3.88 and 5.17 points. Although its Math and Code averages trail ExOPD by 1.23 and 0.94 points, respectively, the Medical gains yield two-domain averages that exceed ExOPD by 1.33 and 2.89 points, demonstrating a better overall result when domains are weighted equally. 4.3FURTHER ANALYSIS Generalizing to lower-conflict settings and more domains.Tables 2 and 3 show that ROAD MOPD also improves joint performance in low-conflict and three-domain settings. For Math × Code, it achieves the best Code and second-best Math averages among the compared multi- domain methods. Although its Math average trails ExOPD by 0.55 points, its 1.25-point Code gain yields the highest All Avg. of 55.40, exceeding ExOPD by 0.35 points. With all three domains, ROAD MOPD leads in every domain and achieves the best All Avg. of 52.89, surpassing CaMOPD by 1.18 points. This setting also tests multiple auxiliary teachers, using aggregated agreement for credit assignment and sequential constraints for gradient protection. Full per-benchmark results are in Appendix C.2 (Tables 9 and 10). 8 Paper Under Review −1 0 1 Math × Medical μ= +0.076 μ= −0.529 a PCGradOurs 1204058 Training step −1 0 1 Code × Medical μ= +0.033 μ= −0.391 MathMed.CodeMed. 35 40 50 60 46.8 48.0 60.9 47.6 47.6 48.2 63.0 47.8 Math × Med.Code × Med. b Mean layer cosine Accuracy (%) Figure 4: Preventive projection vs. PCGrad. (a) Mean layerwise cosine; labels average shaded steps 39–58. (b) Domain accuracy (%). Table 4: Ablations of ROAD MOPD. Domain-average accuracy (%). Avg.: mean across the two domains. Math× Med.MathMed.Avg. w/o Credit Assign.45.7847.7446.76 w/o Grad. Protect.47.1948.1147.65 w/o Adaptive Ret.45.7848.4047.09 ROAD MOPD47.6048.2347.92 Code× Med.CodeMed.Avg. w/o Credit Assign.59.5447.4653.50 w/o Grad. Protect.63.0146.8754.94 w/o Adaptive Ret.62.8547.0354.94 ROAD MOPD63.0247.7555.38 Cross-teacher disagreement is associated with gradient conflict, supporting token-level credit assignment. To examine whether cross-teacher agreement reflects update compatibility, we sample response batches at a fixed student checkpoint, averaging max(−s t , 0) (Section 3.2) over all valid tokens per batch to obtain disagreement S − . Within each domain, we sort batches by S − from low to high into five equal-sized groups. Figure 3 (left) plots each group’s fraction of batches with negative cosine between raw full-parameter OPD gradients under the two teachers on identical responses. Conflict rates trend upward from 24% to 40% in Math and 46% to 76% in Medical. This suggests that teacher disagreement can signal potential update conflicts, providing empirical support for agreement-guided credit assignment. Figure 3 (right) visualizes token-level credit assignment: relative to the shared reference, opposing teacher corrections on Alternatively in Math and The in Medical attenuate the routed teacher’s relative reward signal, whereas aligned corrections on perhaps and Maybe in the respective domains amplify it. Probe details are provided in Appendix B.5; full prompts and visualizations of extended responses appear in Appendix C.3 (Figure 5). Preventive projection improves joint learning over post-hoc correction. Figure 4 compares pre- ventive projection with PCGrad under matched training settings; both retain cross-teacher credit assignment and the same norm-retention mechanism. PCGrad corrects conflicts between gradients computed independently on each domain’s own trajectories. Our method instead constrains target- domain updates using auxiliary-teacher gradients on the same rollout. This directly tests compat- ibility between the target gradient and auxiliary-teacher feedback on the same behavior. Unlike PCGrad’s comparison of independent domain gradients, this constraint can address local conflicts that may persist even when those gradients align. Figure 4a shows negative mean layerwise cosines for same-trajectory pairs during much of training, whereas independently formed domain-gradient pairs are mostly non-negative. These patterns reflect different local compatibility criteria and do not directly measure capability damage. Across Math× Medical and Code× Medical, ROAD MOPD improves all four domain-average accuracies over PCGrad by 0.18–2.12 percentage points (Fig- ure 4b). These gains support the utility of same-trajectory constraints for joint learning. Each component improves joint performance. Table 4 shows that removing either credit assign- ment or gradient protection lowers all four domain averages across the two domain pairs. Without credit assignment, Math and Code accuracy drop by 1.82 and 3.48 points, respectively; removing gradient protection reduces Medical accuracy by 0.88 points in Code× Medical. Replacing adap- tive retention with full projection lowers the two-domain mean by 0.83 and 0.44 points in Math× Medical and Code× Medical, respectively, despite a 0.17-point Medical gain in the former. The full method achieves the highest equally weighted two-domain mean in both settings. These results sup- port adaptive retention as a way to balance conflict reduction with target-domain learning, alongside credit assignment and gradient protection. 9 Paper Under Review 5CONCLUSION We identify update incompatibility as a central challenge in multi-teacher on-policy distilla- tion: domain-routed supervision can produce conflicting updates to shared student parameters. ROAD MOPD addresses this challenge using auxiliary teachers to assess the same student tra- jectories. Agreement-aware token weighting and preventive gradient projection suppress conflict- ing supervision and updates while retaining target-domain learning signals. Across mathematics, medicine, and code, ROAD MOPD mitigates high-conflict degradation, retains gains in compati- ble settings, and achieves the best distilled-model average in each domain under joint three-domain training. Our evaluation is limited to three text-based reasoning domains, and auxiliary scoring and gradient computation add training cost. Extending to broader domains and larger teacher pools calls for efficient teacher selection and compatibility estimation. AI USE STATEMENT We used generative AI tools to assist with English translation, language polishing, and the presenta- tion of figures and tables. This assistance included refining wording, improving clarity, and adjusting captions and layouts. The authors retain full responsibility for the paper’s technical content, results, citations, and conclusions, as well as the accuracy and integrity of the final manuscript. REFERENCES Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist,and Olivier Bachem.On-Policy Distillation of Language Mod- els:Learning from Self-Generated Mistakes.In The Twelfth International Con- ference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024. URL https://proceedings.iclr.c/paper_files/paper/2024/hash/ 5be69a584901a26c521c2b51e40a4c20-Abstract-Conference.html. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. HuatuoGPT-o1, towards medical complex reasoning with LLMs. arXiv preprint arXiv:2412.18925, 2024. Tianlei Chen, Jiao Ou, Ziyuan Liu, Ruiming Tang, Jian Liang, and Han Li. Counteraction-aware multi-teacher on-policy distillation for general capability recovery with domain preservation. arXiv preprint arXiv:2605.27115, 2026. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, et al. Process Reinforcement through Implicit Rewards. arXiv preprint arXiv:2502.01456, 2025. URL https: //arxiv.org/abs/2502.01456. DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948, 2025. URL https://arxiv.org/abs/2501. 12948. DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, et al. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348, 2026. URL https://arxiv.org/abs/2606.19348. FreedomIntelligence.Medical-R1-Distill-Data.HuggingFacedataset,2025. URL https://huggingface.co/datasets/FreedomIntelligence/ Medical-R1-Distill-Data. Huan-ang Gao, Haohan Chi, Yong Yan, Shiyuan Feng, Hanlin Wu, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, and Hao Zhou. Open-mopd: Diagnosing and fixing capability imbalance in multi-teacher on-policy distillation. arXiv preprint arXiv:2608.19098, 2026. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models. In Proceedings of ICLR, 2024. Harvard–MIT Mathematics Tournament. Archive of february 2025. Official tournament archive, 2025a. URL https://w.hmmt.org/w/archive/282. 10 Paper Under Review Harvard–MIT Mathematics Tournament. Archive of november 2025. Official tournament archive, 2025b. URL https://w.hmmt.org/w/archive/291. Haonan He, Haodi Lei, Yun Luo, Haoran Zhang, Shunkai Zhang, Yizhuo Li, Shengji Tang, Zhilin Wang, Runzhe Zhan, Lei Bai, Ganqu Cui, Fangchen Yu, Yafu Li, Peng Ye, Ning Ding, and Yu Cheng. Simpleopd: Simple tokenizer-agnostic on-policy distillation for long-context reason- ing. arXiv preprint arXiv:2608.14277, 2026. Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. DeepMath-103K: A large-scale, challenging, decontaminated, and verifiable mathe- matical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456, 2025. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531, 2015. URL https://arxiv.org/abs/1503.02531. Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, Chengquan Zhang, Zhuotao Tian, Han Hu, Yi Yang, Fei Wu, and Hehe Fan. Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe. arXiv preprint arXiv:2605.03677, 2026. URL https://arxiv.org/abs/2605. 03677. Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. m1: Unleash the Poten- tial of Test-Time Scaling for Medical Reasoning with Large Language Models. arXiv preprint arXiv:2504.00869, 2025. URL https://arxiv.org/abs/2504.00869. Intelligent Internet. I-Medical-8B: Medical reasoning model. Hugging Face model, 2025. URL https://huggingface.co/Intelligent-Internet/I-Medical-8B. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What dis- ease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009.13081, 2020. Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, p. 1317–1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1139. URL https://aclanthology. org/D16-1139/. Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se- Young Yun. DistiLLM-2: A contrastive approach boosts the distillation of LLMs. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 31044–31062. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/ko25a.html. Yuying Li, Leqi Zheng, Yongzi Yu, Wenrui Zhou, Xuchang Zhong, Xing Hu, Jing Jin, Hangjie Yuan, and Tao Feng. Filter, then reweight: Rethinking optimization granularity in on-policy distillation. arXiv preprint arXiv:2606.02684, 2026a. Zhaoyi Li, Deyang Kong, Yuan Wei, Evan Yang, Ranran Shen, Mahardika Krisna Ihsani, Ming Yang, Wei Zhang, Chuan Hao, Jian Yang, Ran Tao, Bryan Dai, Shikun Zhang, Wei Ye, Ying Wei, and Defu Lian. Every coin has two sides: On the dual nature of generalization in on-policy distillation of large language models. arXiv preprint arXiv:2608.16647, 2026b. Yiqing Liang, Jielin Qiu, Wenhao Ding, Zuxin Liu, James Tompkin, Mengdi Xu, Mengzhou Xia, Zhengzhong Tu, Laixi Shi, and Jiacheng Zhu. Modomodo: Multi-domain data mixtures for multimodal llm reinforcement learning. arXiv preprint arXiv:2505.24871, 2025. 11 Paper Under Review Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, Shenzhi Wang, Bingxiang He, and Gao Huang. On-policy distillation with verifiable reward. arXiv preprint arXiv:2608.24696, 2026. Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Conference on Neural Information Processing Systems (NeurIPS), 2021, December 2021. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.Is Your Code Gen- erated by ChatGPT Really Correct?Rigorous Evaluation of Large Language Models for Code Generation. In Thirty-Seventh Conference on Neural Information Processing Systems, 2023.URL https://proceedings.neurips.c/paper_files/paper/2023/ hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract.html. M-A-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, et al. SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines. arXiv preprint arXiv:2502.14739, 2025. URL https://arxiv.org/abs/2502.14739. Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, Jinhao Dong, Zhifang Sui, and Fuli Luo. Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training. arXiv preprint arXiv:2606.30406, 2026. Mathematical Association of America. 2024 AIME I Problems. Problem sets archived by Art of Problem Solving, 2024. URL https://artofproblemsolving.com/wiki/index. php?title=2024_AIME_I_Problems. Mathematical Association of America. 2025 AIME I Problems. Problem sets archived by Art of Problem Solving, 2025. URL https://artofproblemsolving.com/wiki/index. php?title=2025_AIME_I_Problems. OpenAI. GPT-4 Technical Report. Technical Report arXiv:2303.08774, OpenAI, 2023. URL https://arxiv.org/abs/2303.08774. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, p. 248–260. PMLR, 2022. URL https://proceedings.mlr.press/v174/ pal22a.html. Bing Shao, Jiazheng Zhang, Long Ma, Yujiong Shen, Senjie Jin, Xin Guo, Yuming Yang, Mingxu Chai, Zhiheng Xi, Boyang Liu, Junlin Shang, Tao Gui, Qi Zhang, and Xuanjing Huang. A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation. arXiv preprint arXiv:2608.25643, 2026. URL https://arxiv.org/abs/2608.25643. Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu, Feng Xiao, Lu Liu, Yanlong Du, Zifeng Cheng, Ziwei Jiang, and Qing Gu. Step-level on- policy distillation: Interpolating between on-policy distillation and supervised fine-tuning. arXiv preprint arXiv:2608.16333, 2026a. Zechen Sun, Zhiwei Zhang, Fei Zhao, Juntao Li, Mu Chuan, Huayu Deng, Guojian Zhan, Wenliang Chen, Yao Hu, and Min Zhang. D 3 -MOPD: Dynamic Domain ScheDuling for Efficient Multi- Teacher Distillation. arXiv preprint arXiv:2608.24987, 2026b. URL https://arxiv.org/ abs/2608.24987v2. Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, Yihan Cao, Hui Ren, Xiang Li, Xiaoxiao Li, and Yuyin Zhou. MedReason: Eliciting factual medical reasoning steps in LLMs via knowledge graphs. arXiv preprint arXiv:2504.00993, 2025. Xiaomi LLM-Core Team, Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, et al. MiMo-V2-Flash Technical Report. arXiv preprint arXiv:2601.02780, 2026. URL https:// arxiv.org/abs/2601.02780. 12 Paper Under Review Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798– 69818, 2023. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, et al. Qwen3 Tech- nical Report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/ 2505.09388. Chen Yang, Haiyuan Wan, Rengrong Xiong, Yize Chen, and Danny H. K. Tsang. Beyond imitation: Filtering on-policy distillation by reasoning progress. arXiv preprint arXiv:2608.19408, 2026a. Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125, 2026b. URL https://arxiv.org/abs/2602.12125. Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, Renjie Pi, Grace Lam, Nayeon Lee, Alexander Bukharin, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation. arXiv preprint arXiv:2603.19220, 2026c. URL https://arxiv.org/abs/2603.19220. Qixiang Yin, Huanjin Yao, Yuchen Cai, Jianghao Chen, Ziyi Wang, Min Yang, Fei Su, and Zhicheng Zhao. H-opd: Confidence aware heterogeneous multi-teacher multimodal on-policy distillation. arXiv preprint arXiv:2607.02592, 2026. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824–5836, 2020. Zhu Zhang, Jixun Wang, Xiaoang Xu, Xiaorong Wang, Zihan Zhou, Zhiyuan Wang, Shuo Wang, Chaojun Xiao, and Yuezhi Zhou. Beyond teacher likelihood: Group-calibrated on-policy distilla- tion for long-context reasoning. arXiv preprint arXiv:2608.19181, 2026. Yuxin Zuo, Shang Qu, Yifei Li, Zhang-Ren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. MedXpertQA: Benchmarking expert-level medical reasoning and un- derstanding. In Proceedings of the 42nd International Conference on Machine Learning, vol- ume 267 of Proceedings of Machine Learning Research, p. 80961–80990. PMLR, 2025. URL https://proceedings.mlr.press/v267/zuo25a.html. 13 Paper Under Review APPENDIX CONTENTS Page AAdditional Method Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.1OPD Derivation and Training Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.2Derivation of Norm-Constrained Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.3Complete Training Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 BExperimental Setup and Reproducibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.1Models and Teacher–Student Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.2Training Data and Prompt Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.3Training Hyperparameters and Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 B.4Baseline Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 B.5Evaluation Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 CAdditional Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 C.1Detailed Benchmark Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 C.2Full Results for Low-Conflict and Three-Domain Settings . . . . . . . . . . . . . . . . . . . . . . . . . . 23 C.3Extended Token-Level Visualizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.4Training Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 DLimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 14 Paper Under Review AADDITIONAL METHOD DETAILS A.1OPD DERIVATION AND TRAINING LOSS From sequence-level reverse KL to token-level supervision. We expand the OPD reformulation in Section 3.1 before specifying its sampled-token implementation. For a prompt x ∼ D, let y = (y 1 ,...,y T ) ∼ π θ (· | x) and h t = (x,y <t ). Write p t = π θ (· | h t ) and q t = π T (· | h t ) for the student and teacher distributions on the same prefix. WithE θ denoting expectation over these prompts and student-generated responses, autoregressive factorization and the KL chain rule give L OPD (θ) =E x∼D D KL (π θ (·| x)∥π T (·| x)) =E θ log π θ (y | x) π T (y | x) =E θ " T X t=1 log p t (y t ) q t (y t ) # =E θ " T X t=1 D KL (p t ∥q t ) # . (12) The expectations assume finite divergences and common policy support. Thus, the token-level teacher distributions are evaluated on prefixes visited by the student. Equivalent relative-reward objective. Introducing a frozen reference π ref and adding and sub- tracting its log-probability exposes the teacher’s relative reward (Yang et al., 2026b): −L OPD (θ) =E θ [logπ T (y | x)− logπ θ (y | x)] =E θ logπ T (y | x)− logπ ref (y | x) − logπ θ (y | x)− logπ ref (y | x) =E θ log π T (y | x) π ref (y | x) −E x∼D D KL (π θ (·| x)∥π ref (·| x))≡J 1 (θ). (13) Consequently, arg min θ L OPD (θ) = arg max θ J 1 (θ): the reward-maximization objective is the negative of the reverse-KL loss. Scaling only the relative-reward term by λ > 0 gives the objective in Eq. (2), whose sampled-response form is J λ (θ) =E θ λ log π T (y | x) π ref (y | x) − log π θ (y | x) π ref (y | x) =E θ " T X t=1 λ ℓ T t − ℓ ref t − ℓ θ t − ℓ ref t # , (14) where ℓ T t , ℓ θ t , and ℓ ref t are the teacher, student, and reference log-probabilities of the sampled token. At λ = 1, the reference terms cancel and recover OPD; 0 < λ < 1 attenuates the relative reward, whereas λ > 1 amplifies it (ExOPD). Domain routing applies this construction to each teacher– domain pair in Eq. (3). Credit assignment with multiple teachers. For the K > 2 extension of Section 3.2, let P d = j | j ̸= d be the peer teachers for target domain d. Here we use explicit teacher indices: for each peer j, the main-text roles are T r = T d and T a = T j , with ℓ r t = ℓ d t and ℓ a t = ℓ j t . All peers score the same trajectory as the routed teacher, and their token agreements are averaged: ̄s d,t = 1 K− 1 X j∈P d tanh ℓ d t − ℓ ref t τ tanh ℓ j t − ℓ ref t τ ! .(15) We obtain λ d,t by applying Eq. (5) with s t = ̄s d,t . For K = 2, this recovers Eqs. (4)–(5). For K > 2, the K − 1 peers reinforce or attenuate the routed signal according to their mean evidence; opposing assessments can cancel, recovering standard OPD when ̄s d,t = 0. Sampled-token learning signal. The implementation replaces the fixed reward scale in Eq. (14) with the agreement coefficient from Section 3.2, using the sampled-token k 1 log-ratio estimator and a PPO-style policy surrogate. Let π old denote the student policy that generates a stored response, 15 Paper Under Review and let ℓ θ i,t = logπ θ (y i,t | h i,t ) and ℓ old i,t = logπ old (y i,t | h i,t ). For a response from domain d, the routed teacher and shared frozen reference define ∆ d i,t = ℓ d i,t − ℓ ref i,t . The advantage supplied to the policy loss is A agr d,i,t = sg clip λ d,i,t ∆ d i,t − ℓ θ i,t − ℓ ref i,t ,−10, 10 ,(16) where sg denotes stop-gradient. The entire recomputed advantage is clipped to [−10, 10] after agreement modulation and detached before entering the policy surrogate, including its student log- probability term. Teacher and reference parameters are frozen throughout training. When λ d,i,t = 1, Eq. (16) reduces to the plain OPD learning signal sg[clip(ℓ d i,t −ℓ θ i,t ,−10, 10)]. Thus, the coefficient rescales the teacher’s reward relative to the reference; it does not multiply the entire OPD loss. Clipped policy surrogate and masking. The importance ratio and per-token loss are r i,t (θ) = exp ℓ θ i,t − ℓ old i,t ,(17) ℓ agr d,i,t (θ) =− min n r i,t (θ)A agr d,i,t , clip(r i,t (θ), 1− ε PPO , 1 + ε PPO )A agr d,i,t o .(18) Let M i,t indicate a valid response token and T i = P t M i,t . The target-domain loss is L agr d = 1 |B d | X i∈B d P t M i,t ℓ agr d,i,t T i .(19) Prompt and padding positions do not contribute to the loss. The recorded training configuration uses ε PPO = 0.2, without an outcome-based task reward or a separate KL reward added by the trainer. The single-turn rollout mask retains the returned generated tokens, including EOS when returned, and excludes padding. A response reaching the length cap retains supervision on its available prefix. Fully masked responses contribute zero. The agreement coefficient λ d,i,t is bounded by [0.5, 1.5]. Auxiliary supervision. For each peer teacher j ̸= d, we reuse the same stored domain-d responses and replace the advantage by A OPD d→j,i,t = sg[clip(ℓ j i,t − ℓ θ i,t ,−10, 10)]. The auxiliary loss uses the same clipped surrogate, with agreement modulation disabled and the same advantage clipping range. For the selected auxiliary teacher T a = T j , we write the main-text loss L ′ d→a as L ′ d→j . The full method and all ablations retain this range, including when credit assignment is disabled. Advantage clipping is separate from the importance-ratio clipping in Eq. (18). A.2DERIVATION OF NORM-CONSTRAINED PROJECTION For nonzero g and g ′ , let c = ⟨g,g ′ ⟩/(∥g∥ 2 ∥g ′ ∥ 2 ). When c < 0, every candidate inS(g,g ′ ) can be written as v(α) = g− α ⟨g,g ′ ⟩ ∥g ′ ∥ 2 2 g ′ ,0≤ α≤ 1.(20) The objective in Eq. (8) is−(1−α)⟨g,g ′ ⟩, which strictly decreases with α. The optimum therefore uses the largest feasible coefficient. Expanding the squared norm gives ∥v(α)∥ 2 2 ∥g∥ 2 2 = 1− (2α− α 2 )c 2 .(21) This decreases monotonically on [0, 1]. Full projection is feasible if √ 1− c 2 ≥ m; otherwise, the optimum satisfies 1− (2α− α 2 )c 2 = m 2 . Including the no-conflict case gives α ⋆ =        0,c≥ 0, 1,c < 0 and √ 1− c 2 ≥ m, 1− r 1− 1− m 2 c 2 , c < 0 and √ 1− c 2 < m. (22) Substitution into Eq. (9) recovers the unique solution of Eq. (8) for a conflicting pair. The norm constraint trades complete conflict removal for retention of the target-domain signal. In particular, ⟨eg,g ′ ⟩ = (1− α ⋆ )⟨g,g ′ ⟩,(23) 16 Paper Under Review which remains negative when c < 0 and α ⋆ < 1. These statements concern the projected tensor before domain aggregation, global clipping, and AdamW preconditioning. They do not guarantee a non-increasing auxiliary loss after the final optimizer step, nor a lower bound on the norm of the merged update. Multi-teacher projection and shared norm retention. For K > 2, each peer j ∈ P d induces a same-trajectory auxiliary gradient g ′ d→j , computed as in Eq. (6). For each parameter tensor, we select the remaining valid peer with the most negative cosine, project against it, and recompute the cosines; processing stops when no valid negative cosine remains. These projections use the norm- constrained form of Eq. (9) with a shared floor relative to the original target tensor: ∥g (q) d ∥ 2 ≥ m∥g (0) d ∥ 2 , q = 1,...,K− 1,(24) where g (0) d is the original target tensor and g (q) d is the tensor after q projections. Once processing is complete, the resulting tensors formeg d . The floor is measured against the original tensor throughout the sequence. Applying a fresh local retention ratio m independently at every projection would provide only a product bound and would not implement this shared constraint. In addition, a later projection can change alignment with a previously processed peer; sequential projection alone does not establish simultaneous non-conflict with every peer. For each conflicting peer, the implementation computes the coefficient from the remaining squared- norm budget. Write a = ∥g (q−1) d ∥ 2 2 , b = m 2 ∥g (0) d ∥ 2 2 , and u = ⟨g (q−1) d ,g ′ ⟩ 2 /∥g ′ ∥ 2 2 . It uses α = 1 if a− u≥ b; otherwise, α = 1− p 1− clip((a− b)/u, 0, 1). Thus, a tensor that has exhausted its shared budget receives no further projection. These comparisons use numerical tolerance 10 −12 . A.3COMPLETE TRAINING PROCEDURE Algorithm 1 combines teacher scoring, token modulation, and all-peer gradient projection within one optimizer step, using the domain-mean gradient convention of the main text. All target and auxiliary gradients are evaluated at the same student parameters on stored trajectories; only projected target gradients enter the update. Batch normalization is detailed below, and archived projection configurations are specified in Appendix B.3. Token operations are elementwise over the valid responses in B d ; stored scores and masks are re- stricted to that domain when used. MASKEDLOSS computes the loss in Eqs. (18) and (19) using valid response tokens. TENSORWISEPROJECT applies Eq. (9) separately to each parameter ten- sor. PROJECTPEERSSHAREDFLOOR denotes the sequential multi-peer procedure detailed in Ap- pendix A.2, subject to the shared norm floor in Eq. (24). Auxiliary gradients are discarded after projection and never merged into g update . Domain means and global-batch normalization. Equation (19) and the algorithm define each gradient from a domain mean, and the algorithm averages the projected gradients across domains as in Eq. (11). In the implementation path supplied with full-batch normalization metadata, masking or slicing to domain d retains the denominator|B|. After microbatch accumulation and data-parallel averaging, its loss contribution is therefore L batch d = w d L agr d , where w d = |B d |/|B|; auxiliary losses on the same responses receive the same factor. Scaling both gradients by w d > 0 preserves the projection cosines and retention coefficients in exact arithmetic. To express this implementation using the algorithm’s domain-mean gradients, replace its accumulation by g update ← g update + w d eg d . Balanced batches give w d = 1/K, recovering the equal-weight average in Eq. (11); shuffled batches use their observed domain fractions. For imbalanced batches, these weights differ from 1/K before global clipping and AdamW, so the two weighting conventions are not assumed to produce identical optimizer steps. Projection implementation. The paired-domain implementation projects each trainable FSDP parameter tensor separately. Gradient snapshots are stored in BF16, while dot products and squared norms are computed in FP64 and reduced across the data-parallel group. Shards of the same tensor therefore use a common cosine and projection coefficient. Tensors with negligible gradient norms are left unchanged; the numerical stability constant is 10 −12 . With multiple peers, each tensor selects the most negative cosine among its unprocessed peers and recomputes these cosines after 17 Paper Under Review Algorithm 1 One optimizer step of ROAD MOPD Require: Student parameters θ, optimizer stateS, rollout policy π old Require: Frozen teachersπ T d K d=1 , frozen reference π ref , labelled promptsX Require: Modulation parameters (τ,ρ g ,λ min ,λ max ), retention m, clip threshold γ Ensure: Updated student parameters θ and optimizer stateS 1: B,ℓ old ,M ← ROLLOUT(π old ,X ) 2: ℓ 1:K ,ℓ ref ← SCORESAMETRAJECTORIES(π T d K d=1 ,π ref ,B) 3: g update ← 0 4: for d = 1,...,K do 5: B d ←i∈B : domain(i) = d;P d ←1,...,K\d 6: ℓ θ ← STUDENTLOGPROBS(π θ ,B d ); r ← exp(ℓ θ − ℓ old ) 7: ̄s d,i,t ← 1 K−1 P j∈P d tanh (ℓ d i,t − ℓ ref i,t )/τ tanh (ℓ j i,t − ℓ ref i,t )/τ 8: λ d,i,t ← clip(1 + ρ g ̄s d,i,t ,λ min ,λ max ) 9: A agr d,i,t ← sg clip λ d,i,t (ℓ d i,t − ℓ ref i,t )− (ℓ θ i,t − ℓ ref i,t ),−10, 10 10: L agr d ← MASKEDLOSS(r,A agr d ,M d ) 11: g d ←∇ θ L agr d 12:for j ∈P d do 13:A OPD d→j,i,t ← sg[clip(ℓ j i,t − ℓ θ i,t ,−10, 10)] 14: L ′ d→j ← MASKEDLOSS(r,A OPD d→j ,M d ) 15:g ′ d→j ←∇ θ L ′ d→j 16:end for 17:if K = 2 then 18:eg d ← TENSORWISEPROJECT(g d ,g ′ d→j ,m), whereP d =j 19:else 20:eg d ← PROJECTPEERSSHAREDFLOOR(g d ,g ′ d→j j∈P d ,m) 21:end if 22: g update ← g update + 1 K eg d 23: end for 24: g update ← GLOBALNORMCLIP(g update ,γ) 25: (θ,S)← ADAMWSTEP(θ,S,g update ) 26: return θ,S each projection. Ties are broken by the configured peer order. Peers with negligible norms are skipped, and processing stops when no valid negative cosine remains. A selected peer is removed from the candidate set, even if the norm budget forces its coefficient to zero; previously selected peers are not revisited. BEXPERIMENTAL SETUP AND REPRODUCIBILITY B.1MODELS AND TEACHER–STUDENT CONFIGURATIONS Section 4.1 introduces the student and domain teachers. The specific Math and Code teacher check- points are Qwen3-4B-Non-Thinking-RL-Math-Step500 and Qwen3-4B-Non-Thinking-RL-Code- Step300, respectively (Yang et al., 2026b). The medical teacher is I-Medical-8B, a reinforcement- learning (RL) checkpoint based on Qwen3-8B (Intelligent Internet, 2025). The student undergoes full-parameter training; all teachers and the shared reference remain frozen and are used only for scoring. The models use compatible Qwen3 tokenizers and vocabularies, allowing the scorers to evaluate identical response token IDs at identical prefixes. Scoring checks require matching token IDs and log-probability tensor shapes. B.2TRAINING DATA AND PROMPT CONSTRUCTION Each paired training set contains 10,000 prompts per domain, for 20,000 prompts in total, and is randomly shuffled before training. For Math, we filter DeepMath (He et al., 2025) to samples with difficulty level≥ 6, yielding a pool of 57K candidates, and sample 10,000 training prompts from this pool. For Code, we sample 10,000 prompts from Eurus-RL-Code (Cui et al., 2025). The Medical pool contains 10,000 prompts from four sources, as detailed in Table 5, including 4,000 sampled 18 Paper Under Review from the MedMCQA training split (Pal et al., 2022). The MedMCQA validation split is used only for evaluation, not for constructing the training pool. Table 5: Composition of the training prompt pools. Counts denote sampled training prompts. Each paired setting uses the two corresponding domain pools. DomainSourceSampled prompts MathDeepMath (He et al., 2025), difficulty level≥ 6 (57K candidates)10,000 CodeEurus-RL-Code (Cui et al., 2025)10,000 MedicalMedMCQA training split (Pal et al., 2022)4,000 MedReason (Wu et al., 2025)3,000 Medical-R1-Distill-Data (FreedomIntelligence, 2025; Chen et al., 2024)2,000 m23k (Huang et al., 2025)1,000 Domain labels select the routed teacher; only the prompt is supplied as the model input. Ground- truth answers are not used as outcome rewards in the distillation loss. Medical prompts request step-by-step reasoning with the final answer enclosed in . Thinking mode is disabled through the chat template, although generated responses can still contain intermediate reasoning. B.3TRAINING HYPERPARAMETERS AND INFRASTRUCTURE Table 6 records the shared optimization and rollout settings of the archived ROAD MOPD runs. The implementation uses verl with an FSDP student and vLLM for rollout generation and teacher scoring. Baseline-specific update schedules are described in Appendix B.4. Table 6: Shared optimization and rollout settings of ROAD MOPD. Each rollout batch produces one optimizer update. HyperparameterValue Training / PPO minibatch size1,008 / 1,008 prompts Rollout batches (paired / three-domain)58 / 116 PPO epochs per rollout batch1 Responses per prompt1 Rollout temperature / top-p1.0 / 1.0 Maximum prompt / response length2,048 / 16,384 tokens OptimizerAdamW Learning rate / schedule10 −5 / constant, no warm-up Adam betas / weight decay(0.9, 0.999) / 0.01 Global gradient norm clipping1.0, after merging domain gradients Advantage clipping[−10, 10], full method and all ablations PPO ratio-clipping parameter0.2 Distillation estimatorSampled-token k 1 , policy-gradient surrogate Task reward / separate trainer KL rewardDisabled / disabled Checkpoint interval29 optimizer steps In-training evaluationDisabled; offline checkpoint evaluation The shared modulation parameters are τ = 1, ρ g = 0.5, and [λ min ,λ max ] = [0.5, 1.5], with 10 −12 for numerical stability. Paired-domain training uses bidirectional projection with m = 0.6. Because agreement lies in [−1, 1], the unbounded mapping 1 + 0.5s t already lies in [0.5, 1.5]; clipping does not further restrict this mapping in exact arithmetic. The paired- and three-domain configurations use one node with eight NVIDIA B200 GPUs. Six GPUs host the FSDP student and three rollout replicas with tensor-parallel size two. The remaining two GPUs host teacher scoring. In paired settings, one 4B teacher shares a GPU with the frozen reference; in three-domain training, the Math teacher, Code teacher, and reference share one GPU, while the Medical teacher occupies the other. Gradient checkpointing, padding removal, and dy- namic batching are enabled. The three-domain ROAD MOPD sampler forms balanced batches of 336 prompts per domain. 19 Paper Under Review The audited ExOPD, Open-MOPD, and CaMOPD runs use 1,008 prompts per rollout batch, with 58 batches in paired settings and 116 in three-domain settings. ExOPD and CaMOPD perform one optimizer update per batch. Open-MOPD divides that batch into four sequential minibatches of 252 prompts, giving 232 and 464 optimizer updates, respectively, while retaining one response per prompt. Thus, matching rollout budgets does not imply matching optimizer-update counts; CaMOPD also changes the allocation of prompts across domains through its alternating schedule. Student sizes, reference models, and Single OPD. For both ROAD MOPD and ExOPD, the reference is a frozen copy of the student’s initialization: Qwen3-4B in 4B experiments and Qwen3- 1.7B in 1.7B experiments. For Qwen3-1.7B, Table 1(a)–(b) reports multi-domain checkpoints at step 58; Student denotes the original, undistilled model. Open-MOPD uses the sampled-token k 1 estimator. Within each student size, Math, Code, and Medical Single OPD share the same training configuration and differ only in their domain-specific datasets. All three Qwen3-1.7B Single OPD runs start from the original model and train for 50 steps, evaluated with Avg@32, Avg@4, and Avg@1, respectively. Qwen3-4B Single OPD uses the matching in-domain checkpoints in Table 8. B.4BASELINE IMPLEMENTATIONS We implement the baselines within the same MyOPD/verl training stack, using the corresponding Qwen3-4B or Qwen3-1.7B initialization and domain teachers introduced in Section 4.1. Teacher scoring uses the sampled tokens of student-generated responses. The baseline paths use the sampled- token k 1 reverse-KL estimator and a clipped policy-gradient surrogate with clipping parameter 0.2; task rewards, entropy regularization, and a separate trainer KL reward are disabled. The method- specific implementations are as follows. Vanilla MOPD. Each prompt is routed to its domain teacher (Ma et al., 2026). The token advan- tage logπ T d (y t | h t )− logπ θ (y t | h t ) is clipped to [−10, 10] and detached, without cross-teacher modulation or gradient projection. The archived Math–Code and Code–Medical runs each use 58 optimizer updates, while the three-domain run uses 116. ExOPD. We use a fixed extrapolation coefficient of 1.25. The reference is the frozen student initialization, Qwen3-4B or Qwen3-1.7B according to student size (Yang et al., 2026b). For each sampled token, the detached advantage is 1.25(logπ T d − logπ ref )− (logπ θ − logπ ref ), with all log probabilities computed at the same response prefix. The cross-teacher gate, gradient projection, and domain-balancing modules are disabled for this baseline. CaMOPD. The paired-domain implementation alternates homogeneous domain batches and se- lects examples using the rollout-time gap ∆ t = logπ T d (y t | h t )− logπ old (y t | h t ) (Chen et al., 2026). Recovery examples are scored by the mean of |∆ t |, while preservation examples use the mean of max(∆ t , 0). Within each branch, examples are ranked by this score and the smallest prefix covering 80% of the total score mass is retained. For Medical–Math and Medical–Code, Medical is the recovery branch: the update ratio is 3:1, and selected-example advantage scales are 2.0 for Medical and 1.0 for the other domain. Math–Code uses Math as recovery and Code as preservation, a 1:1 update ratio, and unit scales for both branches. The three-domain adaptation treats domains as peer branches, using the positive-gap score, the same 0.8 score-mass threshold, and unit selected- example weights. We sample Medical, Math, and Code data in a 3:1:1 ratio, prioritizing the weaker Medical domain in line with CaMOPD’s capability-recovery objective. Open-MOPD. We adapt its balancing and reward-refresh mechanisms to the sampled-token OPD path (Gao et al., 2026). At rollout time, a domain’s weight is proportional to its uniform target share divided by its observed response-token share, multiplied by its mean absolute teacher–student gap relative to the mean gap across domains. This gap factor uses exponent 1.0 and is clipped to [0.05, 20]. The weights are normalized to have token-weighted mean one and remain fixed within the rollout batch. Dynamic allocation is therefore implemented through domain loss weights. Each batch is split into four sequential optimizer minibatches. On each minibatch, the OPD advantage is recomputed using the current student’s log probabilities and cached teacher scores, providing reward refresh. 20 Paper Under Review PCGrad. For the two-domain comparison (Yu et al., 2020), separate backward passes mask the shared minibatch to each domain’s own responses. For each trainable parameter tensor, the imple- mentation computes the two gradients’ dot product and squared norms across FSDP shards. If the dot product is negative, both gradients are projected against the original opposing gradient; the two corrected gradients are then summed. This symmetric two-task calculation requires no random task ordering. Gradient snapshots use BF16 and dot products and squared norms use FP64. The compar- ison applies global gradient clipping after merging, followed by one optimizer step. These gradients are formed on their respective domains’ responses, following the independent-domain comparison in Section 4.3; same-trajectory auxiliary gradients are not used for this projection. The archived PCGrad runs whose benchmark scores appear in Table 7 also enable reward-agreement modulation and are marked with a dagger in that table. B.5EVALUATION PROTOCOLS We evaluate 11 benchmarks across Code (HumanEval+, MBPP+, and LiveCodeBench) (Liu et al., 2023; Jain et al., 2024), Math (AIME 2024/2025 and HMMT February/November 2025) (MAA, 2024; 2025; Harvard–MIT Mathematics Tournament, 2025a;b), and Medical (MedQA, MedX- pertQA, MedMCQA, and SuperGPQA-Med) (Jin et al., 2020; Zuo et al., 2025; Pal et al., 2022; M-A-P Team et al., 2025). Generation setup. Offline evaluation uses vLLM with temperature 1.0, top-p 1.0, and a max- imum of 16,384 generated tokens per completion.Chat prompts disable thinking mode via enable_thinking=False. Mathematics. Each of the four Math test sets contains 30 questions, with 32 sampled completions per question. We append the instruction “Please reason step by step, and put your final answer within .” The evaluator extracts the last boxed answer and uses math-verify to compare it with the reference answer; a missing or malformed boxed answer is incorrect. The reported Avg@32 is the mean correctness over completions, averaged over questions: Avg@32 = 1 N N X i=1 1 32 32 X r=1 1[ˆa i,r = a i ].(25) Here equality denotes equivalence under the answer verifier. Avg@32 equals the pass@1 estimate computed from the same 32 samples. This differs from pass@32, which counts a question as solved when at least one of its 32 completions is correct. Code. All three Code benchmarks use four completions per problem. We report Avg@4, the fraction of passing completions averaged over problems, which equals the pass@1 estimate from the same four samples. HumanEval+ contains 164 problems and MBPP+ contains 378. A completion must pass both the original and additional EvalPlus tests to count as correct. Prompts request a Python solution in a fenced code block. Our local evaluation protocol (v3) prioritizes Python or untagged Markdown code blocks, selects the candidate with the most nonempty lines, and uses tree- sitter to retain the target entry point and its dependencies. Generation continues until EOS or the token limit, without stopping at the first closing code fence. EvalPlus batches up to 256 generation requests across problems and sets each test’s execution limit to max(5 s, 6t ref ), where t ref is the reference solution’s runtime for that test. LiveCodeBench. We use the v6 code-generation split: 175 problems with contest dates from January 4 to April 6, 2025, giving 700 completions in total. The local loader selects test6.jsonl, with no additional date filtering. The runner uses the Qwen3-4B non-thinking prompt template and extracts the final fenced code block. Each completion is checked against the problem’s public and private tests with a 60-second per-test timeout. Compilation failures, runtime errors, timeouts, and failed tests count as incorrect completions. Medical. The evaluation uses one completion per question and reports Avg@1 multiple-choice ac- curacy. It covers the four-option MedQA-USMLE test set (1,273 questions), MedXpertQA Text test set (2,450; ten options), MedMCQA validation set (4,183; four options), and the SuperGPQA sub- set selected by discipline=Medicine (2,755; ten options). Each prompt includes the question 21 Paper Under Review and labeled options, and requests step-by-step reasoning with the final answer letter in . The parser first checks the last boxed letter, then falls back to an explicit answer phrase such as “An- swer: B” or a terminal “(B)”. Letters are normalized to uppercase and must belong to the question’s available options. Exact match to the reference letter determines correctness; unparseable answers are incorrect. Aggregation. All scores are reported as percentages. Each benchmark weights its questions equally, and domain averages are arithmetic means over the benchmarks within that domain, ir- respective of their different numbers of questions. The 11-task average in Table 8 weights all 11 benchmarks equally; it is distinct from averaging the three domain means, because Code contains three benchmarks and the other domains contain four. Teacher-disagreement and gradient-conflict probe. For the fixed-checkpoint probe in Figure 3 (left), we compute S − = mean t [max(−s t , 0)] over all valid response tokens pooled across each four-trajectory batch, using Eq. (4) with τ = 1. We partition batches into five within-domain quin- tiles, with 50 batches per quintile. A raw conflict is a negative cosine between the full-parameter routed- and auxiliary-teacher OPD gradients on the same batch, without token modulation, projec- tion, or optimizer updates. This five-bin analysis is exploratory; shading connects pointwise 95% Wilson interval limits. The probe is separate from the token-visualization samples in Appendix C.3. CADDITIONAL EXPERIMENTAL RESULTS C.1DETAILED BENCHMARK RESULTS Table 7 provides available benchmark breakdowns for the ablations and PCGrad comparison. Table 7: Per-benchmark breakdown of the ablations and PCGrad comparison. Avg. is the benchmark mean; rounding may cause small differences from the mean of displayed scores. Scores are percentages (higher is better). (a) Math× Medical MathMedical MethodAIME24 AIME25 HMMT 25 Feb. HMMT 25 Nov.Avg.MedQA Med Xpert QA Med- MCQA GPQA MedAvg. w/o Credit Assign.59.6953.5432.7137.1945.7876.9815.8862.0836.0147.74 w/o Grad. Protection61.4654.4833.7539.0647.1976.0415.3963.2837.7148.11 w/o Adaptive Retention58.1355.4233.0236.5645.7877.5315.5963.6136.8448.40 PCGrad † 62.6053.5433.7537.5046.8577.9314.6562.5436.9948.03 ROAD MOPD63.8553.9633.6538.9647.6077.0615.6762.8337.3548.23 (b) Code× Medical CodeMedical Method Human Eval+MBPP+LCBAvg.MedQA Med Xpert QA Med- MCQA GPQA MedAvg. w/o Credit Assign.84.1569.0525.4359.5476.2016.0861.7735.7947.46 w/o Grad. Protection87.6571.9629.4363.0175.1814.1661.5336.6246.87 w/o Adaptive Retention86.7473.6828.1462.8576.1214.0862.2535.6847.03 PCGrad † 84.4570.1128.1460.9076.9015.7162.4935.1747.57 ROAD MOPD87.6572.6928.7163.0275.7315.9662.9736.3347.75 † The archived PCGrad runs also enable reward-agreement modulation. Single-domain reference results. Table 8 lists the available single-domain checkpoints across all 11 benchmarks, complementing Figure 1. 22 Paper Under Review Table 8: Cross-domain evaluation of single-domain training checkpoints. Each Qwen3-4B single-domain model is trained with OPD on 10,000 in-domain prompts for 3 epochs and evalu- ated across all 11 benchmarks. The best and second-best results among the three checkpoints are shown in bold and underlined, respectively. BenchmarkTeacherStudentMath-onlyMedical-onlyCode-only Code HumanEval+85.9876.6881.4081.1084.30 MBPP+79.6365.0869.7172.9570.44 LCB v627.0017.1422.0022.1428.00 Avg.64.2052.9757.7058.7360.91 Math AIME2457.7121.4658.1359.3843.75 AIME2555.0021.7751.6746.15 37.29 HMMT25 Feb.32.2910.0031.5630.3120.00 HMMT25 Nov.39.278.2335.7336.5624.79 Avg.46.0715.3744.2743.1031.46 Medical MedQA87.9064.7370.8677.0667.71 MedXpertQA25.7113.8014.9015.3114.00 MedMCQA71.3855.6360.1762.4457.33 SuperGPQA-Med47.1929.8734.7036.8133.32 Avg.58.0541.0145.1647.9043.09 11-Task Avg.55.3734.9448.2649.1143.72 C.2FULL RESULTS FOR LOW-CONFLICT AND THREE-DOMAIN SETTINGS Tables 9 and 10 provide the full per-benchmark Qwen3-4B results underlying the summaries in Tables 2 and 3, respectively. Domain averages weight benchmarks equally. Single OPD uses the matching in-domain checkpoints from Table 8. Table 9: Full results for low-conflict Math× Code. Per-benchmark Qwen3-4B scores underlying Table 2. Scores are percentages (higher is better); Avg. weights benchmarks equally within each domain. Bold and underlining mark the best and second-best multi-domain methods, respectively. BenchmarkStudent Single OPD Vanilla MOPDExOPD Open- MOPDCaMOPDOurs Math AIME2421.4658.1360.2163.3356.7761.0460.73 AIME2521.7751.6755.2157.7153.3354.1757.40 HMMT25 Feb.10.0031.5631.9833.8530.5231.8733.33 HMMT25 Nov.8.2335.7337.0838.2336.6738.1239.48 Avg.15.3744.2746.1248.2844.3246.3047.73 Code HumanEval+76.6884.3085.0686.5983.0884.9186.89 MBPP+65.0870.4470.3771.16 70.3770.5072.75 LCB v617.1428.0028.2927.7126.0025.7129.57 Avg.52.9760.9161.2461.8259.8260.3763.07 23 Paper Under Review Table 10: Full results for generalization to three domains. Per-benchmark Qwen3-4B scores un- derlying Table 3. Scores are percentages (higher is better); Avg. weights benchmarks equally within each domain. Single OPD uses the matching in-domain checkpoint from Table 8. The best and second-best results among the compared multi-domain methods are shown in bold and underlined, respectively. BenchmarkStudent Single OPD Vanilla MOPDExOPD Open- MOPDCaMOPDOurs Code HumanEval+76.6884.3086.2884.7685.5284.4586.59 MBPP+65.0870.4470.6371.2370.4470.9072.95 LCB17.1428.0026.8628.57 25.7127.5729.29 Avg.52.9760.9161.2661.5260.5660.9762.94 Math AIME2421.4658.1361.0459.9055.7359.4861.88 AIME2521.7751.6757.5055.31 54.1754.9054.48 HMMT25 Feb.10.0031.5631.6734.9031.2531.9833.54 HMMT25 Nov.8.2335.7337.7138.2335.1037.5039.27 Avg.15.3744.2746.9847.0944.0645.9647.29 Medical MedQA64.7377.0664.4973.2960.1776.7577.61 MedXpertQA13.8015.3112.7814.5712.7816.6915.96 MedMCQA55.6362.4457.1460.9455.0662.5463.11 SuperGPQA-Med29.8736.8129.6933.6827.3036.8137.10 Avg.41.0147.9041.0345.6238.8348.2048.45 C.3EXTENDED TOKEN-LEVEL VISUALIZATIONS Figure 5 expands the exact trajectories shown in the right panels of Figure 3 to 256-token con- texts. It includes the original prompts and marks the 52-token main-text windows, preserving every displayed token’s original coefficient. These illustrative windows were selected from 20 sampled re- sponses per prompt to show both attenuation and amplification near an answer mention; they do not estimate the overall coefficient distribution. The visualization samples use the token-modulation- only checkpoint at step 58, temperature/top-p 1/1, and a 2,048-token generation cap. Both selected trajectories reached that cap. 24 Paper Under Review (a) Mathematics AIME2024_0 | rollout 7 | response tokens [1792, 2048) Every morning Aya goes for a 9-kilometer-long walk and stops at a coffee shop afterwards. When she walks at a constant speed of s kilometers per hour, the walk takes her 4 hours, including t minutes spent in the coffee shop. When she walks s+2 kilometers per hour, the walk takes her 2 hours and 24 minutes, including t minutes spent in the coffee shop. Suppose Aya walks at s+1/2 kilometers per hour. Find the number of minutes the walk takes her, including the t minutes spent in the coffee shop. Please reason step by step, and put your final answer within . Scored response context | outlined tokens: Figure 3 [1988, 2040) says:"Findthenumberofminutesthewalktakesher,includingthetminutesspentinthecoffeeshop."So ,whenshewalksats+·1/2km/h,thetimespentwalkingis·9/(s+·1/2)hours.Thenaddt minutes.Buttoconverttheentirethingtominutes,weneedtoconvertthewalkingtimetominutesandadd t.So:·Walkingtime:·9/(s+·1/2)hours.Converttominutes:·9/(s+·1/2)*·60minutes. Thenaddtminutes,whichis·24.Buts+·1/2is·2.5+·0.5=·3,asabove.So·9/·3=·3 hours.·3hoursis·180minutes.·180+·24=·204minutes.Sothat's·204minutestotal. Butletmejustcheckagain,maybeImadeamistakewiththeequations?Wait,perhapsIshouldmakesure thatthestepsarecorrect.Alternatively,let'sseeifthere'sanotherway.Letmecheckifthevalueofsis correct.Fromtheinitial Earlier context omitted. Generation limit reached at token 2048. (b) Medical MedQA_test_0 | rollout 2 | response tokens [869, 1125) A junior orthopaedic surgery resident is completing a carpal tunnel repair with the department chairman as the attending physician. During the case, the resident inadvertently cuts a flexor tendon. The tendon is repaired without complication. The attending tells the resident that the patient will do fine, and there is no need to report this minor complication that will not harm the patient, as he does not want to make the patient worry unnecessarily. He tells the resident to leave this complication out of the operative report. Which of the following is the correct next action for the resident to take? Options: A. Disclose the error to the patient and put it in the operative report B. Tell the attending that he cannot fail to disclose this mistake C. Report the physician to the ethics committee D. Refuse to dictate the operative report Reason briefly, then state the final answer as a single option letter. Scored response context | outlined tokens: Figure 3 [971, 1023) thepatient.·Alternatively,maybetheresidentshouldfollowtheattending'sordersiftheattendinghas authority.However,thiscangettricky.·Also,iftheresidentisnotdisclosingerrors,theymightbe violatingtheirownethicaltraining.·Wait,thekeyisthatevenminorerrorsshouldbereported.The attending'sreasoningisperhapsthatit'snotharmful,butthemistakeitselfneedstobedocumented.So, perhapsthecorrectanswerisA,becausetheresidentshoulddisclosetheerrortothepatient,evenifthe attendingsaysnotto.OrperhapsthecorrectanswerisB,theresidenthastotelltheattendingthattheycan 'tfailtodisclose.·Thequestionaskswhichisthecorrectnextaction.MaybetheanswerisA?·Wait,the operativereportispartofthemedicalrecord.Soiftheresidentisdictatingtheoperativereport,perhaps theycan'tomittheerror.Soiftheattendingsaystonotreportit,buttheresidentisinapositiontotellthe truth,thenperhapstheresidenthastogoagainsttheattending.·Alternatively,maybetheattendingisthe onewhohastheauthoritytodecidewhatisreported.Butgenerally,medicalrecordsmustbeaccurate.· Hmm.MaybetheanswerisA,becausetheresident Earlier context omitted. Response continues beyond this excerpt. 0.51.01.5 λ t attenuationamplification Figure 5: Extended Math and Medical examples from the right panels of Figure 3. Outlines identify the main-text windows within the longer contexts. Blue and orange denote attenuation and amplification on the unchanged scale λ t ∈ [0.5, 1.5]. Token indices are zero-based, with half-open intervals. Omitted context and the generation limit are marked. 25 Paper Under Review Additional fixed-window examples. Figures 6 and 7 show three further responses per prompt. We take the first three rollout IDs after excluding the response already shown in Figure 5: Math IDs 0, 1, and 2; Medical IDs 0, 1, and 3. In that order, we display the fixed 192-token windows [0, 192), [928, 1120), and [1856, 2048). This selection does not inspect coefficients or answer correctness. Prompt summaries accompany the plots; the full prompts appear in Figure 5. Mathematics: three additional sampled responses AIME2024_0 | same prompt; distinct rollout IDs Prompt summary: Aya walks 9 km with a fixed coffee stop. The total time is 4 h at speed s and 2 h 24 min at s + 2. Find the total minutes at speed s + 1/2, including the same stop. The full prompt appears in Figure 5. (a) Beginning | rollout 0 | tokens [0, 192) Okay,soIneedtofigureoutthisproblemaboutAya'swalk.Letmetrytoparseitstepbystep.· First,theproblemsaysthateverymorningAyawalks·9kilometersandthenstopsatacoffeeshop. Whenshewalksataconstantspeedofskm/h,thewalktakesher·4hours,includingtminutesin thecoffeeshop.Then,whenshewalksats+·2km/h,thewalktakesher·2hoursand·24 minutes,includingthesametminutesinthecoffeeshop.Now,weneedtofindouthowlongthe walktakesherwhenshewalksats+·1/2km/h,includingthetminutesinthecoffeeshop.So, thekeyhereisthatthetotaltimeincludesboththewalkingtimeandthetimespentinthecoffeeshop .Butthecoffeeshoptimeisthesametminutesinbothscenarios.So,weneedtofind Response begins here. Later context omitted. (b) Middle | rollout 1 | tokens [928, 1120) 1.25=·11.25?Wait,letmedothatagain.·18dividedby·1.6.Alternatively,·1.6is ·8/5,so·18dividedby(8/5)is·18*·5/8=·90/8=·11.25.Yes,correct.So· 11.25=s(s+·2)Sos^2+·2s-·11.25=·0·Multiplybothsidesby·4toeliminate thedecimal?Let'ssee:s^2+·2s-·11.25=·0·Alternatively,maybeconvert·11.25 to·45/4?Yes,·11.25=·45/4.Soequationiss^2+·2s-·45 Earlier context omitted. Later context omitted. (c) End | rollout 2 | tokens [1856, 2048) tis·24minutes.Wait,butwait.Theproblemsays"thenumberofminutesthewalktakesher, includingthetminutesspentinthecoffeeshop."Butwhenshewalksat·3km/h,thetimespent walkingis·3hours,thenaddingtminutes(24minutes)forcoffee.Sototaltimeis·3hours+· 24minutes.Buttheanswerneedstobeinminutes.Soconvert·3hourstominutes:·3*60=1 80minutes.Plus·24minutes,total·204minutes.Wait,butwait,isthatcorrect?Wait,let mecheckagain.Wait,intheoriginalproblem,thetotaltimewhenwalkingatsis·4hours(which includestminutes).So,whenAyawalksats,hertotaltimeis·4hours,whichistimewalking(9 /shours)plustminutes.Similarly, Earlier context omitted. Generation cap reached at token 2048. 0.51.01.5 λ t attenuationamplification Figure 6: Three additional Math response excerpts. Panels show the beginning, middle, and end of three distinct sampled responses to AIME2024_0. Original token coefficients and the [0.5, 1.5] color scale are preserved. A dot represents a whitespace-only token; omitted context and the gener- ation cap are marked. 26 Paper Under Review Medical: three additional sampled responses MedQA_test_0 | same prompt; distinct rollout IDs Prompt summary: A resident cuts and repairs a tendon; the attending asks that the complication be omitted from the record. Choose: A disclose and document; B tell the attending it must be disclosed; C report to ethics committee; D refuse to dictate. Full prompt: Figure 5. (a) Beginning | rollout 0 | tokens [0, 192) Okay,soIneedtofigureoutthecorrectnextactionfortheresidentinthisscenario.Let'ssee.The situationisthatajuniororthopaedicsurgeryresidentdidacarpaltunnelrepair,butaccidentallycut aflexortendonduringtheprocedure.Thetendonwasrepairedwithoutcomplications,andthe attendingphysiciantoldtheresidentnottoreportthisminorcomplication,sayingthepatientwillbe fineandthere'snoneedtoworry.Theresidentisnowfacedwithadilemmaaboutwhattodonext. TheoptionsareAthroughD.Let'sbreakthemdown.OptionAsaystodisclosetheerrortothe patientandputitintheoperativereport.Buttheattendingtoldtheresidentnottoreportit,so maybetheresidentisconflictedhere.However,theethicalobligationmightrequiredisclosure,even iftheattendingsaysotherwise.OptionBistotelltheattendingthathecan'tfailtodisclosethe mistake.Thisseemslikeadirectconfrontation, Response begins here. Later context omitted. (b) Middle | rollout 1 | tokens [928, 1120) iftheattendingsaysnotto.·Buttheattendingsaysthepatientwilldofine,andit'snotharmful.If theresidentdisclosestheerror,perhapsthepatientmightbeworried,butthepatienthasarightto know.·However,theinitialinstructionisthattheattendingsaysthere'snoneedtoworry.So maybetheresident'snextstepistoconflictwiththeattending'sorder.·Alternatively,maybeinthe UnitedStates,underthelegalframework,it'stheresponsibilityofthesurgeontoreporterrors.So, eveniftheattendingsaysnotto,perhapstheresidentmustreport.·Butlookingforthecorrect answer.·Alternatively,perhapstheanswerisD.Refusetodictatetheoperativereport.So,the residentwouldn'tfollowtheattending'sorderandinsteadchoosenottodictatethereport.So,the residentwouldwritethereporthimselfwithoutincludingtheerror.Butthatseemscontradictory.· Alternatively,thecorrect Earlier context omitted. Later context omitted. (c) End | rollout 3 | tokens [1856, 2048) properactionisA.·Anotherapproach:theoperativereportissupposedtobeaccurate.Theresident 'sdutyistodocumentthecaseaccurately.Soeveniftheattendingtellsthemtoomittheerror,they shoulddocumentit.Therefore,thecorrectanswerwouldbeA.·Therefore,Awouldbecorrect.· However,accordingtotheguidelines,sometimescertainerrorsmightnotbereportediftheyare minor,buttheattending'sorderheremaynotbeconsidered.·Alternatively,thecorrectanswerisD :Refusetodictate.·Wait,theoperativereportistheresident'sresponsibilitytowrite.Soifthe residentisaskedbytheattendingnottomentiontheerror,buttheresidentknowsitwasanerror, perhapstheyshouldrefusetowritethereport.·Thisseemscomplicated,butiftheattendingsays theyshouldnotmentionit,perhapstheresident'sbestcoursewouldbetodocumentit.·Inthe medicalfield,errorsaresupposed Earlier context omitted. Generation cap reached at token 2048. 0.51.01.5 λ t attenuationamplification Figure 7: Three additional Medical response excerpts. Panels show the beginning, middle, and end of three distinct sampled responses to MedQA_test_0, using the same fixed-window rule and unchanged color scale as Figure 6. These are generated reasoning excerpts, not verified answer explanations. 27 Paper Under Review C.4TRAINING DYNAMICS We complement the endpoint evaluations with complete training traces from three archived ROAD MOPD runs: Math × Medical, Code × Medical, and Math × Code. Each run has 58 updates with 1,008 prompts per update, one PPO epoch, and a 16,384-token response cap. Both domains use reward-agreement modulation and tensor-wise adaptive projection with norm-retention floor m = 0.6. All plots retain every logged step without smoothing. Loss and generation behavior. Figure 8 shows the logged signed policy surrogate, per-domain response lengths, and the fraction of responses reaching the cap. The logger captures only the task-0 primary loss contribution: Math in Math × Medical, and Code in the other two runs. This quan- tity can be negative and is neither the full two-domain objective nor a nonnegative KL estimate. It decreases sharply early in training, while generation lengths and cap rates continue to vary. These single-run traces describe training behavior; they do not establish convergence of both domain ob- jectives or comparative test accuracy. −0.01 0 0.01 0.1 Signed surrogate loss (symlog) Math only Math × Medical Code only Code × Medical Code only Math × Code 0 4k 8k 12k 16k Mean response length (tokens) Math Medical Code Medical Math Code 1204058 Training step 0 10 20 30 Responses at cap (%, both domains) 1204058 Training step 1204058 Training step Figure 8: Training loss and generation behavior across three domain pairings. Columns show individual ROAD MOPD runs. Top: the logged task-0 surrogate contribution on a symmetric-log scale (linear within ±0.01); each panel identifies the recorded domain. Middle: mean response length for each domain. Bottom: percentage of responses across both domains that reach 16,384 tokens. Curves connect all 58 logged updates; negative loss values and fluctuations are preserved. 28 Paper Under Review Gradient-norm retention during projection. Figure 9 examines when the norm constraint changes the projection strength. For each primary domain, the logger computes two norm ratios from the same pre-projection gradients: the ratio under full projection and the ratio under the cho- sen adaptive strength. Each curve reports the minimum over conflicting parameter-tensor units at that update (or one if none conflict). These are analytical projection diagnostics before merging and global gradient clipping, rather than measurements of the final optimizer displacement. The adaptive ratios stay at or above 0.6, within numerical precision, in all six domain traces. For example, in the Math branch of Math× Medical, the full-projection estimate falls below 0.6 in 40 of 58 updates and reaches a minimum of 0.253, whereas the adaptive minimum is 0.600. The Medical branch of that same run never crosses the floor, so the two curves coincide. This illustrates how the constraint limits norm loss when needed without changing full projection when its retained norm al- ready satisfies the floor. The full-projection curve uses the gradients encountered by ROAD MOPD; it is not a separately trained baseline. 0.2 0.4 0.6 0.8 1.0 Min. norm ratio Math primaryCode primaryMath primary 1204058 Training step 0.2 0.4 0.6 0.8 1.0 Min. norm ratio Medical primary 1204058 Training step Medical primary 1204058 Training step Code primary Math × MedicalCode × MedicalMath × Code Adaptive projectionFull projection (estimate) Floor m= 0.6 Figure 9: Minimum norm retention over training. Each panel identifies the primary domain whose gradient is projected; columns use the same runs as Figure 8. Blue shows the logged ratio under adaptive projection; dashed orange shows the analytical full-projection estimate on the same gradients. The dotted line marks m = 0.6. Ratios are minima over conflicting parameter-tensor units, before merged-gradient clipping. Overlapping curves indicate identical minimum retention. All logged steps are shown without smoothing. 29 Paper Under Review DLIMITATIONS Computational overhead.ROAD MOPD incurs additional training overhead compared with Vanilla MOPD because of auxiliary-teacher scoring and gradient processing. Reusing student tra- jectories avoids additional rollout generation. Future work will explore more efficient teacher eval- uation and gradient correction to reduce this overhead. Limited evaluation of domain scalability. Our experiments cover mathematics, code, and medicine, with joint training evaluated on at most three domains (Table 3). Future work will as- sess whether the observed gains and training stability extend to broader domain collections and larger teacher pools. 30

Instant research discovery