Paper deep dive

Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts

Hejian Sang, Zhengze Zhou, Shayan Mohajer Hamidi, Xiaomin Li, Rohit Jain, Alborz Geramifard

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 61

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 10/13/2026, 3:17:57 AM

Summary

The paper introduces Delta-MOPD, a method for Multi-Teacher On-Policy Distillation that transfers teacher-relative logit shifts (teacher-minus-base) re-anchored at the student's initialization, rather than endpoint policies. This approach mitigates the 'inherited base pull' problem where post-training shifts are obscured by base model preferences. Experiments show Delta-MOPD improves performance in common-domain composition (especially with three teachers) and routed-domain distillation (phased routing), achieving higher accuracy with fewer computational resources compared to endpoint supervision.

Entities (11)

Delta-MOPD · method · 98%Multi-teacher On-Policy Distillation · method · 95%Inherited Base Pull · concept · 93%DeepSeek-R1-Distill-Qwen-1.5B · model · 92%JustRL · model · 90%Nemotron · model · 90%Polaris-7B · model · 90%Common-Domain Composition · task · 88%Routed-Domain Distillation · task · 88%Five-Benchmark · benchmark · 85%MATH · benchmark · 85%

Relation Signals (7)

Delta-MOPD → isa → Multi-teacher On-Policy Distillation

confidence 95% · We introduce $Δ$-MOPD... compare it with endpoint supervision in both settings

Delta-MOPD → uses → Teacher-Relative Shifts

confidence 94% · Delta-MOPD, which transfers each teacher's teacher-minus-base logit shift

Inherited Base Pull → impedes → Endpoint Transfer

confidence 92% · We first expose the mechanism that impedes endpoint transfer: inherited base pull can exceed the post-training shift.

DeepSeek-R1-Distill-Qwen-1.5B → isinitializationfor → Delta-MOPD

confidence 90% · The anchor and student initialization in every run is DeepSeek-R1-Distill-Qwen-1.5B

Delta-MOPD → outperforms → Endpoint Composition

confidence 90% · With three composed teachers, $Δ$-MOPD exceeds endpoint composition by 4.11 Math

Nemotron → isteacherfor → Science/IF

confidence 88% · Science/IF teacher Nemotron-Reasoning-Qwen-1.5B

Polaris-7B → isteacherfor → Math

confidence 88% · Math teacher Polaris-7B-Preview

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multi-teacher on-policy distillation (MOPD) is used in two settings. In common-domain composition, several teachers score each student rollout from one prompt domain and their signals form a single target; in routed-domain distillation, prompts from different domains are assigned to the corresponding specialist. Both settings usually transfer each teacher's endpoint policy, which mixes what post-training changed with preferences inherited from the teacher's base. We introduce $\Delta$-MOPD, which transfers each teacher's teacher-minus-base logit shift re-anchored at the student's frozen initialization, and compare it with endpoint supervision in both settings while holding teacher selection fixed. We first expose the mechanism that impedes endpoint transfer: inherited base pull can exceed the post-training shift. Removing it reduces the teacher-term norm ratio and target--student KL. Across our experiments, the results suggest that shift targets are particularly useful when teacher signals are combined at a state. With three composed teachers, $\Delta$-MOPD exceeds endpoint composition by $4.11$ Math and $1.95$ five-benchmark points; with two, it matches endpoint accuracy. Under phased routing, it achieves higher mean performance in both phase orders and reduces the observed order gap from $10.50$ to $6.42$ points. Under interleaved routing, where each update involves one teacher, the two targets perform comparably. The phased results provide supporting evidence that the benefit may extend to signals accumulated across training phases. Target construction is thus an independent design axis in MOPD, complementary to teacher selection.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

57,462 characters extracted from source content.

Expand or collapse full text

Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts Hejian Sang11footnotemark: 1 Affiliation: Iowa State University Zhengze Zhou11footnotemark: 1 Shayan Mohajer Hamidi Affiliation: LinkedIn Xiaomin Li Affiliation: Harvard University Rohit Jain Affiliation: LinkedIn Alborz Geramifard Affiliation: LinkedIn Abstract Multi-teacher on-policy distillation (MOPD) is used in two settings. In common-domain composition, several teachers score each student rollout from one prompt domain and their signals form a single target; in routed-domain distillation, prompts from different domains are assigned to the corresponding specialist. Both settings usually transfer each teacher’s endpoint policy, which mixes what post-training changed with preferences inherited from the teacher’s base. We introduce Δ -MOPD, which transfers each teacher’s teacher-minus-base logit shift re-anchored at the student’s frozen initialization, and compare it with endpoint supervision in both settings while holding teacher selection fixed. We first expose the mechanism that impedes endpoint transfer: inherited base pull can exceed the post-training shift. Removing it reduces the teacher-term norm ratio from :15.2\!:\!1 to :11.44\!:\!1 and target–student KL from 0.1520.152 to 0.0310.031. The resulting target reaches the endpoint arm’s best evaluated Math score with 35%35\% fewer allocated H100-hours. Across our experiments, the results suggest that shift targets are particularly useful when teacher signals are combined at a state. With three composed teachers, Δ -MOPD exceeds endpoint composition by 4.114.11 Math and 1.951.95 five-benchmark points; with two, it matches endpoint accuracy. In an unpaired cross-tokenizer extension, a projected fourth shift improves all three Math benchmarks and adds a further 3.323.32 Math and 1.851.85 five-benchmark points. Under phased routing, it achieves higher mean performance in both phase orders and reduces the observed order gap from 10.5010.50 to 6.426.42 points. Under interleaved routing, where each update involves one teacher, the two targets perform comparably. The phased results provide supporting evidence that the benefit may extend to signals accumulated across training phases. Target construction is thus an independent design axis in MOPD, complementary to teacher selection. 1 Introduction On-policy distillation (OPD) trains a student on its own rollouts using scores from a frozen teacher, so supervision arrives at the states the student visits (Gu et al., 2024; Agarwal et al., 2024; Song and Zheng, 2026). Because post-trained specialists are individually incomplete and widely released, multi-teacher OPD (MOPD) aims to combine several of them in one student. MOPD is used in two settings. In common-domain composition, several teachers score the same rollout from one prompt domain and their signals are merged into one target. In routed-domain distillation, prompts from different domains are assigned to the corresponding specialist, so each state is supervised by one teacher. Both settings, and the studies of routing, teacher counteraction, and cascaded overwrite built on them (Ma et al., 2026; Chen et al., 2026a; Li et al., 2026c), transfer each teacher’s endpoint policy. A specialist endpoint, however, combines the change produced by its post-training stage with preferences inherited from its base (DeepSeek-AI, 2025). When a teacher’s base differs from the student’s initialization, endpoint supervision transfers that inherited difference as well. The original MOPD pipeline avoids it because every teacher shares the student’s initial checkpoint (Ma et al., 2026); public specialists rarely do. Single-teacher methods report promising results from teacher-relative differences (Heo et al., 2026; Feng et al., 2026; Yang et al., 2026), but whether such differences are a better transfer object in either MOPD setting has not been tested. We separate two decisions in any MOPD recipe: teacher selection, which teachers supervise a state, and target construction, how their scores define the distribution to be matched. Holding selection fixed, we compare endpoint targets with Δ -MOPD, which measures each teacher against its own base BiB_i and re-anchors the resulting shift at the student’s initialization A. Writing zXz_X for checkpoint X’s logits at a student-generated state and S for the selected teachers, the two targets are endpoint:zE : z_E =zA+∑i∈(zTi−zA), =z_A+Σ _i (z_T_i-z_A), (1) Δ-MOPD:zC $ $-MOPD: z_C =zA+∑i∈(zTi−zBi), =z_A+Σ _i (z_T_i-z_B_i), each normalized by a softmax. Composition selects all teachers; routing selects one. When Bi=AB_i=A, both constructions give the same term, so only cross-origin teachers are affected (Figure 1). Concretely, if a Math specialist both acquires stronger reasoning and keeps a verbosity preference already present in its base, endpoint distillation transfers both; dividing the specialist by its base cancels what the base already preferred, and the anchor expresses the remaining change relative to the student’s own starting policy. We hypothesize that removing inherited base differences yields a target that is closer to the student and, under composition, more balanced across teachers. Figure 1: Endpoint versus shift targets. Each endpoint term zTi−zAz_T_i-z_A contains a post-training shift zTi−zBiz_T_i-z_B_i and an inherited base difference zBi−zAz_B_i-z_A. Δ -MOPD keeps only the shift and re-anchors it at A, giving the effective teacher T~i T_i; the two composite targets differ by the summed base differences. Teacher 1 is drawn with B1=AB_1=A, so T~1=T1 T_1=T_1. Positions are schematic. Our results suggest that shift targets are particularly useful when teacher signals are combined at a state. The mechanism appears directly in the learning signal: endpoint supervision gives the inherited base pull a larger gradient than the post-training shift, whereas base subtraction produces a target that is five times closer to the student and substantially more balanced across teachers. The closer target reaches the endpoint arm’s best Math score with 35%35\% fewer allocated H100-hours. In common-domain composition, shifts match endpoint accuracy with two teachers and lead on Math and the five-benchmark macro with three. In an unpaired cross-tokenizer extension, a projected fourth shift from a second cross-origin teacher improves all three Math benchmarks, raising the Math macro by a further 3.323.32 points and the five-benchmark macro by 1.851.85 points. In routed-domain distillation, phased routing yields higher mean performance for shifts in both phase orders and a smaller observed order gap; interleaved routing, where each update sees a single teacher, yields comparable results. The phased results provide supporting evidence that the benefit may extend to signals accumulated across training phases. Contributions. 1. We identify target construction as a design axis in MOPD, independent of teacher selection, and instantiate it with Δ -MOPD (Eq. 1), which reduces exactly to endpoint OPD for same-origin teachers. 2. We expose the mechanism that impedes endpoint transfer. The inherited base pull can exceed the post-training shift; removing it lowers the teacher-term norm ratio from :15.2\!:\!1 to :11.44\!:\!1, and reduces target–student KL fivefold. 3. The improved geometry yields stronger signal combination. With three composed teachers, Δ -MOPD exceeds endpoint composition by 4.114.11 Math and 1.951.95 five-benchmark points; under an unpaired cross-tokenizer extension, a projected fourth shift improves all three Math benchmarks and adds a further 3.323.32 Math and 1.851.85 five-benchmark points. Under phased routing it achieves higher mean performance in both orders and reduces the observed order gap from 10.5010.50 to 6.426.42 points. Interleaved routing performs comparably. 2 Preliminaries Let x∼x and y=(y1,…,yL)∼πθ(⋅∣x)y=(y_1,…,y_L) _θ(· x), with ht=(x,y<t)h_t=(x,y_<t), pt=πθ(⋅∣ht)p_t= _θ(· h_t), and qtq_t the target distribution at hth_t. OPD minimizes the token-level reverse KL ℒ= = x∼,y∼πθ(⋅∣x) _x ,\;y _θ(· x) (2) [1L∑t=1LDKL(pt∥qt)]. [ 1L _t=1^LD_KL\! (p_t\,\|\,q_t ) ]. For sampled yt∼pty_t p_t, log⁡pt(yt)−log⁡qt(yt) p_t(y_t)- q_t(y_t) is a one-sample estimator of the per-state reverse KL. Training maximizes the sampled-token reward log⁡qt(yt)−log⁡pt(yt) q_t(y_t)- p_t(y_t) with the centered score-function estimator of Appendix B. Notation. πX _X and zXz_X denote checkpoint X’s distribution and logits over a shared effective vocabulary. A is a frozen anchor, in every experiment the student initialization; TiT_i is teacher i’s post-training endpoint; BiB_i is the exact checkpoint that entered that stage. The shift of teacher i is δi=zTi−zBi _i=z_T_i-z_B_i and its inherited base difference is zBi−zAz_B_i-z_A. A shift carries the complete effect of post-training—capability, style, and calibration alike—and is not assumed to isolate capability. Teacher i is same-origin when Bi=AB_i=A and cross-origin otherwise. Teacher selection. A selection rule assigns each state a teacher set ⁡(ht)S(h_t). Common-domain composition uses ⁡(ht)=1,…,MS(h_t)=\1,…,M\, so the teachers’ relative magnitudes at a state determine the update. Routed-domain distillation uses ⁡(ht)=ρ⁡(ht)S(h_t)=\ρ(h_t)\, where ρ maps a prompt to its domain’s teacher; domains may be interleaved within training or assigned to contiguous phases. 3 Δ -MOPD 3.1 Construction For teachers that share a token-id mapping on the 151,665151,665-token effective vocabulary, Δ -MOPD uses the shared-anchor target πC(v∣ht)=1Z⁡(ht)πA(v∣ht)∏i∈⁡(ht)πTi(v∣ht)πBi(v∣ht), _C(v h_t)= 1Z(h_t) _A(v h_t) _i (h_t) _T_i(v h_t) _B_i(v h_t), (3) where Z⁡(ht)Z(h_t) normalizes over the vocabulary. Each model’s log-partition term is constant across tokens, so the target is a logit sum normalized once: zC(v∣ht) z_C(v h_t) =zA(v∣ht)+∑i∈⁡(ht)δi(v∣ht), =z_A(v h_t)+ _i (h_t) _i(v h_t), (4) πC(⋅∣ht) _C(· h_t) =softmax(zC(⋅∣ht)), =softmax\! (z_C(· h_t) ), with sampled-token reward RtC(v)=log⁡πC(v∣ht)−log⁡πθ(v∣ht).R^C_t(v)= _C(v h_t)- _θ(v h_t). (5) Setting qt=πC(⋅∣ht)q_t= _C(· h_t) in Eq. 2 gives the full-vocabulary objective. Some runs approximate only the partition function on a restricted support; details of this approximation are given in Appendix B. Endpoint control. The control replaces each shift by zTi−zAz_T_i-z_A, giving zEz_E in Eq. 1. At a given prefix the two pre-softmax targets differ by exactly ∑i∈⁡(ht)(zBi−zA) _i (h_t)(z_B_i-z_A). With one selected teacher the control is that teacher’s endpoint. Same-origin reduction. If Bi=AB_i=A, then zA+δi=zTiz_A+ _i=z_T_i: a same-origin teacher’s shift target is its endpoint. Routing to a same-origin teacher is therefore exactly endpoint OPD, and if every selected teacher is same-origin the two composites coincide. Only cross-origin terms differ. 3.2 Reference-term interpretation The endpoint reward of teacher j decomposes exactly as log⁡πTj−log⁡πθ= _T_j- _θ= log⁡πTj−log⁡πBj⏟RjΔ:shift _T_j- _B_j_R _j:\ shift (6) +log⁡πBj−log⁡πθ⏟RjB:base-reference pull. + _B_j- _θ_R^B_j:\ base-reference pull. The base-reference pull equals the inherited difference log⁡πBj−log⁡πA _B_j- _A plus the current anchor–student gap log⁡πA−log⁡πθ _A- _θ; at initialization only the inherited difference remains. Up to a token-independent normalizer, Δ -MOPD has the same form with a single shared reference: RtC(v)= R^C_t(v)= ∑i∈⁡(ht)δi(v∣ht) _i (h_t) _i(v h_t) +[log⁡πA(v∣ht)−log⁡πθ(v∣ht)]. + [ _A(v h_t)- _θ(v h_t) ]. Both targets reward following teacher shifts and pull toward a reference. Endpoint supervision pulls toward each selected teacher’s own base; Δ -MOPD replaces these heterogeneous pulls by one pull toward the student’s initialization, which is zero at the start of training. Two consequences are plausible: the target is closer to the student, and, under composition, a cross-origin teacher’s inherited difference no longer sets the scale of its term relative to the others. Section 5.1 examines both. 4 Experimental Setup Checkpoints. The anchor and student initialization in every run is DeepSeek-R1-Distill-Qwen-1.5B (DeepSeek-AI, 2025). Table 1 lists the teachers and the exact precursor used for each shift. Polaris-7B is post-trained from the 7B checkpoint of the same distillation family, so “cross-origin” here means a different and larger base of the same lineage. Nemotron and JustRL are same-origin, so their shift and endpoint terms are identical. In every paired comparison, the two targets therefore differ by one controlled substitution: the endpoint or teacher-relative representation of Polaris. A tokenizer-projected Qwen3-DAPO-449 shift extends the construction to a second cross-origin teacher in Appendix F. Table 1: Public checkpoints. “Base” is the exact precursor used to compute each shift. Nemotron is pinned to revision v1. Role Model Base Origin Specialization Use Anchor / student init DS-R1-Distill-Qwen-1.5B self self – all runs Math teacher JustRL-DeepSeek-1.5B DS-R1-Distill-Qwen-1.5B same Math RL composition (M=3M=3) Science/IF teacher Nemotron-Reasoning-Qwen-1.5B DS-R1-Distill-Qwen-1.5B same multi-domain RL composition, routing Math teacher Polaris-7B-Preview DS-R1-Distill-Qwen-7B cross Math RL all runs Math teacher Qwen3-DAPO-449 Qwen3-8B cross DAPO Math RL extension (App. F) Runs. Common-domain composition uses two runs on BigMath prompts, with every teacher scoring every rollout: the mechanism run composes Nemotron and Polaris (M=2M=2) and records diagnostics; the scaling run compares M=2M=2 with M=3M=3, which adds JustRL. Routed-domain distillation assigns Math prompts to Polaris and Science/IF prompts to Nemotron, either interleaved within each batch or in two phased orders. The acquisition run distills Polaris alone. Paired arms share initialization, prompts, optimizer, update budget, and evaluation; all frozen models score the same prefixes within an arm, although the arms’ on-policy trajectories diverge. Each comparison uses a shared run protocol; key configurations are listed in Table 4. 4.1 Diagnostic metrics We measure Γbase=‖gB‖/‖gΔ‖ _base=\|g_B\|/\|g_ \|, the gradient norm of the base-reference pull relative to the shift. For composition geometry, we report the teacher-term norm ratio, centered-shift cosine, cancellation, and top-1616 sign conflict. Target–student and target–anchor KL are computed over the full effective vocabulary, independently of the training normalizer. The reported forward KL DKL(πC∥πθ)D_KL( _C\| _θ) measures target tracking and is not the training objective. Definitions are in Appendix B. 5 Results The evidence follows a mechanism-to-outcome progression. We first test whether inherited base differences distort the learning signal (§5.1), then examine the predicted consequences for cross-origin acquisition (§5.2), common-domain composition (§5.3), and routed-domain distillation (§5.4). 5.1 Inherited base differences distort composition geometry Figure 2: Composition diagnostics in the mechanism run. (a) For Polaris, the base-reference pull has a larger gradient norm than the shift. (b) The endpoint composite has a :15.2\!:\!1 teacher-term norm ratio; the shift composite has :11.44\!:\!1. (c) The shift target is about five times closer to the student in target–student KL. Values are late-training averages (Table 7). Inherited pull dominates early learning. Under endpoint supervision, the base-reference pull of Polaris has roughly twice the gradient norm of its shift (Γbase≈2 _base\!≈\!2, averaged over checkpoints recorded every 2020 steps). The ratio declines during training because only RjBR^B_j depends on the student: it shrinks as the student moves toward πBj _B_j, whereas RjΔR _j is fixed by the teacher–base pair at a given state. The inherited component therefore has its greatest relative influence while the specialization is still being acquired. Base subtraction restores balance. With the Nemotron term identical in both arms, the teacher-term norm ratio falls from 5.25.2 under endpoint composition to 1.441.44 under shift composition, and target–student KL falls from 0.1520.152 to 0.0310.031 (Figure 2). The shift composite lies near the equal-norm independence reference (cancellation 0.2870.287 versus 0.2930.293, cosine −0.016-0.016), whereas the endpoint composite has low cancellation (0.1360.136), the signature of one dominant term. The ordering appears by step 2020 and persists through training (Appendix C); at M=3M=3, target–anchor KL remains lower for shifts (0.4830.483 versus 0.6950.695). Together, these measurements establish the proposed mechanism: inherited base differences enlarge the target and set the relative scale of the teacher terms. Removing them yields two testable predictions. The closer target should be acquired earlier, and the more balanced terms should make additional teachers more useful. We test these predictions next. 5.2 A closer target accelerates cross-origin acquisition Distilling Polaris alone removes composition and directly tests the first prediction. Δ -MOPD leads the English-Math macro by 2.872.87 p at step 100100, after which the arms approach a similar level. Including precursor-scoring overhead, Δ -MOPD scores 47.90%47.90\% at step 100100, already exceeding Endpoint’s best evaluated 46.54%46.54\% at step 195195; after charging for Δ -MOPD’s slower updates, these checkpoints cost 28.028.0 versus 42.942.9 allocated H100-hours. At initialization, the endpoint target is 4.94×4.94× farther from the student in target–student KL across 16,38416,384 shared prefix states. The closer target is therefore acquired earlier even after charging for the additional precursor forward pass. Details are in Appendix E. 5.3 Balanced shifts improve common-domain composition We next test the second prediction: whether balancing the teacher terms turns additional supervision into usable capability. Two teachers. In the mechanism run, both composites improve substantially over the initial student and reach similar aggregate accuracy: 44.91%44.91\% for Δ -MOPD versus 45.14%45.14\% for the endpoint composite under Avg@K, and 37.73%37.73\% versus 36.25%36.25\% under greedy decoding. Δ -MOPD is higher on Science/IF under both decodings, while the endpoint composite is 3.43.4 p higher on MATH-500 (Appendix C). Three teachers. The scaling run compares M=2M=2 with M=3M=3 under one protocol (Table 2). At M=3M=3, Δ -MOPD exceeds the endpoint composite by 4.114.11 p on the Math macro and 1.951.95 p on the five-suite macro. Adding the same JustRL teacher raises shift composition by 4.404.40 Math and 1.781.78 five-suite p, but endpoint composition by only 0.430.43 and 0.580.58. Because JustRL is same-origin, its term is identical in both arms; the difference lies in how Polaris is represented when a third teacher is added. The gain is concentrated in AIME 2025; Science/IF, which receives no in-domain prompts in this run, is 1.291.29 p below the endpoint composite (Table 9). Table 2: Common-domain composition in the scaling run at step 100100. Greedy pass@11 macros (%). All teachers score every BigMath rollout. Δ3−2 _3-2 is the change from adding same-origin JustRL. Per-benchmark results are in Table 9. Teachers Target Math Sci/IF Five-suite M=2M=2 Endpoint composite 28.96 15.70 23.66 M=2M=2 Δ -MOPD 29.10 17.38 24.41 M=3M=3 Endpoint composite 29.39 16.51 24.24 M=3M=3 Δ -MOPD 33.50 15.22 26.19 Δ3−2 _3-2 Endpoint composite +0.43+0.43 +0.81+0.81 +0.58+0.58 Δ3−2 _3-2 Δ -MOPD +4.40+4.40 −2.16-2.16 +1.78+1.78 Shift composition thus preserves two-teacher accuracy and turns an additional teacher into a substantially larger Math gain. 5.4 Shift targets show consistent gains under phased routing Composition combines teacher signals within each state. Routing separates them across examples or training phases, examining whether the same target construction also changes how specialist signals accumulate over time. Under routing, Polaris supervises Math prompts and Nemotron supervises Science/IF prompts. Because Nemotron is same-origin, the two targets coincide on Science/IF states and differ only in the routed Polaris term. Table 3 reports two schedules. Table 3: Routed-domain distillation. Interleaved routing mixes domains in each batch (50/25/2550/25/25 Math/Science/IF); phased routing trains one teacher–domain phase at a time. Greedy pass@11 macros (%); Δ is Δ -MOPD minus Endpoint in p. Schedule Macro Endpoint Δ -MOPD Δ Interleaved Math 30.61 31.52 +0.91+0.91 Sci/IF 18.26 18.45 +0.19+0.19 Five-suite 25.67 26.29 +0.62+0.62 Phased: Sci/IF → Math Math 28.02 34.39 +6.37+6.37 Sci/IF 18.25 22.89 +4.64+4.64 Five-suite 24.11 29.79 +5.68+5.68 Phased: Math → Sci/IF Math 40.87 42.20 +1.33+1.33 Sci/IF 25.23 27.23 +2.00+2.00 Five-suite 34.61 36.21 +1.60+1.60 Interleaved routing. With both teachers active throughout training, the five-suite macro differs by only 0.620.62 p in favor of Δ -MOPD, with group-macro differences of +0.91+0.91 p on Math and +0.19+0.19 p on Science/IF. The two targets are therefore comparable when each update involves a single teacher, as expected if the benefit of shifts comes from reconciling several teacher signals. Phased routing. When each teacher–domain pair occupies a contiguous phase, Δ -MOPD ends above its same-order endpoint control in both orders, by 5.685.68 and 1.601.60 five-suite p, and both group macros improve in each order. The gap between the two phase orders falls from 10.5010.50 p under endpoint supervision to 6.426.42 p. Phased training couples each teacher with its domain data, providing evidence about the robustness of the combined teacher–data curriculum across phase orders. Together, these results suggest that the benefits of shift targets may extend beyond within-state composition to settings where teacher signals accumulate across training phases. We view the phased-routing results as supporting evidence for this interpretation, while interleaved routing yields comparable performance. 6 Discussion Where shifts help. Shift targets show their clearest benefits when several teachers are composed at a state. Phased routing provides supporting evidence that these benefits may extend to settings where routed teachers occupy separate phases and their signals accumulate over training; under interleaved routing, where each update involves one teacher, the targets are comparable. The paired design holds all same-origin terms fixed and changes only the representation of Polaris, directly attributing the contrast to target construction. The projected four-teacher extension adds a second cross-origin shift, improves all three Math benchmarks, and raises the Math macro by a further 3.323.32 p (Appendix F), extending the construction to a larger heterogeneous teacher set. Geometry and scale. Cosine and sign conflict are invariant to rescaling individual shifts, and cancellation to rescaling all shifts jointly. Their change therefore reflects the relative geometry of the teacher terms, not only the overall distance to the student. The policy-ratio form cancels vocabulary normalization constants but does not calibrate differences in logit temperature or confidence across checkpoints, so shift norms reflect both the magnitude of post-training change and the scale on which it is expressed. Calibration-aware composition is complementary to base subtraction. 7 Related Work Distillation for language models. Knowledge distillation transfers a teacher’s output distribution to a smaller student (Hinton et al., 2015), for language models mostly off-policy on teacher-written rationales (Li et al., 2022; Hsieh et al., 2023; Fu et al., 2023; Singh et al., 2024; Liu et al., 2024b), so the student is never supervised at the states it visits. OPD removes that mismatch by scoring the student’s own rollouts (Agarwal et al., 2024; Song and Zheng, 2026) and has been reframed as compressing reward-shaped behavior (Sang et al., 2026; Xu et al., 2026b; He et al., 2026; Xu et al., 2026a). Work that broadens what the on-policy signal carries, through rubric judgments (Fang et al., 2026), dual sequence- and token-level views (Hou et al., 2026; Wang et al., 2026; Shen et al., 2026a), or context rather than weights (Ye et al., 2026), keeps the teacher’s endpoint as the matched object. Li et al. (2026b) show that single-teacher OPD depends on thinking-pattern compatibility rather than teacher strength; student competence frontiers for on-policy self-distillation are studied by Xu et al. (2026c). Multi-teacher on-policy distillation. MOPD studies of routing, counteraction, and cascaded overwrite (Ma et al., 2026; Chen et al., 2026a; Li et al., 2026c) motivate our paired controls, and the same brittleness surfaces in tool-use, GUI-agent, and expert-to-generalist settings (Shen et al., 2026b; Lian et al., 2026; Chen et al., 2026b). Sample routing connects group-relative and self-distillation policy optimization (Li et al., 2026a). These methods act on selection, scheduling, or rehearsal; we vary what each selected teacher contributes. Differences as transfer objects. Delta-oriented single-teacher methods use teacher–base differences for direct or extrapolated transfer (Heo et al., 2026; Feng et al., 2026; Yang et al., 2026), and task arithmetic and proxy tuning compose parameter- or logit-space differences offline (Ilharco et al., 2023; Liu et al., 2024a). We study such differences as transfer objects in both MOPD settings, on the student’s own prefixes and normalized under one softmax. 8 Conclusion We studied target construction in MOPD by comparing teacher-relative shifts, re-anchored at the student’s initialization, with endpoint supervision. Inherited base differences can dominate a teacher’s post-training shift; removing them yields a target that is balanced across teachers and closer to the student. Across our experiments, shift targets show their clearest benefits when teacher signals are combined at a state: they match endpoints with two composed teachers and raise Math and five-benchmark accuracy with three. In an unpaired cross-tokenizer extension, adding a projected fourth shift from a second cross-origin teacher yields further gains on both macros. Under phased routing, shifts achieve higher mean performance in both tested orders and reduce the observed order gap, providing supporting evidence that their benefits may extend to teacher signals accumulated across phases. Under interleaved routing, where each update sees one teacher, the targets are comparable. Because the construction reduces exactly to endpoint OPD for same-origin teachers, it preserves their signal and acts only on inherited base differences. What is transferred from each teacher is thus a design axis in MOPD, complementary to which teachers are selected. 9 Limitations • Evaluation scope. We study public reasoning specialists and common-domain composition on Math prompts, with Science/IF measured through held-out evaluation. Broader specialist families and domain-mixed composition are natural extensions. • Requirements. Δ -MOPD requires each teacher’s precursor and a compatible tokenizer or an explicit projection rule, unlike black-box OPD (Ye et al., 2025). References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Cited by: §1, §7. Chen et al. (2026a) T. Chen, J. Ou, Z. Liu, R. Tang, J. Liang, and H. Li Counteraction-aware multi-teacher on-policy distillation for general capability recovery with domain preservation. External Links: 2605.27115, Link Cited by: §1, §7. Chen et al. (2026b) Y. Chen, X. Chen, and F. Wang REGEN: replay-recycling for expert-to-generalist distillation with offline reinforcement learning. External Links: 2607.19450, Link Cited by: §7. DeepSeek-AI (2025) DeepSeek-AI DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. Nature 645, p. 633–638. Cited by: §1, §4. Fang et al. (2026) J. Fang, Z. Hong, M. Zheng, M. Song, G. Li, H. Jiang, D. Zhang, H. Guo, X. Wang, and T. Chua Rubric-based on-policy distillation. arXiv preprint arXiv:2605.07396. Cited by: §7. Feng et al. (2026) S. Feng, H. Gao, H. Chi, H. Wu, Z. Zhang, Z. Jiang, B. He, W. Ma, Y. Zhang, and H. Zhou Weak-to-strong generalization via direct on-policy distillation. External Links: 2607.05394, Link Cited by: §1, §7. Fu et al. (2023) Y. Fu, H. Peng, L. Ou, A. Sabharwal, and T. Khot Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12726. Cited by: §7. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang MiniLLM: knowledge distillation of large language models. In International Conference on Learning Representations, Cited by: §1. He et al. (2026) Y. He, S. Kaur, A. Bhaskar, Y. Yang, J. Liu, N. Ri, L. Fowl, A. Panigrahi, D. Chen, and S. Arora Self-distillation zero: self-revision turns binary rewards into dense supervision. External Links: 2604.12002, Link Cited by: §7. Heo et al. (2026) B. Heo, J. Hwang, S. Yun, and D. Han On-policy delta distillation. External Links: 2607.15161, Link Cited by: §1, §7. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: §7. Hou et al. (2026) W. Hou, S. Peng, W. Wang, Z. Ruan, Y. Zhang, Z. Zhou, M. Gao, Y. Chen, K. Wang, H. Yang, C. Zhang, Z. Tian, H. Hu, Y. Yang, F. Wu, and H. Fan Uni-OPD: unifying on-policy distillation with a dual-perspective recipe. arXiv preprint arXiv:2605.03677. Cited by: §7. Hsieh et al. (2023) C. Hsieh, C. Li, C. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C. Lee, and T. Pfister Distilling step-by-step! Outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, Cited by: §7. Ilharco et al. (2023) G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi Editing models with task arithmetic. In International Conference on Learning Representations, Cited by: §7. Li et al. (2026a) G. Li, T. Yang, J. Fang, M. Song, M. Zheng, H. Guo, D. Zhang, J. Wang, and T. Chua Unifying group-relative and self-distillation policy optimization via sample routing. External Links: 2604.02288, Link Cited by: §7. Li et al. (2022) S. Li, J. Chen, Y. Shen, Z. Chen, X. Zhang, Z. Li, H. Wang, J. Qian, B. Peng, Y. Mao, W. Chen, and X. Xie Explanations from large language models make small reasoners better. arXiv preprint arXiv:2210.06726. Cited by: §7. Li et al. (2026b) Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, and N. Ding Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016. Cited by: §7. Li et al. (2026c) Z. Li, D. Kong, Y. Wei, E. Yang, R. Shen, M. K. Ihsani, M. Yang, W. Zhang, C. Hao, J. Yang, R. Tao, B. Dai, S. Zhang, W. Ye, Y. Wei, and D. Lian Every coin has two sides: on the dual nature of generalization in on-policy distillation of large language models. External Links: 2608.16647, Link Cited by: §1, §7. Lian et al. (2026) N. Lian, A. Chen, Z. Yu, C. Duan, F. Liu, H. Liu, P. Fu, J. Luan, Y. Wang, S. Xia, and J. Wang UI-MOPD: multi-platform on-policy distillation for continual GUI agent learning. External Links: 2607.04425, Link Cited by: §7. Liu et al. (2024a) A. Liu, X. Han, Y. Wang, Y. Tsvetkov, Y. Choi, and N. A. Smith Tuning language models by proxy. In International Conference on Learning Representations, Cited by: §7. Liu et al. (2024b) J. Liu, C. Zhang, J. Guo, Y. Zhang, H. Que, K. Deng, Z. Bai, J. Liu, G. Zhang, J. Wang, Y. Wu, C. Liu, W. Su, J. Wang, L. Qu, and B. Zheng DDK: distilling domain knowledge for efficient large language models. In Advances in Neural Information Processing Systems, Cited by: §7. Ma et al. (2026) W. Ma, J. Wei, L. Zhao, H. Zhang, B. Xiao, L. Li, Q. Yang, B. Gao, Y. Wang, R. Li, J. Dong, Z. Sui, and F. Luo MOPD: multi-teacher on-policy distillation for capability integration in LLM post-training. External Links: 2606.30406, Link Cited by: §1, §7. Sang et al. (2026) H. Sang, Y. Xu, Z. Zhou, R. He, Z. Wang, and J. Sun CRISP: compressed reasoning via iterative self-policy distillation. External Links: 2603.05433, Link Cited by: §7. Shen et al. (2026a) G. Shen, L. Huang, X. Cheng, C. Zhao, J. Li, D. Zhao, and X. Yu From generic correlation to input-specific credit in on-policy self-distillation. External Links: 2605.11613, Link Cited by: §7. Shen et al. (2026b) J. Shen, G. Chen, and C. Mao Diagnosing and calibrating tool-call boundary drift in multi-teacher on-policy distillation. External Links: 2607.07050, Link Cited by: §7. Singh et al. (2024) A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P. J. Liu, J. Harrison, J. Lee, K. Xu, A. Parisi, A. Kumar, A. Alemi, A. Rizkowsky, A. Nova, B. Adlam, B. Bohnet, G. Elsayed, H. Sedghi, I. Mordatch, I. Simpson, I. G"ur, J. Snoek, J. Pennington, J. Hron, K. Kenealy, K. Swersky, K. Mahajan, L. Culp, L. Xiao, M. L. Bileschi, N. Constant, R. Novak, R. Liu, T. Warkentin, Y. Qian, Y. Bansal, E. Dyer, B. Neyshabur, J. Sohl-Dickstein, and N. Fiedel Beyond human data: scaling self-training for problem-solving with language models. Transactions on Machine Learning Research. Cited by: §7. Song and Zheng (2026) M. Song and M. Zheng A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626. Cited by: §1, §7. Wang et al. (2026) Y. Wang, S. Lu, Y. Gu, P. Wang, Y. Yang, Z. Yan, C. Xie, J. Wu, and H. Yang Not all disagreement is learnable: token teachability in on-policy distillation. External Links: 2605.26844, Link Cited by: §7. Xu et al. (2026a) Y. Xu, H. Sang, Z. Zhou, R. He, Z. Wang, and A. Geramifard Beyond GRPO and on-policy distillation: an empirical sparse-to-dense reward principle for language-model post-training. External Links: 2605.12483, Link Cited by: §7. Xu et al. (2026b) Y. Xu, H. Sang, Z. Zhou, R. He, Z. Wang, and A. Geramifard TIP: token importance in on-policy distillation. External Links: 2604.14084, Link Cited by: §7. Xu et al. (2026c) Y. Xu, H. Sang, Z. Zhou, R. He, and Z. Wang PACED: distillation and on-policy self-distillation at the frontier of student competence. External Links: 2603.11178, Link Cited by: §7. Yang et al. (2026) W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin Learning beyond teacher: generalized on-policy distillation with reward extrapolation. External Links: 2602.12125, Link Cited by: §1, §7. Ye et al. (2025) T. Ye, L. Dong, Z. Chi, X. Wu, S. Huang, and F. Wei Black-box on-policy distillation of language models. External Links: 2511.10643, Link Cited by: 2nd item. Ye et al. (2026) T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei On-policy context distillation for language models. External Links: 2602.12275, Link Cited by: §7. Appendix A Protocols and Reproducibility Table 4: Training protocols. Values are shared by both arms of a run unless an arm-specific value is shown. Fields marked “–” were not recorded in the run’s frozen configuration summary. Setting Acquisition (M=1M=1) Mechanism (M=2M=2) Scaling (M=2,3M=2,3; proj. M=4M=4) Phased routing Interleaved routing Phase structure single single single 2 phases, optimizer resumed, dataloader reset single, domains interleaved from step 1 On-policy prompts BigMath, 2525K Math BigMath, 2525K Math BigMath Math BigMath; TextbookReasoning ++ Llama-Nemotron IF Math ++ Science/IF mixture (50/25/2550/25/25) Temperature / top-p 1.0 / 1.0 1.0 / 1.0 1.0 / 1.0 1.0 / 1.0 1.0 / 1.0, top-k off Maximum response 10,000 10,000 10,000 10,000 10,000 Learning rate 10−610^-6 10−610^-6 10−610^-6 10−610^-6 10−610^-6 Hardware 4 × H100 80GB 4 × H100 80GB 8 × H200 4 × H100 80GB per arm 8 × H200 (Endpoint) / 8 × H100 (Δ -MOPD) Evaluation Avg@K (Table 5) Avg@K and greedy greedy pass@11 greedy pass@11 greedy pass@11 Table 5: Held-out evaluation suite. Avg@K applies to the acquisition and mechanism runs; other runs decode every benchmark greedily at K=1K=1. Group Benchmark Test items Avg@K Generation / scoring English Math AMC 2023 40 16 sampled, official answer checker English Math MATH-500 500 1 greedy, official answer checker English Math AIME 2025 30 16 sampled, official answer checker Science GPQA-Diamond 198 4 sampled, final option-letter scorer Instruction following IF-Eval 541 4 sampled, vendored rule checker Evaluation. Evaluation uses SGLang with a 16,38416,384-token maximum response length, except that IF-Eval is capped at 2,0482,048 tokens by the evaluation driver; the scaling run and teacher references use a 10,00010,000-token greedy protocol. Greedy decoding uses temperature 00 and top-p=1p=1; Avg@K sampling uses temperature 11 and top-p=0.95p=0.95. MATH-500 reuses its greedy item outcomes for K=1K=1. All runs use the same 1,3091,309 frozen held-out items. Five-suite macros are unweighted averages of the five benchmark scores, not of the two group macros. Entries with error bars are mean ± sample standard deviation over five independently trained seeds with matched seed IDs and prompt orders; macros are computed from the five-seed benchmark means. Reproducibility. Each teacher is paired with its exact precursor and checked for tokenizer compatibility; the Qwen3 extension records the token-string overlap map of Eq. 16. Paired arms share prompts, update counts, and evaluation settings, and all frozen models score identical student prefixes within an arm before target construction. Dataset revisions, sampled IDs, chat templates, model revisions, decoding settings, and verifier versions are fixed across comparisons. Appendix B Estimator and Diagnostic Definitions Estimator and normalization. For valid sampled-token positions ℐI in an optimization batch, training subtracts the empirical mean reward R¯C=|ℐ|−1∑(b,t)∈ℐRb,tC(yb,t) R^C=|I|^-1 _(b,t) R^C_b,t(y_b,t) and optimizes the detached centered reward times the student log-probability. This is sampled-token centering rather than an explicit per-state full-vocabulary expectation. When a restricted support StS_t is used, the sampled-token numerator is exact and only the composite partition is approximated: log⁡πC^(yt∣ht)= _C(y_t h_t)= zC(yt∣ht) z_C(y_t h_t) (7) −log∑u∈StexpzC(u∣ht). - _u∈ S_t z_C(u h_t). If mSt=∑v∈StπC(v∣ht)m_S_t= _v∈ S_t _C(v h_t) is the composite mass captured by the support, then log⁡πC^(yt∣ht)=log⁡πC(yt∣ht)−log⁡mSt _C(y_t h_t)= _C(y_t h_t)- m_S_t, so the restricted normalizer raises the sampled-token log probability by −log⁡mSt- m_S_t. This is a normalizer approximation, not the top-k OPD objective. All logits are restricted to the 151,665151,665-token effective vocabulary before composition. Gradient decomposition. For sampled positions (ht,yt)t=1N\(h_t,y_t)\_t=1^N, define RtΔ=log⁡πT(yt∣ht)−log⁡πB(yt∣ht)R _t= _T(y_t h_t)- _B(y_t h_t) and RtB=log⁡πB(yt∣ht)−log⁡πθ(yt∣ht)R^B_t= _B(y_t h_t)- _θ(y_t h_t). Their gradient estimators and relative magnitude are g∙ g_ =∇θ[−1N∑t=1Nsg(R∙t)logπθ(yt∣ht)], = _θ\! [- 1N _t=1^Nsg\! (R _t )\, _θ(y_t h_t) ], (8) ∙∈Δ,B, ∈\ ,B\, Γbase _base =‖gB‖2‖gΔ‖2+10−8. = \|g_B\|_2\|g_ \|_2+10^-8. At each mechanism checkpoint, Eq. 8 uses the cross-origin teacher–base pair, the first micro-batch, and its first N=min⁡(128,|ℐ|)N= (128,|I|) valid response positions, backpropagating through all trainable parameters. Measurements are recorded every 2020 steps from step 2020 through 380380. Composition geometry. Center each shift by its vocabulary mean, Δi(t,v)= _i(t,v)= zTi(v∣ht)−zBi(v∣ht) z_T_i(v h_t)-z_B_i(v h_t) (9) −1V∑u=1V[zTi(u∣ht)−zBi(u∣ht)], - 1V _u=1^V [z_T_i(u h_t)-z_B_i(u h_t) ], concatenate across sampled positions, and compute pairwise cosine, aggregate norm, and Cancel⁡()=1−‖∑i∈Δi‖2∑i∈‖Δi‖2+10−8.Cancel(T)=1- \| _i _i \|_2 _i \| _i\|_2+10^-8. (10) Sign conflict uses the top-1616 absolute-shift coordinates selected by each teacher, retaining overlaps and aggregating across positions. The endpoint control replaces zTi−zBiz_T_i-z_B_i by zTi−zAz_T_i-z_A. Because Δi _i is evaluated at student-visited prefixes, these statistics depend on the rollout protocol and pooling rule. The mechanism run concatenates the first 128128 valid positions of the first micro-batch on each data-parallel rank; the scaling run computes the same quantities per complete response and averages over micro-batches. We read them within a run. Two orthogonal equal-norm vectors have cosine 00 and Cancel=1−2/2=0.2929Cancel=1- 2/2=0.2929; independent sign-symmetric coordinates have conflict rate 0.50.5. For a=Δsamea= _same, b=Δcrossb= _cross, c=cos⁡(a,b)c= (a,b), and r=‖b‖/‖a‖r=\|b\|/\|a\|, 1−Cancel=r2+1+2rcr+1.1-Cancel= r^2+1+2rcr+1. (11) Since this expression is invariant under r↦1/r 1/r, cosine and cancellation identify the norm ratio max⁡(r,1/r) (r,1/r). Target distance. The target-to-student forward KL is DST=ht∑v=1VπC(v∣ht)log⁡πC(v∣ht)πθ(v∣ht),D_ST=E_h_t _v=1^V _C(v h_t) _C(v h_t) _θ(v h_t), (12) computed over the full V=151,665V=151,665 vocabulary on the first micro-batch of each mechanism checkpoint, using up to 128128 valid response positions. KL to the anchor replaces πθ _θ by πA _A. Appendix C Common-Domain Composition Details C.1 Mechanism run: capability Table 6: Mechanism run (M=2M=2) student evaluation. AMC 2023 and AIME 2025 use Avg@1616, GPQA-Diamond and IF-Eval use Avg@44, and MATH-500 uses greedy decoding. Entries with error bars are mean ± sample standard deviation over training seeds, in percent. Benchmark Initial student Δ -MOPD Endpoint composite Δ -MOPD −- Endpoint (p) AMC 2023 63.12 79.84±1.9979.84± 1.99 79.53±2.7979.53± 2.79 +0.31+0.31 MATH-500 36.00 59.40±1.2959.40± 1.29 62.80±1.3462.80± 1.34 −3.40-3.40 AIME 2025 23.33 32.92±3.0432.92± 3.04 32.92±3.2432.92± 3.24 0.000.00 English Math macro 40.82 57.39 58.42 −1.03-1.03 GPQA-Diamond 34.72 40.28±2.1140.28± 2.11 39.02±2.2239.02± 2.22 +1.26+1.26 IF-Eval 23.43 24.58±1.3124.58± 1.31 24.68±1.0724.68± 1.07 −0.10-0.10 Science/IF macro 29.08 32.43 31.85 +0.58+0.58 Under greedy decoding, the group-balanced totals are 24.10%24.10\% for the initial student, 37.73%37.73\% for Δ -MOPD, and 36.25%36.25\% for the endpoint composite. The per-group greedy macros are 31.44%31.44\%, 50.91%50.91\%, and 48.16%48.16\% for English-Math, and 16.76%16.76\%, 24.54%24.54\%, and 24.34%24.34\% for Science/IF, so greedy decoding favors Δ -MOPD on both groups while the Avg@K ordering is effectively tied. C.2 Mechanism run: diagnostics Table 7: Late-training diagnostics for the mechanism run, averaged over steps 300,320,340,360,380300,320,340,360,380. Target Cancellation Aggregate norm Conflict rate Signal cosine Entropy KL to anchor Target–student KL Δ -MOPD 0.2871 3247.5 0.5540 −0.0163-0.0163 0.2458 0.1243 0.0307 Endpoint composite 0.1363 10231.5 0.4386 0.0592 0.2643 0.2567 0.1522 Equal-norm reference 0.2929 — 0.5000 0.0000 — — — Table 8: Early mechanism-run diagnostics at steps 20,60,10020,60,100. Target Step Cancellation Target–student KL Δ -MOPD 20 0.2728 0.1070 Δ -MOPD 60 0.2914 0.0750 Δ -MOPD 100 0.2832 0.0462 Endpoint composite 20 0.1265 0.2472 Endpoint composite 60 0.1378 0.1222 Endpoint composite 100 0.1381 0.1632 Reference — 0.2929 — Table 7 reports the late-training window and Table 8 early checkpoints; the two arms separate by step 2020 and the ordering is stable. C.3 Scaling run Table 9: Scaling run at step 100100. Greedy pass@11; entries with error bars are mean ± sample standard deviation over training seeds, in percent. Teachers Target AMC 2023 MATH-500 AIME 2025 Math macro GPQA-D IF-Eval Sci/IF macro Five-suite macro M=2M=2 Endpoint composite 35.50±2.5635.50± 2.56 34.04±1.3734.04± 1.37 17.33±1.9317.33± 1.93 28.96 16.77±2.2716.77± 2.27 14.64±1.6614.64± 1.66 15.70 23.66 M=2M=2 Δ -MOPD 38.00±1.6838.00± 1.68 28.64±1.4228.64± 1.42 20.67±3.1120.67± 3.11 29.10 20.30±1.9020.30± 1.90 14.45±1.3314.45± 1.33 17.38 24.41 M=3M=3 Endpoint composite 38.00±1.8038.00± 1.80 32.84±0.9932.84± 0.99 17.33±3.0817.33± 3.08 29.39 17.27±1.5317.27± 1.53 15.75±1.1415.75± 1.14 16.51 24.24 M=3M=3 Δ -MOPD 38.00±1.6138.00± 1.61 31.84±0.9931.84± 0.99 30.67±2.5030.67± 2.50 33.50 17.27±1.4817.27± 1.48 13.16±1.5013.16± 1.50 15.22 26.19 Under the same greedy, 10,00010,000-token protocol, the individual teachers score (Math / Sci-IF / five-suite macro, %) 46.62/28.08/39.2146.62/28.08/39.21 for Nemotron-1.5B, 49.02/27.50/40.4149.02/27.50/40.41 for JustRL-1.5B, and 59.29/36.87/50.3259.29/36.87/50.32 for Polaris-7B. At M=2M=2 the two shifts have norms 13,67413,674 and 6,0736,073 with pairwise cosine 0.0840.084. Adding JustRL gives norms 12,89112,891, 11,11411,114, and 6,4906,490, so the third shift enters at a magnitude comparable to the other two. Two of its pairwise cosines remain near zero (0.0720.072 and 0.0620.062) while the third rises to 0.3150.315. Across the recorded M=2M=2 and M=3M=3 training-signal diagnostics, shift composition lowers target displacement, reward variance, and gradient magnitude; at M=3M=3, its final target-to-anchor KL is 0.4830.483 rather than 0.6950.695. Its per-update time is 11.0%11.0\% higher (167.1167.1 versus 150.6150.6 seconds) because the Polaris precursor must also be scored. Appendix D Routed-Domain Distillation Details Interleaved routing. The router is the prompt’s domain: Math prompts are supervised by Polaris and Science and instruction-following prompts by Nemotron, with exactly one teacher per prompt. The training mixture is 50%50\% Math, 25%25\% Science, and 25%25\% instruction-following. Domains are interleaved from the first update with a deterministic within-batch shuffle. Endpoint matches zTρ⁡(h)z_T_ρ(h) and Δ -MOPD matches zA+δρ⁡(h)z_A+ _ρ(h). Targets are normalized exactly over the full effective vocabulary. The Endpoint arm was trained on 8×8×H200 and the Δ -MOPD arm on 8×8×H100; the comparison is at matched update count, so hardware affects throughput but not the reported accuracies. All checkpoints were evaluated on H100 with greedy decoding. D.1 Phased routing Table 10: Phased routing at the step-300300 checkpoint. Greedy pass@11; group macros are unweighted averages within each group. Entries with error bars are mean ± sample standard deviation over training seeds, in percent. Order Method AMC 2023 MATH-500 AIME 2025 GPQA-Diamond IF-Eval Math macro Sci/IF macro Science/IF → Math Endpoint 25.50±2.4325.50± 2.43 41.24±0.8441.24± 0.84 17.33±2.1917.33± 2.19 15.76±1.9315.76± 1.93 20.74±0.8120.74± 0.81 28.02 18.25 Δ -MOPD 33.00±2.0633.00± 2.06 42.84±1.3142.84± 1.31 27.33±2.6927.33± 2.69 24.85±2.2824.85± 2.28 20.92±1.9220.92± 1.92 34.39 22.89 Math → Science/IF Endpoint 40.50±1.7940.50± 1.79 51.44±0.9451.44± 0.94 30.67±2.2130.67± 2.21 29.90±1.8129.90± 1.81 20.55±1.3520.55± 1.35 40.87 25.23 Δ -MOPD 40.50±2.4840.50± 2.48 55.44±1.1255.44± 1.12 30.67±2.5230.67± 2.52 32.42±2.4332.42± 2.43 22.03±1.4722.03± 1.47 42.20 27.23 Each arm trains two phases with actor and optimizer state carried across the switch. The Science/IF phase is identical across methods, so the contrast lies in the Polaris Math phase. Per-item binary outcomes are retained, so same-order Δ -MOPD-versus-Endpoint comparisons are exact paired McNemar tests on identical items. The only comparison below p=0.05p=0.05 is MATH-500 in Math → Science/IF (5656 versus 3636 discordant items, p=0.047p=0.047), which does not survive correction across the ten benchmark-by-order comparisons. Appendix E Cross-Origin Acquisition and Compute Table 11: Acquisition run at step 100100. AMC 2023 and AIME 2025 use Avg@1616; MATH-500 uses greedy decoding. The macro is the unweighted average of these three benchmarks. Entries with error bars are mean ± sample standard deviation over training seeds, in percent. Target AMC 2023 MATH-500 AIME 2025 English Math macro Δ -MOPD 67.81±1.5367.81± 1.53 48.80±1.4048.80± 1.40 27.08±1.8227.08± 1.82 47.90 Endpoint 64.84±1.7164.84± 1.71 44.20±0.9344.20± 0.93 26.04±1.7326.04± 1.73 45.03 Both arms train only on BigMath prompts. The English-Math macro across the evaluated checkpoints is 47.90/45.65/47.6847.90/45.65/47.68 for Δ -MOPD and 45.03/46.18/46.5445.03/46.18/46.54 for Endpoint at steps 100/150/195100/150/195. On GPQA-Diamond and IF-Eval, the ordering flips with decoding mode: at step 100100, Δ -MOPD leads by 0.910.91 p on the Avg@44 macro while greedy decoding favors Endpoint by 1.071.07 p. The endpoint target remains near 5×5× farther from the student in target–student KL across checkpoints. E.1 Compute accounting For an arm with S updates, measured end-to-end step times tst_s, and NGPUN_GPU H100 GPUs, allocated compute is Calloc=NGPU3600∑s=1StsH100-hours.C_alloc= N_GPU3600 _s=1^St_s 100-hours. (13) Thus, one wall-clock hour of training on four allocated H100s counts as four H100-hours. The timed region includes rollout generation, student optimization, batching, and every teacher, precursor, and anchor forward pass, but excludes held-out evaluation. This measures allocated device time rather than FLOPs and does not correct for utilization. The M=1M=1 run uses each arm’s mean step time and four GPUs; the M=2M=2 run sums its recorded step times before multiplying by four GPUs. For the acquisition comparison, Δ -MOPD first exceeds Endpoint’s best evaluated English-Math macro at step 100100, whereas Endpoint attains that best score at step 195195. With mean step times of approximately 252252 and 198198 seconds, respectively, their allocated costs are CΔ-MOPD=4×100×252/3600=28.0,CEndpoint=4×195×198/3600=42.9H100-hours. aligned C_ -MOPD&=4× 100× 252/3600=28.0,\\ C_Endpoint&=4× 195× 198/3600=42.9 aligned 100-hours. (14) Therefore the matched-capability reduction is (42.9−28.0)/42.9=34.7%(42.9-28.0)/42.9=34.7\%, reported as 35%35\%. The reduction comes from requiring fewer updates, not from cheaper updates: Δ -MOPD is 27.3%27.3\% slower per update because it also scores the precursor. Table 12: Earliest evaluated checkpoint meeting each held-out English-Math threshold in the acquisition run, in allocated H100-hours. Checkpoints were evaluated every 5050 steps, so every entry is an upper bound on first-passage compute. English-Math macro Δ -MOPD Endpoint 46.18%46.18\% ≤28.0≤ 28.0 h ≤33.0≤ 33.0 h 46.54%46.54\% ≤28.0≤ 28.0 h ≤42.9≤ 42.9 h 47.90%47.90\% ≤28.0≤ 28.0 h not observed by 42.942.9 h At equal update count, Δ -MOPD costs 27.3%27.3\% more in M=1M=1 and 12.0%12.0\% more in M=2M=2, below a naive factor of two because rollout generation, optimization, batching, and parallel scoring are shared. Every H100-hour figure, including Table 12, already charges Δ -MOPD for this overhead. In the separate scaling run, the M=3M=3 overhead is 11.0%11.0\% on the same eight-H200 configuration in both arms; this is a within-run wall-clock ratio, not an H200-to-H100 conversion. Convergence criterion. We define the plateau as the first 2525-step block after which every later block-mean rollout accuracy varies by at most 2.52.5 p. Δ -MOPD stabilizes near step 7575 and Endpoint near step 125125, corresponding to 21.021.0 versus 27.527.5 H100-hours (1.31×1.31×). At 21.021.0 hours Δ -MOPD has 53.38%53.38\% block-mean rollout accuracy, against 51.12%51.12\% for Endpoint at a comparable 22.022.0 hours. Best-macro compute, updates to plateau, and accuracy at matched hours therefore agree. Appendix F Cross-Tokenizer Extension This extension adds Qwen3-DAPO-449, our public Qwen3-8B-DAPO iter-449 checkpoint, as a second cross-origin teacher in the scaling run (M=4M=4). It is tokenizer-identical to its Qwen3-8B precursor but not fully special-token compatible with the DeepSeek/Qwen2 anchor. Overlap projection. We compute the Qwen3 shift inside the Qwen3 family and transfer only overlapping action coordinates to the anchor vocabulary. Let VAV_A be the anchor vocabulary, VQV_Q the Qwen3 vocabulary, and τA,τQ _A, _Q their id-to-token-string maps. With δQ(vQ∣ht)= _Q^Q(v_Q h_t)= zDAPOQ(vQ∣ht) z_DAPO^Q(v_Q h_t) (15) −zQwen3-8BQ(vQ∣ht), -z_Qwen3 -8B^Q(v_Q h_t), define m⁡(vA)=vQm(v_A)=v_Q whenever τA(vA)=τQ(vQ) _A(v_A)= _Q(v_Q). The projected shift is δ~Q(vA∣ht)=δQ(m⁡(vA)∣ht),m⁡(vA)∈VQ,0,otherwise, δ_Q(v_A h_t)= cases _Q^Q(m(v_A) h_t),&m(v_A)∈ V_Q,\\ 0,&otherwise, cases (16) and the M=4M=4 target is zA+∑i=13δi+δ~Qz_A+ _i=1^3 _i+ δ_Q. We verified 151,660151,660 token-string overlaps, no duplicate token strings, and 151,658151,658 shared tokens with identical ids. Only <think> and </think> require remapping; five anchor-only special tokens receive zero Qwen3 shift, and Qwen3-only special tokens are dropped. This full-vocabulary overlap projection is narrower than text-span likelihood alignment for general cross-tokenizer OPD. Because cross-tokenizer endpoint logits are not placed on the anchor vocabulary, no paired endpoint control is defined. Table 13: Projected four-teacher extension at step 100100, with the M=3M=3 shift composite for reference. Greedy pass@11, in percent. Teachers Target AMC 2023 MATH-500 AIME 2025 Math macro GPQA-D IF-Eval Sci/IF macro Five-suite macro M=3M=3 Δ -MOPD 38.00±1.6138.00± 1.61 31.84±0.9931.84± 0.99 30.67±2.5030.67± 2.50 33.50 17.27±1.4817.27± 1.48 13.16±1.5013.16± 1.50 15.22 26.19 M=4M=4 Projected Δ -MOPD 40.50±1.9240.50± 1.92 32.64±1.1932.64± 1.19 37.33±2.6337.33± 2.63 36.82 16.77±1.7616.77± 1.76 12.94±1.4212.94± 1.42 14.85 28.04 Adding the projected shift raises the Math macro from 33.5033.50 to 36.8236.82 and the five-suite macro from 26.1926.19 to 28.0428.04. All three Math benchmarks improve, with the largest gain on AIME 2025, while Science/IF does not improve. For reference, the Qwen3-DAPO-449 model card (https://huggingface.co/pb09204048/Qwen3-8B-DAPO-iter449-disable-thinking) reports AIME-24/AIME-25 pass@11 of 73.75%/67.08%73.75\%/67.08\% under its non-thinking sampling protocol (n=8n=8, maximum response length 30,00030,000), compared with 25.00%/17.90%25.00\%/17.90\% for Qwen3-8B under the same protocol.

Instant research discovery