Paper deep dive
DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition
Yuxuan Lou, Kai Yang, Geng Zhang, Yong Liu, Yang You
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/9/2026, 8:39:50 AM
Summary
The paper introduces DivMoE, a framework for fine-grained Mixture-of-Experts (MoE) upcycling that addresses the 'routing collapse' pathology observed when deriving experts from a single source model. DivMoE utilizes domain-specialized fine-grained expert initialization by slicing domain-adapted dense models and enforces diversity-constrained routing to ensure cross-domain expert activation. Experiments on Qwen3-1.7B and Llama3.2-1B show DivMoE significantly outperforms existing upcycling baselines and closes the performance regression gap.
Entities (13)
Relation Signals (8)
DivMoE → evaluatedon → Llama3.2-1B
confidence 95% · Across two base models... Llama3.2-1B
DivMoE → evaluatedon → Qwen3-1.7B
confidence 95% · Across two base models... Qwen3-1.7B-Base
DivMoE → solves → Routing Collapse
confidence 95% · DivMoE introduces... diversity-constrained routing... guaranteeing that each token activates experts from distinct domain groups.
Drop-Upcycling-fine-grained → suffersfrom → Routing Collapse
confidence 94% · Drop-Upcycling-fine-grained reaches only 23.2% average accuracy... essentially matching from-scratch training
DivMoE → usesinitializationfrom → domain-specialized dense models
confidence 93% · DivMoE introduces domain-specialized fine-grained expert initialization, deriving experts from dense models that have undergone domain-adaptive continual pre-training
DivMoE → outperforms → NVIDIA Upcycling
confidence 92% · DivMoE consistently outperforms six upcycling baselines... outperforming a controlled NVIDIA-Upcycling baseline by 6.1 percentage points.
DivMoE → matchesperformanceof → Moonlight-MoE
confidence 90% · our 12B-parameter DivMoE model matches Moonlight-MoE (16B) at 64.5% average accuracy
DivMoE → usesdataset →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, with recent work demonstrating the benefits of fine-grained expert designs. Training such models from scratch is expensive, and sparse upcycling from pre-trained dense models is an attractive alternative. However, we identify a structural pathology of fine-grained upcycling: when fine-grained experts are derived from a single source model, naive routing collapses and downstream accuracy drops to near-random (e.g., on Qwen3-1.7B, Drop-Upcycling-fine-grained reaches only 23.2% average accuracy across 15 benchmarks, essentially matching from-scratch training at 22.2%, while the same method's coarse-grained variant reaches 50.2%). We propose DivMoE, the first framework achieving fine-grained MoE Upcycling with structurally-balanced routing. DivMoE introduces domain-specialized fine-grained expert initialization, deriving experts from dense models that have undergone domain-adaptive continual pre-training, and diversity-constrained routing, a hard structural constraint guaranteeing that each token activates experts from distinct domain groups. Across two base models and 15 benchmarks, DivMoE consistently outperforms six upcycling baselines (55.6% vs. 51.6% for the strongest baseline on Qwen3-1.7B) and strictly improves over the dense base model on every benchmark after Stage 2 continual pre-training -- closing the regression gap that has plagued prior fine-grained upcycling. After supervised fine-tuning on a public reasoning mixture, our 12B-parameter DivMoE model matches Moonlight-MoE (16B) at 64.5% average accuracy while outperforming a controlled NVIDIA-Upcycling baseline by 6.1 percentage points.
Tags
Links
- Source: https://arxiv.org/abs/2610.11317v1
- Canonical: https://arxiv.org/abs/2610.11317v1
Trouble viewing inline? Open PDF directly →
Full Text
64,113 characters extracted from source content.
Expand or collapse full text
DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition Yuxuan Lou †thanks: Equal contribution. Affiliation: School of Computing, National University of Singapore, Singapore Kai Yang11footnotemark: 1 Affiliation: Department of Computer Science and EngineeringShanghai Jiao Tong University, Shanghai, China Geng Zhang Affiliation: School of Computing, National University of Singapore, Singapore Yong Liu Affiliation: School of Computing, National University of Singapore, Singapore Yang You †thanks: Corresponding author: youy@comp.nus.edu.sg. Affiliation: School of Computing, National University of Singapore, Singapore Abstract Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, with recent work demonstrating the benefits of fine-grained expert designs. Training such models from scratch is expensive, and sparse upcycling from pre-trained dense models is an attractive alternative. However, we identify a structural pathology of fine-grained upcycling: when fine-grained experts are derived from a single source model, naive routing collapses and downstream accuracy drops to near-random (e.g., on Qwen3-1.7B, Drop-Upcycling-fine-grained reaches only 23.2% average accuracy across 15 benchmarks, essentially matching from-scratch training at 22.2%, while the same method’s coarse-grained variant reaches 50.2%). We propose DivMoE, the first framework achieving fine-grained MoE Upcycling with structurally-balanced routing. DivMoE introduces domain-specialized fine-grained expert initialization, deriving experts from dense models that have undergone domain-adaptive continual pre-training, and diversity-constrained routing, a hard structural constraint guaranteeing that each token activates experts from distinct domain groups. Across two base models and 15 benchmarks, DivMoE consistently outperforms six upcycling baselines (55.6% vs. 51.6% for the strongest baseline on Qwen3-1.7B) and strictly improves over the dense base model on every benchmark after Stage 2 continual pre-training—closing the regression gap that has plagued prior fine-grained upcycling. After supervised fine-tuning on a public reasoning mixture, our 12B-parameter DivMoE model matches Moonlight-MoE (16B) at 64.5% average accuracy while outperforming a controlled NVIDIA-Upcycling baseline by 6.1 percentage points. 1 Introduction Mixture-of-Experts (MoE) architectures have emerged as a cornerstone for building state-of-the-art large language models, enabling substantial capacity expansion while maintaining computational efficiency through sparse activation [30, 8]. Recent advances have demonstrated the effectiveness of fine-grained MoE designs, where each feed-forward network (FFN) layer is partitioned into many small experts rather than a few large ones [6, 7] by enabling more flexible routing and finer specialization, which contributes to the strong performance of models such as DeepSeek-V2 and Mixtral [17]. Training such models from scratch demands substantial compute, motivating sparse upcycling [20], which converts pre-trained dense checkpoints into MoE architectures while inheriting their learned representations. However, we observe that fine-grained upcycling is fundamentally fragile: when fine-grained experts are derived from a single dense source, downstream accuracy collapses to near-random. On Qwen3-1.7B-Base evaluated across 15 benchmarks, Drop-Upcycling-fine-grained [26] reaches only 23.2% average accuracy (comparable to from-scratch training, 22.2%), versus 50.2% for the same method’s coarse-grained variant. NVIDIA Upcycling [11], which preserves dense-equivalence by copying a single source, recovers the gap (51.1%) but yields homogeneous expert initial scores that suppress routing diversity. The two existing remedies are unsatisfactory: identical weight copying limits routing diversity by construction, while random re-initialization sacrifices pre-trained knowledge. Our central observation is that no prior method simultaneously occupies three design axes that the experiments below show are jointly necessary: (a) domain-specialized expert initialization, (b) fine-grained partitioning, and (c) a structural mechanism that prevents same-source co-activation. Branch-Train-Mix (BTX) [33] addresses (a) but uses coarse experts and standard top-k routing; NVIDIA Upcycling addresses (b); Drop-Upcycling targets diversity through randomness, sacrificing (a) in a different sense. The concurrent FlexOLMo [31] also constructs experts from domain specialists, but for the orthogonal goal of flexible data governance: it uses coarse experts, no joint training, and no routing constraint. We propose DivMoE, the first framework that achieves fine-grained MoE Upcycling, which includes two components: (1) Domain-specialized fine-grained expert initialization. We derive experts from n dense models that have undergone domain-adaptive continual pre-training on public corpora (mathematics, code, science, commonsense), then vertically slice each specialist’s FFN into m shards, yielding N=n×mN=n× m fine-grained experts organized into n domain groups. This preserves all pre-trained knowledge while injecting structured diversity into the routing scores at initialization. (2) Diversity-constrained routing. We enforce a hard structural constraint that each token selects at most one expert per domain group. Unlike soft auxiliary losses [8, 40] that trade routing balance against task accuracy, the diversity constraint operates outside the loss landscape and provides a guarantee regardless of gradient magnitudes. Empirically, DivMoE consistently outperforms six upcycling baselines across 15 benchmarks on two base models (Qwen3-1.7B-Base, Llama3.2-1B). With Stage 2 continual pre-training on the public Dolmino-Mix-1124 corpus [27], DivMoE strictly improves over the dense base model on the majority of benchmarks, eliminating the post-CPT regression that prior upcycling pipelines exhibit. Crucially, the structural constraint improves performance on every domain-specific benchmark (HumanEval +1.8%, GSM8K +2.1%, MATH +1.4%, MBPP +2.3%), refuting the intuition that “coding tokens should only use coding experts.” After Stage 3 supervised fine-tuning on a public reasoning mixture, our 12B-parameter DivMoE model matches Moonlight-MoE (16B) at 64.5% and outperforms a controlled NVIDIA-Upcycling baseline by 6.1 percentage points. The contributions of our work are summarized below: • Observation on a fine-grained upcycling collapse pathology. We show that combining fine-grained expert partitioning with single-source initialization leads to routing collapse and severe performance degradation. • The first fine-grained MoE Upcycling framework. We propose DivMoE, which combines domain-specialized fine-grained expert initialization with a hard diversity-constrained routing mechanism. • Comprehensive experiments with strong downstream performance. Across two base models, three model scales (1B, 1.7B, 4B base → 3.2B, 5.4B, 12B MoE), fifteen benchmarks, and six upcycling baselines, all using public corpora, DivMoE consistently outperforms prior upcycling methods and closes the regression gap of previous fine-grained upcycling approaches. 2 Related Work Mixture-of-Experts in LLMs. MoE architectures scale model capacity efficiently through conditional computation [15, 18, 30]. GShard [22] and Switch Transformer [8] scaled this paradigm to trillion-parameter models with top-k routing and load-balancing losses. Mixtral [17] matches dense quality at reduced cost, while DeepSeek-MoE [6] and DeepSeek-V2 [7] introduced fine-grained expert segmentation with shared experts. OLMoE [25] provides open MoE models and data. ST-MoE [40] introduced the router z-loss for stable routing. DivMoE extends this fine-grained trend by deriving fine-grained experts from multiple domain-specialized dense models and enforcing routing diversity structurally. Dense-to-MoE upcycling. Sparse Upcycling [20] initializes all experts from copies of the same dense FFN. Branch-Train-Mix [33] (BTX, distinct from Branch-Train-Merge [23]) trains separate dense “branch” models on different data slices, then assembles them into a coarse-grained MoE in which each expert is one full FFN. He et al. [11] extend dense-equivalent initialization to fine-grained experts, but derive all experts from a single source model, providing no initial routing-score diversity. Drop-Upcycling [26] introduces diversity via random re-initialization of expert subsets, at the cost of partial knowledge loss; we will show that its fine-grained variant collapses. The concurrent FlexOLMo [31] also constructs experts from domain specialists, but for the orthogonal goal of flexible data governance: experts train independently with no joint pre-training, the granularity is coarse, and no routing constraint is imposed. Routing collapse and load balancing. Top-k routing is prone to routing collapse, in which a few experts dominate selection regardless of input [8]. Standard remedies are auxiliary load-balancing losses [30, 8] or expert-choice routing [39]. The router z-loss [40] penalizes large router logits to stabilize training. All of these are soft mechanisms: they add gradients to the routing logits without changing what selections are reachable. DivMoE’s diversity-constrained routing is a complementary hard mechanism whose guarantee is independent of gradient magnitudes; we retain a small auxiliary load-balancing loss (Section 3.2) only to encourage utilization within domain groups. Continual pre-training. Multi-stage training pipelines that interleave continual pre-training with task-specific fine-tuning are now standard [14, 10]. The choice of CPT data has a large effect on benchmark performance, with mismatches between the base model’s original pre-training distribution and the CPT mixture causing transient regressions on specific tasks. We use Dolmino-Mix-1124 [27], a public mid-training corpus curated for the OLMo-2 family, which is high-enough quality that all upcycling baselines (including ours) post-CPT improve over their dense bases on the majority of benchmarks. Positioning. Table 1 summarizes the design space. DivMoE is the first method to achieve fine-grained MoE Upcycling, with the mechanism occupying all three columns simultaneously. Table 1: Design-space of dense-to-MoE upcycling. ✓indicates the method occupies the design choice. Only DivMoE occupies all three columns; the empirical consequence is that fine-grained partitioning without domain diversity collapses (Section 4.2). Method Domain-specialized Fine-grained Structural routing experts partitioning constraint Sparse Upcycling [20] ✗ ✗ ✗ Branch-Train-Mix [33] ✓ ✗ ✗ NVIDIA Upcycling [11] ✗ ✓ ✗ Drop-Upcycling [26] ✗ ✗ ✗ Drop-Upcycling-fine-grained ✗ ✓ ✗ FlexOLMo [31] (concurrent) ✓ ✗ ✗ DivMoE (ours) ✓ ✓ ✓ 3 DivMoE DivMoE upcycles dense language models into fine-grained MoE architectures through two innovations: (1) domain-specialized fine-grained expert initialization (Section 3.1) that derives experts from n dense models domain-adaptively pre-trained on distinct corpora, and (2) diversity-constrained routing (Section 3.2) that structurally enforces cross-domain expert selection. Figure 1 provides an overview. Figure 1: DivMoE overview. Left: a single base model is domain-adaptively pre-trained into n=4n=4 specialists. Middle: each specialist’s FFN is vertically sliced into m=2m=2 shards, yielding N=8N=8 experts in n=4n=4 domain groups. Right: the router selects the top-1 expert within each group, then top-k groups, guaranteeing k experts from k distinct domains. Given a dense transformer with SwiGLU FFN FFN()=down⋅σ(gate)⊙(up)FFN(x)=W_down·σ(W_gatex) (W_upx), where gate,up∈ℝdff×dW_gate,W_up ^d_f× d and down∈ℝd×dffW_down ^d× d_f, we convert each FFN into an MoE layer. Unlike prior work that derives experts from a single model [20, 11], DivMoE first trains n domain specialists ℳ1,…,ℳn\M_1,…,M_n\ from the base model, then slices each specialist’s FFN into m fine-grained experts. With n=4n=4, m=2m=2, and k=2k=2, DivMoE doubles total capacity while activating half the dense FFN parameters. 3.1 Domain-Specialized Fine-Grained Expert Initialization Domain specialists. Starting from base model ℳbaseM_base, we apply domain-adaptive continual pre-training (next-token-prediction loss on full sequences, not instruction-style SFT) on public domain-specific subsets to obtain n specialists ℳmath,ℳcode,ℳscience,ℳcommon\M_math,M_code,M_science,M_common\. Each specialist encodes distinct domain knowledge while retaining general capabilities, in contrast to identical copies [20] or random perturbations [26]. Expert construction. Each specialist’s FFN weights are vertically sliced into m shards: gate,i _gate,i →gate,i(1),…,gate,i(m), →\W_gate,i^(1),…,W_gate,i^(m)\, (1) up,i _up,i →up,i(1),…,up,i(m), →\W_up,i^(1),…,W_up,i^(m)\, (2) down,i _down,i →down,i(1),…,down,i(m), →\W_down,i^(1),…,W_down,i^(m)\, (3) where expert (i,j)(i,j) has dimensions (dff/m)×d(d_f/m)× d for gate/up and d×(dff/m)d×(d_f/m) for down projections. Indices i∈1,…,ni∈\1,…,n\ and j∈1,…,mj∈\1,…,m\ identify the specialist (domain group) and shard, respectively. Dense-equivalent initialization. Following He et al. [11], we ensure stable initialization via virtual groups: selecting one shard j from any specialist reconstructs a complete FFN. Decomposing each specialist as i=base+iW_i=W_base+ _i where i _i captures domain adaptations, ∑j=1mExpertdj(j)()≈FFNbase(), _j=1^mExpert_d_j^(j)(x) _base(x), (4) preserving the base model’s output distribution at initialization while introducing structured diversity through the i _i. Magnitude correction. We scale expert weights by γ to compensate for softmax attenuation of router scores [11]: expert=γ⋅slice,γ=E⋅G2k3,W_expert=γ·W_slice, γ= [3] E· G^2k, (5) where E is the expansion factor, G is granularity, and k is the active-expert count. Non-expert components. Embeddings, attention, and normalization layers are initialized as the mean of the base model and all specialists: shared=1n+1(base+∑i=1ni).W_shared= 1n+1 (W_base+ _i=1^nW_i ). (6) We ablate this choice in Appendix E. 3.2 Diversity-Constrained Routing Figure 2: Routing comparison. (a) Standard top-k may select both experts from one domain. (b) Diversity-constrained routing guarantees cross-domain selection. Standard MoE routing computes scores =r∈ℝNs=W_rx ^N and selects =TopK(Softmax(),k)S=TopK(Softmax(s),k). When experts are partitioned into n domain groups 1,…,n\D_1,…,D_n\ as in Section 3.1, this top-k rule admits within-group concentration: tokens may route to multiple experts in a single group, leaving other groups inactive (routing collapse, Figure 2a). To prevent this, we impose a diversity constraint that each token activates at most one expert per group: ∀i∈1,…,n:|∩i|≤1,∀ i∈\1,…,n\: |S _i|≤ 1, (7) combined with k≤nk≤ n, Equation 7 guarantees that the k selected experts originate from k distinct domain groups (Figure 2b). The constraint is intentionally structural rather than gradient-based. Conventional remedies for routing imbalance—auxiliary load-balancing losses [8] and router z-losses [40]—are soft: they shape routing through gradient pressure that competes with the language-modeling loss, and they offer no guarantee on what selections are reachable. In contrast, Equation 7 eliminates same-group co-activation by construction, independently of gradient magnitudes. The structural choice is also justified by the initialization: experts within a group are slices of the same domain specialist and encode similar transformations, Experti,j()≈Experti,j′()Expert_i,j(x) _i,j (x) for j,j′∈1,…,mj,j ∈\1,…,m\, so selecting more than one is largely redundant. A potential concern is that obligatory cross-domain composition may waste capacity on ostensibly domain-specific tasks—e.g., forcing a coding token to also draw from math, science, or commonsense experts. We argue, and later confirm empirically (Section 5.1), that the constraint operates at a finer granularity than the prompt: individual tokens within a coding prompt serve heterogeneous functional roles (natural-language description, algorithmic reasoning, syntactic generation) and benefit from complementary expertise. Critically, the constraint does not equalize routing weights; the softmax scores still reflect input distribution, so domain-relevant experts continue to receive higher weights and domain emphasis is preserved (Figure 3). Algorithm 1 realizes the constraint in two deterministic stages: for each domain group i, select the highest-scoring expert ji∗=argmaxjsi,j_i^*= _js_i,j; then select the top-k groups by their representative scores si∗=si,ji∗s_i^*=s_i,j_i^*. This procedure adds no parameters or hyperparameters, integrates with standard MoE infrastructure, and—together with a small auxiliary load-balancing loss (α=0.01α=0.01) retained for within-group balance—guarantees k experts drawn from k distinct domains. Algorithm 1 Diversity-Constrained Routing Input: hidden state x; router weights rW_r; #groups n; experts/group m; active k Output: selected experts S, weights w ←rs _rx ⊳ Router scores ∈ℝn×m ^n× m // Stage 1: Intra-group selection for i=1i=1 to n do ji∗←argmaxj∈[m]si,j_i^* _j∈[m]s_i,j ⊳ Best expert in group i si∗←si,ji∗s_i^*← s_i,j_i^* ⊳ Representative score end for // Stage 2: Inter-group selection ←TopK(s1∗,…,sn∗,k)G (\s_1^*,…,s_n^*\,k) ⊳ Top-k groups ←(i,ji∗):i∈S←\(i,j_i^*):i \ ⊳ Selected experts // Compute normalized routing weights for i∈i do wi←exp(si∗)/∑j∈exp(sj∗)w_i← (s_i^*)/ _j (s_j^*) end for Return S, w Selected expert outputs are aggregated with normalized weights =∑i∈wi⋅Experti,ji∗()y= _i w_i·Expert_i,j_i^*(x). The auxiliary load-balancing loss takes the standard form ℒaux=α⋅N⋅∑i=1Nfi⋅piL_aux=α· N· _i=1^Nf_i· p_i [8], where fif_i is the fraction of tokens routed to expert i and pip_i its mean routing probability. 3.3 Training Pipeline The DivMoE pipeline has three stages, all using only public corpora. Stage 1 (domain-specialist initialization) continually pre-trains four copies of the dense base on the per-domain partitions of Dolmino-Mix [27]—math, code, academic, web—for 4×10B=40B4×10B=40B tokens, producing the specialists from which fine-grained experts are sliced. Stage 2 (joint continual pre-training of the assembled MoE) trains the full DivMoE model on dolmino-mix [27, 32] for 500B tokens; the corpus, optimizer, and step count are identical for every upcycling baseline, ensuring a controlled comparison. Stage 3 (supervised fine-tuning, used only for the open-source MoE comparison in Section 4.3) applies Tülu-3-SFT-Mixture [21] identically to DivMoE, NVIDIA-Upcycling, and OLMoE, isolating architectural effects from data effects. Per-stage corpus, hyperparameter, and compute details are provided in Sections B.2, B.3 and B.4. 4 Experiments 4.1 Experimental Setup Models and architecture. We evaluate two base models, Qwen3-1.7B-Base [36] and Llama3.2-1B, and additionally upcycle Qwen3-4B into a 12B-parameter DivMoE for the open-source MoE comparison. All variants use n=4n=4 domain groups, m=2m=2 experts per group (N=8N=8 total), and k=2k=2 active experts per token. Baselines. We compare against six MoE construction methods, all sharing the Stage 2 CPT corpus, optimizer, and step count: From Scratch (FS) (random fine-grained MoE init), Sparse Upcycling (SU) [20], Branch-Train-Mix (BTX) [33] (coarse-grained MoE assembled from domain branch models), NVIDIA Upcycling (NU) [11] (fine-grained, dense-equivalent, single-source), Drop-Upcycling (DU) [26] (coarse, with random re-initialization), and its fine-grained variant DU-f. Evaluation. We evaluate on 15 benchmarks spanning general knowledge, reasoning, code, and graduate-level science (ARC-c/e [4], BoolQ [3], COPA [9], MMLU [12], OpenBookQA [24], TriviaQA [19], HellaSwag [37], SQuAD2 [28], GSM8K [5], MATH [13], BBH [35], HumanEval [2], MBPP [1], GPQA [29]). Training hyperparameters, per-method compute, and per-benchmark shot counts are tabulated in Sections B.3, B.4 and B.5. 4.2 Comparison with Upcycling Methods Tables 2 and 3 report results after Stage 2 continual pre-training (no SFT) on a representative subset of 11 benchmarks. All upcycling methods share the same Stage 2 corpus, optimizer, and step count; the only differences are the initialization and routing schemes. More results are provided in Appendix C. Table 2: Upcycling comparison on Qwen3-1.7B-Base after 500B Stage 2 CPT tokens (540B for BTX and DivMoE, including Stage 1). Best across upcycling methods in bold. F.G. denotes fine-grained partitioning. “Avg.” averages over the 11 benchmarks shown; full 15-benchmark results in Appendix C. Method F.G. ARC-c BoolQ MMLU TriviaQ HellaS. GSM8K MATH HumanE. MBPP GPQA BBH Avg. From Scratch ✓ 12.2 30.5 40.5 33.0 42.5 5.5 2.3 6.8 11.2 22.6 20.5 20.7 Sparse Upcycling ✗ 38.5 60.2 56.8 43.5 54.5 56.5 28.5 38.0 47.5 24.5 47.5 45.1 BTX ✗ 43.0 71.0 61.5 47.5 61.0 64.0 36.5 44.0 53.0 26.0 51.5 50.8 NVIDIA Upcycling ✓ 42.5 70.5 61.5 47.5 60.5 62.5 35.5 44.5 53.5 25.8 51.0 50.5 Drop-Upcycling ✗ 41.5 71.5 61.0 47.0 59.5 60.0 33.5 41.0 50.5 25.0 50.0 49.1 Drop-Upcycling-f ✓ 14.5 32.5 42.0 33.5 41.5 7.0 3.5 7.8 12.5 22.5 22.0 21.8 DivMoE ✓ 45.0 75.0 64.5 50.0 63.5 68.5 40.5 49.0 57.5 28.0 55.5 54.3 Table 3: Upcycling comparison on Llama3.2-1B after 500B Stage 2 CPT tokens (540B for BTX and DivMoE). Same conventions as Table 2. Method F.G. ARC-c BoolQ MMLU TriviaQ HellaS. GSM8K MATH HumanE. MBPP GPQA BBH Avg. From Scratch ✓ 13.0 27.5 38.0 31.0 36.0 4.0 1.5 4.5 8.0 21.8 18.0 18.5 Sparse Upcycling ✗ 33.5 60.5 48.0 39.5 52.5 26.5 9.0 15.0 21.5 23.5 31.5 32.8 BTX ✗ 36.0 66.0 50.5 39.5 56.0 31.5 11.0 17.5 24.5 25.0 33.0 35.5 NVIDIA Upcycling ✓ 36.0 65.5 50.0 39.0 55.5 30.5 10.8 17.0 24.0 24.8 32.8 35.1 Drop-Upcycling ✗ 35.0 66.0 49.5 39.0 55.0 29.5 10.5 16.5 23.5 24.5 32.5 34.7 Drop-Upcycling-f ✓ 25.5 33.0 43.5 38.0 39.5 6.0 2.5 6.0 9.5 21.5 19.0 22.2 DivMoE ✓ 37.5 67.5 52.5 42.0 56.5 34.5 13.5 20.0 27.0 26.0 34.5 37.4 DivMoE achieves the best overall performance. On Qwen3-1.7B-Base, DivMoE reaches 54.3% averaged over the 11 main-paper benchmarks, +3.5 points over the strongest baseline (BTX, 50.8%) and +3.8 over NVIDIA Upcycling (50.5%). The pattern holds on Llama3.2-1B (37.4% vs. 35.5% for BTX). The advantage is largest on reasoning and code: +4.5 points on GSM8K, +4.0 on MATH, and +5.0 on HumanEval over BTX (Qwen3-1.7B-Base), confirming that domain-specialized initialization transfers reasoning capability into the assembled MoE. The gap is sustained throughout training rather than driven by initialization alone (Appendix F). Stage 2 CPT improves over the dense base. Unlike prior work where post-CPT upcycled models regress relative to the dense base on most benchmarks [20, 11, 26], our use of dolmino-mix as the CPT corpus eliminates this regression: DivMoE surpasses Qwen3-1.7B-Base on every benchmark we evaluate (e.g., MMLU 64.5 vs. 62.6, MATH 40.5 vs. 38.4, BBH 55.5 vs. 53.5; full reference rows in Appendix C). DivMoE also outperforms a data-matched dense baseline (continually pre-trained on the same 540B tokens, no MoE conversion) by ∼ 2 points despite activating only ∼ 2.4B of its 5.4B parameters per token, demonstrating that the MoE architecture provides value beyond the additional CPT data. Fine-grained collapse without domain diversity. From-Scratch (20.7%) and Drop-Upcycling-fine-grained (21.8%) collapse to near-random performance on Qwen3-1.7B-Base—essentially identical, despite very different initialization schemes (random vs. partial re-initialization). The pattern repeats on Llama3.2-1B (18.5% and 22.2%). Coarse-grained Drop-Upcycling (49.1%) does not collapse, isolating the failure to the combination of fine-grained partitioning with non-domain-specialized initialization. DivMoE’s domain-specialized initialization is the only mechanism in our experiments that recovers fine-grained capacity. 4.3 Comparison with Open-Source MoE Models (Controlled SFT) We next compare against state-of-the-art open-source MoE checkpoints: OLMoE [25], DeepSeek-V2-Lite [7], and Moonlight-MoE [38]. A direct comparison would be confounded by the difference in fine-tuning data: the public checkpoints use their own SFT mixtures, while our model is fine-tuned on Tülu-3-SFT-Mixture. To isolate the architectural contribution from the data effect, we additionally fine-tune NVIDIA Upcycling (the strongest fine-grained baseline) and OLMoE on the same Tülu-3-SFT-Mixture under identical Stage 3 hyperparameters as DivMoE and report the controlled rows in Table 4. Table 4: Comparison with open-source MoE models on a representative 7-benchmark subset. Size denotes active/total parameters. DivMoE is upcycled from Qwen3-4B-Base; all baselines use released checkpoints. Rows marked “+Tülu-3 SFT” apply the same fine-tuning to a baseline. Best in bold, second-best underlined. Full 8-benchmark results (including ARC-e) in Appendix D. Model Size ARC-c MMLU TriviaQA HumanE. MBPP GSM8K MATH Avg. OLMoE 1B/7B 49.2 51.9 60.4 51.8 61.2 45.5 23.9 49.1 OLMoE + Tülu-3 SFT 1B/7B 52.4 55.8 63.5 53.5 62.8 56.8 31.2 53.7 DeepSeek-V2-Lite 2.4B/16B 52.1 58.3 65.1 29.9 43.2 41.1 17.1 43.8 NVIDIA Up. + Tülu-3 SFT 2.4B/5.4B 56.3 64.2 67.5 45.8 54.2 66.5 38.8 56.2 Moonlight-MoE 2.4B/16B 65.5 70.0 66.3 48.1 63.8 77.4 45.3 62.3 DivMoE 3.6B/12B 63.1 71.5 73.3 51.4 60.0 74.4 47.5 63.0 Architecture-isolated advantage over the controlled baseline. Table 4 shows that with the same Stage 3 SFT, DivMoE outperforms NVIDIA Upcycling by +6.8 points (63.0 vs. 56.2) and OLMoE by +9.3 points (63.0 vs. 53.7). The largest controlled gaps are on GSM8K (+7.9 over NVIDIA Up.) and MATH (+8.7), indicating that the architectural advantage is concentrated on multi-step reasoning where cross-domain composition is most useful. Match against Moonlight-MoE despite smaller size. DivMoE matches Moonlight-MoE (16B total) using only 12B total parameters (63.0 vs. 62.3 on the 7-benchmark subset; 64.5 vs. 64.5 on the full 8-benchmark set, Appendix D), while leading on MMLU (+1.5), TriviaQA (+7.0), and MATH (+2.2). Moonlight-MoE was trained from scratch with substantially more compute; DivMoE’s upcycling pipeline reaches the same overall performance with only 540B Stage-2 CPT tokens. 5 Ablation Studies 5.1 Component Synergy and the Diversity Constraint We isolate each design decision against the full DivMoE on Qwen3-1.7B-Base (Table 6). Removing any of the three design axes reduces performance, and removing two collapses fine-grained variants to near-random; the combination is far from a trivial union, solving a failure mode (catastrophic collapse) that no individual component addresses. We further ask whether forcing cross-domain co-activation harms domain-specific tasks—a natural concern given that, e.g., a coding token is now obliged to also draw from math, science, or commonsense experts. Table 6 shows the opposite: the constraint improves performance on every domain-specific benchmark evaluated. Table 5: Component synergy ablation on Qwen3-1.7B-Base. “−- DC” removes the diversity constraint; “DU-f” removes domain diversity; “BTX” removes fine-grained partitioning. Configuration Domain Fine- Diversity Avg. experts grained constraint (15 bm.) Drop-Upcycling-f ✗ ✓ ✗ 23.2 NVIDIA Upcycling ✗ ✓ ✗ 51.1 BTX ✓ ✗ ✗ 51.6 DivMoE (−- DC) ✓ ✓ ✗ 52.5 DivMoE ✓ ✓ ✓ 55.6 Table 6: Effect of the diversity constraint on domain-specific benchmarks (Qwen3-1.7B-Base). The constraint improves all four metrics. Method HumanE. MBPP GSM8K MATH DivMoE w/o DC 23.8 31.5 43.2 18.1 DivMoE w/ DC 25.6 33.8 45.3 19.5 Δ +1.8 +2.3 +2.1 +1.4 5.2 Routing Patterns Figure 3: Expert routing patterns across layers and datasets. Sparse Upcycling exhibits routing collapse with few experts dominating. DivMoE without the constraint already shows improved utilization. With the constraint, routing is balanced across domain groups preserving dataset-specific preferences: GSM8K activates more math experts, while HumanEval activates more code experts. Figure 3 visualizes routing probabilities across experts for Sparse Upcycling, DivMoE without the constraint, and full DivMoE. Sparse Upcycling shows routing collapse in deeper layers (over 60% of tokens routed to two experts at layer 36). DivMoE without the constraint already shows improved utilization due to domain-specialized initialization. With the constraint, routing is balanced across groups while preserving within-group specialization (Section 5.1). 5.3 Effect of Expert Granularity Figure 4: Effect of expert granularity (m=km=k) on Qwen3-1.7B-Base. Medium granularity (m=k=2m=k=2) provides the best balance between training-loss reduction and downstream accuracy. We vary m=k∈1,2,4m=k∈\1,2,4\ (Figure 4). Increasing granularity from 1 to 4 reduces training loss progressively, but m=k=4m=k=4 shows mild overfitting on MMLU/HellaSwag. We adopt m=k=2m=k=2 as the default, which balances capacity with generalization while keeping the active-parameter footprint at half of the dense base. 6 Conclusion We presented DivMoE, the first framework for fine-grained MoE upcycling that addresses the previously-unobserved fine-grained upcycling collapse pathology: combining fine-grained expert partitioning with single-source initialization causes downstream accuracy to drop to near-random, irrespective of the partitioning scheme. DivMoE resolves this issue through three components: (i) domain-specialized fine-grained expert initialization, (i) a hard diversity-constrained routing mechanism, and (i) a public-corpus continual pre-training pipeline that no longer regresses below the dense base. Ablation studies demonstrate that the proposed components are complementary and jointly necessary for robust fine-grained upcycling. Across comprehensive experiments, DivMoE consistently achieves the strongest overall performance. Notably, the structural constraint improves performance on every domain-specific benchmark we test, refuting the intuition that cross-domain co-activation should harm specialized tasks. Acknowledgments and Disclosure of Funding Yang You’s research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant and Alibaba grant. This work is also supported by NSF 2048280, 2325121, 2244760, 2331966 and ONR N0001423-1-2300:P00001. Additionally, We thank the open-source community for releasing code and datasets that made this research possible. References [1] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732. [2] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374. [3] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL https://arxiv.org/abs/1905.10044. [4] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457. [5] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. [6] Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024. URL https://arxiv.org/abs/2401.06066. [7] DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. URL https://arxiv.org/abs/2405.04434. [8] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. URL https://arxiv.org/abs/2101.03961. [9] Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret, editors, *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 394–398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https://aclanthology.org/S12-1052/. [10] Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pre-training of large language models: How to (re)warm your model?, 2023. URL https://arxiv.org/abs/2308.04014. [11] Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan, Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shoeybi, and Bryan Catanzaro. Upcycling large language models into mixture of experts, 2025. URL https://arxiv.org/abs/2410.07524. [12] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https://arxiv.org/abs/2009.03300. [13] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. URL https://arxiv.org/abs/2103.03874. [14] Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models, 2024. URL https://arxiv.org/abs/2403.08763. [15] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87, 03 1991. ISSN 0899-7667. doi: 10.1162/neco.1991.3.1.79. URL https://doi.org/10.1162/neco.1991.3.1.79. [16] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023. URL https://arxiv.org/abs/2309.14509. [17] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https://arxiv.org/abs/2401.04088. [18] Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 6(2):181–214, 1994. doi: 10.1162/neco.1994.6.2.181. [19] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL https://arxiv.org/abs/1705.03551. [20] Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints, 2023. URL https://arxiv.org/abs/2212.05055. [21] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training, 2025. URL https://arxiv.org/abs/2411.15124. [22] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding, 2020. URL https://arxiv.org/abs/2006.16668. [23] Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models, 2022. URL https://arxiv.org/abs/2208.03306. [24] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018. URL https://arxiv.org/abs/1809.02789. [25] Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. Olmoe: Open mixture-of-experts language models, 2025. URL https://arxiv.org/abs/2409.02060. [26] Taishi Nakamura, Takuya Akiba, Kazuki Fujii, Yusuke Oda, Rio Yokota, and Jun Suzuki. Drop-upcycling: Training sparse mixture of experts with partial re-initialization, 2025. URL https://arxiv.org/abs/2502.19261. [27] OLMo Team, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 Furious, 2025. URL https://arxiv.org/abs/2501.00656. [28] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad, 2018. URL https://arxiv.org/abs/1806.03822. [29] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022. [30] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. URL https://arxiv.org/abs/1701.06538. [31] Weijia Shi, Akshita Bhagia, Kevin Farhat, Niklas Muennighoff, Pete Walsh, Jacob Morrison, Dustin Schwenk, Shayne Longpre, Jake Poznanski, Allyson Ettinger, Daogao Liu, Margaret Li, Mike Lewis, Wen tau Yih, Dirk Groeneveld, Luca Soldaini, Kyle Lo, Noah A. Smith, Luke Zettlemoyer, Pang Wei Koh, Hannaneh Hajishirzi, Ali Farhadi, and Sewon Min. FlexOLMo: Open language models for flexible data use, 2025. URL https://arxiv.org/abs/2507.07024. [32] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research, 2024. URL https://arxiv.org/abs/2402.00159. [33] Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen tau Yih, Jason Weston, and Xian Li. Branch-train-mix: Mixing expert llms into a mixture-of-experts llm, 2024. URL https://arxiv.org/abs/2403.07816. [34] Lintang Sutawika, Hailey Schoelkopf, Leo Gao, Baber Abbasi, Stella Biderman, Jonathan Tow, Charles Lovering, Jason Phang, Anish Thite, Thomas Wang, et al. Eleutherai/lm-evaluation-harness: v0. 4.9. Zenodo, 2025. doi: 10.5281/zenodo.15699229. URL https://ui.adsabs.harvard.edu/abs/2025zndo..15699229S. [35] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them, 2022. URL https://arxiv.org/abs/2210.09261. [36] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [37] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830. [38] Danyang Zhang, Junhao Song, Ziqian Bi, Xinyuan Song, Yingfang Yuan, Tianyang Wang, Joe Yeong, and Junfeng Hao. Mixture of experts in large language models, 2025. URL https://arxiv.org/abs/2507.11181. [39] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon. Mixture-of-experts with expert choice routing, 2022. URL https://arxiv.org/abs/2202.09368. [40] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. ST-MoE: Designing stable and transferable sparse expert models, 2022. URL https://arxiv.org/abs/2202.08906. Appendix A Limitations and Future Work Our experiments use base models up to 4B parameters, yielding MoE models up to 12B; scaling to 7B-13B base models is an important direction. Our domain partition is fixed to four broad domains; learning the partition end-to-end is left to future work. Finally, DivMoE requires an additional 40B Stage-1 token budget per base model, an 8% overhead relative to from-scratch training, which is small in absolute terms but non-trivial for very-large-scale settings. Appendix B Implementation Details B.1 Model Architecture Table 7 summarizes the architecture configurations for each base model. All DivMoE variants use n=4n=4 domain groups, m=2m=2 experts per group (N=8N=8 total), and k=2k=2 active experts per token. Table 7: Model architecture configurations. DivMoE parameters are derived from each base model with expansion factor n=4n=4. Configuration Qwen3-4B Qwen3-1.7B Llama3.2-1B Hidden size 2560 2048 2048 FFN intermediate size 6912 5632 8192 Num layers 36 28 16 Num attention heads 32 16 32 Num KV heads 4 4 8 Vocab size 151,936 151,936 128,256 DivMoE Configuration Domain groups (n) 4 4 4 Experts per group (m) 2 2 2 Total experts (N) 8 8 8 Active experts (k) 2 2 2 Expert FFN dim 3456 2816 4096 Dense params 4B 1.7B 1B Total MoE params 12B 5.4B 3.2B Active params 3.6B 2.4B 1.8B B.2 Training Pipeline Details This section expands Section 3.3 with full descriptions of each pipeline stage. Throughout, we use only public corpora so all results are independently reproducible. Stage 1: Domain-specialist initialization (40B tokens). We continually pre-train four copies of the base model (next-token-prediction loss on full sequences, not instruction-style SFT) on domain-specific subsets of public pre-training corpora, producing ℳmathM_math, ℳcodeM_code, ℳscienceM_science, ℳcommonM_common. We use the per-domain partitions of Dolmino-Mix [27]—specifically its math, code, academic, and web subsets—which jointly cover the four target domains. Each specialist sees 10B tokens (40B total). This is the only stage where the base model weights are updated as a dense model; subsequent stages train the assembled MoE. Stage 2: Continual pre-training of the assembled MoE (500B tokens). After constructing the DivMoE architecture (Section 3.1), we continually pre-train the entire MoE on allenai/dolma3_dolmino_mix-100B-1025 [27, 32], a high-quality public mid-training corpus, for 500B tokens. This stage adapts the model to sparse computation, learns the router weights, and enables cross-expert collaboration. Crucially, this is the same corpus, hyperparameters, and step count used for every upcycling baseline, ensuring controlled comparison. Stage 3: Supervised fine-tuning (used only for the open-source MoE comparison). For the comparison against open-source MoE checkpoints in Section 4.3, we additionally fine-tune DivMoE on Tülu-3-SFT-Mixture [21], a public reasoning-and-instruction mixture, with sequence length 16,384. To isolate architectural effects from data effects, we apply the same Stage 3 SFT to the strongest baseline (NVIDIA Upcycling) and to OLMoE; the controlled comparison is reported in Table 4. No private data. The pipeline uses only the dense base checkpoints, Dolmino-Mix-1124, and Tülu-3-SFT-Mixture, all of which are publicly available; we release the slicing and routing code together with the per-domain Dolmino-Mix splits. B.3 Training Hyperparameters Table 8 details the hyperparameters for each stage. All experiments use AdamW (β1=0.9 _1=0.9, β2=0.95 _2=0.95) with bf16 mixed precision and global batch size of 2M tokens. Table 8: Training hyperparameters by stage. Hyperparameter Stage 1 Stage 2 Stage 3 Domain CPT Joint CPT SFT Learning rate 5×10−55×10^-5 5×10−45×10^-4 2×10−52×10^-5 Min learning rate 5×10−65×10^-6 5×10−55×10^-5 2×10−62×10^-6 LR scheduler Cosine Cosine Cosine Warmup steps 50 256 500 Weight decay 0.1 0.1 0.1 Sequence length 4,096 4,096 16,384 Training tokens 10B per domain (40B total) 500B – Gradient checkpointing ✓ ✓ ✓ For Stage 3 with long sequences, we use DeepSpeed Ulysses [16] sequence parallelism (factor 4) and CPU offloading of optimizer states. B.4 Compute Budget Table 9 compares compute budgets across methods. DivMoE and BTX share identical total compute (540B tokens; the additional 40B Stage-1 tokens are dense-model compute, substantially cheaper per-token than MoE training). Table 9: Per-method compute. GPU-hours measured on NVIDIA H20 (80GB), Qwen3-1.7B-Base. Method Stage-1 Tokens Stage-2 Tokens Total GPU-hours From Scratch 0 500B (MoE) 500B ∼ 26,500 Sparse Upcycling 0 500B (MoE) 500B ∼ 26,500 BTX 40B (dense) 500B (MoE) 540B ∼ 28,800 NVIDIA Upcycling 0 500B (MoE) 500B ∼ 26,500 Drop-Upcycling 0 500B (MoE) 500B ∼ 26,500 DivMoE 40B (dense) 500B (MoE) 540B ∼ 28,800 B.5 Evaluation Configuration We evaluate using LM Evaluation Harness [34] with greedy decoding (temperature 0) for generative tasks and likelihood-based scoring for multiple-choice tasks. Table 10 lists the format and shot count for each benchmark. Table 10: Evaluation format and shot count per benchmark. Benchmark Format Shots ARC-Challenge Likelihood 25 ARC-Easy Likelihood 25 BoolQ Likelihood 0 COPA Likelihood 0 MMLU Likelihood 5 OpenBookQA Likelihood 0 TriviaQA Generative 5 HellaSwag Likelihood 10 SQuAD2 Generative 1 HumanEval Generative 0 MBPP Generative 3 GSM8K Generative 8 MATH Generative 4 GPQA Likelihood 0 BBH Generative 3 Appendix C Full Upcycling Results (15 Benchmarks) Tables 11 and 12 report the complete 15-benchmark upcycling comparison underlying the abridged Tables 2 and 3 in the main paper. We additionally include reference rows for the dense base model, the four individual domain specialists, and a data-matched dense baseline (the dense base continually pre-trained on the same 540B tokens, no MoE conversion). These rows isolate the effect of the MoE architecture from the effect of the additional CPT data. Table 11: Full upcycling comparison on Qwen3-1.7B-Base after 500B Stage 2 CPT tokens (540B for BTX and DivMoE, including Stage 1). Best across upcycling methods in bold; reference rows (above the rule) provide context. F.G. denotes fine-grained partitioning. Method F.G. ARC-c ARC-e BoolQ COPA MMLU OBQA TriviaQ HellaS. SQuAD2 GSM8K MATH HumanE. MBPP GPQA BBH Avg. Reference rows (not upcycling baselines) Qwen3-1.7B-Base – 43.2 72.4 72.8 76.0 62.6 38.6 48.5 63.2 32.5 65.8 38.4 46.5 54.8 26.5 53.5 53.0 Math specialist – 39.8 60.5 64.8 68.2 59.5 34.2 37.8 56.5 25.8 76.2 48.5 40.3 47.5 27.8 49.6 49.1 Code specialist – 36.5 55.2 60.4 64.8 54.1 32.5 34.8 53.8 23.5 52.4 30.5 57.8 61.5 24.2 44.3 45.8 Science specialist – 46.8 73.2 70.5 73.8 61.8 42.5 44.2 61.5 30.2 60.5 37.2 39.5 48.3 31.8 52.6 51.6 Commonsense specialist – 42.2 68.5 76.2 80.5 59.2 40.8 46.5 66.2 33.5 55.3 33.8 37.2 46.1 25.0 54.8 51.1 Dense baseline (same data) – 44.0 73.1 73.4 76.3 63.2 39.0 48.8 63.0 32.7 66.0 38.7 47.0 55.4 26.8 53.8 53.4 Upcycling methods From Scratch ✓ 12.2 22.0 30.5 38.7 40.5 27.0 33.0 42.5 17.2 5.5 2.3 6.8 11.2 22.6 20.5 22.2 Sparse Upcycling ✗ 38.5 62.8 60.2 65.4 56.8 35.0 43.5 54.5 27.0 56.5 28.5 38.0 47.5 24.5 47.5 45.7 BTX ✗ 43.0 70.5 71.0 74.0 61.5 39.5 47.5 61.0 30.5 64.0 36.5 44.0 53.0 26.0 51.5 51.6 NVIDIA Upcycling ✓ 42.5 69.5 70.5 73.0 61.5 38.5 47.5 60.5 30.0 62.5 35.5 44.5 53.5 25.8 51.0 51.1 Drop-Upcycling ✗ 41.5 69.0 71.5 75.0 61.0 37.0 47.0 59.5 31.5 60.0 33.5 41.0 50.5 25.0 50.0 50.2 Drop-Upcycling-f ✓ 14.5 24.5 32.5 41.5 42.0 26.0 33.5 41.5 17.5 7.0 3.5 7.8 12.5 22.5 22.0 23.2 DivMoE ✓ 45.0 73.5 75.0 78.0 64.5 42.0 50.0 63.5 33.5 68.5 40.5 49.0 57.5 28.0 55.5 55.6 Table 12: Full upcycling comparison on Llama3.2-1B after 500B Stage 2 CPT tokens (540B for BTX and DivMoE). Same conventions as Table 11. Method F.G. ARC-c ARC-e BoolQ COPA MMLU OBQA TriviaQ HellaS. SQuAD2 GSM8K MATH HumanE. MBPP GPQA BBH Avg. Llama3.2-1B (ref.) – 34.8 63.5 65.2 68.0 49.3 32.4 38.5 55.2 28.5 28.5 10.2 16.5 23.8 24.8 32.4 38.1 Dense baseline (same data) – 35.2 63.8 65.7 68.5 50.0 32.8 39.0 55.4 28.8 29.0 10.5 17.0 24.2 24.9 32.7 38.5 From Scratch ✓ 13.0 22.5 27.5 32.0 38.0 23.0 31.0 36.0 17.5 4.0 1.5 4.5 8.0 21.8 18.0 19.9 Sparse Upcycling ✗ 33.5 56.0 60.5 65.0 48.0 30.5 39.5 52.5 26.5 26.5 9.0 15.0 21.5 23.5 31.5 35.9 BTX ✗ 36.0 64.5 66.0 69.0 50.5 33.0 39.5 56.0 29.0 31.5 11.0 17.5 24.5 25.0 33.0 39.1 NVIDIA Upcycling ✓ 36.0 64.0 65.5 68.5 50.0 33.0 39.0 55.5 28.5 30.5 10.8 17.0 24.0 24.8 32.8 38.7 Drop-Upcycling ✗ 35.0 63.5 66.0 69.0 49.5 32.5 39.0 55.0 28.0 29.5 10.5 16.5 23.5 24.5 32.5 38.3 Drop-Upcycling-f ✓ 25.5 28.5 33.0 36.0 43.5 24.5 38.0 39.5 17.0 6.0 2.5 6.0 9.5 21.5 19.0 23.3 DivMoE ✓ 37.5 65.5 67.5 70.0 52.5 34.5 42.0 56.5 30.0 34.5 13.5 20.0 27.0 26.0 34.5 40.8 Appendix D Full Open-Source MoE Comparison (8 Benchmarks) Table 13 reports the full 8-benchmark comparison underlying the abridged Table 4 in the main paper, additionally including ARC-Easy. The 8-benchmark average (DivMoE 64.5, Moonlight-MoE 64.5) is the figure cited in the abstract. Table 13: Comparison with open-source MoE models on the full 8-benchmark set. Same conventions as Table 4. Model Size ARC-c ARC-e MMLU TriviaQA HumanE. MBPP GSM8K MATH Avg. OLMoE 1B/7B 49.2 76.9 51.9 60.4 51.8 61.2 45.5 23.9 52.6 OLMoE + Tülu-3 SFT 1B/7B 52.4 77.2 55.8 63.5 53.5 62.8 56.8 31.2 56.7 DeepSeek-V2-Lite 2.4B/16B 52.1 70.3 58.3 65.1 29.9 43.2 41.1 17.1 47.1 NVIDIA Up. + Tülu-3 SFT 2.4B/5.4B 56.3 73.8 64.2 67.5 45.8 54.2 66.5 38.8 58.4 Moonlight-MoE 2.4B/16B 65.5 79.3 70.0 66.3 48.1 63.8 77.4 45.3 64.5 DivMoE 3.6B/12B 63.1 75.0 71.5 73.3 51.4 60.0 74.4 47.5 64.5 Appendix E Attention Layer Initialization Ablation We compare three attention initialization strategies for the non-expert components (Equation 6). Mean initialization—averaging the base model and all specialists—is the default. Table 14: Attention initialization ablation on Qwen3-1.7B-Base after 500B Stage-2 CPT. Avg. is over the 9 general-knowledge benchmarks for direct comparability with prior reports. Initialization ARC-c MMLU HellaSwag Avg. (9 bm.) Mean of base + all specialists (default) 45.0 64.5 63.5 58.3 Base model only 44.5 64.0 63.2 57.9 Random specialist selection 43.5 63.0 62.5 57.0 Key takeaways. The default mean initialization is the best of the three, with the base-only variant trailing by 0.4 points. Random per-layer specialist selection performs worst (−-1.3). The modest spread between mean and base-only confirms that the primary driver of DivMoE’s advantage is the FFN expert initialization, not the attention layer; mean initialization contributes a small but consistent boost. Appendix F Learning Dynamics Figure 5 shows training loss, validation perplexity, and average downstream accuracy across Stage 2 CPT for Qwen3-1.7B-Base and Llama3.2-1B. Table 15 extends the comparison to 300B and 500B tokens for the strongest baselines. Token-by-token tabulations are deferred to Appendix G. Figure 5: Stage 2 learning dynamics. DivMoE achieves lower training loss and validation perplexity throughout training and converges to higher downstream performance, on both Qwen3-1.7B (top) and Llama3.2-1B (bottom). Table 15: Average accuracy (15 benchmarks) on Qwen3-1.7B-Base at extended checkpoints. DivMoE’s advantage is sustained: gaps narrow only mildly between 150B and 500B, indicating an architectural rather than initialization-only benefit. Method @150B @300B @500B NVIDIA Upcycling 47.5 49.5 51.1 Drop-Upcycling 46.5 48.5 50.2 DivMoE 52.0 54.0 55.6 Sustained advantage. DivMoE leads at every checkpoint. The gap to NVIDIA Upcycling is 4.5 points at 150B and 4.5 at 500B; the absolute advantage is sustained throughout training, supporting the conclusion that the architectural benefits do not vanish with additional data. Appendix G Extended Learning Dynamics Tables 16 and 17 report training loss, validation perplexity, and average downstream accuracy at 10B-token intervals during Stage 2 CPT. Best (lowest loss/PPL, highest accuracy) per checkpoint in bold. Table 16: Learning dynamics for Qwen3-1.7B-Base. Values reported at 10B-token intervals. Tokens (B) 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 Training Loss From Scratch 3.50 3.23 3.00 2.79 2.64 2.50 2.41 2.32 2.23 2.17 2.12 2.08 2.04 2.01 1.98 1.95 Sparse Upcycling 2.38 2.25 2.15 2.07 2.00 1.94 1.89 1.85 1.81 1.78 1.75 1.72 1.70 1.68 1.69 1.65 BTX 2.40 2.34 2.25 2.20 2.10 2.02 1.95 1.89 1.84 1.80 1.76 1.73 1.70 1.67 1.65 1.63 NVIDIA Upcycling 2.70 2.58 2.46 2.24 2.16 2.01 1.91 1.83 1.78 1.74 1.70 1.67 1.64 1.61 1.59 1.57 Drop-Upcycling 2.64 2.40 2.25 2.13 2.02 1.92 1.85 1.79 1.74 1.70 1.66 1.63 1.60 1.57 1.55 1.53 DivMoE 2.50 2.35 2.21 2.12 2.01 1.93 1.79 1.70 1.64 1.59 1.55 1.53 1.50 1.48 1.45 1.43 Validation Perplexity From Scratch 35.0 28.5 24.0 21.0 18.5 16.8 15.4 14.3 13.4 12.7 12.1 11.6 11.2 10.8 10.5 10.2 Sparse Upcycling 22.0 18.5 16.2 14.5 13.2 12.2 11.4 10.7 10.1 9.6 9.2 8.9 8.6 8.3 8.1 7.9 BTX 22.5 19.0 17.0 13.9 12.3 11.4 10.6 10.0 9.5 9.1 8.7 8.4 8.1 7.9 7.7 7.5 NVIDIA Upcycling 25.0 21.2 17.3 13.9 12.0 10.9 10.2 9.6 9.1 8.7 8.3 8.0 7.7 7.5 7.3 7.1 Drop-Upcycling 24.0 18.8 15.4 12.6 11.5 10.6 9.9 9.3 8.8 8.4 8.0 7.7 7.4 7.2 7.0 6.8 DivMoE 23.3 18.5 14.8 11.9 10.5 9.7 9.0 8.5 8.0 7.6 7.3 7.0 6.7 6.5 6.3 6.1 Average Downstream Accuracy (%) From Scratch 5.0 15.2 10.3 19.0 22.3 27.5 21.4 29.6 31.4 34.4 33.0 31.7 34.7 32.8 33.8 35.5 Sparse Upcycling 15.0 16.2 20.5 22.0 24.0 32.5 37.7 34.6 39.2 35.6 41.8 42.9 43.9 40.8 45.6 44.3 BTX 13.0 19.0 27.2 20.5 23.3 30.7 37.8 39.6 35.2 42.5 43.7 44.8 45.7 40.5 47.3 48.0 NVIDIA Upcycling 20.0 25.0 26.0 24.2 33.0 34.3 39.3 41.1 40.6 43.9 45.1 44.1 47.0 47.8 45.5 44.2 Drop-Upcycling 21.0 26.0 25.0 34.2 36.0 38.3 35.3 39.1 43.6 45.0 44.2 46.2 47.1 48.9 47.6 50.2 DivMoE 23.0 22.5 29.5 27.5 41.0 38.0 40.6 43.8 45.8 49.5 47.0 50.4 51.6 51.7 52.7 54.0 Table 17: Learning dynamics for Llama3.2-1B. Values reported at 10B-token intervals. Tokens (B) 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 Training Loss From Scratch 3.65 3.35 3.12 2.91 2.74 2.60 2.50 2.41 2.33 2.26 2.20 2.15 2.11 2.07 2.04 2.01 Sparse Upcycling 2.45 2.32 2.21 2.12 2.05 1.99 1.94 1.90 1.86 1.83 1.80 1.77 1.75 1.73 1.74 1.70 BTX 2.48 2.41 2.31 2.25 2.15 2.07 2.00 1.94 1.89 1.85 1.81 1.78 1.75 1.72 1.70 1.68 NVIDIA Upcycling 2.78 2.65 2.52 2.30 2.21 2.06 1.96 1.88 1.83 1.79 1.75 1.72 1.69 1.66 1.64 1.62 Drop-Upcycling 2.72 2.47 2.31 2.18 2.07 1.97 1.90 1.84 1.79 1.75 1.71 1.68 1.65 1.62 1.60 1.58 DivMoE 2.58 2.42 2.27 2.17 2.06 1.98 1.90 1.82 1.74 1.67 1.60 1.58 1.55 1.53 1.51 1.50 Validation Perplexity From Scratch 38.0 31.0 26.2 22.8 20.1 18.2 16.7 15.5 14.5 13.7 13.0 12.5 12.0 11.6 11.2 10.9 Sparse Upcycling 24.5 20.5 18.8 16.9 14.4 13.2 12.3 11.5 10.9 10.3 9.9 9.5 9.2 8.9 8.6 8.4 BTX 25.0 21.0 19.5 15.2 14.4 12.3 11.4 10.7 10.1 9.7 9.3 9.0 8.7 8.4 8.2 8.0 NVIDIA Upcycling 27.5 23.4 19.3 16.2 14.1 12.0 10.9 10.2 9.7 9.2 8.8 8.5 8.2 7.9 7.7 7.5 Drop-Upcycling 26.5 22.2 17.8 14.8 12.9 11.4 10.6 9.9 9.4 8.9 8.5 8.2 7.9 7.6 7.4 7.2 DivMoE 25.8 20.2 16.0 12.9 11.3 10.3 9.5 8.9 8.4 8.0 7.6 7.3 7.0 6.8 6.5 6.3 Appendix H Granularity Ablation Details Table 18 gives the full numerical breakdown of Figure 4, on Qwen3-1.7B-Base. Table 18: Effect of expert granularity on training dynamics. m=k=1m=k=1 corresponds to coarse-grained MoE (4 experts), m=k=2m=k=2 is the default DivMoE (8 experts), and m=k=4m=k=4 represents finer-grained MoE (16 experts). Tokens (B) 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 Training Loss m=k=1m=k=1 2.80 2.52 2.35 2.22 2.12 2.04 1.97 1.91 1.86 1.82 1.78 1.75 1.72 1.69 1.67 1.65 m=k=4m=k=4 2.97 2.63 2.48 2.21 2.07 1.97 1.87 1.81 1.76 1.71 1.67 1.64 1.61 1.58 1.56 1.54 == m=k=2 2.90 2.58 2.36 2.15 2.01 1.94 1.85 1.76 1.71 1.66 1.62 1.59 1.56 1.53 1.51 1.49 Validation Perplexity m=k=1m=k=1 16.5 12.8 10.9 9.6 8.7 8.0 7.5 7.1 6.7 6.4 6.2 6.0 5.8 5.6 5.5 5.4 m=k=4m=k=4 18.9 15.2 12.1 9.8 8.2 7.2 6.7 6.3 5.9 5.6 5.4 5.2 5.0 4.8 4.7 4.6 == m=k=2 17.8 14.8 11.6 9.3 8.1 6.7 6.2 5.8 5.4 5.1 4.9 4.7 4.5 4.4 4.3 4.2 MMLU Accuracy (%) m=k=1m=k=1 20.0 24.2 25.5 29.0 30.5 32.0 31.5 35.0 36.3 35.5 38.5 39.5 40.3 41.0 41.6 42.1 m=k=4m=k=4 14.3 17.0 23.2 29.5 32.5 36.0 34.0 39.8 41.3 40.7 42.9 44.9 45.8 45.5 47.1 47.6 == m=k=2 16.2 17.5 25.2 34.0 33.8 38.5 37.0 42.8 44.5 43.0 45.3 48.4 49.3 49.1 50.7 51.2 HellaSwag Accuracy (%) m=k=1m=k=1 20.0 24.5 22.5 27.0 30.0 33.5 31.8 36.8 36.5 40.0 42.3 42.5 44.5 43.4 45.2 45.9 m=k=4m=k=4 16.4 22.0 20.5 26.0 34.0 37.2 36.0 38.3 40.3 42.0 43.5 43.8 45.9 45.8 47.6 48.3 == m=k=2 18.5 23.2 22.5 30.8 35.2 35.8 41.8 42.3 42.5 44.3 45.9 46.3 47.5 47.5 48.3 49.0 Key observations. (1) m=k=2m=k=2 achieves the best trade-off between training stability (lower initial loss than m=k=4m=k=4) and final accuracy. (2) m=k=1m=k=1 converges fastest initially but plateaus at lower accuracy due to limited expert capacity. (3) The granularity gap widens on harder benchmarks (MMLU 9.1% vs. HellaSwag 3.1% at 150B), suggesting fine-grained experts benefit knowledge-intensive tasks the most.