Paper deep dive
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Jing Li, Jian Meng, Yingmeng Gao, Suming Qiu, Linyuan Qiu, Dongfang Li, Baotian Hu, Binfan Zheng, Rongqian Zhao, Weijian Sun, Xin Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mixture-of-Experts (MoE) has been widely adopted in recent large language model (LLM) architectures. However, scaling up MoE in LLM training introduces system-level challenges on training, where non-uniform token routing can lead to highly imbalanced workloads across experts and devices, further destabilizing the training process. With trillion-scale LLMs, imbalanced expert workloads further amplify the resource cost of MoE training, resulting in degraded training efficiency and hardware utilization for underloaded experts, while hot experts require additional resources to accommodate excessive workloads. Recent studies address imbalanced MoE training through intricate parallelism strategies or resource reallocation. However, these system-level approaches often introduce additional resource requirements and considerable orchestration complexity, which become increasingly difficult to afford when training trillion-parameter LLMs under constrained computational resources. This work introduces CIPHER-MoE, which mitigates MoE workload imbalance while keeping the router's token-side Top-K selection unchanged. CIPHER-MoE applies affinity-aware Expert-to-Token filtering with explicit capacity control to reduce hotspot expert workloads without additional hardware resources or complex runtime design. The proposed method has been evaluated on large-scale MoE models, including DeepSeek-V4-Pro, showing up to 64.9 percentage points Top-1 expert workload reduction and 1.10$\times$-1.94$\times$ training acceleration, while preserving the training quality. The source code will be released soon.
Tags
Links
- Source: https://arxiv.org/abs/2610.05744v1
- Canonical: https://arxiv.org/abs/2610.05744v1
Trouble viewing inline? Open PDF directly →
Full Text
79,396 characters extracted from source content.
Expand or collapse full text
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training Jing Li1, Jian Meng2,*, Yingmeng Gao, Suming Qiu Linyuan Qiu, Dongfang Li3,4, Baotian Hu3,4, Binfan Zheng Rongqian Zhao, Weijian Sun, Xin Chen 1Tongji University 2Cornell University 3Harbin Institute of Technology, Shenzhen 4AI Training Platform Team, Shenzhen Loop Area Institute †thanks: Equal contribution.†thanks: Corresponding author: jm2787@cornell.edu Abstract Mixture-of-Experts (MoE) has been widely adopted in recent large language model (LLM) architectures. However, scaling up MoE in LLM training introduces system-level challenges on training, where non-uniform token routing can lead to highly imbalanced workloads across experts and devices, further destabilizing the training process. With trillion-scale LLMs, imbalanced expert workloads further amplify the resource cost of MoE training, resulting in degraded training efficiency and hardware utilization for underloaded experts, while hot experts require additional resources to accommodate excessive workloads. Recent studies address imbalanced MoE training through intricate parallelism strategies or resource reallocation. However, these system-level approaches often introduce additional resource requirements and considerable orchestration complexity, which become increasingly difficult to afford when training trillion-parameter LLMs under constrained computational resources. This work introduces CIPHER-MoE, which mitigates MoE workload imbalance while keeping the router’s token-side Top-K selection unchanged. CIPHER-MoE applies affinity-aware Expert-to-Token filtering with explicit capacity control to reduce hotspot expert workloads without additional hardware resources or complex runtime design. The proposed method has been evaluated on large-scale MoE models, including DeepSeek-V4-Pro, showing up to 64.9 percentage points Top-1 expert workload reduction and 1.10×–1.94× training acceleration, while preserving the training quality. The source code will be released soon. 1 Introduction Large language models (LLMs) have become foundational components of modern AI systems (Bommasani et al., 2021). The continued effectiveness of scaling laws has demonstrated that increasing model capacity can substantially improve performance (Kaplan et al., 2020). However, naively scaling dense Transformers toward the trillion-parameter regime incurs prohibitively high computational and memory costs, motivating the adoption of Mixture-of-Experts (MoE) architectures (Fedus et al., 2022). In particular, MoE decouples model capacity from per-token computation by replacing the dense feed-forward network (FFN) with multiple sparsely activated experts and dynamically routing each token to only a small subset of them, leading to a balanced tradeoff between scalability and computational efficiency (Shazeer et al., 2017). Recent studies have demonstrated the strong potential of domain-specific supervised fine-tuning (SFT) and reinforcement learning (RL) for advancing LLM post-training (Guo et al., 2025). Training MoE models introduces substantial system-level challenges due to workload imbalance, which degrades communication efficiency, hardware utilization, and training stability (Jin et al., 2026b). The inherently non-uniform data distribution across domains and tasks can induce highly skewed token routing, resulting in uneven workloads across different experts. For the expensive trillion-parameter LLMs, cross-rank expert parallelism is almost mandatory for efficient training. However, the lopsided computation impedes efficient distributed training with unbalanced communication and computation (Rajbhandari et al., 2022). Moreover, the overloaded experts may incur excessive memory consumption and instantly trigger out-of-memory failures on large-scale computing clusters, ultimately undermining training efficiency and stability (Nguyen et al., 2026). Figure 1: Expert workload imbalance in DeepSeek-V4-Flash. (a) Illustration of the MoE architecture with data-dependent routing, which leads to the imbalanced workload. (b) Lopsided workload statistics across all layers of the DeepSeek-V4-Flash model. With the domain-specific dataset, 10% of the experts consume over 88% routing workload, leading to degraded training efficiency. Recent studies on efficient MoE training are two-fold; system-level approaches redistribute workloads through dynamic expert migration or replication (Nie et al., 2023). Meanwhile, MoE-specific kernels optimize expert-parallel execution through aggressive communication-computation pipeline design (Zhang et al., 2025). At the algorithm level, recent studies investigate global-batch balancing, routing regularization, or transfer the pre-trained routing strategy to fresh training (Lewis et al., 2021). However, system-level approaches largely accommodate routing imbalance after it arises, requiring additional expert replicas, collective communication, or specialized runtime scheduling, whose overhead and implementation complexity become increasingly pronounced with trillion-parameter LLMs training. On the other hand, regularization-based routing penalty failed to handle the instantaneous memory peak (Fedus et al., 2022), while the heuristic batch balancing or experience-based routing distillation exhibits poor scalability to the trillion-parameter MoE models. Given the research gaps in existing algorithm and system-level approaches for MoE LLM training, this work aims to investigate the following problem: How to achieve efficient and stable training for trillion-parameter MoE LLMs without intricate system design or heuristic-based routing? In this work, we propose CIPHER-MoE, a plug-and-play algorithm for stable-and-efficient trillion-parameter MoE model training. Although the fixed routing strategy ensures the perfect load balance with stabilized training process, the reduced expert specialization leads to the suboptimal intelligence in the resulting trained model (Dikkala et al., 2023). On the other hand, freely-exploited expert routing amplifies the workload imbalance (Guo et al., 2026), especially with domain-specific post-training phase. In contrast, the proposed CIPHER-MoE tackles the problem by investigating a bounded learning capacity of different experts and proves the feasibility of dropping tokens from expert training. Different from heuristic token dropping, this work first reveals the relationship between optimal token budget and training efficiency for trillion-parameter MoE training. We validate the proposed method via both theoretical analysis and empirical experiments across multiple large-scale MoE models (from 284B to 1.6T). The major contributions of this paper are: • Simplicity. CIPHER-MoE directly exploits the bounded token budget for MoE training, achieving high efficiency while preserving the LLM training quality. • Theoretical Foundation. CIPHER-MoE derives a sufficient expert capacity for similarity-based admission and a necessary capacity for natural-order admission. • Scalability. CIPHER-MoE scales effectively across different model sizes, architectures, and datasets. Across DeepSeek-V3, DeepSeek-V4-Flash, GLM-5, and DeepSeek-V4-Pro, achieving 1.10×1.10×–1.94×1.94× measured training-time speedups. 2 Related Work System-level workload balancing and efficient MoE execution. Workload imbalance is a major bottleneck for efficient distributed MoE training (Hwang et al., 2023), where skewed token routing creates expert stragglers, uneven communication, and poor accelerator utilization. System-level work mitigates this imbalance by adapting the mapping between experts and hardware. FasterMoE and FlexMoE replicate, resize, or migrate experts according to runtime load (He et al., 2022; Nie et al., 2023), while some works combine balancing with parameter offloading, decoupled parameter/optimizer-state placement, or sharded expert re-layout (Kim et al., 2024; Skiadopoulos et al., 2025; Liu et al., 2026). In parallel, another line of work reduces the overhead of sparse and irregular MoE computation. Tutel and PipeMoE (Hwang et al., 2023; Shi et al., 2023) optimize expert-parallel execution through adaptive parallelism and communication–computation pipelining; MegaBlocks (Gale et al., 2023) uses block-sparse kernels to process variable expert workloads without padding or token dropping; MegaScale-MoE (Jin et al., 2026a) jointly optimizes parallelism, communication–computation overlap, and communication compression. While these techniques improve MoE execution efficiency, they operate after routing and rely on expert replication, parameter migration or re-layout, runtime profiling, and specialized communication scheduling, increasing system complexity at scale. Algorithmic load balancing and expert specialization. Routing offers an alternative approach to load balancing. GShard (Lepikhin et al., 2020) and Switch Transformer (Fedus et al., 2022) use auxiliary load-balancing objectives to encourage uniform expert utilization. However, overly strong local balancing can interfere with the language-modeling objective and suppress expert specialization. Loss-Free Balancing (Wang et al., 2024a), adopted by DeepSeek-V3 (Liu et al., 2024), adjusts per-expert routing biases based on historical loads without introducing auxiliary-loss gradients. Global-Batch Load Balancing (Qiu et al., 2025) aggregates global statistics to preserve domain specialization. Related work further highlights the tension between strict balancing and expert differentiation (Dai et al., 2024), while ϕφ-balancing (Chen et al., 2026) formulates expert utilization as a population-level objective rather than relying on noisy mini-batch statistics. These methods improve average utilization through auxiliary objectives, historical statistics, or batch-level assignments rather than bounding instantaneous loads, although peak memory and latency depend on the most heavily loaded expert. Capacity-aware token–expert assignment. Several works are closely related to our capacity-aware perspective. LocMoE (Li et al., 2024) jointly considers workload balance and communication locality, deriving a minimum expert capacity from the angular relationship between token representations and gating weights. This result motivates capacity-aware routing by showing that expert capacity need not be overprovisioned to preserve model quality. Expert Choice Routing (Zhou et al., 2022) instead fixes per-expert capacity by reversing token-to-expert assignment. These approaches show that expert-side token selection can control workloads, but either impose fixed expert-centric assignments or couple capacity optimization with specific locality and routing assumptions. In contrast, CIPHER-MoE analyzes expert capacity for large-scale training and establishes a sufficient capacity bound that enables workload-aware bidirectional token–expert assignment. Tokens first identify preferred experts, which then select the highest-affinity tokens within the derived budget, while excess assignments are rerouted. This design preserves token–expert affinity and controls hotspot workloads without expert replication, parameter migration, or complex runtime orchestration. We evaluate CIPHER-MoE on MoE models ranging from 284B to 1.6T parameters. 3 Proposed Method: CIPHER-MoE 3.1 Preliminaries of Mixture-of-Expert training Sparse expert computation and training. An MoE layer replaces the dense FFN with E expert networks fe(⋅,θe)e=1E\f_e(·; _e)\_e=1^E and a router that activates only K≪EK E experts for each token (Fedus et al., 2022). Consider N token representations ii=1N\ x_i\_i=1^N, with i∈ℝd x_i ^d, processed by one expert-parallel group in a training micro-batch (the layer index is omitted for clarity). Let r∈ℝE×d W_r ^E× d denote the router parameters and e⊤ w_e its e-th row. A conventional softmax router computes the token–expert affinity pi,ep_i,e and selects the corresponding Top-K expert set iS_i: si,e=e⊤i,pi,e=exp(si,e)∑j=1Eexp(si,j),i=TopKe∈1,…,E(pi,e).s_i,e= w_e x_i, p_i,e= (s_i,e) _j=1^E (s_i,j), _i=TopK_e∈\1,…,E\(p_i,e). (1) Here, si,es_i,e is the routing logit, and iS_i contains the indices of the K selected experts. Define the assignment indicator ai,e=[e∈i]a_i,e=1[e _i] and the sparse gate gi,e=ai,epi,eg_i,e=a_i,ep_i,e. The routed MoE output is i=∑e=1Egi,efe(i,θe),∑e=1Eai,e=K. y_i= _e=1^Eg_i,ef_e( x_i; _e), _e=1^Ea_i,e=K. (2) Training optimizes the task loss ℒtaskL_task over the expert parameters and router parameters. Each token contributes task gradients through its selected expert paths. Consequently, increasing E expands the available model capacity while the expert computation per token remains governed by K. Models Method Latency / Iteration (sec) NL4Opt OptiBench Bench4Opt Feasi. Bench4Opt OR DeepSeek-V4-Flash Fixed Router 28.30 (1.00×) 88.93 66.17 70.93 48.73 DeepSeek-V4-Flash Vanilla MoE Training 38.69 (1.37×) 93.08 68.00 71.51 51.02 DeepSeek-V4-Pro Fixed Router 24.68 (1.00×) 89.27 64.67 74.42 54.31 DeepSeek-V4-Pro Vanilla MoE Training OOM - - - - Table 1: Post-training SFT of the DeepSeek-V4-Flash and DeepSeek-V4-Pro model on linear programming and operations research (OR) dataset. OR capabilities evaluated on NL4Opt (Ramamonjison et al., 2023), OptiBench (Yang et al., 2025), Bench4Opt-Feasi. (Wang et al., 2025), and Bench4Opt-OR (Wang et al., 2025). Fixed router exhibits suboptimal performance compared to the vanilla MoE training due to the homogenized expert training. Detailed training setup and resource configurations are summarized in the Appendix A.5. Fixed Router and homogenized experts. With expert parallelism (EP), experts are distributed across R devices, with ℰrE_r denoting the set hosted on device r. Given the unpredictable routing for each incoming token, fixed routing is introduced to enforce the deterministic workload balancing between different experts. For a local routing tensor with M tokens, the j-th expert assigned to token position t is et,jfix=1+((tK+j)modE),t=0,…,M−1,j=0,…,K−1,e^fix_t,j=1+ ((tK+j) E ), t=0,…,M-1, j=0,…,K-1, (3) For K≤EK≤ E, each token is dispatched to K distinct experts, and each expert receives either ⌊MK/E⌋ MK/E or ⌈MK/E⌉ MK/E assignments from this local tensor. The predictable routing strategy ensures the stable and balanced training workload for MoE model at the cost of removing affinity-based expert selection. Semantically unrelated tokens occupying the same positions across micro-batches are repeatedly dispatched to the same expert combinations, potentially weakening expert differentiation, leading to expert homogenization. As a result, the trained model exhibits suboptimal domain specialization with homogenized experts (Qiu et al., 2025). We compare the routing strategy with large-scale MoE model training with the results in Table 1. The standard training paradigm with freely-explored expert workload reaches the ideal performance, with the cost of the degraded training efficiency. With the scaled computing resources, DeepSeek-V4-Pro (1.6T parameters) even failed before the very first iteration of training with the error of out-of-memory (OOM), where the freely-explored expert routing explodes the imbalanced workload, resulting in the instant OOM error. 3.2 CIPHER-MoE: Bidirectional Expert Workload Balancing Expert-token similarity. Given the token hidden state i x_i and expert gating vector e w_e, their cosine similarity is defined as ci,e=e⊤i‖e‖2‖i‖2.c_i,e= w_e x_i\| w_e\|_2\| x_i\|_2. (4) For each expert e, CIPHER-MoE uses ci,ec_i,e to rank the candidate tokens in eP_e. The similarity threshold τ∈[−1,1]τ∈[-1,1] excludes tokens with low similarity, while the expert capacity CeC_e limits the number of tokens processed in each micro-batch. Specifically, let eτ=i∈e:ci,e≥τP_e^τ=\i _e:c_i,e≥τ\ contain the tokens that satisfy the similarity threshold. If more than CeC_e tokens satisfy the threshold, the expert retains the CeC_e tokens with the highest cosine similarity; otherwise, it retains all tokens in eτP_e^τ: eS=TopminCe,|eτ|(ci,e:i∈eτ),bi,eS=[i∈eS],T_e^S=Top_ \C_e,|P_e^τ|\ (\c_i,e:i _e^τ\ ), b_i,e^S=1[i _e^S], (5) where TopqTop_q returns the indices of the q tokens with the highest similarity. The budget CeC_e can vary with the layer, expert, and training stage, while remaining fixed for each micro-batch. Furthermore, the rejection by one expert does not remove a token from its other selected experts, which will be discussed in Section 3.2.2. Although similarity-based token selection directly controls expert workload, two critical questions require further investigation: 1 Under what conditions can similarity-based selection reduce the required expert capacity while retaining informative tokens? 2 How should the routing weights of rejected assignments be handled? 3.2.1 Theoretical analysis of similarity-based token selection To address the first question, we examine why selecting tokens by similarity can require less expert capacity than processing them in their original order. Fix an expert e and its NeN_e candidate tokens. A token is called discriminative if it carries the expert-specific signal described by our assumptions. Let AeA_e denote the event that the candidate pool contains at least one such token. We compare the two selection policies on the same pool, with the same target: Pr(at least one discriminative token is retained∣Ae)≥1−ε,0<ε<1. (at least one discriminative token is retained A_e)≥ 1- , 0< <1. (6) Natural-order selection processes the first C candidates, whereas CIPHER-MoE ranks the candidates by similarity before allocating its C expert-execution slots. Assumptions. Our analysis uses the assumptions of A1–A3 in Appendix A.4.1. First, candidates are independent draws from an expert’s exposure distribution, with signal availability Pr(Ae)≥a0>0 (A_e)≥ a_0>0. Under the spherical reference model, discriminative tokens have normalized affinity at least δ∈(0,1)δ∈(0,1), and expert exposure has probability at least 1/(βE)1/(β E), where β≥1β≥ 1. Second, conditional on AeA_e, a chosen discriminative token survives candidate filtering and thresholding with probability at least 1−η1-η, where 0≤η<ε0≤η< . Third, after conditioning on its survival and previously examined candidates, the probabilities that competing tokens rank ahead of this token have sum at most μ. Thus, η measures the risk of excluding the signal before ranking, while μ controls the capacity needed to retain it after ranking. The signal margin δ is distinct from the threshold τ. Theorem 1 (Similarity-based Capacity Reduction) Under A1–A3, with d≥2d≥ 2, define ℓ =log1ε−η, = 1 -η, G(d,δ) G(d,δ) =−d−12log(1−δ2), =- d-12 (1-δ^2), (7) Be B_e =minNe,⌈1+μ+ℓ3+2μℓ+ℓ29⌉. = \N_e, 1+μ+ 3+ 2μ + ^29 \. Let Cnatural∗C^*_natural and CCIPHER∗C^*_CIPHER be the minimum capacities required by the respective policies to satisfy Eq. 6. Then CCIPHER∗≤Be,Cnatural∗≥(1−ε)a0βEeG(d,δ).C^*_CIPHER≤ B_e, C^*_natural≥ (1- )a_0β Ee^G(d,δ). (8) Consequently, a factor-κ reduction, CCIPHER∗≤Cnatural∗/κC^*_CIPHER≤ C^*_natural/κ for κ>1κ>1, is guaranteed whenever the natural-order lower bound is at least κBeκ B_e. Key proof steps. 1 Signal scarcity. Spherical concentration bounds the discriminative-token probability by πe≤βEe−G(d,δ) _e≤β Ee^-G(d,δ). Processing the first C tokens retains a discriminative token with conditional probability at most Cπe/a0C _e/a_0. Reaching the target therefore requires C≥(1−ε)a0/πeC≥(1- )a_0/ _e, giving the natural-order lower bound. 2 Similarity ranking. Let S count candidates ranked ahead of the surviving discriminative token. Its rank is S+1S+1. A conditional concentration bound on S, using A3, gives the sufficient capacity BeB_e with ranking-failure probability at most ε−η -η. Including filtering failures keeps the total failure probability at most ε . 3 Capacity comparison. Comparing CIPHER-MoE’s sufficient capacity with natural-order selection’s necessary capacity establishes the reduction. The full proof and explicit dimension condition are given in Appendix A.4.2. Figure 2: The overall design of the proposed CIPHER-MoE. The incoming tokens start with the normal Top-K expert selection, followed by the proposed expert-side token drop and reroute. The result explains the benefit of similarity-based selection: An expert can retain an informative token without processing all candidates that precede it in the original order. Under the stated assumptions, Ce≥BeC_e≥ B_e is sufficient for signal retention. This is a requirement on the available capacity, not a minimum number of tokens that must actually be processed; equation 5 still caps the workload at CeC_e. The theorem does not guarantee unchanged training loss or final model quality, and it does not recover signals excluded by an overly restrictive τ. Stability conditions and output-fidelity bounds are provided in Appendices A.4.3 and A.4.4. 3.2.2 Strict Token Drop vs. Token–Expert Reroute Given the theoretical analysis of bounded expert capacity, handling rejected token–expert assignments becomes critical for both the efficiency and quality of LLM training. Strict Token Drop removes each rejected assignment and sets its routing weight to zero, while retaining the token’s other accepted expert paths. Token--Expert Reroute instead redirects the rejected assignment to another candidate expert that satisfies the similarity threshold and has available capacity. Both strategies apply only to routed experts. Shared experts continue to process all incoming tokens, and rejecting an expert assignment does not remove the token from the training sequence or the task loss. Figure 2 illustrates these two mechanisms within CIPHER-MoE. We evaluate the impact of both strategies on large-sized (MoE models >>250B) with domain-specific training tasks. Surprisingly, rerouting tokens to other experts leads to sub-optimal performance compared to the strict token dropping, as shown in Table 2. Although rerouting preserves more expert computation and distributes overflow to experts with spare capacity and sufficient affinity, the additional computation does not necessarily improve training quality. For domain-specific training, the token–expert rerouting shifts the token distributions on which those experts specialize. Even threshold-qualified tokens can induce updates that interfere with an expert’s existing specialization, since cosine affinity alone does not constrain the direction of the task gradient. Strict token dropping avoids the additional assignments and gradient interference, which allows each expert to train on the highest-affinity candidates retained within its budget. 3.2.3 Training Objective of CIPHER-MoE On top of the proposed token selection during MoE training, we adopt the Locality loss proposed by LocMoE (Li et al., 2024), which minimizes the KL divergence between the current expert-assignment distribution and the ideal distribution target corresponding to all the locally-deployed experts (e.g., within the same EP rank), softly encouraging tokens to be routed to local experts while preserving load balance. Overall, the training objective of CIPHER-MoE is formulated as: ℒCIPHERρ(Θ) _CIPHER^ρ( ) =−1||∑t∈logpΘρ(zt∣z<t)+λloc∑ℓ∈ℳℒlocρ,ℓ(Θ), =- 1|U| _t p_ ^ρ (z_t z_<t )+ _loc _ L_loc^ρ, ( ), (9) whereℒlocρ,ℓ _loc^ρ, =−∫Dcρ,ℓ(x)log(Dlρ,ℓ(x)Dcρ,ℓ(x))dx=KL(Dcρ,ℓ∥Dlρ,ℓ). =- D_c^ρ, (x) ( D_l^ρ, (x)D_c^ρ, (x) )\,dx=KL (D_c^ρ, \, \|\,D_l^ρ, ). (10) where DcD_c is the current distribution of expert assignments for tokens in the batch, and DlD_l is the hypothetical fully localized assignment distribution. The loss pushes the actual routing distribution toward one where tokens are preferentially processed by experts that are local to their current communication group of expert parallelism (EP). Mρ,ℓM^ρ, can be determined by the aforementioned strict or reroute strategies, where ρ∈strict,rerouteρ∈\strict,reroute\. Other auxiliary loss terms (e.g., for MTP head and sparse attention indexer) are omitted for better clarity. 4 Experimental Results We validate the proposed CIPHER-MoE algorithm on MoE models ranging from 284B to 1.6T parameters, including DeepSeek-V4-Pro (1.6T) (DeepSeek-AI, 2026), DeepSeek-V4-Flash (284B) (DeepSeek-AI, 2026), GLM-5 (744B) (GLM-5-Team et al., 2026), and DeepSeek-V3 (671B) (Liu et al., 2024). The selected models are trained on domain-specific datasets including OR-based mathematical reasoning, Medical knowledge MedMCQA (Pal et al., 2022), and Social interactions SIQA (Sap et al., 2019). After training the model on domain-specific datasets, we validate the robustness of the trained model on the general-knowledge datasets, as presented in Appendix A.3. We would like to highlight the fact that this work is not deliberately picking the skewed data distributions to enforce the lopsided expert workloads. In fact, the domain-specific fine-tuning on powerful foundation LLMs is widely employed in practical scenarios. The detailed training configurations are summarized in Appendix A.5. While most existing works are validated on small-sized MoE models rather than the mega-sized model, this work uses the vanilla MoE training (provided by the official recipes from Megatron), Fixed-routing enabled MoE training, and LocMoE (Li et al., 2024) as the comparison counterpart. Table 2: Evaluation results and training speedup of domain-specific SFT. The proposed CIPHER-MoE achieves the optimal speedup and quality. Model Method Operations Research MedMCQA SIQA NL4Opt OptiBench B4O-Feas. B4O-OR W. Avg. trainS_train Acc. trainS_train Acc. trainS_train DeepSeek-V4-Flash Vanilla MoE Training (Baseline) 93.08 68.00 71.51 51.02 69.09 - 76.60 - 51.84 - CIPHER-strict 93.77 66.67 70.64 51.27 68.59 1.42× 77.36 1.50× 54.81 1.58× CIPHER-reroute 90.66 66.33 69.77 51.27 67.73 1.38× 76.79 1.46× 57.83 1.52× LocMoE (τ=0.005τ=0.005) 94.12 68.17 66.28 50.76 68.16 1.17× – – – – LocMoE (τ=0.01τ=0.01) 94.81 67.67 70.06 53.05 69.46 1.12× 77.22 1.12× 55.94 1.19× LocMoE (τ=0.02τ=0.02) 93.08 67.50 72.67 51.52 69.27 1.12× – – – – LocMoE (τ=0.03τ=0.03) 93.77 65.67 70.93 52.79 68.65 1.10× – – – – DeepSeek-V3 Vanilla MoE Training (Baseline) 83.39 60.17 58.43 46.45 60.60 - 72.32 - 53.63 - CIPHER-strict 83.74 59.83 61.34 47.46 61.40 1.94× 72.84 1.84× 52.00 1.78× CIPHER-reroute 82.70 61.50 61.05 47.20 61.71 1.68× 73.44 1.62× 52.25 1.56× LocMoE (τ=0.01τ=0.01) 84.08 62.00 59.30 45.94 61.46 1.75× 72.70 1.62× 53.07 1.50× GLM-5 Vanilla MoE Training (Baseline) 85.47 64.67 61.05 45.94 63.06 - 80.66 - 55.83 - CIPHER-strict 90.66 65.67 57.56 46.95 63.86 1.14× 81.31 1.14× 55.02 1.13× CIPHER-reroute 88.58 63.33 58.14 46.95 62.75 1.10× 80.56 1.11× 53.63 1.10× LocMoE (τ=0.01τ=0.01) 88.93 62.33 57.56 47.72 62.51 1.12× 77.77 1.11× 50.15 1.11× 4.1 Training Efficiency and Memory Usage Figure 3: Training efficiency and memory footprint on DeepSeek-V4-Flash. (a) Per-step training time over the training progress, and CIPHER-MoE achieves up to 1.47× speedup. (b) Step-time distributions, where CIPHER-strict and CIPHER-reroute achieve an average speedup of 1.38–1.42× over the baseline. (c) Allocated memory throughout training. Figure 4: Training stability and convergence. (a) LM loss over elapsed training time. (b) Ten-step trailing LM loss and time to reach the 0.10 target. Training efficiency and accelerator memory usage are reported in Figure 3. Since CIPHER-MoE targets very large-scale MoE models and few existing methods provide directly comparable results at this scale, we compare it with LocMoE (Li et al., 2024), which has already been applied to large-cluster training. All reported training times are measured from end-to-end training runs. CIPHER-MoE achieves 1.42× speedups compared to other baselines. In the meantime, CIPHER-MoE limits the allocated memory to approximately 87.5% of the device memory capacity. These gains arise from capacity-aware expert admission, which reduces the number of executed token–expert assignments and the associated computation and communication. The computational overheads introduced by the strict and reroute policies account for only 1.57% and 3.85% of the total training time, respectively, allowing both policies to preserve most of the efficiency gains. 4.2 LLM Training Quality with CIPHER-MoE Figure 4 demonstrates that CIPHER-MoE accelerates convergence without compromising training stability. Both CIPHER-strict and CIPHER-reroute exhibit smooth loss trajectories and converge reliably to the same low-loss regime as the reference methods. The detailed information of gradient norm is also provided in Appendix A.1. With the smooth training process of CIPHER-MoE, the post-trained LLM achieves the optimal performance compared to the baselines, as presented in Table 2. The proposed CIPHER-MoE achieves the best quality-efficiency tradeoff regardless of the model sizes and architectures. Figure 5: Router-update diversity and expert specialization. (a) Five consecutive training stages connected by arrows, with darker colors denoting later stages. Lower update-vector cosine similarity indicates clearer expert differentiation, while higher entropy effective rank indicates more dispersed, less homogeneous updates; the upper-left is better. (b) PCA of router-input hidden states. Compact same-color clusters indicate similar tokens routed to the same expert, while separation across colors indicates distinct token assignments and stronger specialization. 4.3 Expert Update Distributions and Degree of Homogenization As displayed in Figure 5, CIPHER-MoE consistently promotes stronger expert differentiation and avoids expert homogenization. CIPHER-strict exhibits the clearest separation in router adaptation. The agreement between the update dynamics and routed-representation geometry indicates that the resulting specialization reflects a persistent division of labor rather than a transient routing fluctuation. This behavior arises because affinity-aware capacity control suppresses weak or repetitive assignments, while selective rerouting directs overflow tokens toward compatible experts with available capacity, thereby preserving distinct expert learning signals. 4.4 Training Efficiency and Evaluation of DeepSeek-V4-Pro (a) Training efficiency comparison. Method GBS Elapsed Time (s) Loss Time Range (s) DeepSeek-V4-Pro Baseline 256 — (OOM) — — DeepSeek-V4-Pro CIPHER-MoE 256 45.16 0.29–0.43 5.50 8192 538.65 1.41–1.51 91.11 (b) Evaluation results of OR domain benchmarks (%) for OR SFT. Method NL4Opt OptiBench B4O-Feas. B4O-OR Weighted Avg. DeepSeek-V4-Pro Fixed Router 89.27 64.67 74.42 54.31 68.59 DeepSeek-V4-Pro CIPHER-MoE 92.04 65.33 67.44 59.14 69.02 Table 3: DeepSeek-V4-Pro training efficiency and OR-domain evaluation. As shown in Table 3, the DeepSeek-V4-Pro baseline fails to train under limited memory resources due to OOM errors, whereas CIPHER-MoE’s lower memory footprint enables successful training without fixed routing. Training remains stable even at a global batch size of 8192. CIPHER-MoE also outperforms the fixed-router baseline on multiple OR benchmarks and achieves a higher weighted average score. 5 Conclusion This work addresses the expert workload imbalance that limits the efficiency and stability of MoE training, with empirical evaluations spanning hundred-billion- to trillion-parameter models. We propose CIPHER-MoE, a plug-and-play approach that balances workloads by constraining overloaded experts through capacity-aware token selection and rerouting. More importantly, this work reveals the bounded relationship between expert capacity and workload distribution. Experiments are conducted on the mainstream mega-sized MoE models ranging from 284B to 1.6T parameters, including DeepSeek-V3, DeepSeek-V4-Flash, GLM-5, and DeepSeek-V4-Pro. Across different architectures and datasets, CIPHER-MoE achieves measured training speedups of 1.10×1.10×–1.94×1.94×, accelerating convergence while consistently preserving both domain-specific and general capabilities. These results demonstrate that CIPHER-MoE provides a simple, efficient, stable, and scalable foundation for large-scale MoE training. References Bommasani et al. (2021) R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. Cited by: §1. Chen et al. (2026) L. Chen, J. Li, Q. Wang, R. Liao, S. Li, C. Liang, N. Lao, and Q. Liu Phi-balancing for mixture-of-experts training. arXiv preprint arXiv:2605.15403. Cited by: §2. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: §A.3. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §A.3. Dai et al. (2024) D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al. Deepseekmoe: towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), p. 1280–1297. Cited by: §2. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-v4: towards highly efficient million-token context intelligence. Cited by: §4. Dikkala et al. (2023) N. Dikkala, N. Ghosh, R. Meka, R. Panigrahy, N. Vyas, and X. Wang On the benefits of learning to route in mixture-of-experts models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 9376–9396. Cited by: §1. Dua et al. (2019) D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner DROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 2368–2378. Cited by: §A.3. Fedus et al. (2022) W. Fedus, B. Zoph, and N. Shazeer Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. Cited by: §1, §1, §2, §3.1. Gale et al. (2023) T. Gale, D. Narayanan, C. Young, and M. Zaharia Megablocks: efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems 5, p. 288–304. Cited by: §2. GLM-5-Team et al. (2026) GLM-5-Team, :, A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, C. Zhu, C. Yin, C. Wang, G. Pan, H. Zeng, H. Zhang, H. Wang, H. Chen, J. Zhang, J. Jiao, J. Guo, J. Wang, J. Du, J. Wu, K. Wang, L. Li, L. Fan, L. Zhong, M. Liu, M. Zhao, P. Du, Q. Dong, R. Lu, Shuang-Li, S. Cao, S. Liu, T. Jiang, X. Chen, X. Zhang, X. Huang, X. Dong, Y. Xu, Y. Wei, Y. An, Y. Niu, Y. Zhu, Y. Wen, Y. Cen, Y. Bai, Z. Qiao, Z. Wang, Z. Wang, Z. Zhu, Z. Liu, Z. Li, B. Wang, B. Wen, C. Huang, C. Cai, C. Yu, C. Li, C. Hu, C. Zhang, D. Zhang, D. Lin, D. Yang, D. Wang, D. Ai, E. Zhu, F. Yi, F. Chen, G. Wen, H. Sun, H. Zhao, H. Hu, H. Zhang, H. Liu, H. Zhang, H. Peng, H. Tai, H. Zhang, H. Liu, H. Wang, H. Yan, H. Ge, H. Liu, H. Chu, J. Zhao, J. Wang, J. Zhao, J. Ren, J. Wang, J. Zhang, J. Gui, J. Zhao, J. Li, J. An, J. Li, J. Yuan, J. Du, J. Liu, J. Zhi, J. Duan, K. Zhou, K. Wei, K. Wang, K. Luo, L. Zhang, L. Sha, L. Xu, L. Wu, L. Ding, L. Chen, M. Li, N. Lin, P. Ta, Q. Zou, R. Song, R. Yang, S. Tu, S. Yang, S. Wu, S. Zhang, S. Li, S. Li, S. Fan, W. Qin, W. Tian, W. Zhang, W. Yu, W. Liang, X. Kuang, X. Cheng, X. Li, X. Yan, X. Hu, X. Ling, X. Fan, X. Xia, X. Zhang, X. Zhang, X. Pan, X. Zou, X. Zhang, Y. Liu, Y. Wu, Y. Li, Y. Wang, Y. Zhu, Y. Tan, Y. Zhou, Y. Pan, Y. Zhang, Y. Su, Y. Geng, Y. Yan, Y. Tan, Y. Bi, Y. Shen, Y. Yang, Y. Li, Y. Liu, Y. Wang, Y. Li, Y. Wu, Y. Zhang, Y. Duan, Y. Zhang, Z. Liu, Z. Jiang, Z. Yan, Z. Zhang, Z. Wei, Z. Chen, Z. Feng, Z. Yao, Z. Chai, Z. Wang, Z. Zhang, B. Xu, M. Huang, H. Wang, J. Li, Y. Dong, and J. Tang GLM-5: from vibe coding to agentic engineering. External Links: 2602.15763, Link Cited by: §4. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1. Guo et al. (2026) H. Guo, H. Lu, G. Nan, B. Chu, J. Zhuang, Y. Yang, W. Che, X. Cao, S. Leng, Q. Cui, et al. Advancing expert specialization for better moe. Advances in Neural Information Processing Systems 38, p. 48767–48809. Cited by: §1. He et al. (2022) J. He, J. Zhai, T. Antunes, H. Wang, F. Luo, S. Shi, and Q. Li Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, p. 120–134. Cited by: §2. Hwang et al. (2023) C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, et al. Tutel: adaptive mixture-of-experts at scale. Proceedings of Machine Learning and Systems 5, p. 269–287. Cited by: §2. Jin et al. (2026a) C. Jin, Z. Jiang, Z. Bai, Z. Zhong, J. Liu, X. Li, N. Zheng, X. Wang, C. Xie, Q. Huang, et al. Megascale-moe: large-scale communication-efficient training of mixture-of-experts models in production. In Proceedings of the 21st European Conference on Computer Systems, p. 366–382. Cited by: §2. Jin et al. (2026b) C. Jin, X. Wei, Y. Zhong, C. Yang, B. Wu, R. Zhu, Z. Zhang, Y. Liu, and X. Jin ReLibra: routing-replay-guided load balancing for moe training in reinforcement learning. arXiv preprint arXiv:2605.08639. Cited by: §1. Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: §1. Kim et al. (2024) Y. Kim, H. Lim, and D. Han Scaling beyond the gpu memory limit for large mixture-of-experts model training. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: §2. Lepikhin et al. (2020) D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen Gshard: scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668. Cited by: §2. Lewis et al. (2021) M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer Base layers: simplifying training of large, sparse models. In International Conference on Machine Learning, p. 6265–6274. Cited by: §1. Li et al. (2024) J. Li, Z. Sun, X. He, L. Zeng, Y. Lin, E. Li, B. Zheng, R. Zhao, and X. Chen LocMoE: a low-overhead moe for large language model training. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI ’24. External Links: ISBN 978-1-956792-04-1, Link, Document Cited by: §2, §3.2.3, §4.1, §4. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations, Vol. 2024, p. 39578–39601. Cited by: §A.3. Liu et al. (2024) A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Cited by: §2, §4. Liu et al. (2026) X. Liu, Y. Wang, F. Fu, X. Xiao, H. Li, J. Li, and B. Cui Laer-moe: load-adaptive expert re-layout for efficient mixture-of-experts training. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, p. 1055–1072. Cited by: §2. Mathematical Association of America (2024) Mathematical Association of America 2024 AMC 12a and AMC 12b competitions. Note: https://maa.org/student-programs/amc/American Mathematics Competitions Cited by: §A.3. Mathematical Association of America (2026a) Mathematical Association of America 2026 american invitational mathematics examination i. Note: Administered February 5, 2026 External Links: Link Cited by: §A.3. Mathematical Association of America (2026b) Mathematical Association of America 2026 american invitational mathematics examination i. Note: Administered February 11, 2026 External Links: Link Cited by: §A.3. Nguyen et al. (2026) X. Nguyen, S. Pandit, A. Xu, C. Xiong, and S. Joty Least-loaded expert parallelism: load balancing an imbalanced mixture-of-experts. arXiv preprint arXiv:2601.17111. Cited by: §1. Nie et al. (2023) X. Nie, X. Miao, Z. Wang, Z. Yang, J. Xue, L. Ma, G. Cao, and B. Cui Flexmoe: scaling large-scale sparse pre-trained model training via dynamic device placement. Proceedings of the ACM on Management of Data 1 (1), p. 1–19. Cited by: §1, §2. Pal et al. (2022) A. Pal, L. K. Umapathi, and M. Sankarasubbu MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, G. Flores, G. H. Chen, T. Pollard, J. C. Ho, and T. Naumann (Eds.), Proceedings of Machine Learning Research, Vol. 174, p. 248–260. External Links: Link Cited by: §4. Qiu et al. (2025) Z. Qiu, Z. Huang, B. Zheng, K. Wen, Z. Wang, R. Men, I. Titov, D. Liu, J. Zhou, and J. Lin Demons in the detail: on implementing load balancing loss for training specialized mixture-of-expert models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5005–5018. Cited by: §2, §3.1. Rajbhandari et al. (2022) S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He Deepspeed-moe: advancing mixture-of-experts inference and training to power next-generation ai scale. In International conference on machine learning, p. 18332–18346. Cited by: §1. Ramamonjison et al. (2023) R. Ramamonjison, T. Yu, R. Li, H. Li, G. Carenini, B. Ghaddar, S. He, M. Mostajabdaveh, A. Banitalebi-Dehkordi, Z. Zhou, et al. Nl4opt competition: formulating optimization problems based on their natural language descriptions. In NeurIPS 2022 competition track, p. 189–203. Cited by: Table 1. Rein et al. (2023) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman Gpqa: a graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022. Cited by: §A.3. Sap et al. (2019) M. Sap, H. Rashkin, D. Chen, R. Le Bras, and Y. Choi Social iqa: commonsense reasoning about social interactions. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), p. 4463–4473. Cited by: §4. Shazeer et al. (2017) N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. Cited by: §1. Shi et al. (2023) S. Shi, X. Pan, X. Chu, and B. Li Pipemoe: accelerating mixture-of-experts through adaptive pipelining. In IEEE INFOCOM 2023-IEEE Conference on Computer Communications, p. 1–10. Cited by: §2. Skiadopoulos et al. (2025) A. Skiadopoulos, M. Zhao, S. Gandhi, T. Norrie, S. Mukherjee, and C. Kozyrakis Accelerating mixture-of-experts training with adaptive expert replication. arXiv e-prints, p. arXiv–2504. Cited by: §2. Suzgun et al. (2023) M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, p. 13003–13051. Cited by: §A.3. Wang et al. (2024a) L. Wang, H. Gao, C. Zhao, X. Sun, and D. Dai Auxiliary-loss-free load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408.15664. Cited by: §2. Wang et al. (2024b) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. Mmlu-pro: a more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems (NeurIPS). Cited by: §A.3. Wang et al. (2025) Z. Wang, Z. Zhu, Z. Li, C. Chen, Y. Han, Y. Lin, Z. Lin, A. Gu, X. Hu, R. Sun, et al. ORGEval: graph-theoretic evaluation of llms in optimization modeling. arXiv preprint arXiv:2510.27610. Cited by: Table 1. Yang et al. (2025) Z. Yang, Y. Wang, Y. Huang, Z. Guo, S. Shi, X. Han, L. Feng, L. Song, X. Liang, and J. Tang Optibench meets resocratic: measure and improve llms for optimization modeling. In International Conference on Learning Representations, Vol. 2025, p. 24726–24759. Cited by: Table 1. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi Hellaswag: can a machine really finish your sentence?. In Proceedings of the 57th annual meeting of the association for computational linguistics, p. 4791–4800. Cited by: §A.3. Zhang et al. (2025) S. Zhang, N. Zheng, H. Lin, Z. Jiang, W. Bao, C. Jiang, Q. Hou, W. Cui, S. Zheng, L. Chang, et al. Comet: fine-grained computation-communication overlapping for mixture-of-experts. Proceedings of Machine Learning and Systems 7. Cited by: §1. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. Cited by: §A.3. Zhou et al. (2022) Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Q. V. Le, J. Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems 35, p. 7103–7114. Cited by: §2. Appendix A Appendix A.1 Gradient Dynamics and Training Stability of MoE Models Figure 6: Gradient stability of CIPHER-MoE over training progress for DeepSeek-V4-Flash, DeepSeek-V3, and GLM-5 on MedMCQA, SIQA, and OR, shown on logarithmic scales. In Figure 6, across all evaluated model-dataset configurations, CIPHER-MoE maintains finite, non-degenerate gradient norms that generally decay from their initial transients into stable ranges, with no evidence of gradient explosion or vanishing. CIPHER-strict often produces smoother and lower gradient norms than the baseline, while CIPHER-reroute remains bounded despite occasional task-dependent fluctuations. Together with the consistently decreasing loss curves, these results indicate that CIPHER-MoE preserves well-behaved optimization dynamics and can provide greater training stability than the baseline. Overall, capacity-aware routing enables favorable convergence without destabilizing gradient-based training. In Figure 7, across early, middle, and late layers, both CIPHER policies suppress the severe routing hotspots observed under the baseline and LocMoE, reducing peak expert traffic from as high as 83.6% to 18.7%; CIPHER-strict minimizes routing volume, whereas CIPHER-reroute achieves the most balanced expert utilization while retaining broader expert coverage. Figure 7: Expert routing behavior. Expert traffic across representative MoE layers (left), activated experts per token (middle), and load entropy during training (right). A.2 Training Efficiency and Stability of Qwen Models Figure 8: Training efficiency and optimization stability on Qwen3.6-35B-A3B. (a) Step time over training progress, shown with a 10-step rolling mean and interquartile range. (b) Distribution of post-warm-up step times. (c) Raw gradient norms on a logarithmic scale. CIPHER maintains stable step times and convergent gradient norms, with CIPHER-strict exhibiting smoother late-stage optimization. Qwen3.6-35B-A3B adopts hidden-state dimensions of 2048 for Qwen3.6-35B-A3B, whose relatively limited widths are consistent with failure to satisfy the favorable dimension-margin regime required by CIPHER-MoE in Table 7, as shown in Figure 8. Consequently, the observed training-efficiency improvement remains below 5%, falling short of the gains achieved on larger MoE models. This contrast suggests that CIPHER-MoE is better suited to large-scale MoE architectures with higher-dimensional representations and may deliver increasingly substantial benefits as future models approach the 10-trillion-parameter scale. A.3 Evaluation results of general benchmarks Table 4: Evaluation results of general benchmarks (%) for OR SFT. Model Method AIME-26 AMC ARC BBH DROP GPQA-D GSM8K HellaSwag IFEval MATH-500 MMLU-Pro DeepSeek-V4-Flash Baseline 33.33 71.60 98.10 84.00 86.00 64.14 95.90 85.40 87.80 91.20 80.02 CIPHER-strict 30.00 70.90 98.20 71.20 84.50 67.17 95.70 80.80 83.70 91.80 80.98 CIPHER-reroute 40.00 70.60 97.80 58.00 72.30 55.56 95.70 81.80 80.40 90.40 75.47 LocMoE (τ=0.005) 33.33 70.90 98.10 71.00 86.30 66.67 96.80 82.00 86.50 92.20 80.59 LocMoE (τ=0.01) 33.33 70.90 98.10 71.30 86.30 65.15 96.30 81.80 85.20 93.20 80.86 LocMoE (τ=0.02) 46.67 71.60 98.00 71.20 86.40 65.15 96.40 81.70 85.40 91.80 81.44 LocMoE (τ=0.03) 36.67 72.40 98.10 71.90 86.90 70.20 96.30 81.30 83.90 94.20 81.70 DeepSeek-V3 Baseline 46.67 71.64 97.94 87.87 87.19 52.53 95.68 75.97 73.74 88.80 75.51 CIPHER-strict 46.67 64.18 97.91 85.99 89.24 57.58 96.13 87.45 84.66 87.60 76.77 CIPHER-reroute 43.33 62.69 97.91 86.44 88.51 52.53 96.29 87.48 93.05 88.00 76.14 LocMoE (τ=0.01) 40.00 66.42 97.07 88.44 87.27 56.57 95.68 80.65 81.65 88.60 75.57 GLM-5 Baseline 63.33 86.57 98.20 93.20 82.19 61.11 96.74 87.55 90.94 95.80 81.67 CIPHER-strict 53.33 81.34 98.59 93.44 81.56 64.65 97.19 88.19 91.31 92.80 80.15 CIPHER-reroute 66.67 82.84 98.37 91.34 82.96 58.59 97.80 87.65 92.05 95.20 80.33 LocMoE (τ=0.01) 60.00 82.84 97.27 93.87 80.53 63.64 96.21 85.22 91.50 95.20 77.85 Table 5: Evaluation results of general benchmarks (%) for MedMCQA SFT. Model Method AIME-26 AMC ARC BBH DROP GPQA-D GSM8K HellaSwag IFEval MATH-500 MMLU-Pro DeepSeek-V4-Flash Baseline 50.00 80.60 97.50 85.00 85.20 62.63 95.70 80.20 70.20 92.60 76.38 CIPHER-strict 56.67 75.56 98.03 85.16 81.62 66.67 96.44 76.35 78.74 94.00 80.31 CIPHER-reroute 50.00 75.56 97.74 86.13 81.40 60.10 96.13 80.33 74.12 92.80 78.77 LocMoE (τ=0.01) 56.67 80.00 97.86 84.53 81.51 66.67 96.29 76.86 77.45 94.20 80.38 DeepSeek-V3 Baseline 46.67 77.61 96.65 87.70 82.65 60.61 96.66 74.38 80.10 92.20 74.28 CIPHER-strict 46.67 74.63 97.97 86.28 86.77 55.56 96.29 87.48 82.07 88.00 76.32 CIPHER-reroute 53.33 76.12 97.86 85.33 85.66 59.09 95.83 80.87 80.41 90.20 74.75 LocMoE (τ=0.01) 43.33 76.86 96.99 88.33 85.89 57.07 96.66 75.08 83.93 92.80 75.01 GLM-5 Baseline 63.33 86.57 98.34 91.49 84.49 61.11 96.82 87.46 90.76 94.80 80.38 CIPHER-strict 60.00 85.07 98.68 92.58 82.01 62.63 97.88 86.25 90.20 95.40 81.38 CIPHER-reroute 60.00 88.06 98.39 92.40 79.38 65.15 96.89 85.74 94.64 95.80 81.82 LocMoE (τ=0.01) 60.00 82.09 98.00 93.12 82.50 65.15 96.97 86.31 91.31 96.20 75.07 Table 6: Evaluation results of general benchmarks (%) for SIQA SFT. Model Method AIME-26 AMC ARC BBH DROP GPQA-D GSM8K HellaSwag IFEval MATH-500 MMLU-Pro DeepSeek-V4-Flash Baseline 53.33 75.40 96.30 87.90 84.70 60.10 96.10 87.10 79.30 90.80 77.68 CIPHER-strict 50.00 77.78 98.03 85.64 81.53 62.63 96.44 85.12 80.59 92.00 78.26 CIPHER-reroute 60.00 73.33 98.17 85.73 79.46 60.10 95.75 88.06 81.48 91.60 74.67 LocMoE (τ=0.01) 53.33 73.33 97.91 85.73 81.91 56.57 96.51 84.87 81.33 91.20 78.27 DeepSeek-V3 Baseline 53.33 72.39 96.62 85.30 84.31 54.55 93.10 78.87 76.14 87.60 71.67 CIPHER-strict 46.67 65.67 97.94 86.35 88.39 57.58 95.98 87.61 92.81 87.40 73.99 CIPHER-reroute 53.33 68.66 97.86 84.23 87.45 56.57 95.00 85.23 89.09 88.60 71.01 LocMoE (τ=0.01) 56.67 73.88 97.10 81.97 87.21 54.55 95.68 80.01 87.29 90.80 72.22 GLM-5 Baseline 53.33 82.84 97.49 91.49 83.66 61.11 95.91 86.48 91.50 93.80 80.56 CIPHER-strict 56.67 84.33 98.03 93.56 83.47 62.63 97.04 89.12 91.68 92.40 81.22 CIPHER-reroute 56.67 85.07 97.89 92.87 84.55 61.11 96.89 88.74 90.57 93.80 82.68 LocMoE (τ=0.01) 50.00 82.09 97.66 92.23 81.75 62.63 95.83 86.88 90.20 95.20 75.86 General capabilities are evaluated across five categories: mathematics (AIME-26 (Mathematical Association of America, 2026a; Mathematical Association of America, 2026b), AMC (Mathematical Association of America, 2024), GSM8K (Cobbe et al., 2021), and MATH-500 (Lightman et al., 2024)), general knowledge (ARC (Clark et al., 2018), GPQA-D (Rein et al., 2023), MMLU-Pro (Wang et al., 2024b), and HellaSwag (Zellers et al., 2019)), logical reasoning (BBH (Suzgun et al., 2023)), reading comprehension (DROP (Dua et al., 2019)), and instruction following (IFEval (Zhou et al., 2023)). Tables 4, 5, and 6 demonstrate that CIPHER-MoE can strengthen broad transfer performance rather than merely preserve it. For DeepSeek-V3 after SIQA SFT, CIPHER-strict improves the macro-average across ARC, BBH, GPQA-D, HellaSwag, and MMLU-Pro by 4.3% relative to the baseline. This pattern suggests that selective routing suppresses low-value signals and concentrates expert updates on more informative token–expert associations. The efficiency benefits of CIPHER-MoE therefore need not come at the expense of general capability and may instead be accompanied by better transferable reasoning. A.4 Capacity analysis of Theorem 1 A.4.1 Comparison task and assumptions Consider expert e and its candidate pool in one micro-batch. Let Ne≥1N_e≥ 1 be the number of offered tokens, and let C∈1,…,NeC∈\1,…,N_e\ be the number retained for expert execution. A token is discriminative if it carries the expert-specific signal specified by the model below. Write DiD_i for its indicator and Ae=∑i=1NeDi≥1A_e=\ _i=1^N_eD_i≥ 1\ for the event that the offered pool contains such a token. The common retention target is given in equation 6. Natural-order admission executes the first C offered tokens, whereas CIPHER admission ranks the available candidates before allocating its C slots. Computing affinities and selecting candidates are not counted as expert-execution capacity. The comparison therefore concerns signal retention per execution slot; its implications for end-to-end training time are evaluated separately. A1: Exposure, geometry, and signal availability. The offered tokens are independent draws from a common exposure distribution, with discriminative probability πe∈(0,1) _e∈(0,1) and Pr(Ae)=1−(1−πe)Ne≥a0>0. (A_e)=1-(1- _e)^N_e≥ a_0>0. (11) For the geometric reference model, let X∼Unif(Sd−1)X (S^d-1), let e u_e be the unit gating direction, and let ℛeR_e denote exposure to expert e. Suppose the discriminative event eD_e and exposure probability satisfy e⊆⟨e,X⟩≥δ,Pr(ℛe)≥1βE,0<δ<1,β≥1.D_e \ u_e,X ≥δ\, (R_e)≥ 1β E, 0<δ<1, β≥ 1. (12) The offered distribution is the exposure-conditioned distribution, rather than an unconditioned uniform spherical distribution. Consequently, πe=Pr(e∣ℛe)≤βEPr(e)≤βEpδ(d),pδ(d)=Pr(|⟨e,X⟩|≥δ). _e= (D_e _e)≤β E (D_e)≤β Ep_δ(d), p_δ(d)= (| u_e,X |≥δ). (13) No independence between discrimination and exposure is required for this inequality. The two-sided spherical probability gives a conservative bound for the one-sided discriminative event. Equal-norm orthogonal gates under the symmetric reference model yield β=1β=1; learned gates, expert biases, and Top-K selection require their own exposure justification. The margin δ is a certified property of the signal model and is distinct from the algorithmic affinity threshold τ. A2: Candidate and threshold recall. Conditional on AeA_e, choose one discriminative token as a witness according to a fixed rule. Let eV_e denote the event that this witness reaches the expert’s ranking pool and survives thresholding. Assume Pr(e∣Ae)≥1−η,0≤η<ε. (V_e A_e)≥ 1-η, 0≤η< . (14) This explicitly accounts for signal loss before capacity truncation. Increasing execution capacity cannot recover a witness that has already been excluded from the ranking pool. A3: Stable conditional ranking. Condition on AeA_e and eV_e. An expert ranks candidates using affinity and a fixed deterministic rule for ties. Expose the competing candidates sequentially, and let YjY_j indicate whether competitor j precedes the witness. For the information ℱj−1F_j-1 revealed by previous competitors, assume deterministic bounds Pr(Yj=1∣ℱj−1,Ae,e)≤qj,0≤qj≤1,∑j=1Ne−1qj≤μ. (Y_j=1 _j-1,A_e,V_e)≤ q_j, 0≤ q_j≤ 1, _j=1^N_e-1q_j≤μ. (15) Here, μ bounds the total conditional number of competitors that can rank ahead of the witness. If fewer than NeN_e candidates reach the ranking pool, the remaining indicators are set to zero. With S=∑jYjS= _jY_j, the witness occupies position S+1S+1. This assumption allows dependence between candidates, provided the bound holds after witness selection and after observing earlier competitors. A marginal ranking-error rate alone does not imply this conditional guarantee. A.4.2 Capacity separation for affinity-based admission We prove the capacity bounds in Theorem 1, using BeB_e and G(d,δ)G(d,δ) as defined in equation 7. For any prescribed reduction factor κ>1κ>1, we also derive the following sufficient condition for CCIPHER∗≤Cnatural∗/κC^*_CIPHER≤ C^*_natural/κ: d≥1+2log(κβEBe/[(1−ε)a0])−log(1−δ2).d≥ 1+ 2 \! (κβ EB_e/[(1- )a_0] )- (1-δ^2). (16) Theorem 1 identifies two different controls on execution capacity. Natural-order admission must allocate enough slots to encounter a scarce discriminative token. Affinity admission instead needs enough slots to accommodate candidates ranked ahead of an available witness. When this conditional ranking budget is small, the latter policy can operate with substantially fewer slots. The quantity BeB_e is a sufficient budget, rather than an exact minimum or an optimal training-time configuration. Spherical concentration. We first establish the finite-dimensional geometric bound used in the theorem. Rotational symmetry and the normalized Gaussian representation of a spherical vector give pδ(d)=1−Iδ2(12,d−12)≤(1−δ2)(d−1)/2=e−G(d,δ),p_δ(d)=1-I_δ^2\! ( 12, d-12 )≤(1-δ^2)^(d-1)/2=e^-G(d,δ), (17) where Iz(a,b)I_z(a,b) is the regularized incomplete beta function. To prove the inequality, let Z1,…,ZdZ_1,…,Z_d be independent standard Gaussian variables, set W=∑j=2dZj2W= _j=2^dZ_j^2, and write X=Z/‖Z‖X=Z/\|Z\|. Then X12=Z12/(Z12+W)X_1^2=Z_1^2/(Z_1^2+W) has the beta distribution appearing above. With b=δ2/(1−δ2)b=δ^2/(1-δ^2), the cap event is Z12≥bWZ_1^2≥ bW. The Gaussian tail satisfies 2[1−Φ(v)]≤e−v2/22[1- (v)]≤ e^-v^2/2 for v≥0v≥ 0, where Φ is the standard Gaussian cumulative distribution function. Indeed, the difference between the right and left sides vanishes at zero and at infinity, and its derivative is e−v2/2(2/π−v)e^-v^2/2( 2/π-v), so it remains nonnegative. Conditioning on W therefore yields pδ(d) p_δ(d) =[21−Φ(bW)]≤e−bW/2 =E\! [2\1- ( bW)\ ] ^-bW/2 (18) =∏j=2de−bZj2/2=(1+b)−(d−1)/2=(1−δ2)(d−1)/2. = _j=2^dEe^-bZ_j^2/2=(1+b)^-(d-1)/2=(1-δ^2)^(d-1)/2. This bound holds at finite d and does not require an asymptotic approximation. In particular, G(d,δ)≥(d−1)δ2/2G(d,δ)≥(d-1)δ^2/2. Necessary capacity for natural-order admission. Success within the first C slots implies AeA_e. Independence in A1 thus gives Pnatural(C∣Ae) P_natural(C A_e) =1−(1−πe)C1−(1−πe)Ne, = 1-(1- _e)^C1-(1- _e)^N_e, (19) Cnatural∗ C^*_natural =⌈log(1−(1−ε)[1−(1−πe)Ne])log(1−πe)⌉. = \! (1-(1- )[1-(1- _e)^N_e] ) (1- _e) . The exact expression follows by solving Pnatural(C∣Ae)≥1−εP_natural(C A_e)≥ 1- ; division by log(1−πe)<0 (1- _e)<0 reverses the inequality. For a simpler necessary bound, 1−(1−πe)C≤Cπe1-(1- _e)^C≤ C _e and equation 11 imply Cnatural∗≥(1−ε)a0πe≥(1−ε)a0βEeG(d,δ),C^*_natural≥ (1- )a_0 _e≥ (1- )a_0β Ee^G(d,δ), (20) where the second inequality uses equations 13 and 17. The union-bound argument requires only the marginal scarcity bound and positive availability; independence is needed for the exact expression in equation 19. Conditional concentration of competing candidates. Write ℚQ for the probability law conditioned on AeA_e and eV_e. For θ>0θ>0, A3 gives ℚ[eθYj∣ℱj−1] _Q[e^θ Y_j _j-1] =1+(eθ−1)ℚ[Yj∣ℱj−1] =1+(e^θ-1)E_Q[Y_j _j-1] (21) ≤1+qj(eθ−1)≤expqj(eθ−1). ≤ 1+q_j(e^θ-1)≤ \q_j(e^θ-1)\. Iterating conditional expectations and using ∑jqj≤μ _jq_j≤μ yields ℚeθS≤expμ(eθ−1)E_Qe^θ S≤ \μ(e^θ-1)\. Exponential Markov and minimization over θ>0θ>0 then give, for μ>0μ>0 and x>0x>0, Prℚ(S≥μ+x) _Q(S≥μ+x) ≤exp−μh(x/μ)≤exp−x22(μ+x/3), ≤ \-μ h(x/μ)\≤ \! \- x^22(μ+x/3) \, (22) h(v) h(v) =(1+v)log(1+v)−v. =(1+v) (1+v)-v. The first exponent follows by setting θ=log(1+x/μ)θ= (1+x/μ). For the second, the series inequality eθ−1−θ≤θ2/[2(1−θ/3)]e^θ-1-θ≤θ^2/[2(1-θ/3)], valid for 0<θ<30<θ<3, follows from k!≥2⋅3k−2k!≥ 2· 3^k-2 for k≥2k≥ 2. Substituting θ=x/(μ+x/3)θ=x/(μ+x/3) into the Markov exponent gives the stated bound. If μ=0μ=0, every qjq_j is zero, so S=0S=0 almost surely under ℚQ and one slot suffices for an admitted witness. Sufficient CIPHER capacity and proof of separation. Choose x=ℓ3+2μℓ+ℓ29,x2=2ℓ(μ+x/3).x= 3+ 2μ + ^29, x^2=2 (μ+x/3). (23) Equation 22 bounds the ranking failure by e−ℓ=ε−ηe^- = -η. Since the witness has rank S+1S+1, capacity ⌈1+μ+x⌉ 1+μ+x is sufficient. If this exceeds NeN_e, admitting all available candidates eliminates capacity-induced failure, giving BeB_e in equation 7. The same expression remains sufficient when μ=0μ=0 or Ne=1N_e=1, although it can be conservative. Accounting for candidate-recall failure gives Pr(failure∣Ae)≤η+(1−η)(ε−η)≤ε. (failure A_e)≤η+(1-η)( -η)≤ . (24) Thus CCIPHER∗≤BeC^*_CIPHER≤ B_e. Together with equation 20, this proves equation 8. Requiring the natural-order lower bound to be at least κBeκ B_e is equivalent to G(d,δ)≥logκβEBe(1−ε)a0,G(d,δ)≥ κβ EB_e(1- )a_0, (25) which gives equation 16 after substituting the definition of G. This completes the proof of Theorem 1. Finite-pool feasibility. Availability imposes the additional requirement Ne≥log(1−a0)log(1−πe).N_e≥ (1-a_0) (1- _e). (26) Ranking cannot create a discriminative token absent from the offered pool. If the claimed natural-order lower bound exceeds NeN_e, the assumed scarcity and availability cannot both hold for that pool. The theorem does not prescribe execution of more tokens than are available. The degenerate case πe=0 _e=0 violates positive availability, whereas πe=1 _e=1 requires only one natural-order slot and admits no nontrivial separation of this kind. The assumptions are jointly feasible. For example, consider a single expert, identity exposure, independent spherical tokens, and e=⟨e,X⟩≥δD_e=\ u_e,X ≥δ\. Choose a sufficiently large pool to maintain a0a_0, and choose its highest-projection discriminative token as the witness. Ranking by exact projection places no competitor ahead of this witness, up to probability-zero ties. With complete candidate recall, η=μ=0η=μ=0 and one affinity-selected slot suffices, whereas natural-order admission still follows equation 19. This construction establishes compatibility of the assumptions without asserting that a trained checkpoint follows the reference distribution. Retention guarantee for reroute. For the same input micro-batch, let eST_e^S be the tokens retained by expert e under strict selection and eRT_e^R its final token set after rerouting. If rerouting preserves all strictly accepted assignments and adds only tokens that satisfy the destination expert’s similarity threshold within its remaining capacity, then eS⊆eR,|eR|≤Ce,T_e^S _e^R, |T_e^R|≤ C_e, (27) Every discriminative witness retained by strict is therefore also retained by reroute. The strict-retention event is a subset of the reroute-retention event, so reroute inherits the same lower bound on the conditional retention probability under the assumptions of Theorem 1. This guarantee requires preserving accepted assignments and respecting each expert’s capacity; it does not apply if rerouting evicts a strictly accepted token. Additional assignments can change the expert mixture and the resulting training updates, so the retention guarantee does not imply unchanged model outputs or improved downstream performance. A.4.3 Sharper capacity bounds and stability conditions Sharper conditional ranking bounds. When A3 admits a common bound qj≤q_j≤ q, retaining the exact Bernoulli moment-generating function gives ℚeθS _Qe^θ S ≤(1−q+qeθ)Ne−1, ≤(1-q+qe^θ)^N_e-1, (28) Prℚ(S≥k) _Q(S≥ k) ≤exp[−(Ne−1)kl(kNe−1∥q)], ≤ \! [-(N_e-1)kl ( kN_e-1 \|q ) ], for Ne≥2N_e≥ 2, 0<q<10<q<1, and (Ne−1)q<k≤Ne−1(N_e-1)q<k≤ N_e-1, where kl(u∥q)=ulog(u/q)+(1−u)log[(1−u)/(1−q)]kl(u\|q)=u (u/q)+(1-u) [(1-u)/(1-q)]. Optimizing the Markov exponent yields eθ=u(1−q)/[q(1−u)]e^θ=u(1-q)/[q(1-u)] for u=k/(Ne−1)<1u=k/(N_e-1)<1; the endpoint u=1u=1 follows by taking θ→∞θ→∞. Numerically inverting this bound can reduce the sufficient integer budget. If the blockers are independent Bernoulli variables under ℚQ, the exact witness-retention budget is the corresponding binomial quantile, Crank∗(α)=1+mink∈0,…,Ne−1:Pr(Bin(Ne−1,q)≤k)≥1−α.C^*_rank(α)=1+ \k∈\0,…,N_e-1\: (Bin(N_e-1,q)≤ k)≥ 1-α\. (29) Here α is the conditional ranking-failure allowance, and α=ε−ηα= -η accounts for candidate recall as above. A simpler sufficient budget associated with the comparison is C~=minNe,⌈max2Neq,163log1α⌉. C= \N_e, \2N_eq, 163 1α \ \. (30) Indeed, with m0=(Ne−1)q≤C/2m_0=(N_e-1)q≤ C/2, the Bernstein exponent satisfies 3(C−m0)2/[2(C+2m0)]≥3C/163(C-m_0)^2/[2(C+2m_0)]≥ 3C/16. Admitting the entire pool handles budgets exceeding NeN_e; one slot suffices when q=0q=0. These bounds concern the chosen witness, whereas retaining any one of several discriminative tokens may require fewer slots. Data drift and affinity calibration. To connect A3 with changing training data, suppose a relevant signal has ideal normalized affinity at least δ0 _0, and data drift reduces this margin by at most Δ . Let cj,e⋆c _j,e and cj,ec_j,e be ideal and learned affinities, respectively. Assume both scores in a witness–competitor comparison have error at most ζ, except with conditional probability ϱ . Also assume the conditional competitor tail satisfies Pr(cj,e⋆≥z∣ℱj−1,Ae,e)≤χPr(⟨e,X⟩≥z)+ν,χ≥1,ν≥0. (c _j,e≥ z _j-1,A_e,V_e)≤χ ( u_e,X ≥ z)+ν, χ≥ 1, ν≥ 0. (31) The parameters χ and ν account for tail inflation and contamination after conditioning on exposure, witness admission, and previous competitors. Unconditional isotropy or calibration accuracy does not establish this bound. On a calibrated comparison, a competitor preceding the witness must have ideal affinity at least cj,e⋆≥cj,e−ζ≥c∗,e−ζ≥c∗,e⋆−2ζ≥δ0−Δ−2ζ≡δeff>0.c _j,e≥ c_j,e-ζ≥ c_*,e-ζ≥ c _*,e-2ζ≥ _0- -2ζ≡ _eff>0. (32) For δeff<1 _eff<1, a union bound and the one-sided spherical tail yield qmax≤min1,ϱ+ν+χ2e−G(d,δeff),μ=(Ne−1)qmax.q_ ≤ \1, +ν+ χ2e^-G(d, _eff) \, μ=(N_e-1)q_ . (33) The factor 1/21/2 arises from the one-sided competitor event; equation 13 used a conservative two-sided bound. A bounded total ranking budget is obtained along configurations satisfying Ne(ϱ+ν)=O(1)N_e( +ν)=O(1) and Neχe−G(d,δeff)=O(1)N_eχ e^-G(d, _eff)=O(1). With a confidence gap ε−η -η bounded away from zero, this gives Be=O(1)B_e=O(1). Thus, a larger candidate pool requires correspondingly stronger control of ranking errors. For a certified budget BeB_e and desired factor κ, write Lκ=log[κβEBe/((1−ε)a0)]L_κ= [κβ EB_e/((1- )a_0)]. Applying equation 25 with the conservative margin δeff _eff gives the sufficient drift tolerance Δ+2ζ≤δ0−1−exp−2Lκ/(d−1). +2ζ≤ _0- 1- \-2L_κ/(d-1)\. (34) The right side must be nonnegative, and this condition must hold jointly with candidate recall, availability, and the bound on μ. If the data domain or router changes enough to invalidate these conditions, the admission budget or threshold must be reconsidered. Dimension and finite-horizon scaling. Theorem 1 gives CCIPHER∗Cnatural∗≤βEBe(1−ε)a0e−G(d,δ). C^*_CIPHERC^*_natural≤ β EB_e(1- )a_0e^-G(d,δ). (35) A growing separation follows if G(d,δ)−log(βEBe/a0)→∞G(d,δ)- (β EB_e/a_0)→∞ with fixed confidence. For bounded μ and fixed exposure and availability parameters, dδ(d)2→∞dδ(d)^2→∞ is sufficient. In contrast, when δ=c/dδ=c/ d for fixed c>0c>0, the normalized Gaussian representation gives dX1⇒(0,1) dX_1 (0,1) and hence pc/d(d)→2[1−Φ(c)]p_c/ d(d)→ 2[1- (c)]. The geometric exponent remains of constant order, so increasing dimension alone does not yield an increasing separation in this regime. These conclusions require availability to remain valid, potentially through a growing offered pool. Nominal hidden width or total parameter count does not by itself certify the effective spherical dimension of an anisotropic learned representation. For H expert–micro-batch pairs, set ε=εtot/H = _tot/H, require η<εη< , and recompute BeB_e. If the assumptions hold for each pair, a union bound gives Pr(⋃h=1H(Fh∩Ah))≤∑h=1HPr(Fh∣Ah)Pr(Ah)≤εtot, \! ( _h=1^H(F_h∩ A_h) )≤ _h=1^H (F_h A_h) (A_h)≤ _tot, (36) where FhF_h denotes retention failure and AhA_h signal availability. Independence between micro-batches is unnecessary. The guarantee covers failures to retain an offered signal; it neither guarantees signal availability nor establishes an unlimited-horizon training guarantee. Illustrative capacity regimes. For E=256E=256, β=1β=1, ε=0.1 =0.1, η=0.01η=0.01, a0=0.9a_0=0.9, and μ=1μ=1, equation 7 gives Be=6B_e=6 when Ne≥6N_e≥ 6. Substituting these values into equation 16 gives Table 7. These are conditional illustrations rather than fitted properties of the evaluated models. In particular, Ne≥6N_e≥ 6 establishes only the displayed budget; the offered pool must also satisfy equation 26, and the margin cannot be replaced by τ without calibration. Table 7: Sufficient representation dimensions for the illustrative capacity-separation regimes under A1–A3. Certified margin δ Twofold reduction (κ=2κ=2) Tenfold reduction (κ=10κ=10) 0.050.05 d≥6,586d≥ 6,586 d≥7,872d≥ 7,872 0.060.06 d≥4,571d≥ 4,571 d≥5,464d≥ 5,464 0.070.07 d≥3,357d≥ 3,357 d≥4,012d≥ 4,012 A.4.4 Signal retention and output fidelity Signal retention does not imply that clipping leaves the MoE output unchanged. To quantify a separate local fidelity property, fix a token i with original routed mass mi>0m_i>0, normalize its native gates as ωe=gi,e/mi _e=g_i,e/m_i, and write e=mife(i,θe) v_e=m_if_e( x_i; _e). Then ∑eωe=1 _e _e=1 and the original routed output is ∑eωee _e _e v_e. Assume ‖e‖≤V\| v_e\|≤ V, absorbing the original routed mass and any model-specific output scale into this bound. For a retained expert set U with mass r=∑e∈ωe>0r= _e _e>0, the strict output with renormalized weights satisfies ‖∑eωee−∑e∈ωere‖ \| _e _e v_e- _e _er v_e \| ≤V∑e∉ωe+V(1r−1)∑e∈ωe ≤ V _e _e+V ( 1r-1 ) _e _e (37) =2V(1−r). =2V(1-r). If r=0r=0 and the routed branch returns zero, its deviation is at most V. For general modified weights ωe′ω _e, including reroute, define both weight vectors over the union of their supports. The triangle inequality gives ‖∑eωee−∑eωe′e‖≤V‖ω−ω′‖1. \| _e _e v_e- _eω _e v_e \|≤ V\|ω-ω \|_1. (38) Unchanged shared-expert and residual contributions cancel in this comparison. If the downstream loss is LlossL_loss-Lipschitz in the block output, its one-step change is bounded by 2LlossV(1−r)2L_lossV(1-r) for strict and by LlossV‖ω−ω′‖1L_lossV\|ω-ω \|_1 for a general weight change. These statements hold at fixed expert outputs and fixed downstream computation. They do not bound the accumulation of parameter changes during optimization. Consequently, the capacity analysis motivates affinity-aware workload reduction, while training stability, specialization, and final model capability remain separate empirical evaluations. A.5 Training Configurations of CIPHER-MoE Table 8: Training configurations for MoE models. Model Nodes Parallel configuration Seq. length GBS MoE configuration DeepSeek-V4-Pro 32 TP2/P8/EP64/CP1/ETP1/VPP4 + SP 4K 256/8192 384 experts; learned Top-6 DeepSeek-V4-Flash 8 TP2/P4/EP16/CP1/ETP1/DP16 + SP 8K 128 256 experts; learned Top-6; sqrt-softplus gate DeepSeek-V3 16 TP4/P16/EP4/CP1/DP4 + SP 8K 128 256 experts; Top-8 routing; sigmoid gate GLM-5 16 TP2/P8/EP32/CP1/ETP1/VPP2/DP16/EDP1 8K 128 256 experts; Top-8 routing; sigmoid gate Table 8 summarizes the training and routing configurations used in our experiments. The affinity threshold τ specifies the minimum normalized token–expert affinity required to retain a candidate assignment; assignments with affinity below this threshold are filtered out. On DeepSeek-V4-Flash, we evaluated LocMoE with τ∈0.005,0.01,0.02,0.03τ∈\0.005,0.01,0.02,0.03\ and found that τ=0.01τ=0.01 provided the most favorable overall performance. We therefore adopt τ=0.01τ=0.01 for LocMoE in all subsequent experiments to ensure a consistent configuration across models and datasets. Both variants of CIPHER-MoE use a capacity factor of 88; CIPHER-reroute additionally uses a fill-target factor of 44. The capacity factor denotes the capacity limit as a multiple of the uniform expert load. For locality loss, λloc _loc is set to 0.010.01 for both LocMoE and CIPHER-MoE. These settings remain robust across different models and datasets.