Paper deep dive

Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing

Hanchi Sun, Yixin Liu, Yonghui Wu, Lichao Sun

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 80

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/22/2026, 6:24:42 AM

Summary

The paper introduces Expert Threshold (ET) routing for Mixture-of-Experts (MoE) models, a fully causal mechanism that routes tokens based on an exponential moving average (EMA) threshold of expert scores. This approach enables dynamic computation allocation and load balancing without auxiliary losses or the non-causal batch-dependency of Expert Choice (EC) routing, outperforming Token Choice (TC) routing in cross-entropy loss and CORE benchmark scores.

Entities (5)

Expert Choice Routing · method · 100%Expert Threshold Routing · method · 100%Mixture-of-Experts · architecture · 100%Token-choice Mixture-of-Experts · method · 100%FineWeb-Edu · dataset · 95%

Relation Signals (3)

Expert Threshold Routing → outperforms → Token-choice Mixture-of-Experts

confidence 95% · ET achieves 0.067 lower cross-entropy loss than TC-MoE

Expert Choice Routing → violates → Causality

confidence 95% · EC routing fundamentally violates causality, making it unsuitable for autoregressive language models.

Expert Threshold Routing → enables → Dynamic Computation Allocation

confidence 90% · enabling dynamic computation allocation while achieving load balance without auxiliary losses

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Token-choice Mixture-of-Experts (TC-MoE) routes each token to a fixed number of experts, limiting dynamic computation allocation and requiring auxiliary losses to maintain load balance. We propose Expert Threshold (ET) routing, where each expert maintains an exponential moving average (EMA) threshold estimated from the global token distribution. At both training and inference, each token is independently routed to an expert if its score exceeds the expert's threshold, enabling dynamic computation allocation while achieving load balance without auxiliary losses. This fully causal mechanism eliminates dependence on other tokens in the batch, making it well-suited for autoregressive language modeling. In pretraining experiments scaling to 2.4B parameters on FineWeb-Edu, ET achieves 0.067 lower cross-entropy loss than TC-MoE, equivalent to reaching the same performance with 1.6$\times$ fewer tokens.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

79,857 characters extracted from source content.

Expand or collapse full text

Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing Ryan Sun 1 Yixin Liu 1 Yonghui Wu 2 Lichao Sun 1 Abstract Token-choice Mixture-of-Experts (TC-MoE) routes each token to a fixed number of experts, limiting dynamic computation allocation and re- quiring auxiliary losses to maintain load balance. We propose Expert Threshold (ET) routing, where each expert maintains an exponential moving av- erage (EMA) threshold estimated from the global token distribution. At both training and infer- ence, each token is independently routed to an expert if its score exceeds the expert’s threshold, enabling dynamic computation allocation while achieving load balance without auxiliary losses. This fully causal mechanism eliminates depen- dence on other tokens in the batch, making it well-suited for autoregressive language modeling. In pretraining experiments scaling to 2.4B param- eters on FineWeb-Edu, ET achieves 0.067 lower cross-entropy loss than TC-MoE, equivalent to reaching the same performance with 1.6× fewer tokens. 1. Introduction Mixture of Experts (MoE) architectures (Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al., 2022) have emerged as a leading approach to scale language mod- els efficiently, powering frontier models like DeepSeek- V3 (DeepSeek-AI, 2024). By sparsely activating only a sub- set of expert networks per token, MoE decouples model ca- pacity from computational cost, enabling massive parameter counts with tractable FLOPs. However, sparse routing intro- duces a fundamental tension: without intervention, routers tend to collapse onto a small subset of experts (Shazeer et al., 2017). This harms model quality, as underutilized experts become redundant parameters that waste capacity. It also creates hardware bottlenecks under Expert Paral- Code available at GitHub repository. 1 Computer Science and Engineering, Lehigh University, Bethlehem, PA, USA 2 MD-HOBI- BIOMED INFORMATICS, University of Florida, Gainesville, FL, USA. Correspondence to: Lichao Sun <lis221@lehigh.edu>. Preprint. March 13, 2026. 5k7k10k12k15k17k20k Step 2.6 2.7 2.8 2.9 3.0 Eval loss 0.067 1.6x Dense TC ET Figure 1. evaluation loss for Dense, TC, and ET. Compared to TC, ET achieves a 0.067 final loss gap (TC vs ET), or equivalently reaching same performance level with 1.6x few tokens. lelism (Lepikhin et al., 2021), where skewed loads leave some devices idle and others overloaded. Thus, we need a routing mechanism that roughly maintains load balancing. Prior work falls into two categories. The prevalent token choice (TC) routing (Fedus et al., 2022) fixes the number of experts each token selects. This sparsity constraint not only fails to address load imbalance, but further complicates the routing as it conflicts with load balancing, turning the routing into a combinatorial optimization problem. People resort to heuristics to approximate load balancing, such as auxiliary losses (Lepikhin et al., 2021; Fedus et al., 2022) or PID controllers (Team, 2025a; Wang et al., 2024). In contrast, expert choice (EC) routing (Zhou et al., 2022) relaxes the fixed computation budget per token and only en- forces load balancing within a batch by selecting the top-k tokens for each expert, achieving perfect load balancing by construction while enabling dynamic computation alloca- tion. However, EC routing fundamentally violates causality, making it unsuitable for autoregressive language models. Selecting top-krequires comparing against the entire batch that includes future positions. At training time this mech- anism leaks information (Wang et al., 2024); at inference time future tokens simply do not exist. In this paper, we relax both per-token sparsity and per-batch load balancing, requiring only that load reaches a targeted activation rate in expectation. The resulting mechanism, Expert Threshold (ET) routing, routes each token by com- 1 arXiv:2603.11535v1 [cs.AI] 12 Mar 2026 Expert Threshold Routing Token Choice (TC) Load Imbalance token input Top G r t,i ∀i M 1 M 2 M 3 M 4 Expert Choice (EC) Non Causal Seq top-k M i 1 sequence Top k r t,i ∀t∈seq Batch top-k M i s 1 s 2 s 3 1 batch Top k r t,i ∀t∈batch Expert Threshold (ET) Fully Causal EMA Threshold c i Score r t,i Token index t z t,i = 1r t,i > c i routing pool size token sequence batchpopulation Figure 2. Illustration of TC, EC, and ET routing mechanisms and their routing pools. Left: TC routes each token independently to its top-Gexperts, causing load imbalance. Middle: EC has each expert select its top-ktokens from the batch, requiring access to all tokens including future ones (non-causal). Right: ET routes each token independently by comparing its score against the population’s top-(1/E) quantile estimated by an EMA-tracked threshold c i , enabling fully causal routing over the population. paring its score to a quantile threshold tracked from each expert’s global score distribution. Because the same thresh- old is used at training and inference, ET routing is fully causal with no train-inference mismatch. Pretraining a 2.4B (0.56B active) language model on FineWeb-Edu, ET outperforms TC by 0.067 in cross-entropy loss while achieving near-perfect load balancing. We fur- ther show that EC’s performance improves with batch size, and that models trained with large-batch EC can perform causal inference using our threshold-based routing without retraining. 2. Preliminaries: Routing as Constrained Optimization An MoE layer replaces a dense feed-forward block with a router andGEexperts. Consider a batch ofNtokens with representations x t ∈ R d . The router computes scores r t,i = (W r x t ) i ,(1) collected into a matrixr ∈ R N×GE . Based onr, a routing rule produces a binary assignmentz ∈0, 1 N×GE where z t,i = 1indicates expertiis activated for tokentand0 otherwise. Each selected experticomputes an outputy i,t ∈ R d , weighted by a gate valuep t,i = σ(r t,i ). The MoE output for token t is y t = GE X i=1 z t,i p t,i y i,t .(2) The routing rule that determinesztherefore controls both compute allocation and expert load balance. We formalize MoE routing as findingzthat maximizes the total rout- ing score subject to computational constraints, since higher scores indicate stronger token-expert affinity and, through the gate p t,i , larger expert contributions to the output. Token Choice RoutingThe standard Token Choice rout- ing goal is: max z N X t=1 GE X i=1 z t,i r t,i s.t. GE X i=1 z t,i = G, ∀t (Sparsity) N X t=1 z t,i = k, ∀i (Load Balancing) z t,i ∈0, 1 (3) Here the sparsity constraint ensures each token selects ex- actlyGexperts, and the Load Balancing constraint ensures each expert processes exactlyk = N/Etokens. Solv- ing(3)exactly requires combinatorial algorithms such as theO(N 3 )Hungarian Matching algorithm. Most Token Choice (TC) methods therefore strictly enforce the sparsity constraint by settingz t,i = 1 ⇐⇒ i ∈ Top G (r t,· ), while relying on auxiliary losses (Lepikhin et al., 2021; Fedus et al., 2022) or loss-free load balancing strategies (Wang et al., 2024) to approximate the load balancing constraint. Expert Choice Routing While the load balancing con- straint is essential to avoid routing collapse, the sparsity constraint has no practical benefit. Thus, Expert Choice (EC) (Zhou et al., 2022) removes the sparsity constraint entirely and enforces only load balancing within batches. 2 Expert Threshold Routing The primal problem becomes: max z N X t=1 GE X i=1 z t,i r t,i s.t. N X t=1 z t,i = k, ∀i z t,i ∈0, 1 (4) with trivial closed-form solutionz t,i = 1t∈ Top k (r ·,i ), i.e. picking the top-ktokens in each batch. This design has two key benefits: (1) Perfect load balancing: each ex- pert processes exactlyk = N/Etokens by construction, eliminating the need for auxiliary losses or capacity clip- ping; (2) Dynamic computation: a token may be selected by zero, one, or multiple experts, enabling adaptive compute allocation based on token importance. However, the per sequence load balancing constraint in EC introduces a causality problem for autoregressive genera- tion. The selection indicatorz t,i depends on all tokens’ scoresr 1,i ,...,r N,i —including future tokens unavail- able during inference. Extending EC to batch-level top- k(Ludziejewski et al., 2024) partially alleviates this but does not fully restore causality, as routing still depends on batch composition. 3. Expert Threshold In the preliminaries, we identified the constraints that token choice and expert choice routing impose, yet we question their necessity. To avoid routing collapse, asymptotic load balancing suffices. ET further relaxes the per-sequence or per-batch Load Balancing constraint to a stochastic expecta- tion: max z E data " GE X i=1 z t,i r t,i # s.t. E data [z t,i ] = 1 E , ∀i z t,i ∈0, 1 (5) Essentially, solving this primal problem is equivalent to picking the top1/Efraction of tokens from the full router logit distribution, rather than from a single batch. We may obtain a(1 − 1/E)-quantile estimatec i via exponential moving average (EMA) of thek-th largest router logit of each batch. Then, for both training and inference, we route tokens via binary thresholding, setting z t,i = 1r t,i > c i (6) wherez t,i ∈0, 1is the binary indicator of whether token tis routed to experti. Sincez t,i depends only onr t,i and the global thresholdc i , routing is fully causal while satisfying load balancing in expectation. Algorithm 1 Expert Threshold Routing 1:Input: router logitsr ∈ R N×GE , cutoff-EMAc i , decay rate β, target selection size k = N/E 2: for expert i = 1,...,GE do 3: z t,i ← 1r t,i > c i ∀t 4:if TRAINING then 5: c i ← βc i + (1− β)· kth-largest(r t,i N t=1 ,k) 6:end if 7: end for 8: Return z,c i Connection to EC.Conceptually, ET can be viewed as ex- pert choice routing over an infinitely large batch. In standard EC, each expert selects its top-ktokens within the batch, so the selection threshold depends on all tokens present. As the batch size grows, however, each individual token’s influence on this threshold vanishes, and the routing decision for any token becomes independent of others. ET approximates this limit by maintaining a fixed threshold estimated from the global token distribution. ET and EC handle batch-wise variance differently. EC en- forces perfect load balance per batch by letting the threshold vary, which means routing decisions fluctuate with batch composition. ET instead fixes the threshold for stable rout- ing decisions, accepting small variance in per-batch expert utilization. Despite this difference in training, we show that ET routing can serve as causal inference for EC-trained mod- els without retraining, provided the batch size is sufficiently large. Warmup. At the beginning of training, the router logits’ distribution is not stable yet. The cutoff-EMA requires several thousand steps to converge to a meaningful estimate of the population quantile. During this period, incorrect thresholds cause severe expert starvation—most tokens fail to exceed the threshold, leaving experts underutilized. To address this cold-start problem, we use standard EC routing for the first 4k steps before switching to ET. This allows the cutoff-EMA to accumulate stable statistics under controlled load balance. 4. Experiments 4.1. Experiment Setup We evaluate our methods on Nanochat (Karpathy, 2025), an open-source codebase for training GPT-2-like models. We conduct experiments at two scales: a d12 model (575M parameters, 195M active) with 12 transformer layers, and a d20 model (2.4B parameters, 561M active) with 20 trans- former layers. For MoE layers, we use 16 routed experts with granularityG=1and expansionE=16, plus 1 shared 3 Expert Threshold Routing Table 1. Main results comparing Expert Choice (EC), Token Choice (TC), and Expert Threshold (ET) routing. Batch: token routing pool size. EC uses global selection batch, TC uses per- step batch, ET reports effective EMA pool sizeN/(1− β). TC variants: no load-balancing, auxiliary loss (α=0.001), or loss-free (u=0.005). We report validation cross-entropy (CE) loss (↓) and CORE Eval score (↑). MethodBatchCE loss (↓) CORE (↑) dense—3.00215.743 TC—2.89317.983 TC aux64k2.89215.894 TC loss-free512k2.89818.031 EC2k2.91017.91 EC8k2.84518.83 EC64k2.84118.754 EC512k2.84319.94 ET (β=0.999+warmup) 0.5M→ 500M2.84419.876 expert. Each token activates the shared expert and on aver- age 1 routed expert. We use sigmoid gates (p t,i = σ(r t,i )) instead of softmax gates following LossFree (Wang et al., 2024) and Mixture-of-Depths (Raposo et al., 2024). We add expert capacity factor ofC = 0.5to avoid GPU out-of- memory. The first layer is kept dense following common practice (DeepSeek-AI, 2024; Wang et al., 2024) to allow meaningful routing. We train on 10B and 11.2B tokens for d12 and d20, respectively, from the FineWeb-Edu 100B dataset (Penedo et al., 2024) with a batch size of 0.5M tokens (for d20, we halve the minibatch size and use 2- step gradient accumulation). We report CE loss and CORE benchmark results (Li et al., 2024). Architecture, training, and evaluation details are in Appendices B, C, and D. 4.2. Main Results We compare Expert Threshold (ET) routing against Expert Choice (EC) and Token Choice (TC) routing. All variants share the same architecture and parameter count. For ET, we use EMA decayβ = 0.999and EC warmup for the first 4k steps. For EC, we sweep the global selection batch size from 2k to 512k tokens during training and use ET’s cutoff EMA during inference which makes it fully causal. Unless stated, reported CORE/CE use the causal protocol. For TC, we report variants with no load balancing, auxiliary loss (α=0.001), and loss-free load balancing (u=0.005). Tables 1 and 2 summarize results. ET consistently outper- forms TC in both CE loss (by 0.05 on d12 and 0.067 on d20) and CORE (by 1.89 on d12 and 2.83 on d20). EC with large batch sizes achieves comparable CE loss to ET, con- firming that explicit large-batch selection and EMA-based thresholding reach similar training loss. EC 512k slightly edges out ET on CORE (19.94 vs. 19.88) in d12, though both substantially outperform TC. Table 2. d20 results. MethodBatchCE loss (↓) CORE (↑) dense—2.75120.43 TC aux32k2.68722.31 EC256k2.62124.98 ET (β=0.999+warmup) 256k→ 500M2.62025.14 4k6k8k10k12k14k16k18k Step 1 0 1 Cutoff - EMA (a) Cutoff Raw Deviation from EMA EC (512k) ET 4k6k8k10k12k14k16k18k Step 0.062 0.064 0.066 0.068 0.070 Expert Usage (b) Expert Usage EC (512k) ET Figure 3. Cutoff stability vs expert usage tradeoff. Top Signed cutoff deviation relative to the EMA for EC at 512k batch size. ET stays at zero because routing uses the cutoff EMA directly. Bottom Expert usage for EC at 512k and ET. ET varies around the capacity target while EC remains constant. 4.3. Analysis We analyze key aspects of Expert Threshold routing through cutoff-usage tradeoff, dynamic computation allocation, ex- pert specialization, and supporting EC comparisons on batch size scaling and train-evaluation gap. 4.3.1. CUTOFF VS EXPERT USAGE TRADEOFF EC and ET achieve routing stability through complementary mechanisms. EC enforces a fixed expert usage: each expert selects exactly top-ktokens, guaranteeing usage of1/E per expert. However, the cutoff threshold varies batch-to- batch, with standard deviation scaling asO(1/ √ N ). ET inverts this tradeoff. The cutoff-EMA provides a stable threshold (β = 0.999), while expert usage fluctuates around the capacity target. Figure 3 shows the signed deviation between EC’s per batch cutoff and cutoff-EMA, while ET remains at zero by design. This enables consistent inference without large-batch coordination. In essence, ET trades off hardware consistency for training-inference uniformity. 4 Expert Threshold Routing <|bos|> Question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Answer: Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. #### 72 Token activation intensity GSM8K_0 0141664176 Total fanout (sum across 11 layers) (a) Per-token expert routing on a GSM8K passage. 1 2 3 4 5 6 7 8 9 10 11 ec_bsz2k Layer HumanEvalGSM8K 0.0 0.1 0.2 0.3 0.4 Expert Token Ratio Expert ID 1 2 3 4 5 6 7 8 9 10 11 gec_warmup Layer Expert ID 0.0 0.1 0.2 0.3 0.4 Expert Token Ratio (b) Expert activation heatmap. Top: EC with batch size 2k shows less special- ization. Bottom: ET shows more extreme activation patterns, suggesting more domain-aware routing. Figure 4. Expert specialization analysis. (a) Token-level activation intensity on a GSM8K passage, colored by total fanout (sum of experts activated across layers). The model assigns more computation to structurally important tokens (punctuation, sentence boundaries, numerical results) than to common content words. (b) Expert token ratio heatmaps for HumanEval (code) and GSM8K (math). Top: EC (batch size 2k). Bottom: ET. ET achieves sharper patterns in expert activation, suggesting more domain-aware routing and specialization. 4.3.2. DYNAMIC COMPUTATION ALLOCATION A key advantage of ET and EC is that they do not enforce a fixed amount of computation for every token. We here document its behavior and compare it with EC. For a more drastic comparison, we use the sequence-level EC with batch size 2k. Figure 4(a) gives a qualitative example on a GSM8K passage (Cobbe et al., 2021), where total fanout highlights tokens that receive heavier computation. We further analyze how expert activation relates to posi- tion and token difficulty. Figure 5 shows that both methods allocate more computation to early positions, but EC (2k) exhibits a dramatic spike at the first token (mean fanout ∼10) while ET shows a milder increase (∼2) that decays smoothly. The lower row bins tokens by loss and overlays faint dashed layer traces with a denser global trend. For EC (2k), both the global curve and several layers rise with loss, showing that harder tokens receive more computation. ET remains flatter overall, with layer trajectories crossing and the global curve peaking in the middle before soften- ing at higher loss. Additional layerwise views for the two main runs and extended comparisons for the remaining runs appear in Appendix F.3. 4.3.3. EXPERT SPECIALIZATION We follow Global LBL (Qiu et al., 2025) to evaluate expert specialization across EC with various batch sizes (2k, 8k, 64k, 512k) and ET. For each configuration, we measure the expert token ratio—the fraction of tokens from a given do- main routed to each expert—across HumanEval (Chen et al., 2021) (code) and GSM8K (Cobbe et al., 2021) (math) eval- uation sets. Figure 4(b) compares EC (batch size 2k) with ET. Both exhibit clear specialization: certain experts consis- tently attract domain-specific tokens, visible as concentrated dark cells in the heatmap. ET achieves specialization com- parable to EC without requiring large-batch coordination at inference. The full comparison across all batch sizes (Ap- pendix F, Figure 22) shows that EC specialization sharpens with larger batches—patterns become more concentrated from 2k to 512k—while ET matches the large-batch EC pattern. 4.3.4. BATCH SIZE SCALING We hypothesize that larger batch sizes stabilize EC’s cut- off threshold, yielding better performance and motivating ET’s pursuit of the infinite-batch limit. Figure 6 confirms this trend across four batch sizes (2k, 8k, 64k, 512k to- kens). Training CE loss improves from 2.874 (2k) to 2.844 (8k) to 2.836 (64k), with CORE Eval scores following suit (17.91→18.83→18.75). Top-kselection over larger token pools better approximates the population-level routing deci- sion, explaining this gain. However, performance saturates around 64k tokens, as increasing to 512k provides no further improvement (2.840 CE, 19.94 CORE Eval). Figure 6 visualizes this scaling behavior. Notably, ET achieves comparable performance (2.844 CE, 19.876 CORE Eval) without requiring batch size coordination, making it 5 Expert Threshold Routing (a) EC (2k) fanout vs position(b) ET fanout vs position (c) EC (2k) fanout vs loss by layer(d) ET fanout vs loss by layer Figure 5. Activation dynamics for EC (2k) and ET. Both methods allocate more computation to early positions, with EC (2k) showing a sharper spike. Faint dashed curves show per-layer means and solid red curves show the global mean. When binned by loss, EC (2k) fanout rises monotonically while ET peaks early before declining. 2k8k64k512kET Routing Batch Size 2.83 2.84 2.85 2.86 2.87 2.88 Train CE Loss 17.5 18.0 18.5 19.0 19.5 20.0 20.5 CORE (%) Figure 6. EC performance across routing batch sizes. Training CE loss decreases and CORE Eval score increases with larger batches. practical for autoregressive inference where only single to- kens are available. 4.3.5. TRAIN-EVALUATION GAP A key concern for Expert Choice is the train-inference dis- crepancy when using ET routing at inference. During train- ing, EC selects the top-ktokens for each expert within a batch; at inference, we apply ET’s learned thresholds in- stead, since future tokens are unavailable for batch-level selection. Our results demonstrate that this concern depends critically on the routing batch size. As shown in Table 1, EC with large batch sizes (64k, 512k) achieves validation loss nearly identical to ET (2.841–2.843 vs 2.844), with comparable CORE Eval scores. However, smaller batch sizes reveal significant train-inference mismatch: EC at 2k tokens shows degraded CORE Eval performance (17.91 vs 19.94 at 512k) and evaluation loss (2.910 vs 2.843). This gap arises because top-kselection over a small batch is a noisy esti- mate of the population-level routing decision; at inference (batch size 1), this noise becomes extreme. Figure 7 illustrates this gap. EC (2k) shows a large train- evaluation discrepancy, while EC (512k) maintains close alignment between train loss EMA and eval loss. ET’s cutoff-EMA mechanism addresses this by maintaining a population-level threshold that is independent of batch size, enabling consistent routing at inference without large-batch 6 Expert Threshold Routing 10k12k14k16k18k20k Step 2.8 3.0 3.2 3.4 Loss EC (2k) (train) EC (2k) (eval) EC (512k) (train) EC (512k) (eval) ET (train) ET (eval) Figure 7. Train loss EMA and eval loss for EC at different batch sizes and ET. Solid lines show train loss EMA and dashed lines show eval loss. EC (2k) shows a large train-eval discrepancy, while EC (512k) and ET remain closely aligned. coordination. 4.3.6. ROUTING CONSISTENCY ACROSS CHECKPOINTS To measure how stably each routing rule preserves token expert assignments over training, we compare the routed- expert sets assigned to the same token-layer pairs across checkpoints, excluding the always-active shared expert. We report weighted Jaccard over pooled token-layer-expert edges, weighted_jaccard = |E A ∩ E B | |E A ∪ E B | , whereE A andE B are the pooled active token-layer-expert edges under two checkpoints. A higher weighted Jaccard in- dicates more similar routing behaviors between checkpoints. This gives the clearest separation while preserving the same qualitative ranking as the companion divergence views in Appendix F.2. Figure 8 shows a clear pattern. ET is above EC 2k on every checkpoint pair, indicating that threshold routing pre- serves its token-expert decisions much more consistently than small-pool EC. At the same time, ET remains close to EC 64k across the full matrix, which supports the view that ET tracks the large-pool EC regime without requiring large- batch coordination at inference. TC shows strong short- range consistency, but its longest-range pairs are weaker than ET, so it does not match the same large-pool EC be- havior as cleanly. Appendix F.2 reports the complementary joint JSD heatmap. 5. Related Work 5.1. Mixture of Experts Mixture of Experts (MoE) scales model capacity by routing each token to a small subset of experts while keeping com- pute nearly constant. A learned gate selects top-Gexperts per token (Shazeer et al., 2017), with auxiliary losses to balance load across experts (Lepikhin et al., 2021). The Switch Transformer (Fedus et al., 2022) setsG=1for effi- ciency. Recent LLMs further adopt fine-grained MoE with many small experts and shared experts that remain always active to capture global knowledge (Dai et al., 2024). We incorporate shared experts in our design. 5.2. Load Balancing A critical challenge in MoE systems is load balancing, as routers often favor a small subset of experts without ex- plicit constraints. The standard approach uses an auxiliary lossL aux = α P i f i P i to encourage uniform expert as- signment (Lepikhin et al., 2021; Fedus et al., 2022), where f i = E N P N t=1 z t,i andP i = 1 N P N t=1 p t,i are the normal- ized load and average routing probability for experti. Mini- mizing this loss exerts unbalanced pressure to suppress the router logits based on the load statistics, which makes the router logits biased towards the less loaded experts. How- ever, in distributed training, small local batch sizes cause high variance in load estimation. Global-batch load bal- ancing (Qiu et al., 2025; Team, 2025b) addresses this by computing balance statistics across all devices, yielding more stable gradients and improved expert specialization. This insight motivates our approach to extend the “global” philosophy beyond auxiliary losses. Recent work explores auxiliary-loss-free alternatives. DeepSeekMoE (Dai et al., 2024) introduces expert-specific bias termsb i that dynamically adjust based on load statistics. Expert selection uses biased scoresr t,i + b i , while gating weights use original scoresr t,i , preserving specialization. The bias updates followb i ← b i +u· sign(1−f i ), wheref i is a normalized load statistic for experti(equal to 1 under perfect balance). This eliminates the trade-off between load balancing and task performance inherent in auxiliary loss methods. LongCat-Flash (Team, 2025a) adopts a similar framework but replaces the sign-based update with propor- tional control:∆b i = u · (1 − f i ). While DeepSeek’s approach applies constant-magnitude corrections regardless of imbalance severity, proportional updates scale with the load deviation, enabling smoother convergence. Expert Threshold (ET) combines the above ideas. Instead of a per-batch top-kselection for the original EC, we extend Qwen’s philosophy to compute balance statistics across the entire pretrain population by maintaining a distributional cutoff threshold using EMA. Such number, surprisingly, functions similarly to the bias term for loss-free load balanc- ing. See Table 4 for more details. 5.3. Dynamic Computation Dynamic computation methods adaptively allocate compu- tational resources based on input complexity. Expert Choice 7 Expert Threshold Routing 5k10k15k19k Checkpoint 5k 10k 15k 19k Checkpoint 1.0000.6070.4460.451 0.6071.0000.5400.577 0.4460.5401.0000.618 0.4510.5770.6181.000 EC 2k 5k10k15k19k Checkpoint 5k 10k 15k 19k 1.0000.6830.6640.678 0.6831.0000.7080.752 0.6640.7081.0000.780 0.6780.7520.7801.000 EC 64k 5k10k15k19k Checkpoint 5k 10k 15k 19k 1.0000.6580.6590.662 0.6581.0000.6970.740 0.6590.6971.0000.773 0.6620.7400.7731.000 ET 5k10k15k19k Checkpoint 5k 10k 15k 19k 1.0000.6450.6070.595 0.6451.0000.7450.743 0.6070.7451.0000.797 0.5950.7430.7971.000 TC 0.5 0.6 0.7 0.8 0.9 1.0 Weighted Jaccard Figure 8. Within-family checkpoint-pair routing consistency on a fixed validation stream, measured by weighted Jaccard. ET is consistently more stable than EC 2k and stays close to EC 64k. TC is competitive on nearby checkpoints but degrades more on longer ranges. Table 3. Taxonomy of load balancing methods by scope (Aux loss (Lepikhin et al., 2021); Global LBL (Qiu et al., 2025); Loss- Free (Wang et al., 2024); Seq EC (Zhou et al., 2022); Batch EC (Ludziejewski et al., 2024)). Micro Batch/SeqBatchPopulation Aux lossGlobal LBL– –LossFree Seq ECBatch ECET (ours) Table 4. Conceptual connections between ET and recent work (LossFree (Wang et al., 2024); GShard (Fedus et al., 2022)). ETSimilar ToConnection Cutoff-EMA c i LossFree bias b i Per-expert scalar; no aux loss 1− βLossFree μUpdate rate (EC) (Zhou et al., 2022), detailed in Section 2, achieves this by letting each expert select its top-ktokens, enabling variable computation per token (0 toGEexperts). EC has been applied to upcycling dense checkpoints (Komatsuzaki et al., 2023), attention layer skipping (Raposo et al., 2024), vision (Liu et al., 2024), diffusion (Sun et al., 2024; Shi et al., 2025), and multimodal models (Lin et al., 2024; Ni & team, 2025). Related variants expand the design space (Yan et al., 2025). However, EC’s causality problem limits its use in autoregressive LLMs (Section 5.4). Besides EC, other approaches to dynamic computation rely on other explicit designs. ReMoE (Wang et al., 2025b) replaces discrete TopG routing with fully differentiable ReLU-based routing and adaptive L1 regularization. Other works (Jin et al., 2024; Team, 2025a; Zeng et al., 2024) introduce zero-computation experts (e.g., zero, copy, and constant) that allow tokens to skip expert computation en- tirely, an approach Kilian et al. (2026) extend to multimodal modeling. Top-P routing (Liu et al., 2025b; Jin et al., 2025; Huang et al., 2024; Wang et al., 2025a) selects experts based on cumulative probability mass, adapting expert count to routing confidence, so high-confidence tokens use fewer experts while uncertain ones activate more. XMoE (Yang et al., 2024) is closest to our setting, replacing fixed Top-G routing with a threshold that activates experts until cumula- tive routing mass exceeds a preset value. The key difference is that XMoE uses a fixed probability-mass threshold in token-choice MoE, while ET uses expert-specific EMA cut- offs to causalize expert choice. Auto-tuning methods like DynMoE (Guo et al., 2025) also let each token determine how many experts to activate while reducing sensitivity to MoE hyperparameters. Beyond MoE routing itself, condi- tional computation can also be applied to other Transformer components and long-context settings, e.g., CoLT5 (Ainslie et al., 2023b). Early exit methods (Xin et al., 2020) en- able sample-level dynamics by allowing tokens to exit at intermediate layers. 5.4. Causal Generation of Expert Choice Models EC poses a causality challenge: token selection requires ranking against future tokens, which are unavailable in au- toregressive generation. Prior work addresses this issue in three main ways. Predictor-based methods train an auxiliary predictor or learn per-expert thresholds to approximate ora- cle top-kdecisions, enabling causal routing at inference (Ra- poso et al., 2024; Shi et al., 2025). Alternatively, top-kselec- tion across the current tokens from different sequences pre- serves causality within each sequence (Ludziejewski et al., 2024; Wen et al., 2025). Recent work changes routing gran- ularity: Lory routes at the segment level, using the previous segment to determine the next (Zhong et al., 2024), while Se- qTopK shifts expert budgets to sequence-level selection with an Expert Cache for autoregressive decoding (Wen et al., 2025). All above approaches have significant drawbacks: predictions can be noisy and unstable, and batch-level top-k can impose inference-time topology constraints, leading to a large train–inference mismatch; moreover, routing that depends on global batch composition can be sensitive to batch size/composition and raises privacy/safety concerns in multi-tenant settings (Wen et al., 2025). In contrast to 8 Expert Threshold Routing EC, ET reduces to a simple threshold test (whether token logitr t,i is higher than cutoff EMAc i ) at inference time, thus eliminating the train–inference discrepancy. 5.5. From Batch to Population Level Statistics The progression from sample, batch, to population-level statistics is a recurring theme in deep learning. While tech- niques like Batch Normalization (Ioffe & Szegedy, 2015) and contrastive learning (Radford et al., 2021) rely on batch statistics, momentum-based approaches (He et al., 2020; Caron et al., 2021) and adaptive optimizers like Adam (Kingma & Ba, 2015) use Exponential Moving Av- erages (EMA) to approximate population distributions. ET applies this principle to routing via EMA-based cutoffs. 6. Conclusion We introduce Expert Threshold (ET) routing, a mecha- nism that resolves the fundamental causality issue in Expert Choice (EC) models while preserving their load-balancing advantages. By maintaining an exponential moving average of each expert’s selection threshold, estimated from histor- ical batches rather than within-batch top-kselection, ET routing enables fully causal routing. Each token’s routing decision depends only on past statistics, eliminating the need for future token access at both training and inference time. Our experiments demonstrate that ET routing achieves com- petitive performance with EC routing (matching validation loss at 2.84) while outperforming Token Choice by 0.067 in cross-entropy loss, all while enabling causal autoregressive generation. The cutoff-EMA mechanism provides stable routing thresholds that accurately approximate EC’s top-k boundaries, as evidenced by the minimal train-inference gap observed across all metrics. We further show that a warmup strategy, using EC routing before transitioning to threshold-based selection, stabilizes early training dynamics. These findings suggest that the perceived incompatibility between Expert Choice routing and causal language mod- eling can be effectively bridged through population-level threshold estimation, opening new directions for scalable MoE architectures. Acknowledgments We gratefully acknowledge the support of NVIDIA Corpora- tion and the NVIDIA AI Technology Center (NVAITC) UF program. We thank Hongwu Peng for the generous support and guidance on development of the code. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. References Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points, 2023a. URLhttps://arxiv.org/abs/ 2305.13245. Ainslie, J., Lei, T., de Jong, M., Ontanon, S., Brahma, S., Zemlyanskiy, Y., Uthus, D., Guo, M., Lee-Thorp, J., Tay, Y., Sung, Y.-H., and Sanghai, S. CoLT5: Faster long-range transformers with conditional computation. In Bouamor, H., Pino, J., and Bali, K. (eds.), Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, p. 5085–5100, Singapore, December 2023b. Association for Computa- tional Linguistics. doi: 10.18653/v1/2023.emnlp-main. 309. URLhttps://aclanthology.org/2023. emnlp-main.309/. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In IEEE/CVF Inter- national Conference on Computer Vision, p. 9650–9660, 2021. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Cheng, A., Zhang, K., Sui, J., Zhao, X., Xing, N., Peng, Z., Jie, S., Yang, T., Gao, W., Wang, Q., Zeng, Y., Gao, C., Xiong, R., and Sun, X. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024. URL https://arxiv.org/abs/2401.06066. DeepSeek-AI. Deepseek-v3 technical report, 2024. URL https://arxiv.org/abs/2412.19437. Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. Scaling vision transformers to 22 billion parameters, 2023. URLhttps://arxiv. org/abs/2302.05442. 9 Expert Threshold Routing Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. Guo, Y., Cheng, Z., Tang, X., Tu, Z., and Lin, T. Dy- namic mixture of experts: An auto-tuning approach for efficient transformer models, 2025. URLhttps: //arxiv.org/abs/2405.14297. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9729–9738, 2020. Huang, Q., An, Z., Zhuang, N., Tao, M., Zhang, C., Jin, Y., Xu, K., Chen, L., Huang, S., and Feng, Y. Harder tasks need more experts: Dynamic routing in moe mod- els, 2024. URLhttps://arxiv.org/abs/2403. 07652. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, p. 448–456. PMLR, 2015. Jin, C., Peng, H., Xiang, M., Zhang, Q., Yuan, X., Hasan, A., Dibua, O., Gong, Y., Kang, Y., and Metaxas, D. N. Sparsity-controllable dynamic top-p moe for large founda- tion model pre-training, 2025. URLhttps://arxiv. org/abs/2512.13996. Jin, P., Zhu, B., Yuan, L., and Yan, S. Moe++: Accelerat- ing mixture-of-experts methods with zero-computation experts, 2024. URLhttps://arxiv.org/abs/ 2410.07348. Jordan, K. Muon: An optimizer for hidden layers in neu- ral networks.https://kellerjordan.github. io/posts/muon/, 2024. Blog post. Karpathy, A.nanochat: The best chatgpt that $100 can buy.https://github.com/karpathy/ nanochat, 2025. GitHub repository. Kilian, M., Mkrtchyan, O., Zettlemoyer, L., Shrivastava, A., and Aghajanyan, A. Improving moe compute ef- ficiency by composing weight and data sparsity, 2026. URL https://arxiv.org/abs/2601.15370. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. Komatsuzaki, A., Puigcerver, J., Lee-Thorp, J., Ruiz, C. R., Mustafa, B., Ainslie, J., Tay, Y., Dehghani, M., and Houlsby, N. Sparse upcycling: Training mixture-of- experts from dense checkpoints. In International Confer- ence on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/2212.05055. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. GShard: Scal- ing giant models with conditional computation and auto- matic sharding. In International Conference on Learning Representations, 2021. URLhttps://openreview. net/forum?id=qrwe7XHTmYb. Li, J., Fang, A., Smber, G., Wortsman, M., Gadre, S. Y., Schmidt, L., et al. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794, 2024. Lin, X. V., Shrivastava, A., Luo, L., Iyer, S., Lewis, M., Ghosh, G., Zettlemoyer, L., and Aghajanyan, A. Moma: Efficient early-fusion pre-training with mix- ture of modality-aware experts, 2024. URLhttps: //arxiv.org/abs/2407.21770. Liu, H., Li, Y., Shen, Y., Wang, B., Liang, C., Jiang, C., Li, C., Deng, D., Ding, F., Gao, W., et al. Moonlight: A cost-effective approach for pre-training large language models, 2025a. URLhttps://arxiv.org/abs/ 2502.16456. Liu, T., Blondel, M., Riquelme Ruiz, C., and Puigcerver, J. Routers in vision mixture of experts: An empirical study. Transactions on Machine Learning Research, 2024. URLhttps://openreview.net/forum? id=aHk3vctnf1. Also available as arXiv:2401.15969. Liu, Z., Li, Y., Zhang, X., Teng, Q., Jiang, S., Chen, X., Shi, H., Li, J., Wang, Q., Chen, H., Meng, F., Zhao, M., Xu, Y., He, Y., Hu, B., and Zhang, M. Unimoe-audio: Unified speech and music generation with dynamic- capacity moe, 2025b. URLhttps://arxiv.org/ abs/2510.13344. Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019. URLhttps://openreview. net/forum?id=Bkg6RiCqY7. Ludziejewski, J., Krajewski, J., Adamczewski, K., Pioro, M., Chowdhury, S., Sanyal, A., Miasojedow, B., Pontes, H. R., Jaszczur, S., Pacek, B., Jastrz ̨ebski, S., Bousquet, O., Hoogeboom, E., and Michalewski, H. Scaling laws for fine-grained mixture of experts. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 32790–32809, 2024. URLhttps://proceedings. mlr.press/v235/ludziejewski24a.html. 10 Expert Threshold Routing Ni, J. and team.Openmoe 2:Sparse diffu- sion language models.https://github.com/ JinjieNi/OpenMoE2, 2025. Penedo, G., Kydlí ˇ cek, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2024. URLhttps://openreview.net/forum? id=n6SCkn2QaG. Qiu, Z., Huang, Z., Zheng, B., Wen, K., Wang, Z., Men, R., Titov, I., Liu, D., Zhou, J., and Lin, J. Demons in the detail: On implementing load balancing loss for train- ing specialized mixture-of-expert models, 2025. URL https://arxiv.org/abs/2501.11873. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog, 2019. URLhttps://openai. com/blog/better-language-models/. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, p. 8748–8763. PMLR, 2021. Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parame- ter models, 2020. URLhttps://arxiv.org/abs/ 1910.02054. Raposo, D., Ritter, S., Richards, B., Lillicrap, T., Humphreys, P. C., and Santoro, A. Mixture-of-depths: Dynamically allocating compute in transformer-based lan- guage models. arXiv preprint arXiv:2404.02258, 2024. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. Shi, M., Yuan, Z., Yang, H., Wang, X., Zheng, M., Tao, X., Zhao, W., Zheng, W., Zhou, J., Lu, J., Wan, P., Zhang, D., and Gai, K. Diffmoe: Dynamic token selection for scalable diffusion transformers, 2025. URLhttps:// arxiv.org/abs/2503.14487. So, D. R., Ma ́ nke, W., Liu, H., Dai, Z., Shazeer, N., and Le, Q. V. Primer: Searching for efficient transformers for language modeling, 2021. URLhttps://arxiv. org/abs/2109.08668. Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y.Roformer: Enhanced transformer with ro- tary position embedding.Neurocomputing, 568: 127063, 2024.doi: 10.1016/j.neucom.2023.127063. URLhttps://w.sciencedirect.com/ science/article/pii/S0925231223011864. Sun, H., Lei, T., Zhang, B., Li, Y., Huang, H., Pang, R., Dai, B., and Du, N. Ec-dit: Scaling diffusion transformers with adaptive expert-choice routing, 2024. URLhttps: //arxiv.org/abs/2410.02098. Tan, S., Shen, Y., Panda, R., and Courville, A. Scattered mixture-of-experts implementation, 2024. URLhttps: //arxiv.org/abs/2403.08245. Team, G., Riviere, M., Pathak, S., Sessa, P. G., Cassirer, C., Coppey, L., El-Boukkouri, K., et al. Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv.org/abs/2408.00118. Team, M. A. Longcat-flash technical report, 2025a. URL https://arxiv.org/abs/2509.01322. Team, Q. Qwen3 technical report, 2025b. URLhttps: //arxiv.org/abs/2505.09388. Wang, A., Sun, X., Xie, R., Li, S., Zhu, J., Yang, Z., Zhao, P., Han, W., Kang, Z., Wang, D., Okazaki, N., and Xu, C.- z. HMoE: Heterogeneous mixture of experts for language modeling. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 21943–21957, Suzhou, China, November 2025a. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main. 1115. URLhttps://aclanthology.org/2025. emnlp-main.1115/. Wang, L., Gao, H., Zhao, C., Sun, X., and Dai, D. Auxiliary-loss-free load balancing strategy for mixture- of-experts, 2024. URLhttps://arxiv.org/abs/ 2408.15664. Wang, Z., Zhu, J., and Chen, J. Remoe: Fully differentiable mixture-of-experts with reLU routing. In The Thirteenth International Conference on Learning Representations, 2025b. URLhttps://openreview.net/forum? id=4D0f16Vwc3. Wen, T., Wang, Y., Feng, A., Ma, L., Liu, X., Wang, Y., Guo, L., Chen, B., Jegelka, S., and You, C. Route experts by sequence, not by token, 2025. URLhttps://arxiv. org/abs/2511.06494. Xin, J., Tang, R., Lee, J., Yu, Y., and Lin, J. Deebert: Dy- namic early exiting for accelerating bert inference. In Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, p. 2246–2251, 2020. 11 Expert Threshold Routing Yan, S., Bin, X., Zhang, S., Wang, Y., and Lin, Z. TC-MoE: Augmenting mixture of experts with ternary expert choice. In International Conference on Learning Representations (ICLR), 2025. URLhttps://openreview.net/ forum?id=dsP91M4hDL. Poster. Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer, 2022. URLhttps://arxiv. org/abs/2203.03466. Yang, Y., Qi, S., Gu, W., Wang, C., Gao, C., and Xu, Z. XMoE: Sparse models with fine-grained and adaptive expert selection. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, p. 11664– 11674, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. findings-acl.694. URLhttps://aclanthology. org/2024.findings-acl.694/. Zeng, Z., Miao, Y., Gao, H., Zhang, H., and Deng, Z.AdaMoE: Token-adaptive routing with null ex- perts for mixture-of-experts language models. In Al- Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Find- ings of the Association for Computational Linguis- tics: EMNLP 2024, p. 6223–6235, Miami, Florida, USA, November 2024. Association for Computational Linguistics.doi: 10.18653/v1/2024.findings-emnlp. 361. URLhttps://aclanthology.org/2024. findings-emnlp.361/. Zhang, B. and Sennrich, R. Root mean square layer nor- malization, 2019. URLhttps://arxiv.org/abs/ 1910.07467. Zhong, Z., Xia, M., Chen, D., and Lewis, M.Lory: Fully differentiable mixture-of-experts for autoregres- sive language model pre-training. In Conference on Language Modeling (COLM), 2024.URLhttps: //openreview.net/forum?id=LKEJPySnlt. Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A. M., Le, Q. V., Laudon, J., et al. Mixture-of- experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103–7114, 2022. A. Future Information Leakage for Expert Choice Models In DeepSeek’s loss-free load balancing paper (Wang et al., 2024), they give an upper bound on the future information leakage of Expert Choice (EC) to be superlinear in the number of tokens. They considered all potential selection combinations N k for choosek = N/Etokens out ofN tokens, which makes upper boundlog 2 N k = O(N logN ). We consider two scenarios: when cutoff threshold is ex- pressed as a finite precision float, it is trivial that the total future information leakage is at most the number of bits to represent the cutoff threshold. However, when cutoff thresh- old is of infinite precision, we show that we can indeed leak at leastO(N logN )bits of future information, making the bound tight. We arrange the following sections as follows: 1.First, we provide a formal definition of future informa- tion leakage. 2. Then, we show for finite precision, the total leakage is constant, which means per token leakage is 0 as batch size increases. 3.Then, we show for infinite precision, we describe an encoding strategy that can leak at leastO(N logN ) bits of future information, making the bound tight. The idea is that we can break the cutoff space into2 N small intervals, and injectively (though not surjectively) map each potential selection combination to a unique interval in a memoryless way. 4. We formally prove that the encoding strategy can leak at least O(N logN ) bits of future information. Gladly, since we rely on finite precision float to represent the cutoff threshold, ET is still causal. A.1. Definition of Future Information Leakage Definition A.1 (Future information leakage). Fix a se- quence lengthNand a deterministic selection ruleFthat maps a logit sequencer 1:N to a subsetF (r 1:N ) ⊆ [N ] (where[N ]≜ 1,...,N). Herer 1:N denotes the entire sequence of router scores/logits across tokens, and we write z t ≜ 1[t∈ F (r 1:N )] for the induced selection indicator. An advice variable is any functionA = α(r 1:N )with finite range, whereαis an encoder that maps the full logit se- quence to a finite label. We writeRange(α)≜α(r 1:N ) : r 1:N for the set of labels that can be produced by α. We sayAcausalizesFif there exist functionsg t N t=1 such that for all r 1:N and all t∈ [N ], z t = g t (r 1:t ,A). The future information leakage of F on length N is L F ([1 :N ])≜ min α,g t log 2 |Range(α)|. 12 Expert Threshold Routing A.2. Finite-precision cutoff implies constant leakage With this definition, it is trivial to show that, under finite precision float representation, the total future information leakage is constant, which means per token leakage is 0 as batch size increases. In our case,Fis the Expert Choice selection rule induced by a cutoff threshold: for each expert, tokens are selected by comparing their router scores against the expert’s cut- off (equivalently, selecting those above the cutoff, which matches top-kwhen the cutoff is set to thek-th order statis- tic). Theorem A.2 (Finite-precision cutoff implies constant leak- age). If the cutoff threshold is represented withbbits of precision (e.g.,b = 16forbf16orb = 32forfp32), then the future information leakage satisfiesL F ([1 :N ])≤ bfor all N . Proof. This is an upper bound via a particular advice choice. Let the advice be the cutoff itself:A = α(r 1:N )≜ β, encoded inbbits, so|Range(α)| ≤ 2 b . GivenA = β, the selection indicator at timetis a causal function of the prefix (in fact, ofr t ) andβby thresholding, henceAcausalizesF. ThereforeL F ([1 :N ])≤ log 2 |Range(α)|≤ b. A.3. Upper bound on Infinite-precision future information leakage The strategy we used in the previous subsection does not work for infinite precision cutoff, as it takes infinite bits of communication to represent the cutoff. Thus, we need a different encoding strategy. Previously, Loss-free Load Balancing paper Wang et al. (2024) gave a combinatorial upper bound on the information carried by an Expert Choice allocation when we allow all admissible token-to-expert assignments consistent with the sparsity pattern. Using the token-choice notation from Loss-free Load Bal- ancing, letNdenote the number of tokens in the routing pool andGEthe number of routed experts. Each token activatesGrouted experts, so the MoE sparsity is 1 E . For an MoE layer in Expert Choice, the maximum information leakage L (bits per token) is: L = GE N log 2 N N E > GE N · N E log 2 (E− 1) = G log 2 (E− 1).(7) For a model with sparsity 1 E = 2 16 = 0.125and 9 MoE layers, the total leakage information is more than 50 bits per token. A.4. Encoding Strategy and Decoding Procedure The above upper bound is pretty intuitive to understand. We ask, can we reach this bound? Surprisingly, we can. In this subsection, we describe an encoding strategy where we cre- ate an injective mapping from the set of all possible expert choice combinations to the range of the infinite precision cutoff. Suppose the cutoffc ∈ [0, 1]. We partition this space into 2 N dyadic intervals. Letz ∈ 0, 1 N be the target expert choice combination (codeword). We map z to the interval: I(z)≜ h N X t=1 (1− z t )2 −t , N X t=1 (1− z t )2 −t + 2 −N . This orders codewords in descending order:00· 00maps to the rightmost interval and11· 11maps to the leftmost. Choose a cutoff valuec ∈ I(z). Obviously, this mapping is injective, i.e. each codeword maps to a unique interval. However, since not all combinations are possible, it is not surjective. Information from the Past. For a tokent, what infor- mation about the past routing logitsr 1:t and decisionsz 1:t can we use to guide the selection of the interval? Turns out, the only information we get is the current bracket[ℓ,u) containing the decision boundary. The upper bounduis determined by the lowest logit of the selected tokens, and the lower boundℓby the highest logit of the unselected tokens. To maximize information leakage, we minimize help from the past by always choosing the next query logit to be in the middle of the interval,r t = (ℓ + u)/2. Then, the routing decisionz t = 1r t ≥ creveals exactly one bit of the cutoff, requiring full future information. Decoding Procedure.We formalize this procedure in Al- gorithm 2. A.5. Formal Proof We now formally prove that the amount of information re- quired to implement the Expert Choice selection causally is lower-bounded by the combinatorial entropy of the selection space. This confirms that the upper bound in Eq. (7) is tight and that the infinite-precision construction in Algorithm 2 is optimal in terms of information leakage. Theorem A.3. For a single expert with capacityk = ⌊N/E⌋ , any causal routing mechanism that can realize all possible top-kassignments requires at leastlog 2 N k bits of non-causal information (advice). Proof. LetZ k =z ∈0, 1 N : P N t=1 z t = k be the set of all valid selection indicators for the expert, with|Z k | = 13 Expert Threshold Routing Algorithm 2 Binary-search decoding of an infinite-precision cutoff 1:Input: horizonN; unknown cutoffc ∈ [0, 1]; oracle bit z t = 1r t ≥ c 2: ℓ← 0, u← 1 3: for t = 1,...,N do 4: r t ← (ℓ + u)/2 Query midpoint 5:Observe selection z t ∈0, 1 6:if z t = 1 then 7: u← r t Selected (r t ≥ c) =⇒ c∈ [ℓ,r t ) 8:else 9: ℓ← r t Not selected (r t < c) =⇒ c∈ [r t ,u) 10:end if 11: end for 12: Return (z 1 ,...,z N ) and interval [ℓ,u) Figure 9. Illustration of the binary-search encoding strategy. The cutoff valuecpartitions the interval[0, 1]into regions correspond- ing to different expert selection patterns, enablingNbits of future information to be encoded in a single real-valued threshold. N k . We show that distinct advice is necessary for every distinct pattern inZ k . Consider the specific family of logit sequences generated by the binary search process in Algorithm 2. In this construc- tion, the router logitr t is the midpoint of the current valid interval[ℓ,u), which is determined solely by the history of decisionsz 1 ,...,z t−1 . Consequently, if two selection patternszandz ′ share the same prefixz 1:t−1 , they will generate the exact same logit r t at step t. Suppose there exists an advice encoding with range size strictly less than N k . By the Pigeonhole Principle, at least two distinct valid patternsz,z ′ ∈Z k must share the same advice value. Lettbe the first index where they differ (i.e.,z 1:t−1 = z ′ 1:t−1 butz t ̸= z ′ t ). Since the prefixes are identical, the generated logit sequencer 1:t is identical for both patterns. A causal decoder, which must output a decision at timetbased only onr 1:t and the advice, receives identical inputs for both cases. It must therefore produce the same output. However,z t ̸= z ′ t , so the decoder necessarily fails for at least one of the patterns. Thus, every valid top-kpattern requires a unique advice value. The minimum information leakage islog 2 N k bits per expert. Summing this lower bound overGEexperts recovers the combinatorial quantity in Eq. (7), proving that Expert Choice routing fundamentally requires significant future information to implement. B. Architecture Details Our model architecture follows nanochat (Karpathy, 2025), which differs from standard GPT-2 (Radford et al., 2019) in several ways. Table 5 summarizes the key differences. B.1. Data and Tokenization We train on the FineWeb-Edu 100B shuffle dataset (Penedo et al., 2024), a high-quality educational web corpus. Tok- enization uses RustBPE with a vocabulary of 65,536 tokens (64k, power-of-2 aligned for GPU efficiency). We use a sequence length of 2048 tokens. B.2. Model Size Configurations Table 5 shows the model configurations used in our experi- ments. We follow the nanochat naming convention where d * indicates the number of layers, andn embd = d× 64, with head dimension fixed at 128. Attention configuration. We use grouped-query atten- tion (Ainslie et al., 2023a) with head dimension 128 (larger than GPT-2’s 64). The number of attention heads isn head = n embd /128, giving 6 heads for d12 and 10 heads for d20. QK normalization is applied before the attention computation. MoE configuration.For MoE variants, we use 16 routed experts with granularityG=1and expansionE=16, plus 1 shared expert (17 total). Each routed and shared expert has dimensiond expert = 2 × n embd (half the dense FFN dimension). The shared expert processes every token, while each token activates on average 1 routed expert, matching the dense model’s active parameter count. C. Training Setup Details C.1. Training Hyperparameters C.1.1. WEIGHT INITIALIZATION We follow the nanochat initialization scheme (Karpathy, 2025), which uses aspect-ratio scaled initialization. For a weight matrix W ∈ R d out ×d in : 14 Expert Threshold Routing Table 5. Model architecture and size configurations. Architecture features are shared between d12 and d20 (nanochat-style). For MoE variants withG=1,E=16: 16 routed experts + 1 shared = 17 total experts. Total params include all expert parameters; active params include only the shared expert plus on average one routed expert per token. FeatureGPT-2d12d20 TokenizationGPT-2RustBPE Vocab Size50,25765,536 ActivationGELUReLU 2 (So et al., 2021) NormalizationLayerNormRMSNorm (Zhang & Sennrich, 2019) FFN Dimension4× d model 4× d model Linear Layer BiasYesNo Position EncodingLearnedRoPE (Su et al., 2024) Head Dimension64128 QK NormalizationNoYes (Dehghani et al., 2023) Logits SoftcappingNo15.0 (Team et al., 2024) Embedding WeightsTied (wte = lm_head)Untied First Layer Dense—Yes n embd 7687681280 n layer 121220 n head 12610 KV heads1222 Dense Params124M195M561M MoE Total Params —575M2429M MoE Active Params—195M561M Table 6. Training hyperparameters. HyperparameterValue Total tokens10B / 11.2B Batch size (tokens)524,288 (0.5M) Sequence length2048 Muon Warmup stepsNo warmup AdamW Warmup steps250 Learning rate scheduleLinear decay Min learning rate0.1× peak LR Gradient clippingNone Weight decay0.0 AdamW β 1 ,β 2 0.9, 0.95 std = 1 √ d in · min 1, r d out d in ! (7) This formula reduces to standard1/ √ d in initialization for square or tall matrices, but scales down variance for wide matrices where d out ≫ d in . Component-specific initialization: •Embeddings:N (0, 1)– standard normal initialization •Output projections (lm_head,c_proj): Zero ini- tialization, critical for Muon optimizer stability •Router weights:N (0, 1/ √ d in )– small init for sym- metry breaking • Expert weights: Aspect-ratio scaled for up projections, zero for down projections • Attention weights: Aspect-ratio scaled as above Table 7. Parameter initialization and optimizer configuration. Aspect-ratio scaled init. usesstd = d −1/2 in · min(1, p d out /d in ). LRs includeμP (Yang et al., 2022) scalingλ = (d model /768) −1/2 . ParameterInit.LROpt. W E (embed.) N(0, 1)0.2λAdamW W lm (head) 00.004λAdamW W QKV (attn)Asp.-ratio 0.02λMuon W O (attn proj) 00.02λMuon W router N(0,d − 1 2 ) 0.02λMuon W (e) ↑ (exp. up)Asp.-ratio 0.02λMuon W (e) ↓ (exp. dn) 00.02λMuon W (s) ↑ (shd. up)Asp.-ratio 0.02λMuon W (s) ↓ (shd. dn) 00.02λMuon C.1.2. OPTIMIZER CONFIGURATION We use a hybrid optimizer setup following nanochat (Jordan, 2024; Liu et al., 2025a): •Muon (Jordan, 2024) for 2D/3D weight matrices (at- tention, MLP, experts): momentum-based optimizer with Newton-Schulz orthogonalization • AdamW (Loshchilov & Hutter, 2019) for embed- dings and output head: with learning rate scaling 15 Expert Threshold Routing ∝ 1/ p d model /768 No weight decay is used, as Muon provides implicit regular- ization and language models benefit from memorization. C.2. Hardware Infrastructure Details Hardware.We train our models on a single node with 8x NVIDIA B200 GPUs, each with 180GB of memory. Code. For TC models, we use the ScatterMoE backend (Tan et al., 2024). For EC and ET models, we write our own custom Pytorch MoE implementation. We use padding to handle variable number of tokens per expert. Parallelism. We rely on Nanochat’s implementation of distributed AdamW and Muon optimizer use a ZeRO-2 style gradient synchronization (Rajbhandari et al., 2020). For EC and ET models, we write our own expert paralleliza- tion all-to-all communication framework. This allows us to use maximum batch size during routing instead of micro- batches, reaching a better usage/cutoff variance trade-off while saving memory. D. CORE Evaluation Details We evaluate using the CORE benchmark (Li et al., 2024), which provides a standardized suite of in-context learning tasks for language model evaluation. Task types. The CORE benchmark includes multiple- choice tasks, schema matching tasks, and language model- ing tasks, testing various aspects of language understanding. Metric. The primary metric is centered accuracy, which adjusts for random baseline performance: acc centered = acc− 0.01× baseline random 1.0− 0.01× baseline random (8) This normalization ensures that random guessing yields a score near zero, while perfect accuracy yields 1.0. The final CORE Eval score is the mean of centered accuracy across all tasks. Evaluation protocol. We evaluate at fixed intervals dur- ing training (every 250 steps by default) to track learning dynamics. E. Ablations E.1. Warmup We find warmup crucial for ET. In the early stages of train- ing, the cutoff threshold is not yet stable, while the EMA lags behind the actual cutoff threshold because of slow up- date speed (1/(1 − β) ≈ 1000steps). As a result, the threshold-based routing becomes unreliable: tokens that should be routed are dropped, and the capacity lower bound is frequently triggered (Figure 10c). This leads to under- trained experts during early training. To address this, we warm up the routing by using TopK selection for the first 4,000 steps before switching to threshold-based routing. ET no warmup relies solely on the capacity factor during these early steps, which is suboptimal because capacity control can limit collapse but does not provide a stable threshold estimate or balanced expert learning signal before the cut- off EMA has converged. As shown in Figure 10, warmup stabilizes the cutoff-EMA trajectory (a, d), increases raw expert usage (b), and reduces starvation rate (c). We also observe that ET no warmup exhibits higher variance in both logits (e) and gate outputs (f), suggesting less stable gradient signals. E.2. Comparison to Token Choice In our setup, Token Choice with loss-free load balancing shows a less stable routing trajectory than ET and EC, es- pecially in early layers. Figure 11 compares cutoff-EMA (expert 0) at layer 1. ET and EC stabilize quickly, while DeepSeek’s loss-free controller drifts upward over training. We treat this as an exploratory observation rather than a central claim, since the behavior may depend on hyperpa- rameters and gating parameterization. E.3. Shared Expert We report results of EC and ET no warmup with and without shared expert. For no shared experts, we select 2 experts out of 16 routed experts, roughly matching the number of parameters and compute to the shared variant. In both cases, shared expert improves loss by roughly 0.02. We suspect that while later layers need early layers to empower the router, sometimes early layers have no activated experts, causing ineffective routing. See Table 8 for more details. Table 8. Ablation on the shared expert mechanism. In both ET no warmup and EC, shared expert improves loss by roughly 0.02. MethodSharedCE CORE EC (bsz 512k)Yes2.84319.94 EC (bsz 512k)No2.86216.307 ET no warmup (β=0.999)Yes2.84416.867 ET no warmup (β=0.999)No2.86218.515 E.4. Normalization Initially, we assumed that dynamic expert count would bring instability in training because of the scale expansion. How- ever, we found that normalization was ineffective in our 16 Expert Threshold Routing 02000400060008000 step 2 0 2 4 value L9 E0 cutoff vs EMA gec gec_warmup ec 02000400060008000 step 0.0 0.2 0.4 0.6 0.8 value Raw expert usage gec gec_warmup 02000400060008000 step 0.0 0.2 0.4 0.6 0.8 1.0 value Starvation rate gec gec_warmup (a) L9 cutoff vs EMA(b) Raw expert usage(c) Starvation rate 02000400060008000 step 2 1 0 value Layer cutoff_ema (L6 E0) gec gec_warmup ec 02000400060008000 step 1.00 1.25 1.50 1.75 2.00 2.25 value Router logits std gec gec_warmup ec 02000400060008000 step 0.05 0.10 0.15 0.20 0.25 0.30 value Gate std gec gec_warmup ec (d) L6 cutoff-EMA(e) Router logits std(f) Gate std Figure 10. Effect of TopK warmup on ET training dynamics (first 8k steps). Before 4k steps, ET no warmup exhibits unstable threshold routing: (a) the cutoff-EMA lags behind the actual cutoff, (b) raw expert usage is low, and (c) starvation rate is high as the capacity lower bound is frequently triggered. ET no warmup relies only on the capacity factor during this stage, which is suboptimal because it does not provide a stable threshold estimate or balanced expert learning signal. With warmup, the cutoff-EMA trajectory stabilizes (d), and router outputs show lower variance in both logits (e) and gates (f). Note:ec_shared_bsz512kdoes not log raw usage and underflow metrics, so panels (b) and (c) show only the two ET runs. 05k10k15k20k Step 0 10 20 30 40 50 Cutoff EMA (L1, E0) EC ET DeepSeek LB Figure 11. Layer-1 cutoff-EMA (expert 0) under EC/ET vs DeepSeek loss-free load balancing. setting. No norm outperformed fanout norm by 0.04 in CE loss. We suspect that the norm made experts’ contribution unpredictable (see Figure 12). F. Additional Experiment Results F.1. Capacity Constraints Because ET’s thresholding does not fix the per-batch num- ber of selected tokens for each expert, expert loads can fluctuate around the target, which can risk GPU out-of- memory. Following standard practice (Fedus et al., 2022), 8k10k12k14k16k18k Step 2.85 2.90 2.95 3.00 3.05 Eval Loss No Normalization Fanout Normalization Figure 12. Comparison of evaluation loss with and without nor- malization. The configuration without normalization (blue) con- sistently achieves lower loss than the fanout-normalized variant (orange). we enforce capacity constraints during training: each ex- pert processes between(1− C)· N/Eand(1 + C)· N/E tokens per batch (capacity factorC = 0.5), with excess tokens dropped or capacity padded. Since these constraints are absent at inference, frequent triggering would cause train-inference mismatch. Figure 13 shows that capacity constraints are triggered infrequently: after warmup, both saturation and starvation rates remain low. This confirms that train-inference mismatch from capacity constraints is minimal. Figure 13 shows capacity constraint metrics for ET (with 17 Expert Threshold Routing warmup) from step 4k onward. After warmup, raw expert usage stabilizes around 6.5%, and both saturation and star- vation rates remain low, confirming that capacity constraints are rarely triggered and train-inference mismatch is mini- mal. F.2. Routing Consistency Sweep Section 4.3.6 reports the main weighted Jaccard heatmap strip. Here we define the routing-consistency metrics used throughout the comparison and include the companion joint JSD heatmap for the same four runs. Objects being compared. For a given token-layer pair, letA t andB t denote the sets of active routed experts under checkpointsAandB. The shared expert is excluded, so these sets contain only routed experts. Pooling all active token-layer-expert edges across the full comparison gives E A =(ℓ,t,i) : i∈ A t , E B =(ℓ,t,i) : i∈ B t . The pooled edge setsE A andE B are used by the weighted overlap metrics, while the token-level setsA t andB t are used by the per-token overlap and divergence metrics below. Metric definitions.Our main metric is weighted Jaccard, defined on the pooled edge sets weighted_jaccard = |E A ∩ E B | |E A ∪ E B | . Its pooled Dice companion is weighted_dice = 2|E A ∩ E B | |E A | +|E B | . We also report token-level overlap averaged uniformly over token-layer pairs J t = |A t ∩ B t | |A t ∪ B t | ,Dice t = 2|A t ∩ B t | |A t | +|B t | . The reportedjaccardanddiceare the means ofJ t and Dice t over all token-layer pairs. For the divergence metrics, we convert each token’s binary activation set into a distribution over experts by assigning uniform mass to the active experts P t (i) = ( 1/|A t |, i∈ A t 0,otherwise Q t (i) = ( 1/|B t |, i∈ B t 0,otherwise. Using these distributions, we compute joint_jsd(P t ,Q t ) = 1 2 KL(P t ∥M t ) + 1 2 KL(Q t ∥M t ), M t = 1 2 (P t + Q t ), and total_variation(P t ,Q t ) = 1 2 X i |P t (i)− Q t (i)|. The reportedjoint_jsdandtotal_variationare averages over token-layer pairs, and lower values indicate more stable routing. Empty-routing conventions.If both checkpoints activate no routed expert for a token-layer pair, we set token-level Jaccard and Dice to1, and joint JSD and total variation to 0. If only one checkpoint activates any routed expert, we set token-level Jaccard and Dice to0, and joint JSD and total variation to1. For the pooled metrics, if both pooled edge sets are empty, weighted Jaccard and weighted Dice are both defined as 1. F.3. Activation Dynamics Sweep Across Routing Variants Section 4.3.2 focuses on EC (2k) and ET. This subsection first gives the layerwise continuation for those two main runs using loss binned fanout views, then shows EC 8k in the same overlaid form as the main figure before collecting inverse layerwise diagnostics for the remaining three runs. Across these additional variants, EC 8k is much flatter than EC 2k in the same overlaid inverse view used in the main text. The remaining three inverse layerwise panels show that the loss–fanout relation remains setup dependent, with substantial variation across routing variants and depth. For completeness, Figure 18 shows representative router logit histograms from the warmup ET run. We include them as a qualitative diagnostic of router behavior. This appendix also provides extended expert specialization analysis, complementing the summary in Section 4.3.3. Per-token routing visualizations.Figures 19, 20, and 21 show token-level expert routing for additional passages from GSM8K and HumanEval. Across GSM8K passages, content-bearing tokens—particularly numbers (e.g., “48”, “72”), mathematical operators (“/”, “+”, “=”), and compu- tation markers (“<<”)—consistently receive the highest fanout. Function words and punctuation receive minimal activation, indicating that experts preferentially process se- mantically rich tokens. In HumanEval passages, a similar pattern holds: code-specific tokens (variable names, oper- ators, keywords) receive higher activation than boilerplate text and whitespace. Expert activation heatmaps.Figure 22 shows expert to- ken ratios across all routing configurations. Each heatmap plots expert ID (columns) versus layer (rows), with color 18 Expert Threshold Routing 5k10k15k Step 0.062 0.064 0.066 0.068 0.070 Raw Usage (a) Raw Expert Usage ET 5k10k15k Step 0.00 0.02 0.04 0.06 0.08 Saturation Rate (b) Saturation Rate ET 5k10k15k Step 0.00 0.01 0.02 Starvation Rate (c) Starvation Rate ET Figure 13. Capacity constraint behavior during ET training (from step 4k onward, after warmup). (a) Raw expert usage before capacity capping. (b) Saturation rate: fraction of selected tokens dropped due to capacity limits. (c) Starvation rate: fraction of unused expert capacity. Both saturation and starvation rates remain low, confirming minimal train-inference mismatch. 5k10k15k19k Checkpoint 5k 10k 15k 19k Checkpoint 0.0000.1930.2970.264 0.1930.0000.2450.203 0.2970.2450.0000.205 0.2640.2030.2050.000 EC 2k 5k10k15k19k Checkpoint 5k 10k 15k 19k 0.0000.1620.1810.169 0.1620.0000.1500.126 0.1810.1500.0000.110 0.1690.1260.1100.000 EC 64k 5k10k15k19k Checkpoint 5k 10k 15k 19k 0.0000.1960.1970.197 0.1960.0000.1660.145 0.1970.1660.0000.120 0.1970.1450.1200.000 ET 5k10k15k19k Checkpoint 5k 10k 15k 19k 0.0000.2160.2440.254 0.2160.0000.1460.148 0.2440.1460.0000.113 0.2540.1480.1130.000 TC 0.00 0.05 0.10 0.15 0.20 0.25 Joint JSD Figure 14. Within-family checkpoint-pair routing consistency on a fixed validation stream, measured by joint JSD. Lower values indicate more stable routing. The same broad story remains. ET separates clearly from EC 2k and stays close to EC 64k. intensity indicating the fraction of domain-specific tokens routed to each expert. The left column shows HumanEval (code) and the right column shows GSM8K (math). Several patterns emerge across batch sizes. EC with batch size 2k (top row) shows diffuse activation: while some ex- perts exhibit domain preferences (e.g., concentrated dark cells at specific layer-expert pairs), the overall pattern is noisy with activation spread across many experts. As batch size increases to 8k and 64k, specialization sharpens—dark cells become more concentrated and background activa- tion fades, indicating that experts more consistently capture domain-specific tokens when routing decisions are made over larger token pools. EC at 512k shows the most pro- nounced specialization, with a small number of experts per layer handling the majority of domain tokens. ET (bottom row) achieves specialization comparable to large-batch EC. The activation patterns closely resemble EC at 512k, with concentrated expert-domain associations across layers. This confirms that ET’s population-level threshold mechanism captures the same routing structure as large-batch top-kselection, without requiring batch size coordination at inference. Comparing across domains, the HumanEval and GSM8K columns reveal that experts develop different specializa- tion patterns for code versus math. Certain experts that are heavily activated for code tokens (e.g., dark cells in the Hu- manEval column) show low activation for math, and vice versa. This cross-domain differentiation is consistent across all routing configurations, suggesting that expert special- ization reflects genuine domain-level structure rather than artifacts of a particular routing strategy. 19 Expert Threshold Routing EC (2k)ET Figure 15. Layerwise mean fanout versus loss bin for the two main runs. EC 2k shows a stronger positive dependence in several layers, while ET remains more mixed across depth. Figure 16. Standalone EC 8k activation dynamics in the same form as the main figure. Faint dashed gray curves show the per-layer means and the solid red curve shows the global mean across layers. Compared with EC 2k, EC 8k is noticeably flatter across the loss range. ET no warmupEC (64k)EC (512k) Figure 17. Inverse layerwise activation diagnostics for the remaining three runs. Each panel plots mean loss against fanout by layer, highlighting how the loss–fanout relation remains setup dependent across ET no warmup, EC 64k, and EC 512k. 20 Expert Threshold Routing Layer 1 expert 0Layer 11 expert 15 Figure 18. Router logit histograms from the warmup ET run. The bulk of the distribution is roughly bell shaped, with a heavier right tail. This asymmetric tail is consistent with activated tokens receiving reinforcing gradient signals that can further increase their logits. We view this figure as a qualitative appendix diagnostic rather than a core result. ·sold ·clips ·to · 48 ·of ·her ·friends ·in ·April , ·and ·then ·she ·sold ·half ·as ·many ·clips ·in ·May . ·How ·many ·clips ·did ·Natal ia ·sell ·altogether Token 1 2 3 4 5 6 7 8 9 10 11 Layer Variant A: Per-layer fanout GSM8K_0 0 1 2 4 8 16 Fanout (# experts) (a) GSM8K. def ·has _ close _ elements (n umbers : ·List [ f loat ], ·threshold : ·float ) ·-> ·b ool : · · ·" "" ·Check Token 1 2 3 4 5 6 7 8 9 10 11 Layer Variant A: Per-layer fanout HumanEval_0 0 1 2 4 8 16 Fanout (# experts) (b) HumanEval. Figure 19. Per-layer expert fanout on GSM8K and HumanEval passages. Each cell shows the number of experts activated for a given token at a given layer. Numerical and code-specific tokens receive substantially higher fanout than function words. 21 Expert Threshold Routing 0 7 15 L11 0 7 15 L10 0 7 15 L9 0 7 15 L8 0 7 15 L7 0 7 15 L6 0 7 15 L5 0 7 15 L4 0 7 15 L3 0 7 15 L2 ·sold ·clips ·to · 48 ·of ·her ·friends ·in ·April , ·and ·then ·she ·sold ·half ·as ·many ·clips ·in ·May . ·How ·many ·clips ·did ·Natal ia ·sell ·altogether Token 0 7 15 L1 Variant C: Full routing GSM8K_0 (a) GSM8K. 0 7 15 L11 0 7 15 L10 0 7 15 L9 0 7 15 L8 0 7 15 L7 0 7 15 L6 0 7 15 L5 0 7 15 L4 0 7 15 L3 0 7 15 L2 def ·has _ close _ elements (n umbers : ·List [ f loat ], ·threshold : ·float ) ·-> ·b ool : · · ·" "" ·Check Token 0 7 15 L1 Variant C: Full routing HumanEval_0 (b) HumanEval. Figure 20. Full expert routing on GSM8K and HumanEval passages. Each panel shows binary expert activation (black = activated) across all layers and experts for every token. Routing patterns reveal domain-specific structure. 22 Expert Threshold Routing <|bos|> from typing import List def has _ close _ elements(numbers: List[float], threshold: float) -> b ool: """ Check if in given list of numbers, are any two numbers closer to each other than given threshold. >>> has _ close _ elements([1.0, 2.0, 3.0], 0.5) False >>> has _ close _ elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3) True """ for idx, elem in enumerate(numbers): for idx2, elem2 in enumerate(numbers): if idx != idx2: distance = abs(elem - elem2) if distance < threshold: return True return False Token activation intensity HumanEval_0 01248163264176 Total fanout (sum across 11 layers) Figure 21. Token activation intensity on a HumanEval passage. Each token is colored by total fanout (sum of experts activated across all layers). Code-specific tokens (variable names, operators, keywords) receive higher activation than boilerplate text. 23 Expert Threshold Routing 1 2 3 4 5 6 7 8 9 10 11 ec_bsz2k Layer HumanEvalGSM8K 0.0 0.1 0.2 0.3 0.4 Expert Token Ratio 1 2 3 4 5 6 7 8 9 10 11 ec_bsz8k Layer 0.0 0.1 0.2 0.3 0.4 Expert Token Ratio 1 2 3 4 5 6 7 8 9 10 11 ec_bsz64k Layer 0.0 0.1 0.2 0.3 0.4 Expert Token Ratio 1 2 3 4 5 6 7 8 9 10 11 ec_bsz512k Layer 0.0 0.1 0.2 0.3 0.4 Expert Token Ratio Expert ID 1 2 3 4 5 6 7 8 9 10 11 gec_warmup Layer Expert ID 0.0 0.1 0.2 0.3 0.4 Expert Token Ratio Figure 22. Expert activation heatmaps across routing configurations. Each row corresponds to a routing variant (EC with batch sizes 2k, 8k, 64k, 512k, and ET). Columns show HumanEval (code) and GSM8K (math) domains. Color intensity indicates expert token ratio. Specialization sharpens with larger EC batch sizes, and ET achieves comparable patterns without batch size dependence. 24

Instant research discovery