Paper deep dive
Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
Kening Zheng, Wei-Chieh Huang, Jiahao Huo, Zhonghao Li, Henry Peng Zou, Yibo Yan, Xin Zou, Jungang Li, Junzhuo Li, Hanrong Zhang, Xuming Hu, Philip S. Yu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 4/10/2026, 2:15:14 AM
Summary
The paper introduces 'RISE' (Routing Isolation-guided Subnetwork Enhancement), a framework for multilingual Mixture-of-Experts (MoE) models. It identifies 'Language Routing Isolation'—where high- and low-resource languages activate disjoint expert sets—and a 'layer-wise convergence-divergence' pattern. RISE leverages these findings to select and train language-specific expert subnetworks in shallow/deep layers and universal experts in middle layers, significantly improving low-resource language performance while preserving cross-lingual capabilities.
Entities (5)
Relation Signals (3)
MoE models → exhibits → Language Routing Isolation
confidence 95% · we conduct a systematic analysis of expert routing patterns in MoE models, revealing a phenomenon we term Language Routing Isolation
RISE → improvesperformanceof → Low-resource languages
confidence 95% · RISE substantially improves low-resource language performance while preserving capabilities in other languages.
RISE → utilizes → Specificity scores
confidence 95% · RISE applies a tripartite selection strategy, using specificity scores to identify language-specific experts
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we conduct a systematic analysis of expert routing patterns in MoE models, revealing a phenomenon we term Language Routing Isolation, in which high- and low-resource languages tend to activate largely disjoint expert sets. Through layer-stratified analysis, we further show that routing patterns exhibit a layer-wise convergence-divergence pattern across model depth. Building on these findings, we propose RISE (Routing Isolation-guided Subnetwork Enhancement), a framework that exploits routing isolation to identify and adapt language-specific expert subnetworks. RISE applies a tripartite selection strategy, using specificity scores to identify language-specific experts in shallow and deep layers and overlap scores to select universal experts in middle layers. By training only the selected subnetwork while freezing all other parameters, RISE substantially improves low-resource language performance while preserving capabilities in other languages. Experiments on 10 languages demonstrate that RISE achieves target-language F1 gains of up to 10.85% with minimal cross-lingual degradation.
Tags
Links
- Source: https://arxiv.org/abs/2604.03592v1
- Canonical: https://arxiv.org/abs/2604.03592v1
Trouble viewing inline? Open PDF directly →
Full Text
92,235 characters extracted from source content.
Expand or collapse full text
[ BoldFont=texgyrepagella-bold.otf, ItalicFont=texgyrepagella-italic.otf, BoldItalicFont=texgyrepagella-bolditalic.otf ] _if_language:nTFENG =English Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation Kening Zheng1 Wei-Chieh Huang1 Jiahao Huo1 Zhonghao Li3 Henry Peng Zou1 Yibo Yan2 Xin Zou2 Jungang Li2 Junzhuo Li2 Hanrong Zhang1 Xuming Hu2 Philip S. Yu1 1University of Illinois Chicago 2HKUST (Guangzhou) 3University of Maryland Corresponding author. Abstract Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we conduct a systematic analysis of expert routing patterns in MoE models, revealing a phenomenon we term Language Routing Isolation, in which high- and low-resource languages tend to activate largely disjoint expert sets. Through layer-stratified analysis, we further show that routing patterns exhibit a layer-wise convergence–divergence pattern across model depth. Building on these findings, we propose RISE (Routing Isolation-guided Subnetwork Enhancement), a framework that exploits routing isolation to identify and adapt language-specific expert subnetworks. RISE applies a tripartite selection strategy, using specificity scores to identify language-specific experts in shallow and deep layers and overlap scores to select universal experts in middle layers. By training only the selected subnetwork while freezing all other parameters, RISE substantially improves low-resource language performance while preserving capabilities in other languages. Experiments on 10 languages demonstrate that RISE achieves target-language F1 gains of up to 10.85% with minimal cross-lingual degradation. _if_language:nTFENG =English1 Introduction The MoE paradigm has emerged as a powerful architectural approach in modern language modeling, demonstrating remarkable capabilities across a wide range of tasks, including reasoning, code generation, and multilingual understanding (Abdin et al., 2024; DeepSeek-AI et al., 2024b; a). By leveraging sparse expert activation, MoE models enable scaling to tens or hundreds of billions of parameters while maintaining practical inference efficiency, making them particularly attractive for resource-intensive applications. Despite their empirical success, MoE models exhibit significant performance disparities across languages (Chua et al., 2025). As shown in Table _if_language:nTFENG =English1, MoE models show accuracy gaps exceeding 80 percentage points on the MGSM benchmark between high-resource languages (e.g., English) and low-resource languages (e.g., Bengali and Swahili). Recent studies have explored expert mechanisms in multilingual MoE models. Bandarkar et al. (2026) analyzed layer-wise activation probability distributions to characterize routing patterns, while Chen et al. (2026) identified top experts based on total activation frequencies to understand expert specialization. However, these analyses focus either on layer-wise activation distributions or on global activation frequency, without clarifying the distinct functional roles that different model depths play in multilingual processing. To address this gap, we conduct a comprehensive analysis of multilingual routing behavior in MoE models from two complementary perspectives. At the global level, we uncover the routing isolation phenomenon, where high- and low-resource languages rely on nearly orthogonal sets of experts. Through layer-stratified analysis, we further reveal the layer-wise convergence–divergence phenomenon: routing patterns converge in middle layers (language-agnostic semantic processing) but diverge in shallow and deep layers (language-specific encoding and generation), shedding light on how MoE models process multilingual inputs across layers and revealing a principled structure amenable to multilingual adaptation. !20 Model BN DE EN ES FR JA RU SW TH ZH Avg. Phi-3.5-MoE-instruct 1 79 88.5 81.5 73 56 77 1 26.5 65.5 54.9 Qwen3-30B-A3B 46 88.5 96.5 91.5 82 83.5 92.5 48 87.5 86 80.3 _if_language:nTFENG =EnglishTable 1: Accuracy on MGSM across languages. Language codes: BN (Bengali), DE (German), EN (English), ES (Spanish), FR (French), JA (Japanese), RU (Russian), SW (Swahili), TH (Thai), ZH (Chinese). Building on these findings, we propose RISE (Routing Isolation-guided Subnetwork Enhancement), a method designed to improve low-resource language performance while preserving the model’s capabilities in high-resource languages. RISE adopts a hierarchical selection strategy to identify the expert subnetwork that predominantly supports the target language. Specifically, in shallow and deep layers, RISE uses a specificity score to identify experts that are particularly associated with the target low-resource language. In middle layers, where routing patterns are more shared across languages, RISE uses an overlap score to select universal experts. Extensive experiments show that RISE consistently improves target-language performance while preserving performance on other languages and tasks. In summary, this paper makes the following contributions: _if_language:nTFENG =English❶ Empirical finding. We first conduct a systematic analysis of multilingual routing behavior in MoE models from both global and layer-wise perspectives, through which we uncover two key phenomena: routing isolation and layer-wise convergence–divergence. These findings reveal a strong regularity in routing underlying multilingual processing in MoE models. _if_language:nTFENG =English❷ Methodological contribution. Building on these observations, we then propose RISE, a layer-stratified expert selection method that identifies language-specific subnetworks for low-resource language adaptation by selecting distinctive experts in shallow and deep layers while preserving shared experts in middle layers. _if_language:nTFENG =English❸ Empirical validation. We conduct comprehensive experiments across 10 languages using multiple datasets, validating the effectiveness and efficiency of RISE. _if_language:nTFENG =English2 Related Work Expert Specialization and Routing Analysis. Understanding how experts specialize and how routing mechanisms function has been an active area of research. Early studies focused on analyzing expert utilization patterns and their correlation with input characteristics (Shazeer et al., 2017). More recent work has explored the interpretability of expert behaviors in multilingual settings. Bandarkar et al. (2026) analyzed layer-wise activation probability distributions to characterize how different languages invoke different routing patterns in MoE models. Chen et al. (2026) identified top experts based on total activation frequencies and proposed layerwise steering techniques to understand and control expert specialization across languages. However, these approaches either examine routing from a global frequency perspective or focus on layer-wise distributions independently, without jointly considering the functional roles of different model depths. In contrast, our work reveals a systematic routing isolation phenomenon between high- and low-resource languages, and further reveals a layer-wise convergence–divergence pattern aligned with the functional stages of the model. _if_language:nTFENG =English3 Routing Analysis To understand the multilingual capability disparities in MoE models, we analyze expert routing patterns from two complementary perspectives: global-level activation statistics and layer-wise routing dynamics. These two views operate at different granularities and are complementary rather than contradictory: routing isolation is defined over the globally aggregated top-K routed support across all layers, whereas the layer-wise analysis decomposes similarity at each depth relative to a high-resource reference language. _if_language:nTFENG =English3.1 Routing Isolation at Global Level For a given language l, let cl,jc_l,j denote the total activation count of expert j aggregated across all layers and samples. We select the top-K most frequently activated experts globally: ℰlglobal=Top-K(j:j∈[1,Ne×L],sorted by cl,j)E_l^global=Top-K (\j:j∈[1,N_e× L]\,sorted by c_l,j ) _if_language:nTFENG =English(1) where NeN_e is the number of experts per layer, L is the total number of MoE layers, and K is the total number of experts considered in the statistics. To quantify the routing overlap between languages, we compute the Jaccard similarity between their top-K expert sets: J(l1,l2)=|ℰl1global∩ℰl2global||ℰl1global∪ℰl2global|J(l_1,l_2)= |E_l_1^global _l_2^global||E_l_1^global _l_2^global| _if_language:nTFENG =English(2) where J(l1,l2)∈[0,1]J(l_1,l_2)∈[0,1], with 0 indicating no overlap and 1 indicating complete overlap. We partition languages into high-resource (English, Chinese) and low-resource (Bengali, Swahili, Thai, etc.) groups based on their proportions in the training corpus. Figure _if_language:nTFENG =English1 (a) shows the global expert activation overlap across languages. We provide more examples of routing isolation in Appendix _if_language:nTFENG =EnglishE. _if_language:nTFENG =English(a) Global-level expert activation overlap across languages in MGSM. _if_language:nTFENG =English(b) Layer-wise expert routing similarity with English. _if_language:nTFENG =EnglishFigure 1: Comprehensive routing analysis of _if_language:nTFENG =EnglishQwen3-30B-A3B: global-level (left) expert activation overlap and layer-wise (right) routing similarity with English. Key observations from the global analysis point to a clear phenomenon between high- and low-resource languages: ❶ As high-resource languages, English and Chinese basically don’t share experts and rotate among many experts with little overlap in their Top-30 choices, reflecting flexible, well-trained routing; ❷ Low-resource languages repeatedly trigger the same small subset, implying more fixed patterns likely from limited training; ❸ overlap across the two resource groups is negligible, showing their expert sets are almost orthogonal, here we call this routing isolation. Besides this, we also found that languages from the same family (e.g., Indo-European) exhibit greater similarity in expert selection patterns, consistent with previous linguistic studies (Tang et al., 2024; Chen et al., 2026), thereby grounding our analysis in established theoretical frameworks. _if_language:nTFENG =English3.2 Layer-wise Convergence–Divergence Pattern To understand how expert routing patterns evolve across different model depths, we analyze the layer-wise expert overlap with respect to English as the reference language. For each layer i within language l, we compute the Jaccard similarity with English: Jl(i)=|ℰl(i)∩ℰen(i)||ℰl(i)∪ℰen(i)|,l=[Jl(0),Jl(1),…,Jl(L−1)]J_l(i)\;=\; |E_l^(i) _en^(i) | |E_l^(i) _en^(i) |, _l\;=\; [\,J_l(0),\;J_l(1),\;…,\;J_l(L-1)\, ] _if_language:nTFENG =English(3) This yields a per-layer similarity curve lJ_l that captures routing dynamics across layers. The average similarity within each region is: J¯lℒ=1|ℒ|∑i∈ℒJl(i),ℒ∈ℒshallow,ℒmiddle,ℒdeep J_l^\,L\;=\; 1|L| _i J_l(i), ∈ \L_shallow,\;L_middle,\;L_deep \ _if_language:nTFENG =English(4) We plot the similarity curve lJ_l and compute the average similarity J¯lℒ J_l^L for each region, as shown in Figure _if_language:nTFENG =English1 (b) and Table _if_language:nTFENG =English2, respectively. Language Shallow Middle Deep Bengali (BN) 0.12 0.22 0.05 Swahili (SW) 0.04 0.19 0.04 Thai (TH) 0.25 0.33 0.12 Russian (RU) 0.41 0.43 0.17 Japanese (JA) 0.37 0.37 0.10 _if_language:nTFENG =EnglishTable 2: Average layer-wise expert overlap with English across different languages. Additional analysis results are provided in Table _if_language:nTFENG =English7- _if_language:nTFENG =English10 in Appendix _if_language:nTFENG =EnglishE. As shown in Table _if_language:nTFENG =English2, routing similarity with English is consistently lower in the shallow and deep layers, while the middle layers exhibit noticeably higher overlap across all languages. This pattern suggests that the functional stratification observed in dense transformers is also reflected in MoE routing behavior. Specifically, the shallow and deep layers tend to capture more language-specific processing, whereas the middle layers appear to support more shared cross-lingual representations. These findings provide a strong empirical basis for our layer-aware expert selection strategy in RISE. _if_language:nTFENG =English4 Method Building on the routing isolation identified in Section _if_language:nTFENG =English3, which suggests that MoE models contain language-specific expert subnetworks, we present a systematic approach to identify and train these subnetworks for low-resource languages, thereby improving performance on target languages while minimally affecting other capabilities. Our method consists of three main stages: (1) collecting routing statistics to understand language-expert affinities, (2) selecting language-specific and shared experts based on layer-aware analysis, and (3) training the selected subnetwork while keeping the rest of the model frozen. We detail each stage below. Figure _if_language:nTFENG =English2 illustrates the overall pipeline of our approach. _if_language:nTFENG =English4.1 Routing Statistics Collection The first stage of RISE involves analyzing the routing behavior of the pre-trained model across multiple languages. This analysis reveals which experts are preferentially activated for different languages, providing the foundation for our selection strategy. Data Preparation. We collect evaluation datasets in M languages Λ=λ1,λ2,…,λM =\ _1, _2,…, _M\, including the target language λ∗λ^*. These datasets should cover the same or similar tasks to ensure comparable routing patterns. For each language λ∈Λλ∈ , we run inference on the corresponding dataset and record the discrete routing decisions for all MoE layers. Specifically, for layer l and language λ, we define a binary activation indicator gt,i(l)g^(l)_t,i for each token t and expert i, and aggregate it into an activation frequency aλ,i(l)a^(l)_λ,i over all TλT_λ tokens: aλ,i(l)=1Tλ∑t=1Tλgt,i(l)∈[0,1],gt,i(l)=1,if expert i is activated for token t at layer l0,otherwisea^(l)_λ,i\;=\; 1T_λ _t=1^T_λg^(l)_t,i\;∈\;[0,1], g^(l)_t,i\;=\; cases1,&if expert $i$ is activated for token $t$ at layer $l$\\ 0,&otherwise cases _if_language:nTFENG =English(5) where TλT_λ is the total number of tokens processed for language λ, and aλ,i(l)a^(l)_λ,i represents the empirical probability that expert i is activated when processing language λ. This yields a routing profile matrix (l)∈ℝM×NA^(l) ^M× N for each layer, where each row corresponds to a language and each column to an expert. These matrices capture language–expert affinity patterns that inform our selection strategy. _if_language:nTFENG =English4.2 Layer-Aware Expert Selection _if_language:nTFENG =EnglishFigure 2: Overview of RISE. (a) We first collect routing statistics across multiple languages. (b) Based on layer-aware analysis, we select language-specific experts in shallow/deep layers and cross-lingual shared experts in middle layers. (c) Only the selected experts are trained while keeping all other parameters frozen. Based on the analysis in Section _if_language:nTFENG =English3, we partition the L MoE layers into three groups: ℒshallow=[0,L1],ℒmiddle=(L1,L2],ℒdeep=(L2,L−1]L_shallow=[0,L_1], _middle=(L_1,L_2], _deep=(L_2,L-1] _if_language:nTFENG =English(6) Shallow and deep layers. For shallow and deep layers, we seek experts that are preferentially activated by the target language λ∗λ^*. We quantify this preference using the differential score Sλ∗,i(l)=aλ∗,i(l)/a¯i(l)S^(l)_λ^*,i=a^(l)_λ^*,i/ a^(l)_i, where a¯i(l) a^(l)_i is the average activation probabilities of corresponding expert i in all languages, serving as a baseline for comparison. The specificity score compares the activation probability of expert i on the target language λ∗λ^* to its average activation probability across all languages. When Sλ∗,i(l)S^(l)_λ^*,i is greater than 11, expert i is preferentially activated by λ∗λ^*, indicating language-specific behavior; when Sλ∗,i(l)S^(l)_λ^*,i is close to 11, the expert is activated at a similar rate across languages and thus behaves in a language-agnostic manner; and when Sλ∗,i(l)S^(l)_λ^*,i is less than 11, the expert is more preferred by other languages. Middle layers. For middle layers, we seek experts that capture language-agnostic representations shared across all languages. We measure this property using the overlap score based on the coefficient of variation: Oi(l)=11+cv^i(l)wherecv^i(l)=σ(a⋅,i(l))μ(a⋅,i(l))O^(l)_i= 11+ c_ v^(l)_i c_ v^(l)_i= σ(a^(l)_·,i)μ(a^(l)_·,i) _if_language:nTFENG =English(7) Here, cv c_ v represents the coefficient of variation, while σ(⋅)σ(·) and μ(⋅)μ(·) denote the standard deviation and mean of expert i’s activation probabilities across all languages. The coefficient of variation measures relative dispersion: _if_language:nTFENG =English• Low cv c_ v (high Oi(l)O^(l)_i): Uniform activation across languages → shared expert. _if_language:nTFENG =English• High cv c_ v (low Oi(l)O^(l)_i): Activation varies significantly → language-specific expert. Subnetwork Selection. The specificity and overlap scores measure relative preferences but do not account for the absolute activation magnitude. An expert with high specificity but very low overall activation may be less important than one with moderate specificity but high activation. To address this, we define composite scores that incorporate both factors: Spec(l,i,λ∗)=Sλ∗,i(l)⋅(1+α⋅aλ∗,i(l)),Ovlp(l,i)=Oi(l)⋅(1+α⋅a¯i(l))Spec(l,i,λ^*)=S^(l)_λ^*,i· (1+α· a^(l)_λ^*,i ), (l,i)=O^(l)_i· (1+α· a^(l)_i ) _if_language:nTFENG =English(8) where α>0α>0 is a hyperparameter controlling the importance of absolute activation magnitude. In our experiments, we set α=10α=10. Given a total budget of K experts to train, we allocate them across layer groups using predefined ratios (ρs,ρm,ρd)( _s, _m, _d) satisfying ρs+ρm+ρd=1 _s+ _m+ _d=1. The complete selection procedure is detailed in Algorithm _if_language:nTFENG =English1. With the selected expert set ℰ=(l1,i1),…,(lK,iK)E=\(l_1,i_1),…,(l_K,i_K)\, we proceed to train only these experts while keeping all other parameters frozen. We use standard causal language modeling loss on the target language data λ∗D_λ^* and our loss function is defined as: ℒ(Θtrain)=−x∼λ∗[∑t=1|x|logPΘ(xt∣x<t)]L( _train)=-E_x _λ^* [ _t=1^|x| P_ (x_t x_<t) ] _if_language:nTFENG =English(9) where only Θtrain _train receives gradient updates. We detail the complete procedure in Algorithm _if_language:nTFENG =English1 and visualize the distribution of selected experts in Appendix _if_language:nTFENG =EnglishF. _if_language:nTFENG =English5 Experiments In this section, we conduct extensive experiments to answer the following research questions: (RQ1) Can RISE improve model performance on the target low-resource language? (RQ2) Does RISE preserve performance on non-target languages and more comprehensive tasks? (RQ3) Do the experts identified by RISE predominantly specialize in the target language? (RQ4) Is each layer-group component of RISE’s expert subnet indispensable? (RQ5) How sensitive is RISE to its key hyperparameters? _if_language:nTFENG =English5.1 Settings Datasets We evaluate on multilingual benchmarks: TyDiQA-GoldP (Clark et al., 2020), an extractive QA dataset covering 9 languages (AR, BN, EN, FI, ID, KO, RU, SW, TE); and MGSM (Shi et al., 2022), a math reasoning dataset covering 10 languages (BN, DE, EN, ES, FR, JA, RU, SW, TH, ZH). To verify that RISE does not degrade general capabilities, we further evaluate on TriviaQA, MMLU, HellaSwag, and ARC. Comparison. To demonstrate the effectiveness of RISE, we compare it against the following settings: randomly selecting an equivalent number of experts; selecting only the top-K most-activated experts for each language, and varying the expert budget and target language under the RISE framework. We also include ESFT (Wang et al., 2024) as an expert-selection baseline. Regarding parameter-efficient methods such as LoRA, we further include LoRA as a representative PEFT baseline in Table _if_language:nTFENG =English6. We note that RISE operates at the level of which experts to train, while LoRA addresses how parameters are updated — the two are orthogonal and can be combined; a fuller discussion is deferred to Appendix _if_language:nTFENG =EnglishH. Implementation Details. All experiments were conducted on a single NVIDIA H200 GPU. For GPU memory cost information, we list the details in Table _if_language:nTFENG =English12. Training was performed for 3 epochs with a per-device batch size of 2 and gradient accumulation of 8 steps (effective batch size of 16), using a learning rate of 2×10−52× 10^-5 and _if_language:nTFENG =Englishbfloat16 mixed precision. Further details on the training budget, expert allocation, and layer grouping are provided in Table _if_language:nTFENG =English11 in Appendix _if_language:nTFENG =EnglishG. All hyperparameter settings are held fixed across different target languages and backbone models to ensure a fair comparison. _if_language:nTFENG =English5.2 Performance and Cross-lingual Preservation (RQ1 & RQ2) !20 Setting AR BN EN FI ID KO RU SW TE Avg. Qwen3-30B-A3B (TyDiQA-GoldP, F1 %) Vanilla 49.4949.49 51.5151.51 24.1524.15 22.9822.98 26.8626.86 48.3548.35 27.0427.04 27.0527.05 58.6158.61 37.3437.34 !10 Random (64, BN) 48.9148.91 52.17 23.8723.87 22.3122.31 25.6125.61 48.1948.19 25.6325.63 27.7327.73 59.0359.03 37.0537.05 Random (128, BN) 50.3150.31 53.52 24.4324.43 22.6922.69 26.4926.49 48.3148.31 26.0826.08 27.2227.22 59.7159.71 37.6437.64 !10 TopK (128, BN) 50.2050.20 52.74 24.7524.75 23.1323.13 27.8627.86 47.5147.51 26.9426.94 26.8526.85 59.1159.11 37.6837.68 ESFT (Wang et al., 2024) 49.5149.51 51.79 24.0824.08 23.2623.26 26.8926.89 47.3347.33 27.0927.09 25.8525.85 59.7059.70 37.2837.28 !10 RISE (128, BN) 49.1549.15 54.23 24.0424.04 23.3923.39 27.6127.61 47.7747.77 27.2227.22 27.4027.40 60.3960.39 37.9137.91 RISE (128, RU) 51.0651.06 53.6253.62 23.8823.88 24.1524.15 29.5529.55 49.0849.08 29.62 29.3029.30 60.6660.66 38.9938.99 !10 RISE (128, ID) 51.5951.59 53.9753.97 25.6525.65 23.5823.58 31.79 47.7547.75 29.5429.54 30.1330.13 59.9559.95 39.3339.33 Phi-3.5-MoE-Instruct (TyDiQA-GoldP, F1 %) Vanilla 36.7536.75 36.0436.04 16.9416.94 19.1919.19 22.0922.09 19.7519.75 16.1916.19 11.5511.55 7.687.68 20.6920.69 !10 Random (16, BN) 38.2638.26 41.93 18.9018.90 19.9019.90 23.2323.23 26.8426.84 18.7518.75 11.6411.64 9.349.34 23.2023.20 Random (32, BN) 37.5337.53 43.61 18.8918.89 20.1020.10 23.2823.28 24.5624.56 18.4518.45 11.4411.44 8.938.93 22.9822.98 !10 TopK (16, BN) 39.6339.63 49.51 18.3218.32 20.7820.78 23.5923.59 34.5834.58 18.9118.91 11.5611.56 10.7810.78 25.3025.30 ESFT (Wang et al., 2024) 38.2938.29 45.44 18.1018.10 20.1220.12 23.1523.15 28.8428.84 19.1019.10 12.2412.24 10.0910.09 23.9323.93 !10 RISE (16, BN) 37.4337.43 46.89 17.5517.55 19.6419.64 22.7522.75 21.1021.10 17.2617.26 11.5611.56 9.689.68 22.6522.65 RISE (16, RU) 37.6837.68 34.8334.83 17.6317.63 20.8520.85 22.8522.85 21.9821.98 17.81 10.7010.70 8.328.32 21.4121.41 !10 RISE (16, ID) 39.4339.43 36.8336.83 19.0019.00 21.5121.51 22.33 24.5024.50 20.7820.78 11.0311.03 8.498.49 22.6622.66 _if_language:nTFENG =EnglishTable 3: TyDiQA-GoldP multilingual QA performance (F1, %). Each row corresponds to a training setting; notation (budget, lang) denotes the number of experts selected by RISE and the target language, e.g., (128, BN) selects 128 experts routed by Bengali and trains only those experts on Bengali data. The colored bold value indicates the target language column for each training setting (blue==BN, orange==RU, green==ID); the language code in the first column is colored accordingly. To answer RQ1 and RQ2, we comprehensively compare RISE against other methods across two multilingual MoE models. Table _if_language:nTFENG =English3 presents the main results. We report the following observations. Obs. ❶ RISE consistently improves target-language performance. As shown in Table _if_language:nTFENG =English3, RISE achieves the best Bengali F1 of 54.23% on _if_language:nTFENG =EnglishQwen3-30B-A3B, outperforming all competing methods. On _if_language:nTFENG =EnglishPhi-3.5-MoE-Instruct, RISE similarly improves Bengali F1 by 10.85% over the Vanilla baseline, confirming that routing-based expert selection reliably identifies the parameters most critical for the target language. The gains are consistent across multiple target languages: RISE (128, RU) and RISE (128, ID) yield the best per-language F1 on Russian (29.62%) and Indonesian (31.79%), respectively, demonstrating that the method generalizes beyond a single low-resource setting. !20 Setting TriviaQA MMLU HellaSwag ARC Avg. Qwen 59.60 74.40 81.50 85.76 75.32 !10 RISE 59.00 74.65 81.50 86.78 75.48 Δ −-0.50 ++0.26 0.00 ++0.68 ++0.11 !10 Phi 65.50 77.50 70.70 90.85 76.14 RISE 65.70 76.67 70.60 91.53 76.13 !10 Δ ++0.90 −-0.53 −-0.10 ++0.68 ++0.24 _if_language:nTFENG =EnglishTable 4: General ability preservation before and after RISE training. TriviaQA (world knowledge), MMLU (academic reasoning), HellaSwag (commonsense inference), ARC (science QA). Obs. ❷ RISE delivers targeted improvements without cross-lingual or cross-task degradation. Table _if_language:nTFENG =English6 in Appendix _if_language:nTFENG =EnglishD shows that RISE trained exclusively on TyDiQA incurs virtually no loss on the held-out MGSM benchmark: RISE (128, BN) achieves an average accuracy of 80.7%, marginally above the Vanilla baseline (80.2%), confirming that the selected subnetworks capture language-specific rather than task-general computation. LoRA, by contrast, induces severe cross-lingual interference: on _if_language:nTFENG =EnglishPhi-3.5-MoE-Instruct, Thai accuracy collapses by 16.5 percentage points (26.5%→ 10.0%), and on _if_language:nTFENG =EnglishQwen3-30B-A3B, Swahili drops by 7.0 points (48.0%→ 41.0%). General capability is equally well preserved: Table _if_language:nTFENG =English4 shows that all four benchmark scores shift by less than 1% after RISE training, indicating that selective expert updates leave shared, task-agnostic computation intact. _if_language:nTFENG =English5.3 Causal Expert Verification (RQ3) Obs. ❸ The RISE-selected experts are causally responsible for target-language computation. To verify that the identified subnet is genuinely language-specific, we pruned the experts belonging to the selected expert subnetwork in both backbone models and re-evaluated them on TyDiQA-GoldP, as shown in Table _if_language:nTFENG =English5. The target language suffers catastrophic collapse on Bengali, while other languages decline only moderately, mirroring the selectivity observed during training. This bidirectional evidence, improvement upon activation and collapse upon ablation, confirms that RISE isolates the true language-specific subnetwork rather than a task-general or spuriously correlated subset of parameters. !20 Setting AR BN EN FI ID KO RU SW TE Qwen3-30B-A3B Vanilla 49.49 51.51 24.15 22.98 26.86 48.35 27.04 27.05 58.61 !10 RISE (128, BN) 49.15 54.23 24.04 23.39 27.61 47.77 27.22 27.40 60.39 Pruned (128, BN) 26.83↓ 22.32 1.00↓ 53.23 24.63↑ 0.59 19.37↓ 4.02 30.99↑ 3.38 12.47↓ 35.30 17.87↓ 9.35 21.09↓ 6.31 7.78↓ 52.61 Phi-3.5-MoE-Instruct !10 Vanilla 36.75 36.04 16.94 19.19 22.09 19.75 16.19 11.55 7.68 RISE (16, BN) 37.43 46.89 17.55 19.64 22.75 21.10 17.26 11.56 9.68 !10 Pruned (16, BN) 12.03↓ 25.40 1.85↓ 45.04 16.53↓ 1.02 19.40↓ 0.24 18.79↓ 3.96 16.63↓ 4.47 7.85↓ 9.41 3.46↓ 8.10 1.02↓ 8.66 _if_language:nTFENG =EnglishTable 5: The performance comparison between RISE and its pruned version. _if_language:nTFENG =English5.4 Indispensability of Layer-Group Components (RQ4) _if_language:nTFENG =EnglishFigure 3: Grouped comparison of layer-wise expert subset combinations. w/o means removing the corresponding layer group; Only: retaining only the corresponding layer group. To answer RQ4, we conduct an ablation study on the contribution of each layer group by training variants of RISE that omit or retain only one group at a time. Results are reported in Figure _if_language:nTFENG =English3. Obs. ❹ All three layer groups contribute to RISE, with shallow and deep layers being most critical for target-language adaptation. Removing the shallow or deep layer experts causes the biggest TyDiQA-BN performance drops, whereas w/o Middle yields the smallest degradation. The w/o Middle result further corroborates the routing analysis in Section _if_language:nTFENG =English3: middle layers exhibit language-agnostic, convergent routing and are therefore less critical for language-specific adaptation. By contrast, experts selected in shallow and deep layers capture language-specific syntactic and semantic patterns that are essential for low-resource performance. These results confirm that the layer-aware design of RISE is non-redundant: each group plays a distinct and necessary role, and no single group alone suffices to match the full method. _if_language:nTFENG =English5.5 Sensitivity Analysis (RQ5) To answer RQ5, we examine the sensitivity of RISE to three key hyperparameters: the activation scale factor α, the layer-group budget allocation ratio and the total number of selected training experts K. Results are presented in Figure _if_language:nTFENG =English4. _if_language:nTFENG =English(a) Ablation on activation scale factor α. _if_language:nTFENG =English(b) Ablation on expert budget allocation ratio. _if_language:nTFENG =English(c) Ablation on number of training experts K. _if_language:nTFENG =EnglishFigure 4: Ablation studies for hyperparameters. (a) Effect of the activation scale factor α in the composite expert selection. (b) Effect of the expert budget allocation ratio across shallow, middle, and deep layers. (c) Effect of the total number of selected experts K. Obs. ❺ RISE is robust to hyperparameter choices within a reasonable operating range. As shown in Figure _if_language:nTFENG =English4(a), performance is stable across a broad range of α values, with a modest optimum that reflects the intended balance between activation frequency and routing weight in the composite selection score. Figure _if_language:nTFENG =English4(b) shows that concentrating the budget predominantly on shallow and deep layers yields the best results while a uniform allocation underperforms, consistent with the findings in RQ4. Figure _if_language:nTFENG =English4(c) demonstrates that performance improves as K increases up to 128, beyond which returns diminish. _if_language:nTFENG =English6 Conclusion In this work, we investigate how MoE language models process multilingual inputs and uncover a striking phenomenon of language routing isolation, where MoE models naturally form distinct routing pathways for high-resource versus low-resource languages. Building on this mechanistic insight, we propose RISE, an interpretable framework that identifies target-adaptive expert subnetworks through layer-aware selection, combining language specificity in shallow/deep layers with cross-lingual overlap in middle layers. Through selective training of these identified experts while freezing all other parameters, our method substantially improves low-resource language performance with minimal impact on other tasks. Extensive experiments on TyDiQA and MGSM benchmarks validate that RISE successfully adapts MoE models while avoiding catastrophic forgetting. Our work demonstrates that mechanistic interpretability of routing patterns is not merely an analytical tool, but a practical foundation for efficient and controllable multilingual adaptation. As MoE architectures continue to scale, understanding their internal routing topology will become increasingly crucial for building equitable multilingual systems. References Abdin et al. (2024) Marah Abdin et al. Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404.14219, 2024. Bandarkar et al. (2026) Lucas Bandarkar, Chenyuan Yang, Mohsen Fayyaz, Junlin Hu, and Nanyun Peng. Multilingual routing in mixture-of-experts. In International Conference on Learning Representations (ICLR) 2026, 2026. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=ZoZR0x7tTD. Accepted paper / OpenReview submission. Bertolissi et al. (2025) Ryo Bertolissi, Jonas Hübotter, Ido Hakimi, and Andreas Krause. Local mixtures of experts: Essentially free test-time training via model merging. In Conference on Language Modeling (COLM), 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=X2RXpFA6Vh. Boizard et al. (2025) Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte Miguel Alves, Andre Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, João Alves, Kevin El Haddad, Manuel Faysse, Maxime Peyrard, Nuno M Guerreiro, Patrick Fernandes, Ricardo Rei, and Pierre Colombo. EuroBERT: Scaling multilingual encoders for european languages. In Conference on Language Modeling (COLM), 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=jdOC24msVq. Chen et al. (2026) Yuxin Chen, Zhengzhou Cai, Xiangtian Ji, Weixiang Zhao, An Zhang, Xiang Wang, and Tat-Seng Chua. Understanding multilingualism in mixture-of-experts llms: Routing mechanism, expert specialization, and layerwise steering, 2026. URL _if_language:nTFENG =Englishhttps://arxiv.org/abs/2601.14050. Cho & Hockenmaier (2025) Ikhyun Cho and Julia Hockenmaier. Analyzing multilingualism in large language models with sparse autoencoders. In Second Conference on Language Modeling, 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=NmGSvZoU3K. Chua et al. (2025) Lynn Chua, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chulin Xie, and Chiyuan Zhang. Crosslingual capabilities and knowledge barriers in multilingual large language models, 2025. URL _if_language:nTFENG =Englishhttps://arxiv.org/abs/2406.16135. Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages, 2020. URL _if_language:nTFENG =Englishhttps://arxiv.org/abs/2003.05002. DeepSeek-AI et al. (2024a) DeepSeek-AI et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, abs/2405.04434, 2024a. 2024a. DeepSeek-AI et al. (2024b) DeepSeek-AI et al. Deepseek-v3 technical report. CoRR, abs/2412.19437, 2024b. 2024b. Deng et al. (2026) Guanzhi Deng, Bo Li, Ronghao Chen, Huacan Wang, Lijie Wen, and Linqi Song. Dr-lora: Dynamic rank lora for mixture-of-experts adaptation, 2026. URL _if_language:nTFENG =Englishhttps://arxiv.org/abs/2601.04823. Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. URL _if_language:nTFENG =Englishhttps://arxiv.org/abs/2101.03961. Fleshman & Durme (2025) William Fleshman and Benjamin Van Durme. SpectR: Dynamically composing LM experts with spectral routing. In Conference on Language Modeling (COLM), 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=tK8GHR62EX. Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL _if_language:nTFENG =Englishhttps://arxiv.org/abs/2106.09685. Kim et al. (2025) Yekyung Kim, Jenna Russell, Marzena Karpinska, and Mohit Iyyer. One ruler to measure them all: Benchmarking multilingual long-context language models. In Conference on Language Modeling (COLM), 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=3vxxB3Ar9r. Kumar et al. (2025) Priyanshu Kumar, Devansh Jain, Akhila Yerukola, Liwei Jiang, Himanshu Beniwal, Thomas Hartvigsen, and Maarten Sap. PolyGuard: A multilingual safety moderation tool for 17 languages. In Conference on Language Modeling (COLM), 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=wbAWKXNeQ4. Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding, 2020. URL _if_language:nTFENG =Englishhttps://arxiv.org/abs/2006.16668. Li et al. (2025a) Junzhuo Li, Bo Wang, Xiuze Zhou, and Xuming Hu. Dynamic expert specialization: Towards catastrophic forgetting-free multi-domain MoE adaptation. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 18478–18493, Suzhou, China, November 2025a. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: _if_language:nTFENG =English10.18653/v1/2025.emnlp-main.932. URL _if_language:nTFENG =Englishhttps://aclanthology.org/2025.emnlp-main.932/. Li et al. (2025b) Zhongyang Li, Ziyue Li, and Tianyi Zhou. C3PO: Critical-layer, core-expert, collaborative pathway optimization for test-time expert re-mixing. In Conference on Language Modeling (COLM), 2025b. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=lqC5J7pBP9. Li et al. (2025c) Zichong Li, Chen Liang, Zixuan Zhang, Ilgee Hong, Young Jin Kim, Weizhu Chen, and Tuo Zhao. SlimMoE: Structured compression of large MoE models via expert slimming and distillation. In Conference on Language Modeling (COLM), 2025c. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=oaCUsn391F. Penedo et al. (2025) Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. FineWeb2: One pipeline to scale them all — adapting pre-training data processing to every language. In Conference on Language Modeling (COLM), 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=jnRBe6zatP. Pombal et al. (2025) José Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig, and André Martins. M-Prometheus: A suite of open multilingual LLM judges. In Conference on Language Modeling (COLM), 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=Atyk8lnIQQ. Ramos et al. (2025) Miguel Moura Ramos, Patrick Fernandes, Sweta Agrawal, and Andre Martins. Multilingual contextualization of large language models for document-level machine translation. In Conference on Language Modeling (COLM), 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=Ah0U1r5Ldq. Schmidt et al. (2025) Fabian David Schmidt, Ivan Vulić, Goran Glavaš, and David Ifeoluwa Adelani. Fleurs-SLU: A massively multilingual benchmark for spoken language understanding. In Conference on Language Modeling (COLM), 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=Tqj3fYqhwS. Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. URL _if_language:nTFENG =Englishhttps://arxiv.org/abs/1701.06538. Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners, 2022. URL _if_language:nTFENG =Englishhttps://arxiv.org/abs/2210.03057. Song et al. (2025) Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Yuxuan Li, Zhiyuan Liu, and Maosong Sun. BlockFFN: Towards end-side acceleration-friendly mixture-of-experts with chunk-level activation sparsity. In Conference on Language Modeling (COLM), 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=uLl7tSUOir. Tang et al. (2024) Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. Language-specific neurons: The key to multilingual capabilities in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5701–5715, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: _if_language:nTFENG =English10.18653/v1/2024.acl-long.309. URL _if_language:nTFENG =Englishhttps://aclanthology.org/2024.acl-long.309/. Wang et al. (2024) Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, and Y. Wu. Let the expert stick to his last: Expert-specialized fine-tuning for sparse architectural large language models, 2024. URL _if_language:nTFENG =Englishhttps://arxiv.org/abs/2407.01906. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL _if_language:nTFENG =Englishhttps://arxiv.org/abs/2505.09388. Yu et al. (2025) Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel F Campos. Arctic-Embed 2.0: Multilingual retrieval without compromise. In Conference on Language Modeling (COLM), 2025. URL _if_language:nTFENG =Englishhttps://openreview.net/forum?id=H6so82c2Sw. Appendix _if_language:nTFENG =EnglishAppendix A More Related Work Multilingual Language Models As large language models have gained increasing prominence, a growing body of research has begun to focus on both the mechanistic interpretability and practical applications of multilingual language models. Cho & Hockenmaier (2025) applied sparse autoencoders to dissect multilingual processing in LLMs, revealing how individual features encode language-specific information. On the data side, Penedo et al. (2025) introduced FineWeb2, a scalable pre-training data pipeline adapted to every language, enabling the development of high-quality multilingual corpora at scale. For multilingual encoders, Boizard et al. (2025) proposed EuroBERT, scaling multilingual encoder models across European languages, while Yu et al. (2025) presented Arctic-Embed 2.0, achieving strong multilingual retrieval without sacrificing cross-lingual transfer. Benchmarking efforts have also expanded in scope: Kim et al. (2025) introduced a benchmark for multilingual long-context language models, and Schmidt et al. (2025) proposed Fleurs-SLU, a massively multilingual benchmark for spoken language understanding. On the evaluation and judgment side, Pombal et al. (2025) developed M-Prometheus, a suite of open multilingual LLM judges supporting diverse languages. For safety, Kumar et al. (2025) introduced PolyGuard, a multilingual moderation tool covering 17 languages. Regarding translation, Ramos et al. (2025) studied multilingual contextualization of LLMs for document-level machine translation. Collectively, these works highlight the growing importance of multilingual considerations across all stages of language model development, providing important context for our analysis of multilingual routing behavior in MoE models. Mixture-of-Experts (MoE) Mixture-of-Experts (MoE) architectures have been widely studied as a means to scale neural networks while maintaining computational efficiency (Shazeer et al., 2017; Lepikhin et al., 2020; Fedus et al., 2022). By activating only a subset of experts for each input token, MoE models can achieve high parameter counts without incurring proportional increases in inference cost. This design has been particularly influential in the development of large language models (LLMs), where MoE variants have demonstrated strong performance across various natural language processing tasks (DeepSeek-AI et al., 2024a; Abdin et al., 2024; Yang et al., 2025). A key component of MoE models is the routing mechanism, which determines how input tokens are assigned to different experts. Several routing strategies have been proposed, including top-k gating (Shazeer et al., 2017), learned routing networks (Lepikhin et al., 2020), and dynamic routing based on input features (Li et al., 2025a). Beyond standard routing, recent work has explored more flexible expert composition strategies: Fleshman & Durme (2025) proposed SpectR, which dynamically composes experts using spectral routing to improve model expressivity, while Li et al. (2025b) introduced C3PO, a test-time expert re-mixing approach that optimizes expert pathways at critical layers without retraining. Complementary efforts have focused on efficiency and compression: Li et al. (2025c) proposed SlimMoE to compress large MoE models via expert slimming and distillation, Song et al. (2025) introduced BlockFFN with chunk-level activation sparsity to accelerate inference on end-side devices, and Bertolissi et al. (2025) showed that local mixtures of experts can be constructed via model merging to enable essentially free test-time training. _if_language:nTFENG =EnglishAppendix B The Algorithm of RISE _if_language:nTFENG =EnglishAlgorithm 1 Language-specific Expert Subnetwork Selection _if_language:nTFENG =English0: Routing profiles (l)l=0L−1\A^(l)\_l=0^L-1, target language λ∗λ^*, budget K, layer boundaries (L1,L2)(L_1,L_2), allocation ratios (ρs,ρm,ρd)( _s, _m, _d) _if_language:nTFENG =English0: Selected expert set ℰE _if_language:nTFENG =English1: Initialize: ℰ←∅E← _if_language:nTFENG =English2: Compute budgets: Ks←⌊K⋅ρs⌋K_s← K· _s , Km←⌊K⋅ρm⌋K_m← K· _m , Kd←K−Ks−KmK_d← K-K_s-K_m _if_language:nTFENG =English3: // Phase 1: Select language-specific experts from shallow layers _if_language:nTFENG =English4: for each layer l∈ℒshallowl _shallow and expert i∈[N]i∈[N] do _if_language:nTFENG =English5: Compute Spec(l,i,λ∗)Spec(l,i,λ^*) using Eq. ( _if_language:nTFENG =English8) _if_language:nTFENG =English6: end for _if_language:nTFENG =English7: ℰ←ℰ∪TopK((l,i):l∈ℒshallow,Ks)E (\(l,i):l _shallow\,K_s ) _if_language:nTFENG =English8: // Phase 2: Select cross-lingual shared experts from middle layers _if_language:nTFENG =English9: for each layer l∈ℒmiddlel _middle and expert i∈[N]i∈[N] do _if_language:nTFENG =English10: Compute Ovlp(l,i)Ovlp(l,i) using Eq. ( _if_language:nTFENG =English8) _if_language:nTFENG =English11: end for _if_language:nTFENG =English12: ℰ←ℰ∪TopK((l,i):l∈ℒmiddle,(l,i)∉ℰ,Km)E (\(l,i):l _middle,(l,i) \,K_m ) _if_language:nTFENG =English13: // Phase 3: Select language-specific experts from deep layers _if_language:nTFENG =English14: for each layer l∈ℒdeepl _deep and expert i∈[N]i∈[N] do _if_language:nTFENG =English15: Compute Spec(l,i,λ∗)Spec(l,i,λ^*) using Eq. ( _if_language:nTFENG =English8) _if_language:nTFENG =English16: end for _if_language:nTFENG =English17: ℰ←ℰ∪TopK((l,i):l∈ℒdeep,(l,i)∉ℰ,Kd)E (\(l,i):l _deep,(l,i) \,K_d ) _if_language:nTFENG =English18: return ℰE _if_language:nTFENG =EnglishAppendix C Gradient Isolation In this appendix, we formalize why routing isolation leads to gradient isolation, and why training only the target-language expert subnetwork preserves the model’s general capabilities. The key point is that sparse MoE routing does not merely reduce computation: it also induces a structured partition of the parameter space. Once the routing supports of different languages are separated, gradient updates become correspondingly localized. Setup. Consider an MoE transformer with L MoE layers and NeN_e experts per layer. For an input sequence x=(x1,…,xT)x=(x_1,…,x_T), let ht(l)h_t^(l) denote the hidden state of token t at layer l. At MoE layer l, the output is ht(l+1)=U(l)(ht(l))+∑i=1Negt,i(l)(x)Fi(l)(ht(l);θl,i),h_t^(l+1)=U^(l)\! (h_t^(l) )+ _i=1^N_eg_t,i^(l)(x)\,F_i^(l)\! (h_t^(l); _l,i ), _if_language:nTFENG =English(10) where: (i) U(l)U^(l) collects all frozen shared transformations at layer l (e.g., residual pathway, attention block, layer norm, and any non-expert shared modules), (i) Fi(l)(⋅;θl,i)F_i^(l)(·; _l,i) is expert i at layer l with parameters θl,i _l,i, and (i) gt,i(l)(x)∈0,1g_t,i^(l)(x)∈\0,1\ is the discrete routing indicator, with ∑i=1Negt,i(l)(x)=k _i=1^N_eg_t,i^(l)(x)=k under top-k routing. Let S=Sll=1LS=\S_l\_l=1^L denote the selected expert subnetwork, where Sl⊆1,…,NeS_l \1,…,N_e\ is the set of trainable experts in layer l. During RISE training, only θS=θl,i:i∈Sl _S=\ _l,i:i∈ S_l\ are updated; all other parameters are frozen. For a language λ, define its routing support at layer l as ℛλ(l)=i∈1,…,Ne:Prx∼λ,t[gt,i(l)(x)=1]>0.R_λ^(l)= \i∈\1,…,N_e\: _x _λ,\,t\! [g_t,i^(l)(x)=1 ]>0 \. _if_language:nTFENG =English(11) Intuitively, ℛλ(l)R_λ^(l) contains the experts that language λ actually uses at layer l. We also define the routing overlap mass between language λ and the selected subnetwork S: Ωλ(S)=∑l=1L∑i∈Slpλ,i(l),pλ,i(l)=x∼λ,t[gt,i(l)(x)]. _λ(S)= _l=1^L _i∈ S_lp_λ,i^(l), p_λ,i^(l)=E_x _λ,\,t [g_t,i^(l)(x) ]. _if_language:nTFENG =English(12) When routing isolation is strong, Ωλ(S) _λ(S) is small for non-target languages. _if_language:nTFENG =EnglishC.1 Gradient Isolation We first state the exact gradient form for expert parameters. _if_language:nTF ENG =English Lemma 1 (Exact gradient isolation). For any expert i at layer l, the gradient of the training loss ℒλ∗L_λ^* on target-language data λ∗D_λ^* satisfies ∇θl,iℒλ∗=x∼λ∗[∑t=1Tgt,i(l)(x)∇θl,iℓt(x)], _ _l,iL_λ^*=E_x _λ^* [ _t=1^Tg_t,i^(l)(x)\, _ _l,i _t(x) ], _if_language:nTFENG =English(13) where ℓt(x) _t(x) is the token-level loss contribution. Hence, if expert i is never activated by target-language tokens, i.e., gt,i(l)(x)=0g_t,i^(l)(x)=0 for all (x,t)(x,t) from λ∗D_λ^*, then ∇θl,iℒλ∗=0. _ _l,iL_λ^*=0. _if_language:nTFENG =English(14) Proof. _if_language:nTF ENG =EnglishFrom the MoE forward definition in Eq. ( _if_language:nTFENG =English10), the parameter θl,i _l,i appears only in the term gt,i(l)(x)Fi(l)(ht(l);θl,i)g_t,i^(l)(x)F_i^(l)(h_t^(l); _l,i). Since gt,i(l)(x)∈0,1g_t,i^(l)(x)∈\0,1\ is a multiplicative routing mask, the chain rule gives ∇θl,iℓt(x)=gt,i(l)(x)∂ℓt(x)∂Fi(l)∂Fi(l)(ht(l);θl,i)∂θl,i. _ _l,i _t(x)=g_t,i^(l)(x)\, ∂ _t(x)∂ F_i^(l)\, ∂ F_i^(l)(h_t^(l); _l,i)∂ _l,i. _if_language:nTFENG =English(15) Summing over tokens and taking expectation over x∼λ∗x _λ^* yields Eq. ( _if_language:nTFENG =English13). If gt,i(l)(x)=0g_t,i^(l)(x)=0 for all target-language examples, every term vanishes identically, so the gradient is exactly zero. ∎ This lemma shows that sparse routing induces exact gradient sparsity: an expert receives gradient only from the tokens that actually traverse it. _if_language:nTFENG =EnglishC.2 Exact Preservation Under Disjoint Routing We now formalize the strongest case: if the selected target-language subnetwork is disjoint from the routing support of another language, then training on the target language leaves that other language exactly unchanged. _if_language:nTF ENG =English Theorem 1 (Exact invariance under disjoint routing). Let λ∗λ^* be the target language and let S=Sll=1LS=\S_l\_l=1^L be the selected trainable expert subnetwork. Assume: _if_language:nTFENG =English1. All shared parameters and all unselected experts are frozen. _if_language:nTFENG =English2. For a non-target language λ, the selected experts are disjoint from its routing supports at every layer: Sl∩ℛλ(l)=∅,∀l=1,…,L.S_l _λ^(l)= , ∀ l=1,…,L. _if_language:nTFENG =English(16) Then, after any number of gradient-based training steps on λ∗D_λ^* that update only θS _S, the model’s forward computation on any input x∼λx _λ remains exactly unchanged: fθ′(x)=fθ(x),∀x∈supp(λ),f_θ (x)=f_θ(x), ∀ x (D_λ), _if_language:nTFENG =English(17) where θ denotes the initial parameters and θ′θ the parameters after training. Consequently, the predictive distribution, token losses, and task outputs on language λ are all unchanged. Proof. _if_language:nTF ENG =EnglishWe prove the claim by induction over training steps and model layers. Step 1: one update cannot alter the forward path of language λ. Consider one optimization step that updates only θS _S: θl,i′=θl,i−η∇θl,iℒλ∗,i∈Sl,θl,i,i∉Sl.θ _l,i= cases _l,i-η _ _l,iL_λ^*,&i∈ S_l,\\ _l,i,&i∉ S_l. cases _if_language:nTFENG =English(18) Take any input x∼λx _λ. We show by induction on layer depth that all hidden states remain unchanged. For the first layer, the input embeddings are unchanged because they are frozen, so ht(0)′=ht(0)h_t^(0) =h_t^(0). Assume ht(l)′=ht(l)h_t^(l) =h_t^(l) for all tokens t at layer l. Since the router and all shared modules are frozen, the router receives the same input hidden states before and after the update; therefore the routing decisions are identical: gt,i(l)′(x)=gt,i(l)(x),∀i,t.g_t,i^(l) (x)=g_t,i^(l)(x), ∀ i,t. _if_language:nTFENG =English(19) By the disjointness assumption in Eq. ( _if_language:nTFENG =English16), if gt,i(l)(x)=1g_t,i^(l)(x)=1, then necessarily i∉Sli∉ S_l, because no selected expert belongs to the routing support of language λ. Hence every expert actually activated by language λ at layer l remains frozen: θl,i′=θl,ifor all i such that gt,i(l)(x)=1.θ _l,i= _l,i all i such that g_t,i^(l)(x)=1. _if_language:nTFENG =English(20) Therefore, every active expert output is unchanged, and so is the shared branch: ht(l+1)′ h_t^(l+1) =U(l)(ht(l)′)+∑i=1Negt,i(l)′(x)Fi(l)(ht(l)′;θl,i′) =U^(l)(h_t^(l) )+ _i=1^N_eg_t,i^(l) (x)\,F_i^(l)(h_t^(l) ;θ _l,i) _if_language:nTFENG =English(21) =U(l)(ht(l))+∑i=1Negt,i(l)(x)Fi(l)(ht(l);θl,i)=ht(l+1). =U^(l)(h_t^(l))+ _i=1^N_eg_t,i^(l)(x)\,F_i^(l)(h_t^(l); _l,i)=h_t^(l+1). _if_language:nTFENG =English(22) Thus the induction closes, giving ht(l)′=ht(l)h_t^(l) =h_t^(l) for all layers l and tokens t, and therefore fθ′(x)=fθ(x)f_θ (x)=f_θ(x). Step 2: extension to multiple updates. The argument above applies after each gradient step individually. Since at every step only selected experts are modified, and these experts are never traversed by language λ, the forward computation for λ remains unchanged after every step. By induction over optimization steps, Eq. ( _if_language:nTFENG =English17) holds after arbitrary-length training. ∎ Theorem _if_language:nTFENG =English1 gives the cleanest formal statement of the intuition: if a language never uses the updated experts, then it is mathematically impossible for its forward pass to change. _if_language:nTFENG =EnglishC.3 Approximate Preservation Under Near-Orthogonal Routing In practice, routing isolation is strong but not perfectly disjoint. We therefore derive a stability bound showing that cross-lingual interference is controlled by routing overlap. _if_language:nTF ENG =English Assumption 1 (Expert smoothness). For each expert Fi(l)F_i^(l), there exists a constant Ll,i>0L_l,i>0 such that for all relevant hidden states h and parameter perturbations Δθl,i _l,i, ‖Fi(l)(h;θl,i+Δθl,i)−Fi(l)(h;θl,i)‖≤Ll,i‖Δθl,i‖. \|F_i^(l)(h; _l,i+ _l,i)-F_i^(l)(h; _l,i) \|≤ L_l,i\,\| _l,i\|. _if_language:nTFENG =English(23) _if_language:nTF ENG =English Assumption 2 (Task-head smoothness). The mapping from final hidden states to logits/loss is C-Lipschitz. _if_language:nTF ENG =English Theorem 2 (Cross-lingual perturbation bound). Let Δθ θ be any update supported only on the selected subnetwork S, i.e., Δθl,i=0 _l,i=0 for i∉Sli∉ S_l. Assume the routing pattern of a non-target language λ is locally stable under this small update. Then the expected output perturbation on λD_λ is bounded by x∼λ[‖fθ+Δθ(x)−fθ(x)‖]≤C∑l=1L∑i∈Slpλ,i(l)Ll,i‖Δθl,i‖.E_x _λ [\|f_θ+ θ(x)-f_θ(x)\| ]≤ C _l=1^L _i∈ S_lp_λ,i^(l)\,L_l,i\,\| _l,i\|. _if_language:nTFENG =English(24) Consequently, the expected loss change satisfies x∼λ[|ℓ(x;θ+Δθ)−ℓ(x;θ)|]≤C′∑l=1L∑i∈Slpλ,i(l)Ll,i‖Δθl,i‖E_x _λ [| (x;θ+ θ)- (x;θ)| ]≤ C _l=1^L _i∈ S_lp_λ,i^(l)\,L_l,i\,\| _l,i\| _if_language:nTFENG =English(25) for some constant C′>0C >0. Proof. _if_language:nTF ENG =EnglishFix x∼λx _λ and condition on its routing pattern. Because only experts in S are updated, the change in the MoE contribution at layer l can only come from activated experts in SlS_l. Using Eq. ( _if_language:nTFENG =English10) and the triangle inequality, ‖ht(l+1)′−ht(l+1)‖≤∑i∈Slgt,i(l)(x)‖Fi(l)(ht(l);θl,i+Δθl,i)−Fi(l)(ht(l);θl,i)‖.\|h_t^(l+1) -h_t^(l+1)\|≤ _i∈ S_lg_t,i^(l)(x)\, \|F_i^(l)(h_t^(l); _l,i+ _l,i)-F_i^(l)(h_t^(l); _l,i) \|. _if_language:nTFENG =English(26) Applying the smoothness assumption in Eq. ( _if_language:nTFENG =English23) gives ‖ht(l+1)′−ht(l+1)‖≤∑i∈Slgt,i(l)(x)Ll,i‖Δθl,i‖.\|h_t^(l+1) -h_t^(l+1)\|≤ _i∈ S_lg_t,i^(l)(x)\,L_l,i\,\| _l,i\|. _if_language:nTFENG =English(27) Summing across layers and propagating to the output using the C-Lipschitz property of the remaining frozen network yields ‖fθ+Δθ(x)−fθ(x)‖≤C∑l=1L∑i∈Slgt,i(l)(x)Ll,i‖Δθl,i‖.\|f_θ+ θ(x)-f_θ(x)\|≤ C _l=1^L _i∈ S_lg_t,i^(l)(x)\,L_l,i\,\| _l,i\|. _if_language:nTFENG =English(28) Taking expectation over x∼λx _λ and using [gt,i(l)(x)]=pλ,i(l)E[g_t,i^(l)(x)]=p_λ,i^(l) gives Eq. ( _if_language:nTFENG =English24). The loss bound in Eq. ( _if_language:nTFENG =English25) follows immediately from the Lipschitz continuity of the loss with respect to logits. ∎ Theorem _if_language:nTFENG =English2 shows that cross-lingual interference is not arbitrary: it is proportional to the extent to which a non-target language routes through the updated experts. Therefore, when routing isolation makes pλ,i(l)p_λ,i^(l) very small for i∈Sli∈ S_l, the induced perturbation on language λ is correspondingly small. _if_language:nTFENG =EnglishC.4 Implication for RISE The above results justify the design principle of RISE. The method first identifies a target-language expert subnetwork using routing statistics, and then updates only that subnetwork while freezing all remaining parameters. If the selected experts align with the target language’s routing support and have little overlap with other languages, then: _if_language:nTFENG =English1. gradients from target-language training are concentrated on the selected experts; _if_language:nTFENG =English2. non-target languages do not backpropagate through those experts, or do so only with very small probability; _if_language:nTFENG =English3. thus, non-target behavior is either exactly preserved (in the disjoint case) or perturbed only by a small amount bounded by routing overlap. In this sense, routing isolation induces functional decoupling between language-specific expert subnetworks, while gradient isolation turns this decoupling into optimization locality. This explains why selective expert training can improve the target language without materially degrading the model’s general multilingual capabilities. _if_language:nTFENG =EnglishAppendix D Cross-Task Transfer: MGSM Results After TyDiQA Training !20 Setting BN DE EN ES FR JA RU SW TH ZH Avg. Qwen3-30B-A3B (MGSM, Accuracy %) Vanilla 46.046.0 88.588.5 96.596.5 91.591.5 82.082.0 83.583.5 92.592.5 48.048.0 87.587.5 86.086.0 80.280.2 !10 Random (64,BN) 46.0 87.587.5 96.096.0 90.590.5 81.581.5 84.584.5 91.091.0 46.546.5 87.087.0 89.089.0 80.080.0 Random (128,BN) 49.0 87.587.5 96.096.0 90.090.0 81.581.5 83.083.0 91.091.0 46.046.0 87.087.0 87.087.0 79.879.8 !10 LoRA (Hu et al., 2021) 65.0 88.088.0 97.097.0 89.589.5 79.079.0 82.082.0 90.090.0 41.041.0 87.587.5 88.588.5 80.880.8 ESFT (Wang et al., 2024) 48.0 86.586.5 97.097.0 91.091.0 82.082.0 84.084.0 92.592.5 47.547.5 87.087.0 86.086.0 80.280.2 !10 RISE (128, BN) 55.0 87.587.5 96.596.5 89.589.5 82.582.5 84.584.5 91.091.0 46.546.5 86.086.0 87.587.5 80.780.7 RISE (128, RU) 49.049.0 88.588.5 96.096.0 91.591.5 81.081.0 85.085.0 91.0 43.043.0 87.587.5 89.089.0 80.280.2 !10 RISE (128, ID) 46.546.5 88.588.5 97.597.5 90.590.5 82.082.0 84.584.5 92.092.0 47.047.0 88.088.0 87.087.0 80.480.4 Phi-3.5-MoE-Instruct (vLLM + chat template, MGSM, Accuracy %) Vanilla 1.01.0 79.079.0 88.588.5 81.581.5 73.073.0 56.056.0 77.077.0 1.01.0 26.526.5 65.565.5 54.954.9 !10 Random (16,BN) 3.0 79.579.5 89.589.5 83.083.0 79.079.0 60.060.0 82.082.0 1.51.5 25.025.0 65.565.5 56.856.8 Random (32,BN) 3.0 78.578.5 88.088.0 83.583.5 79.579.5 57.057.0 78.578.5 1.51.5 22.522.5 64.564.5 55.755.7 !10 TopK (16,BN) 1.5 78.578.5 89.089.0 82.082.0 74.074.0 59.559.5 79.079.0 2.52.5 24.524.5 67.567.5 55.855.8 LoRA (Hu et al., 2021) 0.0 82.082.0 91.591.5 80.580.5 73.573.5 60.560.5 78.578.5 0.50.5 10.010.0 64.064.0 54.154.1 !10 ESFT (Wang et al., 2024) 3.5 81.081.0 88.588.5 82.582.5 78.078.0 56.556.5 80.580.5 1.51.5 22.522.5 66.066.0 56.156.1 RISE (16, BN) 3.5 76.576.5 88.588.5 84.084.0 75.575.5 62.562.5 81.581.5 0.50.5 22.022.0 64.064.0 55.955.9 !10 RISE (16, RU) 2.52.5 79.579.5 91.091.0 83.583.5 75.575.5 58.058.0 79.0 0.00.0 19.519.5 66.066.0 55.555.5 RISE (16, ID) 0.50.5 80.580.5 88.588.5 83.583.5 76.576.5 58.558.5 78.078.0 1.51.5 19.519.5 63.563.5 55.155.1 _if_language:nTFENG =EnglishTable 6: Cross-task transfer: MGSM accuracy (%) for models trained on TyDiQA. All models are trained exclusively on TyDiQA and evaluated zero-shot on MGSM to measure cross-task interference. The table is split by backbone: _if_language:nTFENG =EnglishQwen3-30B-A3B and _if_language:nTFENG =EnglishPhi-3.5-MoE-Instruct. The colored bold value indicates the target language column (blue==BN, orange==RU, green==ID); the language code in the first column is colored accordingly. To further validate that RISE does not impair the model’s capabilities on tasks beyond the training domain, we evaluate all TyDiQA-trained models on the MGSM benchmark without any MGSM-specific training. This cross-task evaluation directly tests whether the language-specific subnetworks identified by RISE are truly isolated: training on extractive QA (TyDiQA) should not degrade multilingual mathematical reasoning (MGSM) if the selected experts are genuinely language-specific rather than task-general. As shown in Table _if_language:nTFENG =English6, RISE consistently preserves MGSM performance across all non-target languages, confirming that our subnetwork selection respects the functional boundaries between language-specific and task-general computation. _if_language:nTFENG =EnglishAppendix E Additional Routing Analysis This section consolidates the complete layer-wise expert overlap statistics across both backbone models and both evaluation benchmarks, extending the analysis summarized in Table _if_language:nTFENG =English2 of the main text. !20 Language Shal. Mid. Deep Avg. Telugu (TE) 0.16 0.29 0.06 0.17 !10 Swahili (SW) 0.14 0.30 0.07 0.17 Bengali (BN) 0.19 0.31 0.10 0.20 !10 Korean (KO) 0.27 0.39 0.12 0.26 Finnish (FI) 0.29 0.41 0.14 0.28 !10 Arabic (AR) 0.31 0.42 0.21 0.31 Indonesian (ID) 0.32 0.46 0.19 0.32 !10 Russian (RU) 0.40 0.42 0.21 0.34 Mean 0.26 0.38 0.14 0.26 _if_language:nTFENG =EnglishTable 7: Layer-wise expert overlap with English on TyDiQA (Qwen3-30B-A3B). !20 Language Shal. Mid. Deep Avg. Telugu (TE) 0.14 0.12 0.06 0.11 !10 Bengali (BN) 0.24 0.16 0.00 0.13 Arabic (AR) 0.19 0.23 0.00 0.14 !10 Swahili (SW) 0.19 0.19 0.11 0.16 Finnish (FI) 0.24 0.23 0.06 0.17 !10 Korean (KO) 0.24 0.19 0.11 0.18 Indonesian (ID) 0.29 0.26 0.06 0.20 !10 Russian (RU) 0.33 0.39 0.00 0.24 Mean 0.23 0.22 0.05 0.17 _if_language:nTFENG =EnglishTable 8: Layer-wise expert overlap with English on TyDiQA (Phi-3.5-MoE-Instruct). !20 Language Shal. Mid. Deep Avg. Bengali (BN) 0.12 0.22 0.05 0.13 !10 Swahili (SW) 0.04 0.19 0.04 0.09 Thai (TH) 0.25 0.33 0.12 0.23 !10 Chinese (ZH) 0.36 0.43 0.11 0.30 Japanese (JA) 0.37 0.37 0.10 0.28 !10 Russian (RU) 0.41 0.43 0.17 0.34 German (DE) 0.39 0.46 0.22 0.36 !10 French (FR) 0.46 0.49 0.24 0.40 Spanish (ES) 0.52 0.51 0.25 0.43 Mean 0.32 0.38 0.14 0.28 _if_language:nTFENG =EnglishTable 9: Layer-wise expert overlap with English on MGSM (Qwen3-30B-A3B). !20 Language Shal. Mid. Deep Avg. Swahili (SW) 0.10 0.04 0.00 0.04 !10 Thai (TH) 0.05 0.12 0.00 0.06 Bengali (BN) 0.10 0.09 0.06 0.08 !10 Japanese (JA) 0.19 0.30 0.22 0.24 Russian (RU) 0.29 0.40 0.11 0.27 !10 German (DE) 0.24 0.49 0.11 0.28 French (FR) 0.33 0.49 0.06 0.29 !10 Chinese (ZH) 0.05 0.51 0.39 0.32 Spanish (ES) 0.38 0.46 0.11 0.32 Mean 0.19 0.32 0.12 0.21 _if_language:nTFENG =EnglishTable 10: Layer-wise expert overlap with English on MGSM (Phi-3.5-MoE-Instruct). Qwen3-30B-A3B. Tables _if_language:nTFENG =English7 and _if_language:nTFENG =English9 report per-language Jaccard similarity with English at the shallow, middle, and deep layers for _if_language:nTFENG =EnglishQwen3-30B-A3B on TyDiQA and MGSM, respectively. Bsides, we also provide the global-level overlap hotmap of TyDiQA in Figures _if_language:nTFENG =English5 (a), which is a supplement of the Table _if_language:nTFENG =English1 in the main text. Phi-3.5-MoE-Instruct. Tables _if_language:nTFENG =English8 and _if_language:nTFENG =English10 report the layer-wise expert overlap statistics for Phi-3.5-MoE-Instruct on TyDiQA and MGSM, respectively. Figures _if_language:nTFENG =English5 (b) and (c) provide the corresponding global-level routing overlap analysis, serving as the Phi-side counterparts to Figure _if_language:nTFENG =English1 (a) in the main text. Despite the substantial architectural differences from Qwen3-30B-A3B (32 vs. 48 layers, top-2 vs. top-8 routing, 16 vs. 128 experts per layer), both the routing isolation and layerwise divergence phenomena observed in the main text hold consistently: low-resource languages activate a narrow, repetitive set of experts with little overlap at the global level, while the layer-wise similarity follows the same three-phase hierarchy—shallow layers showing moderate overlap, middle layers peaking, and deep layers dropping to near zero. These results confirm that the two core phenomena identified in our analysis are robust across model families and evaluation tasks, rather than artifacts of a specific architecture or benchmark. _if_language:nTFENG =English(a) Global-level expert activation overlap of Qwen3-30B-A3B across languages in TyDiQA. _if_language:nTFENG =English(b) Global-level expert activation overlap of Phi-3.5-MoE across languages in MGSM. _if_language:nTFENG =English(c) Global-level expert activation overlap of Phi-3.5-MoE across languages in TyDiQA. _if_language:nTFENG =EnglishFigure 5: Global-level expert activation overlap of Qwen3-30B-A3B and Phi-3.5-MoE across languages in TyDiQA and MGSM. _if_language:nTFENG =EnglishAppendix F RISE Selected Expert Visualization _if_language:nTFENG =EnglishFigure 6: Layer-wise expert selection heatmap for Bengali training on Qwen3-30B-A3B. Each column corresponds to a transformer layer and each row to an expert index. Highlighted cells denote experts selected by RISE for gradient updates. The selection density is highest in the middle and deep layers, consistent with the routing isolation observed for Bengali in Table _if_language:nTFENG =English9. _if_language:nTFENG =EnglishFigure 7: Layer-wise expert selection heatmap for Bengali training on Phi-3.5-MoE-Instruct. Despite the architectural differences (32 layers, 16 experts, top-2 routing), RISE again targets experts concentrated in the middle-to-deep range, mirroring the isolation pattern reported in Table _if_language:nTFENG =English10. The sparser overall selection reflects the smaller expert count per layer relative to Qwen3-30B-A3B. Figures _if_language:nTFENG =English6 and _if_language:nTFENG =English7 visualize the layer-wise distribution of training experts selected by RISE for Bengali on Qwen3-30B-A3B and Phi-3.5-MoE-Instruct, respectively. Each cell indicates whether a given expert in a given layer was selected for training. The heatmaps reveal that RISE concentrates its budget in the middle and deep layers—precisely the layers where Bengali routing diverges most from English (cf. Tables _if_language:nTFENG =English9– _if_language:nTFENG =English8). This alignment confirms that the composite score in RISE successfully identifies language-specific experts rather than selecting high-activation experts indiscriminately. _if_language:nTFENG =EnglishAppendix G Layer Partition and Expert Selection Settings Setting Qwen3-30B-A3B Phi-3.5-MoE Total Layers 48 32 Experts per Layer 128 16 Total Expert Pool 6,144 512 Shallow Layers 0–17 (37.5%) 0–11 (37.5%) Middle Layers 18–29 (25%) 12–19 (25%) Deep Layers 30–47 (37.5%) 20–31 (37.5%) Selected Experts K 128 (2.1%) 16 (3.1%) Shallow budget 35% (≈ 44) 12.5% (≈ 2) Middle budget 25% (≈ 32) 68.75% (≈ 11) Deep budget 40% (≈ 52) 18.75% (≈ 3) Shallow criterion Language specificity Language specificity Middle criterion Cross-lingual overlap Cross-lingual overlap Deep criterion Language specificity Language specificity _if_language:nTFENG =EnglishTable 11: Layer-aware expert selection settings for Qwen3-30B-A3B and Phi-3.5-MoE-Instruct. Table _if_language:nTFENG =English11 summarizes the layer partition boundaries and expert selection budget used in RISE for both backbone models. The three functional groups—shallow (language-specific encoding), middle (language-agnostic semantic processing), and deep (language-specific generation)—follow the 37.5%/25%/37.5% split derived from our routing divergence analysis in Section _if_language:nTFENG =English3. The expert budget K is allocated asymmetrically across groups, concentrating the majority of the budget in the middle layers to capture cross-lingual shared computation. K (Experts) Model Weights Activations Gradients Optimizer State Total 8 56.3 GB 7.9 GB 72 MB 432 MB 64.7 GB 16 56.3 GB 7.9 GB 144 MB 864 MB 65.1 GB 32 56.3 GB 7.9 GB 288 MB 1.7 GB 66.1 GB 64 56.3 GB 7.9 GB 576 MB 3.4 GB 68.1 GB 128 56.3 GB 7.9 GB 1.1 GB 6.8 GB 72.0 GB _if_language:nTFENG =EnglishTable 12: GPU memory usage during training for different numbers of selected experts K on Qwen3-30B-A3B. Note that these figures represent the minimum memory footprint without any training data loaded; actual memory consumption may increase significantly with larger batch sizes. _if_language:nTFENG =EnglishAppendix H Why We Do Not Compare with LoRA We do not compare RISE with LoRA because they operate at orthogonal levels of model adaptation and address fundamentally different questions. RISE operates at the routing level: it identifies which experts to update, based on a mechanistic analysis of multilingual routing behavior in MoE models. LoRA operates at the parameter-update level: it specifies how to update the weight matrices efficiently via low-rank adapters. These are orthogonal design dimensions—RISE concerns the selection scope of adaptation, while LoRA concerns the parameterization of the update. The main contribution of this work is the discovery that multilingual MoE models exhibit structured routing isolation, from which language-specific expert subnetworks can be identified in a principled manner. We intentionally train the selected experts with standard full-rank updates to isolate the contribution of expert selection itself, avoiding confounds introduced by rank constraints or adapter initialization. Importantly, RISE and LoRA are fully compatible rather than mutually exclusive. One may readily apply LoRA, QLoRA, or other PEFT techniques within the RISE-selected subnetwork—freezing all non-selected experts and attaching low-rank adapters only to the selected ones. Under this view, RISE answers which part of the MoE model to adapt, while LoRA answers how to update that part efficiently. Figure _if_language:nTFENG =English8 illustrates this orthogonal and composable relationship. _if_language:nTFENG =EnglishFigure 8: RISE and LoRA are orthogonal and composable: RISE operates at the routing level (which experts to adapt), while LoRA operates at the parameter-update level (how to update them). For this reason, treating LoRA as a direct competing baseline would conflate two orthogonal design dimensions and blur the central contribution of this paper. We note that prior work has already explored LoRA-style adaptation within MoE training (Deng et al., 2026), further supporting the view that LoRA is a natural complement to RISE—applicable within the selected subnetwork for additional parameter efficiency—rather than an alternative to it. _if_language:nTFENG =EnglishAppendix I Case Study Case 1 — Qwen3-30B-A3B / MGSM Bengali — Percentage Misinterpretation Question: জোশ একটি বাড়ি ফ্লিপ করার সিদ্ধান্ত নিয়েছেন। তিনি $80,000-এ একটি বাড়ি ক্রয় করলেন এবং $50,000 দিয়ে মেরামত করলেন। এর ফলে বাড়িটির মূল্য 150% বৃদ্ধি পেল। তিনি কত ডলার লাভ করলেন? Josh bought a house for $80k, repaired it for $50k; value rose by 150%. How much profit? Vanilla ✗ আসুন প্রশ্নটি ধীরে ধীরে বিশ্লেষণ করি। প্রথমে মোট খরচ: বাড়ি ক্রয়: $80,000; মেরামত: $50,000 মোট খরচ =$80,000+$50,000=$130,000= 80,000+ 50,000= 130,000 এরপর নতুন মূল্য: বাড়িটির মূল্য 150% বৃদ্ধি পেয়েছে। নতুন মূল্য = পুরানো মূল্য × 150% = $80,000 × 1.5 == $120,000 লাভ নির্ণয়: বিক্রয় মূল্য = $120,000; মোট খরচ = $130,000 লাভ = $120,000 −- $130,000 = −-$10,000 জোশ কোনো লাভ করেনি, বরং $10,000 লোকসান করলেন। RISE (Ours) ✓ আসুন প্রশ্নটি ধাপে ধাপে বিশ্লেষণ করি। প্রথম ধাপ: মোট খরচ জোশের মোট খরচ: 80,000 + 50,000 = 130,000 ডলার দ্বিতীয় ধাপ: নতুন মূল্য বাড়িটির মূল্য 150% বৃদ্ধি পেল, অর্থাৎ: 80,000×(1+1.5)= 80,000×2.5= 200,000 ডলার 80,000× [rgb]0,0.45,0 [named]pgfstrokecolorrgb0,0.45,0(1+1.5)= 80,000× [rgb]0,0.45,0 [named]pgfstrokecolorrgb0,0.45,02.5= [rgb]0,0.45,0 [named]pgfstrokecolorrgb0,0.45,0 200,000~ডলার তৃতীয় ধাপ: লাভ 200,000− 130,000= 70,000 ডলার [rgb]0,0.45,0 [named]pgfstrokecolorrgb0,0.45,0 200,000- [rgb]0,0.45,0 [named]pgfstrokecolorrgb0,0.45,0 130,000= [rgb]0,0.45,0 [named]pgfstrokecolorrgb0,0.45,0 70,000~ডলার উত্তর: 70000 Gold: $70,000 Error: “150% increase” interpreted as ×1.5× 1.5 instead of ×2.5× 2.5 Case 2 — Qwen3-30B-A3B / MGSM Bengali — Generation Truncation Question: জন এক সপ্তাহে 60 মাইল দৌড়ান। তিনি প্রতি সপ্তাহে 3 দিন দৌড়ান। তিনি প্রথম দিনে 3 ঘণ্টা দৌড়ান এবং অন্য দুদিন তার অর্ধেক দৌড়ান। তিনি কত দ্রুত দৌড়ান? 60 mi/week; Day 1: 3 h, other two days: 1.5 h each. What is his speed? Vanilla ✗ প্রথম দিন: সময় = 3 ঘণ্টা; দূরত্ব = 3 ঘণ্টা × v অন্য দুদিন: প্রতিদিন সময় = 32 32 = 1.5 ঘণ্টা দুদিনের মোট সময় = 1.5 + 1.5 = 3 ঘণ্টা; দূরত্ব = 3v3v মোট দূরত্ব: 3v+3v=6v3v+3v=6v; মোট দূরত্ব = 60 মাইল 6v=60⇒v=6v=60\; \;v=\;[output truncated here] Evaluator extracts last number found: 60 RISE (Ours) ✓ প্রথম দিনের দৌড়ের সময়: 3 ঘণ্টা অন্য দুদিনের সময়: প্রতি দিন 3 ঘণ্টার অর্ধেক = 1.5 ঘণ্টা দুদিনের মোট সময় = 1.5×2=31.5× 2=3 ঘণ্টা মোট দৌড়ের সময় (3 দিনে): 3 + 3 = 6 ঘণ্টা দ্রুতি: দ্রুতি=60÷6= 10 মাইল/ঘণ্টা দ্রুতি=60 6= [rgb]0,0.45,0 [named]pgfstrokecolorrgb0,0.45,0 10~মাইল/ঘণ্টা উত্তর: 10 Gold: 10 mph Error: generation cut off; evaluator extracts 60 (distance) not 10 (speed) _if_language:nTFENG =EnglishFigure 9: Qualitative case study (Cases 1–2) comparing Vanilla (base model) and RISE (ours) on Bengali MGSM. Red bold marks incorrect steps; green bold marks correct reasoning. Both cases show mathematical reasoning errors in Qwen3-30B-A3B: Case 1 misinterprets “150% increase” as ×1.5× 1.5 instead of ×2.5× 2.5; Case 2 suffers premature generation truncation, causing the evaluator to extract the distance (60) rather than the speed (10). See Figure _if_language:nTFENG =English10 for Cases 3–4. Case 3 — Phi-3.5-MoE / TyDiQA Bengali — Language Coherence Passage & Question: কর্ণ ক্ষিপ্ত হয়ে অর্জুনের সঙ্গে যুদ্ধে লিপ্ত হন। ভীষণ যুদ্ধ লাগে। দুইজন সমান যোদ্ধা হওয়ার কারণে কেউ কাউকে ছাড় দিচ্ছিলেন না। সবাই নিজ নিজ যুদ্ধ ভুলে অর্জুন এবং কর্ণের যুদ্ধ দেখতে লাগলেন। কর্ণের রথের সারথী ছিলেন শল্য, যিনি ছিলেন সম্পর্কে পাণ্ডবদের মামা। তিনি কর্ণকে নানা কথায় কাবু করতে লাগলেন। হঠাৎ কর্ণের রথের চাকা মাটিতে বসে গেল। […] পরশুরামের অভিশাপে কর্ণ ব্রহ্মাস্ত্রের মন্ত্র ভুলে গেলেন। শেষ পর্যন্ত অর্জুনের হাতে কর্ণের মৃত্যু হল। প্রশ্ন: পুরাণ মতে কর্ণ কার হাতে নিহত হন? (Karna angrily engaged in battle with Arjuna…Karna’s chariot wheel sank in mud…Karna was finally slain by Arjuna. Q: According to mythology, at whose hand was Karna slain?) Vanilla ✗ আ yes, পুরাণ মতে কর্ণ কার হাতে নিহত হয়েছে। (“Ah yes, according to mythology Karna was slain at whose hand.”) Echoes the question verbatim; produces no answer. RISE (Ours) ✓ অর্জুনের হাতে কর্ণ মৃত্যু হয়। (“Karna was killed at the hand of Arjuna.”) Gold: অর্জুনের (Arjuna’s) Error: mixed-language non-answer; question echoed instead of answered Case 4 — Phi-3.5-MoE / MGSM Bengali — Incoherent Multi-step Reasoning Question: জেমস এক সপ্তাহে 3 বার 3টি স্প্রিন্টে দৌড়ানোর সিদ্ধান্ত নিয়েছেন। তিনি প্রতি স্প্রিন্টে 60 মিটার করে দৌড়ান। তিনি এক সপ্তাহে মোট কত মিটার করে দৌড়ান? James: 3 sessions/week × 3 sprints/session × 60 m/sprint. Total meters per week? Vanilla ✗ উপরে উল্লিখিত সমস্যাটি হল জেমস এক সপ্তাহে স্প্রিন্টে দৌড়ানের মোট মিটার গুন নির্ণায়ক। পদক্ষেপ: 1: প্রতি স্প্রিন্টে দৌড়ানের দূরীয়তা = (3 স্প্রিন্টে দৌড়ান = 60 মিটার) [misreads 60 m/sprint as: 3 sprints total = 60 m] 2: জেমস এক সপ্তাহে স্প্রিন্টে দৌড়ানোর সম্পর্কে জানতে চাহিয়েছে যে সপ্তাহে তিনি দৌড়ানো। Step 3: এক সপ্তাহে সাত দিন আছে। [irrelevant; confuses sessions with days-of-week] প্রতি স্প্রিন্টে দৌড়ানের দূরীয়তা [output truncated; no answer] RISE (Ours) ✓ ৬০ মিটার × ৩ = ১৮০ মিটার (60 m/sprint × 3 sprints = 180 m per session) ১৮০ মিটার × ৩ = ৫৪০ মিটার (180 m × 3 sessions = 540 m total) উত্তর: ৫৪০ Gold: 540 m Error: misreads per-sprint distance; introduces spurious 7 days/week _if_language:nTFENG =EnglishFigure 10: Qualitative case study (Cases 3–4) comparing Vanilla (base model) and RISE (ours) on Bengali TyDiQA and MGSM. Red bold marks incorrect steps; green bold marks correct reasoning. Both cases show incoherent or mixed-language outputs in Phi-3.5-MoE: Case 3 produces a question-echoing non-answer with mixed-language output; Case 4 misreads per-sprint distance and introduces a spurious 7-day/week step, with output truncated before any answer is given. RISE’s Bengali-specialized expert activation corrects both failures.