Paper deep dive
SLEM: Spectrum-Aware Latent Expert Merging
Dohyeon Kim, Bedionita Soro, Bruno Andreis, Sung Ju Hwang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mixture-of-Experts (MoE) models scale model capacity efficiently by activating only a small subset of experts for each input, while still requiring all parameters to be stored in memory, resulting in substantial memory overhead. Expert merging techniques have recently emerged as a promising compression paradigm that reduces this overhead by consolidating multiple experts into a smaller set. However, existing approaches have two key limitations: (i) direct convex interpolation in parameter space is not generally guaranteed to yield meaningful interpolation of expert functions; and (ii) estimating expert similarity or merging coefficients often relies on calibration data, introducing dependence on the calibration distribution. To address these limitations, we propose Spectrum-Aware Latent Expert Merging (SLEM), a calibration-free expert merging framework that occurs in a structured latent representation of expert weights using a VAE. SLEM groups experts within and across layers based on latent similarities and derives merging coefficients from expert weight spectra. It then performs geometry-aware latent interpolation and decodes the resulting representations into merged expert weights. Compared with existing expert merging methods, SLEM achieves comparable or better performance on general knowledge benchmarks and higher average performance on generation benchmarks involving mathematical reasoning and code generation.
Tags
Links
- Source: https://arxiv.org/abs/2610.16484v1
- Canonical: https://arxiv.org/abs/2610.16484v1
Trouble viewing inline? Open PDF directly →
Full Text
78,782 characters extracted from source content.
Expand or collapse full text
SLEM: Spectrum-Aware Latent Expert Merging Dohyeon Kim Bedionita Soro Bruno Andreis Affiliation: University of Oxford Affiliation: Slater Labsdhkim011030, sorobedio, sjhwang82@kaist.ac.kr, bruno.andreis@eng.ox.ac.uk Sung Ju Hwang Affiliation: KAIST Affiliation: DeepAuto.ai Abstract Mixture-of-Experts (MoE) models scale model capacity efficiently by activating only a small subset of experts for each input, while still requiring all parameters to be stored in memory, resulting in substantial memory overhead. Expert merging techniques have recently emerged as a promising compression paradigm that reduces this overhead by consolidating multiple experts into a smaller set. However, existing approaches have two key limitations: (i) direct convex interpolation in parameter space is not generally guaranteed to yield meaningful interpolation of expert functions; and (i) estimating expert similarity or merging coefficients often relies on calibration data, introducing dependence on the calibration distribution. To address these limitations, we propose Spectrum-Aware Latent Expert Merging (SLEM), a calibration-free expert merging framework that occurs in a structured latent representation of expert weights using a VAE. SLEM groups experts within and across layers based on latent similarities and derives merging coefficients from expert weight spectra. It then performs geometry-aware latent interpolation and decodes the resulting representations into merged expert weights. Compared with existing expert merging methods, SLEM achieves comparable or better performance on general knowledge benchmarks and higher average performance on generation benchmarks involving mathematical reasoning and code generation. 1 Introduction Mixture-of-Experts (MoE) models scale capacity efficiently by activating only a small subset of experts for each token (Shazeer et al., 2017; Cai et al., 2025). This sparsity reduces per-token computation, but not memory usage: all expert parameters must remain stored regardless of how frequently they are activated. As a result, the memory footprint scales with the total number of experts rather than the number used for each token. Expert pruning and expert merging address this mismatch by either removing experts or consolidating multiple experts into a smaller set (He et al., 2023; Li et al., 2024; Miao et al., 2025). Expert merging is particularly attractive because it seeks to retain information from the experts it consolidates rather than simply discarding them. Existing approaches, however, face two limitations. First, experts are typically combined directly in parameter space, although linear interpolation between neural-network parameters is not generally guaranteed to correspond to meaningful interpolation between the functions they represent. Neuron alignment (Li et al., 2024), curvature-aware merging (Nguyen et al., 2025), and subspace decomposition (Li et al., 2026) mitigate this issue, but the resulting merge remains tied to the original parameter space. Second, deciding which experts to merge and how strongly they should contribute often relies on calibration data through routing frequencies, routing patterns, activations, or expert outputs (Li et al., 2024; Chen et al., 2025; Li et al., 2026). Such statistics make the resulting compression decisions dependent on the calibration distribution. Improving the merging rule alone may therefore be insufficient: both the representation in which experts are combined and the information used to guide compression need to be reconsidered. Figure 1: Average performance comparison of expert merging methods on generation-based benchmarks involving mathematical reasoning and code generation (GSM8K, MATH500, MBPP, and HumanEval). SLEM consistently achieves the highest average performance under all evaluated settings. These limitations motivate reformulating expert merging as a weight-space learning problem (Han et al., 2026), shifting the merging domain from raw parameter space to a learned latent weight space (Soro et al., 2026) in which MoE experts may admit a geometry better suited to merging without calibration data. Building on this view, we propose Spectrum-Aware Latent Expert Merging (SLEM), a calibration-free framework that merges experts in a structured latent representation learned with a variational autoencoder (VAE) (Higgins et al., 2017). SLEM encodes expert weights, groups experts by latent similarity, aligns subnetworks in the same latent space, performs geometry-aware interpolation, and decodes the result into a merged expert, with subnetwork-specific merging coefficients derived from the spectral structure of the pretrained weights. It therefore requires neither calibration data nor additional forward passes: latent geometry replaces direct parameter-space interpolation, while spectral structure replaces input-dependent routing statistics with quantities computed directly from pretrained expert weights. Our contributions are summarized as follows: 1. We propose Spectrum-Aware Latent Expert Merging (SLEM), which introduces an Expert VAE to learn latent representations of expert weights and performs geometry-aware expert interpolation in the learned latent space. 2. We introduce an expert merging strategy combining spectrum-aware merging coefficients with latent-similarity-based expert grouping, enabling fully calibration-free expert merging without additional data. 3. We extend expert merging beyond within-layer constraints and enable cross-layer expert merging by representing experts from all MoE layers in a shared latent space. 4. We demonstrate that SLEM achieves the highest average performance on mathematical reasoning and code generation benchmarks, with ablations validating latent-space merging, cross-layer merging, and spectrum-aware coefficients. Taken together, SLEM provides a new perspective on expert merging by changing both interpolation and merging decisions from input-dependent statistics toward the intrinsic structure of expert weights. 2 Related Work MoE Compression and Expert Merging. One line of work reduces the memory and inference cost of MoE models by pruning redundant parameters or experts (Cai et al., 2025; Liu et al., 2026), scoring them with weight magnitude, activations and router weights (Xie et al., 2024), approximate Shapley values (Huang et al., 2025), learned per-layer ratios (Bai et al., 2025), cross-domain behavior clusters (Hu et al., 2026), or informative tokens (Zong et al., 2026), replacing redundant experts with lightweight novices (Zhang et al., 2026), or exploiting finer-grained parameter structure through low-rank matrix sharing (Li et al., 2025), a shared base with low-rank deltas (Gu et al., 2025), and cross-matrix micro-experts (Xu et al., 2025). Expert merging is complementary, since it preserves information from the experts it consolidates rather than discarding them: MEO approximates several activated experts with one (He et al., 2023), M-SMoE aligns expert neurons and derives groups and coefficients from routing statistics (Li et al., 2024), HC-SMoE clusters experts by output similarity (Chen et al., 2025), Sub-MoE extracts shared subspaces via joint SVD (Li et al., 2026), NAMEx obtains merging weights from a Nash bargaining problem (Nguyen et al., 2026), and PuzzleMoE separates shared from expert-specific parameters for sparse element-wise merging (Zhao et al., 2026). In all of these, the merged expert is still constructed from the original parameters or their explicit decompositions, and grouping and weighting rely on routing frequencies, expert outputs, activations, or calibration examples. Figure 2: Overall framework of Spectrum-Aware Latent Expert Merging (SLEM). Weight-Space Learning. A growing line of work treats trained network parameters as a data modality in their own right (Han et al., 2026; Wang et al., 2026). Early efforts learn representations over populations of trained models to predict model characteristics (Schürholt et al., 2021). Later work extends such representations to recurrent weight matrices (Herrmann et al., 2024) and to downstream domains such as geospatial learning, where representations are learned from trained models rather than raw data (Hanna et al., 2026). Because network functions are invariant to neuron permutations, a parallel line designs permutation-equivariant architectures that process weights directly (Zhou et al., 2023a; Zhou et al., 2023b), recently extended to the symmetries of Transformer weights (Tran et al., 2025). On the generative side, hyper-representations embed weights from model populations and generate new parameter configurations from them (Schürholt et al., 2022), followed by diffusion-based generators (Soro et al., 2025), flow matching with re-basin alignment to handle permutation symmetry (Gupta et al., 2026), and width-agnostic generation over structurally aligned weight spaces (Kim et al., 2026). Weight-space operations also underlie model editing and merging. Task arithmetic combines fine-tuning deltas directly in parameter space (Ilharco et al., 2023), whereas LS-Merge, closest to our setting, encodes pretrained language-model weights with a Transformer-based variational autoencoder, aligns heterogeneous models in the learned representation, and interpolates them in latent space before decoding back to parameters (Soro et al., 2026). While these methods establish weight space as a useful domain for representation, generation, and model-level manipulation, we study learned weight representations at the granularity of individual MoE experts and use their geometry explicitly for compression. Rather than merging complete models or interpolating experts directly in parameter space, we learn latent representations of individual experts and combine their latent geometry with spectral structure to determine expert groupings and merging coefficients without calibration data. Figure 3: Culmulative explained variance ratio (EVR) of expert projection weights across layers in LFM2-8B-A1B. The EVR curves show that variance is concentrated along a subset of principal directions across all projection types, motivating the use of dominant spectral structure for estimating subnetwork importance. Results for the other evaluated models are provided in Appendix G. 3 Spectrum-Aware Latent Expert Merging (SLEM) Consider a pretrained Mixture-of-Experts (MoE) model with L MoE layers, where layer ℓ contains K experts E1(ℓ),…,EK(ℓ)\E_1^( ),…,E_K^( )\, of which only k≪Kk K are activated for each token. Unlike conventional within-layer expert merging, SLEM allows expert pairs to be selected both within and across MoE layers. Accordingly, our objective is to reduce the expert-memory footprint without retraining by replacing an expert pair =Ei(ℓ),Ej(ℓ′)G=\E_i^( ),E_j^( )\, where ℓ and ℓ′ may be identical or different, with a single merged expert. Each pairwise merge therefore reduces the number of stored expert parameter sets by one. Rather than combining the corresponding expert weights directly in weight space, we treat them as structured data, map them to learned latent representations, merge in that space, and decode the result back into a single expert. For notational simplicity, we omit layer indices throughout, while allowing EiE_i and EjE_j to originate from either the same or different MoE layers. 3.1 Hyperspherical Expert VAE Following LS-Merge (Soro et al., 2026), we learn latent representations of expert weights with transformer-based variational autoencoders (VAEs). To reflect the internal structure of MoE experts, we design an Expert VAE that jointly models the three projection types p∈=gate,up,downp∈P=\ gate, up, down\. The training data consist solely of the non-shared expert weights of the pretrained model, obtained by splitting the fused gate_up_proj weights of each expert EiE_i into gate and up projections and retaining the corresponding down_proj weights. Each projection weight is represented as a sequence along the hidden-unit dimension, since a gated feed-forward expert with intermediate width DMoED_MoE decomposes into contributions from its hidden units, which we refer to as subnetworks, Ei(x)=∑m=1DMoEEi,m(x),Ei,m(x)=φ(gi,m⊤x)(ui,m⊤x)di,m,E_i(x)= _m=1^D_MoEE_i,m(x), E_i,m(x)= \! (g_i,m x ) (u_i,m x )d_i,m, (1) where gi,mg_i,m and ui,mu_i,m are the m-th rows of the gate and up projections respectively, di,md_i,m is the m-th column of the down projection, and φ is the activation function. The triplet (gi,m,ui,m,di,m)(g_i,m,u_i,m,d_i,m) defines a subnetwork index common to the three projections. Organizing the projections by this index gives Wi,gate=GiW_i, gate=G_i, Wi,up=UiW_i, up=U_i, and Wi,down=Di⊤W_i, down=D_i , all in ℝDMoE×DR^D_MoE× D, where the m-th row of Wi,pW_i,p holds the component of subnetwork m in projection p∈p and D is the model hidden dimension. A single training sample for each expert is constructed by concatenating the three projection weights along the sequence dimension, Wi=[Wi,gate;Wi,up;Wi,down]∈ℝ3DMoE×DW_i= [W_i, gate;W_i, up;W_i, down ] ^3D_MoE× D. We use type-specific projections to distinguish the gate, up, and down segments while jointly modeling their relationships within each expert. In addition, we omit positional encodings because the absolute ordering of subnetworks has no canonical functional meaning, as reflected in equation 1. Hyperspherical prior and objective. The prior toward which the posterior is regularized determines the geometry in which expert representations are later compared and interpolated, so its choice is central to our setting. Schürholt et al. (2022) observed that learned weight embeddings tend to occupy a shell in high-dimensional space, which is poorly matched by the Gaussian prior commonly used in VAEs. We therefore adopt a hyperspherical VAE (Davidson et al., 2018) with von Mises–Fisher (vMF) prior and use a fixed concentration κ following prior work (Xu and Durrett, 2018). For an expert weight W, the encoder defines qϕ(∣W)=vMF(μϕ(W),κ)q_φ(z W)=vMF\! ( _φ(W),κ ), while the prior is parameterized as pψ()=vMF(ν,κ)p_ψ(z)=vMF\! (ν,κ ), where ν is a learnable prior direction. We optimize ℒ=minθ,ϕ,ψ∼qϕ(∣W)[‖W−θ()‖F2]+βDKL(qϕ(∣W)∥pψ()),L= _θ,φ,ψ\;E_z q_φ(z W) [ \|W-D_θ(z) \|_F^2 ]+β D_KL (q_φ(z W)\, \|\,p_ψ(z) ), (2) subject to ‖2=‖μϕ(W)‖2=‖νℓ‖2=1\|z\|_2=\| _φ(W)\|_2=\| _ \|_2=1, with β controlling the trade-off between reconstruction fidelity and latent regularity (Higgins et al., 2017). Both the sampled latent and the vMF mean directions therefore lie on d−1S^d-1; we denote the latent representation of subnetwork m of expert i in projection p as i,pmz_i,p^m. 3.2 Latent Expert Merging Given the learned latent space, SLEM (i) identifies expert pairs and aligns their subnetworks using latent similarity, (i) assigns subnetwork-specific merging coefficients based on spectral importance, and (i) performs geometry-aware latent interpolation, followed by decoding the merged expert. Latent subnetwork similarity. Since the latent representations lie on the unit hypersphere, their geometry is characterized by direction, which makes cosine similarity a natural choice. We define the compatibility between subnetwork m of expert i and subnetwork n of expert j as Cm,n(i,j)=1||∑p∈(i,pm)⊤j,pn‖i,pm‖2‖j,pn‖2.C_m,n^(i,j)= 1|P| _p (z_i,p^m) z_j,p^n\|z_i,p^m\|_2\|z_j,p^n\|_2. (3) Based on equation 3, we define the similarity of the two experts i,ji,j as the mean compatibility over all subnetwork pairs, (i,j)=1DMoE2∑m=1DMoE∑n=1DMoECm,n(i,j)S(i,j)= 1D_MoE^2 _m=1^D_MoE _n=1^D_MoEC_m,n^(i,j). Expert pairs with higher (i,j)S(i,j) are considered more compatible and are therefore prioritized for grouping. We form pairwise merge groups by greedy one-to-one matching on S: at each step we take the most similar remaining pair, assign it to a merge group, and remove both experts from consideration, until the desired number of pairs is reached. We additionally impose a layer-wise limit on the number of merged experts to prevent merge groups from being overly concentrated in a small subset of layers. Within a group, subnetworks still have no canonical ordering across experts and must be aligned before interpolation. Hungarian matching of hidden units substantially improves expert merging (Li et al., 2024), so we follow this approach but define the criterion in latent space, seeking the permutation π of [DMoE][D_MoE] that maximizes total compatibility defined as π⋆=argmax∑m=1DMoEπCm,π(m)(i,j).π = _π _m=1^D_MoEC_m,π(m)^(i,j). The resulting permutation is applied consistently across the three projections, preserving the shared subnetwork index of equation 1. Spectral importance as merging coefficients. As shown in Figure 3, subnetworks do not contribute equally to the dominant structure of an expert, so we set their coefficients from the spectrum of the pretrained weights Wi,p=Ai,pΣi,pBi,p⊤W_i,p=A_i,p _i,pB_i,p , where σi,p,c _i,p,c is the c-th singular value and Ai,p,mcA_i,p,mc the (m,c)(m,c) entry of Ai,pA_i,p. We quantify the spectral importance of subnetwork m as s~i,p,mr=∑c=1rσi,p,c2Ai,p,mc2,si,p,mr=s~i,p,mr∑ns~i,p,nr,si,mr=1||∑p∈si,p,mr, s_i,p,m^\,r= _c=1^r _i,p,c^2A_i,p,mc^2, s_i,p,m^\,r= s_i,p,m^\,r _n s_i,p,n^\,r, s_i,m^\,r= 1|P| _p s_i,p,m^\,r, (4) which measures the amount of spectral energy in the leading r singular directions attributable to subnetwork m. A larger value therefore indicates that the subnetwork contributes more strongly to the dominant structure of the expert. For a matched pair (m,π⋆(m))(m,π (m)), we then determine the merging coefficient from their relative spectral importance, assigning a larger contribution to the more spectrally important subnetwork: αm=si,mrsi,mr+sj,π⋆(m)r,,απ⋆(m)=sj,π⋆(m)rsi,mr+sj,π⋆(m)r,=1−αm. _m= s_i,m^\,rs_i,m^\,r+s_j,π (m)^\,r,, _π (m)= s_j,π (m)^\,rs_i,m^\,r+s_j,π (m)^\,r,=1- _m. (5) These coefficients replace uniform averaging and routing-frequency weighting, and the rank r, which controls how much of the leading spectrum is retained, is the only free parameter of the rule. Latent expert interpolation. Because the representations are directional, we merge each aligned pair with spherical linear interpolation as ~ijm=SLERP(im,jπ⋆(m),1−αm) z_ij^\,m=SLERP(z_i^m,z_j^π (m);1- _m), where SLERP(a,b,t)SLERP(a,b;t) interpolates from a at t=0t=0 to b at t=1t=1, so a larger αm _m weights the subnetwork from expert i more heavily. The interpolated latents are assembled into Z~ij Z_ij and decoded as W~ij=θ(Z~ij) W_ij=D_θ( Z_ij). The resulting expert then replaces the pair in the compressed MoE layer. Table 1: Downstream performance comparison with existing expert merging methods. All merging methods are evaluated under the same expert reduction rate. The best and second-best results within each expert reduction setting are highlighted in bold and underlined, respectively. General Knowledge (MCQA) Mathematics & Coding (Generation) Expert Reduction Method MMLU ARC-E ARC-C Average ↑ GSM8K MATH500 MBPP HumanEval Average ↑ LFM2-8B-A1B 0.0% Vanilla 63.1163.11 75.5975.59 47.6347.63 62.1162.11 81.7381.73 72.4072.40 45.6045.60 70.7370.73 67.6267.62 VAE-reconstructed 62.93±0.2162.93_ ± 0.21 75.61±0.4575.61_ ± 0.45 47.41±0.6147.41_ ± 0.61 61.98±0.3261.98_ ± 0.32 82.35±0.3082.35_ ± 0.30 72.20±0.6472.20_ ± 0.64 45.52±1.1445.52_ ± 1.14 71.83±3.1771.83_ ± 3.17 67.97±0.7267.97_ ± 0.72 12.5% F-Merging 55.6455.64 68.1868.18 42.3242.32 55.3855.38 69.1469.14 30.6030.60 03.0003.00 03.6603.66 26.6026.60 M-SMoE 54.7254.72 72.9772.97 44.1144.11 57.2757.27 57.2457.24 20.4020.40 13.2013.20 25.6125.61 29.1129.11 HC-SMoE 57.4557.45 69.8269.82 43.8743.87 57.0557.05 64.9764.97 56.0056.00 37.0037.00 57.9357.93 53.9853.98 Sub-MoE 56.2856.28 69.6069.60 42.2742.27 56.0556.05 70.1370.13 59.6059.60 34.8034.80 54.2754.27 54.7054.70 SLEM (r=14) 58.37 74.79 47.87 60.34 77.10 67.60 44.20 62.80 62.93 25.0% F-Merging 47.7247.72 65.0365.03 38.2338.23 50.3250.32 37.8337.83 06.4006.40 00.2000.20 00.6100.61 11.2611.26 M-SMoE 46.0746.07 66.8966.89 40.3140.31 51.0951.09 02.0502.05 02.2002.20 00.0000.00 00.0000.00 01.0601.06 HC-SMoE 50.9850.98 60.4760.47 37.1537.15 49.5349.53 54.1354.13 45.2045.20 30.4030.40 44.5144.51 43.5643.56 Sub-MoE 49.9149.91 64.8264.82 37.7537.75 50.8350.83 43.5943.59 34.2034.20 08.2008.20 28.0528.05 28.5128.51 SLEM (r=14) 51.1051.10 70.62 42.75 54.82 70.36 58.8058.80 36.20 61.59 56.74 ⌞\; within-layer merging 51.29 67.89 41.47 53.55 68.16 59.00 31.40 59.76 54.58 OLMoE-1B-7B-0125-Instruct 0.0% Vanilla 53.4553.45 76.4876.48 47.1747.17 59.0359.03 68.9968.99 23.0023.00 29.6029.60 35.3735.37 39.2439.24 VAE-reconstructed 53.21±0.1053.21_ ± 0.10 76.36±0.5676.36_ ± 0.56 47.39±0.7547.39_ ± 0.75 58.98±0.3258.98_ ± 0.32 68.89±0.8968.89_ ± 0.89 23.64±1.0823.64_ ± 1.08 28.40±0.8928.40_ ± 0.89 33.54±1.0233.54_ ± 1.02 38.62±0.4238.62_ ± 0.42 12.5% F-Merging 45.1345.13 66.1266.12 38.9138.91 50.0550.05 41.3241.32 03.6003.60 00.0000.00 00.0000.00 11.2311.23 M-SMoE 43.5643.56 63.4363.43 35.7535.75 47.5847.58 07.5107.51 03.6003.60 00.0000.00 00.0000.00 02.7802.78 HC-SMoE 50.58 75.4975.49 48.02 58.03 56.0356.03 09.4009.40 12.0012.00 11.5911.59 22.2622.26 Sub-MoE 46.2646.26 69.8269.82 39.6439.64 51.9151.91 48.4548.45 07.2007.20 10.6010.60 11.5911.59 19.4619.46 SLEM (r=16) 47.3747.37 75.80 45.9045.90 56.3656.36 59.74 14.20 20.00 25.00 29.74 25.0% F-Merging 32.6932.69 62.7162.71 35.3235.32 43.5843.58 05.2305.23 03.6003.60 00.0000.00 00.0000.00 02.2102.21 M-SMoE 23.1623.16 42.1042.10 26.3326.33 30.5330.53 00.0000.00 00.0000.00 00.0000.00 00.0000.00 00.0000.00 HC-SMoE 43.52 69.4169.41 41.67 51.54 06.4406.44 03.8003.80 02.4002.40 00.6100.61 03.3103.31 Sub-MoE 33.5633.56 39.9339.93 22.8722.87 32.1232.12 00.9100.91 01.6001.60 02.0002.00 00.0000.00 01.1301.13 SLEM (r=16) 30.3330.33 69.95 40.8740.87 47.0547.05 23.12 06.60 10.40 05.49 11.40 ERNIE-4.5-21B-A3B-PT 0.0% Vanilla 73.8973.89 83.3083.30 56.6256.62 71.2771.27 90.6790.67 74.8074.80 73.2073.20 89.6389.63 82.0882.08 VAE-reconstructed 74.01±0.1574.01_ ± 0.15 82.74±0.3382.74_ ± 0.33 55.68±0.4655.68_ ± 0.46 70.81±0.1770.81_ ± 0.17 90.42±0.5990.42_ ± 0.59 75.48±1.0775.48_ ± 1.07 72.52±0.4172.52_ ± 0.41 88.78±1.1388.78_ ± 1.13 81.80±0.3581.80_ ± 0.35 12.5% F-Merging 62.4662.46 82.15 52.8252.82 65.8165.81 84.6984.69 48.6048.60 36.4036.40 42.0742.07 52.9452.94 M-SMoE 67.6667.66 74.7374.73 48.3648.36 63.5863.58 82.6482.64 51.6051.60 46.0046.00 51.8351.83 58.0258.02 HC-SMoE 70.74 82.0482.04 54.61 69.13 85.90 59.0059.00 55.4055.40 62.8062.80 65.7865.78 Sub-MoE 64.8364.83 74.4674.46 44.1644.16 61.1561.15 84.4684.46 52.8052.80 46.8046.80 60.3760.37 61.1161.11 SLEM (r=12) 65.5765.57 79.9279.92 50.0050.00 65.1665.16 85.7585.75 65.40 64.40 83.54 74.77 25.0% F-Merging 56.9256.92 67.7667.76 40.1940.19 54.9654.96 64.8264.82 21.0021.00 11.8011.80 06.7106.71 26.0826.08 M-SMoE 54.5754.57 61.4561.45 35.1535.15 50.3950.39 49.2049.20 13.2013.20 10.2010.20 07.3207.32 19.9819.98 HC-SMoE 64.53 79.06 52.35 65.32 79.83 37.4037.40 32.2032.20 32.9332.93 45.5945.59 Sub-MoE 51.4251.42 62.8662.86 35.3435.34 49.8749.87 50.4250.42 18.2018.20 20.2020.20 18.9018.90 26.9326.93 SLEM (r=12) 53.0953.09 76.2676.26 46.7546.75 58.7058.70 72.2572.25 44.40 46.80 51.83 53.82 ⌞\; within-layer merging 50.33 75.42 44.80 56.85 59.06 40.00 44.40 45.12 47.15 4 Experiments 4.1 Experimental Setup Models and benchmarks. We conduct experiments on three MoE models ranging 7B-21B parameters: LFM2-8B-A1B (AI, 2025), OLMoE-1B-7B-0125-Instruct (Muennighoff et al., 2025), and ERNIE-4.5-21B-A3B-PT (Team, 2025). We evaluate three categories: general knowledge reasoning with MMLU (Hendrycks et al., 2021a), ARC-Easy, and ARC-Challenge (Clark et al., 2018); mathematical reasoning with GSM8K (Cobbe et al., 2021) and MATH500 (Hendrycks et al., 2021b); and code generation with MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021). General knowledge reasoning is evaluated in an MCQA format, while the other two categories use generation-based evaluation. Baselines. We compare SLEM against F-Merging, M-SMoE (Li et al., 2024), HC-SMoE (Chen et al., 2025), and Sub-MoE (Li et al., 2026). F-Merging uses expert activation frequencies collected from calibration data to guide the merging process. For all calibration-dependent baselines, we use randomly sampled 32×204832× 2048 tokens from C4 dataset (Dodge et al., 2021) to obtain the required routing statistics or expert outputs. Due to computational constraints, we conduct experiments using greedy decoding with a single run unless otherwise specified. Further details on Expert VAE training, baseline descriptions, and benchmark configurations are provided in Appendix B and C. 4.2 Functional Fidelity of Reconstructed Experts We first evaluate whether the trained hyperspherical VAE can effectively represent and reconstruct expert weights before applying latent-space merging. We sample latent representations from the posterior conditioned on the pretrained expert weights and decode them back into the corresponding weights. Table 1 reports the mean and standard deviation of downstream performance over five posterior samples, where the original expert weights are replaced with VAE-reconstructed weights. Across all evaluated tasks, the reconstructed models achieve performance close to that of the original models with modest variations across samples. These results indicate that the learned latent representations retain the information necessary to recover functionally effective expert weights, supporting the use of the learned hyperspherical latent space for subsequent expert interpolation and merging. 4.3 Comparison with Expert Merging Baselines Table 1 compares SLEM with existing expert merging methods across three MoE models and two expert reduction rates. For LFM2, SLEM achieves the highest average performance on both general knowledge and generation benchmarks at 12.5% and 25% expert reduction. For OLMoE and ERNIE-4.5, HC-SMoE performs better on average for general knowledge benchmarks, while SLEM remains competitive. In contrast, SLEM consistently achieves the highest average generation performance across all models and expert reduction settings. Overall, these results show that SLEM maintains comparable or better general knowledge performance while providing consistently stronger performance on generation tasks, demonstrating its effectiveness across different model architectures, scales, and compression levels. Figure 4: Comparison of parameter-space (PS) and latent-space (LS) merging under identical expert pairs and merging coefficients. Across both models and task categories, LS Merging consistently outperforms PS Merging for every evaluated pair. Figure 5: Effect of the truncation rank r for spectrum-aware merging coefficients. Spectrum-aware coefficients consistently outperform the uniform-coefficient baseline across models and tasks. 4.4 Ablation Study We ablate the three key components of SLEM: merging in the learned latent space, spectrum-aware merging coefficients, and cross-layer merging. Merging space. To isolate the effect of the merging space, we randomly construct expert pairs and assign the same uniform coefficients of [0.5,0.5][0.5,0.5], varying only whether interpolation is performed in parameter space or in the learned latent space. We repeat this experiment with five independently sampled expert pairs. As shown in Figure 4, latent-space merging consistently outperforms parameter-space merging for every pair across both benchmark categories and the improvement is particularly pronounced on generation-based benchmarks. These results provide empirical evidence that interpolating experts in the learned latent space offers a more effective merging space than direct interpolation of their parameters. Spectrum-aware coefficients. We next evaluate spectrum-aware merging coefficients by varying the truncation rank r used for spectral importance estimation and comparing against uniform coefficients of [0.5,0.5][0.5,0.5]. As shown in Figure 5, spectrum-aware merging consistently outperforms uniform averaging across models and task categories, supporting the effectiveness of spectral importance for determining subnetwork-specific contributions. Performance also varies with truncation rank r, revealing a trade-off in how concentrated the resulting importance distribution becomes. As r increases, the normalized SVD importance tends to become relatively uniform across subnetworks, reducing the benefit of spectrum-aware weighting. Conversely, smaller ranks produce a more discriminative importance distribution, which can improve performance. However, an overly concentrated distribution can assign negligible importance to some subnetworks, which may discard useful contributions and degrade performance. This suggests that effective merging requires a balance between distinguishing important subnetworks and preserving the contribution of overall subnetworks. We provide the normalized spectral importance distributions under different truncation ranks in Appendix H. Table 2: Downstream performance under constrained maximum expert group size. Limiting the group size generally degrades performance compared with unconstrained grouping. General Knowledge (MCQA) Mathematics & Coding (Generation) Max Group Size Method MMLU ARC-E ARC-C Average ↑ GSM8K MATH500 MBPP HumanEval Average ↑ LFM2-8B-A1B ∞ M-SMoE 46.0746.07 66.8966.89 40.3140.31 51.0951.09 02.0502.05 02.2002.20 00.0000.00 00.0000.00 01.0601.06 HC-SMoE 50.9850.98 60.4760.47 37.1537.15 49.5349.53 54.1354.13 45.2045.20 30.4030.40 44.5144.51 43.5643.56 Sub-MoE 49.9149.91 64.8264.82 37.7537.75 50.8350.83 43.5943.59 34.2034.20 08.2008.20 28.0528.05 28.5128.51 2 M-SMoE 34.5534.55 65.9965.99 38.9938.99 46.5146.51↓4.58 4.58 08.3408.34 03.2003.20 00.0000.00 00.0000.00 02.8902.89↑1.83 1.83 HC-SMoE 49.8449.84 66.4666.46 40.1040.10 52.1352.13↑2.60 2.60 01.5201.52 06.0006.00 01.2001.20 01.2201.22 02.4902.49↓41.07 41.07 Sub-MoE 47.3347.33 63.9363.93 37.1237.12 49.4649.46↓1.37 1.37 44.2044.20 31.4031.40 08.0008.00 03.0503.05 21.6621.66↓6.85 6.85 SLEM (r=14) 51.10 70.62 42.75 54.82 70.36 58.80 36.20 61.59 56.74 ERNIE-4.5-21B-A3B-PT ∞ M-SMoE 54.5754.57 61.4561.45 35.1535.15 50.3950.39 49.2049.20 13.2013.20 10.2010.20 07.3207.32 19.9819.98 HC-SMoE 64.5364.53 79.0679.06 52.3552.35 65.3265.32 79.8379.83 37.4037.40 32.2032.20 32.9332.93 45.5945.59 Sub-MoE 51.4251.42 62.8662.86 35.3435.34 49.8749.87 50.4250.42 18.2018.20 20.2020.20 18.9018.90 26.9326.93 2 M-SMoE 54.4354.43 60.4860.48 34.1334.13 49.6849.68↓0.71 0.71 43.3743.37 11.6011.60 08.8008.80 08.5408.54 18.0818.08↓1.90 1.90 HC-SMoE 55.80 69.5369.53 41.3041.30 55.5455.54↓9.78 9.78 45.2645.26 12.6012.60 09.2009.20 11.5911.59 19.6619.66↓25.93 25.93 Sub-MoE 47.8847.88 62.2962.29 31.8231.82 47.3347.33↓2.54 2.54 48.8248.82 17.8017.80 20.8020.80 15.2415.24 25.6725.67↓1.26 1.26 SLEM (r=12) 53.0953.09 76.26 46.75 58.70 72.25 44.40 46.80 51.83 53.82 Figure 6: Token-level Jensen–Shannon divergence (JSD) and perplexity (PPL) of merged models on sequences generated by the original model. We evaluate 100 sequences only over the response tokens. Lower values indicate better preservation of the original model’s generative behavior. Cross-layer merging. To verify the effectiveness of cross-layer merging, we compare SLEM with a within-layer variant in which the expert-grouping scope is restricted to experts from the same MoE layer, while keeping the remaining components unchanged. As shown in Table 1, allowing cross-layer expert pairs generally improves downstream performance across models and tasks. This suggests that the shared latent space from the trained Expert VAE provides a more flexible representation for identifying compatible experts across layer boundaries. 4.5 Extended Analysis We further analyze SLEM from two perspectives. First, we examine how constraining the maximum expert group size affects existing merging methods, providing a controlled comparison under the pairwise merging setting used by SLEM. Second, we evaluate how closely the merged models preserve the generative behavior of the original model using token-level Jensen–Shannon divergence (JSD) and perplexity (PPL). 4.5.1 Effect of Expert Group Size Constraints Consider a pool of N experts partitioned into M merge groups qq=1M\G_q\_q=1^M, where each group is replaced by a single merged expert. For a fixed expert reduction R, R=∑q=1M(|q|−1),Npreserved=N−∑q=1M|q|=N−R−M.R= _q=1^M (|G_q|-1 ), N_preserved=N- _q=1^M|G_q|=N-R-M. (6) Allowing larger groups therefore achieves the same reduction with fewer merge groups, leaving more original experts untouched. Pairwise merging, in contrast, requires M=RM=R, which gives Npreserved=N−2RN_preserved=N-2R and involves substantially more original experts in the merging process. This raises the question of whether effective expert merging benefits more from preserving many original experts or from limiting each merged group to few constituents. To answer it, we restrict the maximum group size of existing merging methods to two while keeping the expert reduction rate fixed. As shown in Table 2, this constraint generally degrades performance across the baselines, with particularly large drops for HC-SMoE. Lasby et al. (2026) report a high proportion of singleton clusters in HC-SMoE. We hypothesize that limiting the maximum group size reduces the number of preserved experts, which may contribute to the observed performance drop. In contrast, SLEM is designed explicitly around pairwise expert merging and maintains strong performance under the same constraint. These results suggest that SLEM can preserve useful expert functionality even when a larger fraction of the original experts must participate in merging. 4.5.2 Preservation of Generative Behavior We further examine whether the strong downstream performance of SLEM corresponds to better preservation of the original model’s generative behavior. Figure 6 reports token-level JSD between the output distributions of the original and merged models, together with PPL on sequences generated by the original model. Across the evaluated model–task pairs, SLEM exhibits consistently low JSD and achieves the lowest PPL among the merged models, indicating closer agreement with the original model’s token distributions. These results suggest that SLEM more faithfully preserves the generative behavior of the original model. 5 Conclusion In this work, we proposed Spectrum-Aware Latent Expert Merging (SLEM), a calibration-free framework for MoE compression that performs expert merging in a learned latent space. SLEM represents experts across different MoE layers in a shared latent space, enabling both within- and cross-layer merging through geometry-aware interpolation. Expert pairs are selected based on latent similarity, while subnetwork-specific coefficients are derived directly from the spectrum of pretrained expert weights. Across MoE models of different architectures and parameter scales, SLEM consistently achieves strong performance on mathematical reasoning and code generation tasks. Our ablations further demonstrate the effectiveness of latent-space merging, spectrum-aware coefficients, and cross-layer merging. In addition, SLEM more closely preserves the generative behavior of the original model. Overall, these results highlight the potential of learned expert representations and the spectral structure of expert weights for calibration-free MoE compression. AI use statement Generative AI tools were used solely for language editing, improving the clarity and readability of the manuscript, and assisting with literature search. All research ideas, methodological design, experiments, analyses, and conclusions were developed and verified by the authors. Reproducibility statement To support reproducibility, we detail the relevant hyperparameters throughout the Appendix. Our code and merged models will be open-sourced upon acceptance. Ethics statement All models and datasets used in this work are open-source and publicly available. We thank those who produced, compiled, cleaned, and released them. We have cited their work appropriately and complied with all relevant licensing and usage terms. To the best of our knowledge, these models and datasets do not reveal personally identifiable information or violate copyright restrictions. Acknowledgements This work was supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2019-I190075, Artificial Intelligence Graduate School Program (KAIST)), National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00256259), the Institute of Information & Communications Technology Planning & Evaluation (IITP) with a grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research. (No. RS-2024-00469482 & RS-2024-00509279), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.RS-2022-I220713, Meta-learning Applicable to Real-world Problems), the “Advanced GPU Utilization Support Program” funded by the Government of the Republic of Korea (Ministry of Science and ICT), and the InnoCORE program of the Ministry of Science and ICT (No. N10250156). References AI (2025) L. AI LFM2 technical report. arXiv preprint arXiv:2511.23404. Cited by: §B.1, §4.1. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton Program synthesis with large language models. External Links: 2108.07732 Cited by: Table 3, §4.1. Bai et al. (2025) S. Bai, H. Li, J. ZHANG, Z. Hong, and S. Guo DiEP: adaptive mixture-of-experts compression through differentiable expert pruning. In Advances in Neural Information Processing Systems, Vol. 38, Main Conference, p. 56090–56115. External Links: Document Cited by: §2. Cai et al. (2025) W. Cai, J. Jiang, F. Wang, J. Tang, S. Kim, and J. Huang A Survey on Mixture of Experts in Large Language Models . IEEE Transactions on Knowledge & Data Engineering 37 (07), p. 3896–3915. External Links: ISSN 1558-2191 Cited by: §1, §2. Chen et al. (2025) I. Chen, H. Liu, W. Sun, C. Chao, Y. Hsu, and C. Lee Retraining-free merging of sparse MoE via hierarchical clustering. In Proceedings of the 42nd International Conference on Machine Learning, Vol. 267, p. 8594–8620. Cited by: §B.2, §1, §2, §4.1. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating large language models trained on code. External Links: 2107.03374 Cited by: Table 3, §4.1. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the ai2 reasoning challenge. External Links: 1803.05457 Cited by: Table 3, Table 3, §4.1. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. External Links: 2110.14168 Cited by: Table 3, §4.1. Davidson et al. (2018) T. R. Davidson, L. Falorsi, N. De Cao, T. Kipf, and J. M. Tomczak Hyperspherical variational auto-encoders. In 34th Conference on Uncertainty in Artificial Intelligence (UAI), Cited by: §3.1. Dodge et al. (2021) J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner Documenting large webtext corpora: a case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, p. 1286–1305. External Links: Document Cited by: §4.1. Gao et al. (2024) L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou The language model evaluation harness. Zenodo. External Links: Document Cited by: §B.3. Gu et al. (2025) H. Gu, W. Li, L. Li, Z. Qiyuan, M. G. Lee, S. Sun, W. Xue, and Y. Guo Delta decompression for MoE-based LLMs compression. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 20497–20514. Cited by: §2. Gupta et al. (2026) S. Gupta, S. Biggs, M. Laber, Z. Shafi, R. Walters, and A. Paul DeepWeightFlow: re-basined flow matching for generating neural network weights. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Han et al. (2026) X. Han, Z. Wang, B. Zhao, B. Zhang, J. Li, D. Borth, R. Yu, H. Maron, Y. Ye, L. Yin, and F. Neri A survey of weight space learning: understanding, representation, and generation. External Links: 2603.10090, Link Cited by: §1, §2. Hanna et al. (2026) J. Hanna, D. Falk, S. X. Yu, and D. Borth GeoSANE: learning geospatial representations from models, not data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 27804–27814. Cited by: §2. He et al. (2023) S. He, R. Fan, L. Ding, L. Shen, T. Zhou, and D. Tao Merging experts into one: improving computational efficiency of mixture of experts. In The 2023 Conference on Empirical Methods in Natural Language Processing, External Links: Link Cited by: §1, §2. Hendrycks et al. (2021a) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR). Cited by: Table 3, §4.1. Hendrycks et al. (2021b) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset. Advances in neural information processing systems. Cited by: Table 3, §4.1. Herrmann et al. (2024) V. Herrmann, F. Faccio, and J. Schmidhuber Learning useful representations of recurrent neural network weight matrices. External Links: 2403.11998, Link Cited by: §2. Higgins et al. (2017) I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner Beta-VAE: learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, External Links: Link Cited by: §1, §3.1. Hu et al. (2026) W. Hu, M. Zhao, S. Song, X. Zhu, X. Lai, and J. Wang Mosaic pruning: a hierarchical framework for generalizable pruning of mixture-of-experts models. In AAAI Conference on Artificial Intelligence, Cited by: §2. Huang et al. (2025) W. Huang, Y. Zhang, X. Zheng, F. Chao, R. Ji, and L. Cao Discovering important experts for mixture-of-experts models pruning through a theoretical perspective. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2. Ilharco et al. (2023) G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2. Kim et al. (2026) J. Kim, S. Mehta, H. Hsu, H. Ryu, Y. Liu, and M. Pajic NNiT: width-agnostic neural network generation with structurally aligned weight spaces. External Links: 2603.00180, Link Cited by: §2. Lasby et al. (2026) M. Lasby, I. Lazarevich, N. Sinnadurai, S. Lie, Y. Ioannou, and V. Thangarasa REAP the experts: why pruning prevails for one-shot moe compression. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §4.5.1. Li et al. (2026) L. Li, Q. Zhu, J. Wang, X. Qin, W. Li, H. Gu, S. Han, and Y. Guo Sub-moe: efficient mixture-of-expert llms compression via subspace expert merging. In AAAI 2026, Cited by: §B.2, §1, §2, §4.1. Li et al. (2024) P. Li, Z. Zhang, P. Yadav, Y. Sung, Y. Cheng, M. Bansal, and T. Chen Merge, then compress: demystify efficient smoe with hints from its routing policy. External Links: 2310.01334, Link Cited by: §B.2, §1, §1, §2, §3.2, §4.1. Li et al. (2025) W. Li, L. Li, H. Gu, Y. Huang, M. G. Lee, S. Sun, W. Xue, and Y. Guo MoE-SVD: structured mixture-of-experts LLMs compression via singular value decomposition. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 35209–35230. Cited by: §2. Liu et al. (2026) J. Liu, P. Tang, W. Wang, Y. Ren, X. Hou, P. A. Heng, M. Guo, and C. Li A survey on inference optimization techniques for mixture of experts models. ACM Computing Surveys 58 (10), p. 1–37. External Links: ISSN 1557-7341, Document Cited by: §2. Miao et al. (2025) R. Miao, Y. Yao, Z. Wang, Z. Wang, B. Yi, L. Liu, Y. Zhao, and T. Yang MergeMoE: efficient compression of moe models via expert output merging. External Links: 2510.14436, Link Cited by: §1. Muennighoff et al. (2025) N. Muennighoff, L. Soldaini, D. Groeneveld, K. Lo, J. Morrison, S. Min, W. Shi, E. P. Walsh, O. Tafjord, N. Lambert, Y. Gu, S. Arora, A. Bhagia, D. Schwenk, D. Wadden, A. Wettig, B. Hui, T. Dettmers, D. Kiela, A. Farhadi, N. A. Smith, P. W. Koh, A. Singh, and H. Hajishirzi OLMoe: open mixture-of-experts language models. In The Thirteenth International Conference on Learning Representations, Cited by: §B.1, §4.1. Nguyen et al. (2026) D. V. Nguyen, A. Thi, M. H. Nguyen, L. Nguyen, S. Jiang, E. Fetaya, D. L. Tran, G. Chechik, and T. Nguyen Expert merging in sparse mixture of experts with nash bargaining. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026, p. 148047–148075. Cited by: §B.2, §2. Nguyen et al. (2025) V. D. Nguyen, M. N. Hoang, L. Nguyen, R. Teo, T. M. Nguyen, and L. D. Tran CAMEx: curvature-aware merging of experts. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1. Schürholt et al. (2022) K. Schürholt, B. Knyazev, X. Giró-i-Nieto, and D. Borth Hyper-representations as generative models: sampling unseen neural network weights. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. External Links: ISBN 9781713871088 Cited by: §2, §3.1. Schürholt et al. (2021) K. Schürholt, D. Kostadinov, and D. Borth Self-supervised representation learning on neural network weights for model characteristic prediction. In Advances in Neural Information Processing Systems, Vol. 34, p. 16481–16493. Cited by: §2. Shazeer et al. (2017) N. Shazeer, *. Mirhoseini, *. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, Cited by: §1. Soro et al. (2025) B. Soro, B. Andreis, H. Lee, W. Jeong, S. Chong, F. Hutter, and S. J. Hwang Diffusion-based neural network weights generation. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Soro et al. (2026) B. Soro, A. S. Zhang, B. Andreis, J. Jo, S. Chong, and S. J. Hwang LS-merge: merging language models in latent space. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §3.1. Team (2025) B. E. Team ERNIE 4.5 technical report. External Links: Cited by: §B.1, §4.1. Tran et al. (2025) V. Tran, T. Vo, A. Nguyen, T. Tran, M. Nguyen-Nhat, T. Tran, D. Pham, and T. Nguyen Equivariant neural functional networks for transformers. In International Conference on Learning Representations, Vol. 2025, p. 16270–16341. Cited by: §2. Wang et al. (2026) Z. Wang, P. Wang, and K. Wang Position: weight space should be a first-class generative AI modality. In Forty-third International Conference on Machine Learning Position Paper Track, External Links: Link Cited by: §2. Xie et al. (2024) Y. Xie, Z. Zhang, D. Zhou, C. Xie, Z. Song, X. Liu, Y. Wang, X. Lin, and A. Xu MoE-pruner: pruning mixture-of-experts large language model using the hints from its router. External Links: 2410.12013, Link Cited by: §2. Xu and Durrett (2018) J. Xu and G. Durrett Spherical latent spaces for stable variational autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.), Brussels, Belgium, p. 4503–4513. External Links: Link, Document Cited by: §3.1. Xu et al. (2025) Y. Xu, X. Han, Y. Zhang, Y. Wang, Y. Liu, S. Ji, Q. Zhu, and W. Che CAMERA: multi-matrix joint compression for moe models via micro-expert redundancy analysis. External Links: 2508.02322, Link Cited by: §2. Zhang et al. (2026) G. Zhang, H. Yuxuan, Y. Lou, Y. Zhang, W. Zhao, and Y. You MoNE: replacing redundant experts with lightweight novices for structured pruning of moe. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026, p. 39983–40009. External Links: Link Cited by: §2. Zhao et al. (2026) Y. Zhao, Z. Wang, and M. Zhang PuzzleMoE: efficient compression of large mixture-of-experts models via sparse expert merging and bit-packed inference. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §B.2, §2. Zhou et al. (2023a) A. Zhou, K. Yang, K. Burns, A. Cardace, Y. Jiang, S. Sokota, J. Z. Kolter, and C. Finn Permutation equivariant neural functionals. External Links: 2302.14040, Link Cited by: §2. Zhou et al. (2023b) A. Zhou, K. Yang, Y. Jiang, K. Burns, W. Xu, S. Sokota, J. Z. Kolter, and C. Finn Neural functional transformers. In Advances in Neural Information Processing Systems, Vol. 36, p. 77485–77502. External Links: Document Cited by: §2. Zong et al. (2026) Z. Zong, K. Zhang, Y. Wang, W. Tan, Y. Ren, and J. Hu Less token, more signal: moe expert pruning via critical token selection. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §2. Appendix A Limitations and Future Directions Compute constraints limit our evaluation to models with up to 21B parameters and fixed-budget experiments. In addition, as with other MoE compression methods, although SLEM recovers part of the performance lost through compression, the resulting compressed checkpoints do not yet fully match the performance of their uncompressed pretrained counterparts at higher expert reduction rates. SLEM also introduces additional design and computational costs. It requires training an expert VAE before merging, which incurs additional computation and resource overhead. Moreover, the rank used for spectral importance estimation is a hyperparameter whose optimal value can vary across models and tasks. Reducing the need for model-specific rank selection and improving the efficiency of latent-space training are therefore important directions for future work. Finally, the current expert grouping strategy is restricted to one-to-one expert pairs. This pairwise formulation limits the achievable compression under a single merging stage and does not directly support consolidating three or more experts into a single group. Extending SLEM to many-to-one merging will require grouping criteria and aggregation strategies that can characterize the compatibility and relative contributions of multiple experts jointly. Developing such group-level similarity measures and merging rules could enable higher compression ratios while preserving expert functionality. Appendix B Experimental Details We provide further details on the models, baseline implementations, and evaluation protocols used in our experiments. B.1 Models LFM2-8B-A1B (AI, 2025). LFM2-8B-A1B contains 8.3B total parameters, with approximately 1.5B parameters activated per token. Among its 24 layers, the first two employ dense feed-forward layers and the remaining 22 use sparse MoE layers. Each MoE layer consists of 32 routed experts and activates the top-4 experts for each token. The model hidden dimension is D=2048D=2048, and each routed expert has an intermediate dimension of DMoE=1792D_MoE=1792. OLMoE-1B-7B-0125-Instruct (Muennighoff et al., 2025). OLMoE-1B-7B-0125-Instruct has 6.9B total parameters and approximately 1.3B active parameters per token. All 16 layers employ sparse MoE feed-forward networks without shared experts, each containing 64 routed experts with top-8 routing. Its hidden dimension is D=2048D=2048, with an expert intermediate dimension of DMoE=1024D_MoE=1024. ERNIE-4.5-21B-A3B-PT (Team, 2025). ERNIE-4.5-21B-A3B-PT is a 21B-parameter MoE model with approximately 3B active parameters per token. It consists of 28 layers, with the first layer using a dense feed-forward network and the remaining 27 employing MoE layers. Each MoE layer contains 64 routed experts together with two shared experts, and selects the top-6 routed experts for each token. The model uses a hidden dimension of D=2560D=2560 and an MoE intermediate dimension of DMoE=1536D_MoE=1536. B.2 Baselines M-SMoE (Li et al., 2024). M-SMoE performs routing-guided expert grouping followed by frequency-weighted parameter merging. It selects dominant experts according to their activation frequencies from the calibration set and assigns the remaining experts to the most similar dominant expert. Expert similarity is estimated from router logits collected over the calibration set: router(i,j)=cos(i,j),S_router(i,j)=cos\! (r_i,r_j ), (7) where ir_i denotes the router-logit vector of expert i over the calibration tokens. Given a resulting group C, the hidden units of different experts are aligned through neuron permutation matching, and then experts are merged as W~,p=∑i∈fiWi,p∑i∈fi, W_C,p= _i f_iW_i,p _i f_i, (8) where fif_i is the activation frequency of expert i and p∈p denotes the projection type. HC-SMoE (Chen et al., 2025). HC-SMoE groups experts according to their similarity measured from expert outputs. Given a calibration set calD_cal containing T tokens, each expert is represented by its average output: i=1T∑x∈calEi(x).o_i= 1T _x _calE_i(x). (9) Starting from singleton expert clusters, it performs agglomerative hierarchical clustering using the distance between these representations. The distance between two clusters aC_a and bC_b is d(a,b)=1|a||b|∑i∈a∑j∈b‖i−j‖2.d(C_a,C_b)= 1|C_a||C_b| _i _a _j _b\|o_i-o_j\|_2. (10) The closest clusters are iteratively merged until the target number of experts is reached, and the experts within each cluster are merged according to equation 8. Sub-MoE (Li et al., 2026). Sub-MoE groups functionally similar experts using their outputs on the calibration set. For a pair of experts, the functional similarity is defined as the average cosine similarity: out(i,j)=1T∑x∈calEi(x)⊤Ej(x)‖Ei(x)‖2‖Ej(x)‖2,S_out(i,j)= 1T _x _cal E_i(x) E_j(x)\|E_i(x)\|_2\|E_j(x)\|_2, (11) which is used to form expert clusters through K-means clustering. Afterward, the experts within each cluster are aligned in a shared subspace. For projection p, the weight matrices of experts in a cluster C are jointly decomposed as SVD([Wi1,p,…,Wi||,p])=UpΣp[Vi1,p;…;Vi||,p]⊤.SVD ([W_i_1,p,…,W_i_|C|,p] )=U_p _p[V_i_1,p;…;V_i_|C|,p] . (12) The expert-specific V components are then combined according to their activation frequencies, V~,p=∑i∈fiVi,p∑i∈fi, V_C,p= _i f_iV_i,p _i f_i, (13) and the merged projection is reconstructed in the shared subspace as W~,p=UpΣpV~,p⊤. W_C,p=U_p _p V_C,p . (14) Frequency-based Merging. Frequency-based Merging (F-Merging) uses expert activation frequency as both the grouping and merging signal. Given a calibration set, we compute the activation frequency of each expert. Experts are then paired in a one-to-one manner by matching the most frequently activated expert with the least frequently activated expert. The selected pair are merged according to equation 8, and are removed from further pairing. This procedure is repeated until the target number of experts is reached. We do not include PuzzleMoE (Zhao et al., 2026) because its deployment pipeline differs substantially from the standard expert setting considered in our experiments. We also exclude NAMEx (Nguyen et al., 2026) because its original formulation does not target reducing the number of stored experts under a fixed expert-reduction budget, making it not directly comparable to our compression setting. Table 3: Benchmark-specific evaluation settings, including the evaluation metric, few-shot setting, maximum sequence length, and maximum generation tokens. Task Metric Few-shot Maximum Sequence Length Maximum Generation Tokens General Knowledge (MCQA) MMLU (Hendrycks et al., 2021a) Accuracy 0-shot 4,0964,096 N/A ARC-Easy (Clark et al., 2018) Accuracy 0-shot 4,0964,096 N/A ARC-Challenge (Clark et al., 2018) Accuracy 0-shot 4,0964,096 N/A Mathematics (Generation) GSM8K (Cobbe et al., 2021) Exact Match 8-shot (single-turn) 4,0964,096 1,0241,024 MATH500 (Hendrycks et al., 2021b) Exact Match 4-shot (single-turn) 4,0964,096 1,0241,024 Coding (Generation) MBPP (Austin et al., 2021) Pass@1 3-shot (single-turn) 4,0964,096 1,0241,024 HumanEval (Chen et al., 2021) Pass@1 0-shot 4,0964,096 1,0241,024 Table 4: Architecture configurations of the hyperspherical expert VAE for each evaluated model. Training and optimization hyperparameters are provided in Appendix C. Model # layers (enc./dec.) Demb,hidden,latentD_\emb,\,hidden,\,latent\ DFFND_FFN # attn. heads Pre-LN architecture LFM2-8B-A1B 6 3,0723,072 4×3,0724× 3,072 24 yes OLMoE-1B-7B-0125-Instruct 6 3,0723,072 4×3,0724× 3,072 24 yes ERNIE-4.5-21B-A3B-PT 6 3,8403,840 4×3,8404× 3,840 30 yes B.3 Evaluation Protocols We conduct all evaluations using lm_eval framework (Gao et al., 2024). For MCQA tasks, each question and its candidate options are provided as input, and the option with the highest probability is selected as the prediction. For generation-based tasks, the model generates responses autoregressively, after which the final answer is extracted using task-specific regular expressions. The benchmark-specific evaluation settings are summarized in Table 3. All experiments are conducted on a single NVIDIA RTX PRO 6000 Blackwell Max-Q GPU (96GB) using bfloat16 precision without quantization. Due to computational constraints, we use greedy decoding and report results from a single evaluation run unless otherwise specified. Appendix C Hyperspherical Expert VAE Training We provide further details on the hyperspherical expert VAE training, including model-specific architecture configurations, training and optimization hyperparameters, and the computational resources used for training. Training configurations. Table 4 summarizes the model-specific architecture configurations of the hyperspherical Expert VAE, including the number of encoder and decoder layers, representation dimensions, feed-forward dimensions, and attention heads. All models use a transformer-based encoder and decoder with a Pre-LN architecture. For training, we use a von Mises–Fisher (vMF) latent distribution with concentration parameter κ=524,288κ=524,288 and KL weight β=10−3β=10^-3. Expert weights are standardized independently within each MoE layer, and layer-specific priors are initialized from the same randomly sampled direction. We train each VAE for 200 epochs with a batch size of 4, learning rate 5×10−55× 10^-5, weight decay 10−610^-6, and gradient clipping at 1.0. We use a cosine learning rate scheduler and bfloat16 precision training. Computational resources. We train each Expert VAE on a single NVIDIA H200 GPU (140GB). The training cost depends on the model size and VAE configuration; for the largest evaluated model, ERNIE-4.5-21B-A3B-PT, training takes ≈18.7≈ 18.7 hours. Appendix D Analysis of SLEM Design Variants We analyze alternative design choices of SLEM. Specifically, we compare variants that modify the scope of expert grouping and merging coefficient schemes. Table 5: Comparison of SLEM design variants for expert grouping and merging coefficients. We evaluate alternative grouping strategies, uniform and row-norm coefficients, and projection-specific spectral importance using only gate, up, or down projections. All variants are compared with the default SLEM under the same expert reduction setting. General Knowledge (MCQA) Mathematics & Coding (Generation) Expert Reduction Method MMLU ARC-E ARC-C Average ↑ GSM8K MATH500 MBPP HumanEval Average ↑ LFM2-8B-A1B 25.0% SLEM (r=14) 51.10 70.62 42.75 54.82 70.36 58.80 36.20 61.59 56.74 ⌞\; within-layer adaptive 53.12 67.85 41.81 54.26↓0.56 0.56 68.84 60.00 22.00 43.90 48.69↓8.05 8.05 ⌞\; cross-layer adaptive 52.98 69.19 43.60 55.26↑0.44 0.44 68.39 60.40 34.40 55.49 54.67↓2.07 2.07 ⌞\; uniform coefficient 50.48 68.01 41.47 53.32↓1.50 1.50 68.92 58.80 35.40 57.32 55.11↓1.63 1.63 ⌞\; row-norm coefficient 50.71 68.39 41.47 53.52↓1.30 1.30 67.70 60.20 35.40 57.32 55.15↓1.59 1.59 ⌞\; spectral importance: gate-only 52.14 69.11 41.04 54.10↓0.72 0.72 69.83 59.80 30.20 51.83 52.91↓3.83 3.83 ⌞\; spectral importance: up-only 51.58 69.74 42.32 54.55↓0.27 0.27 68.69 59.80 30.40 54.88 53.44↓3.30 3.30 ⌞\; spectral importance: down-only 51.47 67.68 41.55 53.57↓1.25 1.25 71.49 57.80 36.20 62.20 56.92↑0.18 0.18 OLMoE-1B-7B-0125-Instruct 12.5% SLEM (r=16) 47.37 75.80 45.90 56.36 59.74 14.20 20.00 25.00 29.74 ⌞\; within-layer adaptive 46.25 74.41 44.88 55.18↓1.18 1.18 49.58 09.80 18.00 19.51 24.22↓5.52 5.52 ⌞\; cross-layer adaptive 43.60 75.13 46.08 54.94↓1.42 1.42 53.83 10.20 19.00 19.51 25.64↓4.10 4.10 ⌞\; uniform coefficient 44.25 63.89 38.40 48.84↓7.52 7.52 38.82 08.20 10.40 04.88 15.57↓14.17 14.17 ⌞\; row-norm coefficient 46.02 71.97 42.32 53.44↓2.92 2.92 50.87 12.00 15.80 14.02 23.17↓6.57 6.57 ⌞\; spectral importance: gate-only 47.02 74.49 44.80 55.44↓0.92 0.92 59.44 11.80 21.00 21.95 28.55↓1.19 1.19 ⌞\; spectral importance: up-only 47.77 75.80 45.56 56.38↑0.02 0.02 58.38 14.00 22.60 21.95 29.23↓0.51 0.51 ⌞\; spectral importance: down-only 48.07 76.05 45.22 56.45↑0.09 0.09 58.15 14.80 18.60 25.61 29.29↓0.45 0.45 ERNIE-4.5-21B-A3B-PT 25.0% SLEM (r=12) 53.0953.09 76.2676.26 46.7546.75 58.7058.70 72.2572.25 44.40 46.80 51.83 53.82 ⌞\; within-layer adaptive 53.27 76.14 43.26 57.55↓1.15 1.15 63.08 40.20 42.40 39.63 46.33↓7.49 7.49 ⌞\; cross-layer adaptive 51.84 74.92 43.52 56.76↓1.94 1.94 71.57 42.00 43.40 53.66 52.66↓1.16 1.16 ⌞\; uniform coefficient 44.62 73.23 41.04 52.96↓5.74 5.74 66.11 40.60 40.00 32.93 44.91↓8.91 8.91 ⌞\; row-norm coefficient 51.46 74.75 43.86 56.69↓2.01 2.01 71.27 44.00 47.60 51.83 53.68↓0.14 0.14 ⌞\; spectral importance: gate-only 52.86 75.97 44.54 57.79↓0.91 0.91 71.72 38.20 45.20 50.61 51.43↓2.39 2.39 ⌞\; spectral importance: up-only 52.41 74.37 44.03 56.94↓1.76 1.76 70.43 45.20 46.00 46.95 52.15↓1.67 1.67 ⌞\; spectral importance: down-only 54.93 77.40 45.39 59.24↑0.54 0.54 71.49 42.00 42.20 53.66 52.34↓1.48 1.48 D.1 Expert Grouping Variants SLEM constrains the maximum number of merged experts assigned to each layer, preventing merge pairs from being overly concentrated in a small subset of layers. Here, we further examine the role of this layer-wise constraint by removing the maximum merged-expert limit. We consider two variants: within-layer adaptive merging, which selects expert pairs only within each layer, and cross-layer adaptive merging, which allows pairs to be selected across layers. As shown in Table 5, removing the layer-wise constraint generally degrades performance across the three models, with a particularly large drop for within-layer adaptive merging. We hypothesize that this degradation arises from an imbalanced distribution of merge operations across layers. Since expert redundancy, as reflected by latent similarity, is not necessarily uniform across layers, unconstrained similarity-based grouping can concentrate many merges in a small subset of layers, disproportionately reducing their expert capacity while leaving others largely unchanged. This effect is more pronounced for within-layer adaptive merging, where the candidate pool is restricted to each individual layer and highly similar local experts can be repeatedly selected. Cross-layer adaptive merging partially alleviates this issue by providing a broader pool of candidate experts, but still underperforms the balanced grouping used by SLEM. These results suggest that the layer-wise constraint acts as an effective regularizer, preserving balanced expert capacity across the layers while retaining the flexibility of cross-layer merging. D.2 Merging Coefficient Variants SLEM determines subnetwork-specific merging coefficients from the spectrum of pretrained expert weights without relying on calibration data. Here, we examine alternative calibration-free coefficient schemes to understand the contribution of spectral truncation and the aggregation of importance across projection types. Uniform coefficient and row-norm coefficient. A simple calibration-free strategy is to assign equal contributions to the two matched subnetworks. We therefore consider a uniform coefficient of [0.5,0.5][0.5,0.5]. As another alternative, we use the squared row norm of each projection weight as the subnetwork importance, si,p,mnorm=∥Wi,p[m,:]∥22‖Wi,p‖F2s_i,p,m^norm= \|W_i,p[m,:]\|_2^2\|W_i,p\|_F^2, followed by the same projection-wise averaging and coefficient computation used in SLEM. By the SVD identity, this corresponds exactly to the full-rank case of our spectral importance. As shown in Table 5, both uniform and row-norm coefficients generally underperform our spectrum-aware coefficients. The uniform variant ignores differences in subnetwork importance entirely, while the row-norm variant incorporates the full weight energy without distinguishing dominant spectral directions. These results suggest that concentrating importance estimation on the leading singular directions provides a more informative signal for assigning the relative contributions of matched subnetworks. Projection type selection. SLEM computes spectral importance independently for the gate, up, and down projections and averages them to obtain a unified subnetwork importance score. To examine the contribution of this aggregation, we evaluate variants that derive merging coefficients from only one projection type at a time. Table 5 shows that the effectiveness of each individual projection varies substantially across tasks. These results indicate that each projection captures complementary aspects of subnetwork importance. Averaging their importance scores therefore yields a more balanced estimate, resulting in more robust performance across task categories than relying on any single projection alone. Table 6: Downstream performance comparison with and without KL regularization during Expert VAE training. The VAE with β=0β=0 variant removes the KL divergence term while keeping the remaining training settings unchanged. General Knowledge (MCQA) Mathematics & Coding (Generation) Expert Reduction Method MMLU ARC-E ARC-C Average ↑ GSM8K MATH500 MBPP HumanEval Average ↑ LFM2-8B-A1B 0.0% Vanilla 63.1163.11 75.5975.59 47.6347.63 62.1162.11 81.7381.73 72.4072.40 45.6045.60 70.7370.73 67.6267.62 VAE-reconstructed 62.93±0.2162.93_ ± 0.21 75.61±0.4575.61_ ± 0.45 47.41±0.6147.41_ ± 0.61 61.98±0.3261.98_ ± 0.32 82.35±0.3082.35_ ± 0.30 72.20±0.6472.20_ ± 0.64 45.52±1.1445.52_ ± 1.14 71.83±3.1771.83_ ± 3.17 67.97±0.7267.97_ ± 0.72 ⌞\; VAE with β=0 62.94±0.1962.94_ ± 0.19 75.14±0.5075.14_ ± 0.50 47.25±0.6147.25_ ± 0.61 61.78±0.1961.78_ ± 0.19 82.02±0.4982.02_ ± 0.49 72.68±1.8472.68_ ± 1.84 45.56±1.5145.56_ ± 1.51 69.39±1.1869.39_ ± 1.18 67.41±0.6267.41_ ± 0.62 12.5% SLEM (r=14) 58.37 74.79 47.87 60.34 77.10 67.60 44.20 62.80 62.93 ⌞\; VAE with β=0 54.32 68.48 42.41 55.07↓5.27 5.27 78.54 60.80 29.00 39.63 51.99↓10.94 10.94 25.0% SLEM (r=14) 51.10 70.62 42.75 54.82 70.36 58.80 36.20 61.59 56.74 ⌞\; VAE with β=0 44.15 64.27 38.48 48.97↓5.85 5.85 68.76 45.40 17.60 32.93 41.17↓15.57 15.57 OLMoE-1B-7B-0125-Instruct 0.0% Vanilla 53.4553.45 76.4876.48 47.1747.17 59.0359.03 68.9968.99 23.0023.00 29.6029.60 35.3735.37 39.2439.24 VAE-reconstructed 53.21±0.1053.21_ ± 0.10 76.36±0.5676.36_ ± 0.56 47.39±0.7547.39_ ± 0.75 58.98±0.3258.98_ ± 0.32 68.89±0.8968.89_ ± 0.89 23.64±1.0823.64_ ± 1.08 28.40±0.8928.40_ ± 0.89 33.54±1.0233.54_ ± 1.02 38.62±0.4238.62_ ± 0.42 ⌞\; VAE with β=0 53.37±0.1653.37_ ± 0.16 76.26±0.5176.26_ ± 0.51 47.30±1.1447.30_ ± 1.14 58.98±0.5358.98_ ± 0.53 68.19±0.6268.19_ ± 0.62 22.96±0.4822.96_ ± 0.48 29.28±1.1329.28_ ± 1.13 33.66±0.9833.66_ ± 0.98 38.52±0.4238.52_ ± 0.42 12.5% SLEM (r=16) 47.37 75.80 45.90 56.36 59.74 14.20 20.00 25.00 29.74 ⌞\; VAE with β=0 42.68 71.46 44.03 52.72↓3.64 3.64 52.08 13.60 17.80 15.24 24.68↓5.06 5.06 25.0% SLEM (r=16) 30.33 69.95 40.87 47.05 23.12 06.60 10.40 05.49 11.40 ⌞\; VAE with β=0 33.59 62.12 35.58 43.76↓3.29 3.29 14.10 03.60 06.00 01.22 06.23↓5.17 5.17 Appendix E Effect of KL Regularization on Expert VAE Training We analyze how KL regularization during Expert VAE training affects subsequent expert merging. For this purpose, we train an additional VAE with the same posterior sampling procedure but set the KL weight to β=0β=0, while keeping the other training configurations unchanged. As shown in Table 6, both VAEs achieve similar reconstruction performance on LFM2 and OLMoE. However, after latent-expert merging, the VAE trained without KL regularization shows substantially worse downstream performance. This indicates that (i) preserving reconstruction quality alone is not sufficient for effective latent interpolation, and (i) KL regularization encourages the posterior representations to remain organized around the shared prior, while maintaining reconstruction performance. Consequently, KL regularization yields a latent space where interpolated representations remain more compatible with the decoder, leading to more effective expert merging. Appendix F Paired Bootstrap Confidence Intervals Since SLEM shows particularly strong performance on generation-based benchmarks involving mathematical reasoning and code generation, we further assess the statistical reliability of these gains on LFM2-8B-A1B. We compare SLEM with HC-SMoE, the strongest competing baseline, using paired bootstrap confidence intervals. For each benchmark, we resample the evaluation instances with replacement while preserving the pairing between the two methods. We perform 200,000 bootstrap resamples and report the 95% confidence interval (CI) of the performance difference in percentage points. As shown in Table 7, the confidence intervals are consistently positive for GSM8K, MATH500, and MBPP at both expert reduction rates. HumanEval includes zero at 12.5% reduction but becomes clearly positive at 25.0%. The confidence interval of the average generation performance also excludes zero at both reduction rates, supporting the statistical reliability of the overall generation gains of SLEM over HC-SMoE. Table 7: Paired bootstrap 95% confidence intervals (p) of the performance difference between SLEM and HC-SMoE on LFM2-8B-A1B generation benchmarks. We use 200,000 bootstrap resamples. Positive intervals favor SLEM. Expert Reduction GSM8K MATH500 MBPP HumanEval Average n=1,319n=1,319 n=500n=500 n=500n=500 n=164n=164 12.5% [9.10, 14.33] [7.60, 15.60] [3.40, 11.00] [-3.05, 12.80] [6.38, 11.31] 25.0% [13.19, 18.65] [9.40, 17.80] [1.00, 9.80] [7.93, 23.78] [10.07, 15.32] Appendix G Cumulative EVR of Expert Weights We provide additional cumulative explained variance ratio (EVR) analyses for the other evaluated models. As shown in Figure 7, their expert projection weights exhibit trends similar to those observed in the main text: variance is consistently concentrated along a subset of principal directions rather than being uniformly distributed across the spectrum. This suggests that the dominant spectral structure of expert weights is a common characteristic across the evaluated MoE models, further supporting the use of spectral information for estimating subnetwork importance. Appendix H Distribution of Normalized Spectral Importance Figure 8 visualizes the normalized spectral importance across subnetworks under different ranks r for each evaluated model. As r varies, the importance distribution changes in concentration, indicating that the truncation rank controls how selectively spectral energy is assigned across subnetworks. (a) OLMoE-1B-7B-0125-Instruct (b) ERNIE-4.5-21B-A3B-PT Figure 7: Culmulative explained variance ratio (EVR) of expert projection weights across layers in other evaluated models. Figure 8: Normalized spectral importance across subnetworks under different truncation ranks r. Lower ranks produce more concentrated importance distributions, while larger ranks distribute importance more uniformly across subnetworks.