Paper deep dive
Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression
Tianxiao Cao, Jiahe Shao, Yuning Qiu, Kyohei Atarashi, Hisashi Kashima, Qibin Zhao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/8/2026, 7:33:03 AM
Summary
The paper introduces Shared Low-rank Basis Factorization (SLBF), a data-free weight reconstruction method for compressing Mixture-of-Experts (MoE) large language models. The authors analyze three compression families (pruning, merging, weight reconstruction), deriving structural error bounds that show pruning and merging incur non-vanishing errors due to routing and expert heterogeneity, whereas weight reconstruction preserves structure. SLBF improves upon existing weight reconstruction methods (like MoBE) by using shared low-rank bases instead of full-rank bases, allowing for richer cross-expert sharing and lower reconstruction error under fixed budgets. A post-hoc gauge fixing removes redundant parameters. SLBF is evaluated on five MoE architectures (16B to 122B parameters) and consistently outperforms methods from all three compression families.
Entities (9)
Relation Signals (9)
SLBF → isa → Weight Reconstruction
confidence 95% · we propose Shared Low-rank Basis Factorization (SLBF), a data-free weight reconstruction method
SLBF → outperforms → MoBE
confidence 92% · SLBF consistently outperforms methods from all three compression families... MoBE formulation... shows significant strengths... To resolve this limitation, we propose SLBF
Weight Reconstruction → avoids → Structural Costs
confidence 90% · weight reconstruction avoids these structural costs by preserving expert structure and routing.
SLBF → outperforms → Expert Merging
confidence 90% · SLBF consistently outperforms methods from all three compression families.
SLBF → outperforms → Expert Pruning
confidence 90% · SLBF consistently outperforms methods from all three compression families.
SLBF → uses → Low-rank Bases
confidence 90% · SLBF... uses rank-k bases shared among experts
Expert Pruning → hasstructuralerror → Non-vanishing Error
confidence 85% · pruning and merging can incur non-vanishing errors tied to routing and expert heterogeneity.
Expert Merging → hasstructuralerror → Non-vanishing Error
confidence 85% · pruning and merging can incur non-vanishing errors tied to routing and expert heterogeneity.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mixture-of-Experts (MoE) large language models decouple capacity from compute through sparse routing, but their large parameter count creates storage and serving challenges. We analyze three MoE compression families: expert pruning, expert merging, and weight reconstruction, and derive structural error bounds showing that pruning and merging can incur non-vanishing errors tied to routing and expert heterogeneity. In contrast, weight reconstruction avoids these structural costs by preserving expert structure and routing. Motivated by the analysis, we propose Shared Low-rank Basis Factorization (SLBF), a data-free weight reconstruction method that uses rank-$k$ bases shared among experts, enabling richer cross-expert sharing, faster convergence, and lower reconstruction error. A post-hoc gauge fixing removes redundant parameters at no representational cost. Across five MoE architectures spanning 16B to 122B parameters, SLBF consistently outperforms methods from all three compression families.
Tags
Links
- Source: https://arxiv.org/abs/2610.09342v1
- Canonical: https://arxiv.org/abs/2610.09342v1
Trouble viewing inline? Open PDF directly →
Full Text
82,283 characters extracted from source content.
Expand or collapse full text
Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression Tianxiao Cao †thanks: Work done primarily while at RIKEN AIP. Affiliation: Kyoto University Email: cao.tianxiao.65w@st.kyoto-u.ac.jp Jiahe Shao Affiliation: The University of Tokyo Yuning Qiu Affiliation: RIKEN AIP Kyohei Atarashi Affiliation: Kyoto University Hisashi Kashima Affiliation: Kyoto University Qibin Zhao Affiliation: RIKEN AIP Abstract Mixture-of-Experts (MoE) large language models decouple capacity from compute through sparse routing, but their large parameter count creates storage and serving challenges. We analyze three MoE compression families: expert pruning, expert merging, and weight reconstruction, and derive structural error bounds showing that pruning and merging can incur non-vanishing errors tied to routing and expert heterogeneity. In contrast, weight reconstruction avoids these structural costs by preserving expert structure and routing. Motivated by the analysis, we propose Shared Low-rank Basis Factorization (SLBF), a data-free weight reconstruction method that uses rank-k bases shared among experts, enabling richer cross-expert sharing, faster convergence, and lower reconstruction error. A post-hoc gauge fixing removes redundant parameters at no representational cost. Across five MoE architectures spanning 16B to 122B parameters, SLBF consistently outperforms methods from all three compression families. 1 Introduction The Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Shazeer et al., 2017) has become the dominant paradigm for scaling large language models, adopted by a rapidly growing family of production LLMs (Jiang et al., 2024; Liu et al., 2024a; Team, 2025; Google, 2026). By decoupling total capacity from compute through sparse routing, MoE enables models with hundreds of billions of parameters while maintaining efficient inference. Sparse activation solves the compute problem but not the memory problem. Every expert must reside in accelerator memory regardless of how rarely it is activated: Qwen3-235B-A22B deploys 128 expert FFNs per layer yet routes each token to only 8, leaving over 93% of expert parameters idle per forward pass but occupying accelerator memory. At larger scales, the situation is acute: deploying DeepSeek-V3 (671B) requires distributing expert shards across dozens of GPUs, and the storage and transfer cost of expert weights dominates both the serving budget and the model distribution pipeline. To reduce their memory footprint while preserving the routing structure that enables sparse activation, compressing the expert parameters of MoE-based LLMs is therefore a practical and urgent problem. Existing approaches to MoE compression intervene at one of three structural levels: expert pruning (Lasby et al., 2026); expert merging (Chen et al., 2025a); and weight reconstruction (Liu et al., 2025b; Chen et al., 2026). We derive structural error bounds for pruning and merging, and show that both can incur non-vanishing errors tied to routing and expert heterogeneity in modern MoE architectures where load balancing encourages broad expert utilization and expert specialization induces functional diversity. By preserving both expert structure and routing, weight reconstruction avoids these intervention-induced structural costs, while its remaining error arises from imperfect parameter reconstruction. Within the weight reconstruction family, the state-of-the-art MoBE formulation Chen et al. (2026) shows significant strengths by parameterizing expert weights as non-linear mixtures of shared full-rank bases. However, scaling these full-rank bases rapidly consumes the parameter budget, creating a bottleneck that limits the model’s representational capacity. To resolve this limitation, we propose Shared Low-rank Basis Factorization (SLBF), which replaces each full-rank basis with a rank-k factorization jjj=1m\U_jV_j\_j=1^m. The lower per-basis cost allows more bases under the same compression budget. This shift from a few full-rank bases to many low-rank bases enables faster convergence, a richer mixing structure where experts compose across multiple bases rather than concentrating on a single dominant one, and ultimately lower weight reconstruction error. A post-hoc gauge fixing eliminates mk2mk^2 redundant parameters at no representational cost. SLBF is entirely data-free and requires no fine-tuning after compression. In summary, our main contributions are: • We derive a unified structural cost analysis of three MoE compression families, with new error bounds for expert pruning and weight reconstruction, showing that weight reconstruction avoids the intervention-induced structural costs of pruning and merging. • We propose the Shared Low-rank Basis Factorization with bilinear gauge fixing that enables richer cross-expert basis sharing under fixed compression budgets. • We evaluate SLBF on five MoE architectures spanning 16B to 122B parameters against competing methods from all three families, demonstrating consistent improvements. 2 Related Work MoE architecture. Originally proposed by Jacobs et al. (1991) and revived for deep learning by Shazeer et al. (2017), the sparse MoE design enables scaling of parameters without proportional increase in compute (Lepikhin et al., 2021; Fedus et al., 2022). Modern MoE-based LLMs adopt MoE FFNs (Jiang et al., 2024; Liu et al., 2024a; Team, 2025; Google, 2026) with refinements including load balancing (Wang et al., 2024), shared experts (Dai et al., 2024), and fine-grained granularity with hundreds of experts per layer (Team, 2025). These developments have made MoE a leading architecture for scaling LLM capacity. Expert merging. Expert merging groups experts and combines the weights within each group into shared representatives, reducing the number of distinct expert parameter sets. Li et al. (2024b) groups experts by router-logit similarity and merges via frequency-weighted averaging; Chen et al. (2025a) improves clustering quality through hierarchical agglomerative clustering on expert output vectors; Jha et al. (2026) improves grouping quality by using activation-based saliency scores. Expert pruning. Expert pruning reduces MoE model size by removing entire experts based on a saliency criterion. Early work required task-specific fine-tuning (Chen et al., 2022), while subsequent methods operate in retraining-free settings using criteria such as router frequency, reconstruction loss minimization Lu et al. (2024), gradient-free search Liu et al. (2024b), continuous relaxation of the pruning decision (Bai et al., 2025), and expert activation norms (Jaiswal et al., 2025). REAP (Lasby et al., 2026) combines router values with activation norms into the pruning criterion. Weight reconstruction. Weight reconstruction compresses expert weight matrices while preserving all experts and routing. MoLAE (Liu et al., 2025b) applies SVD to represent each expert through a shared latent matrix; D2-MoE (Gu et al., 2025) decomposes weights into shared and delta components; MoBE (Chen et al., 2026) re-parameterizes each expert’s right factor as a non-linear mixture of shared basis matrices. Other compression techniques. Beyond the above three families, quantization (Duanmu et al., 2025; Chen et al., 2025b) and weight pruning (Xie et al., 2024) have also been adapted to MoE compression. Complementary post-pruning techniques, such as converting pruned experts into bias vectors (Zong et al., 2026) or replacing them with sub-networks (Zhang et al., 2026), can further compensate for compression loss. These approaches operate along a different axis from the interventions considered here and are outside the scope of our structural analysis. (a) Pair-level structural costs. (b) Realized per-layer compression error Figure 1: (a) Pair-level structural costs ℰ¯pirr E_p^irr (pruning, left column) and ℰ¯mirr E_m^irr (merging, right column) per MoE layer on the uncompressed model, averaged over expert pairs activated on the calibration set. (b) Realized per-layer compression error ‖δℓ‖2/‖ℓorig‖2\| _ \|^2/\|h_ ^orig\|^2 at 25% MoE compression for REAP (pruning), HC-SMoE (merging), and SLBF (weight reconstruction). Top: Qwen3 (n=128n=128, top-8). Bottom: Mixtral (n=8n=8, top-2). 3 Structural Motivation MoE setup. We focus on sparse Mixture-of-Experts (MoE) models. Each feed-forward network (FFN) in the Transformer is treated as one expert. Modern MoE-based LLMs Jiang et al. (2024); Liu et al. (2024a); Team (2025); Google (2026) typically implement each expert as a SwiGLU-style FFN. A sparse MoE layer consists of a set of n experts fi(⋅)i=1n\f^i(·)\_i=1^n and a router. Specifically, the i-th expert is formulated as: fi()=(SiLU(gatei)⊙(upi))downi,f^i(x)= (SiLU(xW^i_gate) (xW^i_up) )W^i_down, (3.1) where ∈ℝdx ^d is the input hidden state; gateiW^i_gate, upi∈ℝd×dffW^i_up ^d× d_f, and downi∈ℝdff×dW^i_down ^d_f× d are the weight matrices. The router assigns hidden states to experts by applying a Top-K selection over a linear projection of the hidden states, followed by a Softmax normalization. This router mechanism can be formulated as: =Softmax(g), =Softmax(xW_g), (3.2) () (x) =argTopK(,K), =argTopK(g,K), () (x) =∑i∈gifi(), = _i g_if^i(x), where g∈ℝd×nW_g ^d× n parameterizes the linear projection of the router. 3.1 Three interventions, three structural costs Setup. We analyze the structural cost of each family (pruning, merging, and weight reconstruction, as introduced in Section 2) in the simplified setting of compressing a pair of experts (fi,fj)(f^i,f^j) with respective routing gi(),gj()g_i(x),g_j(x), adopting the pairwise analysis framework of Lasby et al. (2026). Each compression family modifies the expert pair in a distinct way, giving rise to a different source of compression error. We summarize the three interventions below; detailed derivations are provided in Section A.1. We call the minimum residual under each specified pairwise intervention its irreducible structural cost, denoting those of pruning and merging by ℰpirrE^irr_p and ℰmirrE^irr_m, respectively. Merging (following Lasby et al. (2026)). The pair contribution gi()fi()+gj()fj()g_i(x)f^i(x)+g_j(x)f^j(x) admits the decomposition (gi+gj)(r()fi+(1−r())fj)(g_i+g_j) (r(x)f^i+(1-r(x))f^j ), where r():=gi/(gi+gj)∈[0,1]r(x):=g_i/(g_i+g_j)∈[0,1] is the input-dependent mixing ratio. Merging replaces (fi,fj)(f^i,f^j) with a single representative f~ij f^ij and replaces their separate routing weights with the summed routing (gi+gj)(g_i+g_j). Even with the optimal static mixing coefficient α⋆α , a residual error remains because the input-dependent r()r(x) is replaced by a static mixture. Under a weak-correlation approximation of the three multiplicative factors, this structural cost is ℰmirr≈[(gi+gj)2]⋅Var[r()]⋅‖Δij‖2,E_m^irr _x [(g_i+g_j)^2 ]·Var [r(x) ]·E_x\| _ij\|^2, (3.3) where Δij():=fi()−fj() _ij(x):=f^i(x)-f^j(x). Remark. Under this approximation, the merging cost grows with the expert functional gap ‖Δij‖2\| _ij\|^2, router variability Var[r]Var[r], and pair routing mass (gi+gj)2(g_i+g_j)^2. It vanishes when routing is static on the pair (Var[r]=0Var[r]=0) or the experts are functionally identical (Δij=0 _ij=0). Pruning. We derive a novel bound for the pairwise pruning intervention. This intervention removes one expert from the pair (fi,fj)(f^i,f^j). Without loss of generality, suppose expert j is removed and its contribution is compensated solely by the surviving expert i through a modified router gi′(⋅)g _i(·) Even with the oracle routing function gi⋆g _i which minimizes the output loss after pruning expert j, we show that expert pruning exhibits an error of ℰpirr=[gj2(‖fj‖2−|⟨fj,fi⟩|2‖fi‖2)].E^irr_p=E_x [g^2_j (\|f_j\|^2- | f_j,f_i |^2\|f_i\|^2 ) ]. (3.4) Remark. The cost is positive when, on a non-trivial subset of inputs, expert-j is active (gj>0g_j>0) and its output is not parallel to the substitute’s (fj|fif_j f_i). Weight Reconstruction. Unlike pruning and merging, weight reconstruction preserves both expert identities and routing, and instead compresses each expert weight tensor kW_k through a reduced-parameter representation ^k W_k. For simplified linear experts fk()=kf^k(x)=xW^k, the resulting output error is bounded directly by the router-weighted reconstruction residual: ℰw≤∑k=1n[gk‖2]‖k−^k‖F2. _w≤ _k=1^nE_x [g_k\|x\|^2 ]\|W^k- W^k\|_F^2. (3.5) An analogous bound holds for SwiGLU FFN experts (Appendix A.1), with a prefactor determined by the original weight norms and the Lipschitz constant of the activation. Remark. In contrast to the preceding interventions, this bound contains no additional structural term induced by removing experts or tying their routing. The error is instead controlled by the quality of parameter reconstruction. Regime in modern MoE-based LLMs. The structural terms in Equations 3.3 and 3.4 are particularly relevant when routing is input-dependent and experts are functionally heterogeneous. Modern MoEs encourage broad expert utilization through load-balancing mechanisms (Lepikhin et al., 2021; Wang et al., 2024), while prior studies report substantial functional specialization among trained experts (Li et al., 2025; Lasby et al., 2026). These properties suggest that the structural costs identified by the pairwise analysis need not be negligible in pretrained MoEs. We examine this empirically. 3.2 Empirical evidence for three interventions Next, we connect our derivations to empirical observations on two MoE-based LLMs: Qwen3-30B-A3B-Instruct-2507 (n=128n=128, top-8; high granularity) and Mixtral-8x7B-v0.1 (n=8n=8, top-2; low granularity). We collect activations from 128 sequences × 2048 tokens from the C4 corpus Raffel et al. (2020). Structural costs are empirically non-trivial. We first verify that the structural costs in Equations 3.3 and 3.4 are non-trivial in pretrained MoEs. Figure 1(a) reports ℰ¯pirr E_p^irr and ℰ¯mirr E_m^irr per MoE layer on the original models. We observe that (i) both quantities are strictly above zero across every MoE layer in both architectures, showing that the structural terms are non-vacuous for these models; (i) both grow ∼ 3–4 orders of magnitude with depth, consistent with expert specialization concentrating in deep layers (Lasby et al., 2026; Bandarkar et al., 2026) — precisely where compression incurs the highest cost; (i) within each model the two quantities trace similar shapes across layers, suggesting that both quantities are shaped by common layerwise changes in routing and expert geometry. Activations drift in compressed models. We examine whether the distinctions highlighted by the structural analysis are reflected in realized compression error. To address this, we apply one representative method per family at 25% MoE compression: REAP (Lasby et al., 2026) for pruning, HC-SMoE (Chen et al., 2025a) for merging, and SLBF (Section 4) for weight reconstruction. For each compressed model, we measure the per-layer relative MoE-block output discrepancy ‖δℓ‖2/‖ℓorig‖2\| _ \|^2/\|h_ ^orig\|^2. Figure 1(b) reports the results. SLBF achieves the lowest discrepancy across all layers on both architectures, consistent with its preservation of expert identities and routing. The relative ordering of REAP and HC-SMoE changes with expert granularity. On Qwen3, HC-SMoE exhibits larger drift, consistent with the difficulty of tying many distinct experts to shared representatives, consistent with prior observations (Lasby et al., 2026); on Mixtral, REAP exhibits larger drift, where the smaller co-active expert set offers fewer substitutes for a removed expert. SLBF remains below both across the two regimes. A Bwnnrrd2d_2d1d_1Φ(⋅) [rgb]0.61,0.61,0.61 (·)mA UVwnnrrd2d_2d1d_1Φ(⋅) [rgb]0.61,0.61,0.61 (·)kkmm (a) Tensor diagrams. (b) Convergence Profile. (c) Mixing weight heatmaps. Figure 2: SLBF vs. MoBE on an MoE layer. (a) Tensor diagrams of the MoBE (blue, top) and SLBF (orange, bottom) parameterizations; nodes are tensors, edges are indices, and shared edges denote contraction. (b) Reconstruction loss during training on Qwen3-30B-A3B, layer 7, gate: SLBF reaches MoBE’s loss in ∼ 2k iterations and improves by >>20% by 20k iterations, despite running for fewer steps. (c) Mixing weight wijw_ij heatmaps under the same compression budget: SLBF’s lower per-basis cost permits m=nm=n, revealing rich cross-expert sharing, while MoBE is restricted to m<nm<n and concentrates each expert on a single dominant basis. 4 Method The structural analysis in Section 3.1 shows that weight reconstruction avoids the intervention-induced structural costs of pruning and merging by preserving expert identities and routing, leaving its compression error controlled by parameter reconstruction quality. This makes parameterization efficiency central to effective weight reconstruction. We therefore address a bottleneck in the state-of-the-art mixture-of-basis-expert formulation (MoBE) (Chen et al., 2026). MoBE representation. MoBE operates on the projection weights i∈gatei,upiW^i∈\W^i_gate,W^i_up\.11 1 We write each weight as i∈ℝd1×d2W^i ^d_1× d_2 with d1≤d2d_1≤ d_2, transposing where necessary to fit this convention. MoBE approximates each expert weight as ^MoBEi=iΦ(∑j=1mwijj), W^i_MoBE=A^i\, \! ( _j=1^mw_ij\,B_j ), (4.1) where jj=1m⊂ℝr×d2\B_j\_j=1^m ^r× d_2 is a shared basis set, wij∈ℝw_ij is a per-expert mixing coefficient, i∈ℝd1×rA^i ^d_1× r is a per-expert linear adapter, and Φ is a non-linearity to enhance the representational power. The parameter cost per basis is r⋅d2r· d_2, restricting m to be small under tight compression budgets. MoBE converts trained MoE-based LLM simply through minimizing weight reconstruction loss: min∑i=1n,ii=1n,jj=1m‖i−^i‖F2 _w,\A^i\_i=1^n,\B_j\_j=1^m _i=1^n\|W^i- W^i\|_F^2 (4.2) =∑i=1n‖i−if(∑j=1mwijj)‖F2. = _i=1^n \|W^i-A^if ( _j=1^mw_ijB_j ) \|_F^2. 4.1 Improved Weight Reconstruction with Shared Low-rank Basis Factorization We propose Shared Low-rank Basis Factorization (SLBF), which replaces each full basis jB_j with a rank-k factorization jjU_jV_j, where j∈ℝr×kU_j ^r× k and j∈ℝk×d2V_j ^k× d_2: ^i=iΦ(∑j=1mwijjj), W^i=A^i\, \! ( _j=1^mw_ij\,U_jV_j ), (4.3) where jjj=1m\U_jV_j\_j=1^m are m shared low-rank bases for reconstructing the right vectors of iW^i, and j∈ℝr×k,j∈ℝk×d2U_j ^r× k,V_j ^k× d_2. Under a fixed compression budget, this factorization affords substantially more bases. Figure 2(a) shows the tensor diagram of the proposed factorization. We discuss the structural and empirical implications of this trade-off. Low-rank mixtures are high-rank. Each individual basis jjU_jV_j has rank at most k. The pre-activation mixture ∑j=1mwijjj _j=1^mw_ijU_jV_j generically attains rank min(mk,r,d2) (mk,r,d_2), matching MoBE’s per-basis rank when mk≥rmk≥ r. SLBF thus trades few-rich bases for many-simple bases composed through the mixture, with Φ providing further non-linear composition. Empirical benefit of richer mixing. Empirically, SLBF exploits this freedom by distributing reconstruction across multiple bases per expert, whereas MoBE tends to concentrate each expert on one dominant basis (Figure 2(c)). SLBF also converges faster and reaches lower reconstruction MSE under fixed budget (Figure 2(b)). On Mixtral-8x7B, this advantage holds uniformly: SLBF achieves lower reconstruction MSE across all 32 MoE layers under matched budget (Figure 3). Figure 3: Per-layer MSE on Mixtral-8x7B gate. SLBF achieves uniformly lower weight reconstruction error across all layers under a matched compression. Connections to block-term decomposition. In tensor decomposition terms, the pre-activation mixture ∑j=1mwijjj _j=1^mw_ij\,U_jV_j corresponds to the i-th matrix in a rank-(k,k,1)(k,k,1) block-term decomposition (BTD) (De Lathauwer, 2008) across experts. SLBF thus generalizes MoBE in its basis structure: MoBE’s mixture ∑jwijj _jw_ijB_j is the Tucker-1 case recovered at k=rk=r, and the rank-k restriction enables the favorable parameter trade-off. BTD has been applied effectively in model compression (Ye et al., 2018; Ma et al., 2019) and signal processing (Giampouras et al., 2022; Min et al., 2026), motivating its use in our formulation. Training. We fit SLBF by minimizing the weight-space reconstruction loss ∑i‖i−^i‖F2 _i\|W^i- W^i\|_F^2 independently for each MoE layer. This objective directly reduces the parameter reconstruction residual appearing in Equation 3.5 while requiring no calibration data. The objective is non-convex and solved with AdamW (Loshchilov and Hutter, 2019). For stability, we use techniques including matrix-sign initialization (details in Sections 5.1 and A.2). 4.2 Parameter Reduction via Bilinear Symmetry Gauge Fixing The bilinear product UV exhibits a known gauge symmetry (Zhao et al., 2025; Lu et al., 2026): for any invertible ∈GL(k)Q (k), the factorization ()(−1)=(UQ)(Q^-1V)=UV is invariant. This leaves k2k^2 parameters redundant in each basis, which we eliminate by fixing the gauge. Specifically, we can write the basis as: =[ab]=[ab]=[Ikba−1]a,UV= bmatrixU_a\\ U_b bmatrixV= bmatrixU_aV\\ U_bV bmatrix= bmatrixI_k\\ U_bU_a^-1 bmatrixU_aV, (4.4) where a∈ℝk×kU_a ^k× k is an invertible matrix. Assuming that ∈ℝr×kU ^r× k has full column rank, such an invertible k×k× k row submatrix always exists. This condition held for all trained factors in our experiments. Therefore, re-parameterizing ^←[Ik;ba−1] U←[I_k;U_bU_a^-1] and ^←a V _aV preserves the product while the fixed identity block in U removes k2k^2 parameters per basis. We select the rows forming aU_a via column-pivoted QR on ⊤U for numerical stability. Parameter count. We summarize the parameter count of one projection in an MoE layer in Table 1 for MoBE and the proposed SLBF. The per-expert i∈ℝd1×rA^i ^d_1× r contributes nd1rnd_1r parameters, identical across MoBE and SLBF. The shared-basis cost differs: mrd2mrd_2 in MoBE versus mk(r+d2)mk(r+d_2) in SLBF, a ratio of k(r+d2)/(rd2)k(r+d_2)/(rd_2). For r≤d2r≤ d_2, SLBF affords proportionally more bases under a fixed compression budget. The gauge-fixing scheme of Section 4.2 further removes mk2mk^2 parameters from ^j U_j at no representational cost (ablated in Section B.1, Figure 4). Method # Parameters Standard nd1d2nd_1d_2 Pruning/Merging n′d1d2n d_1d_2 MoBE nd1r+mrd2+nmnd_1r+mrd_2+nm SLBF nd1r+mk(r+d2−k)+nmnd_1r+mk(r+d_2-k)+nm Table 1: Parameter counts per projection weight. 5 Experiments In this section, we conduct comprehensive experiments on five state-of-the-art MoE-based LLMs, spanning 16B to 122B parameters and diverse architectures, and compare SLBF against representative baselines. Method Ratio Reasoning (↑ ) Knowledge (↑ ) Math/Coding (↑ ) Avg.(↑ ) ARC-C GPQA-D IFEval MMLU CEval CMMLU GSM8K MBPP HEval Qwen3-30B-A3B-2507 95.0 58.6 82.8 87.5 86.3 84.4 95.9 79.6 93.9 84.9 REAP 24% 88.4 36.4 80.6 75.7 65.7 60.9 95.5 67.4 82.3 72.5 HC-SMoE 24% 93.9 52.5 81.9 84.1 69.3 67.5 94.7 44.0 79.9 74.2 REAM 24% 89.9 39.9 81.2 78.5 65.7 58.8 94.7 66.8 86.6 73.6 MoLAE 24% 91.3 46.5 77.1 84.3 73.8 73.3 89.3 64.6 84.2 76.0 MoBE 24% 94.0 54.6 82.4 85.5 82.8 81.6 95.2 77.4 91.5 82.8 SLBF 24% 93.9 60.1 82.6 85.9 85.1 82.2 95.6 78.6 95.1 84.3 MoLAE 32% 84.5 34.3 69.3 75.8 66.3 66.4 80.0 51.0 67.7 66.1 MoBE 32% 91.6 46.5 79.9 82.3 77.1 76.2 92.9 69.6 89.0 78.3 SLBF 36% 93.0 57.6 83.9 85.5 80.5 79.9 94.7 76.2 90.9 82.4 Moonlight-16B-A3B 83.6 31.8 43.4 71.7 71.4 73.4 74.8 57.4 71.3 64.3 MoLAE 15% 65.1 18.2 20.9 48.9 50.7 52.6 16.1 10.8 0.0 31.5 MoBE 15% 79.1 23.7 35.5 61.6 65.2 66.8 52.2 46.0 59.8 54.4 SLBF 15% 80.8 28.8 40.7 69.2 69.0 71.9 71.7 59.6 70.7 62.5 Gemma4-26B-A4B-it 94.1 65.7 90.4 89.3 69.3 73.1 94.7 91.2 98.2 85.1 MoLAE 37% 74.2 29.8 71.2 67.0 27.7 51.0 70.7 7.8 33.5 48.1 MoBE 37% 91.8 46.0 85.6 82.1 28.1 63.4 88.5 35.2 86.6 67.5 SLBF 37% 92.9 61.6 89.3 87.8 63.6 68.6 93.6 84.6 93.9 81.8 Qwen3.5-122B-A10B 96.1 83.3 93.0 88.7 86.7 86.2 97.4 53.0 92.1 86.3 MoLAE 34% 93.3 43.4 70.4 79.5 78.6 79.4 91.3 38.8 76.2 72.3 MoBE 34% 96.2 67.7 87.8 87.7 84.0 84.4 97.0 47.0 93.3 82.8 SLBF 34% 96.3 83.8 93.7 89.6 87.4 86.9 97.4 61.8 94.5 88.0 Table 2: Multi-architecture compression results. We report nine benchmarks grouped into reasoning, knowledge, and math/coding for four instruction-tuned MoE LLMs at varying compression ratios. All methods are applied one-shot without fine-tuning. Ratio denotes total parameter reduction. Best results per model and ratio are bolded. Method Ratio WT (↓ ) ARC-C ARC-E HellaSwag OPQA RTE Winogrande PIQA Avg.(↑ ) Mixtral-8x7B-v0.1 5.64 59.7 82.7 84.2 49.4 71.1 77.3 83.2 72.5 Frequency 24% 9.10 44.8 70.3 66.6 30.2 58.5 56.8 77.6 57.8 MC-SMoE 24% 48.42 29.4 52.0 49.4 30.6 52.7 54.1 66.2 47.8 HC-SMoE 24% 8.54 53.2 78.1 81.3 46.2 67.9 75.2 81.7 69.1 EAN 24% 8.28 54.5 80.6 81.9 45.4 69.7 75.9 83.1 70.2 REAP 24% 8.30 53.9 79.5 81.8 46.8 67.2 77.1 82.5 69.8 Sub-MoE 24% 17.51 43.0 66.5 68.0 35.8 53.1 68.0 74.0 58.3 TD-MoE 24% 16.68 42.5 69.3 66.7 40.4 55.6 69.4 74.8 59.8 SLBF 24% 6.92 56.7 80.6 83.5 50.4 63.9 76.4 83.0 70.7 MoLAE 30% 11.17 51.6 76.6 77.9 46.0 58.1 75.4 80.8 66.6 MoBE 30% 8.59 54.0 79.8 80.8 49.4 62.1 75.6 82.3 69.1 SLBF 30% 7.78 55.8 80.1 82.1 48.6 67.5 76.7 83.1 70.6 Table 3: Compression comparison on Mixtral-8x7B-v0.1. We report WikiText-2 perplexity (WT (↓ )) and zero-shot task accuracy. All methods are applied one-shot without fine-tuning. Best results per compression ratio are bolded. 5.1 Setup Models. We evaluate SLBF on five MoE LLMs spanning a range of architectures, granularities, and scales: Moonlight-16B-A3B-Instruct (Liu et al., 2025a), Qwen3-30B-A3B-Instruct-2507 (Team, 2025), Gemma4-26B-A4B-it Google (2026), Mixtral-8x7B-v0.1 (base) Jiang et al. (2024), Qwen3.5-122B-A10B Qwen Team (2026). Compression ratios (Ratio) refer to parameter reduction with respect to the total model. Baselines. We compare against representative methods from all three families. Pruning: Frequency, REAP (Lasby et al., 2026), and EAN (Jaiswal et al., 2025), each using a different expert-saliency criterion. Merging: MC-SMoE (Li et al., 2024b), HC-SMoE (Chen et al., 2025a), and REAM (Jha et al., 2026). Weight reconstruction: MoLAE (Liu et al., 2025b), and MoBE (Chen et al., 2026). Others: Sub-MoE (Li et al., 2026), TD-MoE (XU et al., 2026). MoLAE, MoBE, and SLBF are data-free, while the remaining baselines use calibration data. For all calibration-based baselines, we use 1024 C4 sequences of 2048 tokens. Evaluation benchmarks. We evaluate on a comprehensive set of benchmark datasets. For the modern instruction-tuned MoE LLMs (Section 5.2), we report nine standard benchmarks grouped as reasoning: ARC-Challenge (Clark et al., 2018), GPQA-Diamond (Rein et al., 2024), IFEval (Zhou et al., 2023); knowledge: MMLU (Hendrycks et al., 2021), CEval (Huang et al., 2023), CMMLU (Li et al., 2024a); and math/coding: GSM8K (Cobbe et al., 2021), MBPP (Austin et al., 2021), HumanEval (Chen et al., 2021). For Mixtral-8x7B, following prior Mixtral compression work, we report perplexity on WikiText-2 and zero-shot accuracy on ARC-Challenge, ARC-Easy (Clark et al., 2018), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018), RTE (Bentivogli et al., 2009), WinoGrande (Sakaguchi et al., 2021), and PIQA (Bisk et al., 2020). Using each model’s conventional suite preserves direct comparability with prior compression work. Evaluations are conducted using lm-evaluation-harness (Gao et al., 2024). Original and compressed checkpoints are evaluated with identical task definitions and decoding configurations. Full evaluation details, including exact task IDs, prompting settings, and decoding settings, are provided in Section C.1. SLBF configuration. We use m=nm=n shared bases throughout, matching the number of bases to the number of experts and yielding an n×n× n expert-to-basis mixing structure (Figure 2(c)). We set r=d1r=d_1 following Chen et al. (2026), so no low-rank bottleneck is introduced in iA^i; the per-basis rank k is chosen to match the target compression ratio. We use the SiLU activation (Elfwing et al., 2018) for Φ . We optimize with AdamW Loshchilov and Hutter (2019). Because of the multiplicative low-rank parameterization, we use separate learning-rate groups for ,\A,w\ and ,\U,V\, together with matrix-sign initialization. Details and remaining hyper-parameters can be found in Section A.2. 5.2 Multi-architecture Results We evaluate SLBF on Qwen3-30B-A3B-Instruct-2507, Qwen3.5-122B-A10B, Moonlight-16B-A3B, and Gemma4-26B-A4B-it. Baselines are the strongest representative from each compression family evaluated at this scale: REAP (pruning), HC-SMoE (merging), REAM (merging), MoLAE (weight reconstruction), and MoBE (weight reconstruction). All results are obtained in the one-shot setting with no fine-tuning after compression. Performance across models and compression ratios. Table 2 shows that SLBF consistently outperforms baselines across four architectures and varying compression ratios. At 24% compression, SLBF preserves near-original accuracy on Qwen3-30B while the strongest baseline MoBE drops 2.1 points and pruning/merging baselines drop over 8 points. The advantage widens at higher compression and on other architectures, reaching a 14.3-point gap over MoBE on Gemma4-26B and a 5.2-point gap on Qwen3.5-122B, where SLBF maintains performance comparable to the original model under the same evaluation configuration. Notably, SLBF at 36% compression on Qwen3-30B still outperforms MoBE at the easier 32% ratio. Performance on Moonlight. Moonlight-16B-A3B differs substantially from the other evaluated models in its training recipe, notably being trained end-to-end with the Muon optimizer (Liu et al., 2025a). SLBF remains effective in this setting: at 15% compression, the average score drops by only 1.8 points, compared with 9.9 points for MoBE and over 30 points for MoLAE. 5.3 Cross-family Comparison on Mixtral Mixtral is more often used in MoE compression literature, on which we compare against a richer set of baselines, including Frequency, EAN (Jaiswal et al., 2025), MC-SMoE (Li et al., 2024b), HC-SMoE (Chen et al., 2025a), Sub-MoE (Li et al., 2026), TD-MoE (XU et al., 2026), MoLAE (Liu et al., 2025b), and MoBE (Chen et al., 2026). Table 3 reports the results. Performance across compression families. At 24% compression, SLBF reaches 70.7 average accuracy and WikiText-2 perplexity of 6.92, outperforming the strongest pruning baseline (EAN: 70.2 / 8.28) and the strongest merging baseline (HC-SMoE: 69.1 / 8.54). The perplexity gap is more pronounced: SLBF is 1.4 lower (16% relative) than the next-best method, while the accuracy gap remains within 0.5 points, suggesting better preservation of language-modeling performance. At 30% compression, SLBF also outperforms the weight-reconstruction baselines MoLAE and MoBE. It maintains the lead with perplexity 7.78 against MoBE’s 8.59. These results are consistent with the structural distinction highlighted in Section 3.1: weight reconstruction preserves expert identities and routing, while SLBF further improves the parameterization within this family. 5.4 Ablation study and runtime analysis We ablate design choices and analyze runtime cost; full details are in the appendix. SLBF without gauge fixing already outperforms MoBE at every compression ratio; gauge fixing adds a secondary gain (Section B.1). Compressing all three projections outperforms two at matched budget (62.5 vs 58.2 on Moonlight-16B; Section B.3), and sweeping m at fixed budget suggests that m=nm=n lies near a strong operating region (Section B.2). On Mixtral gate projections, SLBF takes 40.9 minutes per (layer, projection) pair versus 94.3 minutes for MoBE, using only 40% as many training iterations due to earlier convergence, and the procedure is embarrassingly parallel across pairs (Section B.4). In runtime-decompressed serving on 2×2×H100, SLBF reduces model-weight memory by 30% and increases KV-cache capacity and maximum serving concurrency by 53%, while retaining 0.88×0.88× throughput (Section B.5). 6 Conclusion We introduced Shared Low-rank Basis Factorization (SLBF), a data-free weight reconstruction method for MoE compression that uses shared rank-k bases to improve parameter efficiency under fixed compression budgets. Our structural analysis shows that pruning and merging can incur intervention-induced costs associated with expert removal and tied routing, whereas weight reconstruction preserves expert identities and routing and leaves its error controlled by parameter reconstruction quality. Across five MoE architectures spanning 16B to 122B parameters, SLBF consistently outperforms representative methods. Limitations SLBF operates entirely in weight space without calibration data or post-compression fine-tuning. We therefore do not explore calibration-guided budget allocation or fine-tuning to recover residual accuracy loss; efficient forward computation through the factorized representation is also beyond the scope of this work. Our study focuses on expert FFN weight compression. Interactions with other compression axes, such as quantization or attention compression, are not explored. We also use a fixed decomposition structure with r=d1r=d_1 and m=nm=n across models and layers. Although Section B.2 identifies m=nm=n as a strong operating point, adaptive per-layer or per-projection choices of (r,k,m)(r,k,m) may yield further improvements. Finally, our deployment evaluation covers limited hardware and serving configurations. Runtime-decompressed inference exhibits a memory–throughput trade-off that may differ with optimized MoE kernels or other serving systems. Direct loading of the factorized representation can also incur substantial reconstruction overhead for large MoEs in our current implementation. Acknowledgements This work was supported by JSPS KAKENHI Grant Numbers JP26K02984 and JP23K28109, and JSPS Bilateral Program Number JPJSBP120257420. References Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §5.1. Bai et al. (2025) S. Bai, H. Li, J. ZHANG, Z. Hong, and S. Guo DiEP: adaptive mixture-of-experts compression through differentiable expert pruning. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, p. 56090–56115. External Links: Link Cited by: §2. Bandarkar et al. (2026) L. Bandarkar, C. Yang, M. Fayyaz, J. Hu, and N. Peng Multilingual routing in mixture-of-experts. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §3.2. Bentivogli et al. (2009) L. Bentivogli, P. Clark, I. Dagan, and D. Giampiccolo The fifth pascal recognizing textual entailment challenge.. TAC 7 (8), p. 1. Cited by: §5.1. Bisk et al. (2020) Y. Bisk, R. Zellers, R. Le bras, J. Gao, and Y. Choi PIQA: reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence 34 (05), p. 7432–7439. External Links: Link, Document Cited by: §5.1. Chen et al. (2025a) I. Chen, H. Liu, W. Sun, C. Chao, Y. Hsu, and C. Lee Retraining-free merging of sparse moe via hierarchical clustering. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §A.1, §1, §2, §3.2, §5.1, §5.3. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating large language models trained on code. External Links: 2107.03374 Cited by: §5.1. Chen et al. (2022) T. Chen, S. Huang, Y. Xie, B. Jiao, D. Jiang, H. Zhou, J. Li, and F. Wei Task-specific expert pruning for sparse mixture-of-experts. arXiv preprint arXiv:2206.00277. Cited by: §2. Chen et al. (2026) X. Chen, M. Ha, Z. Lan, J. Zhang, and J. Li MoBE: mixture-of-basis-experts for compressing moe-based LLMs. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §A.2, §1, §1, §2, §4, §5.1, §5.1, §5.3. Chen et al. (2025b) Z. Chen, X. Hu, D. Yang, Z. Xu, X. Chen, Z. Yuan, S. Zhou, and J. Yu MoEQuant: enhancing quantization for mixture-of-experts large language models via expert-balanced sampling and affinity guidance. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 8245–8260. External Links: Link Cited by: §2. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: §5.1. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §5.1. Dai et al. (2024) D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al. Deepseekmoe: towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1280–1297. Cited by: §2. De Lathauwer (2008) L. De Lathauwer Decompositions of a higher-order tensor in block terms—part i: definitions and uniqueness. SIAM Journal on Matrix Analysis and Applications 30 (3), p. 1033–1066. Cited by: §4.1. Duanmu et al. (2025) H. Duanmu, X. Li, Z. Yuan, S. Zheng, J. Duan, X. Zhang, and D. Lin MxMoE: mixed-precision quantization for MoE with accuracy and performance co-design. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 14793–14806. External Links: Link Cited by: §2. Elfwing et al. (2018) S. Elfwing, E. Uchibe, and K. Doya Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks 107, p. 3–11. Cited by: §5.1. Fedus et al. (2022) W. Fedus, B. Zoph, and N. Shazeer Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. External Links: Link Cited by: §2. Gao et al. (2024) L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou The language model evaluation harness. Zenodo. External Links: Document, Link Cited by: Appendix C, §5.1. Geva et al. (2021) M. Geva, R. Schuster, J. Berant, and O. Levy Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 5484–5495. Cited by: §B.3. Giampouras et al. (2022) P. V. Giampouras, A. A. Rontogiannis, and E. Kofidis Block-term tensor decomposition model selection and computation: the bayesian way. IEEE Transactions on Signal Processing 70, p. 1704–1717. Cited by: §4.1. Google (2026) Google Gemma 4 model card. Note: https://ai.google.dev/gemma/docs/core/model_card_4 Cited by: §1, §2, §3, §5.1. Gu et al. (2025) H. Gu, W. Li, L. Li, Z. Qiyuan, M. G. Lee, S. Sun, W. Xue, and Y. Guo Delta decompression for moe-based LLMs compression. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §2. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In International Conference on Learning Representations, External Links: Link Cited by: §5.1. Huang et al. (2023) Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, j. lei, Y. Fu, M. Sun, and J. He C-eval: a multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 62991–63010. External Links: Link Cited by: §5.1. Jacobs et al. (1991) R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton Adaptive mixtures of local experts. Neural computation 3 (1), p. 79–87. Cited by: §1, §2. Jaiswal et al. (2025) A. Jaiswal, J. Wang, Y. Li, P. Li, T. Chen, Z. Wang, C. Wang, R. Pang, and X. Du Finding fantastic experts in moes: a unified study for expert dropping strategies and observations. arXiv preprint arXiv:2504.05586. Cited by: §2, §5.1, §5.3. Jha et al. (2026) S. Jha, M. Hashemzadeh, A. S. Pasand, A. Parviz, M. Lee, and B. Knyazev REAM: merging improves pruning of experts in llms. arXiv preprint arXiv:2604.04356. Cited by: §2, §5.1. Jiang et al. (2024) A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: §1, §2, §3, §5.1. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, Cited by: footnote 2. Lasby et al. (2026) M. Lasby, I. Lazarevich, N. Sinnadurai, S. Lie, Y. Ioannou, and V. Thangarasa REAP the experts: why pruning prevails for one-shot moe compression. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §A.1, §1, §2, §3.1, §3.1, §3.1, §3.2, §3.2, §5.1. Lepikhin et al. (2021) D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen GShard: scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, External Links: Link Cited by: §2, §3.1. Li et al. (2024a) H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin CMMLU: measuring massive multitask language understanding in Chinese. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 11260–11285. External Links: Link, Document Cited by: §5.1. Li et al. (2026) L. Li, Q. Zhu, J. Wang, X. Qin, W. Li, H. Gu, S. Han, and Y. Guo Sub-moe: efficient mixture-of-expert llms compression via subspace expert merging. Proceedings of the AAAI Conference on Artificial Intelligence 40 (27), p. 22994–23002. External Links: Link, Document Cited by: §5.1, §5.3. Li et al. (2024b) P. Li, Z. Zhang, P. Yadav, Y. Sung, Y. Cheng, M. Bansal, and T. Chen Merge, then compress: demystify efficient SMoe with hints from its routing policy. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2, §5.1, §5.3. Li et al. (2025) W. Li, L. Li, H. Gu, Y. Huang, M. G. Lee, S. Sun, W. Xue, and Y. Guo MoE-SVD: structured mixture-of-experts LLMs compression via singular value decomposition. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §3.1. Liu et al. (2024a) A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Cited by: §1, §2, §3. Liu et al. (2024b) E. Liu, J. Zhu, Z. Lin, X. Ning, M. B. Blaschko, S. Yan, G. Dai, H. Yang, and Y. Wang Efficient expert pruning for sparse mixture-of-experts language models: enhancing performance and reducing inference costs. arXiv preprint arXiv:2407.00945. Cited by: §2. Liu et al. (2025a) J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan, Y. Chen, H. Zheng, Y. Liu, S. Liu, B. Yin, W. He, H. Zhu, Y. Wang, J. Wang, M. Dong, Z. Zhang, Y. Kang, H. Zhang, X. Xu, Y. Zhang, Y. Wu, X. Zhou, and Z. Yang Muon is scalable for llm training. External Links: 2502.16982, Link Cited by: §5.1, §5.2. Liu et al. (2025b) Z. Liu, H. Wu, R. She, X. Fu, X. Han, T. Zhong, and M. Yuan MoLAE: mixture of latent experts for parameter-efficient language models. arXiv preprint arXiv:2503.23100. Cited by: §B.3, §1, §2, §5.1, §5.3. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §A.2, §4.1, §5.1. Lu et al. (2024) X. Lu, Q. Liu, Y. Xu, A. Zhou, S. Huang, B. Zhang, J. Yan, and H. Li Not all experts are equal: efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6159–6172. Cited by: §2. Lu et al. (2026) Y. Lu, S. Yu, H. Weng, P. Wang, Y. Hu, L. Hung-Chun, H. Chiang, and K. Wu Skipcat: rank-maximized low-rank compression of large language models via shared projection and block skipping. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 24124–24132. Cited by: §4.2. Ma et al. (2019) X. Ma, P. Zhang, S. Zhang, N. Duan, Y. Hou, M. Zhou, and D. Song A tensorized transformer for language modeling. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, p. . External Links: Link Cited by: §4.1. Meng et al. (2022) K. Meng, D. Bau, A. Andonian, and Y. Belinkov Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 17359–17372. External Links: Link Cited by: §B.3. Mihaylov et al. (2018) T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.), Brussels, Belgium, p. 2381–2391. External Links: Link, Document Cited by: §5.1. Min et al. (2026) Z. Min, T. Huang, and W. Wu Hyperspectral image denoising via deep block term decomposition. IEEE Geoscience and Remote Sensing Letters 23 (), p. 1–5. External Links: Document Cited by: §4.1. Qwen Team (2026) Qwen Team Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §5.1. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (140), p. 1–67. External Links: Link Cited by: Appendix C, §3.2. Rein et al. (2024) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, External Links: Link Cited by: §5.1. Sakaguchi et al. (2021) K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi WinoGrande: an adversarial winograd schema challenge at scale. Commun. ACM 64 (9), p. 99–106. External Links: ISSN 0001-0782, Link, Document Cited by: §5.1. Shazeer et al. (2017) N. Shazeer, *. Mirhoseini, *. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Sun et al. (2025) H. Sun, Z. Wu, B. Xia, P. Chang, Z. Dong, Y. Yuan, Y. Chang, and X. Wang Entropy-based activation function optimization: a method on searching better activation functions. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §A.1. Team (2025) Q. Team Qwen3 technical report. External Links: 2505.09388, Link Cited by: §1, §2, §3, §5.1. Wang et al. (2024) L. Wang, H. Gao, C. Zhao, X. Sun, and D. Dai Auxiliary-loss-free load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408.15664. Cited by: §2, §3.1. Xie et al. (2024) Y. Xie, Z. Zhang, D. Zhou, C. Xie, Z. Song, X. Liu, Y. Wang, X. Lin, and A. Xu Moe-pruner: pruning mixture-of-experts large language model using the hints from its router. arXiv preprint arXiv:2410.12013. Cited by: §2. XU et al. (2026) Y. XU, Y. WANG, X. Peng, H. Zang, C. Minghao, P. Xia, and Z. Wen TD-moe: tensor decomposition for moe models. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §5.1, §5.3. Ye et al. (2018) J. Ye, L. Wang, G. Li, D. Chen, S. Zhe, X. Chu, and Z. Xu Learning compact recurrent neural networks with block-term tensor decomposition. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vol. , p. 9378–9387. External Links: Document Cited by: §4.1. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez (Eds.), Florence, Italy, p. 4791–4800. External Links: Link, Document Cited by: §5.1. Zhang and Sennrich (2019) B. Zhang and R. Sennrich Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d. Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. External Links: Link Cited by: §A.1. Zhang et al. (2026) G. Zhang, H. Yuxuan, Y. Lou, Y. Zhang, W. Zhao, and Y. You MoNE: replacing redundant experts with lightweight novices for structured pruning of moe. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Zhao et al. (2025) J. Zhao, Y. Zhang, and C. V. Cannistraci Pivoting factorization: a compact meta low-rank representation of sparsity for efficient inference in large language models. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §4.2. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. External Links: 2311.07911, Link Cited by: §5.1. Zong et al. (2026) Z. Zong, K. Zhang, Y. Wang, Z. Li, W. Tan, Y. Ren, and J. Hu Not all experts and tokens matter: selective token-guided expert pruning for moe. External Links: Link Cited by: §2. Appendix A Appendix A.1 Missing Derivations in Sec. 3.1 Derivations of merging error (following Lasby et al. (2026)). Recall the original pair contribution (gi+gj)⋅(r()fi()+(1−r())fj()).(g_i+g_j)·(r(x)f^i(x)+(1-r(x))f^j(x)). (A.1) Merging replaces by f~ij=αfi+(1−α)fj f^ij=α f_i+(1-α)f_j for a constant α, a simplification justified in the latest merging method (Chen et al., 2025a). The merged contribution is (gi+gj)f~ij(g_i+g_j) f^ij. The output difference of the pair is δ =(gi+gj)(rfi+(1−r)fj) =(g_i+g_j) (rf^i+(1-r)f^j ) (A.2) −(gi+gj)(αfi+(1−α)fj) -(g_i+g_j) (α f^i+(1-α)f^j ) =(gi+gj)(r−α)(fi−fj) =(g_i+g_j)(r-α)(f^i-f^j) =(gi+gj)(r()−α)Δij(). =(g_i+g_j)(r(x)-α) _ij(x). The expectation of the error squared norm is ℰm(α)=[(gi+gj)2(r()−α)2‖Δij()‖2].E_m(α)=E_x [(g_i+g_j)^2(r(x)-α)^2\| _ij(x)\|^2 ]. (A.3) The optimal mixing ratio, obtained by setting dαℰm=0 ddαE_m=0, is α⋆=[(gi+gj)2‖Δij‖2⋅r()][(gi+gj)2‖Δij‖2].α = E_x [(g_i+g_j)^2\| _ij\|^2· r(x) ]E_x [(g_i+g_j)^2\| _ij\|^2 ]. (A.4) The exact optimal static mixing coefficient is given above. For an interpretable approximation, we additionally assume weak correlation among (gi+gj)2(g_i+g_j)^2, ‖Δij‖2\| _ij\|^2, and r()r(x) across inputs. We refer to the resulting minimum residual within this pairwise static-merging intervention as its irreducible structural cost. The approximate optimal mixing ratio is α^⋆=[r()] α =E_x[r(x)]. We define the irreducible error as the expectation of the approximate error under α^⋆ α : ℰmirr ^irr_m :=ℰm(α^⋆) :=E_m( α ) (A.5) ≈[(gi+gj)2]⋅Var[r()]⋅‖Δij‖2. _x [(g_i+g_j)^2 ]·Var [r(x) ]·E_x\| _ij\|^2. Derivations of pruning error. Recall the original pair contribution gi()fi()+gj()fj().g_i(x)f^i(x)+g_j(x)f^j(x). (A.6) When expert-j is pruned, the output difference is δ =gjfj−(gi′−gi)fi. =g_jf_j-(g _i-g_i)f_i. (A.7) To wipe out the influence of gi′g _i choices, we consider the following oracle router after pruning that minimizes the error squared norm mingi′‖gjfj−(gi′−gi)fi‖2. _g _i\|g_jf_j-(g _i-g_i)f_i\|^2. (A.8) The oracle router is gi⋆=gi+gj⟨fj,fi⟩‖fi‖2g _i=g_i+ g_j f_j,f_i \|f_i\|^2. The oracle router depends on fj()f_j(x), which can only be computed if expert-j is retained. Since the goal of pruning is to remove expert-j entirely, the dependence renders the oracle structurally unrealizable. Substituting back the oracle router, the expectation of the error squared norm is ℰpirr ^irr_p =[‖gjfj−(gi⋆−gi)fi‖2] =E_x [\|g_jf_j-(g _i-g_i)f_i\|^2 ] (A.9) =[‖gjfj−gj⟨fj,fi⟩‖fi‖2⋅fi‖2] =E_x [\|g_jf_j- g_j f_j,f_i \|f_i\|^2· f_i\|^2 ] =[gj2∥fj∥2−2⋅gj2⟨fj,fi⟩‖fi‖2⋅⟨fj,fi⟩ =E_x [g_j^2\|f_j\|^2-2· g_j^2 f_j,f_i \|f_i\|^2· f_j,f_i +gj2|⟨fj,fi⟩|2‖fi‖2] + g_j^2| f_j,f_i |^2\|f_i\|^2 ] =[gj2(‖fj‖2−|⟨fj,fi⟩|2‖fi‖2)]. =E_x [g^2_j (\|f_j\|^2- | f_j,f_i |^2\|f_i\|^2 ) ]. Derivations of weight reconstruction error bound. Each kW^k is replaced by ^k W^k. Routing and expert presence are preserved. We start with the linear case fk()=kf^k(x)=xW^k, and consider the influence of all experts. The output difference is δ=∑k∈()gk(k−^k). =x\, _k (x)g_k(W^k- W^k). (A.10) We use the fact that ∑k=1ngk=1,gk>0 _k=1^ng_k=1,\,g_k>0 and that G():=∑k∈()gk≤1G(x):= _k (x)g_k≤ 1. Using Jensen’s inequality and Cauchy-Schwarz inequality, we have the squared norm of the difference bounded as ‖δ‖2 \| \|^2 =‖∑k∈()gk(k−^k)‖2 =\|x _k (x)g_k(W^k- W^k)\|^2 (A.11) ≤G()∑k∈()gk‖(k−^k)‖2 ≤ G(x) _k (x)g_k\|x(W^k- W^k)\|^2 ≤∑k∈()gk‖(k−^k)‖2 ≤ _k (x)g_k\|x(W^k- W^k)\|^2 ≤∑k=1ngk‖(k−^k)‖2. ≤ _k=1^ng_k\|x(W^k- W^k)\|^2. ≤∑k=1ngk‖2‖k−^k‖F2 ≤ _k=1^ng_k\|x\|^2\|W^k- W^k\|_F^2 Defining ℰw:=‖δ‖2E_w:=E_x\| \|^2, it is bounded as ℰw≤∑k=1n‖k−^k‖F2⋅[gk‖2]. _w≤ _k=1^n\|W^k- W^k\|_F^2·E[g_k\|x\|^2]. (A.12) Modern MoE-based LLMs commonly apply RMSNorm (Zhang and Sennrich, 2019) before the expert FFN. When x denotes the post-RMSNorm activation and the RMSNorm gain is γ, we have ‖2≤d‖∞2≤CRMSd,\|x\|^2≤ d\| γ\|_∞^2≤ C_RMSd, (A.13) where CRMSC_RMS is a finite layer-wise bound on the squared gain magnitude. Therefore, ℰw≤CRMSd∑k=1n[gk]‖k−^k‖F2.E_w≤ C_RMSd _k=1^nE[g_k]\,\|W^k- W^k\|_F^2. (A.14) Weight reconstruction error bound extended to practical SwiGLU-style FFNs. Beyond the linear expert, we can extend to the SwiGLU-style FFN case. For SwiGLU FFNs with SiLU activation σ, the expert is fk()=(σ(gk)⊙uk)dk.f^k(x)= (σ(xW^k_g) ^k_u )W^k_d. (A.15) Compression replaces each of gk,uk,dkW^k_g,W^k_u,W^k_d with ^gk,^uk,^dk W^k_g, W^k_u, W^k_d; denote Δmk:=mk−^mk ^k_m:=W^k_m- W^k_m for m∈u,g,dm∈\u,g,d\. Define intermediate quantities k:=σ(gk),k:=uk,k:=k⊙k,a^k:=σ(xW^k_g), ^k:=xW^k_u, ^k:=a^k ^k, (A.16) and analogously ^k,^k,^k a^k, b^k, z^k for the perturbed expert. The per-expert error telescopes as fk−f^k f^k- f^k =kΔdk+((k−^k)⊙k)^dk =z^k ^k_d+ ((a^k- a^k) ^k ) W^k_d (A.17) +(^k⊙Δuk)^dk. + ( a^k ^k_u ) W^k_d. Note that the SiLU activation is 1.11.1-Lipchitz (shown in Sun et al. (2025), Appendix E). The triangle inequality, together with |σ(t)|≤|t||σ(t)|≤|t|, |σ(a)−σ(b)|≤Lσ|a−b||σ(a)-σ(b)|≤ L_σ|a-b|, the Hadamard inequality ‖⊙‖≤‖∞‖\|p \|≤\|p\|_∞\|q\|, and ‖≤‖‖F\|xM\|≤\|x\|\|M\|_F, yields ∥fk−f^k∥≤∥2[Adk∥Δkd∥F \|f^k- f^k\|≤\|x\|^2 [ A^k_d\,\| ^k_d\|_F (A.18) +Agk∥Δkg∥F+Auk∥Δku∥F], + A^k_g\,\| ^k_g\|_F+ A^k_u\,\| ^k_u\|_F ], with architectural constants Adk:=‖gk‖F2‖uk‖F2, A^k_d:=\|W^k_g\|_F^2\|W^k_u\|_F^2, (A.19) Agk:=Lσ2‖uk‖F2‖^dk‖F2, A^k_g:=L_σ^2\|W^k_u\|_F^2\| W^k_d\|_F^2, Auk:=‖^gk‖F2‖^dk‖F2. A^k_u:=\| W^k_g\|_F^2\| W^k_d\|_F^2. Using the Cauchy-Schwarz inequality with am=Amka_m= A^k_m, bm=‖Δmk‖Fb_m=\| ^k_m\|_F: ‖fk−f^k‖2≤‖4⋅Ak⋅‖Δk‖F2,\|f^k- f^k\|^2≤\|x\|^4· A^k·\| ^k\|_F^2, (A.20) where Ak:=Adk+Agk+AukA^k:=A^k_d+A^k_g+A^k_u is a single per-expert architectural constant and ‖Δk‖F2:=‖Δdk‖F2+‖Δgk‖F2+‖Δuk‖F2\| ^k\|_F^2:=\| ^k_d\|_F^2+\| ^k_g\|_F^2+\| ^k_u\|_F^2 is the total per-expert weight residual. Applying the same Jensen argument across routed experts and using Equation A.13, we obtain ℰw≤(CRMSd)2∑k[gk]Ak‖Δk‖F2.E_w≤(C_RMSd)^2 _kE[g_k]A^k\| ^k\|^2_F. (A.21) Model n d1×d2d_1\!×\!d_2 Ratio k η ρ iters projs. Mixtral-8x7B-v0.1 8 ×143364096\!×\!14336 24% 1152 0.03 0.10 8 000 gate,upgate,up 30% 832 0.03 0.10 8 000 gate,upgate,up Qwen3-30B-A3B-Instruct-2507 128 ×2048768\!×\!2048 24% 226 0.10 0.30 20 000 gate,up,downgate,up,down 36% 147 0.10 0.30 20 000 gate,up,downgate,up,down Qwen3.5-122B-A10B 256 ×30721024\!×\!3072 34% 250 0.10 0.222 8 000 gate,up,downgate,up,down Moonlight-16B-A3B-Instruct 64 ×20481408\!×\!2048 15% 126 0.10 0.359 20 000 gate,up,downgate,up,down Gemma-4-26B-A4B-it 128 ×2816704\!×\!2816 37% 199 0.10 0.267 20 000 gate,up,downgate,up,down Table 4: SLBF training hyperparameters for each evaluated compression setting. n is the number of experts; d1×d2d_1× d_2 is the per-expert FFN projection shape; Ratio denotes total-model parameter reduction, and k is the corresponding per-basis rank. The basis-side parameters ,\A,w\ use learning rate η, while ,\U,V\ use ρηρη. A.2 SLBF training details All reported results are from a single run; the initialization is deterministic, so results are reproducible given fixed hyperparameters. In our experiments, we train SLBF using one NVIDIA RTX A5000 24GB or NVIDIA RTX A6000 48GB. Matrix-sign initialization. Each expert’s basis triple (i,i,i)(A^i,U_i,V_i) is initialized from the SVD i=WWWW^i=U_W _WV_W of the original per-expert weight matrix, with the singular spectrum discarded. Concretely, i:=W[:,:r]A^i:=U_W[:,\!:r] is the left singular matrix (top-r columns) and is full-rank since we set r=d1r=d_1; i:=[k; 0]∈ℝr×kU_i:=[\,I_k\,;\,0\,] ^r× k is a partial-identity block (the first k rows form the identity, remaining rows are zero), giving orthonormal columns; and i:=W[:k,:]∈ℝk×d2V_i:=V_W[\,:k,\,:] ^k× d_2 is the reduced right-singular factor (top-k orthonormal columns). The product iiiA^iU_iV_i therefore lies in the rank-k singular subspace of iW^i but carries no magnitude; the optimizer recovers scale and inter-expert mixing during training. We follow (Chen et al., 2026) to parameterize w by softmax to keep ∑j=1mwij=1 _j=1^mw_ij=1 for stability. For the mixing weights we set the pre-softmax logit w~ij=c⋅[j=i] w_ij=c·1[j=i] with c=1c=1, so the post-softmax weight is a soft (not sharp) one-hot vector: with n=128n=128, softmax(w~i)e≈0.021softmax( w^i)_e≈ 0.021 and the remaining n−1n-1 entries each ≈0.0077≈ 0.0077. This intentional softness keeps gradients flowing through every basis from step 0; a much sharper initialization strongly suppresses off-diagonal mixing gradients, which we observed to hinder experts from learning to combine bases. Shared training recipe. All SLBF runs use AdamW (Loshchilov and Hutter, 2019) with weight decay 2×10−22× 10^-2, (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999), a cosine learning-rate schedule with ηmin=10−4 _min=10^-4, and global_std normalization of the per-expert reconstruction target. Each ((layer, projection ∈gate,up,down∈\gate,up,down\)) pair is trained independently; no calibration data is used, since the reconstruction objective is purely weight-side. Two parameter groups receive separate learning rates: the basis side ,\A,w\ and the spectrum-restoring side ,\U,V\ at η and ρηρη, respectively. Model-specific values of η and ρ are listed in Table 4. Projection scope. For Qwen3, Qwen3.5, Moonlight, and Gemma-4 we compress all three FFN projections (gate_proj, up_proj, down_proj). For Mixtral-8x7B we compress only gate_proj and up_proj and keep down_proj at full rank; with Mixtral’s d2/d1≈3.5d_2/d_1≈ 3.5 aspect ratio the down-projection (14336→409614336→ 4096) is the most error-sensitive slice of the FFN and we observed materially better downstream scores when it was left uncompressed at matched total storage. The reported total compression ratio accounts for this asymmetry on Mixtral. Bilinear gauge-fixing. After training, each (i,i)(U_i,V_i) pair is gauge-fixed (Section 4.2) so the resulting proxy can be saved as a standard HuggingFace MoE checkpoint with no inference-time changes. Per-model hyperparameters. Table 4 lists the model- and compression-specific settings. The bottleneck rank k is chosen for each compression setting to match the target total-model parameter reduction. We use η=0.03η=0.03 and ρ=0.10ρ=0.10 for Mixtral rather than the default scaling rule, due to its unusually large d2/d1d_2/d_1 ratio. We train for 8,000 iterations on Mixtral and Qwen3.5-122B, whose reconstruction losses had plateaued by that point. A small number of (layer, projection) pairs exhibit optimization failures. We identify these by manually inspecting the deterministic loss curves, with failures characterized by a large irreversible loss spike or a loss that fails to decrease. For an affected pair, we retrain from the same initialization with ρ halved once (e.g., →0.050.10\!→\!0.05 on Mixtral); no additional retries are performed. The same procedure is applied across all settings. On Mixtral-8x7B at 30% compression (k=832k=832), this affects 66 out of 32×2=6432× 2=64 pairs, while fewer than 1% of pairs are affected on the other models. A single halving is sufficient in all cases encountered. MoBE implementation and hyperparameters. For MoBE, we directly use the authors’ released implementation and recommended hyperparameters. The only modification is on Mixtral-8x7B, where the recommended learning rate did not converge in our setting; we therefore reduce it to 0.030.03. All other MoBE settings follow the released implementation without modification. Appendix B Ablation Details B.1 Effect of Gauge Fixing on Compression Frontier We isolate the contribution of bilinear gauge fixing (Section 4.2) by comparing three configurations at varying compression ratios on Qwen3-30B gate_proj: SLBF with gauge fixing, SLBF without gauge fixing, and MoBE. For each method, we sweep the compression budget by varying the per-basis rank k (SLBF) or the number of bases m (MoBE), training each configuration to convergence. Figure 4 reports the results. Two conclusions emerge. First, SLBF without gauge fixing already outperforms MoBE across the evaluated compression sweep, showing that the gains are not solely attributable to gauge fixing and providing evidence for the benefit of the low-rank basis design. Second, gauge fixing provides a consistent secondary gain by eliminating mk2mk^2 redundant parameters, which are reinvested as higher per-basis rank k at matched budget. This gain widens at more aggressive compression, where the freed parameters constitute a larger fraction of the total budget. Figure 4: Compression frontier on layer 4 gate_proj of Qwen3-30B-A3B-Instruct-2507. Best reconstruction MSE versus total compression ratio for SLBF (with gauge fixing), SLBF without gauge fixing, and MoBE. SLBF without gauge fixing already outperforms MoBE across the evaluated sweep; gauge fixing provides an additional gain that widens at higher compression. B.2 Effect of Number of Bases at Matched Budget We investigate the tradeoff between the number of bases m and the per-basis rank k under a fixed compression budget. On Qwen3-30B layer 4, gate_proj, we fix the compression ratio at 37.5% of the original expert weight tensor and sweep m∈64,96,128,160,192,224,256m∈\64,96,128,160,192,224,256\, adjusting k for each m to match the budget. Figure 5 shows the reconstruction loss over training steps. Too few bases (m=64m=64, k=314k=314) converge most slowly and to the highest MSE despite each basis being high-rank; too many bases (m=256m=256, k=72k=72) degrade slightly as each basis becomes too rank-deficient to contribute meaningfully. The optimum lies in the range m=160m=160 to 224224, indicating a genuine tradeoff between basis count and per-basis expressiveness. The operating point m=n=128m=n=128 used throughout our experiments lies in a strong operating region, and the gains from further increasing m beyond n are modest. We adopt m=nm=n, matching the number of bases to experts and simplifying the implementation. Figure 5: Reconstruction loss on layer 4 gate_proj of Qwen3-30B-A3B-Instruct-2507 at fixed 37.5% compression, sweeping the number of bases m with per-basis rank k adjusted to match budget. The optimum lies at intermediate m; the m=n=128m=n=128 operating point is near-optimal. B.3 Ablation on Downstream Benchmarks We ablate the effects of the SLBF parameterization, projection scope, and gauge fixing on downstream performance. On Moonlight-16B-A3B at 15% total compression, comparing MoBE and SLBF under the same gate+up scope, SLBF improves average accuracy from 54.5 to 58.2 and reduces WikiText-2 perplexity from 8.22 to 7.77. Extending SLBF to all three projections at the same total compression ratio further improves the average score to 62.2, while gauge fixing provides an additional gain. We further conduct an equal-scope control on Qwen3-30B-A3B at 24% compression. SLBF (gate+up) reaches 83.7 average accuracy compared with 82.8 for MoBE under the same projection scope, while extending SLBF to gate+up+down further improves the score to 84.3. These results separate the two effects: SLBF improves over MoBE even at matched projection scope, while distributing the compression budget across all three projections provides an additional gain. This finding motivates the projection scope used in Table 4. For Qwen3, Qwen3.5, Moonlight, and Gemma4, we compress all three FFN projections. The original MoBE configuration compresses gate and up projections while leaving down unchanged, following prior observations that down projections can be more sensitive to compression (Geva et al., 2021; Meng et al., 2022; Liu et al., 2025b). For Mixtral-8x7B, whose large aspect ratio (d2/d1=3.5d_2/d_1=3.5) makes the down projection particularly sensitive in our experiments, we likewise retain down_proj at full rank and compress only gate and up. Configuration Avg.(↑ ) WT2 (↓ ) C4 (↓ ) Moonlight-16B-A3B, 15% compression Original 64.3 6.93 12.77 MoLAE (g+u) 31.5 13.02 24.33 MoBE (g+u) 54.4 8.22 14.67 SLBF (g+u) 58.2 7.77 14.08 SLBF (g+u+d, w/o GF) 62.2 7.54 13.66 SLBF (g+u+d, w/ GF) 62.5 7.53 13.55 Qwen3-30B-A3B, 24% compression Original 84.9 MoBE (g+u) 82.8 SLBF (g+u) 83.7 SLBF (g+u+d) 84.3 Table 5: Downstream ablation of projection scope and gauge fixing. The Moonlight results additionally report token-level perplexity on WikiText-2 and C4. The Qwen3 control isolates projection scope at the same 24% total-model compression ratio. B.4 Compression Runtime Both MoBE and SLBF solve the same weight-space reconstruction objective; the per-iteration cost is comparable since the dominant operation is the forward pass through the parameterization. On Mixtral-8x7B (gate_proj), MoBE requires a mean of 94.3 minutes per (layer, projection) pair, while SLBF completes in 40.9 minutes using 40% of the iterations, enabled by its faster convergence (Figure 2(b)). Since each (layer, projection) pair is trained independently with no cross-layer coupling, the procedure is embarrassingly parallel. B.5 Inference Analysis Inference cost reduction under gauge-fixed storage Beyond removing k2k^2 stored parameters per basis (Sec. 4.2), the fixed identity block in U also reduces the matrix-multiplication cost when materializing each basis at inference. Writing the product in block form, ^^=[kba−1]^=[^(ba−1)^], U V= bmatrixI_k\\ U_bU_a^-1 bmatrix V= bmatrix V\\[2.0pt] (U_bU_a^-1)\, V bmatrix, (B.1) the top k rows of the basis are simply a copy of V, requiring no multiplication, while only the remaining (r−k)(r-k) rows need an explicit matrix product. The multiply-add cost per basis drops from rkdr\,k\,d to (r−k)kd(r-k)\,k\,d, a saving of k2dk^2d operations. Aggregated over m shared bases this is mk2dm\,k^2\,d multiply-adds per projection per MoE layer. On Mixtral-8x7B with m=8m=8, k=832k=832, d=14336d=14336 the saving is about 20%20\% of the unoptimized basis reconstruction cost per projection per layer. In practice the k pivot rows in U sit at arbitrary positions selected by column-pivoted QR rather than literally the top of the matrix. We therefore store the pivot indices alongside ba−1U_bU_a^-1 and V, and assemble each materialized basis by (i) placing V at the pivot rows of the output via an index-scatter, and (i) computing the remaining (r−k)(r-k) rows from a smaller batched matrix product on ba−1U_bU_a^-1 and V. Runtime storage and throughput We evaluate SLBF’s runtime-decompressed mode on Mixtral-8x7B against an uncompressed baseline that uses the same per-expert dispatch path22 2 Both variants share vLLM’s stock attention backend Kwon et al. (2023) and use a naive (per-expert loop) MoE implementation. This isolates the cost of basis reconstruction from any optimization gap between fused- and loop-based MoE kernels.. Both run on 2× NVIDIA H100 80 GB (pipeline parallelism P=2=2, BF16) under vLLM with enforce_eager=True and gpu_memory_utilization=0.85=0.85 (an effective GPU-memory budget of ∼ 136 GiB across both ranks). Table 6 summarizes the trade-off. SLBF reduces on-device model weights by 30%30\%; the freed memory becomes KV-cache budget (+53%+53\%), so the maximum number of concurrent requests at the configured 4416-token sequence length grows from 75.475.4 to 115.5115.5. At a high-concurrency operating point of 96 prompts (the regime where the uncompressed baseline reaches its KV-cache ceiling), the SLBF throughput is within 12%12\% of the uncompressed baseline (ratio of 0.88×0.88×), indicating that the runtime-decompression overhead becomes a small fraction of the total compute once it is amortized across enough in-flight tokens. Metric Orig SLBF Change Weights (GiB) 87.5 61.5 −30%-30\% KV-cache budget (GiB) 40.9 62.5 +53%+53\% KV-cache size (tokens) 333k 510k +53%+53\% Max concurrent requests 75.4 115.5 +53%+53\% Throughput (tok/s) 9,469 8,319 0.88×0.88× Table 6: SLBF runtime decompression vs. an uncompressed baseline on Mixtral-8x7B (2×H100 80 GB, P=2=2, BF16, vLLM 0.17.1). Memory figures are summed across both P ranks. Throughput is reported at the operating point at which the uncompressed baseline begins to saturate its KV-cache budget. Appendix C Source and licenses Base-model sources and licenses. All base models are obtained from their official HuggingFace repositories and used under their respective public licenses. We release no model weights ourselves; SLBF reconstructions and downstream evaluations operate locally and respect each base model’s terms of use. Specifically, we use Mixtral-8x7B-v0.1 (Apache 2.0), Qwen3-30B-A3B-Instruct-2507 (Apache 2.0), Qwen3.5-122B-A10B (Apache 2.0), Moonlight-16B-A3B-Instruct (MIT), and Gemma-4-26B-A4B-it (Gemma license). Dataset. For baseline methods (REAP, HC-SMoE, MC-SMoE, Sub-MoE, etc.), calibration uses 1024 sequences of 2048 tokens from the C4 corpus (Raffel et al., 2020) (ODC-By 1.0). All downstream evaluations are run via the EleutherAI lm-evaluation-harness (Gao et al., 2024), which we use under Apache 2.0 along with each task’s documented license. Code and evaluation scripts are available in our public repository. C.1 Evaluation Details Implementation. All evaluations use the vLLM backend with lm-evaluation-harness v0.4.11. Qwen3, Moonlight, and Mixtral use vLLM v0.17.1; Gemma-4 and Qwen3.5 use v0.19.1. Original and compressed checkpoints use identical settings. Tasks and prompting. For the instruction-tuned models, we evaluate mmlu_flan_cot_zeroshot, gpqa_diamond_cot_zeroshot, C-Eval, CMMLU, gsm8k, arc_challenge_chat, leaderboard_ifeval, mbpp, and HumanEval. C-Eval and CMMLU use ceval-valid and cmmlu for Qwen3, Moonlight, and Qwen3.5, and ceval-valid-chat and cmmlu-chat for Gemma-4. All tasks are zero-shot except gsm8k (5-shot) and mbpp (3-shot). Generative tasks use each model’s tokenizer-provided chat template; few-shot examples are separate user/assistant turns, while log-likelihood tasks disable chat formatting. Decoding settings. Qwen3 and Moonlight use greedy decoding. Gemma-4 uses temperature 1.01.0, top-p 0.950.95, and top-k 6464; Qwen3.5 uses temperature 1.01.0, top-p 0.950.95, top-k 2020, min-p 00, and presence penalty 1.51.5. Maximum output lengths are 4,096 tokens for Qwen3, task-dependent 1,024–6,144 tokens for Moonlight, task-dependent 32–32,768 tokens for Gemma-4, and 32,768 tokens for Qwen3.5.