Paper deep dive
Shape Mutating Expert Compression:LorExperts and BTExperts
Inesh Chakrabarti, Sourjya Roy, Bowen Bao, Thiago Crepaldi, Spandan Tiwari, Ashish Sirasao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/11/2026, 4:11:33 AM
Summary
The paper introduces LorExperts and BTExperts, a router-preserving compression method for Mixture-of-Experts (MoE) language models. LorExperts clusters experts into co-activation communities, keeping one full-precision dominant per cluster and representing others as low-rank corrections. This approach preserves the original router and all experts, avoiding the need for retraining. BTExperts organizes these dominants and corrections in a tree structure to amortize shared computation during inference. The method achieves ~50% compression with minimal accuracy loss on models like Qwen3-30B-A3B and Gemma-4-26B-A4B.
Entities (10)
Relation Signals (8)
LorExperts → clusters → Experts
confidence 95% · LorExperts... clusters experts, keeps one full-precision dominant per cluster
LorExperts → preserves → Router
confidence 95% · LorExperts retains all experts and the original router (no router retraining).
BTExperts → enables → Inference-time Amortization
confidence 90% · BTExperts... enables inference-time amortization of shared computation.
BTExperts → organizes → Dominants and Corrections
confidence 90% · BTExperts, a tree organization of dominants and corrections
LorExperts → represents → Low-rank Corrections
confidence 90% · represents the remaining members as low-rank corrections to their local dominant.
Experts → organizeinto → Co-activation Communities
confidence 88% · experts nonetheless organize into functional co-activation communities
Expert Pruning → requires → Router Retraining
confidence 85% · Expert pruning... require retraining the router
D^2-MoE → uses → Shared Component
confidence 85% · D^2-MoE... uses a shared (Fisher) mean with per-expert singular value decomposition (SVD) deltas.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mixture-of-Experts (MoE) language models deliver high capacity at low per-token compute, but deploying them cheaply requires compressing their many expert weight matrices. Expert pruning (e.g., REAP) and merging reduce cost but sacrifice accuracy and require retraining the router; low-rank delta decomposition of experts (e.g., D^2-MoE) preserves all experts and the router, but degrades sharply as the expert count grows because a single shared component cannot approximate many near-orthogonal experts. Because MoE expert weights are near-orthogonal, a single shared component (as in prior delta decomposition) scales poorly with the expert count; we show that experts nonetheless organize into functional co-activation communities that are decoupled from weight similarity. Building on this, we introduce LorExperts, a router-preserving compression method that clusters experts, keeps one full-precision dominant per cluster, and represents the remaining members as low-rank corrections to their local dominant. LorExperts retains all experts and the original router (no router retraining). At ~50% expert compression on Qwen3-30B-A3B and Gemma-4-26B-A4B, LorExperts preserves downstream accuracy and perplexity better than the baselines on most of the tasks; the margin over D^2-MoE grows with expert count E. We further give a reconstruction fine-tuning procedure for LorExperts, and BTExperts, a tree organization of dominants and corrections that enables inference-time amortization of shared computation.
Tags
Links
- Source: https://arxiv.org/abs/2608.07814v1
- Canonical: https://arxiv.org/abs/2608.07814v1
Trouble viewing inline? Open PDF directly →
Full Text
47,430 characters extracted from source content.
Expand or collapse full text
Shape Mutating Expert Compression: LorExperts and BTExperts Inesh Chakrabarti Sourjya Roy11footnotemark: 1 Bowen Bao Thiago Crepaldi Spandan Tiwari Ashish Sirasao Advanced Micro Devices inesh.chakrabarti, sourjya.roy, bowen.bao@amd.com thiago.crepaldi, spandan.tiwari, ashish.sirasao@amd.com Equal contribution. (August 7, 2026) Abstract Mixture-of-Experts (MoE) language models deliver high capacity at low per-token compute, but deploying them cheaply requires compressing their many expert weight matrices. Expert pruning (e.g., REAP) and merging reduce cost but sacrifice accuracy and require retraining the router; low-rank delta decomposition of experts (e.g., D2-MoE) preserves all experts and the router, but degrades sharply as the expert count grows because a single shared component cannot approximate many near-orthogonal experts. Because MoE expert weights are near-orthogonal, a single shared component (as in prior delta decomposition) scales poorly with the expert count; we show that experts nonetheless organize into functional co-activation communities that are decoupled from weight similarity. Building on this, we introduce LorExperts, a router-preserving compression method that clusters experts, keeps one full-precision dominant per cluster, and represents the remaining members as low-rank corrections to their local dominant. LorExperts retains all experts and the original router (no router retraining). At ∼ 50% expert compression on Qwen3-30B-A3B and Gemma-4-26B-A4B, LorExperts preserves downstream accuracy and perplexity better than the baselines on most of the tasks; the margin over D2-MoE grows with expert count E. We further give a reconstruction fine-tuning procedure for LorExperts, and BTExperts, a tree organization of dominants and corrections that enables inference-time amortization of shared computation. Shape Mutating Expert Compression: LorExperts and BTExperts Inesh Chakrabarti†thanks: Equal contribution. Sourjya Roy11footnotemark: 1 Bowen Bao Thiago Crepaldi Spandan Tiwari Ashish Sirasao Advanced Micro Devices inesh.chakrabarti, sourjya.roy, bowen.bao@amd.com thiago.crepaldi, spandan.tiwari, ashish.sirasao@amd.com 1 Introduction Large language models (LLMs) have improved rapidly as dense transformers have scaled to hundreds of billions of parameters (Brown et al., 2020; Chowdhery et al., 2023; Touvron et al., 2023). Scaling a dense model, however, means paying for every parameter on every token. Sparse Mixture-of-Experts (MoE) architectures ease this by adding capacity while holding the cost of each forward pass roughly constant. An MoE layer keeps E separate experts, and for every token a small router activates only k≪Ek E of them, leaving the rest idle (Jiang et al., 2024; Yang et al., 2025; Dai et al., 2024). Because total capacity is decoupled from the compute spent per token, models can grow to hundreds of billions of parameters without a comparable rise in inference floating-point operations (FLOPs). The difficulty is that this capacity lives almost entirely in the experts, and each expert is as large as a dense feed-forward block. Since only a few are active per token, MoE inference is limited by the memory traffic of loading expert weights rather than by arithmetic (Eliseev and Mazur, 2023). Recent architectures push this further, favoring more experts that are each smaller and more specialized, and configurations of E=128E=128 are now common (Dai et al., 2024; Yang et al., 2025). As this expert budget grows, serving the models affordably comes down to compressing the experts. The aim is to shrink their memory footprint without losing quality, and without disturbing the routing the model has already learned. 1.1 Related Work Expert pruning. Pruning removes low-importance experts. Candidates are chosen by activation frequency or by an importance criterion, as in REAP (Lasby et al., 2026). The approach is simple and reduces both storage and per-token compute. However, it discards model capacity and changes the set of experts the router can select. The router is therefore miscalibrated after pruning and must be adjusted or retrained. Accuracy also degrades on the capabilities carried by the removed experts. The relative merit of pruning and merging is benchmark-dependent. On discriminative metrics such as perplexity and multiple-choice accuracy, merging has been reported to do better (Li et al., 2024). On generative tasks, REAP (Lasby et al., 2026) finds the opposite. It attributes the gap to an irreducible error in merging, which arises from the loss of independent routing control over the combined experts. Expert merging. Merging fuses similar experts into fewer experts. M-SMoE (Li et al., 2024) groups experts by the cosine similarity of their router logits and merges each group by frequency-weighted averaging. HC-SMoE (Chen et al., 2025) instead clusters experts by the similarity of their output activations, which makes the grouping less dependent on routing statistics. Like pruning, merging shrinks the expert set and usually requires the router to be re-calibrated or retrained. Its quality depends on how losslessly similar experts can be fused. Low-rank / delta decomposition. A third family preserves all experts and the router. It writes each expert as a shared component plus a low-rank per-expert delta, in the spirit of LoRA (Hu et al., 2022). D2-MoE (Gu et al., 2025) uses a shared (Fisher) mean with per-expert singular value decomposition (SVD) deltas. SD-MoE (Huang et al., 2026) instead uses a spectral shared component. Because the expert set and the routing are unchanged, no router retraining is required, which is attractive in practice. Their quality, though, is bounded by how well a single shared component plus a small delta can approximate each expert. Neuron permutation symmetry. Feed-forward neurons have no canonical ordering. Two functionally related networks can match closely only after a permutation of their hidden units. This symmetry is well studied in model merging, for example the activation and weight matching used in Git Re-Basin (Ainsworth et al., 2023). We reuse this alignment machinery when forming inter-expert residuals. 1.2 Motivation We build on delta decomposition, the most deployment-friendly of these families, because it leaves every expert and the router in place. Its limitation is scale. A single shared component cannot represent a growing expert set, since one global anchor sits far from most experts in a large, diverse pool. MoE expert weights are also near-orthogonal, so a small low-rank delta still leaves a large residual. As a result, single-anchor methods degrade as E grows (Sec. 2). We address this with two ideas. The first is to use multiple local anchors. Instead of one global root, we cluster the experts and keep a full-precision dominant per cluster, so every anchor stays close to its members and the method scales with E. The second is to exploit permutation symmetry. Two experts may be similar only up to a neuron permutation, so we align each member’s neurons to its dominant before forming the low-rank residual (Sec. 3). Throughout, every expert and the original router are left untouched, with no rerouting and no router retraining. Contributions. • Analysis (Sec. 2). We characterize the “orthogonal-experts” problem: why a single shared component scales poorly as E grows, and we measure the functional structure that survives it—co-activation communities, and their projection-specific dissociation from weight similarity. We then quantify the two things a compression budget actually buys (Sec. 2.3): expert spectra decay only slowly, so rank is expensive, while routing is strongly concentrated—the 64 busiest of 128 experts absorb 94.5%94.5\% of routed visits—so retaining a well-chosen minority exactly is cheap. Together these motivate the asymmetric budget LorExperts uses. • LorExperts (Sec. 3). A router-preserving compression method that clusters experts, keeps one full-precision dominant per cluster, and encodes the remaining members as permutation-aligned low-rank corrections. Every expert and the original router are retained. • Fine-tuning (Sec. 4). A lightweight distillation step that fits only the low-rank factors to the original expert outputs, while the dominants and the router stay frozen. • BTExperts (Sec. 5). A tree organization of dominants and corrections that amortizes shared computation at inference time. 2 Analysis: The Orthogonal-Experts Problem Router: top-k over all E experts (unchanged — every expert stays individually addressable)Wd1W_d_1Wd1+B1A1W_d_1\!+\!B_1A_1 cluster 1Wd2W_d_2Wd2+B2A2W_d_2\!+\!B_2A_2 cluster 2Wd3W_d_3Wd3+B3A3W_d_3\!+\!B_3A_3 cluster k■ full-precision dominant (kept) ■ member ≈ dominant ++ rank-r correction BmAmB_mA_m Figure 1: LorExperts overview. Experts are clustered (by co-activation or weight similarity); each cluster keeps its highest-firing expert as a full-precision dominant WdW_d and represents every other member as Wd+BmAmW_d+B_mA_m with a rank-r correction. Unlike single-shared-component decomposition, which uses one global root, LorExperts uses k local dominants, so approximation quality does not collapse as the expert count E grows. The router and all E experts are preserved—no rerouting or retraining. This section motivates LorExperts (overview in Fig. 1) by characterizing why shared-component low-rank decomposition of MoE experts is hard, and identifying the structure that makes it tractable. Setup: the single-anchor budget. Let an MoE layer have E experts with weight matrices We∈ℝI×HW_e ^I× H. A single-shared-component method approximates We≈Wshared+ΔeW_e≈ W_shared+ _e with Δe _e low-rank, so its quality is bounded by how much of each expert the one shared component can carry and how low-rank the leftover residual is. The difficulty is an accounting one: a single anchor is amortized across the whole pool, so as E grows it must sit close to ever more, ever more diverse experts at once. For experts that are close to mutually orthogonal in weight space—as MoE experts are, and as the co-activation analysis below takes as its starting point—any one direction can align with only about 1/E1/E of the pool’s inter-expert variance, leaving a near-full-rank residual for the per-expert delta to absorb. Adding rank to Δe _e does not fix this, because the residual is not low-rank to begin with. The remedy we pursue is therefore not a better global anchor but more anchors: k local dominants, each close to its own members (Sec. 3). What remains is to decide which experts belong together. 2.1 What the co-activation structure reveals Although expert weights are near-orthogonal, the router induces rich, non-random functional structure. The expert co-activation graph (edges weighted by normalized pointwise mutual information, NPMI), built across 13 datasets on Qwen3-30B-A3B (Fig. 2), exposes several properties that bear directly on the design choices—and failure modes—of prior compression methods. Figure 2: Expert co-activation structure at Qwen3-30B-A3B layer 23 as a backbone graph: experts are nodes, and each strong co-activation edge is colored by the dataset with the largest NPMI (code=green, math/GSM8K=red, web/wikitext=cyan, …) and widened by its strength. Experts partition into stable communities even though their weights are near-orthogonal, and distinct domains induce visibly specialized structure. A raw heatmap view and clique statistics appear in Appendix B. • Stable communities. Experts partition into consistent co-activation communities that persist across datasets—natural groups that LorExperts clusters over (motivating the Coact-NPMI / MsoftM_soft metrics, Sec. 3). • Depth-varying granularity. The number of co-activation communities grows with network depth. A uniform compression budget is therefore suboptimal, motivating per-layer treatment. • Domain specialists. Some communities are domain-general while others are strongly domain-specific: code and math (GSM8K) induce the most distinctive routing (visible as the colored communities in Fig. 2), whereas web text (C4, Wikipedia) tracks the consensus. These specialists are rare yet load-bearing for their domain. Consequences for prior methods—and for LorExperts. This structure is exactly what the prior families endanger. Frequency-based pruning (Sec. 1.1) discards precisely the rare domain specialists that carry code/math ability, and merging (Sec. 1.1) blurs distinct communities—both then force router re-calibration. LorExperts instead keeps every expert and the router intact, preserving this functional structure by construction. Moreover, the structure lives in the routing while the weights are near-orthogonal—a decoupling we measure directly (Sec. 2.2, Table 1)—so single-shared-component weight decomposition (Sec. 1.1) cannot exploit it, which is why LorExperts uses local per-cluster dominants rather than one global root. 2.2 The functional–weight dissociation is projection-specific Do co-firing experts also have similar weights? We correlate each expert pair’s co-activation (NPMI) with its weight cosine similarity, per layer and per projection (Table 1). The correlation is small but consistently significant for the gate projection (r≈0.08r≈ 0.08–0.210.21, p<10−3p<10^-3) and effectively zero for up and down. Table 1: Correlation between expert co-activation (NPMI) and weight cosine similarity, by layer and projection (Qwen3-30B-A3B). p∗<10−3^**p<10^-3, p∗<0.05^*p<0.05. Layer gate up down mean L5 +0.208∗+0.208^** +0.017+0.017 −0.026∗-0.026^* +0.164∗+0.164^** L11 +0.085∗+0.085^** +0.010+0.010 +0.006+0.006 +0.078∗+0.078^** L23 +0.133∗+0.133^** −0.005-0.005 +0.009+0.009 +0.115∗+0.115^** L35 +0.178∗+0.178^** −0.001-0.001 +0.020+0.020 +0.160∗+0.160^** L47 +0.195∗+0.195^** +0.007+0.007 +0.011+0.011 +0.167∗+0.167^** The pattern is interpretable. The gate projection selects which inputs activate an expert, so co-firing experts share a weak alignment in what excites them; their actual computation (up, down) is essentially orthogonal. Experts that fire together thus respond to similar contexts but compute different things—complementary, not redundant. Even the gate correlation is small (r≤0.21r≤ 0.21), so experts remain largely near-orthogonal in weight space. This is precisely why routing-space structure is not recoverable by a shared-component weight decomposition, and why LorExperts corrects each expert against a local dominant rather than assuming a single shared low-rank basis across experts. 2.3 What the budget can buy: spectra and routing mass Two measurable quantities decide how a compression budget for an MoE layer is best spent: how fast expert spectra decay, which sets what a unit of rank buys, and how concentrated the routing is, which sets what keeping a single expert exact buys. Measured on Qwen3-30B-A3B, together they argue for exactly the asymmetric allocation LorExperts adopts. Spectra decay, but slowly. Table 2 reports the fraction of squared Frobenius energy a rank-r truncation retains, averaged over the three projections of 12 experts at layers 5, 23 and 40. There is genuine low-rank structure: rank 64 retains 26.0%26.0\% where a perfectly flat spectrum would retain only 64/768=8.3%64/768=8.3\%. But the decay is shallow—reaching 72%72\% of the energy already costs rank 311, i.e. 40%40\% of the maximum rank 768. Rank is expensive on these matrices, so a budget spread thinly and uniformly across all E experts buys comparatively little per expert; it is worth asking whether some of it is better spent keeping selected experts exactly. Table 2: Fraction of squared Frobenius energy retained by a rank-r truncation of an expert matrix (I=768I=768, H=2048H=2048, so rank 768768 is lossless), mean over gate/up/down of 12 experts per layer. The last row is what a flat spectrum (r/768r/768) would give. Layer r=64r=64 r=200r=200 r=311r=311 r=450r=450 r=558r=558 L5 0.2660.266 0.5650.565 0.7240.724 0.8600.860 0.9290.929 L23 0.2660.266 0.5660.566 0.7250.725 0.8600.860 0.9290.929 L40 0.2480.248 0.5480.548 0.7120.712 0.8530.853 0.9260.926 mean 0.2600.260 0.5600.560 0.7200.720 0.8580.858 0.9280.928 flat spectrum 0.0830.083 0.2600.260 0.4050.405 0.5860.586 0.7270.727 Routing is strongly concentrated. Expert utilization is meanwhile far from uniform. Over five layers, the 64 busiest of the 128 experts absorb 94.5%94.5\% of routed token–expert visits, against the 50%50\% that uniform routing would give, and 18–30 experts per layer are effectively dead (<10−4<10^-4 of traffic). Concentration this strong is what makes an asymmetric budget attractive: if a minority of experts carries almost all of the traffic, then holding exactly those experts in full precision costs a well-understood share of the budget and removes approximation error where it is encountered most often. Table 3: Routed token–expert visits absorbed by the 6464 busiest of E=128E=128 experts, measured on WikiText-2. Uniform routing would give 0.5000.500. “Dead” counts experts receiving <10−4<10^-4 of traffic. Layer top-64 share dead experts L5 0.9280.928 1818 L12 0.9500.950 2222 L23 0.9420.942 3030 L31 0.9610.961 2727 L40 0.9440.944 2626 mean 0.9450.945 24.624.6 Taken together the two measurements motivate the shape of LorExperts: because rank is expensive, uniform low-rank treatment of every expert is a poor use of the budget; because routing is concentrated, keeping a well-chosen minority exact covers most of what the router actually does. Retaining a full-precision dominant per cluster and spending only low-rank corrections on the remainder follows directly, and it is why LorExperts selects each cluster’s highest-firing member rather than its geometric center. One caveat qualifies how the dominant set should be read. Utilization here is measured on WikiText-2, and Sec. 2 shows that code and math induce visibly different communities, so the busy set is partly domain-dependent. This is a reason to keep every expert addressable rather than pruning the tail: the rare domain specialists are precisely the experts a single-domain utilization estimate will rank last. 2.4 Implication for method design Two consequences drive LorExperts. First, use multiple local anchors (per-cluster dominants) rather than one global root, so each anchor is close to its members and the residual it must correct is small—and the approach scales with E instead of degrading (ablated in Appendix A). Second, by Sec. 2.3, split the budget asymmetrically rather than uniformly: rank buys little on these spectra, while routing mass is concentrated enough that full-precision dominants pay for themselves—so a dominant should be chosen for the traffic it absorbs. 3 LorExperts: Router-Preserving Low-Rank Expert Compression 3.1 Representation Consider one MoE layer with E experts, each a set of weight matrices (gate, up, down) that we write jointly as WeW_e. LorExperts partitions the experts into k clusters. Within each cluster it keeps one dominant expert at full precision and represents every other member m as a low-rank correction of its dominant, Wm≈Wdominant+BmAm,rank(BmAm)=r, gatheredW_m\;≈\;W_dominant+B_mA_m,\\ rank(B_mA_m)=r, gathered with factors Bm∈ℝd×rB_m ^d× r, Am∈ℝr×d′A_m ^r× d . The dominant is stored exactly; each member costs only Bm,AmB_m,A_m. With k≈E/2k≈ E/2 and r=64r=64, expert parameters are reduced by roughly one half; the ratio is controlled by k and r. 3.2 Compression pipeline Stage 1 — Router profiling. We pass a small calibration set (C4, 64 sequences of length 2048) through the model with router hooks, accumulating per-layer firing counts NeN_e. These counts seed clustering and dominant selection. Profiling only reads the router: LorExperts keeps all E experts and leaves the gate unchanged, unlike pruning or merging. Stage 2 — Clustering. Per layer, we form a pairwise distance matrix D=1−MD=1-M with the diagonal zeroed. We then run frequency-seeded k-medoids: the medoids start at the highest-firing experts, and the assign/update loop iterates to convergence. Seeding from frequently-used experts biases the dominants toward experts the router actually selects. Each cluster’s dominant is its highest-firing member, with ties broken by co-activation centrality, and the remaining members are queued for reconstruction. An optional protection set can be pulled out as singleton clusters and kept at full precision. We compare three choices of the distance M empirically (Sec. 6): weight cosine (WS-Frob), co-activation NPMI (Coact-NPMI), and soft co-activation (MsoftM_soft). Stage 3 — Alignment and low-rank decomposition. We apply three steps to each non-dominant member. (i) Neuron alignment. An expert’s hidden feed-forward network (FFN) has no canonical ordering of its intermediate neurons, so a member may store corresponding neurons at different indices than its dominant. This inflates the naive residual Wm−WdominantW_m-W_dominant, which can be full-rank even when the experts match up to a permutation. Aligning networks by such permutations is well established in model merging (Ainsworth et al., 2023), and has been applied to experts before merging in MoE compression (Li et al., 2024). We therefore align the member to its dominant before decomposing. Treating each neuron as the concatenation of its gate, up, and down columns, we score neuron i of the dominant against neuron j of the member by the squared distance Cij=‖ai−bj‖22=∥ai∥2+∥bj∥2−2ai⊤bj,C_ij= a_i-b_j _2^2= a_i ^2+ b_j ^2-2\,a_i b_j, (1) where ai=Wdom[:,i]a_i=W_dom[:,i] and bj=Wm[:,j]b_j=W_m[:,j] are the concatenated neuron vectors. We then seek the permutation P of the member’s neurons that minimizes the total matching cost, P⋆=argminP∑iCi,P(i),P = _P _iC_i,\,P(i), (2) which we solve exactly with the Jonker–Volgenant algorithm. Applying P⋆P gives the aligned member WmPW_m^P. The reported results use this alignment. (i) Truncated-SVD residual. We form the aligned residual and take its rank-r SVD, R=WmP−Wdominant≈UrΣrVr⊤,R=W_m^P-W_dominant≈ U_r _rV_r , (3) computed with a randomized solver (r=64r=64). Splitting the singular values symmetrically yields the low-rank factors Bm=UrΣr1/2,Am=Σr1/2Vr⊤,so BmAm≈R. gatheredB_m=U_r _r^1/2, A_m= _r^1/2V_r ,\\ so B_mA_m≈ R. gathered (4) (i) Installation. The member is replaced by a module that computes Wdominant+BmAmW_dominant+B_mA_m at inference, while the dominants and the router remain untouched. Stage 4 — Reconstruction fine-tuning. A short output-matching pass refines the factors (Sec. 4); it updates only Bm,AmB_m,A_m. Variants. The framework is parameterized by the clustering distance M and an optional protection set, giving four variants we evaluate side by side (Sec. 6, Tables 4–5): • WS-Frob clusters on the weight-space Frobenius distance ∥Wi−Wj∥F W_i-W_j _F. It uses no router information, so it serves as a routing-agnostic reference. • WS-Frob+protK8 uses the same weight-space clustering but keeps eight full-precision protected experts per layer. This isolates the effect of protection on an otherwise routing-agnostic variant. • Coact-NPMI clusters on single-mixture routing co-activation, measured as 1−NPMI1-NPMI over a mixed calibration set. It is our first routing-aware variant. • MsoftM_soft clusters on the cross-domain soft consensus 1−Msoft1-M_soft, optionally with the router-identified protection list, and is our strongest variant. Because k and the protection set are tunable, we report it at several operating points to trace the compression–quality frontier. All variants share the pipeline above and differ only in the Stage-2 distance and whether a protection set is supplied. 3.3 Storage For a cluster with a dominant and c members, storage is one full matrix plus c rank-r factor pairs, versus c+1c+1 full matrices uncompressed. Summed over clusters, the layer’s expert parameters scale with the number of dominants (set by k) plus r times the number of members. Thus k and r jointly set the compression ratio, and every expert remains individually addressable by the unchanged router. 4 Reconstruction Fine-Tuning for LorExperts Truncated-SVD (Stage 3) minimizes weight-space error; a short output-matching refinement then recovers the error that matters for the token distribution—at negligible cost. Objective. On cached calibration activations x, we fit each member’s factors so its output matches the original expert’s, minBm,Am∥(Wdominant+BmAm)x−Wmx∥F2, _B_m,A_m\; (W_dominant+B_mA_m)\,x-W_mx _F^2, training only the rank-r factors Bm,AmB_m,A_m; the dominant weights and the router are frozen. Why it is cheap. Three properties make this the least costly stage of LorExperts: (i) it updates only the small rank-r factors—not the base model and not the router, so no gradients flow through either; (i) the objective is local and per-member and is evaluated on pre-cached activations, so there are no full-model forward/backward passes during fitting—just small matmuls; (i) it is an activation-weighted low-rank regression, which admits a near-closed-form solution: whitening by the activation second moment and taking a (generalized) SVD of the residual recovers the optimal rank-r factors directly, in place of iterative gradient steps. Its wall-clock cost, and its quality contribution relative to the SVD initialization (Stage-3-only vs. Stage-3++4), are reported with the calibration cost (Sec. 6). We expect the SVD initialization to do the heavy lifting and this stage to be a cheap refinement. 5 BTExperts: Tree Organization for Inference-Time Amortization BTExperts organizes the dominants and their corrections in a binary tree. We emphasize up front what BTExperts is not: it does not improve compression quality. On near-orthogonal experts, sharing corrections across a hierarchy does not beat flat per-cluster decomposition: in a controlled study on Mixtral-8×7B (E=8E=8) at matched 7.5×7.5× compression, the hierarchical organization matches flat decomposition within noise in perplexity (PPL; ΔPPL=0.099 =0.099 vs. 0.0980.098 after reconstruction fine-tuning; 0.1120.112 vs. 0.1050.105 zero-shot, 3 seeds; Appendix A). Its sole purpose is therefore inference-time amortization: components shared by several selected experts are computed once and reused. 5.1 Amortized forward pass When multiple selected experts share a cluster (dominant) or a tree ancestor, the shared computation—xWdominant⊤xW_dominant and any shared correction factors—is computed once per token and broadcast to those experts, rather than recomputed per expert. The tree (Fig. 3) makes this sharing explicit: experts on the same path share the ancestors’ contributions. WrootW_rootB6A6B_6A_6B7A7B_7A_7−1-1+1+1B4A4B_4A_4B5A5B_5A_5B2A2B_2A_2B3A3B_3A_3−1-1+1+1−1-1+1+1e0e_0e1e_1e2e_2e3e_3e4e_4e5e_5e6e_6e7e_7 Figure 3: BTExperts tree (E=8E=8). Shared components near the root are computed once and amortized across experts that share a path; leaves are experts. The highlighted red path is path(e3e_3). The tree is an inference-time organization, not a compression-quality mechanism. 5.2 Inference cost: theoretical FLOPs Standard top-k routing evaluates k full experts per token, costing k⋅2IHk· 2IH FLOPs and loading k full weight tensors. Pruning (REAP) does not reduce this: top-k still selects k full survivors, so per-token FLOPs and high-bandwidth memory (HBM) reads are unchanged (a 0×0× reduction—pruning saves storage, not per-token compute). BTExperts instead amortizes: when co-selected experts share a dominant, that dominant is computed once and only the cheap rank-r corrections are added per expert (Fig. 4). For such a group, speedup=k⋅2IH2IH+k⋅2r(I+H)≈ 1.9×speedup\;=\; k· 2IH2IH+k· 2r(I+H)\;≈\;1.9× for Mixtral-8×7B gate_proj (k=2k=2, I=14336I=14336, H=4096H=4096, r=64r=64). Because this projection is memory-bandwidth-bound (arithmetic intensity ≈1≈ 1 FLOP/byte), the reduction is kernel-independent. Two caveats keep it honest: the benefit is routing-dependent (it applies only when the router co-selects experts from the same cluster), and it is shared with any single-shared-component method—D2-MoE can amortize its shared base likewise. (a) Originaltoken xxWe1W_e_1117 MBWe2W_e_2117 MBout235M FLOPs, 235 MB(b) REAP (pruned)token xxWe1W_e_1117 MBWe2W_e_2117 MBout235M FLOPs, 235 MBsame per-token cost(c) LorExperts++BTExpertstoken xxWdomW_dom117 MBsharedBe1Ae1B_e_1A_e_12.4 MBBe2Ae2B_e_2A_e_22.4 MBout122M FLOPs, 122 MB≈ 1.9× fewer Figure 4: Per-token forward pass (gate_proj, top-2 routing). (a) Original: two full expert matmuls. (b) REAP: identical per-token cost—pruning removes experts but top-k still runs k full survivors. (c) LorExperts++BTExperts for two co-selected experts sharing a dominant: the dominant is computed once and reused, plus two cheap rank-r corrections, cutting FLOPs and HBM reads ≈1.9×≈ 1.9× (kernel-independent). Adapted from the prior draft’s FLOP analysis. 6 Results We evaluate on Qwen3-30B-A3B (E=128E=128) and Gemma-4-26B-A4B at ∼ 50% expert compression, reporting perplexity on Wikitext-103 and downstream accuracy via the LM Evaluation Harness (MMLU, ARC-C, HellaSwag, WinoGrande, PIQA). We compare against merging (M-SMoE, HC-SMoE), pruning (Freq-Prune, REAP), and delta decomposition (D2-MoE), all at matched compression (Tables 4–5). Table 4: Qwen3-30B-A3B (∼ 50% expert compression; uncompressed PPL =8.50=8.50). Method Family PPL ↓ MMLU ARC-C HellaS WinoG PIQA Uncompressed — 8.50 77.8 52.6 59.6 71.0 79.4 M-SMoE merge 68.61 24.7 20.8 32.1 55.1 60.4 HC-SMoE merge 39.50 43.3 27.6 32.9 54.1 61.4 Freq-Prune prune 29.64 44.6 21.9 52.9 63.2 65.1 REAP prune 34.14 32.1 23.5 38.9 54.5 63.3 D2-MoE delta 23.58 45.7 40.8 51.7 66.9 70.5 LorExperts (WS-Frob) ours 12.36 42.4 32.4 55.5 71.1 76.4 LorExperts (WS-Frob+protK8) ours 11.79 53.7 41.1 55.7 71.0 76.9 LorExperts (Coact-NPMI) ours 12.20 54.1 37.1 52.6 68.4 75.3 LorExperts (Msoft) @40.6% ours 11.99 54.9 39.1 52.4 68.8 74.3 LorExperts (Msoft) @51% ours 15.37 46.0 28.8 45.5 63.8 68.4 Table 5: Gemma-4-26B-A4B (∼ 50% expert compression; uncompressed PPL ≈7.15≈ 7.15). Method Family PPL ↓ MMLU ARC-C HellaS WinoG PIQA Uncompressed — ∼ 7.15 74.3 64.5 63.4 76.1 82.2 M-SMoE @50% merge 14.94 27.8 38.9 74.8 73.7 79.2 HC-SMoE @50% merge 25.20 35.6 30.0 46.4 55.2 64.1 Freq-Prune @50% prune 11.57 34.8 48.5 79.6 74.6 81.3 REAP @50% prune 15.09 27.4 36.3 64.9 63.5 72.2 D2-MoE @50% delta 21.59 39.0 37.4 56.4 66.7 70.3 LorExperts (WS-Frob) @49.9% ours 13.50 26.3 36.4 74.4 74.4 78.0 LorExperts (WS-Frob+protK8) @49.9% ours 14.66 51.1 45.2 71.9 72.0 76.0 LorExperts (Coact-NPMI) @49.9% ours 14.42 50.1 42.7 55.5 68.0 68.8 LorExperts (Msoft) @41.4% ours 13.32 54.6 50.1 63.9 70.6 72.9 LorExperts (Msoft) @51.6% ours 18.54 47.4 38.8 48.4 62.4 65.8 Reading the tables. Freq-Prune is the strongest prior baseline, but LorExperts preserves knowledge and reasoning accuracy (MMLU, ARC-C, HellaSwag) markedly better: it leads across the board on Qwen3, and on both models holds MMLU/ARC well above the pruning and merging baselines, whereas D2-MoE degrades sharply at this expert count. On Gemma, Freq-Prune retains lower PPL and higher commonsense (WinoGrande/PIQA), so LorExperts’s advantage is in preserved capability rather than a clean sweep of every metric. As a side observation consistent with the near-orthogonality analysis (Sec. 2), LorExperts’s margin over single-root decomposition (D2-MoE) widens as the expert count E grows—the regime where one shared component is least able to fit the experts. Which variant to use. The routing-aware variants win: the soft cross-domain consensus MsoftM_soft is strongest (best MMLU on both models at ∼ 40–50% compression), with Coact-NPMI close behind, because clustering by co-activation groups experts that are functionally related rather than merely close in weight space. Protecting the highest-firing experts also helps sharply (WS-Frob MMLU 42.4→53.742.4→ 53.7 on Qwen3 with protK8), since those load-bearing specialists are the costliest to approximate. 6.1 Compression–quality frontier The tables fix a single operating point, but the compression ratio is a knob we can turn. In Figure 5 we sweep the Coact-NPMI variant on Qwen3-30B-A3B from 0 to about 50%50\% expert compression and track perplexity alongside the five downstream tasks. Accuracy changes little through low and moderate compression, and the clearer drops set in only as the ratio nears 50%50\%; perplexity climbs slowly over the same range. Quality therefore falls off smoothly rather than all at once, and the 50%50\% end of the sweep reproduces the Coact-NPMI row of Table 4. In practice the ratio can then be tuned to a quality target rather than fixed in advance. Figure 5: Compression–quality frontier for LorExperts (Coact-NPMI) on Qwen3-30B-A3B. As the expert compression ratio increases (top to bottom), downstream accuracy (MMLU, ARC-C, HellaSwag, WinoGrande, PIQA; bottom axis) stays near its uncompressed value through moderate compression and falls off mainly near 50%50\%, while perplexity (dashed, top axis) rises gradually. 6.2 Calibration cost Calibration time is dominated by fitting the low-rank factors, and this admits a large, analytically guaranteed speedup. The reconstruction fine-tuning of Sec. 4 reaches the factors through ∼ 500 iterative gradient steps over cached activations. That objective, however, is an activation-weighted low-rank regression, which has a closed-form optimum: whitening by the activation second moment and taking a single (generalized) SVD of the residual recovers the optimal rank-r factors directly. Replacing the ∼ 500 optimization passes with one whitened SVD collapses the factor-fitting stage from O(T)O(T) passes to O(1)O(1)—roughly two orders of magnitude (∼ 100×100×) less compute for that stage—while attaining the same rank-r optimum, so accuracy is unchanged. Because factor fitting dominates the pipeline, this turns whole-model calibration into a short offline step rather than a bottleneck. We report this speedup as a relative multiplier; absolute wall-clock times are omitted. 7 Discussion Why LorExperts works. The gains trace to one design choice grounded in Sec. 2: multiple local anchors (per-cluster dominants) instead of one global root, so each anchor is close to its members and the method scales with E where single-root decomposition degrades. This preserves all experts and the original router. Hessian-weighted fitting (ongoing work). Sec. 6 introduces the activation-weighted closed-form solve as a way to replace the iterative reconstruction pass at equal quality. We note here that the same device is also a quality improvement in its own right over the plain-SVD default, and that it generalizes beyond the corrections. The default fit minimizes weight-space error ∥W−W^∥F W- W _F, which is not the quantity a layer emits. The error reaching the next layer is ∥X(W−W^)⊤∥F X(W- W) _F for the calibration activations X the router actually sends to that expert—a per-expert quantity in an MoE, since each expert sees only its own slice of the tokens. With Ge=Xe⊤Xe=LeLe⊤G_e=X_e X_e=L_eL_e , minimizing the emitted error is again a truncation problem—of WLeWL_e rather than W—so it costs the same rank, the same storage r(I+H)r(I+H), and the same two matmuls at inference. In preliminary runs on Qwen3-30B-A3B this improves perplexity at matched storage for every variant we have tried. One MoE-specific caveat: GeG_e has H2H^2 entries estimated from the tokens a single expert receives, so it needs substantially more calibration data than a diagonal or norm-based statistic, and experts that see fewer tokens than H rely on damping. The same reweighting extends to the shared components themselves, and there it also removes a structural weakness. For a shared component C serving expert set S with corrections ReR_e, the optimal Hessian-weighted choice is closed-form, C⋆=[∑e∈S(We−Re)Ge][∑e∈SGe]−1,C = [ _e∈ S(W_e-R_e)\,G_e ] [ _e∈ SG_e ]^-1, which is a traffic-weighted combination of the cluster rather than a member of it. Because C=0C=0 lies in the feasible set, a solved anchor can never be worse than using no anchor at all—a guarantee that selecting an existing expert as the dominant does not provide, and one that matters precisely when experts are near-orthogonal and the residual against another expert is larger than the expert itself. Note that a solved anchor is no longer one of the experts, so the former dominant also needs a correction; at k=64k=64 this is roughly a 10%10\% increase in the expert budget, which must be returned by a small reduction in r. We are pursuing the full treatment, including its interaction with the BTExperts tree, where the component values change but the sharing structure—and hence the amortization factor—does not. Limitations. 1. Calibration cost. Calibration is dominated by the per-member SVD and the short reconstruction pass; a closed-form activation-weighted solve can replace the SVD (Sec. 6, Gate H) but is not yet the default. 2. Inference amortization is bounded. The BTExperts amortization benefit is routing-dependent and partly shared with any shared-component method; we report only fairly-baselined numbers (Sec. 5) and make no strawman speedup claims. 8 Conclusion We presented LorExperts, a router-preserving method for compressing Mixture-of-Experts weight matrices. Motivated by a mechanistic analysis of the “orthogonal experts” problem—experts are near-orthogonal in weight space yet organize into functional co-activation communities—LorExperts keeps a full dominant per cluster and represents other experts as low-rank corrections to that local dominant, retaining all experts and the original router with no rerouting or router retraining. A lightweight reconstruction fine-tuning refines the factors, and BTExperts organizes dominants/corrections for inference-time amortization. Across Qwen3-30B-A3B and Gemma-4-26B-A4B at ∼ 50% expert compression, LorExperts preserves downstream capability—particularly knowledge- and reasoning-heavy accuracy (MMLU, ARC-C, HellaSwag)—better than pruning, merging, and single-shared-component decomposition (D2-MoE), and its advantage over the latter grows with the expert count. Future work. Higher-compression and full-model operating points; generation-benchmark evaluation; faster closed-form factor fitting at scale; and using the BTExperts structure for upcycling / MoE construction from dense checkpoints. References Ainsworth et al. (2023) Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2023. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations (ICLR). Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS). Chen et al. (2025) I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, and Chun-Yi Lee. 2025. Retraining-free merging of sparse MoE via hierarchical clustering. In International Conference on Machine Learning (ICML), volume 267, pages 8594–8620. Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113. Dai et al. (2024) Damai Dai, Chengqi Deng, Chenggang Zhao, R.X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297. Association for Computational Linguistics. Eliseev and Mazur (2023) Artyom Eliseev and Denis Mazur. 2023. Fast inference of mixture-of-experts language models with offloading. arXiv preprint arXiv:2312.17238. Gu et al. (2025) Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. 2025. Delta decompression for MoE-based LLMs compression. In International Conference on Machine Learning (ICML), volume 267, pages 20497–20514. Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR). Huang et al. (2026) Ruijun Huang, Fang Dong, Xin Zhang, Hengjie Cao, Zhendong Huang, Anrui Chen, Jixian Zhou, Mengyi Chen, Yifeng Yang, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin Lv, Robert P. Dick, Yuan Cheng, Fan Yang, Tun Lu, Chun Zhang, and Li Shang. 2026. SD-MoE: Spectral decomposition for effective expert specialization. arXiv preprint arXiv:2602.12556. Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. Lasby et al. (2026) Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, and Vithursan Thangarasa. 2026. REAP the experts: Why pruning prevails for one-shot MoE compression. In International Conference on Learning Representations (ICLR). Li et al. (2024) Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. 2024. Merge, then compress: Demystify efficient SMoE with hints from its routing policy. In International Conference on Learning Representations (ICLR). Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Appendix A Ablations Tree vs. flat (quality). A recurring question is whether organizing the low-rank corrections as a hierarchy (BTExperts) helps compression quality relative to flat per-cluster decomposition. Table 6 answers it in a controlled study on Mixtral-8×7B (E=8E=8) at matched compression: the two are indistinguishable within noise, both zero-shot and after reconstruction fine-tuning. This confirms that BTExperts is a quality-neutral inference-time organization (Sec. 5), not a compression-quality mechanism—its value is amortization, not accuracy. Table 6: Tree (BTExperts) vs. flat per-cluster decomposition on Mixtral-8×7B (E=8E=8; layer 16, rank 32, 3 seeds), at matched compression. Δ is the increase over the uncompressed baseline (lower is better); the hierarchy matches flat decomposition within noise. Method comp Δ (zero-shot) Δ (+recon-FT) BTExperts (tree) 7.5×7.5× 0.112±0.0120.112±0.012 0.099±0.0110.099±0.011 D2-MoE (flat) 7.4×7.4× 0.105±0.0110.105±0.011 0.098±0.0100.098±0.010 Appendix B Additional Co-Activation Views The backbone graph in the main text (Fig. 2) summarizes the co-activation structure at layer 23. Figure 6 shows the same kind of structure as a raw NPMI heatmap at a late layer, and Fig. 7 reports how the dense co-activation cliques shift with network depth. Figure 6: Union-mean NPMI co-activation matrix at Qwen3-30B-A3B layer 47 (over 13 datasets), experts reordered by community: co-firing experts form clear diagonal blocks even though their weights are near-orthogonal. Figure 7: Per-layer counts of dense expert cliques by size (Qwen3-30B-A3B). Both the number and the size of co-activation cliques vary with depth, so a uniform per-layer compression budget is suboptimal.