Paper deep dive
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
Ali Janati, Kaoutar El Maghraoui, Xinyi Luo, Wenyuan Shen, Owen Zou, Yankai Mao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/11/2026, 4:19:19 AM
Summary
This paper proposes a method for pruning experts in Mixture-of-Experts (MoE) models by measuring router sensitivity under lightweight fine-tuning. Instead of full fine-tuning, the authors use parameter-efficient methods like LoRA or IA3 to induce small changes in router weights or logits. Experts with the smallest induced changes (lowest sensitivity) are pruned. The study demonstrates that router-only LoRA on Mixtral-8x7B-Instruct effectively identifies prunable experts, reducing memory by 49% and latency by 37% with minimal accuracy loss compared to magnitude-based or random pruning. The method is also shown to transfer to Qwen1.5-MoE.
Entities (7)
Relation Signals (6)
Mixtral 8x7B Instruct → evaluatedon → MMLU-Pro
confidence 96% · On Mixtral-8×7B-Instruct (44.83% MMLU-Pro)...
LoRA → usedfor → Router Sensitivity Measurement
confidence 95% · We briefly fine-tune with a parameter-efficient adapter, rank experts by the induced l2 router change...
Router Sensitivity → identifies → Prunable Experts
confidence 94% · Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
Router-guided Pruning → reduces → Memory Usage
confidence 92% · reduces memory by 49% and per-token latency by 37%
LoRA → outperforms → All-module LoRA
confidence 90% · router-only LoRA ... outperforms all-module LoRA at matched rank
Qwen1.5-MoE → transfersto → Mathematics Benchmarks
confidence 88% · The criterion also transfers to Qwen1.5-MoE fine-tuned for mathematics...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert. Recent theory shows that pruning experts with the smallest router-norm changes during fine-tuning can preserve accuracy, but assumes full fine-tuning. We test whether lightweight adaptation can recover this signal. We briefly fine-tune with a parameter-efficient adapter, rank experts by the induced $\ell_2$ router change, and prune the least-changed experts in one shot. On Mixtral-8$\times$7B-Instruct (44.83% MMLU-Pro), router-only LoRA trains 0.002% of parameters and outperforms all-module LoRA at matched rank with half the experts removed (27.54% vs. 24.42%); signal quality declines as adaptation spreads to attention and expert weights. Accuracy improves monotonically with LoRA rank, reaching 28.76%. IA3, which leaves router weights frozen, matches direct router adaptation, whereas unconstrained additive adapters degrade the signal. Router-guided MMLU-Pro accuracy decays quasi-linearly rather than collapsing, remains nearly 1.8 times that of magnitude-based or random pruning at maximal compression, and reduces memory by 49% and per-token latency by 37%. At 25% compression, retention is competitive with methods using full activation statistics. The criterion also transfers to Qwen1.5-MoE fine-tuned for mathematics, retaining 49.7% mean accuracy over eleven benchmarks with half the experts removed while random pruning falls to single digits. Router sensitivity under lightweight fine-tuning therefore makes provably motivated expert pruning practical at scale.
Tags
Links
- Source: https://arxiv.org/abs/2608.07890v1
- Canonical: https://arxiv.org/abs/2608.07890v1
Trouble viewing inline? Open PDF directly →
Full Text
84,642 characters extracted from source content.
Expand or collapse full text
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models Ali Janati1 Kaoutar El Maghraoui2 Xinyi Luo2 Wenyuan Shen2 Owen Zou2 Yankai Mao2 1Data Science Institute, Columbia University, New York, NY, USA 2Department of Computer Science, Columbia University, New York, NY, USA Abstract Mixture-of-Experts (MoE) models decouple parameter count from per-token compute, but deployment requires hosting every expert in memory. Recent theoretical work shows that experts whose router weights change least during fine-tuning can be pruned with provable accuracy preservation, yet the guarantee assumes full fine-tuning, which at scale costs precisely what pruning is meant to save. We study whether this signal can be elicited far more cheaply. Our recipe is to briefly fine-tune with a parameter-efficient adapter, rank experts by the ℓ2 _2 change induced in the router, and prune the least-changed experts in one shot. On Mixtral-8×7B-Instruct (44.83% MMLU-Pro), restricting LoRA to the router weights, 0.002% of parameters trained, outperforms adapting all modules at matched rank (27.54% vs. 24.42% with half the experts removed), and signal quality declines steadily as adaptation spreads to attention and expert weights. Fidelity rises monotonically with LoRA rank, reaching 28.76% at the highest rank tested. Multiplicative modulation (IA3), which writes to no router weight at all, matches direct router adaptation, whereas unconstrained additive adapters degrade the signal. MMLU-Pro accuracy under router-guided pruning decays quasi-linearly rather than collapsing, remaining nearly 1.8 times that of magnitude-based or random pruning at maximal compression, while memory usage falls by 49% and per-token latency by 37%. On benchmarks shared with recent one-shot pruning studies, retention at 25% compression is competitive with methods built on full activation statistics. The criterion transfers to Qwen1.5-MoE fine-tuned for mathematical reasoning, where it retains 49.7% mean accuracy over eleven mathematics benchmarks with half the experts removed while random pruning collapses to single digits. Router sensitivity measured under lightweight fine-tuning thus makes provably motivated expert pruning practical at scale. 1 Introduction Mixture-of-Experts (MoE) architectures decouple parameter count from per-token compute by routing each token to a small subset of expert subnetworks [28, 19, 9]. This has made them the dominant design for frontier open-weight language models [16, 7]. The saving, however, is in computation rather than in memory: every expert must remain resident even though only a few are active per token. A Mixtral-8×7B checkpoint occupies 46.7B parameters while activating 13B, and this gap is what restricts deployment on constrained hardware. Expert pruning, the removal of entire experts from each MoE layer, addresses the memory cost directly and reduces parameters, memory and latency in proportion to the experts removed. The difficulty is deciding which experts to remove. Experts are not interchangeable, and heuristics inherited from dense pruning transfer poorly: weight magnitude ignores functional role, and usage frequency ignores the magnitude of an expert’s contribution when it is selected [18]. Recent work instead locates the signal in the router. Chowdhury et al. [3] prove that pruning the experts whose router weights change least during fine-tuning preserves test accuracy, giving the first provably effective criterion for pruning experts in fine-tuned MoE models. The guarantee is attractive because the quantity it depends on is almost free to compute: one ℓ2 _2 norm per expert, read off the gate matrix before and after adaptation. Two conditions limit how far that result reaches in practice. The theory is established for binary classification on a simplified MoE, and its empirical validation is on vision models fine-tuned for image classification, leaving open whether the criterion holds for instruction-tuned MoE language models evaluated on reasoning benchmarks. More restrictive still, the criterion is defined against full fine-tuning. Fully fine-tuning a 46.7B-parameter MoE in order to decide which experts to delete inverts the economics of the exercise: the compression step costs more than the deployment it enables. We therefore ask whether the same signal can be recovered from an adaptation cheap enough to be worth running. We fine-tune briefly with a parameter-efficient adapter, measure the ℓ2 _2 change it induces in the router, rank experts by that change, and prune the smallest in one shot. Under this view a PEFT method is not a training technique but a measuring instrument, and different methods perturb the router through different geometries: LoRA [12] adds a low-rank update in weight space, IA3 [23] rescales activations multiplicatively, and Houlsby adapters [11] inject additive residuals through a bottleneck. Which instrument is used, and where it is attached, turns out to determine how faithfully the signal is recovered. On Mixtral-8×7B-Instruct, which scores 44.83% on MMLU-Pro before pruning, LoRA confined to the router weights and training 0.002% of its parameters gives the best result we obtain at maximal compression, retaining 28.76% with half the experts removed. At matched rank, confining adaptation to the router beats spreading it across all modules (27.54% against 24.42%); at that compression both are far above the roughly 16% left by magnitude and random selection, and accuracy declines steadily as adaptation is extended to attention and expert weights. Fidelity increases monotonically with LoRA rank. Multiplicative modulation, which leaves every router weight frozen, matches direct router adaptation, while unconstrained additive adapters degrade the signal. On MMLU-Pro, router-guided pruning produces quasi-linear rather than catastrophic decay, which makes the accuracy cost of a target compression ratio predictable in advance. The resulting models use 49% less memory and have 37% lower per-token latency at four experts pruned. A parallel line of work prunes experts with no training at all, scoring them by activation norm, router gate value, or reconstruction error on calibration data [24, 15, 18]. Static router weight magnitude has itself been evaluated at this scale, and Jaiswal et al. [15] report it to be the strongest of the weight-based criteria they survey, which makes it natural to ask what the change in that quantity adds once adaptation enters the picture. Our aim is not to displace training-free criteria but to characterize a regime they do not cover. Practitioners routinely fine-tune before deployment; when they do, the pruning signal is a byproduct of work already performed, and its quality is governed by adaptation choices made for unrelated reasons. Two properties of that position should be stated plainly. First, the cost claim is relative: the adaptations we use are orders of magnitude cheaper than the full fine-tuning the criterion originally assumed, but they still perform optimization, and so cost more than criteria that need only calibration forward passes. Second, the standard we hold the signal to is competitiveness rather than dominance. A ranking recovered at 0.002% of trainable parameters does not need to win every benchmark to be useful; it needs to be competitive at matched pruning budgets while offering what calibration-based scores do not, a task-conditioned signal whose fidelity the practitioner can control through the adaptation configuration. Understanding that dependence is what this paper contributes, and Section 9 situates the signal against this literature on the benchmarks where evaluations overlap; to our knowledge, no published study reports MMLU-Pro for expert-pruned Mixtral. Our results are: • We give, to our knowledge, the first validation of the router norm-change sensitivity criterion of Chowdhury et al. [3] on MoE language models, extending it from the vision classification setting in which it was introduced to Mixtral-8×7B-Instruct and Qwen1.5-MoE evaluated on reasoning benchmarks; • We show the criterion does not require full fine-tuning: router-only LoRA, training 0.002% of parameters, recovers the signal, and enlarging the adapter’s reach beyond the router degrades it; • We characterize how adaptation configuration governs signal quality, isolating the effects of target modules, adapter rank, and perturbation geometry through controlled sweeps in which a single factor varies at a time; • We extend the criterion to adaptations that leave the router frozen, replacing the weight-space score with an ℓ2 _2 delta over router logits, which makes bottleneck-adapter and multiplicative methods comparable within one framework; • We situate the signal against recent one-shot pruning criteria on overlapping benchmarks, provide the first MMLU-Pro and BBH evaluation of expert-pruned Mixtral, report deployment measurements and reproducibility evidence across independently configured runs, and release configurations and pruned checkpoints. 2 Related Work Expert pruning in MoE models. Because MoE layers hold most of a model’s parameters, removing whole experts is the most direct route to a smaller checkpoint. Early work pruned experts progressively during task-specific fine-tuning until a single expert remained per layer [2], and Koishekenov et al. [17] showed that language-specific experts in a multilingual translation model could be removed without any further training. Subsequent criteria have grown more selective: Muzio et al. [26] rank experts by accumulated router gate values, Lu et al. [24] search for the expert subset that minimizes reconstruction loss against the original layer output, and Liu et al. [22] use a gradient-free evolutionary search over expert subsets. Lasby et al. [18] combine gate value with expert activation norm and report near-lossless compression at 50% of experts removed on models up to 1T parameters. Jaiswal et al. [15] survey sixteen criteria on Mixtral and find activation-norm scoring strongest overall. Bai et al. [1] relax the selection into a differentiable optimization that learns a nonuniform pruning rate per layer, and Yang et al. [33] cast expert retention as global path planning over activation trajectories. A related line prunes weights within experts rather than experts themselves [32]. What unites these methods is that they score experts from a frozen checkpoint using calibration data, with no adaptation step. Router-derived importance signals. Several of the above criteria read the router, but they read its current state: gate values on calibration data [26, 18], or the ℓ2 _2 norm of each expert’s row in the gate matrix, which Jaiswal et al. [15] report to be the strongest of the weight-based scores they evaluate. A different signal is available when the model is adapted. Chowdhury et al. [3] define the sensitivity of expert e as the change in its router norm over fine-tuning, Δe=∥we(T)∥−∥we(0)∥ _e= w_e^(T) - w_e^(0) , and prove that removing the experts with smallest Δe _e preserves test accuracy. The distinction matters for what the score can express: a static norm measures how much the pretrained router relies on an expert, whereas Δe _e measures how much adaptation to a task revises that reliance. Their analysis covers binary classification on a simplified MoE, and their empirical validation uses vision MoE models fine-tuned on image classification benchmarks. Adapting the router alone is itself an established technique: Hu et al. [13] attach LoRA to the gating projection of Mixtral, DeepSeek and Qwen to study expert attribution, though they do not use the resulting weight changes to prune. We take the sensitivity criterion as our starting point and ask what happens when the adaptation producing Δe _e is made parameter-efficient rather than full, and when the model is a MoE language model rather than a vision classifier. Expert merging and post-compression repair. Rather than discarding experts, merging collapses similar experts into one, clustering by router-logit similarity and averaging weights [20]. Merging has been reported to beat pruning on multiple-choice benchmarks, but Lasby et al. [18] show it introduces an irreducible error by tying gate values that the router previously controlled independently, and that the advantage reverses on generative tasks. Independently of the criterion used, some accuracy is usually recoverable afterwards: Xie et al. [32] restore 99% of baseline performance through expert-wise distillation, and Jaiswal et al. [15] find iterative dropping with interleaved retraining preferable to one-shot removal. Hyeon and Do [14] argue more strongly that compression leaves the router mismatched to the surviving experts, and recover much of the loss by distilling into the router alone. They find this recovery far larger for fine-grained architectures than for Mixtral, whose eight experts per layer admit only (82) 82 routing configurations. We evaluate in the one-shot, uncalibrated regime throughout, which isolates the quality of the selection criterion from the quality of any subsequent repair, and Section 11 returns to what calibration would add. Parameter-efficient fine-tuning. PEFT methods adapt a frozen model through a small set of additional trainable parameters. LoRA [12] adds a low-rank product to selected weight matrices, and QLoRA [8] makes this feasible on a single node by keeping the base weights in 4-bit precision. Houlsby adapters [11] insert bottleneck MLPs with a residual connection, prefix tuning [21] prepends learned vectors to each layer’s input, and IA3 [23] learns per-channel multiplicative scalings of inner activations. These differ in trainable parameter count, but for our purposes the more consequential difference is geometric: LoRA modifies weights additively in a low-rank subspace, Houlsby adapters modify activations additively through a nonlinear bottleneck, and IA3 modifies them multiplicatively without changing their direction. Section 7 finds that this geometric distinction, and not the parameter budget, is what orders the three as instruments. PEFT as an instrument rather than a training method. The PEFT literature evaluates these methods by how closely they approach full fine-tuning on task loss, and existing guidance about where to attach adapters follows from that objective. We use them differently. Here the adapted model is not the deliverable; the fine-tuning run is an instrument, and what we read off it is which experts the task recruits. The two objectives can favor different configurations, and Section 11 returns to the case where they visibly diverge. 3 Measuring Router Sensitivity 3.1 Preliminaries: MoE Routing We consider a decoder-only transformer in which the feed-forward block of each layer is replaced by an MoE layer holding E experts f1,…,fEf_1,…,f_E, each a map ℝd→ℝdR^d ^d, together with a router. The router is a single linear map with weight matrix Wg∈ℝE×dW_g ^E× d whose e-th row we∈ℝdw_e ^d is the gate vector of expert e. For a token representation x∈ℝdx ^d, the router produces logits ℓ(x)=Wgx∈ℝE,ℓe(x)=⟨we,x⟩, (x)=W_g\,x ^E, _e(x)= w_e,x , (1) and top-k routing selects the index set k(x)T_k(x) of the k largest logits. Gate values are normalized over the selected experts alone, so the layer output is y(x)=∑e∈k(x)ge(x)fe(x),ge(x)=expℓe(x)∑j∈k(x)expℓj(x).y(x)= _e _k(x)g_e(x)\,f_e(x), g_e(x)= _e(x) _j _k(x) _j(x). (2) In Mixtral-8×7B-Instruct, every one of the 32 layers carries E=8E=8 experts with k=2k=2 active per token. Because each expert is a full SwiGLU feed-forward block while wew_e is a single d-dimensional vector, the router accounts for a negligible share of parameters yet fully determines which experts contribute to any given token. 3.2 Sensitivity Scores Router norm change. The criterion we build on measures how far adaptation moves an expert’s gate vector. Writing we(0)w_e^(0) for the gate vector of expert e in the pretrained model and we(T)w_e^(T) for its value after fine-tuning, Chowdhury et al. [3] define the signed change Δe=‖we(T)‖2−‖we(0)‖2 _e= w_e^(T) _2- w_e^(0) _2 (3) and retain the experts with the largest values of Δe _e. The intuition is that fine-tuning revises the router’s reliance on experts the task actually recruits, while experts the task does not use are left near their pretrained gating. We score experts by the magnitude of this revision, se=|‖we(T)‖2−‖we(0)‖2|,s_e= \, w_e^(T) _2- w_e^(0) _2\, , (4) and prune the experts with the smallest ses_e. The two scores agree whenever gate norms only grow, and diverge when a norm contracts: such an expert has a large ses_e and is retained under Equation 4, but has a strongly negative Δe _e and would be an early pruning candidate under Equation 3. We adopt the direction-agnostic form because our question is how much adaptation perturbs the router’s treatment of an expert, not in which direction, and because a contracting gate is still evidence that the task engaged that expert. We note that the theoretical guarantee is stated for the signed quantity, and return in Section 11 to whether the sign carries usable information. Logit deltas when the router is frozen. Equations 3 and 4 require WgW_g to be trainable. Several adaptation methods leave the router weights untouched and alter only the representations reaching it, in which case se=0s_e=0 for every expert and the score is vacuous. The routing behavior nonetheless changes, because the logits in Equation 2 are a function of the incoming representation as much as of WgW_g. We therefore extend the criterion to act on logits rather than weights. Let X be a held-out batch of N token representations, and let Lbase,Ladapt∈ℝN×EL^base,L^adapt ^N× E collect the router logits produced by the pretrained and adapted models on X. With ΔL=Ladapt−Lbase L=L^adapt-L^base, we define selogit=1N‖ΔL:,e‖2,s_e^logit= 1 N L_:,e _2, (5) the root-mean-square shift in expert e’s logit across tokens. The normalization makes the score comparable across batch sizes. Equation 5 reduces to a measure of the same underlying quantity as Equation 4, the extent to which adaptation revises the router’s treatment of an expert, but reads it downstream of the gate rather than in its weights, and is therefore defined for every method we consider. 3.3 Adaptation Families and Their Perturbation Geometry Throughout, a parameter-efficient method is applied for a short fine-tuning run whose only purpose is to induce a measurable change in routing. We consider three families, which differ less in parameter count than in how they perturb the quantities entering Equation 2. LoRA [12] reparameterizes a weight matrix W as W+BAW+BA with B∈ℝdout×rB ^d_out× r, A∈ℝr×dinA ^r× d_in and r≪min(din,dout)r (d_in,d_out). Applied to WgW_g, it perturbs the gate vectors directly and additively within an r-dimensional subspace, so wew_e changes and Equation 4 applies. Applied elsewhere, it perturbs the representation x that reaches an unchanged router. The rank r bounds the expressiveness of the perturbation and is swept in Section 6. IA3 [23] learns a per-channel scaling vector α and replaces x by x⊙αx α. Router logits become ℓ(x)=Wg(x⊙α) (x)=W_g(x α), so gating is modulated multiplicatively while WgW_g is untouched. Because scaling acts coordinate-wise on the input, the induced logit shift is a reweighting of contributions already present in ⟨we,x⟩ w_e,x . Houlsby adapters [11] insert a bottleneck x↦x+Uσ(DLN(x))x x+U\,σ\! (D\,LN(x) ) with D∈ℝr×dD ^r× d, U∈ℝd×rU ^d× r and a nonlinearity σ. The perturbation is additive in representation space and passes through a nonlinearity, so it is unconstrained in direction: the residual may point anywhere in ℝdR^d, including directions with no relation to any wew_e. Only the first of these updates the router weights, so LoRA on the gate admits either score while IA3 and Houlsby require Equation 5. The three perturbation geometries, low-rank additive in weight space, multiplicative in representation space, and unconstrained additive in representation space, are compared in Section 7, where we find that the geometry rather than the parameter budget predicts how cleanly experts separate. 3.4 One-Shot Pruning Procedure Scores are computed and applied independently per layer (Figure 1). Within layer ℓ we rank its E experts by ses_e (or selogits_e^logit), remove the n experts with the smallest scores, and delete their parameters from the checkpoint. We prune the same number n from every layer, so a model with n pruned retains E−nE-n experts throughout. The corresponding rows of WgW_g are removed with them, which leaves Equation 2 well-formed on the surviving experts: top-k selection and gate normalization are simply carried out over E−nE-n logits instead of E. For Mixtral with k=2k=2, pruning up to n=4n=4 leaves at least two experts per layer and requires no change to the routing rule. Two properties of this procedure are worth stating explicitly. Pruning is applied to the original pretrained checkpoint, not to the adapted one: the adapter supplies the ranking and is then discarded, so the deployed model contains no adapter parameters. And pruning is one-shot, with no retraining, distillation, or router recalibration afterwards. This isolates the quality of the selection criterion, which is what we set out to measure, at the cost of forgoing recoverable accuracy; Section 11 discusses what a repair step would add. Figure 1: Router sensitivity across Mixtral-8×7B-Instruct: the per-expert ℓ2 _2 norm change ses_e of Equation 4, for all 32 layers and 8 experts, from an all-module LoRA adaptation at rank 16. The other adaptation runs in this study produce maps with the same scattered structure, differing mainly in overall scale. Sensitivity is not concentrated: most cells sit near their layer’s median while a few expert-layer pairs stand out by an order of magnitude (the strongest here is expert 7 of layer 8), and no single expert column or band of layers dominates. This scatter is why experts are ranked within each layer rather than against a global threshold (Section 3.4). 4 Experimental Setup Models. Our primary model is Mixtral-8×7B-Instruct-v0.1 [16], a decoder-only transformer with 32 layers, E=8E=8 experts per layer and k=2k=2 active per token, totalling 46.7B parameters of which 13B are active. It is a deliberate choice for this study: with only eight coarse experts per layer, each removal is a large intervention, so a pruning criterion has little room to be accidentally right. To test whether the criterion survives a change of model family, training regime and domain, we additionally use Qwen1.5-MoE-A2.7B-Chat [27], whose 60 experts per layer with k=4k=4 active make it the fine-grained counterpart to Mixtral. Because full fine-tuning the original 60-expert model exceeded the memory capacity of our cluster, we first reduced it to 20 experts per layer, using a router-only LoRA adaptation to select the experts to remove. We then treated this reduced checkpoint as the starting point for the full fine-tuning experiments. This hardware-constrained preprocessing means that the Qwen study evaluates the criterion on a checkpoint whose initial expert subset was itself selected using router sensitivity, rather than directly on the original 60-expert model. Data. All Mixtral adaptation runs and the Qwen router adaptation use Na0s/sft-ready-Text-Generation-Augmented-Data, an augmented supervised fine-tuning corpus of general text-generation instructions, training on the completion field at a maximum sequence length of 1024 tokens. The corpus is deliberately generic: we ask which experts a broad instruction distribution recruits, rather than tailoring the signal to any evaluation benchmark. The mathematical-reasoning runs of Section 8 instead use a mixture of MathInstruct and Belle Math, making that transfer test a change of domain as well as of model. Adaptation configurations. Except where a factor is deliberately varied, runs share one recipe: 500 optimizer steps of 8-bit AdamW at learning rate 2×10−42× 10^-4, weight decay 0.01, a cosine schedule with 5 warmup steps, bfloat16 compute, per-device batch size 1 with gradient accumulation, and LoRA dropout 0.1, from seed 3407. Accumulation is 20 for the router-only reference and is held fixed within each sweep; per-run values are given in Appendix A. Mixtral adaptation runs hold the base weights in 4-bit precision following QLoRA [8]; the Qwen router adaptation runs unquantized. Our reference configuration applies LoRA of rank r=8r=8 with α=8α=8 to the router weights alone. From that reference we vary one factor at a time: rank over 2,4,8,16\2,4,8,16\ with the target set fixed (Section 6), and the target module set over router-only, router with query and output projections, router with full attention, router with expert feed-forward weights, and all linear modules, with rank fixed at 8 (Section 5). Every configuration in this sweep includes the router, since excluding it leaves WgW_g untouched and Equation 4 identically zero. Such configurations could in principle be scored with Equation 5 instead, and we return to this gap in Section 11. The multiplicative and additive-adapter runs use a fixed budget of 2000 training examples for one epoch at learning rate 10−410^-4, with context length 2048, micro-batch size 1 and gradient accumulation 16. IA3 [23] attaches scaling vectors to the router, the attention projections and the expert feed-forward weights; Houlsby adapters [11] use a reduction factor of 16 with a GELU nonlinearity. Because this budget differs from that of the LoRA sweeps, Section 7 compares adaptation geometries under their own settings rather than under a matched budget, and we read those results accordingly. Full fine-tuning reference. One setting updates every parameter rather than an adapter. On Qwen we fully fine-tune the 20-expert model on mathematical reasoning, which is feasible only with aggressive offloading: DeepSpeed ZeRO stage 3 with both optimizer states and parameters offloaded to CPU, gradient checkpointing, and an effective batch size of 512 formed from micro-batches of 1 with 256 accumulation steps, using AdamW at learning rate 2×10−42× 10^-4 with weight decay 0.01. Sensitivity is then read from Equation 4 directly, since WgW_g is trainable. Pruning implementation. Pruning is structural rather than masked. For each layer we delete the selected experts from the module list and remove the corresponding rows of WgW_g, so the router’s output dimension shrinks from E to E−nE-n and top-k selection with softmax renormalization proceeds over the surviving experts exactly as in Equation 2. We remove the same number n∈1,2,3,4n∈\1,2,3,4\ from all 32 layers, corresponding to 12.5% through 50% of experts. Pruning is applied to the original pretrained checkpoint rather than to the adapted one, so no adapter parameters survive into any evaluated model. Pruned models are saved with the foundation model’s tokenizer and a corrected expert count in the configuration, then evaluated as ordinary checkpoints. Evaluation. We evaluate with LightEval [10]. MMLU-Pro [31] is the primary benchmark throughout, chosen because its ten-option format and reasoning-heavy questions leave less headroom for a degraded model to guess well than four-option multiple choice does; the unpruned Mixtral baseline is 44.83%. Section 8 widens the evaluation to GSM8K [6], ARC-Challenge [5], BIG-Bench Hard [29] and BoolQ [4], which span chain-of-thought arithmetic, science question answering, multi-step reasoning and binary reading comprehension, and which we use to test whether the shape of the accuracy curve is a property of the criterion or of MMLU-Pro. The Qwen study is evaluated on eleven mathematical-reasoning benchmarks covering college, high school and elementary mathematics, algebra, precalculus, probability, induction and MathQA. Implementation and hardware. Adaptation uses PEFT and TRL’s SFTTrainer over HuggingFace Transformers, with bitsandbytes supplying 4-bit base weights and the 8-bit optimizer. Experiments ran on NVIDIA A100-SXM4 GPUs with 80 GB HBM2 under CUDA 12.8 and driver 550.54.15, using a single GPU for adaptation and up to three for pruning and serialization of the full checkpoint. The memory and latency measurements of Section 10 are taken with Optimum-Benchmark [25] under 4-bit weights on a single A100, which is why the unpruned model is reported at 24.2 GB rather than at its half-precision footprint. Pruned checkpoints, run configurations, and the scoring and pruning code are released. Measurement resolution. Three configurations were run twice over the course of the project under nominally identical settings. Across the twelve resulting pairs of accuracies, repeated runs differ by 0.76 points on average and by at most 2.24. We take 2.2 points as the resolution of a single comparison in what follows, and distinguish throughout between differences that exceed it and orderings that are supported instead by their consistency across many comparisons. Section 8.1 reports these replications in full. Two measurement pipelines. Results were produced in two phases whose evaluation settings differ. The first, from which every result in Sections 5 through 7 is drawn, uses the recipe above and gives an unpruned MMLU-Pro baseline of 44.83%. The second, reported in Section 8.1 and in the benchmark study of Section 8, comprises four independently configured runs that differ in rank, target modules, step budget, batch shape, weight decay and schedule, and that additionally use gradient clipping at maximum norm 0.3, sequence packing and length-grouped batching; their unpruned baselines are 45.35% and 45.57%. The Qwen study is a third setting with its own model, training regime and benchmarks, and is reported against its own unpruned scores throughout. Every comparison is computed within a setting against that setting’s own baseline, and we do not mix settings in any single table or figure. Full configurations appear in Appendix A. 5 Where Does the Pruning Signal Live? The criterion of Section 3.2 says to read expert importance off the router, but it does not say how the router should be made to move. This section fixes the adapter rank at 8 and varies only which modules it is attached to, so the adapted rows of Table 1 differ in nothing except the location of the adaptation. Two selection rules that require no adaptation at all are included as a floor. Figure 2 plots the resulting accuracy against pruning depth. Table 1: MMLU-Pro accuracy (%) after removing n experts from every layer of Mixtral-8×7B-Instruct. The unpruned model scores 44.83% at 46.7B parameters. All adapted rows use LoRA of rank 8 and include the router; rows below the first add further target modules. Remaining parameter counts are identical across selection criteria at a given n. Bold marks the best adapted configuration in each column. Experts pruned per layer Selection criterion 1 2 3 4 (41.1B) (35.4B) (29.8B) (24.2B) Without adaptation Magnitude 43.94 27.45 22.27 15.88 Random 41.94 34.74 29.26 16.31 Router sensitivity, LoRA r=8r=8 Router only 40.18 35.78 31.37 27.54 + q, o 40.06 35.32 30.81 26.74 + q, k, v, o 39.94 34.98 30.40 25.93 + expert MLP 39.82 34.85 29.97 24.97 + all linear modules 39.87 34.86 30.04 24.42 Figure 2: MMLU-Pro accuracy as experts are removed from every layer of Mixtral-8×7B-Instruct, by selection criterion. Router sensitivity is the router-only LoRA reference at rank 8 (Table 1); all curves share the 44.83% unpruned baseline at zero experts pruned. Router-guided selection decays quasi-linearly, whereas magnitude and random selection collapse once more than one expert per layer is removed. Confining adaptation to the router improves the signal. At matched rank, data, and training schedule, attaching LoRA to every linear module yields 24.42% MMLU-Pro accuracy with four experts pruned per layer, compared with 27.54% when LoRA is confined to the router. The gap is 3.1 points at equal rank, equal data, and an identical pruned model size; the two runs differ only in where the adapter was attached. Adapting the whole network is the stronger configuration for producing a capable fine-tuned model, and is what the parameter-efficient fine-tuning literature recommends when task loss is the objective. It is the weaker configuration for deciding what to delete. The penalty grows with the modules adapted and with pruning depth. At four experts pruned the five adapted rows form a ladder: 27.54% for the router alone, then 26.74%, 25.93% and 24.97% as query and output projections, full attention, and the expert feed-forward weights are added, and 24.42% with every linear module adapted. At shallower pruning the ordering is not strict. The five configurations span 0.36 points at one expert pruned, 0.93 at two and 1.40 at three, and within those bands the two largest-footprint configurations exchange places, and no single step of the ladder exceeds the measurement resolution. What the data support is the span rather than the individual rungs: the 3.1 points between confining the adapter to the router and spreading it across the whole network is the one comparison here larger than repeated-run variation. Target module choice is close to irrelevant when one expert is removed and worth three points when four are, because a criterion only needs to be right about the marginal expert until it has to be right about half of them. Reference points without adaptation. Neither adaptation-free rule survives past the first removal. Magnitude selection is the best entry in the n=1n=1 column at 43.94%, since every layer holds one expert the pretrained model already weights lightly and any criterion correlated with that weighting will find it, but it falls 16.5 points at n=2n=2 against 4.4 for router sensitivity and ends at 15.88%. Random selection ends at 16.31% and, in the middle of the range, is the better of the two, which indicates that small weight magnitude is not merely a weak indicator of redundancy but an actively misleading one. We do not treat these rules as competitive baselines; they establish that the comparisons in the rest of the paper are made above a floor rather than against one. Interpretation. The router is the point at which revised expert recruitment becomes observable, whether the change is induced by modifying the router weights directly or by changing the representations reaching them. When the adapter also reaches attention and expert weights, the same reduction in loss becomes available by changing what the experts compute rather than which are consulted. The router then moves less, and less distinctively, and the measurement is diluted by the alternatives available to the optimizer. This reading predicts the two results that follow: capacity spent on the router should sharpen the signal (Section 6), and adaptations that reach the router only indirectly should be judged by how faithfully they preserve its structure rather than by how many parameters they train (Section 7). 6 How Much Adaptation Capacity Is Needed? Section 5 fixed the adapter rank and moved its location. This section does the reverse: the adapter stays on the router and we sweep its rank over 2,4,8,16\2,4,8,16\, scaling α with r and holding data, schedule and step budget constant, in order to isolate the effect of adapter capacity (Figure 3). Rank bounds the dimension of the subspace in which WgW_g may move, so it controls how much of the task-induced revision the adapter is able to express at all. Table 2: MMLU-Pro accuracy (%) after pruning, for router-only LoRA at four ranks. Trainable parameters are given as a percentage of the 46.7B-parameter model. The unpruned model scores 44.83%. Experts pruned per layer Rank Trainable 1 2 3 4 r=2r=2 0.0006% 38.91 33.17 27.65 23.42 r=4r=4 0.0011% 39.58 34.52 30.11 25.87 r=8r=8 0.0022% 40.18 35.78 31.37 27.54 r=16r=16 0.0045% 40.64 36.34 32.21 28.76 Figure 3: MMLU-Pro accuracy after pruning for router-only LoRA at ranks r∈2,4,8,16r∈\2,4,8,16\ (Table 2). Each curve lies above the next-lower rank at all four pruning depths, and the spread widens with pruning depth. The adapter is discarded after ranking, so the curves describe models of identical size that differ only in which experts were kept. Accuracy after pruning increases monotonically with rank. Every column of Table 2 is ordered, without exception: rank 4 beats rank 2, rank 8 beats rank 4 and rank 16 beats rank 8 at all four pruning depths. No single doubling clears the measurement resolution on its own; what supports the trend is that all twelve adjacent comparisons point the same way, and that the endpoints are separated by 5.34 points at four experts pruned. This is worth stating carefully, because rank here does not act on the pruned model. The adapter is discarded once experts have been ranked, and the four rows of Table 2 describe models with identical architecture and identical parameter counts, differing only in which experts were removed. Rank changes the selection, not the artifact. A monotone trend therefore says that larger adapters identify better pruning candidates, not that they leave behind a better model. Returns diminish with each doubling. Averaged across pruning depths, moving from rank 2 to rank 4 is worth 1.73 points, rank 4 to rank 8 a further 1.20, and rank 8 to rank 16 a further 0.77. At the most aggressive setting the same pattern holds with larger absolute values: +2.45+2.45, +1.67+1.67 and +1.22+1.22 points as rank doubles. The curve is concave in logr r, and the practical reading is that rank 8 captures most of what is available, with rank 16 buying a final point at twice the adapter size. Since even rank 16 trains 0.0045% of the model, the argument for stopping at 8 is not cost but the shape of the curve. Capacity matters more the more is pruned. The spread between the smallest and largest rank widens with pruning depth: 1.73 points at one expert pruned, 3.17 at two, 4.56 at three and 5.34 at four. This mirrors the widening observed across target modules in Section 5, and for the same reason. Ranking the single least-used expert is a coarse decision that a low-capacity adapter can get right; ordering half the experts in a layer requires resolving distinctions that a rank-2 update cannot represent. Rank-2 selection is in fact worse than random removal at one, two and three experts pruned, and only overtakes it at four, where random selection collapses. An adapter too small to express the revision the task induces is not a weak instrument but a misleading one. Interpretation. Rank controls the dimension of the subspace within which ΔWg W_g may lie. When that subspace is smaller than the revision the task actually requires, the adapter projects the true update onto whatever directions it can represent, and the norm changes that result reflect the projection as much as the task. Increasing rank relaxes this constraint, and the monotone trend indicates that no rank we tested is large enough to overfit the selection: within this range, more expressive adaptation yields a more faithful ranking. Where that trend would turn is not determined by these experiments; the unconstrained limit of the sequence, an unrestricted update to WgW_g, is the natural point of comparison and we do not reach it here. Section 7 turns instead to adaptations that cannot write to WgW_g at all. 7 How Should Adaptation Perturb the Model? The two preceding sections varied where an adapter is attached and how much capacity it has, holding its form fixed. This section varies the form (Figure 4). LoRA writes a low-rank additive update into WgW_g itself. IA3 leaves WgW_g untouched and rescales the representation reaching it, coordinate by coordinate. Houlsby adapters leave WgW_g untouched and add a nonlinear residual to that representation. Only the first admits the weight score of Equation 4; the other two are read through the logit score of Equation 5. These runs use the budget given in Section 4, which differs from that of the LoRA sweeps, so Table 3 compares three geometries each under its own settings rather than under a matched budget. Table 3: MMLU-Pro accuracy (%) after pruning, by adaptation geometry. The unpruned model scores 44.83%. LoRA is the router-only reference at rank 8 and is scored on router weights; IA3 and Houlsby leave WgW_g frozen and are scored on router logits. IA3 and LoRA differ by less than the measurement resolution at every depth; Houlsby trails both. Experts pruned per layer Adaptation Perturbation 1 2 3 4 IA3 multiplicative, diagonal 40.27 36.11 32.48 28.04 LoRA (router) additive, low-rank 40.18 35.78 31.37 27.54 Houlsby additive, unconstrained 40.08 35.52 30.71 25.39 Figure 4: MMLU-Pro accuracy after pruning by adaptation geometry (Table 3). IA3, which never writes to the router, tracks router-only LoRA within the measurement resolution at every depth, while the unconstrained additive Houlsby adapter separates downward as pruning deepens. An adaptation that never writes to the router matches one that does. IA3 leads at every pruning depth, by 0.09, 0.33, 1.11 and 0.50 points over router-only LoRA. None of these margins clears the measurement resolution of Section 4, and the budgets are unmatched, so we read the table as a tie rather than a win. The tie is itself the finding: a method that modifies no router parameter identifies prunable experts as well as one that modifies them directly. The criterion does not require writing to the object being measured. It requires only that the object be observed while the model is under task pressure. An unconstrained additive perturbation degrades the signal. Houlsby adapters trail at every depth and the gap widens with pruning, from 0.10 points at one expert removed to 2.15 at four. The endpoint gap to IA3, 2.65 points, is the one comparison in this section that exceeds the measurement resolution, and the widening is consistent across all four depths. Measured end to end, accuracy falls 14.69 points between one and four experts pruned under Houlsby, against 12.64 for LoRA and 12.23 for IA3. Houlsby is nonetheless far above the adaptation-free floor of Section 5, so its selections carry real information; they are simply noisier. The scoring rule alone does not explain the ordering. IA3 and Houlsby are both scored using logit deltas on held-out data, whereas LoRA is scored using router-weight changes. If the logit score alone were responsible for IA3’s performance, Houlsby would be expected to show a similar advantage over LoRA. Instead, Houlsby performs worst at every pruning depth despite using the same scoring rule as IA3. This suggests that differences in how the adaptation methods perturb the model contribute to the observed ordering. Because the methods are trained under unmatched budgets and configurations, however, the experiment does not isolate perturbation geometry as the sole causal factor. Reconciling this with Section 5. IA3 attaches to the router, the attention projections and the expert feed-forward weights, which is the spread configuration that Section 5 found harmful for LoRA. The two results are consistent once dilution is understood as a function of capacity available away from the router rather than of mere presence there. LoRA at rank 8 on all modules gives attention and experts substantial capacity, and the optimizer can lower loss by changing what experts compute instead of which are consulted, leaving the router less distinctly moved. IA3 assigns each module a single scaling vector, on the order of 0.01% of parameters in total, which is too little capacity to absorb the task anywhere. Houlsby sits between the two in budget yet performs worst, which shows that capacity alone does not order these methods either. Interpretation: constraint as an instrument property. The three geometries differ in how freely they can reorganize routing for reasons unrelated to expert utility. IA3 multiplies the representation by a vector shared across every expert in a layer, so it cannot introduce a distinction between two experts that their existing weight vectors do not already express; it can only amplify or attenuate distinctions that are there. LoRA writes directly into WgW_g within an r-dimensional subspace, which is more free but still bounded. A Houlsby residual passes through a nonlinearity and may point anywhere in representation space, including directions that happen to align with one expert’s gate vector for reasons having nothing to do with the task. Read alongside Section 6, this suggests that capacity and faithfulness are separate axes. Within a fixed geometry, more capacity gives a better ranking, since the adapter can express more of the revision the task actually induces. Across geometries, less freedom gives a better ranking, since the perturbation is more tightly bound to structure the model already had. An instrument should be expressive enough to register the effect and constrained enough not to create one. 8 Does the Signal Generalize? The controlled sweeps of Sections 5 through 7 vary one factor at a time inside a single pipeline. This section asks what survives outside it: across independently chosen configurations and repeated runs, across benchmarks other than the one used throughout, and across a different model family, adaptation regime and domain. 8.1 Robustness Across Runs and Configurations Four adaptation runs were configured independently, before the controlled sweeps existed, by different members of the project. They differ in rank, target modules, step budget, learning rate, batch shape, weight decay and schedule (Table 4; full settings in Appendix A), and each produced its own expert selection, pruned models and evaluation under the second pipeline of Section 4. Table 4: Four independently configured adaptation runs, second pipeline. MMLU-Pro accuracy (%) by experts pruned per layer, each against its own unpruned baseline. All selections use the weight score of Equation 4. Experts pruned Run Rank Target modules Steps 1 2 3 4 A 16 all linear 1000 41.81 38.40 32.73 26.62 B 4 router + experts 500 41.03 36.46 32.35 26.42 C 4 all linear 200 40.53 36.46 29.87 24.60 D 4 router + attn + gate_projgate\_proj 100 39.91 35.68 29.75 24.39 Configuration moves the result by about as much as rerunning does. The four runs span 1.90 points at one expert pruned, 2.72 at two, 2.98 at three and 2.23 at four. The largest difference we observe between repeated runs of a single configuration is 2.24 points (below). Choosing a different rank, module set, budget and schedule therefore displaces the outcome by roughly the same amount as holding everything fixed and rerunning, and every one of the four curves decays gradually, with a maximum deviation from a straight line between baseline and endpoint of 0.81 to 2.41 points and no collapse anywhere. The shape of the result is a property of the criterion, not of a configuration. The controlled trends reappear as correlations. Ranked by final accuracy, the four runs order exactly by adaptation steps: 1000, 500, 200 and 100 steps give 26.62%, 26.42%, 24.60% and 24.39%. The two longer runs also lose less from their own baselines (18.7 and 18.9 points at four pruned) than the two shorter ones (21.0 and 21.0). The best run combines the highest rank with the longest budget, consistent with the monotone rank effect of Section 6, though since these runs vary several factors at once we read the ordering as corroboration rather than as an independent measurement. Table 5: Replication of the rank sweep. Three configurations were run twice during the project under nominally identical settings; rank 8 was run once. Cells give both runs as first / second. MMLU-Pro (%). Experts pruned per layer Rank 1 2 3 4 2 39.13 / 38.91 33.09 / 33.17 25.97 / 27.65 23.46 / 23.42 4 40.11 / 39.58 33.27 / 34.52 32.35 / 30.11 27.21 / 25.87 16 41.12 / 40.64 36.96 / 36.34 32.66 / 32.21 28.94 / 28.76 What replication preserves and what it does not. Across the twelve paired accuracies of Table 5, repeated runs differ by 0.76 points on average, 0.53 at the median, and 2.24 at most; these are the figures behind the resolution adopted in Section 4. Within that noise, the claims of Section 6 are exactly the ones that survive. Rank 16 is best and rank 2 worst in both runs at every pruning level. What does not survive is finer: in the first run, rank 4 exceeds rank 8 at three experts pruned by 0.98 points, the one inversion of adjacent ranks anywhere in either run, and its size is characteristic of the noise. Replication thus draws the same line the resolution analysis drew: endpoint orderings are stable, adjacent orderings are not guaranteed, and single comparisons below two points should not be read. 8.2 Across Benchmarks MMLU-Pro has carried every result so far. Table 6 evaluates pruned models from the selections of Table 4 on four further benchmarks; each suite is evaluated on one selection, and BIG-Bench Hard on two. Figure 5 shows the retention curves. Table 6: Accuracy (%) by experts pruned, on benchmarks beyond MMLU-Pro. Each row uses pruned models from one selection of Table 4 (BBH from two). Reported standard errors are 1.4 points for GSM8K and ARC. Experts pruned per layer Benchmark Selection 0 1 2 3 4 GSM8K D 51.25 52.08 40.03 27.29 12.05 ARC-Challenge D 54.10 54.52 52.39 51.11 46.59 BIG-Bench Hard A 43.70 41.50 40.75 39.82 36.04 BIG-Bench Hard B 43.70 43.09 40.86 37.92 33.25 BoolQ B 83.70 77.30 81.80 81.40 62.10 Figure 5: Accuracy retained relative to each benchmark’s own unpruned score, for pruned models from the selections of Table 4 (Table 6). Plotting retention rather than raw accuracy makes the task-dependent shape of the decay comparable across benchmarks with different baselines: ARC-Challenge and BIG-Bench Hard fall gently, GSM8K is flat then collapses, and BoolQ is non-monotonic. The dotted line marks the unpruned level. The shape of the curve is task-dependent. Quasi-linear decay, the signature result on MMLU-Pro, is not universal. ARC-Challenge and BIG-Bench Hard decline gently, losing 7.5 and 7.7 to 10.5 points across the full range. GSM8K is flat to one expert pruned and then collapses, from 52.08% to 12.05%, the steepest fall we observe anywhere on Mixtral. BoolQ is non-monotonic, dipping 6.4 points at one expert pruned, returning to within 2.3 points of baseline at two and three, then dropping 19 points at four. For the same selection D, MMLU-Pro loses 21.0 points end to end, placing the primary benchmark mid-pack rather than at either extreme. The selections preserve broad capability at substantial compression while multi-step generative arithmetic degrades first and fastest, which is consistent with such computation depending on a small set of experts that mid-ranked removals eventually reach. The signal is conditioned on the adaptation task. These selections were produced by adapting on a generic text-generation corpus that is disjoint from every benchmark evaluated here, so nothing in the procedure was tuned to the tests. Read in that light, the table shows both faces of task conditioning. A generic corpus recruits the experts that generic instruction-following uses, and pruning by that signal preserves broad capabilities in proportion to how broadly they are exercised; a capability the corpus exercises lightly, such as chained arithmetic, receives no protection, and its experts sit in the middle of the ranking where deeper pruning reaches them. The practical lever this implies is that the adaptation corpus is a control surface: a practitioner who needs arithmetic preserved should adapt on arithmetic. The lever is not specific to our criterion; Bai et al. [1] report that switching a pruning method’s calibration data from generic text to mathematics lifts GSM8K accuracy at half the experts by nine points or more, for their method and for an enumeration baseline alike. Section 8.3 evaluates exactly that regime for router sensitivity, with the adaptation domain matched to the evaluation domain. Flat is not improved. Two curves tick upward at one expert pruned, GSM8K by 0.83 points and ARC by 0.42. Both movements are inside the reported standard errors of 1.4 points, so we read them as flat: removing the lowest-sensitivity expert from every layer leaves these capabilities measurably unchanged, which is itself the behavior the criterion promises at its first increment. 8.3 Across Models and Domains The remaining question is whether any of this is specific to Mixtral, to parameter-efficient adaptation, or to general instruction data. The setting of Section 4 changes all three at once: Qwen1.5-MoE with 20 experts per layer, full fine-tuning of every parameter, and a domain-matched corpus in place of a generic one, with adaptation on mathematical reasoning and evaluation on eleven mathematics benchmarks, experts removed five and ten at a time, half the experts at the deepest cut (Figure 6). Table 7: Qwen1.5-MoE fully fine-tuned on mathematical reasoning: mean accuracy (%) over eleven mathematics benchmarks, with one representative benchmark shown. Experts pruned (of 20) Selection criterion 0 5 10 Mean over eleven benchmarks Router sensitivity 61.5 58.1 49.7 Magnitude 61.5 55.2 39.1 Random 61.5 37.6 4.4 MMLU College Mathematics Router sensitivity 62.4 59.0 50.8 Magnitude 62.4 56.2 39.8 Random 62.4 38.2 4.5 Figure 6: Qwen1.5-MoE fully fine-tuned on mathematical reasoning: mean accuracy over eleven mathematics benchmarks as experts are removed (Table 7). Router sensitivity degrades gradually while random selection collapses to single digits with half the experts removed; magnitude selection lies between them. The criterion transfers, and the alternatives do not. With half the experts removed, router sensitivity retains 49.7% on average against 39.1% for magnitude selection, while random selection falls to 4.4%, single-digit accuracy on generative mathematics, a model that has effectively stopped producing correct answers. The collapse is not an artifact of our setting: Bai et al. [1] report random expert removal at the same ratio scoring 0.68% on GSM8K. The gap to magnitude at ten experts pruned, more than ten points on the mean and present in every one of the eleven benchmarks, is far beyond any noise scale in this paper. The signal read from a fully fine-tuned router on domain data behaves as it did when read from a rank-8 adapter on general instructions. The location result recurs. The 20-expert base model itself was chosen using preliminary Qwen adaptations in which router-only LoRA reached 29.31% MMLU-Pro against 28.13% for LoRA on all modules. The gap of 1.18 points sits within the measurement resolution; its direction repeats, on a second model family, the central finding of Section 5. 9 Positioning against One-Shot Pruning Methods This work evaluates what the literature does not. To our knowledge, no published study reports MMLU-Pro for expert-pruned Mixtral. REAP evaluates its Mixtral compressions on ARC, BoolQ, HellaSwag, MMLU, OpenBookQA, RTE and WinoGrande [18]; DiEP on MMLU, OpenBookQA, BoolQ and RTE [1]; trajectory-based selection on MMLU, HellaSwag, WinoGrande, ARC, GSM8K and MedQA [33]; the task-agnostic line on MMLU, BoolQ, OpenBookQA and RTE [15]. Standard MMLU is the common denominator of that literature, and its four-option format leaves a degraded model substantial guessing headroom. None of the four suites includes BIG-Bench Hard either. Every MMLU-Pro and BBH number in Sections 5 through 8 is therefore the first characterization of expert-pruned Mixtral on these benchmarks, and the quasi-linear decay established on MMLU-Pro across five compression points is a property no previous study, reporting two points on four-option MMLU, was positioned to observe. On the benchmarks the literature does report, the signal is competitive. Table 8 compares retention, each pruned score normalized by its own study’s unpruned baseline, on the three benchmarks where our evaluations and published ones overlap. At 25% compression the signal sits inside the published band everywhere: BoolQ retention of 97.7% against 98.5% for REAP and 99.1% for DiEP, ARC retention of 96.8% above both reported values, GSM8K within six points of a method calibrated on task data. These are methods built on full activation statistics, differentiable search over expert subsets, and global trajectory planning; the row they bracket comes from rank-4 LoRA selections of Table 4, and in its leanest router-only form the signal trains 0.002% of parameters. Table 8: Retention (%) of each study’s own unpruned baseline after expert pruning on Mixtral-8×7B-Instruct, at 25% and 50% expert sparsity, on the benchmarks where evaluations overlap. Published figures are computed within their own harness and settings; REAP’s ARC uses normalized accuracy, ours ordinary accuracy; DiEP and trajectory selection allocate experts nonuniformly across layers, whereas we remove uniformly; the trajectory GSM8K result is calibrated on task data, whereas our selections are conditioned on a generic corpus. Dashes mark benchmarks a study does not report. BoolQ ARC-Challenge GSM8K Selection criterion 25% 50% 25% 50% 25% 50% Router sensitivity (this work) 97.7 74.2 96.8 86.1 78.1 23.5 REAP [18] 98.5 94.4 94.0 83.7 – – DiEP [1] 99.1 96.7 – – – – MoE Pathfinder [33] – – 94.4 85.0 83.9 46.3 At 50% the ordering is task-dependent and tracks conditioning. ARC retention of 86.1% exceeds REAP’s 83.7% and matches trajectory selection’s 85.0%. BoolQ and GSM8K fall below the published values. The GSM8K column is the conditioning effect of Section 8.2 measured across papers: the published figure is task-calibrated, ours is generically conditioned, and DiEP’s own ablation prices that difference at nine points or more of GSM8K accuracy at this compression [1]. Competitive at moderate compression, and the economics run in one direction. The value of the criterion is a combination the table cannot show in one column. On overlapping benchmarks at moderate compression it is competitive with methods that consume full activation statistics or learned allocations, and the economics run in one direction twice over. Against the full fine-tuning the criterion originally required, the router-only signal trains 0.002% of the parameters. Against calibration statistics, which are always a dedicated expense, the common deployment path already includes parameter-efficient adaptation, so the ranking arrives as a byproduct of work performed anyway, at zero marginal cost. It uniquely covers the benchmark class the literature avoids, where it establishes predictable quasi-linear decay. And it is the one criterion whose knobs this paper has mapped: Sections 5 through 7 identify which adaptation choices sharpen the ranking, and Section 8.2 shows the adaptation corpus steers which capabilities pruning protects, a steering that external ablations confirm for calibration data as well [1] but that no other criterion arrives with characterized. 10 Deployment Efficiency Accuracy retention is only half of the pruning case; the other half is what removal buys at inference time. We measure GPU memory and per-token generation latency with Optimum-Benchmark [25] on a single A100, with base weights quantized to 4 bits as in adaptation, for the unpruned model and the four router-sensitivity pruned variants of Section 5. Table 9: Inference cost by experts retained, measured with Optimum-Benchmark on one A100 under 4-bit weights. Memory is in GB and is a 4-bit measurement, so the unpruned figure of 24.2 GB is unrelated to the 24.2B parameter count of the four-experts-pruned model elsewhere in the paper. Relative throughput is the reciprocal of per-token latency and therefore describes single-stream decoding. Model Experts Memory (GB) Latency/token (ms) Rel. throughput Unpruned 8 24.2 40.3 100% 1 pruned 7 21.3 35.8 112.6% 2 pruned 6 18.1 32.4 124.4% 3 pruned 5 15.4 28.7 140.4% 4 pruned 4 12.3 25.2 159.9% Costs fall near-linearly in experts removed. Memory decreases by 2.7 to 3.2 GB per expert removed, 2.97 on average, against the 2.8 GB that one expert’s 5.6B parameters occupy at 4 bits; the small surplus is the removed rows’ share of activation and dequantization overhead. Latency decreases by 3.4 to 4.5 ms per expert. Neither series deviates from a straight line between its endpoints by more than 0.15 GB or 0.73 ms. At the deepest cut the model uses 49% less memory and generates each token 37% faster, and single-stream throughput rises 60%. The latency gain is a memory-traffic effect, not a FLOP effect. Top-2 routing is unchanged by pruning: every token activates two experts before and after, so the arithmetic per token is constant across the rows of Table 9. What shrinks is the set of distinct expert weights touched per forward pass. Across the tokens of a sequence the router reaches most resident experts in every layer, so the weights streamed from memory scale with the number of experts retained rather than with the number activated per token, and in the memory-bandwidth-bound regime of single-stream decoding that traffic is what latency measures. The near-linear latency curve is this mechanism made visible. Both sides of the trade are predictable. The cost curves here are near-linear in experts removed, and Section 5 showed the MMLU-Pro accuracy curve under router-sensitivity selection to be quasi-linear over the same range. A practitioner can therefore price a compression target in advance from the endpoints alone: on this hardware, each expert removed buys roughly 3 GB of memory and 3.5 to 4.5 ms per token, at a quasi-linear accuracy cost whose slope Sections 5 through 7 showed how to minimize. Neither the naive selection rules, which collapse abruptly past two experts, nor benchmarks with cliff behavior such as GSM8K offer that predictability, which is why we regard the shape of the decay, as much as its level, as the practically valuable property of the criterion. 11 Discussion Why the router concentrates the signal. The results of Sections 5 through 7 admit a single mechanism. The gate matrix is the one place in the network where the decision of which experts serve a token is expressed as parameters, so it is the one place where a task’s revision of expert recruitment must appear if it appears at all. Whether it appears cleanly depends on the alternatives the optimizer has. Give the adapter access to attention and expert weights and loss can fall by changing what experts compute rather than which are consulted, and the gate moves less distinctively (27.54% against 24.42% at four experts pruned). Give the router adapter too little rank and the true revision is projected onto directions it can represent, degrading the ranking monotonically as capacity shrinks. Give the adaptation freedom to inject arbitrary directions into the representation and the gate moves for reasons unrelated to expert utility, which is where Houlsby adapters lose to multiplicative scaling. The signal is faithful when the perturbation reaching the router is both forced, because the task cannot be absorbed away from routing, and disciplined, because the perturbation cannot reorganize routing for reasons unrelated to expert utility. Every degradation we measure violates one of the two, and Section 7 shows the perturbation need not write to WgW_g at all to satisfy both. An instrument objective inverts training guidance. Practical guidance for LoRA as a training method recommends attaching it broadly, with the feed-forward and expert layers mattering most, because that configuration best matches full fine-tuning on task loss [8, 30]. Our measurements agree with that guidance on its own terms and invert it on ours: the all-modules configuration is the better way to produce a capable model and the worse way to decide what to delete. The divergence is principled rather than paradoxical. Training optimizes the model’s outputs; an instrument optimizes the legibility of a specific internal change. Sections 5 through 7 can be read as the beginning of a configuration guide for the second objective, for which the PEFT literature’s first-objective guidance does not transfer. What the empirics say back to the theory. The guarantee of Chowdhury et al. [3] is stated for full fine-tuning and for the signed norm change. Our results extend its empirical reach and, in three places, ask questions the theory does not yet answer. First, the monotone rank effect: constraining the router update to a rank-r subspace degrades the ranking gracefully and monotonically rather than abruptly, which invites an analysis of the criterion under constrained adaptation, with the unconstrained update as the limit our sweep approaches from below. Second, perturbation geometry: a multiplicative rescaling that never writes to WgW_g ranks experts as well as a direct low-rank update while an unconstrained additive perturbation ranks them worse, a distinction absent from the current analysis. Third, the sign: our score is the magnitude of the norm change, the guarantee concerns its signed value, and our sensitivity maps show gate norms that contract as well as grow under adaptation. Whether contraction marks an expert as recruited or as released is a well-posed question settled by re-ranking gate norms from the adapted checkpoints at no training cost, and the answer would sharpen both the theory and the practice. The module ladder has two unscored rungs. Attention-only and expert-only adaptation were excluded from the module sweep because they leave WgW_g untouched and Equation 4 identically zero. Equation 5 scores them anyway, by reading the logits the frozen router produces over shifted representations. Running the module ladder under the logit score would separate two claims this paper currently cannot distinguish: that the router must be adapted for the signal to exist, and that it must merely be observed while the model is under task pressure. Section 7 already shows the second suffices for IA3; completing the ladder is the cleanest single experiment this work leaves open. Calibration, and where the one-shot numbers sit. Every result in this paper is one-shot and uncalibrated by design, isolating the selection criterion from any repair applied after it. Hyeon and Do [14] show that distilling into the router after compression recovers substantial accuracy on fine-grained architectures while yielding only marginal recovery on Mixtral, whose eight experts per layer admit few routing configurations. Read together, the two results place our Mixtral numbers near the ceiling of what router-side repair would add, and mark the fine-grained regime, including the Qwen setting of Section 8.3, as where a calibration stage is the natural extension; recovery through retraining the surviving experts is a separate and larger budget that our one-shot scope deliberately excludes. Quasi-linear uncalibrated decay is the floor that stage would build on. Scope. The controlled sweeps cover one model family, and the head-to-head table specified in Section 9 is the pending experiment; the cross-paper retentions stand in for it at the stated confounds. Sweep results are single runs read against a 2.2-point replication resolution, with the twice-run configurations reported as ranges in Section 8.1 and Appendix B. The IA3 and Houlsby comparisons use their own training budget rather than one matched to the LoRA sweeps. Configuration selection throughout was keyed to MMLU-Pro, and Section 8.2 quantifies how far that choice generalizes. The signed-score question above is open, and costs only a re-ranking from the adapted checkpoints to close. 12 Conclusion and Future Work A provably motivated criterion for expert pruning, previously defined against full fine-tuning, is practical at the scale of current MoE language models: a brief parameter-efficient adaptation moves the router enough to rank experts, the magnitude of each gate vector’s norm change (or, when the router is frozen, the shift in its logits) supplies the ranking, and one-shot removal of the lowest-ranked experts yields models whose MMLU-Pro accuracy decays quasi-linearly with compression while memory and latency fall near-linearly. The signal concentrates where the decision it measures is expressed. Confining adaptation to the router sharpens it, capacity spent there improves it monotonically, and constrained perturbation geometry preserves it where unconstrained geometry does not. The ranking survives changes of configuration, benchmark, model family, adaptation regime and domain, and on the benchmarks it shares with recent one-shot pruning methods it is competitive at moderate compression from a fraction of their signal cost. For a practitioner the recipe is short. Attach LoRA to the router alone at moderate rank, adapt briefly on a corpus that exercises the capabilities the pruned model must keep, rank by norm change, and remove experts uniformly per layer to the target budget, stopping earlier for capabilities that cliff. The corpus is a control surface and the rank a fidelity knob, both cost a negligible fraction of the model they compress, and where deployment already includes parameter-efficient adaptation the ranking arrives as a byproduct, at no additional training cost. Several directions follow directly. Owing to computational constraints, the controlled sweeps concentrate on a single coarse-grained family; validating the configuration findings across MoE families, from fine-grained architectures with many small experts and shared-expert designs to larger Mixtral variants, is the most direct extension, and the Qwen results of Section 8.3 mark the path. Running a controlled evaluation of every criterion on the same checkpoint, calibration corpus, and sample budget would extend the cross-paper comparison in Section 9. Such an evaluation should remove the same number of experts per layer, apply no post-pruning repair, and use identical LightEval prompts and configurations across MMLU-Pro, GSM8K, ARC-Challenge, BIG-Bench Hard, and BoolQ. Completing the module ladder under the logit score separates adaptation of the router from observation of it, and re-ranking stored gate norms answers whether the sign of the change carries information. Beyond the one-shot scope, iterative pruning with interleaved healing and a router-calibration stage are the natural recovery mechanisms, with the most to gain in the fine-grained regime. On the theoretical side, the monotone rank effect and the role of perturbation geometry are regularities the current guarantee does not predict, and extending the analysis to constrained adaptation would close the loop this paper opens. Each direction sharpens the same instrument this work set out to build. References [1] S. Bai, H. Li, J. Zhang, Z. Hong, and S. Guo (2025) DiEP: adaptive mixture-of-experts compression through differentiable expert pruning. arXiv preprint arXiv:2509.16105. Cited by: §2, §8.2, §8.3, Table 8, §9, §9, §9. [2] T. Chen, S. Huang, Y. Xie, B. Jiao, D. Jiang, H. Zhou, J. Li, and F. Wei (2022) Task-specific expert pruning for sparse mixture-of-experts. arXiv preprint arXiv:2206.00277. Cited by: §2. [3] M. N. R. Chowdhury, M. Wang, K. El Maghraoui, N. Wang, P. Chen, and C. Carothers (2024) A provably effective method for pruning experts in fine-tuned sparse mixture-of-experts. In Proceedings of the 41st International Conference on Machine Learning (ICML), Note: arXiv:2405.16646 Cited by: 1st item, §1, §11, §2, §3.2. [4] C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova (2019) BoolQ: exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), p. 2924–2936. Cited by: §4. [5] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018) Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: §4. [6] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §4. [7] D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al. (2024) DeepSeekMoE: towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Note: arXiv:2401.06066 Cited by: §1. [8] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023) QLoRA: efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2305.14314 Cited by: §11, §2, §4. [9] W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. Cited by: §1. [10] C. Fourrier, N. Habib, T. Wolf, and L. Tunstall (2023) LightEval: a lightweight framework for LLM evaluation. Note: https://github.com/huggingface/lighteval Cited by: §4. [11] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019) Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML), p. 2790–2799. Cited by: §1, §2, §3.3, §4. [12] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), Note: arXiv:2106.09685 Cited by: §1, §2, §3.3. [13] G. Hu, M. Li, Q. Peng, L. Hu, B. Xu, and R. Cai (2026) What gets activated: uncovering domain and driver experts in MoE language models. arXiv preprint arXiv:2601.10159. Cited by: §2. [14] S. Hyeon and J. Do (2026) Is retraining-free enough? the necessity of router calibration for efficient MoE compression. arXiv preprint arXiv:2603.02217. Cited by: §11, §2. [15] A. Jaiswal, J. Wang, Y. Li, P. Li, T. Chen, Z. Wang, C. Wang, R. Pang, and X. Du (2025) Finding fantastic experts in MoEs: a unified study for expert dropping strategies and observations. arXiv preprint arXiv:2504.05586. Cited by: §1, §2, §2, §2, §9. [16] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, et al. (2024) Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: §1, §4. [17] Y. Koishekenov, A. Berard, and V. Nikoulina (2023) Memory-efficient NLLB-200: language-specific expert pruning of a massively multilingual machine translation model. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), p. 3567–3585. Cited by: §2. [18] M. Lasby, I. Lazarevich, N. Sinnadurai, S. Lie, Y. Ioannou, and V. Thangarasa (2026) REAP the experts: why pruning prevails for one-shot MoE compression. In International Conference on Learning Representations (ICLR), Note: arXiv:2510.13999 Cited by: §1, §1, §2, §2, §2, Table 8, §9. [19] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen (2021) GShard: scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations (ICLR), Note: arXiv:2006.16668 Cited by: §1. [20] P. Li, Z. Zhang, P. Yadav, Y. Sung, Y. Cheng, M. Bansal, and T. Chen (2024) Merge, then compress: demystify efficient SMoE with hints from its routing policy. In International Conference on Learning Representations (ICLR), Note: arXiv:2310.01334 Cited by: §2. [21] X. L. Li and P. Liang (2021) Prefix-tuning: optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), p. 4582–4597. Cited by: §2. [22] E. Liu, J. Zhu, Z. Lin, X. Ning, M. B. Blaschko, S. Yan, G. Dai, H. Yang, and Y. Wang (2024) Efficient expert pruning for sparse mixture-of-experts language models: enhancing performance and reducing inference costs. arXiv preprint arXiv:2407.00945. Cited by: §2. [23] H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel (2022) Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2205.05638 Cited by: §1, §2, §3.3, §4. [24] X. Lu, Q. Liu, Y. Xu, A. Zhou, S. Huang, B. Zhang, J. Yan, and H. Li (2024) Not all experts are equal: efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), p. 6159–6172. Cited by: §1, §2. [25] I. Moutawwakil and Hugging Face (2023) Optimum-benchmark: a framework for benchmarking the performance of transformers models with different hardwares, backends and optimizations. Note: https://github.com/huggingface/optimum-benchmark Cited by: §10, §4. [26] A. Muzio, A. Sun, and C. He (2024) SEER-MoE: sparse expert efficiency through regularization for mixture-of-experts. arXiv preprint arXiv:2404.05089. Cited by: §2, §2. [27] Qwen Team (2024) Qwen1.5-MoE: matching 7B model performance with 1/3 activated parameters. Note: https://qwenlm.github.io/blog/qwen-moe/Accessed 2026 Cited by: §4. [28] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean (2017) Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), Note: arXiv:1701.06538 Cited by: §1. [29] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei (2023) Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics (ACL Findings), Note: arXiv:2210.09261 Cited by: §4. [30] Thinking Machines Lab (2025) LoRA without regret. Note: https://thinkingmachines.ai/blog/lora/Accessed 2026 Cited by: §11. [31] Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. (2024) MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, Note: arXiv:2406.01574 Cited by: §4. [32] Y. Xie, Z. Zhang, D. Zhou, C. Xie, Z. Song, X. Liu, Y. Wang, X. Lin, and A. Xu (2024) MoE-Pruner: pruning mixture-of-experts large language model using the hints from its router. arXiv preprint arXiv:2410.12013. Cited by: §2, §2. [33] X. Yang, Y. Tian, and Y. Song (2025) MoE pathfinder: trajectory-driven expert pruning. arXiv preprint arXiv:2512.18425. Cited by: §2, Table 8, §9. Appendix A Artifacts and Full Hyperparameter Configurations Code for adaptation, sensitivity scoring, pruning and evaluation is available at https://github.com/ianKa1/MoE_pruning. The pruned Mixtral checkpoints evaluated throughout the paper are available as a collection at https://huggingface.co/collections/Na0s/pruned-moes-mixtral-8x7b-instruct-v01. Table 10 lists the settings governing each adaptation run reported in the paper; the multiplicative and additive-adapter budgets and the two full fine-tuning references are given in Section 4. All runs use the 8-bit AdamW optimizer. Table 10: Adaptation configurations. Canonical denotes the reference run of the controlled sweeps; A through D are the independently configured runs of Section 8.1. Batch is per-device batch size × gradient accumulation. Canonical A B C D Rank r / α 8 / 8 16 / 16 4 / 4 4 / 4 4 / 4 Target modules router all linear router + experts all linear router + attn + gate_projgate\_proj Steps 500 1000 500 200 100 Learning rate 2×10−42× 10^-4 2×10−42× 10^-4 1×10−51× 10^-5 2×10−42× 10^-4 1.5×10−41.5× 10^-4 Batch 1×201× 20 1×161× 16 1×161× 16 8×28× 2 1×41× 4 Warmup 5 steps 5 steps ratio 0.05 5 steps ratio 0.03 Weight decay 0.01 0.01 0.03 0.01 0.04 Schedule cosine cosine cosine cosine cosine, restarts Max grad norm – 0.3 0.3 0.3 0.3 LoRA dropout 0.1 0.1 0.1 0.1 0.1 Appendix B Replication Tables Table 5 in the main text reports the three rank configurations run twice under nominally identical settings; their twelve paired accuracies differ by 0.76 points on average, 0.53 at the median and 2.24 at most, which fixes the measurement resolution of Section 4. Two module-sweep configurations also exist in both project phases under differing names, and Table 11 reports them under the mapping that identifies the phase-one names Router+Attention and Router+Experts with the phase-two target sets gate+q,k,v,o+q,k,v,o and gate++MLP. Their eight paired accuracies differ by 0.43 points on average and 0.97 at most, tightening rather than loosening the resolution; the main text conservatively counts only the three rank configurations, whose identity across phases is unambiguous. Table 11: Module-sweep configurations run in both project phases, under the name mapping stated in the text. Cells give both runs as first / second. MMLU-Pro (%). Experts pruned per layer Target modules 1 2 3 4 gate + q,k,v,oq,k,v,o 40.13 / 39.94 35.68 / 34.98 31.11 / 30.40 26.90 / 25.93 gate + MLP 39.96 / 39.82 35.03 / 34.85 30.33 / 29.97 25.19 / 24.97 Appendix C Router Sensitivity Across Configurations Figure 1 shows the sensitivity map for the all-module rank-16 run; the maps from the other independently configured runs of Section 8.1 share its qualitative structure, differing mainly in overall scale with the adaptation budget. In every case sensitivity is far from uniform, most cells sitting near their layer median while a few expert-layer pairs are an order of magnitude larger, and in no case is the mass concentrated in a single expert column or a single band of layers, which is what motivates per-layer ranking rather than a global threshold (Section 3.4). The signed variant of these maps shows gate norms that contract as well as grow under adaptation, the observation behind the sign question of Section 11. Appendix D Sensitivity Scoring Code The two scores of Section 3.2, written over a generic accessor because the router attribute path differs by model family (block_sparse_moe.gate in HuggingFace Mixtral, mlp.gate in Qwen-MoE). def router(layer): # Mixtral: layer.block_sparse_moe.gate ; Qwen-MoE: layer.mlp.gate return layer.block_sparse_moe.gate.weight # [num_experts, hidden] def weight_scores(base, adapted): # s_e = | ||w_e^(T)|| - ||w_e^(0)|| | per layer (Equation 4) scores = [] for lb, la in zip(base.model.layers, adapted.model.layers): n0 = torch.linalg.norm(router.__wrapped__(lb) if False else router(lb), ord=2, dim=1) nT = torch.linalg.norm(router(la), ord=2, dim=1) scores.append((nT - n0).abs()) return scores # rank ascending, prune the smallest per layer def logit_scores(base, adapted, batch): # s_e = ||dL[:, e]||_2 / sqrt(N) over held-out tokens (Equation 5) L0 = collect_router_logits(base, batch) # [layers, N, num_experts] LT = collect_router_logits(adapted, batch) d = LT - L0 return d.norm(dim=1) / math.sqrt(d.shape[1]) # [layers, num_experts]