Paper deep dive
MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation
Lie Li, Wen Li, Junxiao Shen, Gusheng Hu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/18/2026, 6:07:43 AM
Summary
The paper introduces MAPLE, a plug-and-play framework for heterogeneous expert allocation in Mixture-of-Experts (MoE) Transformers. Unlike uniform allocation, MAPLE uses a sensitivity-guided closed-form solution and genetic search to allocate a fixed expert budget across layers based on their individual sensitivity to expert count changes. Evaluated on models like DeepSeek-MoE-16B, MAPLE achieves higher accuracy and lower latency than uniform baselines and pruning methods using only 75% of the routed experts.
Entities (19)
Relation Signals (14)
MaPLe → improves → DeepSeek-MoE-16B
confidence 95% · MAPLE uses only 75% of the experts yet surpasses the original 100% expert-uniform baseline on ARC-E, ARC-C, and BoolQ
MaPLe → uses → sensitivity-guided allocation
confidence 95% · Our core contribution is a closed-form sensitivity-guided allocation
MaPLe → evaluatedon → OLMoE-1B-7B
confidence 90% · We evaluate MAPLE on four representative MoE models: ... OLMoE-1B-7B
MaPLe → evaluatedon → DeepSeek-V2-Lite
confidence 90% · We evaluate MAPLE on four representative MoE models: ... DeepSeek-V2-Lite
MaPLe → evaluatedon → Moonlight-16B-A3B
confidence 90% · We evaluate MAPLE on four representative MoE models: ... Moonlight-16B-A3B
MaPLe → improves → Throughput
confidence 90% · improves throughput by 47.4%
MaPLe → isimplementedin → SGLang
confidence 90% · implementing MAPLE in SGLang reduces single-GPU end-to-end serving latency
MaPLe → outperforms → LExI
confidence 90% · MAPLE outperforms uniform and pruning-based baselines under a 75% routed-expert budget
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy. We demonstrate that this uniformity is systematically suboptimal and propose MAPLE, a plug-and-play framework that reallocates the routed-expert budget heterogeneously across layers of any pretrained MoE LLM, without modifying weights or requiring retraining. Our core contribution is a closed-form sensitivity-guided allocation: we probe each layer's response to variation in expert count, quantify sensitivity using three measures, and derive an analytically optimal budget assignment that directs capacity towards sensitive layers and absorbs reductions in redundant layers. This closed-form solution is further refined by a sensitivity-constrained genetic search that uses layer-wise sensitivity as a prior to guide exploration, yielding faster convergence and superior allocation quality. On four MoE models spanning different scales and architectures, MAPLE outperforms uniform and pruning-based baselines under a 75% routed-expert budget. Notably, on DeepSeek-MoE-16B, MAPLE uses only 75% of the experts yet surpasses the original 100% expert-uniform baseline on ARC-E, ARC-C, and BoolQ, improving accuracy from 65.09 to 71.40, 48.49 to 51.50, and 80.03 to 82.38, respectively. These accuracy gains translate into measured deployment efficiency: implementing MAPLE in SGLang reduces single-GPU end-to-end serving latency by 32.2% and improves throughput by 47.4%. These results show that well-designed heterogeneous allocation can be more effective than simply activating more experts, establishing it as a principled and practical axis for improving MoE efficiency.
Tags
Links
- Source: https://arxiv.org/abs/2608.15299v1
- Canonical: https://arxiv.org/abs/2608.15299v1
Trouble viewing inline? Open PDF directly →
Full Text
49,707 characters extracted from source content.
Expand or collapse full text
MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation Lie Li Affiliation: University of Bristol Email: fx260102@bristol.ac.uk Wen Li Affiliation: University of Bristol Email: wen.li@bristol.ac.uk Junxiao Shen Affiliation: University of Bristol Email: junxiao.shen@bristol.ac.uk Guosheng Hu Affiliation: University of Bristol Email: g.hu@bristol.ac.uk Abstract Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy. We demonstrate that this uniformity is systematically suboptimal and propose MAPLE, a plug-and-play framework that reallocates the routed-expert budget heterogeneously across layers of any pretrained MoE LLM, without modifying weights or requiring retraining. Our core contribution is a closed-form sensitivity-guided allocation: we probe each layer’s response to variation in expert count, quantify sensitivity using three measures, and derive an analytically optimal budget assignment that directs capacity towards sensitive layers and absorbs reductions in redundant layers. This closed-form solution is further refined by a sensitivity-constrained genetic search that uses layer-wise sensitivity as a before guide for exploration, yielding faster convergence and superior allocation quality. On four MoE models spanning different scales and architectures, MAPLE outperforms uniform and pruning-based baselines under a 75% routed-expert budget. Notably, on DeepSeek-MoE-16B, MAPLE uses only 75% of the experts yet surpasses the original 100% expert-uniform baseline on ARC-E, ARC-C, and BoolQ, improving accuracy from 65.09 to 71.40, 48.49 to 51.50, and 80.03 to 82.38, respectively. These accuracy gains translate into measured deployment efficiency: implementing MAPLE in SGLang reduces single-GPU end-to-end serving latency by 32.2% and improves throughput by 47.4%. These results show that well-designed heterogeneous allocation can be more effective than simply activating more experts, establishing it as a principled and practical axis for improving MoE efficiency. Figure 1: Main idea and key result of MAPLE. (a) Overview of MAPLE: uniform expert allocation is replaced by sensitivity-guided heterogeneous allocation under the same budget. (b) Less compute, better performance: with only 75% routed experts, MAPLE surpasses the full-expert uniform baseline on ARC-Easy (ARC-E), ARC-Challenge (ARC-C), and BoolQ reasoning benchmarks. 1 Introduction Plug-and-play adaptability, first formalised in signal processing (52), describes the ability to insert a module into an existing system and immediately reap benefits without redesigning it from scratch. Deep learning has demonstrated this principle through several landmark interventions: Batch Normalization (27) stabilised training across diverse architectures; Residual connections (21) unlocked networks an order of magnitude deeper; Dropout (46) became a widely applicable regulariser requiring minimal modification. In each case, the intervention was lightweight yet delivered outsized returns. As neural networks developed, the plug-and-play philosophy also became more targeted. Rather than applying the same intervention uniformly across an entire model, researchers began introducing lightweight modules that selectively emphasised the most informative components. In convolutional neural networks, Squeeze-and-Excitation blocks (25) and CBAM (54) exemplified this shift by adaptively recalibrating feature responses through simple add-on modules. The broader lesson was clear: when internal components contribute unevenly, targeted modular interventions can be more effective than uniform ones. As Transformer-based language models continued to scale (8; 43; 2; 5; 49; 50), full retraining became increasingly expensive, making plug-and-play adaptation more attractive. Adapter layers (23), LoRA (24), Prefix tuning (34), prompt tuning (32), and BitFit (59) offered efficient alternatives by updating only a small subset of parameters. Yet despite evidence that different model components contribute unequally (1; 51; 39; 53), these methods still largely treat adaptation uniformly rather than tailoring it to the parts of the model that matter most. This untapped dimension becomes especially important in Mixture-of-Experts (MoE) LLMs (28; 45), where model capacity is exposed explicitly through sparse expert activation. By replacing dense feed-forward layers with sparsely activated expert pools, MoE models decouple parameter capacity from per-token compute (15; 31; 12). With continued improvements in stability and routing (64; 63), MoE architectures have also been adopted in a number of recent high-performing models, including OLMoE (41), DeepSeek-MoE (6), and Moonlight (36). Yet despite this flexibility, current pretrained MoE LLMs still typically use a fixed per-layer number of active experts, even though recent work suggests that expert importance and redundancy vary substantially across layers (19; 37; 60). Existing evidence also points to substantial layer-wise heterogeneity. In parameter-efficient fine-tuning, higher layers have been shown to benefit from more LoRA experts (18), suggesting that adaptation demands are not uniform across depth. More importantly, recent studies on expert pruning and layer compression indicate that redundancy in MoE models varies substantially across layers (37; 35; 11; 3). Taken together, these findings suggest that routed expert capacity should not be allocated uniformly, but instead determined in a layer-aware manner. However, implementing such layer-aware allocation through existing approaches remains costly. Unstructured methods (20; 17; 47) and MoE-specific variants (56) often rely on calibration data, weight reconstruction, or additional recovery procedures to maintain performance. Structured methods that permanently remove experts, whether guided by activation statistics (37), low-rank decomposition (57), condensation (3), Shapley-value estimation (26), or domain-specific calibration (11), similarly introduce extra data, retraining, or task-specific overhead. Expert merging (33) adds further computational complexity. In addition, representative MoE pruning methods such as MoNE (60) and EEP (35) also operate by identifying and removing redundant experts or expert parameters. Since these methods alter the original expert structure rather than only reallocating the active expert budget at inference time, they are less amenable to plug-and-play deployment on pretrained MoE models. In this work, we propose MAPLE (MoE Adaptive Plug-and-play Layer-wise Expert allocation), a plug-and-play framework for automatically determining heterogeneous per-layer expert allocations in pretrained MoE Transformers without modifying weights or retraining. MAPLE formulates expert reallocation as a constrained optimisation problem: it first performs a lightweight layer-wise sensitivity scan, then derives a closed-form heterogeneous allocation under a global expert budget, and subsequently applies a sensitivity-guided genetic refinement, all in a fully post-hoc manner without modifying model architectures. Figure 1 provides an overview of MAPLE, showing that sensitivity-guided heterogeneous expert allocation improves performance under a fixed budget and can even surpass the full-expert uniform baseline using only 75% experts. Our main contributions are as follows: • We identify heterogeneous per-layer expert allocation as an underexplored yet effective axis for improving pretrained MoE LLMs. Unlike existing work (17; 47; 62; 35; 60), we propose MAPLE, a fully plug-and-play framework that requires no weight updates, retraining. • We introduce a sensitivity-guided optimisation strategy for expert reallocation, in which a lightweight layer-wise sensitivity scan enables a closed-form allocation under a global expert budget, with genetic search for subsequent refinement. • Under the same expert budget, MAPLE improves overall performance over the uniform baseline and delivers superior results compared with pruning-based and expert-reallocation methods across five reasoning benchmarks. In particular, even with 25% fewer routed experts, MAPLE is able to surpass the original full-expert MoE on multiple tasks, highlighting the value of sensitivity-guided heterogeneous expert allocation. 2 Related Work Layer-wise Sensitivity Analysis. Understanding layer-wise response to capacity reduction is fundamental to model compression. Prior work has shown that network redundancy is substantial but highly non-uniform across layers 20; 16; 19. Sensitivity-based analyses therefore motivate non-uniform compression, where capacity is allocated according to functional importance rather than reduced uniformly 14; 38. Evidence from structured pruning and attention-head analysis further supports this view, showing that layers and sub-components contribute unequally to performance 39; 53. Mixture-of-Experts Models and Expert Compression. MoE models improve parameter efficiency through sparse expert activation (45; 15), as shown by recent architectures such as Mixtral (29) and DeepSeekMoE (6). To reduce deployment cost, early work adapted dense-model pruning methods to MoE experts, including SparseGPT (17), Wanda (47), and router-aware pruning in MoE-Pruner (56). Because unstructured sparsity often brings limited practical acceleration, later studies shifted to structured expert-level compression. These include activation-frequency-based pruning in Not All Experts are Equal (37), evolutionary expert search in EEP (35), and expert merging in MC-SMoE (33). More recent methods further introduce lightweight replacements or calibration-guided compression, such as MoNE (60), ConDense-MoE (3) and domain-specific expert pruning approaches (11). However, these methods generally modify model weights or structure, rely on calibration data, and often overlook heterogeneous layer-wise redundancy. LExI(4) is more closely related to MAPLE: it also preserves the pretrained model and assigns heterogeneous active-expert counts across layers, but derives the allocation in a data-free manner from model-weight statistics. Our method instead reallocates the active expert budget across layers based on sensitivity, without altering model weights. Budget-Constrained Architecture Search. From an optimisation perspective, allocating a fixed budget non-uniformly across layers is a discrete combinatorial problem whose search space grows rapidly with network depth, making exact optimisation impractical for large models (13). Prior work has therefore relied on approximate search strategies. In CNNs, AMC (22) used reinforcement learning to assign layer-wise compression ratios under a global FLOPs budget and showed clear gains over uniform policies. NetAdapt addressed the same problem with iterative greedy adaptation, trading global optimality for search efficiency (58). NAS later generalised this idea to broader architecture search, where evolutionary methods proved effective in discrete design spaces (13; 44; 48). Similar observations have also emerged in Transformers: MoLA showed that heterogeneous LoRA expert assignment outperforms uniform allocation under the same parameter budget (18). Our method follows this line of work, but avoids expensive search in the allocation step by deriving a closed-form layer-wise expert allocation from sensitivity estimates. 3 Method We present MAPLE, a plug-and-play framework for sensitivity-guided heterogeneous expert allocation in budget-constrained MoE Transformers. Layers that are more sensitive to expert count should deviate less from their preferred allocation. MAPLE proceeds in three steps: it estimates layer-wise sensitivity and identifies each layer’s single-layer optimum (Section 3.1); derives a closed-form allocation under the global budget (Section 3.2); and refines the discrete solution with a lightweight genetic search (Section 3.3). 3.1 Layer-wise Sensitivity Modelling MoE Transformers typically use a uniform expert count across layers, although layers play distinct roles and exhibit varying redundancy (19; 37; 60). Because routing is performed independently at each layer (42), we probe layers individually. Specifically, while fixing all other layers at the baseline expert count k0k_0, we vary the expert count of layer l over ℰ=1,…,k0E=\1,…,k_0\, where e∈ℰe denotes the number of routed experts assigned to layer l, and record the validation accuracy Al(e)A_l(e). The single-layer optimum is kl∗=mine∣Al(e)=maxe′∈ℰAl(e′)k_l^*= \e A_l(e)= _e A_l(e )\. The optimum kl∗k_l^* alone does not indicate how strongly layer l responds to expert-count variation. Prior work in pruning, quantisation, and adaptive fine-tuning similarly reports layer-dependent sensitivity (30; 40; 10; 9; 55; 61). We therefore consider three lightweight sensitivity measures: slacc=maxe∈ℰAl(e)−mine∈ℰAl(e)s_l^acc= _e A_l(e)- _e A_l(e), slw=1|Wl|∑W∈WlVar(W)s_l^w= 1|W_l| _W∈ W_lVar(W), and slout=1|ℰ|∑e∈ℰMSE(Hl(e),Hl(k0))s_l^out= 1|E| _e MSE (H_l(e),H_l(k_0) ). Here, WlW_l is the set of parameter tensors in layer l, and Hl(e)H_l(e) is its output when using e experts. These metrics capture performance-, parameter-, and representation-level sensitivity, respectively. We use accuracy range by default and evaluate the alternatives in the ablation study; sls_l denotes the selected metric. 3.2 Sensitivity-guided Closed-form Expert Allocation Given the preferred expert counts kl∗l=1L\k_l^*\_l=1^L, sensitivities sll=1L\s_l\_l=1^L, and global budget K, we seek an allocation =[k1,…,kL]k=[k_1,…,k_L] that stays close to each layer’s optimum. More sensitive layers should deviate less, so we solve min()=∑l=1Lwl(kl−kl∗)2s.t.∑l=1Lkl=K _k\;J(k)= _l=1^Lw_l (k_l-k_l^* )^2 .t. _l=1^Lk_l=K, where wl=sl>0w_l=s_l>0 in our implementation. Introducing a Lagrange multiplier λ gives ℒ(,λ)=()+λ(∑l=1Lkl−K)L(k,λ)=J(k)+λ\! ( _l=1^Lk_l-K ). Setting ∂ℒ/∂kl=0 /∂ k_l=0 yields klcont=kl∗−λ2wlk_l^cont=k_l^*- λ2w_l, with λ=(∑l=1Lkl∗−K)∑l=1L1/wlλ= 2\! ( _l=1^Lk_l^*-K ) _l=1^L1/w_l. Let Δ=K−∑lkl∗ =K- _lk_l^* and Γ=∑lkl∗−K = _lk_l^*-K. Any surplus or deficit relative to K is assigned preferentially to low-sensitivity layers: klcont=kl∗+Δ⋅1/sl∑j=1L1/sj,if ∑lkl∗<K,kl∗−Γ⋅1/sl∑j=1L1/sj,if ∑lkl∗>K.k_l^cont= casesk_l^*+ · 1/s_l _j=1^L1/s_j,&if _lk_l^*<K,\\[5.69054pt] k_l^*- · 1/s_l _j=1^L1/s_j,&if _lk_l^*>K. cases (1) We clip contk^cont to the feasible range [kmin,kmax][k_ ,k_ ] and apply the largest-remainder method to obtain an integer allocation sensk^sens satisfying the per-layer bounds and global budget. 3.3 Sensitivity-constrained Genetic Refinement The closed-form solution sensk^sens provides a principled initialization but may retain rounding artefacts and residual cross-layer interactions. We therefore apply a lightweight genetic algorithm with validation accuracy A()A(k) as fitness. Candidate allocations are generated by sensitivity-controlled Gaussian mutation: k~l=klsens+ϵl,ϵl∼(0,σl2),σl2∝1sl+ε k_l=k_l^sens+ _l, _l (0, _l^2), _l^2 1s_l+ , so high-sensitivity layers receive smaller perturbations. At each generation, we retain the top two candidates as elites and generate the remaining population using this same perturbation, without crossover. The best candidate gives the final allocation bestk^best. 3.4 Deployment Compatibility Because MAPLE only changes, per MoE layer, how many router-ranked experts are activated at inference time, without modifying the router, expert weights, or model architecture, it can be integrated into any MoE serving engine that exposes a per-layer top-k routing parameter. We implement MAPLE in SGLang by introducing a dedicated execution path for layers assigned k=1k=1: after the router computes per-expert scores, only the highest-scoring (argmax) expert is evaluated, bypassing the multi-expert dispatch, weighted combination, and reduction logic required when k>1k>1. Section 4 reports the resulting end-to-end latency and throughput gains measured in this SGLang implementation. The same mechanism applies to vLLM’s fused MoE kernel, which similarly exposes top-k as a per-layer configuration parameter, so we expect comparable integration to be possible without further modification to MAPLE itself. 4 Experiments 4.1 Experimental Settings Model architectures. We evaluate MAPLE on four representative MoE models: DeepSeek-MoE-16B(6), DeepSeek-V2-Lite(7), Moonlight-16B-A3B(36), and OLMoE-1B-7B(41). To assess expert reallocation under constrained resources, we retain 75% of the routed experts in each model while preserving the number of shared experts when present. As described in Section 3.1, the sensitivity scan and allocation search operate only over each model’s MoE layers, and all reported budgets (75% routed budget / full routed+shared budget, computed over these layers only) are: DeepSeek-MoE-16B 108/162+54; DeepSeek-V2-Lite and Moonlight-16B-A3B 104/156+52; and OLMoE-1B-7B, which has no shared experts, 96/128+0. Evaluation benchmarks. We evaluate performance on seven benchmarks spanning both short-form MCQA and long-form generative reasoning. The MCQA benchmarks include RTE for textual entailment, BoolQ for yes/no question answering, PIQA for physical commonsense reasoning, and ARC-E/ARC-C for science question answering. We further include long-form generative benchmarks: GSM8K for mathematical reasoning with 0-shot CoT and BBH for challenging multi-step reasoning with 3-shot CoT. Implementation and deployment details. All experiments use NVIDIA RTX A6000 GPUs with 48 GB memory and a 75% routed-expert budget, with accuracy range as the default sensitivity metric. MAPLE computes a sensitivity-guided allocation and refines it using 10 candidates over 10 generations, retaining the top two elites per generation while enforcing the global budget and per-layer bounds. For end-to-end serving, we implement MAPLE in SGLang 0.4.6 and evaluate single-GPU, two-GPU pipeline-parallel (P), and two-GPU expert-parallel (EP) configurations. Latency and throughput are averaged over five runs. 4.2 Experimental Results Overall performance. According to Table1, MAPLE improves average accuracy over the full-budget baseline by 2.63, 6.44, and 4.13 points on DeepSeek-MoE-16B, DeepSeek-V2-Lite, and OLMoE-1B-7B, respectively, while using only 75% of the routed experts; on Moonlight-16B-A3B, it is 0.62 points below baseline but still the best-performing method at this budget. Compared with the strongest baseline in each column, MAPLE improves average accuracy by 3.52, 5.79, 1.14, and 3.61 points across the four models, achieving the best average accuracy on every model evaluated. These results show that uniform expert allocation is often suboptimal, and that sensitivity-guided heterogeneous allocation can match or exceed it with fewer active experts. Table 1: Main results under the 75% routed-expert budget on four MoE architectures and five reasoning benchmarks. Baseline denotes the original full-budget uniform routed-expert configuration. Bold indicates the best result in each model–task block. DeepSeek-MoE-16B DeepSeek-V2-Lite Method ARC-C ARC-E BoolQ PIQA RTE Avg. ARC-C ARC-E BoolQ PIQA RTE Avg. Baseline 48.49 65.09 80.03 80.09 84.84 71.71 44.82 64.74 67.40 79.82 53.43 62.04 SparseGPT 46.15 62.11 79.69 79.33 81.22 69.70 44.48 64.91 67.34 78.98 57.76 62.69 Wanda 45.81 62.98 79.72 79.27 82.67 70.09 44.15 64.91 67.31 78.67 54.51 61.91 RIA 46.48 63.16 79.72 79.49 80.87 69.94 45.15 64.04 65.08 78.99 49.82 60.62 EPP 47.49 66.14 79.60 77.86 83.03 70.82 42.47 62.46 67.74 78.40 49.82 60.18 MoNE 47.15 64.04 76.52 76.44 81.46 69.12 42.81 62.11 65.86 78.30 53.42 60.50 LExI 47.49 67.19 79.85 78.35 74.37 69.45 47.16 64.21 68.78 76.82 54.15 62.54 MAPLE 51.50 71.40 82.38 80.85 85.55 74.34 50.84 68.60 73.43 80.20 69.31 68.48 Moonlight-16B-A3B OLMoE-1B-7B Method ARC-C ARC-E BoolQ PIQA RTE Avg. ARC-C ARC-E BoolQ PIQA RTE Avg. Baseline 77.26 88.60 76.54 79.71 69.68 78.36 41.47 54.91 65.96 78.35 77.98 63.73 SparseGPT 75.25 85.79 75.14 78.51 65.34 76.01 43.47 55.09 67.06 77.37 77.61 64.12 Wanda 73.58 85.61 75.08 78.18 64.62 75.41 43.48 54.61 66.88 77.20 77.26 63.89 RIA 76.25 84.25 74.20 78.45 68.31 76.29 43.14 55.96 64.74 78.34 79.06 64.25 EPP 76.60 85.82 74.50 79.65 66.43 76.60 43.83 57.19 66.45 75.19 76.17 63.77 MoNE 74.09 85.80 77.37 77.92 65.53 76.14 33.78 46.15 62.41 76.18 77.62 59.23 LExI 67.89 81.58 70.61 78.67 71.12 73.98 30.43 38.95 63.73 76.82 53.07 52.60 MAPLE 77.59 86.26 74.65 80.16 70.04 77.74 44.81 63.16 68.62 78.94 83.75 67.86 Long-form generative tasks. Beyond short-answer MCQA, we evaluate MAPLE on GSM8K and BBH, which require multi-step chain-of-thought generation rather than single-token answer selection. Table 2 shows that at a 75% budget on DeepSeek-MoE-16B, calibrated directly on each benchmark’s chain-of-thought accuracy, MAPLE preserves GSM8K accuracy within 0.61 points of the full-budget baseline while reducing total evaluation time by 53.0%, and improves BBH accuracy by 0.90 points while reducing total evaluation time by 6.3%. These results indicate that MAPLE’s gains are not confined to short-form multiple-choice calibration, and that heterogeneous allocation remains competitive with, and can exceed, full-budget uniform routing on long-form generative reasoning. Table 2: GSM8K and BBH accuracy and total evaluation wall-clock time at a 75% budget, DeepSeek-MoE-16B, directly calibrated on each benchmark’s chain-of-thought accuracy. GSM8K uses the complete 1,319-example test set (0-shot); BBH uses the complete 5,911-example set (3-shot CoT). GSM8K BBH Configuration Acc. Total time Acc. Total time Full uniform (100%) 58.53 492.26 s 43.43 5,776.97 s MAPLE (75%) 57.92 231.48 s 44.33 5,434.15 s Δ −0.61-0.61 −53.0%-53.0\% +0.90+0.90 −6.3%-6.3\% Efficiency profile: offline cost versus serving gains. Table 3 reports both sides of MAPLE’s practical cost–benefit trade-off on DeepSeek-MoE-16B: panel (a) gives the one-time offline calibration cost, incurred once per model before the resulting allocation is reused indefinitely at no further overhead; panels (b) and (c) give measured serving latency and throughput in SGLang 0.4.6 and vLLM respectively, both on an ARC-Easy workload, where MoE layers assigned a single active expert use a dedicated execution path that skips multi-expert dispatch, and the model’s dense layer is unaffected throughout. In vLLM, we additionally report a fused CUDA kernel variant of this k=1k=1 path (“MAPLE 75%”), which further reduces kernel-launch overhead. Gains under the two-GPU pipeline and expert parallelism are smaller than the single-GPU setting in SGLang, since communication and synchronisation overheads do not shrink in proportion to reduced expert computation. MAPLE retains all expert weights throughout, so its benefit is reduced active computation and latency rather than model-size compression. Table 3: MAPLE’s efficiency profile. (a) One-time offline calibration cost per model, summed across five tasks, on a single RTX A6000. (b) Measured SGLang 0.4.6 serving time on DeepSeek-MoE-16B at a 75% budget, averaged over five runs, ARC-Easy workload. (c) Measured vLLM serving throughput and latency on DeepSeek-MoE-16B at a 75% budget, ARC-Easy workload. P: pipeline parallelism; EP: expert parallelism. Model Total (h) DeepSeek-MoE-16B 7.39 DeepSeek-V2-Lite 14.86 Moonlight-16B-A3B 9.81 OLMoE-1B-7B 6.10 (a) Offline cost Setting Full Uniform, 75% MAPLE, 75% Time ↓ Tput. ↑ 1 GPU 2.860 s 2.680 s 1.940 s 32.2% 47.4% 2 GPU, P 3.660 s 3.660 s 3.470 s 5.2% 5.5% 2 GPU, EP 9.056 s 8.809 s 8.701 s 3.9% 4.1% (b) SGLang serving latency and throughput Setting Full Uniform, 75% MAPLE, 75% Time ↓ Tput. ↑ 1 GPU 3.915 s 3.745 s 3.736 s 4.58% 4.80% 2 GPU, P 13.703 s 11.819 s 11.762 s 14.16% 16.50% (c) vLLM serving latency and throughput 4.3 Ablation Study Sensitivity metrics. Table 4 compares different combinations of accuracy-range, weight-variance, and output-deviation sensitivity. Table 4: Ablation of sensitivity metrics (Acc.: accuracy range, Weight: weight variance, Output: output deviation). Best Gen. is the generation at which the best accuracy is reached. Acc. Weight Output Best Gen. Best Acc. Δ 1 – 84.84 – 2 ✓ 2 85.55 +0.71 3 ✓ ✓ 4 85.20 +0.36 4 ✓ ✓ 6 84.12 -0.72 5 ✓ ✓ 10 83.75 -1.08 6 ✓ ✓ ✓ 9 85.55 +0.71 The full-uniform baseline reaches 84.84% (row 1). Accuracy range alone (row 2) achieves the largest gain (+0.71) in just 2 generations. Adding weight variance (row 3) converges more slowly with a smaller gain (+0.36, gen. 4), while adding output deviation instead (row 4) hurts performance (-0.72, gen. 6). Weight and output variance without accuracy range (row 5) perform worst of all (-1.08, gen. 10), showing accuracy range does most of the useful work and the other two signals are redundant at best. Combining all three (row 6) matches the accuracy range alone (+0.71) but needs far more generations (9 vs. 2) to get there. We therefore use accuracy range as the default metric throughout the paper. Figure 2 confirms this across tasks under the same 75% budget, where accuracy range remains consistently competitive while weight and output variance are less stable. We therefore use accuracy range as the default metric. Figure 2: Performance of different sensitivity metrics in MAPLE under the 75% expert budget across five tasks. Bold bar outlines indicate the best result within each task. Refinement and stability. Table 5 ablates the refinement step while keeping the budget and sensitivity metric fixed: refinement improves the closed-form allocation on every benchmark, with gains ranging from +0.82 on PIQA to +7.01 on ARC-E, showing that the closed-form solution provides an effective initial allocation while lightweight task-specific refinement yields further improvements. Since this stage is stochastic, we additionally rerun it with five independent seeds on DeepSeek-MoE-16B, fixing the model, calibration split, prompts, and expert budget, and varying only random initialisation; the resulting seed standard deviation is an order of magnitude smaller than the refinement gain, indicating a stable solution rather than exploited noise. Table 5: Effect of refinement and seed stability. Closed-form denotes the initial allocation, +Refinement applies the proposed refinement step, and Seed Std. reports the standard deviation of +Refinement accuracy across five independent refinement seeds. Task Closed-form +Refinement Δ Seed Std. ARC-C 48.49 51.50 +3.01 ±1.16± 1.16 ARC-E 64.39 71.40 +7.01 ±0.90± 0.90 BoolQ 80.70 82.39 +1.68 ±0.45± 0.45 PIQA 80.03 80.85 +0.82 ±0.21± 0.21 RTE 83.39 85.55 +2.16 ±0.75± 0.75 Figure 3 examines the refinement dynamics across tasks. Despite substantially different layer-wise sensitivity landscapes across benchmarks, most gains occur in the first few generations and the search converges quickly, supporting the use of a lightweight local refinement rather than an expensive global search. (a) (b) Figure 3: (a) Task-wise standardised sensitivity distributions under the 75% expert budget. (b) Normalised convergence trends across tasks, where Remaining Gap denotes the performance difference from the best-found solution. Robustness of the calibration signal. Table 6 tests whether MAPLE’s gains depend on precise, task-specific calibration, or hold under weaker calibration signals on DeepSeek-MoE-16B at a 75% budget. A single allocation jointly calibrated across all five tasks (Reusable) still delivers strong accuracy without any per-task recalibration, showing MAPLE does not require a fresh search for every deployment task. MAPLE also transfers well across tasks: applying each task’s searched allocation directly to the other four (Cross-task transfer, averaged over the 20 off-diagonal source–target pairs shown in the lower rows) performs on par with established pruning methods despite never being calibrated on the target task, indicating that the sensitivity patterns MAPLE identifies reflect genuine, transferable properties of the model’s layers rather than fitting narrowly to one evaluation set. As an adversarial control, we replace task accuracy with negative token-level NLL on two generic corpora, C4 and FineWeb-Edu, sharing no examples, labels, or format with any evaluation task. Taken together, these results show MAPLE is robust along three independent axes: it does not need per-task recalibration, its allocations generalise across tasks, and its gains are not an artefact of the search procedure. Table 6: Robustness of MAPLE’s calibration signal on DeepSeek-MoE-16B at a 75% budget. Reusable, Cross-task transfer, C4-NLL, and FineWeb-Edu-NLL are alternative calibration signals; C4-NLL/FineWeb-Edu-NLL replace task accuracy with generic-corpus perplexity. The remaining rows give the full cross-task transfer matrix underlying the Cross-task transfer average above. Calibration signal ARC-C ARC-E BoolQ PIQA RTE Avg. Reusable (joint 5-task) 49.83 68.95 82.05 78.62 83.39 72.57 Cross-task transfer (avg. of 20 pairs) 43.65 65.09 78.83 78.81 79.33 69.14 C4-NLL 44.15 65.26 78.99 79.43 80.14 69.59 FineWeb-Edu-NLL 44.48 62.11 76.45 79.76 81.59 68.88 ARC-C (source) 51.50 64.56 79.24 78.40 79.06 70.55 ARC-E (source) 44.82 71.40 78.75 78.67 76.90 70.11 BoolQ (source) 40.80 64.56 82.38 79.16 81.95 69.77 PIQA (source) 44.48 64.74 79.14 80.85 79.42 69.73 RTE (source) 44.48 66.49 78.17 79.00 85.55 70.74 Expert budget. MAPLE’s advantage over Wanda and SparseGPT holds at every budget level and widens at the tighter 50% budget, suggesting heterogeneous reallocation becomes more valuable as naive pruning is more likely to remove capacity from layers that still need it. Heterogeneous allocation also improves over the uniform baseline at 100% budget, indicating uniform allocation itself leaves room for better capacity placement independent of any budget reduction. Table 7: ARC-C / RTE accuracy under 50%, 75%, and 100% expert budgets on DeepSeek-MoE-16B. Wanda and SparseGPT are omitted at 100% since their unpruned configuration is identical to the uniform baseline. Budget Wanda SparseGPT MAPLE 50% 41.14 / 72.92 39.46 / 72.20 46.49 / 79.78 75% 45.81 / 82.67 46.15 / 81.22 51.50 / 85.55 100% (uniform baseline: 48.49 / 84.84) – – 51.84 / 88.09 5 Conclusions and Limitations We presented MAPLE, a plug-and-play framework for heterogeneous expert allocation in budget-constrained MoE Transformers. MAPLE estimates layer-wise sensitivity, derives a closed-form allocation, and refines it with a lightweight genetic search, without weight updates, retraining, or architectural modification. Experiments on four MoE models and five benchmarks show that MAPLE outperforms pruning-based baselines under a 75% routed-expert budget, and surpasses the full-budget uniform configuration on three models while using fewer routed experts. These results indicate that uniform expert allocation is often suboptimal, and that layer-wise capacity redistribution can better match the heterogeneous demands of MoE layers. In the future, we plan to evaluate MAPLE on diverse hardware platforms, including GPUs, CPUs, and edge devices, and under broader real-world inference scenarios. This will allow us to further assess its practical efficiency, deployment flexibility, and applicability beyond the current experimental setting. In particular, hardware-aware evaluation can reveal whether reduced routed-expert computation translates into actual latency, memory, and energy benefits across different deployment environments. Such deployment-oriented analysis may also provide deeper insight into how heterogeneous expert allocation interacts with system-level constraints in practical inference systems, including memory bandwidth, batching behaviour, and hardware support for sparse MoE execution. References Bahdanau et al. (2015) D. Bahdanau, K. Cho, and Y. Bengio Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), Cited by: §1. Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS) 33, p. 1877–1901. Cited by: §1. Cao et al. (2024) M. Cao, G. Li, J. Ji, J. Zhang, X. Ma, S. Liu, and L. Yin Condense, don’t just prune: enhancing efficiency and performance in MoE layer pruning. arXiv preprint arXiv:2412.00069. Cited by: §1, §1, §2. Chitty-Venkata et al. (2025) K. T. Chitty-Venkata, S. Madireddy, M. Emani, and V. Vishwanath LExI: layer-adaptive active experts for efficient moe model inference. External Links: 2509.02753, Link Cited by: §2. Chowdhery et al. (2023) A. Chowdhery, S. Narang, J. Devlin, et al. PaLM: scaling language modeling with pathways. Journal of Machine Learning Research 24, p. 1–113. Cited by: §1. Dai et al. (2024) D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al. DeepSeekMoE: towards ultimate expert specialization in mixture-of-experts language models. External Links: 2401.06066 Cited by: §1, §2, §4.1. DeepSeek-AI (2024) DeepSeek-AI DeepSeek-V2: a strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434. Cited by: §4.1. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, p. 4171–4186. Cited by: §1. Dong et al. (2020) Z. Dong, Z. Yao, D. Arfeen, A. Gholami, M. W. Mahoney, and K. Keutzer HAWQ-V2: Hessian aware trace-weighted quantization of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33, p. 18518–18529. Cited by: §3.1. Dong et al. (2019) Z. Dong, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer HAWQ: Hessian AWare quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 293–302. Cited by: §3.1. Dong et al. (2025) Z. Dong, H. Peng, P. Liu, W. X. Zhao, D. Wu, F. Xiao, and Z. Wang Domain-specific pruning of large mixture-of-experts models with few-shot demonstrations. arXiv preprint arXiv:2504.06792. Cited by: §1, §1, §2. Du et al. (2022) N. Du, Y. Huang, A. M. Dai, et al. GLaM: efficient scaling of language models with mixture-of-experts. In Proceedings of ICML, p. 5547–5569. Cited by: §1. Elsken et al. (2019) T. Elsken, J. H. Metzen, and F. Hutter Neural architecture search: a survey. Journal of Machine Learning Research 20 (55), p. 1–21. Cited by: §2. Engelbrecht (2001) A. P. Engelbrecht A new pruning heuristic based on variance analysis of sensitivity information. IEEE Transactions on Neural Networks 12 (6), p. 1386–1399. Cited by: §2. Fedus et al. (2022) W. Fedus, B. Zoph, and N. Shazeer Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. Cited by: §1, §2. Frankle and Carbin (2019) J. Frankle and M. Carbin The lottery ticket hypothesis: finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), Cited by: §2. Frantar and Alistarh (2023) E. Frantar and D. Alistarh SparseGPT: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning (ICML), p. 10323–10337. Cited by: 1st item, §1, §2. Gao et al. (2024) C. Gao, K. Chen, J. Rao, B. Sun, R. Liu, D. Peng, Y. Zhang, X. Guo, J. Yang, and V. S. Subrahmanian Higher layers need more LoRA experts. arXiv preprint arXiv:2402.08562. Cited by: §1, §2. Guo et al. (2023) S. Guo, B. Lai, S. Yang, J. Zhao, and F. Shen Sensitivity pruner: filter-level compression algorithm for deep neural networks. Pattern Recognition 140, p. 109508. External Links: Document Cited by: §1, §2, §3.1. Han et al. (2015) S. Han, J. Pool, J. Tran, and W. J. Dally Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 28. Cited by: §1, §2. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–778. Cited by: §1. He et al. (2018) Y. He, J. Lin, Z. Liu, H. Wang, L. Li, and S. Han AMC: AutoML for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), p. 815–832. Cited by: §2. Houlsby et al. (2019) N. Houlsby, A. Giurgiu, S. Jastrzebski, et al. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML), p. 2790–2799. Cited by: §1. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, et al. LoRA: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: §1. Hu et al. (2018) J. Hu, L. Shen, and G. Sun Squeeze-and-excitation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 7132–7141. Cited by: §1. Huang et al. (2025) W. Huang, Y. Zhang, X. Zheng, F. Chao, R. Ji, and L. Cao Discovering important experts for mixture-of-experts models pruning through a theoretical perspective. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1. Ioffe and Szegedy (2015) S. Ioffe and C. Szegedy Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning (ICML), p. 448–456. Cited by: §1. Jacobs et al. (1991) R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton Adaptive mixtures of local experts. Neural Computation 3 (1), p. 79–87. Cited by: §1. Jiang et al. (2024) A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: §2. LeCun et al. (1990) Y. LeCun, J. S. Denker, and S. A. Solla Optimal brain damage. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 2, p. 598–605. Cited by: §3.1. Lepikhin et al. (2021) D. Lepikhin, H. Lee, Y. Xu, et al. GShard: scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668. Cited by: §1. Lester et al. (2021) B. Lester, R. Al-Rfou, and N. Constant The power of scale for parameter-efficient prompt tuning. In Proceedings of EMNLP, p. 3045–3059. Cited by: §1. Li et al. (2024) P. Li, Z. Zhang, P. Yadav, Y. Sung, Y. Cheng, M. Bansal, and T. Chen Merge, then compress: demystify efficient SMoE with hints from its routing policy. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. Li and Liang (2021) X. L. Li and P. Liang Prefix-tuning: optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), p. 4582–4597. Cited by: §1. Liu et al. (2024) E. Liu, J. Zhu, Z. Lin, X. Ning, M. B. Blaschko, S. Yan, G. Dai, H. Yang, and Y. Wang Efficient expert pruning for sparse mixture-of-experts language models: enhancing performance and reducing inference costs. arXiv preprint arXiv:2407.00945. Cited by: 1st item, §1, §1, §2. Liu et al. (2025) J. Liu, J. Su, X. Yao, et al. Muon is scalable for LLM training. External Links: 2502.16982 Cited by: §1, §4.1. Lu et al. (2024) X. Lu, Q. Liu, Y. Xu, A. Zhou, S. Huang, B. Zhang, J. Yan, and H. Li Not all experts are equal: efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §1, §1, §1, §2, §3.1. Ma et al. (2023) X. Ma, G. Fang, and X. Wang LLM-Pruner: on the structural pruning of large language models. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36. Cited by: §2. Michel et al. (2019) P. Michel, O. Levy, and G. Neubig Are sixteen heads really better than one?. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 32. Cited by: §1, §2. Molchanov et al. (2019) P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 11264–11272. Cited by: §3.1. Muennighoff et al. (2024) N. Muennighoff et al. OLMoE: open mixture-of-experts language models. arXiv preprint arXiv:2409.02060. Cited by: §1, §4.1. Qiu et al. (2025) Z. Qiu, Z. Huang, S. Cheng, Y. Zhou, Z. Wang, I. Titov, and J. Fu Layerwise recurrent router for mixture-of-experts. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §3.1. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (140), p. 1–67. Cited by: §1. Real et al. (2019) E. Real, A. Aggarwal, Y. Huang, and Q. V. Le Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, p. 4780–4789. Cited by: §2. Shazeer et al. (2017) N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. Srivastava et al. (2014) N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15, p. 1929–1958. Cited by: §1. Sun et al. (2024) M. Sun, Z. Liu, A. Bair, and J. Z. Kolter A simple and effective pruning approach for large language models. In International Conference on Learning Representations (ICLR), Cited by: 1st item, §1, §2. Tan et al. (2019) M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, and Q. V. Le MnasNet: platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. Touvron et al. (2023a) H. Touvron, T. Lavril, G. Izacard, et al. LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1. Touvron et al. (2023b) H. Touvron, L. Martin, K. Stone, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §1. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30. Cited by: §1. Venkatakrishnan et al. (2013) S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg Plug-and-play priors for model based reconstruction. In IEEE Global Conference on Signal and Information Processing (GlobalSIP), p. 945–948. Cited by: §1. Voita et al. (2019) E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov Analyzing multi-head self-attention: specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), p. 5797–5808. Cited by: §1, §2. Woo et al. (2018) S. Woo, J. Park, J. Lee, and I. S. Kweon CBAM: convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), p. 3–19. Cited by: §1. Wu et al. (2020) H. Wu, P. Judd, X. Zhang, M. Isaev, and P. Micikevicius Integer quantization for deep learning inference: principles and empirical evaluation. CoRR abs/2004.09602. External Links: Link, 2004.09602 Cited by: §3.1. Xie et al. (2024) Y. Xie, Z. Zhang, D. Zhou, C. Xie, Z. Song, X. Liu, Y. Wang, X. Lin, and A. Xu MoE-pruner: pruning mixture-of-experts large language model using the hints from its router. External Links: 2410.12013, Link Cited by: §1, §2. Yang et al. (2024) C. Yang, Y. Sui, J. Xiao, et al. MoE-I2^2: compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 10456–10466. Cited by: §1. Yang et al. (2018) T. Yang, A. Howard, B. Chen, X. Zhang, A. Go, M. Sandler, V. Sze, and H. Adam NetAdapt: platform-aware neural network adaptation for mobile applications. In Proceedings of the European Conference on Computer Vision (ECCV), p. 285–300. Cited by: §2. Zaken et al. (2022) E. B. Zaken, S. Ravfogel, and Y. Goldberg BitFit: simple parameter-efficient fine-tuning for transformer-based masked language models. arXiv preprint arXiv:2106.10199. Cited by: §1. Zhang et al. (2026) G. Zhang, Y. Han, Y. Lou, Y. Zhang, W. Zhao, and Y. You MoNE: replacing redundant experts with lightweight novices for structured pruning of MoE. In International Conference on Learning Representations (ICLR), Note: arXiv:2507.00390 Cited by: 1st item, §1, §1, §2, §3.1. Zhang et al. (2023) Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao AdaLoRA: adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Representations (ICLR), Cited by: §3.1. Zhang et al. (2024) Y. Zhang, H. Bai, H. Lin, J. Zhao, L. Hou, and C. V. Cannistraci Plug-and-play: an efficient post-training pruning method for large language models. In International Conference on Learning Representations (ICLR), Cited by: 1st item. Zhou et al. (2022) Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Q. V. Le, and J. Laudon Mixture-of-experts with expert choice routing. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. Cited by: §1. Zoph et al. (2022) B. Zoph, I. Bello, S. Kumar, et al. ST-MoE: designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906. Cited by: §1. Appendix A Appendix You may include other additional sections here. A.1 Full Algorithm Algorithm 1 summarises the complete three-step MAPLE pipeline. Input: MoE model with L layers; base expert count k0k_0; total budget K; valid range [kmin,kmax][k_ ,k_ ]; validation set valD_val; population size P; generations G Output: Final allocation bestk^best 1 Step 1: Layer-wise Sensitivity Modelling; 2 for l←1l← 1 to L do 3 Probe layer l by varying its expert count from 11 to k0k_0, with all other layers fixed at k0k_0; 4 Record Al(e)A_l(e) for each candidate count e, and set the preferred count kl∗=mine∣Al(e)=maxe′Al(e′)k_l^*= \e A_l(e)= _e A_l(e )\; 5 Compute the sensitivity score sl=maxeAl(e)−mineAl(e)s_l= _eA_l(e)- _eA_l(e) (accuracy range; see Section 3.1 for alternative metrics); 6 7 Step 2: Sensitivity-guided Closed-form Expert Allocation; 8 if ∑l=1Lkl∗<K _l=1^Lk_l^*<K then 9 Let Δ=K−∑l=1Lkl∗ =K- _l=1^Lk_l^*; 10 Allocate the surplus to less sensitive layers: klcont=kl∗+Δ⋅1/sl∑j=1L1/sjk_l^cont=k_l^*+ · 1/s_l _j=1^L1/s_j for each layer l; 11 else 12 Let Γ=∑l=1Lkl∗−K = _l=1^Lk_l^*-K; 13 Remove the excess from less sensitive layers: klcont=kl∗−Γ⋅1/sl∑j=1L1/sjk_l^cont=k_l^*- · 1/s_l _j=1^L1/s_j for each layer l; 14 Clip contk^cont to [kmin,kmax][k_ ,k_ ] and project it to integers via the largest-remainder method to obtain sensk^sens; 15 16 Step 3: Sensitivity-constrained Genetic Refinement; 17 Initialise the population P around sensk^sens by sampling k~l=klsens+ϵl k_l=k_l^sens+ _l, with ϵl∼(0,σl2) _l (0, _l^2) and σl2∝1sl+ε _l^2 1s_l+ ; 18 for g←1g← 1 to G do 19 Evaluate each candidate ∈k on valD_val and compute its fitness f()=A()f(k)=A(k); 20 Keep the top-2 candidates in P as elites; 21 Refill P by perturbing elite solutions with sensitivity-controlled Gaussian noise, then project each result back to the feasible set; 22 return best=argmax∈f()k^best= _k f(k); 23 Algorithm 1 Pseudocode for MAPLE