Paper deep dive
APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference
Alish Kanani, Layan Badawi, Umit Y. Ogras
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mixture-of-Experts (MoE) models are attractive for edge deployment because they provide high model capacity while activating only a small subset of parameters per token, improving compute efficiency. However, MoE inference at the edge is fundamentally limited by memory. Expert parameters are large and often reside in off-chip memory due to capacity, cost, and power constraints, putting expert loading to the critical path. We present APEX: Adaptive Expert Prefetching, a predictive resource management framework that overlaps expert loading with useful computation. APEX introduces a lightweight prefetch router that predicts candidate experts before the attention block to dynamically fetch additional experts using a learned confidence model. This adaptive strategy achieves over 99% overlap accuracy, significantly outperforming fixed top-k prefetching techniques. APEX supports two execution modes: a correctness-preserving mode that guarantees exact routing semantics, and a stall-free mode that eliminates residual stalls by operating on available experts with negligible impact on application accuracy. Across multiple MoE models, the correctness-preserving mode reduces per-token latency by up to 26% and improves energy-delay product (EDP) by up to 41% over state-of-the-art baselines, while the stall-free mode provides additional efficiency gains with negligible impact on application accuracy. These results establish adaptive, confidence-driven expert prefetching as an effective approach for efficient MoE inference on edge systems.
Tags
Links
- Source: https://arxiv.org/abs/2608.11688v1
- Canonical: https://arxiv.org/abs/2608.11688v1
Trouble viewing inline? Open PDF directly →
Full Text
81,603 characters extracted from source content.
Expand or collapse full text
APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference This work was partially supported by the Intel CAD SRS program. Alish Kanani, Layan Badawi and Umit Y. Ogras Affiliation: University of Wisconsin–Madison; ahkanani, lbadawi, uogras@wisc.edu Abstract Mixture-of-Experts (MoE) models are attractive for edge deployment because they provide high model capacity while activating only a small subset of parameters per token, improving compute efficiency. However, MoE inference at the edge is fundamentally limited by memory. Expert parameters are large and often reside in off-chip memory due to capacity, cost, and power constraints, putting expert loading to the critical path. We present APEX: Adaptive Expert Prefetching, a predictive resource management framework that overlaps expert loading with useful computation. APEX introduces a lightweight prefetch router that predicts candidate experts before the attention block to dynamically fetch additional experts using a learned confidence model. This adaptive strategy achieves over 99% overlap accuracy, significantly outperforming fixed top-k prefetching techniques. APEX supports two execution modes: a correctness-preserving mode that guarantees exact routing semantics, and a stall-free mode that eliminates residual stalls by operating on available experts with negligible impact on application accuracy. Across multiple MoE models, the correctness-preserving mode reduces per-token latency by up to 26% and improves energy-delay product (EDP) by up to 41% over state-of-the-art baselines, while the stall-free mode provides additional efficiency gains with negligible impact on application accuracy. These results establish adaptive, confidence-driven expert prefetching as an effective approach for efficient MoE inference on edge systems. I Introduction Large language models (LLMs) are increasingly deployed at the edge, where low-latency response, energy efficiency, data privacy, and reduced cloud dependence are critical requirements [33, 55]. Mixture-of-Experts (MoE) models are particularly well-suited for this setting, as they decouple model capacity from per-token compute [10, 22]. By activating only a small subset of experts per token, MoEs achieve high accuracy while operating within tight compute (FLOPs) budgets. This technique enables edge accelerators to support much larger models that would otherwise be computationally prohibitive. MoE inference at the edge faces a fundamental memory bottleneck [12]. Unlike dense models with deterministic weights, MoEs repeatedly fetch large, sparse, and irregularly accessed expert parameters [26, 43]. Keeping all experts resident in high-bandwidth on-package memory (e.g., HBM or GDDR) is impractical on edge platforms due to capacity, cost, thermal, and power constraints [55, 38, 42]. In practice, expert parameters are stored in lower-cost off-chip memory (e.g., LPDDR), while non-expert weights, activations, and the key-value (KV) cache remain on-package. Although this organization is more realistic for edge AI accelerators, it places expert loading on the critical path of token generation. Figure 1 illustrates the memory bottleneck on a representative high-end edge accelerator composed of a compute chiplet with on-package memory and an I/O chiplet. The modeled compute capability and on-package memory bandwidth are comparable to high-end edge GPU platforms such as NVIDIA RTX 3090 [35]. To reflect practical deployments, expert weights reside in host-attached LPDDR5X and are streamed to the accelerator over a PCIe 6.0 ×16 link. Non-expert weights, activations, and the KV cache remain on-package. Full timing and per-bit energy parameters are provided in Section V. Under this realistic memory hierarchy, expert loading becomes a dominant cost. For single-token generation with the Granite-3.1-3B-A800M model [21], expert loading contributes 43% of latency and 29% of total energy. Importantly, this cost is not purely a bandwidth problem. The compute package remains underutilized yet continues to dissipate static power while waiting for expert transfers. Fig. 1: Latency and energy breakdown for the Granite-3.1-3B-A800M model on a representative edge platform with off-chip expert memory and 1024-token context length. We argue that the solution is not static memory scaling, but adaptive resource management [23]. If the required experts can be predicted early, they can be prefetched to reduce exposed I/O latency. Prior work follows this idea by statically prefetching top-k predicted experts, assuming that prediction accuracy (i.e., the overlap between the predicted and actual top-k experts) is sufficient to ensure coverage [49]. However, routing uncertainty varies significantly across layers and tokens, and prediction errors can lead to missing experts. Even a relatively high average prediction accuracy (e.g., 70–85%) is insufficient, since some layers may miss consistently more experts and experience stalls that negate much of the prefetching benefit [49, 56, 52]. Hence, existing approaches with a fixed prefetch budget fail to adapt to layer and input variations, leaving significant performance and energy gains unrealized. TABLE I: Comparison of state-of-the-art MoE inference methods for edge deployment Approach Early Expert Prediction Dynamic Prefetch Budget Correctness Guarantee Stall-free Key Idea Pre-gated MoE [19] × × × ✓ Router in layer L selects experts for layer L+1L+1 MoE-Infinity [54] × × ✓ × Prefetching is guided by historical traces Fate [9] ✓ × × × Cross-Layer prefetch router HOBBIT [52] ✓ × a × × Prefetching mixed precision experts Pre-Attn Pred. [57] ✓ × ✓ × Same-layer pre-attention expert prediction ProMoE [49] ✓ × ✓ × Learned proactive expert caching APEX ✓ ✓ ✓ ✓ Adaptive expert prefetching using per-token confidence a It uses an adaptive predictor that continues predicting future layers while the predicted experts remain in cache, but it is still not adaptive per-token. We address this problem with APEX, an adaptive expert prefetching framework that overlaps expert loading with attention computation. APEX adds a lightweight prefetch router before the attention block to predict a ranked list of candidate experts for the current layer. Rather than always prefetching a fixed top-k set, APEX exploits the observation that most routing misses can be avoided by fetching only a small, token-dependent number of additional experts. Therefore, it prefetches a top-(k+δ^(x))(k+ δ(x)) set, where δ^(x) δ(x) is dynamically selected per token using a learned confidence model. This enables APEX to fetch just enough additional experts to suppress misses without paying the energy cost of aggressive over-prefetching on every token. While fetching additional experts slightly increases communication energy, it substantially reduces the latency and stall time associated with exposed expert loading, resulting in a net gain in both energy and performance. APEX supports two execution modes: (1) a correctness-preserving mode guarantees exact routing semantics by fetching any rarely missing experts, and (2) an optional stall-free mode, which eliminates residual stalls by operating on available experts when the application can tolerate the resulting accuracy trade-off. We evaluate APEX across IBM Granite-3.1-1B-A400M [20], Granite-3.1-3B-A800M [21], Microsoft Phi-mini-MoE-7B-A2.4B [1] and DeepSeek-V2-Lite-16B-A2.4B [30] models on edge configurations with off-chip expert memory. It achieves near-oracle overlap (>>99%), substantially reducing expert-loading stalls and improving end-to-end EDP by up to 41% over state-of-the-art baselines. We further show that the stall-free variant achieves an additional 2–14% EDP improvement with negligible impact on application accuracy. These results demonstrate that adaptive expert prefetching is a practical and effective approach for enabling efficient MoE inference at the edge. The key contributions of this work are as follows: • An adaptive top-(k+δ^(x)k+ δ(x)) prefetching strategy that adaptively selects the number of prefetched experts using a learned confidence model. • Two execution modes: a correctness-preserving mode, which guarantees exact routing semantics, and an optional stall-free mode, which further reduces latency and energy with negligible impact on application accuracy. • Comprehensive evaluations across multiple MoE models on an edge platform with off-chip expert memory, achieving >>99% expert overlap and up to 41% improvement in end-to-end EDP over state-of-the-art baselines. In the rest, Section I reviews related work, while Section I provides background on MoE inference. Section IV presents our adaptive expert prefetching method, followed by evaluations in Section V, and Section VI concludes the paper. I Related Work MoE models scale model capacity by replacing dense feed-forward networks with multiple experts and a router that activates only a sparse subset per token [48]. This conditional-computation paradigm has been extended to large-scale training systems such as GShard [26] and later improved through routing stability, load-balancing, and training-stability techniques [58, 27, 59, 4]. More recently, MoE architectures have been adopted in both open and proprietary LLMs. Models such as Mixtral [22], DeepSeekMoE [6], and OLMoE [34] extend MoE designs to modern decoder-only LLMs, enabling efficient scaling in practical deployments. A substantial body of work addresses system-level challenges in MoE training and inference at a datacenter scale [17, 7]. Early systems, such as FastMoE [16], DeepSpeed-MoE [2], and MegaBlocks [11], optimize token dispatch, expert parallelism, and all-to-all communication to enable efficient large-scale training. However, these systems assume that expert parameters reside in accelerator memory and rely on high-bandwidth interconnects. These assumptions do not hold in edge deployments, where memory capacity is limited, and experts are often offloaded to external memory. As a result, expert movement becomes a dominant runtime bottleneck. Hot-expert caching methods, such as MoE-Infinity [54], exploit temporal locality by keeping frequently activated experts in GPU memory and prefetching others from host memory. This is effective when persistent device memory is available and expert reuse is stable across tokens or requests. In edge settings, however, on-package memory must also hold non-expert weights, activations, the KV cache, and current-layer staging buffers. Reserving capacity for static hot experts can reduce the space available for token-specific expert movement. MoE optimization methods for edge inference can be broadly divided into heuristic-based and predictive approaches, as summarized in Table I. Heuristic methods reduce memory pressure through caching, reuse, or architectural modifications, without explicitly predicting future expert usage. Pre-gated MoE shifts routing earlier by having layer L select experts for layer L+1L+1, enabling partial overlap through buffering, but cannot preserve the correctness of the original routing decisions [19]. AdapMoE combines cross-layer prefetching and cache allocation, but changes the number of active experts and is orthogonal to fixed top-k routing [56]. MoE-SpAc uses speculative decoding to anticipate expert demand, but it operates at the sequence level rather than modeling per-layer routing decisions [28]. Predictive approaches forecast future expert usage and prefetch weights ahead of execution to hide data movement latency. Fate [9] leverages cross-layer routing similarity to predict expert usage in subsequent layers, while HOBBIT [52] extends this idea with mixed-precision expert loading, combining prediction with hardware-aware optimizations, but without strict correctness guarantees. Recent pre-attention expert prediction work [57] uses same-layer pre-attention activations to predict expert selections earlier in the layer, improving prediction accuracy. While such methods improve the predictor, they still rely on a fixed predicted set unlike APEX, which focuses on adaptive resource management. ProMoE, the closest state-of-the-art work, uses a learned predictor based on intermediate activations to anticipate routing decisions and prefetch experts [49]. However, it employs a static top-k prefetch policy that does not adapt the prefetch budget, which we evaluate as a primary baseline in Section V. Despite these advances, existing prediction-based methods prefetch only the predicted top-k experts. As shown in Table I, none of them adapts the prefetch budget using prediction confidence. In contrast, APEX uses an adaptive top-(k+δ^(x))(k+ δ(x)) prefetch strategy that dynamically determines the minimal additional budget needed to meet a target coverage. By using a learned confidence model, APEX improves coverage reliability while maintaining energy efficiency and enabling both correctness-preserving and stall-free execution modes. I Background and Motivation This section reviews the execution characteristics of MoE layers in edge settings, highlights the resulting memory bottlenecks, and discusses the limitations of static prefetching. I-A MoE Inference at the Edge Modern LLMs are built on the Transformer architecture, where each layer consists of a self-attention block and a feed-forward network (FFN) [53]. The attention mechanism captures token-to-token interactions for contextual reasoning, while the FFN provides most of the model capacity. In MoE models, the dense FFN is replaced by a set of N experts, each implemented as a smaller FFN and a router, as shown in Figure 2(a). For each token, the router computes a probability distribution over experts and selects the top-k experts [48]. These experts process the token independently, and their outputs are combined using a weighted sum. Training objectives encourage balanced expert utilization [26]. Thus, routing decisions vary across tokens and layers, making expert selection prediction challenging at inference. A key property of MoE inference is the sparse and irregular expert activation, which varies across tokens and layers. As a result, most experts remain idle for most of the execution while still occupying memory. In edge deployments, keeping all experts in high-bandwidth memory is impractical due to capacity, cost, and power constraints. Instead, expert weights are stored in off-package memory, while non-expert weights, activations, and the KV cache remain on-package. This creates a hierarchical memory system where expert weights must be fetched over a slower off-package path, such as PCIe. During inference, the router selects the top-k experts, triggering on-demand loading of their weights before computation can proceed. This introduces a serialization between routing and expert execution, placing expert loading on the critical path. Due to the size and irregular access of expert parameters, this data movement dominates latency and energy while leaving compute resources idle during the transfer. Fig. 2: Comparison of (a) standard MoE block; and (b) our proposed architecture with an additional prefetch router. Attention computation provides a natural opportunity to overlap expert loading with useful work. If expert usage can be predicted before attention, the corresponding weights can be prefetched while attention is executing, partially or fully hiding expert-loading latency. However, imperfect predictions may miss required experts, reintroducing stalls and limiting the effectiveness of naive top-k prefetching, as discussed next. I-B Why Fixed Top-k Prefetch is Insufficient? While predictive prefetching overlaps expert transfers with attention computation, existing approaches prefetch only the predicted top-k experts. We show that this strategy is insufficient to reliably eliminate stalls in MoE inference. Challenge 1: High overlap accuracy does not imply stall-free execution: A key limitation of top-k prefetching is its sensitivity to even small prediction errors. If any expert selected by the original router is missing from the prefetched set, it must be fetched on demand, exposing correction latency on the critical path. Hence, average overlap accuracy across all layers can be misleading. For instance, predicting k−1k-1 out of k experts yields an overlap of k−1k k-1k (e.g., 75% for k=4k=4 and 87.5% for k=8k=8). Despite these seemingly high values, tokens at every layer still incur a correction stall, revealing a fundamental limitation. Therefore, we consider per-layer, per-token overlap and complement it with application-level accuracy. Challenge 2: Token-level heterogeneity: Routing uncertainty varies significantly across tokens and layers. Tokens corresponding to common patterns (e.g., frequent words or simple continuations) exhibit predictable routing, while more complex or context-dependent tokens are harder to predict. A fixed prefetch policy cannot adapt to these variations. A small fixed δ leads to frequent misses on difficult tokens, while a large δ wastes communication energy on easy ones. Therefore, determining how many additional experts to prefetch is a non-trivial, per-token decision. Challenge 3: Runtime decision must be lightweight: Determining the prefetch budget must (1) be early enough to maximize overlap with attention and (2) introduce minimal area, power, and performance overhead. These constraints limit the available decision window and rule out complex or compute-intensive policies. Key Insight: Small additional prefetch enables near-complete coverage. Eliminating stalls requires full coverage, i.e., ensuring all experts selected by the original router are present in the prefetched set. Our measurements show that most token-layer instances require only a small number of additional experts beyond top-k to achieve this. Figure 3 illustrates this behavior for the Granite-3.1-1B-A400M model. While top-k prefetching results in variable and suboptimal overlap across layers, extending the prefetched set to top-(k+δ)(k+δ) rapidly improves coverage, approaching near-oracle levels with small δ. Even modest increases (e.g., δ=2δ=2 or δ=4δ=4) significantly reduce misses, indicating that most prediction errors are localized and can be corrected with minimal additional prefetches. Fig. 3: Per-layer overlap accuracy for Granite-3.1-1B-A400M (k=8k=8). Increasing the prefetch budget (δ) stabilizes coverage near 100%, effectively mitigating the routing stochasticity and misprediction stalls. These observations motivate moving beyond fixed top-k prefetching toward an adaptive strategy that determines the prefetch budget per token and per layer. The goal is to select the minimal additional budget needed to achieve high coverage without incurring unnecessary communication overhead. APEX achieves this by selecting a token-dependent extra-prefetch budget δ^(x) δ(x) that balances coverage and efficiency. Fig. 4: Overview of the MoE memory bottleneck and APEX. (a) Conventional execution stalls while loading experts from off-package memory. (b) APEX overlaps expert loading with attention using a prefetch router that dynamically selects the extra-prefetch budget δ^(x) δ(x) based on token-level confidence. IV Adaptive Prefetching: Top-(k+δ^(x))(k+ δ(x)) Expert Prefetching with Logistic CDF IV-A Overview of Adaptive Expert Prefetching Our goal is to reduce exposed expert-loading latency by making expert movement a predictive, token-aware runtime decision. To achieve this, we augment each MoE layer with a lightweight prefetch router placed before the attention block. The original router selects the exact top-k experts after attention. In contrast, the proposed prefetch router runs before attention to predict a ranked list of candidate experts and initiate data movement while attention is still executing. This transforms the conventional route → load → execute pipeline into predict → prefetch → execute, as shown in Figure 4. Importantly, APEX does not modify the original routing semantics. The prefetch router acts as an auxiliary prediction path that exposes early information for scheduling data movement, while the original router continues to determine the final expert set used for computation. This clean separation decouples expert prediction for memory scheduling from expert selection for model execution. The key question is how many experts to prefetch as discussed in Section I-B. APEX prefetches the top-(k+δ^(x))(k+ δ(x)) experts, where the additional prefetch budget δ^(x) δ(x) is determined dynamically for each token. As shown in Figure 4(b), δ^(x) δ(x) depends on token-level confidence. Easy, high-confidence tokens require only a small prefetch set, while uncertain tokens prefetch a few additional experts to improve coverage. APEX operates in two stages. 1. Prefetch stage: The prefetch router produces a ranked expert list, and a confidence model selects δ^(x) δ(x). The runtime then issues asynchronous direct memory access (DMA) transfers for the top-(k+δ^(x))(k+ δ(x)) experts while attention executes. 2. Execution stage: After attention, the original router determines the exact top-k experts for computation. If all routed experts are already loaded to on-package memory, execution proceeds immediately. Otherwise, APEX uses one of two modes: correctness-preserving mode fetches the missing routed experts while computing the available ones, whereas stall-free mode avoids the correction stall by executing with the available routed experts and high-scoring prefetched substitutes. IV-B Problem Formulation and Learning Objective Fig. 5: Illustration of expert prefetch coverage. The prefetch router produces a ranked list of N=8N=8 experts, while the original router selects k=4k=4 experts (rK_r). Prefetching only the top-k experts may miss some required experts. Extending the prefetched set to top-(k+δ)(k+δ) achieves full coverage. Let ℰ=e1,e2,…,eNE=\e_1,e_2,…,e_N\ denote the ordered list of experts produced by the prefetch router, sorted from the highest (most likely to be used) to the lowest predicted score, where N is the total number of experts. Let r⊆ℰK_r , with |r|=k|K_r|=k, denote the set of experts selected by the original router. Figure 5(a) illustrates an example where the prefetch router outputs a ranked list of 8 experts, and the original router selects 4 experts (k=4k=4). For any extra-prefetch budget δ∈0,1,…,N−kδ∈\0,1,…,N-k\, the prefetched expert set is: p(δ)=e1,e2,…,ek+δK_p^(δ)=\e_1,e_2,…,e_k+δ\ (1) A prefetch decision is successful if all required experts are already present in the prefetched set: r⊆p(δ)⇔r∖p(δ)=∅K_r _p^(δ) _r _p^(δ)= (2) As shown in Figure 5(b), prefetching only the top-k experts (i.e., δ=0δ=0) can miss required experts, even when most predictions are correct. Extending the prefetched set with a small additional budget δ can eliminate these misses and achieve full coverage. We define the oracle extra-prefetch budget δ∗δ^* as the minimum δ that guarantees full coverage: δ∗=minδ∈0,1,…,N−k|r∖p(δ)=∅δ^*= \δ∈\0,1,…,N-k\\; |\;K_r _p^(δ)= \ (3) Figure 6(a) shows a representative distribution of δ∗δ^* obtained by profiling the Granite-1B model (Layer 3) on the WikiText [32] test set. Most tokens require little or no additional prefetch, while a small fraction require larger δ∗δ^* due to higher routing uncertainty. Fig. 6: Adaptive expert prefetching. (a) Representative distribution of oracle extra-prefetch budget (δ∗δ^*) for Granite-1B (Layer 3) on WikiText [32], showing a long-tail across tokens. (b) Selection of prefetch budget δ^(x) δ(x) using a learned logistic CDF to satisfy a target coverage threshold (τ). Learning Objective: Our goal is to select a token-dependent prefetch budget δ^(x) δ(x), where x denotes the token representation available before attention, that minimizes unnecessary data movement while ensuring high coverage probability. Formally, we require that the prefetched set covers the original router’s selection with probability at least τ: Pr(r∖p(δ^(x))=∅∣x)≥τ, (K_r _p^( δ(x))= x )≥τ, (4) where τ∈[0,1]τ∈[0,1] is a user-defined coverage target. Since full coverage is achieved if and only if δ^(x)≥δ∗ δ(x)≥δ^*, the objective can be equivalently written as: Pr(δ^(x)≥δ∗∣x)≥τ ( δ(x)≥δ^* x )≥τ (5) This formulation highlights the core objective: for each token, select the smallest token-dependent δ^(x) δ(x) that satisfies the coverage constraint, thereby balancing latency reduction against communication cost. IV-C Confidence Modeling via CDF The adaptive prefetching problem reduces to selecting the smallest extra-prefetch budget that satisfies a target coverage probability. To enable this decision at runtime, we use a lightweight probabilistic model that estimates whether a given number of additional experts is sufficient. Coverage Probability: We define the probability that a candidate budget δ is sufficient as: pδ(x)=Pr(δ≥δ∗∣x),δ∈0,…,N−kp_δ(x)= (δ≥δ^* x), δ∈\0,…,N-k\ (6) Since larger prefetch budgets can only improve coverage, these probabilities are monotonic: p0(x)≤p1(x)≤⋯≤pN−k(x),p_0(x)≤ p_1(x)≤·s≤ p_N-k(x), (7) which can be interpreted as the cumulative distribution function (CDF) of the oracle δ∗δ^*. Figure 6(b) provides intuition for this formulation. The learned CDF enables APEX to select the smallest δ that satisfies a target threshold τ. Ordinal Logistic CDF Model: We model this CDF using an ordinal logistic formulation that captures the ordered structure of δ. The cumulative probability is modeled by: pδ(x)=σ(θδ−w⊤x),δ∈0,…,N−k,p_δ(x)=σ( _δ-w x), δ∈\0,…,N-k\, (8) where w is a learned parameter vector, θδ\ _δ\ are ordered bias terms satisfying θ0≤θ1≤⋯≤θN−k _0≤ _1≤·s≤ _N-k, and σ(⋅)σ(·) is the sigmoid function. The ordered biases partition the latent confidence space, with each threshold corresponding to a different prefetch budget. By construction, this formulation guarantees monotonicity of pδ(x)p_δ(x) and therefore defines a valid CDF. TABLE I: Summary of notations used in this work. Notation Description ℰ=e1,e2,…,eNE=\e_1,e_2,…,e_N\ Ordered list of experts sorted by prefetch router N Total number of experts k Number of experts to be executed δ∈0,1,…,N−kδ∈\0,1,…,N-k\ Extra-prefetch budget rK_r Expert set selected by the original router p(δ)K_p^(δ) Prefetched expert set e1,…,ek+δ\e_1,…,e_k+δ\ δ∗δ^* Oracle delta: minimum δ such that r⊆p(δ)K_r _p^(δ) δ^(x) δ(x) Token-dependent (dynamic) prefetch budget τ∈[0,1]τ∈[0,1] Target coverage probability threshold pδ(x)p_δ(x) Coverage probability: pδ(x)=Pr(δ≥δ∗∣x)p_δ(x)= (δ≥δ^* x) w Learned parameter vector in CDF model θ0≤θ1≤⋯≤θN−k _0≤ _1≤·s≤ _N-k Ordered thresholds (cutpoints) for ordinal model At runtime, given a token representation x and a target coverage τ, we select the smallest prefetch budget that satisfies: δ^(x)=minδ∈0,…,N−k|pδ(x)≥τ. δ(x)= \δ∈\0,…,N-k\\; |\;p_δ(x)≥τ \. (9) This yields a confidence-aware adaptive policy: • easy tokens (tokens whose routed experts are likely covered by a small budget) require a small δ^(x) δ(x); • difficult tokens with low confidence require a larger δ^(x) δ(x). Thus, unlike static overfetching, APEX fetches just enough additional experts to satisfy the desired coverage target, reducing misses while avoiding unnecessary communication. IV-D Prefetch Router: Training and Integration The key design goal is to enable accurate early prediction for prefetching without modifying or retraining the base MoE model. Accordingly, the LLM remains frozen, and training is limited to the auxiliary prefetching components, i.e., the prefetch router and the CDF model. Prefetch Router Architecture and Placement: The prefetch router mirrors the original MoE router: a linear layer that produces expert logits, followed by a softmax to produce a probability distribution over experts. It is placed before the attention block of each MoE layer and operates on the layer’s hidden representation. This placement serves two purposes. First, it hides expert loading latency by initiating expert transfers during attention. Second, it remains within the same layer, preserving a strong correlation between the prefetch prediction and the final routing decision made later by the original router. Prefetch Router Distillation: For each MoE layer, let qrq_r and qpq_p denote the expert softmax distributions produced by the original router and the prefetch router, respectively. The prefetch router is trained to match the original router using a KL divergence loss: ℒKL=∑iqr(i)logqr(i)qp(i).L_KL= _iq_r(i) q_r(i)q_p(i). (10) Training requires only forward passes through the frozen base model to obtain qrq_r, and gradients are applied exclusively to the prefetch router. No changes are made to expert execution during this stage. The result is a layer-wise early predictor that approximates the original router’s expert selections. CDF Model Training: After training the prefetch router, we fit the CDF model described in Section IV-C through a separate lightweight stage. For each token and layer, we use the trained prefetch router to obtain expert probabilities, derive the ranked expert list, and compute the oracle budget δ∗δ^* by comparing with the original router. These are converted into cumulative binary targets [δ≥δ∗]I[δ≥δ^*] for each δ. The CDF model is trained using a cumulative binary cross-entropy loss: ℒCDF=∑δ=0N−kℋ(pδ(x),[δ≥δ∗]),L_CDF= _δ=0^N-kH (p_δ(x),~I[δ≥δ^*] ), (11) where ℋ(⋅,⋅)H(·,·) denotes binary cross-entropy [14]. The model consists of a single weight vector and a small set of thresholds, making training inexpensive and feasible with a modest held-out validation set. Since the prefetch router is distilled from the frozen original router, it is tied to the base model’s routing function rather than to a specific prompt or downstream task. Thus, the same trained prefetch router and CDF model can be reused across prompts and datasets; retraining is only needed if the base MoE model is fine-tuned and its router behavior changes. Runtime Operation: The prefetch router and CDF model are evaluated before attention to determine the adaptive prefetch budget δ^(x) δ(x). The system then asynchronously prefetches the top-(k+δ^(x))(k+ δ(x)) experts while attention executes. After attention, the original router produces the final top-k selection, and execution proceeds using either correction or stall-free mode, as described next. The overhead is minimal, as detailed in Section V-F. The prefetch router mirrors the lightweight original router, and the CDF model introduces only a small number of parameters, enabling early prediction and scheduling with negligible compute and storage overhead. IV-E Correctness-preserving and Stall-free Execution Modes Even with adaptive top-(k+δ^(x))(k+ δ(x)) prefetching, rare mispredictions may occur. Hence, we support two execution modes. 1) Correctness-preserving mode: The system strictly follows the original router decision. If any experts in rK_r are missing from the prefetched set, they are fetched while execution begins on available experts, and the final output uses the complete original expert set. This preserves exact MoE semantics, but incurs a small residual correction latency, which is evaluated in Section V-E. Asynchronous miss correction: APEX does not use a rigid wait-then-execute pipeline when a prefetch miss occurs. Since the selected experts in an MoE layer are independent before the final weighted sum, correctness-preserving execution begins immediately on the routed experts that are already available in the on-package buffer. Missing routed experts are fetched asynchronously in parallel with this available expert computation. After the missing experts arrive, APEX computes only the remaining expert outputs and then performs the final weighted aggregation using the original router weights. Therefore, the correction overhead is the unhidden portion of the miss transfer after overlap with available expert computation, rather than the full load latency. This preserves exact MoE semantics while exploiting inter-expert pipelining. 2) Stall-free mode: This mode eliminates correction latency by executing only with experts that are already present in the prefetched set. Instead of waiting to fetch missing experts, we approximate the original routing operation using the available experts. If all experts selected by the original router are already prefetched (Algorithm 1, lines 3–4), we directly execute them without any deviation from the behavior of the original architecture. Otherwise, if one or more routed experts are missing, stall-free mode keeps the correctly prefetched routed experts and replaces the missing experts with the same number of highest-weight candidates from the remaining prefetched set, ranked by the original router’s softmax weights (lines 7–11). Intuitively, this procedure selects the highest-scoring available substitutes from the prefetched set according to the original routing scores, minimizing deviation from the original decision. Because overlap is already very high, such substitutions are rare. Moreover, the missing expert are often a lower-ranked choice, so the resulting approximation can have small model-dependent accuracy impact, as evaluated in Section V-D. We note that, the stall-free mode is an optional mode, while correctness-preserving mode remains the default when exact routing semantics are required. Algorithm 1 Expert Selection in the Stall-free Mode 1: Input: Routed experts rK_r, original router logits qr∈ℝNq_r ^N, prefetched experts p(δ^)K_p^( δ) 2: Output: Execution experts ~ K 3: α←softmax(qr)α (q_r) // original-router expert weights 4: if r⊆p(δ^)K_r ^( δ)_p then 5: ~←r K _r // all correct experts are available 6: else 7: ~←r∩p(δ^) K _r ^( δ)_p // keep correctly routed experts 8: ℳ←r∖p(δ^)M _r ^( δ)_p // missing routed experts 9: ←(δ^)∖~C ^( δ) K // available replacement candidates 10: ←Top∗|ℳ|(,α)S *|M|(C;α) // highest-weight candidates 11: ~←~∪ K← K 12: end if 13: return ~ K V Experimental Evaluation V-A Experimental Setup Models: We evaluate APEX on four representative MoE LLMs spanning different model scales and configurations: • Granite-1B (IBM Granite-3.1-1B-A400M [20]): a 1B-scale model with N=32N=32 experts and top-k routing with k=8k=8. • Granite-3B (IBM Granite-3.1-3B-A800M [21]): a 3B-scale model with N=40N=40 experts and top-k routing with k=8k=8. • Phi-7B (Microsoft Phi-mini-MoE-7B-A2.4B [1]): a 7B model with N=16N=16 experts and top-k routing with k=2k=2. • DeepSeek-16B (DeepSeek-V2-Lite-16B-A2.4B) [30]: a 16B model with N=64N=64 experts and top-k routing with k=6k=6. These models capture varying numbers of experts, routing sparsity, and layer depths. Granite-1B, Granite-3B, Phi-7B, and DeepSeek-16B contain 24, 32, 32, and 27 Transformer layers, respectively. In DeepSeek-16B, the first Transformer layer uses a dense FFN, while the remaining 26 layers use MoE FFNs. DeepSeek also includes two shared experts per MoE layer that are activated for every token in addition to the six routed experts selected from the 64 routed experts. Thus, for DeepSeek-16B, shared experts are deterministic per-token computation and are not part of APEX’s adaptive routing or extra-prefetch-budget decision. Workload: We focus on the decode phase (autoregressive token generation), since it is latency-critical in edge deployments and exposes expert-loading bottlenecks. We evaluate across diverse context lengths, including 512, 1024, and 2048 tokens. For application-level accuracy, we use standard benchmarks including WikiText [32] (perplexity), AI2 Reasoning Challenge [5], Massive Multitask Language Understanding [18], WinoGrande [45], and TruthfulQA [29]. Prefetch Router and CDF Training: Training is performed on an NVIDIA RTX 3090 GPU [35] for Granite and Phi models and A100 [36] for DeepSeek using the WikiText dataset [32]. We use a learning rate of 5×10−45× 10^-4, batch size of 8, sequence length of 1024, and train for 1000 steps. The training takes approximately 10 minutes for Granite-1B, 30 minutes for Granite-3B, and about 1 hour for Phi-7B and DeepSeek-16B. After distillation, the CDF model is fitted using collected oracle δ∗δ^* values on a held-out validation set. No task-specific retraining or hyperparameter search is used for downstream benchmarks; the auxiliary components trained on WikiText are reused for all evaluated application-level tasks. Baselines: We compare APEX against three baselines: (i) No Prefetch, the conventional execution scheme in which expert loading occurs on the critical path [8]; (i) ProMoE, a state-of-the-art prediction-based top-k prefetching approach [49]; and (i) static overfetch policies with fixed additional-prefetch budgets. We evaluate both correctness-preserving and stall-free modes. To decouple the effect of expert prediction from adaptive budget selection, the static overfetch baselines use the same prefetch-router ranking as APEX and replace the learned (δ^(x) δ(x)) with a fixed (δ) for every token and layer. Therefore, differences between static overfetching and APEX isolate the benefit of adaptive per-token budgeting. The evaluation metrics are (i) overlap accuracy between prefetched and routed experts, (i) application-level accuracy across benchmarks, and (i) system efficiency, in terms of latency, energy, and energy-delay-product (EDP). V-B Hardware Platform and Evaluation Methodology Target Edge Architecture: We model a modern edge-class accelerator consisting of a compute chiplet tightly coupled with I/O and memory chiplets, following the prior approach [46]. The compute chiplet integrates multiple vector processing arrays and SRAM buffers [24], while all expert weights reside in off-package LPDDR5X memory. The compute microarchitecture is based on the vector processing array illustrated in Figure 7. Each array is organized as a 16×1616× 16 grid of vector units with vector length 32, supporting bfloat16 (BF16) matrix-multiply operations. We instantiate four such arrays operating at 750 MHz, achieving a peak throughput of 24 TFLOPS, as summarized in Table I. Each array is provisioned with 2 MB of SRAM (8 MB total) and includes 32 special-function units [44] to support non-linear activations. We use BF16 as the default evaluation datatype to preserve the original model behavior and isolate the impact of prefetching; Section V-G further studies how APEX behaves when expert weights are represented with lower-precision datatypes. Fig. 7: Microarchitecture of a vector processing array. Fig. 8: Per-layer overlap accuracy using a CDF confidence threshold of τ=0.90τ=0.90 for APEX. (a) Granite-1B model. The static baselines prefetch top-(k+2k+2) and top-(k+4k+4), while APEX prefetches on average 4.17. (b) Granite-3B model (keeping the same static-baseline settings). (c) Phi-7B model. The static baselines prefetch top-(k+1k+1) and top-(k+2k+2), while APEX prefetches on average 0.67. (d) DeepSeek-16B model. The static baselines prefetch top-(k+2k+2) and top-(k+4k+4), while APEX prefetches on average 1.98. To reflect realistic edge constraints, on-package memory capacity is varied across model scales (1 GB, 2 GB, and 6 GB), holding only non-expert weights, activations, and KV cache, following the literature [25, 2]. All expert parameters are stored in off-package LPDDR and accessed over a PCIe interconnect. On-package memory bandwidth is modeled at 819 GB/s (HBM3-class), while off-package bidirectional bandwidth is set to 256 GB/s (PCIe 6.0 x16) [40, 39]. The evaluations are performed on a latency-sensitive single-user edge inference setting, but the proposed method supports batching multiple concurrent requests. Platform Positioning: The target configuration represents a high-end, near-future edge AI accelerator rather than a low-power smartphone-class SoC. This class is motivated by emerging robotics, industrial, and local GenAI platforms that already integrate high compute throughput with large local memory capacity, such as NVIDIA Jetson AGX Orin/Thor [37], and Hailo-10H [15] class edge accelerators. At the same time, APEX is not tied to this specific bandwidth point. Section V-G evaluates sensitivity to lower and higher off-chip bandwidths, including a 32 GB/s setting representative of more constrained mobile-SoC-class memory systems. Evaluation Methodology: Our evaluation combines hardware synthesis with cycle-accurate co-simulation rather than pure simulation. The compute arrays are implemented in RTL, synthesized in TSMC 28 nm, and their active/leakage power is characterized using Synopsys PrimeTime [51]. Memory timing is modeled using Ramulator, which captures DRAM bank conflicts, refresh, and controller scheduling [31], while LPDDR and PCIe energy costs are incorporated using established per-bit energy models [13, 40]. The arrays are clock-gated, allowing us to accurately model idle periods during stalls. We obtain active power by running the arrays under high activity, and leakage power by evaluating the design under no-activity conditions with clock gating enabled. On-package communication is modeled using UCIe-based interconnect parameters that capture low-energy, high-bandwidth chiplet communication [47]. We use an open-source, cycle-accurate compute–communication co-simulation framework, CHIPSIM [41], which integrates: (i) compute timing from RTL-derived models, (i) memory access timing from DRAM simulation [31], and (i) per-layer execution traces collected from GPU-based model runs [35]. The simulator models token-level execution, including attention compute, expert routing, data transfers, and expert execution, enabling us to model overlap among attention computation, expert routing, DMA transfers, and expert execution. The asynchronous expert transfers are not modeled as ideal parallel fetches. Each prefetched expert is injected as a DMA transfer into the shared PCIe/LPDDR path, where concurrent requests experience bandwidth limits and queuing delay. CHIPSIM coordinates compute, routing, DMA transfers, and expert execution on a unified timeline. Thus, overlap is limited by the modeled attention window and contention in the off-chip memory path. TABLE I: Hardware configuration and power parameters Component Configuration Perf./Energy Compute Arrays 4×(16×16×32)4×(16× 16× 32) 24 TFLOPS @ 750 MHz SRAM [3] 8 MB (2 MB per array) CACTI-based modeling HBM Bandwidth [39] 819 GB/s 7 pJ/bit LPDDR5X [13] Off-package experts 3 pJ/bit PCIe 6.0 x16 [40] 256 GB/s 5 pJ/bit On-chip Comm. [47] UCIe Adv. 0.5 pJ/bit Prefetch-overlap Bound: APEX does not assume that all prefetched experts are fully hidden behind attention. For an extra-prefetch budget (δ), the ideal hiding condition is: Tprefetch(δ)=δSexpertBeff+Tqueue≤Tattn,T_prefetch(δ)= δ S_expertB_eff+T_queue≤ T_attn, (12) where (SexpertS_expert) is the expert size, (BeffB_eff) is the effective off-chip bandwidth, and (TqueueT_queue) captures memory-controller/interconnect contention. When this condition is not satisfied, the unhidden portion of (TprefetchT_prefetch) appears as exposed stall time in the simulation. Thus, APEX improves latency and energy only when the saved stall/idle cost exceeds the additional transfer cost. This trade-off is evaluated directly through CHIPSIM and further stressed in the bandwidth sensitivity study in Section V-G. Methodology Scope and Limitations: CHIPSIM has been validated against hardware measurements on a real chiplet platform under concurrent workloads. However, the target APEX accelerator itself is evaluated through synthesis-calibrated co-simulation rather than measurements on a hardware prototype. We model PCIe/LPDDR contention through the memory/interconnect timing model and account for OS-level DMA setup and Input-Output Memory Management Unit (IOMMU) overhead using a fixed per-DMA-descriptor cost, which is amortized over multi-MB expert transfers. Building an FPGA or silicon prototype to further validate these system-level effects and full end-to-end OS-stack modeling are left as future work. V-C Expert Prediction Overlap Accuracy This section analyzes the overlap accuracy between the experts selected by the original router and the prefetched top-(k+δ^(x))(k+ δ(x)) set, since a high overlap is the key enabler for hiding expert-loading latency. Then, the next subsection analyzes the impact of the rare routing mismatches in the stall-free mode on the end-to-end application-level accuracy. Per-layer Overlap Across Models: We first evaluate how the overlap accuracy changes across layers and model scales to ensure that APEX consistently offers high coverage throughout the network. In this experiment, the confidence threshold is set as τ=0.90τ=0.90 to ensure a high and consistent overlap target across all layers. The effect of varying this threshold and its impact on overlap accuracy is analyzed later. Figure 8 plots the per-layer overlaps between the original router’s selections and the prefetched experts for Granite-1B, Granite-3B, Phi-7B, and DeepSeek-16B. Across all models, APEX consistently maintains over 97%97\% overlap across layers, greatly reducing the variability seen in alternative methods. ProMoE exhibits noticeable degradation in layers 5 and 8 of Granite-1B, with ∼ 79% and ∼ 84% overlap, respectively. Similarly, it struggles with multiple layers in the Granite-3B, Phi-7B and DeepSeek-16B models, where ProMoE often stays in the high-80% range and drops near 82%. In contrast, APEX consistently achieves around 97–98% overlap. This stability is essential to guarantee that no individual layer acts as a bottleneck, effectively removing layer-wise performance differences caused by mispredictions. Figure 8 also shows that APEX significantly outperforms static policies that always prefetch 2 and 4 extra experts. Fixed policies must account for worst-case layers, which causes unnecessary overfetching in simpler layers. Even with this over-provisioning, they struggle to prefetch the correct set of experts for many layers (e.g., 5th^th layer of Granite-1B and first layer of Phi-7B), which likely exhibit higher routing ambiguity and weaker correlation between the pre-attention representation and the final routing decision. In contrast to static policies, APEX adapts its prefetch budget to runtime uncertainty, eliminating the low-overlap tail while avoiding excessive overfetch. As a result, APEX maintains over 97%97\% overlap across layers by prefetching 4.17 (Granite-1B), 2.86 (Granite-3B), 0.67 (Phi-7B) 1.98 (DeepSeek-16B) experts on average. These results show that APEX enhances not just average overlap but also worst-case reliability, which is essential for latency-sensitive inference. Fig. 9: Average overlap accuracy vs. extra-prefetch budget (δ δ). Increasing the CDF confidence threshold (τ) from 0.60 to 0.97 consistently raises overlap toward 100% across all models, validating that the CDF-based selection effectively meets the desired coverage targets. Overlap vs. Adaptive Target Threshold: Next, we evaluate how the target confidence threshold τ affects the overlap accuracy and prefetch budget. Figure 9 shows that increasing τ monotonically increases the overlap percentage across all models, as expected. For Granite-1B, the overlap improves from 93.4% with τ=0.60τ=0.60 to 98.2% with τ=0.90τ=0.90, and further to 99.4% with τ=0.97τ=0.97, while the average δ^(x) δ(x) increases from 1.36 to 4.17 and 6.70, respectively. Granite-3B shows a similar trend, with overlap increasing from 93.5% to 98.1% and 99.4%, while the average number of prefetched experts increases from δ^(x) δ(x) of 0.67 to 2.86 and 4.93. Similarly, for Phi-7B, the average overlap percentage increases from 94.7% to 97.5%, and then 99.0%, while prefetching only 0.16, 0.67, and 1.75 additional experts on average. Finally, DeepSeek-16B also follows the same pattern, with overlap increasing from 92.9% to 97.4% and 99.0%, while the average prefetched experts increases from 0.37 to 1.98 and 3.66. These results highlight two key insights. First, the CDF-based selector is well-designed: higher τ consistently yields higher overlap, confirming that APEX can reliably meet the coverage targets. Second, near-complete overlap (>>99%) can be achieved with a modest increase in prefetch budget, indicating that APEX efficiently allocates additional experts only when necessary rather than uniformly overfetching. We use τ=0.90τ=0.90 in the remaining experiments because it captures most of the overlap gains before saturation. Further increase causes additional prefetching with only marginal benefits. Adaptive Behavior Compared to Oracle: Next, we analyze how effectively APEX adapts its prefetch budget to the intrinsic difficulty of each layer by comparing it to an Oracle that stores the ground truth. Figure 10 plots the number of required experts (given by the Oracle) and prefetched by APEX with τ=0.90τ=0.90 for the representative Granite-3B model. We observe that APEX closely follows the ground truth. APEX prefetches three or fewer experts for most layers, which require one or two additional experts according to the Oracle. That is, it learns the layers for which experts can be predicted accurately and assigns small budgets to them. In more challenging layers that require three or more additional experts, APEX slightly overestimates the oracle budget, which is desirable to provide a safety margin to maintain high overlap. We observe the same adaptive behavior across other benchmark models. Overall, these results confirm that APEX does not use a coarse global heuristic but dynamically adjusts its prefetch budget in response to layer- and token-level runtime variability, enabling it to achieve consistently high overlap accuracy while remaining efficient. Hence, APEX achieves its primary goal of transforming a highly variable expert prediction problem into a controlled, confidence-driven prefetch policy, which consistently delivers very high overlap accuracy while avoiding the inefficiencies of static overfetching. Fig. 10: Per-layer additional experts selected by APEX (δ^(x) δ(x)) versus oracle extra-prefetch budget (δ∗δ^*) for Granite-3B (τ=0.90τ=0.90). Layers are ordered by oracle δ∗δ^* to highlight relative difficulty. Per-token Prefetch Budget Distribution: Beyond average (δ^(x) δ(x)), we also examine the per-token distribution of the total prefetched experts (k+δ^(x)k+ δ(x)), since large bursts could stress the PCIe path or on-package buffers. Figure 11 plots the token distribution for Granite-3B at (τ=0.90τ=0.90) across representative easy, typical, and hardest layers. For easy and typical layers, the distribution is concentrated near (k), with mean (δ δ) below 3 and fewer than 0.2% of tokens prefetching more than (N/2) experts. The hardest layer requires a larger budget because the pre-attention representation is less correlated with the final router decision. However, APEX prefetches experts layer by layer; each layer’s experts are loaded, used, and evicted before the next layer begins, so peak buffer occupancy is bounded by the maximum single-layer prefetch set rather than accumulating across layers. Thus, the adaptive budget does not create an unbounded memory-pressure. Fig. 11: Distribution of total prefetched experts per token (k+δ^(x)k+ δ(x)) for Granite-3B at τ=0.90τ=0.90. Fig. 12: Comparison of (a) average per-token latency and (b) per-token energy consumption across context lengths for Granite-3B model. V-D Application-Level Accuracy This section evaluates the impact of the APEX on the application-level accuracy measured by perplexity and standardized benchmarks AI2 Reasoning Challenge (ARC), Massive Multitask Language Understanding (MMLU), WinoGrande (WG), and TruthfulQA (TQA). For clarity, we report the perplexity and the average of these standardized scores. These accuracy metrics are compared against the base system, which always uses the original router’s expert choices. The correctness-preserving mode always matches the accuracy of the baseline by definition. Therefore, Table IV reports perplexity and downstream task performance across four model scales under different confidence thresholds τ. The impact on the perplexity is negligible for the Granite-1B, 3B and DeepSeek-16B models, especially at higher thresholds. Specifically, for Granite-1B moving from correctness-preserving execution to stall-free at τ=0.90τ=0.90 results in only a marginal increase in perplexity (7.88 → 7.99) and a small drop in average downstream accuracy (43.3 → 42.8). A similar trend is observed in Granite-3B, where the average score decreases modestly from 54.0 to 53.2, with perplexity increasing slightly from 6.79 to 6.83. DeepSeek-16B follows the same trend, with only a small PPL increase at τ=0.90τ=0.90 (7.02 → 7.10) and a modest average-score drop ( 51.6 → 50.8). These results indicate that in high-overlap regimes, the effect of occasional expert mispredictions is negligible for Granite models. The robustness in accuracy can be attributed to the routing structure of Granite and DeepSeek models, where each token is processed by multiple experts (k=8k=8 for Granite and k=6k=6 plus two shared experts for DeepSeek) and the final output is a weighted average. In this setting, mispredictions are more likely to occur among lower-ranked experts that contribute less to the final output, thereby limiting their impact on model accuracy. In contrast, Phi-7B exhibits a more noticeable degradation in stall-free mode, particularly at lower thresholds. At τ=0.60τ=0.60, the average accuracy drops significantly from 64.6 to 58.6, with large reductions in ARC and MMLU scores. Increasing τ improves performance, with the average recovering to 60.4 at τ=0.90τ=0.90, but a gap relative to the baseline remains. The degradation is primarily driven by a subset of benchmarks (notably ARC and MMLU), while others, such as WinoGrande, remain relatively stable. This behavior is explained by the different routing configuration of Phi models, which use only k=2k=2 experts per token. In this case, each expert contributes a larger fraction of the final output, making the model more sensitive to mispredictions. As a result, even occasional overlap misses can have a more pronounced effect on accuracy compared to Granite models. Deployment Guidance: These results show that stall-free execution should be treated as a complementary opt-in mode rather than the default APEX configuration. Correctness-preserving APEX always preserves the original routing semantics and is therefore preferable for accuracy-critical deployments. Stall-free mode is most appropriate when the application can tolerate small accuracy changes in exchange for additional performance gains. Empirically, we observe that models with larger active expert counts, such as Granite-1B/3B with (k=8k=8) and DeepSeek-16B with (k=6k=6) and two shared experts, are more robust because each missed expert contributes a smaller fraction of the final weighted output. In contrast, low-top-k models such as Phi-7B (k=2k=2) are more sensitive to substitutions; therefore, the correctness-preserving mode is the recommended deployment choice, treating stall-free execution for an optional performance improvement. TABLE IV: Application-level accuracy of APEX in stall-free (APEXSF) mode compared to correctness-preserving execution (Base). Model Config PPL↓ ARC/MMLU/WG/TQA ↑ Avg ↑ 1B Base 7.88 40.2 / 28.5 / 60.3 / 44.1 43.3 APEXSF (τ=0.90τ=0.90) 7.99 39.4 / 28.1 / 60.1 / 43.4 42.8 APEXSF (τ=0.75τ=0.75) 8.06 39.3 / 28.0 / 60.0 / 43.5 42.7 APEXSF (τ=0.60τ=0.60) 8.04 39.0 / 28.0 / 60.0 / 43.5 42.6 3B Base 6.79 52.7 / 49.4 / 67.9 / 45.9 54.0 APEXSF (τ=0.90τ=0.90) 6.83 52.6 / 49.0 / 67.4 / 44.0 53.2 APEXSF (τ=0.75τ=0.75) 6.87 52.5 / 48.7 / 67.4 / 43.9 53.1 APEXSF (τ=0.60τ=0.60) 6.91 52.1 / 48.6 / 66.8 / 43.9 52.9 7B Base 6.27 63.4 / 70.6 / 76.0 / 48.4 64.6 APEXSF (τ=0.90τ=0.90) 7.47 57.9 / 62.7 / 74.7 / 46.2 60.4 APEXSF (τ=0.75τ=0.75) 7.49 57.5 / 62.4 / 74.5 / 45.8 60.1 APEXSF (τ=0.60τ=0.60) 7.52 51.8 / 62.6 / 74.6 / 45.6 58.6 16B Base 7.02 52.6 / 48.3 / 70.5 / 34.8 51.6 APEXSF (τ=0.90τ=0.90) 7.10 52.5 / 46.7 / 70.2 / 33.6 50.8 APEXSF (τ=0.75τ=0.75) 7.11 52.1 / 46.3 / 70.2 / 33.5 50.5 APEXSF (τ=0.60τ=0.60) 7.13 52.3 / 45.7 / 66.4 / 31.8 49.0 V-E Performance and Energy Analysis This section evaluates the system-level benefits of APEX in terms of latency, energy, and combined EDP efficiency. Latency Analysis: Figure 12(a) presents the average per-token latency across context lengths for the Granite-3B model as a representative case; other models exhibit similar trends, with combined EDP comparisons provided next. At a context length of 512, APEX correctness-preserving mode achieves 11.41 ms average per-token latency. This is 42% lower than no prefetching (19.77 ms) and 26% lower than ProMoE (15.39 ms). Similarly, APEX reduces the average latency by 20% and 40% relative to static (k+4)(k+4) and static (k+8)(k+8) prefetching techniques. Longer contexts continue to exhibit the same trend. APEX achieves 41% and 39% lower latency than no prefetching at 1024 and 2048 token contexts, outperforming ProMoE by 24% and 20% lower latency. Figure 12 (a) also show that static overfetching performs poorly. When too many experts are statically prefetched, expert loading itself begins to dominate and can exceed the attention window available for hiding it, turning prefetch traffic into the new bottleneck. Finally, we observe that APEX’s stall-free mode provides 2.0%–2.8% lower latency because it does not stall to bring in missing experts. The latency difference is small since the overlap accuracy is high (i.e., stalls happen rarely). Energy Consumption Analysis: Figure 12(b) plots the energy consumption results, which reinforce the superiority of APEX over the baselines. At context length 512, APEX correctness-preserving mode consumes 287.3 mJ energy, which is 9.5% lower than no prefetching and 5.8% lower than ProMoE. We observe that static prefetching leads to the largest energy consumption (9.9% and 21.8% more than APEX) due to over-provisioning. Results with longer context lengths confirm that both modes of APEX consistently achieve the lowest energy consumption. The stall-free mode has again a slightly (≈ 1%) lower energy by avoiding additional (rare) prefetches. This small gap indicates that correctness-preserving mode already removes nearly all of the exploitable stall overhead, consistent with the very high overlap accuracy reported earlier. Notably, the static (k+8)(k+8) is consistently the most energy-consuming policy, as it attempts to avoid misses more aggressively. This highlights an important point: overfetching does not come for free. Excessive expert transfers increase memory and interconnect activity and can outweigh any stall reduction. To further clarify the energy trade-off, Table V reports an energy breakdown for Granite-3B at 1024-token context. APEX does not reduce transfer energy; off-chip I/O energy increases from 48 mJ to 56 mJ due to additional prefetched experts. However, by overlapping these transfers with attention computation, APEX reduces exposed stall time and lowers idle/leakage energy from 57 mJ to 8 mJ. This reduction more than offsets the added I/O energy, reducing total energy from 340 mJ to 299 mJ. In contrast, static (k+8) prefetching overfetches aggressively, increasing I/O energy to 97 mJ and total energy to 377 mJ. APEX, by comparison, remains close to the Oracle behavior by allocating the additional-prefetch budget needed for each token and layer, which is why it achieves the best combined latency and energy efficiency. TABLE V: Energy breakdown for Granite-3B at 1024-token context. Component No Prefetch APEX Static (k+8)(k+8) On-chip active memory 233 mJ 234 mJ 234 mJ Off-chip I/O transfer 48 mJ 56 mJ (+8)(+8) 97 mJ Idle / leakage 57 mJ 8 mJ (−49)(-49) 44 mJ Total energy 340 mJ 299 mJ 377 mJ End-to-End Efficiency (EDP): Figure 13 shows that APEX consistently achieves the lowest normalized EDP. For Granite-1B, APEX correctness-preserving mode lowers the EDP by 36%–42% over no prefetching, and 16%–27% over ProMoE, across all context lengths. The stall-free mode provides an additional of 4%, 3%, and 2% gain over correctness-preserving at 512, 1024, and 2048 tokens, again indicating that residual misses are already rare. Granite-3B follows the same trend, with APEX achieving the lowest EDP, with 22%–30% lower EDP than ProMoE across all context lengths. For both models, the static (k+8)(k+8) consistently has the largest EDP, aligned with the latency and energy analysis, as it attempts to avoid misses more aggressively. Both modes of APEX continue to achieve the best EDP compared to baselines for the Phi-7B model, as shown in the third subplot in Figure 13. Specifically, the correctness-preserving mode achieves 48%–49% lower EDP than no prefetching, and 24%–28% lower EDP than ProMoE. Unlike the Granite models, the Phi-7B model results in a larger difference between the correctness-preserving and stall-free modes of APEX. It lowers the EDP by 14%, 8%, and 7% compared to correctness-preserving for different context lengths. These results are consistent with Phi’s routing structure and accuracy analysis in Section V-C. Each token activates only k=2k=2 experts, and missing even one expert has a larger effect than in Granite models, where each token aggregates across k=8k=8 experts. Thus, eliminating the final residual stalls yields a larger benefit for Phi, at the cost of a larger accuracy degradation. DeepSeek-16B further confirms the same trend at a larger scale as shown in Figure 13 (D). APEX reduces EDP by 30–41% over ProMoE, while stall-free execution adds a further 4–6% improvement over correctness-preserving mode. This smaller stall-free gap compared to Phi-7B is consistent with DeepSeek’s routing structure, where each token uses k=6k=6 routed experts plus two shared experts, making occasional routed-expert substitutions less disruptive. Fig. 13: Normalized EDP all models. For Granite models, k=8k=8, x=4x=4, and y=8y=8; for Phi-7B, k=2k=2, x=1x=1, and y=2y=2 and for DeepSeek-16B, k=6k=6, x=4x=4 and y=8y=8. APEX variants consistently achieve the lowest EDP across all context lengths, demonstrating superior hardware efficiency. TABLE VI: Overhead analysis of the APEX prefetch router. Model Additional Parameters % of model weights % of Performance Granite-1B 0.79M 0.059% 0.051% Granite-3B 1.97M 0.060% 0.046% Phi-7B 2.10M 0.027% 0.009% DeepSeek-16B 34.11M 0.022% 0.036% V-F Overhead Analysis The additional parameters required by APEX comprise the lightweight prefetch router and the CDF model. For example, in Granite-1B, each layer includes a 1024×321024× 32 prefetch router and 2×322× 32 CDF parameters, resulting in a total of 0.79M parameters across 24 layers. This is only 0.059% of the 1.3B model weights. The overhead remains similarly negligible for the larger models, accounting for just 0.060%, 0.027% and 0.022% of total parameters for Granite-3B, Phi-7B and DeepSeek-16B, respectively, as shown in Table VI. On our target edge platform (described in Section V-B), APEX adds only 0.051%, 0.046%, 0.009% and 0.036% performance overhead for Granite-1B, Granite-3B, Phi-7B, and DeepSeek-16B, respectively. Since both the added weights and computation are negligible relative to the base model, this overhead is expected to remain insignificant on other platforms as well. Overall, these results show that APEX is an extremely low-overhead approach. It adds only a lightweight auxiliary predictor on top of the base MoE model, while delivering substantial improvements in latency and energy efficiency. V-G Ablation Study Sensitivity to I/O Bandwidth: We evaluate APEX across different off-chip I/O bandwidths using Granite-3B as a representative model at a context length of 1024, while keeping all other parameters fixed to isolate the impact of bandwidth on prefetch effectiveness. The sweep ranges from 32 to 1024 GB/s. The low-bandwidth points (32 and 64 GB/s) represent constrained mobile/embedded-class settings where expert transfers receive only a fraction of total DRAM bandwidth due to sharing with CPU/GPU/NPU activity, KV-cache traffic, and memory-controller contention. The 128 and 256 GB/s points correspond to PCIe 5.0 ×16 and PCIe 6.0 ×16 bandwidths, respectively. As expected, increasing bandwidth reduces latency, even for the no-prefetch baseline, since expert transfers become less expensive at higher bandwidth. Figure 14 shows that APEX consistently decreases the latency by 14%–42% across the entire bandwidth range by minimizing the number of exposed expert-loading stalls. At very low bandwidths, the benefit is limited by the fact that only part of the expert transfer can be hidden behind the attention window, so the unhidden portion remains exposed as stall time. However, APEX still consistently improves latency across the full range by overlapping the transferable portion of expert loading with attention and by avoiding the unnecessary traffic of fixed overfetching. At the same time, APEX remains beneficial even at high-bandwidth settings approaching NVLink-C2C class interconnects, showing that adaptive prefetching continues to improve latency even when I/O is less constrained. Overall, these results highlight that APEX is robust across a wide range of deployment scenarios, with particularly strong benefits in the bandwidth-limited edge setting. Fig. 14: Per-token latency versus bidirectional I/O bandwidth for Granite-3B (1024 context length). Sensitivity to Expert Weight Precision: Since many edge deployments use quantized weights, we further evaluate APEX across different expert-weight data types on Granite-3B (1024 context). This study scales the expert transfer size and compute cost for 4-, 8-, 16-, and 32-bit expert weights while keeping the routing behavior fixed. Therefore, it isolates the system-level impact of datatype and does not quantify quantization-induced accuracy loss. As shown in Figure 15, reducing precision lowers absolute latency for both No Prefetch and APEX because each expert transfer becomes smaller. However, APEX continues to reduce latency across all datatypes by overlapping expert movement with attention computation. The benefit is especially clear when expert transfers remain exposed on the critical path, while at very low precision, the execution is dominated by attention. These results show that APEX complements quantization. Quantization reduces the amount of data moved [50], whereas APEX reduces the portion of that movement exposed. Fig. 15: Per-token latency sensitivity to expert-weight datatype for Granite-3B (1024 context length). VI Conclusion Edge MoE inference is fundamentally bottlenecked by expert I/O, where irregular and off-chip expert accesses stall execution and waste energy. In this work, we presented APEX, an adaptive expert prefetching framework that transforms expert loading into a predictive, confidence-driven decision. By introducing a lightweight prefetch router and a learned CDF-based mechanism to dynamically select the minimal top-(k+δ^(x))(k+ δ(x)) prefetch budget, APEX achieves near-oracle expert coverage while avoiding the inefficiencies of static overfetching. This adaptive approach delivers substantial system-level benefits. At higher confidence thresholds, APEX achieves >>99% overlap accuracy and effectively eliminates expert-loading stalls while preserving application-level accuracy. Across multiple models and settings, it reduces latency and improves energy-delay product (EDP) by up to 41% compared to state-of-the-art baselines, with negligible model and runtime overhead. Overall, APEX demonstrates that adaptive, confidence-aware prefetching is key to unlocking efficient MoE inference on edge systems, bridging the gap between predictive accuracy and system performance. Disclosure: Dr. Ogras is affiliated with Samsung Austin Research & Development Center and Advanced Computing Lab (SARC/ACL). This relationship has been approved under applicable outside activities policies. References [1] M. Abdin et al. (2024) Phi-3 technical report: a highly capable language model locally on your phone. External Links: 2404.14219 Cited by: §I, 3rd item. [2] R. Y. Aminabadi et al. (2022) DeepSpeed-Inference: enabling efficient inference of transformer models at unprecedented scale. In Proc. Int. Conf. Performance Computing, Networking, Storage and Analysis (SC), Cited by: §I, §V-B. [3] R. Balasubramonian, A. B. Kahng, N. Muralimanohar, A. Shafiee, and V. Srinivas (2017) CACTI 7: new tools for interconnect exploration in innovative off-chip memories. ACM Trans. Architecture and Code Optimization (TACO) 14 (2), p. 1–25. Cited by: TABLE I. [4] A. Cheng et al. (2025) ERMoE: eigen-reparameterized mixture-of-experts for stable routing and interpretable specialization. External Links: 2511.10971 Cited by: §I. [5] P. Clark et al. (2018) Think you have solved question answering? try ARC, the AI2 reasoning challenge. External Links: 1803.05457 Cited by: §V-A. [6] D. Dai et al. (2024) DeepSeekMoE: towards ultimate expert specialization in mixture-of-experts language models. In Proc. Annu. Meet. Assoc. Computational Linguistics (ACL), p. 1280–1297. Cited by: §I. [7] J. Dong et al. (2025) UbiMoE: a ubiquitous mixture-of-experts vision transformer accelerator with hybrid computation pattern on FPGA. In Proc. IEEE Int. Symp. Circuits Systems (ISCAS), p. 1–5. Cited by: §I. [8] A. Eliseev and D. Mazur (2023) Fast inference of mixture-of-experts language models with offloading. External Links: 2312.17238 Cited by: §V-A. [9] Z. Fang et al. (2025) FATE: fast edge inference of mixture-of-experts models via cross-layer gate. External Links: 2502.12224 Cited by: TABLE I, §I. [10] W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Machine Learning Research 23 (120), p. 1–39. Cited by: §I. [11] T. Gale, D. Narayanan, C. Young, and M. Zaharia (2023) MegaBlocks: efficient sparse training with mixture-of-experts. In Proc. Machine Learning and Systems (MLSys), Vol. 5. Cited by: §I. [12] A. Gholami et al. (2024) AI and the memory wall. IEEE Micro 44 (3), p. 33–39. Cited by: §I. [13] S. Ghose et al. (2018) Understanding DRAM power consumption: experimental characterization and analysis. In Proc. ACM SIGMETRICS, Cited by: §V-B, TABLE I. [14] I. Goodfellow, Y. Bengio, and A. Courville (2016) Deep learning. MIT Press. Cited by: §IV-D. [15] Hailo (2024) Hailo-10H AI Accelerator: LLM and VLM Generative AI Accelerator. Note: https://hailo.ai/products/ai-accelerators/hailo-10h-ai-accelerator/Accessed: 2026-06-14 Cited by: §V-B. [16] J. He et al. (2021) FastMoE: a fast mixture-of-expert training system. External Links: 2103.13262 Cited by: §I. [17] S. He et al. (2025) Hydra: harnessing expert popularity for efficient mixture-of-expert inference on chiplet systems. In Proc. ACM/IEEE Design Automation Conf. (DAC), p. 1–7. Cited by: §I. [18] D. Hendrycks et al. (2020) Measuring massive multitask language understanding. External Links: 2009.03300 Cited by: §V-A. [19] R. Hwang et al. (2024) Pre-gated MoE: an algorithm-system co-design for fast and scalable mixture-of-expert inference. In Proc. Annu. Int. Symp. Computer Architecture (ISCA), p. 1018–1031. Cited by: TABLE I, §I. [20] IBM Research (2024) Granite-3.1-1B-A400M Base: a sparse mixture-of-experts language model. Note: model card, accessed 2026-03-27 External Links: Link Cited by: §I, 1st item. [21] IBM Research (2024) Granite-3.1-3B-A800M Base: a sparse mixture-of-experts language model. Note: model card, accessed 2026-03-27 External Links: Link Cited by: §I, §I, 2nd item. [22] A. Q. Jiang et al. (2024) Mixtral of experts. External Links: 2401.04088 Cited by: §I, §I. [23] A. Kanani et al. (2025) THERMOS: thermally-aware Multi-Objective Scheduling of AI Workloads on Heterogeneous Multi-Chiplet PIM Architectures. ACM Transactions on Embedded Computing Systems 24 (5s), p. 1–26. Cited by: §I. [24] A. Kanani et al. (2026) DUET: Disaggregated Hybrid Mamba-Transformer LLMs with Prefill and Decode-Specific Packages. In Proc. ACM/IEEE Design Automation Conf. (DAC), p. 1–7. Cited by: §V-B. [25] W. Kwon et al. (2023) Efficient memory management for large language model serving with PagedAttention. In Proc. ACM Symp. operating systems principles (SOSP), p. 611–626. Cited by: §V-B. [26] D. Lepikhin et al. (2021) GShard: scaling giant models with conditional computation and automatic sharding. In Proc. Int. Conf. Learning Representations (ICLR), Cited by: §I, §I, §I-A. [27] M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer (2021) BASE layers: simplifying training of large, sparse models. In Proc. Int. Conf. Machine Learning (ICML), p. 6265–6274. Cited by: §I. [28] S. Li, J. Lin, D. Ge, and Y. Ye (2026) MoE-SpAc: efficient MoE inference based on speculative activation utility in heterogeneous edge scenarios. External Links: 2603.09983 Cited by: §I. [29] S. Lin, J. Hilton, and O. Evans (2022) TruthfulQA: measuring how models mimic human falsehoods. In Proc. Annu. Meet. Assoc. computational linguistics (ACL), p. 3214–3252. Cited by: §V-A. [30] A. Liu et al. (2024) Deepseek-v2: a strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434. Cited by: §I, 4th item. [31] H. Luo et al. (2024) Ramulator 2.0: a modern, modular, and extensible DRAM simulator. IEEE Computer Architecture Letters 23 (1), p. 112–116. Cited by: §V-B. [32] S. Merity, C. Xiong, J. Bradbury, and R. Socher (2016) Pointer sentinel mixture models. Note: Introduces the WikiText language modeling datasets External Links: 1609.07843 Cited by: Fig. 6, §IV-B, §V-A, §V-A. [33] T. Meuser et al. (2024) Revisiting edge AI: opportunities and challenges. IEEE Internet Computing 28 (4), p. 49–59. Cited by: §I. [34] N. Muennighoff et al. (2024) OLMoE: open mixture-of-experts language models. External Links: 2409.02060 Cited by: §I. [35] NVIDIA Corporation (2020) GeForce RTX 3090. Note: accessed 2026-03-11 External Links: Link Cited by: §I, §V-A, §V-B. [36] NVIDIA Corporation (2020) NVIDIA A100 Tensor Core GPU Architecture. Note: accessed 2026-06-01 External Links: Link Cited by: §V-A. [37] NVIDIA (2026) Jetson Thor: Advanced AI for Physical Robotics. Note: https://w.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-thor/Accessed: 2026-06-14 Cited by: §V-B. [38] J. Park et al. (2024) Thermal modeling and management challenges in heterogeneous integration: 2.5D chiplet platforms and beyond. In Proc. IEEE VLSI Test Symp. (VTS), p. 1–4. Cited by: §I. [39] M. Park et al. (2023) A 192-gb 12-high 896-gb/s HBM3 DRAM with a TSV auto-calibration scheme and machine-learning-based layout optimization. IEEE J. Solid-State Circuits 58 (1), p. 256–269. Cited by: §V-B, TABLE I. [40] PCI-SIG (2022) PCI Express Base Specification Revision 6.0. Note: PCI-SIG specification External Links: Link Cited by: §V-B, §V-B, TABLE I. [41] L. Pfromm et al. (2025) CHIPSIM: a co-simulation framework for deep learning on chiplet-based systems. IEEE Open J. Solid-State Circuits Society 5, p. 410–423. Cited by: §V-B. [42] L. Pfromm et al. (2025) MFIT: multi-Fidelity Thermal Modeling for 2.5d and 3d Multi-Chiplet Architectures. ACM Transactions on Design Automation of Electronic Systems. External Links: Document Cited by: §I. [43] R. Pope et al. (2023) Efficiently scaling transformer inference. In Proc. Maching Learning System (MLSys), Vol. 5. Cited by: §I. [44] E. Reggiani, R. Andri, and L. Cavigelli (2023) Flex-SFU: accelerating DNN activation functions by non-uniform piecewise approximation. In Proc. ACM/IEEE Design Automation Conference (DAC), p. 1–6. Cited by: §V-B. [45] K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi (2021) WinoGrande: an adversarial winograd schema challenge at scale. Commun. ACM 64 (9), p. 99–106. Cited by: §V-A. [46] G. Shan, Y. Zheng, C. Xing, D. Chen, G. Li, and Y. Yang (2022) Architecture of computing system based on chiplet. Micromachines 13 (2), p. 205. Cited by: §V-B. [47] D. D. Sharma, G. Pasdast, Z. Qian, and K. Aygun (2022) Universal chiplet interconnect express (UCIe): an open industry standard for innovations with chiplets at package level. IEEE Trans. Components, Packaging and Manufacturing Technology 12 (9), p. 1423–1431. Cited by: §V-B, TABLE I. [48] N. Shazeer et al. (2017) Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In Proc. Int. Conf. Learning Representations (ICLR), Cited by: §I, §I-A. [49] X. Song, Z. Zhong, R. Chen, and H. Chen (2024) ProMoE: fast MoE-based LLM serving using proactive caching. External Links: 2410.22134 Cited by: TABLE I, §I, §I, §V-A. [50] M. Sun, A. Kanani, K. Shroff, and U. Ogras (2026) LEXI: Lossless Exponent Coding for Efficient Inter-Chiplet Communication in Hybrid LLMs. In Proc. ACM/IEEE Design Automation Conf. (DAC), p. 1–7. Cited by: §V-G. [51] Synopsys Inc. (2023) Synopsys PrimeTime. Note: static timing and power analysis tool External Links: Link Cited by: §V-B. [52] P. Tang et al. (2024) HOBBIT: a mixed precision expert offloading system for fast MoE inference. External Links: 2411.01433 Cited by: TABLE I, §I, §I. [53] A. Vaswani et al. (2017) Attention is all you need. In Proc. Adv. Neural Information Processing System (NeurIPS), Vol. 30, p. 5998–6008. Cited by: §I-A. [54] L. Xue et al. (2024) MoE-Infinity: efficient MoE inference on personal machines with sparsity-aware expert cache. External Links: 2401.14361 Cited by: TABLE I, §I. [55] Y. Zheng et al. (2025) A review on edge large language models: design, execution, and applications. ACM Computing Survey 57 (8), p. 1–35. Cited by: §I, §I. [56] S. Zhong et al. (2024) AdapMoE: adaptive sensitivity-based expert gating and management for efficient MoE inference. In Proc. IEEE/ACM Int. Conf. Computer-Aided Design (ICCAD), p. 1–9. Cited by: §I, §I. [57] S. Zhu, S. Bohl, R. Oester, and G. Alonso (2025) Pre-attention expert prediction and prefetching for mixture-of-experts large language models. arXiv preprint arXiv:2511.10676. Cited by: TABLE I, §I. [58] B. Zoph et al. (2022) ST-MoE: designing stable and transferable sparse expert models. External Links: 2202.08906 Cited by: §I. [59] Y. Zou et al. (2024) FED-MOE: efficient federated learning for mixture-of-experts models via empirical pruning. In Proc. Int. Conf. Parallel Distributed Computing: Applications and technologies (PDCAT), p. 128–139. Cited by: §I.