Paper deep dive
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
Dengke Han
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU. In this setting, self-speculative decoding faces a new bottleneck: increasing the draft expert set improves accuracy but triggers extra expert loading, while cheap small-footprint drafts have low acceptance; moreover, verifying a multi-token block activates the union of target experts and is no longer close to one target step. We propose DraftExpert, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference. DraftExpert trains one lightweight accelerator-resident draft expert per layer by self-distilling residual, logit/token, and router-agreement signals from the frozen target MoE. At inference time, it uses a fixed-footprint shared+top-1+draft-expert drafter together with confidence--expansion truncation and target-expert prefetching, while final tokens are still exactly verified by the target model. On DeepSeek-V2-Lite and Moonlight-16B-A3B across CPU-GPU and Flash-NPU offload, DraftExpert improves decode throughput by 1.45x on average, raises draft acceptance to 84~87%, and achieves 86~88% prefetch hit rates.
Tags
Links
- Source: https://arxiv.org/abs/2607.24434v1
- Canonical: https://arxiv.org/abs/2607.24434v1
Trouble viewing inline? Open PDF directly →
Full Text
41,684 characters extracted from source content.
Expand or collapse full text
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference Dengke Han Abstract Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU. In this setting, self-speculative decoding faces a new bottleneck: increasing the draft expert set improves accuracy but triggers extra expert loading, while cheap small-footprint drafts have low acceptance; moreover, verifying a multi-token block activates the union of target experts and is no longer close to one target step. We propose DraftExpert, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference. DraftExpert trains one lightweight accelerator-resident draft expert per layer by self-distilling residual, logit/token, and router-agreement signals from the frozen target MoE. At inference time, it uses a fixed-footprint shared+top-1+draft-expert drafter together with confidence–expansion truncation and target-expert prefetching, while final tokens are still exactly verified by the target model. On DeepSeek-V2-Lite and Moonlight-16B-A3B across CPU→ and Flash→ offload, DraftExpert improves decode throughput by 1.45×1.45× on average, raises draft acceptance to 84–87%, and achieves 86–88% prefetch hit rates. Introduction Speculative decoding accelerates autoregressive generation by drafting multiple future tokens and verifying them with the target model in parallel (Leviathan et al. 2023; Chen et al. 2023; Miao et al. 2024). Its performance rests on three conditions: the draft must be cheap, parallel verification must be close to one target step, and the draft must be accepted often enough. This paper argues that end-device Mixture-of-Experts (MoE) inference changes the performance paradigm behind all three conditions. When routed experts are offloaded to CPU memory for consumer CPU→ execution or to Flash for mobile Flash→ execution, speculative efficiency is no longer governed mainly by the number of drafted tokens; it is governed by how quickly the draft and verification stages expand the set of experts that must be loaded. This setting is increasingly important. Large MoE language models are attractive for personal devices because each token activates only a small fraction of experts (Shazeer et al. 2017; Lepikhin et al. 2021; Fedus et al. 2022; Du et al. 2022; Jiang et al. 2024; DeepSeek-AI 2024; Liu et al. 2025). We target latency-critical single-user end-device inference, where interactive generation has limited batching or cross-request cache reuse and therefore exposes per-request expert movement. However, the routed experts dominate the parameter count and often exceed the memory of consumer GPUs or mobile accelerators. A practical runtime therefore keeps routed experts in CPU memory or mobile Flash and loads them on demand, while the rest of the model stays accelerator-resident because it is used every forward pass (Xue et al. 2024; Yi et al. 2023; Kamahori et al. 2025). Expert movement becomes a first-order latency cost. Under this expert-offload cost model, existing MoE self-speculative methods struggle with all three conditions. First, the draft is not necessarily cheap. Training-free self-drafters often use the shared path plus router top-r routed experts. Increasing r improves the approximation to the target MoE, but it also makes the drafter load more offloaded experts as the draft grows. Second, verification is not nearly free. A block of K draft tokens may activate the union of many target experts, so the cost of one parallel verify pass grows with expert-set expansion. Third, acceptance is not high when r is kept small. A top-1 or small-r drafter omits routed-expert residuals, causing hidden-state drift, lower token agreement, and weaker router agreement. The key performance unit therefore shifts from tokens per target forward to accepted tokens per expert-set expansion. A useful end-device MoE speculative decoder should restore the three classical conditions under this new unit: it should keep draft expansion bounded, make verification expansion predictable and controllable, and recover the accuracy lost by cheap drafting. We propose DraftExpert, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference. For the draft condition, DraftExpert uses a fixed-footprint shared+top-1+draft-expert path. Each MoE layer receives one lightweight accelerator-resident draft expert, while the original target MoE remains frozen. The draft expert is trained to compensate for the routed experts omitted by top-1 drafting, so the drafter can improve accuracy without increasing r or loading more experts. For the acceptance condition, DraftExpert uses self-distillation from the full target MoE. Residual, logit, and token losses teach the draft expert to recover missing expert behavior and improve token agreement. For the verification condition, DraftExpert adds router-agreement distillation, making cheap draft hidden states more predictive of the target experts that verification will use. These predictions enable two exact runtime policies: a confidence–expansion controller truncates drafts when a low-confidence candidate would introduce too many new verifier experts, and draft-router prefetching moves likely verifier experts before the target reaches them. All emitted draft tokens remain verified by the frozen full MoE. This paper makes three contributions: • We formulate expert-offloaded MoE speculative decoding as an expert-set expansion problem and show how expansion breaks the three classical conditions for speculative speedup. • We introduce DraftExpert, a fixed-footprint shared+top-1+draft-expert self-drafter that reduces draft-stage expert loading while self-distillation restores acceptance. • We use router-agreement distillation to predict verifier expert sets, enabling expansion-aware dynamic truncation and draft-router expert prefetching for exact target verification. Background and Motivation Speculation Conditions Under Expert Offload Figure 1(a) shows the parameter asymmetry that makes expert offloading both necessary and natural on memory-constrained end devices. In DeepSeek-V2-Lite, routed experts account for 14.4B parameters, or 91.6% of the model weights. In BF16, this routed pool alone occupies 26.8 GiB, while all non-routed weights together occupy only 2.4 GiB. At the same time, the routed pool is sparsely activated: each token selects only top-6 out of 64 routed experts per MoE layer, i.e., 9.4% of the routed pool. By contrast, attention, shared experts, dense layers, routers, norms, and output projection are exercised on every forward pass. Figure 1: Motivation for expert-offloaded MoE inference on end devices. (a) Routed experts dominate DeepSeek-V2-Lite weight storage, but each token activates only a small selected subset; non-routed weights are much smaller and used every forward pass. (b) With non-routed weights resident and routed experts loaded on demand, expert loading dominates decode-128 profiled time in both CPU→ and Flash→ settings. Flash→ uses llama.cpp on a Hexagon HTP v81 NPU and excludes runtime API gap. Therefore, the most feasible way to run large MoE models on end devices is to keep non-routed weights resident on the accelerator and place the large routed-expert pool in slower memory: CPU DRAM for consumer GPUs or Flash storage for mobile NPU/GPU execution. Selected experts are loaded on demand. This deployment enables models whose total parameters exceed accelerator memory, but it also changes the performance model: Figure 1(b) shows that expert movement, rather than arithmetic, becomes the dominant decode bottleneck. Speculative decoding is fast only when three conditions hold: drafting is cheap, verifying a block is close to one target step, and the accepted prefix is long enough to amortize the draft work. In dense models, these conditions are often analyzed in units of target forward passes. In expert-offloaded MoE inference, the natural unit is different. Because routed experts live outside accelerator memory and are loaded on demand, the cost of both drafting and verification depends on the unique experts touched by a token block. For an MoE layer l, let the target router select TopKl(h)⊆1,…,Nl,|TopKl(h)|=kl.TopK_l(h) \1,…,N_l\, |TopK_l(h)|=k_l. (1) For a block B, define its expert set and offload cost as l(B)=⋃v∈BTopKl(hl,v),(B)=∑l∑e∈l(B)cl,e,U_l(B)= _v∈ BTopK_l(h_l,v), (B)= _l _e _l(B)c_l,e, (2) where cl,ec_l,e captures expert loading and execution. The central bottleneck is expert-set expansion: each additional token may introduce newly loaded experts. Because Figure 1(b) shows that expert loading dominates decode, expansion directly turns into latency. Condition 1: Drafting Must Stay Cheap MoE self-drafting avoids a separate draft model by reusing the target model itself. A common strategy evaluates the shared path and router-selected top-r routed experts. This creates a direct acceptance–cost trade-off. Larger r better approximates the full target MoE and usually improves draft accuracy, but it also makes the drafter load more offloaded experts. Thus the drafter becomes less cheap exactly when it becomes more accurate. Figure 2 quantifies this trade-off in our cold expert-offload setting. Averaged over draft lengths, increasing r raises acceptance from 22% to 31% and 42%, but it also increases draft latency relative to top-1. More importantly, the drafter’s speedup over the full target drops from 4.7× for top-1 to 2.7× for top-2 and 2.0× for top-3. Thus naive top-r self-drafting buys acceptance by spending the cheap-draft budget on additional expert loading. Figure 3 further illustrates the underlying mechanism: as draft length or r grows, the cumulative unique routed experts used by drafting expands rapidly. Therefore, recovering the cheap-draft condition requires a fixed-footprint drafter: accuracy should come from a small accelerator-resident learned module, not from loading more routed experts. Figure 2: Top-r acceptance–cheapness trade-off under cold expert offloading. Increasing router top-r improves draft acceptance, but it also makes drafting less cheap: draft latency rises relative to top-1, while draft speedup over the full target falls from 4.7× to 2.0×. Figure 3: Draft-side expert-set expansion. Shared+top-r self-drafters improve approximation by loading more routed experts, so draft cost grows with r and draft length. Condition 2: Parallel Verification Must Be Controlled Dense speculative decoding benefits from the fact that verifying multiple tokens reuses the same dense FFN weights. Offloaded MoE verification is different: a verification block may route different tokens to different experts, so one parallel pass can still load many unique experts. Figure 4 compares dense FFN offload with MoE expert offload after normalizing each runtime by its own one-token decode cost. Dense verification remains close to one step, while MoE verification grows with block length. Figure 4: Dense versus MoE verification under offloaded weights. Dense FFN verification reuses the same weights across tokens, while MoE verification becomes more expensive as longer blocks activate more unique routed experts. For a verification block D=(v1,…,vK)D=(v_1,…,v_K), the verifier may access l(D)=⋃i=1KTopKl(hl,vi⋆).U_l(D)= _i=1^KTopK_l(h_l,v_i ). (3) Figure 5 directly measures this expansion. The verify condition should therefore be reframed as controlling marginal expert expansion, not merely maximizing parallel block length. Figure 5: Verify-side expert-set expansion. Multi-token decode or verification activates a growing union of routed layer-experts, so parallel verification is not free when target experts are loaded from CPU memory or Flash. Condition 3: Acceptance Requires Learned Drafting One might keep draft and verify expansion small by using only shared+top-1 drafting and short fixed blocks. However, this sacrifices the third condition: high acceptance. Top-1 drafting omits the residual contribution of other target experts, causing hidden-state drift and lower next-token agreement. Simply spending more expert-loading cost on naive shared+top-r drafting does not solve the problem either: Figure 2 shows that even the much more expensive top-3 path remains below 50% acceptance. In other words, naive self-drafting pays a large movement cost while still rejecting most candidates, which is close to guessing from the verifier’s perspective. Low acceptance also weakens router agreement, which matters for verification because offload runtimes often use the drafter’s router to predict or prefetch verifier experts. If the cheap drafter’s hidden states drift, expert prediction becomes inaccurate, prefetch hit rate falls, and verification stalls remain. This motivates a dedicated self-distilled draft expert: acceptance should be recovered by learning the missing routed-expert residual, not by loading a larger top-r expert set during drafting. These observations suggest an expansion-aware design principle: optimize accepted tokens per newly loaded expert. A method should (i) bound draft expert-set expansion, (i) recover acceptance without increasing r, and (i) make verifier expert expansion predictable enough for truncation and prefetching. DraftExpert Method Overview DraftExpert restores the three speculative decoding conditions under an expert-set expansion cost model. To make the draft cheap, it replaces shared+top-r self-drafting with a fixed-footprint shared+top-1+draft-expert path. To keep acceptance high, it self-distills the draft experts from the frozen full MoE so the cheap path recovers information from omitted routed experts. To make verification close to cheap again, it improves router agreement and uses the predicted verifier expert set for cost-aware truncation and prefetching. The deployment assumption is that target routed experts are offloaded to CPU memory or Flash and loaded on demand, while the non-expert components and DraftExpert’s added draft experts are accelerator-resident. The original target MoE is frozen. In exact mode, every emitted draft token is verified by the full target MoE; truncation and prefetching change only how many candidates are proposed and when experts are moved. Figure 6: Expansion-aware DraftExpert overview. Expert-offloaded MoE breaks the three classical speculative decoding conditions through expert-set expansion: top-r drafting is expensive yet still yields low acceptance, parallel verification activates a growing target-expert union, and naive small-footprint drafting drifts from the verifier. DraftExpert restores these conditions with a fixed-footprint shared+top-1+draft-expert path, self-distillation with router agreement, and expansion-aware truncation plus target-expert prefetching for exact verification. Fixed-Footprint Drafting Let hlh_l be the hidden state entering MoE layer l, SlS_l be the shared expert path, and ℛl(hl)R_l(h_l) be the original router distribution. The full target MoE output is yl⋆=Sl(hl)+∑e∈TopKl(hl)gl,eEl,e(hl),y_l =S_l(h_l)+ _e _l(h_l)g_l,eE_l,e(h_l), (4) where El,eE_l,e are frozen routed experts. A shared+top-r drafter reduces cost by using fewer routed experts than the target, but its cost still grows with r and with the unique experts touched by the draft block. DraftExpert instead adds one draft expert EldE_l^d per MoE layer and drafts with yld y_l^d =Sl(hl)+gl,e1El,e1(hl)+γlEld(hl), =S_l(h_l)+g_l,e_1E_l,e_1(h_l)+ _lE_l^d(h_l), (5) e1 e_1 =argmaxeℛl(hl)e. = _eR_l(h_l)_e. Only the draft experts and optional scales are trainable. This keeps routed-expert usage close to top-1 while adding a small accelerator-resident module that can learn the missing top-k information. Self-Distillation for Accuracy and Router Agreement Self-distillation has two roles. The first is to improve draft accuracy without increasing r. For a teacher-forced sequence, define the missing routed-expert residual after removing the shared and top-1 paths: ml,t⋆=yl,t⋆−Sl(hl,t⋆)−gl,e1El,e1(hl,t⋆).m_l,t =y_l,t -S_l(h_l,t )-g_l,e_1E_l,e_1(h_l,t ). (6) The draft expert is trained to approximate this residual, ℒres=∑l,t‖γlEld(hl,td)−ml,t⋆‖22.L_res= _l,t \| _lE_l^d(h_l,t^d)-m_l,t \|_2^2. (7) We also distill final logits and top-1 target tokens: ℒlogit=T2∑tKL(softmax(zt⋆/T)∥softmax(ztd/T)),L_logit=T^2 _tKL (softmax(z_t /T)\|softmax(z_t^d/T) ), (8) ℒtok=∑tCE(argmaxzt⋆,softmax(ztd)).L_tok= _tCE ( z_t ,softmax(z_t^d) ). (9) The second role is verify-side expert prediction. If the draft hidden states drift from the target path, the draft router becomes a poor predictor of verifier experts. We therefore include router-agreement supervision, ℒrouter=∑l,tKL(ℛl(hl,t⋆)∥ℛl(hl,td)),L_router= _l,tKL (R_l(h_l,t )\|R_l(h_l,t^d) ), (10) and optionally track top-k expert-set agreement as a validation metric. The complete objective is ℒ=ℒlogit+αℒtok+βℒres+ηℒrouter.L=L_logit+ _tok+ _res+ _router. (11) All supervision comes from unlabeled text and the frozen target MoE. Cost-Aware Dynamic Truncation For the i-th draft token, let qiq_i be a confidence score from the draft logits. Let ℰ^l,i E_l,i be the verifier experts predicted by the draft router for layer l. DraftExpert maintains a predicted block-level verifier set ^i=⋃j≤i⋃lℰ^l,j, U_i= _j≤ i _l E_l,j, (12) with marginal expansion Δi=|^i|−|^i−1|. _i=| U_i|-| U_i-1|. (13) The controller stops before appending token i when the token is unlikely to be accepted and likely to add verifier cost: qi<τandΔi>Bi,q_i<τ _i>B_i, (14) where BiB_i can be a fixed layer-expert budget or a budget derived from the current expert cache. This policy is exact because discarded candidates are never emitted, and all emitted candidates are verified by the target MoE. Draft-Router Expert Prefetch The same router-aligned draft states support prefetching. As soon as draft token i is kept, DraftExpert launches non-blocking prefetches for newly predicted verifier experts ℰ^l,i E_l,i that are not already cached or in flight. These transfers overlap with the remaining draft-token computation; verification only waits for unfinished predicted transfers when it reaches the target pass, and any missed expert is still loaded on demand. Prefetching does not change model semantics: if a prefetched expert is used, verification avoids an on-demand stall; if it is unused, the cost is wasted bandwidth; if a required expert is missed, the verifier loads it on demand. Router-agreement distillation is therefore important because it directly affects prefetch hit rate and wasted movement. Algorithm 1 Exact DraftExpert Decoding 1: D←∅D← , ^←∅ U← , ←∅P← 2: ←C← current accelerator expert cache 3: for i=1i=1 to KmaxK_ do 4: Draft viv_i with shared+top-1+draft-expert path 5: Compute confidence qiq_i and predicted verifier experts ℰ^l,i E_l,i 6: ^i←^∪⋃lℰ^l,i U_i← U∪ _l E_l,i 7: Δi←|^i|−|^| _i←| U_i|-| U| 8: if qi<τq_i<τ and Δi>Bi _i>B_i then 9: Stop drafting before appending viv_i 10: break 11: end if 12: Append viv_i to D and set ^←^i U← U_i 13: Launch async prefetch for ^i∖(∪) U_i (C ) 14: Add launched experts to in-flight set P 15: end for 16: Wait for useful in-flight prefetches; load misses on demand 17: Verify all tokens in D with the frozen full target MoE 18: Accept the target-consistent prefix and continue decoding Experimental Evaluation We evaluate DraftExpert on two expert-offloaded MoE targets, DeepSeek-V2-Lite (DS) (DeepSeek-AI 2024) and Moonlight-16B-A3B (ML) (Liu et al. 2025). The experiments are organized around the three conditions in Section 2: a cheap drafter, controlled verification, and high acceptance. Unless noted otherwise, all throughput numbers are decode-stage tokens per second after prefill, and speedups are normalized to autoregressive (AR) expert-offloaded decoding on the same model and platform. Setup and Metrics We use the same models, prompts, and decoding policies across two end-device memory hierarchies. In the consumer-GPU setting, routed experts are resident in host CPU memory and copied to a discrete GPU on demand. In the mobile setting, CPU, GPU, and NPU share physical DRAM, but DRAM is too small to keep all routed experts resident; routed experts are stored in Flash and staged into shared memory for NPU execution. Mobile measurements use llama.cpp (Gerganov 2023) with Q4_0 quantized model weights on the Hexagon HTP NPU. Attention, embeddings, output heads, shared experts, routers, and DraftExpert’s added draft experts remain accelerator-resident. Routed target experts are not kept in a persistent accelerator cache unless an experiment explicitly studies the resident-memory reference. Item Configuration CG NVIDIA GeForce RTX 4090, 24 GB memory; BF16 PyTorch 2.11.0+cu128; CPU-resident routed experts loaded on demand to GPU. MN device Xiaomi 2509FPN0BC / MIX Flip 2 (popsicle/canoe), Android 16 API 36, MIUI V816, Snapdragon 8 Elite (SM8850, TSMC N3E). MN NPU Hexagon HTP v81 fused AI accelerator; INT8/INT4 peak about 73/146 TOPS; INT4, INT8, INT16, FP16, BF16 support; micro-tile inference; about 8 MB on-chip SRAM. MN runtime llama.cpp with Q4_0 model weights. Routed experts are Flash-resident and staged into shared CPU/GPU/NPU memory on demand. The Flash-staged AR decode baseline averages 10.18 TPS; if experts are already memory-resident, the DS Q4_0 AR reference reaches about 14.36 TPS. Prompts Chat, GSM8K, MBPP, summarization, and instruction following. Models DS and ML each use 27 layers, 64 routed + 2 shared experts, and top-6 active routed experts. Table 1: Evaluation platforms. CG denotes CPU→ expert offload, and MN denotes Flash→ expert offload. Mobile throughput is reported for the Q4_0 llama.cpp NPU runtime; the resident-memory value is an AR reference rather than the offloaded baseline. We compare three method groups. AR offload decodes with the frozen target MoE and loads routed experts on demand. Shared+top-r is the strongest training-free self-drafting baseline after sweeping r∈1,2,3r∈\1,2,3\. DraftExpert uses the fixed-footprint shared+top-1+draft-expert path, router-agreement distillation, confidence–expansion truncation, and target-expert prefetch. All final tokens are verified by the frozen target MoE, so DraftExpert preserves exact target outputs. Decode Throughput Table 2 reports decode-stage throughput. DraftExpert improves average decode TPS by 1.45×1.45× over AR offload across the two models and two platforms, but the gains are intentionally not uniform across platforms. CG uses BF16 weights over PCIe into an RTX 4090, whereas MN uses Q4_0 weights staged from Flash into a shared-memory Hexagon HTP runtime; both absolute TPS and normalized speedup therefore reflect different loading bandwidths, quantization formats, and accelerator compute ratios. On MN, the DS Q4_0 Flash-staged AR baseline reaches 10.18 TPS, while the resident-memory AR reference reaches about 14.36 TPS; this gap quantifies the loading headroom before speculative gains are applied. Model Plat. AR TPS top-r TPS DE TPS Speedup DS CG 2.19 1.19 2.99 1.36× DS MN 10.18 4.80 15.47 1.52× ML CG 1.94 1.01 2.50 1.29× ML MN 8.50 3.82 13.69 1.61× Avg. – – – – 1.45× Table 2: Decode-stage exact speculative throughput. AR TPS is the expert-offloaded autoregressive baseline. top-r is the best training-free shared+top-r exact self-drafting setting, including drafting, verification, and repair. DE is DraftExpert with truncation and prefetch. The main trend is consistent, but its magnitude differs across CG and MN. Training-free shared+top-r improves draft accuracy but is slower than AR once exact verification and repair are included, because it pays extra expert movement before many candidates are rejected. DraftExpert keeps the draft footprint close to top-1 and recovers acceptance through self-distillation, so it converts expert-loading savings into end-to-end decode speedup. The larger MN gains reflect the Flash-staging bottleneck despite Q4_0 quantization, while CG gains are more compute- and PCIe-balanced. Draft-Side Cost and Acceptance The draft-side experiment isolates the first condition: drafting must remain cheap. For draft length K=4K=4, Table 3 reports the cumulative unique routed layer-experts touched by the drafter, accepted/drafted ratio, and draft cost normalized to shared+top-1 on the same platform. Increasing r raises acceptance, but the unique expert set expands much faster. DraftExpert keeps the expert footprint close to top-1 while reaching high acceptance. Model Drafter Unique Acc. Cost-CG Cost-MN DS top-1 84 22.5% 1.00 1.00 DS top-2 183 31.4% 1.36 1.47 DS top-3 244 45.9% 1.59 1.83 DS DraftExpert 88 85.0% 1.05 1.08 ML top-1 86 21.8% 1.00 1.00 ML top-2 186 30.2% 1.34 1.45 ML top-3 248 44.0% 1.61 1.86 ML DraftExpert 90 83.0% 1.06 1.09 Table 3: Draft-side acceptance–movement trade-off at K=4K=4. Cost is draft latency normalized to shared+top-1 on the same model and platform. MN grows faster because additional experts are staged from Flash. Verify-Side Control The verify-side experiment isolates the second condition: verifying a block should be close to one target step. In expert-offloaded MoE, fixed long verification blocks activate a growing union of target experts. Table 4 compares fixed-K verification with confidence–expansion truncation and prefetch. The reported cost is normalized to one-token AR decode on the same model and platform. Model Policy Cost-CG Cost-MN Hit/Waste DS Fixed K 2.35× 3.10× – DS Conf.+exp. 1.48× 1.70× – DS + Prefetch 1.25× 1.35× 88/12% ML Fixed K 2.55× 3.30× – ML Conf.+exp. 1.52× 1.76× – ML + Prefetch 1.28× 1.40× 86/14% Table 4: Verify-side cost control. Confidence–expansion truncation avoids low-value candidates that would add many verifier experts, and prefetch hides part of the remaining expert-load stall. These results explain why the final throughput gain is moderate rather than equal to the full resident-memory upper bound. DraftExpert reduces and overlaps expert movement, but exact verification still exposes misses, prefetch waste, and non-expert runtime overhead. Training Objective Ablation Finally, Table 5 keeps the training ablation next to the loss description. The metrics are platform-independent because the same frozen target and DraftExpert checkpoint are evaluated on both CG and MN; their platform-specific latency impact appears in Tables 2–4. Residual distillation teaches the draft expert to approximate omitted routed-expert residuals. Logit and token losses improve acceptance. Router-agreement distillation improves prediction of target experts during verification, which is required for high prefetch hit rate. Model Loss KL Acc. Router Hit DS Top-1 1.63 47.7% 0.62 61% DS + Residual 0.15 79.5% 0.68 69% DS + Logit/token 0.07 87.1% 0.72 73% DS + Router 0.07 87.0% 0.86 88% ML Top-1 1.70 45.0% 0.61 59% ML + Residual 0.18 76.5% 0.66 67% ML + Logit/token 0.08 84.0% 0.70 71% ML + Router 0.08 84.5% 0.85 86% Table 5: Training objective ablation. Router denotes the router-agreement loss. Residual/logit/token terms recover draft acceptance, while router agreement improves prefetch hit rate. Overall, DraftExpert turns expert-set expansion into the optimization target: it keeps draft movement nearly fixed, truncates verify blocks when marginal expert expansion is too high, and prefetches the experts most likely to be needed by exact verification. Related Work Speculative decoding. Speculative decoding accelerates language-model generation by drafting multiple candidate tokens and verifying them in one target-model pass (Leviathan et al. 2023; Chen et al. 2023; Miao et al. 2024). Subsequent systems improve the drafter with token trees, extra decoding heads, feature-level prediction, or early-exit self-drafting (Cai et al. 2024; Li et al. 2024; Zhang et al. 2024; Elhoushi et al. 2024). Most analyses are shaped by dense-model deployments where verification can reuse the same weights across a token block. DraftExpert focuses on the offloaded MoE case, where the draft stage and verify stage are both governed by unique expert movement. MoE models and MoE speculation. Sparse MoE models increase parameter count while activating only a subset of experts per token (Shazeer et al. 2017; Lepikhin et al. 2021; Fedus et al. 2022; Du et al. 2022; Jiang et al. 2024; Dai et al. 2024; DeepSeek-AI 2024; Liu et al. 2025). MoE-specific speculative decoding methods exploit routing structure, expert budgets, or target-model components to reduce draft cost (McDanel et al. 2026; Huang et al. 2025; Zheng et al. 2026; Bang et al. 2026). DraftExpert addresses the offloaded version of this problem: using more routed experts in the drafter can improve acceptance but directly increases CPU/Flash-to-accelerator transfers. Expert offloading and prefetching. Expert offloading systems move routed experts through heterogeneous memory when accelerator memory is insufficient. Prior systems study expert caching, storage-hierarchy partitioning, CPU–GPU orchestration, quantization, scheduling, and prefetching (Xue et al. 2024; Yi et al. 2023; Kamahori et al. 2025; Wang et al. 2025b; Chen et al. 2025; Wang et al. 2025a; Li et al. 2026). DraftExpert is complementary: it trains the drafter to have better router agreement, then uses those predictions for prefetching. The key distinction is that the learned drafter is designed to be both cheap and predictive. Dynamic speculation control. Dynamic speculation methods adapt draft length based on confidence, utility, or serving conditions (Saxena et al. 2025; Cheng et al. 2026). DraftExpert specializes this principle for offloaded MoE by combining token confidence with predicted marginal expert expansion. The controller stops drafts when an additional token is both uncertain and likely to add verification expert-loading cost. Distillation for drafters. Many speculative systems train a separate draft model or align a smaller model to the target, often building on knowledge distillation or self-distilled draft heads (Hinton et al. 2015; Cai et al. 2024; Li et al. 2024). DraftExpert instead distills the frozen target MoE into small per-layer draft experts while reusing the original model. The loss is designed for both sides of the system: residual/logit/token terms improve draft acceptance, while router-agreement terms improve verify-time expert prediction and prefetching. Method family Main idea Difference from DraftExpert Standard SD External cheap drafter Extra model stresses memory MoE-SD / MoE-Spec MoE-aware speculation or budgets Does not learn fixed-footprint experts S-MoE / SpecMoE Self-drafting from target experts Top-r draft footprint grows Offload systems Prefetch/schedule target experts Need accurate cheap router lookahead Dynamic control Adapt speculation length Usually not expert-expansion aware DraftExpert Distilled draft experts + cost gate Optimizes both draft and verify costs Table 6: Positioning of DraftExpert relative to speculative decoding and MoE offloading work. Conclusion This paper reframes speculative decoding for end-device MoE inference as a two-sided offload problem. On the draft side, shared+top-r self-drafting improves acceptance by loading more routed experts, making the drafter less cheap. On the verify side, parallel verification activates a growing union of target experts, and prefetching those experts requires router predictions that remain accurate despite cheap drafting. DraftExpert addresses both sides with a fixed-footprint shared+top-1+draft-expert drafter trained by self-distillation. Residual, logit, and token losses recover draft accuracy without increasing routed-expert usage; router-agreement loss improves prediction of verifier experts. The resulting predictions drive cost-aware dynamic truncation and draft-router prefetching while exact target verification is preserved. This aligns speculative decoding with the memory hierarchy of consumer CPU→ and mobile Flash→ MoE deployment. References J. Bang, E. Cho, R. Hwang, J. Chung, and M. Rhu (2026) SpecMoE: a fast and efficient mixture-of-experts inference via self-assisted speculative decoding. External Links: 2604.10152 Cited by: MoE models and MoE speculation.. T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao (2024) Medusa: simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, p. 5209–5235. Cited by: Speculative decoding., Distillation for drafters.. C. Chen, S. Borgeaud, G. Irving, J. Lespiau, L. Sifre, and J. Jumper (2023) Accelerating large language model decoding with speculative sampling. External Links: 2302.01318 Cited by: Introduction, Speculative decoding.. L. Chen, Z. Wen, T. Wu, X. Zhang, and C. Wu (2025) SP-MoE: speculative decoding and prefetching for accelerating MoE-based model inference. External Links: 2510.10302 Cited by: Expert offloading and prefetching.. X. Cheng, X. Yu, C. Shao, J. Li, Y. Xiong, et al. (2026) DSpark: confidence-scheduled speculative decoding with semi-autoregressive generation. External Links: Link Cited by: Dynamic speculation control.. D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang (2024) DeepSeekMoE: towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, p. 1280–1297. External Links: Document Cited by: MoE models and MoE speculation.. DeepSeek-AI (2024) DeepSeek-V2: a strong, economical, and efficient mixture-of-experts language model. External Links: 2405.04434 Cited by: Introduction, Experimental Evaluation, MoE models and MoE speculation.. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. Bosma, Z. Zhou, et al. (2022) GLaM: efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, Cited by: Introduction, MoE models and MoE speculation.. M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Roman, A. A. Aly, B. Chen, and C. Wu (2024) LayerSkip: enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, p. 12622–12642. External Links: Document Cited by: Speculative decoding.. W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. Cited by: Introduction, MoE models and MoE speculation.. G. Gerganov (2023) Llama.cpp: LLM inference in C/C++. External Links: Link Cited by: Setup and Metrics. G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, Cited by: Distillation for drafters.. Z. Huang, L. Zhu, Z. Zhan, T. Hu, W. Mao, X. Yu, Y. Liu, and T. Zhang (2025) MoESD: unveil speculative decoding’s potential for accelerating sparse MoE. External Links: 2505.19645 Cited by: MoE models and MoE speculation.. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, et al. (2024) Mixtral of experts. External Links: 2401.04088 Cited by: Introduction, MoE models and MoE speculation.. K. Kamahori, T. Tang, Y. Gu, K. Zhu, and B. Kasikci (2025) Fiddler: CPU-GPU orchestration for fast inference of mixture-of-experts models. In International Conference on Learning Representations, Cited by: Introduction, Expert offloading and prefetching.. D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen (2021) GShard: scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, Cited by: Introduction, MoE models and MoE speculation.. Y. Leviathan, M. Kalman, and Y. Matias (2023) Fast inference from transformers via speculative decoding. External Links: 2211.17192 Cited by: Introduction, Speculative decoding.. S. Li, J. Lin, D. Ge, and Y. Ye (2026) MoE-SpAc: efficient MoE inference based on speculative activation utility in heterogeneous edge scenarios. External Links: 2603.09983 Cited by: Expert offloading and prefetching.. Y. Li, F. Wei, C. Zhang, and H. Zhang (2024) EAGLE: speculative sampling requires rethinking feature uncertainty. External Links: 2401.15077 Cited by: Speculative decoding., Distillation for drafters.. J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan, Y. Chen, H. Zheng, Y. Liu, S. Liu, B. Yin, W. He, H. Zhu, Y. Wang, J. Wang, M. Dong, Z. Zhang, Y. Kang, H. Zhang, X. Xu, Y. Zhang, Y. Wu, X. Zhou, and Z. Yang (2025) Muon is scalable for LLM training. External Links: 2502.16982 Cited by: Introduction, Experimental Evaluation, MoE models and MoE speculation.. B. McDanel, S. Li, S. Surineni, and H. Khaitan (2026) MoE-Spec: expert budgeting for efficient speculative decoding. External Links: 2602.16052 Cited by: MoE models and MoE speculation.. X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y. Y. Wong, A. Zhu, L. Yang, X. Shi, C. Shi, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia (2024) SpecInfer: accelerating generative large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Cited by: Introduction, Speculative decoding.. A. Saxena, P. Tsai, H. Taneja, A. Jaleel, and M. Qureshi (2025) Utility-driven speculative decoding for mixture-of-experts. External Links: 2506.20675 Cited by: Dynamic speculation control.. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean (2017) Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. International Conference on Learning Representations. Cited by: Introduction, MoE models and MoE speculation.. W. Wang, J. Liu, X. Hou, X. Xia, P. Tang, M. Zhang, C. Li, and M. Guo (2025a) MoE-SpeQ: speculative quantized decoding with proactive expert prefetching and offloading for mixture-of-experts. External Links: 2511.14102 Cited by: Expert offloading and prefetching.. Z. Wang, Z. Zhang, Y. Zhou, Z. Wang, M. Zhou, P. Jiang, W. Cai, C. Huan, R. Gu, S. Zhong, and C. Tian (2025b) Accelerating mixture-of-experts inference by hiding offloading latency with speculative decoding. External Links: 2508.21706 Cited by: Expert offloading and prefetching.. L. Xue, Y. Fu, Z. Lu, L. Mai, and M. Marina (2024) MoE-Infinity: activation-aware expert offloading for efficient MoE serving. External Links: 2401.14361 Cited by: Introduction, Expert offloading and prefetching.. R. Yi, L. Guo, S. Wei, A. Zhou, S. Wang, and M. Xu (2023) EdgeMoE: fast on-device inference of MoE-based large language models. External Links: 2308.14352 Cited by: Introduction, Expert offloading and prefetching.. J. Zhang, J. Wang, H. Li, L. Shou, K. Chen, G. Chen, and S. Mehrotra (2024) Draft & verify: lossless large language model acceleration via self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, p. 11263–11282. External Links: Document Cited by: Speculative decoding.. P. Zheng, W. Xu, and H. Wang (2026) Self-speculative decoding for on-device MoE acceleration. In Proceedings of the ACM Web Conference, External Links: Document Cited by: MoE models and MoE speculation..