Paper deep dive
SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
Hongliang Cao, Xiuyu Li, Siqi Wang, Ziming Wang, Jingmin Zhu, Hailong Yang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified target verifies the result. On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97x and 1.72x as fast as vanilla EAGLE-3. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.15417v1
- Canonical: https://arxiv.org/abs/2610.15417v1
Trouble viewing inline? Open PDF directly →
Full Text
57,586 characters extracted from source content.
Expand or collapse full text
SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference Hongliang Cao Affiliation: Beihang University Email: caohongliang@buaa.edu.cn Xiuyu Li Affiliation: Monash University Email: hailong.yang@buaa.edu.cn Siqi Wang Affiliation: Beihang University Ziming Wang Affiliation: The Hong Kong University of Science and Technology Jingmin Zhu Affiliation: Monash University Hailong Yang †thanks: *Equal contribution.†thanks: †Corresponding author. Affiliation: Beihang University Abstract Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified target verifies the result. On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97× and 1.72× as fast as vanilla EAGLE-3. Code is available at https://github.com/Chlience/SpecFit. 1 Introduction Mixture-of-experts (MoE) layers route each token to a few expert feed-forward networks, decoupling the parameters a model stores from the computation it spends per token (Shazeer et al., 2017; Fedus et al., 2022; Wang et al., 2026), and have become a central design for recent large language models (Jiang et al., 2024; DeepSeek-AI et al., 2024; Yang et al., 2025; Li et al., 2026b). Sparse activation reduces computation, but not memory. When GPU memory cannot hold every expert, inference systems keep expert weights in host memory and cache, prefetch, or compute on the CPU the experts they need (Eliseev and Mazur, 2023; Xue et al., 2024; Kamahori et al., 2025), and host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding is a natural remedy. A small drafter proposes several tokens and the target verifies them together in one forward pass (Leviathan et al., 2023; Chen et al., 2023; Li et al., 2026a), so the weights that forward reads serve every committed token. This is especially attractive when those weights must first be copied from host memory (Svirschevski et al., 2024). Existing work improves speculative decoding along two lines. Tree-based methods draft candidate branches and choose which to verify by their expected acceptance, ranking branches by draft confidence (Li et al., 2024; Li et al., 2025), maximizing the expected acceptance length (Wang et al., 2025), or sizing the tree against measured hardware cost (Chen et al., 2024). Methods for MoE models act on the experts, prefetching the experts predicted during drafting (Chen et al., 2025), limiting the experts loaded during verification (McDanel et al., 2026), or training a resident draft expert (Han, 2026). Tree selection, however, treats verification cost as a function of how many tokens are verified, which holds in a dense model, where every token reads the same weights. An MoE model routes each verified token independently, so a draft tree activates the union of its tokens’ experts, including experts needed only by branches that will be rejected. A larger tree commits more tokens per cycle but also transfers more experts, so much of the speedup that its acceptance length promises is lost. Verification cost depends not only on how many draft tokens are verified but also on which ones, and a selector that values trees by acceptance and size cannot see this difference. We propose SpecFit, which selects the verification subtree that maximizes predicted throughput rather than acceptance length. It scores each ancestor-closed subtree of an unmodified EAGLE-3 candidate pool by the ratio of a draft-probability reward to a cycle-time estimate that grows with the number of distinct experts the subtree activates in each layer, with coefficients fitted once on calibration queries. An exact depth-first search with incremental expert bitmaps then decides both which nodes and how many to verify. Run after each draft level, the same search also decides when to stop drafting. The target is untouched: its router is unchanged, and every routed expert of every selected token is executed. The design rests on four properties. The expected committed length of a subtree is one plus the sum of its nodes’ commit probabilities, with no independence assumption; the expert-union cost is monotone and submodular; a smaller subtree raises throughput if and only if it saves a larger fraction of cycle time than of committed tokens; and selecting by the per-cycle ratio stays within a computable margin of the throughput-optimal rule. On 40 summarization articles and an 80-question GSM8K subset, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97× and 1.72× as fast as vanilla EAGLE-3. The gains come from cheaper verification and drafting rather than longer accepted paths, and hold on GSM8K, where EAGLE-3 already accepts long draft paths. In summary, we make the following contributions: • We identify and quantify a conflict between speculative decoding and expert offloading in MoE models: verifying a draft tree enlarges the union of activated experts, so that verification cost depends on which draft tokens are verified, not only on how many. • We propose SpecFit, which selects the verification subtree and the drafting depth by the ratio of a draft-probability reward to an expert-union cost and solves this selection exactly, without retraining the drafter or altering the target’s routing and expert execution. • We characterize when a smaller subtree raises throughput, bound the distance of per-cycle ratio selection from the throughput-optimal rule, and prove the size-limited selection NP-hard; on summarization and GSM8K, SpecFit improves decoding throughput over both vanilla EAGLE-3 and EAGLE-3 with an expert cache. 2 Background This section introduces the two techniques that SpecFit combines and shows why, together, they make verification cost depend on which draft tokens are verified. MoE inference with expert offloading. An MoE model has L MoE layers. Layer ℓ contains N expert feed-forward networks and a router that sends each token x to the set Eℓ(x)E_ (x) of the k experts with the highest gate scores; the layer output combines those experts’ outputs with the gate weights (Shazeer et al., 2017; Fedus et al., 2022). When the experts do not fit in GPU memory, an offloading runtime keeps them in host memory and uses a limited number of GPU slots per layer as an expert cache (Eliseev and Mazur, 2023; Xue et al., 2024; Song et al., 2024). A forward pass over a query Q must execute, in every layer, each expert in ⋃x∈QEℓ(x) _x∈ QE_ (x). An expert missing from the cache is copied host-to-device before the layer can finish and evicts another expert. The cache avoids a transfer when a later forward needs an expert that is still in the cache, but the per-layer union over the query remains the demand that each forward must meet. Tree-based speculative decoding. In EAGLE-2 and EAGLE-3 (Li et al., 2024; Li et al., 2025), the drafter starts from the pending root token s and expands its most likely continuations for several steps into a candidate pool V. Each node v∈Vv∈ V has a parent pa(v)∈V∪spa(v)∈ V∪\s\ and a path probability pvp_v, the product of draft conditional probabilities from the root to v. A subset S⊆VS V is then verified: the target processes S∪sS∪\s\ in one forward with a tree attention mask and, under greedy verification, accepts the longest root-to-node path whose tokens match its own predictions. The cycle commits the root and the accepted path, and the target’s next prediction becomes the new root. EAGLE-3 verifies the nodes with the highest pvp_v up to a fixed budget. If a cycle commits A tokens in time T, long-run decoding throughput is [A]/[T]E[A]/E[T], so choosing S trades committed tokens against cycle time. Why the verified set matters. Combining the two, write Eℓ(v)E_ (v) for EℓE_ of the token of node v in the tree forward, which depends on the node’s ancestors as well as on its own token. A forward that verifies S∪sS∪\s\ must execute D(S)=∑ℓ=1L|Eℓ(s)∪⋃v∈SEℓ(v)|D(S)= _ =1^L |E_ (s)∪ _v∈ SE_ (v) | (1) experts. Draft tokens at different positions and on different branches are routed separately, so the union grows with the tree, and a rejected branch pays for its experts as fully as an accepted one. Choosing the verified set under this cost is difficult for four reasons. First, the reward is uncertain: acceptance is revealed only by verification, so selection must rely on estimates available before the target runs. Second, the cost is not additive: tokens that share an expert in a layer pay for it once, so the expert union is a coverage function and a node’s marginal cost depends on the rest of the tree. Third, the choices are structured: a token can be committed only with its prefix, so only ancestor-closed subsets are meaningful. Fourth, the objective is a ratio: maximizing acceptance ignores cost, minimizing expert demand is trivially achieved by verifying no draft token, and the best tree size varies from cycle to cycle. The decision also lies on the critical path of every cycle, so it must be cheap, and it should not buy speed by pruning or substituting routed experts, which would change the target’s computation. Existing selectors value trees by acceptance and, where they model cost, by size or depth (Li et al., 2025; Wang et al., 2025; Chen et al., 2024); none accounts for the expert union. Section 3 formulates and solves this selection. 3 Method In each speculative cycle, SpecFit decides how deep EAGLE-3 drafts and which draft tokens the target verifies. While the drafter builds the candidate pool level by level, SpecFit selects the legal subtree with the largest ratio of predicted reward to predicted cycle time and stops drafting once that subtree takes no node from the newest level; the target then verifies the selected subtree and commits an accepted path (Figure 1). Section 3.1 defines the legal subtrees. Section 3.2 predicts a subtree’s reward from draft probabilities and its cycle time from the experts its nodes activate. Section 3.3 solves the resulting objective exactly, uses it to decide the drafting depth, relates it to the throughput-optimal rule, and describes the route information and target execution. 1DraftEAGLE-3 candidate pool VVsspvp_v012SelectSpecFit: exact search over legal subtreesssselected S Snot verifiedestimated experts of S Sℓ=1 =1ℓ=2 =2ℓ=L =Lrootadded by S Ssaved by pruningS^=argmaxS∈KR^(S)/T^(S) S= *arg\,max_S _K\; R(S)\,/\, T(S)R^(S)=1+∑v∈Spv,T^(S)=b|S|+βF(S) R(S)=1+ _v∈ Sp_v, T(S)=b_|S|+β F(S)3Verifyoffloaded MoE targetGPUtargetforwardon S Sexpert cacheHost memoryall expertsH2D over PCIemissescommit the accepted path4 Figure 1: One SpecFit decoding cycle. (1) EAGLE-3 drafts a candidate pool V with path probabilities pvp_v level by level; drafting stops once the selection takes no node from the newest level. (2) SpecFit searches the ancestor-closed subtrees exactly and selects the one with the largest R^/T R/ T; experts needed only by unselected nodes are excluded from F(S^)F( S). (3) The offloaded target verifies S S. Its own router decides which experts run, and experts missing from the expert cache are copied from host memory. (4) The accepted path is committed. 3.1 Selecting a Verification Subtree With the root s, candidate pool V, and parents pa(v)pa(v) of Section 2, a selected node requires its ancestors, which defines the legal family of subtrees with at most K draft nodes, K=S⊆V:|S|≤K,v∈S⇒pa(v)∈S∪s.A_K=\S V:\ |S|≤ K, v∈ S (v)∈ S∪\s\\. (2) Our candidate generator uses top-2 expansion, depth at most 5, and at most 18 nodes, with K=min(7,|V|)K= (7,|V|); the pool becomes smaller near the generation limit or when SpecFit stops drafting early (Section 3.3). 3.2 Estimating Reward and Expert Cost A useful subtree must retain likely accepted paths while limiting the work spent verifying all its nodes. Let Iv(S)I_v(S) indicate that draft node v is committed when S is verified. By linearity of expectation, the expected number of tokens committed per cycle is R(S)=1+∑v∈SPr(Iv(S)=1)≈ 1+∑v∈Spv=R^(S),R(S)=1+ _v∈ S (I_v(S)=1 )\;≈\;1+ _v∈ Sp_v= R(S), (3) in which the root contributes one and each draft node its commit probability. The first equality holds for any dependence among the commit events, with each probability allowed to depend on the whole tree. Before verification reveals the commit events, SpecFit estimates their probabilities with the path probabilities pvp_v of Section 2, which the drafter already computes while building the pool. Let E^ℓ,v E_ ,v be the estimate of Eℓ(v)E_ (v) supplied to the selector for node v in layer ℓ . Substituting these estimates into D of Equation 1 gives the union feature F, and SpecFit predicts cycle time from it and the node count, F(S)=∑ℓ=1L|E^ℓ,s∪⋃v∈SE^ℓ,v|,T^(S)=b|S|+βF(S).F(S)= _ =1^L | E_ ,s∪ _v∈ S E_ ,v |, T(S)=b_|S|+β F(S). (4) The node-count intercepts b0,…,b7b_0,…,b_7 and the shared coefficient β are fitted once by nonnegative least squares to cycle times measured on GSM8K calibration questions, then frozen (Appendix B). The fitted cycle time covers drafting, target verification, and commit; the selector’s own time is measured and included in the reported throughput. F(S)F(S) estimates D(S)D(S), the expert demand that the expert cache serves; every selected branch contributes to it, whether or not it is later accepted. SpecFit with the expert cache uses coefficients fitted with the cache active, so β is the average time per expert of demand with cache hits folded in; without the cache, it uses coefficients fitted with the cache cleared before every target forward. Either way, T T needs only the routes of the pool. For fixed estimates, F is a coverage function. Adding v to S increases it by ∑ℓ|E^ℓ,v∖U^ℓ(S)| _ | E_ ,v U_ (S)|, where U^ℓ(S) U_ (S) is the layer-ℓ union in Equation 4; since U^ℓ(S) U_ (S) only grows with S, F is monotone and submodular. Each node adds only experts new to the subtree, so its marginal expert demand can only fall as the subtree grows. 3.3 Selection, Verification, and Available Information SpecFit selects S^=argmaxS∈KR^(S)T^(S). S= *arg\,max_S _K R(S) T(S). (5) This objective jointly chooses node identities and cardinality. For positive rewards and times, a subtree with (R,T)(R,T) outperforms one with (R0,T0)(R_0,T_0) exactly when RT>R0T0⟺T0−T0>R0−R0. RT> R_0T_0 T_0-TT_0> R_0-RR_0. (6) Pruning nodes therefore raises throughput if and only if it saves a larger fraction of cycle time than of committed tokens. Equation 5 maximizes the ratio within each cycle, whereas throughput is a ratio of expectations over cycles (Section 2). To relate the two, let the cycle states, each a pool with its probabilities and routes, be independent and identically distributed and independent of earlier selections, and let Θ(π)=[R^(Sπ)]/[T^(Sπ)] (π)=E[ R(S_π)]/E[ T(S_π)] be the predicted throughput of a rule π that selects SπS_π in each cycle. The throughput-optimal rule selects S∗=argmaxS∈KR^(S)−λ∗T^(S)S =arg\,max_S _K R(S)-λ T(S) and attains the largest value λ∗=maxπΘ(π)λ = _π (π) (Dinkelbach, 1967). This rule needs λ∗λ , which depends on the distribution of all cycles; Equation 5 needs only the current pool. Proposition 1. Let ρ=R^(S^)/T^(S^)ρ= R( S)/ T( S) be the largest ratio in a cycle, and let Θ be the predicted throughput of Equation 5. Then 0≤λ∗−Θ≤[(ρ−λ∗)(T^(S∗)−T^(S^))][T^(S^)],0≤λ - ≤ E [(ρ-λ ) ( T(S )- T( S) ) ]E [ T( S) ], (7) and the quantity inside the expectation is nonnegative in every cycle. The margin vanishes when every cycle has the same largest ratio, and it is large only when cycles differ widely in their largest ratio and the two rules then select trees of widely different time. Appendix A proves the proposition and evaluates the margin on recorded pools. The same argument shows that, for fixed intercepts, a larger β can only lower the selected tree’s predicted expert demand per committed token, F(S^)/R^(S^)F( S)/ R( S). For general pools, the maximization in Equation 5 is hard. Proposition 2. Deciding whether some S∈KS _K reaches a given ratio R^(S)/T^(S) R(S)/ T(S) is NP-complete, even for a pool of depth one, one MoE layer, and equal intercepts. Without the size limit K and with intercepts affine and nondecreasing in |S||S|, the largest ratio is computable in polynomial time by parametric minimum cut (Picard, 1976; Gallo et al., 1989). SpecFit’s pools have at most 18 nodes and K≤7K≤ 7, so the legal family is small, and SpecFit solves Equation 5 exactly by enumeration (Algorithm 1 in Appendix A). A depth-first traversal reaches every subtree in KA_K exactly once and carries the per-layer expert unions as bitmaps from parent states. Equal ratios prefer larger reward, smaller union, fewer draft nodes, and then lexicographically smaller node IDs, fixing a unique choice among the maximizers of Equation 5 over KA_K. Adaptive drafting depth. The same objective also decides how deep EAGLE-3 drafts. The drafter adds one level of the pool per draft forward; let V≤dV_≤ d be the nodes of the first d levels. After drafting level d, SpecFit solves Equation 5 with V≤dV_≤ d in place of V. If the selected subtree contains a level-d node, SpecFit drafts the next level; otherwise it stops drafting and verifies that subtree. At the maximum depth it selects over the full pool. The stop rests on the tree structure: a deeper node can be selected only together with its level-d ancestor and adds no more reward than that ancestor, which the selection has already left out. The stopped selection equals the full-pool selection whenever the latter stays within the first d levels (Lemma 2 in Appendix A). Route information. In our experiments, E E comes from a route atlas, a separate target query over the full candidate pool on SpecFit’s reference prefix, prepared outside the request timer. During measurement, each cycle’s prefix and pool are checked against the atlas. Selection uses only the draft probabilities, the atlas routes, and the frozen coefficients. Target execution. After selection, the target verifies S S with its own router and executes every routed expert of every selected token; experts missing from the expert cache are copied from host memory. 4 Experiments We compare SpecFit with EAGLE-3 and autoregressive (AR) decoding, each without and with the expert cache, on summarization and GSM8K, decompose the gain with verification and transfer counters, vary the cache capacity and the GPU, and account for route prediction and selection cost. 4.1 Setup The target is Qwen3-30B-A3B-Instruct-2507 (Yang et al., 2025; Qwen Team, 2025) with its RedHatAI EAGLE-3 drafter (Red Hat AI, 2025), and the workloads are 40 CNN/DailyMail articles (Nallapati et al., 2016) and 80 GSM8K questions (Cobbe et al., 2021) from Spec-Bench (Xia et al., 2024). All measurements run at request batch size one, on a single A100 PCIe 40GB GPU except in Section 4.5. Every configuration, including AR, runs on our offload backend, which keeps all expert weights in pinned host memory. With the expert cache, each layer holds 48 experts in GPU slots with LRU replacement; without it, the cache is cleared before every target forward. Table 1 lists the compared methods; SpecFit and EAGLE-3 use the same drafter and differ only in drafting depth and in which and how many nodes are verified. EAGLE-3 without the cache is the vanilla EAGLE-3 of the introduction; the cached pair is our primary comparison, and SpecFit without qualification denotes its cached configuration. The cost coefficients are fitted on GSM8K calibration questions, with and without the cache, and used unchanged on both workloads. Speedups are relative to AR without the expert cache, and committed tokens are the tokens a verification cycle appends to the output, namely the root and the accepted draft nodes (Section 2). Appendix B gives the backend, timing, calibration, and data details. Table 1: Compared methods. Method Verification-node selection AR Serial target decoding, without a drafter EAGLE-3 Highest cumulative probability, at most 7 nodes SpecFit Equation 5 with adaptive depth, at most 7 nodes 4.2 Summarization To test whether SpecFit gains by leaving out draft nodes that do not pay for their experts, we first use summarization, on which EAGLE-3 accepts short draft paths. Each method runs on the 40 CNN/DailyMail articles without and with the expert cache; Figure 2 reports decode throughput. With expert cacheWithout expert cache02468108.275.546.804.216.644.67Decode TPSSpecFitEAGLE-3AR Figure 2: Results on the CNN/DailyMail summarization dataset. SpecFit decodes 8.27 tokens/s, 21.6% above cached EAGLE-3 (19.9% including prefill) and 1.77× AR. Without the cache, it decodes 31.6% faster than vanilla EAGLE-3, which is itself slower than AR (0.90×). The verification and transfer counters in Table 2 show where the gain comes from. EAGLE-3 verifies seven draft nodes per cycle but accepts only 1.54 of them. SpecFit verifies 2.88 draft nodes instead. It stops drafting before the fifth level in 85.8% of cycles, which cuts drafting time per cycle by 44.9%. It commits 13.1% fewer tokens per cycle than cached EAGLE-3, but each cycle takes 28.5% less decode time, so it decodes 21.6% faster. Verification accounts for 78.3% of the time saved per cycle, and SpecFit transfers 22.1% fewer experts per output token. This is the trade the objective makes: a node with a low acceptance probability adds little to the expected commit but still adds its experts to the verified union, so leaving such nodes out shortens the cycle by a larger fraction than it reduces committed tokens. Table 2: Verification and transfer counters on the CNN/DailyMail summarization dataset. Method Cache Draft nodes Committed Cycles ms/cycle H2D/token AR No – 1.001.00 9,652 214.2 400.7 EAGLE-3 No 7.007.00 2.542.54 3,823 600.9 486.5 SpecFit No 2.292.29 2.082.08 4,702 373.6 347.7 AR Yes – 1.001.00 9,652 150.6 49.7 EAGLE-3 Yes 7.007.00 2.542.54 3,823 371.7 73.4 SpecFit Yes 2.882.88 2.212.21 4,389 265.8 57.2 4.3 GSM8K To test whether SpecFit still gains when EAGLE-3 accepts long draft paths, we run the same comparison on the 80 GSM8K questions; Figure 3 reports decode throughput. With expert cacheWithout expert cache02468101210.797.0810.086.266.534.70Decode TPSSpecFitEAGLE-3AR Figure 3: Results on the GSM8K dataset. SpecFit decodes 10.79 tokens/s, 7.0% above cached EAGLE-3 (6.3% including prefill) and 2.29× AR; without the cache, it decodes 13.1% faster than vanilla EAGLE-3. The verification and transfer counters in Table 3 show where this gain comes from. With the cache, EAGLE-3 commits 4.35 tokens per cycle, against 2.54 on summarization, so fewer of its nodes are worth leaving out. SpecFit still verifies 5.09 draft nodes instead of 7.00 and commits 7.8% fewer tokens per cycle, but each cycle takes 13.8% less decode time, and it transfers 11.6% fewer experts per output token; adaptive depth stops drafting before the fifth level in 33.7% of cycles and cuts drafting time per cycle by 11.7%. The gain is smaller than on summarization but has the same source, shorter cycles rather than longer accepted paths: cycle time falls by a larger fraction than committed tokens, as Equation 6 requires. Table 3: Verification and transfer counters on the GSM8K dataset. Method Cache Draft nodes Committed Cycles ms/cycle H2D/token AR No – 1.001.00 17,791 212.5 398.2 EAGLE-3 No 7.007.00 4.354.35 4,097 692.3 345.7 SpecFit No 4.274.27 3.813.81 4,662 536.7 294.1 AR Yes – 1.001.00 17,791 153.1 76.5 EAGLE-3 Yes 7.007.00 4.354.35 4,097 429.8 92.7 SpecFit Yes 5.095.09 4.014.01 4,407 370.4 82.0 4.4 Sensitivity to Offload Pressure To test whether SpecFit’s gain grows as the expert cache shrinks, we rerun EAGLE-3 and SpecFit on the 40 summarization articles with 24 and 12 expert slots per layer and compare them with the 48 slots of the main runs and with the configurations without the cache (0 slots); Figure 4a reports decode throughput. 482412002468108.277.686.925.546.806.185.364.21Expert slots per layerDecode TPSSpecFitEAGLE-3 (a) Decode throughput. 0200400600800265.8286.3317.5373.6371.7409.3471.8600.9ms/cycleSpecFitEAGLE-34824120020040060057.2140.4227.4347.773.4200.3340.0486.5Expert slots per layerH2D/token (b) Cycle time and transfers. Figure 4: Results on CNN/DailyMail with different numbers of expert slots per layer. SpecFit’s decoding gain over EAGLE-3 grows from 21.6% at 48 slots to 24.3% and 29.2% at 24 and 12 slots, and it is 31.6% without the cache. With the cache, SpecFit verifies 2.88 draft nodes and commits 13.1% fewer tokens per cycle than EAGLE-3 at every capacity, so the growing gain comes from shorter cycles. As Figure 4b shows, at 48, 24, and 12 slots, its time per cycle is 28.5%, 30.1%, and 32.7% shorter, and it transfers 22.1%, 29.9%, and 33.1% fewer experts per output token. Because the cache capacity does not enter selection, the trees at 48, 24, and 12 slots are the same; for these capacities, let a configuration spend decode time T=B+τHT=B+τ H on R decode tokens, a fixed part B plus H expert transfers of time τ each. For SpecFit and EAGLE-3 (subscripts S and E), RS/TSRE/TE=(1−w)RS/BSRE/BE+wRS/HSRE/HE,w=τHSTS, R_S/T_SR_E/T_E=(1-w)\, R_S/B_SR_E/B_E+w\, R_S/H_SR_E/H_E, w= τ H_ST_S, (8) so the gain averages SpecFit’s advantages in tokens per unit of fixed time and per decode-stage transfer, weighted by its transfer-time share w. If B and τ do not depend on capacity, a smaller cache raises w, increasing the weight of the second term. The full-request counters, including prefill, show token-per-transfer advantages of 28.3%, 42.7%, and 49.5% at 48, 24, and 12 slots (Figure 4b), above each measured decoding gain. 4.5 Another GPU To test whether the gain carries over to a GPU with less memory and a slower host link, we repeat the GSM8K comparison on a single RTX 4090 24GB. It holds 32 expert slots per layer instead of 48, and with the cache cleared, the fitted time per loaded expert is 0.78 ms, against 0.41 ms on the A100. The cost coefficients are refitted on this GPU with the same protocol (Appendix B), and the comparison uses the 40 odd-numbered of the 80 questions; Table 4 reports the results. Table 4: Results on the GSM8K dataset on a single RTX 4090 24GB. Method Cache Decode TPS Speedup Draft nodes Committed Cycles ms/cycle H2D/token AR No 2.672.67 1.00×1.00× – 1.001.00 8,714 374.4 398.6 EAGLE-3 No 3.293.29 1.23×1.23× 6.996.99 4.324.32 2,009 1,306.7 346.4 SpecFit No 3.893.89 1.46×1.46× 3.783.78 3.683.68 2,339 942.8 288.0 AR Yes 4.244.24 1.59×1.59× – 1.001.00 8,714 236.1 126.9 EAGLE-3 Yes 6.056.05 2.27×2.27× 6.996.99 4.324.32 2,009 711.1 150.0 SpecFit Yes 6.706.70 2.51×2.51× 4.534.53 4.004.00 2,162 593.7 128.3 With the expert cache, SpecFit decodes 6.70 tokens/s on this GPU, 10.8% above cached EAGLE-3 and 2.51× AR; without the cache, it decodes 18.1% faster than vanilla EAGLE-3. As on the A100, it verifies fewer draft nodes (4.53 instead of 6.99) and commits 7.5% fewer tokens per cycle, but each cycle takes 16.5% less decode time, and it transfers 14.5% fewer experts per output token. SpecFit thus makes the same trade on a GPU whose memory and transfer costs differ. 4.6 Route Prediction and Selection Cost Our measurements take routes from the atlas, outside the request timer, so that selection is evaluated on its own; a deployment replaces the atlas with a route predictor. Lightweight expert predictors reach 84.7% average accuracy (Song et al., 2024) and 94.7% on Qwen3-30B-A3B (Zhu et al., 2025). SpecFit reads only the predicted expert sets E E, and verification still executes every expert that the target’s router chooses, so route errors can change only which nodes are selected. With a simulated predictor of 60% accuracy, which replaces each routed expert by a random expert outside the true top-8 with probability 40%, SpecFit still decodes 2.26× as fast as AR on 40 GSM8K questions, against 2.27× with exact routes, because the correct predictions still show which draft tokens share experts, while wrong ones rarely coincide across nodes and add a nearly uniform cost per node. A linear-head predictor, which maps each draft node’s hidden state to logits over all 48×12848× 128 experts, has 12.6M parameters, the memory of about three expert slots, and needs about 25 MFLOP per node; applied after each drafted level, it adds an estimated 0.1–0.2 ms per cycle. SpecFit’s own selection, included in every measurement above, takes a median of 2.82 ms per cycle on GSM8K (0.7% of decode time) and 1.43 ms on summarization (0.6%), both small compared with the 59 ms by which a SpecFit cycle is shorter than a cached EAGLE-3 cycle on GSM8K. Training route predictors is left to future work. 5 Related Work Expert offloading. Offloading systems keep MoE expert weights in host memory and cache and prefetch the experts they expect to need (Eliseev and Mazur, 2023; Xue et al., 2024; Song et al., 2024), or compute some experts on the CPU (Kamahori et al., 2025). SpecFit runs on an on-demand expert cache without prefetching; it changes which tokens create expert demand, not how it is served. Draft-tree construction and selection. Leviathan et al. (2023) and Chen et al. (2023) develop draft-and-verify acceleration with target-distribution-preserving sampling rules under their assumptions. EAGLE-2 constructs context-dependent trees ranked by draft confidence, and EAGLE-3 develops the draft model and feature interface (Li et al., 2024; Li et al., 2025). OPT-Tree maximizes the expected acceptance length (Wang et al., 2025), Sequoia chooses tree size and depth against measured hardware cost (Chen et al., 2024), and SpecExec verifies large trees to amortize weight loading for dense models with offloaded weights (Svirschevski et al., 2024). SpecFit keeps a fixed EAGLE-3 drafter and candidate generator and changes only how deep it drafts and which subset is sent to verification, valuing both by an expert-union cost rather than by size or shape. Speculative decoding for MoE models. SP-MoE prefetches experts predicted during drafting and pipelines their execution (Chen et al., 2025). MoE-Spec limits the experts loaded during verification and thereby alters the target computation (McDanel et al., 2026), whereas SpecFit executes all routed experts for its selected tokens. DraftExpert combines a trained resident draft expert with expansion-aware truncation and prefetching for offloaded inference (Han, 2026). SpecFit instead evaluates a complete-subtree reward/cost ratio over a bounded feasible family, with cardinality included in the search, and trains no model component. 6 Conclusion Under expert offloading, speculative verification in MoE models pays for every expert that the verified tree activates. SpecFit therefore selects each cycle’s verification subtree and drafting depth by the ratio of a draft-probability reward to an expert-union cost, solved exactly over ancestor-closed subtrees, without changing the drafter, the router, or expert execution. With Qwen3-30B-A3B, it verifies fewer draft nodes when their expert cost outweighs their expected tokens. On summarization, where EAGLE-3 accepts short draft paths, this raises decoding throughput by 21.6% over EAGLE-3 with the same expert cache. On GSM8K, where EAGLE-3 accepts long paths, each cycle is still 13.8% shorter and the gain is 7.0%, to 2.29× that of AR decoding without the cache. As the expert cache shrinks from 48 to 12 slots per layer, the summarization gain grows to 29.2%, and on an RTX 4090, whose host link is slower, the GSM8K gain is 10.8%. AI use statement We used AI assistants for software development, experiment inspection, mathematical exposition, literature search, and drafting of the manuscript. The authors reviewed all AI-assisted content and take full responsibility for the claims, references, and results in this paper. Ethics statement This work speeds up decoding of existing MoE language models whose experts are offloaded to host memory. It uses a publicly released target model, drafter, and datasets, involves no human subjects, and collects no new data. SpecFit changes only which draft tokens are sent to verification, and the target model verifies them unmodified, so the method adds no capability or risk beyond those of the model it accelerates. Reproducibility statement Section 4.1 and Appendix B give the model, the workloads, the expert cache configuration, the calibration protocol, and the fitted cost coefficients. Section 3 and Algorithm 1 specify the selection and the drafting-depth rule, and Appendix A states the assumptions and gives complete proofs of both propositions and both lemmas. The supplementary material contains the source code, with the configuration files and scripts that run the calibration and the comparisons in Section 4. References Chen et al. (2023) C. Chen, S. Borgeaud, G. Irving, J. Lespiau, L. Sifre, and J. Jumper Accelerating Large Language Model Decoding with Speculative Sampling. External Links: 2302.01318, Link Cited by: §1, §5. Chen et al. (2025) L. Chen, Z. Wen, T. Wu, X. Zhang, and C. Wu SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference. External Links: 2510.10302, Link Cited by: §1, §5. Chen et al. (2024) Z. Chen, A. May, R. Svirschevski, Y. Huang, M. Ryabinin, Z. Jia, and B. Chen Sequoia: scalable and robust speculative decoding. In Advances in Neural Information Processing Systems, Vol. 37, p. 129531–129563. External Links: Document Cited by: §1, §2, §5. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, et al. Training Verifiers to Solve Math Word Problems. External Links: 2110.14168, Link Cited by: §4.1. DeepSeek-AI et al. (2024) DeepSeek-AI, A. Liu, B. Feng, et al. DeepSeek-V3 Technical Report. External Links: 2412.19437, Link Cited by: §1. Dinkelbach (1967) W. Dinkelbach On Nonlinear Fractional Programming. Management Science 13 (7), p. 492–498. External Links: Document, Link Cited by: §A.1, §3.3. Eliseev and Mazur (2023) A. Eliseev and D. Mazur Fast Inference of Mixture-of-Experts Language Models with Offloading. External Links: 2312.17238, Link Cited by: §1, §2, §5. Fedus et al. (2022) W. Fedus, B. Zoph, and N. Shazeer Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. External Links: Link Cited by: §1, §2. Gallo et al. (1989) G. Gallo, M. D. Grigoriadis, and R. E. Tarjan A Fast Parametric Maximum Flow Algorithm and Applications. SIAM Journal on Computing 18 (1), p. 30–55. External Links: Document, Link Cited by: §A.3, Proposition 2. Han (2026) D. Han DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference. External Links: 2607.24434, Link Cited by: §1, §5. Jiang et al. (2024) A. Q. Jiang, A. Sablayrolles, A. Roux, et al. Mixtral of Experts. External Links: 2401.04088, Link Cited by: §1. Kamahori et al. (2025) K. Kamahori, T. Tang, Y. Gu, K. Zhu, and B. Kasikci Fiddler: CPU-GPU orchestration for fast inference of mixture-of-experts models. In International Conference on Learning Representations, p. 56099–56115. External Links: Link Cited by: §1, §5. Karp (1972) R. M. Karp Reducibility among Combinatorial Problems. In Complexity of Computer Computations, p. 85–103. External Links: Document, Link Cited by: §A.3. Leviathan et al. (2023) Y. Leviathan, M. Kalman, and Y. Matias Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 19274–19286. External Links: Link Cited by: §1, §5. Li et al. (2026a) X. Li, J. Zhang, M. Yi, Y. Li, L. Wang, Y. Wang, and J. Fan ETS: energy-guided test-time scaling for training-free RL alignment. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §1. Li et al. (2026b) Y. Li, X. Li, M. Yi, J. Wang, L. Zhang, Z. Xing, and Z. Chen Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment. External Links: 2607.04728, Link Cited by: §1. Li et al. (2024) Y. Li, F. Wei, C. Zhang, and H. Zhang EAGLE-2: faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 7421–7432. External Links: Document, Link Cited by: §1, §2, §5. Li et al. (2025) Y. Li, F. Wei, C. Zhang, and H. Zhang EAGLE-3: scaling up inference acceleration of large language models via training-time test. In Advances in Neural Information Processing Systems, Vol. 38, p. 136737–136756. External Links: Document Cited by: §1, §2, §2, §5. McDanel et al. (2026) B. McDanel, S. Li, S. Surineni, and H. Khaitan MoE-Spec: Expert Budgeting for Efficient Speculative Decoding. External Links: 2602.16052, Link Cited by: §1, §5. Nallapati et al. (2016) R. Nallapati, B. Zhou, C. dos Santos, Ç. Gülçehre, and B. Xiang Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, p. 280–290. External Links: Document, Link Cited by: §4.1. Picard (1976) J. Picard Maximal Closure of a Graph and Applications to Combinatorial Problems. Management Science 22 (11), p. 1268–1272. External Links: Document, Link Cited by: §A.3, Proposition 2. Qwen Team (2025) Qwen Team Qwen3-30B-A3B-Instruct-2507 model card. External Links: Link Cited by: §4.1. Red Hat AI (2025) Red Hat AI Qwen3-30B-A3B-Instruct-2507-speculator.eagle3 model card. External Links: Link Cited by: §4.1. Shazeer et al. (2017) N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Song et al. (2024) X. Song, Z. Zhong, R. Chen, and H. Chen ProMoE: Fast MoE-based LLM Serving using Proactive Caching. External Links: 2410.22134, Link Cited by: §2, §4.6, §5. Svirschevski et al. (2024) R. Svirschevski, A. May, Z. Chen, B. Chen, Z. Jia, and M. Ryabinin SpecExec: massively parallel speculative decoding for interactive LLM inference on consumer devices. In Advances in Neural Information Processing Systems, Vol. 37, p. 16342–16368. External Links: Document Cited by: §1, §5. Wang et al. (2026) C. Wang, W. He, J. Zhu, Q. Yin, H. Xu, and X. Li RailWave: Adaptive Spatial and Temporal Scheduling for Expert-Parallel Communication. External Links: 2610.03415, Link Cited by: §1. Wang et al. (2025) J. Wang, Y. Su, J. Li, Q. Xia, Z. Ye, X. Duan, Z. Wang, and M. Zhang OPT-Tree: speculative decoding with adaptive draft tree structure. Transactions of the Association for Computational Linguistics 13, p. 188–199. External Links: Document, Link Cited by: §1, §2, §5. Xia et al. (2024) H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui Unlocking efficiency in large language model inference: a comprehensive survey of speculative decoding. In Findings of the Association for Computational Linguistics: ACL 2024, p. 7655–7671. External Links: Document, Link Cited by: §4.1. Xue et al. (2024) L. Xue, Y. Fu, Z. Lu, C. Sun, L. Mai, and M. Marina MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache. External Links: 2401.14361, Link Cited by: §1, §2, §5. Yang et al. (2025) A. Yang, A. Li, B. Yang, et al. Qwen3 Technical Report. External Links: 2505.09388, Link Cited by: §1, §4.1. Zhu et al. (2025) S. Zhu, S. Bohl, R. Oester, and G. Alonso Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models. External Links: 2511.10676, Link Cited by: §4.6. Appendix A Selection Theory and Proofs A.1 Throughput of a Selection Rule A cycle state comprises the pool with its probabilities, routes, and legal family KA_K, and a rule π selects Sπ∈KS_π _K from the current state. Suppose that the states of successive cycles are independent and identically distributed and independent of earlier selections. If R R and T T equal the expected committed tokens and cycle time given the state, the renewal–reward theorem gives the long-run throughput Θ(π)=[R^(Sπ)]/[T^(Sπ)] (π)=E[ R(S_π)]/E[ T(S_π)]; otherwise Θ(π) (π) is the throughput that the proxies predict. Let h(λ)=[maxS∈K(R^(S)−λT^(S))].h(λ)=E [ _S _K ( R(S)-λ T(S) ) ]. (9) The intercepts are positive, so T T is bounded below by a positive constant, and h is continuous and strictly decreasing, with h(0)≥1h(0)≥ 1 and h(λ)→−∞h(λ)→-∞; let λ∗λ be its root. Every rule satisfies [R^(Sπ)−λ∗T^(Sπ)]≤h(λ∗)=0E[ R(S_π)-λ T(S_π)]≤ h(λ )=0, so Θ(π)≤λ∗ (π)≤λ , and the rule selecting S∗S attains equality. Hence λ∗=maxπΘ(π)λ = _π (π) (Dinkelbach, 1967). Lemma 1. In any cycle, let SλS_λ maximize R^(S)−λT^(S) R(S)-λ T(S) over KA_K. Then (ρ−λ)(T^(Sλ)−T^(S^))≥0(ρ-λ) ( T(S_λ)- T( S) )≥ 0. In particular, T^(S∗)≤T^(S^) T(S )≤ T( S) when ρ<λ∗ρ<λ , and T^(S∗)≥T^(S^) T(S )≥ T( S) when ρ>λ∗ρ>λ . Proof. Since ρ is the largest ratio, R^(S)−ρT^(S)≤0 R(S)-ρ T(S)≤ 0 for every S, with equality at S S, so S S maximizes R^−ρT R-ρ T. Adding R^(Sλ)−λT^(Sλ)≥R^(S^)−λT^(S^) R(S_λ)-λ T(S_λ)≥ R( S)-λ T( S) and R^(S^)−ρT^(S^)≥R^(Sλ)−ρT^(Sλ) R( S)-ρ T( S)≥ R(S_λ)-ρ T(S_λ) gives the claim. ∎ See 1 Proof. The lower bound holds because λ∗λ is the largest predicted throughput. Since R^(S^)=ρT^(S^) R( S)=ρ T( S), λ∗−Θ=−[(ρ−λ∗)T^(S^)]/[T^(S^)]λ - =-E[(ρ-λ ) T( S)]/E[ T( S)]. Since R^(S∗)≤ρT^(S∗) R(S )≤ρ T(S ), 0=h(λ∗)=[R^(S∗)−λ∗T^(S∗)]≤[(ρ−λ∗)T^(S∗)]0=h(λ )=E[ R(S )-λ T(S )] [(ρ-λ ) T(S )]. Adding this nonnegative quantity to the numerator gives Equation 7, and Lemma 1 with λ=λ∗λ=λ gives the sign of each term. If ρ is the same in every cycle, R^(S)≤ρT^(S) R(S)≤ρ T(S) holds for every selection, so λ∗=ρ=Θλ =ρ= . ∎ Expert demand and β. Fix the intercepts, and let S^1 S_1 and S^2 S_2 be the selections under β1<β2 _1< _2. Write the inverse ratio as T^(S)/R^(S)=x(S)+βy(S) T(S)/ R(S)=x(S)+β y(S) with x(S)=b|S|/R^(S)x(S)=b_|S|/ R(S) and y(S)=F(S)/R^(S)y(S)=F(S)/ R(S). Each selection minimizes the inverse ratio under its own coefficient, so x(S^1)+β1y(S^1)≤x(S^2)+β1y(S^2)x( S_1)+ _1y( S_1)≤ x( S_2)+ _1y( S_2) and x(S^2)+β2y(S^2)≤x(S^1)+β2y(S^1)x( S_2)+ _2y( S_2)≤ x( S_1)+ _2y( S_1). Adding them gives (β2−β1)(y(S^2)−y(S^1))≤0( _2- _1) (y( S_2)-y( S_1) )≤ 0, so F(S^2)/R^(S^2)≤F(S^1)/R^(S^1)F( S_2)/ R( S_2)≤ F( S_1)/ R( S_1). A.2 Adaptive Depth Lemma 2. Let S^d S_d be the selection over V≤dV_≤ d, with S^0 S_0 the root-only choice. The legal families are nested in d, so the largest ratio is nondecreasing in d. Adaptive depth stops at the first level d with S^d=S^d−1 S_d= S_d-1, and its selection equals the full-pool selection S S unless S S contains a node deeper than d. Proof. A legal subtree of V≤dV_≤ d has at most min(7,|V≤d|) (7,|V_≤ d|) nodes and is therefore a legal subtree of V≤d′V_≤ d for every d′>d >d, which gives the nesting and the monotone largest ratio. The tie rule orders all subtrees independently of the pool, so a subtree that is optimal in a family is optimal in every subfamily containing it. If S^d S_d has no level-d node, it lies in the family of V≤d−1V_≤ d-1 and hence equals S^d−1 S_d-1; conversely, S^d−1 S_d-1 has no level-d node. Likewise, if S^⊆V≤d S V_≤ d, then S^d=S S_d= S. ∎ In cycles whose largest ratio is below λ∗λ , the throughput-optimal rule selects no more predicted time than Equation 5 (Lemma 1), so a stopped tree with less predicted time than S S departs from Equation 5 in the same direction as that rule. A.3 Complexity and Exact Search See 2 Proof. A subtree is a certificate checkable in polynomial time, so the problem is in NP. We reduce from CLIQUE (Karp, 1972): given a graph with n vertices, m edges, and an integer 2≤r≤n2≤ r≤ n, decide whether some r vertices are pairwise adjacent. Let k=r(r−1)/2k=r(r-1)/2 and p=1/(m+1)p=1/(m+1). The pool has one node per edge, each a child of the root. The single MoE layer has one expert per vertex; the node of edge u,w\u,w\ routes to u,w\u,w\, and the root routes to no expert. Set pv=p_v=p for every node, bj=b>0b_j=b>0 for every j, K=kK=k, 0<β<bp/(n(1+(k−1)p))0<β<bp/ (n(1+(k-1)p) ), and threshold θ=(1+kp)/(b+βr)θ=(1+kp)/(b+β r). Then F(S)F(S) is the number of vertices covered by the edges in S, and the edges of an r-clique reach θ. Conversely, a subtree with fewer than k nodes has ratio at most (1+(k−1)p)/b<θ(1+(k-1)p)/b<θ by the choice of β, so a subtree reaching θ has k edges covering at most r vertices. Since q vertices span at most q(q−1)/2q(q-1)/2 edges, these edges cover exactly r vertices and are all pairs among them, an r-clique. Without the size limit, let bj=b0+ajb_j=b_0+aj with b0>0b_0>0 and a≥0a≥ 0. Apart from the root’s experts, F(S)F(S) counts the pairs (ℓ,e)( ,e) such that some v∈Sv∈ S routes to expert e in layer ℓ . For fixed λ≥0λ≥ 0, maximizing R^(S)−λT^(S) R(S)-λ T(S) over ancestor-closed S is then a maximum-weight closure problem (Picard, 1976). Its network has a vertex for each node and each pair, an arc of capacity pvp_v from the source to each node, arcs of capacity λaλ a from each node and λβλβ from each pair to the sink, and infinite arcs from each node to its parent and to its pairs; the source side of a minimum cut is an optimal subtree with its pairs. Source capacities are constant and sink capacities are nondecreasing in λ, which after reversing all arcs is the form handled by Gallo et al. (1989): one parametric maximum flow yields nested minimum cuts for all λ, at most as many as the network has vertices. At λ equal to the largest ratio, the maximum of R^−λT R-λ T is zero and is attained only by subtrees of that ratio, so the best ratio among the nested cuts is the largest ratio. ∎ Exact search. Algorithm 1 gives the search of Section 3.3. Pool node IDs put parents before descendants, and the depth-first traversal adds nodes in increasing ID order, each only when its parent is present, so it reaches every subtree in KA_K exactly once. Algorithm 1 SpecFit selector 0: Pool V, root s, atlas E E, frozen coefficients (b,β)(b,β) 1: Read cumulative draft probabilities pvp_v and encode expert bitmaps BvB_v 2: Initialize the incumbent to the root-only choice 3: for all ancestor-closed S⊆VS V, |S|≤min(7,|V|)|S|≤ (7,|V|), in depth-first order do 4: Carry B(S)=BsOR⋁v∈SBvB(S)=B_s OR _v∈ SB_v from the parent state 5: r←1+∑v∈Spvr← 1+ _v∈ Sp_v; f←popcount(B(S))f (B(S)) 6: t←b|S|+βft← b_|S|+β f 7: Update incumbent by (r/t,r,−f,−|S|)(r/t,r,-f,-|S|) and the node-ID tie rule 8: end for 9: return Incumbent subtree A.4 Evaluation on Recorded Pools We evaluate these quantities on the candidate pools that SpecFit drafted in every cycle of its runs with the expert cache (Appendix B), with the coefficients fitted with the cache: 4,389 cycles on the 40 summarization articles and 4,407 cycles on the 80 GSM8K questions. The legal family of a cycle is that of its drafted pool, expectations are taken over the recorded cycles with the trajectory held fixed, and λ∗λ is computed by Dinkelbach’s iteration (Table 5). The optimality gap of Equation 5 is below 1% of λ∗λ and lies within the bound of Equation 7, although the two rules select different trees in about a quarter of the cycles. With the recorded acceptances in place of R R, S∗S commits 0.5% and 0.7% more tokens per unit of predicted time than S S. Doubling β lowers both the expert demand per committed token and the number of selected nodes. Drafting stops before level 5 in 85.8% of the summarization cycles and 33.7% of the GSM8K cycles. Table 5: Selection theory evaluated on the candidate pools drafted by SpecFit, with the coefficients fitted with the cache. Percentages of λ∗λ refer to the predicted throughput. Quantity Summarization GSM8K Cycles 4,389 4,407 λ∗−Θλ - (% of λ∗λ ) 0.74 0.95 Bound of Equation 7 (% of λ∗λ ) 1.38 1.75 Cycles with S∗≠S^S ≠ S (%) 26.6 24.5 Coefficient of variation of ρ 0.31 0.25 Gain of S∗S with recorded acceptances (%) 0.53 0.69 Mean F(S^)/R^(S^)F( S)/ R( S) at β / 2β2β 365.5 / 351.5 305.7 / 298.0 Mean selected nodes at β / 2β2β 2.88 / 2.35 5.09 / 4.54 Mean drafted levels 2.86 4.39 Cycles stopped before level 5 (%) 85.8 33.7 Appendix B Experimental Details Model and data. The target model has 48 MoE layers with 128 experts per layer and top-8 routing. The summarization workload consists of the 40 even-numbered of the 80 CNN/DailyMail articles in Spec-Bench, with inputs of 156–1,496 tokens and a limit of 2048 input plus output tokens. The GSM8K workload is the 80-question subset of Spec-Bench, with a limit of 512 input plus output tokens. Backend and baselines. The offload backend replaces the expert modules of the Hugging Face Transformers Qwen3-MoE model. With the expert cache, missing experts are loaded on demand, with at most two copies in flight. A configuration without the expert cache clears the cache before every target forward, including prefill, and the clearing is timed; an expert shared by tokens of one forward is still loaded once, and the KV cache is kept. EAGLE-3 drafts the full five-level pool with the same generator as SpecFit and verifies all nodes when the pool has fewer than seven nodes near the generation limit. Timing and aggregation. Each method’s request stream begins with an empty expert cache, whose contents carry over between questions, while KV and draft state are new for every question. Measured request time includes prefill, drafting, atlas conversion, selection, target verification, commit, and required synchronization; model loading, atlas acquisition, and post-request scoring are excluded. For output count NiN_i, request time tireqt_i req, and prefill time tipret_i pre, the throughput measures are TPSdec=∑i(Ni−1)∑i(tireq−tipre),TPSreq=∑iNi∑itireq.TPS_ dec= _i(N_i-1) _i(t_i req-t_i pre), _ req= _iN_i _it_i req. (10) The first output token is attributed to prefill. Draft nodes, committed tokens, and ms/cycle are means per target verification cycle; an AR cycle is one decode forward and commits one token. H2D/token counts expert-weight transfers per output token, including prefill. Calibration. The probes use the states of GSM8K questions 401–403 at cycle boundaries 0, 4, and 8. In each of these nine states, two probe trees are measured for each size k=0,…,7k=0,…,7, a probability-prefix tree and a minimum-expert-union tree, twice each and without clearing the expert cache. Averaging the repeats gives 9×8×2=1449× 8× 2=144 points (ki,Fi,yi)(k_i,F_i,y_i), where yiy_i is cycle time without selector time. The fit minimizes ∑i(yi−bki−βFi)2 _i(y_i-b_k_i-β F_i)^2 subject to β,b0,…,b7≥0β,b_0,…,b_7≥ 0. The same probe trees are also timed with the expert cache cleared before every target forward, and SpecFit without the cache uses the fit to these 144 points. The RTX 4090 of Section 4.5 repeats this protocol with 32 expert slots per layer. In milliseconds, β=0.210715513β=0.210715513 with the cache and 0.4080010220.408001022 without it on the A100, and 0.2060371210.206037121 and 0.7815939550.781593955 on the RTX 4090; the intercepts bkb_k are listed below. k 0 1 2 3 4 5 6 7 A100 With cache 142.214 147.532 150.522 152.269 152.743 163.204 155.175 156.068 Without cache 122.702 117.975 118.338 116.389 115.309 114.770 113.887 113.671 RTX 4090 With cache 245.640 293.125 319.436 343.284 367.087 386.660 401.359 416.368 Without cache 151.724 158.827 164.033 164.883 169.941 171.869 173.418 175.833 The intercepts are fitted independently and need not increase with k. Reproduction uses the full-precision frozen coefficients. Measurement runs. On GSM8K, AR and EAGLE-3 with the expert cache come from the main run on the 80 questions, and SpecFit with the cache from a second run; a third run measures AR and EAGLE-3 without the expert cache, and a fourth SpecFit without it. On summarization, AR without the cache and EAGLE-3 and SpecFit with the cache come from one run on the 40 articles, AR with the cache and EAGLE-3 without it from a second run, and SpecFit without the cache from a third run; the slot sweep runs EAGLE-3 and SpecFit once at 24 and once at 12 slots. On the RTX 4090, SpecFit with the cache comes from one run on the 40 questions, AR without the cache and EAGLE-3 with and without it from a second run, SpecFit without the cache from a third run, and AR with the cache from a fourth. Every configuration has 80 requests on GSM8K on the A100 and 40 otherwise.