Paper deep dive
ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping
Zhi-Kai Chen, Jun-Jie Tao, Wei-Xiang Mao, De-Chuan Zhan, Han-Jia Ye
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/29/2026, 4:37:14 AM
Summary
The paper introduces ResiSpec, a framework designed to enhance the efficiency of multi-candidate speculative sampling in Large Language Models (LLMs). It identifies 'Residual Drift' as a bottleneck where sequential rejections cause the residual target distribution to diverge from the draft model's predictions, rendering subsequent candidates ineffective. ResiSpec resolves this by reshaping the proposal distribution during verification to anchor residual mass within the draft model's high-confidence regions, achieving up to 1.92x speedup over state-of-the-art methods while maintaining exactness.
Entities (7)
Relation Signals (6)
ResiSpec → mitigates → Residual Drift
confidence 96% · To resolve this, we propose ResiSpec... By mathematically re-aligning the verification process... ResiSpec prevents candidate obsolescence
Residual Drift → negativelyimpacts → Multi-candidate schemes
confidence 95% · these schemes are bottlenecked by Residual Drift... This shift renders subsequent candidates ineffective
ResiSpec → achievesspeedupover → state-of-the-art multi-candidate methods
confidence 93% · ResiSpec... achieves up to 1.92× speedup over state-of-the-art multi-candidate methods
Speculative Decoding → utilizes → Draft Model
confidence 92% · using a lightweight draft model to speculate future tokens
Speculative Decoding → utilizes → Target Model
confidence 92% · validated by the LLM in a single parallel forward pass
ResiSpec → uses → Proxy Distribution
confidence 90% · ResiSpec reformulates the verification criteria by integrating an auxiliary proxy of draft density
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The efficiency of Large Language Model (LLM) serving is fundamentally limited by the sequential nature of autoregressive decoding. Speculative Decoding (SD) mitigates this by using a lightweight draft model to speculate future tokens, which are then validated by the LLM in a single parallel forward pass. To further boost efficiency, multi-candidate schemes propose diverse candidate sets to increase the likelihood of token acceptance. However, we show that these schemes are bottlenecked by Residual Drift: a phenomenon where the rejection of initial candidates causes the residual target distribution to diverge from the draft model's predictions. This shift renders subsequent candidates ineffective and forces the system into expensive resampling. To resolve this, we propose ResiSpec, a framework that strategically reforms the proposal distribution during verification to anchor the residual target mass within the draft model's high-confidence regions. By mathematically re-aligning the verification process without compromising output exactness, ResiSpec prevents candidate obsolescence and achieves up to 1.92$\times$ speedup over state-of-the-art multi-candidate methods. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.24411v1
- Canonical: https://arxiv.org/abs/2608.24411v1
Trouble viewing inline? Open PDF directly →
Full Text
68,289 characters extracted from source content.
Expand or collapse full text
ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping Zhi-Kai Chen Jun-Jie Tao Wei-Xiang Mao De-Chuan Zhan Han-Jia Ye Abstract The efficiency of Large Language Model (LLM) serving is fundamentally limited by the sequential nature of autoregressive decoding. Speculative Decoding (SD) mitigates this by using a lightweight draft model to speculate future tokens, which are then validated by the LLM in a single parallel forward pass. To further boost efficiency, multi-candidate schemes propose diverse candidate sets to increase the likelihood of token acceptance. However, we show that these schemes are bottlenecked by Residual Drift: a phenomenon where the rejection of initial candidates causes the residual target distribution to diverge from the draft model’s predictions. This shift renders subsequent candidates ineffective and forces the system into expensive resampling. To resolve this, we propose ResiSpec, a framework that strategically reforms the proposal distribution during verification to anchor the residual target mass within the draft model’s high-confidence regions. By mathematically re-aligning the verification process without compromising output exactness, ResiSpec prevents candidate obsolescence and achieves up to 1.92× speedup over state-of-the-art multi-candidate methods. Code is available at https://github.com/Czzzk/Resispec. 1School of Artificial Intelligence, Nanjing University, China 2National Key Laboratory for Novel Software Technology, Nanjing University, China 3Nanjing University, China Introduction The widespread adoption of Large Language Models (LLMs) (Achiam et al. 2023; Touvron et al. 2023b) has made efficient inference a critical priority for reducing both user latency and operational costs (Pope et al. 2023). However, standard auto-regressive (AR) decoding is fundamentally limited by its sequential nature, generating tokens one-by-one (Chen et al. 2023). This process creates a severe I/O bottleneck, as the system must repeatedly access the expanding Key-Value (KV) cache to compute attention for each new output (Yu et al. 2022). This sequential dependency prevents the system from fully leveraging the massive parallel processing power of modern GPUs, leaving compute cores significantly underutilized while waiting for memory traffic (Dao et al. 2022; Fu et al. 2024; Leviathan et al. 2023). Consequently, the efficiency of LLM serving remains low, as most execution time is consumed by the overhead of accessing context rather than performing computation. To overcome these sequential constraints, Speculative Decoding (SD) (Leviathan et al. 2023; Chen et al. 2023; Xia et al. 2024) introduces a “speculate-then-verify” paradigm. The core idea is to decouple the generation process: a lightweight draft model rapidly predicts a sequence of candidate tokens at a fraction of the cost, which the larger target model then validates in a single, parallel forward pass (Spector and Re 2023; He et al. 2024; Huang et al. 2024). This approach directly addresses the I/O bottleneck by exploiting the Transformer’s inherent ability to process multiple tokens concurrently with nearly the same memory latency as a single-token step (Vaswani et al. 2017; Narayanan et al. 2021; Pope et al. 2022). By shifting from one-by-one generation to bulk verification, SD effectively increases the number of generated tokens per memory-access cycle. Consequently, performance is contingent upon acceptance length, where longer validated sequences translate into a substantial reduction in total inference steps. To fully exploit this parallel capacity and maximize the “tokens-per-load” efficiency, recent advancements have transitioned from linear, single-sequence speculation to multi-candidate schemes, such as tree-based or batch-sampling verification (Miao et al. 2024; Cai et al. 2024; Li et al. 2024b). The core idea is to expand the search space by pre-sampling multiple candidate tokens for the same or branching positions. By presenting the target model with a diverse set of “choices” in a single forward pass, these schemes aim to increase the statistical probability that at least one high-quality continuation is accepted. This strategy effectively seeks to maximize the step size—the number of confirmed tokens—of each inference iteration. While multi-candidate speculative decoding schemes—such as tree-based or batch verification—have significantly advanced LLM inference efficiency by expanding the search space beyond linear sequences, they encounter a critical scaling bottleneck. Theoretically, presenting more candidates should increase the probability of finding a high-quality match; however, in practice, the marginal utility of additional candidates diminishes rapidly as the batch size or tree width grows. We observe that as the verification chain lengthens, the acceptance probability of each subsequent candidate drops progressively, often leading to performance plateaus or even efficiency regressions. This raises a fundamental question: why do existing multi-candidate frameworks fail to fully leverage a larger search space to achieve proportional speedups? We formalize this scalability limit as Residual Drift. This phenomenon stems from the mathematical necessity of maintaining “exactness” during verification: when the target model rejects a draft candidate, it must compensate by shifting its sampling distribution to a residual distribution. Crucially, this residual is naturally concentrated in the draft model’s “blind spots”—semantic regions where the lightweight draft model has low predictive confidence or fails to capture complex dependencies. Because all candidates in a multi-candidate scheme are pre-sampled from the original draft distribution (before any rejections occur), they are inherently ill-suited for this shifted residual. Consequently, an early rejection often triggers a “cascade of obsolescence”: the target model’s updated criteria move so far away from the draft’s knowledge base that the remaining pre-sampled candidates become statistically irrelevant, forcing expensive resampling and negating the parallelism benefits of the multi-candidate design. To resolve this, we propose ResiSpec (Figure 2). Our key insight is that the mathematical path to exactness is not unique; the drift observed in prior work is a byproduct of sub-optimal verification criteria rather than an inevitable cost of zero-bias sampling. ResiSpec reformulates the verification criteria by integrating an auxiliary proxy of draft density, which steers the rejection residuals back toward the draft model’s high-confidence zones. By reshaping the transition logic, ResiSpec ensures that pre-sampled candidates remain viable throughout the verification chain without compromising the statistical integrity of the target model. • Characterization of Residual Drift: We formalize Residual Drift as a fundamental bottleneck in multi-candidate speculative decoding, where sequential rejections shift the target distribution into the draft model’s blind spots, invalidating parallel candidates. • The ResiSpec Framework: We propose ResiSpec, which reshapes verification criteria by introducing an auxiliary proxy to keep rejection residuals proximal to the draft model’s high-density regions. This ensures the viability of pre-sampled candidates even after preceding rejections. • Theoretical Rigor and Speedup: We prove ResiSpec’s mathematical exactness and demonstrate up to 1.92× speedup over multi-candidate methods while maintaining zero-bias sampling across benchmarks. Related Work Auto-regressive Acceleration and Speculative Decoding. Large Language Models (LLMs) generate text auto-regressively (Radford et al. 2019; Mann et al. 2020; Achiam et al. 2023; Touvron et al. 2023a), a process constrained by memory bandwidth. Speculative Decoding (SD) (Leviathan et al. 2023; Chen et al. 2023) emerged as a pivotal lossless acceleration paradigm, using a small draft model to generate candidate tokens that the target model verifies in parallel. While SD guarantees mathematical equivalence to the original distribution, its performance is capped by the draft model’s hit rate on a single linear sequence (Xia et al. 2024; Hu et al. 2025a; Yin et al. 2024). Multi-Candidate and Tree-Based Speculation. To overcome the limited acceptance rate of linear speculative decoding, recent work explores multiple candidate paths in parallel. SpecInfer (Miao et al. 2024) introduced tree-based attention, enabling the target model to verify branching token sequences in a single forward pass. Subsequent methods improve this paradigm along two directions: candidate generation and tree construction. For candidate generation, draft-model-free methods such as EAGLE (Li et al. 2024b; Li et al. 2024a), Medusa (Cai et al. 2024), Hydra (Ankner et al. 2024), and KOALA (Zhang et al. 2024a) use specialized heads or lightweight modules to generate candidates from the target model’s hidden states. For tree construction, Sequoia (Chen et al. 2024) uses dynamic programming to optimize tree shape under a budget, while prior work (Hu et al. 2025b) studies theoretical criteria for selecting tokens in the verification tree. Scalability Limits of Standard Verification Protocols Despite the success of multi-candidate speculation, standard sampling protocols fail to scale efficiently with candidate counts, leading to a bottleneck in acceptance length. We bridge this gap with ResiSpec, a new sampling paradigm tailored for multi-candidate verification. By optimizing the sampling logic for large-scale branching trees, ResiSpec significantly enhances the marginal utility of additional candidates and achieves a higher expected token yield. Preliminary Speculative Decoding. The key idea behind speculative decoding is to accelerate autoregressive sampling by leveraging a draft model to propose candidate tokens, which are then verified by the target model. Let the target model define a conditional distribution Ptgt(x∣x<t)P_ tgt(x x_<t) and the draft model define Pdft(x∣x<t)P_ dft(x x_<t), where t is the current step. The algorithm operates in two stages: drafting and verification. During the drafting stage, the draft model generates γ∈ℤ+γ ^+ candidate tokens from its own sampling distribution Pdft(xt∣x<t)P_ dft(x_t x_<t). Since these tokens are only approximate, a verification step is performed. In this stage, the target model evaluates the speculative tokens t+1:t+nt+1 t+n in parallel using a single forward pass. Each token is accepted with probability min(1,ptgt(x)/pdft(x)) (1,p_ tgt(x)/p_ dft(x)). If none of the speculative tokens are accepted, the target model resamples according to the adjusted distribution ptgt′(x)=norm(max(0,ptgt(x)−pdft(x)))p_ tgt (x)=norm ( (0,\,p_ tgt(x)-p_ dft(x)) ) (1) ensuring correctness while still leveraging the draft model for potential speedup. Multi-candidate Tree-structured Verification. To explore multiple potential future paths, recent schemes organize N speculative tokens into a tree-structured proposal T. The tree T is characterized by its depth d, which specifies the number of tokens predicted sequentially forward along each path, and its branching factor b, which denotes the number of parallel candidates proposed at each position. Specifically, at any given node in the tree, the draft model proposes b independent tokens x1,…,xb\x_1,…,x_b\ for the same speculative position. To preserve the exactness of the target distribution, these b candidates are verified sequentially. Let ptgt(1)(x)p_ tgt^(1)(x) be the initial target distribution for the first candidate x1x_1. For each i∈1,…,bi∈\1,…,b\, candidate xix_i is accepted with probability: min(1,ptgt(i)(xi)/qdft(xi)) (1,p_ tgt^(i)(x_i)/q_ dft(x_i)) where qdftq_ dft is the draft distribution. If xix_i is rejected, the target distribution for the next candidate xi+1x_i+1 is updated to the residual distribution ptgt(i+1)p_ tgt^(i+1): ptgt(i+1)(x)=max(0,ptgt(i)(x)−qdft(x))∑x′∈max(0,ptgt(i)(x′)−qdft(x′)).p_ tgt^(i+1)(x)= (0,p_ tgt^(i)(x)-q_ dft(x) ) _x (0,p_ tgt^(i)(x )-q_ dft(x ) ). (2) In a tree structure, this verification process is applied at each level. If a candidate xix_i is accepted, the process continues to its children in the next level (depth); if all b candidates at a certain position are rejected, the branch is truncated, and a final token is resampled from the cumulative residual distribution ptgt(b+1)(x)p_ tgt^(b+1)(x). By using a tree-attention mask, the target model can verify all paths within T in a single forward pass, significantly increasing the potential number of accepted tokens per step. Limitation of Multi-candidate Speculative Decoding Multi-candidate speculative decoding aims to maximize accepted tokens by verifying a batch of N candidates 1,…,N\x_1,…,x_N\ for one position. Although a broader search space should improve acceptance, its efficiency is throttled by Residual Drift. In standard verification (Equation 2), candidates are checked sequentially to preserve exactness. Once 1x_1 is rejected, 2x_2 must be verified against the residual distribution Pres(x)=max(0,p(x)−q(x))/∑x′max(0,p(x′)−q(x′))P_res(x)= (0,p(x)-q(x))/ _x (0,p(x )-q(x )). By construction, Pres(x)P_res(x) assigns zero probability where q(x)q(x) already meets or exceeds p(x)p(x), concentrating mass in the draft model’s “blind spots.” This creates a severe support mismatch: all N candidates are sampled i.i.d. from the original draft distribution q(x)q(x), while later verification criteria shift away from q(x)q(x)’s high-density regions. Figure 1: The Decay of Acceptance Probability. Sequential candidate acceptance rates on CNN/DM using Llama-2-7B and Vicuna-7B (Peng et al. 2023) as targets with Llama-68M as the draft. Across temperatures (T∈0.6,1.0T∈\0.6,1.0\), acceptance drops sharply after the first candidate, leaving later candidates with near-zero marginal utility. Our empirical results confirm this precipitous decline in marginal utility. As illustrated in Figure 1, the first candidate 1x_1 maintains a meaningful acceptance rate, while the probability for subsequent candidates i≥2x_i≥ 2 rapidly approaches zero. This pattern indicates a cascade of obsolescence: because candidates in the same batch are sampled from the high-density support of q(x)q(x), an early rejection shifts the verification criteria toward a residual distribution where the remaining candidates have little statistical visibility. To quantify the resulting scaling inefficiency, we vary the number of candidates N under a fixed speculation depth and report the efficiency score η, defined as the marginal EAL gain per additional candidate. Appendix Table 5 shows that increasing N yields only modest EAL improvements while η drops sharply, indicating that wider speculation spends increasing parallel verification compute on candidates with diminishing acceptance utility. These observations reveal a scaling wall: expanding tree width incurs linear verification cost but yields sub-linear acceptance gains because the residual distribution diverges from the draft distribution. This raises a central question: Can we break this bottleneck by mitigating Residual Drift? If rejection residuals can remain aligned with high-density regions of q(x)q(x), subsequent candidates may stay useful even after earlier failures. By re-aligning the verification criteria with the draft model’s support, we can transform multi-candidate decoding from a game of diminishing returns into a scalable acceleration paradigm. Next, we present ResiSpec. Figure 2: Comparison of ResiSpec vs. standard multi-candidate verification. In traditional schemes, sequential rejections trigger Residual Drift, shifting the target distribution into regions mathematically incompatible with the draft model (red areas), causing candidates to become obsolete. ResiSpec mitigates this drift by employing an auxiliary proxy to reshape the residual distribution (green areas), re-aligning it with the draft model’s high-density support. This proximity ensures that pre-sampled candidates remain viable, effectively sustaining high acceptance rates across the entire verification batch. Method: Residual Distribution Shaping In this section, we present ResiSpec, a framework that ensures high acceptance rates in multi-candidate speculative decoding by actively shaping the residual distribution. The innovation is the introduction of an auxiliary proxy distribution k(x)k(x), which replaces the original draft distribution q(x)q(x) during the verification step. By strategically constructing k(x)k(x), we ensure that if a candidate is rejected, the resulting residual distribution p′(x)p (x) remains proximal to the draft distribution q(x)q(x), thereby maximizing the acceptance probability for all subsequent candidates in the batch. Problem Formulation In standard speculative decoding, a candidate x is accepted with probability min(1,p(x)/q(x)) (1,p(x)/q(x)). If rejected, the algorithm samples from a residual distribution. To gain control over the failure state, we propose verifying candidates using a proxy distribution k(x)k(x) in place of q(x)q(x). Definition 1 (Proxy Mechanism). Given a target distribution p(x)p(x) and an unnormalized residual mass r(x)r(x), the proxy distribution k(x)k(x) is defined as: k(x)=p(x)−r(x)+s(x)k(x)=p(x)-r(x)+s(x) (3) where r(x)r(x) represents the mass reserved for the residual distribution p′(x)=r(x)/Zp (x)=r(x)/Z, and s(x)≥0s(x)≥ 0 is a slack function. The slack function s(x)s(x) serves as a compensatory mass to restore normalization of k(x)k(x). To prevent this injected mass from interfering with the reserved distribution, s(x)s(x) is only permitted to be non-zero where the residual mass is absent (r(x)=0r(x)=0). This leads to the disjoint support constraint: supp(s)∩supp(r)=∅,i.e., s(x)⋅r(x)=0,∀x∈supp(s) (r)= , .e., s(x)· r(x)=0,∀ x (4) For the sampling process to remain exact (i.e., the marginal distribution is p(x)p(x)), we impose the constraints on k(x)k(x): 1. Probability Constraint: ∑xk(x)=1 _xk(x)=1 and k(x)≥0k(x)≥ 0. 2. Local Alignment: At the sampled point xix_i, k(xi)=q(xi)k(x_i)=q(x_i), ensuring consistency with the draft model’s local probability. Proposition 1 (Global Mass Identity). For the proxy k(x)k(x) to be a valid probability distribution, the total mass of the slack function s(x)s(x) must be exactly equal to the total reserved residual mass Z=∑xr(x)Z= _xr(x), provided r(x)≤p(x)r(x)≤ p(x) for all x∈x . Proof. To satisfy the normalization axiom ∑xk(x)=1 _xk(x)=1, we expand the definition: ∑xk(x) _xk(x) =∑x(p(x)−r(x)+s(x))=1 = _x(p(x)-r(x)+s(x))=1 (5) ⇒∑xp(x)−∑xr(x)+∑xs(x)=1 _xp(x)- _xr(x)+ _xs(x)=1 ⇒1−Z+∑xs(x)=1⇒∑xs(x)=Z. 1-Z+ _xs(x)=1 _xs(x)=Z. This identity reveals a fundamental conservation law in our framework: any probability mass “reserved” from the target to form the residual must be precisely balanced by the slack mass “injected” to construct the proxy. ∎ Definition 2 (The Construction Objective). Since subsequent candidates are pre-sampled from q(x)q(x), the probability of future acceptance is maximized when the overlap between the residual p′(x)p (x) and the draft q(x)q(x) is maximized. This is equivalent to minimizing the L1L_1 distance: min∑xk|p′(x)−q(x)|,where p′(x)=r(x)Z,Z=∑xr(x). split _k _x|p (x)-q(x)|, p (x)= r(x)Z,\\ Z= _xr(x). split (6) Theoretical Boundaries: Ideal Shaping and Limits To derive the optimal residual shaping strategy, we seek a formulation that satisfies the rigorous exactness constraints established in Definition 1 while attaining the global minimum of the L1L_1 objective defined in Definition 2. Ideal Alignment and the Capacity Bound. We begin by identifying the ideal shape of the residual mass r(x)r(x) and then determine the maximum budget Z that the system can sustain without violating probability axioms. Lemma 1 (Global Optimality of Residual Matching). To achieve the global minimum of the L1L_1 objective =∑x∈|r(x)/Z−q(x)|J= _x |r(x)/Z-q(x)|, the residual mass must satisfy the ideal scaling relationship: r(x)=Z⋅q(x),∀x∈r(x)=Z· q(x), ∀ x (7) Proof. By the properties of the L1L_1 norm, ≥0J≥ 0, with equality if and only if the normalized residual p′(x)=r(x)/Zp (x)=r(x)/Z is identical to q(x)q(x). This directly implies r(x)=Zq(x)r(x)=Zq(x). ∎ Proposition 2 (The Capacity Bound of Ideal Shaping). For the ideal residual r(x)=Zq(x)r(x)=Zq(x) to be part of a valid proxy distribution k(x)=p(x)−r(x)+s(x)k(x)=p(x)-r(x)+s(x) while adhering to the disjoint support constraint (s⋅r=0s· r=0), the total reserved mass Z is bounded by the minimum density ratio: Z≤minx:q(x)>0p(x)q(x)Z≤ _x:q(x)>0 p(x)q(x) (8) Proof. Substituting the ideal shape r(x)=Zq(x)r(x)=Zq(x) into the non-negativity constraint k(x)≥0k(x)≥ 0 from Definition 1, we have: p(x)−Zq(x)+s(x)≥0p(x)-Zq(x)+s(x)≥ 0 (9) For any x where q(x)>0q(x)>0, the ideal residual r(x)=Zq(x)r(x)=Zq(x) is strictly positive. According to the disjoint support constraint in Definition 1, we must have s(x)=0s(x)=0 for these tokens. The inequality thus simplifies to: p(x)−Zq(x)≥0⟹Z≤p(x)q(x)p(x)-Zq(x)≥ 0 Z≤ p(x)q(x) (10) Taking the minimum over all x in the support of q ensures the proxy remains non-negative, yielding the stated bound. ∎ The Degeneracy and Support Paradox. In real-world LLM inference, the theoretical feasible region defined by Z≤minx:q(x)>0p(x)q(x)Z≤ _x:q(x)>0 p(x)q(x) is often degenerate or operationally unsustainable. This stems from two fundamental conflicts: 1. The Vanishing Margin: Due to the “long-tail” (Baevski and Auli 2018) nature of language models, there frequently exist tokens where the target model p(x)→0p(x)→ 0 while the draft model q(x)>0q(x)>0. In such cases, the upper bound minp(x)q(x) p(x)q(x) collapses to near-zero, effectively shrinking the feasible region to 0\0\. A budget of Z≈0Z≈ 0 would deplete the residual mass r(x)r(x), stripping the system of the probability budget needed to support subsequent candidates. 2. The Alignment-Identity Contradiction: To achieve the global minimum of the L1L_1 objective (L1=0L_1=0), we require r(x)=Z⋅q(x)r(x)=Z· q(x). If the draft model has full support (q(x)>0,∀xq(x)>0,∀ x), then r(x)>0r(x)>0 must also hold. Under the disjoint support constraint (s⋅r=0s· r=0), this forces the slack function s(x)s(x) to be zero everywhere, which, by Proposition 1, implies Z=∑s(x)=0Z=Σ s(x)=0. Crucially, accepting the degenerate solution Z=0Z=0 causes the proxy distribution to revert to the target (k(x)=p(x)k(x)=p(x)). This directly violates the Local Alignment constraint k(xi)=q(xi)k(x_i)=q(x_i), as the rejected candidate xix_i is typically a point of over-estimation where q(xi)>p(xi)q(x_i)>p(x_i). We therefore reformulate the optimal shaping of the residual as a constrained approximation problem, relaxing the ideal shape to maintain a functional residual budget. Target LLM Draft Model T Dataset ResiSpec SpecInfer ResiSpec vs SpecInfer Acc.↑ Thru.↑ Acc.↑ Thru.↑ Acc. Gain Speedup Llama-2-7B JF68M 1 CNN/DM 4.68 96.43 2.51 57.29 1.86× 1.68× Llama-2-7B JF68M 1 OpenWebText 4.69 96.06 2.51 56.76 1.86× 1.69× Llama-2-7B JF68M 1 C4 4.67 96.90 2.64 61.02 1.77× 1.59× Llama-2-7B JF68M 0.6 CNN/DM 4.14 85.03 2.59 59.41 1.60× 1.43× Llama-2-7B JF68M 0.6 OpenWebText 4.16 85.62 2.49 56.48 1.67× 1.52× Llama-2-7B JF68M 0.6 C4 4.13 85.91 2.66 61.69 1.55× 1.39× Table 1: Performance comparison between ResiSpec and the baseline SpecInfer. Regarding the configuration of the tree, we use a 3-branch complete tree with a depth of 5. The results in the table are averaged over 200 runs to reduce variance and ensure statistical reliability. Experimental results demonstrate that ResiSpec significantly increased the accepted length (by up to 1.86×) and throughput (by up to 1.68×), validating its effectiveness in accelerating large language model inference. Method Acc. Len↑ Thru. (tok/s)↑ Acc. Len Gain Speedup Sequoia 3.07 88.14 – – Sequoia+ResiSpec 4.32 115.32 1.41× 1.31× EAGLE 3.12 61.82 – – EAGLE+ResiSpec 3.85 71.17 1.23× 1.15× Table 2: Compatibility of ResiSpec with existing multi-candidate speculative decoding methods at T=1.0T=1.0. ResiSpec is evaluated as an add-on to Sequoia (Chen et al. 2024) and EAGLE (Li et al. 2024b) under matched decoding settings, improving both metrics. Practical Implementation: The ResiSpec Algorithm Given that the theoretical ideal is unattainable, we propose a minimal-budget approximation strategy. According to Theorem 2, the L1L_1 distance between the shaped residual p′(x)p (x) and the draft distribution q(x)q(x) is strictly monotonically increasing with respect to Z whenever Z>Z∗Z>Z^*. Therefore, to maintain the fidelity of the draft model’s trajectory and maximize the acceptance utility for subsequent candidates, we should seek the smallest possible Z that satisfies the physical constraints of the verification step. Since x0x_0 is a point where the draft model over-estimates the target (q(x0)>p(x0)q(x_0)>p(x_0)), and no residual mass can be reserved at a point where the target density is already exhausted (i.e., r(x0)=0r(x_0)=0), the definition of k(x)k(x) at x0x_0 simplifies to q(x0)=p(x0)+s(x0)q(x_0)=p(x_0)+s(x_0). This necessitates a local mass injection of: s(x0)=q(x0)−p(x0)s(x_0)=q(x_0)-p(x_0) (11) Given the Global Mass Identity Z=∑xs(x)Z= _xs(x) and the non-negativity requirement s(x)≥0s(x)≥ 0, the total budget Z is strictly lower-bounded by this local discrepancy: Z=s(x0)+∑x≠x0s(x)≥|q(x0)−p(x0)|Z=s(x_0)+ _x≠ x_0s(x)≥|q(x_0)-p(x_0)| (12) This bound represents the minimum “price” the system must pay in terms of residual mass to bridge the gap between the draft’s prediction and the target’s reality at the failure point. Algorithm 1 ResiSpec Sampling and Verification (The underlined step marks ResiSpec’s proxy computation, distinguishing it from standard multi-candidate protocols.) 1: Input: Prefix x<nx_<n, target model P, draft model Q, and candidates k. 2: Output: A verified token x sampled via the ResiSpec protocol. 3: Initialize residual R←R , draft D←D 4: for i=1→ki=1→ k do 5: Sample xi∼D,ri∼Uniform(0,1)x_i D,r_i (0,1) 6: if ri<R(xi)D(xi)r_i< R(x_i)D(x_i) then 7: Return xix_i ⊳ Accept xix_i 8: else 9: ⊳ Construct Proxy Distribution 10: K←GetProxy(R,Q,xi)¯ K (R,Q,x_i) ⊳ ResiSpec proxy step 11: ⊳ Use Proxy Distribution to Calculate Residual Distribution instead of Draft distribution 12: R←norm(max(R−K,0))R ( (R-K,0)) 13: end if 14: end for 15: ⊳ Resampling From the Residual Distribution 16: Return x∼Rx R 16: 17: Function GetProxy(p,q,x0) GetProxy(p,q,x_0) 18: Z←q(x0)−p(x0)Z← q(x_0)-p(x_0) ⊳ Total residual budget 19: ri←min(Z⋅qi,pi)r_i← (Z· q_i,p_i) for all i≠x0i≠ x_0, and rx0←0r_x_0← 0 20: Δ←Z−∑iri ← Z- _ir_i ⊳ Mass lost due to clipping at pip_i 21: Update r by distributing Δ into the remaining capacity (p−r)(p-r) ⊳ Ensures ∑r=ZΣ r=Z and r≤pr≤ p 22: k←p−r+sk← p-r+s, then set k(x0)←q(x0)k(x_0)← q(x_0) ⊳ Align with q at x0x_0 23: Return k Guided by the monotonicity principle (Theorem 2)—which suggests that any Z larger than necessary will further deviate the residual from the draft’s manifold—we select the minimal Z that accommodates this local gap. In practice, we employ the following heuristic: Zapplied=max(Z∗,|q(x0)−p(x0)|)Z_applied= (Z^*,|q(x_0)-p(x_0)|) (13) This adjustment ensures that k(x)k(x) remains a valid probability distribution while preventing the residual mass from collapsing due to long-tail noise. The complete procedure for dynamically adjusting r(x)r(x) and Z to balance theoretical exactness and inference speed is detailed in Algorithm 1. In the event of a verification failure, GetProxy intervenes to adjust the residual distribution R prior to the next verification step. Its primary role is to prevent the target distribution from drifting into regions of low draft density. By aligning the mass of R with the draft model Q, the system ensures that the remaining candidates xi+1,…,xk\x_i+1,…,x_k\ encounter verification criteria compatible with the draft model’s knowledge. The function’s logic is as follows: Step 1: Budgeting and Initial Projection. It first calculates the mass budget Z=|q(x0)−p(x0)|Z=|q(x_0)-p(x_0)|. It then performs an initial projection by setting ri=min(Zqi,pi)r_i= (Zq_i,p_i), effectively “clipping” the desired draft shape into the target model’s available capacity. Step 2: Capacity-Aware Redistribution. Any mass deficit Δ caused by clipping is redistributed to tokens where pi>rip_i>r_i. This “water-filling” approach ensures that the total residual mass strictly meets the budget Z without violating the physical upper bound p(x)p(x). Step 3: Proxy Formulation. Finally, it constructs k=p−rk=p-r, with a local adjustment at x0x_0 to ensure k(x0)=q(x0)k(x_0)=q(x_0). This ensures that the acceptance ratio at the failed point remains mathematically consistent with the original draft. Theoretical Verification A prerequisite for any speculative sampling framework is the preservation of the target model’s distribution, ensuring that acceleration is achieved without compromising output quality. We formally prove that ResiSpec satisfies this exactness property; specifically, our iterative residual-shaping process, mediated by the proxy distribution k(x)k(x), is shown to be provably unbiased, maintaining a marginal distribution identical to the target model p(x)p(x) across all verification steps. A rigorous theoretical justification is provided in Appendix A.1. This theoretical guarantee is further corroborated by empirical validation, where we demonstrate zero distributional shift compared to standard auto-regressive decoding. Experiment Experiment Setup Model and Baseline. We evaluate ResiSpec in comparison with SpecInfer (Miao et al. 2024) using Llama-2-7B (Touvron et al. 2023b) as the target model and JackFram/Llama-68M (JF68M) (Miao et al. 2024) as the draft model. To ensure evaluation coverage, we conduct experiments on three widely-used text corpora: CNN/DM (See et al. 2017), OpenWebText (Gokaslan and Cohen 2019), and C4 (Raffel et al. 2020). All evaluations are performed under two temperature settings, T=1.0T=1.0 and T=0.6T=0.6, to assess performance across different sampling regimes. Metrics. We assess the efficiency of speculative decoding using two metrics. The first metric is Accepted Length (Acc. Len), which measures the average number of tokens successfully accepted per speculative decoding step; higher values indicate more effective draft utilization. The second metric is Throughput (Thru.), defined as the number of generated tokens per second, which directly reflects end-to-end inference efficiency. All experiments are conducted on a single NVIDIA RTX 3090 GPU and averaged over multiple runs. Main Results We perform an extensive evaluation comparing ResiSpec with SpecInfer, with detailed results presented in Table 1. Considering that both ResiSpec and SpecInfer employ a tree-based speculative decoding method, we use a 3-branch complete tree with a depth of 5 for all experiments. The findings demonstrate that ResiSpec achieves statistically significant improvements over the baseline method. Specifically, we observe consistent enhancements in accepted length (up to 1.86×) and throughput (up to 1.68×) across all tested configurations. Notably, these performance gains are sustained across diverse data domains and varying sampling temperatures, highlighting the robustness and generalizability of our proposed distribution shaping mechanism. Compatibility with Existing Methods. To examine whether residual shaping is tied to a specific baseline, we integrate ResiSpec with two different multi-candidate speculative decoding methods. EAGLE improves the drafter side by accelerating candidate generation, while Sequoia optimizes the verification tree used by the target model. As shown in Table 2, ResiSpec is evaluated as an add-on, reporting accepted-length and throughput improvements under matched decoding settings. Computational Overhead. We further profile the runtime of ResiSpec’s proxy construction and residual update relative to a single target-model verification step. Most of this overhead comes from memory-bound GPU operations over token distributions, rather than heavy arithmetic; fused kernels could therefore further reduce ResiSpec’s latency. Importantly, ResiSpec does not introduce an additional model forward pass, and its cost comes from distribution-level operations after verification. Figure 3 reports per-step latency broken into drafting, verification, and ResiSpec shaping, and further decomposes ResiSpec’s time into budgeting, redistribution, and proxy formulation. Figure 3: Runtime analysis of ResiSpec. Left: per-step latency across SpecInfer, Sequoia, and EAGLE when augmented with ResiSpec, decomposed into drafting, target verification, and ResiSpec shaping. Right: ResiSpec’s internal runtime breakdown across budgeting, redistribution, and proxy formulation, shown by percentage. Ablation Study To further dissect the efficacy of ResiSpec, we analyze both tree topology and target-draft model pairing. For topology, we compare tree structures under an iso-compute constraint by selecting configurations with similar verification costs, ensuring that performance differences reflect distribution alignment rather than extra parallel computation. Table 3 shows that ResiSpec consistently outperforms SpecInfer, with larger gains on deeper and narrower trees, indicating stronger mitigation of accumulated Residual Drift over longer speculation horizons. We also evaluate additional target-draft model pairs and datasets in Appendix D, where ResiSpec consistently improves accepted length and throughput over SpecInfer, suggesting that residual shaping is not tied to a single model family or capacity gap. Tree Verification Costs ResiSpec SpecInfer SpeedUp Thru.↑ Thru.↑ d7b2 127 101.01 52.69 1.92× d5b3 121 94.25 56.92 1.66× d4b5 156 60.83 42.46 1.43× d3b11 133 44.21 37.89 1.17× Table 3: Impact of tree topologies under comparable verification budgets on Llama-2-7B with JF68M. Verification cost is the number of candidate positions verified concurrently. ResiSpec gains more over SpecInfer on deeper, narrower trees. Empirical Validation of Distributional Exactness To empirically verify the theoretical exactness proven in Theorem 1 in Appendix A.1, we measure the Kullback-Leibler (KL) divergence (Kullback and Leibler 1951) between the original target distribution p(x)p(x) and the effective marginal distribution produced by ResiSpec. The effective distribution is reconstructed by exhaustively marginalizing the probability mass across all possible verification trajectories within a candidate batch, including all sequential acceptance and rejection paths (see Appendix B for the full derivation). As shown in Table 4, we compare ResiSpec against standard multi-candidate speculative decoding. Our results demonstrate that ResiSpec maintains the target distribution with high fidelity. Specifically, the KL divergence between the ResiSpec-generated distribution and the target model remains on the order of 10−1010^-10, a level of discrepancy that is indistinguishable from standard multi-candidate methods. This negligible error is consistent with numerical floating-point noise rather than any systematic algorithmic bias. These findings confirm that ResiSpec is stochastically equivalent to the target model, ensuring that our distribution shaping does not sacrifice generation quality for speed. Method Data KL (T=1.0T=1.0) KL (T=0.6T=0.6) ResiSpec CNN/DM 2.64e-10 4.38e-10 OpenWebText 7.15e-10 4.37e-10 C4 1.02e-9 1.39e-10 SpecInfer CNN/DM 1.04e-9 3.55e-10 OpenWebText 1.63e-9 3.89e-10 C4 9.10e-10 1.09e-10 Table 4: KL divergence between the target distribution p(x)p(x) and the sampling distributions of ResiSpec and SpecInfer on Llama-2-7B with JF68M. Near-zero values across datasets and temperatures confirm distributional preservation. Conclusion We address the “residual drift” bottleneck in multi-candidate speculative decoding by proposing ResiSpec, a distribution-shaping framework. By aligning rejection residuals with the draft model through a proxy distribution, ResiSpec prevents the collapse of acceptance rates during sequential verification. As a verification-time add-on, ResiSpec can also complement existing candidate-generation and tree-construction methods. Our method maintains exact sampling while achieving up to 1.92× speedup, providing a robust and efficient solution for high-throughput LLM acceleration. References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: Introduction, Related Work. Ankner et al. (2024) Z. Ankner, R. Parthasarathy, A. Nrusimha, C. Rinard, J. Ragan-Kelley, and W. Brandon Hydra: sequentially-dependent draft heads for medusa decoding. arXiv preprint arXiv:2402.05109. Cited by: Related Work. Baevski and Auli (2018) A. Baevski and M. Auli Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. Cited by: Theoretical Boundaries: Ideal Shaping and Limits. Bapna et al. (2020) A. Bapna, N. Arivazhagan, and O. Firat Controlling computation versus quality for neural sequence models. arXiv preprint arXiv:2002.07106. Cited by: Appendix D. Bishop and Nasrabadi (2006) C. M. Bishop and N. M. Nasrabadi Pattern recognition and machine learning. Springer. Cited by: Appendix D. Burton (2012) F. W. Burton Speculative computation, parallelism, and functional programming. IEEE Transactions on Computers 100 (12), p. 1190–1193. Cited by: Appendix D. Cai et al. (2024) T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao Medusa: simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774. Cited by: Introduction, Related Work. Chen et al. (2023) C. Chen, S. Borgeaud, G. Irving, J. Lespiau, L. Sifre, and J. Jumper Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. Cited by: Introduction, Introduction, Related Work. Chen et al. (2024) Z. Chen, A. May, R. Svirschevski, Y. Huang, M. Ryabinin, Z. Jia, and B. Chen Sequoia: scalable, robust, and hardware-aware speculative decoding. arXiv preprint arXiv:2402.12374. Cited by: Related Work, Table 2. Dao et al. (2022) T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré Flashattention: fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems 35, p. 16344–16359. Cited by: Introduction. Fu et al. (2024) Y. Fu, P. Bailis, I. Stoica, and H. Zhang Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057. Cited by: Introduction. Gokaslan and Cohen (2019) A. Gokaslan and V. Cohen OpenWebText corpus. Cited by: Experiment Setup. He et al. (2024) Z. He, Z. Zhong, T. Cai, J. Lee, and D. He Rest: retrieval-based speculative decoding. In Proceedings of the 2024 conference of the North American chapter of the association for computational linguistics: Human language technologies (volume 1: long papers), p. 1582–1595. Cited by: Appendix D, Introduction. Hu et al. (2025a) Y. Hu, Z. Liu, Z. Dong, T. Peng, B. McDanel, and S. Q. Zhang Speculative decoding and beyond: an in-depth survey of techniques. arXiv preprint arXiv:2502.19732. Cited by: Related Work. Hu et al. (2025b) Z. Hu, T. Zheng, V. Viswanathan, Z. Chen, R. A. Rossi, Y. Wu, D. Manocha, and H. Huang Towards optimal multi-draft speculative decoding. arXiv preprint arXiv:2502.18779. Cited by: Related Work. Huang et al. (2024) K. Huang, X. Guo, and M. Wang Specdec++: boosting speculative decoding via adaptive candidate lengths. arXiv preprint arXiv:2405.19715. Cited by: Appendix D, Introduction. Kullback and Leibler (1951) S. Kullback and R. A. Leibler On information and sufficiency. The annals of mathematical statistics 22 (1), p. 79–86. Cited by: Empirical Validation of Distributional Exactness. Leviathan et al. (2023) Y. Leviathan, M. Kalman, and Y. Matias Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. Cited by: Introduction, Introduction, Related Work. Li et al. (2024a) Y. Li, F. Wei, C. Zhang, and H. Zhang Eagle-2: faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858. Cited by: Related Work. Li et al. (2024b) Y. Li, F. Wei, C. Zhang, and H. Zhang EAGLE: speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: Introduction, Related Work, Table 2. Luo et al. (2025) X. Luo, Y. Wang, Q. Zhu, Z. Zhang, X. Zhang, Q. Yang, and D. Xu Turning trash into treasure: accelerating inference of large language models with token recycling. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6816–6831. Cited by: Appendix D. Mann et al. (2020) B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 1 (3), p. 3. Cited by: Related Work. Miao et al. (2024) X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y. Y. Wong, A. Zhu, L. Yang, X. Shi, C. Shi, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia SpecInfer: accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS ’24, p. 932–949. External Links: Document Cited by: Introduction, Related Work, Experiment Setup. Narayanan et al. (2021) D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia Efficient large-scale language model training on gpu clusters using megatron-lm. External Links: 2104.04473, Link Cited by: Introduction. Peng et al. (2023) B. Peng, C. Li, P. He, M. Galley, and J. Gao Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277. Cited by: Figure 1. Pope et al. (2023) R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean Efficiently scaling transformer inference. Proceedings of machine learning and systems 5, p. 606–624. Cited by: Introduction. Pope et al. (2022) R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean Efficiently scaling transformer inference. External Links: 2211.05102, Link Cited by: Introduction. Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog 1 (8), p. 9. Cited by: Related Work. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (140), p. 1–67. Cited by: Experiment Setup. Robert et al. (1999) C. P. Robert, G. Casella, and G. Casella Monte carlo statistical methods. Vol. 2, Springer. Cited by: Appendix D. See et al. (2017) A. See, P. J. Liu, and C. D. Manning Get to the point: summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, p. 1073–1083. Cited by: Experiment Setup. Spector and Re (2023) B. Spector and C. Re Accelerating llm inference with staged speculative decoding. arXiv preprint arXiv:2308.04623. Cited by: Introduction. Touvron et al. (2023a) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: open and efficient foundation language models. arxiv. arXiv preprint arXiv:2302.13971. Cited by: Related Work. Touvron et al. (2023b) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: Introduction, Experiment Setup. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. Advances in neural information processing systems 30. Cited by: Introduction. Xia et al. (2024) H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui Unlocking efficiency in large language model inference: a comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851. Cited by: Introduction, Related Work. Xia et al. (2023) M. Xia, T. Gao, Z. Zeng, and D. Chen Sheared llama: accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694. Cited by: Table 6. Yin et al. (2024) M. Yin, M. Chen, K. Huang, and M. Wang A theoretical perspective for speculative decoding algorithm. Advances in Neural Information Processing Systems 37, p. 128082–128117. Cited by: Related Work. Yu et al. (2022) G. Yu, J. S. Jeong, G. Kim, S. Kim, and B. Chun Orca: a distributed serving system for \transformer-based\ generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), p. 521–538. Cited by: Introduction. Zhang et al. (2024a) K. Zhang, J. Zhao, and R. Chen KOALA: enhancing speculative decoding for llm via multi-layer draft heads with adversarial learning. arXiv preprint arXiv:2408.08146. Cited by: Related Work. Zhang et al. (2024b) P. Zhang, G. Zeng, T. Wang, and W. Lu Tinyllama: an open-source small language model. arXiv preprint arXiv:2401.02385. Cited by: Table 6, Appendix D. Appendix A Theoretical Proof for ResiSpec Exactness A.1 Correctness and Exactness Theorem 1 (Universal Exactness of ResiSpec). Let p(x)p(x) be the target distribution and q(x)q(x) be the draft distribution. For a candidate token x∼q(x)x q(x), let k(x)k(x) be a proxy distribution that satisfies the Local Alignment condition: k(x)=q(x)k(x)=q(x) at the specific value of the sampled candidate. If we define the acceptance probability as: α(x)=min(1,p(x)k(x))α(x)= (1, p(x)k(x) ) (14) and the residual distribution as: pres(x)=p(x)−min(p(x),k(x))1−βp_res(x)= p(x)- (p(x),k(x))1-β (15) where β=∑x∈min(p(x),k(x))β= _x (p(x),k(x)), then the marginal distribution of the output token X is exactly p(x)p(x). Proof. The core intuition of ResiSpec is that while the proxy distribution k(x)k(x) is used to shape the residual, the sampling exactness is preserved as long as k(x)k(x) “mimics” the draft distribution q(x)q(x) at the point of verification. To prove this, we consider the marginal probability P(X=x′)P(X=x ) for any token x′∈x by summing the probabilities of two mutually exclusive events: (1) x′x is proposed and accepted, or (2) a candidate is proposed and rejected, and the system subsequently samples x′x from the residual distribution. Step 1: Analysis of the Acceptance Path. For the output X to be x′x via acceptance, the draft model must first propose x∼q(x)x q(x) where x=x′x=x , and the verification step must subsequently accept it. Crucially, at this specific point of verification, the Local Alignment condition k(x′)=q(x′)k(x )=q(x ) is active. The probability mass contributed by this path is: P(Accept,X=x′) P(Accept,X=x ) =q(x′)⋅α(x′) =q(x )·α(x ) (16) =q(x′)⋅min(1,p(x′)k(x′)) =q(x )· (1, p(x )k(x ) ) By substituting the Local Alignment condition k(x′)=q(x′)k(x )=q(x ), we can rewrite the acceptance probability in terms of k: P(Accept,X=x′)=k(x′)⋅min(1,p(x′)k(x′))=min(k(x′),p(x′))P(Accept,X=x )=k(x )· (1, p(x )k(x ) )= (k(x ),p(x )) (17) This step is vital: it shows that by aligning k with q locally, the acceptance path effectively samples from the intersection of p and our proxy k. Step 2: Analysis of the Rejection Path. The rejection path occurs if any proposed candidate x is rejected. The total probability of rejection, P(Reject)P(Reject), is the complement of the total acceptance mass: P(Reject)=1−∑x∈P(Accept,X=x)=1−∑x∈min(p(x),k(x))=1−βP(Reject)=1- _x P(Accept,X=x)=1- _x (p(x),k(x))=1-β (18) Once a rejection occurs, the system samples x′x from the residual distribution pres(x′)p_res(x ). The probability mass contributed by this path is the product of the total rejection probability and the conditional probability of sampling x′x : P(Reject,X=x′) P(Reject,X=x ) =P(Reject)⋅pres(x′) =P(Reject)· p_res(x ) (19) =(1−β)⋅p(x′)−min(p(x′),k(x′))1−β =(1-β)· p(x )- (p(x ),k(x ))1-β =p(x′)−min(p(x′),k(x′)) =p(x )- (p(x ),k(x )) Note that the normalization constant 1−β1-β cancels out, leaving a residual mass that perfectly complements the acceptance mass defined in Eq. 17. Step 3: Summation of Marginal Probabilities. According to the law of total probability, the final marginal distribution P(X=x′)P(X=x ) is the sum of the results from Eq. 17 and Eq. 19: P(X=x′) P(X=x ) =P(Accept,X=x′)+P(Reject,X=x′) =P(Accept,X=x )+P(Reject,X=x ) (20) =min(p(x′),k(x′))+[p(x′)−min(p(x′),k(x′))] = (p(x ),k(x ))+ [p(x )- (p(x ),k(x )) ] =p(x′) =p(x ) Since this identity holds for all x′∈x , the output distribution is identically the target distribution p(x)p(x). Conclusion. The proof demonstrates that ResiSpec remains an exact sampling algorithm. By coupling the residual distribution to the same proxy k(x)k(x) used in the acceptance test, any deviation of k(x)k(x) from the original draft distribution q(x)q(x) is mathematically compensated for in the failure state, provided that k and q are aligned at the point of verification. ∎ A.2 Monotonicity of Residual Shaping Loss with Redistribution Theorem 2 (Monotonicity with Redistribution). Let q(x)q(x) and p(x)p(x) be the draft and target distributions, respectively. Define the unnormalized residual mass r(x)r(x) subject to the capacity constraint 0≤r(x)≤p(x)0≤ r(x)≤ p(x) and the global mass constraint ∑xr(x)=Z _xr(x)=Z. Let p′(x)=r(x)/Zp (x)=r(x)/Z be the normalized residual distribution. The minimum L1L_1 distance L1(Z)=∑x|p′(x)−q(x)|D_L1(Z)= _x|p (x)-q(x)| is: 1. Identically zero for Z∈(0,Z∗]Z∈(0,Z^*], where Z∗=minx:q(x)>0p(x)q(x)Z^*= _x:q(x)>0 p(x)q(x). 2. Strictly monotonically increasing for Z∈(Z∗,1]Z∈(Z^*,1]. Proof. To minimize L1(Z)=1Z∑x|r(x)−Zq(x)|D_L1(Z)= 1Z _x|r(x)-Zq(x)|, we define the local deviation as E(x)=r(x)−Zq(x)E(x)=r(x)-Zq(x). Since ∑xr(x)=Z _xr(x)=Z and ∑xZq(x)=Z _xZq(x)=Z, the total deviation sums to zero: ∑xE(x)=0 _xE(x)=0. This implies that the sum of positive deviations must exactly balance the sum of negative deviations: ∑x:E(x)>0|E(x)|=∑x:E(x)<0|E(x)| _x:E(x)>0|E(x)|= _x:E(x)<0|E(x)| (21) Consequently, the L1L_1 distance can be expressed solely in terms of the negative deviations: L1(Z)=1Z(∑E(x)>0|E(x)|+∑E(x)<0|E(x)|)=2Z∑x:E(x)<0|E(x)|D_L1(Z)= 1Z ( _E(x)>0|E(x)|+ _E(x)<0|E(x)| )= 2Z _x:E(x)<0|E(x)| (22) Under the capacity constraint r(x)≤p(x)r(x)≤ p(x), a negative deviation (E(x)<0E(x)<0) is mandatory if and only if p(x)<Zq(x)p(x)<Zq(x). To minimize the total deviation, we must set r(x)=p(x)r(x)=p(x) in these “bottleneck” regions and r(x)≥Zq(x)r(x)≥ Zq(x) elsewhere (Redistribution). Thus, the minimal negative deviation at any point x is |E(x)|=max(0,Zq(x)−p(x))|E(x)|= (0,Zq(x)-p(x)). Substituting this into the L1L_1 objective: L1(Z)=2Z∑xmax(0,Zq(x)−p(x))=2∑xmax(0,q(x)−p(x)Z)D_L1(Z)= 2Z _x (0,Zq(x)-p(x))=2 _x (0,q(x)- p(x)Z ) (23) We now analyze the behavior of L1(Z)D_L1(Z) across the two regimes of Z: Case 1: Z∈(0,Z∗]Z∈(0,Z^*]. By the definition of Z∗=minp(x)q(x)Z^*= p(x)q(x), for any Z≤Z∗Z≤ Z^*, we have p(x)Z≥p(x)Z∗≥q(x) p(x)Z≥ p(x)Z^*≥ q(x) for all x. Thus, q(x)−p(x)Z≤0q(x)- p(x)Z≤ 0 for all x, and L1(Z)=0D_L1(Z)=0. In this regime, the target model has sufficient capacity to perfectly mirror the draft distribution’s shape. Case 2: Z∈(Z∗,1]Z∈(Z^*,1]. In this regime, the set of bottleneck tokens Z=x∣q(x)>p(x)ZS_Z=\x q(x)> p(x)Z\ is non-empty. Let f(Z)=2∑x∈Z(q(x)−p(x)Z)f(Z)=2 _x _Z(q(x)- p(x)Z). Taking the derivative with respect to Z: dL1dZ=2∑x∈Zp(x)Z2 dD_L1dZ=2 _x _Z p(x)Z^2 (24) Since p(x)>0p(x)>0 for x∈Zx _Z and Z2>0Z^2>0, we have dL1dZ>0 dD_L1dZ>0. This proves that the L1L_1 distance is strictly monotonically increasing with Z once the residual mass exceeds the critical bottleneck capacity Z∗Z^*. Conclusion: Any mass Z allocated beyond Z∗Z^* necessitates a “clipping” of the draft shape at bottleneck points and a corresponding “redistribution” of mass to non-bottleneck points. Both actions contribute equally to the increase in L1L_1 distance, justifying the use of a minimal mass budget to preserve distribution fidelity. ∎ Appendix B Calculation of Empirical Validation of Distributional Exactness To empirically verify that ResiSpec maintains zero-bias sampling in a multi-candidate setting (e.g., batch or tree-based speculation), we measure the discrepancy between the theoretical target distribution p(x)p(x) and the aggregate marginal distribution produced by the ResiSpec verification chain. In a multi-candidate scenario with n candidates x1,x2,…,xn\x_1,x_2,…,x_n\, the final output token X is determined by a sequential search for the first accepted candidate, or a fallback to a residual distribution if all candidates are rejected. B.1 Path-based Marginal Probability Calculation The empirical marginal distribution P(X=x′)P(X=x ) is calculated by summing the probability masses of all mutually exclusive execution paths. Let αi _i denote the acceptance probability of the i-th candidate and 1−βi1- _i denote the probability of rejection at step i. The possible paths are: 1. Path 1: Accepted at the first candidate. The probability that the first candidate x1x_1 is x′x and is accepted is: P1(x′)=min(k1(x′),p(x′))P_1(x )= (k_1(x ),p(x )) (25) 2. Path i (2≤i≤n2≤ i≤ n): Accepted at the i-th candidate. For the system to accept the i-th candidate, all preceding i−1i-1 candidates must have been rejected. Due to the exactness property of the ResiSpec residual (see Appendix A.1), the target distribution effectively shifts to pres,i−1(x)p_res,i-1(x) after i−1i-1 rejections. The probability mass is: Pi(x′)=(∏j=1i−1(1−βj))⋅min(ki(x′),pres,i−1(x′))P_i(x )= ( _j=1^i-1(1- _j) )· (k_i(x ),p_res,i-1(x )) (26) where pres,i−1p_res,i-1 is the residual distribution reshaped by ResiSpec after the (i−1)(i-1)-th rejection. 3. Path n+1n+1: All candidates rejected. If all n candidates fail verification, the system samples x′x directly from the final residual distribution pres,n(x′)p_res,n(x ). The probability mass is: Pfail(x′)=(∏j=1n(1−βj))⋅pres,n(x′)P_fail(x )= ( _j=1^n(1- _j) )· p_res,n(x ) (27) B.2 Verification via KL Divergence According to Theorem 1, the total marginal probability p^(x′)=∑i=1nPi(x′)+Pfail(x′) p(x )= _i=1^nP_i(x )+P_fail(x ) must identically equal the target distribution p(x′)p(x ). We evaluate this identity by calculating the Kullback-Leibler (KL) Divergence between the reconstructed distribution p p and the ground-truth target distribution p: DKL(p^∥p)=∑x∈p^(x)logp^(x)p(x)D_KL( p p)= _x p(x) p(x)p(x) (28) In our implementation, we recorded the full logits for both the draft and target models across various benchmarks. Across all test cases, the measured DKLD_KL consistently remained near zero (typically <10−8<10^-8), which is within the expected range of floating-point numerical error. This empirical result confirms that ResiSpec introduces no distributional bias, ensuring that the accelerated inference remains mathematically equivalent to standard auto-regressive decoding. Appendix C Reproducibility Details Execution environment. All throughput experiments reported in the revised evaluation are run inside the same Docker environment to avoid differences caused by host-side Python, CUDA, or library versions. For each matched comparison, the baseline and the corresponding ResiSpec variant use the same container image, the same model checkpoints, and the same dataset order. We bind each run to a single GPU with CUDA_VISIBLE_DEVICES=0. Baseline and ResiSpec runs are executed sequentially, with one independent Python process per dataset-method pair, so that the GPU memory allocator state and kernel scheduling of one method do not interfere with the other. We record raw logs and parsed CSV summaries for every run, and compute throughput from the number of generated tokens divided by measured wall-clock decoding time after warmup. Models and datasets. The main fixed-tree experiments use Llama-2-7B as the target model and JackFram/Llama-68M as the draft model. To address experimental-scope concerns, we additionally evaluate Vicuna-7B-v1.3 with Llama-68M, Sheared-LLaMA-2.7B with TinyLlama-1.1B, and Sheared-LLaMA-2.7B with Sheared-LLaMA-1.3B under the same fixed-tree SpecInfer-style setting. The compatibility experiments use Llama-2-7B with Llama-68M for Sequoia, and Llama-2-7B-Chat with the corresponding EAGLE Llama-2-Chat drafter for EAGLE. Unless otherwise stated, all model checkpoints are loaded from the same local HuggingFace cache used by the Docker container. The dataset suite is CNN/DM, OpenWebText, and C4. For the 200-example runs, we use a fixed dataset order and evaluate the same prompt indices for the baseline and ResiSpec under each configuration. EAGLE prompt-generation runs use two warmup prompts followed by 200 measured prompts. Decoding configurations. For the SpecInfer-style fixed-tree experiments, both the baseline and ResiSpec use the same complete 3-branch tree with depth 5 and a maximum sequence length budget of 512. We evaluate both T=0.6T=0.6 and T=1.0T=1.0 with nucleus probability p=1.0p=1.0. For dynamic-tree Sequoia, we use the original Sequoia tree-selection logic with the same L40 grow maps for the baseline and ResiSpec, and set M=384M=384, T=1.0T=1.0, and p=1.0p=1.0 for the 200-example comparison. For EAGLE, we use total_token=60, depth=5, and max_new_tokens=128; the main compatibility table uses top_k=10, and the sensitivity rerun with top_k=15 follows the same Docker, dataset, and sequential-execution protocol. ResiSpec integration. In fixed-tree SpecInfer-style decoding, ResiSpec reshapes the residual distribution at each sequential verification position in the candidate tree. In Sequoia, ResiSpec is added after Sequoia has selected its dynamic tree, so the tree budget and topology are controlled by the original Sequoia policy and only the verification-time sampling distribution is changed. In EAGLE, ResiSpec is applied only to consecutive candidate siblings that share the same parent node. After each rejection, the implementation updates the current target distribution p and draft/proxy distribution q exactly as in the ResiSpec recurrence before verifying the next sibling. Randomness control. All sampling-based experiments use fixed pseudo-random seeds. Before each benchmark run, we initialize Python’s random module, NumPy, PyTorch CPU random state, and PyTorch CUDA random states with the same base seed, and set cuDNN to deterministic mode. To make per-example sampling reproducible while avoiding identical random streams across prompts, the i-th evaluated prompt uses seed s+is+i, where s is the base seed for that run. The same seed schedule is applied to ResiSpec and the corresponding baseline under each matched model, dataset, temperature, and tree configuration. Timing and diagnostics. Throughput runs disable auxiliary KL diagnostics and other distribution-checking instrumentation, because these checks require additional logit materialization and are not part of the decoding algorithm. Distribution preservation is evaluated separately using the procedure in Appendix B. For the runtime breakdown in Figure 3, we add CUDA synchronization around the measured regions and report per-speculation-step latency for draft generation, target-model verification, and ResiSpec proxy computation. These profiling runs are used only for component attribution; the synchronized timings are not mixed with the end-to-end throughput measurements. Appendix D Ablation Experiment Results Candidates (N) EAL↑ Marginal Gain (Δ ) Efficiency (η) 1 1.89 - - 2 2.17 0.28 0.280 4 2.48 0.31 0.155 8 2.76 0.28 0.070 Table 5: Scaling Inefficiency of Multi-Candidate Schemes. With speculation depth fixed at 4, increasing candidates N yields only modest EAL gains while the efficiency score η drops sharply. Results are averaged over CNN/DM using Llama-2-7B. Target Draft T Data Tree Resi. Acc. Resi. Thru. Spec. Acc. Spec. Thru. Sheared-2.7B Sheared-1.3B 1 CNN/DM d5b3 4.88 78.309 3.88 66.313 Sheared-2.7B Sheared-1.3B 1 CNN/DM d4b4 3.94 82.988 3.42 76.695 Sheared-2.7B Sheared-1.3B 1 OpenWebText d5b3 4.86 78.447 3.92 67.660 Sheared-2.7B Sheared-1.3B 1 OpenWebText d4b4 3.95 83.060 3.45 76.864 Sheared-2.7B Sheared-1.3B 1 C4 d5b3 4.84 78.362 3.98 68.679 Sheared-2.7B Sheared-1.3B 1 C4 d4b4 3.95 82.850 3.44 77.340 Sheared-2.7B Sheared-1.3B 0.6 CNN/DM d5b3 4.88 78.927 4.12 71.174 Sheared-2.7B Sheared-1.3B 0.6 CNN/DM d4b4 3.96 83.110 3.55 79.681 Sheared-2.7B Sheared-1.3B 0.6 OpenWebText d5b3 4.89 78.998 4.10 70.922 Sheared-2.7B Sheared-1.3B 0.6 OpenWebText d4b4 3.97 81.833 3.54 78.424 Sheared-2.7B Sheared-1.3B 0.6 C4 d5b3 4.88 77.939 4.13 70.685 Sheared-2.7B Sheared-1.3B 0.6 C4 d4b4 3.97 82.559 3.51 78.083 Sheared-2.7B TinyLlama-1.1B 1 CNN/DM d5b3 4.79 82.170 3.71 68.587 Sheared-2.7B TinyLlama-1.1B 1 CNN/DM d4b4 3.92 86.656 3.30 78.432 Sheared-2.7B TinyLlama-1.1B 1 OpenWebText d5b3 4.83 82.798 3.69 68.074 Sheared-2.7B TinyLlama-1.1B 1 OpenWebText d4b4 3.92 86.496 3.30 78.125 Sheared-2.7B TinyLlama-1.1B 1 C4 d5b3 4.81 82.645 3.71 68.493 Sheared-2.7B TinyLlama-1.1B 1 C4 d4b4 3.93 86.656 3.27 77.580 Sheared-2.7B TinyLlama-1.1B 0.6 CNN/DM d5b3 4.84 82.508 3.87 71.609 Sheared-2.7B TinyLlama-1.1B 0.6 CNN/DM d4b4 3.95 86.957 3.43 81.699 Sheared-2.7B TinyLlama-1.1B 0.6 OpenWebText d5b3 4.79 82.441 3.80 70.423 Sheared-2.7B TinyLlama-1.1B 0.6 OpenWebText d4b4 3.95 87.395 3.32 79.005 Sheared-2.7B TinyLlama-1.1B 0.6 C4 d5b3 4.81 82.508 3.79 70.077 Sheared-2.7B TinyLlama-1.1B 0.6 C4 d4b4 3.93 87.036 3.32 78.858 Sheared-2.7B Tiny-Vicuna-1B 1 CNN/DM d5b3 4.81 82.508 3.53 65.274 Sheared-2.7B Tiny-Vicuna-1B 1 CNN/DM d4b4 3.93 86.496 3.18 75.429 Sheared-2.7B Tiny-Vicuna-1B 1 OpenWebText d5b3 4.81 82.170 3.51 65.050 Sheared-2.7B Tiny-Vicuna-1B 1 OpenWebText d4b4 3.93 86.656 3.17 75.302 Sheared-2.7B Tiny-Vicuna-1B 1 C4 d5b3 4.81 82.441 3.58 66.327 Sheared-2.7B Tiny-Vicuna-1B 1 C4 d4b4 3.93 86.957 3.21 76.278 Sheared-2.7B Tiny-Vicuna-1B 0.6 CNN/DM d5b3 4.78 81.967 3.71 68.823 Sheared-2.7B Tiny-Vicuna-1B 0.6 CNN/DM d4b4 3.95 87.103 3.25 77.519 Sheared-2.7B Tiny-Vicuna-1B 0.6 OpenWebText d5b3 4.79 82.051 3.64 67.510 Sheared-2.7B Tiny-Vicuna-1B 0.6 OpenWebText d4b4 3.95 86.957 3.21 76.588 Sheared-2.7B Tiny-Vicuna-1B 0.6 C4 d5b3 4.79 81.882 3.65 67.660 Sheared-2.7B Tiny-Vicuna-1B 0.6 C4 d4b4 3.95 87.191 3.24 77.058 Llama-2-7B JF68M 1 CNN/DM d5b3 4.68 95.057 2.51 56.851 Llama-2-7B JF68M 1 CNN/DM d4b4 3.88 84.317 2.47 58.962 Llama-2-7B JF68M 1 OpenWebText d5b3 4.69 94.004 2.51 56.980 Llama-2-7B JF68M 1 OpenWebText d4b4 3.87 84.246 2.47 59.172 Llama-2-7B JF68M 1 C4 d5b3 4.67 94.429 2.64 59.952 Llama-2-7B JF68M 1 C4 d4b4 3.89 84.674 2.55 60.976 Llama-2-7B JF68M 0.6 CNN/DM d5b3 4.14 84.603 2.59 58.997 Llama-2-7B JF68M 0.6 CNN/DM d4b4 3.77 82.035 2.47 59.172 Llama-2-7B JF68M 0.6 OpenWebText d5b3 4.16 84.531 2.49 56.690 Llama-2-7B JF68M 0.6 OpenWebText d4b4 3.75 81.566 2.42 58.072 Llama-2-7B JF68M 0.6 C4 d5b3 4.13 84.388 2.66 60.570 Llama-2-7B JF68M 0.6 C4 d4b4 3.74 81.499 2.57 61.652 Notes: T: temperature; Tree: prediction tree shape; Acc.: accepted length; Thru.: tokens per second. Table 6: Ablation Experiment Results of Different Model Pair Combinations. Here, we use Llama-2-7B and Sheared-LLaMA-2.7B (Xia et al. 2023) as the target model and Sheared-LLaMA-1.3B, TinyLlama-1.1B (Zhang et al. 2024b), Tiny-Vicuna-1B and JF68M as the draft model. The experimental results show that ResiSpec significantly outperforms SpecInfer in terms of accepted length and throughput under different configurations. 2116134641305