Paper deep dive
Approximate Speculative Decoding
Yuannuo Feng, Zegang Peng, Yuxin Xie, Yubing Ye, Yizhe Chen, Wenshuai Yao, Wenyong Zhou, Wang Kang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel. Under standard greedy verification, decoding stops at the first draft token that differs from the target argmax, discarding the remaining target-scored suffix. Although accepting such a mismatch changes the decoding trajectory, it can make a contiguous suffix reusable when its tokens remain target-greedy under the realized prefix. In this paper, we introduce \textbf{Approximate Speculative Decoding (ASD)}, a training-free verifier that replaces binary first-mismatch truncation with budgeted longest-prefix selection. ASD accepts selected mismatches subject to a local target-logit regret gate, a per-block exception cap, and a persistent request-level regret budget, then reuses the contiguous target-greedy suffix without additional approximate decisions or target-model forward passes. ASD requires neither a new draft model nor fine-tuning, and exactly reduces to standard greedy verification when the budget is zero. Experiments show that ASD improves fixed-workload throughput by $3.05\%$--$15.26\%$ over matched strict verification and averages a $7.78\%$ gain across seven Qwen3-14B + DSpark-14B tasks. On DeepSeek-V4-Flash (284B) with DSpark it also raises verifier-side acceptance by roughly $10\%$--$16\%$ on GSM8K and MATH-500 in an FP4-to-FP8 compatibility setting. The source code is publicly available at: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2608.03447v1
- Canonical: https://arxiv.org/abs/2608.03447v1
Trouble viewing inline? Open PDF directly →
Full Text
40,427 characters extracted from source content.
Expand or collapse full text
Approximate Speculative Decoding Yuannuo Feng1, Zegang Peng2 Yuxin Xie1, Yubing Ye3, Yizhe Chen1, Wenshuai Yao5, Wenyong Zhou4 , Wang Kang1 Abstract Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel. Under standard greedy verification, decoding stops at the first draft token that differs from the target argmax, discarding the remaining target-scored suffix. Although accepting such a mismatch changes the decoding trajectory, it can make a contiguous suffix reusable when its tokens remain target-greedy under the realized prefix. In this paper, we introduce Approximate Speculative Decoding (ASD), a training-free verifier that replaces binary first-mismatch truncation with budgeted longest-prefix selection. ASD accepts selected mismatches subject to a local target-logit regret gate, a per-block exception cap, and a persistent request-level regret budget, then reuses the contiguous target-greedy suffix without additional approximate decisions or target-model forward passes. ASD requires neither a new draft model nor fine-tuning, and exactly reduces to standard greedy verification when the budget is zero. Experiments show that ASD improves fixed-workload throughput by 3.05%3.05\%–15.26%15.26\% over matched strict verification and averages a 7.78%7.78\% gain across seven Qwen3-14B + DSpark-14B tasks. On DeepSeek-V4-Flash (284B) with DSpark it also raises verifier-side acceptance by roughly 10%10\%–16%16\% on GSM8K and MATH-500 in an FP4-to-FP8 compatibility setting. The source code is publicly available at: https://github.com/Kissmetothemoon/ASD Introduction Autoregressive large language models (LLMs) generate tokens sequentially, requiring repeated execution of a large target model and making decoding a major inference bottleneck. Early blockwise decoding exposed parallelism by predicting several future tokens at once (Stern et al. 2018); speculative decoding subsequently paired a lightweight proposal model with parallel target verification while preserving the target behavior (Leviathan et al. 2023; Chen et al. 2023). This framework has since expanded across learned drafters, self-speculation, and structured draft trees, making both proposal quality and verification policy central to practical acceleration. Under standard greedy verification, the verifier commits the longest prefix whose draft tokens exactly match the target argmax, stops at the first mismatch, and appends a target recovery token. This rule exactly preserves the target greedy trajectory, but it is inherently binary: a nearly tied draft token is treated identically to a strongly disfavored one. Figure 1: ASD verification. Strict verification stops at mismatch E. ASD accepts E within budget, reuses the target-greedy suffix F,GF,G under the realized prefix, and recovers at H after the same target pass. Figure 2: Different token paths can yield the same task-level answer. In this example, distinct reasoning trajectories produce the same normalized final answer. This motivates a separate task-quality audit, but does not imply that any approximate trajectory is individually exact or safe. The first mismatch also truncates a block that the target has already scored. In a teacher-forced verification pass, each later target row is conditioned on the preceding draft tokens, including any earlier mismatch. Consequently, if a verifier elects to accept an early mismatch, a subsequent contiguous draft suffix may already be target-greedy under the resulting realized prefix. These tokens can be committed without another target forward pass or additional approximate token decisions. Figure 1 illustrates this opportunity: strict verification stops at mismatch E, whereas accepting E enables reuse of F and G, which are already target-greedy under the prefix containing E. Figure 2 shows why token-level divergence does not necessarily imply task-level failure. Alternative connectives, commutative expressions such as 37×4837 ×48 and 48×3748 ×37, and answer formats such as “1776,” “1,776,” and 1776 \1776\ can preserve the same result. However, token mismatch is only an imperfect proxy for task quality and does not make an approximate trajectory exact or safe. Accepting a non-greedy draft token therefore changes the decoding trajectory and cannot be treated as a lossless optimization. The problem is not to ignore mismatches, but to permit a small number of explicit exceptions while controlling their cumulative magnitude over an entire request. A purely local verifier may repeatedly accept individually inexpensive mismatches across decoding rounds, allowing approximation to accumulate with output length. Similarly, resetting an allowance in every draft block fails to account for deviations already introduced earlier in the same completion. In this paper, we introduce Approximate Speculative Decoding (ASD), a training-free verifier that replaces first-mismatch truncation with budgeted longest-prefix selection. At each mismatch, ASD measures local target-logit regret and accepts the longest prefix satisfying a regret threshold, block-level exception cap, and request-level regret budget. It reuses subsequent draft tokens that remain target-greedy under the realized prefix and requires only the standard target verification pass. With a zero budget, ASD exactly recovers strict token-ID greedy verification. ASD bounds accumulated local regret but does not guarantee identical outputs, semantic preservation, or task correctness. Our contributions are threefold: • We formulate approximate greedy verification as a budgeted longest-contiguous-prefix selection problem, using a persistent request-level ledger to account for accepted draft–target mismatches. • We identify and formalize realized-prefix suffix reuse: after an accepted exception, a contiguous target-greedy suffix can be committed without additional approximate token decisions, provided that target and draft caches are updated consistently. • Experiments show that training-free verifier-side ASD improves fixed-workload throughput by up to 15.26%15.26\% over matched strict speculative decoding and by 7.78%7.78\% on average across seven Qwen3-14B + DSpark-14B tasks. It also improves acceptance on DeepSeek-V4-Flash (285B) with DSpark by roughly 10%10\%–16%16\% on GSM8K and MATH-500 under FP4-to-FP8 compatibility. Related Work Speculative decoding and proposal mechanisms. Speculative decoding accelerates autoregressive generation by using a lightweight drafter to propose tokens that are verified in parallel by a target model (Leviathan et al. 2023; Chen et al. 2023). Prior work improves proposal quality and diversity through recurrent, retrieval-augmented, non-autoregressive, adaptive, multi-head, and self-speculative drafters (Zhao et al. 2024; Cheng et al. 2024; Cai et al. 2024; Li et al. 2024; Liu et al. 2024; Gao et al. 2026). Tree-based methods further optimize the structure or width of candidates evaluated in one target pass (Miao et al. 2024; Chen et al. 2024; Wang et al. 2025a; Xiong et al. 2025; Qin et al. 2025). ASD is complementary: rather than changing proposal generation or organization, it changes the verifier’s commitment decision after a proposed block has been scored. Exact and efficient verification. Exact speculative sampling uses rejection-sampling corrections to preserve the target distribution, while greedy verification accepts draft tokens only when they match the target argmax (Chen et al. 2023; Leviathan et al. 2023). Block Verification and Polybasic Speculative Decoding improve or analyze exact multi-token and multi-model verification (Sun et al. 2025b; Wang et al. 2025c). Other methods reduce draft–target coordination or verifier cost through adaptive policies, overlapped drafting and verification, partial KV-state checks, or sparse target computation (Liu et al. 2025; Kumar et al. 2026; Tan et al. 2025; Wang et al. 2025b). In contrast, ASD deliberately permits a bounded number of non-greedy draft tokens, targeting an explicit speed–behavior trade-off rather than exact target-output identity. Relaxed verification. Recent relaxed methods improve acceptance using criteria beyond exact drafter–target agreement, including semantic judgments, divergence constraints, uncertainty-aware thresholds, and target-logit margins (Bachmann et al. 2025; Holsman et al. 2025; Sun et al. 2025a; Hao and Mou 2026; Song et al. 2026). ASD differs by formulating deterministic greedy verification as a longest contiguous prefix-selection problem with a persistent request-level regret ledger. This ledger bounds cumulative target disagreement across decoding rounds, while suffix reuse commits subsequent target-greedy tokens without further approximate decisions or target-model forward passes. Figure 3: Verifier-side opportunity. (a) Ω^(τ,3) (τ,3) measures low-regret mismatches followed by at least three target-greedy draft tokens. (b) Fixed-workload TPS change over strict DSpark. The panels are complementary, not causal. Approximate Speculative Decoding Verifier-side Opportunity Consider a committed history h0h_0. A draft model proposes a block x1:K=(x1,…,xK)x_1:K=(x_1,…,x_K), and a teacher-forced target pass produces logits zi(⋅)=Target(h0,x<i),yi∗=argmaxvzi(v),z_i(·)=Target(h_0,x_<i), y_i^*= _vz_i(v), (1) where yi∗y_i^* denotes the target greedy token at position i. Standard greedy verification commits only the longest prefix that exactly matches the target greedy tokens: astrict=maxa∈0,…,K:xi=yi∗,∀i≤a.a_strict= \a∈\0,…,K\:x_i=y_i^*,\ ∀ i≤ a \. (2) If astrict<Ka_strict<K, it then appends the target recovery token yastrict+1∗y_a_strict+1^*. This rule recovers the target greedy trajectory, but treats every mismatch identically regardless of how strongly the target prefers its argmax. The first mismatch does not imply that all later target computation is useless. In particular, the target row zjz_j for a later position j is conditioned on the draft-forced prefix (h0,x<j)(h_0,x_<j), which includes any earlier draft mismatch. If that mismatch is accepted, this teacher-forced history becomes the realized decoding history. Thus, later draft tokens that equal their corresponding target argmax may already be target-greedy under the realized prefix. Figure 1 illustrates this case: accepting E makes the already scored tokens F and G reusable, whereas strict verification discards them after the mismatch at E. To quantify the availability of this pattern, we define the local target-logit regret ri=zi(yi∗)−zi(xi)≥0.r_i=z_i(y_i^*)-z_i(x_i)≥ 0. (3) Because the softmax normalizer cancels, rir_i is also the target’s conditional log-probability preference for yi∗y_i^* over xix_i. For a threshold τ and a desired reusable suffix length p, we measure the fraction of mismatch positions that are both low-regret and followed by p contiguous target-greedy draft tokens: Ω^(τ,p)=1|ℳ|∑(x,i)∈ℳ[ri≤τ][i+p≤K]∏ℓ=i+1i+p[xℓ=yℓ∗], (τ,p)= 1|M| _(x,i) I[r_i≤τ]I[i+p≤ K] _ =i+1^i+pI[x_ =y_ ^*], (4) where ℳM is the set of draft–target mismatch positions over the evaluated blocks. Figure 3 shows that such potentially reusable suffixes occur across tasks and local-regret thresholds. The fixed-workload TPS changes in Figure 3(b) provide complementary evidence that verifier-side reuse can translate into system-level gains, but do not establish a causal mapping from opportunity rate to throughput. Figure 4: Preliminary speed–quality frontier. Budget and cap move the discovery trade-off, motivating joint local and request-level controls. The frontier selects a conservative configuration; it is not a holdout result. These opportunities cannot be exploited by an unconstrained relaxed verifier. Accepting a non-greedy token changes the decoding trajectory, and individually inexpensive mismatches can accumulate across draft blocks and over long completions. Figure 4 shows that the request budget and per-block exception cap move the throughput–accuracy trade-off, motivating separate controls over local, block-level, and request-level approximation. ASD implements these controls through a budgeted contiguous-prefix verifier. Budgeted Prefix Verification ASD treats a draft–target mismatch as an explicit exception. Let di=[xi≠yi∗]d_i=I[x_i≠ y_i^*] (5) denote the mismatch indicator. For a mismatch at position i, the regret in Equation 3 measures the local target preference sacrificed by accepting xix_i. It is an accounting quantity, rather than a calibrated error probability or a guarantee about downstream task quality. Earlier exceptions can potentially unlock a longer already scored suffix. We therefore use the remaining proposal length qi=K−i+1q_i=K-i+1 (6) as a training-free proxy for this opportunity. ASD applies a normalized local gate, riqi≤g, r_iq_i≤ g, (7) where g controls the largest local regret accepted relative to the remaining proposal length. This proxy is not an optimality claim; it simply favors earlier exceptions only when their local cost is proportionate to their potential suffix-reuse opportunity. Each request maintains a persistent state (B,s)(B,s), where B is the total regret budget and s is the regret already spent by accepted exceptions in previous decoding rounds. For a candidate prefix ending at position t, define its cumulative regret and exception count as Ct=∑i=1tdiri,Nt=∑i=1tdi.C_t= _i=1^td_ir_i, N_t= _i=1^td_i. (8) Given a per-block exception cap M, the prefix x1:tx_1:t is feasible if Ct C_t ≤B−s, ≤ B-s, (9) Nt N_t ≤M, ≤ M, (10) riqi r_iq_i ≤gfor every mismatch i≤t. ≤ g every mismatch i≤ t. (11) ASD returns the longest feasible prefix: aASD=maxt∈0,…,K:x1:tis feasible.a_ASD= \t∈\0,…,K\:x_1:t\ is feasible \. (12) The contiguous-prefix requirement is essential. ASD never skips an infeasible token to accept a later one: doing so would create a gap between the target rows used for verification and the history represented by the committed cache. The three constraints have distinct roles. The local gate filters individually costly exceptions, the block cap prevents exceptions from concentrating within one proposal block, and the persistent budget bounds their cumulative local regret throughout the request. Algorithm 1 ASD for one greedy draft block 0: Draft block x1:Kx_1:K; target greedy tokens y1:K+1∗y_1:K+1^*; target logits zi(yi∗)z_i(y_i^*) and zi(xi)z_i(x_i) for i∈[1,K]i∈[1,K]; state (B,s)(B,s); local gate g; block cap M 1: if B=0B=0 then 2: return exact greedy verifier output and unchanged state (B,s)(B,s) 3: end if 4: di←[xi≠yi∗]d_i [x_i≠ y_i^*] and ri←zi(yi∗)−zi(xi)r_i← z_i(y_i^*)-z_i(x_i) for i∈[1,K]i∈[1,K] 5: qi←K−i+1q_i← K-i+1 for i∈[1,K]i∈[1,K] 6: Ci←∑j=1idjrjC_i← _j=1^id_jr_j and Ni←∑j=1idjN_i← _j=1^id_j for i∈[1,K]i∈[1,K] 7: Construct f1:Kf_1:K using Equation 14 8: a←a← length of the longest all-true prefix of f1:Kf_1:K 9: s←s+∑i=1adiris← s+ _i=1^ad_ir_i 10: if a<Ka<K then 11: return x1:ax_1:a followed by recovery token ya+1∗y_a+1^*, updated state (B,s)(B,s) 12: else 13: return x1:Kx_1:K followed by bonus token yK+1∗y_K+1^*, updated state (B,s)(B,s) 14: end if Decoding Procedure and Cache Semantics Suffix reuse under the realized prefix. Suppose ASD commits x1:aASDx_1:a_ASD. For every committed position j≤aASDj≤ a_ASD, the target row zjz_j was computed under the history (h0,x<j)(h_0,x_<j). Since ASD commits the same contiguous prefix, this is exactly the realized history before emitting xjx_j. Proposition 1 (Realized-prefix suffix reuse) If a committed draft token xjx_j satisfies xj=yj∗x_j=y_j^*, then xjx_j is target-greedy under the realized ASD history and introduces no additional token-level exception. The proposition does not restore equivalence to the strict target-greedy trajectory. An earlier accepted mismatch may already have changed the history on which subsequent target decisions are conditioned. Rather, it establishes that a contiguous suffix of target-greedy draft tokens following an accepted exception can be reused without spending further regret or executing another target forward pass. Benchmark Speedup Threshold Target-only Strict SD ASD ASD vs. Strict (%) Strict τ ASD τ ASD Δ . GSM8K 1.001.00 5.585.58 6.306.30 10.08±0.5010.08± 0.50 4.424.42 4.914.91 0.150.15 MATH-500 1.001.00 6.886.88 7.327.32 11.73±0.3811.73± 0.38 5.355.35 6.026.02 1.171.17 HumanEval 1.001.00 4.684.68 5.145.14 6.90±0.616.90± 0.61 3.753.75 4.054.05 −0.61-0.61 MBPP 1.001.00 5.045.04 5.365.36 3.64±0.263.64± 0.26 3.973.97 4.154.15 0.540.54 MMLU 1.001.00 3.303.30 3.543.54 4.41±0.574.41± 0.57 2.622.62 2.772.77 0.730.73 MT-Bench 1.001.00 4.684.68 5.165.16 7.79±0.687.79± 0.68 3.623.62 3.923.92 −0.64-0.64 Alpaca 1.001.00 4.224.22 4.664.66 9.94±0.359.94± 0.35 3.243.24 3.603.60 0.100.10 Table 1: Results for Qwen3-14B with DSpark-14B (B=8B=8, g=0.25g=0.25, M=2M=2). Speedups are relative to target-only decoding. ASD consistently outperforms matched strict speculative decoding, with small task-dependent accuracy changes under natural-EOS decoding. Parentheses denote confidence-interval half-widths; accuracy changes are reported per task and not averaged across heterogeneous metrics. Ledger update and recovery. After committing the selected prefix, ASD updates the persistent ledger: s′=s+CaASD.s =s+C_a_ASD. (13) By construction, Equation 9 ensures s′≤Bs ≤ B. Thus, the ledger bounds the sum of local regrets from all accepted exceptions along the realized request trajectory. It is not a complete-sequence likelihood-ratio bound, and it does not imply output identity, distributional equivalence, semantic preservation, safety, or task correctness. If aASD<Ka_ASD<K, the target row zaASD+1z_a_ASD+1 already provides the recovery token yaASD+1∗y_a_ASD+1^* under the returned prefix. ASD emits this token and begins the next round from the updated history. If aASD=Ka_ASD=K, ASD appends the bonus token yK+1∗y_K+1^* from the final target row. In both cases, the target cache, draft cache, and request-level ledger must be advanced only along the emitted sequence. Proposal-only cache state beyond the returned prefix is discarded. Vectorized implementation. Algorithm 1 implements ASD after the ordinary target verification pass. Let fi=(1−di)∨[riqi≤g∧Ci≤B−s∧Ni≤M]f_i=(1-d_i)\ \ [ r_iq_i≤ g\ \ C_i≤ B-s\ \ N_i≤ M ] (14) be the per-position feasibility mask. The accepted length is the longest all-true prefix of f1:Kf_1:K, which can be computed using cumulative sums followed by a cumulative product. The verifier therefore adds O(K)O(K) arithmetic and temporary storage, but no target-model forward pass. ASD requires only the target top token and top logit, together with the target logit assigned to each proposed draft token. In a tensor-parallel implementation, these values can be obtained without materializing an additional K×||K×|V| logit buffer. Finally, when B=0B=0, ASD dispatches to ordinary token-ID greedy verification rather than accepting tokens solely because ri=0r_i=0. This strict fallback preserves standard greedy behavior even when multiple tokens tie for the maximum target logit. Experiments Experimental Setup Models and runtime. We evaluate Qwen3-8B and Qwen3-14B targets with matched DSpark-8B-block7 and DSpark-14B-block7 drafters, and test transfer to EAGLE3 with Llama-3.1-8B and Qwen2.5-7B and Medusa with Llama-3.1-8B. ASD is inserted only in the verifier and uses the logits from the standard target verification pass; it adds neither training nor target-model forward passes. All comparisons use greedy decoding, an unchanged target and prompt template, and identical stop-token handling. Unless stated otherwise, each task evaluation is repeated four times, with each strict–ASD–strict triplet run sequentially on one NVIDIA L20 GPU (46,068 MiB) with Linux 5.4, Python 3.12.13, PyTorch 2.8.0+cu128, Transformers 4.55.2, and vLLM 0.11.0. Tasks and evaluation protocol. We evaluate GSM8K, MATH-500, HumanEval, MBPP, MMLU, MT-Bench, and Alpaca. Hyperparameter discovery uses disjoint GSM8K fixed-workload (64×25664× 256 tokens) and natural-EOS (128 prompts, at most 512 tokens) slices; selected controls are frozen before multi-task evaluation. The full configuration and per-run records are provided in the supplementary material. Fixed-work decoding equalizes completion-token counts and measures ASD TPS against the mean of adjacent strict roles; we retain only triplets with matching strict hashes and token counts and at most 3%3\% baseline drift. Reported TPS intervals are 95% Student-t half-widths in percentage points of relative TPS gain. Natural-EOS decoding separately audits accuracy, length, and hash divergence, where hash changes diagnose altered trajectories rather than task quality. Main Results We ask two questions: does relaxing only budget-feasible mismatches improve end-to-end throughput over strict verification, and does the benefit persist when the drafter family changes? Table 1 answers the first question on Qwen3-14B with DSpark-14B using the frozen (B,g,M)=(8,0.25,2)(B,g,M)=(8,0.25,2) configuration. ASD improves fixed-workload TPS on all seven tasks, by 3.64%3.64\%–11.73%11.73\% (mean 7.78%7.78\%) over matched strict SD. The strongest gains occur on MATH-500 (11.73%11.73\%) and GSM8K (10.08%10.08\%), where mean accepted length also rises by 0.670.67 and 0.490.49 tokens per round, respectively. Averaged over all tasks, τ increases from 3.853.85 to 4.204.20. These aligned changes are the expected systems signature of ASD: a permitted low-regret exception exposes a contiguous suffix that is already target-greedy under the realized prefix, allowing the verifier to commit more tokens per target pass. Method Target model Benchmark Speedup Threshold Target-only Strict SD ASD ASD vs. Strict(%) Strict τ ASD τ ASD Δ . DSpark Qwen3-14B GSM8K 1.001.00 5.585.58 6.306.30 10.08±0.5010.08± 0.50 4.424.42 4.914.91 0.150.15 Qwen3-14B MMLU 1.001.00 3.303.30 3.543.54 4.41±0.574.41± 0.57 2.622.62 2.772.77 0.220.22 Qwen3-8B GSM8K 1.001.00 5.865.86 6.166.16 4.88±0.434.88± 0.43 5.315.31 5.625.62 −0.15-0.15 Qwen3-8B MMLU 1.001.00 2.782.78 2.962.96 3.05±0.903.05± 0.90 2.552.55 2.672.67 0.560.56 EAGLE3 Llama-3.1-8B GSM8K 1.001.00 4.744.74 5.345.34 15.26±0.1515.26± 0.15 3.363.36 3.883.88 −1.17-1.17 Llama-3.1-8B MMLU 1.001.00 2.882.88 3.223.22 11.94±0.3011.94± 0.30 2.102.10 2.372.37 0.000.00 Qwen2.5-7B GSM8K 1.001.00 4.104.10 4.304.30 6.08±0.306.08± 0.30 2.862.86 3.033.03 −0.39-0.39 Qwen2.5-7B MMLU 1.001.00 2.502.50 2.662.66 8.11±0.218.11± 0.21 1.761.76 1.911.91 0.500.50 Medusa Llama-3.1-8B GSM8K 1.001.00 2.702.70 2.942.94 6.72±0.136.72± 0.13 2.032.03 2.172.17 −1.52-1.52 Llama-3.1-8B MMLU 1.001.00 1.821.82 1.941.94 4.70±0.144.70± 0.14 1.371.37 1.441.44 0.000.00 Table 2: Cross-family ASD results under matched fixed-work and natural-EOS protocols. Speedups are measured relative to target-only decoding. Values in parentheses denote the corresponding confidence intervals. Control Value Fixed-work gain (%) Δ . (p) B 4 4.96 −0.78-0.78 8 6.53 +2.34+2.34 12 6.98 +0.78+0.78 16 6.64 +1.56+1.56 g 0.125 2.84 +1.56+1.56 0.25 6.53 +2.34+2.34 0.5 7.62 +0.78+0.78 ∞ 6.62 +0.00+0.00 M 1 6.20 +0.00+0.00 2 6.53 +2.34+2.34 ∞ 6.70 +2.34+2.34 Table 3: One-at-a-time ASD control sweeps on GSM8K. Blue cells indicate the selected operating points. Figure 5: Regret-gate sweep on GSM8K under natural-EOS decoding (n=256n=256, one seed). The natural-EOS audit separates this throughput result from behavioral preservation. On the primary DSpark-14B matrix, five of seven task rows have non-negative accuracy changes, while the two observed reductions are small (−0.61-0.61 points on HumanEval and −0.64-0.64 on MT-Bench). At the same time, GSM8K and MATH-500 exhibit hash divergence above 95%95\%, even though their measured accuracy does not decrease. Thus, the result is not an output-preserving speedup: the accuracy audit shows that the selected budget can retain task performance on most measured tasks, and the hash audit makes the altered trajectories explicit. Table 2 tests generalization across DSpark, EAGLE3, and Medusa. ASD improves throughput in every reported cell, spanning 3.05%3.05\%–15.26%15.26\% with a 7.52%7.52\% mean gain across the ten cells. The accepted length increases in each cell as well (+0.07+0.07 to +0.52+0.52 tokens), which ties the cross-family gains to the same verifier-side mechanism rather than a family-specific change in proposal generation. In particular, the largest gains, 15.26%15.26\% on Llama-3.1-8B + EAGLE3 for GSM8K and 11.94%11.94\% for MMLU, show that budgeted prefix selection remains effective with a different draft architecture. Taken together, the two tables show that ASD’s throughput benefit is systematic rather than driven by a small subset of favorable workloads. Gains are positive on all seven tasks in the primary DSpark-14B evaluation and in all ten cells of the cross-family study; moreover, every reported 95% confidence interval remains strictly above zero. These improvements are obtained on top of already strong strict-SD baselines, which span 1.82×1.82×–6.88×6.88× target-only speedup, with ASD raising the corresponding range to 1.94×1.94×–7.32×7.32×. The benefit also covers a broad range of baseline acceptance levels. Across the two tables, strict accepted length ranges from 1.371.37 to 5.355.35 tokens per round, yet ASD increases it in every setting. This consistency across four target models and three drafter families indicates that the verifier does not depend on a particular proposal architecture or a narrow baseline-acceptance regime. Because ASD operates only after ordinary target scoring, these gains require neither retraining the drafter nor an additional target-model forward pass, making the method complementary to improvements in proposal generation. The accuracy columns nevertheless reinforce that ASD should be viewed as a controlled approximation rather than an output-preserving optimization. Five of seven primary-task rows and six of ten cross-family cells show non-negative measured accuracy changes, while the remaining reductions are task dependent and no larger than 1.521.52 percentage points. Thus, throughput improvement is considerably more consistent than the direction of the quality change. Together with the observed hash divergence, this asymmetry supports ASD’s intended deployment model: freeze the verifier controls on disjoint data, then audit natural-EOS task quality for each target, drafter, and workload. Analysis and Discussion The ablations test whether ASD’s improvement follows the behavior predicted by budgeted prefix verification rather than reflecting a fixed increase in acceptance. Table 3 varies the request budget B, local gate g, and per-block cap M around the frozen (8,0.25,2)(8,0.25,2) configuration. These controls address complementary failure modes: B bounds cumulative request-level deviation, g rejects locally costly mismatches, and M prevents exceptions from concentrating within a draft block. In each sweep, the other two controls remain fixed; boldface marks the highest fixed-work gain, and blue shading indicates the frozen configuration used in the main results. Reporting fixed-work gain together with the separate natural-EOS accuracy audit exposes the operating trade-off. The selected configuration therefore balances throughput gains against the measured task behavior used for selection. Budget control and reusable suffix opportunity. Figure 6(a) makes the request budget an interpretable operating knob. At B=0B=0, ASD’s verifier is exactly strict greedy verification. Increasing B admits low-regret exceptions and yields substantial throughput gains on Alpaca, GSM8K, MATH-500, and MT-Bench; the flattening at larger budgets shows that the useful suffix opportunity is finite rather than an artifact of unconstrained acceptance. The selected B=8B=8 therefore lies on the rising part of this response, before the high-budget plateau. Figure 6(b) independently varies the draft horizon under fixed-work decoding (n=64n=64, one seed). ASD outperforms strict verification at every tested horizon, and the gain increases with K on all three workloads. This coupled horizon response is the expected signature of realized-prefix suffix reuse: more target-scored positions remain available after an early draft mismatch. Figure 6: ASD ablations with Qwen3-14B + DSpark-14B: (a) request-budget response, (b) draft-horizon response under fixed-work decoding (n=64n=64, one seed), (c) MARS-style and Fuzzy-style local controls at B=8B=8, and (d) synchronized per-token latency. Figure 7: DeepSeek-V4-Flash + DSpark acceptance ablations on eight H20 GPUs: (a) GSM8K gate sweep (n=256n=256), (b) GSM8K power-set evaluation (n=744n=744), (c) GSM8K-Confirm validation (n=1,000n=1,000), and (d) cross-task accepted tokens per proposal (A/P). Budget-matched verifier controls and latency. Figure 6(c) compares ASD with audited MARS-style and Fuzzy-style local verifier controls after applying the same maximum-realized-regret criterion. ASD is best on both tasks: +6.5%+6.5\% on GSM8K, versus +5.3%+5.3\% and +5.1%+5.1\%, and +9.7%+9.7\% on MATH-500, versus +6.0%+6.0\% and +7.3%+7.3\%, respectively. This comparison distinguishes ASD’s request-level accounting from a locally permissive rule that has no persistent budget. Figure 6(d) closes the systems loop. In the synchronized profiling shown, verifier logic increases by only 0.0830.083 ms per output token on Alpaca and 0.0450.045 ms on GSM8K, while target verification falls by 1.4791.479 and 1.5091.509 ms, respectively. Thus, the gain is not created by hiding verifier overhead: the bounded exceptions commit useful target-scored suffixes and reduce expensive verifier rounds. The regret gate controls the speed–behavior frontier. Figure 5 isolates g under natural-EOS decoding. This GSM8K audit uses n=256n=256 prompts and one seed. Relaxing the gate increases both request coverage and throughput gain, as more near-tied draft tokens become eligible exceptions. The selected g=0.25g=0.25 covers 95.3%95.3\% of requests and yields a 7.2%7.2\% TPS gain without a measured GSM8K accuracy change. More permissive gates produce small accuracy fluctuations, demonstrating why local target disagreement must be gated rather than treating every mismatch as reusable. Collectively, these ablations connect the controls to ASD’s intended mechanism: a bounded, low-regret exception unlocks subsequent target-greedy suffix tokens and thereby reduces the number of verifier rounds. Large-model acceptance ablation. Figure 7 summarizes the development, confirmation, and fresh-validation acceptance experiments; complete numerical results and per-run records are provided in the supplementary material. In the development GSM8K sweep, relaxing g increases accepted draft tokens per proposal (A/P), although accuracy fluctuations motivate confirmation. The 744-example GSM8K power-set evaluation preserves this trend. On the independent 1,000-example GSM8K-Confirm split, g∈2,3g∈\2,3\ improves A/P by 10.08%10.08\%–11.48%11.48\% while changing accuracy by at most 0.300.30 percentage points. HumanEval and MATH-500 show similar acceptance gains. DSpark uses FP4 mixed precision for DeepSeek-V4-Flash, whereas the Hopper-based H20 supports native FP8 rather than FP4; the resulting compatibility path inserts additional quantization–dequantization (Q/DQ) operations, so these experiments therefore characterize task accuracy and verifier-side acceptance only. Conclusion ASD is a training-free, verifier-side method that uses bounded low-regret exceptions to reuse target-scored suffix tokens. It reaches up to 15.26%15.26\% throughput improvement over strict verification, with a 7.78%7.78\% average gain on seven Qwen3-14B + DSpark-14B tasks and positive gains in every reported DSpark, EAGLE3, and Medusa setting. By requiring no target or drafter training, ASD provides a broadly compatible path to faster greedy blockwise speculative decoding, subject to task-level quality auditing. On DeepSeek-V4-Flash with DSpark, ASD also raises verifier-side acceptance across additional datasets, including roughly 10%10\%–16%16\% gains on GSM8K and MATH-500. References G. Bachmann, S. Anagnostidis, A. Pumarola, M. Georgopoulos, A. Sanakoyeu, Y. Du, E. Schoenfeld, A. Thabet, and J. Kohler (2025) Judge decoding: faster speculative sampling requires going beyond model alignment. In International Conference on Learning Representations (ICLR), External Links: 2501.19309 Cited by: Relaxed verification.. T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao (2024) Medusa: simple LLM inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning (ICML), External Links: 2401.10774 Cited by: Speculative decoding and proposal mechanisms.. C. Chen, S. Borgeaud, G. Irving, J. Lespiau, L. Sifre, and J. Jumper (2023) Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. Cited by: Introduction, Speculative decoding and proposal mechanisms., Exact and efficient verification.. Z. Chen, A. May, R. Svirschevski, Y. Huang, M. Ryabinin, Z. Jia, and B. Chen (2024) Sequoia: scalable and robust speculative decoding. In Advances in Neural Information Processing Systems (NeurIPS), p. 129531–129563. External Links: Document, 2402.12374 Cited by: Speculative decoding and proposal mechanisms.. Y. Cheng, A. Zhang, X. Zhang, C. Wang, and Y. Wang (2024) Recurrent drafter for fast speculative decoding in large language models. arXiv preprint arXiv:2403.09919. Cited by: Speculative decoding and proposal mechanisms.. X. Gao, W. Xie, Lixin, X. Fang, C. Hang, C. Li, Y. Lin, and X. Xu (2026) Talon: breaking the synchronization barrier in speculative decoding with hybrid model-based and retrieve-based drafting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 21216–21224. External Links: Document Cited by: Speculative decoding and proposal mechanisms.. Y. Hao and L. Mou (2026) Cactus: accelerating auto-regressive decoding with constrained acceptance speculative sampling. In International Conference on Learning Representations (ICLR), External Links: 2604.04987 Cited by: Relaxed verification.. M. Holsman, Y. Huang, and B. Dhingra (2025) Fuzzy speculative decoding for a tunable accuracy-runtime tradeoff. In Findings of the Association for Computational Linguistics: ACL 2025, p. 26257–26273. External Links: Document Cited by: Relaxed verification.. T. Kumar, T. Dao, and A. May (2026) Speculative speculative decoding. In International Conference on Learning Representations (ICLR), External Links: 2603.03251 Cited by: Exact and efficient verification.. Y. Leviathan, M. Kalman, and Y. Matias (2023) Fast inference from transformers via speculative decoding. In International Conference on Machine Learning (ICML), External Links: 2211.17192 Cited by: Introduction, Speculative decoding and proposal mechanisms., Exact and efficient verification.. Y. Li, F. Wei, C. Zhang, and H. Zhang (2024) EAGLE: speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning (ICML), External Links: 2401.15077 Cited by: Speculative decoding and proposal mechanisms.. F. Liu, Y. Tang, Z. Liu, Y. Ni, D. Tang, K. Han, and Y. Wang (2024) Kangaroo: lossless self-speculative decoding for accelerating LLMs via double early exiting. In Advances in Neural Information Processing Systems (NeurIPS), p. 11946–11965. External Links: Document, 2404.18911 Cited by: Speculative decoding and proposal mechanisms.. X. Liu, B. Lei, R. Zhang, and D. Xu (2025) Adaptive draft-verification for efficient large language model decoding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 24668–24676. External Links: Document Cited by: Exact and efficient verification.. X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y. Y. Wong, A. Zhu, L. Yang, X. Shi, C. Shi, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia (2024) SpecInfer: accelerating large language model serving with tree-based speculative inference and verification. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), p. 932–949. External Links: Document Cited by: Speculative decoding and proposal mechanisms.. Z. Qin, Z. He, N. Prakriya, J. Cong, and Y. Sun (2025) Dynamic-width speculative beam decoding for LLM inference. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 25056–25064. External Links: Document Cited by: Speculative decoding and proposal mechanisms.. J. Song, X. Wang, H. Wang, X. Lei, B. Shi, S. Han, E. Yang, X. Chang, and L. Ai (2026) MARS: unleashing the power of speculative decoding via margin-aware verification. In Findings of the Association for Computational Linguistics: ACL 2026, p. 8652–8663. External Links: Document Cited by: Relaxed verification.. M. Stern, N. Shazeer, and J. Uszkoreit (2018) Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 31. External Links: 1811.03115 Cited by: Introduction. C. Sun, A. Mao, L. Xu, and M. Chen (2025a) Efficient adaptive rejection sampling for accelerating speculative decoding in large language models. arXiv preprint arXiv:2512.13194. Cited by: Relaxed verification.. Z. Sun, U. Mendlovic, Y. Leviathan, A. Aharoni, J. H. Ro, A. Beirami, and A. T. Suresh (2025b) Block verification accelerates speculative decoding. In International Conference on Learning Representations (ICLR), External Links: 2403.10444 Cited by: Exact and efficient verification.. Z. Tan, X. Zhang, C. Hu, J. Peng, and K. Xia (2025) SpecPV: improving self-speculative decoding for long-context generation via partial verification. arXiv preprint arXiv:2512.02337. Cited by: Exact and efficient verification.. J. Wang, Y. Su, J. Li, Q. Xia, Z. Ye, X. Duan, Z. Wang, and M. Zhang (2025a) OPT-Tree: speculative decoding with adaptive draft tree structure. Transactions of the Association for Computational Linguistics 13, p. 188–199. External Links: Document Cited by: Speculative decoding and proposal mechanisms.. J. Wang, J. Tan, Y. Hu, J. Qin, Y. Sun, Y. Xie, X. Cai, J. Li, and M. Zhang (2025b) Accelerate speculative decoding with sparse computation in verification. arXiv preprint arXiv:2512.21911. Cited by: Exact and efficient verification.. R. Wang, H. Li, Y. Ma, X. Zheng, F. Chao, X. Xiao, and R. Ji (2025c) Polybasic speculative decoding through a theoretical perspective. arXiv preprint arXiv:2510.26527. Cited by: Exact and efficient verification.. Y. Xiong, R. Zhang, Y. Li, and L. Zou (2025) DySpec: faster speculative decoding with dynamic token tree structure. World Wide Web 28 (3). External Links: Document Cited by: Speculative decoding and proposal mechanisms.. W. Zhao, Y. Huang, X. Han, W. Xu, C. Xiao, X. Zhang, Y. Fang, K. Zhang, Z. Liu, and M. Sun (2024) Ouroboros: generating longer drafts phrase by phrase for faster speculative decoding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 13378–13393. External Links: Document Cited by: Speculative decoding and proposal mechanisms..