Paper deep dive
TAPS: Task Aware Proposal Distributions for Speculative Sampling
Mohamad Zbib, Mohamad Bazzi, Ammar Mohanna, Hasan Abed Al Kader Hammoud, Bernard Ghanem
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/31/2026, 1:59:16 AM
Summary
The paper introduces TAPS (Task Aware Proposal Distributions for Speculative Sampling), a framework that optimizes speculative decoding by aligning draft model training distributions with downstream workloads. It evaluates HASS and EAGLE-2 drafters on various datasets, finding that task-specific training yields significant specialization. The authors demonstrate that inference-time composition strategies, such as confidence-based routing and merged-tree verification, outperform naive weight-space checkpoint averaging.
Entities (6)
Relation Signals (4)
TAPS → utilizes → HASS
confidence 95% · we study this question with lightweight HASS and EAGLE-2 drafters
TAPS → utilizes → EAGLE-2
confidence 95% · we study this question with lightweight HASS and EAGLE-2 drafters
MathInstruct → trains → HASS
confidence 90% · MathInstruct-trained drafts are strongest on reasoning benchmarks
Meta-Llama-3-8B-Instruct → verifies → HASS
confidence 90% · the verifier is Meta-Llama-3-8B-Instruct
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Speculative decoding accelerates autoregressive generation by letting a lightweight draft model propose future tokens that a larger target model then verifies in parallel. In practice, however, draft models are usually trained on broad generic corpora, which leaves it unclear how much speculative decoding quality depends on the draft training distribution. We study this question with lightweight HASS and EAGLE-2 drafters trained on MathInstruct, ShareGPT, and mixed-data variants, evaluated on MT-Bench, GSM8K, MATH-500, and SVAMP. Measured by acceptance length, task-specific training yields clear specialization: MathInstruct-trained drafts are strongest on reasoning benchmarks, while ShareGPT-trained drafts are strongest on MT-Bench. Mixed-data training improves robustness, but larger mixtures do not dominate across decoding temperatures. We also study how to combine specialized drafters at inference time. Naive checkpoint averaging performs poorly, whereas confidence-based routing improves over single-domain drafts and merged-tree verification yields the highest acceptance length overall for both backbones. Finally, confidence is a more useful routing signal than entropy: rejected tokens tend to have higher entropy, but confidence produces much clearer benchmark-level routing decisions. These results show that speculative decoding quality depends not only on draft architecture, but also on the match between draft training data and downstream workload, and that specialized drafters are better combined at inference time than in weight space.
Tags
Links
- Source: https://arxiv.org/abs/2603.27027v1
- Canonical: https://arxiv.org/abs/2603.27027v1
Trouble viewing inline? Open PDF directly →
Full Text
48,563 characters extracted from source content.
Expand or collapse full text
TAPS: Task Aware Proposal Distributions for Speculative Sampling Mohamad Zbib 1,2 , Mohamad Bazzi 2 , Ammar Mohanna 2 , Hasan Abed Al Kader Hammoud 1,‡ , Bernard Ghanem 1,‡ 1 King Abdullah University of Science and Technology (KAUST) 2 American University of Beirut (AUB) ‡ Equal advising authors Speculative decoding accelerates autoregressive generation by letting a lightweight draft model propose future tokens that a larger target model then verifies in parallel. In practice, however, draft models are usually trained on broad generic corpora, which leaves it unclear how much speculative decoding quality depends on the draft training distribution. We study this question with lightweight HASS and EAGLE-2 drafters trained on MathInstruct, ShareGPT, and mixed-data variants, evaluated on MT-Bench, GSM8K, MATH-500, and SVAMP. Measured by acceptance length, task-specific training yields clear specialization: MathInstruct-trained drafts are strongest on reasoning benchmarks, while ShareGPT-trained drafts are strongest on MT-Bench. Mixed-data training improves robustness, but larger mixtures do not dominate across decoding temperatures. We also study how to combine specialized drafters at inference time. Naive checkpoint averaging performs poorly, whereas confidence-based routing improves over single-domain drafts and merged-tree verification yields the highest acceptance length overall for both backbones. Finally, confidence is a more useful routing signal than entropy: rejected tokens tend to have higher entropy, but confidence produces much clearer benchmark-level routing decisions. These results show that speculative decoding quality depends not only on draft architecture, but also on the match between draft training data and downstream workload, and that specialized drafters are better combined at inference time than in weight space. Date: March 2026 Code Repository: https://github.com/Moe-Zbeeb/TAPS Model Weights: https://huggingface.co/collections/zbeeb/taps Datasets: https://huggingface.co/datasets/zbeeb/TAPS-Datasets Contact: mbz02@mail.aub.edu, hasanabedalkader.hammoud@kaust.edu.sa Introduction Large language models (LLMs) achieve strong results across many tasks, but autoregressive decoding remains a major inference bottleneck because each token depends on the full previously generated prefix Brown et al., Leviathan et al. (2023), Chen et al. (2023). Speculative decoding addresses this bottleneck by letting a lightweight draft model propose several future tokens that a larger target model then verifies in parallel. The appeal of the approach is that it can improve throughput without changing the target model’s output distribution. Its usefulness, however, depends critically on the quality of the proposal distribution produced by the drafter. Figure 1 summarizes the setting studied in this paper. Most prior work improves speculative decoding through better draft architectures or more efficient verification procedures. Early speculative decoding methods use a separate lightweight drafter Leviathan et al. (2023), Corresponding author(s): Mohamad Zbib mbz02@mail.aub.edu; Hasan Abed Al Kader Hammoud hasanabedalkader.hammoud@kaust.edu.sa arXiv:2603.27027v1 [cs.CL] 27 Mar 2026 TAPS: Task Aware Proposal Distributions for Speculative Sampling Chen et al. (2023), while later methods improve feature-level drafting and tree construction, as in EAGLE, EAGLE-2, EAGLE-3, and HASS Li et al. (2024a,b, 2025), Zhang et al. (2025). Other lines of work explore tree verification, self-speculative decoding, hierarchical drafting, cascaded drafters, and retrieval-assisted proposals Miao et al. (2024), Zhang et al. (2024), Elhoushi et al. (2024), Liu et al. (2024), Sun et al. (2024), Chen et al. (2024), He et al. (2024). Despite this progress, draft models are still typically trained on broad generic corpora such as ShareGPT, so the role of the draft training distribution remains under-studied. Figure 1: Schematic of the speculative decoding pipeline. Given contextual in- formation from the target LLM, the draft model generates latent proposed tokens, which are converted by the LM head and sampling module into multiple candidate future tokens. These candidates are provi- sional and are later verified by the target model. Importantly, the trainable compo- nent in this framework is the draft model, whose role is to efficiently approximate the target model’s next-token behavior while preserving the target model’s final output distribution after verification. This gap matters for two reasons. First, if the draft model is trained on a distribution that is poorly aligned with the target workload, ac- ceptance length may degrade even when the speculative decoding algorithm itself is unchanged. Second, the open-weight ecosys- tem increasingly offers multiple specialized checkpoints Sun et al. (2025), which raises a practical question: when several specialized drafters are available, is it better to mix their data during training, merge them in weight space, or compose them at inference time Ilharco et al. (2022), Mu and Lin (2026)? We study these questions with lightweight drafters trained under HASS and EAGLE-2 on MathInstruct and ShareGPT. The paper is organized around five research questions. RQ1. Does task-specific training improve speculative decoding on matched downstream tasks? RQ2. Can mixed-data training recover cross-domain robust- ness without erasing specialization? RQ3. How should multiple specialized draft models be combined: weight averaging, routing, or merged-tree verification? RQ4. Are confidence, entropy, and depth-wise acceptance useful signals for explaining routing and acceptance behavior? RQ5. How does speculative depth affect the exploration and exploitation balance in task aware drafting? Our answers are consistent across both speculative backbones. Sin- gle domain training produces clear specialization: MathInstruct- trained drafts are strongest on GSM8K, MATH-500, and SVAMP, while ShareGPT-trained drafts are strongest on MT-Bench. Mixed- data training improves robustness, but larger mixtures are not uni- formly better across temperatures. When multiple specialists are available, naive weight-space averaging performs poorly, whereas inference-time composition is substantially stronger. Confidence- based routing improves over single-domain baselines, and merged- tree verification yields the highest acceptance length overall. Confi- dence is also more useful than entropy as a routing signal, although entropy remains informative as a diagnostic of likely rejection. Depth analysis also shows that speculative decoding becomes increasingly dominated by the task matched specialist at larger depths. The rest of the paper follows the same logic: we first review the decoding backbones, then describe the controlled setup, and finally answer the five research questions with matched evidence. All training and evaluation experiments in this paper were run on a single node with four NVIDIA A100 GPUs. 2 TAPS: Task Aware Proposal Distributions for Speculative Sampling Preliminaries We briefly review only the pieces of speculative decoding that are needed later: the lossless verification rule and the two drafting backbones used in our experiments, EAGLE-2 and HASS. Throughout the paper, the verifier and acceptance rule are fixed; what changes is the draft training distribution and the way multiple specialized drafts are composed at test time. Speculative Decoding Speculative decoding uses a lightweight draft modelpto proposeKfuture tokens and a target modelqto verify them in parallel, thereby reducing the number of expensive target-model calls. Given a prefixx 1∶n , the draft model generates ̃ x n+1∶n+K autoregressively. The target model then scores these candidates, and each drafted token ̃ x n+t is accepted sequentially with probability α n+t = min ( 1, q( ̃ x n+t ∣ x 1∶n+t−1 ) p( ̃ x n+t ∣ x 1∶n+t−1 ) ) .(1) If rejection occurs, decoding instead samples from r(x) ∝ max(0, q(x ∣ x 1∶n+t−1 )− p(x ∣ x 1∶n+t−1 )) .(2) This rejection-sampling correction preserves the target model’s output distribution exactly while allowing multiple proposed tokens to be checked in a single verifier call. In the rest of the paper, acceptance length measures how often this lossless verification procedure approves long draft continuations. EAGLE-2 EAGLE-2 inherits the EAGLE feature-level drafter and improves inference by replacing a fixed draft tree with a context-dependent dynamic tree Li et al. (2024b). Leth t denote the target model’s second-to-last-layer feature at step t. Instead of autoregressing directly on tokens, the draft model g φ predicts future features, ˆ h t+1 = g φ ( ˆ h ≤t , x ≤t+1 ),(3) and the predicted features are mapped to token probabilities through the target model’s LM head. A compact training objective is ℒ EAGLE =∑ t ∥ ˆ h t+1 − h t+1 ∥ 2 2 + λ∑ t CE ( softmax(W ˆ h t+1 ), x t+1 ) .(4) During verification, a drafted token ˆ x j+i is accepted with probability α j+i = min ( 1, p j+i ( ˆ x j+i ) ˆ p j+i ( ˆ x j+i ) ) ,(5) which preserves the target model’s output distribution. EAGLE-2 uses draft confidence to rank frontier nodes in a dynamic draft tree, V i ≈∏ v j ∈Path(root,v i ) c j ,(6) and expands the highest-valued frontier nodes before verification Li et al. (2024b). In this paper, we keep that decoding rule fixed and vary only how the drafter is trained or combined. 3 TAPS: Task Aware Proposal Distributions for Speculative Sampling HASS HASS uses the same lossless speculative decoding framework, but improves the drafter by reducing objective mismatch and context mismatch between training and inference Zhang et al. (2025). Its harmonized objective distillation term focuses learning on the verifier’s most likely next tokens. Letq(⋅)andp(⋅)denote the next-token distributions of the target and draft models, respectively, and let ˆ Ω ⊂Ωbe the set of top-K tokens under q. The Top-K distillation loss is ℒ Top-K = −∑ x∈ ˆ Ω q(x) log p(x).(7) HASS also introduces harmonized context alignment so that later draft predictions are trained on imperfect draft features rather than only clean target features. At alignment step j, the draft model predicts P (s) (x t+1 ∣ x ≤t ) = Head( f (s j ) t+1 ) = Head(M (s) ( f (s j−1 ) t , f (l) 1 ⊕⋯⊕ f (l) t−j+1 ⊕ f (s 1 ) t−j+2 ⊕⋯⊕ f (s j−1 ) t )) , (8) with training objective ℒ (j) HASS = T−1 ∑ t=1 [CE ( P (l) (x t+1 ∣ x ≤t ), P (s) (x t+1 ∣ x ≤t ) ) +ℒ aux ],(9) whereℒ aux includes Top-Kdistillation and feature regression. As with EAGLE-2, our experiments keep the verifier and the lossless acceptance rule fixed and study how different training distributions and composition strategies affect acceptance. Experimental Setup and Composition Strategies Common Setup We study task-aware draft construction for speculative decoding under a fixed verifier. Across all experiments, the verifier is Meta-Llama-3-8B-Instruct Llama Team (2024), and the draft model is a lightweight LLaMA- style decoder with one transformer layer, hidden size 4096, and roughly 0.8B parameters. Draft and target share the same tokenizer and vocabulary so that acceptance differences are not confounded by tokenization mismatch. We evaluate on MT-Bench together with three reasoning-heavy benchmarks, GSM8K, MATH-500, and SVAMP, at temperatures 0 and 1. Our primary metric is acceptance length averaged over the evaluation distribution under the lossless speculative decoding constraint. Unless stated otherwise, the only factors that change across experiments are the draft training distribution or the way multiple specialized drafts are combined. All draft checkpoints are trained for 20 epochs with learning rate3 × 10 −5 , batch size 8, and gradient accumulation 1. HASS runs use the same auxiliary settings throughout: top-Kdistillation withK = 10, loss weight 1.0, and three forward-alignment steps. The study varies along four axes: speculative backbone (EAGLE-2 or HASS), training domain (ShareGPT for conversational data or MathInstruct for mathematical reasoning), mixed-data supervision (35k+35k or 70k+70k), and test-time composition strategy (Averaged, Confidence Routed, or Merged Trees). For an inputx, verifierM T , and drafterM D , letA(x; M D , M T )denote the number of consecutively accepted draft tokens. We compare methods through E x∼풟 [A(x; M D , M T )],(10) 4 TAPS: Task Aware Proposal Distributions for Speculative Sampling (a) Checkpoint weight merging by point-wise parameter averag- ing. Draft models trained on ShareGPT and MathInstruct are com- bined by averaging corresponding parameters in weight space, producing a single merged draft that retains information from both domains. (b) Confidence-based routing between specialized draft models. At inference time, the system selects the draft with the highest confidence for the cur- rent prompt, enabling task-aware use of specialized models without merging their parameters. Figure 2: Two strategies for combining specialized draft models. Left: checkpoint weight merging in parameter space. Right: confidence-based routing at inference time. subject to the lossless speculative decoding constraint that the final output distribution remains identical to that of the verifier. Unless stated otherwise, the only factors that change across experiments are the draft training data or the way multiple specialized drafts are combined. Draft Variants We study seven main draft variants for each backbone. Two are single-domain checkpoints: one trained on 70k MathInstruct examples and one trained on 70k ShareGPT examples. Two are mixed-data checkpoints: Mixed 35k+35k and Mixed 70k+70k. The remaining three use the single-domain checkpoints as building blocks for composition: Averaged, Confidence Routed, and Merged Trees. These variants map directly to the research questions in the experiments section. The single-domain checkpoints answer RQ1, the mixed-data checkpoints answer RQ2, and the three composition strategies answer RQ3. RQ4 then interprets these results through routing statistics, entropy, and depth-wise acceptance. Checkpoint Averaging Our simplest composition baseline is checkpoint averaging. Letθ math andθ chat denote the parameters of the MathInstruct and ShareGPT draft models. We define the merged checkpoint as θ merge = λθ math + (1− λ)θ chat ,(11) whereλ ∈ [0, 1]controls the contribution of each checkpoint. We useλ = 0.5for the main table and sweep λ in Figure 6. 5 TAPS: Task Aware Proposal Distributions for Speculative Sampling Figure 3: Confidence Routing Between Specialized Trees. The MathInstruct and ShareGPT checkpoints generate separate draft trees from the same prefix, with node labels indicating draft confidence. Confidence routing selects the tree with the higher mean node confidence before verification. Inference-Time Composition We also study two inference-time alternatives that keep the single-domain checkpoints separate. Both strategies operate within one backbone at a time: HASS drafts are combined only with HASS drafts, and EAGLE-2 drafts only with EAGLE-2 drafts. This isolates the effect of composition from any cross-backbone differences. Confidence routing. Given an input prefix, we decode one draft tree from the MathInstruct checkpoint and one from the ShareGPT checkpoint. We score each tree by its mean draft confidence. Let풯 math and풯 chat denote the two trees and let c(v) denote the confidence assigned to node v. The tree-level score is Score(풯 ) = 1 ∣풯 ∣ ∑ v∈풯 c(v),(12) and the selected tree is 풯 ∗ = arg max 풯 ∈풯 math ,풯 chat Score(풯 ).(13) Only풯 ∗ is passed to the verifier. Merged-tree verification. Instead of selecting one tree and discarding the other, we can verify both trees jointly by packing them under a shared root. We preserve the node indices of one subtree, offset the other subtree by the size of the first, and build an attention mask that allows each node to attend only to the shared root and its own ancestors. Candidates from the MathInstruct subtree therefore do not attend to candidates from the ShareGPT subtree, and vice versa. Position ids are assigned by tree depth so that each child is placed one step deeper than its parent. The merged tree increases proposal diversity at each verifier call because both specialists contribute candidate continuations. At the same time, it is a stricter test than routing because the verifier must process a larger tree. In this paper we report the acceptance length benefit of this strategy, but we do not claim an end-to-end latency improvement without a separate systems analysis. 6 TAPS: Task Aware Proposal Distributions for Speculative Sampling Algorithm 1: Merged-Tree Verification. Require:Prefixy 1∶t , targetM T , drafts M math , M chat 1: Generate draft trees풯 math and풯 chat from the same root token 2:Merge the two trees under a shared root by concatenating nodes and remapping indices 3: Build ancestor-preserving attention masks and depth-based position ids 4:Verify the merged tree in one parallel pass with M T 5:Extract candidate paths and apply standard speculative acceptance 6: Commit the accepted prefix 7: return accepted length Figure 4: Merged Verification Tree. The MathInstruct and ShareGPT subtrees are packed under a shared root while preserving their internal ancestry. This lets the verifier evaluate both specialists in one pass and tests whether broader proposal coverage is more useful than selecting a single specialist. Figure 5: Flattened Merged-Tree Input. The merged tree is serialized for verification while ancestry is preserved through the tree attention mask and depth-based position ids. This lets the verifier process both specialized subtrees without cross-subtree attention. Experiments We report acceptance length, the average number of draft tokens accepted per verifier call. Higher acceptance length indicates that the drafter is better aligned with the verifier on the evaluated workload. Table 1 reports the main numbers. The section then reads those results in four passes: single-domain specialization (RQ1), mixed-data robustness (RQ2), specialist composition (RQ3), and routing/diagnostic signals (RQ4). Unless stated otherwise, all conclusions below are drawn from Table 1, Table 2, and the supporting figures. Each subsection reads the relevant block of Table 1, points to the supporting figures when needed, and states the narrowest conclusion supported by that evidence. RQ1: Does task-specific training improve matched-domain acceptance? Question. Do drafters trained on a matched domain achieve longer acceptance lengths than drafters trained on a mismatched domain? Setup. We compare the two single-domain checkpoints in Table 1. MathInstruct is intended to specialize on mathematical reasoning, while ShareGPT is intended to specialize on conversational generation. Figure 8 provides a depth-wise view of the same behavior. Answer. Yes. The specialization pattern is clear at temperature0for both backbones. Under HASS, ShareGPT is stronger than MathInstruct on MT-Bench (3.98 vs. 2.90), while MathInstruct is stronger on GSM8K and MATH-500 (5.02 vs. 4.09 and 5.35 vs. 3.98). The same pattern appears under EAGLE-2, where ShareGPT is strongest on MT-Bench (3.57 vs. 2.54) and MathInstruct is strongest on GSM8K, MATH-500, and SVAMP. Figure 8 shows that the specialization persists across speculative depth, especially on reasoning-heavy tasks. 7 TAPS: Task Aware Proposal Distributions for Speculative Sampling Table 1: Main Results by Research Question. Average acceptance length on MT-Bench, GSM8K, MATH-500, and SVAMP for HASS and EAGLE-2 at temperatures0and1. Rows are grouped by the question they answer: RQ1 tests single-domain specialization, RQ2 mixed-data robustness, and RQ3 composition strategies. Higher is better. Temperature 0Temperature 1 Model VariantMethod MT-Bench GSM8K MATH-500 SVAMP Average MT-Bench GSM8K MATH-500 SVAMP Average RQ1. Task-specific training: single-domain checkpoints MathInstructHASS2.905.025.353.134.102.314.754.632.463.54 MathInstructEAGLE-22.545.045.284.814.422.434.714.614.534.07 ShareGPTHASS3.984.093.984.444.123.504.033.613.953.77 ShareGPTEAGLE-23.573.723.813.713.703.383.723.433.653.54 RQ2. Mixed-data training: robustness checkpoints Mixed 35k+35kHASS3.924.775.024.154.473.464.664.474.574.29 Mixed 35k+35kEAGLE-23.374.124.444.164.023.104.084.024.033.81 Mixed 70k+70kHASS4.135.535.675.385.183.174.163.424.013.69 Mixed 70k+70kEAGLE-23.754.684.854.644.482.993.763.203.083.26 RQ3. Combining specialists: weight averaging vs. inference-time composition AveragedHASS2.292.803.122.132.592.102.782.902.692.62 AveragedEAGLE-22.072.532.572.502.422.012.492.422.452.34 Confidence RoutedHASS3.935.015.374.894.803.514.724.554.714.37 Confidence RoutedEAGLE-23.634.915.254.714.633.364.654.624.464.27 Merged TreesHASS4.055.425.655.315.113.765.214.985.054.75 Merged TreesEAGLE-23.935.325.635.255.033.555.014.794.934.57 Takeaway. Draft quality is not only a property of the speculative decoding backbone. It also depends on whether the draft training distribution matches the downstream workload. RQ2: Can mixed-data training recover cross-domain robustness? Question. If single-domain drafts specialize strongly, can mixed-data training produce a more robust single checkpoint? Setup. We compare the two mixed-data checkpoints against the single-domain checkpoints in Table 1. The Mixed 35k+35k variant keeps the total sample count modest and balanced, while Mixed 70k+70k doubles the amount of mixed supervision. Answer. Mixed-data training improves robustness, but the effect is not monotonic. Under HASS at temper- ature0, Mixed 70k+70k is the strongest trained checkpoint overall with average acceptance length 5.18, but at temperature1it falls to 3.69, below Mixed 35k+35k at 4.29. Under EAGLE-2, the same pattern appears: Mixed 70k+70k is strongest at temperature0(4.48), whereas Mixed 35k+35k is more stable at temperature1(3.81 vs. 3.26). Mixed training broadens coverage, but larger mixtures do not uniformly improve generalization across decoding temperatures. Takeaway. Mixed-data training is a useful robustness strategy, but it does not remove the need to tune the mixture for the decoding regime of interest. RQ3: How should multiple specialized drafters be combined? Question. When multiple specialized drafts are available, is it better to merge them in weight space or compose them at inference time? 8 TAPS: Task Aware Proposal Distributions for Speculative Sampling Figure 6: Interpolation Sweep for Checkpoint Averaging. Acceptance length is plotted against the interpolation weight between the MathInstruct and ShareGPT draft checkpoints under a fixed verifier setup. Weight-space averaging is unstable and remains well below the strongest inference-time composition methods. Setup. We compare checkpoint averaging, confidence routing, and merged-tree verification in Table 1. Figure 6 provides the interpolation sweep for checkpoint averaging. Answer. Inference-time composition is substantially stronger than weight-space averaging. Averaged checkpoints are consistently the weakest variants in the main table, with average acceptance length between 2.34 and 2.62 across methods and temperatures. By contrast, confidence routing improves to 4.80 and 4.63 average acceptance length at temperature0under HASS and EAGLE-2, respectively. Merged-tree verification is strongest overall, reaching 5.11 for HASS and 5.02 for EAGLE-2 at temperature0, and remaining the best variant at temperature1as well. Figure 6 reinforces this result: interpolating between the two checkpoints produces unstable behavior and never approaches the best inference-time composition methods. Takeaway. If multiple specialists are available, they should be kept separate and combined at inference time. Weight-space averaging is a weak baseline for this problem. RQ4: What do confidence, entropy, and depth reveal about acceptance behavior? Question. Are confidence and entropy useful signals for routing, and does depth-wise acceptance help explain the observed specialization? Setup. Table 2 compares benchmark-level routing decisions under confidence-based and entropy-based selection for EAGLE-2. Figure 7 compares accepted and rejected-token entropy, and Figure 8 reports acceptance by speculative depth for all main variants. Answer. Confidence is useful for routing; entropy is mainly diagnostic. Under confidence routing, the MathInstruct drafter is selected for 90.8% of GSM8K, 97.0% of MATH-500, and 93.0% of SVAMP examples, while ShareGPT is selected for 81.2% of MT-Bench examples. Entropy routing is far less discriminative, producing near-balanced splits across all benchmarks. The entropy figures still show a consistent descriptive pattern: rejected tokens tend to have higher entropy than accepted tokens for both HASS and EAGLE- 2. Figure 8 adds a second clue: acceptance falls with speculative depth for every variant, but domain specialization remains visible and often becomes more pronounced deeper in the tree. Takeaway. Confidence is the stronger decision signal for routing between specialized drafters. Entropy and depth-wise acceptance are useful for interpreting verifier failures, but they do not by themselves justify a stronger routing policy. 9 TAPS: Task Aware Proposal Distributions for Speculative Sampling (a) EAGLE-2 Entropy. Draft entropy for accepted and rejected tokens at temperature0across benchmarks and checkpoints. The averaged checkpoint is omitted for readability. Rejected tokens consistently exhibit higher entropy. (b) HASS Entropy. Draft entropy for accepted and rejected tokens at temperature0across benchmarks and checkpoints. The averaged checkpoint is omitted for readability. The same accepted-versus-rejected separation remains visible. Figure 7: Accepted vs. Rejected Token Entropy. Both panels compare draft entropy at temperature0for EAGLE-2 and HASS on the same benchmark suite and checkpoint families. Entropy is a useful diagnostic of rejection, but Table 2 shows that it is weaker than confidence for routing. Confidence RoutingEntropy Routing BenchmarkMathInstructShareGPTTotalMathInstructShareGPTTotal MT-Bench15 (18.8%)65 (81.2%)8042 (52.5%)38 (47.5%)80 GSM8K1198 (90.8%)121 (9.2%)1319720 (54.6%)599 (45.4%)1319 MATH-500485 (97.0%)15 (3.0%)500312 (62.4%)188 (37.6%)500 SVAMP279 (93.0%)21 (7.0%)300159 (53.0%)141 (47.0%)300 Table 2: Routing Decisions by Benchmark. Benchmark-level routing counts for EAGLE-2 under confidence-based and entropy-based selection. Confidence routing separates conversational and mathematical workloads much more clearly than entropy routing. RQ5: How does speculative depth affect the exploration and exploitation balance in task aware drafting? Question. Does speculative depth reveal a shift from broad proposal coverage at shallow levels to stronger reliance on a task matched specialist at deeper levels? Setup. We use Figure 8 and the depth tables to compare how the main draft variants behave as speculative depth increases across benchmarks. 10 TAPS: Task Aware Proposal Distributions for Speculative Sampling Figure 8: Acceptance by Speculative Depth. Acceptance rate is shown by draft depth for HASS and EAGLE-2 at temperatures0and1across MT-Bench, GSM8K, MATH-500, and SVAMP. Acceptance declines with depth for all variants, while domain specialization remains visible and often sharpens on reasoning-heavy tasks. Answer. The results show a clear depth effect. At shallow depths, mixed data drafts often perform best, suggesting an exploration benefit where broader proposal coverage increases the chance of producing acceptable early branches. As depth increases, the task matched specialist becomes more dominant, especially on reasoning benchmarks. This suggests exploitation, where deeper acceptance depends on sustained agreement between the drafter and the verifier. The composition results follow the same pattern because merged trees perform best by preserving diversity across specialists, while confidence routing helps when the system must choose a single drafter. 11 TAPS: Task Aware Proposal Distributions for Speculative Sampling Takeaway. Speculative decoding appears to be both task aware and depth aware. Early proposal steps benefit more from coverage, while deeper accepted paths increasingly favor the better matched specialist. Discussion and Limitations The experiments support a simple but practically important conclusion: speculative decoding depends not only on the drafting backbone, but also on the relationship between the draft training distribution and the target workload. Once that dependency appears, the draft model becomes a systems choice rather than a fixed auxiliary component. A verifier paired with a mismatched draft is not merely weaker in the abstract; it is predictably weaker on specific task families. This perspective also changes how multiple draft models should be used. Mixed-data training is a reasonable way to broaden coverage, but it does not eliminate the underlying specialization. When separate specialists are already available, keeping them separate and composing them at inference time is much more effective than collapsing them into one averaged checkpoint. The weight-space baseline is useful precisely because it fails: it shows that the relevant behavior is not preserved by naive interpolation. Compared to the strongest single checkpoint, confidence routing reduces average speedup by0.32×and0.35× under EAGLE-2 and by0.40×and0.47×under HASS at temperatures0and1, while merged-tree verification incurs a larger drop of0.59×and0.62×under EAGLE-2 and0.72×and0.78×under HASS. However, in a deployment setting that must serve two distinct task families, this overhead may be partly or fully offset when the best single checkpoint is weak on one of the tasks, since combining specialists can recover acceptance that a single drafter would lose. At the same time, the present evidence is narrower than the strongest version of the claim. We evaluate one target model, two source domains, two speculative backbones, and four benchmarks. Acceptance length is our primary metric, so this paper does not establish end-to-end deployment trade-offs for routing or merged-tree verification. The routing policy is intentionally simple and confidence-based rather than learned or cost-aware. These limitations matter because they define the boundary of what the paper can currently claim. Conclusion We asked whether speculative decoding improves when the drafter is trained for the downstream task and how multiple specialized drafters should be combined. The answer to the first question is yes: task-specific training produces clear domain specialization. The answer to the second is that inference-time composition is substantially stronger than weight-space averaging. Across both HASS and EAGLE-2, confidence routing improves over single-domain baselines and merged-tree verification achieves the highest acceptance length overall. More broadly, proposal quality in speculative decoding should be studied as a function of both draft architecture and draft training distribution, not architecture alone. 12 TAPS: Task Aware Proposal Distributions for Speculative Sampling References Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, NeurIPS, pages 1877–1901. Curran Associates, Inc. URLhttps://proceedings.neurips.c/paper_files/ paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023. URLhttps://arxiv. org/abs/2302.01318. Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Kevin C Chang, and Jie Huang. Cascade speculative drafting for even faster llm inference. NeurIPS, 37:86226–86242, 2024. Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculative decoding. In ACL, pages 12622–12642, 2024. Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. REST: Retrieval-based speculative decoding. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, ACL, pages 1582–1595, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.88. URL https://aclanthology.org/2024.naacl-long.88/. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR, 23–29 Jul 2023. URLhttps: //proceedings.mlr.press/v202/leviathan23a.html. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: speculative sampling requires rethinking feature uncertainty. In ICML, ICML’24. JMLR.org, 2024a. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. In EMNLP, pages 7421–7432, 2024b. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. In ACL, 2025. Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding via double early exiting. arXiv preprint arXiv:2404.18911, 2024. AI @ Meta Llama Team. The llama 3 herd of models, 2024. URLhttps://arxiv.org/abs/2407.21783. 13 TAPS: Task Aware Proposal Distributions for Speculative Sampling Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS ’24, page 932–949, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703867. doi: 10.1145/3620666.3651335. URL https://doi.org/10.1145/3620666.3651335. Siyuan Mu and Sen Lin. A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications, 2026. URL https://arxiv.org/abs/2503.07137. Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.11912, 2024. Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al. A survey of reasoning with foundation models: Concepts, methodologies, and outlook. ACM Computing Surveys, 57(11):1–43, 2025. Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft& verify: Lossless large language model acceleration via self-speculative decoding. In ACL, pages 11263–11282, 2024. Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized representations for speculative sampling. In ICLR, 2025. 14 TAPS: Task Aware Proposal Distributions for Speculative Sampling A. Appendix This appendix collects the supporting entropy tables, the tree-merging utility used for merged-tree verification, and the depth-wise acceptance tables that complement Figure 8. The goal is to make the evidence behind the main-text claims easy to audit without interrupting the main narrative. A.1. EAGLE-2 Entropy (Temperature 0) Checkpoint Benchmark Draft Accepted Draft Rejected∆ Draft Verifier Accepted Verifier Rejected∆ Verifier AveragedGSM8K8.02579.2397 +1.21410.15670.2698+0.1131 AveragedMATH-5008.84059.8344+0.99380.20130.3787+0.1774 AveragedMT-Bench8.51289.4886+0.97580.24270.5796+0.3368 AveragedSVAMP7.82339.0891+1.26580.17030.2833+0.1131 MathInstructGSM8K0.52841.0756+0.54730.15000.4246+0.2746 MathInstructMATH-5000.45670.8555+0.39880.19840.5067+0.3083 MathInstructMT-Bench2.38673.8516+1.46490.23410.6212+0.3871 MathInstructSVAMP0.59281.1607+0.56790.16390.4111+0.2473 MixedGSM8K0.70741.2153+0.50790.15250.4671+0.3146 MixedMATH-5000.63021.0409+0.41070.19250.5430+0.3505 MixedMT-Bench1.18391.8407+0.65680.25610.6736+0.4175 MixedSVAMP0.81621.3367+0.52050.17170.4621+0.2904 ShareGPTGSM8K0.74341.0952+0.35180.15000.5075+0.3574 ShareGPTMATH-5000.89261.2558+0.36320.18980.5766+0.3868 ShareGPTMT-Bench1.04041.6292+0.58870.25390.6600+0.4061 ShareGPTSVAMP0.85541.2391+0.38370.16670.5006+0.3339 Table 3: EAGLE-2 Entropy at Temperature 0. Each row reports accepted-token and rejected-token entropy for one benchmark. Positive∆ means higher entropy for rejected tokens. 15 TAPS: Task Aware Proposal Distributions for Speculative Sampling A.2. HASS Entropy (Temperature 0) Checkpoint Benchmark Draft Accepted Draft Rejected∆ Draft Verifier Accepted Verifier Rejected∆ Verifier MathInstruct GSM8K1.07311.4475 +0.37440.16130.5508+0.3895 MathInstructMATH-5002.08062.1836+0.10300.19280.6625+0.4698 MathInstructMT-Bench4.47775.2238+0.74610.23581.3139+1.0780 MathInstructSVAMP1.39271.6778+0.28510.17650.5660+0.3896 ShareGPTGSM8K1.06901.7551+0.68610.15490.5937+0.4388 ShareGPTMATH-5002.24602.3454+0.09950.18140.7575+0.5761 ShareGPTMT-Bench1.91622.2477+0.33150.27960.8882+0.6086 ShareGPTSVAMP1.34452.0791+0.73450.16950.6139+0.4444 MixedGSM8K1.23691.7589+0.52200.15970.5287+0.3690 MixedMATH-5001.96262.2128+0.25020.18760.6563+0.4687 MixedMT-Bench2.04822.4916+0.44330.27660.9013+0.6247 MixedSVAMP1.33061.8352+0.50470.17420.5501+0.3759 AveragedGSM8K2.62083.2743+0.65350.16100.8306+0.6696 AveragedMATH-5003.83843.3019−0.53640.17860.9489+0.7703 AveragedMT-Bench4.20613.5950−0.61100.23621.4447+1.2085 AveragedSVAMP2.54093.1844+0.64340.17390.8523+0.6784 Table 4: HASS Entropy at Temperature 0. Each row reports accepted-token and rejected-token entropy for one benchmark. Positive∆ means higher entropy for rejected tokens. 16 TAPS: Task Aware Proposal Distributions for Speculative Sampling A.3. Tree Merge Utility _merge_trees def _merge_trees( draft_tokens1, retrieve_indices1, tree_mask1, tree_pos1, draft_tokens2, retrieve_indices2, tree_mask2, tree_pos2, ): n1 = draft_tokens1.shape[1] - 1 n2 = draft_tokens2.shape[1] - 1 N = n1 + n2 + 1 device = draft_tokens1.device dtype = tree_mask1.dtype merged_draft = torch.cat([draft_tokens1, draft_tokens2[0, 1:][None]], dim=1) merged_mask = torch.zeros(N, N, device=device, dtype=dtype) merged_mask[0, 0] = 1.0 merged_mask[1:n1 + 1, :n1 + 1] = tree_mask1[0, 0, 1:, :] merged_mask[n1 + 1:, 0] = 1.0 merged_mask[n1 + 1:, n1 + 1:] = tree_mask2[0, 0, 1:, 1:] merged_mask = merged_mask[None, None] merged_pos = torch.cat([tree_pos1, tree_pos2[1:]]) d1, d2 = retrieve_indices1.shape[1], retrieve_indices2.shape[1] max_d = max(d1, d2) ri1 = F.pad(retrieve_indices1, (0, max_d - d1), value=-1) ri2 = retrieve_indices2.clone() ri2[ri2 > 0] += n1 ri2 = F.pad(ri2, (0, max_d - d2), value=-1) merged_retrieve = torch.cat([ri1, ri2], dim=0) return merged_draft, merged_retrieve, merged_mask, merged_pos A.4. Correctness of routing and merged-tree verification LetQ(⋅ ∣ y 1∶t )denote the target model’s continuation distribution from prefixy 1∶t . For any (packed) draft tree풯rooted aty 1∶t , letDec(y 1∶t ;풯 )denote the random continuation produced by running one verifier call on풯 and then continuing with the standard speculative procedure. Assumption A.1 (Lossless verification for a fixed valid tree). A packed tree풯is valid if the verifier pass on풯produces, for every node, the same target-side conditionalsq(⋅ ∣ its path-prefix)that the target model would produce under standalone autoregressive evaluation along that node’s path. For every valid풯and every measurable set of continuations B, Pr ( Dec(y 1∶t ;풯 ) ∈ B ∣ y 1∶t ,풯 ) = Q(B ∣ y 1∶t ). (This is the standard lossless speculative-decoding guarantee used throughout the paper.) Lemma A.1 (Mixtures over valid trees remain lossless). Let풯be any random valid tree (possibly generated by any draft model(s)). Then for every set B, Pr ( Dec(y 1∶t ;풯 ) ∈ B ∣ y 1∶t ) = Q(B ∣ y 1∶t ). 17 TAPS: Task Aware Proposal Distributions for Speculative Sampling Proof. By the tower property, Pr(Dec ∈ B ∣ y 1∶t ) =E[Pr(Dec ∈ B ∣ y 1∶t ,풯 ) ∣ y 1∶t ] =E[Q(B ∣ y 1∶t ) ∣ y 1∶t ] = Q(B ∣ y 1∶t ). Proposition A.1 (Correctness of routing). Let풯 math and풯 chat be two valid draft trees generated from the same prefixy 1∶t . Letgbe any (possibly randomized) routing rule that depends only on draft-side quantities available before verification (e.g., confidences/entropies/tree statistics), and define the selected tree풯 ⋆ =풯 g(y 1∶t ,풯 math ,풯 chat ) . Then routing is distribution-preserving: Pr ( Dec(y 1∶t ;풯 ⋆ ) ∈ B ∣ y 1∶t ) = Q(B ∣ y 1∶t ) for all B. Proof.풯 ⋆ is a random valid tree (a draft-side function of(풯 math ,풯 chat )), so the claim follows immediately from Lemma A.1. Lemma A.2 (Verifier invariance under masked concatenation). Let the verifier be any transformer-style model that computes per-token logits from (token ids, position ids, attention mask). Consider two packed verifier inputs(X, M, P)and(X ′ , M ′ , P ′ ) with a shared index setSsuch that: (i)X∣ S = X ′ ∣ S andP∣ S = P ′ ∣ S ; (i)M∣ S×S = M ′ ∣ S×S ; and (i) tokens inSdo not attend outsideSin either input, i.e. for alli ∈ Sandj ∉ S, M ij = M ′ ij = 0. Then the verifier logits on indices inSare identical under the two packed inputs. Proof. Induct over transformer layers. At layer0, hidden states onSmatch because token embeddings and position encodings match. Assume hidden states onSmatch at layerℓ − 1. At layerℓ, each tokeni ∈ Sattends only to tokensjwithM ij = 1, and by (i) all suchjlie inS. By (i) the mask onS× Smatches, and by the inductive hypothesis the keys/values of all visiblej ∈ Smatch. Therefore the attention output for eachi ∈ S matches; the remaining sublayers are pointwise with shared parameters, so hidden states onSmatch at layerℓ. Hence the final logits on S match. Proposition A.2 (Correctness of merged-tree verification). Let풯 math and풯 chat be valid trees from prefix y 1∶t , each with its own packed representation (tokens, tree attention mask, and depth-based position ids) used for standalone tree verification. Construct the merged tree풯 ∪ by (a) sharing the root, (b) concatenating the non-root nodes of both trees, (c) using an attention mask that preserves each subtree’s ancestry relations and masks all cross-subtree attention, and (d) assigning each node the same depth-based position id it had in its source tree. Then (i) every node in the merged verifier pass receives exactly the same target-side conditional distribution as in standalone verification of its source subtree, and consequently (i) merged-tree verification is distribution- preserving: Pr ( Dec(y 1∶t ;풯 ∪ ) ∈ B ∣ y 1∶t ) = Q(B ∣ y 1∶t ) for all B. Proof. Fixs ∈ math, chatand letS s denote the index set of the shared root together with all nodes coming from subtreesinside the merged packing. By construction, the merged packed input agrees with the standalone packed input onS s (tokens, depth-based positions, and within-subtree attention), and nodes in S s do not attend to nodes outsideS s because all cross-subtree attention is masked. Therefore, by Lemma A.2, the verifier logits (henceq(⋅ ∣ ⋅)) on all nodes in subtreesare identical to standalone verification. This holds for both subtrees, so풯 ∪ is a valid tree in the sense of Assumption A.1. Applying Assumption A.1 to the fixed valid tree풯 ∪ yieldsPr(Dec(y 1∶t ;풯 ∪ ) ∈ B ∣ y 1∶t ,풯 ∪ ) = Q(B ∣ y 1∶t )for allB. Unconditioning (or equivalently applying Lemma A.1) gives the claimed distribution preservation. Corollary A.1. Both routing (Proposition A.1) and merged-tree verification (Proposition A.2) preserve the target-model output distribution. They may change proposal quality, acceptance length, and runtime, but not the verifier’s sampling law. 18 TAPS: Task Aware Proposal Distributions for Speculative Sampling A.5. EAGLE-2 Acceptance Rates by Depth A.5.1. Temperature 0 DepthVariantMT-Bench GSM8K MATH-500 SVAMP Avg 1MathInstruct 72.0%96.1%97.9%94.9% 90.2% 1ShareGPT89.9%93.1%90.6%91.9%91.4% 1Mixed 35k+35k88.7%95.3%96.4%94.3%93.7% 1Averaged71.3%88.3%89.3%87.0%84.0% 1Mixed 70k+70k80.0%80.8%86.6%80.7%82.0% 2MathInstruct52.6%88.8%92.5%86.8%80.2% 2ShareGPT74.6%78.3%73.7%76.3%75.7% 2Mixed 35k+35k70.8%84.2%85.7%82.4%80.8% 2Averaged42.0%62.2%63.1%60.2%56.9% 2Mixed 70k+70k61.0%61.8%68.6%60.1%62.9% 3MathInstruct48.6%84.6%87.3%83.8%76.1% 3ShareGPT66.2%67.3%66.9%68.1%67.1% 3Mixed 35k+35k60.2%74.9%76.3%75.6%71.8% 3Averaged17.6%30.2%30.1%31.4%27.3% 3Mixed 70k+70k52.2%50.1%60.0%53.9%54.0% 4MathInstruct52.8%78.6%82.5%77.9%72.9% 4ShareGPT64.1%62.0%64.9%63.3%63.6% 4Mixed 35k+35k58.3%65.7%73.4%65.8%65.8% 4Averaged19.3%22.5%28.0%27.5%24.3% 4Mixed 70k+70k52.9%47.2%57.0%46.1%50.8% 5MathInstruct61.0%76.4%81.5%75.0%73.5% 5ShareGPT61.5%57.5%65.5%58.8%60.8% 5Mixed 35k+35k57.2%61.9%70.6%63.1%63.2% 5Averaged39.0%25.7%32.2%23.1%30.0% 5Mixed 70k+70k53.3%43.5%58.0%42.2%49.3% Table 5: EAGLE-2 Acceptance by Depth at Temperature 0. Higher rows correspond to shallower draft positions in the speculative tree. 19 TAPS: Task Aware Proposal Distributions for Speculative Sampling A.5.2. Temperature 1 DepthVariantMT-Bench GSM8K MATH-500 SVAMP Avg 1MathInstruct 68.5%94.1%91.5%93.1% 86.8% 1ShareGPT85.7%91.7%83.7%90.7%88.0% 1Mixed 35k+35k85.1%93.8%89.9%93.3%90.5% 1Averaged67.6%86.1%81.2%84.0%79.7% 1Mixed 70k+70k75.5%76.9%75.6%78.1%76.5% 2MathInstruct50.1%86.3%82.0%84.7%75.8% 2ShareGPT68.7%76.4%65.1%73.9%71.0% 2Mixed 35k+35k65.7%81.6%75.9%80.7%76.0% 2Averaged41.0%60.9%55.6%59.0%54.1% 2Mixed 70k+70k56.5%58.0%58.5%58.4%57.8% 3MathInstruct45.5%82.0%76.8%81.3%71.4% 3ShareGPT60.9%65.7%57.7%66.2%62.6% 3Mixed 35k+35k57.3%72.7%67.2%72.6%67.5% 3Averaged17.7%28.1%24.5%29.5%24.9% 3Mixed 70k+70k48.9%46.7%50.6%50.6%49.2% 4MathInstruct52.2%77.4%74.6%76.3%70.1% 4ShareGPT59.7%60.6%58.0%60.8%59.8% 4Mixed 35k+35k56.4%65.2%65.4%65.6%63.1% 4Averaged17.9%21.9%25.2%28.6%23.4% 4Mixed 70k+70k49.9%45.5%51.8%43.2%47.6% 5MathInstruct56.5%75.2%75.2%73.7%70.1% 5ShareGPT58.4%56.2%58.2%58.9%57.9% 5Mixed 35k+35k57.0%60.0%63.7%60.1%60.2% 5Averaged30.4%26.8%21.8%25.7%26.2% 5Mixed 70k+70k52.5%42.1%52.0%41.6%47.0% Table 6: EAGLE-2 Acceptance by Depth at Temperature 1. Higher rows correspond to shallower draft positions in the speculative tree. 20 TAPS: Task Aware Proposal Distributions for Speculative Sampling A.6. HASS Acceptance Rates by Depth A.6.1. Temperature 0 DepthVariantMT-Bench GSM8K MATH-500 SVAMP Avg 1MathInstruct 61.6%94.6%95.9%94.8% 86.7% 1ShareGPT84.5%90.9%88.0%89.4%88.2% 1Mixed 35k+35k84.9%95.2%96.1%94.7%92.7% 1Averaged68.1%87.5%90.9%87.4%83.5% 1Mixed 70k+70k82.3%85.1%90.6%86.2%86.0% 2MathInstruct50.9%84.8%88.9%82.3%76.7% 2ShareGPT73.7%78.6%71.5%75.3%74.8% 2Mixed 35k+35k74.2%84.0%87.2%81.0%81.6% 2Averaged48.2%68.0%72.9%63.3%63.1% 2Mixed 70k+70k67.6%66.8%75.3%66.8%69.1% 3MathInstruct41.6%77.9%81.8%76.1%69.4% 3ShareGPT56.8%63.1%51.1%61.7%58.2% 3Mixed 35k+35k58.3%71.6%71.8%69.7%67.9% 3Averaged27.0%45.3%44.5%41.8%39.6% 3Mixed 70k+70k51.5%55.0%60.3%53.1%55.0% 4MathInstruct51.1%73.8%78.8%73.5%69.3% 4ShareGPT53.4%50.6%44.4%48.6%49.2% 4Mixed 35k+35k53.8%61.9%63.9%62.0%60.4% 4Averaged32.5%33.4%33.1%30.7%32.4% 4Mixed 70k+70k48.7%47.5%54.4%42.2%48.2% 5MathInstruct62.1%70.5%75.9%66.4%68.7% 5ShareGPT52.2%44.2%44.5%41.3%45.5% 5Mixed 35k+35k54.2%55.2%59.2%51.2%54.9% 5Averaged30.3%26.0%29.1%22.8%27.0% 5Mixed 70k+70k50.2%44.7%53.5%36.1%46.1% Table 7: HASS Acceptance by Depth at Temperature 0. Higher rows correspond to shallower draft positions in the speculative tree. 21