Paper deep dive

AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs

Sheng Liang, Yongyue Zhang, Nathanael Brian, Hang Lv, Hao Wang, Chen Zhang, Yong Liu

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 92

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/29/2026, 3:47:25 AM

Summary

The paper introduces AsymSpec, a speculative decoding framework that decouples context access between a lightweight drafter and a large verifier. The drafter processes the full input context to capture rich reasoning signals, while the verifier operates on a compressed view for efficiency. A contrastive delta-fusion mechanism steers the verifier's logits using the difference between full and compressed context outputs, modulated by a divergence-aware acceptance gate. This approach achieves approximately 90% of full-context accuracy with 1.3-1.7x throughput speedups and significantly reduced compute costs compared to standard speculative decoding.

Entities (9)

AsymSpec · method · 98%Huawei Technologies Co., Ltd. · organization · 95%Speculative Decoding · technique · 95%University of Science and Technology of China · organization · 95%Context-Divergence Acceptance · mechanism · 92%Contrastive δ-fusion · mechanism · 92%GAIA · benchmark · 90%LongBench · benchmark · 90%Qwen3-32b · model · 90%

Relation Signals (8)

Hang Lv → affiliatedwith → University of Science and Technology of China

confidence 95% · Hang Lv Affiliation: University of Science and Technology of China

Sheng Liang → affiliatedwith → Huawei Technologies Co., Ltd.

confidence 95% · Sheng Liang Affiliation: Huawei Technologies Co., Ltd.

AsymSpec → improvesover → Speculative Decoding

confidence 95% · Speculative decoding (SD) ... assumes the drafter and verifier share an identical context ... We propose AsymSpec ... that breaks this symmetry

AsymSpec → usesmechanism → Contrastive δ-fusion

confidence 92% · The drafter steers the verifier via a contrastive δ-fusion of logits

AsymSpec → usesmechanism → Context-Divergence Acceptance

confidence 92% · modulated by a divergence-aware acceptance gate that preserves verification stability

AsymSpec → evaluatedon → GAIA

confidence 90% · Evaluated across ... two end-to-end agent benchmarks ... GAIA

AsymSpec → evaluatedon → LongBench

confidence 90% · Evaluated across four agentic capabilities ... LongBench

Qwen3-32b → usedas → Verifier

confidence 90% · Our primary experiments use Qwen3-32B ... as the verifier

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Agentic LLM pipelines face escalating inference costs as context accumulates across retrieval, tool use, and multi-turn interactions. To control latency, deployments routinely compress inputs, but this degrades task accuracy. Speculative decoding (SD) accelerates generation losslessly, yet it assumes the drafter and verifier share an identical context, preventing SD from resolving the accuracy-overhead trade-off. We propose AsymSpec, an asymmetric speculative decoding framework that breaks this symmetry: a lightweight drafter reads the full input while the large verifier operates on the compressed view. The drafter steers the verifier via a contrastive $\delta$-fusion of logits, modulated by a divergence-aware acceptance gate that preserves verification stability and high draft acceptance rates. Evaluated across four agentic capabilities and two end-to-end agent benchmarks, AsymSpec reaches $\approx 90\%$ of full-context accuracy on average, delivering $1.3$--$1.7\times$ throughput speedups at $0.2$--$0.3\times$ the compute cost on isolated text capabilities. These results show that asymmetric context access yields substantial gains precisely when compression discards critical reasoning signals.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

88,804 characters extracted from source content.

Expand or collapse full text

AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs Sheng Liang Affiliation: Huawei Technologies Co., Ltd. Yongyue Zhang Affiliation: Huawei Technologies Co., Ltd. Nathanael Brian Affiliation: Huawei Technologies Co., Ltd. Hang Lv Affiliation: University of Science and Technology of China Hao Wang Affiliation: University of Science and Technology of China Chen Zhang Affiliation: Huawei Technologies Co., Ltd. Yong Liu Affiliation: Huawei Technologies Co., Ltd. Abstract Agentic LLM pipelines face escalating inference costs as context accumulates across retrieval, tool use, and multi-turn interactions. To control latency, deployments routinely compress inputs, but this degrades task accuracy. Speculative decoding (SD) accelerates generation losslessly, yet it assumes the drafter and verifier share an identical context, preventing SD from resolving the accuracy–overhead trade-off. We propose AsymSpec, an asymmetric speculative decoding framework that breaks this symmetry: a lightweight drafter reads the full input while the large verifier operates on the compressed view. The drafter steers the verifier via a contrastive δ-fusion of logits, modulated by a divergence-aware acceptance gate that preserves verification stability and high draft acceptance rates. Evaluated across four agentic capabilities and two end-to-end agent benchmarks, AsymSpec reaches ≈ 90% of full-context accuracy on average, delivering 1.3–1.7× throughput speedups at 0.2–0.3× the compute cost on isolated text capabilities. These results show that asymmetric context access yields substantial gains precisely when compression discards critical reasoning signals. 1 Introduction Modern LLM deployments increasingly operate as agentic pipelines—retrieving documents (Lewis et al., 2020; Gao et al., 2023; Wu et al., 2025a), invoking tools (Yao et al., 2023; Schick et al., 2023), maintaining multi-turn dialogue and memory (Sirdeshmukh et al., 2025; Zhang et al., 2025b; Wu et al., 2025b), and processing multimodal inputs (Liu et al., 2023). These settings issue repeated LLM calls over context that grows with every step. As retrieved passages, tool observations, and interaction histories accumulate, the forward pass becomes the dominant latency bottleneck, making context length the primary driver of inference overhead in production. To control this overhead, deployments routinely compress the context (Pan et al., 2024b): RAG pipelines summarize retrieved passages, tool-use agents pass only API signatures instead of full documentation, and multimodal workflows feed short captions instead of raw images. Compression reduces serving cost but systematically discards fine-grained details critical for task accuracy. Deployments thus face a rigid accuracy–overhead trade-off: absorb the prohibitive latency of full-context generation, or accept significant degradation. Speculative decoding (SD) (Leviathan et al., 2023; Chen et al., 2023) has become the standard approach for accelerating LLM inference: a lightweight drafter proposes candidate tokens that a large verifier checks in parallel, guaranteeing lossless generation under the target distribution. Despite extensive improvements to drafter architectures (Li et al., 2024c; Cai et al., 2024; Li et al., 2024b) and contrastive logit fusion in the speculative loop (Yuan et al., 2023), all existing SD methods share a foundational constraint: drafter and verifier process the same input tokens. SD accelerates a fixed target distribution without changing what the target model sees; once the verifier is compressed, SD can only accelerate the compressed model—it cannot recover what compression removed. Either both models pay the full-context cost, or both inherit the compression loss. Our key observation is a structural compute asymmetry: per-step latency is dominated by the large verifier, while a lightweight drafter adds negligible overhead. Compressing only the verifier captures most of the latency savings, but standard SD cannot exploit this because it enforces identical input. We break this symmetry with AsymSpec, an asymmetric speculative decoding framework that explicitly decouples context access. The verifier operates strictly on the compressed view for efficiency, while the drafter reads the full input to reconstruct the discarded information. We realize this recovery through a contrastive mechanism: the drafter processes both context views, and subtracting their output distributions removes the drafter’s context-independent preferences, isolating the information gain the uncompressed input provides. This gain signal, δ, is fused into the verifier’s logits and modulated by a parameter-free Context-Divergence Acceptance (CDA) gate. By scaling injection strength with the context divergence, the gate maintains stable verification and high acceptance rates. Since only the drafter processes the full input, AsymSpec recovers full-context reasoning fidelity at the compressed verifier’s latency, and extends to cross-modal settings (e.g., a vision–language drafter on raw images steering a text-only verifier on captions). Contributions. 1. We propose AsymSpec, a context-asymmetric speculative decoding framework: the verifier runs on a compressed view while the drafter consumes the full input, opening an operating point—compressed cost with near-ceiling accuracy—inaccessible to symmetric SD, and extending naturally to cross-modal settings. 2. Two coupled mechanisms instantiate the framework: a same-model cross-context δ-fusion that cancels drafter capacity biases to isolate the context-gain signal, and a parameter-free Context-Divergence Acceptance (CDA) gate that bounds steering strength without per-dataset tuning. 3. Across four agentic capabilities and two end-to-end agent benchmarks, AsymSpec recovers ≈ 90% of full-context accuracy at 0.20.2–0.3×0.3× compute and 1.31.3–1.7×1.7× throughput. 2 Related Work 2.1 Speculative decoding Speculative decoding (SD) (Leviathan et al., 2023; Chen et al., 2023) accelerates autoregressive generation by having a lightweight drafter propose candidate tokens that a large verifier validates in parallel, preserving the target distribution via rejection sampling. Subsequent work improves drafter quality (EAGLE (Li et al., 2024c; Li et al., 2024b; Li et al., 2025b), Medusa (Cai et al., 2024)) or targets long-context inference latency via hierarchical or sparse-KV speculation (TriForce (Sun et al., 2024), MagicDec (Sadhukhan et al., 2025)). All these methods feed the drafter and verifier the same input tokens, even when KV-cache structure differs across stages. This symmetry prevents SD from exploiting the compute asymmetry inherent in long-context decoding. We relax this constraint in §3, letting the two models operate on distinct context views without breaking the speculative verification loop. 2.2 Context compression Context compression is the standard mechanism for reducing long-context inference overhead (Lv et al., 2026b). Hard-prompt methods prune tokens by importance (LLMLingua (Jiang et al., 2024; Pan et al., 2024b)), soft-context approaches learn compact latents (Gist Tokens (Mu et al., 2023), ICAE (Ge et al., 2024)), and KV-cache techniques operate directly on cached states (StreamingLLM (Xiao et al., 2024), SnapKV (Li et al., 2024a)). Comprehensive surveys (Li et al., 2024d) document these approaches. Across this literature, accuracy degradation is treated as an unavoidable cost. We treat any compressor as a black box and show in §3 how a full-context drafter can systematically recover the discarded information, turning compression from a lossy shortcut into a steerable efficiency knob. 2.3 Contrastive decoding and logit fusion Contrastive decoding (Li et al., 2023b) improves generation quality by subtracting an amateur model’s logits from an expert’s, amplifying expert-specific signals. This principle has been extended to multi-step reasoning (O’Brien and Lewis, 2023) and integrated into the speculative loop as Speculative Contrastive Decoding (SCD) (Yuan et al., 2023). However, SCD and its variants operate on a single shared context, using logit differences to bridge a model-capacity gap. The same subtractive principle has also been used to remove content-independent positional priors with CapCal (Lv et al., 2026a), mitigate linguistic inertia after reasoning-chain compression with LICD (Zhang et al., 2026b), and transfer local context-induced preferences to a remote model with CoSteer (Lv et al., 2025). Together, these methods frame logit differences as signal isolators. AsymSpec brings this view into speculative verification: both logit terms come from the same drafter under full and compressed context views, so δ isolates the context gain rather than a model-capacity gap. We formalize this cross-context transfer and its bias-cancellation property in §3.2. 2.4 Asymmetric and multimodal speculation Recent work has begun to explore asymmetric configurations. RAPID (Chen et al., 2025) adopts an inverse design: a retrieval-truncated drafter steers a full-context verifier to inject external knowledge, prioritizing full-context fidelity at full-context latency. Speculative RAG (Wang et al., 2025b) drafts answer candidates from retrieved-document subsets and then verifies them. Concurrent multimodal SD methods (SpecVLM, Spec-LLaVA, ViSpec (Huang et al., 2025; Huo et al., 2025; Kang et al., 2025); MASSV (Ganesan et al., 2025)) focus on visual-token prefill acceleration or keep both models within a single modality. SD2 (Berdoz et al., 2026) reverses the steering direction by conditioning the drafter on verifier signals. SpecSteer uses asymmetric local–cloud speculation, keeping private personalization context with a local drafter while drawing on cloud-scale reasoning (Lv et al., 2026c). Across these designs, asymmetry serves personalization and privacy, retrieval allocation, or modality-specific acceleration. AsymSpec instead uses asymmetric context access to change the efficiency operating point: a rich-input drafter compensates for a strictly compressed verifier, recovering accuracy at compressed-verifier latency (§3; design-space comparison in Table 8). 2.5 Speculation in agentic settings Most SD work targets single-turn generation; long-context variants (Sun et al., 2024; Sadhukhan et al., 2025) extend the regime but remain prompt-level. Recent agent-oriented methods accelerate the decision loop by speculating over high-level action sequences (Ye et al., 2025), tool invocations (Nichols et al., 2025), or plans (Hua et al., 2024). DynaThink likewise reduces inference cost by choosing between fast and deliberative reasoning at the request level (Pan et al., 2024a). Such action- and request-level mechanisms are orthogonal to token-level decoding engines. We demonstrate in §5.3 that AsymSpec composes with live agentic loops, providing token-level accuracy recovery whenever tool outputs or retrieved contexts are compressed online. Figure 1: AsymSpec speculation step. (a) Drafting & verification: drafter S reads xfullx_full (logits a, drafts d1:Kd_1:K) and xcompx_comp (logits b); verifier L reads xcompx_comp (logits t). (b) CDA gate & δ-fusion: γeff(i) _eff(i) relaxes as full-vs-compressed divergence grows; accepted drafts are committed, otherwise the δ-fused arg⁡max is emitted. Only S consumes xfullx_full. 3 Method 3.1 Problem setup Let L be a large verifier and S a lightweight drafter (|S|≪|L||S| |L|). A task provides a full prompt xfullx_full; a black-box compressor produces a compressed view xcompx_comp with |xcomp|≪|xfull||x_comp| |x_full|. Per-step latency is dominated by L’s forward pass, which scales super-linearly with context length. This overhead structure defines two baseline operating points: • L(⋅∣xfull)L(· x_full): peak accuracy, but prohibitive latency in agentic loops; • L(⋅∣xcomp)L(· x_comp): low latency, but accuracy degraded by compression. Compressing only L’s input captures the majority of latency savings, while S’s extra forward on xfullx_full remains far cheaper than the verifier’s saved forward on xfullx_full. Standard speculative decoding cannot exploit this asymmetry because it forces identical context access. AsymSpec explicitly decouples the two: L operates strictly on xcompx_comp for efficiency, while S reads xfullx_full to reconstruct discarded information. 3.2 Contrastive δ-fusion Each speculation step executes three forward passes over the current sequence (Figure 1a): 1. Augmented drafter: S⁡(xfull)S(x_full) produces logits a and samples K draft tokens d1:Kd_1:K. 2. Base drafter: S⁡(xcomp)S(x_comp) produces logits b at the same K+1K+1 positions. 3. Verifier: L⁡(xcomp)L(x_comp) scores all K drafts in parallel, yielding logits t. We define the context-gain signal in logit space: δi=ai−bi, _i\;=\;a_i-b_i, -5.0pt (1) where ai,bi∈ℝ||a_i,b_i ^|V| are the per-position logits. Subtracting bib_i from aia_i removes the drafter’s context-independent preferences, isolating the shift induced by the additional context. Upon draft rejection, δi _i is fused into the verifier’s distribution: di′=arg⁡max⁡(ti+βδi),d _i\;=\; (t_i+β\, _i ), -5.0pt (2) where β∈[0,1]β∈[0,1] controls the steering strength. This injection shifts the verifier’s prediction toward what the full-context input would have produced, without requiring L to attend to xfullx_full. 3.3 Context-divergence acceptance (CDA) When the full and compressed views diverge sharply, a fixed acceptance threshold γ usually over-rejects the drafts and wastes context-gain signals. We replace it with a threshold that relaxes proportionally to context divergence, instantiating the per-position divergence DiD_i as the Jensen–Shannon divergence (JSD): γeff(i)=γ⋅exp⁡(−Di),Di=JSD(softmax(ai)∥softmax(bi)). gathered _eff(i)\;=\;γ· (-D_i),\\ D_i\;=\;JSD (softmax(a_i)\,\|\,softmax(b_i) ). gathered -5.0pt (3) We choose JSD specifically for its strict upper bound, which guarantees γeff∈[γ/2,γ] _eff∈[γ/2,γ] without clipping or introducing additional hyperparameters; the exponential form is the unique solution to a multiplicative-composition axiom (derivation in Appendix A). A large DiD_i signals context-induced divergence (not capacity-induced)—these are the positions where the compressed verifier most likely under-scores useful full-context drafts, so relaxing acceptance there is appropriate. The drafted token did_i is accepted iff: [softmax⁡(ti)]di>γeff(i)⋅[softmax⁡(bi)]di.[softmax(t_i)]_d_i\;>\; _eff(i)·[softmax(b_i)]_d_i. -5.0pt (4) Upon the first rejection, we emit the δ-fused token from Equation 2 (Figure 1b). The mechanism degenerates to standard verification on xcompx_comp when β=0β=0 and γ=1γ=1. The strict range γeff∈[γ/2,γ] _eff∈[γ/2,γ] keeps the acceptance criterion stable across positions, preserving predictable draft acceptance rates even under severe compression. Unlike standard SD, AsymSpec does not preserve a strict target distribution; it is a speculative-style steering scheme calibrated for greedy emission. 3.4 Cross-modal extension Cross-modal asymmetry follows directly from the framework. Since drafter and verifier share an output token vocabulary V, the signals δ∈ℝ||δ ^|V| and γeff∈ℝ _eff are computed entirely on the output side (Equations 1, 3, 4 and 2) and remain well-defined regardless of the drafter’s input modality: a vision–language drafter can process raw images as xfullx_full while a text-only verifier reads captions as xcompx_comp, with the speculative loop unchanged. The vision encoder runs once per request and its outputs are cached on the drafter’s KV side, so the per-token overhead of cross-modal δ vanishes at long generations and asymptotically matches that of text-only AsymSpec. Routing vision embeddings through the speculative engine’s drafter prefill is a non-trivial vLLM modification detailed in Appendix E. 4 Experiments 4.1 Tasks & compression protocol We evaluate four isolated agentic capabilities under realistic compression: long-context multi-hop QA (LongBench (Bai et al., 2024), using its three multi-hop subsets), multi-turn instruction following (MultiChallenge (Sirdeshmukh et al., 2025)), tool use (API-Bank (Li et al., 2023a)), and multimodal reasoning (MathVista (Lu et al., 2024)). End-to-end agentic performance is measured on GAIA (Mialon et al., 2024) and SimpleQA (Wei et al., 2024), orchestrated via the smolagents (Roucher et al., 2025) framework. We enforce a strict asymmetric protocol across all benchmarks: the verifier receives only a compressed view (e.g., per-turn LLMLingua-2 (Pan et al., 2024b) summaries or API signatures), while the drafter reads the full uncompressed input. Dataset statistics, compression ratios, and metrics are summarized in Table 9; per-benchmark compression pipelines and tool-execution protocols are in Appendix B. Algorithm 1 AsymSpec Decoding 1: Verifier L, drafter S, prompts xfull,xcompx_full,x_comp, spec length K, threshold γ∈(0,1]γ∈(0,1], fusion weight β∈[0,1]β∈[0,1] 2: Generated sequence y 3: y←[]y←[\,] 4: while generation not complete do 5: Autoregressively sample d1:Kd_1:K from S⁡(xfull⊕y)S(x_full y) (KV cache reused); let a1:K+1a_1:K+1 be the logits at the corresponding positions 6: b1:K+1←S(xcomp⊕y⊕d1:K)b_1:K+1← S(x_comp y d_1:K) at the same positions 7: t←L(xcomp⊕y⊕d1:K)t← L(x_comp y d_1:K) 8: for i=1i=1 to K do 9: δi←ai−bi _i← a_i-b_i 10: Di←JSD(softmax(ai)∥softmax(bi))D_i (softmax(a_i)\,\|\,softmax(b_i)) 11: γeff(i)←γexp⁡(−Di) _eff(i)←γ (-D_i) 12: end for 13: accepted←Trueaccepted 14: for i=1i=1 to K do 15: if [softmax⁡(ti)]di>γeff(i)[softmax⁡(bi)]di[softmax(t_i)]_d_i> _eff(i)[softmax(b_i)]_d_i then 16: y←y⊕[di]y← y [d_i] 17: else 18: y←y⊕[arg⁡max⁡(ti+βδi)]y← y [ (t_i+β _i)] 19: accepted←Falseaccepted ; break 20: end if 21: end for 22: if accepted then 23: y←y⊕[arg⁡max⁡(tK+1)]y← y [ (t_K+1)] 24: end if 25: end while 26: return y GAIA harness. GAIA runs through the smolagents CodeAgent ReAct loop with cached DuckDuckGo search and visit-webpage as the only tools, identical across both AsymSpec variants reported in Table 4. The first variant uses our default Qwen3-4B text drafter, same as every other benchmark in this paper. The second variant swaps the drafter to Qwen3-VL-2B-Instruct so it can read image attachments directly through the cross-modal patches of Appendix E; the verifier still receives pre-computed VL captions, as it cannot consume pixels. Compression, K=2K=2, and β=1.0β=1.0 are shared across both runs; the two variants differ only in drafter capacity and in whether image attachments reach the drafter as pixels or as captions. 4.2 Models & hyperparameters Our primary experiments use Qwen3-32B (Yang et al., 2025) as the verifier; Appendix D additionally evaluates Llama-3.3-70B-Instruct and Llama-3.2-3B-Instruct (Llama Team, 2024) in same- and cross-family Qwen–Llama pairings. Qwen3 offers open weights at 0.6B/1.7B/4B/32B from a common training recipe, post-training for tool use and multi-turn dialogue, and stable speculative-decoding support in vLLM (Kwon et al., 2023); primary experiments run on vLLM with the extensions in Appendix E. For text tasks, we sweep drafter sizes (0.6B, 1.7B, 4B) and report the 4B configuration as primary; for multimodal reasoning, we use Qwen3-VL-2B. All generation uses greedy decoding (τ=0τ=0). Headline results use speculation depth K=2K=2, fusion weight β=1.0β=1.0, and base threshold γ=0.5γ=0.5. Per-benchmark output length, context window, and harness bounds are listed in Table 7; complete hyperparameter grids, sensitivity analyses, and benchmark-specific drafter selections are detailed in Appendix B. 4.3 Metrics We report task accuracy with each benchmark’s official metric (Table 9). Efficiency is reported along two axes: Speedup is the wall-clock token throughput ratio over the full-context Ceiling on a single accelerator (Table 11; Appendix F); FLOPs are per-step prefill compute normalized by the Ceiling’s verifier-only prefill, computed from the Qwen3 architecture and measured context lengths following Kaplan et al. (2020) (per-benchmark accounting in Table 5). Speedup measures realized latency while FLOPs measure energy / compute cost—the two diverge because decoding is memory-bandwidth-bound at 30B+ scale. 4.4 Baselines We compare against five references: (1) Floor: L on xcompx_comp alone; (2) Ceiling: L on xfullx_full alone; (3) SD: standard speculative decoding on shared context, evaluated on xfullx_full (SD on xcompx_comp matches Floor by construction); (4) SCD (Yuan et al., 2023): combines verifier logits t with amateur logits b on a shared context (L,SL,S both on xcompx_comp here), closing a model-capacity gap at SD speed (details in Appendix B); (5) RAPID (Chen et al., 2025): inverse asymmetric design (S on xcompx_comp, L on xfullx_full)—preserves the verifier’s target distribution at full-context compute. In cross-modal settings, the text-only Ceiling is undefined; we use the VL drafter alone as the reference bound. All throughput and compute metrics are normalized to the full-context Ceiling. 5 Results 5.1 Agentic capabilities Long-context multi-hop (LongBench) Multi-turn Tool use Speedup FLOPs Method hotpotQA 2WikiMQA MuSiQue (MultiChal.) (API-Bank) over Ceiling over Ceiling Floor 49.4 52.8 32.7 23.4 57.7 1.17× 0.11× Ceiling 64.9 76.5 55.0 26.4 66.1 1.00× 1.00× SD 65.5 76.6 55.1 26.7 66.1 1.73× 1.04× SCD (Yuan et al., 2023) 46.6 52.7 32.0 22.6 56.7 1.04×1.04× 0.12×0.12× RAPID (Chen et al., 2025) 63.2 75.3 52.5 25.8 64.3 1.38× 1.01× AsymSpec (Ours) 64.0 66.8 48.4 23.5 63.5 1.45× 0.23× Table 1: Accuracy and efficiency on isolated agentic capabilities (K=2K=2). AsymSpec closes 5959–94%94\% of the Floor–Ceiling gap on long-context multi-hop QA and tool use, at 0.23×0.23× the Ceiling’s compute—a trade-off point inaccessible to symmetric SD or SCD. Speedup and FLOPs are averaged across the three text benchmarks; FLOPs follow the parameters × token-ratio convention. Drafter-alone accuracy across drafter sizes is in Appendix H. Metric definitions in §4.3; per-benchmark FLOPs in Table 5. We first evaluate the three text-based agentic capabilities (§4). Table 1 reports accuracy and efficiency, with LongBench broken into its hotpotQA / 2WikiMQA / MuSiQue multi-hop subsets. AsymSpec occupies an operating point inaccessible to symmetric methods: it delivers near-ceiling accuracy while retaining compressed-verifier latency. It reaches 8787–99%99\% of the full-context performance, closing the Floor–Ceiling gap to 0.90.9–9.79.7 residual points. The residual gap is structural: the verifier remains input-constrained and cannot fully reconstruct multi-hop reasoning chains or complex tool dependencies from logit steering alone. Full K and drafter-size sweeps are deferred to §6. Baseline comparisons reinforce the design’s advantage. Standard SD preserves the target distribution but is structurally bound to symmetric context: it either inherits the Floor’s accuracy (on compressed input) or the Ceiling’s latency (on full input). SCD stays at or below the Floor across all five cells (e.g., LongBench mean 43.843.8 vs. Floor 45.045.0; API-Bank 56.756.7 vs. 57.757.7), showing that single-context contrastive fusion cannot recover compression-induced loss when expert and amateur both read the compressed view—the gain is specific to our asymmetric construction. RAPID’s inverse asymmetric design (drafter on xcompx_comp, verifier on xfullx_full) reaches near-Ceiling accuracy but at near-Ceiling compute (1.01×1.01× FLOPs); it targets the opposite trade-off point and is dominated on the compute axis where AsymSpec operates. Running the drafter alone on full context cuts cost but degrades sharply on reasoning-heavy subsets where the 4B model lacks sufficient capacity (per-size numbers in Appendix H). AsymSpec bridges these extremes by retaining the 32B verifier’s reasoning while offloading context recovery to the lightweight drafter. The gains are directly tied to compression-induced information loss, not task-specific overfitting. On MultiChallenge, where compression is near-lossless (Ceiling–Floor gap of 3.0 points), AsymSpec yields negligible improvement (23.5 vs. 23.4). This inertness supports the diagnostic that the mechanism activates only when compression discards critical signals. We further verify this diagnostic on a continuous axis by sweeping the verifier’s truncation budget on LongBench while keeping the drafter on full passages (Table 2). The recovered accuracy scales monotonically with compression severity: at a 500-token budget, AsymSpec restores over two-thirds of the Floor–Ceiling gap; at 12k tokens, the gain naturally vanishes as the verifier approaches the uncompressed ceiling. Stable acceptance rates (0.851–0.855) suggest the gap variation is due to information recovery, not unstable verification. 5.2 Multimodal reasoning We next evaluate the cross-modal extension, where a vision–language drafter processes raw images while the text-only verifier reads captions and OCR text. Since a text-only verifier cannot consume images, the full-context Ceiling is undefined; we use the VL drafter alone as the reference bound. Table 3 reports results on MathVista. AsymSpec reaches 53.9%53.9\% overall accuracy, outperforming symmetric SD by 10.110.1 points. The per-task breakdown reveals a clear complementarity pattern. On geometry problem solving (GPS), caption and OCR text already capture the necessary structure, yielding minimal gain. On visual question answering (VQA) and figure question answering (FQA), the drafter provides visual grounding while the verifier supplies causal logic, lifting accuracy over the Floor by 10.010.0 and 16.716.7 points respectively. The remaining gap to the VL drafter alone (53.953.9 vs. 60.5%60.5\%) is structural: the text-only verifier cannot fully internalize pixel-level cues. Nevertheless, the result demonstrates that organizations with fixed text-only verifiers can extend them to vision-reasoning tasks without re-provisioning multimodal infrastructure. Realizing these gains requires proper routing of vision-tower embeddings through the speculative engine; ablation without these patches drops accuracy to 30.5%30.5\% (implementation details in Appendix E). 5.3 End-to-end agentic loops We finally evaluate AsymSpec inside live agent loops where context accumulates and is re-compressed online. Table 4 reports results on GAIA and SimpleQA, orchestrated via smolagents. AsymSpec reaches 24.2%24.2\% on GAIA and 65.0%65.0\% on SimpleQA, matching or exceeding the available full-context reference (per-subset for GAIA, Table 4) with no degradation as context accumulates. The aggregate GAIA gain masks distinct subset dynamics: on the web-only ReAct loop, compression forces tighter history encoding and benefits token-hungry planning; on the file-attachment subset, the drafter’s full-text access compensates for the verifier’s 2000-token truncation. SimpleQA favors the Ceiling due to its minimal compression headroom (1.33×1.33× token ratio). Across both benchmarks, the method maintains stable draft acceptance (0.880.88–0.900.90, Table 13), confirming that online re-compression does not erode verification quality. Compute efficiency directly tracks compression severity. GAIA applies a 1.91×1.91× per-turn compression ratio, yielding 0.78×0.78× full-context FLOPs. SimpleQA’s lighter 1.33×1.33× compression yields 0.80×0.80×. This linear relationship matches the core design premise: the verifier’s prefill reduction scales proportionally with context shortening, while the drafter’s dual pass remains the smaller component. In live agent loops as in static benchmarks, compression headroom reliably predicts both accuracy recovery and efficiency gains. To verify that the modality-agnostic property (§3.4) holds inside live agent loops, we additionally swap the drafter to Qwen3-VL-2B in the identical harness. AsymSpec reaches 23.0%23.0\% on the full n=165n=165 split, +3.6+3.6 p over Floor and +3.0+3.0 p over Ceiling—the operating-point gain survives drafter-modality substitution. The gain concentrates on the web subset (+7.1+7.1 p over Floor); on the file-attachment subset the 2B drafter’s text capacity limits the gain (consistent with the ≥ 1.7B threshold from §6.4) and AsymSpec falls below the Floor. Trunc tokens Floor AsymSpec Δ 500 25.8 52.5 +26.7+26.7 1500 32.6 53.7 +21.1+21.1 3000 39.5 55.5 +16.0+16.0 6000 50.6 59.2 +8.6+8.6 12000 63.1 63.9 +0.8+0.8 Table 2: Truncation-budget sweep on LongBench (K=2K=2, β=1β=1, 4B drafter; overall F1). Recovery scales monotonically with compression severity and vanishes as the verifier approaches the uncompressed Ceiling (65.565.5 F1, full context). 6 Ablations We systematically ablate the speculation depth K, core components (CDA gate vs. δ-fusion), δ-source construction, divergence metrics, and drafter capacity. Extended grids and baseline comparisons against fixed-γ are provided in Appendices G and H. 6.1 Speculation depth (K) Table 12 justifies K=2K=2 as the default speculation depth. MultiChallenge acts as the binding constraint: all methods degrade at K=4K=4 under the llm-judge, and none recovers its K=2K=2 performance. API-Bank is flat across K, while LongBench shows only marginal gains at K=4K=4 (→61.159.7\!→\!61.1) and saturates at K=6K=6 (58.758.7). Since K=2K=2 matches the standard SD default and yields the most stable accuracy–efficiency trade-off, we adopt it for all text benchmarks. The cross-modal setting is the sole exception where deeper speculation helps (MathVista K=4K=4 outperforms K=2K=2); we report it at its optimal depth (Table 3). Method GPS VQA FQA Overall VL drafter alone 53.4 56.4 67.7 60.5 Floor 62.0 49.1 29.0 44.5 SD 62.0 46.4 28.6 43.8 AsymSpec (ours) 62.1 59.1 45.7 53.9 Table 3: MathVista results under the cross-modal setup. The text-only verifier cannot consume raw images, so the Ceiling is undefined; the VL drafter alone (Qwen3-VL-2B reading the image) is the reference upper bound. Floor is the verifier on the official Bard caption plus EasyOCR text; SD is symmetric speculative decoding with a text drafter on the same caption input. Setting Method Acc Spd FLOPs GAIA Web-only Floor 17.3 Ceiling 18.9 AsymSpec(4B) 22.0 AsymSpec(vl-2B) 24.4 File-attach Floor 26.3 Ceiling 23.7 AsymSpec(4B) 31.6 AsymSpec(vl-2B) 18.4 Full Floor 19.4 1.25× 0.49× Ceiling 20.0 1.00× 1.00× AsymSpec(4B) 24.2 1.41× 0.78× AsymSpec(vl-2B) 23.0 1.65× 0.53× SimpleQA Aggregate Floor 63.0 1.17× 0.74× Ceiling 66.0 1.00× 1.00× AsymSpec(4B) 65.0 1.38× 0.80× Table 4: End-to-end accuracy and efficiency in live agentic loops, with GAIA per-subset breakdown and SimpleQA. On GAIA, “Ceiling” is the best per-subset reference available under each harness: Qwen3-32B-on-full for the web subset; the vl-2B-drafter-alone reference for the file-attachment subset. Speedup and FLOPs measure LLM-only inference at the aggregate level. Setting LfullL_full LcompL_comp ratio Verifier Total GAIA 4650 2434 1.9× 49% 0.78× SimpleQA 3606 2712 1.3× 74% 0.80× LongBench 12437 1532 8.1× 9% 0.28× MultiChallenge 1598 211 7.6× 13% 0.28× API-Bank 6701 909 7.4× 12% 0.19× Table 5: Per-step prefill FLOPs across all benchmarks, computed from the Qwen3 architecture and measured context lengths. Verifier is the 3232B forward on xcompx_comp as a fraction of the full-context baseline; Total adds both drafter forwards (conservatively counted as full prefills). Compute reduction ranges from 0.19×0.19× on heavy-compression benchmarks (API-Bank, 7.4×7.4× token ratio) to 0.80×0.80× on light-compression SimpleQA (1.33×1.33× ratio), tracking compression headroom monotonically. Variant LongBench F1 Floor 45.0 ++ CDA gate (β=0β=0) 52.8 ++ δ-fusion, raw-aug (a only) 56.9 ++ δ-fusion, SCD-style (t−bt-b) 48.0 ++ δ-fusion, ours (a−ba-b) 59.7 Ceiling 65.5 Table 6: Mechanism ablation on LongBench (K=2K=2, 4B drafter, γ=0.5γ=0.5). The CDA gate and δ-fusion are both necessary; our same-model δ source (a−ba-b) outperforms the raw-augmented (a) and SCD-style (t−bt-b) alternatives by 33–1212 F1 points. Full hyperparameter and compressor sweeps are in Appendix G. 6.2 Mechanism ablation Disabling fusion (β=0β=0) isolates each mechanism’s contribution (Table 6). The CDA gate alone lifts LongBench from 45.045.0 to 52.852.8 by admitting context-aware drafts that a fixed threshold would over-reject; adding δ-fusion drives recovery to 59.759.7. The δ source itself is non-trivial: replacing our same-model a−ba-b with raw augmented logits costs 2.82.8 points, and the two-model SCD-style contrast collapses 11.711.7 points. This confirms that the same-model cross-context construction—not generic logit fusion—is what makes δ informative: only by subtracting two passes through identical weights can we isolate the context-induced shift from the drafter’s own preferences. A token-level walkthrough on API-Bank (Section 6.3) makes this concrete: δ-fusion redirects emission from Floor’s free-text formats to the structured patterns visible only in the full spec. 6.3 Case study To illustrate the mechanism at single-token resolution, we walk through one tool-use instance (RecordHealthData, level-1 dialog 2). Under Method A compression, the verifier sees only the bare signature, while the drafter sees the full API spec specifying time format %Y-%m-%d %H:%M:%S and a structural example for health_data. The three outputs: Floor […time="2021-09-17 10:30", health_data="Blood pressure: 120/80, …")] GT …time="2021-09-17 10:30:00", health_data="[’name’: ’blood_pressure’, ’value’: ’120/80’, …]" AsymSpec […time="2021-09-17 10:30:00", health_data [’name’: ’blood_pressure’, ’value’: ’120/80’, …])] Red marks where Floor diverges from the schema (missing :00 seconds field; free-text health_data instead of dict list). Green marks drafter tokens accepted by the CDA gate (δ-fusion unused; speculation merely accelerates). Orange marks tokens emitted via δ-fusion at rejection points: Floor closes time at " and renders health_data as free text, but δ redirects emission toward the structural pattern (:00 seconds field; bracketed dict list) recovered from the spec—both schema details exist only in xfullx_full. 6.4 Robustness Beyond the core mechanisms, extensive sweeps (Appendices H and G) confirm AsymSpec’s robustness across three axes. Hyperparameter insensitivity: Performance is flat across β∈[1.0,2.0]β∈[1.0,2.0] and γ∈[0.4,0.7]γ∈[0.4,0.7], requiring no per-dataset calibration. Compressor agnosticism: Swapping the verifier’s compressor (summarization, LLMLingua-2, truncation) yields a stable 6363–70%70\% Floor–Ceiling recovery; SCD falls below the Floor in every cell. Drafter capacity: Models ≤0.6≤ 0.6B fail to extract reliable context-gain signals; ≥1.7≥ 1.7B is the practical minimum. Cross-family portability. To test whether the mechanism depends on a shared model family, we evaluate bidirectional Qwen–Llama pairings on LongBench. Following Timor et al. (2025), heterogeneous runs restrict generation to 109,566109,566 string-identical tokens plus paired special tokens and map δ from the drafter vocabulary to the verifier. A Qwen3-4B drafter raises the Llama-3.3-70B verifier’s compressed-context Floor from 50.650.6 to 58.458.4 F1, while a Llama-3.2-3B drafter raises the Qwen3-32B Floor from 45.045.0 to 47.147.1. These pairings demonstrate feasible cross-family transfer, with recovery varying across model pairs; Appendix D reports the full comparison. 7 Conclusion We introduced AsymSpec, breaking the symmetric context constraint of standard speculative decoding. By allowing a lightweight drafter to read the full context and steer a compressed-context verifier via contrastive δ-fusion, it achieves near-ceiling accuracy at a fraction of the compute cost. Empirically, AsymSpec recovers ≈ 90% of full-context performance using only 0.20.2–0.3×0.3× the FLOPs on text tasks. Crucially, the accuracy gain scales monotonically with the severity of compression loss, providing practitioners with a clear, predictable criterion for when asymmetric steering is warranted in production agentic pipelines. Limitations AsymSpec’s recovery mechanism is fundamentally bounded by the information retained in the compressed view and the drafter’s capacity to extract it. On near-lossless tasks, the method correctly yields minimal intervention, demonstrating that it does not introduce spurious hallucinations or over-fit to the uncompressed context. For cross-modal settings, the upper bound of accuracy recovery is constrained by the fidelity of the modality translation (e.g., image-to-caption quality); integrating richer multi-modal drafters that process raw pixels directly alongside the verifier remains an exciting avenue for future work. Cross-family δ-fusion requires an explicit vocabulary and logit-space alignment. Our Qwen–Llama study demonstrates feasibility for two shared-token-aligned pairings, but the observed recovery varies across model pairs; broader transfer may require richer mappings. AsymSpec also requires access to verifier logits and therefore does not apply to proprietary APIs that expose only generated text. Furthermore, our evaluation focuses on deterministic decoding (τ=0τ=0). This is a deliberate design choice rather than a constraint: agentic workflows strictly demand reproducible, parsable structured outputs (e.g., JSON, tool calls), where stochastic sampling (τ>0τ>0) fundamentally degrades pipeline reliability. Generalizing the CDA bound to stochastic sampling (e.g., via Gumbel-Softmax relaxations) is a promising theoretical extension. Finally, in end-to-end agentic loops, wall-clock latency is a composite of LLM inference, tool execution, and network I/O. While AsymSpec strictly optimizes the LLM inference bottleneck—which becomes dominant as context scales into the compute-bound regime—it is designed to be highly complementary to system-level optimizations, such as I/O overlapping and asynchronous tool execution, in production agent frameworks. References Bai et al. (2024) Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li LongBench: a bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §4.1. Berdoz et al. (2026) F. Berdoz, P. Rheinboldt, and R. Wattenhofer Steering pretrained drafters during speculative decoding. Proceedings of the AAAI Conference on Artificial Intelligence 40 (36), p. 30067–30075. External Links: Document, 2511.09844 Cited by: Table 8, §2.4. Cai et al. (2024) T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao Medusa: simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning (ICML), External Links: 2401.10774 Cited by: §1, §2.1. Chen et al. (2023) C. Chen, S. Borgeaud, G. Irving, J. Lespiau, L. Sifre, and J. Jumper Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. Cited by: §1, §2.1. Chen et al. (2025) G. Chen, Q. Feng, J. Ni, X. Li, and M. Q. Shieh RAPID: long-context inference with retrieval-augmented speculative decoding. In Proceedings of the 42nd International Conference on Machine Learning (ICML), Note: Spotlight External Links: 2502.20330 Cited by: Table 8, §2.4, §4.4, Table 1. Dong et al. (2026) K. Dong, S. Huang, F. Ye, W. Han, Z. Zhang, D. Li, W. Li, Q. Yang, G. Wang, Y. Wang, C. Zhang, and Y. Liu Doc-Researcher: a unified system for multimodal document parsing and deep research. In Proceedings of the ACM Web Conference 2026, p. 2349–2360. External Links: Document Cited by: Appendix C. Feng et al. (2025) R. Feng, B. Zhang, S. Liang, and Z. Yuan Steer-MoE: efficient audio-language alignment with a mixture-of-experts steering module. External Links: 2510.13558 Cited by: Appendix C. Ganesan et al. (2025) M. Ganesan, S. Segal, A. Aggarwal, N. Sinnadurai, S. Lie, and V. Thangarasa MASSV: multimodal adaptation and self-data distillation for speculative decoding of vision-language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, p. 12265–12276. External Links: 2505.10526 Cited by: §2.4. Gao et al. (2023) Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang Retrieval-augmented generation for large language models: a survey. arXiv preprint arXiv:2312.10997. Cited by: §1. Ge et al. (2024) T. Ge, J. Hu, L. Wang, X. Wang, S. Chen, and F. Wei In-context autoencoder for context compression in a large language model. In International Conference on Learning Representations (ICLR), Note: arXiv:2307.06945 Cited by: §2.2. Guo et al. (2024) W. Guo, H. Wang, L. Zhang, J. Y. Chin, Z. Liu, K. Cheng, Q. Pan, Y. Q. Lee, W. Xue, T. Shen, K. Song, K. Wang, W. Xie, Y. Ye, H. Guo, Y. Liu, D. Lian, R. Tang, and E. Chen Scaling new frontiers: insights into large recommendation models. arXiv preprint arXiv:2412.00714. Cited by: Appendix C. Hua et al. (2024) W. Hua, M. Wan, S. Vadrevu, R. Nadel, Y. Zhang, and C. Wang Interactive speculative planning: enhance agent efficiency through co-design of system and user interface. Note: arXiv:2410.00079 Cited by: §2.5. Huang et al. (2025) H. Huang, F. Yang, Z. Liu, X. Yin, D. Li, P. Ren, and E. Barsoum SpecVLM: fast speculative decoding in vision-language models. Note: arXiv:2509.11815 Cited by: Table 8, §2.4. Huo et al. (2025) M. Huo, J. Zhang, H. Wang, J. Xu, Z. Chen, H. Tai, and Y. Chen Spec-LLaVA: accelerating vision-language models with dynamic tree-based speculative decoding. In ICML 2025 Workshop on Tiny Titans (TTODLer-FM), External Links: 2509.11961 Cited by: Table 8, §2.4. Jiang et al. (2024) H. Jiang, Q. Wu, X. Luo, D. Li, C. Lin, Y. Yang, and L. Qiu LongLLMLingua: accelerating and enhancing LLMs in long context scenarios via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1658–1677. Note: arXiv:2310.06839 Cited by: §2.2. Kang et al. (2025) J. Kang, H. Shu, W. Li, Y. Zhai, and X. Chen ViSpec: accelerating vision-language models with vision-aware speculative decoding. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2509.15235 Cited by: Table 8, §2.4. Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: §4.3. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), Cited by: §4.2. Leviathan et al. (2023) Y. Leviathan, M. Kalman, and Y. Matias Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML), Cited by: Table 8, §1, §2.1. Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1. Li et al. (2023a) M. Li, Y. Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, and Y. Li API-Bank: a comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 3102–3116. External Links: Document Cited by: Appendix B, §4.1. Li et al. (2025a) W. Li, Z. Chen, J. Lin, H. Cao, W. Han, S. Liang, Z. Zhang, K. Dong, D. Li, C. Zhang, and Y. Liu Reinforcement learning foundations for deep research systems: a survey. External Links: 2509.06733 Cited by: Appendix C. Li et al. (2023b) X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. Hashimoto, L. Zettlemoyer, and M. Lewis Contrastive decoding: open-ended text generation as optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: Table 8, §2.3. Li et al. (2024a) Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen SnapKV: LLM knows what you are looking for before generation. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2404.14469 Cited by: §2.2. Li et al. (2024b) Y. Li, F. Wei, C. Zhang, and H. Zhang EAGLE-2: faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §1, §2.1. Li et al. (2024c) Y. Li, F. Wei, C. Zhang, and H. Zhang EAGLE: speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning (ICML), Cited by: Table 8, §1, §2.1. Li et al. (2025b) Y. Li, F. Wei, C. Zhang, and H. Zhang EAGLE-3: scaling up inference acceleration of large language models via training-time test. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2503.01840 Cited by: §2.1. Li et al. (2024d) Z. Li, Y. Liu, Y. Su, and N. Collier Prompt compression for large language models: a survey. Note: arXiv:2410.12388 Cited by: §2.2. Liang et al. (2025a) S. Liang, H. Lv, Z. Wen, Y. Wu, Y. Zhang, H. Wang, and Y. Liu Adaptive schema-aware event extraction with retrieval-augmented generation. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, p. 7927–7946. External Links: Document Cited by: Appendix C. Liang et al. (2025b) S. Liang, Y. Zhang, Y. Wu, R. Tang, and Y. Liu Schema as parameterized tools for universal information extraction. External Links: 2506.01276 Cited by: Appendix C. Liang et al. (2022) S. Liang, M. Zhao, and H. Schütze Modular and parameter-efficient multimodal fusion with prompting. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, p. 2976–2985. External Links: Document Cited by: Appendix C. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1. Llama Team (2024) Llama Team The Llama 3 herd of models. CoRR abs/2407.21783. External Links: Document, Link Cited by: §4.2. Lu et al. (2024) P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K. Chang, M. Galley, and J. Gao MathVista: evaluating mathematical reasoning of foundation models in visual contexts. In Proceedings of the International Conference on Learning Representations (ICLR), Note: Oral Cited by: §4.1. Lv et al. (2026a) H. Lv, H. Gu, R. Yang, L. Li, Z. Chen, D. Lian, H. Wang, and E. Chen Learning from emptiness: de-biasing listwise rerankers with content-agnostic probability calibration. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 817–826. External Links: Document Cited by: §2.3. Lv et al. (2026b) H. Lv, S. Liang, H. Gu, W. Guo, D. Lian, Y. Liu, H. Wang, and E. Chen IE as cache: information extraction enhanced agentic reasoning. External Links: 2604.14930 Cited by: §2.2. Lv et al. (2025) H. Lv, S. Liang, H. Wang, H. Gu, Y. Wu, W. Guo, D. Lian, Y. Liu, and E. Chen CoSteer: collaborative decoding-time personalization via local delta steering. External Links: 2507.04756 Cited by: §2.3. Lv et al. (2026c) H. Lv, S. Liang, H. Wang, Y. Zhang, H. Gu, W. Guo, D. Lian, Y. Liu, and E. Chen SpecSteer: synergizing local context and global reasoning for efficient personalized generation. External Links: 2603.16219 Cited by: §2.4. Mialon et al. (2024) G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom GAIA: a benchmark for general AI assistants. In International Conference on Learning Representations (ICLR), Note: arXiv:2311.12983 Cited by: §4.1. Mu et al. (2023) J. Mu, X. L. Li, and N. D. Goodman Learning to compress prompts with gist tokens. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2304.08467 Cited by: §2.2. Nichols et al. (2025) D. Nichols, P. Singhania, C. Jekel, A. Bhatele, and H. Menon Optimizing agentic language model inference via speculative tool calls. Note: arXiv:2512.15834 Cited by: §2.5. O’Brien and Lewis (2023) S. O’Brien and M. Lewis Contrastive decoding improves reasoning in large language models. arXiv preprint arXiv:2309.09117. Cited by: Appendix B, §2.3. Pan et al. (2024a) J. Pan, Y. Zhang, C. Zhang, Z. Liu, H. Wang, and H. Li DynaThink: fast or slow? a dynamic decision-making framework for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 14686–14695. External Links: Document Cited by: §2.5. Pan et al. (2025) Q. Pan, H. Wang, G. An, L. Zhang, W. Guo, and Y. Liu Revisiting scalable sequential recommendation with multi-embedding approach and mixture-of-experts. arXiv preprint arXiv:2510.25285. Cited by: Appendix C. Pan et al. (2024b) Z. Pan, Q. Wu, H. Jiang, M. Xia, X. Luo, J. Zhang, Q. Lin, V. Rühle, Y. Yang, C. Lin, H. V. Zhao, L. Qiu, and D. Zhang LLMLingua-2: data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Association for Computational Linguistics: ACL 2024, External Links: 2403.12968 Cited by: Appendix B, §1, §2.2, §4.1. Roucher et al. (2025) A. Roucher, A. Villanova del Moral, T. Wolf, L. von Werra, and E. Kaunismäki smolagents: a barebones library for agents that think in code. Note: https://github.com/huggingface/smolagents Cited by: §4.1. Sadhukhan et al. (2025) R. Sadhukhan, J. Chen, Z. Chen, V. Tiwari, R. Lai, J. Shi, I. E. Yen, A. May, T. Chen, and B. Chen MagicDec: breaking the latency-throughput tradeoff for long context generation with speculative decoding. In International Conference on Learning Representations (ICLR), Note: arXiv:2408.11049 Cited by: §2.1, §2.5. Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1. Shen et al. (2026) T. Shen, H. Wang, C. Qin, R. Sun, Y. Song, D. Lian, H. Zhu, and E. Chen Prompting is not enough: exploring knowledge integration and controllable generation on large language models. Big Data Mining and Analytics 9 (2), p. 563–579. External Links: Document Cited by: Appendix C. Shen et al. (2025) T. Shen, H. Wang, C. Wu, J. Y. Chin, W. Guo, Y. Liu, H. Guo, D. Lian, R. Tang, and E. Chen P-Law: predicting quantitative scaling law with entropy guidance in large recommendation models. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Appendix C. Shen et al. (2024) T. Shen, H. Wang, J. Zhang, S. Zhao, L. Li, Z. Chen, D. Lian, and E. Chen Exploring user retrieval integration towards large language models for cross-domain sequential recommendation. arXiv preprint arXiv:2406.03085. Cited by: Appendix C. Shi et al. (2025) W. Shi, H. Tan, C. Kuang, X. Li, X. Ren, C. Zhang, H. Chen, Y. Wang, L. Hou, and L. Shang DeepDiver: adaptive search intensity scaling via open-web reinforcement learning. CoRR abs/2505.24332. External Links: 2505.24332 Cited by: Appendix C. Sirdeshmukh et al. (2025) V. Sirdeshmukh, K. Deshpande, J. Mols, L. Jin, E. Cardona, D. Lee, J. Kritz, W. Primack, S. Yue, and C. Xing MultiChallenge: a realistic multi-turn conversation evaluation benchmark challenging to frontier LLMs. Note: arXiv:2501.17399 Cited by: Appendix B, §1, §4.1. Sun et al. (2024) H. Sun, Z. Chen, X. Yang, Y. Tian, and B. Chen TriForce: lossless acceleration of long sequence generation with hierarchical speculative decoding. In Conference on Language Modeling (COLM), Note: arXiv:2404.11912 Cited by: §2.1, §2.5. Timor et al. (2025) N. Timor, J. Mamou, D. Korat, M. Berchansky, G. Jain, O. Pereg, M. Wasserblat, and D. Harel Accelerating LLM inference with lossless speculative decoding algorithms for heterogeneous vocabularies. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 59598–59620. External Links: Link Cited by: Appendix D, §6.4. Wang et al. (2025a) H. Wang, W. Guo, L. Zhang, J. Y. Chin, Y. Ye, H. Guo, Y. Liu, D. Lian, R. Tang, and E. Chen Generative large recommendation models: emerging trends in llms for recommendation. In Companion Proceedings of the ACM on Web Conference 2025, p. 49–52. Cited by: Appendix C. Wang et al. (2025b) Z. Wang, Z. Wang, L. Le, H. S. Zheng, S. Mishra, V. Perot, Y. Zhang, A. Mattapalli, A. Taly, J. Shang, C. Lee, and T. Pfister Speculative RAG: enhancing retrieval augmented generation through drafting. In International Conference on Learning Representations (ICLR), Cited by: §2.4. Wei et al. (2024) J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368. External Links: 2411.04368 Cited by: Appendix B, Table 9, §4.1. Wen et al. (2025) Z. Wen, S. Liang, Y. Wu, Y. Zhang, and Y. Liu Effective and efficient schema-aware information extraction using on-device large language models. External Links: 2505.14992 Cited by: Appendix C. Wu et al. (2025a) Y. Wu, J. Bo, Y. Zhang, S. Liang, and Y. Liu Query-centric graph retrieval augmented generation. External Links: 2509.21237 Cited by: §1. Wu et al. (2025b) Y. Wu, S. Liang, C. Zhang, Y. Wang, Y. Zhang, H. Guo, R. Tang, and Y. Liu From human memory to AI memory: a survey on memory mechanisms in the era of LLMs. External Links: 2504.15965 Cited by: §1. Wu et al. (2025c) Y. Wu, Y. Zhang, S. Liang, and Y. Liu SGMem: sentence graph memory for long-term conversational agents. External Links: 2509.21212 Cited by: Appendix C. Xia et al. (2024) H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui Unlocking efficiency in large language model inference: a comprehensive survey of speculative decoding. In Findings of the Association for Computational Linguistics: ACL 2024, Note: arXiv:2401.07851; Spec-Bench Cited by: Appendix F. Xiao et al. (2024) G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), Note: arXiv:2309.17453 Cited by: §2.2. Xie et al. (2025) W. Xie, H. Wang, M. Fang, R. Yu, W. Guo, Y. Liu, D. Lian, and E. Chen Breaking the bottleneck: user-specific optimization and real-time inference integration for sequential recommendation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, p. 3333–3343. Cited by: Appendix C. Xu et al. (2025) X. Xu, H. Wang, W. Guo, L. Zhang, W. Yang, R. Yu, Y. Liu, D. Lian, and E. Chen Multi-granularity interest retrieval and refinement network for long-term user behavior modeling in ctr prediction. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, p. 2745–2755. Cited by: Appendix C. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. arXiv preprint arXiv:2505.09388. External Links: 2505.09388 Cited by: §4.2. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), Note: arXiv:2210.03629 Cited by: §1. Ye et al. (2025) N. Ye, A. Ahuja, G. Liargkovas, Y. Lu, K. Kaffes, and T. Peng Speculative actions: a lossless framework for faster agentic systems. Note: arXiv:2510.04371 Cited by: §2.5. Ye et al. (2026) Y. Ye, W. Guo, H. Wang, L. Zhang, H. Chang, H. Zhu, Y. Ye, Y. Liu, D. Lian, and E. Chen FuXi-Linear: unleashing the power of linear attention in long-term time-aware sequential recommendation. arXiv preprint arXiv:2602.23671. Cited by: Appendix C. Yu et al. (2025) H. Yu, Y. Wu, H. Wang, W. Guo, Y. Liu, Y. Li, Y. Ye, J. Du, and E. Chen Thought-augmented planning for llm-powered interactive recommender agent. arXiv preprint arXiv:2506.23485. Cited by: Appendix C. Yuan et al. (2023) H. Yuan, K. Lu, F. Huang, Z. Yuan, and C. Zhou Speculative contrastive decoding. arXiv preprint arXiv:2311.08981. Cited by: Appendix B, Table 8, §1, §2.3, §4.4, Table 1. Zhang et al. (2025a) C. Zhang, D. Chong, F. Jiang, C. Tang, A. Gao, G. Tang, and H. Li Aligning language models using follow-up likelihood as reward signal. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence, p. 25832–25841. External Links: Document Cited by: Appendix C. Zhang et al. (2025b) C. Zhang, X. Dai, Y. Wu, Q. Yang, Y. Wang, R. Tang, and Y. Liu A survey on multi-turn interaction capabilities of large language models. CoRR abs/2501.09959. External Links: 2501.09959 Cited by: §1. Zhang et al. (2026a) C. Zhang, K. Dong, D. Li, W. Li, Q. Yang, W. Han, and Y. Liu SRR-Judge: step-level rating and refinement for enhancing search-integrated reasoning in search agents. CoRR abs/2602.07773. External Links: 2602.07773 Cited by: Appendix C. Zhang et al. (2024) C. Zhang, C. Tang, D. Chong, K. Shi, G. Tang, F. Jiang, and H. Li TS-Align: a teacher-student collaborative framework for scalable iterative finetuning of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 8926–8946. External Links: Document Cited by: Appendix C. Zhang et al. (2026b) L. Zhang, Y. Huang, H. Lv, X. Zhi, M. Yin, Y. Ye, W. Guo, H. Wang, and E. Chen Why thinking hurts: diagnosing and rectifying linguistic inertia in large language models for recommendation. External Links: 2602.16587 Cited by: §2.3. Zhang et al. (2026c) L. Zhang, H. Lv, Q. Pan, K. Wang, Y. Huang, X. Miao, Y. Xu, W. Guo, Y. Liu, H. Wang, and E. Chen The next paradigm is user-centric agent, not platform-centric service. arXiv preprint arXiv:2602.15682. Cited by: Appendix C. Zhang et al. (2025c) L. Zhang, K. Song, Y. Q. Lee, W. Guo, H. Wang, Y. Li, H. Guo, Y. Liu, D. Lian, and E. Chen Killing two birds with one stone: unifying retrieval and ranking with a single generative recommendation model. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 2224–2234. Cited by: Appendix C. Zhang et al. (2026d) L. Zhang, H. Wang, Z. Liu, M. Yin, Y. Huang, J. Li, W. Guo, Y. Liu, H. Guo, D. Lian, and E. Chen Can recommender systems teach themselves? a recursive self-improving framework with fidelity control. arXiv preprint arXiv:2602.15659. Cited by: Appendix C. Zhang et al. (2025d) L. Zhang, H. Wang, S. Zhang, M. Yin, Y. Han, J. Zhang, D. Lian, and E. Chen A unified framework for adaptive representation enhancement and inversed learning in cross-domain recommendation. In Database Systems for Advanced Applications, p. 115–130. External Links: Document Cited by: Appendix C. Zhi et al. (2026) X. Zhi, P. Zhou, C. Lu, H. Lv, Y. Liang, R. Zhang, Y. Gao, Yiwu, Y. Hu, H. Gu, D. Lian, H. Wang, and E. Chen SPARD: self-paced curriculum for RL alignment via integrating reward dynamics and data utility. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States, p. 47402–47422. External Links: Document, Link Cited by: Appendix C. Zhou et al. (2026) R. Zhou, Q. Jia, B. Chen, P. Xu, Y. Sun, S. Lou, C. Fu, M. Fu, G. Shen, Z. Zhou, J. Jiao, N. Zhou, S. Guan, Y. Qi, S. Wang, X. Luo, Q. Hu, C. Ma, X. Lv, Q. Luo, Y. Ye, L. Zhang, D. Lian, R. Tang, G. Zhou, H. Li, K. Gai, H. Wang, and E. Chen A survey of user lifelong behavior modeling: perspectives on efficiency and effectiveness. Preprints. External Links: Link Cited by: Appendix C. Appendix A Derivation of the CDA gate We require the per-position effective threshold γeff _eff (§3.3) to satisfy: • γeff(0)=γ _eff(0)=γ (no divergence ⇒ standard SD rule); • γeff _eff continuous and monotonically non-increasing in DiD_i; • multiplicative composition over independent divergence signals (independent context shifts should compose multiplicatively, mirroring independent probabilities): γeff(D1+D2)=γeff(D1)γeff(D2)/γ _eff(D_1+D_2)= _eff(D_1)\, _eff(D_2)/γ. Defining f⁡(D)=γeff(D)/γf(D)= _eff(D)/γ, the third property is a Cauchy-type multiplicative equation f⁡(D1+D2)=f⁡(D1)f(D2)f(D_1+D_2)=f(D_1)f(D_2). Under continuity, this fixes f to an exponential form f(D)=exp(−D/T)f(D)= (-D/T) for some scale T>0T>0, giving γeff(i)=γ⋅exp(−Di/T). _eff(i)\;=\;γ· (-D_i/T ). (5) To eliminate T without introducing a hyperparameter, we instantiate DiD_i as the Jensen–Shannon divergence. Let Zi∼Bernoulli⁡(12)Z_i ( 12) be a latent indicator selecting xfullx_full vs. xcompx_comp, and XiX_i the drafter’s next token. Then Di=I⁡(Xi,Zi)D_i=I(X_i;Z_i), the mutual information between the token and the context source. Since I⁡(Xi,Zi)≤H⁡(Zi)=ln⁡2I(X_i;Z_i)≤ H(Z_i)= 2 for any binary channel, DiD_i is universally bounded. Setting T=1T=1 absorbs the scale into the bound, yielding Equation 3 and guaranteeing γeff∈[γ/2,γ] _eff∈[γ/2,γ] without clipping. Appendix B Design space and benchmark details Models and hyperparameters (full). The verifier is Qwen3-32B in all primary experiments; the portability study in Appendix D additionally uses a Llama-3.3-70B-Instruct verifier and a Llama-3.2-3B-Instruct drafter. For text capabilities the drafter is one of Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B (default 4B drafter, except API-Bank where 1.7B is the sweet spot, §6.4); for multimodal reasoning (cross-modal setup) the drafter is Qwen3-VL-2B-Instruct. All inference runs in bf16 with greedy decoding (τ=0τ=0) and Qwen3’s thinking mode disabled. Default runs use speculation depth K=2K=2 (the standard speculative-decoding default; the K-sweep justifying this choice is §6.1) and β=1.0β=1.0 (β∈0.5,1.0β∈\0.5,1.0\ swept on MathVista). The acceptance threshold is γ=0.5γ=0.5 for CDA (our gate with a tuning-free divergence scale, §3.3). Two reference baselines are reported in Appendices G and H: a fixed-γ gate (standard SD acceptance, γ=0.5γ=0.5, no divergence modulation) and a manually-tuned λ-variant of the acceptance gate, γeff=γexp⁡(−λΔ) _eff=γ (-λ ), where Δ=log⁡softmax⁡(ai)−log⁡softmax⁡(bi) = (a_i)- (b_i) is the drafter’s per-position log-prob difference and λ is set per dataset (default λ=0.1λ=0.1); the tuned variant serves as a foil isolating the value of CDA’s parameter-free JSD bound. Generation parameters. Table 7 lists per-benchmark output length, model context, and harness bounds. All entries inherit greedy decoding (τ=0τ=0) and Qwen3 thinking mode disabled from the global setup above. Benchmark max_new max_model_len Harness / prompt LongBench 1024 24576 QA on passage(s) MultiChallenge 2000 8192 official dataset prompt API-Bank (Method A) 256 16384 API specs + dialog → [Tool(args)] MathVista 1024 8192 VQA prompt; image via VL drafter GAIA (web, n=127n=127) 1024 32768 smolagents CodeAgent, max_steps=8 GAIA (file, n=38n=38) 1024 32768 single-shot file-QA SimpleQA 1024 32768 smolagents CodeAgent, max_steps=6 Table 7: Per-benchmark generation parameters and harness bounds. max_new is the per-call output token cap; max_model_len is the vLLM context window. GAIA splits into a web-only ReAct loop and a single-shot file-attachment harness, listed separately. Per-benchmark compression. Each benchmark uses its native compression scheme, applied only to the verifier’s input. LongBench: the dataset’s auto-generated multi-document summary replaces the full passages (8.1×8.1× token reduction). MultiChallenge: the most recent turn is kept verbatim; all prior turns are LLM-summarized into a single context block (7.6×7.6×). API-Bank: only the API name and signature are exposed in place of the full documentation (7.4×7.4×). MathVista: the official Bard caption plus EasyOCR-extracted text substitute for the raw image, the cross-modal substitution exercised by the framework’s modality-agnostic property (§3.4). GAIA and SimpleQA: the per-turn ReAct context is compressed online with LLMLingua-2 (Pan et al., 2024b) at target ratio 0.30.3; the two most recent turns are kept verbatim (keep_last_k=2keep\_last\_k=2), and everything older—including the system prompt—is compressed. Agentic setup (GAIA, SimpleQA). Both end-to-end benchmarks run the smolagents CodeAgent ReAct loop with cached DuckDuckGo-search and visit-webpage tools so tool returns are deterministic and runs reproducible; the agentic configuration is Qwen3-4B drafter, K=2K=2, β=1.0β=1.0. GAIA: full validation split (Levels 1–3, n=165n=165). The web-only subset (n=127n=127) is evaluated through the agentic ReAct loop above; the file/image-attachment subset (n=38n=38) is evaluated under a single-shot file-QA harness in which the verifier reads a 2000-token truncation of the extracted file content (xlsx, pdf, pptx/docx/txt, csv, etc.) and the drafter reads the full extracted text, with image attachments substituted by the pre-computed VL caption from §3.4; per-subset numbers are listed in Table 4. On the file subset the “Ceiling” reference reported in Table 4 is the drafter-alone harness cell (Qwen3-4B on the full file), since the file-QA setup does not include a Qwen3-32B-on-full-file run. SimpleQA: random n=500n=500 subset; each query runs through the same smolagents ReAct harness as GAIA-web with max_steps=6max\_steps=6; we report accuracy under the official Wei et al. grader (llm-judge, prompt verbatim from Wei et al. (2024)). Agentic run with VL drafter. An identical harness is rerun with the drafter swapped from Qwen3-4B to Qwen3-VL-2B-Instruct. Tools, prompt templates, ReAct step bound, online LLMLingua-2 compression, K=2K=2, and β=1.0β=1.0 are unchanged from the 4B-drafter run above. On web-only samples the VL drafter receives only text from the agentic loop, so the effective change is drafter capacity—2B versus 4B. On file-attachment samples, image attachments are routed directly into the VL drafter’s vision tower via the cross-modal patches in Appendix E; the text-only verifier still receives only the pre-computed VL caption since it cannot consume pixels. This run produces the AsymSpec(vl-2B) rows in Table 4. API-Bank (Method A subset). We evaluate the Method A subset of Li et al. (2023a): single-call API invocation, where each instance presents the model with a compact API specification plus a user query and the model must emit a single well-formed API call. We use n=200n=200 instances and report api-acc (fraction of calls matching the gold API name and argument set). The complementary Method B subset (multi-call API trajectories) introduces tool-trajectory complexity orthogonal to the compression-recovery effect we study; we leave it to follow-up work. MultiChallenge judge protocol. For MultiChallenge we use llm-judge over the official prompt and rubric (Sirdeshmukh et al., 2025), with each cell scored across 3 independent judge runs (per-cell std ≤0.9≤ 0.9 p; Table 1). All judge prompts, responses, and scoring rubrics are logged to our reproducibility repository. Transient API degradations during collection were identified via response-length anomalies (judge responses truncated to <<50 characters in >>15% of cases) and excluded from reported means; full logs are preserved for audit. SCD reimplementation. We implement a faithful reproduction of Improved Contrastive Decoding (Yuan et al., 2023; O’Brien and Lewis, 2023): greedy (1+β)Ye−βYa(1+β)Y_e-β Y_a over the plausibility set Ye>logα+maxYe\Y_e> α+ Y_e\ with α=0.5,β=1.0α=0.5,β=1.0. The expert is Qwen3-32B on xcompx_comp; the amateur is the drafter on xcompx_comp. Method Drafter ctx Verifier ctx δ source SD (Leviathan et al., 2023) = = — EAGLE (Li et al., 2024c) = = — CD (Li et al., 2023b) expert — capacity SCD (Yuan et al., 2023) expert amateur capacity RAPID (Chen et al., 2025) short long — VL-SD (Huang et al., 2025; Huo et al., 2025; Kang et al., 2025) VL VL — SD2 (Berdoz et al., 2026) V→ steer = — Ours full compressed context-gain Table 8: Design-space placement. “δ source” is what the linear logit combination measures. Capability Benchmark n Compression Metric Long-context multi-hop QA LongBench 600 8.1× per-subset F1 Multi-turn instruction following MultiChallenge 271 7.6× llm-judge acc Tool use API-Bank 200 7.4× API-call exact match Multimodal reasoning MathVista 587 cross-modal accuracy End-to-end GAIA L1–3 (full) 165 per-turn / per-file GAIA exact match End-to-end SimpleQA 500 per-turn llm-judge (Wei et al., 2024) Table 9: Capabilities and benchmarks. Compression is the token ratio |xfull|/|xcomp||x_full|/|x_comp|; per-turn LLMLingua-2 for GAIA/SimpleQA (Appendix B). LongBench is 200200 examples each from hotpotQA, 2WikiMQA, MuSiQue, reported per-subset (Table 1). Appendix C Broader Connections to Adaptive and Agentic LLM Systems Agent memory and deep-research systems. Long-horizon agents organize interaction histories as persistent memory or allocate additional computation to search and refinement. Representative systems structure conversational memory as sentence graphs (Wu et al., 2025c), develop reinforcement-learning foundations for deep research (Li et al., 2025a), refine search trajectories with step-level feedback (Zhang et al., 2026a), unify multimodal document parsing with deep research (Dong et al., 2026), or adapt search intensity to problem difficulty (Shi et al., 2025). These methods operate above the token-level decoder; AsymSpec provides a complementary decoding substrate when their accumulated evidence is compressed. Small–large collaboration and inference-time steering. Adjacent training-time mechanisms use teacher–student alignment, fidelity-controlled recursive self-improvement, or self-paced curricula over reward dynamics and data utility (Zhang et al., 2024; Zhang et al., 2026d; Zhi et al., 2026). Other work uses follow-up likelihood as an alignment signal (Zhang et al., 2025a), lightweight steering modules across modalities (Feng et al., 2025), or on-device specialist models for efficient information extraction (Wen et al., 2025). These approaches optimize training or task-specific modules, whereas AsymSpec coordinates a small drafter and large verifier during decoding. Structured and multimodal interfaces. Retrieval-based knowledge integration with controllable generation (Shen et al., 2026), parameter-efficient multimodal fusion (Liang et al., 2022), retrieval-augmented schema adaptation (Liang et al., 2025a), and parameterized tool schemas (Liang et al., 2025b) provide complementary ways to expose structured or multimodal information to language models. AsymSpec does not prescribe these upstream interfaces; it transfers information across their rich and compact views at generation time. Recommendation as an agentic long-context workload. User-centric and interactive recommendation increasingly combines device–cloud agents, multi-step planning, and persistent or lifelong user histories (Zhang et al., 2026c; Yu et al., 2025; Zhou et al., 2026). These workloads motivate efficient handling of long behavioral contexts through retrieval and refinement (Xu et al., 2025; Shen et al., 2024), cross-domain representation transfer (Zhang et al., 2025d), real-time user-specific inference (Xie et al., 2025), and scalable long-sequence architectures (Ye et al., 2026; Pan et al., 2025). Related work also unifies retrieval and ranking (Zhang et al., 2025c), develops LLM-based generative recommendation (Wang et al., 2025a), and studies large-model design and scaling behavior (Guo et al., 2024; Shen et al., 2025). These application- and architecture-level advances are orthogonal to AsymSpec, which optimizes token-level generation through asymmetric drafter–verifier context allocation. Appendix D Cross-family portability We retain the LongBench data, summary compressor, and default decoding configuration (K=2K=2, β=1.0β=1.0, γ=0.5γ=0.5), changing only the drafter–verifier model pair. Same-family controls use their native vocabularies. For heterogeneous pairs, we follow the vocabulary-alignment scheme of Timor et al. (2025): δ is computed in the drafter vocabulary and mapped to the verifier through 109,566109,566 string-identical tokens. Committed tokens are restricted to this shared set together with paired special tokens. Drafter → verifier Floor Ours Ceiling Recovery Llama-3B → Llama-70B 50.6 54.2 66.3 23% Qwen-4B → Llama-70B 50.6 58.4 66.3 50% Llama-3B → Qwen-32B 45.0 47.1 65.5 10% Qwen-4B → Qwen-32B 45.0 59.7 65.5 72% Table 10: LongBench portability across model families (overall mean F1). Floor and Ceiling run each verifier on the compressed and full contexts, respectively; recovery is (AsymSpec−Floor)/(Ceiling−Floor)( AsymSpec-Floor)/(Ceiling-Floor). The Qwen–Llama rows improve over their respective Floors, with recovery varying across model pairs. Appendix E Cross-modal Implementation Patches The cross-modal extension required five patches to the vLLM speculative-decoding code path. Patches target vLLM v0.19.0; the speculative-decoding APIs (DraftModelProposer, triton_utils, etc.) are restructured in newer vLLM releases and on newer Qwen variants, requiring re-porting. 1. Per-request cache for pixel_values and image_grid_thw delivered via sampling_params.extra_args. 2. A vision-tower forward + embedding merge that runs once per request and caches the image embeddings. 3. A hand-computed Qwen3-VL 3-D M-RoPE positional encoder for the drafter’s aug prompt; we verified this matches the official _get_mrope_input_positions bit-for-bit. 4. Critically: routing the merged image embeddings through the speculative-decoding engine’s m_embed_inputs parameter, so the drafter’s prefill actually receives image embeddings rather than text-only embeddings of image_pad tokens. 5. A relaxation of the engine’s aug-substitution gate that originally required |xfull|>|xcomp||x_full|>|x_comp| (false in cross-modal where image tokens are typically fewer than caption tokens). Without patches (4) and (5), the drafter never sees the image and MathVista collapses to 30.5%30.5\% (vs. 53.0%53.0\% with patches), demonstrating that the implementation is non-trivial. Appendix F Throughput Table Table 11 reports eager-mode throughput (tokens/s). MathVista uses VL drafter alone as reference (text-only ceiling undefined). Text-benchmark variance is consolidated in the accuracy-equivalent efficiency metrics in Table 1. LongBench MultiChallenge API-Bank MathVista Method (4B, K=2) (4B, K=2) (1.7B, K=2) (VL-2B, K=4) Floor (verifier on compressed) 50.0 52.3 51.2 49.2 Ceiling (verifier on full) 37.5 51.5 49.5 81.7 SD 50.9 77.5 70.0 65.6 fixed-γ (no gate) 57.8 68.6 67.2 44.4 ++ tuned λ (0.10.1) 88.2 96.0 66.6 46.9 ++ CDA (ours) 63.3 66.6 66.1 48.0 Speedup, CDA vs. full 1.69× 1.29× 1.34× — Table 11: Throughput (tokens/s) across four benchmarks at our default configurations on a single accelerator; eager mode (no graph capture). MathVista has no text-only Ceiling since Qwen3-32B cannot consume images; the VL drafter alone (Qwen3-VL-2B reading the image) at 81.7 tokens/s is the closest reference upper bound. Eager throughput exhibits non-trivial per-run variance on text benchmarks; the accuracy-equivalent main-table efficiency is consolidated in Table 1. CDA adds only one scalar division per token over the tuned λ-variant, so observed gaps reflect measurement variance, not gate overhead. MathVista throughput includes the per-request vision-tower forward, amortized via the cache in Appendix E (patch 2); batched, graph-captured measurement would tighten these further. The final row reports CDA throughput as a ratio over Ceiling—the per-benchmark speedup—with MathVista omitted as it has no text-only Ceiling. Patterns. • Fixed-γ throughput exceeds compressed baseline because successful drafting amortizes the third forward over K tokens and skips verifier autoregressive steps. • AsymSpec vs. SD operating points. SD-full reaches Ceiling accuracy at full-context cost; AsymSpec reaches near-Ceiling accuracy at compressed-verifier cost (∼ 0.2–0.7× compute, Table 5). Throughputs are comparable on every benchmark (LongBench 63.363.3 vs. 50.950.9; MultiChallenge 66.666.6 vs. 77.577.5; API-Bank 66.166.1 vs. 70.070.0), so the two methods are differentiated by cost regime, not by speed. • Cross-modal overhead. MathVista is the slowest regime due to the per-request vision-tower forward, the necessary cost of cross-modal capability extension. Why throughput << FLOPs reduction. LLM decoding is memory-bandwidth-bound. Reducing FLOPs to 0.20.2–0.7×0.7× does not translate linearly to wall-clock; even optimized vanilla SD reaches ≈ 1.5× at 30B+ scale (Xia et al., 2024). AsymSpec’s 1.31.3–1.7×1.7× aligns with this hardware regime. FLOPs quantify compute/energy savings; throughput reflects realized latency gains. Acceptance dynamics. The realized speedup is also bounded by drafter–verifier agreement. Table 13 records the drafter acceptance rate (AR) and mean accepted length (MAL) at our default configurations: AR sits in [0.78,0.92][0.78,0.92] and MAL in [2.6,2.8][2.6,2.8] of the available K+1=3K+1=3 positions for K=2K=2 rows, persisting through the multi-turn GAIA loop where context is re-compressed online. Asymmetric context therefore does not erode verifier–drafter agreement; the gap between FLOP reduction and realized throughput is bounded by the memory-bandwidth regime above, not by acceptance failures. LongBench F1 MultiChallenge acc MathVista acc Method K=2K=2 K=4K=4 K=6K=6 K=2K=2 K=4K=4 K=6K=6 K=2K=2 K=4K=4 K=6K=6 fixed-γ (no gate) 57.0 59.7 57.4 25.5 22.9 20.2 53.0 51.1 50.4 ++ tuned λ (0.10.1) 59.0 59.3 57.8 23.2 21.8 17.3 52.5 52.5 50.7 ++ CDA (ours) 59.7 61.1 58.7 22.5 19.6 18.5 52.6 53.9 52.1 Table 12: K-sweep at Qwen3-4B drafter (Qwen3-VL-2B for MathVista), γ=0.5γ=0.5, β=1.0β=1.0. MultiChallenge declines at K=4K=4 for every method and none recovers its K=2K=2 level — the binding constraint behind the K=2K=2 default (§6.1); cross-modal MathVista is the lone K=4K=4 beneficiary (Table 3). The K=6K=6 sweep saturates on LongBench (CDA 58.758.7, below the K=4K=4 peak of 61.161.1) and degrades on MultiChallenge / MathVista, confirming the K=2K=2 default. Setting AR MAL LongBench, K=2K=2, 4B, summary 0.86 2.72 LongBench, K=4K=4, 4B, truncate 0.78 4.13 MultiChallenge, K=2K=2, 4B, llmlingua 0.80 2.61 API-Bank, K=2K=2, 1.7B, signature 0.92 2.84 GAIA, K=2K=2, 4B, per-turn llmlingua 0.90 2.81 Table 13: Speculative-acceptance diagnostics at our default configurations. AR and MAL remain near vanilla-SD levels under asymmetric context, including in the multi-turn GAIA agent loop. Quality–throughput tradeoff. Each of the three components adds compute: δ-fusion requires the drafter’s compressed-context forward (cost reduced to ⁡(K)O(K) per step by maintaining a separate KV cache across speculation steps), the fixed-γ acceptance is essentially free, and CDA adds one scalar division per token. For cross-modal runs the vision-tower forward is amortized once per request via the per-request embedding cache (Appendix E, patch 2), so its per-token contribution is inverse in the number of decoded tokens and vanishes at long generations. Appendix G Additional ablation grids Full grids referenced in §6: Table 12 (K-sweep), Table 14 (JSD vs. KL), Table 15 (γ-robustness), Table 16 (β-sweep), and Table 17 (compressor probe). CDA’s parameter-free design matches or exceeds tuned baselines across all grids without introducing dataset-specific scales. Benchmark DiD_i gate K=2K=2 K=4K=4 LongBench KL (unbounded) 58.6 59.6 JSD (≤ln⁡2≤ 2, ours) 59.7 61.1 API-Bank KL (unbounded) 63.5 62.6 JSD (≤ln⁡2≤ 2, ours) 63.5 62.6 Table 14: JSD vs. KL instantiation of CDA’s divergence DiD_i (Equation 3; γ=0.5γ=0.5; LongBench overall F1, 4B drafter; API-Bank api-acc, 1.7B drafter). On API-Bank the two gates are identical down to the pass count (339/534 at K=2K=2, 334/534 at K=4K=4): API-Bank’s small context divergences leave JSD and KL with the same accept/reject decisions. On LongBench they differ by ≤1.5≤ 1.5 F1. JSD is chosen for the universal bound (§3.3), not for accuracy, and is used in all other tables. γ 0.4 0.5 0.6 0.7 LongBench F1 56.8 59.7 58.8 59.1 API-Bank acc 63.7 63.5 63.7 63.5 MultiChallenge acc 23.3 23.3 22.1 23.6 Table 15: CDA is flat in its only free quantity, the standard speculative-decoding threshold γ (parameter-free gate γe−Dγ e^-D, K=2K=2; LongBench overall F1, 4B drafter; API-Bank api-acc, 1.7B; MultiChallenge acc, 4B). No cliff on any of the three. K=2K=2 K=4K=4 β F1 AR MAL F1 AR MAL 0.5 50.3 0.853 2.71 57.5 0.781 4.12 1.0 53.7 0.852 2.70 58.0 0.783 4.13 2.0 53.9 0.854 2.71 57.4 0.779 4.12 Table 16: β sweep on LongBench (truncate-1500, 4B drafter, γ=0.5γ=0.5). Within β∈1.0,2.0β∈\1.0,2.0\ F1 is flat (within 0.20.2 at K=2K=2 and 0.60.6 at K=4K=4); β=0.5β=0.5 underperforms by 33–3.63.6 F1 at K=2K=2, motivating β=1.0β=1.0 as a stable midpoint. Acceptance rate (AR) and mean accepted length (MAL) are flat throughout. xcompx_comp source Floor SCD AsymSpec Recovery Summary 45.0 42.3 58.6 66%66\% LLMLingua-2 36.1 31.7 54.6 63%63\% Truncate-1500 32.6 28.8 53.7 64%64\% Question-only 29.5 25.8 54.8 70%70\% Ceiling 65.5 Table 17: Compressor probe on LongBench (4B drafter, K=2K=2, γ=0.5γ=0.5). Recovery is (AsymSpec−Floor)/(Ceiling−Floor)(AsymSpec-Floor)/(Ceiling-Floor). Appendix H Extended drafter-size sweep Tables 18, 20 and 21 report the full drafter capacity sweep. Key takeaways: (1) CDA requires ≥ 1.7B drafters to reliably extract context-gain signals; (2) on API-Bank, the tuned λ-variant and CDA are numerically equivalent due to small context divergences, confirming CDA’s bound does not penalize low-divergence regimes; (3) MultiChallenge differences remain within the ≤ 3 p headroom band, consistent with its near-lossless compression profile. CDA’s advantage is hyperparameter-free robustness, not peak accuracy on near-lossless tasks. Tuned-λ reference baseline. The tuned λ-variant (defined in Appendix B) requires a per-dataset λ. On judge-independent metrics, CDA’s parameter-free gate matches or exceeds it at 4B—identical on API-Bank and +0.7+0.7/+1.8+1.8 F1 on LongBench at K=2/4K=2/4 (Tables 18 and 20). Tables 20 and 21 extend it to API-Bank and MultiChallenge. API-Bank. Two regimes. At K=2K=2 the size-accuracy relation is inverted-U with 1.7B as sweet spot (59.6 → 63.5 → 60.7 for tuned-λ / CDA), confirming the fixed-γ pattern (§5.1). At K=4K=4 it becomes monotone-increasing and the best API-Bank cell of the sweep is 4B K=4K=4 (tuned-λ / CDA both 63.7). The tuned variant and CDA give numerically identical numbers in all six cells — API-Bank’s relatively small context divergences between drafter-on-aug and drafter-on-main make the tuned λ=0.1λ=0.1 gate and CDA’s parameter-free JSD gate effectively interchangeable (the JSD and KL instantiations of CDA are themselves numerically identical on API-Bank, Table 14). MultiChallenge. MC has only ≈ 3 p headroom; the per-cell differences in Table 21 are too small to order the methods consistently (e.g. at 4B K=2K=2 fixed-γ / tuned-λ / CDA =25.5/23.2/22.5=25.5/23.2/22.5, a ≤ 3 p band that re-shuffles across drafter sizes). The only MC takeaway is that AsymSpec does not improve a near-lossless task and sits mildly below the compressed floor there (Table 1). Drafter K Drafter alone fixed-γ ++ tuned λ ++ CDA (ours) 0.6B 2 23.7 50.6 48.4 47.8 0.6B 4 23.7 48.5 46.6 46.4 1.7B 2 38.9 52.9 51.1 50.8 1.7B 4 38.9 48.5 53.6 53.9 4B 2 54.6 57.0 59.0 59.7 4B 4 54.6 59.7 59.3 61.1 Table 18: Drafter-size sweep on LongBench F1 (γ=0.5γ=0.5, λ=0.1λ=0.1 for the tuned variant). CDA is the parameter-free JSD gate (§3.3); the 4B row (our default drafter) is its JSD instantiation of record, the smaller-drafter rows the accuracy-equivalent KL instantiation (Table 14). Drafter alone: the SLM on full passages, no verifier — reference upper bound for what the small model can do unassisted (K-independent). λ MultiChallenge acc 0.05 21.0 0.10 (default tune) 23.2 0.20 23.6 CDA (no scale) 22.5 Table 19: λ sweep of the tuned variant on MultiChallenge under the llm-judge (n=271n=271, 4B drafter, γ=0.5γ=0.5, K=2K=2). The spread is only 2.62.6 p (21.021.0–23.623.6). CDA (tuning-free, 22.522.5) lies within the same band with no λ to set. Drafter K Drafter alone fixed-γ ++ tuned λ ++ CDA (ours) 0.6B 2 31.3 59.4 59.6 59.6 0.6B 4 31.3 58.2 57.9 57.9 1.7B 2 61.2 63.7 63.5 63.5 1.7B 4 61.2 62.4 62.5 62.5 4B 2 64.0 60.3 60.7 60.7 4B 4 64.0 63.3 63.7 63.7 Table 20: Drafter-size sweep on API-Bank api_acc (γ=0.5γ=0.5, λ=0.1λ=0.1 for the tuned variant). Drafter alone: the SLM alone on full API context, no verifier (K-independent). Drafter K Drafter alone fixed-γ ++ tuned λ ++ CDA (ours) 0.6B 2 13.3 20.3 19.6 21.4 0.6B 4 13.3 21.8 22.9 22.1 1.7B 2 14.0 19.6 20.7 19.6 1.7B 4 14.0 21.8 18.1 21.4 4B 2 25.8 25.5 23.2 22.5 4B 4 25.8 22.9 21.8 19.6 Table 21: Drafter-size sweep on MultiChallenge accuracy (γ=0.5γ=0.5, llm-judge, n=271n=271, λ=0.1λ=0.1 for the tuned variant; single run). Every per-cell gap lies within the ≤ 3 p headroom on this near-lossless task (§6.4).

Instant research discovery