Paper deep dive
TreeWY: Speculative Verification for Gated DeltaNet Hybrids
Sneha Murthy Ghantasala
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/24/2026, 5:44:30 AM
Summary
The paper introduces TreeWY, a method for speculative verification in hybrid language models using Gated DeltaNet (GDN) layers. It replaces memory-intensive full-state snapshotting with a tree-structured WY transform, allowing efficient parallel verification of draft trees by solving a single triangular system for pseudo-values. This reduces recurrent-state memory and KV-cache pressure, enabling higher throughput and lower time-to-first-token (TTFT) in memory-bound scenarios, and making wider draft trees affordable.
Entities (8)
Relation Signals (7)
TreeWY → evaluatedon → Qwen3.5
confidence 95% · In serving benchmarks on two scales of one hybrid model family (Qwen3.5 35B and 397B)
TreeWY → optimizes → Gated DeltaNet
confidence 95% · TreeWY: Speculative Verification for Gated DeltaNet Hybrids
TreeWY → uses → WY transform
confidence 95% · Using a tree-structured WY transform of the gated delta rule, we compute every draft node's output with a single triangular solve
TreeWY → implementedin → vLLM
confidence 90% · We implement TreeWY for vLLM, as a fork of its main branch
TreeWY → replaces → full-state snapshotting
confidence 90% · We remove the snapshots. ... The standard fix, which we call full-state snapshotting, keeps the committed state plus one full snapshot per draft position
ReplaySSM → issimilarto → TreeWY
confidence 85% · ReplaySSM has also been incorporated in SGLang ... It is the closest prior point in vLLM; we benchmark against it
Bole → issimilarto → TreeWY
confidence 85% · Bole [12] ... is the most direct overlap with this paper: it also rewrites the linear-attention recurrence into a tree-structured closed form
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Modern open models are hybrids: most layers are linear-attention (Gated DeltaNet, GDN) layers carrying a small fixed-size recurrent state instead of a growing key-value (KV) cache. This makes ordinary decoding memory-efficient, but hurts speculative decoding. To verify a batch of draft tokens and then roll back the rejected ones, today's systems snapshot the full recurrent state at every draft position for GDN layers, and those snapshots cannot be shared across branches of a draft tree, so a wide, high-acceptance tree becomes memory-infeasible. We remove the snapshots. Using a tree-structured WY transform of the gated delta rule, we compute every draft node's output with a single triangular solve and reconstruct only the one accepted state on commit, storing a small pseudo-value matrix instead of per-node states; the derivation depends only on the gated delta rule, not on any other architectural detail. In serving benchmarks on two scales of one hybrid model family (Qwen3.5 35B and 397B) this cuts speculative recurrent-state memory and KV-cache pressure at identical acceptance length, turning the freed HBM into higher throughput and much lower time-to-first-token (TTFT) wherever memory binds, and costing a few percent where it does not. For tree width the same memory buys affordability: a wider, higher-acceptance draft becomes possible, though not yet a throughput win.
Tags
Links
- Source: https://arxiv.org/abs/2608.20961v1
- Canonical: https://arxiv.org/abs/2608.20961v1
Trouble viewing inline? Open PDF directly →
Full Text
30,103 characters extracted from source content.
Expand or collapse full text
TreeWY: Speculative Verification for Gated DeltaNet Hybrids Sneha Murthy Ghantasala Thomson Reuters sneha.ghantasala@thomsonreuters.com Abstract Modern open models are hybrids: most layers are linear-attention (Gated DeltaNet, GDN) layers carrying a small fixed-size recurrent state instead of a growing key- value (KV) cache. This makes ordinary decoding memory-efficient, but hurts speculative decoding. To verify a batch of draft tokens and then roll back the rejected ones, today’s systems snapshot the full recurrent state at every draft position for GDN layers, and those snapshots cannot be shared across branches of a draft tree, so a wide, high-acceptance tree becomes memory-infeasible. We remove the snapshots. Using a tree-structured WY transform of the gated delta rule, we compute every draft node’s output with a single triangular solve and reconstruct only the one accepted state on commit, storing a small pseudo-value matrix instead of per-node states; the derivation depends only on the gated delta rule, not on any other architectural detail. In serving benchmarks on two scales of one hybrid model family (Qwen3.5 35B and 397B) this cuts speculative recurrent-state memory and KV-cache pressure at identical acceptance length, turning the freed HBM into higher throughput and much lower time-to-first-token (TTFT) wherever memory binds, and costing a few percent where it does not. For tree width the same memory buys affordability: a wider, higher-acceptance draft becomes possible, though not yet a throughput win. 1 Background Speculative decoding. Autoregressive decoding emits one token per forward pass, so it is memory- bandwidth bound and the GPU sits mostly idle. Speculative decoding [1,2] fills that idle compute: a cheap drafter (a small model, or a lightweight “MTP” head shipped with the target) proposesk candidate tokens; the target verifies allkin one pass and accepts the longest prefix it would have generated itself, leaving the output distribution unchanged but emitting several tokens per pass. A tree drafter proposes multiple alternatives per position [3]. Hybrid models and GDN layers. A hybrid model mixes softmax-attention with linear-attention layers (for e.g., Gated DeltaNet, GDN). A softmax layer keeps a KV cache: it stores a key and value vector for every previous token, so its memory grows linearly with context. A Gated DeltaNet (GDN) layer [4] instead maintains one recurrent state matrixS ∈R d v ×d k per head summarizing the whole prefix in fixed size. For tokentwith keyk t , valuev t , queryq t , scalar decay gateα t ∈ (0, 1)and write strength β t ∈ (0, 1), the state evolves by the gated delta rule S t = α t S t−1 I − β t k t k ⊤ t + β t v t k ⊤ t , o t = S t q t .(1) The transitionT t = α t (I − β t k t k ⊤ t ) is a scalar decay times a rank-1 correction: theβ t k t k ⊤ t term “erases” what the state already predicts fork t before writing the new value (the “delta” rule). Crucially, Sis a lossy summary of the whole prefix. It cannot be truncated or partially rolled back the way a KV cache can. That single fact is the source of the speculative-decoding problem. Preprint. arXiv:2608.20961v1 [cs.AI] 21 Aug 2026 2 The memory problem We make the contrast concrete on the two Qwen3.5 [5] hybrids we evaluate, both3:1GDN-to-softmax withd k =d v = 128and softmaxd head = 256, 2 KV heads: 35B-A3B (30 GDN, 32 value heads; 10 softmax) and 397B-A17B (45 GDN, 64 value heads; 15 softmax). Normal decoding. Summed over the softmax layers, the KV cache costs2(K,V) × n softmax × n KV-head × d head × 2B (bf16) per token:20KiB/token at 35B and30KiB at 397B, and it grows with context (0.625and0.94GiB/seq at a 32K context). The GDN layers instead hold one fixed state, n GDN × n v-head × d v × d k × 2B (bf16) per sequence:30and90MiB/seq, forever, which is why hybrids are attractive. Speculative verification: where GDN blows up. Speculation inverts the picture. To verifykdraft tokens (or a tree of N nodes): •Softmax layers stay cheap. Draft KV entries are appended, and rejected ones are dropped by moving a pointer, i.e., rollback is free. A tree even shares its common-prefix KV across branches. The marginal cost is a handful of KiB. •GDN layers explode during verification. Verification is one parallel forward pass over all ktokens at once. In that single pass the GDN recurrent state advances all the way to the end (past tokenk) before any acceptance is checked. And so, we need to rollback to the state at the accepted node. The standard fix, which we call full-state snapshotting, keeps the committed state plus one full snapshot per draft position:k+1blocks per sequence per GDN layer for a chain ofkdraft tokens (a tree ofNnodes needsN +1, one per node). Fork = 3that is4×the committed state,120MiB/seq (35B) and360MiB/seq (397B). Moreover, the snapshots are unshareable across branches. Snapshot memory scales with draft tokens, caps concurrency and rules out wide trees. This is the bottleneck we remove. Existing approaches and trade-offs. •(i) Full-state snapshotting. The current default in both vLLM [6] (storeallin our figures) and SGLang [7]: one recurrent-state snapshot per draft position, rolled back by indexing to the accepted one. Rollback is free; memory scales with draft size and snapshots cannot be shared across tree branches. • (i) Deferred materialization with a rank-1 cache (ReplaySSM [8], shipping concurrently July 2026 in a vLLM RFC and TensorRT-LLM) keeps a checkpoint plus a short cached history and defers materializing the state until a periodic flush, so state is written far less often than every draft step. Inside the verify window it solves the same triangular system we do (Section 3), so the distinction is when state is materialized, not how it is computed. It covers Mamba2 and GDN, and the implementation linked in the RFC is chain-only (not a tree). It is the closest prior point in vLLM; we benchmark against it in Appendix D. ReplaySSM has also been incorporated in SGLang [9] as an opt-in feature. • (i) Transition-matrix tree recompute (STree [10,11]) verifies a tree, but only for Mamba2: its transition is a bare scalar decay, so its speedup rests on turning a product of gates into a cheap cumulative sum. This does not exist for GDN’s non-commuting matrix transition (scalar decay× rank-1 correction). •(iv) Tree-structured closed form in SGLang (Bole [12], concurrent, Aug. 2026) is the most direct overlap with this paper: it also rewrites the linear-attention recurrence into a tree- structured closed form so a whole draft tree can be verified in parallel rather than branch by branch, reporting82–99×transient-memory reduction and3.4–7.7×faster tree verification from its kernel. It targets SGLang rather than vLLM and, by its own description, is framed at the level of hybrid-attention recurrences generally. We have not run a head-to-head, since Bole’s paper does not link to a code implementation; it is the closest concurrent work and the comparison we most want to make next. Our method is the tree-structured WY/UT transform of the gated delta rule (Section 3), implemented in vLLM (Section 4): one triangular solve verifies every draft node. It generalizes DeltaNet’s [13] chain WY transform to a tree strict-ancestor form reused for verification and rollback. A single small pseudo-value matrix written only on commit stands in for the per-node snapshots. (iv) is architecturally closest to us and was arrived at independently on a different serving stack. 2 3 Method: TreeWY Key rewrite. Expanding(1)shows the delta rule is decayed additive attention with a corrected value:S t = α t S t−1 + ̃v t k ⊤ t with pseudo-value ̃v t = β t (v t − α t S t−1 k t )(raw value minus what the state already predicts). Given the ̃v t , both the state and every output are decay-weighted sums over the preceding tokens, so obtaining the ̃v t without walking the recurrence removes the need for intermediate states. Verifying a draft. A draft can be a chain or a tree: a chain proposes one candidate per position, a tree lets a node fan out into several. Either way, we lay out itsNnodes in DFS pre-order from the committed stateS 0 , so every ancestor of nodethas a smaller index (i ≺ t; on a chain this is just i < t, andN =k). Chaining the per-token rewrite along every root-to-node path turns verification into one linear system for all the pseudo-values at once, ̃ V ∈R N×d v : I + diag(β)G ̃ V = R, G[t,i] = g t g i (k ⊤ t k i ) [i≺ t ], R[t] = β t v t − β t g t (S 0 k t ),(2) withg t the cumulative decay fromS 0 tot. Parents precede children, soGis strictly lower-triangular, and the whole draft solves in one forward substitution: no recurrence, no per-node state, and no change to the system as the tree widens. This is DeltaNet’s [13] WY/UT transform applied to the whole verify window at once; every node’s output reads out from that same solve. Reconstruct on commit. Once verification accepts nodea, we rebuild the continuation state directly from ̃ V , summing over a’s ancestors: S a = g a S 0 + X i⪯a g a g i ̃v i k ⊤ i ,(3) which becomes the next round’sS 0 . We store only ̃ V,O(Nd v ), instead of one full state per node: a 128×smaller object per head atd k =d v =128, which is what turns snapshotting’sN +1state blocks per sequence per layer into one, for a chain or a tree alike. The scalar-decay ancestor mask is STree’s [10]; extending it to carry GDN’s rank-1 delta correction, so it covers GDN’s non-commuting transition rather than only Mamba2’s scalar one, is the piece we add. 4 Implementation We implement TreeWY for vLLM, as a fork of its main branch (not yet upstreamed as of this writing) through twoSpeculativeConfigoptions:mamba_state_commit="reconstruct"turns on our commit strategy in place of the default"store_all", anddraft_tree_widthsturns a chain into a tree by giving the branching factor per level. All evaluations here run with prefix caching disabled. A chain (or an all-ones tree, which is really just a chain) verifies and commits in one fused, CUDA- graph-capturable Triton kernel. A real tree (w>1) needs a non-causal ancestor mask that cannot be replayed from a CUDA graph, so vLLM drops the whole model to piecewise capture, which also evicts the GDN mixer from graphs, making the cost far larger than the mask itself. Trees must also be scheduled atomically: a DFS prefix of a tree is a different topology, so a request whose tree does not fit the per-step token budget skips speculation rather than being truncated. All results use greedy drafting and verification. We check correctness two ways: the closed form matches the per-node recurrence to∼ 10 −15 (fp64) and∼ 10 −7 (fp32), and the production kernel matches that reference within bf16 tolerance. Token streams are therefore not bit-identical to the baseline; we gate correctness against a shared no-speculation reference and compare the acceptance length directly between treewy and storeall. 5 Evaluation Setup. We serve Qwen3.5-35B-A3B (tensor-parallel degree 1, TP1) and Qwen3.5-397B-A17B (TP8) in vLLM on B200 GPUs (178GiB HBM/device) with a depth-3 MTP draft chain, comparingtreewy (ours) against snapshotting (vLLM’s default,storeall). We sweep GPU memory utilization (gmu)∈ 0.6, 0.75, 0.9over six workloads (ShareGPT [14], spec-bench [15], BurstGPT [16], and synthetic balanced-chat, generation-heavy, summarize-heavy). 1 Five of the six sweep fixed 1 Licenses: Qwen3.5 [5] and vLLM [6] are Apache-2.0, as is spec-bench; BurstGPT is C-BY-4.0; the ShareGPT mirror we use is tagged Apache-2.0 on Hugging Face. The synthetic workloads use vLLM’s own random benchmark generator. 3 Table 1: Concurrency sweep at each model’s tightest measured budget (35B:gmu0.6; 397B:gmu0.75), geomean over 5 work- loads, > 1 favours TreeWY. tput is output throughput; KV red. is peak KV-cache usage reduction; TTFT is p99 TTFT; TPOT is mean time-per-output-token (TPOT).bis the ratio of peak admitted requests, the mechanism behind the other columns past the knee (remaining budgets: Table 3). 35B-A3B (TP1, gmu 0.6)397B-A17B (TP8, gmu 0.75) conc. tput KV red. TTFT TPOTbtput KV red. TTFT TPOTb 10.942.431.020.941.000.932.000.950.941.00 81.022.401.031.021.000.972.191.080.961.00 321.012.451.021.001.001.012.190.991.011.00 641.002.441.010.991.011.012.191.001.021.00 1281.201.645.620.831.451.022.091.070.991.04 2561.401.043.970.602.501.151.613.350.901.24 Table 2: Tree width vs. acceptance length (Qwen3.5-35B-A3B, TP1, gmu0.9, spec_bench, greedy, no prefix caching; more information in Appendix B). shapeNacc. storeall chain33.24 reconstruct33.23 treewy tree (1, 1, 1)33.23 (2, 1, 1)63.30 (2, 2, 1)103.38 (2, 2, 2)143.38 (3, 2, 2)213.41 (3, 3, 2)303.45 (3, 3, 3)393.58 max-concurrency∈1, 8, 32, 64, 128, 256; BurstGPT instead sweeps its own Poisson arrival-rate ∈4, 8, 16, 32, 64 req/s, a different independent variable. Correctness. A matched point is one (dataset, concurrency, gmu) combination run under both treewyandstoreall:5workloads× 6concurrencies+BurstGPT× 5arrival rates= 35per gmu, times3 gmuat 35B and2at 397B gives175total (105at 35B,70at 397B), the same partition Fig. 3 splits by memory pressure. Across all of them, acceptance length is essentially identical to the baseline (mean|∆| = 0.039, max 0.33), matching to within 0.01 at every depth on real prompts. Memory and throughput. TreeWY’s peak KV usage is2–3×lower at the same load (one state block per layer vs.k+1), so it preempts far less under pressure (1365vs.2531requests; just51of treewy’s are at 397B, vs.909for storeall). That headroom drives Table 1 and Fig. 2: TreeWY wins outright where memory binds (up to1.49×throughput,∼ 40×lower p99 TTFT), and trails by a few percent on throughput (0.97–0.99×) where it doesn’t, still at2–3×less KV. That per-token dip is the price of admitted concurrency, not a slower kernel (Appendix C). The knee migrates predictably with the budget: 128 concurrency atgmu=0.6, 256 at0.75, never at 0.9(Appendix A); and whereverb=1.00the freed memory is real but unspent: KV reduction still holds its full2.2–2.5×while every other column sits within3%of parity. Fig. 3 decomposes every metric by memory pressure. We separately benchmark against ReplaySSM on the identical 35B sweep (Appendix D). Tree width. Table 2 sweeps tree widthwat depth 3 against a store-all chain baseline (more shapes and block costs in Appendix B). TreeWY holds one state block per layer regardless ofw(the same freed- memory mechanism as Table 1, now spent on width instead of concurrency), and acceptance length keeps rising withweven as the per-position rate falls (Appendix B). Width is therefore affordable under reconstruction where snapshotting can’t afford it at all: a store-all baseline’s per-request cost grows withNwhile TreeWY’s stays flat at one block. It is not yet a throughput win, though: a wider tree pushesN +1tokens per step through the target and runs piecewise rather than fused (Section 4), so we report trees as enabled and correct, not as a speedup. 6 Conclusion Treating the gated delta rule as a tree-structured WY transform lets us verify a GDN draft tree with one triangular solve and reconstruct only the accepted state on commit, tradingO(N )state snapshots for one small pseudo-value matrix, on chains and trees alike. On chains this is already a fused, graph-capturable kernel that turns freed HBM into higher throughput and much lower TTFT wherever memory binds, at a residual per-step cost that is an implementation artifact rather than a property of the closed form. On trees the same mechanism holds the stored state flat at one block regardless of width (Table 4), which makes a wider, higher-acceptance draft affordable, though not yet a throughput win, since acceptance is capped by draft depth and the non-capturable tree verify kernel costs more than the extra acceptance returns. Fusing the tree path into a single graph-capturable kernel and testing the deferred-state-write levers that ReplaySSM and Bole both point to are the next immediate engineering steps, alongside extending to a second model family. 4 References [1]Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML, 2023. [2]C. Chen et al. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318, 2023. [3]T. Cai et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv:2401.10774, 2024. [4]S. Yang, J. Kautz, A. Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR, 2025. [5]Qwen Team. Qwen3.5: Towards Native Multimodal Agents. Blog, Feb. 2026.https://qwen.ai/blog? id=qwen3.5 ; model cards for the two checkpoints evaluated here athttps://huggingface.co/Qwen/ Qwen3.5-35B-A3B and https://huggingface.co/Qwen/Qwen3.5-397B-A17B. [6]W. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). SOSP, 2023. [7]L. Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS, 2024. (Hybrid state pool as of v0.5.16, 2026.) [8] Dao AI Lab and NVIDIA. ReplaySSM: Cache SSM Inputs, Not State. Blog, 2026; vLLM RFC #47572, PR #47576; TensorRT-LLM PR #14203. [9]Y. Luo. [RFC] Porting ReplaySSM to SGLang: Faster Decode and Speculative Decoding for Hybrid (GDN/KDA) Models. sgl-project/sglang Issue #28511, 2026.https://github.com/sgl-project/ sglang/issues/28511. [10] Y. Wu et al. STree: Speculative Tree Decoding for Hybrid State-Space Models. arXiv:2505.14969, 2025. [11]Y. Wu et al. Snakes and Ladders: Accelerating State Space Model Inference with Speculative Decoding. NeurIPS ENLSP Workshop, 2024. [12]L. Wang et al. Bole: Efficient Tree Speculation for Hybrid-Attention Language Models. arXiv:2608.01651, 2026. [13]S. Yang et al. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. arXiv:2406.06484, 2024. [14] ShareGPT (user-shared ChatGPT conversations); we use theanon8231489123mirror. Hugging Face. https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered. [15]H. Xia et al. Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (introduces Spec-Bench). ACL Findings, 2024. [16]Y. Wang et al. BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems. arXiv:2401.17644, 2024. 5 A Remaining memory budgets Table 3: The higher memory budgets, same axes and orientation as Table 1. At 397B the knee appears only at0.75/256. Whereverbreads1.00the freed memory is real but unspent, as discussed in Section 5. conc.tputKV red.TTFTTPOT b Qwen3.5-35B-A3B, gmu 0.75 10.952.321.090.941.00 80.982.441.001.001.00 321.002.451.010.991.00 641.002.480.941.001.00 1281.022.430.991.021.01 2561.181.624.610.801.46 Qwen3.5-35B-A3B, gmu 0.9 (no knee) 10.932.320.910.931.00 80.972.440.650.981.00 320.982.451.060.981.00 640.982.460.940.981.00 1281.002.480.990.991.00 2561.032.261.440.951.07 Qwen3.5-397B-A17B, gmu 0.9 (no knee) 10.932.000.950.921.00 80.972.190.940.961.00 320.992.190.940.991.00 640.992.191.000.991.00 1280.992.220.980.991.00 2561.002.151.020.981.03 B Wide draft trees Wider trees. Snapshotting cannot afford a wide tree; TreeWY can, and Table 2 in the main text is the primary width sweep. Table 4 below adds more shapes on the block-cost side of that trade: a store-all baseline’s per-request cost grows10×(4→ 40blocks) from(1, 1, 1)to(3, 3, 3), while TreeWY’s stays at one block regardless ofw. Acceptance length keeps rising over that same range (1.883→ 2.786) even though it is bounded by draft depth rather than width (nearly flat from(2, 2, 2) to(3, 3, 3)while blocks go15→ 40): the gain comes from matching each level’s probability mass more often, not from a longer accepted path. Table 4: More tree-width shapes (Qwen3.5-35B-A3B, depth-3, branching factorw; primary width sweep in Table 2). “blk” is a store-all baseline’s per-request block cost (N +1); TreeWY holds one block at every w. Acceptance length matches the storeall reference within sampling noise. shapeNblkstorealltreewy (1, 1, 1)341.9791.883 (2, 2, 2)14152.8542.776 (3, 3, 3)39402.8682.786 C Admission vs. per-step cost Reading the latency columns. Admitting more requests raises per-request latency at unchanged per-step cost, so a TPOT ratio below 1 is partly the price of that concurrency: across the 35B points the penalty tracks batch growth almost exactly (Pearsonr =−0.88against the running-batch ratio, +0.88for throughput). At the 31 points where the baseline saturated, TreeWY pays0.83×on TPOT yet its mean end-to-end latency still improves (1.17×), because requests the baseline could not admit stop waiting; at the other 74 its batch is matched to within1%and TPOT is0.98; that, not the aggregate, is our per-step cost (same decomposition for ReplaySSM, r =−0.79). 6 D ReplaySSM comparison Comparison with ReplaySSM. ReplaySSM [8] (approach (i)) is the closest concurrent point with a chain-mode implementation, so we ran the identical 35B sweep against it (Tables 5, 6 below). The two arms sit on different vLLM builds, so each is normalised to the snapshotting baseline from its own image; Table 5 lists both baselines so the drift is visible: the store-all throughputs agree to within1%, so the two arms’ own-baseline ratios below are directly comparable to each other. Both delete the per-position snapshots and free the same order of KV headroom, TreeWY marginally more (0.96×ReplaySSM’s peak usage), and both preserve acceptance on real prompts (mean|∆| acceptance length0.029vs.0.037). ReplaySSM turns its headroom into more throughput on a chain (1.12–1.20×its baseline vs. our0.99–1.08×) and preempts less often. That margin is not the solve (the solve is shared): our working hypothesis is that it is the scheduling around it, since they defer writing state on a verify step where we materialize the accepted state on every commit, but this is an attribution, not a validated claim: we have not prototyped a deferred-write commit path ourselves to check that it actually closes the gap. If it holds, it is an implementation lever open to us and orthogonal to the closed form; testing it, alongside the analogous sweep we have not yet run against Bole [12] (approach (iv), Section 2, which targets a different serving engine), is future work. Table 5: Why the ratios differ, decomposed at 256 concurrency (Qwen3.5-35B-A3B, all 15 points: 5 workloads (ShareGPT, spec-bench, synthetic balanced-chat, generation-heavy, summarize-heavy) ×3gmu; BurstGPT excluded, its own independent variable). Absolute levels first, then the iden- tity. Both methods admit essentially the same work: 238 vs 224 peak running requests, and at gmu0.6 per workload they are indistinguishable (253/255, 254/255, 256/256, 256/256, 109/107 for ShareGPT/spec-bench/balanced-chat/generation-heavy/ summarize-heavy), because both simply reach the offered concurrency once the snapshots are gone. TreeWY’s larger batch ratio in Table 6 is therefore a denominator artifact: its own baseline happened to admit slightly fewer (92 vs 97 on balanced-chat, 61 vs 63 on summarize-heavy). What separates the two is per-token cost, and once both are saturated and decode-bound the throughput ratio is justbatch÷ cost, which closes to within 3%for both. So at this operating point the memory mechanism is a tie and the whole difference is per-step efficiency. OursReplaySSM Store-allTreeWYStore-allReplaySSM peak running requests151.1238.2153.6223.9 output tok/s846910094825011357 mean TPOT (ms)16.421.217.017.6 p99 TTFT (ms)8167274689652958 ratio vs own baseline, and the decode-bound identity admitted batch1.5771.458 per-token cost1.295×1.035× batch÷ cost (predicted)1.2181.408 throughput (measured) 1.192 1.376 7 Table 6: The same load axis for both methods (Qwen3.5-35B-A3B), each normalised to the store-all baseline from its own vLLM build, so the two columns are comparable even though the builds are not. Per-token speedup is based on mean TPOT. ReplaySSM leads on throughput at every load, and the gap is widest exactly where neither method is admitting anything extra (batch1.00, loads 1–64), which is what identifies it as per-step cost rather than a memory effect. Past the knee the two converge on the memory-driven columns: at 256 concurrency the peak-KV reduction (1.56vs.1.57) and the admitted batch (1.58vs.1.46) are within a few percent, i.e. both free the same headroom and spend it the same way. On the BurstGPT arrival-rate sweep at64req/s the p99 TTFT improvements are10.72×and10.20×respectively. The per-token column is the clearest read on per-step cost: ReplaySSM leads it at every load, and Table 5 decomposes the 256-concurrency point. Output tput.Peak KVp99 TTFTPer-token concurrencygain↑reduction↑speedup↑speedup↑ ours / Replayours / Replayours / Replayours / Replay 10.94 / 1.062.36 / 2.151.00 / 0.980.94 / 1.07 80.99 / 1.082.43 / 2.260.87 / 1.201.00 / 1.07 321.00 / 1.112.45 / 2.291.03 / 0.990.99 / 1.11 640.99 / 1.172.46 / 2.290.96 / 1.010.99 / 1.17 1281.07 / 1.272.15 / 2.021.76 / 1.600.94 / 1.14 2561.19 / 1.381.56 / 1.572.97 / 3.030.77 / 0.97 E Additional serving curves 183264128256 max concurrency (in-flight requests) 0.0 0.2 0.4 0.6 0.8 1.0 peak KV-cache usage (fraction of pool) Qwen3.5-35B-A3B · gpu-mem-util=0.6 183264128256 max concurrency (in-flight requests) 0.0 0.2 0.4 0.6 0.8 1.0 peak KV-cache usage (fraction of pool) Qwen3.5-35B-A3B · gpu-mem-util=0.75 183264128256 max concurrency (in-flight requests) 0.0 0.2 0.4 0.6 0.8 1.0 peak KV-cache usage (fraction of pool) Qwen3.5-35B-A3B · gpu-mem-util=0.9 183264128256 max concurrency (in-flight requests) 0.0 0.2 0.4 0.6 0.8 1.0 peak KV-cache usage (fraction of pool) Qwen3.5-397B-A17B · gpu-mem-util=0.75 183264128256 max concurrency (in-flight requests) 0.0 0.2 0.4 0.6 0.8 1.0 peak KV-cache usage (fraction of pool) Qwen3.5-397B-A17B · gpu-mem-util=0.9 storealltreewypool full (100%) KV-cache saturation vs offered load synthetic_generation_heavy (store-all hits the 100% ceiling early; tree-WY keeps headroom at the same load) Figure 1: Peak KV-cache usage (fraction of pool) vs. offered load, per model and GPU memory utilization (gmu). The snapshotting baseline (storeall) hits the 100% ceiling early and must then queue the requests it cannot fit; TreeWY holds2–3×more headroom at the same load. That freed capacity buys the admission, throughput and TTFT wins (Fig. 2). 8 183264128256 max concurrency (in-flight requests) 10 2 10 3 10 4 10 5 p99 TTFT (ms) treewy 3.1× faster at c256 Qwen3.5-35B-A3B · gmu=0.6 183264128256 max concurrency (in-flight requests) 10 2 10 3 10 4 p99 TTFT (ms) treewy 29.7× faster at c256 Qwen3.5-35B-A3B · gmu=0.75 183264128256 max concurrency (in-flight requests) 10 2 10 3 p99 TTFT (ms) treewy 4.5× faster at c256 Qwen3.5-35B-A3B · gmu=0.9 183264128256 max concurrency (in-flight requests) 10 2 10 3 10 4 p99 TTFT (ms) treewy 25.6× faster at c256 Qwen3.5-397B-A17B · gmu=0.75 183264128256 max concurrency (in-flight requests) 10 2 10 3 p99 TTFT (ms) treewy ~same at c256 Qwen3.5-397B-A17B · gmu=0.9 KV mode storealltreewy p99 Time-To-First-Token vs offered load synthetic_generation_heavy (store-all's KV starves at high concurrency, so TTFT blows up; tree-WY sustains it) Figure 2: p99 time-to-first-token (TTFT) vs. offered load (generation-heavy workload). As concur- rency rises, the snapshotting baseline’s KV pool saturates and it must queue requests, so its TTFT blows up (log scale): at 256 concurrency it is up to∼ 30×(35B, utilization 0.75) and26×(397B, utilization 0.75) worse than TreeWY. The largest gap over the whole sweep,40×(683vs.27489ms), is at 128 concurrency (35B, utilization 0.6), where the baseline saturates but TreeWY still does not. The gap closes only when memory is slack (rightmost panels). 0.81.01.52.03.0 ratio vs store-all at matched load ( > 1 = tree-WY better) output tok/s peak KV usage p99 TTFT mean end-to-end per-token (TPOT) 1.15× 1.60× 2.94× 1.17× 0.83× Qwen3.5-35B-A3B (TP=1) 31 memory-bound vs 74 headroom points · control band 0.972.60× 0.81.01.52.03.0 ratio vs store-all at matched load ( > 1 = tree-WY better) output tok/s peak KV usage p99 TTFT mean end-to-end per-token (TPOT) 1.06× 1.63× 1.66× 1.06× 0.95× Qwen3.5-397B-A17B (TP=8) 17 memory-bound vs 53 headroom points · control band 0.972.34× The memory win is regime-dependent: freed HBM only pays where the baseline had run out of it baseline had KV headroom (control)baseline out of KV (memory-bound) Figure 3: Every ratio in Table 1 split by whether the baseline had run out of KV headroom at that load, the partition behind Appendix C. Open markers are the headroom control: by construction every metric sits within a few percent of parity there. Filled markers are where the freed HBM is actually spent: at 35B (31/105 points memory-bound) TreeWY reaches1.15×throughput,2.94×lower p99 TTFT and1.17×lower mean end-to-end latency at1.60×more KV headroom, for a0.83× per-token cost; 397B (17/70points memory-bound) shows the same shape at smaller magnitude (1.06×throughput,1.66×lower p99 TTFT,1.63×more KV headroom,0.95×per-token cost), since its sweep reaches fewer points past the knee. This is the single figure that makes the paper’s central qualifier legible at a glance: the win is not a fixed multiplier, it is conditional on memory pressure, and averaging over both regimes (as a single headline number would) hides exactly that. 9 F Compute resources GPU-hours. All runs use B200 GPUs (178GiB HBM/device). Each job serves one (model, mode, gmu) combination through its full35-point concurrency/arrival-rate sweep in one continuous server session. The 35B main sweep (Table 1, Appendix A) is6jobs (storeall/treewy× 3 gmu) at TP1, ≈ 11.8GPU-hours; the 397B main sweep is4jobs (storeall/treewy × 2 gmu) at TP8,≈ 62.5 GPU-hours; the ReplaySSM comparison (Appendix D) is6further 35B/TP1 jobs,≈ 11.1GPU-hours. Total:≈ 85GPU-hours across the three reported sweeps. This is a lower bound: it excludes one-time model-loading/server-startup overhead at the start of each sweep (∼ 1h for the 397B model,∼ 30min for 35B), which does not recur across jobs within the same sweep. 10