Paper deep dive
Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy
Naji Alazhar, Wassel Alazhar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/13/2026, 2:57:53 AM
Summary
The paper introduces Adaptive DHiSparse, a system for long-context LLM serving that combines hierarchical sparse key-value (KV) caching with speculative decoding (DSpark and DFlash2). It adapts KV residency to system load, storing history in host memory and fetching only selected entries, while supporting draft-model speculation. This approach significantly improves throughput and concurrency compared to HiSparse and non-speculative adaptive serving, particularly at high context lengths (100K-500K) and concurrency levels.
Entities (8)
Relation Signals (8)
Adaptive DHiSparse → adapts → Residency
confidence 95% · Adaptive DHiSparse... adapts residency to load.
Adaptive DHiSparse → integrates → DSpark
confidence 95% · We introduce Adaptive DHiSparse, which brings DSpark and DFlash2 to hierarchical sparse KV serving
Adaptive DHiSparse → integrates → DFlash2
confidence 95% · We introduce Adaptive DHiSparse, which brings DSpark and DFlash2 to hierarchical sparse KV serving
HiSparse → storeshistoryin → Host Memory
confidence 95% · HiSparse stores history in host memory and fetches only the selected entries
Adaptive DHiSparse → supports → Speculative Decoding
confidence 95% · We present Adaptive DHiSparse, which supports speculative decoding over hierarchical sparse KV
GLM-5.3 → usedwith → DFlash2
confidence 95% · GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU
DeepSeek V4.1 Flash → usedwith → DSpark
confidence 95% · On DeepSeek-V4.1-Flash... Adaptive DHiSparse with DSpark delivers 1.85× the output
Adaptive DHiSparse → outperforms → HiSparse
confidence 90% · GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU, 1.45× HiSparse
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at every load and does not handle draft-model speculative decoding such as DSpark and DFlash2. We present Adaptive DHiSparse, which supports speculative decoding over hierarchical sparse KV and adapts residency to load. It protects the required working set, retains additional history while GPU memory is spare, and recovers the rest through exact fused retrieval. DSpark and DFlash2 keep their proposal and acceptance rules: queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification staging is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU, 1.45$\times$ HiSparse and 1.27$\times$ non-speculative adaptive serving. Held to the same latency bounds as HiSparse, it sustains up to 4$\times$ the concurrency and 2.3 to 3.7$\times$ the output, depending on the per-token bound. On DeepSeek-V4.1-Flash at 500K context and concurrency 512, Adaptive DHiSparse with DSpark delivers 1.85$\times$ the output of resident serving with DSpark. In a load step, optional residency reduces host fetches from 14.2% to 1.4% of selected rows at low load and releases memory under pressure.
Tags
Links
- Source: https://arxiv.org/abs/2610.13708v1
- Canonical: https://arxiv.org/abs/2610.13708v1
Trouble viewing inline? Open PDF directly →
Full Text
73,058 characters extracted from source content.
Expand or collapse full text
Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy Naji AlazharWassel Alazhar Umans AI research@umans.ai Abstract Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at ev- ery load and does not handle draft-model speculative decoding such as DSpark and DFlash2. We present Adaptive DHiSparse, which supports speculative de- coding over hierarchical sparse KV and adapts resi- dency to load. It protects the required working set, retains additional history while GPU memory is spare, and recovers the rest through exact fused retrieval. DSpark and DFlash2 keep their proposal and accep- tance rules: queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification stag- ing is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU, 1.45×HiSparse and 1.27×non-speculative adaptive serving. Held to the same latency bounds as HiSparse, it sustains up to 4×the concurrency and 2.3 to 3.7×the output, de- pending on the per-token bound. On DeepSeek-V4.1- Flash at 500K context and concurrency 512, Adaptive DHiSparse with DSpark delivers 1.85×the output of resident serving with DSpark. In a load step, optional residency reduces host fetches from 14.2% to 1.4% of selected rows at low load and releases memory under pressure. 1. Introduction Long-context agents and assistants retain large histo- ries while producing relatively short continuations; a coding request may carry material for several later de- cisions, leaving the cost of its KV state in place between decoding steps. Sparse attention reduces the entries read by each query, but conventional serving still re- serves the complete history in GPU high-bandwidth memory (HBM). As context length and the number of sessions increase, this storage can prevent further ad- mission before the hardware reaches its useful through- put (DeepSeek-AI, 2025, 2026; Xie et al., 2026). Reduc- ing the computation of a query therefore makes the placement of its history increasingly important. Hierarchical KV caching addresses this mismatch by retaining history in host memory and recovering only the entries selected by the model. HiSparse com- bines a bounded GPU cache, fused miss resolution, and prefetch for shared selections (Xie et al., 2026). A recently used row can remain in the cache and serve subsequent queries, exploiting locality while bound- ing each request’s device footprint. Its fixed budget imposes a deployment-level trade-off: a small working set permits more sessions but can incur avoidable host reads while HBM is available; a larger reservation re- duces retrieval but consumes admission capacity under pressure. A deployment with changing load encoun- ters both sides of this trade-off. Section 4.3.4 measures it. HiSparse also has no path for draft-model specula- tive decoding: it maintains no draft KV, resolves one query per request, and provides no multi-query verifi- cation staging. This matters because draft models can amortize target-model computation across several out- put tokens. DSpark selects request-dependent verifica- tion prefixes using confidence and engine cost, while DFlash2 verifies a configured block generated through its conditional selector (Cheng et al., 2026; Inco AI, 2026). Their benefits depend on both the return from verification and the memory available to sustain con- current requests. A hierarchy that supports only single- query decoding leaves this acceleration path unavail- able. Verification expands the immediate KV working set because several queries can select the same historical row while retaining separate causal boundaries and producing tentative state that may later be rejected. Resolving each query independently can repeat host reads; retaining every selection independently can re- quire substantially more staging than the hot cache. Draft KV introduces another persistent allocation, and keeping it in a full-history layout can consume the ca- pacity released by target offloading. The KV layer must therefore manage shared historical reads, temporary verification state, and the prefix retained after accep- tance as parts of the same execution path. We introduce Adaptive DHiSparse, which brings DSpark and DFlash2 to hierarchical sparse KV serv- ing and adapts history residency to the memory avail- able at each load. We call the same system without a 1 arXiv:2610.13708v1 [cs.AI] 9 Oct 2026 83264128 Offered concurrency (requests) 0 1,000 2,000 3,000 4,000 Generation throughput (tok/s) a 01,0002,0003,0004,000 Generation throughput (tok/s) 0.3 1 3 10 30 100 300 Mean time to first token (s) b 01,0002,0003,0004,000 Generation throughput (tok/s) 0 10 20 30 40 50 60 Mean time per output token (ms) c ResidentHiSparseAdaptive HiSparseResident + DFlash2Adaptive DHiSparse Figure 1 | GLM-5.3 at 32K input and 8K output cap, using the input and output lengths of the HiSparse evaluation. (a) Generation throughput against offered concurrency, (b) mean TTFT on a logarithmic axis and (c) mean TPOT against generation throughput. drafter Adaptive HiSparse. We protect a fixed work- ing set for exact recovery, retain additional immutable history while HBM is spare, and reclaim that history when other requests need its pages. The fixed hot-cache structure remains small as optional residency grows, allowing the resolver to exploit additional hits with- out scanning the full resident history. Under pressure, backup and reclamation make those pages available for admission while selected entries remain recoverable from host memory. This couples the allocation decision to the execution that uses it. We follow the reuse structure of each model: com- patible layers in DeepSeek-V4.1-Flash consume a shared global KV source, whereas GLM-5.3 shares sparse indices while retaining distinct layer payloads (DeepSeek-AI, 2026; Z.ai, 2026). Shared positions can consequently announce several future copies without making their bytes interchangeable. We exploit both forms of reuse and bound the additional speculative storage by its lifetime: the drafter keeps its existing visible window in a ring, while target layers reuse stag- ing after their consumers complete. Together, these choices let verification operate over the hierarchy while preserving proposal generation, sparse selections, and acceptance. We evaluate GLM-5.3 and DeepSeek-V4.1-Flash us- ing matched cache settings, reporting output together with first-token and per-token latency. The experi- ments cover fixed offered loads, a same-build load step, separate resolver measurements, and a sweep of the hot-cache budget with kernel profiles. They distin- guish the capacity of an allocation from the activity that serving actually sustains, and show where more active requests improve aggregate output while slowing in- dividual requests. Comparisons emphasize relative performance under the shared setup; absolute rates are in Table 2 and the figures. Our contributions are: •Speculative serving on hierarchical KV. At 100K context and offered concurrency 128 (C128), GLM- 5.3 with DFlash2 delivers 793 tokens/s per decode GPU, 1.45×HiSparse and 1.27×non-speculative adaptive serving, with mean time to first token (TTFT) 2.1×lower than HiSparse; held to the same latency bounds as HiSparse, it sustains up to 4×the concurrency and 2.3 to 3.7×the output, depending on the bound on time per output token (TPOT). • Residency that follows load. Optional history re- duces host-fetched selections from 14.2% to 1.4% at low load in the on/off experiment, yields its mem- ory under pressure, and repopulates as requests drain. •Bounded speculative state. A windowed draft ring and reusable verification staging add 0.060 GiB of required GPU state per GLM session while supporting the full configured verification block. Figure 1 gives an early view at 32K input and an 8K output cap, matching the input/output lengths of the HiSparse evaluation. There, Adaptive DHiSparse de- livers 1.7 to 1.9×the output of HiSparse from C8 to C64 and 1.45×at C128, where its mean TTFT is 2.3×lower. Resident serving with DFlash2 is 5 to 9% ahead up to C32, while every history still fits, and stops scaling at C64. Our main GLM experiments use 100K context, and the DeepSeek experiments use 500K context. 2. Background 2.1. Sparse Attention and KV Residency At query position푡, a sparse indexer assigns a score 퐼 푡,푠 to each historical KV entry푠and selects푘entries, denoted byS 푡 . The model’s sparse-attention opera- torAttncombines the query hidden state h 푡 with the selected stored rows c 푠 to produce u 푡 : S 푡 =푠 | 퐼 푡,푠 ∈ Top-k(퐼 푡,: ), u 푡 = Attn ( h 푡 ,c 푠 : 푠∈S 푡 ) . (1) The stored rows may be compressed latent representa- tions (DeepSeek-AI, 2025). We keepS 푡 as the ordered list the model’s top-푘returns, with its tie handling, causal masking and any additional local-attention branch, and write set operations on it when only mem- 2 bership matters. The indexer runs before the selected payload is read, providing an explicit demand signal to the memory system. Keeping selection state on the GPU lets that computation continue over the logical history while the larger attention payload is distributed across memory tiers. The resolver then supplies the announced rows before attention consumes them. An entry omitted fromS 푡 can appear in a later selec- tion, so sparse attention reduces what a query reads while the complete history remains logically available. HiSparse exploits this distinction by changing where entries are stored rather than which entries attention selects (Xie et al., 2026). LetR 푡 contain the entries ready on the GPU in the tail, hot cache, or optional history. The selected entries that require host recovery form D 푡 =S 푡 푡 .(2) Placement changesD 푡 while preservingS 푡 . Retaining additional history may reduce recovery, but consumes memory that could admit another request, and popu- lating that history also incurs work. 2.2. KV Residency Strategies Serving approaches differ in how much history they reserve on the GPU and how selected entries reach attention, with the resulting storage and retrieval costs determining the useful active batch. Full-history residency. Full-resident serving keeps ev- ery admitted request’s KV cache in HBM, giving each selected position an existing device location. For a source containing퐿historical rows, payload storage grows in proportion to퐿. Across concurrent requests, these histories can exhaust the available cache bud- get and cap the active decode batch (Xie et al., 2026). Paging reduces allocation waste, and source sharing removes duplicated state, but each distinct resident history still occupies device memory (Kwon et al., 2023; DeepSeek-AI, 2026). Fixed-cache offloading. HiSparse retains complete history in pinned host memory and gives each request a bounded GPU cache of퐵entries for each KV source (Xie et al., 2026). After the indexer selects positions, a fused kernel identifies hits, protects useful cached rows, chooses least-recently-used (LRU) victims, fetches miss- ing entries, and returns their device locations. Compact indexer state remains on the GPU, allowing the model to select across the full history while its larger KV pay- load uses the hierarchy. The cache preserves temporal locality: a selected en- try can remain in the hot set rather than being copied on every step. Increasing퐵retains more history, whereas decreasing it leaves room for additional requests. For models that share selections across layers, HiSparse also prefetches layer-local payload after an earlier in- dex producer announces the positions (Xie et al., 2026; Bai et al., 2026). The transfer can begin during interven- ing computation and complete before the consuming attention layer. A fixed cache keeps its configured size at every load: it fetches rows from host while memory is spare, and a larger reservation would cost admission capacity when load rises. This motivates separating the small replace- ment working set from additional, reclaimable history. Physical-source sharing. A KV source produces bytes consumed by one or more layers: different layers may select the same positions from distinct sources, whereas DeepSeek-V4.1-Flash shares global KV and indexer keys across compatible layers. A Full layer produces a source and a selection; a Reindex layer selects again us- ing shared KV; a Reuse layer consumes the latest appli- cable selection (DeepSeek-AI, 2026). A residency man- ager can consequently organize payload by request and physical source, while selection reuse determines when a location vector can also be shared. 2.3. Parallel Drafting and Verification Speculative decoding proposes several tokens for veri- fication in one target pass: DFlash conditions a parallel draft block on target features (Chen et al., 2026); DSpark adds a lightweight sequential head and confidence- scheduled verification prefixes (Cheng et al., 2026); DFlash2 uses a conditional selector over its candidate lattice (Inco AI, 2026). The target accepts a prefix and emits a correction or bonus according to the existing verification rule (Leviathan et al., 2023). These methods determine proposal and acceptance behavior; the KV layer supplies the historical and tentative state required to execute their queries. 3. Architecture We combine load-adaptive residency, residency-aware resolution, and speculative KV-state management around the selected set in equation(1): the model chooses positions, and our runtime supplies their de- vice locations. Figure 2 follows one decode step from selection to attention. We first describe the division between required storage and optional history, then how resolution and speculative execution use it. 3.1. Load-Adaptive Residency We reserve a hot cache, a mutable tail and metadata for each admitted request and KV source. With a drafter we also reserve verification staging (Section 3.4). The hot cache holds up to퐵recently useful rows; the tail holds newly computed state until it can become im- mutable history. Additional history occupies pages that can be returned when required allocations grow, letting a request retain many more than퐵rows while keeping the replacement structure small. Optional history occupies reclaimable capacity within the configured KV pools: unclaimed dedi- cated payload pages and eligible unused index back- ing. Let푀be the capacity of those pools. At time 휏, let푀 req (휏)be the memory held by required alloca- 3 129853177639940 12314056377979899 IndexerResolverAttention 112 3 top-k locations GPU HBM moves with load index stateoptional historyhot cachetail Host DRAM host backing selected rowin tailin optional historyin hot cachecopied from hostnot selectedreserve pagenot backed up Figure 2 | One decode step and where its KV lives. The indexer reads index state and emits the top-푘positions, colored here by where each row is found. The resolver first takes rows that are in the tail or in optional history (1), then scans the hot cache (2). A row found in none of them is copied from host DRAM into the hot cache (3) before attention reads the selected rows. Optional history holds imported history in reclaimable capacity of the KV pools, drawn schematically as one divider that moves with load. The hot cache and the tail are kept per request and KV source and are never lent. Host backup can be deferred while HBM is spare: a row without host backing stays protected on the GPU, so every missing row has host backing. A lookup skips steps 2 and 3 only when every selected row resolves in step 1. Row positions are illustrative. tions and푀 opt (휏)the memory held by optional history. Keeping an admission reserve 푀 res , we require 푀 req (휏)+ 푀 opt (휏)+ 푀 res ≤ 푀.(3) Required allocations and the reserve take priority; hot- cache and tail storage are never lent. Optional residency covers complete pages of a re- quest’s imported history. Rows generated during de- coding and the partial boundary page stay under the tail and hot-cache rules, and a row may be in the op- tional map and in the hot cache at once. Let푎 푟 be the number of optional pages held by request푟. When an admission leaves a deficit ofΔpages, the runtime re- claims whole requests: it walks the admitted requests in its reclaim order푟 1 , 푟 2 , . . .and takes the pages of the first 푞, with 푞= min n 푞 ′ : 푞 ′ ∑︁ 푖=1 푎 푟 푖 ≥Δ o .(4) The other requests keep their pages. If the optional pages do not coverΔ, the admission waits. When pages are free again, a bounded round-robin scheduler repop- ulates history, a limited number of pages per attempt. We track ready locations separately from allocated pages so attention uses rows only after their copies com- plete. A pending page remains unavailable through the optional map, while the hot-cache recovery path continues to serve selections. The fixed퐵therefore governs replacement work even when optional history is much larger. Increasing optional residency retains more possible future selections without enlarging the LRU scan, so allocation can respond to available capac- ity while execution retains the same bounded fallback during admission and partial refill. We reclaim pages only after their rows have host backing and earlier readers complete, then return them to required allocations. Population reverses this order: reserve eligible pages, copy or retain their bytes, and publish ready locations after completion. This ordering lets a request continue through partial coverage while protecting device-only rows until their host backup completes. Every row of a request’s history is therefore, at all times, ready on the GPU or has a host copy. On admission we reserve the request’s hot cache and keep its imported history on device while HBM is spare, deferring its host backup. We call this resident- first admission. Under pressure, we select whole re- quests until their optional pages cover the deficit, back up their history, and retire those pages once backup and earlier readers complete; the new request takes the freed capacity and the other requests keep their history on device. A reclaimed request then resolves its history through its hot cache, fetching misses from host. Completion releases a request’s capacity, which later population refills with history or leaves free. 3.2. Residency-Aware Resolution The fused resolver takes the model’s selected positions and returns their physical KV locations to attention. It 4 first locates rows in the tail and ready optional-history map (Figure 2, callout 1), then scans the bounded hot cache for unresolved positions, protecting selected rows and identifying unselected LRU victims (call- out 2). It recovers the remaining misses from pinned host memory and emits locations in selection order (callout 3). Hit detection, replacement, retrieval, and location generation share request-local intermediate state. The copy path launches over a grid that scales with the batch, from 4 to 32 thread blocks, so that row movement uses the available parallelism. Rows resolved in the first step (callout 1) never reach replacement or host recovery. In a census of selected rows on GLM-5.3 at C8, without speculation, 69.7% resolved directly, through the optional-history map or the tail, 30.3% through the hot cache and 0.05% from host. The hot scan remains bounded by퐵rather than the entire resident history. A lookup skips the hot-cache scan only when every selected row resolves through a ready mapping or the tail. Rows generated during decoding follow hot-cache rules, so keeping the whole imported history on device does not by itself make a lookup wholly direct (Section 4.3.3). We organize the cache by physical KV source so that a DeepSeek-V4.1-Flash Reuse consumer can inherit the preceding selected-location vector together with its ma- terialized payload. A Reindex consumer chooses a fresh top-푘from the shared source, reusing rows al- ready present and recovering any new misses. The cache update follows earlier attention readers, allowing several layers to share storage safely as their selections change. A shared source therefore carries one cache state across its consumers: rows fetched for an earlier selection remain available to later layers, and a fresh selection adds only the rows absent from that state. Reusing a selection also avoids another resolution call for the same location vector. GLM-5.3 instead has 21 index producers announcing positions for 78 distinct payload layers, allowing one selection to start several follower-layer transfers while each layer retains its own bytes. We use HiSparse’s exact layer-wise prefetch (Xie et al., 2026) with source- specific locations, reusing selection and transfer plan- ning through shared indices and reusing payload itself where the model shares KV. This distinction determines which consumers share a resolved vector and which need separate transient storage, whose size we bound by the simultaneously live layer group in Section 3.4. 3.3. Speculative Verification over the Hierarchy DSpark chooses a per-request prefix with its confidence scheduler, whereas DFlash2 uses a fixed block (Cheng et al., 2026; Inco AI, 2026). For request푟, we denote the number of target queries in a round by퐾 푟 , includ- ing the anchor, and the maximum supported width by퐾 max . Our KV path leaves proposals and accep- tance unchanged. Figure 3 shows how the common retrieval and state-management path handles the re- sulting block. Let푛 푟 be the number of token positions covered by committed target KV before the round. Query푗, with 0≤ 푗 < 퐾 푟 , sits at position푛 푟 + 푗;푗=0 is the anchor. Its visible token positions form V 푟,푗 = 푣 : 푣 < 푛 푟 ∪ 푛 푟 + 푢 : 0≤ 푢≤ 푗,(5) the committed history, served through the hierarchy, and the current block up to the query itself, read from the tail. We writeS 푟,푗 for the query’s selected rows; at token granularity,S 푟,푗 ⊆ V 푟,푗 . Compressed source rows must also be complete, as described below. We keep these query boundaries when resolving historical demand, even though all queries execute in one target pass. For a fixed request and payload source, letR 푟 be the rows ready on the GPU, as in equation(2), andH 푟 the rows with a complete host copy. The retrieval union U 푟 and the missing-row setD 푟 are U 푟 = 퐾 푟 −1 Ø 푗=0 S 푟,푗 , |U 푟 | ≤ 퐾 푟 푘, D 푟 =U 푟 푟 ⊆ H 푟 . (6) The inclusion is the invariant of Section 3.1: a row that is not ready on the GPU always has a host copy. We deduplicate these misses within the request and pay- load source, copy each distinct row once, and return its location to every selecting query. In Figure 3, callout 2, row a is selected three times and row b twice, yet each requires one host copy while attention retains each orderedS 푟,푗 and its causal mask. Newly computed rows and incomplete boundary state remain in the tail, where their visibility follows the individual query. A union can exceed the퐵-row hot cache even when each single-query selection fits. We reserve staging for these overflow rows and protect all selected lo- cations until their attention consumers finish. The bound퐾 max 푘per live request/source supplies space for disjoint selections; overlapping selections reduce the copies required within that allocation. Thus the config- ured hot-cache budget continues to preserve temporal locality while verification can materialize a larger im- mediate working set. Section 3.4 describes how layer groups reuse this staging. Write푥 푟,0 for the anchor and푥 푟,1 , . . . , 푥 푟,퐾 푟 −1 for the proposed tokens. The existing acceptance procedure (Leviathan et al., 2023) accepts퐴 푟 of the퐾 푟 −1 proposals, 0≤ 퐴 푟 ≤ 퐾 푟 −1, and emits a correction or bonus푏 푟 that starts the next round. The retained state and new committed frontier 푛 ′ 푟 satisfy new output=[푥 푟,1 , . . . , 푥 푟,퐴 푟 , 푏 푟 ], committed prefix=[푥 푟,0 , . . . , 푥 푟,퐴 푟 ], 푛 ′ 푟 = 푛 푟 + 1+ 퐴 푟 . (7) The next forward computes the target KV of푏 푟 . In Figure 3, callout 4, the target has computed D, E, F, and G; accepting E and F retains D–F, discards G, and emits G ∗ as the next anchor. 5 DDEFGDEFGG* a a a b b ab DEFG ab DE DrafterTarget model Resolver 1 2 3 4 next round top-k GPU HBM draft KV ring hot cachestaging arena tail Host DRAM host backing target tokendraft tokenrow on devicerow copied from hostrow written this round backup after verification Figure 3 | One speculative round over the KV hierarchy. The drafter, conditioned on target features, extends the anchor D into a block, and the target model verifies its퐾queries in one pass. Here E and F are accepted, G is rejected, and the correction G ∗ opens the next round. (1) Draft KV is held in a ring bounded by the drafter ’s window. (2) For each request and KV source, the resolver handles the top-푘selections of the퐾queries together, so each distinct missing row is copied from host once (a is selected by three queries, b by two), and each query still attends over its own selection. (3) Rows that do not fit the hot cache go to a staging arena, reused across layer groups after its readers finish and promoted rows are copied back. (4) The target writes the rows of all퐾queries to the tail. Acceptance commits the computed rows of the anchor and the accepted drafts; the rejected row is dropped and G ∗ has no row until the next forward. Host backup of the committed rows follows verification. Sizes are illustrative (퐾= 4, 푘= 3). For a source that packs푚consecutive token positions into each stored row (푚=1 without compression), the source-row indices 푠 completed by the round satisfy ⌊ 푛 푟 /푚 ⌋ ≤ 푠 < 푛 ′ 푟 /푚 .(8) These rows join the immutable history, and their host backup is scheduled after verification; any remain- ing incomplete group stays in the tail until it is com- plete. Committed rows follow the accepted prefix, re- jected and unverified rows are never published, and the drafter’s state advances with the same prefix. This interface separates how many queries the drafter asks the target to verify from how much history those queries fetch. DSpark can change퐾 푟 across re- quests and rounds, while DFlash2 uses its configured width; both retain their original verification decisions. The storage system handles the resulting union and ac- cepted state, including repetitions of a selected position across queries. 3.4. Bounded Speculative Storage Speculation adds persistent draft KV and temporary target-verification state, allocations that can become the next concurrency constraint once the target history is offloaded. We size the draft by its visible window and the target staging by the layer payloads that are simul- taneously live, preserving the information consumed by the models while limiting its physical footprint. A drafter that reads at most푊prior positions needs storage for that window and its current block of at most 퐾 max positions. With푝rows per page, alignment adds at most푝−1 positions. We allocate a ring of푅 푑 rows per request, where 푅 푑 = 푝 푊+ 푝− 1+ 퐾 max 푝 .(9) This covers the visible interval when previous read- ers complete before overwrite. For request slot푖, the physical ring address 휙(푖, 푡) of draft position 푡 is 휙(푖, 푡)= 푖푅 푑 +(푡 mod 푅 푑 ).(10) Two live positions of one request differ by less than푅 푑 , so they never share a row, and slots occupy disjoint ranges. Figure 3, callout 1, places this bounded draft state in HBM. Logical positions advance through the ring while the drafter continues to read the same suffix, making its KV allocation independent of the length of the target’s full context. In GLM-5.3/DFlash2,푊=2,048,푝=64, and 퐾 max =8 give푅 푑 =2,176, a page-rounded size that also covers the six- and four-query blocks. Six draft transformer layers have twelve K/V allocations total- 6 Table 1 | Memory per session for GLM-5.3, in GiB. Re- quired GPU memory at 32K and 100K context, and the full host backup at 100K. Model weights and shared workspaces are excluded.†excludes optional residency, which only uses HBM that is spare. GPU, requiredHost backup Method32K100K100K Resident1.4564.444– HiSparse0.2590.4324.186 Adaptive HiSparse0.259 † 0.432 † 4.186 Resident + DFlash21.4814.469– Adaptive DHiSparse0.319 † 0.492 † 4.186 ing 12,288 bytes per row. For 35 request slots they oc- cupy about 0.87 GiB per GPU. The target can continue selecting historical rows far outside the draft window through the KV hierarchy, while the drafter retains the suffix and target features it already uses. Target verification instead reuses temporary storage, with overflow rows in Figure 3, callout 3, occupying staging separate from the hot cache. Layers that share selected positions may still read different payloads, so each simultaneously active source receives a distinct arena. After all consumers of a layer group complete, the next group can reuse those arenas. Promoted rows are copied back to their stable hot-cache addresses be- fore an arena is released. With휒payload sources live at once,푁request slots per GPU, and푑 KV bytes per stored KV row, the staging reservation is 푀 stage = 휒푁퐾 max 푘 푑 KV .(11) The reservation푀 stage covers the maximum verification width, with allocator rounding, maps, and persistent cache state accounted separately, while each round uses locations selected by its live queries. GLM-5.3 processes its index-sharing layer groups sequentially, with at most four payload layers live to- gether. Reusing four arenas instead of dedicating one to each of 78 layers divides this transient term by 19.5, from 24.0 to 1.23 GiB per GPU at 35 slots. The win- dowed ring and reusable staging together add 0.060 GiB of required GPU state per session in the evaluated layout. Copy-back remains part of execution cost. Ta- ble 1 gives the per-session memory: at 100K Adaptive HiSparse requires 0.432 GiB of GPU memory against 4.444 GiB for resident serving, 10.3×less; with DFlash2 the two figures are 0.492 and 4.469 GiB. Capacity accounting. Speculation also adds fixed memory on each decode GPU, outside the per-session figures of Table 1: the DFlash2 drafter takes 4.9 GiB, the target model 1.3 GiB more, and verification metadata 0.4 GiB. These allocations reduce the memory left for the KV and index pools. The index pool then holds 3.72 million rows per GPU against 5.37 million, enough for 36 complete histories of 100K input and 2,048 output tokens against 52. We configure 35, which gives 140 sessions on four GPUs against 208. 3.5. Exactness Our hierarchy changes where KV rows are stored, while each query receives the model’s ordered selec- tion, payload and scales, and causal mask. Sharing a retrieval reuses the copy rather than merging attention inputs. For the payload source at hand,휋 푟 (푠)is the GPU location the resolver returns for row푠of request 푟; a row held in two places has one of them chosen. If 푠 푟,푗,푖 is the푖-th entry in query푗’s ordered selection, the resolver returns a location ℓ 푟,푗,푖 satisfying ℓ 푟,푗,푖 = 휋 푟 (푠 푟,푗,푖 ),KV[ℓ 푟,푗,푖 ]= c 푟, 푠 푟,푗,푖 .(12) HereKV[ℓ]is the GPU payload at locationℓ, and c 푟,푠 is the original KV row푠of request푟. Each query receives its own ordered locations; queries selecting the same row share its location. We preserve target features, proposal probabilities, acceptance, and accepted-prefix updates, and make committed rows recoverable before reusing their device storage. Given equal model state and numerical execution, these properties preserve at- tention inputs and the verifier ’s output law (Leviathan et al., 2023). The ring retains the drafter’s visible win- dow, and staging is reused only after its readers and required copy-back complete. These are the design conditions for exact placement. 4. Experiments We evaluate GLM-5.3 and DeepSeek-V4.1-Flash serv- ing, then examine how optional residency responds to load, how verification width affects output, what a miss costs, and how large the hot cache should be. 4.1. Experimental Setup Hardware and workloads. Experiments use 8×B300 nodes (269 GiB of HBM per GPU, 4.3 TB of host mem- ory over PCIe). GLM-5.3 uses four GPUs for prefill and four for decode at 100K context, while DeepSeek-V4.1- Flash uses two per role at 500K. We evaluate end-to-end serving with the closed-loop concurrency benchmark used by HiSparse (Xie et al., 2026), sweeping offered concurrency퐶at a fixed input length and output cap. GLM-5.3 runs use 100K input and a 2,048-token output cap; Figure 1 uses HiSparse’s 32K input and 8K out- put. DeepSeek-V4.1-Flash runs use 500K input and a 2,048-token output cap. The 100K and 500K workloads reuse prepared prefixes, one for GLM-5.3 and 16 for DeepSeek-V4.1-Flash, with independent decode KV per request. We report output throughput per decode GPU, time to first token (TTFT) and time per output token (TPOT). Configurations. Wecompareresidentserving, HiSparse, and our adaptive configurations in SGLang (Zheng et al., 2024; Xie et al., 2026), with DSpark or DFlash2 where supported. For both models, HiSparse and the adaptive configurations use the same model- computation kernels.All adaptive configurations add the KV management of Section 3; Section 4.2.1 attributes the gain over HiSparse. GLM offloading 7 03264128256 Offered concurrency (requests) 0 200 400 600 800 Output throughput (tok/s/GPU) a 0200400600800 Output throughput (tok/s/GPU) 0.3 1 3 10 30 100 300 Mean time to first token (s) b 0200400600800 Output throughput (tok/s/GPU) 0 10 20 30 40 50 60 Mean time per output token (ms) c ResidentHiSparseAdaptive HiSparseResident + DFlash2Adaptive DHiSparse Figure 4 | GLM-5.3 serving at 100K context. (a) Output throughput against offered concurrency, (b) mean TTFT and (c) mean TPOT against output throughput. Color gives the KV policy and filled markers add DFlash2. TTFT in (b) is on a logarithmic axis. Adaptive DHiSparse uses static blocks of 8 at C4 to C32, 6 at C64 and 4 at C128 and C256. configurations use FP8 KV,푘=2,048, and퐵=4,096, with cache settings matched across methods (SGLang contributors, 2026); Section 4.3.4 sweeps퐵in this configuration: at C128 output peaks at퐵=2푘for both systems. DeepSeek uses our optimized HiSparse port as its reference, with푘=512 and hot caches of 8,192 rows for the KV source that five layers select from and 4,096 rows for each of the three sources that one layer selects from. Figure 4 specifies the DFlash2 block schedule; Figure 7 shows the block-size comparison. Measurement. Output throughput counts returned tokens per second per decode GPU, excluding prefill GPUs; Figure 1 reports aggregate output across four decode GPUs. Post-wave latency excludes the first퐶 submitted requests. 4.2. Experimental Results 4.2.1. GLM-5.3 at 100K context Figure 4 shows the serving curves and Table 2 gives the C128 and C256 results. Where the 1.45×comes from. At C128, Adaptive DHiSparse delivers 793 tokens/s per decode GPU, 1.45×the output of HiSparse. The gain is the product of two steps. The first, from HiSparse to non-speculative adaptive serving, is 1.14×. The copy path launches over a grid that scales with the batch, 4 to 32 thread blocks against a fixed four, exposing more parallelism for row movement. Mean time per token is 7% lower. Resident- first admission places incoming history directly in GPU storage and defers its host backup until memory pres- sure, which keeps that backup off the admission path: the prefill/KV handoff falls from 10.4 to 4.0 seconds, for 1.8 seconds of added allocation wait. In a closed loop a shorter wait keeps more of the 128 requests active, 126 against 121. The second step, speculation, is 1.27×. A round verifies four queries and emits 2.88 tokens in 2.2×the time of a non-speculative step, so each session runs 1.30× faster while 99% of the sessions stay active. The gain follows the tokens a round emits: with 2.67 tokens per round instead of 2.88, speculation delivered 1.13 to 1.16×non-speculative adaptive serving instead of 1.27×, and 1.29 to 1.31×HiSparse instead of 1.45×. Mean TTFT ends 2.1×lower than HiSparse and mean TPOT 29% lower. Why speculation stops paying at C256. With푏active sessions, output is proportional to푏/TPOT, so the out- put푄of speculative serving (푠) against non-speculative adaptive serving (0) is 푄 푠 푄 0 ≈ 푏 푠 푏 0 · TPOT 0 TPOT 푠 .(13) Speculation pays while it keeps a larger share of the sessions than its TPOT ratio. At C128 it keeps 99% of the sessions at 0.77 of the TPOT, and the relation gives 1.29×against 1.27×measured. At C256 the speculative configuration admits 140 sessions against 208. It keeps 69% of the sessions at 0.71 of the TPOT, which gives 0.96×against 1.00×measured (826 and 823 tokens/s per decode GPU). Speculation then serves fewer sessions, each faster, for the same output, and re- quests wait longer for a slot: mean TTFT is 1.9×higher. Both adaptive configurations remain about 13% above HiSparse. Why the gap to resident serving grows with load. Resident serving requires 4.444 GiB of GPU memory per session at 100K, against 0.432 GiB for Adaptive HiSparse (Table 1). The memory of 52 offloaded ses- sions on a GPU therefore holds five resident ones, 20 on four GPUs. With DFlash2 the memory of its 35 ses- sions holds three, 12 on four GPUs. This is where the resident curves stop scaling: output stays near 178 to- kens/s per decode GPU from C32 to C256, and near 220 with DFlash2 from C16, while mean TTFT without DFlash2 grows from 0.6 seconds at C16 to 30 seconds at C32. At C256, Adaptive DHiSparse delivers 4.6× the output of resident serving and 3.7×that of resident serving with DFlash2. Below the limit, resident serving has no rows to resolve and stays ahead: at C4 to C16, Adaptive HiSparse is 5 to 7% below it and 19 to 23% above HiSparse. A kernel profile at C8 records a shorter step with Adaptive HiSparse (16.6 versus 20.9 ms) and 8 Table 2 | GLM-5.3 at 100K context, offered concurrency 128 and 256. Output is in tokens/s per GPU. Post-wave latency excludes the first퐶submitted requests. Active is the mean number of requests decoding at once over the central half of a run. AgainstTTFT (s)TPOTActive 퐶MethodOutputHiSparsemeanpost-wavepost-wave p99(ms)/ slots 128HiSparse548–10.685.1126.9750.3121.1 / 208 Adaptive HiSparse6221.14×6.132.4811.8546.6125.6 / 208 Adaptive DHiSparse7931.45×5.132.259.6735.9124.7 / 140 256HiSparse730–47.6142.54102.8958.0190.5 / 208 Adaptive HiSparse8231.13×34.3631.0181.9954.4201.5 / 208 Adaptive DHiSparse8261.13×64.6267.7987.6038.9138.2 / 140 1.4 ms less cache-kernel work. In the same-build exper- iment of Section 4.3.1, optional residency alone makes each session 3.8% faster. At high load, where optional residency disappears, resident-first admission and the copy path remain active. Where first-token time goes. Table A2 splits mean TTFT into allocation wait and handoff.At C128, the handoff is 97% of HiSparse’s TTFT. With non- speculative adaptive serving it is 66%, and with DFlash2 43%. Below the session limit the handoff is thus the larger part for HiSparse and Adaptive HiSparse, and the adaptive configurations shorten it by 2.6×and 4.6×. With DFlash2, allocation wait, 49% of mean TTFT at C128, is already the larger part. At C256 the bottleneck moves to admission: allocation wait is 85 to 97% of mean TTFT for the adaptive configura- tions and 67% for HiSparse. Speculative TTFT stays as high after the first wave of requests, so the pressure is sustained. Capacity under fixed latency bounds. Table A3 lists latency at every load. With post-wave p99 TTFT at most 10 s and mean TPOT at most 40 ms, HiSparse sustains C32 (212 tokens/s per decode GPU), Adap- tive HiSparse C64 (393), and Adaptive DHiSparse C128 (793): 4×the concurrency and 3.7×the output of HiSparse. The TPOT bound is what holds HiSparse at C32: it reaches 42 ms at C64, while speculation holds 36 ms at C128. With the bound at 50 ms, HiSparse reaches C64 (348) and the output ratio is 2.3×. The ratios also depend on the first-token bound: Adaptive DHiSparse meets it at C128 with 9.67 s, and with a bound of 8 s it sustains C64, 2×the concurrency and 2.6× the output of HiSparse. 4.2.2. DeepSeek-V4.1-Flash at 500K context Figure 5 repeats the comparison on a second model, at five times the context and with DSpark as the drafter. The same limit, at a higher load. Resident serving stops scaling at C128: its output stays near 1,800 to- kens/s per decode GPU up to C512 while mean TTFT grows from 11 to 115 seconds. Adaptive serving keeps scaling, from 1.16×resident at C128 to 2.3×at C512 (4,217 tokens/s per decode GPU), where its mean TTFT is 5.6×lower. Against HiSparse it is within 4% up to C256 and 9.4% higher at C512. Context length sets the crossover. At C512, adaptive output is 2.6% below resident at 32,736 tokens, 1.57× resident at 200K and 2.3×at 500K. At the shortest con- text every history fits in HBM and resident serving has nothing to resolve. As the history grows, fewer resident sessions fit and the hierarchy takes over. Speculation follows the same pattern. With DSpark, adaptive output is 1.85×resident serving with DSpark at C512 (5,054 tokens/s per decode GPU), with mean TTFT 3.2×lower. TPOT is 1.9×higher, because each request shares a step with many more active sessions. The return of speculation over non-speculative adap- tive serving falls as load rises, from 1.54×at C128 to 1.20× at C512, as on GLM-5.3 above C32. 4.3. Experimental Analysis 4.3.1. What residency buys and what it costs On GLM-5.3 at 100K context, without speculation, the same build runs with optional residency on and off under a C16–C256–C16 sequence, with five minutes at each offered load (Figure 6, Table A1). At low load. With residency on, 75 to 100% of the eligible history is on the GPU, and the share of selected rows fetched from host is 10.3×lower. In this first phase per-session speed is 3.8% higher and time to first token is 1.65 seconds against 3.22. When load rises. The pages have to be returned before new requests can use them. The runtime backs up and retires the resident history: coverage falls below 1% within 16 seconds, and the 208 slots fill after 53 seconds, against 3 seconds with residency off. In the C256 phase, allocation wait is 1.4×longer and per-session speed 3% lower, with the same host-fetched share in both arms. The prefill/KV handoff takes 8.5 seconds against 26.8, so time to first token in the phase is still 21% lower, and 11% more requests complete. When load falls. In-flight requests drain first, then population refills the freed pages. Coverage is back above 50% after 142 seconds. The host-fetched share is again 5.8×lower and per-session speed 13.1% higher, while time to first token is the same in both arms (0.84 and 0.87 seconds). While memory is spare, optional residency thus re- moves 83 to 90% of host fetches, and sessions run 3.8% faster before the step and 13.1% faster after recovery. It costs 3% of per-session speed and slower slot filling for the minute that follows a load step. 9 064128256512 Offered concurrency (requests) 0 1,500 3,000 4,500 Output throughput (tok/s/GPU) a 01,5003,0004,500 Output throughput (tok/s/GPU) 0.5 1 2 5 10 20 50 100 Mean time to first token (s) b 01,5003,0004,500 Output throughput (tok/s/GPU) 0 10 20 30 40 Mean time per output token (ms) c ResidentHiSparseAdaptive HiSparseResident + DSparkAdaptive DHiSparse Figure 5 | DeepSeek-V4.1-Flash serving at 500K context. Panels and encoding as in Figure 4; filled markers add DSpark. HiSparse is our optimized port to DeepSeek-V4.1-Flash. 051015 Time (minutes) 0 50 100 150 200 Allocated request slots C16C256C16 a 051015 Time (minutes) 0 25 50 75 100 Ready history coverage (%) C16C256C16 b 051015 Time (minutes) 0 5 10 15 Selections fetched from host (%) C16C256C16 c Optional residency onOptional residency off Figure 6 | Residency under a load step. GLM-5.3 at 100K context, without speculation: the same build with optional residency on and off, offered C16, then C256, then C16 for five minutes each; in-flight requests drain when the load falls. (a) Allocated request slots. (b) Coverage: ready optional history over eligible history. (c) Share of selected rows fetched from host. One-second counters shown in five-second bins. Per-phase request statistics are in Table A1. 4.3.2. What a wider verification block returns Figure 7 compares static DFlash2 blocks of 4, 6 and 8 tar- get queries at C64, C128 and C256. A wider block emits more tokens per round and takes longer per round. At C64 the two cancel: block 6 emits 18% more tokens than block 4 in a round that is 17% longer (Figure 11a), and the three blocks are within 3% in output. At higher load the wider blocks delivered less. At C128, blocks of 6 and 8 emit 17% and 25% more tokens per round than block 4 but deliver 6% and 9% less output, and at C256 they deliver 1% and 8% less. At these three loads the serving curve uses the block with the highest output, 6 at C64 and 4 at C128 and C256. Resolution and copying run alone for 17% of the block-4 round and 20% of the block-6 round. Shared retrieval keeps that share down: at C128 with four queries, verification counters show 1.41 missing se- lections per distinct host-fetched row, so fetching each row once removes 29% of the fetches the queries would otherwise issue. 4.3.3. What a miss costs Figure 8 times the resolver apart from the model. Time follows bytes. On GLM-5.3 (panel a), at verifica- tion width 8, batch 32 per GPU and퐵=4,096, resolver time is linear in the unique host-miss payload of a call. Six cases, from 0.031 ms with no miss to 6.23 ms for 302 MB, fit a rate of 49.4 GB/s (푅 2 =0.9992). Eight disjoint selections cost 7.9×eight identical ones. On GLM-5.3 the cost of resolution is the bytes it moves, and the two mechanisms that remove fetches act on it directly: optional residency, with 83 to 90% fewer host-fetched rows at low load, and shared retrieval, with 29% fewer during verification. The scan grows with the hot cache. On DeepSeek- V4.1-Flash (panel b), a hot cache four times larger makes a call 1.9×slower at batch 8 and 2.2×slower at batch 160 with no miss at all, and at퐵=4,096 the direct path is 2.5×faster than the scan. Optional history adds rows without lengthening the scan. The scan is a small term in serving: on DeepSeek-V4.1-Flash at C128 the whole resolver takes 0.5 to 0.7 ms of a 25 ms step and does not grow with the budget, and on GLM-5.3 at C8 resolver work grows by 0.5 ms per step from퐵= 푘 to 8푘(Section 4.3.4). Misses add in proportion to the batch (panel c): 77 misses per request add a median of 1.3휇s per request at both batch sizes, which makes a 10 468 Block size (target queries per round) 2.6 2.9 3.2 3.5 3.8 Emitted tokens per round a 468 Block size (target queries per round) 500 600 700 800 Output throughput (tok/s/GPU) b C64C128C256 Figure 7 | Block size for DFlash2 at 100K context. (a) Emitted tokens per speculative round and (b) output throughput for static blocks of 4, 6 and 8 target queries at three offered loads. 0100200300 Unique host-miss payload (MB per call) 0 2 4 6 Resolver time (ms per call) shared partial disjoint 49.4 GB/s a GLM-5.3, width 8, batch 32 cold cache retained cache 2,0484,0968,192 Hot-cache size B (rows) 0 20 40 60 Resolver time (μs per call) direct path, batch 8 b DeepSeek-V4.1-Flash, no host miss batch 8 batch 160 batch 8batch 160 0 100 200 300 400 Resolver time (μs per call) 18.7 29.3 36.9 243.3 c DeepSeek-V4.1-Flash, B = 4,096 no host miss 77 host misses Figure 8 | What resolution costs. (a) GLM-5.3, verification width 8, batch 32 per GPU,퐵=4,096: resolver time against the unique host-miss payload of a call, for synthetic shared, partial and disjoint selections across the eight queries, with a cold or a retained cache. The dashed line is the least-squares fit, 49.4 GB/s (푅 2 =0.9992). One query’s top-푘over the 32 requests of a call is 37.75 MB, so disjoint selections fetch eight such payloads and shared selections one. (b) DeepSeek-V4.1-Flash at 500K, single-query resolver with no host miss: time against hot-cache size for two batch sizes. The dotted line is the direct path at batch 8. (c) The same resolver at퐵=4,096 with and without 77 host misses per request. Error bars span the minimum and maximum of nine timings in (a) and the 10th to 90th percentiles in (b) and (c). These are isolated kernel measurements with synthetic selections and real payload copies. call 1.6×slower at batch 8 and 6.6×slower at batch 160, because the scan grows far less than the batch. Share of a step. On DeepSeek-V4.1-Flash, resolver kernel work is 0.4 to 1.4% of the step interval at C16 and 4.75% at C512 (Figure 11b). In a GLM-5.3 round with DFlash2 at C64, resolution and copying are the only kernels running for 17 to 20% of the time (Fig- ure 11a). Figure 10 compares the two models at one load, without speculation. In the census of Section 3.2, about 70% of selected rows resolved directly and 30% through the hot cache. Rows generated during decod- ing go through the hot cache, so no lookup was wholly direct. 4.3.4. How large the hot cache should be We sweep the hot-cache budget from퐵= 푘to 8푘on GLM-5.3 and from 2푘to 16푘on DeepSeek-V4.1-Flash, at C8 and C128, without speculation (Figure 9). On GLM-5.3 the budget trades step time against ses- sions. Here the sweep changes only퐵: the KV pools keep the size configured for퐵=2푘and every other setting is as in Section 4.1, so a larger hot cache leaves room for fewer sessions. At C128, HiSparse’s host-miss rate falls from 33% at퐵= 푘to 3.9% at퐵=8푘, and its TPOT from 57 to 38 ms. Output nevertheless peaks at퐵=2푘and then collapses: from 2푘to 8푘it loses 44%, and mean TTFT grows from 11 to 115 seconds. Adaptive HiSparse follows the same curve, 14% higher at the peak, and its profile shows the trade: from 2푘 to 8푘a decode step shortens from 44.7 to 35.3 ms but holds 13 sessions per GPU instead of 28. A lower miss rate does not buy throughput once the cache takes the memory that sessions need. In these pools,퐵=2푘, the setting of our GLM-5.3 comparisons, gives the highest output of the four budgets for both systems. At C8 the picture changes. With optional residency the host-miss rate is 0.05% at every budget and output stays within 3%, so the smallest cache is enough. It is also the fastest: with the miss rate unchanged, resolver work per step grows from 0.35 to 0.85 ms between퐵= 푘and 8푘, and 11 1248 0 40 80 120 Output throughput (tok/s/GPU) a GLM-5.3, C8 1248 0 200 400 600 b GLM-5.3, C128 24816 0 100 200 300 400 c DeepSeek, C8 24816 0 800 1,600 2,400 d DeepSeek, C128 1248 Hot-cache budget B/k 0 10 20 30 40 Fetched from host (% of rows) e 1248 Hot-cache budget B/k 0 10 20 30 40 f 24816 Hot-cache budget B/k 0 10 20 30 40 g 24816 Hot-cache budget B/k 0 10 20 30 40 h HiSparseAdaptive HiSparse Figure 9 | Output and host fetches against the hot-cache budget. (a–d) Output per decode GPU and (e–h) share of selected rows fetched from host, for HiSparse and Adaptive HiSparse as퐵/푘varies, without speculation. GLM-5.3 runs at 100K context on one prepared prefix, and DeepSeek-V4.1-Flash at 500K on 16 prepared prefixes. Table 3 | Retrieval volume of the two models. Per generated token and request. Ranges run from the smallest to the largest budget of the sweep, for HiSparse at C128. The hot cache is that of Adaptive HiSparse at the budgets of Section 4.1. GLM-5.3DeepSeek Selecting layers× 푘21× 2,0488× 512 Selections per token43,0084,096 KV sources78, one per layer4, shared Bytes per missed selection2.4 to 2.5 KB288 B Host bytes per token34 to 4.2 MB 0.42 to 0.11 MB Hot cache per session184 MB6 MB Resolver calls per step218 Follower copies per step57none TPOT from 18.9 to 19.4 ms. Best against best, Adap- tive HiSparse stays 21% ahead of HiSparse, the gap analyzed in Section 4.2.1. On DeepSeek-V4.1-Flash the budget does not matter. At C128, output stays within 2% across an eightfold range of퐵for both systems, while the host-miss rate falls from 36% to 9.5% for HiSparse and from 18% to 5.9% for Adaptive HiSparse. Mean TPOT and mean TTFT are flat as well, the latter at 2.5 to 2.8 seconds. At C8, Adaptive HiSparse fetches nothing from host at any budget, and TPOT shows no trend with the budget for either system; the lower HiSparse output at퐵=2푘 comes from first-token time. Why the two models differ. Table 3 gives the rea- son. For each generated token, GLM-5.3 makes 43,008 selections through 21 index producers, and each selec- tion feeds payload layers that keep their own bytes. DeepSeek-V4.1-Flash makes 4,096 selections through eight selecting layers that read four KV sources shared across its layers. A missed selection moves 288 bytes on DeepSeek-V4.1-Flash and about 2.4 KB on GLM-5.3, where it carries the rows of 3.7 payload layers on aver- age. At the smallest budget, where both models miss about a third of their selections, HiSparse fetches 34 MB per generated token on GLM-5.3 and 0.42 MB on DeepSeek-V4.1-Flash, about 80×less. The DeepSeek- V4.1-Flash hot cache is also about 6 MB per session at the budgets of Section 4.1, against 184 MB, so enlarging it costs no sessions. What the cache costs inside a step. Figure 10 profiles the kernels of Adaptive HiSparse at C128. On GLM-5.3 a step makes 21 resolver calls and 57 follower-layer copies. At퐵=2푘they take 15.1 ms of kernel work, three quarters of it in the follower copies, and both shrink as the budget grows (panel a); per session, the work falls from 0.55 ms at 2푘to 0.29 ms at 4푘and 0.27 ms at 8푘. Most of this work runs under the model’s own kernels: 11.0 of the 15.1 ms overlap them, and the cache runs alone for 9% of the step (panel b). On DeepSeek-V4.1-Flash a step makes eight resolver calls and no follower copy. The resolver takes 0.5 to 0.7 ms of a 25 ms step at every budget, and the residency machinery itself (host backup and page bookkeeping) adds 0.05 ms. Expert matrix products run alone for 38% of the GLM-5.3 step and 53% of the DeepSeek- V4.1-Flash step. 5. Discussion When resident serving is enough. If every active his- tory fits in HBM, resident serving has nothing to re- solve: Adaptive HiSparse is 5 to 7% below it at C4 to C16 on GLM-5.3, and 2.6% below it on DeepSeek-V4.1- Flash at 32K. The hierarchy pays once resident histories stop fitting, at about 20 sessions for 100K on our GPUs 12 124824816 Hot-cache budget B/k 0 10 20 30 Cache kernel work (ms per step) 25.4 15.1 7.1 3.4 0.7 0.5 0.5 0.5 GLM-5.3DeepSeek a Cache work against the budget, C128 follower-layer copy resolver 0255075100 Share of step time (%) GLM-5.3 44.7 ms DeepSeek 25.1 ms b Where a C128 step goes 38768309 53108178 expert matrix products dense projections indexing and attention other model kernels several families at once cache kernels alone no kernel running Figure 10 | What the cache costs inside a step, Adaptive HiSparse at C128. (a) Summed duration of the cache kernels per GPU step as퐵/푘varies, on one scale for both models: the fused resolver and, on GLM-5.3, the copies that load selected rows into the follower payload layers. A GLM-5.3 step holds 24 to 28 sessions per GPU up to퐵/푘=4 and 13 at퐵/푘=8; a DeepSeek-V4.1-Flash step holds 62 to 63. (b) Share of step time by kernel family at the budget used in the paper (퐵/푘=2 for GLM-5.3, 16 for DeepSeek-V4.1-Flash), each family counted where it runs alone. “Several families at once” is time in which kernels of more than one family run; “no kernel running” is time between kernels, 1.5% and 1.9% of the step. and 12 with DFlash2 (Table 1), and its advantage then grows with both load and context length. How to operate speculation. Equation(13)says when speculation pays: while the share of sessions it keeps exceeds its TPOT ratio, 0.77 at C128 and 0.71 at C256 in our runs. Once both configurations fill their slots, the share kept is the slot ratio, 140/208=0.67, below the 0.71 measured at C256, so speculation adds no output there and lengthens the wait for a slot. Our schedule narrows the block as load approaches that limit (8 up to C32, 6 at C64, 4 from C128); selecting the block online from load and memory is future work. What limits capacity next. With KV offloaded, GPU memory is still the binding resource. A session requires 0.432 GiB at 100K, or 0.492 GiB with DFlash2, and spec- ulation adds 6.6 GiB of fixed allocations per GPU for the drafter, the target model and metadata; the con- figurations admit 208 and 140 sessions. Host memory is far from binding: 208 sessions back up 0.93 TB of a 4.3 TB host, which would hold about 950. At C256, allocation wait is 85 to 97% of time to first token, and after a load step slots take 53 seconds to fill. The next gains in capacity will come from a lower required state per session and faster reclaim. Implications for model design. The cost of offloading follows the bytes a token’s selections can pull from host. Against GLM-5.3, DeepSeek-V4.1-Flash has a푘 four times smaller, 2.6×fewer selecting layers and rows about half as wide, and a selection reads one source row where GLM-5.3 reads 3.7 layer rows. Together these give about 80×fewer host bytes per token at a comparable miss rate. A DeepSeek-V4.1-Flash step needs eight resolver calls and no follower copy, and its output is insensitive to the cache budget. GLM- 5.3 keeps 78 payload layers with their own bytes: a step needs 57 follower copies, and output falls 44% when the budget grows from 2푘to 8푘in pools sized for 2푘(Section 4.3.4). A drafter with a bounded window makes draft KV independent of context: the ring adds 0.025 GiB per session at 32K and at 100K alike. Scope. Our evaluation covers one GPU platform, two sparse-attention models, and closed-loop workloads at fixed input lengths over a fixed set of prepared prefixes, with cache settings matched across methods. Specula- tive gains depend on acceptance, which varied between runs of the same workload. 6. Related Work Hierarchical KV caching. HiSparse establishes LRU- managed hierarchical storage, fused resolution with GPU-assisted host reads, and exact layer-wise prefetch from shared indices (Xie et al., 2026), keeping com- plete history available through a fixed device cache. HiSparse has no draft-model speculative path; we ex- tend it with load-adaptive residency, shared verifica- tion retrieval, bounded draft KV, and reusable staging. Strata provides complementary hierarchical context- loading and GPU-assisted I/O mechanisms (Xie et al., 2025). InfiniGen and ShadowKV use prediction or recon- struction for offloaded execution, and DualDecoder predicts future retrieval (Lee et al., 2024; Sun et al., 2025; Liang et al., 2026). Model-side reuse. Cross-Layer Attention and YOCO share KV state, while IndexCache reuses sparse selec- tions (Brandon et al., 2024; Sun et al., 2024b; Bai et al., 2026). DeepSeek-V4.1-Flash combines global-KV and index reuse (DeepSeek-AI, 2026); GLM-5.3 illustrates shared indices with distinct payloads (Z.ai, 2026). We organize storage around those model-defined relation- ships, sharing recovered bytes where the model shares a source and preserving distinct bytes where it shares only positions. 13 Drafting and verification. Speculative decoding has moved from separate draft models (Leviathan et al., 2023) to drafters that reuse the target’s heads or fea- tures, as in Medusa (Cai et al., 2024) and the EAGLE series (Li et al., 2024a,b, 2025). DSpark develops semi- autoregressive generation on a parallel DFlash back- bone and schedules verification prefixes using confi- dence and engine cost (Chen et al., 2026; Cheng et al., 2026). DFlash2 retains parallel drafting with a con- ditional candidate-lattice selector (Inco AI, 2026); we support both through their existing proposals and ac- ceptance procedures by supplying the KV retrieval and storage required for hierarchical verification. Two systems study speculative decoding at long context. MagicDec (Sadhukhan et al., 2025) shows that speculation can raise throughput at large batch sizes once sequences are long, using drafters with a sparse KV cache. TriForce (Sun et al., 2024a) combines retrieval-based drafting with hierarchical speculation to relieve the KV bottleneck of long sequences, includ- ing an offloading setting. Both give the drafter a sparse or partial KV cache; we keep the target’s sparse selec- tions exact and manage where its full history lives. Serving infrastructure. PagedAttention, SGLang, and DistServe address paged allocation, efficient execution, and prefill–decode separation (Kwon et al., 2023; Zheng et al., 2024; Zhong et al., 2024). Active-request resi- dency complements prefix caching: a prefix hit avoids repeated prefill computation, while the decode hier- archy manages the physical history used by ongoing requests. Our bounded draft and staging allocations operate alongside these serving mechanisms. 7. Conclusion Hierarchical KV caching lets long histories live in host memory, but with a fixed cache budget requests keep fetching from host while HBM sits idle below peak load, and with single-query retrieval draft-model speculation stays outside the hierarchy. Adaptive DHiSparse closes both gaps by bringing draft-model speculative decoding to hierarchical sparse KV serving and adapting history residency to load. A protected working set supports exact recovery, spare HBM re- tains additional history, and a source-aware resolver shares missing-row retrieval across verification queries. Window-bounded draft KV and reusable staging con- trol the extra memory required by DSpark and DFlash2 while preserving their proposals and accepted-state updates. At 100K context and C128, GLM-5.3 with DFlash2 delivers 1.45×the output of HiSparse and, held to the same latency bounds, sustains up to 4×the concurrency. At 500K context and C512, DeepSeek- V4.1-Flash with DSpark delivers 1.85×the output of resident serving with DSpark. With Adaptive DHiSparse, GPU residency becomes a runtime decision where it was a deployment constant. Optional history removes most host fetches while HBM is spare and returns its pages when sessions need them. More broadly, sparse attention, hierarchical KV place- ment and draft-model speculation compose. Each re- moves a different cost of long-context serving (what a query computes, where its history lives, how many tar- get passes a token takes), and combining them leaves selections, proposals and acceptance unchanged. How cheaply they combine is decided in the model and the drafter. A model that selects few, narrow rows from KV sources shared across its layers leaves the hierarchy little to move (about 80×fewer host bytes per token on DeepSeek-V4.1-Flash than on GLM-5.3 at a compara- ble miss rate), and a drafter with a bounded window leaves speculation little to store at any context length. References Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, and Juanzi Li. IndexCache: Accelerating sparse attention via cross-layer index reuse. arXiv:2603.12201, 2026. URL https://arxiv.org/abs/2603.12201. William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan-Kelley. Reducing transformer key-value cache size with cross-layer attention. In Advances in Neural Information Processing Systems, volume 37, pages 86927–86957, 2024. URL https://arxiv.org/abs/2405.12981. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5209–5235. PMLR, 2024. URL https: //proceedings.mlr.press/v235/cai24b.html. Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block diffusion for flash speculative decoding. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 15343–15355. PMLR, 2026. URL https: //proceedings.mlr.press/v306/chen26bv.html. Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, et al. DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv:2607.05147, 2026. URL https://arxiv.org/abs/2607.05147. DeepSeek-AI. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv:2512.02556, 2025. URL https://arxiv.org/abs/2512.02556. DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. Technical report, DeepSeek-AI, 2026. URL https: //huggingface.co/deepseek-ai/DeepSeek-V4.1-F lash/blob/main/DeepSeek_V41_Tech_Report.pdf. Inco AI. DFlash 2: Keep drafting parallel. Technical article, 18 August 2026. URL https://inco.ai/blog/dflash2/. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles, pages 611–626, 2023. doi: 10.1145/3600006.3613165. URL https://arxiv.org/abs/2309.06180. Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation, pages 155–172. USENIX Association, 2024. URL https://w.usenix.org/conference/osdi24 /presentation/lee. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In 14 Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR, 2023. URLhttps://pr oceedings.mlr.press/v202/leviathan23a.html. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 28935–28948. PMLR, 2024a. URL https: //proceedings.mlr.press/v235/li24bt.html. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421–7432. Association for Computational Linguistics, 2024b. URL https: //aclanthology.org/2024.emnlp-main.422/. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. In Advances in Neural Information Processing Systems, volume 38, 2025. URL https://proceedings.neurips.c/paper_files/p aper/2025/hash/c7b5a35ea98b62512a869c19ea7b0 3cb-Abstract-Conference.html. Zuning Liang, Zhiyi Yao, Qi Chen, Yuedong Xu, Hao Dai, Zhiqiang Ding, Tongkai Yang, Jinlong Hou, and Yuan Cheng. DualDecoder: Accelerate long context LLM inference by predictive prefetch. arXiv:2607.26475, 2026. URL https://arxiv.org/abs/2607.26475. Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. MagicDec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2408.11049. SGLang contributors. HiSparse: Hierarchical sparse attention. SGLang documentation, 2026. URL https: //github.com/sgl-project/sglang/blob/main/do cs/docs/advanced_features/hisparse_guide.mdx. Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. TriForce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. In First Conference on Language Modeling (COLM), 2024a. URL https://arxiv.org/abs/2404.11912. Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. ShadowKV: KV cache in shadows for high-throughput long-context LLM inference. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 57355–57373. PMLR, 2025. URL https: //proceedings.mlr.press/v267/sun25b.html. Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. In Advances in Neural Information Processing Systems, volume 37, pages 7339–7361, 2024b. URL https://arxiv.org/abs/2405.05254. Zhiqiang Xie, Ziyi Xu, Mark Zhao, Yuwei An, Vikram Sharma Mailthody, Scott Mahlke, Michael Garland, and Christos Kozyrakis. Strata: Hierarchical context caching for long context language model serving. arXiv:2508.18572, 2025. URL https://arxiv.org/abs/2508.18572. Zhiqiang Xie, Zhangheng Huang, Tingwei Huang, Ziyi Xu, Ruiyang Ma, and Christos Kozyrakis. HiSparse: Scaling sparse-attention decoding with hierarchical KV cache management. arXiv:2608.07009, 2026. URL https://arxiv.org/abs/2608.07009. Z.ai. GLM-5.3. Model card, 2026. URL https://huggingface.co/zai-org/GLM-5.3. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2312.07104. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation, 2024. URL https://arxiv.org/abs/2401.09670. A. Additional Measurements This appendix gives the measurements behind the main re- sults. Table A1 lists load-step statistics; Table A2 partitions first-token latency into adjacent request intervals; and Ta- ble A3 gives latency at every measured load. Figure 11 supplies the round and step breakdowns discussed in Sec- tions 4.3.2 and 4.3.3. 15 020406080100 Profiled round time (ms) block 4 block 6 80.1 94.0 a GLM-5.3 with DFlash2, C64 13.921.316.215.613.1 18.723.121.018.312.9 cache kernels alone expert matrix products other model kernels several families at once no kernel coverage C16 fill C16 middle C16 drain C512 pressure 0 2 4 6 Resolver kernel work (% of step) 0.41% 1.36% 1.33% 4.75% b DeepSeek-V4.1-Flash, no speculation Figure 11 | Where step time goes. (a) GLM-5.3 with DFlash2 at C64: mean profiled round time for static blocks of 4 and 6, as an exclusive partition with a separate category for overlapping kernels. (b) DeepSeek-V4.1-Flash without speculation: resolver kernel work as a share of the step interval in serving profiles, at C16 (fill, middle, drain) and C512 under pressure. Table A1 | Load step, optional residency on / off. GLM-5.3 at 100K context, without speculation. Each cell gives the value with residency on, then off. The host-fetched share is counted over each phase; the other columns of the first two rows cover requests that start and finish inside the phase. The last row covers the time and the requests after the in-flight count returned to 16. PhaseRequestsFrom host (%)TPOT (ms)Tok/s per sessionGainTTFT (s)Alloc. wait (s) C16 before80 / 801.38 / 14.2125.18 / 26.0739.83 / 38.38+3.8%1.65 / 3.220.03 / 0.00 C256417 / 37513.79 / 13.7953.77 / 52.2418.61 / 19.18 −3.0%36.89 / 46.6328.19 / 19.74 C16 after recovery62 / 522.43 / 13.9824.44 / 28.1040.92 / 36.19+13.1%0.84 / 0.870.02 / 0.01 Table A2 | Where first-token latency is spent, in seconds. Allocation wait precedes slot allocation; handoff covers prefill readiness and KV transport. The right half excludes the first 퐶 submitted requests. All requestsAfter the first wave 퐶MethodAllocationHandoffOtherTTFTAllocationHandoffTTFT 128HiSparse0.0010.360.3110.680.004.985.11 Adaptive HiSparse1.754.030.346.130.092.242.48 Adaptive DHiSparse2.512.230.395.130.251.772.25 256HiSparse32.1014.970.5347.6133.708.7142.54 Adaptive HiSparse29.154.700.5134.3627.093.7731.01 Adaptive DHiSparse62.501.500.6164.6266.461.0967.79 Table A3 | First-token and per-token latency at every load, GLM-5.3 at 100K. The middle columns exclude the first퐶 submitted requests. TTFT (s)TTFT after the first wave (s)TPOT (ms) Method퐶meanp99meanp99meanp99 HiSparse40.791.500.700.9519.419.5 81.052.810.751.8223.423.6 161.855.491.203.4928.428.7 323.8210.812.497.4034.836.4 645.0921.352.689.6342.243.4 12810.6843.535.1126.9750.351.8 25647.61158.8642.54102.8958.059.1 Adaptive HiSparse40.540.790.480.7116.016.1 80.761.070.690.9719.119.2 161.243.140.901.2324.124.7 321.595.701.253.0930.831.7 643.4716.742.056.6638.139.9 1286.1336.162.4811.8546.648.3 25634.36131.6131.0181.9954.455.4 Adaptive DHiSparse40.530.790.470.7011.413.2 80.651.350.490.7213.316.4 160.862.550.641.0316.218.8 321.476.880.741.4019.424.4 642.7615.451.123.0226.231.8 1285.1332.422.259.6735.942.1 25664.62111.8967.7987.6038.948.3 16