Paper deep dive

Kwai Summary Attention V2 Technical Report

OneRec Team, Chenglong Chu, Guohong Mu, Guowang Zhang, Hao Peng, Hui Wang, Jian Liang, Jiangxia Cao, Jiao Ou, Jiaxin Deng, Lu Ren, Muhao Wei, Ruiming Tang, Tianle Xu, Wenwu Ou, Xingmei Wang, Zhixin Ling, Zhiyuan Liang, Ziqi Wang, Chaoyi Ma, Haixiao Gao, Kuo Cai, Renpu Liu, Ruochen Yang, Wenlong Xu, Xinqi Jin, Yan Sun, Yifan Wang, Yingzhi He, Yiyu Wang, Yusen Huo, Zhaojie Liu, Zihan Zhou

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 99

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Summary-token architectures such as Kwai Summary Attention (KSA) compress distant context into learnable Summary token per chunk, and hybrid variants interleave these layers with full attention to retain exact access to the original text. In such hybrid KSA-Full architecture, the interleaving inserted summary tokens are served as a streaming flow role across different layers. Although effective, however, it also introduce some bottlenecks: (1) the different layers summary compression chunk must be same, therefore hard to reach a flexible KV Cache compression, e.g., different layer set different summary compression chunk window. (2) in the full-attention layers, the summary token introduced additional computation cost, but such layer ignore them and still pay quadratic prefilling, therefore unfriendly to the computation side. Motivated by them, we developed the KSA V2, which not only achieves a more flexible KV cache memory compression but also reduces the computation cost in Full attention layer. Specifically, we found that the summary tokens can serve two roles: the KV memory and sparse indexer.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

97,083 characters extracted from source content.

Expand or collapse full text

Kwai Summary Attention V2 Technical Report OneRec Team Abstract Summary-token architectures such as Kwai Summary Attention (KSA) compress distant context into learnable Summary token per chunk, and hybrid variants interleave these layers with full attention to retain exact access to the original text. In such hybrid KSA-Full architecture, the interleaving inserted summary tokens are served as a streaming flow role across different layers. Although effective, however, it also introduce some bottlenecks: (1) the different layers summary compression chunk must be same, therefore hard to reach a flexible KV Cache compression, e.g., different layer set different summary compression chunk window. (2) in the full-attention layers, the summary token introduced additional computation cost, but such layer ignore them and still pay quadratic prefilling, therefore unfriendly to the computation side. Motivated by them, we developed the KSA V2, which not only achieves a more flexible KV cache memory compression but also reduces the computation cost in Full attention layer. Specifically, we found that the summary tokens can serve two roles: the KV memory and sparse indexer. As KV memory, they can be further compressed without coarsening the representation passed between layers: Compressed KSA (C-KSA) pools groups of Summary KV into layer-local Meta entries, raising a larger chunk distant history compression. As sparse index, the summary token could conduct exact-access layers where to look: Sparse KSA (S-KSA) builds block-level routing keys from the Summaries and attends only to selected original-text blocks. We conduct experiments based on the 1.9B (MHA) and 4B (GQA) Hybrid KSA backbones, and we mostly adopt them by training the new compressed and sparse modules. In 1.9B (MHA) and 128K sequence scale, it reduces resident cache size by 40.4% and achieves 1.51×1.51× prefill and 1.27×1.27× decode throughput. Figure 1: Left: long-context and general capabilities of 1.9B C-KSA+S-KSA and the reference models reported in KSA. Right: per token inference FLOPs and cache size (1GiB=2301GiB=2^30 bytes). 1 Introduction Serving long contexts incurs substantial storage and computation costs. With full attention, the key–value (KV) cache grows linearly with context length, prefill attention is quadratic, and each decoding query reads the full history. Per-token KV compression (Ainslie et al., 2023; DeepSeek-AI et al., 2024) reduces storage per position but retains an entry for every token. Sparse attention (Lu et al., 2025; Yuan et al., 2025; DeepSeek-AI et al., 2025) reduces each query’s read set while retaining token-level KV. Sliding-window and recurrent linear attention (Jiang et al., 2023; Yang et al., 2025) reduce both storage and computation, but do not preserve direct access to distant tokens. Hybrid architectures balance these trade-offs by interleaving compact-memory layers with full-attention layers that retain direct access to the original text. Summary-token hybrids are a strong instance of this design. Kwai Summary Attention (KSA) (Chu et al., 2026) appends a learned Summary token after every ksk_s text tokens; in KSA layers, text queries read nearby chunks at full resolution and distant chunks through their Summaries. Hybrid KSA interleaves three such layers with one full-attention layer (Figure 2) and retrieves long-range information better than a same-size full-attention model. In these hybrids, however, the Summaries are underused. They are computed and refined at every layer, yet serve only as low-resolution memory for KSA layers, leaving an inefficiency on each side. On the efficient side, compression is tied to chunk size: a KSA layer stores one entry per distant chunk, so compressing further requires larger chunks, which also coarsen the Summaries that every later layer reads and refines. The KSA report’s 4B ablation shows that coarser Summary chunks can degrade retrieval performance. On the exact side, full-attention layers read every token, although most barely matter. At 128K, oracle block selection restricts the main-attention read set to about 4% of text positions, yet retains about 82% of decode attention mass and near-dense retrieval performance in a frozen Hybrid KSA (Figure 13; Table 9). What is missing is a cheap way to know where to look, and the Summaries, which describe every chunk, are never consulted. We argue that a single summary hierarchy can play both roles. As memory, Summaries can be compressed a second time when stored, without coarsening the representation that is propagated; as an index, they can direct exact-access layers to the few original-text blocks worth reading. Efficient layers store coarse, exact layers read fine, and the Summaries connect the two. We instantiate this idea with two drop-in layer types, which together form KSA V2. Compressed KSA (C-KSA) pools every kmk_m consecutive Summary KV entries into one layer-local Meta entry, using learned dimension-wise weights as in DeepSeek-V4 (DeepSeek-AI et al., 2026). Meta entries are never fed forward, so the Summary sequence keeps its granularity while distant-history storage shrinks by a factor of kskmk_sk_m (1024×1024× with ks=8k_s=8 and km=128k_m=128). Sparse KSA (S-KSA) replaces full attention: each Summary pools the tokens of each b-token block in its chunk into a routing key, a Lightning Indexer (DeepSeek-AI et al., 2025) selects the top blocks, and the main attention reads their original KV with attention sinks and a local window. Routing needs b×b× fewer candidates than token-level indexing in DSA, and unlike CSA (DeepSeek-AI et al., 2026), the main attention never reads compressed text. Because both layer types reuse existing Summaries, neither adds tokens nor changes the Summary layout, so both can be retrofitted into a pretrained hybrid. The adapted 1.9B model largely preserves Hybrid KSA’s retrieval and general capabilities while outperforming Full on retrieval. Ablations show that C-KSA alone matches its parent and that sparse adaptation incurs a smaller retrieval loss on the compressed backbone than on the original (Section 3.3). At 128K, the model reduces resident cache size by 40.4% and achieves 1.51×1.51× prefill and 1.27×1.27× decode throughput relative to Hybrid KSA. Compared with Full, it reduces cache size by 73.7% and achieves 2.16×2.16× prefill throughput (Section 3.6). The remaining cache is mostly original-text KV in the exact-access layers, and aggregation-heavy tasks such as frequent-word extraction degrade under block selection (Table 11). The approach also retains near-parent retrieval performance on a 4B GQA backbone, supporting transfer beyond the 1.9B MHA setting. Our contributions are threefold: • We propose C-KSA, which compresses Summary KV at storage time into layer-local Meta entries, so that further compression of distant history no longer requires coarser Summaries. • We propose S-KSA, which turns existing Summaries into block-level routing keys for sparse attention over selected original-text blocks. • We provide a retrofit recipe and GPU kernels that convert a pretrained Hybrid KSA model with RULER performance comparable to its parent, and we report where the remaining cost lies and where the approach fails. Figure 2: Architecture overview: from Hybrid KSA to KSA V2. 2 Methodology KSA V2 uses the Summaries in two roles: C-KSA compresses them further as long-range memory, and S-KSA uses them as an index to select original-text blocks for sparse attention. As shown in Figure 2, we retain Hybrid KSA’s 3:13:1 layout, repeating three C-KSA layers followed by one S-KSA layer throughout the backbone. We denote text tokens by TiT_i, chunk Summaries by ScS_c, and Meta KV entries by MgM_g. The chunk size ksk_s specifies the number of text tokens per Summary, kmk_m the number of Summaries per Meta entry, and b the number of text tokens per routing block. For a text query at position t, qtq_t and OtO_t denote the attention query and output, respectively. 2.1 Compressed KSA (C-KSA) C-KSA adds a second level of compression to KSA: KSA inserts one Summary token after every ksk_s text tokens, and C-KSA aggregates every kmk_m consecutive Summary KV entries into one Meta KV entry. This increases the compression factor for distant history from ksk_s to kskmk_sk_m, while keeping recent text directly accessible (Figure 3). Figure 3: Compressed KSA with ks=4k_s=4 and km=2k_m=2. Complete remote Summary groups are compressed into Meta KV, while boundary Summary KV and local text KV remain directly accessible. Text-to-Summary aggregation. As in KSA, we partition the text sequence into chunks of ksk_s tokens and append a Summary token ScS_c to each chunk c. All Summary tokens share a learned initial embedding and use the same attention and feed-forward parameters as text tokens. At each layer, ScS_c attends to its chunk’s text tokens and itself to aggregate chunk-level information. Text tokens retain their original positional indices, and ScS_c uses the position of its chunk’s last text token. Summary-to-Meta aggregation. In each C-KSA layer, we divide the Summary sequence into non-overlapping groups of kmk_m consecutive Summaries. Let d denote the hidden size and dKVd_KV the total key/value width across heads. For a group g, let hr∈ℝdh_r ^d denote the normalized layer input of its r-th Summary, where 0≤r<km0≤ r<k_m. We compute its key, value, and aggregation logits as kr k_r =RMSNorm(WKhr),vr=WVhr, =RMSNorm(W_Kh_r), v_r=W_Vh_r, (1) zr z_r =Ahr. =Ah_r. Here, WK,WV,A∈ℝdKV×dW_K,W_V,A ^d_KV× d; WKW_K and WVW_V are the backbone’s existing projections, and A is an additional learned projection. RMSNorm is applied independently to each attention head. We add a learned within-group position bias Br∈ℝdKVB_r ^d_KV and normalize the logits across the kmk_m Summaries to obtain weight vectors αr∈ℝdKV _r ^d_KV: [α0,…,αkm−1]=softmaxgroup⁡([z0+B0,…,zkm−1+Bkm−1]).[ _0,…, _k_m-1]=softmax_group ([z_0+B_0,…,z_k_m-1+B_k_m-1] ). (2) The softmax operates independently for each feature coordinate. The same weights aggregate the keys and values: k¯=∑r=0km−1αr⊙kr,v¯=∑r=0km−1αr⊙vr, k= _r=0^k_m-1 _r k_r, v= _r=0^k_m-1 _r v_r, (3) where ⊙ denotes element-wise multiplication. Finally, we apply rotary position embedding (RoPE) (Su et al., 2024) to the pooled key at position pgp_g of the group’s last Summary, producing the Meta KV entry Mg=(RoPE⁡(k¯,pg),v¯).M_g= (RoPE( k,p_g),\, v ). (4) The aggregation parameters A and Brr=0km−1\B_r\_r=0^k_m-1 are specific to each layer and shared across groups. Meta entries are used only within the layer that constructs them, while the original Summary states continue to propagate across layers. The group size kmk_m can therefore vary across C-KSA layers without changing the shared Summary token layout. We omit the layer index for clarity. Visible KV entries. For each text query, C-KSA attends to three types of KV entries: local text, boundary Summaries, and remote Meta entries (Figure 3). The local window contains the causal prefix of the current chunk and up to W preceding complete chunks. Outside this window, each complete group of kmk_m Summaries is represented by one Meta KV entry. The remaining fewer than kmk_m Summaries immediately preceding the window remain individually accessible until their entire group lies outside the window. All three types of entries participate in a single attention softmax. Attention-mass correction. Compressing remote Summary entries can shift attention toward the local window by reducing the number of terms in the softmax denominator, even if local logits remain unchanged. We therefore add log⁡km k_m to each Meta logit s before normalization. Since exp⁡(s+log⁡km)=kmexp⁡(s) (s+ k_m)=k_m (s), this helps preserve the balance between local and remote attention. 2.2 Sparse KSA (S-KSA) S-KSA replaces full attention with Summary-guided block selection and sparse reads of original text (Figure 4). It uses the Summary tokens already present in KSA to construct one index key for each text block. For each query, a Lightning Indexer selects relevant remote blocks, and the main attention reads their original KV together with local context. This design reduces the routing overhead of token-level selection and avoids dense attention over the entire text history, while preserving direct access to the selected blocks’ original tokens. Figure 4: Sparse KSA with ks=4k_s=4 and b=2b=2. Summary-guided block aggregation. We divide each chunk into blocks of b consecutive text tokens, where b divides ksk_s. For a block u, let h1,…,hb∈ℝdh_1,…,h_b ^d denote its normalized layer inputs and hS∈ℝdh_S ^d the normalized input of its chunk’s Summary. For one auxiliary attention head of width dAd_A, omitting the head index, we compute qS q_S =RoPE⁡(RMSNorm⁡(WQAhS),pS), =RoPE\! (RMSNorm(W_Q^Ah_S),p_S ), (5) ki k_i =RoPE⁡(RMSNorm⁡(WKAhi),pi), =RoPE\! (RMSNorm(W_K^Ah_i),p_i ), vi v_i =WVAhi. =W_V^Ah_i. Here, WQA,WKA,WVA∈ℝdA×dW_Q^A,W_K^A,W_V^A ^d_A× d are auxiliary projections specific to each S-KSA layer and independent of the backbone’s attention projections. The positions pSp_S and pip_i follow the original sequence positions, with the Summary sharing its chunk’s last text position. Writing KA=[k1,…,kb]K^A=[k_1,…,k_b] and VA=[v1,…,vb]V^A=[v_1,…,v_b], the head’s block representation is z^u=VAsoftmax⁡((KA)⊤qSdA). z_u=V^Asoftmax\! ( (K^A) q_S d_A ). (6) The softmax operates over the block’s b text tokens; the Summary supplies only the query. Concatenating the outputs of the HAH_A auxiliary heads gives zu∈ℝHAdAz_u ^H_Ad_A. Blocks in a chunk share the Summary query but aggregate their own text tokens. This resembles the landmark pooling of HiLS-Attention (Hu et al., 2026), but reuses the existing Summary as the query rather than inserting additional landmark tokens. Block indexing. We adapt the Lightning Indexer (DeepSeek-AI et al., 2025) to score text blocks using their Summary-guided representations. For a text query at position t, the score for block u is ρu=∑h=1HIwhReLU⁡((qhI)⊤kuIdI). _u= _h=1^H_Iw_h\,ReLU\! ( (q_h^I) k_u^I d_I ). (7) The Indexer uses HIH_I query heads of dimension dId_I. For each text query, its normalized layer input hth_t is mapped to a query vector qhI∈ℝdIq_h^I ^d_I and a scalar weight whw_h for each head h. Each block representation zuz_u is mapped to a single index key kuI∈ℝdIk_u^I ^d_I, shared across all heads. The weights whw_h, scaled by 1/HI1/ H_I, combine the per-head relevance scores into the block score ρu _u. For the 4B model, we additionally apply RMSNorm to the projected index queries and keys before RoPE to improve adaptation convergence. Section 3.4 discusses this choice. Before scoring, partial RoPE encodes queries at position t and keys at their blocks’ last text positions. Sparse access to original text. Given the block scores ρu\ _u\, S-KSA selects up to K/bK/b remote blocks, where K denotes the remote budget measured in original-text tokens. Candidates must belong to completed chunks and lie entirely before the causal local window. Eligible sink blocks at the beginning of the sequence are retained within this budget, and the remaining slots are filled by the highest-scoring candidates. Let ℒtL_t and ℛtR_t denote local and selected remote positions, respectively. The main attention computes Ot=Attn⁡(qt,[Kℒt,Kℛt],[Vℒt,Vℛt]),O_t=Attn\! (q_t,\,[K_L_t,K_R_t],\,[V_L_t,V_R_t] ), (8) where brackets denote concatenation along the sequence dimension. The attention uses the backbone’s original-text Q/K/V, with a single softmax over local and remote tokens in each head. For N text tokens, S-KSA constructs approximately N/bN/b index keys, reducing candidate comparisons by a factor of b relative to token-level indexing. Its routing cost scales as O⁡(N2/b)O(N^2/b) during prefill, while the main attention reads a budgeted set of original-text blocks for each query. 2.3 KV Cache for C-KSA The C-KSA KV cache combines a contiguous attention region with a Summary ring buffer to support incremental decoding and second-level compression. We first describe its layout and update rules, then analyze its memory cost and how sequence compression combines with GQA and MLA. 2.3.1 Cache Structure and Updates To support efficient decoding, we design a compact KV cache structure for C-KSA (Figure 5). This structure allows each text query to read all visible KV entries through a single contiguous slice, avoiding per-step concatenation of cache segments. Each C-KSA layer maintains two components: an Attention KV Cache and a Summary Ring Buffer. The illustration uses ks=2k_s=2, W=4W=4, and km=4k_m=4, with TiT_i, ScS_c, and MgM_g denoting text, Summary, and Meta KV entries, respectively. (a) Cache layout. Text queries read one contiguous KV slice (blue outline). The Summary Ring Buffer stores local Summary KV and compression state. (b) Text insertion. Writing T23T_23 immediately left of T22T_22 completes chunk 11. (c) Summary insertion. S11S_11 attends only to T22T_22, T23T_23, and its own KV. (d) Ring update. Chunk 11 replaces chunk 7, moving S7S_7 to remote history. S11S_11 takes its slot, and the pointer advances. (e) Meta compression. S4S_4–S7S_7 become M1M_1, which overwrites the group’s first slot and frees the other three. Figure 5: C-KSA cache layout and decoding updates with ks=2k_s=2, W=4W=4, and km=4k_m=4. Gray (T), orange (S), and purple (M) denote text, Summary, and Meta KV. Blue marks indicate reads or writes. Cache layout. The Attention KV Cache stores keys and values in separate tensors with the same layout: the current chunk, a local text ring, and remote history, followed by free slots (Figure 5(a)). The current chunk reserves ksk_s text slots, and the local ring reserves ksWk_sW slots for W preceding complete chunks. Remote history contains Meta entries followed by fewer than kmk_m Summaries awaiting compression. The visible KV entries are contiguous along the sequence dimension, allowing each text query to read them with a single slicing operation. The Summary Ring Buffer stores the local chunks’ Summary KV and auxiliary compression state. While a chunk remains in the local window, text queries attend to its original-text KV rather than its Summary KV. When the chunk leaves the window, its Summary KV enters remote history. A shared pointer identifies the next chunk slot to replace in both rings. Text and Summary token insertion. New text KV entries are written from right to left in the current-chunk region, keeping the filled portion adjacent to the local text ring. Each text query attends to the visible KV, including its own entry. At a chunk boundary, the Summary query attends only to the completed chunk’s text KV and its own KV. In the example, T23T_23 completes chunk 11 (Figure 5(b)), and S11S_11 attends to T22T_22, T23T_23, and itself (Figure 5(c)). Updating the text and Summary rings. Once text and Summary attention finish, the completed chunk’s text KV is copied into the local text ring, and its Summary KV and auxiliary compression state are stored in the Summary Ring Buffer. If the rings are not yet full, the new entries occupy unused slots. Otherwise, they replace the oldest local chunk and its Summary record; the old Summary KV moves into remote history, with its auxiliary state retained for compression. The shared pointer advances to the next chunk slot. In Figure 5(d), chunk 11 replaces chunk 7: S7S_7 enters remote history, S11S_11 takes its Summary ring slot, and the pointer advances to the slot holding S8S_8. Compressing remote Summaries. When kmk_m uncompressed Summaries accumulate in remote history, they are compressed into a single Meta KV entry. This entry overwrites the group’s first slot, leaving the remaining km−1k_m-1 slots available for reuse. In Figure 5(e), S4,S5,S6,S7S_4,S_5,S_6,S_7 are compressed into M1M_1, stored immediately after M0M_0. The visible KV entries remain contiguous, allowing each text query to read them through a single slice. 2.3.2 KV Cache Memory Analysis We extend the KV-cache analysis of KSA with C-KSA’s second compression level. Table 1 retains the original mechanisms for comparison. Table 1 counts stored scalars per layer and request, excluding allocation headroom and auxiliary buffers. Table 1: KV-cache cost comparison for a sequence of N text tokens. Example settings use ks=8k_s=8, km=128k_m=128, H=128H=128, dh=128d_h=128, HKV=8H_KV=8, dc=512d_c=512, and dr=64d_r=64. Compression rates are relative to MHA. Mechanism Effective Context KV Cache Size (N→∞N→∞) Compression Rate MHA N (exactly) 2⋅N⋅H⋅dh2· N· H· d_h — GQA N (exactly) 2⋅N⋅HKV⋅dh2· N· H_KV· d_h HKVH≈6.25% H_KVH≈ 6.25\% MLA N (exactly) N⋅(dc+dr)N·(d_c+d_r) dc+dr2Hdh≈1.76% d_c+d_r2Hd_h≈ 1.76\% GDN N (ambiguous) H⋅dh2H· d_h^2 dh2N≈0% d_h2N≈ 0\% SWA w (exactly) 2⋅w⋅HKV⋅dh2· w· H_KV· d_h wN⋅HKVH≈0% wN· H_KVH≈ 0\% KSA N (summarized) 2⋅Nks⋅H⋅dh2· Nk_s· H· d_h 1ks≈12.5% 1k_s≈ 12.5\% KSA with GQA N (summarized) 2⋅Nks⋅HKV⋅dh2· Nk_s· H_KV· d_h 1ks⋅HKVH≈0.78% 1k_s· H_KVH≈ 0.78\% KSA with MLA N (summarized) Nks⋅(dc+dr) Nk_s·(d_c+d_r) 1ks⋅dc+dr2Hdh≈0.22% 1k_s· d_c+d_r2Hd_h≈ 0.22\% C-KSA N (summarized) 2⋅Nkskm⋅H⋅dh2· Nk_sk_m· H· d_h 1kskm≈0.098% 1k_sk_m≈ 0.098\% C-KSA with GQA N (summarized) 2⋅Nkskm⋅HKV⋅dh2· Nk_sk_m· H_KV· d_h 1kskm⋅HKVH≈0.0061% 1k_sk_m· H_KVH≈ 0.0061\% C-KSA with MLA N (summarized) Nkskm⋅(dc+dr) Nk_sk_m·(d_c+d_r) 1kskm⋅dc+dr2Hdh≈0.0017% 1k_sk_m· d_c+d_r2Hd_h≈ 0.0017\% KV Cache Cost. Let N denote text length, H and HKVH_KV the query and KV head counts, and dhd_h the head width. In Table 1, dcd_c and drd_r are MLA’s latent and rotary-key widths, and w is the local text window. KSA’s leading cache term is 2NHKVdh/ks2NH_KVd_h/k_s. Under the compression rule in Section 2.1, C-KSA replaces it with MC-KSA=2HKVdh[Nkskm+O⁡(ksW+W+km)].M_C-KSA=2H_KVd_h [ Nk_sk_m+O(k_sW+W+k_m) ]. (9) The bounded term accounts for local text, recent Summaries, and unfinished compression groups. Thus C-KSA reduces the growing history term by a further factor of kmk_m while preserving the local window. With ks=8k_s=8 and km=128k_m=128, the growing history term is 1/10241/1024 of uncompressed attention with the same KV width. This nominal 1024×1024× compression applies to distant history; the local and auxiliary terms remain. For fixed compression factors, cache growth remains linear in N. Orthogonality of KV Cache Compression. C-KSA preserves KSA’s orthogonality to per-position compression: the additional kmk_m factor multiplies the sequence compression without changing the KV head count or representation width. The resulting GQA and MLA combinations are listed in Table 1. Figure 6 uses the settings in Table 1 with 60 layers, batch size 1, BF16, and w=ksW=1024w=k_sW=1024 local tokens. The KSA and C-KSA curves use N/ks+wN/k_s+w and N/(kskm)+wN/(k_sk_m)+w entries, respectively, omitting other bounded auxiliary buffers and allocation headroom. Figure 6: KV cache size comparison across different mechanisms as sequence length grows. The inset uses a logarithmic vertical axis to separate the low-cache curves and labels C-KSA values at 128K. Hybrid KSA and C-KSA. With matched layer counts and KV widths and a fraction f of Full layers, the leading cache ratio is MC-KSA+FullMHybridKSA≃f+(1−f)/(kskm)f+(1−f)/ks. M_C-KSA+FullM_Hybrid\ KSA f+(1-f)/(k_sk_m)f+(1-f)/k_s. (10) With f=1/4f=1/4 in the 3:13:1 layout, the Full layers limit the whole-model savings. 2.4 Kernel Design C-KSA and S-KSA reduce attention computation and KV reads through compression and sparse access. Realizing these savings on GPUs requires efficient KV addressing, block selection, and attention execution. Figure 7 summarizes the method-specific addressing and routing alongside the tiled prefill pattern used by both variants. (a) C-KSA: direct addressing. (b) S-KSA: fused block scoring. (c) Shared tiled prefill. Figure 7: Kernel design for C-KSA and S-KSA: (a) direct KV addressing, (b) fused block scoring and selection under a K-token budget, and (c) shared tiled prefill with KV reuse. Shared prefill execution. Both C-KSA and S-KSA follow FlashAttention’s tiled execution and online softmax (Dao et al., 2022), as illustrated in Figure 7(c). Let ℬB denote a group of text queries arranged into one or more query tiles, with query and output matrices QℬQ_B and OℬO_B. Each box in these matrices represents one token vector; blue cells in the query–key grid indicate visible pairs, while white cells are masked. Each loaded KV tile (Kj,Vj)(K_j,V_j) is reused on chip within the group, while a per-query mask enforces each query’s visible set. Building on this shared execution pattern, C-KSA and S-KSA specialize the KV addressing and tile schedule described below. 2.4.1 C-KSA Logical intervals and direct addressing. C-KSA’s regular layout allows the kernel to compute the KV locations needed by each query using simple arithmetic. During prefill, text and Summary Q/K/V are interleaved, while precomputed Meta K/V are stored in a separate buffer. We treat text, Summary, and Meta entries as three logical segments, each with its own zero-based index space. Figure 7(a) illustrates this addressing process with ks=2k_s=2, km=2k_m=2, and W=1W=1. Blue outlines mark entries visible to T9T_9. For a text query at position t, let a=max⁡(0,⌊t/ks⌋−W)a= (0, t/k_s -W) denote the first chunk in its local window and g=⌊a/km⌋g= a/k_m the number of complete Meta groups preceding that window. We first determine the visible logical indices in each segment: ℐT(t)=[ksa,t+1),ℐS(t)=[kmg,a),ℐM(t)=[0,g).I_T(t)=[k_sa,\,t+1), _S(t)=[k_mg,\,a), _M(t)=[0,\,g). (11) These intervals contain the causal local text, boundary Summaries, and complete remote Meta groups, respectively. We then map these logical indices to physical positions in the interleaved storage. For text index i and Summary index j, the mappings are ϕT(i)=i+⌊iks⌋,ϕS(j)=j⁡(ks+1)+ks. _T(i)=i+ ik_s , _S(j)=j(k_s+1)+k_s. (12) The kernel applies these mappings when loading Q/K/V. Meta indices directly address entries in the separate Meta buffer. For each query tile, the logical intervals determine the KV tile ranges to visit, and a position-based mask excludes entries outside each query’s visible set. The ranges and masks are computed within the kernel, with no dense attention mask or block-index table stored in memory. Prefill integration. The text-query kernel carries each query’s online-softmax state across the text, Summary, and Meta segments, applying the log⁡km k_m correction only to Meta logits. A separate local kernel computes each Summary query over its own text chunk and self-KV. Both kernels write outputs directly to their interleaved positions. Decoding. The KV cache in Section 2.3.1 stores the visible KV entries in a contiguous region, allowing attention to be computed directly with scaled dot-product attention (SDPA). 2.4.2 S-KSA S-KSA uses fused block scoring and a sparse KV-tile schedule to support query-dependent access to original text. The scoring optimization also applies during decoding. Fused block scoring. Using separate kernels for dot products and head aggregation requires storing intermediate scores in GPU memory. To reduce this memory traffic, we compute block scores with a fused kernel that performs dot products and combines weighted head scores on chip. In Figure 7(b), QIQ^I denotes the stacked query matrices of the indexer heads and KIK^I their shared block keys. The superscript I denotes the indexer. Each QIQ^I matrix has queries along rows and features along columns. The central grayscale matrix has queries along rows and candidate blocks along columns, with darker cells indicating higher scores. A separate selection step then chooses remote blocks independently for each query under a K-token budget (up to K/bK/b blocks); each row of the blue IDs matrix on the right lists the block indices selected for one query. KV-tile scheduling. For each query group, we map its selected remote blocks and local windows to KV tiles and construct a conservative schedule of tiles to visit. The schedule skips inactive KV tiles, while per-query masks enforce block selection, local visibility, and causality within the scheduled tiles. We supply this schedule and the masks to the FlashAttention forward kernel for tiled prefill execution. Decoding. Gathering selected KV entries into temporary buffers incurs extra memory reads and writes before attention computation. To avoid this overhead during decoding, we use a custom sparse kernel that reads directly from the KV cache using the selected block indices. 3 Experiments Our experiments examine whether continued pretraining can adapt Hybrid KSA to more aggressive KV compression and sparse original-text access while retaining its capabilities. We first evaluate C-KSA with Full attention, then add S-KSA to the compressed backbone, assessing long-context retrieval, general capabilities, and inference efficiency. We further analyze the effects of compression granularity, block size, teacher pooling, and routing on model performance. Unless otherwise stated, all experiments and analyses use the 1.9B model. 3.1 Experimental Setup Pretrained model and variants. We start from the 128K checkpoints of the 1.9B and 4B Hybrid KSA models in the KSA report (Chu et al., 2026). The 1.9B backbone has 24 multi-head attention (MHA) layers; the 4B backbone has 36 grouped-query attention (GQA) layers with 32 query heads and eight KV heads. Both interleave three KSA layers with one Full layer and insert one Summary after every ks=8k_s=8 text tokens. We evaluate two successive variants: C-KSA+Full compresses remote Summary KV in KSA layers; C-KSA+S-KSA further replaces the Full layers with S-KSA. Although C-KSA supports layer-specific compression, each evaluated configuration uses a common kmk_m across its C-KSA layers. Unless otherwise specified, all experiments use the backbone and module configurations in Table 2. Baselines and training data. We reuse the KSA report’s from-scratch results at 1.9B and continual-pretraining results at 4B. The baselines include Full, Hybrid SWA, Hybrid SCA, Pure KSA, and Hybrid KSA at both scales, plus Hybrid GDN at 1.9B and Hybrid Linear at 4B. Hybrid KSA serves as the parent-model reference for capability retention, while Full provides a dense-attention reference. We report module-only and full-parameter training budgets separately. A continuation study found no consistent RULER improvement for 1.9B Hybrid KSA under the tested recipe and budgets (Table 3.7), supporting our use of its original checkpoint as the capability reference. Adaptation reuses KSA’s training corpus and data mixture, with data sources and mixture proportions detailed in the original report. We use a fixed context length of 128K. Evaluation. Following the KSA report, we assess long-context modeling and general capabilities. Long-context modeling is evaluated using RULER (Hsieh et al., 2024) at sequence lengths from 4K to 128K. General capabilities cover knowledge (MMLU, CMMLU, C-Eval, and MMLU-Pro (Hendrycks et al., 2021; Li et al., 2024; Huang et al., 2023; Wang et al., 2024)), mathematics (GSM8K, CMATH, and MATH (Cobbe et al., 2021; Wei et al., 2023; Hendrycks et al., 2021a)), and code generation (MBPP and HumanEval (Austin et al., 2021; Chen et al., 2021)). Table 2: Configurations of the selected 1.9B and 4B models. Module settings apply where the corresponding modules are used. Configuration 1.9B 4B Shared backbone Number of layers 24 36 Hidden size 2048 2560 Intermediate size 6144 9728 Attention heads (Q/KV) 16/16 32/8 Head dimension 128 128 Hybrid architecture ratio 3:13:1 3:13:1 Summary chunk size ksk_s 8 8 Sliding chunk number (KSA/C-KSA) 128 128 Tied embeddings False True C-KSA Summary-to-Meta group size kmk_m 128 32 KV width dKVd_KV 2048 1024 Aggregation projection A shape 2048×20482048× 2048 1024×25601024× 2560 Position-bias shape km×dKVk_m× d_KV 128×2048128× 2048 32×102432× 1024 Additional parameters 80.2M 71.7M S-KSA Block size b 4 4 Remote token budget K 4096 4096 Sink tokens, included in K 8 8 Local window length (text tokens) 1024 1024 Aggregation heads (Q/KV) 16/16 32/8 Aggregation head dimension dAd_A 128 128 Indexer heads 8 8 Indexer head dimension 128 128 Query low-rank dimension 512 512 Final Indexer Q/K RMSNorm No Yes Additional parameters 86.6M 163.0M 3.2 Adaptation Protocol Figure 8 summarizes our sequential adaptation route at 1.9B: we first study compression granularity in C-KSA+Full, then compare sparse-attention schemes on the selected km=128k_m=128 checkpoint. Module-only adaptation trains only newly introduced parameters while freezing all inherited parameters; full-parameter adaptation updates the entire model. C-KSA uses module-only adaptation followed by full-parameter adaptation; the subsequent S-KSA stage uses module-only adaptation. C-KSA adaptation. During module-only adaptation, we train the aggregation projection A and position biases Br\B_r\ while freezing all inherited parameters. The new parameters are initialized to zero, yielding uniform aggregation weights. We align compressed attention outputs with those of uncompressed KSA. Let Ol,t,dCO^C_l,t,d and Ol,t,dKO^K_l,t,d denote the d-th coordinate of the attention outputs from C-KSA and uncompressed KSA, respectively, at layer l and text position t. Both outputs are computed from the same layer input, before the output projection. The alignment loss is ℒalign=1|ℐC|||D∑l∈ℐC∑t∈∑d=1Dmin⁡(Ol,t,dC−sg⁡(Ol,t,dK))2, 0.1,L_align= 1|I_C|\,|T|\,D _l _C _t _d=1^D \! \ (O^C_l,t,d-sg(O^K_l,t,d) )^2,\;0.1 \, (13) where ℐCI_C is the set of C-KSA layers, T contains the text positions in a minibatch, D is the concatenated attention width, and sg⁡(⋅)sg(·) denotes stop-gradient. Each squared coordinate error is clipped before averaging. We then transition from module-only to full-parameter adaptation, using the objectives ℒcomp=ℒCE+λalignℒalign,module-only adaptation,ℒCE,full-parameter adaptation,L_comp= casesL_CE+ _alignL_align,&module-only adaptation,\\[3.0pt] L_CE,&full-parameter adaptation, cases (14) where ℒCEL_CE is the text-token cross-entropy loss and λalign=5 _align=5. Figure 8: Sequential adaptation and parameter selection for KSA V2 at 1.9B. S-KSA adaptation. Module-only adaptation trains the new aggregation modules and Indexer while freezing all inherited parameters, including the C-KSA aggregation parameters. During training, we retain dense attention in Full layers and use its probabilities to supervise block selection. For query t, let tU_t contain all eligible remote blocks, ℬuB_u the text positions in block u, and at,ia_t,i the dense causal attention probability averaged across backbone heads. Suppressing the layer index, we define the block-level teacher distribution ptp_t, Indexer distribution πt _t, and training objective as pt,u p_t,u =sg⁡(maxi∈ℬu⁡at,i∑v∈tmaxj∈ℬv⁡at,j), =sg\! ( _i _ua_t,i _v _t _j _va_t,j ), (15) πt _t =softmax⁡(ρt), =softmax( _t), ℒS-KSA _S-KSA =l,t[DKL(pt∥πt)], =E_l,t\! [D_KL\! (p_t\,\|\, _t ) ], where sg⁡(⋅)sg(·) denotes stop-gradient and ρt _t contains the Indexer scores defined in Equation 7 for all blocks in tU_t. The loss is averaged over valid text queries within each adapted layer, then across layers. Following QSA (Qiu et al., 2026), the teacher takes the blockwise maximum, after averaging over heads; we compare max and sum pooling in Section 3.7. At inference, the trained Indexer selects remote blocks for sparse attention. Training configuration. Table 3 summarizes our adaptation hyperparameters. We reinitialize the optimizer and learning-rate schedule at each stage transition. Table 3: Adaptation hyperparameters and token budgets for the selected models. Configuration C-KSA S-KSA Module-only Full-parameter Module-only Trainable parameters Aggregation modules All Aggregation and Indexer Training tokens (B), 1.9B 6.7 3.4 18.5 Training tokens (B), 4B 6.7 15.9 4.2 Target learning rate 10−410^-4 10−410^-4 10−310^-3 LR warmup (updates) 200 400 200 Sequence length 128K Global batch size 32 Initial learning rate 10−610^-6 LR schedule Warmup then constant Optimizer AdamW AdamW (β1,β2)( _1, _2) (0.9,0.95)(0.9,0.95) Weight decay 0.01 Gradient clipping 1.0 3.3 Model Quality Table 4: Long-context retrieval and general capabilities at 1.9B (%; pass@1 for code). Bold and underlined values mark the best and second-best scores, respectively, here and in subsequent tables. Baselines Ours Benchmark Full Hybrid SWA Hybrid SCA Hybrid GDN Pure KSA Hybrid KSA C-KSA+Full C-KSA+S-KSA [0pt][0pt] Long-context retrieval RULER-4K 76.08 74.54 77.72 79.83 70.44 80.65 81.52 81.48 RULER-8K 72.85 71.69 75.22 76.01 65.91 73.35 72.24 72.42 RULER-16K 73.24 69.54 72.55 74.04 66.74 74.07 74.44 75.67 RULER-32K 69.06 67.86 67.74 70.41 62.54 72.30 71.93 70.78 RULER-64K 65.32 63.03 63.54 69.39 57.13 69.95 69.61 68.99 RULER-128K 48.75 56.64 58.01 59.87 39.29 65.35 66.31 65.01 RULER mean 67.55 67.22 69.13 71.59 60.34 72.61 72.68 72.39 [0pt][0pt] General knowledge MMLU 44.99 46.84 46.77 46.23 46.83 46.83 46.97 47.09 CMMLU 44.41 45.89 46.42 47.19 45.59 46.88 46.49 46.39 C-Eval 44.28 43.54 47.62 45.54 45.69 44.13 43.83 42.94 MMLU-Pro 19.48 20.46 20.10 21.22 21.72 22.52 22.71 22.83 [0pt][0pt] Mathematics GSM8K 48.29 47.46 52.39 50.95 54.81 59.14 55.95 56.33 CMATH 55.33 54.83 62.33 58.00 58.50 61.83 64.50 63.50 MATH 23.38 31.46 28.82 33.30 30.04 36.92 36.60 36.72 [0pt][0pt] Code MBPP 30.60 30.00 31.60 34.80 35.80 36.40 35.60 35.20 HumanEval 25.61 28.05 26.83 27.44 29.88 31.71 28.05 29.88 Table 3.3 compares C-KSA+Full and C-KSA+S-KSA with the 1.9B baselines from the KSA report. Long-context retrieval. C-KSA+S-KSA achieves a six-length RULER mean of 72.39% and a 128K score of 65.01%, compared with 72.61% and 65.35% for Hybrid KSA and 67.55% and 48.75% for Full. The combined model thus recovers near-parent retrieval performance while retaining the advantage of Hybrid KSA over Full. Compression and sparsity compose. At km=128k_m=128, C-KSA+Full increases the distant-history compression factor within KSA layers from 8 to 1024, with mean and 128K RULER scores of 72.68% and 66.31%. Adding S-KSA to this backbone lowers the 128K score to 65.01%, whereas the same sparse adaptation applied to Hybrid KSA lowers it from 65.35% to 61.06% (Table 8(c)). General capabilities. C-KSA+S-KSA remains broadly comparable to Hybrid KSA across the nine general-capability benchmarks, with gains on MMLU, MMLU-Pro, and CMATH and lower scores on the remaining tasks. The largest decrease is 2.81 percentage points on GSM8K, while MBPP and HumanEval decrease by 1.20 and 1.83 points, respectively. Compared with C-KSA+Full, adding S-KSA changes each score by at most 1.83 points, with both gains and losses, indicating no uniform decline from sparse adaptation. The combined model exceeds Full on eight of the nine benchmarks, with C-Eval as the exception. Adaptation dynamics. Figure 9 shows how retrieval performance recovers during adaptation. For C-KSA, retrieval performance improves during module-only adaptation and approaches Hybrid KSA after full-parameter adaptation. On the selected C-KSA+Full backbone, S-KSA with b=4b=4 approaches Hybrid KSA’s performance by training only its newly introduced aggregation modules and Indexer. Figure 9: Mean RULER scores during compression and sparse adaptation. Circles mark the checkpoints reported in Table 3.3. 3.4 Scaling to a 4B GQA Model To test transfer beyond the 1.9B MHA backbone, we adapt the 4B GQA model with km=32k_m=32 and b=4b=4. With ks=8k_s=8, C-KSA compresses distant-history KV by 256×256× along the sequence dimension while preserving GQA’s shared KV heads. The C-KSA+Full checkpoint uses 6.7B tokens of module-only adaptation followed by 15.9B tokens of full-parameter adaptation. Starting from this checkpoint, S-KSA uses a further 4.2B tokens of module-only adaptation, leaving all inherited parameters frozen. The remote budget, local window, and sink settings remain unchanged from the 1.9B experiments; the 4B Indexer additionally normalizes its projected queries and keys as described in Section 2.2. Table 5: Long-context retrieval and general capabilities at 4B (%; pass@1 for code). Baselines are from the KSA report (Chu et al., 2026). Baselines Ours Benchmark Full Hybrid SWA Hybrid SCA Hybrid Linear Pure KSA Hybrid KSA C-KSA+Full C-KSA+S-KSA [0pt][0pt] Long-context retrieval RULER-4K 92.88 91.30 86.02 86.39 91.55 92.97 92.77 93.38 RULER-8K 91.38 88.03 84.28 83.86 86.78 90.53 88.12 87.78 RULER-16K 89.12 82.87 80.67 78.06 84.78 88.86 86.27 85.45 RULER-32K 84.74 78.94 76.89 76.48 80.30 86.65 82.68 83.60 RULER-64K 78.16 73.88 68.88 73.50 76.09 76.04 78.58 80.28 RULER-128K 65.86 66.27 60.94 67.98 66.81 71.67 73.92 74.41 RULER mean 83.69 80.22 76.28 77.71 81.05 84.45 83.72 84.15 [0pt][0pt] General knowledge MMLU 71.83 70.57 69.83 64.33 70.73 70.50 68.79 68.79 CMMLU 75.00 73.69 72.59 68.41 73.29 72.63 70.88 71.06 C-Eval 73.66 72.36 71.66 67.42 72.14 72.66 70.36 70.58 MMLU-Pro 46.36 45.23 45.11 38.83 45.70 45.39 43.66 43.58 [0pt][0pt] Mathematics GSM8K 82.75 81.92 80.10 72.44 81.09 79.50 79.00 79.00 CMATH 83.41 84.84 83.16 79.09 84.58 84.25 81.50 81.67 MATH 47.48 48.24 47.45 42.57 48.15 47.56 46.10 46.40 [0pt][0pt] Code MBPP 61.30 61.70 59.60 55.30 61.50 62.20 60.40 60.00 HumanEval 58.54 61.89 61.89 54.58 60.97 62.50 60.98 60.98 Table 3.4 compares the adapted models with the six 4B baselines reported in KSA. C-KSA+S-KSA achieves a six-length RULER mean of 84.15%, compared with 84.45% for Hybrid KSA and 83.72% for its C-KSA+Full starting point. At 128K, it scores 74.41%, compared with 71.67% for Hybrid KSA and 65.86% for Full. On general-capability benchmarks, C-KSA+S-KSA stays within 0.40 percentage points of C-KSA+Full and 0.50–2.58 points below Hybrid KSA. Overall, C-KSA+S-KSA retains retrieval performance close to Hybrid KSA, with modest declines in general capabilities. These results support its effectiveness at a larger scale and compatibility with GQA. Indexer Q/K normalization. Without Q/K normalization, the 4B Indexer loss remains high, and RULER performance stays below its C-KSA+Full parent even after 21.8B adaptation tokens. Under the same adaptation setup, normalizing the projected queries and keys restores near-parent performance with only 4.2B tokens. A possible explanation is that differences in feature scale between the pretrained backbones affect index-logit magnitudes and optimization; RMSNorm reduces this scale sensitivity. We retain the original 1.9B setup for consistency and have not repeated its full experimental suite with this change. Whether normalization also reduces the adaptation data required at 1.9B remains an open question for future work. Compression granularity. Table 6 compares compression granularities on the 4B backbone, using 6.7B tokens of module-only adaptation followed by 15.9B tokens of full-parameter adaptation for each configuration. Compared with km=8k_m=8, km=32k_m=32 reduces the number of remote Meta entries by a further factor of four while retaining similar RULER performance. We select km=32k_m=32 for the 4B sparse-adaptation experiments. Table 6: RULER scores (%) of 4B C-KSA+Full with different compression granularities. kmk_m 4K 8K 16K 32K 64K 128K Mean 8 91.13 88.22 85.71 82.62 81.35 74.58 83.94 16 91.45 89.51 88.29 83.06 75.47 69.56 82.89 32 92.77 88.12 86.27 82.68 78.58 73.92 83.72 64 89.83 87.17 85.62 81.67 79.56 70.93 82.46 128 92.23 89.83 86.36 81.25 74.38 71.90 82.66 3.5 Long-Context Retrieval Analysis We evaluate whether C-KSA+S-KSA retains the parent model’s ability to retrieve information from different positions in a long context. In the Needle-in-a-Haystack test (Kamradt, 2023), a target item is inserted into the context and retrieved through a targeted question. Figure 11 compares the models on identical inputs across six context lengths from 4K to 128K and eleven insertion depths from 0% to 100%. Table 11 complements the grid with a paired comparison of RULER subtasks at 128K. Figure 10: Single-needle retrieval across context lengths and needle depths. C-KSA+S-KSA maintains high accuracy across the grid, averaging 97.88% over all depths at 128K. Figure 11: RULER-128K subtask scores (%). C-KSA+S-KSA exceeds Hybrid KSA on Single, Multiquery, and SQuAD. Subtask Hybrid KSA C-KSA+S-KSA NIAH-Single 93.33 98.33 NIAH-Multikey 85.00 83.33 NIAH-Multivalue 91.25 88.75 NIAH-Multiquery 91.25 96.25 VT 25.00 23.00 FWE 45.00 26.67 SQuAD 30.00 35.00 3.6 Inference Efficiency C-KSA reduces inference costs by compressing remote Summary KV, while S-KSA replaces dense original-text attention with block-sparse access. Figure 1 summarizes the resulting compute and cache costs for C-KSA+S-KSA with km=128k_m=128 and b=4b=4. At 128K, its estimated matrix-FLOP count per text token is lower by a factor of 6.386.38 relative to Full and 2.762.76 relative to Hybrid KSA. Its BF16 KV and index-key cache is smaller by factors of 3.863.86 and 1.351.35, respectively. Figure 12 compares the resident cache size, prefill throughput, and decode throughput of Full, Hybrid KSA, and C-KSA+S-KSA from 16K to 128K using the 1.9B models with BF16 and batch size one on a single NVIDIA H800 GPU. C-KSA+S-KSA uses km=128k_m=128 and b=4b=4. Cache. At 128K, C-KSA+S-KSA’s resident cache occupies 6.31 GiB, compared with 10.59 GiB for Hybrid KSA and 24.00 GiB for Full, reducing cache size by 40.4% and 73.7%, respectively. Measurements include indexer, auxiliary state, KV storage, and resident workspaces. Prefill. At 128K, C-KSA+S-KSA reaches 44,971 tokens/s, compared with 29,837 for Hybrid KSA and 20,830 for Full, corresponding to 2.16× of Full attention. Its gain over Hybrid KSA increases from 1.23×1.23× at 16K to 1.51×1.51× at 128K. Decode. C-KSA+S-KSA achieves 215.50 tokens/s at 16K and 198.85 tokens/s at 128K, compared with 162.87 and 157.02 for Hybrid KSA, respectively. Across the four sequence lengths, this gives a 1.23–1.32× speedup. Figure 12: Inference cache size and throughput of 1.9B Full, Hybrid KSA, and C-KSA+S-KSA (km=128k_m=128, b=4b=4). 3.7 Analysis and Ablation Studies Direct versus hierarchical compression. An alternative way to obtain 1024×1024× compression of distant-history KV is to increase the Summary chunk size directly to ks=1024k_s=1024. Table 7 compares this approach with C-KSA+Full, using the original ks=8k_s=8 Hybrid KSA as a reference. All three configurations retain Full-attention layers. The large-chunk model is trained from scratch, while C-KSA builds on small-chunk Hybrid KSA. Table 7: RULER performance of direct and hierarchical compression with Full attention retained. Configuration 4K 8K 16K 32K 64K 128K Mean Hybrid KSA, ks=8k_s=8 80.65 73.35 74.07 72.30 69.95 65.35 72.61 Hybrid KSA, ks=1024k_s=1024 75.65 72.14 70.44 70.81 69.11 65.62 70.63 C-KSA+Full, ks=8k_s=8, km=128k_m=128 81.52 72.24 74.44 71.93 69.61 66.31 72.68 Direct compression achieves a mean RULER score of 70.63, compared with 72.61 for the small-chunk reference and 72.68 for C-KSA+Full. Hierarchical compression thus matches the reference performance and exceeds direct compression by 2.05 percentage points. This finding is consistent with the 4B-model ablation in the KSA report (Chu et al., 2026, Table 7), where increasing ksk_s from 8 to 64 reduces the mean RULER score over 4K–64K from 82.97 to 77.79. This advantage may stem from the additional computation devoted to fine-grained Summary processing in C-KSA, which may help preserve retrieval-relevant information at the same remote-memory compression ratio. Actually, the additional computation as in two aspects: (1) the C-KSA takes 8 as the summary chunk, which will add more tokens at input sequence level; (2) the compressed KV memory is further computed by an additional module, which can be seen as a two-step summarization. Table 8: Ablations of compression and sparse attention. Dashes indicate unreported scores since the sequence length is not enough to activate sparse mechanism. Configuration Long-context retrieval (RULER) Knowledge Mathematics Code 4K 8K 16K 32K 64K 128K Mean MMLU CMMLU C-Eval MMLU Pro GSM8K CMATH MATH MBPP Human Eval (a) C-KSA + Full: compression granularity km=4k_m=4 79.54 72.37 74.22 71.37 68.21 65.78 71.92 46.77 46.27 43.76 23.06 57.39 64.00 36.80 36.40 28.05 km=8k_m=8 79.42 72.56 74.35 71.69 68.95 66.15 72.19 46.78 46.41 43.76 23.12 57.16 64.50 36.46 36.60 29.27 km=16k_m=16 79.95 73.31 74.92 72.31 68.46 67.46 72.74 46.95 46.30 43.39 22.61 56.71 63.17 36.56 37.40 31.10 km=32k_m=32 80.69 72.73 74.04 71.86 69.15 66.90 72.56 47.04 46.30 43.46 22.83 56.33 62.67 37.06 36.20 28.66 km=64k_m=64 80.94 72.00 73.68 72.30 68.41 67.04 72.40 47.12 46.19 43.46 22.92 56.25 63.50 36.70 36.00 27.44 km=128k_m=128 (selected) 81.52 72.24 74.44 71.93 69.61 66.31 72.68 46.97 46.49 43.83 22.71 55.95 64.50 36.60 35.60 28.05 (b) C-KSA, km=128k_m=128: sparse attention and teacher pooling DSA (token routing) 81.38 72.22 75.12 71.22 69.37 65.37 72.45 — — — — — — — — — QSA, b=4b=4, max 81.48 72.19 75.15 70.26 67.57 61.39 71.34 — — — — — — — — — S-KSA, b=2b=2, max 81.08 72.67 75.30 71.58 69.21 65.23 72.51 — — — — — — — — — S-KSA, b=4b=4, max (selected) 81.48 72.42 75.67 70.78 68.99 65.01 72.39 — — — — — — — — — S-KSA, b=8b=8, max 81.38 72.47 74.15 68.29 64.83 57.62 69.79 — — — — — — — — — S-KSA, b=4b=4, sum 81.48 72.15 74.81 70.28 68.51 61.68 71.49 — — — — — — — — — (c) Backbone for sparse adaptation: S-KSA, b=4b=4, max teacher KSA + S-KSA 80.71 73.82 74.76 72.12 68.57 61.06 71.84 46.60 46.99 43.83 22.62 58.83 61.67 37.40 36.80 31.71 C-KSA + S-KSA, km=128k_m=128 81.48 72.42 75.67 70.78 68.99 65.01 72.39 47.09 46.39 42.94 22.83 56.33 63.50 36.72 35.20 29.88 Compression granularity with Full attention. Table 8(a) compares km∈4,8,16,32,64,128k_m∈\4,8,16,32,64,128\ in C-KSA+Full. The six-length RULER mean ranges from 71.92 to 72.74, with no systematic decline as compression increases. The Full layers may help preserve retrieval quality by providing access to original-text details lost during compression. At 128K, km=128k_m=128 leaves about 128 Meta entries per C-KSA layer, so larger kmk_m would save little further cache; we therefore select km=128k_m=128 and leave higher factors to million-token contexts. Block size and routing methods. Table 8(b) compares sparse-attention schemes on the same km=128k_m=128 C-KSA+Full backbone, including S-KSA with b∈2,4,8b∈\2,4,8\, token-level DSA (DeepSeek-AI et al., 2025), and Qwen Sparse Attention (QSA) (Qiu et al., 2026). QSA provides a simple alternative to Summary-guided aggregation: it averages the projected token-level index keys within each block. We follow Qwen’s QSA indexer design, using b=4b=4 and eight indexer heads to match our comparison settings. All methods are compared under the same experimental setup. We report only RULER in this panel, since most general-capability sequences fit within the combined remote and local attention budget and offer limited discrimination of routing quality. S-KSA with b=4b=4 achieves a mean RULER score comparable to those of b=2b=2 and token-level DSA, with only half and one quarter as many routing candidates, respectively. It also outperforms QSA at the same block size. Increasing to b=8b=8 reduces the candidate count further but degrades retrieval quality, particularly at 128K. We therefore select b=4b=4 as a balance between retrieval quality and routing cost. Max versus sum teacher pooling. At km=128k_m=128 and b=4b=4, we compare the max-pooled teacher in Equation 15, which follows QSA (Qiu et al., 2026), with a sum-pooled teacher that uses ∑i∈ℬuat,i _i _ua_t,i as each block’s unnormalized weight. Both runs use the same training setup, differing only in teacher pooling. Max pooling improves the mean RULER score from 71.49% to 72.39% and the 128K score from 61.68% to 65.01% (Table 8(b)). Retrieval likely depends on a few salient tokens, whose signal sum pooling dilutes with diffuse attention. These results confirm QSA’s choice on the KSA backbone. Failure mode. The largest 128K subtask decrease is frequent-word extraction, from 45.00% to 26.67% (Table 11), an aggregation task. Backbone for sparse adaptation. Table 8(c) compares S-KSA adapted on Hybrid KSA and C-KSA+Full with km=128k_m=128. Both use b=4b=4, a max-pooled teacher, and 18.5B tokens of module-only sparse adaptation with the same remote budget, local window, and sink settings. C-KSA+S-KSA achieves a mean RULER score of 72.39 and a 128K score of 65.01, compared with 71.84 and 61.06 for KSA+S-KSA. Relative to their respective dense references, the 128K reduction is 1.30 points for C-KSA versus 4.29 for Hybrid KSA, indicating better retention of the backbone’s retrieval performance in this comparison. Figure 13(c) provides a complementary observation: at the same token budget, oracle block selection retains slightly more decode attention mass on C-KSA than on Hybrid KSA, including local context and sinks. This is consistent with the smaller retrieval loss after sparse adaptation and suggests that the compressed backbone remains well suited to sparse original-text access. Together, these results support the compatibility of hierarchical compression and sparse attention. Sparse selection using Full attention scores. To test whether sparse attention requires backbone adaptation, we freeze all model parameters and select remote tokens or blocks directly using Full attention probabilities averaged across heads. Token selection ranks individual probabilities; block selection ranks the maximum probability within each block of b=4b=4 tokens, matching the Indexer’s teacher pooling rule. We call this oracle selection and compare it with the same backbone’s unchanged Full layers. The remote budget K includes eight sink tokens; a 1024-token local window is retained separately. Figure 13: Oracle selection at 128K with frozen Hybrid KSA (blue) and C-KSA+Full (km=128k_m=128, red). (a,b) RULER scores. (c) Decode attention mass retained by token and block selection (b=4b=4), including local context and sinks before sparse renormalization. Table 9: Dense and oracle RULER results with frozen backbones. Oracle selection uses K=4096K=4096; Mean averages all six context lengths (4K–128K). Backbone Dense Token oracle Block-4 oracle Mean 128K Mean 128K Mean 128K Hybrid KSA 72.61 65.35 73.12 65.13 73.05 65.00 C-KSA+Full, km=16k_m=16 72.74 67.46 73.24 68.94 73.17 69.41 C-KSA+Full, km=32k_m=32 72.56 66.90 72.69 66.93 72.81 67.22 C-KSA+Full, km=64k_m=64 72.40 67.04 72.87 66.56 72.63 66.89 C-KSA+Full, km=128k_m=128 72.68 66.31 72.84 66.67 72.66 66.24 Figure 13 shows rapid recovery toward dense performance as the remote budget increases on both Hybrid KSA and C-KSA with km=128k_m=128. On C-KSA, block-level selection reaches 64.71 at K=512K=512. At K=4096K=4096, the selected remote tokens and the 1024-token local window together cover only about 3.9% of a 128K context, yet retain 82.83% of decode attention mass. The corresponding RULER score is 66.24, compared with 66.31 for dense attention. This concentration of attention on a small subset of tokens helps explain why sparse attention can preserve retrieval performance when relevant tokens are selected. At the same budget, Hybrid KSA retains 81.71% of decode attention mass. Local context and sinks alone already retain substantial attention mass but yield poor retrieval performance, highlighting the importance of selecting relevant remote content. Table 9 extends the comparison to Hybrid KSA and four C-KSA+Full backbones at K=4096K=4096, reporting mean RULER performance across all six context lengths and the 128K score. Both selection methods remain within 0.51 percentage points of the dense references in mean RULER score across all five backbones. Accurate selection therefore recovers near-dense RULER performance without backbone updates in these settings, including on models already adapted to C-KSA. Full-parameter sparse continuation also showed no clear RULER improvement under the training recipe and budget we tested. Together with the recovery observed after module-only training in Section 3.3, these results support focusing sparse adaptation on the aggregation module and Indexer rather than requiring backbone updates. Figure 14: Per-layer attention and block selection on a shared 16K NIAH example for Hybrid KSA (left) and C-KSA+S-KSA (right). Teal strips mark selected remote blocks, dashed lines mark the answer, and gray shading denotes the local window. Attention patterns and block selection. Figure 14 compares layer-wise attention in Hybrid KSA and our main C-KSA+S-KSA model on a shared 16K NIAH example, using km=128k_m=128, b=4b=4, and K=4096K=4096 for the latter. Both models correctly retrieve the answer from outside the local window. We visualize attention when predicting the first answer digit. For C-KSA, the plotted attention probabilities include the log⁡km k_m correction to Meta logits before softmax. For S-KSA, we plot dense attention probabilities and mark the text blocks selected by the Indexer. For the visualized query, Meta entries receive 7.7–13.7% of C-KSA’s attention mass in the displayed shallow layers, compared with 73.1–87.8% in the middle and deep layers, showing substantial use of compressed remote memory. In all three displayed S-KSA layers, the selected blocks cover the answer span. Together with local context, these blocks retain 87.6–97.3% of the dense attention mass while accessing only 31.5% of the original-text positions. Continued pretraining of Hybrid KSA. Because our adapted models receive additional training while the reported baselines do not, we examine whether continued pretraining of the unchanged Hybrid KSA backbone would provide a stronger quality reference. Table 3.7 reports a full-parameter continuation experiment from the same 128K checkpoint, using a global batch size of 256, a peak learning rate of 4×10−44× 10^-4, and a warmup–stable–decay schedule. Across five evaluated checkpoints covering 5.7–51.3B additional tokens, the six-length RULER mean remains below the original checkpoint’s 72.60. The 128K score improves at 28.5B tokens but falls back at later checkpoints. Under the tested recipe and data budgets, continued pretraining does not yield a consistent improvement in Hybrid KSA’s RULER performance. This supports using the original checkpoint as the quality reference for assessing whether architectural adaptation can preserve its capabilities at lower inference cost. Table 10: RULER scores after continued pretraining of Hybrid KSA. Added tokens (B) Mean 128K 0 (original) 72.60 65.27 5.7 70.84 63.26 17.1 70.77 63.72 28.5 71.86 66.76 39.9 69.74 63.17 51.3 71.95 63.62 4 Related Work Multi-head attention (MHA) is a fundamental component of Transformer language models, with separate query, key, and value projections for each head (Vaswani et al., 2017). Full attention provides direct access to preceding tokens, but its computation during training and prefill grows quadratically with sequence length, while its KV cache grows linearly during autoregressive decoding. FlashAttention reduces memory traffic for exact attention (Dao et al., 2022), while PagedAttention improves KV-cache allocation and sharing (Kwon et al., 2023). These system-level optimizations complement research on memory compression, sparse attention, and hybrid architectures. 4.1 Memory Compression One line of memory compression reduces storage at each token position. Multi-query attention (MQA) shares a single KV head across all query heads (Shazeer, 2019), while grouped-query attention (GQA) shares KV heads within groups of query heads (Ainslie et al., 2023). Multi-head latent attention (MLA) stores compact per-token latent states (DeepSeek-AI et al., 2024), and quantization methods such as KIVI reduce the precision of cached entries (Liu et al., 2024). These approaches reduce storage per position without removing historical positions. Other methods retain only a subset of token-level KV entries: H2O (Zhang et al., 2023) and SnapKV (Li et al., 2024a) use attention statistics to identify important positions, while StreamingLLM (Xiao et al., 2024) retains initial attention sinks together with a recent window. Another direction compresses historical context into fixed-size recurrent states rather than retaining token-level KV entries. Linear Transformers (Katharopoulos et al., 2020) achieve this through kernel feature maps and recurrent accumulation of key–value statistics. Subsequent designs introduce decay in RetNet (Sun et al., 2023), data-dependent gating in GLA (Yang et al., 2024), and gated delta-rule updates in Gated DeltaNet (Yang et al., 2025). Kimi Delta Attention (KDA) (Kimi Team et al., 2025) further extends Gated DeltaNet with finer-grained gating for recurrent memory updates. Related approaches include RWKV (Peng et al., 2024), with matrix-valued recurrent states, and selective state-space models such as Mamba (Gu & Dao, 2024) and Mamba-2 (Dao & Gu, 2024). In their recurrent forms, these models maintain a state size independent of context length, but access historical information through aggregated states rather than separately retained memory entries. Sequence-level aggregation represents multiple context positions with fewer memory entries that remain accessible to attention. Compressive Transformer (Rae et al., 2020) compresses older hidden-state memories, while Dynamic Memory Compression (Nawrot et al., 2024) learns adaptive KV-cache compression. Learned summary and memory representations are explored in Gist (Mu et al., 2023), ICAE (Ge et al., 2024), and UniGist (Deng et al., 2025). AutoCompressors (Chevalier et al., 2023) uses summary vectors as soft prompts, whereas Activation Beacon (Zhang et al., 2025a) retains compressed KV activations at each layer. GMSA (Tang et al., 2026) combines group merging and layer semantic alignment in an encoder–decoder context-compression framework. Cartridges (Eyuboglu et al., 2026) instead optimizes corpus-specific KV representations offline for reuse across subsequent queries. KSA (Chu et al., 2026) interleaves Summary tokens with fixed-size text chunks. In KSA layers, text queries directly access nearby text and retrieve distant context through Summary KV, while the Summary hidden states continue to be updated across layers. A systematic study of gist-based compression identifies task-dependent limitations, particularly in synthetic recall (Deng et al., 2025a). These findings underscore the importance of evaluating retained capabilities rather than compression ratio alone. Other designs organize compressed memory at multiple resolutions. MELODI (Chen et al., 2025) combines recurrent short-term memory with further-compressed long-term memory constructed at an intermediate layer. AdmTree (Li et al., 2025) organizes gist representations into adaptive semantic trees for hierarchical context compression. H-SSA (Mao et al., 2026) inserts meta-gist tokens to summarize groups of gist tokens and support coarse-to-fine retrieval. DeepSeek-V4 (DeepSeek-AI et al., 2026) applies learned dimension-wise aggregation to text-token KV, using different compression granularities in Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). C-KSA instead applies dimension-wise aggregation to Summary KV, forming Meta entries used only within the layer that constructs them, rather than introducing additional tokens into the hidden-state sequence. This further reduces distant-history storage without coarsening the Summary sequence passed between layers, and permits each layer to choose its own KV compression ratio. 4.2 Sparse Attention Sparse attention reduces computation by restricting the positions read by each query. Sparse Transformer (Child et al., 2019) uses structured sparse factorizations, Longformer (Beltagy et al., 2020) combines local windows with global attention, and BigBird (Zaheer et al., 2020) additionally incorporates random connections. Sliding-window attention, used in Mistral (Jiang et al., 2023), restricts each layer’s direct access to nearby tokens. Another line exploits attention sparsity in pretrained models without additional training. QUEST (Tang et al., 2024) estimates the relevance of KV-cache pages to each query and reads only the selected pages. MInference (Jiang et al., 2024) combines head-specific pattern templates with input-dependent sparse indices to accelerate prefill, while SpargeAttention (Zhang et al., 2025) and XAttention (Xu et al., 2025) estimate attention importance to skip low-contribution computations. These methods sparsify attention through prescribed patterns or inference-time selection, without training an auxiliary routing module. Content-dependent sparse architectures integrate routing into the attention mechanism. SeerAttention (Gao et al., 2025) learns block-level gates by self-distilling dense attention, while DeepSeek Sparse Attention (DSA) (DeepSeek-AI et al., 2025) uses a Lightning Indexer to select individual tokens. At block granularity, MoBA (Lu et al., 2025) scores mean-pooled token keys, whereas Qwen Sparse Attention (QSA) (Qiu et al., 2026) mean-pools auxiliary index keys within micro-blocks; both attend to the selected blocks’ original-token KV. Native Sparse Attention (NSA) (Yuan et al., 2025) uses learned block compression to guide fine-grained selection and combines compressed, selected, and local attention branches. In contrast, DeepSeek-V4’s CSA (DeepSeek-AI et al., 2026) selects compressed KV entries for its remote attention. S-KSA also reduces routing to block granularity, but reuses existing KSA Summaries as aggregation queries to construct routing representations and retains original-token KV for the main attention. Learned memory tokens can also provide an interface for block retrieval. Landmark Attention (Mohtashami & Jaggi, 2023) represents each context block with a landmark token and learns to retrieve blocks through attention to these landmarks. Simplified Sparse Attention (SSA) (Mao et al., 2026) scores gist keys and unfolds selected chunks to recover their original tokens. HiLS-Attention (Hu et al., 2026) uses landmark queries to pool token keys, then combines within-chunk attention outputs using entropy-corrected chunk weights. This allows the language-modeling loss to train routing; SAS (Li et al., 2026) similarly learns context ranking through differentiable gates inside attention normalization. In S-KSA, Summaries guide auxiliary aggregation rather than serving directly as routing keys. We train the aggregation modules and Indexer by attention distillation and use the resulting scores only for block selection. The main attention applies a single softmax over selected original tokens and local context, without using routing scores as additional attention weights. 4.3 Hybrid Architectures Hybrid architectures seek to retain the fine-grained retrieval of attention while reducing the cost of processing long contexts. A common strategy interleaves several efficient layers with a smaller number of full-attention layers. Hybrid H3 (Fu et al., 2023), Mamba-2-Hybrid (Waleffe et al., 2024), and Jamba (Lenz et al., 2025) combine state-space layers with attention, while Kimi Linear (Kimi Team et al., 2025) combines KDA with MLA. Local attention provides another complementary component: RecurrentGemma (Botev et al., 2024) and Samba (Ren et al., 2025) pair it with recurrent or state-space layers, whereas Gemma 3 (Gemma Team et al., 2025) and MiMo-V2-Flash (Xiaomi LLM-Core Team et al., 2026) interleave it with global attention. Beyond layer-wise interleaving, other designs exploit sharing or finer-grained mixing. Zamba (Glorioso et al., 2024) reuses a shared attention module at multiple depths of a Mamba backbone. YOCO (Sun et al., 2024) stacks a cross-decoder on an efficient self-decoder and shares a global KV cache across cross-decoder layers. HySparse (Gao et al., 2026) reuses token selections and global KV from full-attention layers in subsequent sparse layers, while DeepSeek-V4.1 (DeepSeek-AI, 2026) introduces cross-layer KV and index reuse within compressed sparse attention. At a finer granularity, Hymba (Dong et al., 2025) runs attention and state-space heads in parallel within the same layer. Hybrid KSA (Chu et al., 2026) combines Summary-based KSA layers with full-attention layers in a 3:13:1 layout. KSA V2 retains this layer-wise structure while improving both components: C-KSA further compresses the Summary KV used for distant-history access, and S-KSA replaces dense attention over original text with Summary-guided block-sparse attention. 5 Conclusion and Future Work We showed that the Summaries of a summary-token hybrid can serve both as compressible memory and as a routing index. C-KSA compresses Summary KV within each layer without coarsening the Summaries passed between layers, and S-KSA replaces full attention with Summary-guided block-sparse attention over original text. Retrofitted into a pretrained 1.9B Hybrid KSA model, KSA V2 retains its retrieval performance and largely preserves general capabilities, while reducing cache size and improving inference throughput. Additional experiments on a 4B GQA backbone retain near-parent RULER performance, supporting transfer to a larger model with shared KV heads. The remaining cost lies mainly in the original-text KV stored by the sparse layers, a natural target for future work. Our evaluation covers 1.9B MHA and 4B GQA backbones with contexts up to 128K. General-capability evaluations and efficiency measurements are limited to 1.9B; the 4B validation covers RULER retrieval. Future work will explore longer contexts and larger models, including whether additional compression levels can provide further savings while preserving useful information. We will also explore how to allocate compression factors across layers. Evaluating long-document understanding and multi-step reasoning will also help establish how well the observed retrieval performance transfers to broader long-context tasks. 6 Author List and Acknowledgement Core Contributors Chenglong Chu, Guohong Mu, Guowang Zhang, Hao Peng, Hui Wang, Jian Liang, Jiangxia Cao† , Jiao Ou, Jiaxin Deng, Lu Ren*, Muhao Wei, Ruiming Tang*, Tianle Xu, Wenwu Ou, Xingmei Wang, Zhixin Ling, Zhiyuan Liang*, Ziqi Wang* Contributors Chaoyi Ma, Haixiao Gao, Kuo Cai, Renpu Liu, Ruochen Yang, Wenlong Xu, Xinqi Jin, Yan Sun, Yifan Wang, Yingzhi He, Yiyu Wang, Yusen Huo, Zhaojie Liu, Zihan Zhou All the authors listed alphabetically by first name. *individuals who have departed from our team. † Leader: caojiangxia@kuaishou.com References Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan. Gomez, Łukasz Kaiser and Illia Polosukhin “Attention is All you Need” In Advances in Neural Information Processing Systems 30, 2017 URL: https://proceedings.neurips.c/paper_files/paper/2017/hash/3f5e243547dee91fbd053c1c4a845a-Abstract.html Child et al. (2019) Rewon Child, Scott Gray, Alec Radford and Ilya Sutskever “Generating Long Sequences with Sparse Transformers” In arXiv preprint arXiv:1904.10509, 2019 URL: https://arxiv.org/abs/1904.10509 Shazeer (2019) Noam Shazeer “Fast Transformer Decoding: One Write-Head is All You Need” In arXiv preprint arXiv:1911.02150, 2019 URL: https://arxiv.org/abs/1911.02150 Beltagy et al. (2020) Iz Beltagy, Matthew. Peters and Arman Cohan “Longformer: The Long-Document Transformer” In arXiv preprint arXiv:2004.05150, 2020 URL: https://arxiv.org/abs/2004.05150 Katharopoulos et al. (2020) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas and François Fleuret “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention” In International Conference on Machine Learning PMLR, 2020, p. 5156–5165 URL: https://proceedings.mlr.press/v119/katharopoulos20a.html Rae et al. (2020) Jack. Rae, Anna Potapenko, Siddhant. Jayakumar, Chloe Hillier and Timothy. Lillicrap “Compressive Transformers for Long-Range Sequence Modelling” In International Conference on Learning Representations, 2020 URL: https://openreview.net/forum?id=SylKikSYDH Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang and Amr Ahmed “Big Bird: Transformers for Longer Sequences” In Advances in Neural Information Processing Systems 33, 2020, p. 17283–17297 URL: https://proceedings.neurips.c/paper_files/paper/2020/hash/c8512d142a2d849725f31a9a7a361ab9-Abstract.html Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le and Charles Sutton “Program Synthesis with Large Language Models” In arXiv preprint arXiv:2108.07732, 2021 URL: https://arxiv.org/abs/2108.07732 Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pinto, Jared Kaplan, Harri Edwards, Yuri Burda and Nicholas Joseph “Evaluating Large Language Models Trained on Code” In arXiv preprint arXiv:2107.03374, 2021 URL: https://arxiv.org/abs/2107.03374 Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse and John Schulman “Training Verifiers to Solve Math Word Problems” In arXiv preprint arXiv:2110.14168, 2021 URL: https://arxiv.org/abs/2110.14168 Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song and Jacob Steinhardt “Measuring Massive Multitask Language Understanding” In International Conference on Learning Representations, 2021 URL: https://openreview.net/forum?id=d7KBjmI3GmQ Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song and Jacob Steinhardt “Measuring Mathematical Problem Solving With the MATH Dataset” In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, 2021 URL: https://datasets-benchmarks-proceedings.neurips.c/paper_files/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra and Christopher Ré “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness” In Advances in Neural Information Processing Systems 35, 2022, p. 16344–16359 DOI: 10.52202/068431-1189 Ainslie et al. (2023) Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron and Sumit Sanghai “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints” In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, p. 4895–4901 DOI: 10.18653/v1/2023.emnlp-main.298 Chevalier et al. (2023) Alexis Chevalier, Alexander Wettig, Anirudh Ajith and Danqi Chen “Adapting Language Models to Compress Contexts” In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, p. 3829–3846 DOI: 10.18653/v1/2023.emnlp-main.232 Fu et al. (2023) Daniel. Fu, Tri Dao, Khaled. Saab, Armin. Thomas, Atri Rudra and Christopher Ré “Hungry Hungry Hippos: Towards Language Modeling with State Space Models” In International Conference on Learning Representations, 2023 URL: https://openreview.net/forum?id=COZDy0WYGg Huang et al. (2023) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, jiayi lei, Yao Fu, Maosong Sun and Junxian He “C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models” In Advances in Neural Information Processing Systems 36, 2023, p. 62991–63010 DOI: 10.52202/075280-2749 Jiang et al. (2023) Albert. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Chaplot, Diego Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix and William Sayed “Mistral 7B” In arXiv preprint arXiv:2310.06825, 2023 URL: https://arxiv.org/abs/2310.06825 Kamradt (2023) Greg Kamradt “Needle In A Haystack - Pressure Testing LLMs” Evaluation repository, accessed September 17, 2026, Official website / repository, 2023 URL: https://github.com/gkamradt/LLMTest_NeedleInAHaystack Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joseph. Gonzalez, Hao Zhang and Ion Stoica “Efficient Memory Management for Large Language Model Serving with PagedAttention” In Proceedings of the 29th Symposium on Operating Systems Principles, 2023, p. 611–626 DOI: 10.1145/3600006.3613165 Mohtashami & Jaggi (2023) Amirkeivan Mohtashami and Martin Jaggi “Random-Access Infinite Context Length for Transformers” In Advances in Neural Information Processing Systems 36, 2023, p. 54567–54585 DOI: 10.52202/075280-2378 Mu et al. (2023) Jesse Mu, Xiang Li and Noah Goodman “Learning to Compress Prompts with Gist Tokens” In Advances in Neural Information Processing Systems 36, 2023, p. 19327–19352 DOI: 10.52202/075280-0848 Sun et al. (2023) Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang and Furu Wei “Retentive Network: A Successor to Transformer for Large Language Models” In arXiv preprint arXiv:2307.08621, 2023 URL: https://arxiv.org/abs/2307.08621 Wei et al. (2023) Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong and Bin Wang “CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?” In arXiv preprint arXiv:2306.16636, 2023 URL: https://arxiv.org/abs/2306.16636 Zhang et al. (2023) Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang"Atlas" Wang and Beidi Chen “H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models” In Advances in Neural Information Processing Systems 36, 2023, p. 34661–34710 DOI: 10.52202/075280-1506 Botev et al. (2024) Aleksandar Botev, Soham De, Samuel Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu and Pier Sessa “RecurrentGemma: Moving Past Transformers for Efficient Open Language Models” In arXiv preprint arXiv:2404.07839, 2024 URL: https://arxiv.org/abs/2404.07839 Dao & Gu (2024) Tri Dao and Albert Gu “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality” In International Conference on Machine Learning PMLR, 2024, p. 10041–10071 URL: https://proceedings.mlr.press/v235/dao24a.html DeepSeek-AI et al. (2024) DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr and Chong Ruan “DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model” In arXiv preprint arXiv:2405.04434, 2024 URL: https://arxiv.org/abs/2405.04434 Ge et al. (2024) Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen and Furu Wei “In-context Autoencoder for Context Compression in a Large Language Model” In International Conference on Learning Representations, 2024, p. 2591–2607 URL: https://proceedings.iclr.c/paper_files/paper/2024/hash/0b276510ec2d3f6613a8b60c41f0438-Abstract-Conference.html Glorioso et al. (2024) Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim and Beren Millidge “Zamba: A Compact 7B SSM Hybrid Model” In arXiv preprint arXiv:2405.16712, 2024 URL: https://arxiv.org/abs/2405.16712 Gu & Dao (2024) Albert Gu and Tri Dao “Mamba: Linear-Time Sequence Modeling with Selective State Spaces” In Conference on Language Modeling, 2024 URL: https://openreview.net/forum?id=tEYskw1VY2 Hsieh et al. (2024) Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang and Boris Ginsburg “RULER: What’s the Real Context Size of Your Long-Context Language Models?” In Conference on Language Modeling, 2024 URL: https://openreview.net/forum?id=kIoBbc76Sy Jiang et al. (2024) Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang and Lili Qiu “MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention” In Advances in Neural Information Processing Systems 37, 2024, p. 52481–52515 DOI: 10.52202/079017-1663 Li et al. (2024) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan and Timothy Baldwin “CMMLU: Measuring massive multitask language understanding in Chinese” In Findings of the Association for Computational Linguistics: ACL 2024, 2024, p. 11260–11285 DOI: 10.18653/v1/2024.findings-acl.671 Li et al. (2024a) Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis and Deming Chen “SnapKV: LLM Knows What You are Looking for Before Generation” In Advances in Neural Information Processing Systems 37, 2024, p. 22947–22970 DOI: 10.52202/079017-0722 Liu et al. (2024) Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen and Xia Hu “KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache” In International Conference on Machine Learning PMLR, 2024, p. 32332–32344 URL: https://proceedings.mlr.press/v235/liu24bz.html Nawrot et al. (2024) Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan and Edoardo Ponti “Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference” In International Conference on Machine Learning PMLR, 2024, p. 37396–37412 URL: https://proceedings.mlr.press/v235/nawrot24a.html Peng et al. (2024) Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Kranthi GV, Haowen Hou, Satyapriya Krishna, Ronald McClelland Jr., Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Jian Zhu and Rui-Jie Zhu “Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence” In Conference on Language Modeling, 2024 URL: https://openreview.net/forum?id=soz1SEiPeq Su et al. (2024) Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo and Yunfeng Liu “RoFormer: Enhanced transformer with Rotary Position Embedding” In Neurocomputing 568, 2024, p. 127063 DOI: 10.1016/j.neucom.2023.127063 Sun et al. (2024) Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang and Furu Wei “You Only Cache Once: Decoder-Decoder Architectures for Language Models” In Advances in Neural Information Processing Systems 37, 2024, p. 7339–7361 DOI: 10.52202/079017-0235 Tang et al. (2024) Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci and Song Han “QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference” In International Conference on Machine Learning PMLR, 2024, p. 47901–47911 URL: https://proceedings.mlr.press/v235/tang24l.html Waleffe et al. (2024) Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi and Bryan Catanzaro “An Empirical Study of Mamba-based Language Models” In arXiv preprint arXiv:2406.07887, 2024 URL: https://arxiv.org/abs/2406.07887 Wang et al. (2024) Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue and Wenhu Chen “MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark” In Advances in Neural Information Processing Systems 37, 2024, p. 95266–95290 DOI: 10.52202/079017-3018 Xiao et al. (2024) Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han and Mike Lewis “Efficient Streaming Language Models with Attention Sinks” In International Conference on Learning Representations, 2024, p. 21875–21895 URL: https://proceedings.iclr.c/paper_files/paper/2024/hash/5e5fd18f863cbe6d8ae392a93fd271c9-Abstract-Conference.html Yang et al. (2024) Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda and Yoon Kim “Gated Linear Attention Transformers with Hardware-Efficient Training” In International Conference on Machine Learning PMLR, 2024, p. 56501–56523 URL: https://proceedings.mlr.press/v235/yang24ab.html Chen et al. (2025) Yinpeng Chen, DeLesley Hutchins, Aren Jansen, Andrey Zhmoginov, David Racz and Jesper Andersen “MELODI: Exploring Memory Compression for Long Contexts” In International Conference on Learning Representations, 2025, p. 67589–67611 URL: https://proceedings.iclr.c/paper_files/paper/2025/hash/a8e3197f627b9c88f86c0d3eb1ade5d7-Abstract-Conference.html DeepSeek-AI et al. (2025) DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu and Bowei Zhang “DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models” In arXiv preprint arXiv:2512.02556, 2025 URL: https://arxiv.org/abs/2512.02556 Deng et al. (2025) Chenlong Deng, Zhisong Zhang, Kelong Mao, Shuaiyi Li, Tianqing Fang, Hongming Zhang, Haitao Mi, Dong Yu and Zhicheng Dou “UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression” In Advances in Neural Information Processing Systems 38, 2025, p. 19775–19797 DOI: 10.52202/085713-0665 Deng et al. (2025a) Chenlong Deng, Zhisong Zhang, Kelong Mao, Shuaiyi Li, Xinting Huang, Dong Yu and Zhicheng Dou “A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression” In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, p. 4861–4879 DOI: 10.18653/v1/2025.acl-long.241 Dong et al. (2025) Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz and Pavlo Molchanov “Hymba: A Hybrid-head Architecture for Small Language Models” In International Conference on Learning Representations, 2025, p. 97729–97755 URL: https://proceedings.iclr.c/paper_files/paper/2025/hash/f32def07618040e540e0a6182e290562-Abstract-Conference.html Gao et al. (2025) Yizhao Gao, Zhichen Zeng, DaYou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden So, Ting Cao, Fan Yang and Mao Yang “SeerAttention: Self-distilled Attention Gating for Efficient Long-context Prefilling” In Advances in Neural Information Processing Systems 38, 2025, p. 55846–55869 DOI: 10.52202/085713-1869 Gemma Team et al. (2025) Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova and Alexandre Ramé “Gemma 3 Technical Report” In arXiv preprint arXiv:2503.19786, 2025 URL: https://arxiv.org/abs/2503.19786 Kimi Team et al. (2025) Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men and Songlin Yang “Kimi Linear: An Expressive, Efficient Attention Architecture” In arXiv preprint arXiv:2510.26692, 2025 URL: https://arxiv.org/abs/2510.26692 Lenz et al. (2025) Barak Lenz, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor and Clara Fridman “Jamba: Hybrid Transformer-Mamba Language Models” In International Conference on Learning Representations, 2025, p. 67959–67984 URL: https://proceedings.iclr.c/paper_files/paper/2025/hash/a9ed43fa31dc8b4a7d7a673d713dcb5f-Abstract-Conference.html Li et al. (2025) Yangning Li, Shaoshen Chen, Yinghui Li, Yankai Chen, Hai-Tao Zheng, Hui Wang, Wenhao Jiang and Philip Yu “AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees” In Advances in Neural Information Processing Systems 38, 2025, p. 40389–40415 DOI: 10.52202/085713-1348 Lu et al. (2025) Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Yutao Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang and Jiezhong Qiu “MoBA: Mixture of Block Attention for Long-Context LLMs” In Advances in Neural Information Processing Systems 38, 2025, p. 17790–17815 DOI: 10.52202/085713-0602 Ren et al. (2025) Liliang Ren, Yang Liu, Yadong Lu, yelong shen, Chen Liang and Weizhu Chen “Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling” In International Conference on Learning Representations, 2025, p. 53551–53575 URL: https://proceedings.iclr.c/paper_files/paper/2025/hash/84a7fc24ed52e8eff514c33e8ac76ea3-Abstract-Conference.html Xu et al. (2025) Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo and Song Han “XAttention: Block Sparse Attention with Antidiagonal Scoring” In International Conference on Machine Learning PMLR, 2025, p. 69819–69831 URL: https://proceedings.mlr.press/v267/xu25ag.html Yang et al. (2025) Songlin Yang, Jan Kautz and Ali Hatamizadeh “Gated Delta Networks: Improving Mamba2 with Delta Rule” In International Conference on Learning Representations, 2025, p. 29687–29707 URL: https://proceedings.iclr.c/paper_files/paper/2025/hash/4904fad153f6434a7bcf04465d4be2c-Abstract-Conference.html Yuan et al. (2025) Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang and Wangding Zeng “Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention” In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, p. 23078–23097 DOI: 10.18653/v1/2025.acl-long.1126 Zhang et al. (2025) Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu and Jianfei Chen “SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference” In International Conference on Machine Learning PMLR, 2025, p. 76397–76413 URL: https://proceedings.mlr.press/v267/zhang25ch.html Zhang et al. (2025a) Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye and Zhicheng Dou “Long Context Compression with Activation Beacon” In International Conference on Learning Representations, 2025, p. 101932–101948 URL: https://proceedings.iclr.c/paper_files/paper/2025/hash/fc797c61eb18f7b3ce9a74af0ef0d876-Abstract-Conference.html Chu et al. (2026) Chenglong Chu, Guorui Zhou, Guowang Zhang, Han Li, Hao Peng, Hongtao Cheng, Hui Wang, Jian Liang and Jiangxia Cao “Kwai Summary Attention Technical Report” In arXiv preprint arXiv:2604.24432, 2026 URL: https://arxiv.org/abs/2604.24432 DeepSeek-AI (2026) DeepSeek-AI “DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression” In arXiv preprint arXiv:2609.19969, 2026 URL: https://arxiv.org/abs/2609.19969 DeepSeek-AI et al. (2026) DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang and Chaofan Lin “DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence” In arXiv preprint arXiv:2606.19348, 2026 URL: https://arxiv.org/abs/2606.19348 Eyuboglu et al. (2026) Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Atri Rudra, James Zou, Azalia Mirhoseini and Christopher Re “Cartridges: Lightweight and general-purpose long context representations via self-study” In International Conference on Learning Representations, 2026, p. 42642–42687 URL: https://proceedings.iclr.c/paper_files/paper/2026/hash/4681359a7b1e94571598ad1adda35e6e-Abstract-Conference.html Gao et al. (2026) Yizhao Gao, Jianyu Wei, Qihao Zhang, Yu Cheng, Shimao Chen, Zhengju Tang, Zihan Jiang, Yifan Song, Hailin Zhang, Liang Zhao, Bo Yang, Gang Wang, Shijie Cao and Fuli Luo “HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing” In arXiv preprint arXiv:2602.03560, 2026 URL: https://arxiv.org/abs/2602.03560 Hu et al. (2026) Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, Tian Liang, Huayang Li, Lei Zhu, Yan Wang, Sirui Han, Yushi Bai, Kewei Tu, Haitao Mi and Leo Liang “Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling” In arXiv preprint arXiv:2607.02980, 2026 URL: https://arxiv.org/abs/2607.02980 Li et al. (2026) Zhiwei Li, Lei Zhu, Hao Gu, Xiang Hu, Yan Wang, Haitao Mi, Sirui Han, Leo Liang and Zhijiang Guo “SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking” In arXiv preprint arXiv:2609.13141, 2026 URL: https://arxiv.org/abs/2609.13141 Mao et al. (2026) Yuzhen Mao, Michael. Li and Emily. Fox “Simplified Sparse Attention via Gist Tokens” In arXiv preprint arXiv:2604.20920, 2026 URL: https://arxiv.org/abs/2604.20920v2 Qiu et al. (2026) Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men and Bochao Mao “On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability” In arXiv preprint arXiv:2608.30320, 2026 URL: https://arxiv.org/abs/2608.30320 Tang et al. (2026) Jiwei Tang, Zhicheng Zhang, Shunlong Wu, Jingheng Ye, Lichen Bai, Zitai Wang, Tingwei Lu, Lin Hai, Yiming Zhao, Hai-Tao Zheng and Hong-Gee Kim “GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment” In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026, p. 28690–28704 DOI: 10.18653/v1/2026.acl-long.1324 Xiaomi LLM-Core Team et al. (2026) Xiaomi LLM-Core Team, Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He and Chiheng Lou “MiMo-V2-Flash Technical Report” In arXiv preprint arXiv:2601.02780, 2026 URL: https://arxiv.org/abs/2601.02780

Instant research discovery