Paper deep dive
AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD
Sara Abdali, Jongwoo Ko, Pashmina Cameron
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/7/2026, 8:07:39 AM
Summary
The paper introduces AttSVD, a training-free, prompt-adaptive low-rank compression method for the Key-Value (KV) cache in autoregressive transformers. Unlike eviction-based methods, AttSVD retains all tokens by compressing them along the feature axis using an online, per-prompt truncated Singular Value Decomposition (SVD) guided by attention geometry. It employs two caching strategies (accumulating and streaming) and adaptive rank selection based on matrix energy, achieving up to 50% memory reduction while maintaining performance comparable to dense caches.
Entities (10)
Relation Signals (8)
AttSVD → targets → KV cache
confidence 95% · AttSVD: Prompt-Adaptive Low-Rank KV Cache Compression
AttSVD → uses → SVD
confidence 95% · AttSVD computes an online attention-aware factorization... an online, per-prompt truncated SVD
Sara Abdali → affiliatedwith → Microsoft Applied Sciences Group
confidence 90% · Affiliation: Microsoft Applied Sciences Group (ASG) Affiliation: {saraabdali...}
Jongwoo Ko → affiliatedwith → Microsoft Applied Sciences Group
confidence 90% · Affiliation: Microsoft Applied Sciences Group (ASG) Affiliation: {...jongwooko...}
Pashmina Cameron → affiliatedwith → Microsoft Applied Sciences Group
confidence 90% · Affiliation: Microsoft Applied Sciences Group (ASG) Affiliation: {...pcameron...}
AttSVD → evaluatedon → LongBench
confidence 90% · Across multiple models, on both an agentic benchmark and the full LongBench suite AttSVD stays on par with the dense cache
AttSVD → outperformsorimproves → SnapKV
confidence 80% · Most training-free remedies evict low-importance tokens... We instead keep every token... AttSVD stays on par with the dense cache while using up to 50% of the KV-cache memory.
AttSVD → outperformsorimproves →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free remedies evict low-importance tokens, an irreversible choice along the sequence axis. We instead keep every token and store it more cheaply along the "feature" axis. We therefore propose AttSVD, a new "interpretable" low-rank compression whose basis is derived from each prompt's own attention geometry: an online, per-prompt truncated SVD that keeps only the directions attention actually reads, cutting persistent per-head KV memory in proportion to the retained rank. We propose two decode-time caching strategies, accumulating and streaming, for short and long generation regimes. Furthermore, we propose two refinements that make compression adaptive. A per-matrix energy rule sizes the logit space and the attention mass independently. An attention-aware basis truncates only in the spaces attention actually reads, preserving both the attention logits and the attention output. The same factors also provide free, per-head interpretability insights into the effective rank and the geometry attention consumes. Across multiple models, on both an agentic benchmark and the full LongBench suite AttSVD stays on par with the dense cache while using up to 50% of the KV-cache memory.
Tags
Links
- Source: https://arxiv.org/abs/2610.06927v1
- Canonical: https://arxiv.org/abs/2610.06927v1
Trouble viewing inline? Open PDF directly →
Full Text
77,872 characters extracted from source content.
Expand or collapse full text
AttSVD: Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD Sara Abdali Jongwoo Ko Pashmina Cameron Affiliation: Microsoft Applied Sciences Group (ASG) Affiliation: saraabdali, jongwooko, pcameron@microsoft.com Abstract The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free remedies evict low-importance tokens, an irreversible choice along the sequence axis. We instead keep every token and store it more cheaply along the “feature” axis. We therefore propose AttSVD, a new interpretable low-rank compression whose basis is derived from each prompt’s own attention geometry: an online, per-prompt truncated SVD that keeps only the directions attention actually reads, cutting persistent per-head KV memory in proportion to the retained rank. We propose two decode-time caching strategies, accumulating and streaming, for short and long generation regimes. Furthermore, we propose two refinements that make compression adaptive. A per-matrix energy rule sizes the logit space and the attention mass independently. An attention-aware basis truncates only in the spaces attention actually reads, preserving both the attention logits and the attention output. The same factors also provide free, per-head interpretability insights into the effective rank and the geometry attention consumes. Across multiple models, on both an agentic benchmark and the full LongBench suite AttSVD stays on par with the dense cache while using up to ∼50% 50\% of the KV-cache memory. 1 Introduction Training-free KV-cache compression methods fall into two families. Eviction methods (e.g. H2O (Zhang et al., 2023), SnapKV (Li et al., 2024), KeyDiff (Park et al., 2025), KVzip (Kim et al., 2025)) keep a budget of tokens and drop the rest entirely; they differ only in how they score importance. Eviction is a hard, irreversible choice along the sequence axis: a wrongly dropped token can never be attended to again. Reconstruction methods (e.g. VECTOR (Lin et al., 2026), Palu (Chang et al., 2025)) instead keep all positions but store them approximately. We propose AttSVD, which is not only a feature-axis compression method but also introduces two decode-time caching strategies for different generation regimes. The core rationale builds on the observed fact that a prompt’s keys and values are effectively low-rank, the keys especially (Saxena et al., 2024; Sun et al., 2025); we extend this low-rank observation from the K and V spaces to the attention logit space and the attention mass space. AttSVD computes an online attention-aware factorization in the subspace attention actually reads (rather than by plain spectral energy), per layer and per key/value head, both at prefill (factoring the prompt bulk) and during decode (the streaming scenario), where generated tokens are factored once they age out of a short recent-token window so long generations stay compressed as well (Figure 1). Because the basis is the SVD of the prompt’s own activations, it is, by Eckart-Young (Eckart and Young, 1936), the best possible rank-r approximation of that prompt’s attention logits and attention mass, obtained with no calibration corpus and no training; no token is discarded, and every position is still attended to, just from reconstructed rank-r factors. Our contributions are: • A per-prompt, calibration-free low-rank KV cache that retains all token positions and is exact at full rank, with a dynamic per-matrix rank rule that sizes keys and values separately so memory adapts to each prompt instead of a fixed schedule; • An attention-aware basis that truncates in the subspace attention actually reads, preserving both the attention logits and the attention output; • Two decode-time caching strategies that both compress at prefill but differ at decode: an accumulating mode for long-prompt, short-decode scenarios (e.g., no-think mode) that keeps generated tokens exact, and a streaming mode for long-prompt, long-decode scenarios (e.g., think mode) that compresses aged-out tokens into thin blocks for a bounded cache. • Interpretability at no extra cost: because the basis is derived from each prompt’s own attention geometry, the retained per-matrix ranks and attention-mass profile also serve as per-head diagnostics, exposing each head’s effective-rank compressibility, query-subspace focus, and token saliency without calibration. 2 Related Work Most low-rank KV-cache methods extract the basis ahead of time and reuse it for every prompt. Weight decompositions factor the key/value projection weights offline (ASVD (Yuan et al., 2023), Palu (Chang et al., 2025), LoRC (Zhang et al., 2024)); calibration methods estimate the basis from a corpus (Eigen Attention (Saxena et al., 2024), ReCalKV (Yan et al., 2025), KQ-SVD (Lesens et al., 2026)); and cross-layer or trained methods share or learn it (xKV (Chang et al., 2026), MatryoshkaKV (Lin et al., 2025)). OjaKV (Zhu et al., 2026) is also online and per-context but approximate and variance-greedy. It keeps a running incremental-PCA basis seeded from a small calibration set, using attention only to pick which tokens stay full-rank rather than to shape its basis. How AttSVD differs. Prior methods either extract or learn a single basis reused for every prompt (the offline families above), or track an approximate running basis seeded from calibration (OjaKV). AttSVD differs on three axes. (i) Per-prompt and exact. It computes the Eckart-Young-optimal truncated SVD of this prompt’s keys and values at prefill, with no calibration corpus, no training, and no running-basis bookkeeping; the basis is then frozen during generation (unlike OjaKV’s seeded, incremental estimate). (i) Attention-guided, not variance-greedy. Rather than keeping the directions of largest K,VK,V variance, it truncates in the space attention actually consumes; weighting keys by the query covariance CQ=Q⊤QC_Q=Q Q (which preserves the logits QK⊤QK ) and values by per-token attention mass (which preserves the output PVPV), so the retained subspace is functional rather than merely descriptive. This is a distinct form of attention-awareness from KQ-SVD: rather than low-rank-approximating the products KQ⊤KQ and VWOVW_O learned offline from calibration, AttSVD keeps the ordinary per-matrix factors and merely reweights their SVD metric, so one calibration-free rule aligns both keys (via CQC_Q, preserving the scores) and values (via attention mass, preserving the output) with this prompt’s own attention. (i) Dynamic per-matrix rank. A spectral-energy rule factors each matrix independently from the data so that memory adapts to each prompt instead of following a fixed schedule. For a comprehensive related work discussion, see Appendix A. 3 Method Motivation: Existing KV-cache methods save memory by evicting tokens, permanently discarding their keys and values. We instead retain every token in a cheaper form, re-expanded transiently at inference time, with no eviction or training. Unlike other low-rank reconstruction methods, we aim at a low-rank compression that is per-matrix adaptive, calibration-free, prompt-adaptive, robust across prefill & decode length scenarios, and attention-importance aware. 3.1 Proposed AttSVD Setup and notation. Let K,V∈ℝS×dK,V ^S× d denote the keys and values for a prompt of length S, where d is the head dimension. Standard attention computes P=softmax(QK⊤/d+M)P=softmax\! (QK / d+M ) and outputs PVPV, where M is the causal mask and Q∈ℝS×dQ ^S× d the queries. A single head stores 2Sd2Sd entries (K and V), for a total KV-cache footprint of 2SdLH2SdLH across L layers and H key/value heads. In case of grouped-query attention (GQA), H is the number of key/value heads, typically a small multiple fewer than the number of query heads HqH_q; this matters for the attention-aware variant, where a key head is shared by several query heads. Throughout the paper, K denotes the post-RoPE keys as they sit in the cache,11 1 Rotary embeddings rotate Q and K but never V. so our attention-weighted factorization operates on the rotated keys; this differs from ShadowKV (Sun et al., 2025), which compresses the pre-RoPE keys because the position-dependent rotation spreads the keys across more singular directions, so the pre-RoPE keys are lower-rank and compress better, at the price of storing and replaying the rotation at reconstruction. This pre-RoPE trick does not apply to us, because we never run an SVD on raw K or V; we factor in the attention-logit and attention-mass geometry instead. The quantity our key basis must preserve is the logit inner product QK⊤QK , which is exact only when the keys carry the same rotation as the queries, so operating post-RoPE is required and a pre-RoPE key basis would be misaligned with the post-RoPE query metric that our attention-aware truncation uses. The modestly higher rK~r_ K this positional spreading induces is in turn absorbed by the per-prompt adaptive rank. Standard SVD on weighted inputs. AttSVD introduces no new decomposition. It runs the same standard truncated SVD as SVD on raw KV, only on attention-weighted K and V rather than on K and V directly, so it is standard SVD on weighted inputs, not a separate weighted-SVD routine. Figure 1: AttSVD keeps a thin, compressed side cache of attention-aware factors A,BA,B plus a small exact cache of recent tokens, cutting persistent memory to a fraction of dense while retaining all token positions. At decode, accumulating keeps every token exact (small decoded tokens); streaming compresses tokens leaving a sliding window, keeping the exact buffer bounded. 3.2 Per-prompt and attention-aware low-rank compression In general, for an S×dS× d matrix Z with S>dS>d the (full) singular value decomposition (SVD) is Z=UΣ⊤Z=U , where at full rank U∈ℝS×dU ^S× d, Σ=diag(σ1≥⋯≥σd≥0)∈ℝd×d =diag( _1≥·s≥ _d≥ 0) ^d× d, and ∈ℝd×dV ^d× d (so Z has at most d nonzero singular values). The rank-r truncation (r≤dr≤ d) Zr=U:,1:rΣ1:r:,1:r⊤=argminrank(X)≤r∥Z−X∥FZ_r\;=\;U_:,1:r\, _1:r\,V_:,1:r \;=\; _rank(X)≤ r\; Z-X _F (1) is, by the Eckart-Young theorem, the optimal rank-r approximation of Z, with error ∥Z−Zr∥F=(∑i>rσi2)1/2 Z-Z_r _F= ( _i>r _i^2 )^1/2, where we store the factors A=U:,1:rΣ1:r∈ℝS×r,B=:,1:r⊤∈ℝr×d,Z~=AB.A\;=\;U_:,1:r\, _1:r ^S× r, B\;=\;V_:,1:r ^r× d, Z\;=\;AB. (2) AttSVD is attention-aware, i.e. the SVD is taken in the score and output spaces: it preserves the attention logits QK⊤QK by weighting the keys in the query geometry CQ=Q⊤QC_Q=Q Q, and the output O=PVO=PV by weighting the values by their per-key attention mass, changing only the basis, not the factor shapes. SVD on raw K and V instead minimizes ∥K−Kr∥F K-K_r _F and ∥V−Vr∥F V-V_r _F, so it can spend rank on directions attention never queries; we truncate in the spaces attention uses, keeping the decomposed storage of Eq. (2) unchanged. Either way the decomposition is the same standard truncated SVD; only its input differs, raw versus attention-weighted K and V. Weighted keys via the query geometry. Replacing K by KrK_r perturbs the logits by Q(K−Kr)⊤/dQ(K-K_r) / d, so the relevant objective is Kr⋆=argminrank(X)≤r∥Q(K−X)⊤∥F=argminrank(X)≤r∥(K−X)CQ1/2∥F,CQ=Q⊤Q,K_r = _rank(X)≤ r Q(K-X) _F= _rank(X)≤ r (K-X)\,C_Q^1/2 _F, C_Q=Q Q, (3) the optimal rank-r approximation of K in the query-covariance metric CQC_Q (the two norms are equal since both are tr((K−X)CQ(K−X)⊤)tr\! ((K-X)\,C_Q\,(K-X) )). By the change of variables K~=KCQ1/2 K=K\,C_Q^1/2 the generalized Eckart-Young solution is the ordinary truncated SVD of K~ K mapped back: factor K~=UΣ⊤ K=U and store AK~=U:,1:rΣ1:rA_ K=U_:,1:r _1:r and BK~=:,1:r⊤CQ−1/2B_ K=V_:,1:r C_Q^-1/2, so that AK~BK~=Kr⋆A_ KB_ K=K_r in the original coordinates. Only the basis changes; the cache layout and decode reconstruction are identical. Weighted values via the attention mass. Replacing V by VrV_r perturbs the output by P(V−Vr)P(V-V_r). The exact minimizer of ∥P(V−Vr)∥F P(V-V_r) _F needs the S×S× S Gram P⊤P P; we use the diagonal (per-key mass) approximation ai=∑qPqia_i= _qP_qi, the total attention received by key i, giving Vr⋆=argminrank(X)≤r∑i=1Sai∥Vi−Xi∥22=argminrank(X)≤r∥diag(a)(V−X)∥F,V_r = _rank(X)≤ r _i=1^Sa_i\, V_i-X_i _2^2= _rank(X)≤ r ( a)\,(V-X) _F, (4) solved by a change of variables: form the row-weighted values V~=diag(a)V=UΣ⊤ V=diag( a)\,V=U , whose best rank-r fit is the standard truncated SVD V~r=U:,1:rΣ1:r:,1:r⊤ V_r=U_:,1:r _1:rV_:,1:r (Eq. 1); un-weighting its rows by diag(1/a)diag(1/ a) recovers the store AV~=diag(1/a)U:,1:rΣ1:rA_ V=diag(1/ a)\,U_:,1:r _1:r and BV~=:,1:r⊤B_ V=V_:,1:r , so that AV~BV~=diag(1/a)V~r=Vr⋆A_ VB_ V=diag(1/ a)\, V_r=V_r . In both cases the kept directions are exactly the ones attention reads; full proofs of both truncations are given in Appendix B. Obtaining Q and a. Both are available at prefill: we capture the post-RoPE queries with lightweight forward hooks (a short observation window per layer) and read the value mass a from one attention-shaped pass, with GQA expansion and small numerical guards for the un-weighting. This composes with any rank rule of Section 3.4: the weighting sets the basis, the rule sets the rank. 3.3 Two KV-cache compression regimes tailored to prefill and decode length AttSVD always compresses the prompt at prefill; the two regimes differ only in how the cache is maintained during decode. We first describe the shared prefill phase, then the two decode variants: an accumulating and a streaming cache. Prefill. Given the prompt’s K~,V~ K, V for a layer, split off a small recent window of the newest w tokens, kept exact; call the remaining bulk K(b),V(b)∈ℝ(S−w)×dK^(b),V^(b) ^(S-w)× d. AttentionSVD each via Eq. (2) into (AK~,BK~)(A_ K,B_ K) and (AV~,BV~)(A_ V,B_ V) and store only the factors in a compressed side cache, writing just the exact tail to the exact cache. The prefill forward itself is served with the exact K,VK,V that are already in hand; reconstruction at prefill would only inject a lossy step we do not need, so prefill output is bit-identical to the dense cache. The compressed factors are a lossy approximation, consulted only when those tokens are later reconstructed at decode. Exact recent-window tail. The newest w tokens are stored exactly rather than decomposed. They carry the highest-impact local context, and a handful of tokens cannot be usefully compressed by a rank-r SVD (and a single decode token is appended one at a time, where factoring is counter-productive). Setting w=0w=0 recovers a fully decomposed prefill. Decode. At each generation step the decomposed bulk is re-materialized transiently as K^(b)=AK~BK~∈ℝ(S−w)×d K^(b)=A_ KB_ K ^(S-w)× d and V^(b)=AV~BV~ V^(b)=A_ VB_ V (the stored B factors carry the reverse map, so these products approximate the original K(b),V(b)K^(b),V^(b), not K~,V~ K, V), concatenated with the exact tail and consumed by attention, which forms P from QK⊤QK and O=PVO=PV on the recovered K,VK,V; these tensors exist for one layer at a time and are freed on return, so the transient peak is (Sd)O(Sd), never the whole cache. Only the low-rank factors persist across steps. The two strategies differ in decode-time caching: Decode strategy 1: accumulating cache. The prefill factors are fixed and each new token is appended exactly to the cache, never compressed; no decode-time SVD is performed and the decode-factor side cache stays empty, so the cache grows with generation length. This is best when the prompt is long but generation is short (e.g. multiple-choice tasks or no-think mode), where the exact generated tokens stay a small fraction of the cache. Decode strategy 2: streaming cache (sliding window). Streaming AttSVD bounds the decode-time footprint by compressing during decode (Figure 1, bottom), holding the cache at a fixed size regardless of output length, which suits long think-mode generations. It maintains a sliding exact window of the most recent w tokens: each generated token is appended to the window, and once the window overflows by a full chunk of ℓ tokens (the streaming branch of Algorithm 1, App. C), the oldest ℓ tokens (still held exactly) are factored with the same decomposition into a per-layer list of frozen decode blocks, then dropped from the window. Decode attention runs on the full sequence, formed by concatenation (⊕ ) of prefill bulk ⊕ decode blocks ⊕ exact window in position order. Two properties keep it sound: aged-out tokens are factored from their exact values (never by compressing the already-lossy prefill bulk), so error does not compound across the generation, and each block is frozen, so a token is compressed at most once. The scheme composes with any rank rule of Section 3.4 and the attention-aware basis of Section 3.2. 3.4 Choosing the rank: compression-ratio or per-matrix energy AttSVD offers two strategies for setting the rank, each suited to a different use case: a (1) fixed compression ratio c when a specific memory budget is required, and a (2) per-matrix energy rule when factor denoising and minimizing reconstruction error matter more than hitting a target budget. Compression-ratio rank (unified). If an absolute rank r or a compression ratio c is supplied, both K and V use the same r (with r=round((1−c)d)r=round((1-c)d) for a ratio), recovering the basic decomposition of Eq. (2). This uniform setting trades away per-matrix adaptivity for an exact, predictable budget; note that c selects a rank (a fraction of the d feature directions), not a fraction of tokens, so all S tokens are retained. This is the operating point used for the matched-memory settings in Section D.3 (e.g. c=0.5⇒r=64c=0.5 r=64). The two ranks may also be set individually as explicit rK~r_ K and rV~r_ V, overriding the shared value; this exposes the per-matrix budget as a direct hyperparameter and enables controlled study of the asymmetric rK~≪rV~r_ K\! \!r_ V regime. Per-matrix energy rank (dynamic). Keys and values are not equally compressible. Building on the pre-RoPE-key observation of ShadowKV (Sun et al., 2025) that keys and values differ in their intrinsic compression capacity, the attention-weighted matrices we factor, K~ K and V~ V, likewise likely differ, so a single shared rank over-spends on K~ K and under-serves V~ V (or vice versa) and the two may need different ranks. Here AttSVD lets each matrix choose its own rank from its own spectrum. Ordering a matrix M’s singular values σ1≥σ2≥⋯ _1\!≥\! _2\!≥\!·s, we keep the fewest leading directions that retain a fraction τ of its energy ∥M∥F2=∑iσi2 M _F^2= _i _i^2, rM(τ)=minr:∑i≤rσi2(M)≥τ∑iσi2(M),M∈K~,V~,r_M(τ)\;=\; \\,r:\ _i≤ r _i^2(M)\ ≥\ τ _i _i^2(M)\, \, M∈\ K, V\, (5) the effective rank of M at tolerance τ (Roy and Vetterli, 2007) (default 0.990.99): a fast-decaying spectrum reaches τ in a few directions, a flat one needs many. Applying the same τ to both matrices yields per-matrix ranks rK~r_ K and rV~r_ V that adapt to each prompt’s own spectrum rather than being hand-set, and may differ from one another. Equation (5) gives one rank per (batch,head)(batch,head); since the cache stores a single rank per matrix, these are reduced by a configurable aggregator: max (the default; the largest, so every head retains at least τ), mean, or a percentile pNN (allowing a few low-energy heads to fall below τ to save memory). An optional rank_max caps the result; passing an explicit τ selects this dynamic mode even when a rank or ratio is otherwise set. 3.5 Memory cost The saving is explicit. A dense head stores K,V∈ℝS×dK,V ^S× d, i.e. 2Sd2Sd entries; AttSVD instead keeps the factors that reconstruct them. For the prompt bulk it stores the factors AK~∈ℝ(S−w)×rK~,BK~∈ℝrK~×dA_ K ^(S-w)× r_ K,B_ K ^r_ K× d for the keys and AV~∈ℝ(S−w)×rV~,BV~∈ℝrV~×dA_ V ^(S-w)× r_ V,B_ V ^r_ V× d for the values. A rank-r factorization of an (S−w)×d(S-w)× d matrix costs r(S−w+d)r\,(S-w+d) entries, so the two factor pairs together use (rK~+rV~)(S−w+d)(r_ K+r_ V)(S-w+d) entries, plus a short exact tail of the w most recent tokens (2wd2wd entries). The tail is (wd)O(wd) with w fixed and w≪Sw\! \!S, so it is negligible against the bulk; dropping it and using S−w≈S-w\!≈\!S leaves a persistent factor cost of (rK~+rV~)(S+d)(r_ K+r_ V)(S+d). Dividing by the dense 2Sd2Sd and expanding, (rK~+rV~)(S+d)2Sd=rK~+rV~2d(1+dS)=rK~+rV~2d+rK~+rV~2S→S≫drK~+rV~2d. (r_ K+r_ V)(S+d)2Sd\;=\; r_ K+r_ V2d (1+ dS )\;=\; r_ K+r_ V2d+ r_ K+r_ V2S\; [S d]\; r_ K+r_ V2d. (6) The residual (rK~+rV~)/(2S)(r_ K+r_ V)/(2S) term vanishes once the context is long (S≫dS d), leaving the compression ratio (rK~+rV~)/(2d)(r_ K+r_ V)/(2d). The asymmetric regime rK~≪rV~r_ K\! \!r_ V shrinks this further, while a shared rank rK~=rV~=r_ K=r_ V=r reduces it to simply r/dr/d. The recent window adds a further (w/S)O(w/S) term, and as context grows the measured ratio tightens to r/dr/d. At decode, the accumulating cache keeps every generated token exact, so its exact part is the recent tail plus all g generated tokens, ((w+g)d)O ((w+g)d ); the streaming cache instead bounds it, oscillating in [w,w+ℓ][w,\,w+ ] independent of g, while the compressed part grows by one (ℓr+rd)O( r+rd) block per ℓ tokens (≈r/d≈ r/d of dense per token once ℓ≫d d, the same asymptotic ratio as the prefill bulk). Streaming thus trades a small fixed exact buffer plus extra decode-time SVDs (one per compressed chunk) for its (g)O(g) tail, winning when g is large. The chunk ℓ sets the granularity. It must exceed the rank for the compression to be lossy, and larger ℓ relative to d tightens the storage toward its r/dr/d asymptote; taking w→∞w\!→\!∞ recovers the accumulating cache (further details in Table 1). We use ℓ=128 =128 in our experiments, above every rank we use so each aged-out chunk is genuinely truncated, and a larger ℓ is equally valid. 3.6 Interpretability: what the retained basis exposes Insight. By compressing in the score and output space, weighting K by the query covariance CQ=Q⊤QC_Q=Q Q and V by the per-token attention mass, AttSVD retains a head’s functional subspace. Because this basis is derived from the prompt’s own attention geometry rather than from fixed weights or calibration data, these same quantities also serve as free, per-head interpretability insights: they reveal how each head geometrically uses its context, rather than what it attends to semantically. (i) Effective rank. Roy and Vetterli (2007) observe that the effective rank of a matrix such as K or V measures its intrinsic dimensionality. Because our compression happens in the score and output geometry, the per-matrix ranks rK~,rV~r_ K,r_ V of Eq. (5) report the intrinsic dimensionality of the scores P (through the query-weighted keys K~ K) and of the output O (through the mass-weighted values V~ V). The per-matrix ranks can be read as the intrinsic compression capacity of each matrix, and the per-(layer, head) rank map shows which heads carry low-rank, highly compressible context. We empirically observe that, on the models we test, the query- and mass-weightings redistribute singular-value energy yet still leave a systematic gap rK~≪rV~r_ K\! \!r_ V (Appendix D, Figures 6 and 6). (i) Query subspace. The covariance CQ=Q⊤QC_Q=Q Q is the subspace the head’s queries actually probe; the decay of its spectrum quantifies how focused (few-directional) vs. diffuse a head’s attention is. (i) Token saliency. The per-key mass ai=∑qPqia_i= _qP_qi is a token-importance profile (the same heavy-hitter signal exploited by eviction (Zhang et al., 2023)) emerged here freely, so even an eviction-only pipeline can read off which tokens to keep and which to drop. We emphasize that this is geometric, not semantic. The basis vectors are singular directions, not named concepts, they mix position and content (post-RoPE), and they describe each prompt and head rather than explain a prediction. Within those limits, the effective-rank map, query-subspace concentration, and attention-mass profile are concrete, reusable quantities the compression produces at no cost (interpretability analysis in Appendix D). 4 Experiments Baselines. We compare AttSVD against three reference points at a token budget matched to AttSVD’s memory at each compression setting, so every method shares the same model, prompts, decoding, and cache size. The dense cache is the uncompressed upper bound. We compare against eviction methods: H2O (heavy-hitter attention mass), KeyDiff (attention-free cosine similarity), and KVzip (query-agnostic reconstruction scoring); and low-rank compression methods: SVD on raw KV (standard top-r truncated SVD applied directly to K and V, without AttSVD’s attention weighting), KQ-SVD (attention-aware but from an offline calibration set), and OjaKV (an online Oja’s-rule basis seeded from calibration). Datasets and benchmarks. We evaluate on two complementary benchmarks. (i) Agentic benchmark: 184184 prompts spanning 1313 common scenarios across three difficulty tiers, each annotated with the correct action to take (from ∼77 77 candidates) or no action when none applies, examining action selection and necessity. An LLM judge scores each output 11 to 55; we report the mean judge score and pass@4pass@4 (fraction ≥4≥ 4). Prompts are short but the reasoning-and-response generations are long (median ∼800 800 tokens, up to ∼10 10K). (i) LongBench (Bai et al., 2024): the full public long-context suite (1,7501,750 examples over 99 tasks), each scored by its native metric; we report per-task accuracy and the average (00 to 100100). Its long-context inputs and outputs stress the cache at real context length and exercise the streaming schedule’s sliding window. The two examine different axes: the agentic eval isolates answer quality, LongBench tests generality at long context, so agreement across both guards against over-reliance on either. Memory sums the actual stored tensor bytes (factors plus exact tail) from one prefill forward, cross-checked against the allocator’s peak (full setup and model choice in Appendix D). Table 1: Measured KV-cache memory on Phi-4-14B (40 layers, 40 query / 10 KV heads) and Qwen3-8B (36 layers, 32 query / 8 KV heads); d=128d=128, bf16. Phi-4-14B Qwen3-8B Rank r KV @32K (GiB) Measured ratio KV @9.4K (GiB) KV @32K (GiB) Measured ratio KV @9.4K (GiB) Dense cache 128128 6.2506.250 1.0001.000 1.8041.804 4.5004.500 1.0001.000 1.2991.299 AttSVD (c=0.50c=0.50) 6464 3.1393.139 0.5020.502 0.9160.916 2.2602.260 0.5020.502 0.6600.660 AttSVD (c=0.75c=0.75) 3232 1.5711.571 0.2510.251 0.4590.459 1.1311.131 0.2510.251 0.3300.330 AttSVD (c=0.875c=0.875) 1616 0.7870.787 0.1260.126 0.2310.231 0.5670.567 0.1260.126 0.1660.166 4.1 AttSVD vs. eviction methods On the agentic benchmark at the aggressive half-rank (50%50\% cache) point (Figure 2), attention-aware AttSVD-Stream reaches 4.074.07 mean judge / 62.5%62.5\% pass@4pass@4, edging out the strongest eviction baselines (H2O 4.044.04/59.0%59.0\%, KVzip 3.953.95/55.4%55.4\%, KeyDiff 3.903.90/51.1%51.1\%), while its AttSVD-Accum cache (4.014.01/59.2%59.2\%) is on par with H2O at half the memory. On Qwen3-8B (Figure 2) the mean-judge gaps among Full, AttSVD (Accum/Streaming) and the eviction baselines H2O/KVzip all fall within the reported 95%95\% CI, they are statistically tied with full attention, and only KeyDiff differs. The ordering is stable across the full rank range (Figure 3). The same holds for long context: on LongBench at the same 50%50\% cache point (Table 2), AttSVD-Accum leads every eviction baseline on Phi-4-14B (36.036.0 vs. KVzip 35.435.4, H2O 34.934.9, KeyDiff 33.533.5) and ties the strongest eviction methods on Qwen3-8B (KVzip 51.951.9, H2O 51.251.2), while staying within 0.70.7 of the dense cache. Figure 2: At the aggressive half-rank (50%50\% cache) point, AttSVD stays within noise of the dense cache and matches or beats the strongest baselines on both Phi-4-14B and Qwen3-8B. Figure 3: Agentic benchmark (n=184n=184): mean judge score (a, c) and pass@4pass@4 (b, d) as the cache is reduced from 100%100\% down to 50%50\%, on Phi-4-14B (a, b) and Qwen3-8B (c, d). AttSVD (Accum/Stream) follows the dense cache far more closely than SVD on raw KV across the rank range. Table 2: At 50%50\% cache on LongBench, AttSVD (Accum/Stream) matches the dense cache average on both Phi-4-14B and Qwen3-8B and leads every reconstruction and eviction baseline. Phi-4-14B Qwen3-8B Reconstruction Eviction Reconstruction Eviction LongBench task Dense AttSVD-Accum (ours) AttSVD-Stream (ours) SVD on raw KV KeyDiff KVzip H2O Dense AttSVD-Accum (ours) AttSVD-Stream (ours) SVD on raw KV KeyDiff KVzip H2O PassageRetrieval-en 98.598.5 98.598.5 98.598.5 99.099.0 98.098.0 98.598.5 96.096.0 100.0100.0 100.0100.0 100.0100.0 78.078.0 100.0100.0 100.0100.0 99.599.5 Qasper 27.927.9 26.926.9 28.128.1 21.821.8 23.423.4 28.128.1 26.726.7 41.041.0 41.441.4 41.441.4 37.037.0 30.730.7 41.241.2 37.537.5 MultiFieldQA-en 32.932.9 33.033.0 31.331.3 23.423.4 32.432.4 29.629.6 32.332.3 53.853.8 53.853.8 53.853.8 51.351.3 44.944.9 54.154.1 51.751.7 HotpotQA 21.421.4 20.720.7 20.820.8 14.814.8 20.820.8 20.520.5 20.420.4 59.459.4 59.159.1 59.159.1 53.553.5 56.856.8 58.958.9 58.358.3 2WikiMQA 36.536.5 35.635.6 35.835.8 21.921.9 27.227.2 33.333.3 33.833.8 43.043.0 43.343.3 43.343.3 38.138.1 40.940.9 44.044.0 43.043.0 GovReport 14.314.3 14.114.1 14.214.2 13.813.8 14.014.0 14.414.4 14.614.6 18.518.5 18.618.6 18.918.9 16.816.8 17.117.1 18.618.6 18.618.6 TriviaQA 28.628.6 26.626.6 26.826.8 20.320.3 26.826.8 27.627.6 27.327.3 88.588.5 89.089.0 89.089.0 87.187.1 85.285.2 88.588.5 88.588.5 TREC 52.552.5 53.353.3 51.951.9 52.352.3 47.847.8 58.558.5 51.251.2 59.559.5 57.757.7 57.757.7 25.825.8 35.135.1 62.162.1 61.161.1 PassageCount 17.517.5 15.515.5 14.014.0 12.012.0 11.011.0 8.08.0 12.012.0 2.52.5 3.53.5 3.53.5 2.52.5 6.06.0 0.00.0 2.52.5 average 36.736.7 36.036.0 35.7¯ 35.7 31.031.0 33.533.5 35.435.4 34.934.9 51.851.8 51.851.8 51.951.9 43.343.3 46.346.3 51.9¯ 51.9 51.251.2 4.2 AttSVD vs. other low-rank compression methods Against other low-rank / SVD compressors the basis is what matters, not just the rank. SVD on raw KV truncation trails even the eviction family at r=64r=64 on the agentic benchmark (3.793.79/41.3%41.3\%, Figure 2), reflecting that a softmax-pretrained model has not adapted to low-rank K/VK/V; attention-aware AttSVD closes this gap outright, adding +0.13+0.13 to 0.290.29 mean judge and +7+7 to 1818 points pass@4pass@4 over SVD on raw KV at matched memory across the rank range (Figure 3). The same holds on LongBench, where AttSVD (36.036.0) is the strongest reconstruction compressor and SVD on raw KV trails (31.031.0; Table 2). We further test two calibrated SVD baselines on a third backbone, Llama-3.1-8B-Instruct,22 2 We use Llama-3.1-8B-Instruct here because both baselines ship official code for it. KQ-SVD evaluates on LLaMA/Mistral, and OjaKV uses Llama-3.1-8B as its reference model. (Table 3): KQ-SVD and OjaKV. We run each with its official implementation and released settings, matched only in cache budget, so the numbers reflect the methods as released. AttSVD-Accum matches the dense cache almost exactly on the average (48.548.5 vs. 48.548.5 at 80%80\% cache; 47.847.8 vs. 48.548.5 at 60%60\%) and leads KQ-SVD (47.447.4 and 45.245.2) by 1.11.1 to 2.62.6 points, with the largest gaps on the QA tasks (HotpotQA, MultiFieldQA-en) where a fixed calibration basis transfers least well across prompts. Both AttSVD schedules lead the OjaKV baseline (42.942.9 at 80%80\%), which collapses to 30.530.5 at 60%60\%. These gaps show that AttSVD’s per-prompt, attention-aware basis generalizes better than both KQ-SVD and OjaKV, widening its advantage as the cache tightens. Table 3: On Llama-3.1-8B-Instruct, AttSVD matches the dense cache average at 80%80\% (48.548.5 vs. 48.548.5) and stays within 0.70.7 at 60%60\%, beating KQ-SVD and OjaKV at both matched cache points. 80%80\% cache (c=0.2c=0.2, r≈102r≈102) 60%60\% cache (c=0.4c=0.4, r≈77r≈77) LongBench task Dense AttSVD-Accum (ours) AttSVD-Stream (ours) KQ-SVD OjaKV Dense AttSVD-Accum (ours) AttSVD-Stream (ours) KQ-SVD OjaKV MultiFieldQA-en 53.553.5 53.353.3 53.353.3 50.550.5 48.348.3 53.553.5 51.251.2 51.251.2 48.148.1 45.645.6 HotpotQA 54.254.2 54.354.3 54.354.3 50.950.9 35.835.8 54.254.2 53.553.5 53.553.5 48.248.2 26.926.9 2WikiMQA 44.644.6 45.345.3 45.345.3 40.940.9 40.140.1 44.644.6 45.445.4 45.445.4 40.440.4 31.831.8 TREC 7.57.5 7.07.0 7.07.0 12.012.0 8.28.2 7.57.5 6.06.0 6.06.0 7.87.8 3.03.0 PassageCount 6.26.2 6.86.8 6.86.8 7.27.2 3.03.0 6.26.2 7.27.2 7.27.2 7.67.6 4.54.5 PassageRetrieval-en 99.599.5 99.599.5 99.599.5 99.599.5 89.089.0 99.599.5 99.599.5 99.599.5 99.599.5 36.236.2 Qasper 47.447.4 47.647.6 47.647.6 46.546.5 45.145.1 47.447.4 46.346.3 46.346.3 45.345.3 37.737.7 GovReport 34.734.7 33.833.8 33.833.8 32.232.2 32.632.6 34.734.7 31.331.3 31.331.3 26.426.4 26.326.3 TriviaQA 89.189.1 89.289.2 89.289.2 86.686.6 84.184.1 89.189.1 89.589.5 89.589.5 83.583.5 62.262.2 average 48.548.5 48.548.5 48.548.5 47.447.4 42.942.9 48.548.5 47.847.8 47.847.8 45.245.2 30.530.5 4.3 Dynamic energy-rule rank Section 3.4 sets rK~,rV~r_ K,r_ V per prompt from each matrix’s own spectrum at a single tolerance τ (Eq. (5)), with no fixed rank or ratio. On the agentic benchmark (Figure 4) this dynamic mode matches the dense cache within 0.050.05 mean judge by τ=0.999τ=0.999 (4.0984.098/64.7%64.7\% vs. 4.1454.145/66.8%66.8\%), picking rK~,rV~r_ K,r_ V per prompt with neither a rank nor a ratio preset. Running the same rule on the full LongBench suite (Phi-4-14B, attention-aware basis; Table 4), at τ=0.997τ=0.997 it reads rK~≈65r_ K\!≈\!65 and rV~≈127r_ V\!≈\!127 off the data, keys collapse to about half rank while values stay near-full, exactly the rK~≪rV~r_ K\! \!r_ V asymmetry of Section 3.4. At the operating point the data itself selects (≈75%≈\!75\% cache), the average (36.536.5) lands within 0.20.2 of the dense cache (36.736.7); the tolerance τ alone recovers near-exact quality while each prompt sets its own rank. For a detailed interpretability analysis on real models, see Appendix D. Figure 4: Under the dynamic energy rule, quality saturates by τ=0.99τ=0.99 and matches the dense cache from there on (agentic benchmark, n=184n=184). Table 4: The dynamic per-matrix energy rule picks rK~,rV~r_ K,r_ V from each prompt’s own spectrum at tolerance τ and reaches near-exact LongBench quality on Phi-4-14B and Qwen3-8B. Phi-4-14B Qwen3-8B LongBench task Dense Energy τ=0.997τ=0.997 Energy τ=0.995τ=0.995 Dense Energy τ=0.997τ=0.997 Energy τ=0.995τ=0.995 PassageRetrieval-en 98.598.5 99.599.5 98.598.5 100.0100.0 100.0100.0 100.0100.0 Qasper 27.927.9 27.027.0 28.728.7 41.041.0 40.540.5 41.241.2 MultiFieldQA-en 32.932.9 33.133.1 31.231.2 53.853.8 53.853.8 53.253.2 HotpotQA 21.421.4 21.421.4 22.722.7 59.459.4 59.359.3 59.359.3 2WikiMQA 36.536.5 35.835.8 35.635.6 43.043.0 43.543.5 43.343.3 GovReport 14.314.3 14.414.4 14.514.5 18.518.5 19.219.2 19.219.2 TriviaQA 28.628.6 28.328.3 28.428.4 88.588.5 88.588.5 88.588.5 TREC 52.552.5 53.953.9 52.952.9 59.559.5 60.160.1 61.461.4 PassageCount 17.517.5 15.515.5 15.015.0 2.52.5 3.03.0 3.53.5 average 36.736.7 36.536.5 36.4¯ 36.4 51.851.8 52.0¯ 52.0 52.252.2 5 Conclusion AttSVD shows that the KV cache can be compressed without discarding tokens: it retains every token yet stores each in an instance-optimal rank-r subspace obtained from an online, per-prompt SVD and reconstructed lazily at decode, so the persistent cache shrinks to ≈r/d≈r/d of dense. Its contributions are threefold: an online, calibration-free basis that is provably optimal (Eckart-Young) for the prompt at hand; adaptive per-matrix ranks and an attention-aware objective that spend the budget where attention actually concentrates; and accumulating and streaming decode modes for different generation scenarios. Because the factors are exact, they are also directly interpretable: the effective-rank map exposes the intrinsic compressibility of the query-weighted keys K~ K and mass-weighted values V~ V, the query subspace shows each head’s focus, and the attention-mass profile flags salient tokens, all without calibration. Empirically, across several models, AttSVD matches the dense cache on the agentic benchmark and LongBench at roughly half the KV-cache memory, positioning instance-optimal reconstruction as a strong, training-free alternative to eviction. Use of AI Tools All ideas presented in this paper are original to the authors, or build upon the related work cited herein. AI-based tools were used only in an assistive capacity: to improve the clarity of the authors’ initial statements, to help draft sections of the paper in LaTeX, to improve the quality of figures and tables, to correct grammatical errors, to make sentences more salient, and to serve as a coding assistant that helps implement the authors’ ideas. All technical content, claims, and conclusions were conceived, verified, and approved by the authors. Ethics Statement The authors confirm that this work adheres to established ethical standards for machine learning research. AttSVD is a training-free, inference-time KV-cache compression method. Because the method operates purely on the key and value caches of a frozen pretrained model, it introduces no new training signal; any bias or fairness limitations already present in the underlying language models are inherited unchanged, and AttSVD is designed to alter neither their behavior nor their outputs beyond compressing the cache. The authors report no competing interests and no external funding that could have influenced the findings. Reproducibility Statement We have taken concrete steps to make each reported result reproducible. The method, including the attention-guided per-prompt truncated SVD, the energy-based rank rule, and the accumulating and streaming decode strategies, is specified in Section 3, with the full procedures given as explicit pseudocode in Appendix C (Algorithms 1 and 2) and the attention-aware truncation derived with a complete proof in Appendix B. The experimental setup, including implementation details, baselines, models, and extended studies, is documented in Appendix D. To further support exact reproduction, we release the core code of AttSVD, the model-independent dynamic cache class (a DynamicCache subclass that implements Algorithms 1 and 2) together with its unit tests, as anonymized supplementary material, and defer the release of the full repository to acceptance. References Bai et al. (2024) Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li LongBench: a bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 3119–3137. External Links: Link, Document Cited by: §4. Benfenati et al. (2026) L. Benfenati, M. Risso, A. Vannozzi, A. C. Yüzügüler, L. Cavigelli, E. Macii, D. J. Pagliari, and A. Burrello Don’t be so stief! learning kv cache low-rank approximation over the stiefel manifold. arXiv preprint arXiv:2601.21686. Cited by: Appendix A. Chang et al. (2025) C. Chang, W. Lin, C. Lin, C. Chen, Y. Hu, P. Wang, N. Huang, L. Ceze, M. S. Abdelfattah, and K. Wu Palu: KV-cache compression with low-rank projection. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix A, Table 5, §1, §2. Chang et al. (2026) C. Chang, W. Lin, C. Lin, H. Chiang, Y. Akhauri, X. Dai, H. Jiang, Y. Li, L. Ceze, K. Wu, and M. S. Abdelfattah xKV: cross-layer KV-cache compression via aligned singular vector extraction. In Forty-third International Conference on Machine Learning, Cited by: Appendix A, Table 5, §2. Eckart and Young (1936) C. Eckart and G. Young The approximation of one matrix by another of lower rank. Psychometrika 1 (3), p. 211–218. Cited by: Appendix B, §1. Hao et al. (2026) J. Hao, Q. Huang, Y. Wang, M. Zhang, and J. Yu DeltaKV: residual-based KV cache compression via long-range similarity. arXiv preprint arXiv:2602.08005. Cited by: Appendix A. Hoerl and Kennard (1970) A. E. Hoerl and R. W. Kennard Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12 (1), p. 55–67. Cited by: Appendix B. Kim et al. (2025) J. Kim, J. Kim, S. Kwon, J. W. Lee, S. Yun, and H. O. Song KVzip: query-agnostic KV cache compression with context reconstruction. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Appendix A, §1. Lei and Ilager (2026) J. Lei and S. Ilager ARKV: adaptive and resource-efficient KV cache management under limited memory budget for long-context inference in LLMs. arXiv preprint arXiv:2603.08727. Cited by: Appendix A. Lesens et al. (2026) D. Lesens, B. T. Rakhshan, and G. Rabusseau KQ-SVD: compressing the KV cache with provable guarantees on attention fidelity. In The 29th International Conference on Artificial Intelligence and Statistics, External Links: Link Cited by: Appendix A, Table 5, §2. Li et al. (2024) Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen SnapKV: LLM knows what you are looking for before generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1. Lin et al. (2025) B. Lin, Z. Zeng, Z. Xiao, S. Kou, T. Hou, X. Gao, H. Zhang, and Z. Deng MatryoshkaKV:adaptive KV compression via trainable orthogonal projection. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix A, Table 5, §2. Lin et al. (2026) Y. Lin, J. Ding, Y. Xing, P. He, J. Tang, and S. Mukherjee A simple plug-in for improving eviction-based KV cache compression. arXiv preprint arXiv:2605.23258. Cited by: Appendix A, §1. Mirsky (1960) L. Mirsky Symmetric gauge functions and unitarily invariant norms. The Quarterly Journal of Mathematics 11 (1), p. 50–59. Cited by: Appendix B. Park et al. (2025) J. Park, D. Jones, M. J. Morse, R. Goel, M. Lee, and C. Lott KeyDiff: key similarity-based KV cache eviction for long-context LLM inference in resource-constrained environments. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Appendix A, §1. Roy and Vetterli (2007) O. Roy and M. Vetterli The effective rank: a measure of effective dimensionality. In European Signal Processing Conference (EUSIPCO), p. 606–610. Cited by: §3.4, §3.6. Saxena et al. (2024) U. Saxena, G. Saha, S. Choudhary, and K. Roy Eigen attention: attention in low-rank space for KV cache compression. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 15332–15344. External Links: Link, Document Cited by: Appendix A, Table 5, Appendix B, §1, §2. Shutova et al. (2025) A. Shutova, V. Malinovskii, V. Egiazarian, D. Kuznedelev, D. Mazur, S. Nikita, I. Ermakov, and D. Alistarh Cache me if you must: adaptive key-value quantization for large language models. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: Appendix A. Sun et al. (2025) H. Sun, L. Chang, W. Bao, S. Zheng, N. Zheng, X. Liu, H. Dong, Y. Chi, and B. Chen ShadowKV: KV cache in shadows for high-throughput long-context LLM inference. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: Appendix A, Table 5, §1, §3.1, §3.4. Wan et al. (2025) Z. Wan, X. Wu, Y. Zhang, Y. Xin, C. Tao, Z. Zhu, X. Wang, S. Luo, J. Xiong, L. Wang, and M. Zhang D2OD_2O: Dynamic discriminative operations for efficient long-context inference of large language models. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix A. Xu et al. (2025) Y. Xu, Z. Jie, H. Dong, L. Wang, X. Lu, A. Zhou, A. Saha, C. Xiong, and D. Sahoo ThinK: thinner key cache by query-driven pruning. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix A, Table 5. Yan et al. (2025) X. Yan, Z. Li, T. Zhang, L. Kong, Y. Zhang, and X. Yang ReCalKV: low-rank KV cache compression via head reordering and offline calibration. arXiv preprint arXiv:2505.24357. Cited by: Appendix A, Table 5, §2. Yuan et al. (2023) Z. Yuan, Y. Shang, Y. Song, Q. Wu, Y. Yan, and G. Sun ASVD: activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821. Cited by: Appendix A, Table 5, §2. Zhang et al. (2024) R. Zhang, K. Wang, L. Liu, S. Wang, H. Cheng, C. Zhang, and Y. Shen LoRC: low-rank compression for LLMs KV cache with a progressive compression strategy. arXiv preprint arXiv:2410.03111. Cited by: Appendix A, Table 5, §2. Zhang et al. (2023) Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Re, C. Barrett, Z. Wang, and B. Chen H2O: heavy-hitter oracle for efficient generative inference of large language models. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: Appendix A, §1, §3.6. Zhou et al. (2025) Y. Zhou, S. Song, B. Liu, Z. Xi, S. Jin, X. Fan, Z. Zhang, W. Li, and X. Huang EliteKV: scalable KV cache compression via RoPE frequency selection and joint low-rank projection. arXiv preprint arXiv:2503.01586. Cited by: Appendix A. Zhu et al. (2026) Y. Zhu, D. H. Yang, M. M. Amiri, K. Murugesan, T. Pedapati, and P. Chen OjaKV: context-aware online low-rank KV cache compression. In Findings of the Association for Computational Linguistics: ACL 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 10161–10178. External Links: Link, Document, ISBN 979-8-89176-395-1 Cited by: Appendix A, Table 5, §2. Zweiger et al. (2026) A. Zweiger, X. Fu, H. Guo, and Y. Kim Fast KV compaction via attention matching. arXiv preprint arXiv:2602.16284. Cited by: Appendix A. Appendix A Extended related work Eviction methods. Eviction methods take an orthogonal route, keeping a token budget and dropping the rest by scoring tokens on accumulated heavy-hitter attention mass (H2O (Zhang et al., 2023)), attention-free key cosine similarity (KeyDiff (Park et al., 2025)), or query-agnostic reconstruction importance (KVzip (Kim et al., 2025)); we compare against all three. A parallel line keeps eviction as the backbone but relaxes its binary keep/drop decision into a non-binary retain / approximate / evict allocation (D2O (Wan et al., 2025), ARKV (Lei and Ilager, 2026)). Compression and reconstruction methods. Most methods extract the basis ahead of time and reuse it for every prompt; Table 5 groups them by basis source. Weight decompositions apply an offline SVD to the key/value projection weight matrices (ASVD (Yuan et al., 2023), Palu (Chang et al., 2025), LoRC (Zhang et al., 2024)). Calibration methods estimate the basis from a corpus (Eigen Attention (Saxena et al., 2024), ReCalKV (Yan et al., 2025)); KQ-SVD (Lesens et al., 2026) is closest in spirit, factoring the query-key product KQ⊤KQ to target the attention scores rather than K alone, but still from an offline calibration set of queries fixed across prompts. Cross-layer or trained methods share or learn the basis (xKV (Chang et al., 2026), MatryoshkaKV (Lin et al., 2025), Stiefel-manifold projections (Benfenati et al., 2026), EliteKV (Zhou et al., 2025)). All of these tie the basis to weights, a calibration distribution, or a learning phase. ShadowKV (Sun et al., 2025) is per-prompt, applying an online SVD to the pre-RoPE keys with no calibration, but it compresses keys only and offloads the full values to CPU. ThinK (Xu et al., 2025) is likewise per-prompt, pruning key channels by query-driven scoring with no training (keys only). OjaKV (Zhu et al., 2026) is also online and per-context, i.e., it maintains a running incremental-PCA basis (Oja’s rule), seeded from a small calibration set and re-orthonormalized during decoding, and uses attention only to choose which tokens stay full-rank rather than to shape the basis, trading exactness for within-generation adaptation. A separate cache-style line predicts or residual-codes values from keys via a fixed map V≈KWV\!≈\!KW fit offline (DeltaKV (Hao et al., 2026)), keeping keys and approximating values; these share our “keep all positions” philosophy but with a fixed, generic map. In the same reconstruction-and-compression vein, VECTOR (Lin et al., 2026) reconstructs evicted values from retained keys by least squares, and Attention Matching (Zweiger et al., 2026) fits the compressed cache to match the original attention output rather than the raw K,VK,V; orthogonally, adaptive KV quantization lowers the bit-width of the stored cache rather than its rank (AQUA-KV (Shutova et al., 2025)). These are complementary to AttSVD, which instead sets the per-prompt low-rank basis itself from this prompt’s own attention. Table 5: SVD / low-rank KV-cache methods by basis source. †ShadowKV low-rank-compresses keys and offloads full values to CPU. Method AttentionSVDs Basis source Calib./train? K&V ASVD (Yuan et al., 2023) K/V proj. weights offline, activation-aware calib. weights Palu (Chang et al., 2025) K/V proj. weights offline, one-time calib. K & V LoRC (Zhang et al., 2024) KV weight matrices offline, layer-progressive calib. K & V Eigen Attn (Saxena et al., 2024) K/Q/V activations offline calibration calib. K/Q/V KQ-SVD (Lesens et al., 2026) score KQ⊤KQ /VWOVW_O offline calibration (queries) calib. K & V ReCalKV (Yan et al., 2025) K (grouped) / V offline calibration calib. K & V ShadowKV (Sun et al., 2025) pre-RoPE keys per-prompt SVD (keys only) none K only† xKV (Chang et al., 2026) KV across layers post-hoc, shared none K & V MatryoshkaKV (Lin et al., 2025) KV feature dim trained projections train K & V ThinK (Xu et al., 2025) key channels query-aware (axis-aligned) none K only OjaKV (Zhu et al., 2026) K/V activations online Oja PCA (seeded) seed K & V AttSVD (ours) KCQ1/2KC_Q^1/2, diag(a)Vdiag( a)V, post-RoPE, per prompt online exact per-prompt SVD none K & V Appendix B Proof of Eq. (3) (attention-aware key truncation) We show that the query-metric key objective of Eq. (3) is (i) identical to a plain Frobenius objective in the metric CQC_Q, and (i) solved exactly by an ordinary truncated SVD after a change of variables, yielding the factors AK~,BK~A_ K,B_ K stored in Algorithm 2. Throughout, Q∈ℝm×dQ ^m× d are the (post-RoPE) queries, K∈ℝn×dK ^n× d the keys, CQ=Q⊤Q∈ℝd×dC_Q=Q Q ^d× d, and E=K−XE=K-X for a candidate rank-r approximation X. As CQ⪰0C_Q 0 is symmetric positive semidefinite it admits a unique symmetric PSD square root CQ1/2C_Q^1/2; with the ridge CQ=Q⊤Q+ϵc¯Id≻0C_Q=Q Q+ε\, c\,I_d 0 (Hoerl and Kennard, 1970) used in Algorithm 2, CQ1/2C_Q^1/2 is invertible with inverse CQ−1/2C_Q^-1/2. (i) Norm equivalence. Using ∥M∥F2=tr(MM⊤) M _F^2=tr(M ) and the cyclic invariance of the trace, ∥QE⊤∥F2 QE _F^2 =tr(QE⊤(QE⊤)⊤)=tr(QE⊤EQ⊤)=tr(E⊤EQ⊤Q)=tr(E⊤ECQ), =tr\! (QE (QE ) )=tr\! (QE E\,Q )=tr\! (E E\,Q Q )=tr\! (E E\,C_Q ), ∥ECQ1/2∥F2 E\,C_Q^1/2 _F^2 =tr(ECQ1/2CQ1/2E⊤)=tr(ECQE⊤)=tr(E⊤ECQ), =tr\! (E\,C_Q^1/2C_Q^1/2E )=tr\! (E\,C_Q\,E )=tr\! (E E\,C_Q ), where both chains use cyclicity of the trace and the second uses CQ1/2CQ1/2=CQC_Q^1/2C_Q^1/2=C_Q. Hence ∥Q(K−X)⊤∥F=∥(K−X)CQ1/2∥F Q(K-X) _F= (K-X)\,C_Q^1/2 _F for every X, i.e. the two objectives in Eq. (3) coincide; both equal tr((K−X)CQ(K−X)⊤)1/2tr\! ((K-X)\,C_Q\,(K-X) )^1/2. Up to the constant 1/d1/ d this is precisely the perturbation of the logits Q(K−X)⊤/dQ(K-X) / d, so minimizing the reconstruction error of K in the CQC_Q metric minimizes the logit error. Weighting the factorization by the query-side second moment CQ=Q⊤QC_Q=Q Q, rather than by a fixed calibration basis, is the metric analogue of learning a low-rank attention subspace (Saxena et al., 2024). (i) Generalized Eckart-Young. Introduce K~=KCQ1/2 K=K\,C_Q^1/2 and X~=XCQ1/2 X=X\,C_Q^1/2. Since CQ1/2C_Q^1/2 is invertible, rank(X~)=rank(X)rank( X)=rank(X), so X↦X~X X is a bijection on the set of rank-≤r≤ r matrices, and (K−X)CQ1/2=K~−X~(K-X)C_Q^1/2= K- X. The problem of Eq. (3) thus reduces to the ordinary rank-constrained Frobenius problem minrank(X~)≤r∥K~−X~∥F. _rank( X)≤ r K- X _F. By the Eckart-Young theorem (Eckart and Young, 1936; Mirsky, 1960) (Eq. (1)) its minimizer is the rank-r truncated SVD of K~ K: writing K~=UΣ⊤ K=U with singular values nonincreasing, X~⋆=U:,1:rΣ1:r:,1:r⊤ X =U_:,1:r _1:rV_:,1:r . Mapping back through CQ−1/2C_Q^-1/2, X⋆=X~⋆CQ−1/2=U:,1:rΣ1:r⏟AK~:,1:r⊤CQ−1/2⏟BK~,X = X C_Q^-1/2= U_:,1:r _1:r_A_ K\; V_:,1:r C_Q^-1/2_B_ K, which is exactly the decomposed store AK~BK~=Kr⋆A_ KB_ K=K_r of Eq. (3) and Algorithm 2. Because the change of variables preserves the objective value and is a bijection, X⋆X is a global minimizer of the original query-metric problem; it is unique whenever σr(K~)>σr+1(K~) _r( K)> _r+1( K). □ Remark (values, Eq. (4)). The value objective is the same statement with a left (row) metric diag(a)diag(a) replacing the right (column) metric CQC_Q: for E=V−XE=V-X, ∑iai∥Ei∥22=∥diag(a)E∥F2 _ia_i E_i _2^2= ( a)\,E _F^2, so by the identical change-of-variables argument the minimizer is the truncated SVD of diag(a)Vdiag( a)\,V with the left factor rescaled by diag(1/a)diag(1/ a), giving AV~,BV~A_ V,B_ V as in Algorithm 2. Why the diagonal (per-key mass) metric. We deliberately replace the exact row metric P⊤P P by diag(a)diag(a), ai=∑qPqia_i= _qP_qi, for four reasons; all stem from P⊤P P being an S×S× S object that grows with the context length, whereas the key metric CQC_Q stays a fixed d×d× d gram. (i) Cost. P⊤P P is quadratic in S to form and its symmetric square root is an (S3)O(S^3) eigendecomposition, per layer and forward pass; at S=32S=32K that is a ∼4 4 GB, ∼1013 10^13-flop object per head, against (S)O(S) to form, store, and invert a. A metric that costs more than the low-rank cache saves would defeat the method. (i) No materialized P. P⊤P P needs the dense S×S× S attention matrix, which memory-efficient kernels (FlashAttention) never form; the mass a=∑qPq,:a= _qP_q,: is one attention-shaped reduction that accumulates incrementally as tokens arrive, so it stays compatible with both the memory goal and standard kernels. (i) Conditioning. P⊤P P has rank at most the number of query rows and is singular whenever queries are fewer than keys, so (P⊤P)−1/2(P P)^-1/2 is ill-defined on its null space; diag(a)diag(a) with the floor a←a+10−3a¯a\!←\!a+10^-3 a is strictly positive and trivially invertible. (iv) First-order sufficiency. Attention mass concentrates on a few heavy tokens (the first-token sink alone carries a large share; Fig. 6), so the diagonal already encodes which rows the output reads, while the off-diagonal terms of P⊤P P are second-order. Empirically this diagonal weighting alone lowers the value output error ∥P(V−Vr)∥F/∥PV∥F P(V-V_r) _F/ PV _F from 0.1540.154 to 0.0120.012 at 50%50\% cache, most of the attainable gain. Appendix C Algorithms Algorithm 1 AttSVD; online per-prompt low-rank KV cache (one layer, one key/value head). A single Decode serves both schedules: the streaming flag switches on decode-time compression (branch-specific steps for both schedules are highlighted in blue). 1: keys K, values V∈ℝS×dV ^S× d; post-RoPE queries Q (only for the attention-aware basis); recent window w; streaming flag with chunk ℓ 2: factors (AK~,BK~),(AV~,BV~)(A_ K,B_ K),(A_ V,B_ V); exact tail (TK,TV)(T_K,T_V) (a sliding window when streaming); decode blocks (K,V)(D_K,D_V) 3: 4: Rank rule (the one knob, fixed once): fixed r=round((1−c)d)r=round ((1-c)d ) for both K,VK,V, or per-matrix energy τ (set r=Noner= None, so rK,rVr_K,r_V are read from each matrix’s own spectrum, giving rK≪rVr_K\! \!r_V). The prompt bulk is factored attention-aware (AttentionSVD, Alg. 2); streaming decode chunks, which carry no queries, use the unweighted TruncSVD. 5: 6: procedure Prefill(K,V,QK,V,Q)once, after the prompt is encoded 7: r←round((1−c)d)r \! ((1-c)\,d ) if ratio c is given else None fixed unified rank, else per-matrix energy rule 8: K(b),TK←K1:S−w,KS−w+1:SK^(b),T_K← K_1:S-w,\,K_S-w+1:S; V(b),TV←V1:S−w,VS−w+1:SV^(b),T_V← V_1:S-w,\,V_S-w+1:S split bulk / exact tail 9: (AK~,BK~),(AV~,BV~)←AttentionSVD(K(b),V(b),Q,r)(A_ K,B_ K),(A_ V,B_ V)← AttentionSVD(K^(b),V^(b),Q,r) attention-aware factors (Alg. 2); discard the bulk 10: K,V←(),()D_K,D_V←(\,),(\,) decode-block lists; populated only when streaming 11: return K,VK,\,V prefill served exactly from the original K,VK,V 12: end procedure 13: 14: procedure Decode(k,vk,v)every generated token 15: TK←[TK;k],TV←[TV;v]T_K←[\,T_K;\,k\,], T_V←[\,T_V;\,v\,] append the new token exactly 16: if streaming thenSTREAMING: factor aged-out tokens during decode 17: while rows(TK)−w≥ℓrows(T_K)-w≥ doexact window overflowed by a full chunk 18: CK,CV←C_K,C_V← oldest ℓ rows of TK,TVT_K,T_V still exact values 19: (aK,bK)←TruncSVD(CK,r)(a_K,b_K)← TruncSVD(C_K,r), (aV,bV)←TruncSVD(CV,r)(a_V,b_V)← TruncSVD(C_V,r); append to K,VD_K,D_V aged-out tokens carry no queries ⇒ unweighted SVD; freeze 20: drop those oldest ℓ rows from TK,TVT_K,T_V window stays in [w,w+ℓ][w,\,w+ ] 21: end while 22: elseSTATIC: no decode-time compression 23: keep TK,TVT_K,T_V exact the tail grows by this token; prefill factors fixed, K,VD_K,D_V stay empty 24: end if 25: Kfull←[AK~BK~;Recon(K);TK]K_full←[\,A_ KB_ K;\, Recon(D_K);\,T_K\,]; Vfull←[AV~BV~;Recon(V);TV]V_full←[\,A_ VB_ V;\, Recon(D_V);\,T_V\,] bulk ⊕ decode blocks ⊕ exact tail/window 26: return Kfull,VfullK_full,\,V_full accumulating: D empty ⇒[AK~BK~;TK] [A_ KB_ K;T_K]; freed after this layer 27: end procedure 28: 29: Recon()(D): concatenate ABAB over all blocks (A,B)∈(A,B) (oldest first); empty in accumulating mode. AttentionSVD is the attention-aware factorization of Alg. 2. Algorithm 2 AttentionSVD: the attention-aware low-rank factorization for the prompt bulk (one layer/head). Keys are truncated in the query geometry CQC_Q and values in the attention-mass metric, so the kept directions are exactly the ones attention reads. TruncSVD is the rank-r thin SVD (rank rule). 1: function AttentionSVD(K,V,Q,rK,V,Q,r)attention-aware basis (our method); r fixed, or None⇒ energy rule 2: if Q unavailable thenrare: capture hook did not fire ⇒ no weighting available 3: return TruncSVD(K,r),TruncSVD(V,r) TruncSVD(K,r),\ TruncSVD(V,r) degenerate unweighted fallback only (Eq. 2) 4: end if 5: Keys truncate in the query geometry CQC_Q, preserving the logits QK⊤QK 6: CQ←Q⊤Q+ϵc¯Id,c¯=1dtr(Q⊤Q)C_Q← Q Q+ε\, c\,I_d,\ \ c= 1dtr(Q Q); W,W−1←CQ1/2,CQ−1/2W,\,W^-1← C_Q^1/2,\ C_Q^-1/2 d×d×d gram; ridge ϵε (def. 10−310^-3) 7: (AK~,BK~)←TruncSVD(KW,r);BK~←BK~W−1(A_ K,B_ K)← TruncSVD(KW,\,r); B_ K← B_ K\,W^-1 AK~BK~=Kr⋆A_ KB_ K=K _r (Eq. 3) 8: Values weight rows by attention mass a, preserving the output PVPV 9: a←∑qsoftmax(QK⊤/d)q,:+10−3a¯a← _qsoftmax(QK / d)_q,:+10^-3\, a per-key mass (GQA-expanded), floored 10: (AV~,BV~)←TruncSVD(diag(a)V,r);AV~←diag(1/a)AV~(A_ V,B_ V)← TruncSVD(diag( a)\,V,\,r); A_ V (1/ a)\,A_ V AV~BV~=Vr⋆A_ VB_ V=V _r (Eq. 4) 11: return (AK~,BK~),(AV~,BV~)(A_ K,B_ K),\,(A_ V,B_ V) 12: end function 13: 14: function TruncSVD(X,rX,r)rank-r thin SVD with the rank rule (linear-algebra primitive) 15: U,Σ,←SVD(fp32(X))U, ,V← (fp32(X) ) singular values Σ in descending order 16: if r=Noner= None thendynamic per-matrix rank from X’s own spectrum 17: r←minρ:∑i≤ρΣii2≥τ∑iΣii2r← \ρ:\ _i≤ρ _i^2\ ≥\ τ _i _i^2 \ energy rule (Eq. 5); default τ=0.99τ=0.99 18: end if 19: return (U:,1:rΣ1:r,:,1:r⊤) (\,U_:,1:r _1:r\,,\ \ V_:,1:r \, ) ∈ℝn×r×ℝr×d ^n× r\!×R^r× d; Eckart-Young (Eq. 1) 20: end function Appendix D Experimental setup and extended experiments On a small model we verify that full-rank (r=dr=d) decomposition reproduces the dense-attention output bit-for-bit; the reconstruction error then decreases monotonically as r increases, consistent with Eq. (1). Table 1 reports the KV-cache footprint on Phi-4-14B: the measured ratio matches the theoretical r/dr/d to the decimal at 3232K context; at shorter context the fixed rdrd and exact-tail terms make the ratio slightly larger, exactly as Eq. (6) predicts. D.1 Streaming vs. accumulating decode The accumulating and streaming schedules differ only in the decode tail. Accumulating keeps every earlier decode token exact (an (g)O(g) tail that grows with the generation), while streaming factors aged-out tokens into frozen rank-r blocks so the exact buffer stays bounded. The agentic benchmark generations are mostly too short to overflow the w-token window, so streaming rarely evicts; to stress it where it genuinely compresses decode history we repeat the comparison on the full LongBench suite (n=1,750n=1,750), whose long-context outputs routinely overflow the window and trigger repeated chunk compression (Table 6). Across all ranks, streaming stays within 0.30.3 points of the accumulating (decode-exact) cache on average (Δ∈[−0.29,+0.24] ∈[-0.29,+0.24]), a gap well inside run-to-run noise, and it follows the dense cache (36.736.7) down to the 50%50\% budget. This holds even though streaming repeatedly factors aged-out tokens into frozen low-rank blocks rather than keeping them exact. Across both a short-output agentic eval (Figure 3) and a long-output academic suite, then, bounding the (g)O(g) decode tail to a fixed window costs no measurable quality. Per task, the two schedules are in fact identical on 88 of 99 LongBench tasks and differ only on GovReport, the sole task with a long (512512-token) generation budget, where streaming compression fires once the generation overflows the w=64w=64 window; the remaining tasks generate ≤128≤\!128 tokens (mostly ≤64≤\!64) and end early under greedy decoding, so the streaming path never fires and reduces exactly to accumulating. At the aggressive r=64r=64 the streaming cache holds 62.5%62.5\% pass@4pass@4 with the w=64w=64 window, vs. 56.0%56.0\% at w=16w=16 in an earlier run; enlarging the exact window removes the only point where compressing decode history had cost quality. The result is decisive in one direction: streaming loses nothing relative to the accumulating cache, while replacing the accumulating cache’s (g)O(g) exact tail (which grows with every one of the g generated tokens) with a bounded one; a large memory-efficiency gain on long generations at no quality cost. Table 6: Streaming vs. accumulating decode on LongBench (Phi-4-14B, full suite, n=1,750n=1,750; average, Δ the streaming-minus-accumulating gap), showing that streaming stays within run-to-run noise of the accumulating cache at every budget and lets the growing (g)O(g) exact decode tail be bounded to a fixed window at no quality cost. Phi-4-14B Qwen3-8B Cache c r SVD on raw KV AttSVD-Accum (decode exact) (ours) AttSVD-Stream (w=64w=64) (ours) Δ SVD on raw KV AttSVD-Accum (ours) AttSVD-Stream (ours) Δ 100%100\% −- Dense cache −- 36.736.7 −- −- 51.851.8 −- 80%80\% 0.20.2 102102 33.8333.83 36.2136.21 36.4536.45 +0.24+0.24 48.8448.84 51.7551.75 51.7551.75 +0.00+0.00 70%70\% 0.30.3 9090 33.1333.13 37.2137.21 36.9336.93 −0.28-0.28 48.3348.33 52.1052.10 52.0952.09 −0.01-0.01 60%60\% 0.40.4 7777 33.1633.16 36.6436.64 36.6536.65 +0.01+0.01 46.8846.88 52.5552.55 52.5452.54 −0.01-0.01 50%50\% 0.50.5 6464 31.031.0 36.0236.02 35.7335.73 −0.29-0.29 43.3443.34 51.8151.81 51.8551.85 +0.04+0.04 D.2 Streaming-window ablation on a long-generation task Table 7: Streaming-window ablation on GovReport (Llama-3.1-8B-Instruct, c=0.2c=0.2) for different exact decode-tail windows w, where ROUGE-L stays flat as w shrinks from ∞ down to 1616 so even a small exact window preserves quality. Decode schedule Exact decode window w GovReport (ROUGE-L) FullKV (parity anchor) n/a 34.6634.66 AttSVD-Accum (decode exact) ∞ 33.7933.79 AttSVD-Stream 256256 33.8033.80 AttSVD-Stream 128128 33.8133.81 AttSVD-Stream 6464 33.7733.77 AttSVD-Stream 3232 33.8833.88 AttSVD-Stream 1616 33.7433.74 The streaming schedule bounds the exact decode tail to a sliding window of w tokens, factoring aged-out tokens into frozen low-rank blocks (§3.3). On standard LongBench QA this path rarely fires, because most tasks generate ≤128≤\!128 tokens and never overflow the window, so streaming reduces to the accumulating (decode-exact) cache (§4; Table 3). To isolate the effect of w where the schedule does fire, we run a single-job ablation on GovReport, the LongBench task with the longest generation budget (512512 tokens), at a fixed cache ratio (c=0.2c=0.2, i.e. 80%80\%), varying the window w and including the accumulating cache as the w=∞w=∞ limit. All variants share the same attention-aware prefill factorization and are scored together against the FullKV anchor. The result is a flat curve where ROUGE-L varies by under 0.150.15 across w from 256256 down to 1616 and stays level with the w=∞w=∞ accumulating cache, so bounding the exact decode window to as few as 1616 tokens costs no measurable quality. D.3 Interpretability analysis D.3.1 Adversarial case on a controlled synthetic head We begin with a controlled synthetic head that isolates the core effect. Table is built so that K’s query-relevant directions are deliberately misaligned with its dominant singular directions, the adversarial case for energy-ordered SVD. At matched rank we compare SVD on raw KV, which minimizes ∥K−Kr∥F K-K_r _F, against the attention-aware truncation of Eqs. (3) and (4). On the keys, attention-aware truncation cuts the relative logit error ∥Q(K−Kr)⊤∥F/∥QK⊤∥F Q(K-K_r) _F/ QK _F from 0.980.98 to 0.270.27 at rank r=4r=4, a 3.6×3.6× reduction on the quantity attention actually consumes, even though its raw ∥K−Kr∥F K-K_r _F is larger. On the values, mass-weighted truncation lowers the output error ∥P(V−Vr)∥F/∥PV∥F P(V-V_r) _F/ PV _F from 0.920.92 to 0.640.64 at r=8r=8 and from 0.610.61 to 0.000.00 at r=16r=16, recovering the exact attention output once the rank spans the mass-weighted subspace. The retained basis, not just the rank, governs quality at a fixed memory budget. D.3.2 Real-world interpretability analysis We then move from the synthetic head to a real one. Figures 7, 6, and 6 inspect a single layer/head of SmolLM2-135M33 3 We use SmolLM2-135M for the factor-level and real-head visualizations because its small hidden size and few heads keep a single head’s attention map, singular spectrum, and per-token factor weights legible in one figure, and the full per-head SVD runs on CPU; the same attention-aware effect holds on the larger models used in our main quantitative experiments. under both weightings. Figure 7 contrasts the attention-aware and raw-KV caches directly, while Figures 6 and 6 give a full decomposition of K and V on that head; each reports four panels: (a) the attention map, (b) the singular spectrum under raw vs. attention-aware weighting, (c) the per-token factor weights |U||U|, and (d) the top-contributing tokens of each factor. The three quantities of §3.6 map onto this head as follows. (i) Effective rank. Panel (b) of each factor figure reports the per-matrix compression capacity as the decay rate of its energy curve. The energy rule of Eq. (5) measures this decay on the attention-aware matrices K~ K and V~ V, not the raw K and V, and sets each its own rank accordingly. K~ K’s spectrum is steep, with ∼67% 67\% of the energy in a single leading factor and 99.5%99.5\% in the top-3232 (a 50%50\% cache), whereas V~ V’s spectrum is markedly flatter, with only ∼36% 36\% in its leading factor. The raw K and V spectra (red in panel (b)) track these attention-aware curves closely in overall decay, so the steep versus flat gap is intrinsic to this head rather than an artifact of the weighting; the query-weighted K~ K curve is only somewhat sharper, concentrating on the query-relevant directions that item (i) examines. That asymmetry is the concrete per-head cause of the rK~≪rV~r_ K\! \!r_ V allocation that Eq. (5) exploits, with V~ V genuinely less compressible than K~ K. (i) Query subspace. Panel (b) also shows that the attention-aware basis re-ranks factors by query relevance rather than raw energy, keeping a different rank-r subspace than SVD on raw KV; the gap between the raw-energy (red) and attention-weighted (blue) curves makes visible that ordering factors by energy is not the same as ordering them by what the queries consume. Figure 7 confirms on the real head the same effect Table showed on the synthetic one: at 50%50\% cache the attention-aware keys (blue, panel (a)) sit almost exactly on the full-attention distribution (black) while SVD on raw KV (red) distorts it, and the query-relevant logit error stays more than 10×10× lower across budgets (0.0130.013 vs. 0.1410.141 at 50%50\%, panel (b)). (i) Token saliency. Panels (c) and (d) tie the factors back to tokens. The per-token factor weights |U||U| in panel (c) group the leading K factors into roughly semantic clusters, for example a colour factor \color, purple, green\ identified from these weights, and panel (d) lists the top-contributing tokens of each factor with their energy shares. Row-weighting each token by its attention mass ai=∑qPqia_i= _qP_qi reconstructs the heavily attended tokens, such as the first-token sink, most faithfully, lowering ∥P(V−Vr)∥F/∥PV∥F P(V-V_r) _F/ PV _F from 0.1540.154 to 0.0120.012 at 50%50\% cache, roughly 13×13×. End-to-end on Phi-4-14B agentic benchmark (Figure 3) the same attention-aware basis adds +0.13+0.13 to 0.290.29 mean judge and +7+7 to 1818 points pass@4pass@4 over SVD on raw KV, reaching 4.0674.067 / 62.5%62.5\% at 50%50\% cache, within 0.080.08 of dense. Taken together, the three items justify all of AttSVD’s design choices at once: low-rank caching (both spectra decay), asymmetric per-matrix ranks (rK~≪rV~r_ K\! \!r_ V), and the attention-aware basis (blue ≠ red). Figure 5: SVD factors of the low-rank KV cache, K (SmolLM2-135M, one layer/head). (a) attention map; (b) K’s singular spectrum, SVD on raw KV (energy) vs. attention-aware (query-weighted); (c) per-token factor weights |U||U|; (d) top tokens per factor with energy shares. Figure 6: SVD factors of the low-rank KV cache, V (same layer/head and prompt as Figure 6). (a) attention map, with the per-key attention mass that weights V; (b) V’s singular spectrum, SVD on raw KV vs. attention-aware (mass-weighted); (c) per-token factor weights |U||U|; (d) top tokens per factor with energy shares. Figure 7: The attention-aware cache preserves what attention computes (a real layer/head of SmolLM2-135M). (a) At 50%50\% cache the attention-aware keys (blue) lie almost on the full-attention target (black), while SVD on raw KV (red) distorts it. (b) Query-relevant logit error vs. cache size: attention-aware stays below SVD on raw KV at every budget (0.0130.013 vs. 0.1410.141 at 50%50\%). Real-model analogue of Table .