Paper deep dive
Compressible Softmax-Attended Language under Incompressible Attention
Wonsuk Lee
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 4/10/2026, 2:49:01 AM
Summary
The paper investigates the compressibility of softmax attention in transformer language models by comparing the spectral properties of the learned interaction matrix (W_Q^T W_K) and the generated logit energy field (E~). It finds that while the learned interaction matrix is spectrally uniform and incompressible, the generated logit energy field exhibits sharp spectral decay, indicating that attention compressibility is a property of the input data rather than the model architecture.
Entities (4)
Relation Signals (2)
Logit Energy Field → exhibits → Compressibility
confidence 95% · The compressibility of softmax-attended language is a property of the data
Interaction Matrix → is → Incompressible
confidence 95% · The flat spectrum of M=W_Q^T W_K implies that no fixed, low-rank projection can capture most of the interaction.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Softmax attention defines an interaction through $d_h$ head dimensions, but not all dimensions carry equal weight once real text passes through. We decompose the attention logit field into a learned component and a generated component and measure their spectra separately. For all 5,888 KV heads in five transformer language models (124M--7B parameters, four architecture families), the logit energy field $\tilde{E}$ reaches 90\% of its variance in 2--11 singular components. The learned interaction matrix $W_Q^\mathrm{T} W_K$ needs 38--75 components for the same threshold out of $d_h \in {64, 128}$. The spectral gap is 5--25$\times$ in effective rank. The compressibility of softmax-attended language is a property of the data, not the frame that analyzes it.
Tags
Links
- Source: https://arxiv.org/abs/2604.04384v2
- Canonical: https://arxiv.org/abs/2604.04384v2
Trouble viewing inline? Open PDF directly →
Full Text
18,929 characters extracted from source content.
Expand or collapse full text
Compressible Softmax-Attended Language under Incompressible Attention Wonsuk Lee Seoul National University & SK Hynix America wonsuk.lee@snu.ac.kr (April 5, 2026) Abstract Softmax attention defines an interaction through dhd_h head dimensions, but not all dimensions carry equal weight once real text passes through. We decompose the attention logit field into a learned component and a generated component and measure their spectra separately. For all 5,888 KV heads in five transformer language models (124M–7B parameters, four architecture families), the logit energy field E~ E reaches 90% of its variance in 2–11 singular components. The learned interaction matrix WQTWKW_Q^TW_K needs 38–75 components for the same threshold out of dh∈64,128d_h∈64,128. The spectral gap is 5–25× in effective rank. The compressibility of softmax-attended language is a property of the data, not the frame that analyzes it. 1 Introduction The key–value cache dominates the memory cost of autoregressive transformer inference and grows linearly with context length. Reducing it requires exploitable structure in the attention pattern. The row-centered attention logit energy field E~ E has rank at most dh+1d_h+1, where dhd_h is the head dimension [3], and its effective rank is much lower in practice. The dhd_h-dimensional key cache is redundant. The question is where the redundancy originates. Two possibilities. First, the weight matrices WQW_Q and WKW_K impose spectral structure on the interaction, making the mechanism itself low-rank. A fixed projection from the model parameters then compresses the key cache for any input. Second, the input data concentrates the interaction into a low-dimensional subspace even though the weights are spectrally uniform. Existing KV-cache compression methods assume one or the other. Low-rank weight factorization and static head pruning treat the redundancy as architectural. Token eviction and attention-aware merging treat it as input-dependent. Neither tests the assumption. Comparing the two spectra answers the question and predicts which class of methods should work. We measure both spectra across five models in four architecture families (GPT-2, LLaMA, Qwen, Mistral). They differ by 55–25×25× in effective rank. The learned matrix M=WQTWKM=W_Q^TW_K distributes its variance uniformly across all dhd_h directions in every model tested. The generated matrix E~ E concentrates almost all variance into a few. The compression lives in the data, not the weights. Effective key cache compression requires data-adaptive projections that change with every context. The compressibility of softmax-attended language is a property of how language activates attention, not of how attention is built. 2 Setup A single attention head [4] maps L input embeddings x0,…,xL−1∈ℝdmodelx_0,…,x_L-1 ^d_model to queries and keys through learned weight matrices: qi=WQxi∈ℝdh,kj=WKxj∈ℝdh.q_i=W_Q\,x_i ^d_h, k_j=W_K\,x_j ^d_h. (1) The attention logit is Zij=qiTkj/dhZ_ij=q_i^Tk_j/ d_h, and the attention probability is pij=softmaxj(Zij)p_ij=softmax_j(Z_ij). 2.1 The logit energy field (generated) Row-centering the logit matrix removes the position-dependent baseline: E~ij=Zij−Z¯i,Z¯i=1L∑j=0L−1Zij. E_ij=Z_ij- Z_i, Z_i= 1L _j=0^L-1Z_ij. (2) Since softmax is shift-invariant, pij=softmaxj(E~ij)p_ij=softmax_j( E_ij). The matrix E~ E carries the same information as Z for computing attention. The row-sum identity ∑jE~ij=0 _j E_ij=0 holds by construction. Therefore E~ 1=0 E\,1=0 and 1 is in the right null space. Since Z=QKT/dhZ=QK^T/ d_h has rank at most dhd_h, and row-centering adds a rank-one term, rank(E~)≤dh+1rank( E)≤ d_h+1. The SVD of E~ E, with rank R≤dh+1R≤ d_h+1, E~=∑k=1RσkukvkT, E= _k=1^R _k\,u_k\,v_k^T, (3) decomposes the interaction into R channels ordered by variance [2]. The k-th channel has strength σk _k, left singular vector uk∈ℝLu_k ^L over queries, and right singular vector vk∈ℝLv_k ^L over keys. If the singular values decay rapidly, then a few channels approximate the full matrix and the attention pattern is compressible. The right singular vectors satisfy vk⟂v_k 1. This follows from E~ 1=0 E\,1=0. The vector 1 is an eigenvector of E~TE~ E^T E with eigenvalue zero, while each vkv_k has nonzero eigenvalue σk2 _k^2. Eigenvectors of a symmetric matrix with distinct eigenvalues are orthogonal. 2.2 The interaction matrix (learned) The logit is a bilinear form in the input embeddings: Zij=xiTMxjdh,whereM=WQTWK∈ℝdmodel×dmodel.Z_ij= x_i^TM\,x_j d_h, M=W_Q^TW_K ^d_model× d_model. (4) The matrix M factors through ℝdhR^d_h (WQ∈ℝdh×dmodelW_Q ^d_h× d_model, WK∈ℝdh×dmodelW_K ^d_h× d_model), so rank(M)≤dhrank(M)≤ d_h. Its SVD, M=∑k=1dhλkpkrkT,M= _k=1^d_h _k\,p_k\,r_k^T, (5) decomposes the mechanism’s capacity into dhd_h channels. The singular values λk _k and their decay rate measure how much spectral structure the weights alone impose. The generated spectrum reflects both the weights and the input. The learned spectrum reflects the weights alone. Comparing the two reveals whether the compression is architectural or input-driven. 3 Experiments We computed the singular value spectra of E~ E and M for every attention head in five models spanning four architecture families and two head dimensions. Models. GPT-2 (124M, dh=64d_h\!=\!64, 144 heads). LLaMA-3.2-1B (1.2B, dh=64d_h\!=\!64, 512 heads). LLaMA-3.2-3B (3.2B, dh=128d_h\!=\!128, 672 heads). Qwen-2.5-3B (3B, dh=128d_h\!=\!128, 576 heads). Mistral-7B (7B, dh=128d_h\!=\!128, 1024 heads). The three larger models use grouped query attention, where multiple query heads share a single key-value head. Protocol. For M, we compute singular values directly from the weight matrices via QR factorization. For E~ E, we extract Q and K (with RoPE applied where applicable) on five texts from the Gutenberg Project [1] (Dickens, Darwin, Shakespeare, the King James Bible, and Adam Smith) at L=256L=256 and compute Z=QKT/dhZ=QK^T/ d_h. All SVDs use float64 arithmetic. Tables report the median across all heads pooled over the five texts. The effective rank at 90% varies by at most ±1± 1 across texts (standard deviation ≤1.2≤ 1.2). Results. Table 1 compares the cumulative fraction of variance (∑k=1rσk2/‖E~‖F2 _k=1^r _k^2/\| E\|_F^2) captured by the top r components. Table 2 translates these fractions into effective rank. Table 1: Cumulative variance captured by the top r singular components (median across all heads). E~ E concentrates its variance in a few components. WQTWKW_Q^TW_K does not. Models are grouped by head dimension: dh=64d_h=64 (left) and dh=128d_h=128 (right). GPT-2 LLaMA-1B LLaMA-3B Qwen-3B Mistral-7B r M E~ E M E~ E M E~ E M E~ E M E~ E 1 5% 72% 6% 43% 4% 42% 4% 41% 4% 46% 2 10% 90% 12% 63% 7% 61% 7% 60% 8% 64% 5 20% 96% 25% 83% 15% 81% 16% 80% 17% 82% 10 33% 98% 42% 93% 25% 90% 27% 90% 29% 91% 20 53% 99% 66% 98% 42% 96% 45% 96% 47% 96% 40 81% 100% 92% 100% 66% 99% 70% 99% 72% 99% Table 2: Effective rank: number of singular components for the stated variance threshold (median across all heads). The spectral gap between M and E~ E persists across all five models. GPT-2 LLaMA-1B LLaMA-3B Qwen-3B Mistral-7B Threshold M E~ E M E~ E M E~ E M E~ E M E~ E 80% 40 2 29 5 57 5 53 5 50 5 90% 49 2 38 8 75 11 70 11 66 10 95% 55 4 44 13 88 18 83 17 80 17 99% 61 18 54 23 107 39 104 40 102 38 At 90% variance, E~ E needs 2–11 components; M needs 38–75. The ratio ranges from 5×5× (LLaMA-1B) to 25×25× (GPT-2). The three dh=128d_h=128 models converge to E~ E effective rank 10–11 despite learned effective ranks of 66–75, a gap of 66–7×7×. The learned spectrum is flat in every case. 4 Conjecture 4.1 Incompressible attention framework The flat spectrum of M implies that no fixed, low-rank projection can capture most of the interaction. Discarding any subset of the dhd_h directions loses a proportional fraction of the mechanism’s capacity. The attention framework is, in spectral terms, incompressible. If M had concentrated its spectrum, compression would be straightforward. Project queries and keys onto the dominant eigenvectors, discard the rest, and reduce the head dimension from dhd_h to r≪dhr d_h. The flat spectrum rules this out. Each head retains roughly equal capacity in all dhd_h directions, able to represent whichever query–key interaction the input demands. 4.2 Compressible logit energy field While M describes what the mechanism can do, E~ E describes what it actually does on a specific input. The sharp spectral decay of E~ E means the query–key interaction concentrates into a few dominant patterns for any given context. The concentration comes from the data. Layer normalization and preceding transformer blocks shape the input embeddings xix_i into a context-dependent manifold rather than spanning all of ℝdmodelR^d_model. Queries and keys inherit this low-dimensional structure. The matrix Z=QKT/dhZ=QK^T/ d_h has formal rank dhd_h but much lower effective rank, determined by the data manifold. The empirical regularity μK=O(1) _K=O(1) (key incoherence), measured across 16 transformer language models [3], reflects the same geometry. The weights permit large μK _K. Language does not require it. Training preserves μK≈2 _K≈ 2. Two properties of the data support the pattern’s generality. First, it holds across both dh=64d_h=64 and dh=128d_h=128. Doubling the head dimension roughly doubles the learned effective rank but leaves the generated effective rank at 10–11. Second, it is stable across five texts from different centuries and genres (standard deviation ≤1.2≤ 1.2). 4.3 Low-rank structure survives softmax Whether the sharp generated spectrum survives softmax is not obvious. A rank-r approximation E~r E_r captures most of the Frobenius-norm variance, but softmax is nonlinear and amplifies residuals at peak-attention positions. Proposition 4.1 (Softmax stability of low-rank approximation). Let E~∈ℝL×L E ^L× L be the row-centered logit matrix with SVD E~=∑k=1RσkukvkT E= _k=1^R _k\,u_k\,v_k^T, and let E~r=∑k=1rσkukvkT E_r= _k=1^r _k\,u_k\,v_k^T be its rank-r truncation. Let pi=softmax(E~i)p_i=softmax( E_i) and pi(r)=softmax((E~r)i)p_i^(r)=softmax(( E_r)_i) be the attention distributions in row i. If the singular vectors satisfy the delocalization condition maxjuj2≤βL,maxjvj2≤βL _j\,u_j^2≤ β L, _j\,v_j^2≤ β L (6) for a parameter β, then for every row i: ‖pi−pi(r)‖1≤βL∑k=r+1Rσk. \|p_i-p_i^(r) \|_1\;≤\; β L\, _k=r+1^R _k\,. (7) Proof. The residual Δij=∑k=r+1Rσk(uk)i(vk)j _ij= _k=r+1^R _k\,(u_k)_i\,(v_k)_j satisfies |Δij|≤∑k=r+1Rσk|(uk)i||(vk)j|≤βL∑k=r+1Rσk| _ij|\;≤\; _k=r+1^R _k\,|(u_k)_i|\,|(v_k)_j|\;≤\; β L _k=r+1^R _k by the delocalization condition (6), since |(uk)i||(vk)j|≤β/L|(u_k)_i|\,|(v_k)_j|≤β/ L for all i,j,ki,j,k. This bounds ‖Δi‖∞\| _i\|_∞ uniformly over rows. Applying Theorem A.1 (Appendix A) with a=E~ia= E_i and b=(E~r)ib=( E_r)_i, ‖pi−pi(r)‖1=‖softmax(E~i)−softmax((E~r)i)‖1≤‖Δi‖∞≤βL∑k=r+1Rσk.∎\|p_i-p_i^(r)\|_1\;=\;\|softmax( E_i)-softmax(( E_r)_i)\|_1\;≤\;\| _i\|_∞\;≤\; β L\, _k=r+1^R _k\,. The delocalization condition (6) is verified directly. Measuring β=maxjvj2⋅Lβ= _jv_j^2· L across every singular vector of every head, from five models and five texts at L=256L=256, gives median β=2.5β=2.5–5.65.6 and maximum β≤14β≤ 14. The value does not grow with model size or head dimension. The delocalization lemma of [3] shows that β is controlled by the key incoherence μK _K and the condition number κ(K)κ(K), both of which are bounded across trained models. The bound shrinks with both context length (through 1/L1/ L) and rank (through the tail ∑k>rσk _k>r _k). At r=20r=20, the tail accounts for 2% of the total variance. The empirical evidence at L=64L=64–1,0241,024 shows no growth in the tail. Direct measurement confirms the prediction. At L=256L=256 across five models and five texts, the median ‖pi−pi(r)‖1\|p_i-p_i^(r)\|_1 is 0.31–0.46 (mean per row) at r=10r=10, 0.18–0.29 at r=20r=20, and 0.05–0.14 at r=40r=40. The worst row follows the same trend: 0.90–1.31, 0.52–0.84, and 0.16–0.39 respectively. 4.4 Conjecture statement We conjecture that the low effective rank of E~ E is a universal property of natural language processed by trained transformers. The source is the low intrinsic dimensionality of contextualized embeddings, not any spectral constraint in the attention weights. Models across four architecture families demonstrate the pattern. Whether it extends beyond language remains open. 5 Discussion The spectral gap between M and E~ E has consequences for both compression practice and architectural understanding. The flat spectrum of WQTWKW_Q^TW_K rules out weight-based compression. Fixed, input-independent projections cannot exploit the concentration in E~ E because the low-rank projection that captures nearly all of the attention variance changes with every context. Effective compression requires data-adaptive methods. This separates existing KV-cache work into two classes. Input-independent methods (low-rank weight factorization, static head dimension reduction) compress the mechanism, which is already spectrally uniform. Context-adaptive methods (eviction policies, token merging, attention-aware pruning) can exploit the sharp data spectrum, but they must track a projection that shifts as the context grows during autoregressive generation. The gap between the mechanism’s full-rank frame and the data’s low-rank activation serves a purpose. Each head allocates uniform spectral capacity across dhd_h dimensions, yet language fills only a few at any given moment. The unused dimensions allow the same head to serve diverse contexts across the training distribution. Idle capacity is flexibility. Proposition 4.1 bounds per-head, per-layer attention fidelity. Three quantities remain open: 1. The output error ‖oi−oi(r)‖=‖∑j(pij−pij(r))vj‖\|o_i-o_i^(r)\|=\| _j(p_ij-p_ij^(r))v_j\| depends on the value matrix V, not just the attention weights. 2. Errors compound across layers. Twenty layers each contributing a small ℓ1 _1 attention error could cancel or amplify depending on whether they are correlated. 3. The accumulated effect on end-to-end perplexity remains an open experimental question. The spectral gap reported here is measured on language models. If structured sequential input drives the concentration, vision transformers and protein models would show similar patterns. If it is specific to natural language, other modalities would exhibit higher effective rank in E~ E. Either outcome would clarify what makes attention compressible. References [1] Project Gutenberg. https://w.gutenberg.org/, 1971–present. [2] G. H. Golub and C. F. Van Loan. Matrix Computations. Johns Hopkins University Press, 4th edition, 2013. [3] W. Lee. On the invariants of softmax attention. arXiv preprint, 2026. [4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017. Appendix A Appendix Theorem A.1 (Softmax Lipschitz bound). For any a,b∈ℝka,b ^k, ‖softmax(a)−softmax(b)‖1≤‖a−b‖∞.\|softmax(a)-softmax(b)\|_1\;≤\;\|a-b\|_∞. (8) Proof. Let c=b−ac=b-a and define p(t)=softmax(a+tc)p^(t)=softmax(a+tc) for t∈[0,1]t∈[0,1], so p(0)=softmax(a)p^(0)=softmax(a) and p(1)=softmax(b)p^(1)=softmax(b). By the fundamental theorem of calculus, softmax(b)i−softmax(a)i=∫01dtpi(t)dt.softmax(b)_i-softmax(a)_i= _0^1 ddt\,p_i^(t)\,dt. The softmax Jacobian gives dtpi(t)=pi(t)(ci−μt) ddt\,p_i^(t)=p_i^(t)(c_i- _t), where μt=∑jpj(t)cj=p(t)[c] _t= _jp_j^(t)c_j=E_p^(t)[c]. Applying the triangle inequality inside the integral, ‖softmax(b)−softmax(a)‖1≤∫01∑ipi(t)|ci−μt|dt.\|softmax(b)-softmax(a)\|_1\;≤\; _0^1 _ip_i^(t)|c_i- _t|\,dt. The integrand is the mean absolute deviation MAD(c;q)=q[|c−q[c]|]MAD(c;\,q)=E_q[|c-E_q[c]|] evaluated at q=p(t)q=p^(t). Fix t and write q=p(t)q=p^(t), μ=μtμ= _t. By Jensen’s inequality (φ(x)=x2 (x)=x^2 is convex), MAD(c;q)2≤Varq(c)MAD(c;\,q)^2 _q(c). By Popoviciu’s inequality, for c bounded in [m,M][m,M], Varq(c)≤(M−m)2/4Var_q(c)≤(M-m)^2/4. Combining, MAD(c;q)≤Varq(c)≤M−m2=maxici−minici2.MAD(c;\,q)\;≤\; Var_q(c)\;≤\; M-m2\;=\; _ic_i- _ic_i2. Since maxici≤‖c‖∞ _ic_i≤\|c\|_∞ and −minici≤‖c‖∞- _ic_i≤\|c\|_∞, the range satisfies (maxici−minici)/2≤‖c‖∞( _ic_i- _ic_i)/2≤\|c\|_∞. Substituting back, ‖softmax(b)−softmax(a)‖1≤∫01‖c‖∞t=‖c‖∞=‖a−b‖∞.∎\|softmax(b)-softmax(a)\|_1\;≤\; _0^1\|c\|_∞\,dt\;=\;\|c\|_∞\;=\;\|a-b\|_∞.