Paper deep dive
Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction
Isaac, Li
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas inference saves memory only when KV entries are physically removed. We ask whether the attention substrate affects this soft-to-hard transition. Using GPT-2-scale Transformers trained on OpenWebText, we run a controlled $2\times2\times2$ comparison over attention type, learned gating, and positional encoding. Although sigmoid attention is worse as a dense language model, learned hard eviction changes the useful operating points: sigmoid-gated models delete KV entries with negligible PPL change relative to their own no-eviction references. Under a matched live-cache protocol on the same dense backbones, learned sigmoid gates obtain lower PPL than our H$_2$O and KeyDiff implementations, whereas softmax gates do not uniformly beat these post-hoc methods. The results suggest that attention normalization can substantially affect whether a training-time soft gate transfers cleanly to hard KV deletion.
Tags
Links
- Source: https://arxiv.org/abs/2608.23296v1
- Canonical: https://arxiv.org/abs/2608.23296v1
Trouble viewing inline? Open PDF directly →
Full Text
34,125 characters extracted from source content.
Expand or collapse full text
Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction Isaac (Rucheng) Li Affiliation: University of Pittsburgh, Pittsburgh, PA, USA Abstract Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas inference saves memory only when KV entries are physically removed. We ask whether the attention substrate affects this soft-to-hard transition. Using GPT-2-scale Transformers trained on OpenWebText, we run a controlled 2×2×22× 2× 2 comparison over attention type, learned gating, and positional encoding. Although sigmoid attention is worse as a dense language model, learned hard eviction changes the useful operating points: sigmoid-gated models delete KV entries with negligible PPL change relative to their own no-eviction references. Under a matched live-cache protocol on the same dense backbones, learned sigmoid gates obtain lower PPL than our H2O and KeyDiff implementations, whereas softmax gates do not uniformly beat these post-hoc methods. The results suggest that attention normalization can substantially affect whether a training-time soft gate transfers cleanly to hard KV deletion. Keywords: KV cache compression, sigmoid attention, efficient inference, resource-adaptive inference, learned eviction 1 Introduction Learned KV-cache eviction has a soft-to-hard mismatch. During training, a model can only learn a differentiable proxy for deletion: a token is attenuated by a soft gate, but its key and value remain in memory. During inference, however, memory is freed only when the corresponding KV entry is physically removed. A useful learned eviction policy must therefore be threshold-stable: replacing soft attenuation by hard deletion at some threshold τ should introduce little or no loss in next-token prediction quality. Most KV-cache compression work does not directly ask which attention substrate makes this soft-to-hard transition easier to learn. Online post-hoc methods operate on a frozen model and select which cache entries to retain using signals available at inference time. H2O maintains accumulated attention-based heavy-hitter scores and retains high-score tokens together with recent tokens (20). KeyDiff is an attention-free, key-similarity-based method that ranks cached keys by their similarity to a key-space anchor and retains distinctive keys (10). Learned compression methods, including Dynamic Context Pruning and Dynamic Memory Compression, show that retention policies can be trained (1; 9). Our question is complementary: holding the learned eviction mechanism fixed, does the attention normalization itself affect how learnable hard KV deletion is? We test this question with a controlled 2×2×22× 2× 2 experiment: softmax versus sigmoid attention, dense versus learned eviction, and RoPE versus NoPE. NoPE denotes the removal of rotary embeddings from queries and keys. All models share the same GPT-2-scale setup. Gated models use the same soft-to-hard rule: token gates attenuate logits and values during training, then delete KV entries with g<τg<τ at inference. The results separate dense modeling quality from eviction learnability. Without eviction, softmax remains stronger: sigmoid attention is worse by ∼0.36 0.36 PPL with RoPE and ∼1.21 1.21 PPL without RoPE. With learned hard eviction, however, the useful operating points come from sigmoid attention. Against their own dense no-eviction references, Sig+G RoPE deletes 19.8%19.8\% of KV entries with no measurable PPL penalty (22.42422.424 vs. 22.44022.440), while Sig+G NoPE deletes 32.2%32.2\% with only a small increase (24.63724.637 vs. 24.60324.603). Under the same matched live-cache protocol, both learned sigmoid gates obtain lower PPL than our H2O and KeyDiff implementations. By contrast, softmax-gated models lose to H2O at matched compression. Further analysis clarifies why this pattern is substrate-specific. A random support-removal probe shows that sigmoid attention is not generically more robust to arbitrary K/V deletion: when random K/V positions are removed in dense RoPE models, sigmoid attention outputs are more perturbed than softmax at larger removal rates. Thus the advantage does not come from sigmoid tolerating deletion indiscriminately. It appears when the deletion pattern is learned, where non-row-normalized attention gives soft gates a cleaner local attenuation channel. Code and notebooks for reproducing the eight experimental cells are available at https://github.com/IsaacLi74/sigmoid-kv-eviction. 2 Method 2.1 Factorial model family We evaluate eight models in a 2×2×22× 2× 2 design over attention (softmax / sigmoid), gate (dense / learned), and rotary position treatment (RoPE (13) / NoPE). In this paper, NoPE denotes the no-RoPE condition used in our implementation: the RoPE rotation is removed from queries and keys. It does not use GPT-2-style learned absolute position embeddings; the input representation is the tied token embedding only, xi(0)=WE[ti],x_i^(0)=W_E[t_i], with no added learned position vector. All cells share a GPT-2–style decoder-only Transformer (14; 11) with nlayer=12n_layer=12, nhead=12n_head=12, dmodel=768d_model=768, dh=64d_h=64, dff=3072d_f=3072, and sequence length 512512. Training uses AdamW (8) for three epochs on a 1B1B-token OpenWebText training split (5), corresponding to roughly 3B3B token exposures. We use held-out splits of 100K100K tokens for validation and 1M1M tokens for test. 2.2 Attention variants All variants apply per-head QK-RMSNorm (19) before computing attention scores: q~i=RMSNorm(qi),k~j=RMSNorm(kj). q_i=RMSNorm(q_i), k_j=RMSNorm(k_j). In RoPE cells, we then rotate queries and keys with RoPE; in NoPE cells, this rotation is omitted: (q¯i,k¯j)=(RoPE(q~i,i),RoPE(k~j,j)),RoPE,(q~i,k~j),NoPE.( q_i, k_j)= cases(RoPE( q_i,i),RoPE( k_j,j)),&RoPE,\\ ( q_i, k_j),&NoPE. cases Softmax. Aij=softmaxj(q¯i⊤k¯j/dh).A_ij=softmax_j\! ( q_i k_j/ d_h ). Sigmoid attention. Aij=σ(q¯i⊤k¯j/dh+b(i)),b(i)=−log(i+1).A_ij=σ\! ( q_i k_j/ d_h+b(i) ), b(i)=- (i+1). Following 12, we include a negative bias term to stabilize sigmoid attention norms. One practical choice discussed by 12 is a negative scalar bias on the order of −logn- n, where n is the maximum training sequence length. Since a causal query at position i attends to only i+1i+1 keys, we use the per-query bias b(i)=−log(i+1)b(i)=- (i+1) to stabilize the expected row mass at each position. This row-wise bias would be a mathematical no-op under softmax, since adding the same constant to all logits in a row leaves the softmax distribution unchanged. 2.3 Learned gate For gated models, each layer ℓ<L−1 <L-1 produces a token-wise gate gj(ℓ)=σ(wℓ⊤xj(ℓ)+bℓ)∈(0,1)g_j^( )=σ\! (w_ x_j^( )+b_ )∈(0,1) which controls how the next layer sees token j. Let zij(ℓ+1)z_ij^( +1) denote the pre-attention logit in layer ℓ+1 +1. We inject the gate through both the attention logits and the values: zij(ℓ+1)←zij(ℓ+1)+log(gj(ℓ)+ε),vj(ℓ+1)←gj(ℓ)vj(ℓ+1).z_ij^( +1)← z_ij^( +1)+ \! (g_j^( )+ ), v_j^( +1)← g_j^( )v_j^( +1). where ε=10−8 =10^-8 avoids log0 0. This is the same injection for both attention types; the last layer carries no gate (no subsequent KV cache to evict). Gates are initialized with wℓ=0w_ =0 and bℓ=5b_ =5, so gj(ℓ)=σ(5)≈0.993g_j^( )=σ(5)≈ 0.993 at step 0 and early training behaves like the no-gate model. The two-channel injection (logg g on logits, g on values) is a differentiable simulation of dropping: as g→0g→ 0, both channels drive token j’s contribution to zero, matching the limiting behavior of hard deletion. In inference, we convert this soft retention signal into physical cache removal by dropping entries with gj<τg_j<τ. Entries removed after prefill are unavailable to all subsequent decode steps; during decode, the same threshold rule is applied to each newly written entry. Entries with gj≥τg_j≥τ remain in the cache and continue to use the same learned logit and value scaling. 2.4 Objective We use a simple training objective: cross-entropy plus a mean budget term that pulls gates toward small values, ℒ=ℒCE+λg¯,g¯=1(L−1)T∑ℓ=0L−2∑j=1Tgj(ℓ),L\;=\;L_CE\;+\;λ\, g, g\;=\; 1(L-1)\,T _ =0^L-2 _j=1^Tg_j^( ), with λ=0.03λ=0.03. We do not tune λ per cell; the same budget coefficient is used for all gated variants. 2.5 Evaluation Test protocol. On the OpenWebText test split, each sequence is split into a prefix of 384384 tokens for prefill and a window of 128128 tokens for single-token decode; PPL is computed on the decode window. Threshold selection. For each gated model we sweep a threshold grid from 00 to 0.90.9 on 100100 sequences from the held-out validation split and select the highest-compression τ⋆τ satisfying ΔPPL<0.1 <0.1 versus that gated model’s no-eviction reference. The selected τ⋆τ is then frozen for test. Physical eviction. Learned-gate evaluation performs thresholded physical eviction: after prefill we remove every cached entry with gj<τ⋆g_j<τ , and during decode the same rule is applied to each newly written entry. Entries that remain in the cache still use the learned logit and value scaling from Section 2. Reported compression rates count only physically removed KV entries, not the additional soft attenuation applied to retained entries. Matched live-cache size. For a sequence, the live-cache size is the number of KV entries that remain after hard eviction. Compression is measured relative to the corresponding no-eviction cache. When comparing learned gates with post-hoc baselines, we first measure the learned gate’s realized final live-cache size, then run post-hoc methods with the same final live-cache size on the paired dense no-gate backbone. Thus the baselines need not delete the same tokens, but they operate at the same cache size. Online post-hoc baselines. On each dense backbone we run two online post-hoc KV-cache eviction baselines at the matched final live-cache size. H2O is an attention-score-based heavy-hitter method: it maintains layer-wise, per-head cumulative attention scores and retains a mixture of high-score historical tokens and recent tokens (20). For H2O, we split the matched cache size equally between heavy-hitter and recent tokens and do not tune this ratio on validation. KeyDiff is an attention-free baseline: for each layer and head, it computes an unnormalized cached-key anchor μ(K)μ(K), ranks cached entries by cosine similarity to this anchor, and retains the least similar keys (10). Both baselines are run on the corresponding dense backbone, not on the gated checkpoint. This comparison isolates learned soft-to-hard eviction against post-hoc eviction under the same attention–RoPE substrate. 3 Results We report results in three steps. First, dense no-eviction PPL separates language-modeling quality from eviction behavior (Section 3.1). Second, validation threshold sweeps select one frozen hard-deletion operating point per gated model (Section 3.2). Third, matched final live-cache size test comparisons evaluate whether learned gates improve over post-hoc eviction on the corresponding dense backbones. Threshold selection uses each gated checkpoint at τ=0τ=0 as the no-eviction reference. In the matched live-cache comparison, post-hoc baselines are run on the paired dense no-gate backbone, so we also report the learned gate’s change relative to that dense backbone. 3.1 Dense baselines: softmax has lower no-eviction PPL Table 1 reports dense no-eviction decode-window PPL for the four attention–RoPE backbones under the same prefill–decode protocol. Table 1: Dense no-eviction test PPL. Backbone Dense PPL SM RoPE 22.079 Sig RoPE 22.440 SM NoPE 23.398 Sig NoPE 24.603 Sigmoid attention is not a stronger dense language model in this setup: softmax is better by 0.3610.361 PPL with RoPE and 1.2051.205 PPL without RoPE. 3.2 Validation threshold selection For each gated model we sweep the hard-deletion threshold over a 15-point validation grid and select the highest-compression threshold satisfying ΔPPL<0.1 <0.1 against that model’s τ=0τ=0 reference. The selected threshold is frozen before test evaluation. Figure 1 shows how validation PPL changes as compression increases under the threshold sweep, and Table 2 summarizes the selected operating points. The full numerical sweep is reported in Table 5. Figure 1: Validation threshold sweep. Each point shows the validation PPL and realized hard-KV compression rate for one threshold τ. Stars mark the selected τ⋆τ for each gated model. The threshold curves already separate the substrates. Both softmax models select τ⋆=0.01τ =0.01, while both sigmoid models select τ⋆=0.05τ =0.05. With RoPE, softmax and sigmoid reach nearly matched compression, but Sig+G has essentially no validation PPL increase. Without RoPE, Sig+G reaches 31.5%31.5\% compression, compared with 9.7%9.7\% for SM+G. Table 2: Validation-selected operating points. Compression is hard KV-entry deletion during prefill and decode. Model τ⋆τ ΔPPL Compression SM+G RoPE 0.010 +0.077+0.077 18.7%18.7\% Sig+G RoPE 0.050 −0.001-0.001 18.8%18.8\% SM+G NoPE 0.010 +0.016+0.016 9.7% 09.7\% Sig+G NoPE 0.050 +0.033+0.033 31.5%31.5\% 3.3 Matched live-cache size results On test, we freeze the validation-selected τ⋆τ and compare each learned gate to post-hoc H2O and KeyDiff run on the corresponding dense backbone at the matched final live-cache size. Table 3: Frozen-threshold test operating points. H2O and KeyDiff are run on the corresponding dense backbone at the same final live-cache size as the learned gate. Model τ⋆τ Comp. Learned PPL H2O KeyDiff SM+G RoPE 0.010 19.2%19.2\% 22.514 22.145 22.984 Sig+G RoPE 0.050 19.8%19.8\% 22.424 22.480 22.474 SM+G NoPE 0.010 10.2%10.2\% 23.576 23.423 23.588 Sig+G NoPE 0.050 32.2%32.2\% 24.637 25.107 25.786 The test results show that learned gates are not uniformly better than post-hoc eviction. On softmax backbones, H2O is stronger than the learned gate at the matched final live-cache size. On sigmoid backbones, the pattern reverses under the same matched-cache protocol. Sig+G RoPE deletes 19.8%19.8\% of KV entries and slightly improves the measured PPL relative to its dense no-eviction reference (22.42422.424 vs. 22.44022.440), while obtaining lower PPL than both post-hoc baselines at the same final live-cache size. Sig+G NoPE deletes 32.2%32.2\% with only a small increase over its dense reference (24.63724.637 vs. 24.60324.603) and again obtains lower PPL than the matched H2O and KeyDiff runs. Thus the main result is substrate-specific. The same simple learned soft-to-hard rule is weak under softmax, but becomes useful under sigmoid attention. The RoPE and NoPE regimes show different forms of this effect: with RoPE, sigmoid mainly improves quality at nearly matched compression; without RoPE, it mainly expands the safe compression range. 4 Mechanism Analysis The matched live-cache results suggest that the advantage is not a generic property of sigmoid attention, but a property of learned deletion under sigmoid attention. We use three diagnostics to support this interpretation: random support removal, row-sum and gate statistics, and token-level selectivity. 4.1 Not arbitrary-deletion robustness A possible alternative explanation is that sigmoid attention simply tolerates arbitrary K/V removal better than softmax. We test this with a random support-removal probe on the two dense RoPE backbones, with no learned gates involved. For each sampled test sequence and each layer, we first record the normal attention output Os,ℓfull=As,ℓfullVs,ℓfull.O_s, ^full=A_s, ^fullV_s, ^full. We then run a second pass in which a random fraction p of K/V positions is removed from the attention support at every layer. The first four positions are excluded to avoid confounding from attention-sink artifacts (17; 6). Removed positions are masked out of the attention logits and their value vectors are zeroed, so they cannot receive attention mass or contribute content. We measure the resulting layerwise perturbation by ρs,ℓ(p)=‖Os,ℓdrop(p)−Os,ℓfull‖F‖Os,ℓfull‖F. _s, (p)= \|O_s, ^drop(p)-O_s, ^full \|_F \|O_s, ^full \|_F. For each model and drop rate, we sample one random mask per sequence and per layer. Figure 2 plots the resulting perturbation for each layer after averaging over 5050 test sequences. This probe is a representation-level sensitivity test, not a random-eviction PPL baseline. Figure 2: Random support-removal sensitivity on dense RoPE models. For each drop rate p, we randomly remove K/V support and measure the relative perturbation of each layer’s attention output. The probe does not support generic deletion robustness. To summarize the curves, we average the sequence-averaged perturbations over all 1212 layers. At p=0.1p=0.1, softmax and sigmoid are nearly tied (0.5740.574 vs. 0.5790.579). At larger removal rates, sigmoid is more perturbed: 0.9430.943 vs. 1.0511.051 at p=0.3p=0.3, and 1.1311.131 vs. 3.6663.666 at p=0.5p=0.5. Thus the sigmoid operating-point advantage is unlikely to come from tolerating arbitrary K/V removal. It appears when the deletion pattern is learned. 4.2 Row-sum freedom and gate stability The next diagnostic asks why learned soft gates may transfer more cleanly to hard deletion under sigmoid attention. The key structural difference is row normalization. In softmax attention, each row sums to one, so suppressing one token redistributes attention mass to the other tokens in the same row. Thus a gate changes not only the gated token’s contribution, but also the relative weights of the remaining tokens. Sigmoid attention does not impose this constraint. Its row mass ∑jaij _ja_ij can vary with the query, layer, and sequence. A small gate can therefore mainly attenuate the selected token’s own contribution without forcing the rest of the row to compensate. This can make the training-time soft operation a closer approximation to hard deletion for the gated token, because the remaining row is not forced to renormalize. We measure this row-sum freedom using the coefficient of variation CV=stdi(∑jaij)meani(∑jaij),CV= std_i ( _ja_ij )mean_i ( _ja_ij ), computed across query positions on 19531953 test sequences of length 512512. For softmax, this value is zero by construction because every row sums to one. For sigmoid, a nonzero value means that different query positions can carry different total attention mass. We also track gate binarization, defined as the fraction of gates with g<0.05g<0.05 or g>0.95g>0.95, as a measure of whether training forms a thresholdable keep/delete structure. Figure 3: Per-layer mechanism evidence. Left: gate binarization. Right: coefficient of variation of attention row sums. Figure 3 shows these two ingredients separately. The right panel confirms the structural difference: softmax has zero row-sum variation by construction, while sigmoid retains nonzero row-sum variation across layers. Averaged over layers, the row-sum CV is 0.4840.484 for Sig+G RoPE and 0.4440.444 for Sig+G NoPE, compared with 00 for both softmax models. The left panel shows that learned gates can also form near-binary structure, especially in the high-compression Sig+G NoPE model. Averaged across gate layers and test token positions, Sig+G NoPE has the strongest aggregate binarization (43.3%43.3\%), while SM+G RoPE is also substantially binarized (36.8%36.8\%). However, binarization alone does not explain the result: Sig+G RoPE has lower aggregate binarization than SM+G RoPE (24.4%24.4\% vs. 36.8%36.8\%), yet transfers more cleanly to hard deletion. The useful regime therefore appears to require both pieces: the gates must learn a thresholdable structure, and the attention substrate must let soft attenuation approximate hard deletion locally. 4.3 Token selectivity The final diagnostic asks what the learned gates choose to suppress. If the gates were merely uniform cache shrinkers, then all token types would receive similar gate values. Instead, the gates are token-selective. We bucket GPT-2 surface tokens into coarse categories: content words, stopwords, punctuation, newline, whitespace, and digits. For each bucket, Table 4 reports the mean learned gate value on the test split. Lower values mean the model is more willing to remove that kind of token under hard thresholding; higher values mean the model tends to preserve it. Table 4: Mean gate value by token type on the test split. Buckets are assigned from the GPT-2 decoded surface form. Model content stopword punct. newline whitesp. digit SM+G RoPE 0.213 0.144 0.224 0.170 0.274 0.196 Sig+G RoPE 0.272 0.184 0.270 0.176 0.328 0.235 SM+G NoPE 0.223 0.144 0.195 0.135 0.334 0.224 Sig+G NoPE 0.281 0.178 0.254 0.273 0.455 0.263 The pattern is consistent across all four gated models. Across models, stopwords and newlines tend to receive the lowest gates, while whitespace tokens consistently receive the highest gates. Digit tokens are generally assigned higher gates than stopwords, but are not always among the highest-retention categories. For example, Sig+G NoPE assigns stopwords a mean gate of 0.1780.178, compared with 0.2810.281 for content tokens and 0.4550.455 for whitespace tokens. The token-level rankings show the same qualitative pattern. Soft-gate rankings frequently place common function words such as the, a, an, The, and It among low-gate tokens, while the actual hard-threshold eviction rankings include frequent function words such as the, a, an, of, in, on, for, to, and be. This means the learned policy is not deleting a fixed fraction of the cache uniformly; it assigns different retention values to different token classes. This also rules out a simple “smaller gates are better” explanation. Sigmoid models assign higher mean gates than softmax models in every bucket, so the sigmoid advantage does not come from more aggressive soft suppression during training. The advantage appears when those gates are hard-thresholded: under the sigmoid substrate, the learned gate values form a more useful keep/delete partition. Qualitative gate heatmaps on five short prompts are shown in Appendix B; they illustrate the same non-uniform structure across layers, token positions, and substrates. Taken together, these diagnostics support a substrate-level finding. Sigmoid attention is not generically robust to arbitrary K/V removal: random support removal perturbs it more at larger removal rates. Its advantage appears when deletion is learned. Because sigmoid attention is not row-normalized, soft attenuation of a token can be closer to hard deletion of that token than under a row-normalized softmax substrate. The learned gates then select token classes non-uniformly rather than shrinking the cache by a fixed ratio. 5 Discussion 5.1 Interpretation The main implication of our results is not that sigmoid attention is a strictly better language-modeling substrate. In the dense no-eviction setting, softmax remains stronger. Rather, sigmoid attention appears to be a better substrate for a specific systems objective: training a soft retention signal that can later be converted into hard physical KV deletion. This distinction is important. Post-hoc eviction methods test how well a frozen model tolerates externally imposed cache removal. Our experiment instead asks whether the attention substrate makes a learned eviction policy easier to train in the first place. This view also changes how to interpret the strongest operating point. Sig+G RoPE physically deletes 19.8%19.8\% of KV entries while achieving slightly lower measured PPL than its dense no-eviction backbone (22.42422.424 vs. 22.44022.440). We do not interpret this small negative ΔPPL as a reliable language-modeling gain. Its significance is more operational: learned eviction can remove a nontrivial fraction of the cache without measurable quality loss, and may sometimes act as selective cache denoising rather than merely lossy compression. 5.2 Implications If this substrate effect persists at larger scale and longer context length, it would motivate joint design of attention mechanisms and KV-retention policies. Rather than treating KV eviction only as an inference-time heuristic, our results frame it as an architectural co-design problem. 5.3 Limitations Our study has several limitations. First, we pick a single τ⋆τ per gated model from a discrete validation grid by maximizing compression subject to ΔPPL<0.1 <0.1. This rule is simple and frozen before test evaluation, but the cutoff is still a design choice. Finer threshold grids, per-layer thresholds, or alternative operating criteria could select different cache sizes. We also compare post-hoc baselines against learned gates at matched final live-cache size rather than matched PPL. This directly controls memory use, which is often the deployment constraint of interest, but it does not answer every deployment question; sigmoid threshold sweeps can have sharp cliffs, making matched-PPL comparison less stable. Second, experiments use one 123M123M GPT-2–scale model family trained for three epochs on a 1B1B-token OpenWebText split, with a single seed per cell and context length 512512. This is sufficient for a controlled substrate study, but not sufficient to establish deployment-scale behavior. Whether the effect persists at larger model scales, longer contexts, multiple seeds, other text distributions, or downstream long-context tasks remains open. Third, we use one gate design: token-wise gates with both logit attenuation and value scaling, trained with λ=0.03λ=0.03. We do not ablate logits-only versus values-only injection, alternative λ values, per-head gates, or per-layer thresholds. These ablations are needed to separate the contribution of the sigmoid substrate from the contribution of the particular gate parameterization. Finally, H2O and KeyDiff cover two complementary post-hoc signals: accumulated attention importance and attention-free key similarity. We use a fixed H2O heavy-hitter/recent split and do not tune post-hoc baseline hyperparameters, so these comparisons should be interpreted as matched-cache reference points rather than fully optimized SOTA benchmarking (2; 18; 3; 7; 16; 15). We also report PPL and compression rates, not wall-clock speed. We did not implement a production fused kernel for the specific gated sigmoid attention path studied here, despite existing work on fused attention kernels for softmax and sigmoid attention (4; 12). Thus the current results should be read as evidence about cache quality and learnability, not as an end-to-end inference-speed claim. References Anagnostidis et al. (2023) S. Anagnostidis, D. Pavllo, L. Biggio, L. Noci, A. Lucchi, and T. Hofmann Dynamic context pruning for efficient and interpretable autoregressive transformers. In Advances in Neural Information Processing Systems, Cited by: §1. Bui et al. (2025) N. Bui, S. Sharma, S. Lamba, S. Mishra, and R. Ying Cache what lasts: token retention for memory-bounded KV cache in LLMs. arXiv preprint arXiv:2512.03324. Cited by: §5.3. Chari et al. (2025) V. Chari, G. Qin, and B. Van Durme KV-distill: nearly lossless learnable context compression for LLMs. arXiv preprint arXiv:2503.10337. Cited by: §5.3. Dao et al. (2022) T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré FlashAttention: fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, Cited by: §5.3. Gokaslan and Cohen (2019) A. Gokaslan and V. Cohen OpenWebText corpus. Cited by: §2.1. Gu et al. (2025) X. Gu, T. Pang, C. Du, Q. Liu, F. Zhang, C. Du, Y. Wang, and M. Lin When attention sink emerges in language models: an empirical view. In International Conference on Learning Representations, Cited by: §4.1. Kim et al. (2026) J. Kim, D. Han, and S. Yun Fast KVzip: efficient and accurate LLM inference with gated KV eviction. arXiv preprint arXiv:2601.17668. Cited by: §5.3. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, Cited by: §2.1. Nawrot et al. (2024) P. Nawrot, A. Łańcucki, M. Chochowski, D. Tarjan, and E. M. Ponti Dynamic memory compression: retrofitting LLMs for accelerated inference. In International Conference on Machine Learning, Cited by: §1. Park et al. (2025) J. Park, D. Jones, M. J. Morse, R. Goel, M. Lee, and C. Lott KeyDiff: key similarity-based KV cache eviction for long-context LLM inference in resource-constrained environments. In Advances in Neural Information Processing Systems, Cited by: §1, §2.5. Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever Language models are unsupervised multitask learners. OpenAI Technical Report. Cited by: §2.1. Ramapuram et al. (2025) J. Ramapuram, F. Danieli, E. G. Dhekane, F. Weers, D. Busbridge, P. Ablin, T. Likhomanenko, J. Digani, Z. Gu, A. Shidani, and R. Webb Theory, analysis, and best practices for Sigmoid self-attention. In International Conference on Learning Representations, Cited by: §2.2, §5.3. Su et al. (2021) J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu RoFormer: enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864. Cited by: §2.1. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. In Advances in Neural Information Processing Systems, Cited by: §2.1. Wan et al. (2025) Z. Wan, X. Wu, Y. Zhang, Y. Xin, C. Tao, Z. Zhu, X. Wang, S. Luo, J. Xiong, L. Wang, and M. Zhang D2O: dynamic discriminative operations for efficient long-context inference of large language models. In International Conference on Learning Representations, Cited by: §5.3. Xiao et al. (2025) G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y. Fu, and S. Han DuoAttention: efficient long-context LLM inference with retrieval and streaming heads. In International Conference on Learning Representations, Cited by: §5.3. Xiao et al. (2024) G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis Efficient streaming language models with attention sinks. In International Conference on Learning Representations, Cited by: §4.1. Zeng et al. (2024) Z. Zeng, B. Lin, T. Hou, H. Zhang, and Z. Deng In-context KV-cache eviction for LLMs via attention-gate. arXiv preprint arXiv:2410.12876. Cited by: §5.3. Zhang and Sennrich (2019) B. Zhang and R. Sennrich Root mean square layer normalization. In Advances in Neural Information Processing Systems, Cited by: §2.2. Zhang et al. (2023) Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen H2O: heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems, Cited by: §1, §2.5. Appendix A Additional Validation Sweep Results Table 5 reports the full validation threshold grid. Each entry gives decode-window PPL and realized hard-KV compression at that threshold on the 100-sequence validation set. Bold entries are the selected operating points under the ΔPPL<0.1 <0.1 rule. Table 5: Full validation threshold sweep. τ SM+G RoPE Sig+G RoPE SM+G NoPE Sig+G NoPE 0.000 19.62 / 0.0% 19.48 / 0.0% 20.24 / 0.0% 21.12 / 0.0% 0.005 19.63 / 13.4% 19.48 / 14.2% 20.24 / 6.5% 21.11 / 27.3% 0.010 19.69 / 18.7% 19.48 / 14.6% 20.26 / 9.7% 21.11 / 28.4% 0.020 19.96 / 25.3% 19.48 / 15.6% 20.36 / 13.7% 21.11 / 29.3% 0.050 21.03 / 32.9% 19.48 / 18.8% 21.57 / 21.8% 21.16 / 31.5% 0.100 23.13 / 40.7% 26.02 / 24.8% 995.48 / 34.6% 22.35 / 36.6% 0.150 29.71 / 48.4% 31.73 / 36.2% 1524.68 / 44.4% 28.46 / 43.3% 0.200 117.85 / 56.6% 42.51 / 46.2% 2786.99 / 52.5% 78.84 / 50.6% 0.300 504.08 / 70.0% 79.86 / 59.7% 2224.26 / 62.6% 228.40 / 60.3% 0.400 736.59 / 77.9% 111.25 / 68.7% 2727.09 / 72.7% 206.53 / 68.7% 0.500 856.37 / 82.7% 156.11 / 76.0% 3489.68 / 79.0% 214.38 / 73.2% 0.600 1125.44 / 86.7% 215.09 / 81.0% 4659.00 / 84.4% 270.07 / 75.6% 0.700 1368.69 / 88.5% 280.60 / 83.7% 4585.06 / 87.7% 292.23 / 77.9% 0.800 1557.72 / 90.0% 361.24 / 85.7% 5063.09 / 89.5% 297.74 / 80.2% 0.900 1913.99 / 90.8% 451.68 / 87.4% 5493.55 / 90.6% 309.61 / 81.3% Appendix B Qualitative Gate Heatmaps This appendix shows qualitative gate heatmaps on five short prompts: narrative, technical, dialogue, factual, and question answering. Rows correspond to gate layers and columns correspond to token positions. Green indicates larger gate values and red indicates smaller gate values. These figures are not used as quantitative metrics, but they illustrate that learned gates vary substantially across layers, token positions, and substrates rather than acting as uniform cache shrinkers. Figure 4: Gate heatmap on a narrative prompt. Figure 5: Gate heatmap on a technical prompt. Figure 6: Gate heatmap on a dialogue prompt. Figure 7: Gate heatmap on a factual prompt. Figure 8: Gate heatmap on a question-answering prompt.