Paper deep dive
Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression
Xingyu Zhu, Pu, Yi, Ziheng Cheng, Ang Lv, Jing Liu, Lexing Ying, Yiyuan Ma, Xin Dong
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Chunked KV-cache compression reduces the memory and attention costs of long-context inference by compressing windows of consecutive tokens into fewer cache entries at a fixed stride. Such compression also introduces a new positional coordinate: a token's phase, or its position relative to compression-window boundaries. We uncover a systematic asymmetry in models using such compression: the same information can be easy to retrieve at one phase and difficult at another. We call this periodic variation in retrieval performance phase sensitivity. In large open-weight models with such compression, long-context retrieval accuracy can differ by up to 40 percentage points across phases, revealing periodic weak spots that average benchmark scores can conceal. To investigate this behavior, we pretrain a family of transformers from scratch across multiple KV-compression designs, reproducing phase sensitivity across the variants. Mechanistic analysis using causal interventions in these models reveals phase specialization: different attention components contribute asymmetrically to retrieving information at different source phases. We further analyze idealized retrieval models, showing how gradient flow dynamics may favor sharp phase specialization. Evaluating models with chunked KV-cache compression thus requires measuring across compression phases: high average accuracy can coexist with systematic positional failures.
Tags
Links
- Source: https://arxiv.org/abs/2609.36322v1
- Canonical: https://arxiv.org/abs/2609.36322v1
Trouble viewing inline? Open PDF directly →
Full Text
202,882 characters extracted from source content.
Expand or collapse full text
on any item or to use differnt the four listed to the left Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression Xingyu Zhu* 1,2 , Pu (Luke) Yi* 1,3 , Ziheng Cheng* 1,4 , Ang Lv* 1 , Jing Liu 1 , Lexing Ying 1 , Yiyuan Ma 1,† , Xin Dong 1,† 1 ByteDance Seed 2 Princeton University, 3 Stanford University, 4 University of California, Berkeley ∗ Core contributors. Work done during internship at ByteDance Seed, † Corresponding authors Abstract Chunked KV-cache compression reduces the memory and attention costs of long-context inference by compressing windows of consecutive tokens into fewer cache entries at a fixed stride. Such compression also introduces a new positional coordinate: a token’s phase, or its position relative to compression-window boundaries. We uncover a systematic asymmetry in models using such compression: the same information can be easy to retrieve at one phase and difficult at another. We call this periodic variation in retrieval performance phase sensitivity. In large open-weight models with such compression, long-context retrieval accuracy can differ by up to 40 percentage points across phases, revealing periodic weak spots that average benchmark scores can conceal. To investigate this behavior, we pretrain a family of transformers from scratch across multiple KV- compression designs, reproducing phase sensitivity across the variants. Mechanistic analysis using causal interventions in these models reveals phase specialization: different attention components contribute asymmetrically to retrieving information at different source phases. We further analyze idealized retrieval models, showing how gradient flow dynamics may favor sharp phase specialization. Evaluating models with chunked KV-cache compression thus requires measuring across compression phases: high average accuracy can coexist with systematic positional failures. Date: September 30, 2026 Correspondence: Yiyuan Ma at mayiyuan.unicorn@bytedance.com Xin Dong at xin.dong@bytedance.com 1 Introduction Long-context inference in transformers can be bottlenecked by the KV cache, whose memory and attention costs grow with context length. A practical remedy, used in models such as DeepSeek-V4 [11], ischunked KV-cachecompression: the model groups consecutive tokens into fixed-size windows, compresses each window into fewer cache entries, and uses these summaries for long-context retrieval [28,36]. The compression happens as context is processed, before a later query specifies what information will need to be recalled. Each summary is therefore query-agnostic and must preserve information that may be needed later. 1 arXiv:2609.36322v1 [cs.LG] 28 Sep 2026 This design imposes a periodic structure on the context. A new compression window begins everyStokens, whereSis the compression stride. Each token thus has a new coordinate relative to these boundaries beyond its absolute position. We call this coordinate itsphase, defined as its position moduloS. Shifting the input by a few tokens can change which tokens are compressed together without changing their content or relative positions. The same information may thus be compressed differently depending on its phase, and hence retained with different fidelity. In LLMs trained with chunked KV-cache compression, we find that retrieval performance varies sharply and periodically with the phase of the source information, a failure mode we termphase sensitivity(Section 2). A motivating example comes from asking DeepSeek-V4 to complete its own inference code (Figure 1). When we prepend a decorative comment and vary only its length, the model’s preferred completion of one token flips with a period of four tokens, matching its compression stride. We then measure retrieval directly through a controlled needle-in-a-haystack evaluation (Figure 2). At 128K context, retrieval accuracy in the DeepSeek-V4 family can differ by up to 40 percentage points across phases. To the best of our knowledge, this is the first report of such internal periodic variation, which average benchmark scores can conceal. To test whether chunked KV compression is the source of such variation, we pretrain from scratch a broad family of transformers with the Qwen3-0.6B architecture [27] as backbone and KV compression as the only substantial architectural modification (Section 3). Every compressed model we evaluate shows periodicity matching the compression stride, including models that simply average token representations within each window. By contrast, full-attention baselines have no comparable periodicity. A natural first explanation is the window boundary itself. In our needle-in-a-haystack task, each key is immediately followed by its value, so with non-overlapping windows a key–value pair either falls inside one window or is split across two, and a split pair is plausibly harder to retrieve. However, boundaries alone do not account for the full periodic pattern: accuracy also varies widely across phases that keep the key and its value within a single window. This suggests a deeper look into how the model writes information into compressed memory and reads it back. Causal interventions on KV heads and layers support this view: retrieval at different phases depends differently on different heads, a systematic pattern we callphasespecialization(Section 4.1). The pattern appears in both our pretrained models and DeepSeek-V4. To examine how such specialization can develop, we study compression gates, which determine how much each within-window position contributes to a compressed entry (Section 4.2). In a pretrained model whose gates can learn separate positional preferences, some heads develop persistent preferences during pretraining that recur across inputs. Their intervention effects are strongest near the phases they favor, supporting a division of retrieval work among the examined heads. The emergence of these preferences suggests that training can favor selective retention even when future queries are unknown. We examine this possibility in an idealized induction task with compressed attention (Section 5), where mixing information in the compression window can weaken the match between a compressed key and a later query. We show that the optimal compressor favors particular phases, and gradient flow drives gates toward persistent preferences. Together, these findings show that average accuracy is not enough to evaluate models with chunked KV-cache compression. Their reliability should also be measured across phases, for example by shifting the input by a few tokens while keeping the queried information fixed. 2 Stride-Periodic Behavior in DeepSeek-V4 and DeepSeek-V4.1 2.1 A Periodically Flipping Code-Completion Example We begin by asking DeepSeek-V4-Flash-Base to complete DeepSeek-V4’s official inference code (Figure 1a). The prompt ends atFPinside a type cast in an FP8 quantization function, and the original source continues with “8”. The surrounding code favors this continuation. The function quantizes to FP8, while the outer cast already converts to compute_dtype, which is FP32. Completing “32” would duplicate the outer cast and skip the rounding step. To test whether this prediction depends on position, we prepend a decorative docstring and vary its length by changing the number of repeated “=” characters on its second line. The code is unchanged, and each additional filler token shifts the code and the completion position by one. Across 16 filler lengthsLfrom 2 ... FP8 = "float8_e4m3" FP4 = "float4_e2m1fn" FP32 = "float32" ... """Block-wise FP8 quantization.""" ... compute_dtype = FP32 ... s_local[i] = ... * fp8_max_inv ... T.Cast(compute_dtype, T.Cast(FP """ This module contains ... ==== ... """ (decorative prefill of length L) "32""8""32""8" 0 1 Avg. Prob. 0.713 0.264 L≡ 0, 1 (mod 4) 0.072 0.915 L≡ 2, 3 (mod 4) 2426283032343638 Decorative prefill length L (tokens) 0 1 Next-token probability P(8)P(32) (a) Prompt schematic inference/kernel.py (b) Completion preferences "32" ×"8" ✓ (c) Next-token probabilities DeepSeek-V4 Flash Base Figure 1 Next-token predictions by DeepSeek-V4-Flash-Base on a code snippet. The prediction flip periodically as a variable-length decorative filler shifts the input. (a) The decorative filler is a docstring whose lengthLvaries by the number of “=” characters. The ground-truth completion is “8”, supported by the highlighted components in the context. (b) Averaged over the eight lengths in each group, the model places higher probability on the incorrect “32” whenL mod4∈0,1and on the correct “8” whenL mod4∈2,3. (c) The preferred completion switches every two tokens and the pattern repeats every four, matching DeepSeek-V4’s compression stride. 24 to 39 tokens, the top-1 prediction flips periodically (Figure 1b,c). In particular, the model predicts8 whenL mod4∈2,3and32at the remaining eight lengths, andP(8)− P(32) ranges from−0.79 to 0.95. Changing only the docstring length thus reverses the preferred continuation. The reversals repeat every four tokens, matching the stride S = 4 of DeepSeek-V4’s compressed sparse attention (CSA). This match suggests that the prediction depends on where the code falls relative to the compression windows. The reversals are not limited to one filler. In all four filler families we tested, most predictions follow the same four-token pattern (Appendix B.1), and the post-trained DeepSeek-V4-Flash-0731 and DeepSeek-V4.1- Flash [12] also show periodic reversals under several filler sweeps, matching their compression strides (S= 4 andS= 2 respectively). In contrast, DeepSeek-V3.1-Base, without chunked KV compression, ranks32above 8at only 4 of the same 64 inputs, with very small margins (Appendix B.2). We give the exact inputs, results for the other models, and two further code-completion examples in Appendix B. 2.2 Measuring Phase Sensitivity in Retrieval To measure this effect more systematically, we turn to a controlled key–value based retrieval task in which we can measure the model’s capability of retrieving the same piece of information, but at different relative positions with respect to the compression boundaries. For compression strideS, we callt mod Sthephase of a key-value pair starting at positiont, and we say that retrieval isphase-sensitivewhen certain retrieval metric, such as accuracy or the probability assigned to the correct answer, changes systematically with the phase of the source. We evaluate the base and post-trained versions of DeepSeek-V4-Flash and DeepSeek-V4-Pro, as well as the post-trained DeepSeek-V4.1-Flash, on single-token needle-in-a-haystack (NIAH) retrieval. Each 128K-token prompt holds 16k key–value pairs and asks for the value of a source key near its middle (Figure 2a). We sort the prompts into eight residue groups by the source key’s position modulo eight, which covers two stride cycles for DeepSeek-V4 and four for DeepSeek-V4.1. All groups share the same prompt length, query position, and key–value bindings, and we choose their filler lengths so that the mean and variance of the source key’s absolute position are the same in every group. We defer details about the construction of the retrieval dataset to Appendix C.1. In Figure 2, we can see that the base checkpoints are sharply phase-sensitive, with maximum gaps of up to 40.2 percentage points. Post-training raises accuracy and narrows these gaps, but the gaps of the post-trained DeepSeek-V4 models remain large. DeepSeek-V4.1-Flash has the smallest gap, yet all four of its even residue 3 Some special key–value pairs are hidden in the text below. (variable length natural prefill ...) (variable length natural postfill ...) K1:V1. K2:V2. ... ... Ki:Vi. ... ... Kj:Vj. ... Kn:Vn. Answer using the pairs above. What is the value for key Ki? The value for key Ki is ? (a) Prompt schematic 048 0 1 Accuracy (b) DeepSeek-V4-Flash Δ 40.2 p Δ 19.1 p BasePost-trained-0731 048 (c) DeepSeek-V4-Pro Δ 34.8 p Δ 14.8 p BasePost-trained-0813 02468 (d) DeepSeek-V4.1-Flash Δ 6.1 p Post-trained-0910 Source-key's Position Index Modulo 8 Figure 2 NIAH retrieval accuracy of DeepSeek-V4 family for key-value pairs with different offsets. All five models exhibitphase sensitivity, with a period of 2 or 4 matching the CSA stride. An answer counts as correct if, after stripping whitespace, it begins with the gold value. The marker for residuerof the target key’s position modulo 8 sits between ticksrandr+1. Gray bands mark stride boundaries, shading shows two-sided 95% Wilson intervals, and ∆ is the best-to-worst gap. groups remain above all four odd ones. Average accuracy alone can conceal these differences across phases. 3 Controlled Pretraining Studies DeepSeek-V4 differs from a standard transformer in many ways besides chunked KV compression, and we cannot retrain it without compression to see whether the periodicity disappears. To test whether chunked compression is the source of phase sensitivity, we pretrain a Qwen3-derived family of models from scratch in which chunked compression is the only substantial architectural change, together with full-attention baselines trained the same way. We also vary how compression is performed to see which aspects of phase sensitivity, such as its period, depend on that design. These models also support the mechanistic analyses in Section 4. 3.1 A Family of KV Compression Kernels Each kernel compresses a window into one key and one value per KV head. Apayloadmapsets what each token contributes, and acompressiongate sets how strongly it enters the summary. For one head, leth j,i ∈ R d be the hidden state at offset 0≤ i < Win a windowjof sizeW. For branch B ∈ K,V, the payload map, gate map, and positional bias have shapesC B i ∈ R d h ×d ,Z B i ∈ R d g ×d , and b B i ∈ R d g . Heredis the hidden width,d h is the head dimension, andd g is the gate dimension. For the key branch, we compute gate logits s K j,i = Z K i h j,i +b K i .(1) A softmax over the window turns these logits into gate scores, which weight the payloads: α K j,i = exp(s K j,i ) P W−1 u=0 exp(s K j,u ) , b K j = W−1 X i=0 α K j,i ⊙ C K i h j,i .(2) Scalar gates (d g = 1) weight all payload dimensions together, while vector gates (d g =d h ) weight them separately. Softmax acts along the offset dimension, and⊙broadcasts scalar scores or multiplies element-wise. The value branch is analogous. We vary this family to test candidate explanations of phase sensitivity. Our reference model has one KV head per layer and uses non-overlapping eight-token windows with scalar gates. Window sizeWsets how many tokens an entry summarizes, and strideSsets how often an entry is produced, so the reference is W8/S8. Varying them separately shows whether the period follows the window or the stride, and whether overlapping windows (S < W), in which a token appears at several offsets while its phase stays fixed, remove the effect. 4 Replacing learned gates with uniform averaging, which fixes each gate score at 1/W, tests whether phase sensitivity requires learned gates. We describe parameter-sharing variants in Appendix D. The family also includes DeepSeek-V4’s CSA kernel [11], which tests whether the kernel studied in Section 2 produces phase sensitivitywhile being used on the Qwen backbone. To measure phase sensitivity in these models, we adapt the NIAH task of Section 2.2 to 64 random key–value pairs with adjacent single-token keys and values, and report full-vocabulary next-token accuracy.Prefix Paddingfollows the construction in Section 2.2, shifting all records together with outer fillers. Because the records sit back to back, this shift changes every distractor’s phase along with the target’s, so a phase effect could in principle come from the distractors rather than the target. We therefore addIn-sequencePadding, which varies the gaps between records so that distractor phases no longer move in lockstep with the target. Within each protocol, we fix the query position and match the mean and variance of the target-position sampling distributions across residue groups of source-key position modulo 24, a common multiple of the tested strides (Appendix C.3). 3.2 Phase Sensitivity across the Kernel Family If phase sensitivity arises from chunked compression, compressed models should show it and full-attention baselines should not. We see this contrast in Figure 3. Every compressed model varies periodically with the source-key phase, including the DeepSeek-style kernel (Figure 3i), while the full-attention baselines stay flat (Figure 3a). The same holds for every compressed run in Table 8, with Prefix Padding gaps of up to 78 percentage points against at most 6.1 for the baselines. Note that summary metrics conceal much of this difference. With eight KV heads, the W8/S8 compressed model nearly matches its full-attention baseline in mean accuracy and validation loss, yet its worst residue is only 9.9%, against 58.4% for the baseline. At one and four KV heads, the compressed models reach higher mean accuracy than the baselines at comparable validation loss. Varying window and stride separately shows that the period follows the stride in the tested geometries, whether or not windows overlap. AtW= 8, stridesS ∈4,6,8give periods of about four, six, and eight tokens (Figure 3b–d). W4/S4 shares W8/S4’s four-token period despite its smaller window (Figure 10), and W12/S12 produces an approximately twelve-token period (Figure 3e). In these experiments, phase sensitivity needs neither rotary position embeddings (RoPE) nor learned gates: it persists without RoPE (Figure 3h) and when uniform averaging replaces the gates (Figure 11). In Appendix F, we extend these results across 04812162024 0 0.25 0.5 0.75 1.0 Accuracy (a) Full attention 1/4/8 KV baselines 1 KV: val 2.466 4 KV: val 2.448 8 KV: val 2.437 04812162024 (b) W=8, S=4 Compression Baseline Δ 39.8 p val loss 2.467 06121824 (c) W=8, S=6 Compression Baseline Δ 31.9 p val loss 2.470 081624 (d) W=8, S=8 Compression Baseline Δ 75.3 p val loss 2.478 01224 (e) W=12, S=12 Compression Baseline Δ 63.2 p val loss 2.487 081624 0 0.25 0.5 0.75 1.0 Accuracy (f) W=8, S=8 Vector gate ablation Δ 34.0 p val loss 2.478 081624 (g) W=8, S=8 Tied K/V ablation Δ 39.1 p val loss 2.495 081624 (h) W=8, S=8 NoPE ablation Δ 40.9 p val loss 2.504 04812162024 (i) W=8, S=4 DSV4 gate ablation Δ 38.3 p val loss 2.476 081624 (j) W=8, S=8 Tail-concat SWA ablation Δ 33.9 p val loss 2.520 Source-key's Position Index Modulo 24 Figure 3 Phase-sensitive retrieval accuracy by source-key position modulo 24. (a) shows full-attention baselines with one, four, and eight KV heads per layer. (b)–(e) show W8/S4, W8/S6, W8/S8, and W12/S12 models that otherwise match the one-KV-head reference. Ablations (f) uses vector gates and (g)–(j) vary K/V tying, positional encoding, compression kernel, and local-memory policy. Phase sensitivity persists across the compressed-model panels. 5 head counts, random seeds, and other variants, and to In-sequence Padding, so the pattern does not require all records to shift together. A natural first explanation is the window boundary, since adjacent keys and values can fall in separate windows. In the W8/S8 reference, only a key at phase seven has its value in the next window. Yet accuracy varies by more than 50 percentage points among phases zero through six, whose pairs share a window (Figure 3d). The boundary does not capture the full accuracy trend. This remaining variation suggests that the answer lies in how the model writes to and reads from compressed memory, so we next ask how retrieval at each phase depends on individual KV heads (Section 4). 4 Phase Specialization Retrieval at each phase runs through the attention heads that write and read compressed entries, and a head may serve some phases more reliably than others. We call thisphasespecialization: the contributions of some heads to retrieval depend systematically on phase, so different heads contribute asymmetrically to information at different phases. The definition presupposes no cause, whether boundary straddling or within-window differences. We measure this asymmetry with knockouts in our pretrained models and DeepSeek-V4-Flash-Base (Sec- tion 4.1), and then examine a sharper form in one model, where a head’s preference for some phases is also visible in how it compresses (Section 4.2). 4.1 Probing Phase Specialization through KV-Head Knockouts To find which heads matter for retrieval at each phase, we knock out one KV head at a time [24,38] and see where accuracy drops. In particular, we usemeanreplacement[35], which replaces its output at the intervened positions with its average over many other prompts, so that output no longer carries anything specific to the current prompt, such as which value belongs to the queried key. A drop in accuracy at a phase is evidence that the head’s prompt-specific output contributes causally to retrieval at that phase. In our pretrained models, we run these knockouts on the same evaluation sets used in Section 3, replacing the outputs of the query heads that read the KV head at the final position with their means over a disjoint set of 01234567 0 4 8 12 16 20 24 Layer (a) Qwen3 Full Attention (1KV) 01234567 0 4 8 12 16 20 24 (b) Qwen3 Compressed (W=8 S=8 1KV) 01234567 0 4 8 12 16 20 24 28 32 36 40 (c) DeepSeek-V4 Flash Base -100 -50 0 50 100 Relative Accuracy Change (%) Source-key's Position Index Modulo 8 Figure 4 Retrieval contributions differ across phases. (a) Qwen3 full attention, (b) Qwen3 with W8/S8 compression, and (c) DeepSeek-V4-Flash-Base, all with one KV head per layer. Each row knocks out one layer’s KV head, and columns group source positions modulo eight. Colors show accuracy change divided by clean accuracy at each phase. The model without chunked KV compression (a) shows uniform horizontal stripes across residue groups, indicating that each layer contributes similarly at every source position, while the models with chunked KV compression (b,c) show heterogeneous patterns across the residues of the target key. In (c), the pattern also repeats with period four, matching the CSA stride, and the odd layers, which use heavily compressed attention (HCA) with compression rate 128, contribute little to retrieval. 6 calibration prompts. The computation at other token positions stays unchanged. We adopt a similar knockout strategy for DeepSeek-V4-Flash-Base and replace a whole layer’s attention output at a band of query suffix, with 256 evaluation prompts per residue group (see Appendix E.6 for details). We visualize the effect of knocking out each head on retrieval performance for information at each phase in Figure 4. The knockout maps differ between full attention and compression. In the full-attention baseline, the layers with large losses reduce accuracy by similar amounts at every position (Figure 4a). In the W8/S8 compressed model, some heads have losses concentrated on a few phases, and different heads have different phase-dependent effects (Figure 4b). In DeepSeek-V4-Flash-Base, the largest losses affect every phase unevenly and a few earlier layers affect only some phases, repeating every four positions, its compression stride (Figure 4c). This pattern is not specific to one model. In Appendix F, we give knockout maps for every compressed model in Table 8 under both padding protocols. In most compressed models, some heads have losses concentrated on a few phases that repeat every stride, and in the others, losses spread over all phases with periodic variation. With several KV heads, some individual heads still show losses concentrated on one or two phases, so the asymmetry is not only a property of layers (Figure 9). 4.2 A Case Study of Emerging Gate Preferences Knockouts show which heads matter at which phases, but not why. A natural candidate for the source of cause is the compression gate, which sets each offset’s weight in a compressed entry. A head whose gate persistently favors some offsets should matter most at the corresponding phases. We test this in the W8/S8 reference model, whose scalar key and value gates have separate parameters for each offset and whose windows do not overlap (W = S), so offset i is phase i. Because gate scores depend on the input, a gate can concentrate its scores among offsets in two ways.Static concentrationfavors the same offsets in every window, whereasinput-dependentconcentrationfavors offsets that vary with content. Only static concentration ties a head to fixed phases. We measure both with the effectivenumberofoffsets, defined asexp(H(·)), whereHis Shannon entropy. This metric equals 1 for a one-hot distribution andWfor a uniform one. Withα∈ R W the gate scores of a window and expectations over held-out natural-language text, we measure static concentration byexp(H(E[α])) and the average per-window concentration byE[exp(H(α))]. A gate that applies the same scores to every window would be near the diagonal of Figure 5a, since expectation would commute when α is approximately constant. As shown in Figure 5, all gates appear to be near uniform around initialization, showing neither kind of 12468 1 2 4 6 8 step 1 step 47,5 Other Layers Layer 9 Layer 10 Layer 14 01234567 0.0 0.3 0.6 Layer 9: Key 01234567 Layer 9: Value 01234567 0.0 0.3 0.6 Gating Score Layer 10: Key 01234567 Layer 10: Value 01234567 Key offset 0.0 0.3 0.6 Layer 14: Key 01234567 Value offset Layer 14: Value 01234567 Source-key phase mod 8 Layer 0 Layer 4 Layer 8 Layer 12 Layer 16 Layer 20 Layer 24 −100 0 100 Relative Accuracy Change (%) (a) Emergent Dynamics of Specialized Heads (b) Gate Scoring of Different Specialized Heads (c) Causal Contribution to Retrieval by Specialized Heads exp ¡ H ¡ 피[® K ] ¢ 피 £ exp ¡ H ( ® K ) ¢¤ 18 Figure 5 Emergence and retrieval roles of gate preferences. (a) Key-gate trajectories during pretraining: some heads drift along the diagonal and become statically concentrated (as highlighted), while others remain unbiased in expectation withexp(H(E[α]))≈8. (b) The highlighted layers 14, 10, and 9 develop static preferences for early, middle, and late offsets in the compression window. (c) For the statically concentrated heads, their peaking phases directly match where they are causally important for retrieval. This is the same figure as Figure 4b for Qwen3 with W8/S8 compression. 7 concentration (open circles in Figure 5a, close to the upper right diagonal). During pretraining, some gates, such as those in layers 9, 10, and 14, move toward the lower left along the diagonal, forming static concentration. The other key gates end near the right edge, with different levels of input-dependent concentration but no consistent bias (gray points). In Figure 5b we visualize the mean gate scores of these heads. Each key gate peaks at a few adjacent offsets, early in the window for layer 14, in the middle for layer 10, and late for layer 9, and each value gate peaks one offset later, so each head favors a token and its successor, as in a bigram key–value pair. These peaks match the phases each head specializes in as observed in Section 4.1: its knockout losses concentrate where its key gate favors the queried token and its value gate favors the successor (Figure 5c). Apart from layer 0, the largest knockout losses come from statically concentrated heads. With more heads per layer, static concentration becomes sharper: in the W8/S8 model with four KV heads, many heads peak at a single offset and lose accuracy at a single phase (Figure 17). We defer further results across KV-head counts, random seeds, and compression configurations, including the final gate concentration of every head (Figure 14), to Appendix G. If learned gate preferences help determine which phases are retrieved well, cycling the offset-specific gate parameters bykoffsets should also approximately shift the accuracy pattern bykphases. In Appendix H, we test this prediction in five models with gates exhibiting sharp static concentration (including the W8/S8 reference model). For cycles of one to three offsets in either direction, within-window accuracy patterns largely shift as predicted. These interventions support a causal contribution of learned gate preferences to within-window retrieval asymmetry in these models. We conjecture that the observed phase sensitivity in accuracy reflects both boundary and within-window asymmetries. The first distinguishes key–value pairs that share a window from pairs that straddle one. The second concerns differences among phases within a window, with static gate concentration as one mechanism that can shape which phases each head retains. Accuracy in the reference model varies by more than 50 percentage points among phases whose key–value pairs all share a window (Section 3.2), while uniform averaging also exhibits within-window variation, showing that this second asymmetry can arise without learned gates. 5 Theory: Emergence of Phase Specialization in Compression In Section 4.2, some gates develop persistent preferences for the same offsets across windows, and these heads matter most at the corresponding phases. In an idealized induction model, we ask why compression should concentrate when the future query is unknown, why the concentration becomes static, and whether heads divide the phases among themselves. 5.1 Concentration in Optimal Solutions Consider a contextX= (x ℓ,r ) ℓ≤L,r≤S ofLSdistinct token embeddings, sampled uniformly without replace- ment from a vocabularyV ⊂ R d of sizeNand arranged inL ≥4 non-overlapping chunks of strideS ≥3. Positionrwithin a chunk is offsetr−1 and thus a single phase. After compression, a queryqrepeats a token sampled uniformly from theL(S−1) nonfinal chunk positions and requests its successoryin the same chunk [4, 24]. In particular, each of 1≤ H < S heads compresses chunk ℓ into k h,ℓ = S X r=1 p K h,ℓ (r)x ℓ,r ∈ R d , v h,ℓ = S X r=1 p V h,ℓ (r)x ℓ,r ∈ R d ,(3) where the compression coefficientsp K h,ℓ ,p V h,ℓ are probability vectors inR S and are chosen before the query is revealed. For cleaner analysis, we fix the embeddings during training and set the query, key, and value maps to identities. 8 Each head weights its compressed values by a softmax over the chunk scoresγ⟨q,k h,ℓ ⟩, withγ ∈(0,1], and a tied unembedding of scaleβ >0 maps the summed head outputs to next-token logits. We writeLfor the resulting population cross-entropy loss of predicting y (Appendix A). We assume an idealized embedding geometry with a common component κ > 0 and error ε≥ 0: x ⊤ x ′ − κ− 1x = x ′ ≤ ε ∀x,x ′ ∈V.(4) The common component reflects the anisotropy of token representations [14,26,29], and exact geometry means ε = 0. Under exact geometry, a head’s target-specific logit is proportional to its attention on the correct chunk times its value weight on the answer. We say that a headselectspositionrwhen it puts all its key weight onr and all its value weight onr+ 1, the most concentrated compression. Selection strengthens both factors for queries at r but gives up direct evidence for queries elsewhere. To see what the loss itself favors, we minimizeLover an oracle classG oracle , whose compression coefficients may depend arbitrarily on the context but not on the query, without imposing a gate parameterization. Writing e r for the one-hot vector at position r, we obtain the following result. Theorem 1 (Informal: optimal compression is concentrated). For sufficiently largeNand sufficiently small ε, every global minimizer overG oracle setsp K h,ℓ =e r andp V h,ℓ =e r+1 in every context, chunkℓ, and headh, wherer ∈1,...,S−1may depend on all three. Whenε= 0, the selected positions are distinct across heads within each context and chunk. Although the query is unknown, optimal compression is maximally concentrated, with one effective offset per head and per K/V branch in each chunk. The selected position may vary across contexts and chunks, however, so optimality requires concentration in each chunk but not static concentration. 5.2 Static Concentration under Gradient Flow To study when concentration becomes static, we restrict the oracle class toG linear , a simplified form of the gates in Equation (1). Each head has a key and a value gate vectorc B h,r ∈ R d for every positionr, shared across contexts and chunks, which are the only parameters we train: p B h,ℓ (r) = exp ⟨c B h,r ,x ℓ,r ⟩ P S j=1 exp ⟨c B h,j ,x ℓ,j ⟩ , B ∈K,V.(5) Under exact geometry, the component ofc B h,r along the embedding mean acts as an input-independent bias for positionr, so these gates can express static concentration, as in Section 4.2, although the parameterization alone does not force it. We analyze gradient flow onL, the continuous-time limit of gradient descent, from a small random initialization. Theorem 2 (Informal: gradient flow yields static concentration). Assumeε= 0 and uniform initialization in a Euclidean ball centered at zero of sufficiently small radiusρ, in the effective gate-parameter space of Assumption 11. For fixed model scales and sufficiently largeNrelative tolog(e/ρ), with high probability over the initialization, gradient flow onLyields, for each headh, a positionr h ∈ 1,...,S−1such that p K h,ℓ (r h ),p V h,ℓ (r h + 1)→1 as training time tends to infinity, uniformly over contexts and chunks. Different heads may select the same r h . For each head, the key gate scores converge to the one-hot vector atr h and the value gate scores to the one atr h + 1 in every context and chunk, so both the static and the per-window effective numbers of offsets tend to one, the lower-left corner of Figure 5a. Whereas Theorem 1 lets the selected position vary across inputs, gradient flow fixes it for each head. This gives an idealized account of the adjacent key and value preferences of layers 9, 10, and 14 (Figure 5b), whose gate scores remain softer than the one-hot limit. We defer the formal theorem statements and proofs to Appendix A, with relaxed conditions on the number of chunksL and attention scale γ. 9 6 Related Works Learned compressed memories reduce the cost of retaining long contexts. Compressive Transformer summarizes past activations [28], LoMA trains models to use compressed KV representations [36], Activation Beacon replaces raw activations with the KV entries of regularly spaced learned tokens [44], and CAT lets later chunks attend to compressed vectors of earlier chunks [25]. Sparse attention [8,11,20,41] and cache eviction [18,45] impose related block or token structure on memory. Sparse Transformer already defines a strided head whose visibility depends on position modulo a stride [8], a precedent for the periodic structure we study. In hybrids that keep an exact branch over distant tokens, such as NSA [41], that branch could mask a failure of the compressed branch alone. DeepSeek-V4 keeps exact tokens only in its local window [11], and we find phase sensitivity in the full models (Section 2). On the failure mode side, the Lost in the Middle effect shows that retrieval quality depends on where relevant information appears in a long context [19]. RULER shows that easy needle tests can overstate usable context length [16], and Chen et al.[5]show that aggregate scores can hide method-specific failures of KV cache compression. In gist-based compression, Deng et al.[13]identify degradation near generation-segment boundaries. We instead focus on the phase of the source token, which recurs every compression stride and is distinct from overall context depth or the query’s position within a segment. Mechanistic studies identify retrieval heads whose ablation impairs long-context retrieval [38], and cache policies such as RazorAttention and DuoAttention keep full caches only for such heads [31,39]. These works establish that heads differ in their retrieval roles. We show that in compressed models, a head’s retrieval contribution also depends on phase and, in the examined heads, aligns with the offsets its compression gate prefers (Section 4). On the theory side, gradient descent can turn softmax attention into hard token selectors [32,33], and multi-head training dynamics can allocate tasks across heads [6,42]. We do not claim the first theory of attention hardening or head specialization. Our analysis instead concerns compression chosen before the query is known, and asks when concentrated compression is optimal and when training makes it static (Section 5). We give a more extensive discussion of related work in Appendix I. 7 Conclusion and Discussion In this paper, we identify phase sensitivity as a systematic failure mode in models with chunk-compressed KV caches: retrieval accuracy depends on the source information’s phase relative to the compression windows. The phenomenon appears in DeepSeek-V4, V4.1, and all compressed models we pretrained, with a period following the compression stride. KV-head knockouts in our pretrained models and attention-readout interventions reveal phase-dependent contributions to retrieval, providing causal evidence for phase specialization in these tested models. Finally, our optimization dynamics analysis shows how training can produce static concentration of compression gates in a simplified bigram retrieval model, illustrating a mechanism for phase specialization under the stated assumptions. Our mechanistic conclusions are strongest in the controlled settings studied here and do not establish a universal cause of phase sensitivity in large, heterogeneous models. In particular, gate cycling does not restore the boundary phase (Appendix H), leaving a unified explanation of boundary and within-window asymmetries unresolved. Whether explicit coordination across heads and layers can make retrieval quality more uniform across phases, without sacrificing average performance or compression efficiency, remains open. Beyond controlled retrieval, our code-completion examples document periodic errors in DeepSeek-V4 and DeepSeek-V4.1. These observations motivate investigating whether phase sensitivity also affects robustness- relevant behavior. For example, do meaning-preserving shifts of a request within its context alter refusal or compliance, and do any such changes track the compression phase? We leave a systematic study of their potential robustness implications in real application scenarios to future work. 10 References [1]agentlans. High-quality English sentences. Hugging Face dataset, revision5dfcb23, 2024. URLhttps:// huggingface.co/datasets/agentlans/high-quality-english-sentences. Train split, text field. [2]Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. InProceedingsof the2023ConferenceonEmpiricalMethodsinNaturalLanguageProcessing, pages 4895–4901. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.298. URLhttps://aclanthology.org/ 2023.emnlp-main.298/. [3] Simran Arora, Sabri Eyuboglu, Aman Timalsina, et al. Zoology: Measuring and improving recall in efficient language models. InInternationalConferenceonLearningRepresentations, 2024. URLhttps://arxiv.org/ abs/2312.04927. [4]Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Hervé Jégou, and Léon Bottou. Birth of a transformer: A memory viewpoint. InAdvancesinNeuralInformationProcessingSystems, 2023. URLhttps://arxiv.org/ abs/2306.00802. [5]Alex Chen, Renato Geh, Aditya Grover, Guy Van Den Broeck, and Daniel Mingyi Israel. The pitfalls of KV cache compression. InProceedingsofthe64thAnnualMeetingoftheAssociationforComputational Linguistics(Volume1:LongPapers), pages 41530–41553. Association for Computational Linguistics, 2026. doi: 10.18653/v1/2026.acl-long.1926. URL https://aclanthology.org/2026.acl-long.1926/. [6]Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality (extended abstract). InConferenceonLearning Theory, volume 247, page 4573, 2024. URL https://proceedings.mlr.press/v247/siyu24a.html. [7]Yinpeng Chen, DeLesley Hutchins, Aren Jansen, Andrey Zhmoginov, David Racz, and Jesper Andersen. MELODI: Exploring memory compression for long contexts. InInternationalConferenceonLearningRepresentations, 2025. URL https://arxiv.org/abs/2410.03156. [8] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXivpreprintarXiv:1904.10509, 2019. URL https://arxiv.org/abs/1904.10509. [9] Chenglong Chu, Guorui Zhou, Guowang Zhang, Han Li, Hao Peng, Hongtao Cheng, Jian Liang, Jiangxia Cao, Kun Gai, et al. Kwai summary attention technical report.arXivpreprintarXiv:2604.24432, 2026. URL https://arxiv.org/abs/2604.24432. [10] DeepSeek-AI. DeepSeek-V3 technical report.arXivpreprintarXiv:2412.19437, 2025. URLhttps://arxiv.org/ abs/2412.19437. [11] DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence.arXivpreprint arXiv:2606.19348, 2026. URL https://arxiv.org/abs/2606.19348. [12]DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression.arXivpreprintarXiv:2609.19969, 2026. URL https://arxiv.org/abs/2609.19969. [13]Chenlong Deng, Zhisong Zhang, Kelong Mao, Shuaiyi Li, Xinting Huang, Dong Yu, and Zhicheng Dou. A silver bullet or a compromise for full attention? A comprehensive study of gist token-based context compression. InProceedingsofthe63rdAnnualMeetingoftheAssociationforComputationalLinguistics(Volume1:Long Papers), pages 4861–4879. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.241. URL https://aclanthology.org/2025.acl-long.241/. [14]Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. Representation degeneration problem in training natural language generation models. InInternationalConferenceonLearningRepresentations, 2019. URL https://openreview.net/forum?id=SkEYojRqtm. [15] Zifan He, Yingqi Cao, Zongyue Qin, Neha Prakriya, Yizhou Sun, and Jason Cong. HMT: Hierarchical memory transformer for efficient long context language processing. InProceedingsofthe2025ConferenceoftheNations oftheAmericasChapteroftheAssociationforComputationalLinguistics:HumanLanguageTechnologies, 2025. URL https://aclanthology.org/2025.naacl-long.410/. 11 [16]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? InFirstConference onLanguageModeling, 2024. URL https://arxiv.org/abs/2404.06654. [17]Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. InInternationalConferenceonMachineLearning, pages 21502–21521, 2024. URL https://proceedings.mlr.press/v235/jelassi24a.html. [18]Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. InAdvancesin NeuralInformationProcessingSystems, 2024. URL https://arxiv.org/abs/2404.14469. [19] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.TransactionsoftheAssociationforComputational Linguistics, 12:157–173, 2024. doi: 10.1162/tacl_a_00638. URL https://aclanthology.org/2024.tacl-1.9/. [20]Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, et al. MoBA: Mixture of block attention for long-context LLMs. InAdvancesinNeuralInformationProcessingSystems, 2025. URL https://arxiv.org/abs/2502.13189. [21]Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. InAdvancesinNeuralInformationProcessingSystems, 2023. URLhttps://arxiv.org/abs/2305. 16300. [22]Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. Learning to compress prompts with gist tokens. InAdvancesin NeuralInformationProcessingSystems, 2023. URL https://arxiv.org/abs/2304.08467. [23]Piotr Nawrot, Jianing Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, and Edoardo M. Ponti. The sparse frontier: Sparse attention trade-offs in transformer LLMs. InFindingsoftheAssociationforComputational Linguistics:ACL2026, pages 38667–38701. Association for Computational Linguistics, 2026. doi: 10.18653/v1/ 2026.findings-acl.1926. URL https://aclanthology.org/2026.findings-acl.1926/. [24]Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, et al. In-context learning and induction heads.TransformerCircuitsThread, 2022. URL https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html. [25]Jatin Prakash, Aahlad Puli, and Rajesh Ranganath. Controllably efficient language models.arXivpreprint arXiv:2511.05313, 2025. URL https://arxiv.org/abs/2511.05313. [26] Enrique Queipo-de Llano, Alvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael Bronstein, Yann LeCun, and Ravid Shwartz-Ziv. Attention sinks and compression valleys in LLMs are two sides of the same coin. In InternationalConferenceonLearningRepresentations, pages 13911–13937, 2026. URLhttps://proceedings. iclr.c/paper_files/paper/2026/hash/1734b19d9afe7d2c7f1154954eaf0d5a-Abstract-Conference.html. [27]Qwen Team. Qwen3 technical report.arXivpreprintarXiv:2505.09388, 2025. URLhttps://arxiv.org/abs/ 2505.09388. [28]Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. InInternationalConferenceonLearningRepresentations, 2020. URL https://openreview.net/forum?id=SylKikSYDH. [29]Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov. The shape of learning: Anisotropy and intrinsic dimensions in transformer-based models. InFindingsof theAssociationforComputationalLinguistics:EACL2024, pages 868–874. Association for Computational Linguis- tics, 2024. doi: 10.18653/v1/2024.findings-eacl.58. URLhttps://aclanthology.org/2024.findings-eacl.58/. [30]Margarita Sagitova, Odilon Duranthon, and Lenka Zdeborová. Specialization of softmax attention heads: Insights from the high-dimensional single-location model.arXivpreprintarXiv:2603.03993, 2026. URLhttps: //arxiv.org/abs/2603.03993. [31]Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Danning Ke, Shikuan Hong, Yiwu Yao, and Gongyi Wang. RazorAttention: Efficient KV cache compression through retrieval heads. InInternationalConferenceonLearning Representations, 2025. URL https://arxiv.org/abs/2407.15891. [32]Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak. Transformers as support vector machines.arXivpreprintarXiv:2308.16898, 2023. URL https://arxiv.org/abs/2308.16898. 12 [33]Davoud Ataee Tarzanagh, Yingcong Li, Xuechen Zhang, and Samet Oymak. Max-margin token selection in attention mechanism. InAdvancesinNeuralInformationProcessingSystems, 2023. URLhttps://arxiv.org/ abs/2306.13596. [34]Aditya Varre, Mark Rofin, and Nicolas Flammarion. Gradient flow polarizes softmax outputs towards low-entropy solutions.arXivpreprintarXiv:2603.06248, 2026. URL https://arxiv.org/abs/2603.06248. [35] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. InInternationalConferenceonLearning Representations, 2023. URL https://arxiv.org/abs/2211.00593. [36]Yumeng Wang and Zhenyang Xiao. LoMA: Lossless compressed memory attention.arXivpreprint arXiv:2401.09486, 2024. URL https://arxiv.org/abs/2401.09486. [37]Zihao Wang, Bin Cui, and Shaoduo Gan. SqueezeAttention: 2D management of KV-cache in LLM inference via layer-wise optimal budget. InInternationalConferenceonLearningRepresentations, 2025. URLhttps: //arxiv.org/abs/2404.04793. [38]Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. InInternationalConferenceonLearningRepresentations, 2025. URLhttps://arxiv. org/abs/2404.15574. [39] Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads. InInternational ConferenceonLearningRepresentations, 2025. URL https://arxiv.org/abs/2410.10819. [40]Hongru Yang, Zhangyang Wang, Jason D. Lee, and Yingbin Liang. Transformers provably learn two-mixture of linear classification via gradient flow. InInternationalConferenceonLearningRepresentations, 2025. URL https://openreview.net/forum?id=AuAj4vRPkv. [41]Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. InProceedingsofthe63rdAnnualMeeting oftheAssociationforComputationalLinguistics(Volume1:LongPapers), pages 23078–23097. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.1126. URLhttps://aclanthology.org/2025. acl-long.1126/. [42]Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, and Nicolas Flammarion. Incremental learning of sparse attention patterns in transformers. InInternationalConferenceonMachineLearning, 2026. URLhttps://arxiv.org/ abs/2602.19143. [43] Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M. Susskind. Stabilizing transformer training by preventing attention entropy collapse. In InternationalConferenceonMachineLearning, pages 40770–40803, 2023. URLhttps://proceedings.mlr. press/v202/zhai23a.html. [44]Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Long context compression with activation beacon. InInternationalConferenceonLearningRepresentations, 2025. URLhttps://arxiv. org/abs/2401.03462. [45]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, et al. H2O: Heavy-hitter oracle for efficient generative inference of large language models. InAdvancesinNeuralInformation ProcessingSystems, 2023. URL https://arxiv.org/abs/2306.14048. 13 Appendix A Deferred Proofs for Theorem 1 and Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.1 A Two-Phase Toy Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.2 Model and Compressor Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 A.3 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 A.4 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 B Additional Details on the Code-Completion Examples . . . . . . . . . . . . . . . . . . . . . . 31 B.1 FP8 Example for DeepSeek-V4-Flash-Base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 B.2 FP8 Example for Other DeepSeek Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 B.3 Additional Code-Completion Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 C Additional Details on the Needle-in-a-Haystack Evaluations . . . . . . . . . . . . . . . . . 38 C.1 NIAH Evaluation for the DeepSeek-V4 Family . . . . . . . . . . . . . . . . . . . . . . . . . . 38 C.2 NIAH Evaluation for Qwen3-based Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 C.3 Padding Protocols for Qwen3-based Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 D Additional Details on the Architecture and Pretraining of Qwen3-based Models . . . . . 43 D.1 Backbone and Compression Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 D.2 Windows, Attention Memory, and Positional Encoding . . . . . . . . . . . . . . . . . . . . . . 43 D.3 Training Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 E Additional Details on KV-Head Knockouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 E.1 Replaced Attention Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 E.2 Calibration and Mean Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 E.3 Evaluation Prompts, Metrics, and Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . 47 E.4 Knockout Sweeps for Qwen3-based Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 E.5 Scope of the Causal Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 E.6 Knockouts for DeepSeek-V4-Flash-Base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 F Complete Results for Qwen3-based Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 G Gate Preferences and Knockout Effects across Models . . . . . . . . . . . . . . . . . . . . . 54 G.1 Measuring Gate Preferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 G.2 Variation across Random Seeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 G.3 Other Configurations and Overlapping Windows . . . . . . . . . . . . . . . . . . . . . . . . . 55 H Redirecting Gate Preferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 H.1 Gate Cycling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 H.2 One-Offset Cycles in a W12/S12 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 H.3 Replication across Models and Cycle Sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 I Extended Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 14 A Deferred Proofs for Theorem 1 and Theorem 2 Here we state and prove formal versions of Theorem 1 and Theorem 2. Theorem 8 shows that every minimizer over the oracle class uses hard adjacent selection, with each head placing all key weight on one phase and all value weight on the next. We call such a pair of key and value phases a route. The selected route may change across contexts and chunks. Theorem 12 shows that gradient flow on linear compressors from a small random initialization reaches with high probability hard adjacent selection with one route per head for all inputs, although different heads may share a route. We state the formal conditions more generally than in Section 5. The assumptionsL≥4 andγ ≤1 made there implyγ < log3≤ log(L−1), so the formal conditions hold in the main-text setting. Both results turn on a tradeoff between averaging and selecting. A head that averages the positions of a chunk keeps some evidence about every position, whereas a head that selects one position keeps that position’s evidence at full strength and gives up direct evidence about the others. Selection is favored when the gain at the selected position outweighs the loss elsewhere. A second question is whether several heads divide the positions among themselves, which we call complementary coverage, since sharp heads can serve as many positions as possible only if their selections differ. In the general model, this tradeoff is hard to see directly. Each head has key and value coefficients overS positions in every chunk and attends to chunks through its own softmax, all heads are coupled through the softmax over the vocabulary, and under gradient flow the coefficients come from gates that depend on the input embeddings. We would therefore like to see the tradeoff in a setting small enough to solve by hand before proving the general results. In Appendix A.1, we study a toy example with two heads and two phases, in which each head’s compression is set by a single number and the loss is a function of two variables. There we can compute in closed form why a head sharpens its compression and why the two heads prefer to split the phases between them. The toy example is not a special case of the general theorems, since it hasS= 2, keeps only keys, and idealizes the value readout. We use it for intuition, and at its end we explain which parts of its argument carry over to the general model and which do not (Appendix A.1.2). We then return to the general model. In Appendix A.2, we define the model and the two compressor classes. In Appendix A.3, we prove Theorem 8, in which sharpening reappears for paired key and value coefficients. In Appendix A.4, we prove Theorem 12, using the toy example’s strategy of starting near a symmetric point and asking along which directions gradient flow leaves it. A.1 A Two-Phase Toy Example In the toy example, each chunk holds two key entries, each head compresses them with a single trainable logit, and an idealized readout replaces the values. Unlike Theorem 2, which does not ensure that different heads select different routes, gradient flow here assigns the two heads different phases, provided the output scaleβ is large enough (condition (16) below). Each ofLchunks contains two fixed key entriesk ℓ,1 ,k ℓ,2 . Each headh∈1,2uses a trainable logitθ h to form one compressed key: p h = σ(θ h ), ̄ k h,ℓ = p h k ℓ,1 + (1− p h )k ℓ,2 . Hereσ(t) = (1 +e −t ) −1 ,p h = 1 selects phase 1,p h = 0 selects phase 2, andp h = 1/2 mixes them equally. A subsequent query requests either phase with equal probability. Under orthogonal addressing, the analogue of exact geometry withκ= 0, a query has inner product one with its requested key in the unique target chunk and zero with every other key. With attention scale one, the target-chunk logit of headhis thereforep h for phase 1 and 1− p h for phase 2, while allL−1 distractor logits are zero. The total correct-chunk attention masses of the two heads for phases 1 and 2 are ̄ R 1 = 2 X h=1 e p h e p h + L− 1 , ̄ R 2 = 2 X h=1 e 1−p h e 1−p h + L− 1 . The value readout is idealized: each head contributes one unit of correct-class evidence per unit of attention mass on the target chunk, and the contributions of the two heads add. With a fixed output scaleβ >0 and no 15 additional constant offset, the correct-class margins for phases 1 and 2 areβ ̄ R 1 andβ ̄ R 2 . Training minimizes the population cross-entropy, which is the logistic loss of this margin averaged over the two phases: L(p 1 ,p 2 ) = log(1 + e −β ̄ R 1 ) + log(1 + e −β ̄ R 2 ) /2.(6) Set D = (L− 1)e −1/2 , so that D > 1 for L≥ 3. Informally, moving any mixedp h ∈(0,1) to either endpoint increases ̄ R 1 + ̄ R 2 , and at a fixed sum the convexity of the logistic loss in attention mass favors ̄ R 1 = ̄ R 2 . Together these favor the two complementary endpoint optima established below. A.1.1 Complementary Two-Phase Selection Training updates the logitsθ h rather thanp h , so we rewrite the objective in terms of them. Letθ= (θ 1 ,θ 2 ), and write head h’s signed phase preference as χ(θ h ) = 2p h − 1 = tanh(θ h /2).(7) Fors= +1 (phase 1) ors=−1 (phase 2), the target-chunk logit of headhis (1+sχ(θ h ))/2, so its correct-chunk attention mass is a(sχ(θ h )), a(x) = 1 1 + De −x/2 .(8) In these coordinates, the two total attention masses and the logistic loss are ̄ R 1 = a(χ(θ 1 )) + a(χ(θ 2 )), ̄ R 2 = a(−χ(θ 1 )) + a(−χ(θ 2 )),(9) φ(m) = log(1 + e −βm ),L(p 1 ,p 2 ) = 1 2 φ( ̄ R 1 ) + φ( ̄ R 2 ) ,(10) which is Equation (6) rewritten throughχ(θ h ) = 2p h −1. The corresponding gradient flow is ̇ θ= −∇ θ L(σ(θ 1 ),σ(θ 2 )), and φ ′ < 0 < φ ′ . To analyze this flow, split retrieval into its even and odd parts: a + (x) = a(x) + a(−x) 2 , a − (x) = a(x)− a(−x) 2 , ̄a + (θ) = a + (χ(θ)), ̄a − (θ) = a − (χ(θ)). (11) Then ̄ R 1 = M + I and ̄ R 2 = M − I, where M = ̄a + (θ 1 ) + ̄a + (θ 2 ), I = ̄a − (θ 1 ) + ̄a − (θ 2 ),(12) and hence L(p 1 ,p 2 ) = 1 2 [φ(M + I) + φ(M − I)].(13) This decomposition separates sharpening from balancing. For x > 0, set t = e x/2 > 1. Then 2a + (x) = t t + D + 1 1 + Dt ,2 d dt a + (2 logt) = D 1 (t + D) 2 − 1 (1 + Dt) 2 > 0(14) forD >1 andx >0. Thus increasing|θ h |increases its contribution to the mean attention mass. At fixedM, strict convexity givesL(p 1 ,p 2 )≥ φ(M), with equality only atI= 0. Hence retrieval competition favors sharp heads, whereas loss curvature favors balanced coverage. Both pressures are satisfied together only when the two heads sit at opposite endpoints, and the following theorem makes this precise and shows that gradient flow from near the uniform point reaches these endpoints whenβis large enough. LetG(x) =a(x) +a(−x) = 2a + (x), which is strictly increasing on [0,1] by the calculation above. 16 Theorem 3 (Complementary two-phase selection). SupposeD >1. On [0,1] 2 , the only global minimizers of L(p 1 ,p 2 ) are (1, 0) and (0, 1), corresponding in the compactified θ-space to (+∞,−∞) and(−∞, +∞).(15) If also 2βσ 2β 1 + D > D− 1 D ,(16) then any absolutely continuous initialization supported in a sufficiently small neighborhood of the origin converges almost surely under gradient flow to one of these minimizers. If its density is bounded, then for δ,η ∈(0,1/2), outside an event of probability at mostδ, the time until (p 1 (t),p 2 (t)) lies withinℓ ∞ -distanceη of (1, 0) or (0, 1) is T η,δ = O log 1 δ + 1 η ,(17) where the hidden constant depends onL,β, and the initialization law through its fixed neighborhood and density bound. We establish the loss landscape first, then show that nearby trajectories escape toward and remain near a complementary route. Lemma 4 (Global minimizers). If D > 1, the only minimizers of L on [0, 1] 2 are (1, 0) and (0, 1). Proof. The mean attention mass is ̄ R 1 + ̄ R 2 2 = G(|χ(θ 1 )|) + G(|χ(θ 2 )|) 2 ≤ G(1).(18) Jensen’s inequality and monotonicity of φ yield L(p 1 ,p 2 )≥ φ ̄ R 1 + ̄ R 2 2 ≥ φ(G(1)).(19) Equality in the second inequality requires|χ(θ 1 )|=|χ(θ 2 )|= 1, and strict convexity makes the first tight only when ̄ R 1 = ̄ R 2 , which forces opposite signs. The remaining lemmas analyze the flow near the complementary line (θ 1 ,θ 2 ) = (ξ,−ξ). Use the orthonormal coordinates u = θ 1 − θ 2 √ 2 , v = θ 1 + θ 2 √ 2 .(20) For ξ ≥ 0, put ̄ M ξ = 2 ̄a + (ξ) = G(χ(ξ))(21) and let λ ⊥ (ξ) = φ ′ ( ̄ M ξ ) ̄a ′ + (ξ) + 2φ ′ ( ̄ M ξ ) ̄a ′ − (ξ) 2 .(22) This is the Hessian eigenvalue in the common-head direction (1,1)/ √ 2at (ξ,−ξ). At the uniform point, ̄ M 0 = 2/(1 + D), and the escape rate along (1,−1)/ √ 2 is λ esc =−φ ′ ( ̄ M 0 ) D(D− 1) 16(1 + D) 3 > 0.(23) Lemma 5 (Stability of complementary routing). SupposeD >1 and Equation (16) holds. The complementary line is invariant, λ ⊥ (ξ) > 0 for every ξ ≥ 0, and inf ξ≥0 e ξ λ ⊥ (ξ) > 0.(24) 17 Proof.Invariance follows by exchanging the two heads and phases. Letx=χ(ξ). Where ̄a ′ + (ξ)>0, the curvature satisfies ̄a ′ + (ξ) 2( ̄a ′ − (ξ)) 2 ≤ 1 2 D− 1 D .(25) Indeed, a ′ + (x) > 0 for x > 0, and ̄a ′ + (ξ) 2( ̄a ′ − (ξ)) 2 = a ′ + (x)− 2xa ′ + (x)/(1− x 2 ) 2(a ′ − (x)) 2 ≤ a ′ + (x) 2(a ′ − (x)) 2 .(26) To verify the last bound, putt=e x/2 ≥1 andQ= (1 +Dt) 2 + (D+t) 2 . Substitutinga(x) = (1 +De −x/2 ) −1 and clearing positive denominators gives (D 2 − 1)tQ 2 − 2(D + t)(1 + Dt) × (D− t)(1 + Dt) 3 + (Dt− 1)(D + t) 3 = (t− 1) 2 (D− 1)P D (t), where P D (t) = (t + t 3 )(1 + D 5 ) + (t + 8t 2 + t 3 )(D + D 4 ) + (2 + 6t + 4t 2 + 6t 3 + 2t 4 )(D 2 + D 3 ). Every coefficient is positive, proving Equation (25). Furthermore, φ ′ ( ̄ M ξ ) −φ ′ ( ̄ M ξ ) = βσ(β ̄ M ξ )≥ βσ(β ̄ M 0 ) = φ ′ ( ̄ M 0 ) −φ ′ ( ̄ M 0 ) ,(27) because ̄ M ξ =G(χ(ξ)) is increasing. Combining this inequality, Equation (16), and Equation (25) proves λ ⊥ (ξ) > 0 whenever ̄a ′ + (ξ) > 0. If ̄a ′ + (ξ)≤ 0, positivity is immediate from φ ′ < 0 < φ ′ . Finally, e ξ λ ⊥ (ξ) is continuous and positive for finite ξ, and direct differentiation gives e ξ λ ⊥ (ξ)−→ 2[−φ ′ (G(1))]a ′ + (1) > 0.(28) Compactness after adjoining ξ =∞ proves Equation (24). We next use this curvature bound to trap trajectories in a tube around the complementary line. Lemma 6 (Attracting complementary tube). SupposeD >1 and Equation (16) holds. For everyξ ∗ >0, there are ζ T ,c T ,C T ,c N > 0 such that, in the coordinates (θ 1 ,θ 2 ) = ξ + v √ 2 ,−ξ + v √ 2 , ξ ≥ ξ ∗ , |v|≤ ζ T , the gradient flow satisfies c T e −ξ ≤ ̇ ξ ≤ C T e −ξ , v ̇v ≤−c N e −ξ v 2 .(29) This tube is forward invariant, and its trajectories have ξ(t)→∞ and v(t)→ 0. Proof. The tangential speed at v = 0 is Ω(ξ) :=− 1 4 φ ′ (G(χ(ξ)))G ′ (χ(ξ)) 1− χ(ξ) 2 > 0.(30) On the line v = 0, ̇ ξ = Ω(ξ) > 0, and the transverse linearization is ̇v =−λ ⊥ (ξ)v. Since χ(ξ) = tanh(ξ/2), e ξ Ω(ξ)−→−φ ′ (G(1))G ′ (1) > 0,(31) 18 while Equation (24) boundse ξ λ ⊥ (ξ) away from zero. Every derivative with respect to a saturated logit carries a factor χ ′ (ξ) = O(e −ξ ). Taylor expansion in v, uniform after weighting the vector field by e ξ , gives ̇ ξ = Ω(ξ) + O(e −ξ |v|),(32) ̇v =−λ ⊥ (ξ)v + O(e −ξ v 2 ).(33) The weighted coefficients extend continuously toξ=∞. Their positive lower bounds and finite upper bounds on [ξ ∗ ,∞] yield Equation (29) after choosingζ T sufficiently small. The sign ofv ̇vmakes the tube forward invariant, and the tangential bounds giveξ(t)→∞. Moreover, wheneverv ̸= 0, dlog|v|/dξ ≤−c N /C T , so v(t)→ 0. It remains to show that trajectories starting near the uniform point enter this tube. Lemma 7 (Exit into the complementary tube). Under the assumptions of Lemma 6, there areu ∗ ,ρ 0 >0 such that every trajectory with ∥(u(0),v(0))∥ 2 ≤ ρ 0 and u(0)̸= 0 reaches |u| = u ∗ within time τ ≤ 2 λ esc log u ∗ |u(0)| .(34) At exit, the trajectory lies in the tube from Lemma 6 with ξ ∗ = u ∗ / √ 2, after exchanging heads if u(0) < 0. Proof.The symmetry linesu= 0 andv= 0 are invariant. Smoothness and the two Hessian signs at the origin give ̇u = u (λ esc + O(∥(u,v)∥ 2 )), ̇v = v (−λ ⊥ (0) + O(∥(u,v)∥ 2 )).(35) Chooseu ∗ >0 small enough that the coefficients in Equation (35) stay within half their origin magnitudes whenever|u|,|v| ≤ u ∗ . Apply Lemma 6 atξ ∗ =u ∗ / √ 2and reduce its tube widthζ T to at mostu ∗ . Then whenever |u|≤ u ∗ and |v|≤ ζ T , d dt log|u|≥ λ esc 2 , d dt |v|≤− λ ⊥ (0) 2 |v|. The second inequality holds forv ̸= 0, and the linev= 0 is invariant. Chooseρ 0 < minu ∗ ,ζ T . Then |v|cannot reachζ T before|u|reachesu ∗ . Integrating the first inequality gives Equation (34). At exit ξ =|u(τ )|/ √ 2 = ξ ∗ and |v(τ )|≤ ρ 0 < ζ T , so the point lies in the tube. Proof of Theorem 3.Lemma 4 identifies the only global minima, while Lemmas 5 to 7 give convergence for every sufficiently small initialization offu= 0, which has probability zero under an absolutely continuous law. For the rate, enclose the fixed initialization support in a ball of radiusρ≤ ρ 0 and let its density be bounded byC. ThenPr(|u(0)|≤ t)≤4Cρt. Outside an event of probability at mostδ, we have|u(0)| > δ/(4Cρ), so Equation (34) gives tube entry inO(log(1/δ)) time. The normal coordinate then remains withinζ T . To reach ℓ ∞ -error at mostη, it suffices thatξ ≥ log((1− η)/η) +ζ T / √ 2. Integrating the tube bound ̇ ξ ≥ c T e −ξ up to this threshold (if it exceedsξ ∗ ) takes at moste ζ T / √ 2 /(c T η) additional time. This proves Equation (17) for δ,η ∈ (0, 1/2), and the same tube bounds ensure convergence to the corresponding minimizer. A.1.2 From the Toy Example to the General Model The toy example isolates two pressures, one favoring sharp heads and one favoring balanced coverage of the phases. Sharpening comes from competition with theL−1 distractor chunks. BecauseD >1, a uniform head gives the correct chunk less than half of its attention, and in this regime the correct-chunk attention mass rises more when the target logit increases than it falls when the logit decreases by the same amount. A head that moves toward one phase therefore gains more attention mass on that phase than it loses on the other, which raises the mean attention mass over the two phases (Equation (14)). Balance comes from the loss. Both heads add evidence to the same margin for each phase, and the logistic loss flattens as a margin grows, so evidence added to the phase with the larger margin lowers the loss less than evidence added to the 19 phase with the smaller one. For a fixed mean, the loss is therefore smallest when both phases receive the same attention mass. Both pressures are satisfied together only at the complementary endpoints, which is why these are the only global minimizers (Lemma 4). The same two pressures govern the dynamics. At the uniform point, the two heads are identical. Along the complementary direction (1,−1)/ √ 2, in which the heads move toward different phases, the two phases stay balanced, so only sharpening acts and trajectories escape at rateλ esc . Along the common direction (1,1)/ √ 2, in which both heads move toward the same phase, sharpening and balance compete. Condition(16) is sufficient for balance to win, which makes this direction stable and the uniform point a saddle (Lemma 5). A small random initialization therefore leaves the uniform point along the complementary direction and then stays near the complementary line. Both pressures reappear in the general model, but they change in different ways. Sharpening survives, but it now comes from pairing keys with values rather than from the asymmetry thatD >1 provides. Under exact geometry, a head’s contribution to the target logit for a query at positionris its correct-chunk attention, which grows with its key weight at r, times its value weight at r + 1. Because the value weights sum to one, this contribution summed over query positions is at most the attention that a one-hot key receives, and it reaches that value only when the key coefficient is one-hot at somerand the value coefficient is one-hot at r+ 1. In Appendix A.3, we use this bound to rule out mixed coefficients, in the role that Lemma 4 plays for the toy example. Balance, by contrast, becomes weak when the vocabulary is large. In that regime, the cross-entropy loss is nearly linear in the target logit, because its normalizer is dominated by the many vocabulary items that do not appear in the context. Under exact geometry, two heads that select the same position keep the same expected target logit and pay only through the normalizer, by an amount at most of order 1/N. This penalty still makes complementary coverage optimal whenε= 0 (Appendix A.3). Our dynamics proof, however, treats all normalizer terms as small perturbations, and the remaining target terms do not couple different heads, so the proof has no force that separates them (Appendix A.4). Unlike the toy example, Theorem 12 therefore guarantees persistent selection without guaranteeing complementary coverage. A.2 Model and Compressor Classes We now state the general model of Section 5 with full notation. We index the key and value branches by P ∈K,V, which Section 5 writes as B. For every positive integerm, write [m] :=1,...,m. LetS ≥3, use 1≤ H ≤ S−1 heads, and useL≥2 chunks. The chunks are non-overlapping and each containsSpositions, so the compression stride and effective compression ratio are bothS. LetV ⊂ R d be a vocabulary ofNdistinct embedding vectors, whereN ≥ LS. A context X consists of the L ordered chunks X = (x ℓ,1 ,...,x ℓ,S ) L ℓ=1 ,(36) where theLSembeddings are sampled uniformly without replacement fromV. Independently,ℓ ∗ ∼ Unif[L] andr ∗ ∼ Unif[S−1]. The query vector isq=x ℓ ∗ ,r ∗ and the target embedding isy=x ℓ ∗ ,r ∗ +1 . LetXbe the finite set of all such ordered contexts with distinct embeddings. For the first global theorem, we assume that for some κ∈ R and 0≤ ε < 1, x ⊤ x ′ − κ− 1x = x ′ ≤ ε(x,x ′ ∈V).(37) This is the embedding geometry of Equation (4) in Section 5. For fixedε >0, it permits dimension logarithmic in N. The dynamics result uses its idealized high-dimensional limit ε = 0. Let E ∈ R N×d have row x ⊤ indexed by x∈V, and fix 0 < γ,β <∞. We consider two compressor classes. In the oracle class, after observingXand before (ℓ ∗ ,r ∗ ) is sampled, a compressor rule may compute arbitrary vectors p K h,ℓ (X),p V h,ℓ (X)∈ ∆ S ,∆ S := ( p∈ R S : p(r)≥ 0, S X r=1 p(r) = 1 ) (38) 20 for everyℓ ∈[L] andh ∈[H]. These key and value compression coefficients may depend arbitrarily onX. The class of all such pre-query compressor rules is G oracle = g :X −→ (∆ S × ∆ S ) LH , g(X) = p K h,ℓ (X),p V h,ℓ (X) ℓ,h .(39) Thus the information order isX 7→ g(X), followed by (ℓ ∗ ,r ∗ )∼ Unif([L]×[S−1]). Here and below,e r is the rth vertex of ∆ S . When a context and query are fixed, we suppress their dependence in the notation. Positionris window offsetr−1 in Section 4.2 and corresponds to one phase. Both coefficient vectors use physical phase indices. No phase is masked, and the useful adjacent router ∈[S−1] pairs key phaserwith value phase r + 1. The compressed KV entries are k h,ℓ = S X r=1 p K h,ℓ (r)x ℓ,r , v h,ℓ = S X r=1 p V h,ℓ (r)x ℓ,r .(40) For query q, the attention weights are α h,ℓ = expγ⟨q,k h,ℓ ⟩ P L j=1 expγ⟨q,k h,j ⟩ .(41) The attention output and logits are TF(X,q) = H X h=1 L X ℓ=1 α h,ℓ v h,ℓ , O(X,q) = βE TF(X,q).(42) For a fixed contextX, letpdenote the full collection of oracle coefficients and substitute it into this architecture. Let O(X,q) x denote the output coordinate indexed by x∈V. The fixed-context objective is L X (p) = 1 L(S− 1) L X ℓ ∗ =1 S−1 X r ∗ =1 " log X x∈V e O(X,q) x −O(X,q) y # .(43) Its population average is L(g) = 1 |X| X X∈X L X (g(X)).(44) The finite product G oracle is compact, and L is continuous on it. For the training-dynamics result, we restrict the oracle class to linear compressors, whose gate logits are linear in the embeddings. They use vectorsc K h,r ,c V h,r ∈ R d shared across contexts and chunks, with no scalar biases. Write c = (c P h,r ) P,h,r . For P ∈K,V, its compression coefficients are p P h,ℓ (r;c,X) = exp⟨c P h,r ,x ℓ,r ⟩ P S j=1 exp⟨c P h,j ,x ℓ,j ⟩ (45) and p P h,ℓ (c;X) := (p P h,ℓ (r;c,X)) S r=1 , g c (X) = p K h,ℓ (c;X),p V h,ℓ (c;X) ℓ,h .(46) The resulting linear compressor class is G linear = g c : c K h,r ,c V h,r ∈ R d for all h,r ⊂G oracle ,(47) and only the vectorsc K h,r ,c V h,r are trainable. The inclusion can be strict because an oracle rule may use the whole context in an arbitrary way, whereas g c uses shared phasewise linear gate logits. A.3 Proof of Theorem 1 We first ask what the loss itself favors when compression may depend on the context in any way. As in the toy example, we compare mixed coefficients with one-hot ones, now with keys and values paired. 21 Symmetric comparison loss. We compare the true loss with an auxiliary functional of the same oracle coefficients that isolates the symmetric part κ + 1x = x ′ of the Gram matrix. For 0≤ u≤ 1, define R(u) = e γu e γu + L− 1 .(48) For a fixed context, query (ℓ ∗ ,r ∗ ), and coefficient collection p, set A h ℓ = ( R(p K h,ℓ ∗ (r ∗ )),ℓ = ℓ ∗ , 1− R(p K h,ℓ ∗ (r ∗ ))/(L− 1), ℓ̸= ℓ ∗ . (49) These are the comparison attention weights. Using the same value compression coefficients as the true model, define the unscaled comparison output coordinate at x∈V by F x = H X h=1 L X ℓ=1 A h ℓ S X r=1 p V h,ℓ (r)1x ℓ,r = x.(50) Since bothA h andp V h,ℓ have unit mass, P x∈V F x =H. Vocabulary items absent from the context have F x = 0. Hence the comparison query loss islog P x∈V e βF x − βF y . LetL sym,X (p) be its average over the L(S− 1) queries, and define its population average by L sym (g) = 1 |X| X X∈X L sym,X (g(X)).(51) Whenε= 0, the comparison functional equals the true loss. Indeed,x ⊤ x ′ =κ+ 1x=x ′ . For queryq, the true attention scores areγ(κ+p K h,ℓ ∗ (r ∗ )) on chunkℓ ∗ andγκon every other chunk, so their softmax isA h . The output coordinate atx∈Visβ(κH+F x ), and the common termβκHcancels from cross-entropy loss. For ε > 0, the proof below controls the change from L sym to L along the same coefficient path. For later use, set R 1 = R(1) and Q 1 = (1− R 1 )/(L− 1). Theorem 8 (Formal version of Theorem 1). Under Equation (37), there are constantsc γ,β ,C γ,β >0 such that the following holds. Suppose N ≥ C γ,β LS exp C γ,β H L , ε≤ c γ,β L 2 HS .(52) Then every minimizer overG oracle uses hard adjacent selection: for everyX ∈X,ℓ∈[L], andh∈[H], there is r X,ℓ,h ∈ [S− 1] such that p K h,ℓ (X) = e r X,ℓ,h , p V h,ℓ (X) = e r X,ℓ,h +1 .(53) Ifε= 0, the minimizers are exactly those assignments for which, for everyXandℓ, the maph7→ r X,ℓ,h is injective. The first lemma rules out mixed coefficients, and the second compares the remaining one-hot route assignments under exact geometry. Lemma 9 (Projection to adjacent vertices). Under the assumptions of Theorem 8, for each contextXand coefficient collectionp, there is a one-hot adjacent collection with strictly smallerL X unlesspalready has this form. Proof. Fix a context and writeEfor its uniform-query average. For each (ℓ,h), chooseˆr ℓ,h ∈[S−1] maximizing p K h,ℓ (ˆr) +p V h,ℓ (ˆr+ 1), and replace the two coefficient vectors bye ˆr ℓ,h ande ˆr ℓ,h +1 . Denote the resulting one-hot 22 adjacent collection by b p . Suppressing (ℓ,h), setζ=minR 1 , min u∈[0,1] R ′ (u) >0. The mean-value theorem and maximality of ˆr give R 1 − S−1 X r=1 R(p K (r))p V (r + 1)≥ ζ 1− S−1 X r=1 p K (r)p V (r + 1) ! ≥ ζ 4 ∥p K −e ˆr ∥ 1 +∥p V −e ˆr+1 ∥ 1 . The last step usesuv ≤(u+v) 2 /4 and the fact that a simplex point’s distance to a vertex is twice its missing vertex mass. Let the query-normalized projection distance be ∆ = 1 L(S− 1) X ℓ,h ∥p K h,ℓ − b p K h,ℓ ∥ 1 +∥p V h,ℓ − b p V h,ℓ ∥ 1 .(54) Averaging this per-head gap over the uniform query gives E[ b F y − F y ]≥ ζ 4 ∆.(55) Inserting the output with projected attention but original values, usingE b A h ℓ = 1/L, and applying the Lipschitz bound for R give E∥F − b F∥ 1 ≤ maxS− 1, 2 max u∈[0,1] R ′ (u)∆.(56) Both outputs have massHand entries in [0,HR 1 ]. Along the segment between them, subtracting 1 from each exponential in the derivative of the log-normalizer bounds its change byβ(e βHR 1 − 1)E∥F − b F∥ 1 /(2N). Hence Equations (55) and (56) give L sym,X (p)−L sym,X ( b p)≥ β∆ ( ζ 4 − maxS− 1, 2 max u∈[0,1] R ′ (u)(e βHR 1 − 1) 2N ) .(57) Here ζ ≥ γ/[L(e γ + 1)] and R 1 ≤ e γ /L. The theorem’s vocabulary bound makes the braces at least ζ/8. It remains to transfer this gap to the true Gram matrix. The softmax JacobianJ(q) =diag(q)− q ⊤ satisfies ∥J(q)v∥ 1 ≤2∥v∥ ∞ and∥(J(q)−J(q ′ ))v∥ 1 ≤3∥q−q ′ ∥ 1 ∥v∥ ∞ . By Equation (37), each attention score changes by at mostγε, and its path derivative by at mostγε P ℓ ∥ ̇ p K h,ℓ ∥ 1 . Applying the Jacobian bounds to attention and then to output cross-entropy gives, for any differentiable coefficient path, d dt (L X −L sym,X )(p(t)) ≤ C γ,β (1 + βH)ε X ℓ,h,P ∥ ̇ p P h,ℓ (t)∥ 1 .(58) Indeed, after subtracting the commonβκHfrom the output logits, their true–comparison difference and its path derivative are bounded respectively byO γ,β (Hε) andO γ,β (ε P ℓ,h,P ∥ ̇ p P h,ℓ ∥ 1 ). The cross-entropy gradient is 2-bounded inℓ 1 and 2-Lipschitz fromℓ ∞ toℓ 1 . Integrating Equation (58) alongp(t) = (1−t) b p +tpchanges the comparison gap by at mostC γ,β L(S−1)(1 +βH)ε∆. The theorem’s bound onεmakes this less than βζ∆/16. With Equation (57), the true gap is positive whenever ∆>0. An oracle can make the replacement for one context without changing any other context. Lemma 10 (Distinct routes under exact geometry). Under the assumptions of Theorem 8 withε= 0, all injective one-hot adjacent assignments have the same fixed-context loss, and every assignment with a head collision has strictly larger loss. 23 Proof.Fix a context, writer ℓ,h ∈[S−1] for the route of headhin chunkℓ, and letk= P h 1r ℓ ∗ ,h =r ∗ for a uniform query. Then Ek = H S− 1 ≤ 1,Ek 2 ≤ H.(59) Every assignment has the same expected target logitEF y =HR 1 /(S−1), so only the output normalizer matters. Write Z inj for this normalizer under an injection. Then Z inj = Z 0 + kd inj , where Z 0 := N − LH + LHe β/L , d inj := e βR 1 + (L− 1)e βQ 1 − Le β/L > 0.(60) TheLnumbersR 1 ,Q 1 ,...,Q 1 have mean 1/Land are not all equal, so the last inequality follows from strict convexity. Thus all injections have the same expected target and normalizer terms, and E[logZ inj ]≤ logZ 0 + d inj H (S− 1)Z 0 .(61) For a collided assignment, contributions to a shared output coordinate add before exponentiation. Writing e βA = 1 + (e βA −1), the pair product from any collision adds at leastd col := (e βQ 1 −1) 2 >0 to the normalizer, so Z col ≥ Z 0 + kd inj + d col . Using log(1 + t)≥ t− t 2 /2 and Ek 2 ≤ H gives E[logZ col ]− E[logZ inj ]≥ d col Z 0 − d 2 inj H + 2d inj d col + d 2 col 2Z 2 0 .(62) Taylor’s theorem at 1/LandR 1 =e γ Q 1 gived inj ≤ C γ,β Q 2 1 , whereasd col ≥ β 2 Q 2 1 andd col ≤(e β −1) 2 . The numerator in the error term is therefore at mostC γ,β Hd col . SinceZ 0 ≥ Nand the vocabulary bound can be enlarged to ensureN > C γ,β H, the displayed gap is positive. The expected target term is the same for both assignments, so every collision strictly increases the fixed-context loss. Proof of Theorem 8.The oracle class is a product over contexts, so each fixed-context loss can be minimized separately. Lemma 9 makes every minimizer one-hot and adjacent. Whenε= 0, Lemma 10 shows that precisely the injective route assignments minimize the loss among them. The injection may depend on the context and chunk, so the oracle result does not assign a fixed route to each named head. A.4 Proof of Theorem 2 The oracle result says which compressors are optimal but not whether training finds them, or whether a trained head keeps the same selection for every input. As in the toy example, we answer this by starting gradient flow near zero, where all heads agree and no useful phase is preferred over another, and asking along which directions the flow leaves the symmetric path that starts there. Letμ:=N −1 P x∈V x. We first collect the assumptions for the dynamics result. Its effective parameter space is C eff := ( c : c P h,r ∈ span(V), S X r=1 μ ⊤ c P h,r = 0 for every P ∈K,V, h∈ [H] ) .(63) This removes embedding-orthogonal directions and a common mean-logit shift in each head and branch, which do not change the gates. Assumption 11 (Setting for the gradient-flow result). 1.Retrieval distribution. We use the model of Appendix A.2, withN ≥ LSvocabulary embeddings. Contexts containLnon-overlapping chunks ofSdistinct tokens each, sampled uniformly without replacement across the entire context. After compression, the query is uniform over theL(S−1) nonfinal positions and requests the next token in the same chunk. 24 2.Fixed model parameters. We fixL≥3,S ≥3, 1≤ H ≤ S−1, 0< γ < log(L−1), andβ >0 asN varies. 3.Exact geometry and positive mean. The embeddings satisfy Equation (37) withε= 0, and the mean norm ∥μ∥ > 0 is fixed as N varies. 4.Gate-only population training. Only the offset-specific linear gate vectors inG linear , shared across contexts and chunks, are trained; the embeddings and all other maps remain fixed as in Appendix A.2. We use Euclidean gradient flow on the population loss: ̇ c =−∇ c L(g c )/β.(64) The factor 1/β only rescales time. 5. Initialization. Forρ >0,c(0) is uniform with respect to Euclidean volume on the radius-ρball centered at zero in C eff . The required joint bounds on ρ and N are specified in Theorem 12. This initialization includes input-dependent gate directions. The proof below establishes the emergence of static preferences from these starts. Under exact geometry, averaging the Gram identity givesx ⊤ μ=∥μ∥ 2 =κ+1/N, and hence μ ⊤ (x−μ) = 0,(x−μ) ⊤ (x ′ −μ) = 1x = x ′ − 1 N (x,x ′ ∈V).(65) The exact Gram matrix has eigenvalues 1 on vocabulary contrasts andN∥μ∥ 2 >0 on the constant direction, so its rank isNand this idealization requiresd≥ N. The ambient gradient is tangent toC eff : each parameter gradient is a linear combination of embeddings, and the gate-logit gradients in each row sum to zero. Since ∥x∥ 2 =∥μ∥ 2 + 1−1/Nforx∈V, the vector field in Equation (64) is smooth and globally bounded for each N. Its trajectories therefore exist for all positive time. Below, p P h,ℓ (r;t,X) := p P h,ℓ (r;c(t),X). Theorem 12 (Formal version of Theorem 2). Under Assumption 11, for every 0< δ <1, there are 0< c δ ≤1 and C δ <∞ such that the following holds. If 0 < ρ≤ c δ , N ≥ C δ log e ρ ,(66) then, with probability at least 1− δ, there are r 1 ,...,r H ∈ [S− 1] such that max h∈[H] sup X∈X,ℓ∈[L] max 1−p K h,ℓ (r h ;t,X), 1−p V h,ℓ (r h + 1;t,X) −→ 0 as t→∞.(67) The routesr h need not be distinct. The constantsc δ ,C δ can be chosen uniformly for∥μ∥in any compact subset of (0,∞). Hence fixing κ > 0 suffices, since then ∥μ∥ 2 = κ + 1/N ∈ (κ,κ + 1]. The lemmas below track the dynamics from the symmetric zero-start path to aligned winners and then to persistent one-hot routing. Winners are aligned when each head’s key gate favors some phaser h ∈[S−1] and its value gate favors the next phase r h + 1. In the supporting lemmas,Cdenotes a positive constant depending only onL,S,H,γ,β, and the fixed value of ∥μ∥, and its value may change between displays. Let M be the mean-aligned subspace c P h,r = b P h,r ∥μ∥ 2 μ, S X r=1 b P h,r = 0, P ∈K,V.(68) Sinceμ ⊤ x=∥μ∥ 2 forx∈V,b P h,r is the input-independent gate logit of phaser. WriteP M for the orthogonal projection onto M and T :=M ⊥ ∩C eff . The linear class raises a question that the toy example cannot pose. There, each head’s compression is a single trainable logit with no input, so any preference is automatically the same for every input. With linear gates, the gate logits depend on the embeddings, and a preference could change from one context to another. On 25 M, however, the gate logitb P h,r of each phase is the same for every input, just asθ h is in the toy example, so we track the flow nearM. The flow from zero stays onMwith all heads identical, which plays the role of the toy example’s uniform point (Lemma 14). Differences between routes are unstable along this path, and the unstable directions raise a key phase together with the value phase after it. As in the toy example, selection thus comes from an instability of the symmetric state, but here the instability acts on each head separately, since different heads do not couple in the limit. From a small random start, each head acquires such a route with high probability (Lemma 15), which is then amplified and persists for every input (Lemmas 16 and 17). Lemma 13 (Vocabulary suppression). For fixed context and query, the first three derivatives of the output log-normalizer divided byβareO(N −1 ), uniformly over (∆ S ×∆ S ) LH . This holds for derivatives in the compression coefficients, sampled gate logits⟨c P h,r ,x ℓ,r ⟩, and compressor parameters, using Euclidean product norms. In particular, its contributions to coefficient gradients and the parameter linearization are O(N −1 ). Proof.Whenε= 0, the common vocabulary logit cancels and the remaining logits areβF x , withF x as in Appendix A.3 after substituting the softmax coefficients. ThusF x = 0 outside the context, 0≤ F x ≤ H, and the normalized log-normalizer is 1 β log ( N + L X ℓ=1 S X r=1 (e βF x ℓ,r − 1) ) .(69) The first three derivatives ofF x in sampled gate logits or compression coefficients are uniformly bounded because they use fixed-dimensional softmax maps. The nonconstant part of the expression inside the logarithm, and its first three derivatives, are bounded independently ofN, whereas that expression is at least N. Differentiation gives theO(N −1 ) bounds. The sampled-logit map is linear in the compressor parameters and has operator norm at most C sup x∈V ∥x∥≤ C, yielding the parameter bound. Define the rowwise route-contrast space W := ( U ∈ R H×(S−1) : S−1 X r=1 U h,r = 0 for every h ) .(70) LetF(c) =−∇ c L(g c )/β, letc 0 N (t) be its trajectory from zero, and letA N (t) be the derivative onMof the target part of F at c 0 N (t). Lemma 14 (Symmetric baseline and route-contrast growth). For sufficiently largeN, the trajectoryc 0 N (t) remains inM. All heads agree, and key phases 1,...,S−1 and value phases 2,...,Sare respectively identical. Call key phaseSand value phase 1 the dummy phases, since no useful route uses them and they serve no query. If ω K ,ω V denote the dummy-phase coefficients, then ω K (t),ω V (t)≍ (1 + t) −1 ,log 1− ω P (t) (S− 1)ω P (t) −→∞ (P ∈K,V).(71) Moreover,A N (t) converges to an operatorA ∞ that acts on each route contrast, in the order key then successor value, by ∥μ∥ 2 times the block R ′ (1/(S− 1)) (S− 1) 4 R ′ (1/(S− 1)) (S− 1) 3 R ′ (1/(S− 1)) (S− 1) 3 0 ,(72) and vanishes on the useful-versus-dummy directions, with ∥A N (t)− A ∞ ∥≤ C 1 + t .(73) The positive eigenspace of A ∞ is a copy of W, with same-sign key and successor-value coordinates. 26 Proof.Vocabulary-permutation, head, and useful-phase symmetries keepc 0 N (t) inMand give useful coefficients (1− ω K )/(S− 1) and (1− ω V )/(S− 1). For either row, ̇ω P =−∥μ∥ 2 S S− 1 d P (t)ω 2 P (1− ω P ) 2 , P ∈K,V,(74) whered P (t) is the useful-minus-dummy fitness gap forL/β. Here the fitness of a phase is the negative derivative ofL/βwith respect to its compression coefficient, and the fitness gap is the fitness of a useful phase minus that of the dummy phase in the same row. The target term givesd V ≥1/[L(S−1)]. Onceω V ≤1/S, it also givesd K ≥ R ′ (0)/[S(S−1)]. By Lemma 13, these lower bounds survive for largeN, and both gaps are also uniformly bounded above. Integrating the displayed equation from ω P (0) = 1/S proves Equation (71). Route symmetry separates useful contrasts from useful–dummy directions. At the limiting coefficients, softmax sends a useful contrastytoy/(S−1), and its second derivatives sum to zero. The target Hessian on key and successor-value contrasts is therefore D 2 ( 1 S− 1 S−1 X r=1 R(p K h (r))p V h (r + 1) ) [(y K ,y V )] 2 = R ′ (1/(S− 1))∥y K ∥ 2 (S− 1) 4 + 2R ′ (1/(S− 1))⟨y K ,y V ⟩ (S− 1) 3 . (75) This gives Equation (72). The useful–dummy directions vanish in the limit because the dummy coefficient vanishes, and different heads do not couple. The block has negative determinant, and its positive eigenvector has same-sign coordinates. Smoothness and ω P (t) = O((1 + t) −1 ) give the stated convergence rate. Lemma 15 (Aligned route gaps from random initialization). For every 0< δ <1, there areη δ ,C δ >0 and 0< c δ ≤1 such that, if 0< ρ≤ c δ andN ≥ C δ log(e/ρ), and ifc(0) is uniform on the radius-ρball inC eff , then, with probability at least 1− δ, there are a time τ ≤ C δ log e √ N ρ (76) and a point ̄ c∈M satisfying ∥c(τ )− ̄ c∥≤ C δ ρ,(77) such that every head has a route r h ∈ [S− 1] with b K h,r h − max j̸=r h b K h,j ≥ 2η δ , b V h,r h +1 − max j̸=r h +1 b V h,j ≥ 2η δ .(78) Proof. The proof first splits the flow linearized alongc 0 N (t) into growing and complementary parts, and then shows that deviations orthogonal toMstay small. It next bounds the random initialization, defines an exit timeτfrom the linearized flow, and compares the nonlinear flow with its linearization up toτ. At that time, the growing part carries aligned route gaps. The target is a sum over heads, and simultaneous permutation of useful key and successor-value indices separates useful contrasts from useful–dummy directions. Thus, in theb-coordinates,A N (t) is (B N (t)⊗I W )⊕ D N (t), whereB N (t) is two-dimensional,B ∞ is the block in Equation (72), including its factor∥μ∥ 2 , and ∥B N (t)− B ∞ ∥+∥D N (t)∥≤ C/(1 +t). Letλ + >0> λ − be the eigenvalues ofB ∞ and put ∆ λ =λ + − λ − . In an orthonormal eigenbasis, writeB N (t) =diag(λ + ,λ − ) +E N (t). ChooseT, uniformly in largeN, so that ∥E N (t)∥,∥D N (t)∥ ≤ η ∗ fort ≥ T, withη ∗ sufficiently small. Two invariant contrast lines arex − =r(t)x + and x + = z(t)x − , defined by ̇r =−∆ λ r + E 21 + (E 22 − E 11 )r− E 12 r 2 , r(T ) = 0, z(t) =− Z ∞ t e −∆ λ (u−t) [E 12 + (E 11 − E 22 )z− E 21 z 2 ](u) du. 27 Forζ= 4η ∗ /∆ λ <1/2, the first vector field points inward atr=±ζ, and the second equation is a contraction on∥z∥ ∞ ≤ ζ. Hence these lines are uniformly transverse. The scalar convolution in the first equation and E N (t)→0 giver(t)→0, uniformly in largeN. Their scalar rates areλ + +E 11 +E 12 randλ − +E 22 +E 21 z, respectively. Takingη ∗ small, the former is at leasta >0, the latter is negative, andη ∗ ≤ a/2. Use these same lines for every head and route contrast, placing the whole useful–dummy sector in the complement. This defines invariant projectionsP + (t) andP ⊥ (t) =I − P + (t) with uniformly bounded norms. Extend them to [0,T] by the propagator Φ(t,s) of ̇u=A N (t)u; boundedness ofA N on this fixed interval bounds both transport directions uniformly inN. Write Φ j (t,s) =P j (t)Φ(t,s)P j (s) forj ∈+,⊥. The preceding scalar rates and ∥D N (t)∥≤ η ∗ give, after adjusting C, for t≥ s≥ 0, ∥Φ + (t,s) −1 ∥≤ Ce −a(t−s) , ∥Φ + (t,s)∥≤ Ce C(t−s) , ∥Φ ⊥ (t,s)∥≤ Ce a(t−s)/2 . (79) The inverse is on the growing fiber; the zero limiting modes are included in the complementary bound. All constants are uniform in large N. In the original key/value coordinates, choose the positive unit eigenvectore + ofB ∞ and letQ N (t) be the contrast solution withQ N (T) =e + . The growing fiber isQ N (t)⊗WandQ N (t)/∥Q N (t)∥ → e + . Letα N take the coefficient ofQ N (0) in the splitting along the complementary contrast line, and define Λ N c by applyingα N ⊗ I W to the useful-contrast part ofP M c, expressed inb-coordinates. Fixed-interval transport bounds∥Q N (0)∥above and away from zero; transversality andα N Q N (0) = 1 do the same for∥α N ∥. The useful-contrast projection is an orthogonal surjection, and∥b∥=∥μ∥c∥onM, so all nonzero singular values of Λ N are uniformly bounded above and below. In particular, every growing solution isQ N (t)⊗ wfor a fixed w ∈W. To control deviations from the mean-aligned trajectory, for a parameter direction w define ∥w∥ 2 X = X P,h,ℓ,r (w P h,r ) ⊤ x ℓ,r 2 .(80) The ε = 0 geometry and sampling without replacement imply E X ∥w∥ 2 X = L N ∥w∥ 2 (w ∈T ), ∥w∥ X = √ L∥μ∥w∥ (w ∈M).(81) The fixed-context objective is smooth in the sampled gate logits. Its target derivatives are bounded, and its nor- malizer derivatives areO(N −1 ) by Lemma 13. Vocabulary equivariance givesF( ̄ c)∈MandP M DF( ̄ c)[u] = 0 for ̄ c∈M,u∈T. Applying Equation (81) to the first two derivatives and using this cancellation gives, for u∈T with ∥u∥≤ 1, ∥(I − P M )F ( ̄ c + u)∥≤ C N ∥u∥, ∥P M F ( ̄ c + u)−F ( ̄ c)∥≤ C N ∥u∥ 2 .(82) The effective parameter space has dimension 2H(SN −1). Uniform radius-ρball initialization is an isotropic Gaussian direction times an independent radial factor. Gaussian concentration, the uniform singular-value bounds for Λ N , and a finite union bound over the nondegenerate Gaussian head–route contrasts give a δ ,A δ ,q δ > 0 such that, with probability at least 1− δ, a δ ρ √ N ≤ g 0 :=∥Λ N c(0)∥≤ A δ ρ √ N , ∥P M c(0)∥≤ A δ ρ √ N ,min h gap((Λ N c(0)) h )≥ q δ g 0 .(83) Here gap is the largest minus second-largest row entry. OnM, use the Euclidean norm of theb-array, which is uniformly equivalent to the parameter norm because ∥b∥=∥μ∥c∥there. Letu lin solve ̇u=A N (t)ufromu lin (0) =P M c(0). The estimates below hold for all 28 exit radii in a fixed sufficiently small interval. Chooser δ in that interval below, and letτbe the first time ∥P + (τ )u lin (τ )∥ = r δ . From Equations (79) and (83), A −1 δ log r δ √ N ρ ≤ τ ≤ A δ log r δ √ N ρ .(84) The lemma’s lower bound on N, after increasing C δ , makes τ/N arbitrarily small. Define u M (t) := P M (c(t)−c 0 N (t)), u T (t) := (I − P M )c(t).(85) The first bound in Equation (82) and Gronwall give ∥u T (t)∥≤ A δ ρ(0≤ t≤ τ ).(86) Shrinkingc δ closes the assumed unit token tube. Taylor’s theorem, Lemma 13, and the second bound in Equation (82) give ̇u M = A N (t)u M + E(t), ∥E(t)∥≤ C ∥u M ∥ N +∥u M ∥ 2 + ∥u T ∥ 2 N .(87) Put v(t) = P + (t)u lin (t). By Equations (79) and (83) and scalarity on W, for 0≤ s≤ t≤ τ, ∥v(s)∥≤ Ce −a(t−s) ∥v(t)∥, ∥Φ + (t,s)∥v(s)∥≤ C∥v(t)∥, ∥P ⊥ (t)u lin (t)∥≤ A δ e −at/2 ∥v(t)∥. (88) In particular,∥v(t)∥≤ Cr δ e −a(τ−t) before exit. The final bound is the relative decay: the inverse growing propagator contributes e −at and the complementary propagator contributes e at/2 . Variation of constants for Equation (87) givesu M (t) =u lin (t) + R t 0 Φ(t,s)E(s) ds. Scalarity, bounded projections, and Equation (88) give the full-propagator bounds, for 0≤ s≤ t≤ τ, ∥Φ(t,s)∥v(s)∥≤ C∥v(t)∥, ∥Φ(t,s)∥≤ C ∥v(t)∥ g 0 e −as . Indeed, the complementary contribution to the first bound is at mostCe −a(t−s)/2 ∥v(t)∥, and∥v(s)∥ ≥ C −1 g 0 e as gives the second. On the bootstrap bound∥u M (s)∥≲ δ ∥u lin (s)∥+ρ 2 ∥v(s)∥/(Ng 0 ), the right-hand side isO δ (∥v(s)∥) whenρ 2 /(Ng 0 )≤1. Together with R t 0 ∥v(s)∥ds≤ C∥v(t)∥≤ Cr δ and Equation (86), these kernel bounds yield, uniformly for t≤ τ, Z t 0 ∥Φ(t,s)∥u M (s)∥ 2 ds≲ δ r δ ∥v(t)∥, 1 N Z t 0 ∥Φ(t,s)∥u M (s)∥ ds≲ δ t N ∥v(t)∥, 1 N Z t 0 ∥Φ(t,s)∥u T (s)∥ 2 ds≲ δ ρ 2 Ng 0 ∥v(t)∥. (89) For smallr δ ,τ/N, andρ 2 /(Ng 0 ), these estimates improve the bootstrap, so continuity closes it. Since g 0 ≥ a δ ρ/ √ N, the last term at t = τ is O δ (r δ ρ/ √ N ). Adding the complementary linear component gives ∥u M (τ )− v(τ )∥≲ δ r 2 δ + r δ τ N + r δ ρ √ N + r δ e −aτ/2 .(90) The growing fiber converges to the positive eigenvector of Equation (72), whose key and value coordinates have the same positive sign. Thus Equation (83) gives the leading termv(τ) the same winning route in the key row and the successor-value row, with every winner–loser gap among useful phases at least a fixedδ-dependent multiple ofr δ . Chooser δ small, thenC δ large andc δ small, so Equation (90) is smaller than these gaps. The lower exit-time bound and Equation (71) make the useful–dummy gaps positive as well. Hence ̄ c:=P M c(τ) has aligned adjacent winners with a common gap 2η δ > 0, while Equation (86) gives ∥c(τ )− ̄ c∥≤ A δ ρ. 29 Lemma 16 (Mean amplification). The space M is invariant. Fix 0 < η < M. For fixed head h and branch P, call the phase with the largest gate logitb P h,r the winner and the other phases losers. For all sufficiently largeN, suppose a trajectory inMhas, in every head, a key winnerr h , the value winnerr h + 1, and all corresponding winner–loser gate-logit gaps at leastη. These winners persist, every gap reachesMin time bounded independently of N, and every gap then diverges. Proof.Every vocabulary permutation is represented by an orthogonal map that fixesμand sendsx−μto the corresponding permuted residual. The loss is invariant and its gradient field is equivariant under this action. Its common fixed space inC eff isM, proving invariance. The induced flow in the coordinates of Equation (68) is the gate-logit flow multiplied by ∥μ∥ 2 . OnM, writep P h (r) for the coefficient shared by all chunks. A compressor row is the coefficient vector of one head and branch. Within a compressor row, write its coefficients asp i , its winner ass, and the fitness of phase iasf i =−∂(L/β)/∂p i . Differentiating the target term in Equation (75), the target contributions tof i for key phaser < Sand value phaser+ 1 areR ′ (p K h (r))p V h (r+ 1)/(S−1) andR(p K h (r))/(S−1), respectively. The dummy key and value fitnesses are zero. If every winner gap is at least η > 0, then p s − p j ≥ 1− e −η 1 + (S− 1)e −η (j ̸= s).(91) Becauseγ < log(L−1) makesR ′ increasing on [0,1], the target-fitness advantage of either winner is at least g η := 1 S− 1 min R ′ (0) 1− e −η 1 + (S− 1)e −η , 1 L > 0.(92) By Lemma 13, f s − f j ≥ g η /2 when N is large. Direct differentiation of the gate-logit flow gives d dt (b s − b j ) =∥μ∥ 2 " p j (f s − f j ) + (p s − p j ) f s − X i p i f i !# .(93) The second factor in the last term equals P i p i (f s − f i ), so both terms are nonnegative. Whileb s − b j ≤ M, the losing coefficient is at leaste −M /S, so the gap grows at rate at least∥μ∥ 2 g η e −M /(2S). This gives an N-independent time bound to reach M. Repeating at successively larger levels proves divergence. Lemma 17 (Persistence of aligned winners). For everyr 1 ,...,r H ∈[S−1] and all sufficiently largeN, there isM 0 <∞, independent ofNand the route choice, such that the terminal cone, defined as the region where, for every head and all distinct x,x ′ ∈V, ⟨c K h,r h ,x⟩−⟨c K h,j ,x ′ ⟩≥ M 0 (j ∈ [S]\r h ), ⟨c V h,r h +1 ,x⟩−⟨c V h,j ,x ′ ⟩≥ M 0 (j ∈ [S]\r h + 1)(94) is forward invariant, and every displayed margin tends to +∞. Proof. For each context and compressor row, define f P h,ℓ (i;X) :=− ∂(L X /β) ∂p P h,ℓ (i) , ψ P h,ℓ (i;X) := p P h,ℓ (i) f P h,ℓ (i;X)− X j p P h,ℓ (j)f P h,ℓ (j;X) .(95) Thusψ P h,ℓ (i;X) is the negative gradient with respect to the corresponding sampled gate logit. Fix a context and row, suppress these indices asp i ,f i ,ψ i , and writesfor the selected phase. At a one-hot adjacent corner, the target fitness advantage isR ′ (1)/L(S−1)for the selected key and at leastR(1)−1/L/L(S−1) for the selected value. Both are positive. Set d ∗ := 1 L(S− 1) min R ′ (1),R(1)− 1 L > 0.(96) 30 Uniform continuity and Lemma 13 imply that, throughout the terminal cone for sufficiently largeM 0 and for all large N, f s − f j ≥ d ∗ 2 , |f i − f j |≤ C.(97) ChooseM 0 large enough that these bounds hold and (S−1)e −M 0 C < d ∗ / 4. Then 1− p s ≤(S−1)e −M 0 p s , and ψ s = p s X i̸=s p i (f s − f i ) > 0, −ψ j ≥ d ∗ 4 p j p s > 0.(98) For N ≥∥μ∥ −2 , the Gram matrix is entrywise nonnegative: x ⊤ x ′ =∥μ∥ 2 − 1 N + 1x = x ′ ≥ 0(x,x ′ ∈V).(99) The parameter flow is ̇ c P h,r = 1 |X| X X∈X L X ℓ=1 ψ P h,ℓ (r;X)x ℓ,r .(100) Thus the derivative of a selected-versus-loser margin is a sum of termsψ s x ⊤ ℓ,s x− ψ j x ⊤ ℓ,j x ′ , all nonnegative. The terminal cone is therefore forward invariant. Suppose one margin, for embeddingsx,x ′ , stayed bounded above byB. Restrict the context average to contexts placingxandx ′ in the corresponding phases of one fixed chunk. This event has probability 1/N(N−1). On it,p s ≥1/S,p j ≥ p s e −B , andψ s ≥ d ∗ p s p j /2. Since all other contributions are nonnegative and ∥x∥ 2 ≥ 1, the derivative of this margin is at least d ∗ e −B 2S 2 N (N − 1) > 0,(101) contradicting boundedness. Every margin diverges. Proof of Theorem 12.Fixδand choose the constants so that the three dynamical lemmas apply. With probability at least 1− δ, Lemma 15 gives a timeτand ̄ c∈Mwith aligned adjacent gaps at least 2η δ , while ∥c(τ )− ̄ c∥≤ C δ ρ. Let ̄ c(t) be the mean flow from ̄ c(0) = ̄ c. Apply Lemma 16 withη=η δ andM=M 0 + 2. After enlargingC δ to meet its large-Nthreshold and that of Lemma 17, all mean gaps exceedM 0 + 2 within a timeTindependent ofN. Bounded softmax derivatives control the target field, while Lemma 13 controls the normalizer Hessian. Together withsup x∈V ∥x∥ 2 ≤∥μ∥ 2 + 1, this gives anN-independent Lipschitz constant for the parameter field. Gronwall gives ∥c(τ + t)− ̄ c(t)∥≤ C δ e Ct ρ,0≤ t≤ T.(102) Shrink c δ further so that this perturbation changes every gate-logit margin by less than one. Then c(τ + T ) lies in the terminal cone, and Lemma 17 makes every relevant margin diverge. For fixedNthere are finitely many such margins, and softmax gives 1−p P h,ℓ (r;t,X)≤ (S− 1) exp − min j̸=r ⟨c P h,r (t),x ℓ,r ⟩−⟨c P h,j (t),x ℓ,j ⟩ (103) for the selected phaser. The convergence is therefore uniform over contexts and chunks, proving the theorem. B Additional Details on the Code-Completion Examples Here we give the details behind the code-completion example of Section 2.1: the exact token accounting, the same example on post-trained models and on DeepSeek-V3.1-Base, which has no chunked compression, and two further code-completion examples. As in Section 2, a token at zero-based positionthas phaset mod S for compression stride S. 31 B.1 FP8 Example for DeepSeek-V4-Flash-Base In Section 2.1, we show the code-completion example with a single docstring filler. Here we give the exact inputs needed to reproduce it, together with three further filler families that test whether the reversals depend on the filler’s format or wording. Unless noted otherwise, the results are measured on DeepSeek-V4-Flash-Base using the official TileLang inference implementation with model parallelism four and FP8 expert weights. Source example and reference continuation. The example comes from the official DeepSeek-V4 inference code, inference/kernel.py 1 , which implements TileLang kernels and PyTorch wrappers for quantized computation. The scored prefix contains the first 88 source lines, ending immediately afterT.Cast(compute_dtype, T.Cast(FP. The original source continues with8. The preceding code supports this reference: the function is documented as block-wise FP8 quantization, defines FP8 range and scale quantities, and setscompute_dtypetoFP32for internal computation. At the scored location, an inner cast toFP8followed by a cast back tocompute_dtypeis consistent with this quantization operation. These cues motivate reference-token matching, but they do not establish that no alternative token admits a syntactically valid continuation. Exact input construction and scoring. Using zero-based token indices, the reference token8in the unmodified source occurs at index 983. A filler ofLtokenizer tokens moves this token tot= 983 +L. ForL= 24,...,39, the scored token positions are 1007 through 1022. Each input is one contiguous raw prefix containing the filler followed by the 983 source-code tokens preceding8. No instruction, beginning-of-sequence token, or chat template is added, and the target token and subsequent source code are omitted. We read the logits at the final input position t− 1, immediately after FP. The prefix ends withFP(token ID 20199). The reference continuation8and competing continuation32have token IDs 26 and 2111, respectively. Thus,FP8tokenizes as [20199,26], whereasFP32tokenizes as [20199,2111]. We report their next-token probabilities P (8) and P (32), together with their difference ∆ = P (8)− P (32). These probabilities are selected from the full next-token distribution, soP(8) +P(32) need not equal one. The reported input-shift coordinate is filler lengthL, distinct from the scored-token positiont. The filler shifts both the source-code prefix and the scoring position, so this test does not isolate the position of an individual source token. Filler constructions. Family A (the filler shown in Section 2.1) prepends a triple-quoted Python docstring. At a filler length of 24 tokens, its exact text is """This module contains optimized tensor kernels used by the model inference runtime. = = = = = = = = = """ The second line initially contains nine space-separated=characters. Each successive variant appends one additional space-separated=, ending with 24 such characters. The total filler lengths are therefore L = 24,..., 39, giving 16 prompts. The source-code tokens following the filler remain unchanged. The other families are B, C, and D. As summarized in Table 1, families A, B, and C keep their natural-language sentence fixed throughout the sweep and add one punctuation token at each successive length. These families differ in whether the filler is a Python comment or docstring, in its description of the source file, and in the repeated separator. Family D instead realizes the 16 token counts using semantically similar but lexically distinct single-line comments, without a repeated-symbol padding line. Thus, A/B/C provide content-matched position shifts within each family, whereas D additionally tests robustness to natural-language rephrasing. Results across filler families. In Table 2, we report the probability difference ∆ for all 16 lengths and all four filler families, next to the post-trained DeepSeek-V4-Flash-0731 results discussed in Appendix B.2. Positive values favor the reference continuation8, and negative values favor32. No ties occur in these measurements. 1 https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/0c18385a15b3b34253d6cf7132f3f4a0442d4ea9/ inference/kernel.py 32 Table 1 Filler-family constructions. Quoted sentences are reproduced verbatim. Repeated symbols are separated by spaces in the actual inputs. Family Fixed text or 24-token endpointLength variation ATriple-quoted docstring: “This module con- tains optimized tensor kernels used by the model inference runtime.” The next docstring line contains 9–24=characters before the closing triple quotes. BComment: “This module contains optimized tensor kernels used by the model inference runtime.” A second comment line contains 9–24 = characters. CComment: “Internal model inference kernel implementations.” A second comment line contains 16–31*characters, with one added at each successive length. DAt 24 tokens: “This file defines TileLang ker- nels for block-wise FP8 quantization and the related low-precision operations for model inference.” One natural-language comment is used at each length, with no padding line. Family A exhibits an exact four-token pattern: ∆>0 at all lengths withL mod4∈ 2,3, and ∆<0 at the remaining lengths. The full-vocabulary top-1 prediction is8when ∆>0 and32when ∆<0. Since t= 983 +L, filler residuesL mod4∈ 2,3correspond to scored-token residuest mod4∈ 1,2. Across Family A’s 16 lengths, the mean combined probabilityP(8) +P(32) is 0.982, while ∆ ranges from−0.786 to +0.948. At lengths satisfyingL mod4∈2,3, all 32 measurements across the four families favor8. At the remaining lengths, 28 of 32 favor32, and four favor8, with no ties. These four exceptions are Family B atL= 29, Family C atL= 36, and Family D atL= 24 andL= 36, with probability differences +0.119, +0.527, +0.060, and +0.249, respectively. Across the complete sweep, ∆ ranges from−0.786 to +0.948 for A, from−0.791 to +0.953 for B, from −0.533 to +0.895 for C, and from −0.764 to +0.790 for D. B.2 FP8 Example for Other DeepSeek Models To test whether preference reversals persist under chat prompting, we evaluate the post-trained DeepSeek- V4-Flash-0731 and DeepSeek-V4.1-Flash models on the FP8 example in Appendix B.1. We retain the same source-code prefix and reference continuation, and give both models identical chat-formatted causal prefixes. 2 To check whether the reversals depend on chunked compression, we also evaluate DeepSeek-V3.1-Base [10], which has none. Chat-formatted inputs. We disable thinking and place the following instruction at the start of the user message: Continue the function without changing its block-wise quantization semantics; avoid a redundant nested cast The instruction is followed by the filler and then the first 974 tokens of the source prefix, with blank lines separating these components. The remaining nine source tokens,compute_dtype, T.Cast(FP, form the assistant prefill. This preserves the completion point in Appendix B.1 while presenting the task as a user request: [BOS][USER]instruction filler code head [ASSISTANT]</think>compute_dtype, T.Cast(FP The two displayed lines are concatenated. a newline, and the bracketed role labels stand for the template’s special tokens. We score the next token after the supplied assistant prefill. For filler lengthL, the scored position is t = 1004 + L, including the chat-template tokens. 2 DeepSeek-V4-Flash-0731 uses the official TileLang implementation with model parallelism four in mixed precision, with FP4 (MXFP4) routed-expert weights and FP8 for most other weights. DeepSeek-V4.1-Flash uses vLLM with tensor parallelism eight and expert parallelism. The comparison therefore includes differences in inference backend. 33 Table 2 DeepSeek-V4-Flash-Base (left) and DeepSeek-V4-Flash-0731 (right) results on the FP8 example. Each entry is ∆ =P(8)− P(32), computed from the reported probabilities and rounded to three decimal places. Green marks ∆>0, where the reference8is ranked above32, and red marks ∆<0. “Tokens” is the filler length, and thin rules separate groups of four lengths, matching the stride of four. DeepSeek-V4-Flash-Base receives the raw prefix with fillers of 24–39 tokens, and DeepSeek-V4-Flash-0731 receives the chat-formatted prompt of Appendix B.2 with fillers of 23–38 tokens. DeepSeek-V4-Flash-Base (raw prefix)DeepSeek-V4-Flash-0731 (chat prompt) TokensABCDTokensABCD 24-0.786-0.791-0.007+0.06023-0.910-0.775-0.364-0.789 25-0.636-0.383-0.415-0.48424+0.615+0.451+0.644-0.829 26 +0.586+0.751+0.845+0.61425-0.831-0.854-0.822-0.989 27+0.914+0.862+0.809+0.68126-0.936-0.939-0.824-0.875 28-0.248-0.410-0.354-0.33727-0.679-0.668-0.266-0.151 29 -0.043+0.119-0.533-0.76428+0.860+0.073+0.241-0.420 30+0.826+0.808+0.724+0.69629-0.901-0.238-0.713-0.813 31 +0.927+0.933+0.822+0.73730-0.960-0.954-0.803-0.608 32-0.219-0.660-0.119-0.12931-0.882-0.159-0.160-0.715 33-0.644-0.276-0.118-0.62432+0.282-0.384+0.822-0.566 34+0.746+0.806+0.543+0.52433-0.917-0.828-0.874-0.946 35+0.904+0.829+0.895+0.65334-0.968-0.940-0.786-0.968 36-0.468-0.503+0.527+0.24935-0.869-0.438-0.826-0.787 37 -0.553-0.198-0.510-0.53236+0.706+0.597+0.715-0.895 38+0.887+0.852+0.768+0.59137-0.945-0.824-0.834-0.892 39+0.948+0.953+0.866+0.79038-0.903-0.790-0.770-0.880 Table 3 Filler constructions for the post-trained models. The fixed sentences and formats follow Table 1, and each family spans filler lengths L = 23,..., 38. Family FormatLength variation ATriple-quoted docstring8–23 space-separated = characters on the second line. BTwo-line Python comment8–23 space-separated = characters on the second line. CShorter two-line comment15–30 space-separated * characters on the second line. DSingle-line comment16 natural-language rephrasings, with no repeated-symbol line. Filler families. We use families A, B, C, and D from Appendix B.1, now withL= 23,...,38 tokens, giving 16 variants per family (Table 3). The instruction and source code remain fixed throughout. Results. We report ∆ =P(8)− P(32) in Table 2 (right) and Table 4 (left), in the same format as the base-model results. Across these runs, the full-vocabulary top-1 prediction is always either8or32, so the sign of ∆ also determines whether the reference token is selected. DeepSeek-V4-Flash-0731 exhibits a four-token cycle for A and C: it selects8exactly whenL mod4 = 0, giving 4/16 correct predictions (25%) in each family. B follows the same pattern except atL= 32, where it selects 32, reducing accuracy to 3/16 (18.75%). With D, it selects 32 at every length. DeepSeek-V4.1-Flash instead alternates between the two continuations for A, B, and C, selecting8at odd filler lengths and32at even lengths. Each family therefore gives 8/16 correct predictions (50%). For D, the model selects8at all seven lengths from 23 to 29, then follows the same odd–even alternation, yielding 11/16 (68.75%). These periods describe the top-1 decisions over the measured range, and the probability differences also vary within each residue group. 34 Table 4 DeepSeek-V4.1-Flash (left) and DeepSeek-V3.1-Base (right) results on the FP8 example. Entries, shading, and “Tokens” follow Table 2. DeepSeek-V4.1-Flash receives the chat-formatted prompt with fillers of 23–38 tokens, and thin rules separate pairs of lengths, matching its stride of two. DeepSeek-V3.1-Base receives the same raw-prefix inputs as DeepSeek-V4-Flash-Base, grouped in fours as in Table 2. DeepSeek-V4.1-Flash (chat prompt)DeepSeek-V3.1-Base (raw prefix) TokensABCDTokensABCD 23+0.124+0.245+0.358+0.55524+0.316+0.362+0.355+0.242 24-0.555-0.555-0.462+0.12425+0.477+0.315+0.359-0.003 25+0.124+0.358+0.462+0.46226+0.600+0.316+0.348+0.203 26-0.555-0.635-0.124+0.12427+0.274+0.486-0.026+0.140 27+0.358+0.555+0.462+0.24528+0.430+0.363+0.303+0.011 28-0.555-0.124-0.635+0.12429+0.303+0.259+0.273+0.463 29+0.358+0.462+0.124+0.12430+0.240+0.062+0.081+0.140 30 -0.462-0.555-0.245-0.24531+0.392+0.316+0.464+0.236 31+0.358+0.358+0.358+0.55532+0.346+0.539+0.026-0.054 32-0.635-0.704-0.462-0.24533-0.065+0.284+0.139+0.478 33+0.462+0.245+0.462+0.12434+0.139+0.146+0.109+0.325 34-0.555-0.462-0.245-0.46235+0.398+0.453+0.062+0.017 35+0.555+0.555+0.358+0.73436+0.166+0.081+0.276+0.370 36-0.635-0.555-0.245-0.35837+0.388+0.498+0.027+0.347 37+0.462+0.635+0.462+0.35838+0.397+0.324+0.396+0.143 38-0.635-0.635-0.245-0.35839+0.409+0.197+0.404+0.342 The reversals can be large. Increasing the A filler from 23 to 24 tokens changes ∆ from−0.910 to +0.615 in DeepSeek-V4-Flash-0731, while the same change moves DeepSeek-V4.1-Flash from +0.124 to −0.555. Thus, the sensitivity seen in the base-model example persists with post-trained models and chat-formatted inputs, although its period and dependence on filler content differ across models. The reported accuracies summarize shifted variants of this example under the selected prompt, rather than performance on a code-completion benchmark. A model without chunked compression. DeepSeek-V3.1-Base compresses each token’s keys and values into a low-rank latent vector, but it does not pool tokens into compressed entries, so its attention has no stride. 3 The two models share a tokenizer, so we give DeepSeek-V3.1-Base the raw-prefix inputs of Appendix B.1, token-for-token identical to those of DeepSeek-V4-Flash-Base. In Table 4 (right), we see no periodic reversal. DeepSeek-V3.1-Base ranks32above8at only 4 of the 64 inputs, each with|∆| <0.07, compared with 28 for DeepSeek-V4-Flash-Base. The difference lies in the dependence on position rather than in overall confidence. In family A, the mean ofP(8) is 0.609 for DeepSeek-V3.1-Base and 0.589 for DeepSeek-V4-Flash-Base, but only DeepSeek-V4-Flash-Base changes its preference systematically with the scored position. Across the four families, it errs at 15 of 16 inputs witht mod4 = 0 and 13 of 16 witht mod4 = 3, whereas the four errors of DeepSeek-V3.1-Base fall on three different residues. This contrast is consistent with chunked compression being the source of the sensitivity, although the two models also differ in architecture, training, and inference backend. 3 DeepSeek-V3.1-Base (revisiond3d4eafd) uses the official DeepSeek-V3 inference implementation with model parallelism eight and FP8 weights, because its release does not include inference code of its own. The comparison with DeepSeek-V4-Flash-Base therefore also differs in inference backend. With the implementation’s BF16 output projection, 12 of the 64 comparisons between 8and32were exact ties, so we compute only the final vocabulary projection in FP32 and leave the rest of the model unchanged. The BF16 outputs replicated exactly on a second machine, and the FP32 results come from a single run. 35 Table 5 Omarchy lid-close completion across 16 filler variants. The reference token is0, and every mismatch predicts 1. Green marks correct predictions and red marks mismatches. Spaces group consecutive predictions for readability. ModelTop-1 sequenceMismatches/16 P (0) range DeepSeek-V4-Flash-Base (TileLang)00000001000100013 0.175–0.951 DeepSeek-V4-Flash-073111001100110011008 0.270–0.725 DeepSeek-V4.1-Flash 00000000101010104 0.378–0.867 B.3 Additional Code-Completion Examples To show that the reversals are not peculiar to the FP8 example, we give two further examples in which the same kind of filler sweep produces top-1 reversals, periodic for most of the evaluated models: a shell test from the Omarchy GitHub repository and a SQL task adapted from thegretelai/synthetic_text_to_sql dataset. We evaluate DeepSeek-V4-Flash-Base, DeepSeek-V4-Flash-0731, and DeepSeek-V4.1-Flash. B.3.1 Omarchy Lid-Close Test Omarchy is a Linux distribution whose system utilities handle desktop configuration and hardware events. This example comes fromtest/shell.d/lid-close-test.sh, which tests the response to closing a laptop lid. 4 The test distinguishes an undocked laptop, which should lock before suspending, from a laptop connected to an external display, which should remain usable in clamshell mode. The excerpt below retains the scenario setup and the cues preceding the completion point. Ellipses mark omitted code or comment text, and [CURSOR] denotes the next-token prediction position. setup_scenario() scenario_dir="$tmpdir/$1" # ... local closed="$2" docked="$3" cat >"$mock_bin/omarchy-hw-laptop-closed" <<SH #!/bin/bash exit $closed SH # ... # An undocked lid close is about to suspend ... setup_scenario undocked 0 1 # ... # A docked lid close is clamshell mode: # logind leaves the machine awake and the session # stays in use on the external display, # so locking it would be wrong. setup_scenario docked [CURSOR] The first argument names the scenario. The second controls the exit status of the mocked lid-closed detector, and the third controls the external-display detector. Shell status0indicates that the detected condition holds. Thus,setup_scenario undocked 0 1represents a closed lid without an external display. The docked scenario also explicitly describes a closed lid, so the next token should again be0. Choosing1would instead simulate a lid that is not closed, contradicting the scenario being tested. The comment about avoiding screen locking specifies the expected response to this state, whereas the argument being completed specifies the lid state itself. We vary a natural-language filler over 16 consecutive lengths, from 24 to 39 tokens under the DeepSeek-V4 tokenizer, while keeping the source prefix fixed. We report the next-token predictions in increasing filler-length order in Table 5. 4 Full source file: test/shell.d/lid-close-test.sh, commit f99d33a8. 36 DeepSeek-V4-Flash-0731 exhibits an exact four-position cycle, alternating between two incorrect and two correct predictions. DeepSeek-V4-Flash-Base makes errors at inputs 8, 12, and 16, while DeepSeek-V4.1-Flash alternates between incorrect and correct predictions over the final eight inputs. B.3.2 SQL Station-Distance Query This example is adapted from test record 2113 ofgretelai/synthetic_text_to_sql, a dataset pairing natural-language questions with database contexts and reference SQL queries. 5 The original task asks for the distance between stations 5 and 12. Its context supplies a table containing the two stations’ coordinates, and its reference query computes their spherical distance. We retain the original schema, seed rows, and reference query, and adapt the example into a constrained SQL replay task in which a recorded output determines which operator is correct. We construct a contrasting query by replacing one+in the distance formula with-, and include this complete query together with its recorded output. The reference query is then presented asq_replay, with its recorded output marked authoritative, and truncated immediately before the same arithmetic operator. The task requires choosing+ or - to reproduce that output without changing the remaining SQL. The excerpt below shows the seed data, contrasting query, and completion point. Line breaks are adjusted for readability, ellipses mark omitted text, and [CURSOR] denotes the prediction position. -- Source request: What is the distance between -- station 5 and station 12? -- At the next arithmetic slot, write exactly + or -; -- only one reproduces q_replay. CREATE TABLE stations (...); INSERT INTO stations (station_id, name, latitude, longitude) VALUES (5, ’Station 5’, 40.712776, -74.005974), (12, ’Station 12’, 40.718261, -74.004790); -- q_control captured row multiset (different label): -- [[5626.688008276917]] -- q_control: SELECT 3959 * acos( ... - sin(radians(stations.latitude)) * sin(radians(( SELECT latitude FROM stations WHERE station_id = 12)))) as distance FROM stations WHERE station_id = 5; -- q_replay captured row multiset (authoritative): -- [[0.38403957581396597]] -- q_replay: SELECT 3959 * acos( cos(radians(stations.latitude)) * cos(radians(( SELECT latitude FROM stations WHERE station_id = 12))) * cos(radians(stations.longitude) - radians(( SELECT longitude FROM stations WHERE station_id = 12))) [CURSOR] The reference continuation is+. It adds the product of the latitude sine terms to the preceding cosine product, yielding a distance of approximately 0.38404 miles. Substituting-instead yields approximately 5626.68801 miles, matching the contrasting query’s output rather than the authoritative replay output. Thus, both operators admit executable SQL, but only+satisfies the stated replay constraint. The complete contrasting query supplies a competing local pattern, while the output labels explicitly distinguish its result from the required one. 5 Full source file: synthetic_text_to_sql_test.snappy.parquet, row 2112 (record 2113), revision 740ab23. 37 Table 6 Station-distance completion across 16 filler variants, ordered by increasing filler length. The reference token is+, and every mismatch predicts-. Green marks correct predictions and red marks mismatches. Spaces group consecutive predictions for readability. ModelTop-1 sequenceMismatches/16 P (+) range DeepSeek-V4-Flash-Base (TileLang)++++-++++++++-++2 0.329–0.569 DeepSeek-V4-Flash-0731 ---+--+---+---++11 0.123–0.638 DeepSeek-V4.1-Flash+-+-+-+-+-+-+-+-8 0.119–0.724 In Table 6, we report results for fillers of 24–39 tokens written as SQL comments. DeepSeek-V4.1-Flash alternates exactly between+and-across all 16 inputs. DeepSeek-V4-Flash-Base and DeepSeek-V4-Flash-0731 also switch between the two operators, with 2 and 11 mismatches, respectively. C Additional Details on the Needle-in-a-Haystack Evaluations Here we describe the needle-in-a-haystack evaluations of Section 2.2 and Section 3. We first give the construction and per-residue results for the DeepSeek models, and then the retrieval task and the two padding protocols that we use for our pretrained models. C.1 NIAH Evaluation for the DeepSeek-V4 Family Here we give the setup and per-residue results behind Section 2.2. As in Section 2, a token at zero-based positionthas phaset mod S, and the modulo-eight residues that we use to report the results are distinct from this phase. The construction shifts the target’s phase across residue groups while fixing the prompt length, the query position, and the mean and variance of the target position, so that the groups differ in phase but not in these position statistics. Position coordinate. DeepSeek-V4’s CSA compression kernel uses eight-token windows at stride four,W= 8,S= 4 [11]. We sample eight source-key position residuesr=t K mod8, wheret K is the target key’s zero-based token index, and call the prompts sharing a residue a residue group. For the DeepSeek-V4 models, the eight residue groups cover two cycles of the native CSA phaset K mod4. DeepSeek-V4.1-Flash compresses with stride two [12], so the eight residue groups cover four phase cycles. The modulus eight defines the residue groups and does not denote a compression stride of eight. Data generation. The construction below describes the 2,048-prompt set shared by the four DeepSeek-V4 variants. The DeepSeek-V4.1-Flash evaluation draws ten times as many prompts from the same distribution, using ten times as many bindings, where a binding is one fixed set of 16,000 key–value pairs. This gives 2,560 prompts per residue group, or 20,480 in total (Table 7). Each prompt contains a sequence of key–value records followed by a query asking for the value associated with one target key. Each prompt contains 128,000 tokens. An intro with the instruction and three format examples is followed by a natural prefill ofℓtokens, which sets the target’s residue, and by a padding block of about 63,000 filler tokens whose length is a multiple of eight. Next come 16,000 key–value pairs, with four tokenizer tokens per pair in the formkey:value。, and then a natural postfill of 48− ℓtokens, tail padding, and the query. The target is the 249th pair (zero-based index 248), so its key starts at token 63,976 +ℓ, which is 64,000 on average. The evaluation set contains 256 prompts in each of the eight residue groups, for 2,048 prompts in total. The key–value candidates are generated deterministically using the DeepSeek-V4 tokenizer. We select candidate strings with tokenizer IDs of at least 10,000 as a heuristic for less common tokens. Candidate strings are also alphabetic, contain at most 16 Unicode characters, and occupy exactly one tokenizer token. We additionally require tokenization stability at record and query boundaries. We construct 256 deterministic bindings, each containing 16,000 key–value pairs, and use each binding once in every residue group. The same evaluation prompts and prompt IDs are reused across the four DeepSeek-V4 variants, giving comparisons on identical prompts within that family. 38 Natural-language filler is drawn from the training split of theagentlans/high-quality-english-sentences dataset at revision5dfcb23. Sentences are shuffled and grouped by exact tokenizer length. The evaluation set uses 2,048 distinct sentences to form 1,024 sentence pairs. Each pair appears twice with reversed prefill/postfill roles. This balances the residue schedule and the target’s phase. The filler text is not identical across all eight residue groups. For residuer, the natural prefill lengthℓsatisfies 8≤ ℓ≤40 andℓ≡ r(mod8). Every residue group contains 256 prompts with E[ℓ] = 24,E[ℓ 2 ] = 604. The natural postfill has length 48− ℓ, so the two natural filler segments always contribute exactly 48 tokens. The finite histograms are H 0 =(8, 1), (16, 52), (24, 150), (32, 52), (40, 1), H 1 =(9, 2), (17, 68), (25, 146), (33, 40), H 2 =(10, 4), (18, 84), (26, 140), (34, 28), H 3 =(11, 7), (19, 101), (27, 129), (35, 19), H 4 =(12, 12), (20, 116), (28, 116), (36, 12), with H 8−r , for r ∈1, 2, 3, obtained by replacing each length ℓ in H r with 48− ℓ. Each prompt is assembled as [intro; natural prefill; body prefix; pair body; natural postfill; tail; query separator; query]. The body prefix is constructed so that the target-pair start is s = 64,000 + ℓ− 24, which gives a mean start position of 64,000 and variance 604−24 2 = 28 in every residue group while preserving the desired modulo-eight residue. The tail is then filled to reach exactly 128,000 prompt tokens. The query’s token position is fixed across residue groups. The terminal query is: Now answer the question using the key-value pairs above. What is the value for key <KEY>? Answer with only the value. The value for key <KEY> is Generation is followed by independent retokenization. We verify the prompt length, all component boundaries, pair-record tokenization, target span, source-key residue, and gold answer before saving the evaluation set. Scope of the phase comparison. Changing the outer filler shifts the target and distractor records together, and each four-token record spans a whole number of compression strides for both DeepSeek-V4 and DeepSeek-V4.1. These comparisons hold the query position and the first two moments of target position fixed, but do not vary the target’s phase independently of the distractors. The In-sequence Padding evaluation in Section 3.1 and Appendix C.3 tests variable gaps between records in our pretrained models. Inference protocol and metrics. For each prompt in the set shared by the four DeepSeek-V4 models, we set temperature = 0, max_tokens = 8. The task uses direct-answer prompting without chain-of-thought generation. Let g i be the gold value and y i the raw completion. The primary metric is answer-prefix accuracy: Acc prefix = 1 N N X i=1 1[strip(y i ) begins with g i ], wherestripremoves leading and trailing whitespace. This metric tolerates punctuation or parser-added text after an otherwise correct answer. It is computed from the generated answer and is distinct from full-vocabulary next-token accuracy. In Figure 2, the shaded bands are two-sided 95% Wilson intervals (z = 1.96) for each residue group. 39 Table 7 Answer-prefix accuracy (%) by source-key residuer=t K mod8 for the five DeepSeek models shown in Figure 2.nis the number of prompts in each residue group, and ∆ res =max r Acc r − min r Acc r is reported in percentage points. Accuracies are rounded to two decimal places from exact integer correct counts. Modeln r = 0 r = 1 r = 2 r = 3 r = 4 r = 5 r = 6 r = 7 ∆ res DeepSeek-V4-Flash-Base 256 43.75 69.53 66.02 29.69 44.92 69.92 64.45 32.81 40.23 DeepSeek-V4-Flash-0731 256 80.47 88.67 98.05 96.48 81.25 90.23 99.61 96.09 19.14 DeepSeek-V4-Pro-Base 256 60.94 88.28 78.91 56.25 62.50 91.02 80.47 58.20 34.77 DeepSeek-V4-Pro-0813256 78.13 89.84 91.41 77.34 77.34 89.45 92.19 79.69 14.84 DeepSeek-V4.1-Flash2560 95.00 89.22 95.12 89.69 94.69 90.31 95.31 89.73 6.09 Results across model variants. In Table 7, we report the per-residue results. We evaluate DeepSeek-V4- Flash-Base, DeepSeek-V4-Flash-0731, DeepSeek-V4-Pro-Base, and DeepSeek-V4-Pro-0813 on the same 2,048 prompts. Averaged over the eight residue groups, their answer-prefix accuracies are 52.6%, 91.4%, 72.1%, and 84.4%, respectively. Their best–worst residue gaps are 40.23, 19.14, 34.77, and 14.84 percentage points, respectively. Because the eight residue groups span two cycles of DeepSeek-V4’s stride of four, a four-token recurrence should make residuesrandr+ 4 nearly equal. For each model, we therefore average the absolute accuracy difference between residuesrandr+ 4 overr ∈0,1,2,3. This difference ranges from 1.07 to 1.95 percentage points across the four models, consistent with an approximate four-token recurrence in the accuracy pattern across residues. These measurements describe the positional pattern but do not by themselves isolate its mechanistic source. DeepSeek-V4.1-Flash, evaluated on 2,560 prompts per residue group, has a mean accuracy of 92.38% and a best–worst residue gap of 6.09 percentage points. C.2 NIAH Evaluation for Qwen3-based Models Here we specify the retrieval task that we use to evaluate our pretrained models in Section 3 and the two padding protocols, Prefix Padding and In-sequence Padding, that vary the phase of the target key. Within each protocol, the query position is fixed across phases, and the target’s expected depth and expected distance to the query are the same for every phase. We construct token-exact animal–name retrieval tasks. Each logical example independently permutes a curated pool of 64 animal keys and a disjoint pool of 64 human names, then pairs them to obtain a random one-to-one mapping. Each key and value is one token under the Qwen3 tokenizer. The target is the pair at zero-based index j ∗ = 24 in the 64-pair body. Both evaluations use a task instruction, a mapping header and all 64 pairs, and a recall prefix followed by eight demonstrations and the target animal key. Prefix Padding places natural-language filler before and after the mapping body. In-sequence Padding instead uses fixed outer guards and spacers within the body, as detailed below. The demonstrations are distinct pairs drawn from the body, excluding the target pair. Body records and demonstrations usekeyvalue : the two fields have no intervening delimiter token, but their token strings contain leading spaces, giving records such asant Alice. The prompt ends at the target key. The associated name is the gold next token and is not appended to the input. We show an abridged example in Figure 6. Scoring. We score the gold name using logits at the final input token. Candidate-restricted accuracy takes the argmax over all 64 candidate names (uniform-guess chance 1/64), while full-vocabulary accuracy takes the argmax over the entire tokenizer vocabulary. We also record the gold-token probability and negative log-likelihood. C.3 Padding Protocols for Qwen3-based Models Source phase and matched examples. Chunked compression gives each source token aphase: its alignment relative to the regularly spaced right boundaries of compression windows. For a model with compression stride S, phase is the source-token position moduloS. To compare models with different strides, we use the common 40 Example retrieval prompt (abridged) Remember each animal’s assigned human name. [ prefix filler omitted ] Animal-name pairs: snake Alex beetle Carl fox Lucy · bear Susan shark Helen · crow Nancy [ suffix filler omitted ] Recall the requested animal names: camel Anna spider Lisa chicken Molly bee Alan sheep Adam lizard Emma llama Ethan dog Eric bear Gold next token (not part of the prompt): Susan, including its leading space. Figure 6 An animal--name retrieval example with Prefix Padding. Natural-language fillers are abbreviated, and ·denotes omitted mapping records. All eight demonstrations and the final query are shown. The target mapping, bear Susan, occurs at body index 24 and is excluded from the demonstrations. sampling coordinate moduloM=lcm(4,6,8,12) = 24. For each of 1,000 logical examples, we construct all Msource-key residues, giving 24,000 prompts. The 24 prompts of a logical example share the mapping, pair order, and demonstration indices and differ only in their fillers or spacers, whose offsets are sampled separately for each residue. Letℓ desc andℓ hdr be the instruction and mapping-header lengths,ℓ pair the token count per pair, andL pre the effective inserted-token offset before the target, which the two protocols below produce in different ways. All lengths are measured with the tokenizer, including template whitespace. We define c = ℓ desc + ℓ hdr + j ∗ ℓ pair , t K = c + L pre , r = t K mod M.(104) Requesting source-key residuertherefore requiresL pre ≡ r− c(mod M). The target value is att K + 1 in the concatenated grammar, but neither the value position nor the final query key defines the swept phase. Since each tested strideSdividesM, the model’s phase isr mod S, and phase-wise results group prompts by r mod S, with equal numbers of prompts in each group. We also keep the results for each of the 24 residues alongside these stride-specific summaries. Sampling offsets with equal expected filler lengths. To vary phase without changing the expected insertion depth, we construct the integer-valued length variable X = M + U 1 + U 2 + U 3 + F, U i iid ∼ Unif0,...,M − 1, F ∼ Unif0,...,M + 3,(105) with all four draws independent. For each requested residuer, the sampler draws directly from the conditional length distribution and assigns the complementary suffix filler length: L pre ∼L(X | X ≡ r− c (mod M )), L post = 6M − L pre .(106) 41 ForM= 24, both lengths lie between 24 and 120 tokens, and their sum is exactly 144 tokens in every prompt. Moreover, for every residue r, E[L pre | r] = E[L post | r] = 3M = 72, Var(L pre | r) = Var(L post | r) = M 2 + 2M + 3 3 = 209. (107) Conditioning preserves these moments because every term in the expansion ofXorX 2 involves at most two random summands and so leaves at least one ofU 1 ,U 2 ,U 3 unused, which makes the residue of the sum uniform regardless of the variables in the term. The first and second raw moments are therefore the same under every residue, and the complementary construction gives the suffix filler length the same mean and variance. These equalities concern the sampling distribution, so the 1,000 random draws per residue need not have identical empirical means. By contrast, the total filler length is fixed for every example. Together with the fixed template and body lengths, this fixes the total prompt length and the absolute query and scoring positions across phases. The design therefore matches expected source depth and source-to-query distance while changing the source key’s phase. Prefix Padding.PrefixPaddingplaces a natural-language prefix filler of the sampled lengthL pre from Equation (106) directly before the mapping body and a natural-language suffix filler of lengthL post = 6M−L pre after it. VaryingL pre moves the packed mapping body, including the target pair, while the complementary suffix filler keeps the query and scoring positions fixed. Because the conditional mean ofL pre is the same for every source phase, no phase is systematically assigned an earlier or later target. The prefix and suffix fillers come from the training split of the high-quality-english-sentences dataset [1]. Each source sentence is rendered with a final , which is included in its token length. We build exact-length bins for every length from 24 through 120, retaining at most 64 distinct sentences per bin, and exclude sentences containing any curated key or value token ID. Sentences are used without padding or truncation. The generator first samples a length using Equation (106), then samples a sentence uniformly from that length bin, and draws the suffix filler from the complementary bin as a different sentence. Every length from 24 through 120 must have a nonempty bin, so missing bins do not change the conditional length law. For reproducibility, we use seed 20260720 for the source shuffle and seed 20260722 for mapping, demonstration, and filler generation, and each example and residue has its own seed. In-sequence Padding. Prefix Padding moves all records together, soIn-sequencePaddinginstead produces the offset inside the mapping body. It keeps the same conditional draw as a latentreferenceprefixlength, denoted e L pre , places a fixed 24-token guard before the mapping header and another fixed 24-token guard after the body, and defines x = e L pre − 24. The 64-pair body has 63 inter-record gaps. Letg 0 ,...,g 62 ∈0,...,4be the numbers of one-token spacers inserted after the corresponding records. Each spacer is the fixed one-token Unicode characterU+F5C (FULLWIDTH VERTICAL LINE), not the tokenizer’s special padding token. For the target at indexj ∗ = 24, we sample uniformly from the feasible bounded compositions satisfying 23 X i=0 g i = x, 62 X i=24 g i = 126− x.(108) Thus the effective offset before the target is 24 +x= e L pre , so the target obeys the same phase equation and phase-conditioned length law as Prefix Padding. The total number of in-body spacers is always 126, so the body boundaries, query position, scoring position, and total sequence length are fixed across phases. In-sequence Padding thus changes phase by redistributing bounded gaps around the target rather than by moving the entire body with a variable-length prefix filler. 42 The guards and spacers contribute 24 + 126 + 24 = 174 tokens, compared with 144 filler tokens in Prefix Padding. Thus the query is fixed across phases within each protocol but occurs 30 tokens later in In-sequence Padding. The source-key position has the same conditional distribution in both. The two protocols also reuse the same logical mappings, pair order, demonstrations, logical example indices, and 1,000 examples per source residue, so they test the same retrieval task under two distinct ways of producing the phase-balanced offset. Tokenization checks. Because phase is defined on exact token positions, we verify that every prompt tokenizes as constructed. For Prefix Padding, we check exact text/token round trips of each filler sentence and its additive joins with adjacent prompt components, and require the assembled prompt and the prompt followed by the gold answer to each re-encode exactly to their constructed token IDs. For In-sequence Padding, we independently retokenize every assembled prompt and verify the guard lengths, all 63 gap lengths and their sum, the two partition sums in Equation (108), the target span and phase, the fixed query and scoring positions, and the gold answer. D Additional Details on the Architecture and Pretraining of Qwen3-based Models Here we describe the architecture, attention memory, positional encoding, and training settings behind our pretrained models in Section 3. We report the complete sweep in Appendix F. D.1 Backbone and Compression Kernels Backbone and reference compression kernel. The Qwen3-0.6B backbone has 28 layers, hidden width 1,024, 16 query heads, and head dimensiond h = 128. We retain its feed-forward blocks, residual connections, and normalization, so that chunked KV compression is the only substantial architectural change. The reference compression kernel uses non-overlapping windows with W = S = 8 and one KV head per layer (a key–value head, as opposed to the 16 query heads). Its K/V branches are separate and use scalar gates: payload maps are shared across offsets (C B i =C B ), while gate maps and biases depend on the offset. The same configuration is used in every layer. The ablations vary KV-head count, pooling, parameter sharing, gate dimensionality, positional bias, window overlap, positional encoding, normalization, and local-memory policy. Gate granularity and parameter sharing. Using the notation of Section 3.1, scalar gates haved g = 1, with one gate score per token broadcast across all projected dimensions. Vector gates haved g =d h , allowing projected dimensions to prefer different window offsets. Separate K/V branches use their own payload maps, gate maps, and positional biases. Tied K/V branches share all three and produce a common compressed representation b K j = b V j , before the normalization and positional transformations described below. Within each branch, payload and gate maps can be shared independently across the whole window, within groups of offsets, or remain offset-specific. Sharing these maps does not require sharing positional biases across offsets. Uniform averaging fixes each gate-score coordinate at 1/Win a full window. The evaluated configurations are listed in Table 8. DeepSeek-style compression kernel. Combining these options recovers DeepSeek-V4’s CSA compression kernel [11], which lets us test whether the kernel studied in Section 2 produces phase sensitivity in our backbone. It usesW= 8,S= 4, vector gates, and tied K/V branches. The preceding four-token half, with offsetsi∈0,1,2,3, uses the payload and gate maps (C (0) ,Z (0) ). The current four-token half, with offsets i∈4,5,6,7, uses a separate pair (C (1) ,Z (1) ). Each pair is shared within its half. Positional biases remain offset-specific and are shared between the tied K/V branches. In a full window, one softmax spans all eight offsets for each gate dimension. D.2 Windows, Attention Memory, and Positional Encoding Each kernel above produces one compressed entry per window. To use these entries in attention, we specify when each entry is emitted, which entries a query sees alongside its local window, and at which position each entry is rotated. 43 Window indexing and initial boundary. Token positions start at zero, and compressed entries use one-based indicesj ≥1. Entryjcloses atjS−1 and pools offsets 0≤ i < Wat source positionsjS− W+i. When W > S, the first entries may extend before the sequence start. These entries are emitted as soon as their stride block closes. Negative positions are masked from the softmax, which normalizes over the valid offsets only. The corresponding payload terms are zero. Thus the full-window equation in Section 3.1 applies away from this initial boundary. Attention memory and visibility. For one query head, letq t ∈ R d h be its query after normalization and positional encoding. Writek u ,v u for local entries and ˆ k j , ˆv j for compressed entries after the branch-specific transformations below. With local-window widthT= 16, the visible raw positions areL t =max(0,t− T+ 1),...,t. Let J t index the visible compressed entries and n t =|J t | +|L t |. Stacking vectors as rows gives K t = ( ˆ k ⊤ j ) j∈J t (k ⊤ u ) u∈L t , V t = (ˆv ⊤ j ) j∈J t (v ⊤ u ) u∈L t ∈ R n t ×d h .(109) One softmax normalizes scores jointly across the two memories: a t = Softmax K t q t √ d h ∈ R n t , o t = V ⊤ t a t ∈ R d h .(110) Each KV head supplies the memory for its corresponding group of query heads, which retain their own queries and attention distributions. With a fixed-width local window, compressed visibility can follow one of two policies. The first delays visibility until the entry’s closure leaves the local window:J t =j ≥1 :jS−1≤ t− T. The second admits every causally closed entry,J t =j ≥1 :jS−1≤ t, allowing compressed and local entries to represent overlapping source tokens. Our fixed-width models use the second policy, with an exact 16-token raw window. The open-tail local memory instead admits entries as soon as they close but retains raw tokens only in the current incomplete stride block, so its raw tail is empty at the final token of each stride block. Positional encoding and K/V tying. LetR p ∈ R d h ×d h denote the rotary transformation at positionp. In the reference setting, queries and local keys receive per-head RMSNorm before RoPE, and compressed keys are normalized and rotated after pooling: q t = R t RMSNorm Q (W Q h t ), k u = R u RMSNorm K (W K h u ), ˆ k j = R (j−1)S RMSNorm K ( b K j ), (111) whereW Q ,W K ∈ R d h ×d are the query and local-key projections for the relevant heads. Under the one-based entry indexing above, position (j−1)Sis the start of the compressed window’s newest stride block. For an even number 0 < d r ≤ d h of rotary dimensions, the frequencies are constructed within that subspace: ω ℓ = θ −2ℓ/d r , ℓ = 0,...,d r /2− 1, θ = 10 6 .(112) The transformationR p rotates the finald r coordinates with anglespω ℓ , using Qwen3’s split-half pairing, and leaves the firstd h − d r coordinates unchanged. The reference usesd r =d h = 128, and partial-RoPE variants used r = 16, a rotary fraction of 1/8. NoPE usesd r = 0 andR p =I. No additional frequency or context-length scaling is applied. The variant without Q/K normalization replaces the RMSNorm operations with the identity. The partial-RoPE model builds this 16-dimensional rotary table directly and applies it to the last 16 of the 128 head dimensions. It does not select the firstd r /2 frequencies of the full-head table, whose exponent denominator would be d h rather than d r . With separate K/V branches, local values arev u =W V h u , withW V ∈ R d h ×d , and compressed values are ˆv j = b V j . Neither receives key normalization or rotation. With tied K/V branches, values instead equal the normalized, rotated keys:v u =k u andˆv j = ˆ k j . The combined attention output is then inverse-rotated at the query position, ̃o t =R −1 t o t , before concatenating query-head outputs and applying the standard output projection. The inverse acts on the same dimensions as RoPE and applies to the combined local and compressed readout. With separate branches, ̃o t = o t . 44 D.3 Training Settings The reference 100B-token training run uses AdamW with (β 1 ,β 2 ) = (0.9,0.95),ε= 10 −8 , weight decay 0.1, and gradient clipping at norm 1. Training uses BF16 and seed 42. The packed corpus is traversed once with sequence length 2,048 and a global batch of 1,024 sequences. The learning rate warms up for 2% of training to 2×10 −3 and decays by a cosine schedule to 2×10 −5 . The seed-replication sweep additionally uses seeds 43–45. Unless stated otherwise, results for our pretrained models use the final checkpoint at step 47,518. E Additional Details on KV-Head Knockouts Here we define the KV-head knockouts of Section 4.1. A knockout measures how retrieval changes when the prompt-specific output of an attention component is replaced by a fixed reference vector. We use the component’s mean output over separate calibration prompts as this reference, which removes what the output carries about the particular prompt while retaining its average. The intervention is evaluated on held-out prompts, and its effects are resolved by the phase of the source key. This mean replacement builds on causal analyses of attention heads and their retrieval roles [24,35,38]. We first define which output is replaced and how its reference is fitted, then describe the prompts and metrics, the sweep over our pretrained models, and what the resulting effects can and cannot show. Finally, we adapt the intervention to DeepSeek-V4-Flash-Base, whose single shared KV head and long prompts call for a position-aligned variant. E.1 Replaced Attention Output Our models use grouped-query attention (GQA) [2]. A layerℓhasH q query heads andH kv KV heads, and each KV headhis read by a groupQ(h) ofG=H q /H kv query heads, which share its keys and values but keep their own queries and attention distributions. In a compressed layer, the entries that a KV head exposes to a query combine visible compressed entries (Section 3.1) with uncompressed local entries. Knockouts change neither these entries nor the rule that decides which ones are visible. For query headaat positiont, the attention readoutr ℓ,a,t is the attention-weighted sum of the value vectors it reads. These attention weights are distinct from the compression gate scoresαthat form each compressed entry. Some tied-KV variants apply an inverse positional rotation to the readout before the output projection. We writez ℓ,a,t for the readout after this rotation, withz=rwhen none is applied, so the attention output is W O ℓ Concat a z ℓ,a,t +b O ℓ . The intervention acts onz, immediately before the output projection, where each query head’s contribution is still a separate vector. WithT x the final input position of promptx, whose logits predict the answer, we writeofor the stacked outputs of all query heads that read KV headhat this scoring position: o ℓ,h (x) = Concat a∈Q(h) z ℓ,a,T x (x)∈ R Gd v ,(113) whered v is the value dimension. This is a group of query-head outputs rather than an additional output of the KV head. WithH q = 16,H kv = 4, andd v = 128, one knockout replaces four query-head vectors, a 512-dimensional group output. With one KV head, it replaces all 16 query-head outputs at that layer and position, which is the whole attention readout at that site. E.2 Calibration and Mean Replacement An unmodified forward pass over a calibration setD cal =x (n) N cal n=1 , disjoint from evaluation, records every query head’s output at the scoring position. Because query heads that share a KV head keep their own queries and attention distributions, each query head receives its own mean, μ ℓ,a = 1 N cal N cal X n=1 z ℓ,a,T x (n) (x (n) ), ̄ o ℓ,h = Concat a∈Q(h) μ ℓ,a ,(114) so a knockout retains each head’s own average output, which a mean pooled over the group would not. The calibration set is balanced over source-key residues, so no phase dominates the mean, and each mean also averages over mappings and filler texts. The reference is therefore independent of the evaluation prompt and its phase, and a phase-dependent knockout effect cannot reflect a phase-specific reference. It remains specific 45 to the model, layer, query head, and calibration distribution. A phase-conditioned mean, or replacement by zero, would define a different intervention. To knock out KV headh ∗ in layerℓ ∗ , we replace the outputs of every query head that reads it at the scoring position (Figure 7): e z ℓ ∗ ,a,t (x) = ( μ ℓ ∗ ,a ,a∈Q(h ∗ ), t = T x , z ℓ ∗ ,a,t (x), otherwise. (115) Other heads in the layer and all earlier computation are left unchanged, and later computation at the scoring position is rerun. Because the patch sits at the final input position of a causal decoder, it cannot change earlier context representations, and neither the model parameters nor the context cache are edited. A Extract the substitute: mean head outputs at t=T over N phase-balanced NIAH prompts continue the unpatched forward pass O projection W O over the concatenation of all H q head outputs at t=T q 1 softmax P j p j v j q 2 softmax P j p j v j K 1 , V 1 z 1 z 2 q 2h−1 softmax P j p j v j q 2h softmax P j p j v j K h , V h z 2h−1 z 2h · group 1 group h q a from contextual state u ℓ,T ; K h , V h : visible memory (compressed entries + local window) input: N calibration prompts, n = 1 . . . N, source phase r balanced over 0 . . . S−1; unpatched forward pass record every z a (T ) z (n) a (T ), all query heads a one vector per prompt n mean over n ̄z a = 1 N P n z (n) a (T ) kept for every a = 1 . . . H q nothing is substituted in this pass B Knock out KV head h: substitute the means for its readers before W O , on a test prompt → residual add→ MLP→ · → logits at t=T : accuracy by source phase r vs. unpatched O projection W O over the concatenation of all H q head outputs at t=T q 1 softmax P j p j v j q 2 softmax P j p j v j K 1 , V 1 z 1 z 2 q 2h−1 softmax P j p j v j × q 2h softmax P j p j v j × K h , V h ̄z 2h−1 ̄z 2h · group 1 group h q a from contextual state u ℓ,T ; K h , V h : visible memory (compressed entries + local window) input: one test prompt, source key at phase r; patch only the selected layer at T ; subsequent computation is rerun ̄z 2h−1 , ̄z 2h from A substitute at t=T only the same vectors for every test prompt and every phase r KV head h knocked out: its readers q 2h−1 , q 2h still run, their outputs are discarded (×); K h , V h themselves are untouched Figure 7 Calibration and mean replacement in a GQA block. (A) Unmodified calibration prompts supply a separate mean for every query-head output at the scoring position. (B) On each evaluation prompt, the query heads that read the selected KV head receive those fixed means before the output projection, and the KV head’s keys and values are left intact. The schematic shows two query heads per KV head (G= 2) and suppresses the layer index and the inverse positional rotation. Each query head’s outputz a corresponds toz ℓ,a,T x and its mean ̄z a toμ ℓ,a . Concatenating the selected query-head outputs giveso ℓ,h in Equation (113). The patch is confined to the selected layer and position, and downstream representations are recomputed. 46 E.3 Evaluation Prompts, Metrics, and Uncertainty Knockouts in our pretrained models use the retrieval task of Appendix C.2 under both padding protocols of Appendix C.3. Each protocol has its own calibration set, built with that protocol’s prompts. Each prompt is indexed by its source-key residuer=t K mod24, wheret K is the source key’s zero-based position, and a model with strideSsees phaser mod S. The 24 phase replicas of a logical example stay together when calibration is separated from evaluation. Logical examples [1000,2000) form the calibration set and [0,1000) the evaluation set, each with 24,000 prompts. A prompt counts as correct when the gold name has the largest final logit, either among the 64 candidate names (candidate-restricted accuracy) or over the whole vocabulary (full-vocabulary accuracy). We also record the gold logit, the gold negative log-likelihood, and the total probability of the candidate set. Let D r be the evaluation prompts at residuerandY c (x)∈0,1the correctness of promptxin the clean run (c = 0) or under the knockout (c = KO(ℓ,h)). The paired effect, in percentage points, is ∆ ℓ,h (r) = 100 |D r | X x∈D r Y KO(ℓ,h) (x)− Y 0 (x) ,(116) which is negative when the knockout impairs retrieval. Both terms use the same prompts. For strideS, folded effects average the residues that share a phase, for example residuesr,r+ 8, andr+ 16 for W8/S8. Full-attention baselines have no phase and are grouped by the same residues only for comparison. A phase with low clean accuracy has less room for a further decrease. In Figure 4 and the figures of Appendix F, we therefore plot the relative accuracy change 100(A KO (r)− A 0 (r))/A 0 (r), which scales each loss by the clean accuracy available to lose. HereA c (r) is the accuracy at residuer. This ratio is undefined when clean accuracy is zero, so the figures apply a validity mask. The independent sampling unit is the logical example, not each of its phase replicas. For an overall effect, we average paired differences within each logical example and take the standard error across examples, and we cluster by logical example in the same way for folded or between-phase contrasts. These clustered standard errors are computed but not plotted, and the heatmaps show point estimates that do not establish significance for every cell. The retrieval curves in the main text show two-sided 95% Wilson intervals, and the clean curves in Appendix F show pointwise 95% cluster-bootstrap intervals over logical examples. Selecting the largest loss from a sweep is exploratory, and we do not test gate-nominated heads on held-out prompts. E.4 Knockout Sweeps for Qwen3-based Models To check whether phase-dependent knockout effects extend beyond the model of Figure 4b, we sweep the 23 compressed models listed in Table 8, which have one, two, four, or eight KV heads, at checkpoint 47,518. Each model is evaluated over all 28 layers with the disjoint 24,000-prompt calibration and evaluation sets above. In the six figure groups of Appendix F, we compare Prefix Padding and In-sequence Padding and report unfolded relative full-vocabulary accuracy changes with group-wise zero-centered scales. These are point estimates rather than independently confirmed selections. E.5 Scope of the Causal Interpretation The experiment changes an internal representation while holding the evaluation prompt and model parameters fixed. A loss of accuracy therefore measures a causal effect of this intervention on the tested retrieval behavior. It supports the importance of the selected readout under mean replacement, rather than exclusive storage of the target in that KV head. Other KV heads may contain redundant information, and a head can contribute to query processing or downstream computation in addition to carrying an answer. Conversely, a small knockout effect does not establish that a component is unused: other components can compensate. The change in the attention output is linear in ̄ o ℓ ∗ ,h ∗ −o ℓ ∗ ,h ∗ (x), but its effect on the final logits generally is not, so the effects of knocking out different heads separately need not add up to the effect of removing them together. In compressed layers, one attention softmax covers compressed and local entries, and mean replacement acts on their combined readout. It does not selectively ablate compressed memory, remove one source association, 47 or change a compression gate. Mean replacement preserves a reference first moment but does not guarantee an in-distribution activation or a matched norm. It removes within-head variation and the selected outputs’ prompt-specific correlations with the rest of the network, and the calibrated mean can itself retain common task and template information. Consequently, the intervention is neither a literal deletion of a memory nor a proof of a unique retrieval circuit. We also do not report an identity-intervention comparison or a comparison that replaces the outputs of an equal number of other query heads in the same layer. E.6 Knockouts for DeepSeek-V4-Flash-Base The DeepSeek-V4-Flash-Base knockout follows the same mean-replacement logic but differs in what it replaces and at which positions. The model has one shared KV head per layer, so, as in our pretrained models with one KV head, a knockout replaces the whole attention readout of a layer. We take this readout after the attention output projection and before the hyper-connection, so the patched vector is the entire 4,096-dimensional layer output rather than a query-head slice of the pre-projection tensor in Equation (113). The patch can also cover several positions of the query suffix rather than only the final one. Leta ℓ,s (x)∈ R 4096 denote that output at suffix indexs∈0,...,15. Calibration produces a separate mean for each layer and eachsuffixposition: ν ℓ,s = 1 N cal N cal X n=1 a ℓ,s (x (n) ), e a ℓ,s (x) = ( ν ℓ,s , s∈P, a ℓ,s (x), s /∈P. (117) The maskPselects one of four conditions: the first occurrence of the query key, its second occurrence, both occurrences, or the complete 16-token suffix. In Figure 4c, we show the full-suffix condition, which patches all 16 suffix positions. Suffix positions play different roles in the query, such as the two occurrences of the query key, so each receives its own mean rather than an average over the suffix, just as each query head receives its own mean in our pretrained models. The same position-aligned means are used for every evaluation prompt and source phase. Patching earlier suffix positions can also affect later suffix computation, so the full-suffix condition has a broader scope than the final-position-only intervention in our pretrained models. For Figure 4c, we use 256 evaluation prompts per residue group, each 128,000 tokens long with 16,000 key–value pairs. Calibration and evaluation use disjoint prompts with distinct texts. The source-key coordinate is modulo eight, so these residue groups span two cycles of the CSA stride of four. The 127,984-token prefix is prefilled in 1,024-token chunks, and the remaining 16 tokens form the query suffix. We score the designated gold target token without autoregressive answer generation. This target-token accuracy is distinct from the answer-prefix accuracy used in the behavioral benchmark of Section 2. Because DeepSeek-V4-Flash-Base’s outputs vary with batch size, each clean run uses the same batch composition as the intervened run it is compared with. In the full-suffix condition of Figure 4c, the largest phase-averaged target-token accuracy losses occur at layers 22 and 24 (zero-based). Effects vary across source residues. They establish phase-dependent reliance on the patched layer readout at this intervention granularity, but they do not identify a particular query head, a compressed-only branch, or the compression operation that wrote the relevant information. F Complete Results for Qwen3-based Models In Section 3.2, we plot a subset of our pretrained models. Here we give fuller results, so that each design change can be compared with the reference and the full-attention baselines. In Table 8, we summarize the configurations shown in Figure 3 and the grouped figures below, reporting validation loss alongside mean and worst-residue retrieval accuracy and the gap across the 24 evaluated source-key residues under Prefix Padding. We include 23 compressed runs and three full-attention baselines. These summary numbers cannot show where the weak spots fall or which layers support retrieval at each phase. In the following figures, we therefore show our pretrained models at the end of training (step 47,518) in six groups that follow the blocks of compressed runs in Table 8. In each figure, we pair clean full-vocabulary 48 retrieval across the 24 source-key residues with layer-by-phase relative accuracy changes under KV-head knockouts (Appendix E). Across the six groups, no tested change removes the periodic weak spots, although some narrow them substantially. Uniform averaging, for example, reduces the best-to-worst gap from 75.3 to 19.2 percentage points. Random seeds, positional encoding, and normalization reshape the accuracy pattern across phases, and the weak spots repeat with the compression stride. We show every group under both padding protocols of Appendix C.3. Prefix Padding shifts all records together, so the distractors change phase along with the target. In-sequence Padding instead varies the gaps between records, so a pattern that appears under both protocols does not require all records to shift together. Rows 1 and 2 show clean accuracy and knockout effects under Prefix Padding, and rows 3 and 4 show the same under In-sequence Padding. Curves use 1,000 logical examples and show pointwise 95% cluster-bootstrap intervals. Heatmaps within each group share a zero-centered scale. 49 Table 8 Summary of the plotted pretrained models. Prefix Padding results for the 26 runs shown in the cited main-text and appendix figures: 23 compressed runs and three full-attention baselines, evaluated at the end of training (step 47,518). For full-vocabulary retrieval accuracyA ρ at source-key residueρ=t K mod24, we report the mean over all 24 residues, the minimum, and the gapmax ρ A ρ − min ρ A ρ in percentage points. The same coordinate is used for full-attention baselines, which have no compression phase. Validation loss is reported separately from retrieval. Retrieval accuracy Configuration / changeW/S KV Val. loss Mean (%) Min. (%) Gap (p) Full-attention baselines (Figure 3) Full attention–12.46618.817.23.1 Full attention–42.44847.144.86.1 Full attention–82.43761.158.44.3 Reference and random-seed replication (Figure 8) Reference, seed 428/8 12.47850.65.775.3 Reference, seed 438/8 12.48232.12.559.2 Reference, seed 448/8 12.47756.111.670.3 Reference, seed 458/8 12.47840.515.636.6 Native KV-head count (Figure 9) Two KV heads8/8 22.46558.39.876.8 Four KV heads8/8 42.44868.713.678.0 Eight KV heads8/8 82.43159.49.971.1 Compression-window size and stride (Figure 10) Non-overlapping windows4/4 12.46938.58.050.5 Overlapping windows8/4 12.46766.946.239.8 Overlapping windows8/6 12.47053.438.531.9 Larger windows, one KV head12/12 12.48733.05.063.2 Compression kernels (Figure 11) Tied K/V branches8/8 12.49521.83.639.1 Uniform averaging8/8 12.49712.54.719.2 Uniform averaging, tied K/V, postnorm8/8 12.50318.61.939.3 Vector gates8/8 12.47819.44.034.0 Positional encoding and normalization (Figure 12) No Q/K normalization8/8 12.48538.523.834.6 No positional gate biases8/8 12.47934.34.460.7 No RoPE (NoPE)8/8 12.50429.45.040.9 RoPE on 16 dimensions8/8 12.48274.232.957.2 Local-memory policy and DeepSeek-style kernels (Figure 13) Uncompressed open tail8/8 12.52075.057.433.9 Uncompressed open tail8/8 82.46053.025.549.3 DeepSeek-style, vector gates8/4 12.47635.618.738.3 DeepSeek-style, scalar gates8/4 12.47321.514.218.7 Notes. KV denotes the number of KV heads per layer. The reference uses scalar gates, separate K/V branches, payload maps shared across offsets, and offset-specific gate maps and positional biases (Section 3.1 and Appendix D). Postnorm denotes post-projection normalization. Each run appears once, and the seed-42 reference supplies the one-KV-head model for the head-count comparison. 50 0 0.25 0.5 0.75 1.0 Pre fi x Padding Accuracy (a) W8/S8 · 1 KV · seed 42 Δ 75.3 p val loss 2.478 0816 0 7 14 21 27 Pre fi x Padding Layer KV-head knockout relative accuracy change (%) (b) W8/S8 · 1 KV · seed 43 Δ 59.2 p val loss 2.482 0816 KV-head knockout relative accuracy change (%) (c) W8/S8 · 1 KV · seed 44 Δ 70.3 p val loss 2.477 0816 KV-head knockout relative accuracy change (%) (d) W8/S8 · 1 KV · seed 45 Δ 36.6 p val loss 2.478 0816 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 0 0.25 0.5 0.75 1.0 In-sequence Padding Accuracy Δ 44.0 p 0816 0 7 14 21 27 In-sequence Padding Layer KV-head knockout relative accuracy change (%) Δ 28.9 p 0816 KV-head knockout relative accuracy change (%) Δ 54.4 p 0816 KV-head knockout relative accuracy change (%) Δ 20.1 p 0816 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 Source-key position modulo 24 Figure 8 Random-seed replication. Rows 1–2 use Prefix Padding and rows 3–4 use In-sequence Padding. Rows 1 and 3 show full-vocabulary accuracy, and rows 2 and 4 show relative accuracy change under KV-head knockout. Seed changes alter the accuracy pattern across phases while preserving periodic weak spots. 0 0.25 0.5 0.75 1.0 Pre fi x Padding Accuracy (a) W8/S8 · 1 KV · seed 42 Δ 75.3 p val loss 2.478 0816 0 7 14 21 27 Pre fi x Padding Layer KV-head knockout relative accuracy change (%) (b) W8/S8 · 2 KV Δ 76.8 p val loss 2.465 0816 KV-head knockout relative accuracy change (%) (c) W8/S8 · 4 KV Δ 78.0 p val loss 2.448 0816 KV-head knockout relative accuracy change (%) (d) W8/S8 · 8 KV Δ 71.1 p val loss 2.431 0816 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 0 0.25 0.5 0.75 1.0 In-sequence Padding Accuracy Δ 44.0 p 0816 0 7 14 21 27 In-sequence Padding Layer KV-head knockout relative accuracy change (%) Δ 50.6 p 0816 KV-head knockout relative accuracy change (%) Δ 88.7 p 0816 KV-head knockout relative accuracy change (%) Δ 59.8 p 0816 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 Source-key position modulo 24 Figure 9 Number of KV heads. Rows 1–2 use Prefix Padding and rows 3–4 use In-sequence Padding. Rows 1 and 3 show full-vocabulary accuracy, and rows 2 and 4 show relative accuracy change under KV-head knockout. Phase sensitivity persists from one to eight KV heads. 51 0 0.25 0.5 0.75 1.0 Pre fi x Padding Accuracy (a) W=4, S=4 Compression Baseline Δ 50.5 p val loss 2.469 048121620 0 7 14 21 27 Pre fi x Padding Layer KV-head knockout relative accuracy change (%) (b) W=8, S=4 Compression Baseline Δ 39.8 p val loss 2.467 048121620 KV-head knockout relative accuracy change (%) (c) W=8, S=6 Compression Baseline Δ 31.9 p val loss 2.470 061218 KV-head knockout relative accuracy change (%) (d) W=12, S=12 Compression Baseline Δ 63.2 p val loss 2.487 012 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 0 0.25 0.5 0.75 1.0 In-sequence Padding Accuracy Δ 43.3 p 048121620 0 7 14 21 27 In-sequence Padding Layer KV-head knockout relative accuracy change (%) Δ 61.8 p 048121620 KV-head knockout relative accuracy change (%) Δ 26.4 p 061218 KV-head knockout relative accuracy change (%) Δ 32.0 p 012 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 Source-key position modulo 24 Figure 10 Compression geometry. Rows 1–2 use Prefix Padding and rows 3–4 use In-sequence Padding. Rows 1 and 3 show full-vocabulary accuracy, and rows 2 and 4 show relative accuracy change under KV-head knockout. Weak spots repeat with the compression stride. 0 0.25 0.5 0.75 1.0 Pre fi x Padding Accuracy (a) W8/S8 · shared KV Δ 39.1 p val loss 2.495 0816 0 7 14 21 27 Pre fi x Padding Layer KV-head knockout relative accuracy change (%) (b) W8/S8 · mean pooling Δ 19.2 p val loss 2.497 0816 KV-head knockout relative accuracy change (%) (c) W8/S8 · mean/shared/postnorm Δ 39.3 p val loss 2.503 0816 KV-head knockout relative accuracy change (%) (d) W8/S8 · vector gate Δ 34.0 p val loss 2.478 0816 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 0 0.25 0.5 0.75 1.0 In-sequence Padding Accuracy Δ 34.6 p 0816 0 7 14 21 27 In-sequence Padding Layer KV-head knockout relative accuracy change (%) Δ 37.0 p 0816 KV-head knockout relative accuracy change (%) Δ 23.3 p 0816 KV-head knockout relative accuracy change (%) Δ 25.7 p 0816 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 Source-key position modulo 24 Figure 11 Compression kernels. Rows 1–2 use Prefix Padding and rows 3–4 use In-sequence Padding. Rows 1 and 3 show full-vocabulary accuracy, and rows 2 and 4 show relative accuracy change under KV-head knockout. Phase sensitivity persists across pooling, sharing, and gate variants. 52 0 0.25 0.5 0.75 1.0 Pre fi x Padding Accuracy (a) W8/S8 · no QK norm Δ 34.6 p val loss 2.485 0816 0 7 14 21 27 Pre fi x Padding Layer KV-head knockout relative accuracy change (%) (b) W8/S8 · no Z bias Δ 60.7 p val loss 2.479 0816 KV-head knockout relative accuracy change (%) (c) W8/S8 · NoPE Δ 40.9 p val loss 2.504 0816 KV-head knockout relative accuracy change (%) (d) W8/S8 · RoPE-16 Δ 57.2 p val loss 2.482 0816 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 0 0.25 0.5 0.75 1.0 In-sequence Padding Accuracy Δ 25.0 p 0816 0 7 14 21 27 In-sequence Padding Layer KV-head knockout relative accuracy change (%) Δ 63.2 p 0816 KV-head knockout relative accuracy change (%) Δ 40.0 p 0816 KV-head knockout relative accuracy change (%) Δ 39.0 p 0816 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 Source-key position modulo 24 Figure 12 Position and normalization. Rows 1–2 use Prefix Padding and rows 3–4 use In-sequence Padding. Rows 1 and 3 show full-vocabulary accuracy, and rows 2 and 4 show relative accuracy change under KV-head knockout. Position and normalization choices reshape but do not remove phase sensitivity. 0 0.25 0.5 0.75 1.0 Pre fi x Padding Accuracy (a) Tail concat · 1 KV Δ 33.9 p val loss 2.520 0816 0 7 14 21 27 Pre fi x Padding Layer KV-head knockout relative accuracy change (%) (b) Tail concat · 8 KV Δ 49.3 p val loss 2.460 0816 KV-head knockout relative accuracy change (%) (c) DSV4 kernel · vector gate Δ 38.3 p val loss 2.476 048121620 KV-head knockout relative accuracy change (%) (d) DSV4 kernel · scalar gate Δ 18.7 p val loss 2.473 048121620 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 0 0.25 0.5 0.75 1.0 In-sequence Padding Accuracy Δ 28.1 p 0816 0 7 14 21 27 In-sequence Padding Layer KV-head knockout relative accuracy change (%) Δ 40.9 p 0816 KV-head knockout relative accuracy change (%) Δ 26.0 p 048121620 KV-head knockout relative accuracy change (%) Δ 25.8 p 048121620 KV-head knockout relative accuracy change (%) −100 −50 0 50 100 Source-key position modulo 24 Figure 13 Memory policy and implementation variants. Rows 1–2 use Prefix Padding and rows 3–4 use In-sequence Padding. Rows 1 and 3 show full-vocabulary accuracy, and rows 2 and 4 show relative accuracy change under KV-head knockout. Open-tail local memory and DeepSeek-style kernels remain phase-sensitive. 53 G Gate Preferences and Knockout Effects across Models In Section 4.2, we link gate preferences to knockout effects for three heads of one model. Here we show the same comparison in more models: every head of the four W8/S8 seed replications, and every head of five other configurations. Sharp gate preferences with knockout damage at the favored phases recur across seeds and configurations, but not in every head, and the phase-folded gate curves show little preference when windows overlap. G.1 Measuring Gate Preferences In Section 4.2, we summarize each gate by two effective numbers of offsets (Figure 5a). For one windowjof inputx, collect the gate scores of itsWoffsets into the probability vectorα B ℓ,h,j (x)∈ R W , and letH(p) = − P i p i logp i . The vertical axis is the mean per-window effective number of offsets,E x,j [exp(H(α B ℓ,h,j (x)))], averaged over all compression windows of a held-out validation set of natural-language pretraining text. The horizontal axis is the effective number of offsets of the mean gate,exp(H(E x,j [α B ℓ,h,j (x)])). Both range from 1 toW, and smaller values mean greater concentration. In Section 4.2, we call the horizontal quantity the gate’s staticconcentration. The vertical quantity also includesinput-dependentconcentration, which places a gate below the diagonal. A small per-window value alone does not imply a stable preference. A gate that selects one offset in each window but visits every offset equally often across inputs has a uniform mean gate and lies near (W,1). For vector gates, averaging over channels can conceal preferences that differ across channels, and our channel-averaged curves (Figure 18b) cannot reveal them. In Figure 14, we place the key gate of every KV head in this plane at the end of training, for every run in Table 8 with learned scalar gates and for the W12/S12 model with four KV heads used in Appendix H. With non-overlapping windows, a few heads per model lie along the diagonal toward the lower left, while most gates end near the right edge with varying input-dependent concentration, as in Figure 5a. With more KV heads per layer, the most concentrated heads reach further toward the lower left. With overlapping windows at stride four, including the DeepSeek-style kernel, many gates sit well left of the right edge but below the diagonal, so they favor some offsets across windows while also varying with the input. In the figures of this appendix, we overlay each head’s gate preferences on its knockout effects. Each panel shows one KV head, with the phaserof the source key on the horizontal axis. The background shows the head’s knockout effect at each phase (Section 4.1), with red indicating impairment. The black curve,g K (r), is the head’s mean gate score on key tokens at phaser, scaled so that a uniform gate equals one (dashed line). The purple curve,g V (r), shows the value gate read one phase later. Retrieval needs the key, to locate the entry, and the value one token later, which is the answer. For a key at phaser, we therefore plot the value gate’s score at phase (r+ 1)mod S, so that both curves and the knockout background are indexed by the key’s phase. A head that retains a key and its value shows both curves peaking at the samer, and if the head matters for retrieval, its knockout damage should peak there as well. WhenW=S, the value of a key at the last phase S− 1 falls in the next window, and hollow squares mark these points. G.2 Variation across Random Seeds The four seed replications develop similarly sharp gate preferences, but in different places (Figure 16). Each seed has four to six heads with concentrated mean key gates (entropy below 90% of a uniform gate’s), whose key-gate peaks reach 2.0 to 4.0 times the uniform level and whose knockout damage falls mainly at the favored phases. These heads lie between layers 4 and 18, but no layer has such a head in all four seeds, and their preferred key phases cover all eight phases. Among these heads, the only recurring feature is a head whose key gate favors phase 0 and whose value gate favors the value token one phase later, found at layers 11, 14, 13, and 12 in seeds 42 to 45, respectively. This variation is consistent with Section 5.2, where in an idealized model each head settles on a sharp selection of positions that depends on initialization, and heads need not cover complementary positions. Outside these heads, knocking out layer 0 impairs retrieval at every phase in all seeds, despite its near-uniform gate. 54 G.3 Other Configurations and Overlapping Windows Sharp preferences also arise without rotary position embeddings. Without them, heads in layers 11, 12, 14, and 15 still favor particular phases, with knockout damage at those phases (Figure 15a). We show the same comparison with larger windows (W12/S12, one KV head) in Figure 15b and with four KV heads per layer (W8/S8) in Figure 17. With four KV heads, the preferences become sharper than with one. Many heads spike on a single offset, and their knockout losses concentrate on a single phase, as for heads 2 of layers 5 and 16 and heads 0 of layers 10 and 12. Together, these sharp heads favor all eight phases. Overlapping windows are the exception. With overlapping eight-token windows at stride four, including the DeepSeek-style kernel, most phase-folded gate curves stay close to uniform (Figure 18), leaving little phase structure for knockout damage to follow. Each phase sums two offsets, however, and many gates in these models do favor some offsets across windows (Figure 14), a preference that folding into phases hides. Vector-gate scores are also averaged over channels, which can hide preferences among channels. The gate account of Section 4.2 therefore applies most directly to non-overlapping windows. 55 12468 1 2 4 6 8 W8/S8 1 KV · seed 42 12468 1 2 4 6 8 W8/S8 1 KV · seed 43 12468 1 2 4 6 8 W8/S8 1 KV · seed 44 12468 1 2 4 6 8 W8/S8 1 KV · seed 45 12468 1 2 4 6 8 W8/S8 2 KV 12468 1 2 4 6 8 W8/S8 4 KV 12468 1 2 4 6 8 W8/S8 8 KV 1234 1 2 3 4 W4/S4 1 KV 12468 1 2 4 6 8 W8/S4 1 KV 12468 1 2 4 6 8 W8/S6 1 KV 14812 1 4 8 12 W12/S12 1 KV 14812 1 4 8 12 W12/S12 4 KV 12468 1 2 4 6 8 W8/S8 shared KV 12468 1 2 4 6 8 W8/S8 no QK norm 12468 1 2 4 6 8 W8/S8 no Z bias 12468 1 2 4 6 8 W8/S8 NoPE 12468 1 2 4 6 8 W8/S8 RoPE-16 12468 1 2 4 6 8 Tail concat 1 KV 12468 1 2 4 6 8 Tail concat 8 KV 12468 1 2 4 6 8 DSV4 kernel scalar gate 0 4 8 12 16 20 24 Layer exp ¡ H ¡ 피[® K ] ¢ 피 £ exp ¡ H ( ® K ) ¢ ¤ Figure 14 Final key-gate concentration across configurations (step 47,518). Each point is one KV head’s key gate, placed by its static effective number of offsetsexp(H(E[α K ])) (horizontal) and its mean per-window effective numberE[exp(H(α K ))] (vertical), and colored by layer. The dashed line marks equality, which holds for a gate with the same scores in every window, and lower values mean more concentration. Panels cover every run in Table 8 with learned scalar gates and the W12/S12 model with four KV heads of Appendix H. We omit uniform averaging, which has no learned gates, and the vector-gate runs. Both quantities are defined in Appendix G.1. 56 0 5 L0L1L2L3L4L5L6 0 5 L7L8L9L10L11L12L13 0 5 L14L15L16L17L18L19L20 01234567 0 5 L21 01234567 L22 01234567 L23 01234567 L24 01234567 L25 01234567 L26 01234567 L27 0 3 6 L0L1L2L3L4L5L6 0 3 6 L7L8L9L10L11L12L13 0 3 6 L14L15L16L17L18L19L20 0369 0 3 6 L21 0369 L22 0369 L23 0369 L24 0369 L25 0369 L26 0369 L27 −100 −50 0 50 100 Relative Accuracy Change (%) (a) No RoPE (NoPE), W8/S8, 1 KV head Gate Profile g B ( r ) Source-Key Phase r mod 8 (b) Larger windows, W12/S12, 1 KV head Gate Profile g B ( r ) Source-Key Phase r mod 12 Figure 15 Gate preferences and knockout effects without positional embeddings and with larger windows. (a) No rotary position embeddings (NoPE, W8/S8, one KV head). (b) Larger windows (W12/S12, one KV head). Without RoPE, heads still develop sharp phase preferences, with knockout damage at the favored phases (layers 11, 12, 14, and 15). Both checkpoints are final (step 47,518) and use Prefix Padding. Vertical scales are shared within each model. Encoding and gate inputs follow Figure 16. 57 0 5 L0L1L2L3L4L5L6 0 5 L7L8L9L10L11L12L13 0 5 L14L15L16L17L18L19L20 01234567 0 5 L21 01234567 L22 01234567 L23 01234567 L24 01234567 L25 01234567 L26 01234567 L27 0 5 L0L1L2L3L4L5L6 0 5 L7L8L9L10L11L12L13 0 5 L14L15L16L17L18L19L20 01234567 0 5 L21 01234567 L22 01234567 L23 01234567 L24 01234567 L25 01234567 L26 01234567 L27 0 5 L0L1L2L3L4L5L6 0 5 L7L8L9L10L11L12L13 0 5 L14L15L16L17L18L19L20 01234567 0 5 L21 01234567 L22 01234567 L23 01234567 L24 01234567 L25 01234567 L26 01234567 L27 0 5 L0L1L2L3L4L5L6 0 5 L7L8L9L10L11L12L13 0 5 L14L15L16L17L18L19L20 01234567 0 5 L21 01234567 L22 01234567 L23 01234567 L24 01234567 L25 01234567 L26 01234567 L27 −100 −50 0 50 100 Relative Accuracy Change (%) (a) Seed 42 Gate Profile g B ( r ) (b) Seed 43 Gate Profile g B ( r ) (c) Seed 44 Gate Profile g B ( r ) (d) Seed 45 Gate Profile g B ( r ) Source-Key Phase r mod 8 Figure 16 Gate preferences and knockout effects across random seeds (W8/S8, one KV head, seeds 42–45, step 47,518). Each panel shows one layer’s KV head, labeled Lℓ. Black circles show the key gateg K , purple squares the value gateg V read one phase later, and the dashed line a uniform gate. Hollow squares mark values that fall in the next window. The background shows the relative accuracy change under the head’s knockout, saturating at±100%, with red indicating impairment (Prefix Padding, 1,000 logical examples per phase, folded moduloS). Gate curves average 2,048 held-out natural-language sequences of 2,048 tokens. All seeds share one vertical scale. 58 0 7 L0 H0L0 H1L0 H2L0 H3L1 H0L1 H1L1 H2L1 H3 0 7 L2 H0L2 H1L2 H2L2 H3L3 H0L3 H1L3 H2L3 H3 0 7 L4 H0L4 H1L4 H2L4 H3L5 H0L5 H1L5 H2L5 H3 0 7 L6 H0L6 H1L6 H2L6 H3L7 H0L7 H1L7 H2L7 H3 0 7 L8 H0L8 H1L8 H2L8 H3L9 H0L9 H1L9 H2L9 H3 0 7 L10 H0L10 H1L10 H2L10 H3L11 H0L11 H1L11 H2L11 H3 0 7 L12 H0L12 H1L12 H2L12 H3L13 H0L13 H1L13 H2L13 H3 0 7 L14 H0L14 H1L14 H2L14 H3L15 H0L15 H1L15 H2L15 H3 0 7 L16 H0L16 H1L16 H2L16 H3L17 H0L17 H1L17 H2L17 H3 0 7 L18 H0L18 H1L18 H2L18 H3L19 H0L19 H1L19 H2L19 H3 0 7 L20 H0L20 H1L20 H2L20 H3L21 H0L21 H1L21 H2L21 H3 0 7 L22 H0L22 H1L22 H2L22 H3L23 H0L23 H1L23 H2L23 H3 0 7 L24 H0L24 H1L24 H2L24 H3L25 H0L25 H1L25 H2L25 H3 01234567 0 7 L26 H0 01234567 L26 H1 01234567 L26 H2 01234567 L26 H3 01234567 L27 H0 01234567 L27 H1 01234567 L27 H2 01234567 L27 H3 −100 −50 0 50 100 Relative Accuracy Change (%) Four KV heads, W8/S8 Gate Profile g B ( r ) Source-Key Phase r mod 8 Figure 17 Gate preferences and knockout effects with four KV heads per layer (W8/S8, step 47,518, Prefix Padding). Panels are labeled Lℓ Hh for layer ℓ and KV head h. Encoding and gate inputs follow Figure 16. 59 0 3 L0L1L2L3L4L5L6 0 3 L7L8L9L10L11L12L13 0 3 L14L15L16L17L18L19L20 0123 0 3 L21 0123 L22 0123 L23 0123 L24 0123 L25 0123 L26 0123 L27 0 1 2 L0L1L2L3L4L5L6 0 1 2 L7L8L9L10L11L12L13 0 1 2 L14L15L16L17L18L19L20 0123 0 1 2 L21 0123 L22 0123 L23 0123 L24 0123 L25 0123 L26 0123 L27 −100 −50 0 50 100 Relative Accuracy Change (%) (a) Overlapping windows, W8/S4, 1 KV head Gate Profile g B ( r ) Source-Key Phase r mod 4 (b) DeepSeek-style, vector gates, W8/S4, 1 KV head Gate Profile g B ( r ) Source-Key Phase r mod 4 Figure 18 Gate preferences and knockout effects with overlapping windows. (a) Overlapping windows (W8/S4, one KV head). (b) DeepSeek-style kernel with vector gates (W8/S4, one KV head). Each phase receives two of the eight window offsets, and the curves add their scores, rescaled so that a uniform gate still equals one. In (b), K and V share one gate, so the purple curve is the shared gate read one token later, and vector-gate scores are averaged over channels. No hollow markers appear becauseW ̸=S. Encoding, protocol, gate inputs, and checkpoint step follow Figure 16. 60 H Redirecting Gate Preferences In Section 4.2, we found a few gates with sharp offset preferences that persist across windows and inputs, and knocking out three of the corresponding heads in the reference model costs the most accuracy near the phases their gates favor. If such preferences help decide which phases are weak, editing them after training may move the weak phases with them. We test this by cycling each gate’s preference by a fixed number of offsets and checking whether the accuracy pattern across phases shifts by as many phases. H.1 Gate Cycling In models with non-overlapping windows (W=S), window offsetiis also phasei. A gate’s preference for offseticomes from its gate mapZ B i and positional gate biasb B i (Section 3.1). Cycling byδmoves both forward by δ offsets, while the payload maps stay fixed: e Z B i = Z B (i−δ) mod W , e b B i = b B (i−δ) mod W , B ∈K,V.(118) The gate parameters of offsetiare thus applied at offset (i+δ)mod W. We move maps and biases together, not the biases alone, because both enter the gate logits. The hidden states stay at their original offsets, so this does not move the realized gate scores exactly, and whether the preferences move with the parameters is an empirical question. If they do, and retrieval follows the gates, the accuracy pattern should shift byδphases, A δ (r)≈ A 0 (r− δ) mod S ,(119) whereA 0 (r) andA δ (r) are the accuracies at source-key phaserbefore and after cycling. The prediction concerns associations within one window. Each value immediately follows its key, so the two share a window except at the boundary phase r = S− 1, which we exclude from R 2 as specified for each experiment. In each experiment, we cycle the key and value gates of every head in layers 3 to 23 (zero-based) at step 47,518 and measure full-vocabulary top-1 accuracy on 1,000 logical examples of the retrieval task in Appendix C.2 with Prefix Padding, folding the 24 source-key residues moduloS.R 2 , the squared Pearson correlation between predicted and observed accuracy changes across phases, measures linear agreement rather than agreement with the identity line, which the figures draw. Intervals are bootstrapped over logical examples. H.2 One-Offset Cycles in a W12/S12 Model We first cycle a separate W12/S12 model with four KV heads per layer by one offset in each direction. Away from the boundary phase, both cycles shift the accuracy pattern approximately as predicted (Figure 19), with R 2 of 0.984 forδ= +1 and 0.995 forδ=−1 over comparisons in which phase 11 is neither the observed nor the source phase. Intervals use 2,000 bootstrap resamples. This supports a causal contribution of gate 01234567891011 Source-key's Position Index Modulo 12 0 25 50 75 100 Retrieval accuracy (%) (a) Gate cycling (δ= +1) 01234567891011 Source-key's Position Index Modulo 12 (b) Gate cycling (δ= −1) −50−2502550 Predicted Acc. Change (p) −50 −25 0 25 50 Observed Acc. Change (p) (c) Predicted vs. observed changes Phase 11 Non-boundary comparisons δ= +1: R 2 =0.984 δ= −1: R 2 =0.995 Baseline (W=12, S=12)Predicted translationCycled +1Cycled −1Boundary cases Figure 19 Gate cycling approximately shifts retrieval accuracy. (a,b) Accuracy by source-key phase in the W12/S12 model after cycling byδ= +1 andδ=−1 (markers), with the uncycled model (gray) and the prediction (dashed). (c) Predicted versus observed changes and the identity line (dashed). Shaded and hollow points involve the boundary phase and are excluded from R 2 . 61 preferences to within-window retrieval in this model. Phase 11 departs from the prediction under both cycles, which limits a purely within-window account but does not by itself identify a separate retrieval circuit. H.3 Replication across Models and Cycle Sizes One model and one-offset cycles leave open whether the shift is specific to this configuration. We therefore repeat the intervention on five models: the four-KV-head W12/S12 model above, a W12/S12 model with one KV head, the W8/S8 reference model of Section 4.2 (seed 42), and two of its seed replications (seeds 43 and 44), each W8/S8 model with one KV head. We apply each cycleδ ∈±1,±2,±3to the original parameters. Unlike in the one-offset experiment,R 2 here excludes only the observed phaseS−1, and we also report it over all phases. Intervals again use 2,000 bootstrap resamples. For the four-KV-head W12/S12 model, the cycles δ =±1 reproduce the accuracies of Figure 19. Away from the boundary phase, the observed accuracy patterns largely follow the prediction for every model and cycle (Figure 20). Excluding phaseS−1,R 2 ranges from 0.96 to 1.00 for both W12/S12 models, from 0.88 to 1.00 for the W8/S8 reference model, and from 0.66 to 1.00 for the two seed replications. Editing the gates alone thus largely shifts the weak within-window phases in all five models, most closely in the W12/S12 models. It does not restore the boundary phase, whose accuracy stays at or below 2.9% under every cycle, compared with 2.7% to 23.9% in the uncycled models, even where the prediction is high. Including this phase lowersR 2 to between 0.36 and 0.98. Cycling also lowers overall full-vocabulary accuracy by 3.3 to 18.4 percentage points, so it redirects the weak within-window phases rather than repairing the model. Beyond Figure 20, a fourth W8/S8 seed (45) behaves like the others, withR 2 excluding phaseS−1 between 0.68 and 0.99, but agreement weakens as W8/S8 models gain KV heads. The minimum over cycles is 0.75, 0.43, and 0.36 with 2, 4, and 8 KV heads, and with 8 KV heads no cycle exceeds 0.86. Because every head in layers 3 to 23 is cycled together, these results do not show whether individual heads in these models lack preferences that would shift accuracy when cycled. The layer range was carried over from the one-offset experiment rather than selected for each model. 62 0246810 0 50 100 W12/S12, 1 KV Acc. (%) δ= −3 −60060 −60 0 60 Obs. change (p) excl. last R 2 =0.96 all R 2 =0.62 0246810 δ= −2 −60060 excl. last R 2 =0.97 all R 2 =0.48 0246810 δ= −1 −60060 excl. last R 2 =0.99 all R 2 =0.83 0246810 δ= +1 −60060 excl. last R 2 =1.00 all R 2 =0.82 0246810 δ= +2 −60060 excl. last R 2 =0.99 all R 2 =0.98 0246810 δ= +3 −60060 excl. last R 2 =0.98 all R 2 =0.80 0246810 0 50 100 W12/S12, 4 KV Acc. (%) −60060 −60 0 60 Obs. change (p) excl. last R 2 =0.99 all R 2 =0.78 0246810 −60060 excl. last R 2 =0.98 all R 2 =0.47 0246810 −60060 excl. last R 2 =0.99 all R 2 =0.71 0246810 −60060 excl. last R 2 =0.98 all R 2 =0.77 0246810 −60060 excl. last R 2 =0.98 all R 2 =0.57 0246810 −60060 excl. last R 2 =0.96 all R 2 =0.36 01234567 0 50 100 W8/S8, seed 42 Acc. (%) −60060 −60 0 60 Obs. change (p) excl. last R 2 =0.88 all R 2 =0.61 01234567 −60060 excl. last R 2 =0.95 all R 2 =0.42 01234567 −60060 excl. last R 2 =1.00 all R 2 =0.68 01234567 −60060 excl. last R 2 =1.00 all R 2 =0.70 01234567 −60060 excl. last R 2 =0.95 all R 2 =0.91 01234567 −60060 excl. last R 2 =1.00 all R 2 =0.87 01234567 0 50 100 W8/S8, seed 43 Acc. (%) −40040 −40 0 40 Obs. change (p) excl. last R 2 =0.92 all R 2 =0.80 01234567 −40040 excl. last R 2 =0.89 all R 2 =0.79 01234567 −40040 excl. last R 2 =0.93 all R 2 =0.53 01234567 −40040 excl. last R 2 =0.99 all R 2 =0.92 01234567 −40040 excl. last R 2 =0.97 all R 2 =0.92 01234567 −40040 excl. last R 2 =0.80 all R 2 =0.72 01234567 0 50 100 W8/S8, seed 44 Acc. (%) −60060 −60 0 60 Obs. change (p) excl. last R 2 =0.66 all R 2 =0.46 01234567 −60060 excl. last R 2 =0.95 all R 2 =0.79 01234567 −60060 excl. last R 2 =0.90 all R 2 =0.74 01234567 −60060 excl. last R 2 =1.00 all R 2 =0.65 01234567 −60060 excl. last R 2 =0.93 all R 2 =0.63 01234567 −60060 excl. last R 2 =0.94 all R 2 =0.89 Baseline profilePredicted translation / y = xCycled δ>0Cycled δ<0Last position (excluded) Figure 20 Gate cycling shifts retrieval accuracy across models and cycle sizes. Each cell follows Figure 19 for one model and one cycleδ, with accuracy by source-key phase above and predicted versus observed accuracy changes below. Rows show the two W12/S12 models, the W8/S8 reference model (seed 42), and two of its seed replications (seeds 43 and 44), each W8/S8 model with one KV head. Columns showδfrom−3 to +3, and arrows give its direction and size. Error bars are 95% bootstrap intervals. Each cell reportsR 2 without the boundary phaseS−1 (“excl. last”) and over all phases (“all”). Unlike in Figure 19, “excl. last” excludes only comparisons whose observed phase isS−1. 63 I Extended Related Work Learned summary memories. Compressive Transformer down-samples old activations into a FIFO compressed memory using stride-matched pooling or convolution [28]. Gist tokens and Landmark Attention use special tokens to condense prompts or route random access to blocks [21,22]. LoMA adds an explicit reconstruction objective [36]. Activation Beacon inserts regularly spaced learned beacon tokens whose KVs replace raw activations, and MELODI builds a recurrent short-term memory across multiple layers and a long-term memory in a single middle layer [7,44]. CAT is particularly close at the architectural level: later chunks attend to compressed prior-chunk vectors, and its appendix reports high loss at the first token of each new chunk and recall degradation as chunks grow [25]. None of these works examines retrieval as a function of source phase within compressed memory. Cross-chunk failure modes. Deng et al.[13]systematically compare recurrent, coarse-KV, and fine-grained-KV gist compression. Their reconstruction probe shows a severe information bottleneck as compression rises, and their “lost by the boundary” diagnostic finds a segment-scale sawtooth: with 2K segments, generation is weakest immediately after a segment boundary, when predictions depend most strongly on compressed prior segments, and improves as raw tokens accumulate in the current segment. They also document semantically surprising facts being dropped and long exact strings being lost during generation. KSA’s chunk-aligned sliding window addresses a separate case in which a partially visible chunk is neither locally complete nor yet represented by a summary [9]. HMT uses the phrase “periodical recall pattern” for successful tracking of an artificially periodic topic-switch sequence, not an architecture-induced failure [15]. Among these works, the sawtooth of Deng et al.[13]is the direct precedent for recurring cross-chunk degradation, but its phase variable is thequery/generationpositionwithin the current 2K segment. Our phase variable is thesource position modulo the much smaller compression stride inside already-compressed memory. Sparse and hybrid routing. Sparse Transformer explicitly defines a strided head with modulo connectivity and notes that the pattern may route poorly on nonperiodic text [8]. It is thus a precedent for residue-class visibility as a general attention concept. NSA forms overlapping compressed block representations for coarse selection and retains an exact fine-grained branch [41]. MoBA routes queries to selected raw-token blocks [20], and the Sparse Frontier compares the quality–efficiency trade space across such sparse patterns [23]. DeepSeek-V4 selects sparsely among compressed KV entries and adds an uncompressed local window [11]. In hybrids with an exact branch over distant tokens, such as NSA, that branch could mask a failure of the compressed branch alone. DeepSeek-V4 keeps exact tokens only in its local window, and in Section 2, we find phase sensitivity in the full models. Cache allocation and specialized retrieval components. H2O and SnapKV evict or cluster tokens using estimated importance [18,45]. SqueezeAttention allocates different cache budgets across layers [37], while RazorAttention and DuoAttention retain long-range caches for retrieval heads and use compressed or streaming policies elsewhere [31,39]. Wu et al.[38]causally identify a small subset of heads that dominate long-context factual retrieval. Generic heterogeneity across heads and layers is therefore already established. Our empirical claim is more specific: KV-head knockouts in compressed models have phase-dependent effects that, in the examined heads, match the phases their gates prefer. A supplementary gate-cycling intervention on five models with non-overlapping windows complements this association (Appendix H). Retrieval diagnostics and position bias. Zoology’s multi-query associative recall isolates a major capability gap between attention and efficient sequence models [3]. The copying results of Jelassi et al.[17]and induction-circuit studies [4,24] motivate our synthetic retrieval task. Lost in the Middle establishes broad U-shaped sensitivity to absolute target position [19], and RULER shows that easy needle tests can overstate usable context length [16]. The Pitfalls of KV Cache Compression further shows that aggregate scores hide instruction- and method-specific failures [5]. We therefore match the mean and variance of the target’s absolute position across residue groups and report retrieval by phase rather than only mean accuracy. 64 Optimization and specialization. Max-margin analyses show that gradient descent can turn softmax attention into hard token selectors [32,33], although entropy concentration can also be pathological [43]. Chen et al.[6] prove staged multi-head task allocation in in-context linear regression, and Yang et al.[40]analyze population gradient flow in a symmetric two-headed classification transformer. Most closely related, Yüksel et al.[42] analyze near-symmetric heads that first compete for a dominant sparse pattern and later specialize through competition and cooperation. High-dimensional SGD likewise exhibits sequential head specialization [30], while generic softmax factorization can bias gradient flow toward low-entropy outputs [34]. We therefore do not claim the first theory of attention hardening, symmetry breaking, or division of labor. Our theory (Section 5) concerns an idealized induction model of retrieval from compressed memory. In this model, optimal compressors select sharply, gradient flow from a small initialization reaches, with high probability, a selection that persists across inputs, and nothing guarantees that heads choose distinct positions. 65