Paper deep dive
How Local Mixing Encodes Relative Position in Global NoPE Attention
Cutter Dawes, Nick Alonso, Tom Figliolia, Beren Millidge
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of explicit position encodings have been developed in transformer-based models, such as rotary position encoding (RoPE). Although explicit position encodings have long been assumed to be required, recent methods that interleave local mixing layers, such as sliding window attention (SWA) and gated linear attention, while not encoding position (NoPE) in global attention layers has recently been shown to be successful at scale. How and why this approach works is not well-understood. In this paper, we develop an explanation of how hybrid models of this sort can implicitly encode position at global NoPE layers. Supported by both theoretical and empirical evidence, our central argument is that SWA and gated linear attention induce a recency bias in the residual stream that propagates to, and is selected by, the global attention logits. Moreover, in contrast to the implicit position encodings found in models with only global NoPE attention, in which positional information arises solely from the causal mask, the recency bias in hybrid models can be maintained across long sequences. In addition to deepening our understanding of how hybrid models encode position, these findings may provide insights for how to encode position in a way that can extrapolate to longer sequence lengths indefinitely.
Tags
Links
- Source: https://arxiv.org/abs/2609.38109v1
- Canonical: https://arxiv.org/abs/2609.38109v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
77,397 characters extracted from source content.
Expand or collapse full text
How Local Mixing Encodes Relative Position in Global NoPE Attention Cutter Dawes 1,* , Nick Alonso 1,* , Tom Figliolia 1 , Beren Millidge 1 1 Zyphra Research* Equal contribution Abstract—The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of explicit position encodings have been developed in transformer-based models, such as rotary position encoding (RoPE). Although explicit position encodings have long been assumed to be required, recent methods that interleave local mixing layers, such as sliding window attention (SWA) and gated linear attention, while not encoding position (NoPE) in global attention layers has recently been shown to be successful at scale. How and why this approach works is not well-understood. In this paper, we develop an explanation of how hybrid models of this sort can implicitly encode position at global NoPE layers. Supported by both theoretical and empirical evidence, our central argument is that SWA and gated linear attention induce a recency bias in the residual stream that propagates to, and is selected by, the global attention logits. Moreover, in contrast to the implicit position encodings found in models with only global NoPE attention, in which positional information arises solely from the causal mask, the recency bias in hybrid models can be maintained across long sequences. In addition to deepening our understanding of how hybrid models encode position, these findings may provide insights for how to encode position in a way that can extrapolate to longer sequence lengths indefinitely. I. INTRODUCTION Distinguishing tokens by their position is a vital component of language modeling (Dufter et al., 2022). Since self-attention by itself is permutation invariant, attention layers in large language models (LLMs) typically utilize position encodings (PEs), which explicitly represent either the absolute or relative position of queries and keys. However, commonly used PEs such as rotary positional encoding (RoPE; Su et al., 2024) incur computational overhead and make length extension and extrapolation difficult. Removing explicit PE (a method called NoPE) has met with mixed success: NoPE does not match the in-domain performance of the same architecture with explicit PE, although it can provide some improvements in length extrapolation whereas the performance of RoPE and other explicit position embeddings tend to collapse quickly beyond their training sequence length (Haviv et al., 2022; Kazemnejad et al., 2023). Separately, to mitigate the computational and memory costs of attention across long sequences, so-called hybrid architectures that interleave either sliding window attention (SWA) or gated linear layers with global attention layers are increasingly used at scale. Interestingly, as hybrid models have become more widespread, there has been a shift in the global attention layers from using RoPE (Gemma Team, 2024) to p-RoPE (Qwen Team, 2025) to, most recently, NoPE. Notably, Kimi K3 interleaved global NoPE with a gated linear variant called KDA (Kimi Team et al., 2026). However, there are remaining questions concerning how such models utilize and represent position in their NoPE attention layers, an issue that is critical to their long context performance. To our knowledge, only one prior paper has studied how hybrid models represent position in NoPE layers, and their main finding is that they do not represent absolute position (Puvvada et al., 2025) at all. However, this result leaves open the question of whether such models may learn to encode relative position in their NoPE layers. In this paper, we present novel theoretical and empirical evidence that hybrid architectures can and do develop relative PEs at their NoPE layers. Unlike other explicit relative PEs such as RoPE, hybrid architectures implicitly encode relative position via the interactions between local or linear layers and subsequent NoPE layers. Specifically, the SWA or gated linear layers introduce a recency bias in the residual stream that propagates to, and is selected by, the global NoPE attention logits and then accentuated through depth. We support this account in two complementary ways: (i)Theoretically, we show mathematically how SWA ap- proximates a moving-average convolution in expectation, inducing a recency bias in its output that transfers into the residual stream, propagates through intervening layers, and is selected by the global NoPE attention logits. (i)Empirically, we demonstrate the existence of the pre- dicted recency bias in the residual stream and NoPE attention logits across depth. We show this bias is present at initialization and strengthens throughout training. We validate this in hybrid models interleaving global NoPE attention with SWA (with RoPE and NoPE) and KDA (Appendix E) An overview of our hypothesized mechanism is provided in Figure 1. In the following sections, we first note that existing rel- ative PEs generally induce a recency bias (Section I), then we introduce our core argument in three steps (Section I). First, SWA produces a residual stream recency bias (Section IV); second, this recency bias propagates through the intervening modules such as normalization and multi-layer perceptrons (MLPs; Section V); and third, this recency bias propagates to and is selected by the global NoPE attention logits (Section VI). arXiv:2609.38109v1 [cs.CL] 29 Sep 2026 (a) Embeddings, no positional encoding states are exchangeable Similarity (b) Local layer mixes neighbours SWA or gated linear layer Similarity (c) NoPE attention inherits the band q i ⋅k j picks up a b(i − j) term NoPE logits x 1 x 2 x 3 x 4 x 5 x 1 x 2 x 3 x 4 x 5 y 1 y 2 y 3 y 4 y 5 k 1 k 2 k 3 k 4 k 5 q 5 Figure 1: Proposed mechanism for implicit relative position encoding. (a) Tokens begin as initially uncorrelated. (b) Local mixing correlates nearby residual states. (c) Learned query and key projections can read out this lag-dependent structure as recency-biased global NoPE logits. The shading intensity denotes correlation or logit strength, while line width denotes mixing or attention weight. I. BACKGROUND: RECENCY BIAS AS RELATIVE POSITION ENCODING Relative PEs, as opposed to absolute PEs, are the industry standard in LLM architectures (Shaw et al., 2018; Dufter et al., 2022; Su et al., 2024). These PEs alter attention logits, directly or indirectly, in a way that depends on the relative distance between query and key. There are a variety of methods for encoding relative positions in attention, but the majority share the following feature: Common and state-of-the-art relative position encoding methods encode relative position by creating a recency bias in attention logits. That is, given a queryq i at position i, and a keyk j atj, relative position encodings bias the attention logits downward as the relative distancei− j increases. Relative PEs encoding a recency bias is not strictly necessary (Chen et al., 2025). In principle, a relative PE applied to attention can induce any kind of bias in attention logits, as long as this bias depends directly on relative position. However, using a recency bias to encode relative position appears to work well in practice, because: (i) it provides a particularly simple positional signal for the model to learn and use (i.e., in which distance is represented directly by a smooth, monotonic change in the attention strength); and (i) it fits the statistics of natural sequential data (i.e., the tokens most important for predicting next tokens tend to be the most recent ones; Tan et al., 2025; Wu et al., 2025; Kim et al., 2026). Survey of relative PEs.There are generally two ways to create a relative PE in attention logits: additively or multiplica- tively (Zhang et al., 2026). Additive encodings directly add a bias,b i−j , to attention logits; i.e.,s = q ⊤ i k j + b i−j , wherei andjare the position of query and key andi− jthe relative position. Examples include Alibi and Fire (Press et al., 2022; Li et al., 2024), which add a fixed, data-independent bias that decays linearly with relative distance, as well as the more recent Fox PE (Lin et al., 2025) that uses a data-dependent bias. All of these explicitly induce a recency bias in attention logits, such that keys farther from the query have a larger negative bias added to their attention logits. Multiplicative PEs include a matrix,R i−j , in the query-key multiplication; i.e.,s = q ⊤ i R i−j k j , whereR i−j is unique to relative position i − j. RoPE, arguably the best-known relative PE, rotates pairs of channels using rotation matrices in both the queries and keys. It has been observed that learned attention heads with RoPE either focus on very low-frequency channels with little resulting influence on logit decay, or they focus on high- frequency channels in such a way that they create a strong recency bias (Barbero et al., 2025). Other multiplicative PEs, like the recent Wall attention (Yang et al., 2025), create a similar kind of recency bias in a way that is data-dependent (Yang et al., 2025; Zhang et al., 2026). 2 PE in NoPE.Among previous work studying the position encoding resulting from NoPE, two papers are particularly relevant to this work. First, Puvvada et al. (2025) studied implicit position encodings in models with global NoPE attention, including hybrid architectures that interleave global NoPE with SWA. In particular, they trained probes to predict absolute position encoding in the residual stream embeddings, finding that the probes could predict absolute position in global NoPE architectures but not in hybrid architectures (see Haviv et al., 2022; Chi et al., 2023 for more evidence of absolute position encodings in global NoPE architectures). This provides evidence that hybrid NoPE architectures, unlike their global NoPE counterparts, do not encode absolute position, but leaves open the possibility that hybrid architectures could encode relative positions. Second, Zuo et al. (2025) showed that architectures with global NoPE attention at every layer could learn a recency bias- based PE in their residual stream, created by the asymmetric token mixing effects of attention with a causal mask. However, their analysis and empirical results only demonstrate a recency bias effect on very short sequences (approximately 30 tokens), leaving open the question of how this bias may change with sequence length. Additionally, they do not explain the conditions under which attention can make use of this recency bias in the residual stream. In this work, we apply a similar analysis to hybrid architectures with SWA layers and compare to global NoPE models. Importantly, we show that SWA has a recency bias effect on the residual stream independent of total sequence length, whereas for initialized global NoPE residuals the recency bias loses resolution even at moderate sequence lengths. Further, unlike previous works, we explain the conditions required of attention parameters to make use of this recency bias in the residual stream, and we scale our empirical results to models with 350M+ parameters trained on natural language data. I. THESIS: HYBRID ARCHITECTURES IMPLICITLY ENCODE RELATIVE POSITION AT GLOBAL NOPE LAYERS We develop and support the following thesis about hybrid architectures with global NoPE attention: Local layers (e.g., SWA) output a sequence of vectors which, in expectation, have a recency bias (i.e., smaller cosine similarity with greater relative distance). These outputs then create a recency bias in the post-attention residual stream that is persistent across sequence lengths. This residual stream bias can (and does after training) get transferred through the intervening layers and to attention logits at NoPE layers, thereby creating an implicit relative position encoding in NoPE layers. We provide two lines of evidence that present complementary views of this effect. The theoretical analysis isolates the struc- tural effect of one type of local layer (SWA) and outlines how the resulting recency bias can propagate through intervening modules and into the global NoPE logits. However, our analysis does not fully account for the variety of computation the model is being trained for, particularly the effects of content- dependent attention and MLPs. Our empirical analysis then verifies the existence of the residual stream and logit-based recency biases in multiple models, at varying stages of training, on both random and natural language data (for random-token controls, see AppendixD-B). We also show how these effects generalize to hybrid architectures that use KDA (Appendix E) as the local layer instead of SWA. Below, we first provide theoretical and then empirical preliminaries common to each step in our proposed mechanism. A. Theory First, we formally introduce several notions of similarity in the residual stream, and what we mean by a recency bias. LetX = [x 1 ,...,x T ] ∈ R D×T be the residual stream at some layer. The most simple and intuitive measurement of similarity is cosine similarity; i.e.,c(d) = E[ x ⊤ i x i−d /∥x i ∥x i−d ∥] . However, cosine similarity captures just one slice of the full similarity structure; in particular, it does not account for cross- channel correlations that may be picked up by the global attention projections. Therefore, we also introduce the lagged cross-moment matrix,G X (d) = E[x i x ⊤ i−d ], where thex i are unit-normalized. To obtain scalar measurements of similarity, one can consider various slices of this matrix; i.e., the Frobenius inner product ⟨G X (d),M⟩ F = tr(G X (d) ⊤ M )with some matrixM. Differ- ent slices of the lagged cross-moment correspond precisely to our similarity measures of interest: cosine similarity is the isotropic component, and the NoPE logits capture alignment with the composed query-key projection. That is, Cosine: c(d) =⟨G X (d),I⟩ F , NoPE logits: ℓ(d) =⟨G Z (d),M⟩ F . Here,Iis the identity matrix,Zis the actual post-normalization input to a NoPE head at its (unnormalized) model scale, and M = W ⊤ Q W K / √ D H with head dimensionD H . Given one of the above scalar similarity profilesf (d), we compute the recency gap asf (1) − f (L), whereLis the far-distance reference lag (4096 in our experiments); then,f (L)is the similarity floor, or the underlying correlation structure in the sequence present at long distances. B. Empirics For the empirical analysis, we study hybrid architectures at both the 120M and 350M scales, varying the SWA window sizew ∈64, 128, 256, 512, 1024, 2048, 4096and comparing to a global NoPE baseline (i.e., no SWA interleave) at each scale. Unless otherwise noted, the SWA layers use RoPE and are interleaved with global NoPE at a1:1ratio. The main text shows results for the 350M models, and thew = 128 variant for single-model analyses. We train the models on the Prolong dataset (Gao et al., 2025). For evaluations periodically throughout training, we use TextbookChapters (Chevalier et al., 2024). 3 Using the measures defined in SectionIII-A, we track residual stream cosine similarity and global NoPE attention logits across training, depth, and SWA window size. For more details on the experimental setup, including training setup, statistics sampling, and head aggregation, see Appendix A. IV. STEP 1: SWA PRODUCES A RECENCY BIAS IN THE RESIDUAL STREAM A. Theory Unlike full attention, SWA restricts its attention operation to a local window of keys and values. Here, we show that this local operation gives the attention output a distance-dependent cosine similarity profile. Furthermore, because its scale is fixed by the window size rather than the sequence length (in contrast to full attention), this effect does not dilute as the total sequence length grows. For a single attention head, defineq i = W Q x i ,k i = W K x i , v i = W V x i . For a givenq i andk i−d , the attention weight at distance d is α i (d) = exp q ⊤ i k i−d / √ D H P w−1 r=0 exp q ⊤ i k i−r / √ D H ,0≤ d < w, wherewis the sliding window size andD H the head dimension. Writing o i = W O W V x i , the SWA output is y i = W O w−1 X d=0 α i (d)v i−d = w−1 X d=0 α i (d)o i−d . Hence, SWA acts akin to a data-dependent convolution. Intuition.To illustrate why such a local filter may produce a recency bias in its output, consider a simplified scenario that closely reflects SWA in an LLM at initialization, in which the projected valueso i are zero-mean, of similar magnitude, and uncorrelated across positions, while the attention weights are roughly uniform. Then we may approximate the SWA output as a moving average, y i ≈ 1 w w−1 X r=0 o i−r . In this simplified scenario, the cosine similarity of two outputs, sayy i = 1 w (o i−w+1 +· + o i )andy i−d = 1 w (o i−d−w+1 + · +o i−d ), depends on the fraction of projected values shared by their sums. The closer they are in position, the larger this shared fraction; e.g., for a window size of 64,y i shares 63 terms withy i−1 , 62 withy i−2 , and so on. Thus, the expected output cosine similarity decreases with relative distance and approaches zero once the windows no longer overlap. Length scaling and comparison with global NoPE.As noted in Section I, Zuo et al. (2025) did a similar analysis, but for global NoPE architectures; i.e., they showed that a global NoPE attention layer could induce an adjacency pattern in its outputs. However, they considered only tiny sequences (∼ 30tokens) and did not explore whether this effect scaled with sequence length. Here, the uniform-attention toy case helps explain why global NoPE cannot sustain a well-resolved recency bias even out to 1000-token sequences. Specifically, the output of a global NoPE layer will be y i ≈ 1 i + 1 i X t=0 o t . Unlike SWA, the model averages over the entire preceding sequence. Therefore, ifi = 1000, theny i andy i−10 share99% of their terms, so the uniform-attention cosine changes only slightly over fixed lags. Together, these calculations predict a persistent window-scale recency profile for SWA, but not a comparably resolved fixed-lag profile for global NoPE. To make the contrast with SWA more concrete, under the same initialization-like assumptions as above, the output’s cosine similarity with distance is: c SWA (d)≈ 1− d w + , c global (d)≈ p (i− d + 1)/(i + 1)∝ 1− O(d/i). Hence, at initialization, the strength of SWA’s recency bias decays with the window sizew, with a smaller window resulting in a stronger, steeper recency bias; conversely, global attention’s positional signal decays with the sequence length i, and therefore dilutes for longer sequences. That is, SWA’s locality scale is length-independent by construction, whereas global attention’s is not. General mean-attention case.We now describe how SWA affects the similarity structure in its output after training, where the attention profile is no longer uniform and the incoming residual stream no longer zero-mean Gaussian. Again, we approximate attention by its mean profilea d = E[α i (d)](esti- mated across some number of relevant sequences). Suppose that Xhas some incoming lagged cross-moment structureG X (d); letA = W O W V be the composed projected-value matrix and g a (k) = P s a s a s+k the mean attention autocorrelation. In this setting, the SWA output isy i ≈ A P w−1 d=0 a d x i−d , and its lagged cross-moment is G Y (d)≈ A X k g a (k)G X (d− k)A ⊤ = A(g a ∗ G X )(d)A ⊤ , if output norms are concentrated (see AppendixB-Afor the full derivation). Thus, the similarity structure in the output is the incoming structure mixed across distances by the mean attention autocorrelation, and then mixed across channels by the composed attention matrices. The structural effect of SWA is still present, asg a (k) > 0if|k| < w, and 0 otherwise. So, the window bounds the scale of the recency bias, but its effect can be amplified or dampened by the attention profile within the window. Finally, the extent to which this mixing promotes or dilutes a recency bias in the output with respect to each similarity metric depends on its alignment to the relevant projections. B. Empirics Initialization and length scaling.Empirically, we first val- idate the argument that SWA’s effect is structurally different 4 0.0 0.2 0.4 0.6 0.8 Full NoPE 1162564096 Distance d 0.0 0.2 0.4 0.6 0.8 SWA-RoPE (w= 128) 0.6 0.7 0.8 0.9 1.0 Query position (fraction) Cosine similarity Figure 2: Attention outputs at initialization. Raw cosine of the isolated first attention-branch outputs for global NoPE and SWA (w = 128; marked by dashed line), across four fixed query-position bins. The recency bias in the SWA output is stronger and remains constant with increased sequence length, whereas the recency bias of full attention weakens significantly at longer sequences. from that of global NoPE, and that only the former produces a well-resolved recency bias at long sequence lengths. Figure 2 compares the isolated first attention-branch outputs before residual addition at initialization. With increasing sequence lengths (probed via query position along an 8k-token sequence), global NoPE activations become so highly correlated that positions cannot be resolved for most distances; in contrast, SWA retains a strong distance-dependent profile across the window while flattening beyond, regardless of sequence length. Emergence with training and depth.Though the analysis above corroborates the hypothesis that SWA produces a recency bias at initialization, it is possible that this effect dissipates during training as the model focuses on learning content- dependent interactions. However, this is not the case – in fact, the recency bias strengthens during training. Figure 3 visualizes the recency profile during training at the first, middle, and last SWA layers. Across depth, the trained profiles show strengthening and subsequent attenuation relative to their peak, while remaining stronger than at initialization. Note that this effect stabilized early in training (the colorbar is logarithmic in the fraction of that scale’s total training tokens). Furthermore, the recency bias strengthens significantly with increased depth. Further analysis as well as layer-by-layer profiles are shown in Appendix B-B. Window-size dependence.The theoretical analysis in Sec- tionIV-Asuggests that smaller SWA windows will produce a narrower and steeper recency bias in the outgoing residual stream. The distance profiles (top) and layer-wise recency gaps (bottom) in Figure 4a verify this trend; smaller windows lead to stronger recency biases across depth (for full profile comparisons, see Figure 9b in Appendix B). Further, to mitigate the confound of the strong logit-based recency bias induced by RoPE within the window (which as we discussed in SectionIV-Ahas an important contribution to the recency bias), we also verify this for models with NoPE-in-SWA. Figure 4b shows the distance profiles and layer-wise recency gaps across window sizes for NoPE-in-SWA; in fact, the effect of window size is more pronounced, with smaller windows producing significantly stronger recency biases (for further NoPE-in-SWA results, please see Figure 12f in Appendix D). Given the pronounced effect of SWA window size on recency bias strength, a further question one might ask is how window size affects model performance. We find that smaller windows achieve lower loss than their larger-windowed counterparts (see Figure 5 here and Figure 12g–h in the appendix, which report the late-training validation-loss trajectories; note that larger NoPE-in-SWA windows cause loss collapse). Although preliminary, this is promising evidence that the recency bias strength is important to such hybrid models’ success. In fact, the benefit of strengthening the recency bias effect by decreasing 1162564096 Distance d 0.00 0.06 0.12 0.18 0.24 First SWA (L1) 1162564096 Distance d Middle SWA (L13) 1162564096 Distance d Last SWA (L23) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Cosine profile Figure 3: Residual cosine during training. Post-attention cosine profiles at the first, middle, and last SWA layers (w = 128), with color denoting the fraction of training completed on a logarithmic scale. This demonstrates that residual stream recency bias increases with training and depth. 5 1162564096 Distance d 0.00 0.05 0.10 Cosine profile (a) RoPE-in-SWA 1162564096 Distance d −0.05 0.00 0.10 0.20 (b) NoPE-in-SWA 04812162024 Layer 0.00 0.05 0.10 0.15 Cosine gap 04812162024 Layer 0.0 0.1 0.2 0.3 64 128 256 512 1024 2048 4096 SWA window size Figure 4: Window dependence of residual cosine. (a) RoPE-in-SWA and (b) NoPE-in-SWA, with the top panels showing the cosine gapc(d)−c(4096)at the middle SWA layer (L13), and the bottom panels showing the cosine gapc(1)−c(4096)across depth (crosses mark SWA layers and circles global NoPE layers). With or without RoPE, smaller windows result in stronger recency biases across depth. 2.94 2.97 3.00 3.03 (a) RoPE-in-SWA 18202224 Training tokens (billions) 3.0 4.5 6.0 7.5 (b) NoPE-in-SWA 64 128 256 512 1024 2048 4096 SWA window size Validation loss (nats/token) Figure 5: Validation loss across window sizes. (a) RoPE-in- SWA and (b) NoPE-in-SWA. Mean 350M DCLM token cross- entropy (nats) over the final 30% of training. With or without RoPE, smaller windows result in lower loss. window size may outweigh the cost of the associated reduction in training compute so that the performance of the model improves on shorter window sizes. Also note that, since these experiments did not match compute across window sizes, the advantage for small windows is even stronger than shown here. V. STEP 2: PRESERVATION BY INTERVENING TRANSFORMATIONS A. Theory Addition of the residual stream.Suppose there is some similarity structureG Y (d)in the SWA outputY, andG X (d) in its inputX. Denote the following residual stream by Z = X + Y; we wish to understand its similarity structure G Z (d)as a function of its components. Expanding the outer- product in G Z (d) = E[z i z ⊤ i−d ], we obtain G Z (d) = G X (d) + E[x i y ⊤ i−d ] + E[y i x ⊤ i−d ] + G Y (d) For uncorrelated white inputs, the first and second terms are 0, and the third term becomesa d AΣ, and so we obtainG Z (d) = a d AΣ + g a (d)AΣA ⊤ . In either case, the similarity structure G Y (d)is carried into the residual stream, and the resulting similarity structure depends on its interactions with the existing similarity structure G X (d). Effect of normalization.The actual normalization operation, whether this is the (numerically stable, scaled by √ D) L2 normalization for RMSNorm or first subtracting the channel mean for LayerNorm, has little effect on similarity structure. RMSNorm in fact exactly preserves the unit-normalized cross- moment structure; and, though LayerNorm additionally projects prior to L2 normalization, prior work has found that they behave very similarly in practice (Zhang & Sennrich, 2019; Gupta et al., 2026). 1 In contrast, the gain (and offset for LayerNorm; though we omit that here for simplicity) of the normalization module can 1 In particular, Gupta et al. (2026) found that the residual streams of transformers with LayerNorm are approximately orthogonal to the one-vector in the models studied, which they show to be the normal vector of the projection prior to normalization. 6 alter the similarity structure, including its floor and gap. Here, letXdenote the output of the normalization stage. Decompose X into its variation around a position-independent mean ̄x = E[x i ] , so that eachx i = ̄x+ε i whereE[ε i ] = 0. For per-channel gains D γ = diag(γ) and output Y = D γ X , we have E[y i y ⊤ i−d ] = D γ ̄x ̄x ⊤ D γ |z shared-mean contribution + D γ E[ε i ε ⊤ i−d ]D γ | z fluctuation cross-moment . WriteM = D γ ̄x ̄x ⊤ D γ for the shared-mean contribution ande M = tr(M ) = ∥D γ ̄x∥ 2 for its energy, andC(d) = D γ E[ε i ε ⊤ i−d ]D γ for the fluctuation cross-moment ande C = tr(C(0)) = E[∥D γ ε i ∥ 2 ]for its energy. Assuming approxi- mately stationary second moments and concentrated norms, the unit-normalized similarity gap and far-distance floor are G Y (d)−G Y (L)≈ C(d)− C(L) e M + e C , G Y (L)≈ M + C(L) e M + e C . How the normalization module’s gain affects the similarity gap and floor (and in turn, the cosine similarity, or another projection or magnitude of the cross-moment) depends on its effect on the shared-mean contribution and fluctuation cross- moments and their energies. The gain’s effect on each of these terms depends in turn on how the channel-pair entries are reweighted byγ c γ k , with the energies depending on the diagonal weightsγ 2 c . In practice, the broad pattern appears to be that the normalization’s gain reduces the shared-mean energy relative to the fluctuation energy; when the fluctuation profileC(d)/e C is approximately preserved, this can reduce the cosine similarity floor while amplifying the similarity gap. Effect of the MLP.For normalized inputs, a wide MLP at random initialization admits a simple kernel description of its effect on cosine similarity (Daniely et al., 2016); i.e., at initialization an MLP preserves the ordering of the recency-bias profile. Suppose the MLP branch isy = W 2 f (W 1 x), with wide random GaussianW 1 ,W 2 , and consider two inputsx,x ′ with cosine similarityp. For each hidden unit, the corresponding preactivationsU = w ⊤ xandV = w ⊤ x ′ are jointly Gaussian with correlationp. We therefore reduce the problem to asking how the nonlinearityftransforms this correlation, defining the resulting kernelk f (p); the subsequent projectionW 2 approximately preserves this kernel at large width. Expanding f in the Gaussian Hermite basis gives k f (p) = P n a 2 n p n P n a 2 n . Then = 0term contributes a constant similarity floor, the n = 1term transmits similarity linearly, and higher-order terms nonlinearly reshape the profile. Since the coefficients enter asa 2 n ,k f is nondecreasing forp ≥ 0. Thus, at random initialization, a wide MLP preserves the ordering of a nonnegative recency-bias profile, although it may change its floor, magnitude, and shape. This initialization argument does not guarantee preservation by trained MLPs. Depending on the projections and nonlin- earities learned in training, the MLP can suppress, amplify, or reorient the incoming similarity structure. We therefore assess these effects empirically in SectionV-B, for the similarity gap and floor each. B. Empirics We observed in SectionV-Athat residual stream addition, normalization, and MLPs each have a complex effect on −0.4 −0.2 0 0.2 0.4 Cosine gap LN 1 Attn.+Res. Attn. block LN 2 MLP+Res. MLP block LN 1 Attn.+Res. Attn. block LN 2 MLP+Res. MLP block −0.6 −0.3 0 0.3 0.6 Cosine fl oor SWAFull Figure 6: Module changes in cosine gap and floor. After-minus-before changes in cosine gapc(1)− c(4096)(top) and cosine floorc(4096)(bottom), averaged across SWA and full-attention blocks atw = 128. Residual-addition steps compare with the isolated branch; shaded block summaries compare the complete update with its incoming residual. Note that the effect of each module on the similarity structure is largely as predicted by our theory. 7 1162564096 Distance d 0.0 1.5 3.0 4.5 6.0 First full (L2) 1162564096 Distance d Middle full (L14) 1162564096 Distance d Last full (L24) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Mean logit profile Figure 7: Global NoPE logits during training. Head-mean profiles at the first, middle, and last global layers (w = 128), with color denoting the fraction of training completed on a logarithmic scale. This shows that, through training, the global NoPE heads learn to select the residual stream recency bias and thereby develop a corresponding recency bias in their logits. the full cross-moment similarity structure, including both its gap and floor. To empirically characterize these effects, Figure 6 measures the difference in the cosine gap and floor, before and after each module (averaged across each of its appearances in depth; the grey block summaries show the cumulative effect across each block). Proceeding through each module appearance from left to right, we generally observe the following trends: (i) normalization, including its learned affine transformation, reduces the cosine floor and increases the gap; (i) SWA provides the strongest boost to the gap (and interestingly, reduces the floor); (i) the residual addition reduces the gap and increases the floor; (iv) the MLP reduces the gap and increases the floor; and (v) global NoPE attention also increases the gap, though much less so than SWA. The cumulative effect across each block is much smaller, with full-attention and MLP blocks largely preserving the incoming gap. VI. STEP 3: GLOBAL NOPE INHERITS THE RESIDUAL STREAM RECENCY BIAS A. Theory Suppose the recency bias has been preserved until the input to global NoPE attention; the final step is its successful readout into the global NoPE logits. Letx i denote the post- normalization input to global NoPE’s query and key projections. For a fixed attention head, letM = W ⊤ Q W K / √ D H and recall G X (d) = E[x i x ⊤ i−d ]. The expected logit profile is ℓ(d) = E[x ⊤ i Mx i−d ] =⟨G X (d),M⟩ F . Thus, the logit recency gapℓ(1)− ℓ(L)is positive precisely whenMaligns positively withG X (1)−G X (L); its magnitude depends on the extent of that alignment. Notably, in contrast to cosine, this similarity profile can depend on the skew component of the full cross-moment structure. Depending on the slice orientation, the projections can preserve, distort, or reverse the residual-similarity ordering (see Appendix C for further discussion). Note also that, for independent zero-mean query and key initializations,E[M ] = 0; therefore, the recency bias does not immediately propagate to the full attention logits in expectation over initialization. However, training can alignW Q andW K , allowing their logits to inherit the residual stream recency bias, which predicts that the effect should become visible in NoPE logits as training induces favorable alignment. B. Empirics Emergence with training.To empirically verify this, Figure 7 tracks the global NoPE attention logits during training. Indeed, at initialization the distance dependence is weak; but, as training progresses, a recency-biased profile emerges across depth, con- sistent with learning the projection alignment described above. Similarly to the development of the residual stream recency bias following SWA, the recency bias in the full attention logits strengthens and stabilizes very early in training (the colorbar is logarithmic in the fraction of training completed). But, whereas the residual stream recency bias increases consistently with depth, the logit recency bias quickly strengthens then stabilizes (layer-by-layer profiles are shown in Appendix C-B). Window-size dependence.We found in Section IV that smaller windows induce a stronger residual stream recency bias. To investigate the extent to which this trend propagates into the global NoPE logits, Figure 8 summarizes the layer and head distribution of the recency gap. In comparison to the residual stream, a few distinct patterns emerge. For both RoPE-in-SWA and NoPE-in-SWA, the logit gap remains fairly stable across depth after initially strengthening (as seen above). For RoPE-in-SWA (Figure 8a), there is also little variation by window size; however, for NoPE-in-SWA (Figure 8b), the logit gaps for larger windows swing drastically positive and negative (note that the recency gap axis is symmetric-log scaled to account for the wider variation). Further analysis as well as layer-by-layer profiles are shown in Appendix C-B. These patterns suggest that the model seeks to learn a consistent logit recency bias across depth. However, if window size is too large (and therefore, the residual stream recency bias 8 −10 0 0 10 0 10 1 (a) RoPE-in-SWA 4812162024 Layer −10 3 −10 1 0 10 1 10 3 (b) NoPE-in-SWA 64 128 256 512 1024 2048 4096 SWA window size Logit gap Figure 8: Global NoPE logit gaps. RoPE-in-SWA (top) and NoPE-in-SWA (bottom), showing the head-mean gap ̄ ℓ(1)− ̄ ℓ(4096) (shading spans the full headwise range and vertical axes are symmetric-log). Though the window dependence is less clear in RoPE-in-SWA, it appears the logit recency bias collapses for large windows with NoPE-in-SWA. too weak) then the model is unable to develop a logit recency bias, resulting in performance collapse. This aligns with the spirit of our proposed mechanism: the role of local layers is to produce a residual stream recency bias available to be picked up by the global NoPE logits; within the global NoPE logits, the model learns to pick up and maintain a consistent recency bias from the residual stream across heads to serve as its position encoding in attention. VII. DISCUSSION Despite their empirical success, how hybrid models interleav- ing SWA and global NoPE attention encode position in their global NoPE layers has remained a mystery. In this paper, we develop a novel mechanistic explanation for how such models accomplish this encoding. First, SWA creates a recency bias in its output by correlating neighboring embeddings, which emerges as a fundamental and in-built inductive bias present from initialization. Second, this recency bias is preserved by the intervening transformations (viz. residual stream addition, normalization, and MLPs). Third, it is selected by the global attention projections (to varying degrees across heads) and propagated to the global attention logits. Furthermore, this proposed mechanism may be extended to hybrid models that interleave linear layers like KDA in place of SWA (Appendix E). This recency bias acts as a kind of implicit relative position encoding, creating effects similar to explicit relative position encodings. Our account of how such models encode position is particularly interesting given the special capacity of hybrid models with global NoPE attention for length extension and generalization. We hypothesize that this occurs for a few reasons. First, the window provides a finite-length local scale that does not dilute as sequence length increases, and which is invariant to the absolute sequence length. Also, the position encoding learned does not radically change at distances beyond those seen in training, as is the case with RoPE, implying that extrapolation to longer sequence lengths does not introduce any serious discontinuities. Finally, this position encoding may be uniquely flexible due to residing within the residual stream geometry, which may enable expressive and easily-learned interactions with content that may be adaptively read out by the global attention heads. Nevertheless, there are still many unknowns about how hybrid models with SWA and global NoPE attention encode position. For example, we still have not developed a complete quantitative account of how the recency bias propagates through all modules of the model, or how this affects the attenuation of the recency bias across depth and varying sequence lengths. Nor have we established a quantitative relationship between these characteristics and the extent of length generalizability. These topics are left to future work. REFERENCES Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veli ˇ ckovi ́ c. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Repre- sentations, 2025. URL https://arxiv.org/abs/2410.06205. Yuhan Chen, Ang Lv, Jian Luan, Bin Wang, and Wei Liu. HoPE: A novel positional encoding without long-term decay for enhanced context awareness and extrapolation.In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 23044–23056, 2025. doi: 10.18653/v1/2025.acl-long.1123. URL https://aclanthology.org/2025.acl-long.1123/. Alexis Chevalier, Jiayi Geng, Alexander Wettig, Howard Chen, Sebastian Mizera, Toni Annala, Max Aragon, Arturo Rodriguez Fanlo, Simon Frieder, Simon Machado, Akshara Prabhakar, Ellie Thieu, Jiachen T. Wang, Zirui Wang, Xindi Wu, Mengzhou Xia, Wenhan Xia, Jiatong Yu, Junjie Zhu, Zhiyong Ren, Sanjeev Arora, and Danqi Chen. Language models as science tutors.In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 8310–8335, 2024. URL https://proceedings.mlr.press/v235/ chevalier24a.html. Ta-Chung Chi, Ting-Han Fan, Li-Wei Chen, Alexander Rud- nicky, and Peter Ramadge. Latent positional information is in the self-attention variance of transformer language models without positional embeddings. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), p. 1183– 1193, 2023. doi: 10.18653/v1/2023.acl-short.102. URL https://aclanthology.org/2023.acl-short.102/. Amit Daniely, Roy Frostig, and Yoram Singer.To- ward deeper understanding of neural networks: The power of initialization and a dual view on expres- 9 sivity.In Advances in Neural Information Process- ing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.c/paper_files/paper/2016/ file/abea47ba24142ed16b7d8fbf2c740e0d-Paper.pdf. Philipp Dufter, Martin Schmitt, and Hinrich Sch "utze. Position information in transformers: An overview. Computational Linguistics, 48(3):733–763, 2022. doi: 10. 1162/coli_a_00445. Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660, 2025. URL https://arxiv. org/abs/2410.02660. Gemma Team. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. URL https://arxiv.org/abs/2408.00118. Akshat Gupta, Atahan Ozdemir, Caoqinwei Gong, and Gopala Anumanchipalli. Geometric interpretation of layer normaliza- tion and a comparative analysis with rmsnorm. In Findings of the Association for Computational Linguistics: EACL 2026, p. 375–407, 2026. Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information.In Findings of the Association for Computational Linguistics: EMNLP 2022, p. 1382–1390, 2022. doi: 10.18653/v1/2022.findings-emnlp.99. URL https://aclanthology.org/2022.findings-emnlp.99/. AmirhosseinKazemnejad,InkitPadhi,Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. In Advances in Neural Information Processing Systems, volume 36, p. 24892–24928, 2023.URL https://proceedings.neurips.c/paper_files/paper/2023/hash/ 4e85362c02172c0c6567ce593122d31c-Abstract-Conference. html. Junu Kim, Xiao Liu, Zhenghao Lin, Lei Ji, Yeyun Gong, and Edward Choi. LayerNorm induces recency bias in transformer decoders.In Findings of the Association for Computational Linguistics: ACL 2026, 2026.URL https://aclanthology.org/2026.findings-acl.1430/. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, et al. Kimi K3: Open frontier intelligence.arXiv preprint arXiv:2607.24653, 2026. URL https://arxiv.org/abs/2607. 24653. Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. In The Twelfth International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/ 2310.04418. Zhixuan Lin, Evgenii Nikishin, Xu Owen He, and Aaron Courville. Forgetting transformer: Softmax attention with a forget gate. In The Thirteenth International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/ 2503.02130. Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Rep- resentations, 2022. URL https://arxiv.org/abs/2108.12409. Krishna C. Puvvada, Faisal Ladhak, Santiago Akle Serrano, Cheng-Ping Hsieh, Shantanu Acharya, Somshubra Majumdar, Fei Jia, Samuel Kriman, Simeng Sun, Dima Rekesh, and Boris Ginsburg. SWAN-GPT: An efficient and scalable approach for long-context language modeling. arXiv preprint arXiv:2504.08719, 2025. URL https://arxiv.org/abs/2504. 08719. Qwen Team.Qwen3-Next-80B-A3B-Instruct model card. Hugging Face, 2025. URL https://huggingface.co/Qwen/ Qwen3-Next-80B-A3B-Instruct. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.Self- attention with relative position representations. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 464–468, 2018. doi: 10.18653/ v1/N18-2074. URL https://aclanthology.org/N18-2074/. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. doi: 10.1016/j.neucom.2023.127063. URL https:// arxiv.org/abs/2104.09864. Shawn Tan, Songlin Yang, Aaron Courville, Rameswar Panda, and Yikang Shen. Scaling stick-breaking attention: An efficient implementation and in-depth study. arXiv preprint arXiv:2410.17980, 2025. URL https://arxiv.org/abs/2410. 17980. Xinyi Wu, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the emergence of position bias in transformers. In Pro- ceedings of the 42nd International Conference on Machine Learning, 2025. URL https://arxiv.org/abs/2502.01951. Songlin Yang, Yikang Shen, Kaiyue Wen, Shawn Tan, MayankMishra,LiliangRen,RameswarPanda, and Yoon Kim.PaTH attention: Position encoding via accumulating householder transformations.In Advances in Neural Information Processing Systems, volume 38, 2025.doi: 10.52202/085713-2080.URL https://proceedings.neurips.c/paper_files/paper/2025/hash/ 59c27bf8d56d3d50c7aeaf7535dee975-Abstract-Conference. html. Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural information processing systems, 32, 2019. Yifan Zhang, Zixiang Chen, Yifeng Liu, Zhen Qin, Huizhuo Yuan, Kangping Xu, Yang Yuan, Quanquan Gu, and Andrew Chi-Chih Yao. Group representational position encoding. In The Fourteenth International Conference on Learning Rep- resentations, 2026. URL https://arxiv.org/abs/2512.07805. Chunsheng Zuo, Pavel Guerzhoy, and Michael Guerzhoy. Position information emerges in causal transformers without 10 positional encodings via similarity of nearby embeddings. In Proceedings of the 31st International Conference on Computational Linguistics, p. 9418–9430, 2025. URL https://aclanthology.org/2025.coling-main.632/. 11 Appendix Appendix A: Measurement and Experimental Details12 A-ASimilarity measures and estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 A-BModels, training, and evaluation inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 A-CAggregation and uncertainty conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 Appendix B: Supplement to Step 1: SWA Produces a Recency Bias in the Residual Stream13 B-ATheory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 B-BEmpirics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Appendix C: Supplement to Step 3: Global NoPE Inherits the Residual Stream Recency Bias13 C-ATheory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 C-BEmpirics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 Appendix D: Controls15 D-A120M-scale robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 D-BRandom-token controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 Appendix E: KDA extension21 E-ATheory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 E-BEmpirics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 APPENDIX A MEASUREMENT AND EXPERIMENTAL DETAILS A. Similarity measures and estimation We measure the cosine similarity after attention is added to the residual stream, except for isolated attention outputs at initialization and explicitly named module states. For the matrix plots, we average outer products of unit-length states before taking norms since this captures shared structure, though sampling noise can contribute. Our cosine and logit profiles use every integer lag from 1 to 4096. B. Models, training, and evaluation inputs SWA configurations.We study the 120M and 350M hybrid models described in SectionIII-B, alternating SWA with global NoPE attention. SWA windows are64, 128, 256, 512, 1024, 2048, 4096; single-model comparisons usew = 128. The SWA layers use RoPE unless stated otherwise. The two model scales were trained for 12B and 24B tokens, respectively, with a sequence length of 8192. Training and natural-language evaluation text use the GPT-2 tokenizer. The model embedding vocabulary is padded to 50,304 IDs. The figures in the main text use the 350M models and TextbookChapters unless their captions state otherwise. KDA configurations.The KDA controls interleave KDA and global NoPE layers at both model scales. The supplementary KDA figures use final-checkpoint measurements on both TextbookChapters and random-token inputs; their aggregation conventions are stated below. Evaluation support.Recency measurements use 48 sequences of length 8192. The natural-language inputs come from TextbookChapters; random-token controls sample IDs uniformly from the padded model vocabulary with a seed of 0. Queries are fixed at zero-based positions 4352–8191 for every lag1≤ d≤ 4096, giving 184320 query–key pairs per lag. Initialization plots partition this support into four bins of 960 query positions. Validation-loss plots instead use DCLM sequences of length 4096 and show raw trajectories over the final 30% of training. C. Aggregation and uncertainty conventions Global NoPE logits.In the SWA and KDA profile and gap figures, we average the actual pre-softmax logits across heads at each lag. The logit gap is ̄ ℓ(1)− ̄ ℓ(4096)from this head-mean profile. Where shown, shading spans the range of headwise gaps, not uncertainty. Module changes and matrix measurements.Figure 6 uses the final 350M RoPE-in-SWA,w = 128checkpoint. Each point is an after-minus-before change, averaged equally over the twelve SWA or twelve full-attention blocks. Residual addition steps compare with the isolated branch while block summaries compare the updated residual with its incoming residual. For Figure 10, matrix norms are taken after averaging matrices across queries and sequences separately at each state and layer. Query–key pairs are not treated as independent samples. 12 APPENDIX B SUPPLEMENT TO STEP 1: SWA PRODUCES A RECENCY BIAS IN THE RESIDUAL STREAM This appendix provides additional theoretical and empirical results that corroborate those in Section IV, using the same setting of 350M scale models using natural language inputs (TextbookChapters). A. Theory Mean-attention cross-moment derivation.As in SectionIV-A, we assume that the norms of vectors entering and leaving the attention branch are concentrated, and therefore we work directly on unit-normalized momentsG X andG Y . We assume these moments are approximately stationary away from the causal boundary. For a fixed mean attention profilea d , that is zero outside, 0≤ d < w, let y i = A P d a d x i−d . Expanding both outputs gives G Y (d) = A w−1 X r,s=0 a r a s E[x i−r x ⊤ i−d−s ]A ⊤ = A w−1 X r,s=0 a r a s G X (d + s− r)A ⊤ = A w−1 X k=−(w−1) g a (k)G X (d− k)A ⊤ , g a (k) = X s a s a s+k . Heredis the output lag andk = r− sis the difference between two filter offsets. For learned attention, takinga r = E[α i (r)] is a fixed-profile approximation that omits weight fluctuations and their dependence on the inputs. For uncorrelated white inputs,G X (d) = Σ1d = 0, and thereforeG Y (d) = g a (d)AΣA ⊤ . If we further assume attention is uniform, then the triangular profile introduced in the text is a special case of the same convolution. B. Empirics Depth-wise and window-size dependence.As noted in SectionIV-B, the residual stream recency bias accumulates across depth. To investigate this at finer resolution, Figure 9a shows the residual stream recency profile following every SWA layer at the last checkpoint of training, which demonstrates that the trend of increasing recency bias through depth is notably smooth across layers. Comparing Figure 9a to its 120M-scale companion Figure 12c in AppendixD-A, it is particularly notable how consistent this layer-wise recency bias trend is across model scales, not only in ordering, but also in magnitude. This supports that the residual stream recency bias provides a functional role regardless of scale. To supplement the comparison of window sizes in Figure 4, Figure 9b displays the full cosine profiles at the first, middle, and last SWA layers of different window sizes; Figure 5 in the main text shows the validation loss curves for the same range of window sizes. Together, these results show that varying window size has a clear and predictable effect on both cosine profiles and language modeling (though we cannot fully isolate its causal role). (a) Depth(b) Window size Relative layer depth 1162564096 Distance d 0.00 0.04 0.08 0.12 First Middle Last 1162564096 Distance d 0.00 0.04 0.08 0.12 0.16 First SWA (L1) 1162564096 Distance d Middle SWA (L13) 1162564096 Distance d Last SWA (L23) 64 128 256 512 1024 2048 4096 Cosine profileCosine profile SWA window size Figure 9: Residual cosine at 350M. Post-attention gapc(d)− c(4096): (a) every SWA layer at the finalw = 128checkpoint; (b) first, middle, and last SWA layers across windows 64–4096. APPENDIX C SUPPLEMENT TO STEP 3: GLOBAL NOPE INHERITS THE RESIDUAL STREAM RECENCY BIAS This appendix examines how the recency structure induced by SWA and shaped by the intervening transformations analyzed in Section V is read out by global NoPE. It provides a cross-moment analysis and additional measurements supporting the readout in Section VI. We also explain how cross-moment magnitude and alignment help us understand the recency bias in both the residual stream and logits. 13 A. Theory Decomposing the general similarity structure.To analyze this readout, we study the full lagged cross-moment. Central to our understanding of this general structure is that different slices of it reduce to the cosine similarity and per-head logit curves, via the Frobenius inner product. Just as a vector inner product depends on both magnitude and alignment, it is instructive to decompose the Frobenius inner product into the contributions of cross-moment magnitude and its vectorized alignment with either the isotropic component I or the composed query-key projection M = W ⊤ Q W K / √ D H . For nonzero matrices G and X of the same shape, the Frobenius inner product decomposes as ⟨G,X⟩ F =∥G∥ F ∥X∥ F cos(G,X),cos(G,X) = tr(G ⊤ X) ∥G∥ F ∥X∥ F . Hence, the matrix cosine is just the ordinary cosine between the vectorized matrices: the two norms set the scale, while their alignment determines the sign. To see how this pertains to a recency bias, letB X = G X (1) − G X (L), andB Z = G Z (1) − G Z (L)for the actual post-normalization input Z to global attention. Taking X = I or X = M gives c(1)− c(L) = √ D∥B X ∥ F cos(B X ,I), ℓ(1)− ℓ(L) =∥B Z ∥ F ∥M∥ F cos(B Z ,M ). Here∥I∥ F = √ D, and∥M∥ F is the head’s readout gain. Therefore, the recency gaps for both cosine similarity and the global NoPE logits depend on the underlying matrix lag contrast of the cross-moment, which is unsigned (i.e., a large magnitude alone does not imply a positive cosine or logit gap). Meanwhile, its vectorized cosine alignment determines the sign. Next, we empirically assess its magnitude and alignment. B. Empirics Cross-moment magnitude and readout alignment.For a broader view on the cross-moment magnitude and its alignment to both the isotropic component and per-head projections, Figure 10 measures the matrix lag contrastBof the cross-moment and its cosine similarity withIandMacross depth. Indeed, as is expected from the separate analyses of residual stream cosine and the logits, the total similarity magnitude increases through depth along with its alignment toI(which together imply the increased recency bias through depth we see in Section IV); meanwhile, the global NoPE projections maintain a positive alignment withBthat remains relatively constant, producing positive and consistent logit gaps across depth (as seen in Section VI-B). 181624 Layer 0.02 0.05 0.1 ‖ B ‖ F 181624 SWA layer 0.00 0.02 0.04 cos( B , I ) 181624 Global attention layer −0.02 0.00 0.02 0.04 cos( B , M h ) Figure 10: Cross-moment magnitude and readout alignment. Final 350M RoPE-in-SWA model (w = 128). Left:∥B X ∥ F at post-block residual states; crosses mark SWA and circles global NoPE layers. Middle:cos(B X ,I)at SWA layers. Right: cos(B Z ,M )at model-scale post-LN1 inputs to global heads; small points show eight heads and connected circles their unweighted mean. Magnitude uses a logarithmic axis; alignments use linear axes. Depth-wise and window-size dependence.For a more fine-grained view of depth-wise patterns than provided in Figure 8, Figure 11a shows the mean logit profile at every global NoPE layer. As noted in SectionVI-B, the pattern is different from that of the residual stream recency bias: after an initial rise, the logit recency bias is largely consistent across layers. Similarly, comparing the logit profiles for different window sizes, Figure 11b shows a weaker relationship between recency bias and window size than in the residual stream. The relative lack of variation in the logit profiles with depth and window size suggests that, as we discussed in SectionVI-B, the model seeks to develop a consistent logit-based recency bias to serve as its position encoding in attention. 14 (a) Depth(b) Window size Relative layer depth 1162564096 Distance d 0 2 4 6 First Middle Last 1162564096 Distance d 0.0 2.5 5.0 7.5 10.0 First full (L2) 1162564096 Distance d Middle full (L14) 1162564096 Distance d Last full (L24) 64 128 256 512 1024 2048 4096 Mean logit profileMean logit profile SWA window size Figure 11: Global NoPE logits at 350M. Head-mean gap ̄ ℓ(d)− ̄ ℓ(4096): (a) every global layer at the finalw = 128checkpoint; (b) first, middle, and last global layers across window sizes. APPENDIX D CONTROLS A. 120M-scale robustness To verify that the 350M-scale results of the main text are robust across model scales, we train 120M models with the same configurations of SWA window size and varying RoPE and NoPE within the window, on the same dataset and for a number of tokens comparable given the scale difference (see Appendix A). These 120M-scale results corroborate the main-text 350M-scale results, with the same qualitative conclusions from each panel provided in the composite of Figure 12. That is, SWA’s recency bias is present from initialization (a), strengthens with training and depth (b–c), becomes sharper with smaller windows (d–e), and propagates to and is selected by the logits (i–l). Furthermore, Figure 12f–h displays the cosine gap for NoPE-in-SWA, as well as a comparison of the validation losses for both RoPE and NoPE within the window. Of particular note is that, in both cases, loss decreases with smaller windows despite fewer training flops, indicating that smaller windows may be beneficial, perhaps due to their ability to resolve a strong recency bias. In further support of this, with NoPE-in-SWA loss collapses entirely for larger windows; this is suggestive that, without a sufficiently strong architectural mechanism to resolve a recency bias and therefore encode position, such models fail to learn language effectively. B. Random-token controls The main analyses use natural-language sequences, as these are the natural domain of the language models we train. However, language has an underlying correlation structure that may confound the evidence for our proposed architectural mechanism. Hence, repeating the complete SWA analysis on uniformly sampled tokens from the model’s vocabulary is an important control to isolate the structural contribution from natural-language statistics. The following controls use the same query positions and lags as the TextbookChapters analyses and report results for both RoPE and NoPE within SWA (Figures 13–16; each composite groups the observables for one SWA encoding and model scale). These controls also corroborate the main-text findings. Across all four composites, we observe the recency bias present at initialization and strengthening with training and depth (a–c). SWA’s position encoding has a larger impact on the effect of window size (d–e): for NoPE-in-SWA at both model scales, we observe the same pattern of smaller windows promoting stronger recency biases; however, for RoPE-in-SWA at both scales, the trends were inconclusive, likely due to random tokens altering the learned RoPE decay profile. Finally, across all four settings, the recency bias transfers to the logits particularly for smaller windows, but this transfer is less consistent than in natural language (f–i). Overall, these results are consistent with our proposed architectural mechanism functioning, but being untuned for data outside its training distribution. 15 1162564096 Distance d 0.00 0.25 0.50 0.75 1.00 Raw cosine Full NoPE 1162564096 Distance d RoPE-in-SWA (w= 128) early late Query bin 1162564096 Distance d 0.00 0.06 0.12 0.18 0.24 First SWA (L1) 1162564096 Distance d Middle SWA (L7) 1162564096 Distance d Last SWA (L11) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Cosine profile 1162564096 Distance d 0.00 0.03 0.06 0.09 0.12 First Middle Last Relative layer depth Cosine profile 1162564096 Distance d 0.00 0.07 0.14 First SWA (L1) 1162564096 Distance d Middle SWA (L7) 1162564096 Distance d Last SWA (L11) 64 128 256 512 1024 2048 4096 SWA window size Cosine profile 24681012 Layer 0.00 0.05 0.10 0.15 Cosine gap 64 128 256 512 1024 2048 4096 SWA window size 24681012 Layer 0.00 0.05 0.10 0.15 0.20 Cosine gap 64 128 256 512 1024 2048 4096 SWA window size 9101112 Training tokens (billions) 3.225 3.250 3.275 3.300 Validation loss 64 128 256 512 1024 2048 4096 SWA window size (nats) 9101112 Training tokens (billions) 4 5 6 7 Validation loss 64 128 256 512 1024 2048 4096 SWA window size (nats) 1162564096 Distance d 0.0 1.5 3.0 4.5 6.0 First full (L2) 1162564096 Distance d Middle full (L8) 1162564096 Distance d Last full (L12) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Mean logit profile 1162564096 Distance d 0.0 1.5 3.0 4.5 6.0 First Middle Last Relative layer depth Mean logit profile 1162564096 Distance d 0 2 4 6 8 First full (L2) 1162564096 Distance d Middle full (L8) 1162564096 Distance d Last full (L12) 64 128 256 512 1024 2048 4096 SWA window size Mean logit profile 24681012 Layer −10 −1 0 10 −1 10 0 10 1 Logit gap 64 128 256 512 1024 2048 4096 SWA window size (a) Initialization(b) Residual training (c) Residual depth(d) Residual windows (e) RoPE cosine gap(f) NoPE cosine gap (g) RoPE validation loss(h) NoPE validation loss (i) Logit training(j) Logit depth (k) Logit windows(l) Head-mean logit gap Figure 12: 120M TextbookChapters robustness and validation loss. (a) Attention-branch cosine at initialization. (b–f) Post- attention residual cosine during training, across depth and SWA windows, including RoPE- and NoPE-in-SWA gaps. (g–h) Validation loss over the final 30% of training on 4096-token sequences. (i–l) Global-NoPE head-mean logit profiles and gaps during training, across depth and windows. Profile gaps are relative to lag 4096; window sizes are 64–4096. Window comparisons do not isolate the contribution of recency to the loss. 16 1162564096 Distance d 0.00 0.25 0.50 0.75 1.00 Raw cosine Full NoPE 1162564096 Distance d RoPE-in-SWA (w= 128) early late Query bin 1162564096 Distance d 0.00 0.04 0.08 0.12 First SWA (L1) 1162564096 Distance d Middle SWA (L13) 1162564096 Distance d Last SWA (L23) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Cosine profile 1162564096 Distance d 0.000 0.015 0.030 0.045 First Middle Last Relative layer depth Cosine profile 1162564096 Distance d 0.00 0.05 0.10 0.15 0.20 First SWA (L1) 1162564096 Distance d Middle SWA (L13) 1162564096 Distance d Last SWA (L23) 64 128 256 512 1024 2048 4096 SWA window size Cosine profile 04812162024 Layer 0.00 0.05 0.10 0.15 0.20 Cosine gap 64 128 256 512 1024 2048 4096 SWA window size 1162564096 Distance d 0 1 2 3 First full (L2) 1162564096 Distance d Middle full (L14) 1162564096 Distance d Last full (L24) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Mean logit profile 1162564096 Distance d 0.0 0.8 1.6 2.4 First Middle Last Relative layer depth Mean logit profile 1162564096 Distance d 0.0 1.5 3.0 4.5 First full (L2) 1162564096 Distance d Middle full (L14) 1162564096 Distance d Last full (L24) 64 128 256 512 1024 2048 4096 SWA window size Mean logit profile 4812162024 Layer −10 0 −10 −1 0 10 −1 10 0 Logit gap 64 128 256 512 1024 2048 4096 SWA window size (a) Initialization (b) Residual training(c) Residual depth (d) Residual windows(e) Residual cosine gap (f) Logit training(g) Logit depth (h) Logit windows(i) Head-mean logit gap Figure 13: Random-token control: 350M, RoPE within SWA. (a) Attention-branch cosine at initialization. (b–e) Post-attention residual cosine during training, across depth and windows, including the depthwise gap. (f–i) Global-NoPE head-mean logit profiles and gaps during training, across depth and windows. Profile gaps are relative to lag 4096; window sizes are 64–4096. 17 1162564096 Distance d 0.00 0.25 0.50 0.75 1.00 Raw cosine Full NoPE 1162564096 Distance d NoPE-in-SWA (w= 128) early late Query bin 1162564096 Distance d 0.00 0.15 0.30 First SWA (L1) 1162564096 Distance d Middle SWA (L13) 1162564096 Distance d Last SWA (L23) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Cosine profile 1162564096 Distance d 0.0 0.1 0.2 0.3 First Middle Last Relative layer depth Cosine profile 1162564096 Distance d 0.0 0.1 0.2 0.3 First SWA (L1) 1162564096 Distance d Middle SWA (L13) 1162564096 Distance d Last SWA (L23) 64 128 256 512 1024 2048 4096 SWA window size Cosine profile 04812162024 Layer 0.00 0.08 0.16 0.24 0.32 Cosine gap 64 128 256 512 1024 2048 4096 SWA window size 1162564096 Distance d 0 50 100 150 200 First full (L2) 1162564096 Distance d Middle full (L14) 1162564096 Distance d Last full (L24) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Mean logit profile 1162564096 Distance d 0 25 50 75 100 First Middle Last Relative layer depth Mean logit profile 1162564096 Distance d −200 0 200 400 600 First full (L2) 1162564096 Distance d Middle full (L14) 1162564096 Distance d Last full (L24) 64 128 256 512 1024 2048 4096 SWA window size Mean logit profile 4812162024 Layer −10 3 −10 2 −10 1 −10 0 −10 −1 0 10 −1 10 0 10 1 10 2 10 3 Logit gap 64 128 256 512 1024 2048 4096 SWA window size (a) Initialization (b) Residual training(c) Residual depth (d) Residual windows(e) Residual cosine gap (f) Logit training(g) Logit depth (h) Logit windows(i) Head-mean logit gap Figure 14: Random-token control: 350M, NoPE within SWA. (a) Attention-branch cosine at initialization. (b–e) Post-attention residual cosine during training, across depth and windows, including the depthwise gap. (f–i) Global-NoPE head-mean logit profiles and gaps during training, across depth and windows. Profile gaps are relative to lag 4096; window sizes are 64–4096. 18 1162564096 Distance d 0.00 0.25 0.50 0.75 1.00 Raw cosine Full NoPE 1162564096 Distance d RoPE-in-SWA (w= 128) early late Query bin 1162564096 Distance d 0.00 0.04 0.08 0.12 First SWA (L1) 1162564096 Distance d Middle SWA (L7) 1162564096 Distance d Last SWA (L11) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Cosine profile 1162564096 Distance d 0.000 0.015 0.030 0.045 First Middle Last Relative layer depth Cosine profile 1162564096 Distance d 0.000 0.015 0.030 0.045 0.060 First SWA (L1) 1162564096 Distance d Middle SWA (L7) 1162564096 Distance d Last SWA (L11) 64 128 256 512 1024 2048 4096 SWA window size Cosine profile 24681012 Layer 0.000 0.015 0.030 0.045 0.060 Cosine gap 64 128 256 512 1024 2048 4096 SWA window size 1162564096 Distance d −0.5 0.0 0.5 1.0 1.5 First full (L2) 1162564096 Distance d Middle full (L8) 1162564096 Distance d Last full (L12) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Mean logit profile 1162564096 Distance d 0.0 0.5 1.0 1.5 First Middle Last Relative layer depth Mean logit profile 1162564096 Distance d −0.6 0.0 0.6 1.2 1.8 First full (L2) 1162564096 Distance d Middle full (L8) 1162564096 Distance d Last full (L12) 64 128 256 512 1024 2048 4096 SWA window size Mean logit profile 24681012 Layer −10 0 −10 −1 0 10 −1 10 0 Logit gap 64 128 256 512 1024 2048 4096 SWA window size (a) Initialization (b) Residual training(c) Residual depth (d) Residual windows(e) Residual cosine gap (f) Logit training(g) Logit depth (h) Logit windows(i) Head-mean logit gap Figure 15: Random-token control: 120M, RoPE within SWA. (a) Attention-branch cosine at initialization. (b–e) Post-attention residual cosine during training, across depth and windows, including the depthwise gap. (f–i) Global-NoPE head-mean logit profiles and gaps during training, across depth and windows. Profile gaps are relative to lag 4096; window sizes are 64–4096. 19 1162564096 Distance d 0.00 0.25 0.50 0.75 1.00 Raw cosine Full NoPE 1162564096 Distance d NoPE-in-SWA (w= 128) early late Query bin 1162564096 Distance d 0.00 0.04 0.08 0.12 0.16 First SWA (L1) 1162564096 Distance d Middle SWA (L7) 1162564096 Distance d Last SWA (L11) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Cosine profile 1162564096 Distance d 0.000 0.025 0.050 0.075 0.100 First Middle Last Relative layer depth Cosine profile 1162564096 Distance d 0.00 0.04 0.08 0.12 First SWA (L1) 1162564096 Distance d Middle SWA (L7) 1162564096 Distance d Last SWA (L11) 64 128 256 512 1024 2048 4096 SWA window size Cosine profile 24681012 Layer 0.00 0.04 0.08 0.12 Cosine gap 64 128 256 512 1024 2048 4096 SWA window size 1162564096 Distance d 0 60 120 180 First full (L2) 1162564096 Distance d Middle full (L8) 1162564096 Distance d Last full (L12) 10 −4 10 −3 10 −2 10 −1 1 Fraction of total training tokens Mean logit profile 1162564096 Distance d −80 −40 0 40 First Middle Last Relative layer depth Mean logit profile 1162564096 Distance d 0 150 300 First full (L2) 1162564096 Distance d Middle full (L8) 1162564096 Distance d Last full (L12) 64 128 256 512 1024 2048 4096 SWA window size Mean logit profile 24681012 Layer −10 2 −10 1 −10 0 −10 −1 0 10 −1 10 0 10 1 10 2 10 3 Logit gap 64 128 256 512 1024 2048 4096 SWA window size (a) Initialization (b) Residual training(c) Residual depth (d) Residual windows(e) Residual cosine gap (f) Logit training(g) Logit depth (h) Logit windows(i) Head-mean logit gap Figure 16: Random-token control: 120M, NoPE within SWA. (a) Attention-branch cosine at initialization. (b–e) Post-attention residual cosine during training, across depth and windows, including the depthwise gap. (f–i) Global-NoPE head-mean logit profiles and gaps during training, across depth and windows. Profile gaps are relative to lag 4096; window sizes are 64–4096. 20 APPENDIX E KDA EXTENSION A. Theory KDA updates its recurrent state as S t = (I − β t k t k ⊤ t ) Diag(α t )S t−1 + β t k t v ⊤ t . Here,α t controls how much of each state channel is retained, while the delta update revises the association with the current key. A write from positionireaches positiontthrough the intervening transition products. We now use a simplified approximation to show how this repeated mixing can produce a recency bias, following the same intuition as for SWA. Intuition.Consider a simplified scenario in which the projected valueso t are zero-mean, of similar magnitude, and uncorrelated across positions, and the effective contribution of a past value decreases by a fixed factor0 < λ < 1at each step. Averaging over the content-dependent gates, writes, and query readout, we approximate the attention output before normalization and gating as an exponential moving average, y t ≈ λy t−1 + (1− λ)o t = (1− λ) ∞ X r=0 λ r o t−r , where the infinite sum describes positions sufficiently far from the start of the sequence. As in SWA, nearby outputs are similar because they combine many of the same values; consider, for example,y t andy t−d . In particular, all values contributing to y t−d also contribute toy t , but with their weights reduced byλ d . Hence, if output norms are concentrated, their expected cosine similarity is approximately c KDA (d)≈ λ d . Thus, the same shared-values argument that gives a triangular profile for uniform SWA gives an exponential profile in this simplified KDA case. Length scaling.The retention factor plays a role analogous to the SWA window size: a smallerλproduces a narrower, steeper recency profile, while a largerλretains correlations over longer distances. Critically, like SWA, this scale is length-independent and therefore remains equally well resolved with sequence length. Correlated inputs.As in the general SWA case, the incoming similarity structure also affects the output. LetG O (d)and G Y (d)denote the lagged cross-moments of the unit-normalized projected values and outputs, respectively. Assume norms are concentrated, then for the same exponential weightsa r = (1− λ)λ r , the filter calculation then gives the approximate relation G Y (d)∝ X k∈Z g a (k)G O (d− k), g a (k) = 1− λ 1 + λ λ |k| . Thus, the incoming structure is mixed across distances by a smoothly decaying overlap profile. As in SWA, the subsequent transformations determine how this structure enters the residual stream and whether later NoPE projections read it out as larger logits for recent keys. B. Empirics We empirically test whether our proposed mechanism extends to KDA-interleaved models at different scales, when using both natural language data and random-token controls. Figures 17 and 18 below show the KDA results; each row compares both model scales, and all logit panels use the head-mean convention specified in AppendixA-C. Indeed, the same empirical pattern appears: KDA layers primarily increase residual stream recency bias across depth (a–d); and later NoPE layers learn recency-biased logits (e–h). This indicates that the general mechanism we identify is not specific to SWA, but can apply to many local sequence mixers including the variety of increasingly common linear gated attention types. 21 1162564096 Distance d 0.00 0.04 0.08 0.12 0.16 First Middle Last Relative layer depth Cosine profile 1162564096 Distance d 0.00 0.04 0.08 0.12 First Middle Last Relative layer depth Cosine profile 0510152025 Layer 0.00 0.04 0.08 0.12 0.16 Cosine gap 24681012 Layer 0.00 0.04 0.08 0.12 Cosine gap 1162564096 D is ta n c e d 0 2 4 6 8 First Middle Last Relative layer depth Mean logit profile 1162564096 D is ta n c e d 0 2 4 6 First Middle Last Relative layer depth Mean logit profile 510152025 Layer 0 2 4 6 8 Logit gap 24681012 Layer 0 2 4 6 Logit gap (a) Residual profiles | 350M(b) Residual profiles | 120M (c) Residual cosine gap | 350M(d) Residual cosine gap | 120M (e) Global NoPE logits | 350M(f) Global NoPE logits | 120M (g) Head-mean logit gap | 350M(h) Head-mean logit gap | 120M Figure 17: KDA–NoPE natural-language extension. Columns compare 350M (left) and 120M (right). (a–b) Residual cosine gap profiles across depth; (c–d) lag-1 residual gaps; (e–f) global-NoPE head-mean logit gap profiles; (g–h) lag-1 head-mean logit gaps. All gaps are relative to lag 4096. 22 1162564096 D is ta n c e d 0.00 0.01 0.02 0.03 0.04 First Middle Last Relative layer depth Cosine profile 1162564096 D is ta n c e d 0.00 0.01 0.02 0.03 First Middle Last Relative layer depth Cosine profile 0510152025 Layer 0.00 0.01 0.02 0.03 0.04 Cosine gap 24681012 Layer 0.00 0.01 0.02 0.03 Cosine gap 1162564096 D is ta n c e d 0 3 6 9 12 First Middle Last Relative layer depth Mean logit profile 1162564096 D is ta n c e d 0 1 2 3 First Middle Last Relative layer depth Mean logit profile 510152025 Layer 0 3 6 9 12 Logit gap 24681012 Layer 0 1 2 3 Logit gap (a) Residual profiles | 350M(b) Residual profiles | 120M (c) Residual cosine gap | 350M(d) Residual cosine gap | 120M (e) Global NoPE logits | 350M(f) Global NoPE logits | 120M (g) Head-mean logit gap | 350M(h) Head-mean logit gap | 120M Figure 18: KDA–NoPE random-token extension. Columns compare 350M (left) and 120M (right). (a–b) Residual cosine gap profiles across depth; (c–d) lag-1 residual gaps; (e–f) global-NoPE head-mean logit gap profiles; (g–h) lag-1 head-mean logit gaps. All gaps are relative to lag 4096. 23