Paper deep dive
Attention Sinks Induce Gradient Sinks
Yihong Chen, Quanming Yao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/12/2026, 2:11:34 AM
Summary
The paper investigates the relationship between attention sinks and massive activations in Transformer models. Through empirical and theoretical analysis of backpropagation, the authors demonstrate that attention sinks induce pronounced gradient concentration during training, termed 'gradient sinks.' They propose that massive activations are not the direct cause of attention sinks but rather an adaptive response to this localized gradient pressure, particularly in pre-norm architectures with RMSNorm. To validate this mechanism, they introduce V-scale, a modification that attenuates value-path backpropagated gradients, successfully suppressing massive activations while preserving attention sinks. This establishes gradient sinks as the key training-time mediator linking the two phenomena.
Entities (6)
Relation Signals (6)
Attention Sinks → induces → Gradient Sinks
confidence 95% · Empirically and theoretically, we show that under causal mask, attention sinks can induce pronounced gradient concentration, which we term gradient sinks.
Gradient Sinks → mediates → Massive Activations
confidence 94% · These results support the interpretation that gradient sink is a key training-time mediator linking attention sinks and massive activations.
Massive Activations → adaptiveresponseto → Gradient Sinks
confidence 93% · massive activations can be understood as an adaptive response to this localized gradient pressure during training.
V-scale → preserves → Attention Sinks
confidence 92% · In pretrained V-scale models, attention sinks are preserved whereas massive activations are suppressed.
V-scale → suppresses → Massive Activations
confidence 92% · In pretrained V-scale models, attention sinks are preserved whereas massive activations are suppressed.
Pre-norm Architecture → utilizes → RMSNorm
confidence 90% · in pre-norm architectures with RMSNorm, massive activations can be understood as an adaptive response to this localized gradient pressure during training.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Attention sinks and massive activations are recurring and closely related phenomena in Transformer models. Existing studies have largely focused on the forward pass, making it unclear whether their connection is direct or mediated by a training-time mechanism. We study this question from the perspective of backpropagation. Empirically and theoretically, we show that under causal mask, attention sinks can induce pronounced gradient concentration, which we term gradient sinks. Furthermore, in pre-norm architectures with RMSNorm, massive activations can be understood as an adaptive response to this localized gradient pressure during training. To test this hypothesis, we introduce V-scale, a modification that adjusts value-path backpropagated gradients. In pretrained V-scale models, attention sinks are preserved whereas massive activations are suppressed. These results support the interpretation that gradient sink is a key training-time mediator linking attention sinks and massive activations.
Tags
Links
- Source: https://arxiv.org/abs/2603.17771v1
- Canonical: https://arxiv.org/abs/2603.17771v1
Trouble viewing inline? Open PDF directly →
Full Text
37,556 characters extracted from source content.
Expand or collapse full text
Attention Sinks Induce Gradient Sinks Yihong Chen Department of Electronic Engineering Tsinghua University Beijing, China chenyihong@tsinghua.edu.cn &Quanming Yao Department of Electronic Engineering Tsinghua University Beijing, China qyaoaa@tsinghua.edu.cn Abstract Attention sinks and massive activations are recurring and closely related phenomena in Transformer models. Existing studies have largely focused on the forward pass, making it unclear whether their connection is direct or mediated by a training-time mechanism. We study this question from the perspective of backpropagation. Empirically and theoretically, we show that under causal mask, attention sinks can induce pronounced gradient concentration, which we term gradient sinks. Furthermore, in pre-norm architectures with RMSNorm, massive activations can be understood as an adaptive response to this localized gradient pressure during training. To test this hypothesis, we introduce V-scale, a modification that adjusts value-path backpropagated gradients. In pretrained V-scale models, attention sinks are preserved whereas massive activations are suppressed. These results support the interpretation that gradient sink is a key training-time mediator linking attention sinks and massive activations. 1 Introduction Two prominent and frequently co-occurring phenomena in Transformer-based large language models (LLMs) are attention sinks (AS), where a small number of tokens attract disproportionate attention mass Xiao et al. (2024); Gu et al. (2025), and massive activations (MA), where activations become unusually large on a small set of tokens and features Bondarenko et al. (2023); Sun et al. (2024); An et al. (2025). They are predominantly observed on the first token especially in pre-norm LLMs. These phenomena have attracted lots of research interest. A line of work treats AS as exploitable structure for long-context inference and fine-tuning Xiao et al. (2024); Su and Yuan (2025); Liu et al. (2026b, 2025); Fu et al. (2026); Liu et al. (2026a) and structural work studies how to migrate or reshape these phenomena by attention or normalization variants Henry et al. (2020); Miller (2023); Kaul et al. (2025); Zuhri et al. (2025); Qiu et al. (2025); Bu et al. (2025); Sun et al. (2026). Particularly, mechanistic studies seek to explain their functions and origins. AS is mainstreamly explained as a way to implement “no-ops” attention heads that contribute little to the current token Gu et al. (2025); Barbero et al. (2025); Guo et al. (2025), driven by the sum-to-one constraint of the Softmax operator. Meanwhile, MA is often described to help create sink formation Sun et al. (2024); Su et al. (2026); Queipo-de-Llano et al. (2026). Together, these prior studies have established that AS and MA are functionally correlated structural artifacts that play important roles in LLMs, even as they impose severe numerical outliers. Yet one question remains unresolved: Why should optimization learn such massive token activations at all? This question is especially nontrivial for modern pre-norm Transformers. In this architecture, both attention and MLP sublayers operate on normalized inputs Zhang and Sennrich (2019); Xiong et al. (2020); Touvron et al. (2023). Locally, the forward computation depends much more strongly on representation direction than on raw token norm. This makes MA appear somewhat unnecessary. Most interventions in the literature modify trained models post hoc by changing MLP outputs or token activations and then observe joint changes in AS and MA. While informative, such interventions do not constitute sufficiently clean causal evidence for their relationship, because both magnitude and direction are perturbed at the same time. Recent works that examine training dynamics Gallego-Feliciano et al. (2025); Queipo-de-Llano et al. (2026) provide valuable additional perspective, but still do not directly isolate the backward mechanism linking sink structure to activation growth. Likewise, showing that AS and MA can be decoupled by architectural changes Sun et al. (2026) does not by itself explain why they are learned to be coupled in standard architectures. This paper studies this problem from the perspective of backpropagation. Our starting point is that, under causal mask, sink tokens should not only accumulate attention mass in the forward computation, but also accumulate gradients during backpropagation. As later tokens place attention weights on the first token, their gradients aggregate there. We call this effect a gradient sink (GS). In pre-norm Transformers with RMSNorm, such localized gradient pressure can be partially offset because RMSNorm attenuates backpropagated gradients approximately in inverse proportion to activation norm. This suggests a different interpretation of massive activations: rather than being the direct mechanistic cause of attention sinks, they may emerge as an adaptive response to the gradient imbalance induced by attention sinks. We substantiate this view with complementary evidence. Empirically, we track gradient sinks during training and show that it is concentrated in attention blocks, while RMSNorm-mediated gradient compression scales almost inversely with activation norms. Theoretically, we show that under causal mask, sink structure provides a natural route to gradient aggregation. Finally, we validate the mechanism by intervening the backward path. Our proposed V-scale acts as a gradient valve that attenuates value-path backpropagation of sink tokens. Pretrained V-scale models retain strong AS yet exhibit weaker MA, which is consistent with GS serving as the training-time mediator between the two phenomena. 2 Preliminaries Here, we formalize the Transformer architecture, along with forward and backward observables. These definitions lay the groundwork for studying gradient concentration in subsequent sections. Pre-norm Transformers We study decoder-only Llama-like Transformers with pre-norm residual blocks, RMSNorm, RoPE positional encoding, and SwiGLU MLPs Touvron et al. (2023); Zhang and Sennrich (2019); Su et al. (2021); Shazeer (2020). Let Hℓ=(h1ℓ,…,hTℓ)⊤∈ℝT×dmodelH =(h_1 ,…,h_T ) ^T× d_model denote the hidden states inputting to layer ℓ . A pre-norm block updates the sequence as H~ℓ H =RMSNorm(Hℓ), =RMSNorm(H ), Rattn,ℓ R^attn, =Attention(H~ℓ), =Attention( H ), Hℓ+12 H + 12 =Hℓ+Rattn,ℓ, =H +R^attn, , H~ℓ+12 H + 12 =RMSNorm(Hℓ+12), =RMSNorm(H + 12), Rmlp,ℓ R^mlp, =MLP(H~ℓ+12), =MLP( H + 12), Hℓ+1 H +1 =Hℓ+12+Rmlp,ℓ. =H + 12+R^mlp, . The Attention block can be either multi-head attention (MHA) Vaswani et al. (2017) or a variant such as grouped-query attention (GQA) Ainslie et al. (2023). For simplicity, we fix one layer ℓ and one attention head h, and suppress the layer/head superscripts unless needed. With RoPE, the query, key, and value states at token position t are qt=RtWQh~tq_t=R_tW_Q h_t, kt=RtWKh~tk_t=R_tW_K h_t, and vt=WVh~tv_t=W_V h_t, where Rt∈ℝdhead×dheadR_t ^d_head× d_head is the RoPE rotation matrix. Then the causal attention logits and weights are computed by ztj=⟨qt,kj⟩d+mtjz_tj= q_t,k_j d+m_tj and atj=softmax(zt,:)ja_tj=softmax(z_t,:)_j, where mtj=0m_tj=0 for j≤tj≤ t otherwise −∞-∞. Finally we have the value aggregation yt=∑j≤tatjvjy_t=Σ _j≤ ta_tjv_j and head output ot=WOyto_t=W_Oy_t. Forward metrics for attention sinks and activations For a candidate sink position s (typically the first token), we follow Gu et al. (2025); Queipo-de-Llano et al. (2026) and use thresholded criteria to count sink-like behavior: Sinksϵ,ℓ=1H∑h=1H(αsℓ,h>ϵ),αsℓ,h=1Tϵ∑t=s+Tϵ−1atsℓ,h,Sink_s^ε, = 1HΣ _h=1^HI( _s ,h>ε), _s ,h= 1T_εΣ _t=s^s+T_ε-1a_ts ,h, (1) where H is the number of attention heads in each layer. Following prior work Gu et al. (2025) we set the threshold ϵ=0.3ε=0.3 and Tϵ=64T_ε=64. In this paper, for theory and training dynamics measurements, we also introduce the following simpler column statistics: M¯sℓ,h=Msℓ,hT−s+1=1T−s+1∑t=sTatsℓ,h,S¯sℓ,h=Ssℓ,hT−s+1=1T−s+1∑t=sT(atsℓ,h)2.\!\! M_s ,h\!=\! M_s ,hT-s+1\!=\! 1T-s+1Σ _t=s^Ta_ts ,h, S_s ,h\!=\! S_s ,hT-s+1\!=\! 1T-s+1Σ _t=s^T(a_ts ,h)^2. (2) The quantity M¯s M_s is the average attention mass routed through token s by all causally allowed future queries and S¯s S_s is a second-moment version. In experiments, we focus on the first token (s=1s=1) and further summarize (2) across heads and layers. For activations, many prior studies identify MA as single activation magnitudes that are large where both token and feature axes are considered Bondarenko et al. (2023); Sun et al. (2024). In this work, we concern token-wise behaviors, so we identify MA as large token-wise activation norms at an explicitly chosen computational site Guo et al. (2025), including the input of Transformer layers ‖ht,inℓ‖=‖htℓ‖2\|h_t,in \|=\|h_t \|_2, the output of Transformer layers ‖ht,outℓ‖=‖htℓ+1‖2\|h_t,out \|=\|h_t +1\|_2, and the hidden states after Attention ‖ht,halfℓ‖=‖htℓ+1/2‖2\|h_t,half \|=\|h_t +1/2\|_2. We also measure the output of the MLP and the value state. Backward observables We introduce token-wise gradient observables and local gradient-reshaping ratios that expose where gradient pressure is created and how it is subsequently transformed inside pre-norm residual branches. Let ℒL denote the training loss. Recalling that htℓ+1/2=htℓ+rtattn,ℓh_t +1/2=h_t +r_t^attn, for Attention blocks, we have ∇rtattn,ℓℒ=∇htℓ+1/2ℒ _r_t^attn, L= _h_t +1/2L. Moreover, the difference ∇htℓℒ−∇htℓ+1/2ℒ _h_t L- _h_t +1/2L isolates the additional gradient contribution propagated through the attention branch, excluding the identity skip path. We then define three ratios by Bloattattn,ℓ _t^attn, :=‖∇h~tℓℒ‖2/‖∇rtattn,ℓℒ‖2, =\| _ h_t L\|_2\,/\,\| _r_t^attn, L\|_2, (3) Compresstattn,ℓ _t^attn, :=‖∇htℓℒ−∇htℓ+1/2ℒ‖2/‖∇h~tℓℒ‖2, =\| _h_t L- _h_t +1/2L\|_2\,/\,\| _ h_t L\|_2, Changetattn,ℓ _t^attn, :=‖∇htℓℒ‖2/‖∇htℓ+1/2ℒ‖2 =\| _h_t L\|_2\,/\,\| _h_t +1/2L\|_2 Analogously, we define ratios Bloattmlp,ℓBloat_t^mlp, , Compresstmlp,ℓCompress_t^mlp, , and Changetmlp,ℓChange_t^mlp, in the same way for MLP branch. With these definitions in place, we are ready to examine the training dynamics induced by attention structure. Section 3 shows that attention sinks lead to concentrated gradients, forming gradient sinks. In Section 4, we analyze how such localized gradient pressure is reshaped by the network and associated with massive activations. Finally, Section 5 validates this mechanism through targeted intervention on the gradient flow. 3 From attention sink to gradient sink We show both empirically and theoretically that AS induce gradient concentration, forming gradient sinks. This establishes the key link between forward attention patterns and backward training signals. 3.1 Empirical observations We train LlamaForCausalLM models from scratch and analyze checkpoints throughout training. The models are pretrained on C4 dataset Raffel et al. (2020) using AdamW optimizer. We consider two model sizes, approximately 0.1B and 0.3B parameters. We save checkpoints every 1000 steps during training. At each checkpoint, we run one forward-backward pass using the same causal language modeling objective as in training and collect token-wise statistics. Unless otherwise stated, these statistics are averaged over the full evaluation batches, and the AS metrics are additionally averaged over heads. Figure 1: Forward phenomena in baseline models. Top row: 0.1B model; bottom row: 0.3B model. From left to right: attention sink mass, thresholded sink rate, residual-stream output norm, and MLP output norm. We compare the first token (token 0) with the mean and maximum over the rest early tokens (positions 1–15). The results confirm the co-occurrence of AS and MA. Figure 1 summarizes the forward patterns. In both model sizes, the sink token separates sharply from the other early tokens on both Sink Mass M¯sℓ M_s in (2) and Sinksϵ,ℓSink_s^ε, in (1). On the other hand, the residual outputs are broadly elevated on the first token, and the strongest amplification is concentrated in a small number of MLP blocks. These results confirm that AS and MA already co-occur clearly in small pre-norm Transformers. We now turn to the backward pass and collect the gradients of qtq_t, ktk_t, and vtv_t in attention blocks for every token position t. For each pathway, we first aggregate gradients over the full batch exactly as in one optimization step, and then take the l2l_2 norm for each token. This makes the resulting token-wise gradient curves directly comparable to the effective training signal seen by the optimizer. The statistics are averaged over layers. Figure 2: Token-wise gradient norms of QKV across training checkpoints. Top row: 0.1B model; bottom row: 0.3B model. From left to right: gradient norms of query, key, and value as functions of token position, averaged over layers. In both scales, key and especially value gradients exhibit a pronounced spike at token 0. Figure 2 shows a clear and highly structured asymmetry. On the key and value pathways, the gradient norm exhibits a pronounced spike at the first token across checkpoints, after which it drops rapidly and remains relatively flat over most of the sequence. Moreover, the value-path spikes are observed to be substantially larger than the key-side ones. The query pathway looks qualitatively different and its token-wise gradient profile is comparatively flat across most positions. In fact, the query gradient at the first token is exactly zero under causal mask. Taken together, these observations establish the central empirical fact of this section: in pretrained baseline models, the sink token is not only a forward attention sink but also a backward gradient sink. 3.2 Theoretical results We next show that the link of AS and GS is not just an empirical regularity. This subsection focuses exclusively on the most central theoretical framework. Let gt:=∇ytℒ=WO⊤∇otℒg_t:= _y_tL=W_O _o_tL denote the upstream gradient. Recall that MsM_s and SsS_s in (2) denote the attention column mass and its second moment, respectively. Then by direct computation we have the exact V-side gradients ∇vsℒ=∑t=1Tatsgt _v_sL=Σ _t=1^Ta_tsg_t. This identity exposes that the gradient on vsv_s is an attention-weighted sum of all upstream gradients whose outputs read value from token s. If many later tokens attend to token s, then many later gradients also pass through vsv_s. A convenient quantitative version follows from a mean-plus-noise decomposition gt=μ+εtg_t=μ+ _t for stochastic gradients Mandt et al. (2017); McCandlish et al. (2018). Theorem 1 (V-side gradient control by sink statistics). Assume gt=μ+εtg_t=μ+ _t, where [εt]=0E[ _t]=0, Tr(Cov(εt))≤σ2Tr(Cov( _t))≤σ^2 for all t, and |Tr(Cov(εt,εt′))|≤ρ|Tr(Cov( _t, _t ))|≤ρ for all t≠t′t≠ t . Then we have Ms2‖μ‖22≤[‖∇vsℒ‖22]≤Ms2‖μ‖22+σ2Ss+ρ(Ms2−Ss).M_s^2\|μ\|_2^2 [\| _v_sL\|_2^2 ]≤ M_s^2\|μ\|_2^2+σ^2S_s+ρ(M_s^2-S_s). Hence stronger sink columns imply systematically larger value-path gradient concentration. Theorem 1 shows that sink structure provides a natural and quantitatively explicit route to value-path gradient concentration. The same derivation also yields exact query/key-path gradients. The crucial asymmetry is that K-side gradients aggregate column-wise just as V-side ones, while Q-side gradients are row-local. These match our measurements that the sink token shows strong value/key gradient concentration but no special query-path advantage. 4 Gradient reshaping by massive activations This section examines how the localized gradient pressure is reshaped and whether the observed MA pattern aligns with a concrete gradient-compression mechanism in pre-norm architectures. For each layer of a checkpoint, we compare the activation norms of the attention branch input ‖hinℓ‖\|h_in \|, against three ratios defined in (3): Bloatattn,ℓBloat^attn, , Compressattn,ℓCompress^attn, , and Changeattn,ℓChange^attn, . Figure 3: Scatter plots relating gradient reshaping to input activation norms of Attention block. Top row: 0.1B model; bottom row: 0.3B model. From left to right: log10Bloatattn _10Bloat^attn vs. log10‖hin‖ _10\|h_in\|, log10Compressattn _10Compress^attn vs. log10‖hin‖ _10\|h_in\|, and log10Changeattn _10Change^attn vs. log10‖hin‖ _10\|h_in\|. Each point is colored by token group (token 0, token 1, tokens 2–3, tokens 4–7, and the rest early tokens 8–15). Large-activation points of token 0 occupy the high-bloat regime. Compression exhibits an approximately linear inverse relationship with activation scale. The net residual-level change remains concentrated near log101=0 _101=0 except for a small set of first-layer points. Figure 3 shows that these three ratios play sharply different roles. The scatter plots are colored by token groups. In the left panel, large-activation points concentrate in the high-bloat regime dominated by token 0, while the remaining tokens occupy a much milder regime. This means that the gradient norms of the sink token are not merely large in an absolute sense but are most strongly amplified in the attention branch. The middle panel shows that CompressattnCompress^attn lies on an almost linear trend with slope −1-1, indicating that RMSNorm-mediated attenuation is closely aligned with activation scale. The right panel explains why this alignment matters. Although BloatattnBloat^attn can become very large on the first token, the corresponding ChangeattnChange^attn values remain concentrated near log101=0 _101=0, except for some points located in the lower-right corner from the first layer. Thus the localized gradient pressure revealed by BloatattnBloat^attn is largely reshaped into a regime compatible with stable residual propagation. This observation is consistent with the broader literature, in which residual networks are thought to remain optimizable at depth since the signal propagation stays appropriately close to identity, rather than drifting into systematic explosion or collapse He et al. (2016); Zhang et al. (2019); Liu et al. (2020); Bachlechner et al. (2021); Wang et al. (2024). The empirical alignment between massive activations and strong compression is not accidental. It follows from a basic property of RMSNorm. Theorem 2 (Activation-dependent compression under RMSNorm). Denote y=RMSNorm(x)=γ⊙xrms(x)y=RMSNorm(x)=γ xrms(x) with rms(x)=1d‖x‖22+ϵrms(x)= 1d\|x\|_2^2+ε, and let gy=∇yℒg_y= _yL be the upstream gradient at the RMSNorm output. Then we have ∇xℒ=Jrms(x)⊤gy _xL=J_rms(x) g_y, and the Jacobian satisfies ‖Jrms(x)‖op≤‖γ‖∞rms(x)\|J_rms(x)\|_op≤ \|γ\|_∞rms(x). The theorem formalizes a key intuition that larger incoming activation norm provides a direct route to stronger local attenuation of backpropagated gradients. In total, we observe that localized gradients are not merely large but selectively processed by pre-norm computation. In the attention branch, the sink token exhibits strong branch-local amplification, strong normalization-induced attenuation, and only limited net gradient change. These results support a mechanism that massive activations attenuate localized gradient pressure through RMSNorm. 5 V-scale: a value-path gradient valve The previous sections suggest a concrete and testable prediction of our mechanism. If massive activations are primarily a response to localized gradient pressure, then reducing that pressure should weaken MA even when attention sinks are largely preserved. This section implements this prediction. Several recent architectural variants mitigate extreme phenomena like MA and AS by changing the Softmax function, inserting gates, or modifying normalization structure Zuhri et al. (2025); Qiu et al. (2025); Bu et al. (2025); Sun et al. (2026). While valuable, these changes strongly alter the forward geometry and therefore do not cleanly test the specific prediction of our hypothesis. Here we instead seek a mathematically transparent modification remaining close to the original Llama-like model. The value path is the natural target channel for two reasons. First, our backward observations in Section 3 confirm that value-path gradient norms of sink token are several times larger than key-path ones. Second, prior studies have repeatedly reported that sink tokens tend to have unusually small value norms Guo et al. (2025); Su and Yuan (2025); Bu et al. (2025). We observe the same pattern and are inspired by this useful structural prior: if sink values are already small, then a radial transform depending on ‖v‖2\|v\|_2 can target sink tokens selectively. 5.1 Definition and design intuition For each attention head, after the standard value projection and before value aggregation, we replace yt=∑j≤tatjvjy_t=Σ _j≤ ta_tjv_j by v^j=ϕ(‖vj‖22)vj,ϕ(r)=r+C,y^t=∑j≤tatjv^j, v_j=φ(\|v_j\|_2^2)\,v_j, φ(r)= rr+C, y_t=Σ _j≤ ta_tj v_j, where C>0C>0 is a scale parameter. We call this modification V-scale, which is illustrated in Figure 4. This map is chosen to satisfy three design desiderata. First, it does not affect the attention weights that depend only on Q and K. Second, it is close to identity when ‖vj‖22≫C\|v_j\|_2^2 C, because ϕ(r)→1φ(r)→ 1 as r→∞r→∞. Third, it strongly suppresses small-norm values, since ϕ(r)≈r/Cφ(r)≈ r/C when r≪Cr C. This makes V-scale a targeted modification rather than a uniform shrinkage. Moreover, because those small-norm tokens already contribute little through the forward value sum, the induced forward perturbation is limited exactly in the regime where the gradient intervention is strongest. Figure 4: Schematic of V-scale inside a pre-norm Transformer block. In practice we use a scaled reparameterization Cℓ,h=(dheadσ)2λℓ,hC_ ,h=(d_headσ)^2λ ,h, where σ=0.02σ=0.02 is the standard deviation employed for initializing the weight matrix in LlamaForCausalLM baselines and λℓ,hλ ,h is either fixed to 11 or learned as λℓ,h=exp(θℓ,h)λ ,h= ( _ ,h). This learnable version introduces only one parameter per layer and per head, which is negligible relative to the total parameter count. We implement this learnable version in this work. As expected, the backward behavior of V-scale is analyzable. Let r=‖v‖22r=\|v\|_2^2 and v^=ϕ(r)v v=φ(r)v with ϕ(r)=r/(r+C)φ(r)=r/(r+C). Then the Jacobian of the V-scale map v↦v^v v is Jϕ(v)=ϕ(r)I+2C(r+C)2vv⊤J_φ(v)=φ(r)I+ 2C(r+C)^2v , which has eigenvalue λ⟂(r)=r+C _ (r)= rr+C on the (dhead−1)(d_head-1)-dimensional subspace orthogonal to v, and eigenvalue λ∥(r)=r2+3Cr(r+C)2 _ (r)= r^2+3Cr(r+C)^2 along the radial direction v. Thus if ‖v‖22≪C\|v\|_2^2 C for the sink token, the value-path backward signal is strongly attenuated. Remark 1. The gradient arriving at the token representation after RMSNorm is not a pure value-path quantity but the sum of the Q, K, and V paths ∇h~sℓℒ=WQ⊤∇qsℓℒ+WK⊤∇ksℓℒ+WV⊤∇vsℓℒ _ h_s L=W_Q _q_s L+W_K _k_s L+W_V _v_s L, up to the fixed RoPE rotations and the sum over heads. Therefore, modifying only the value path cannot completely eliminate gradient sinks. Our goal is to show that once the dominant value-path pressure is supplied with an alternative valve, the optimizer no longer needs to build the same degree of MA. 5.2 Experiments We train V-scale models using the same data, optimizer, and model configurations as the corresponding baselines. These models can achieve slightly lower validation losses compared to the baselines. Although the practical significance may be limited for such small-scale models, this result demonstrates that our modified models remain well trainable. The main empirical comparison is given in Figure 5. We compare baseline and V-scale models on the most relevant token-wise forward observables for AS and MA: thresholded Sink Rate Sinksϵ,ℓSink_s^ε, in (1), residual-stream output norm, MLP output norm, and Attention output norm. Figure 5: Forward phenomena in baseline and V-scale models. Top row: 0.1B models; bottom row: 0.3B models. From left to right: thresholded sink rate, residual-stream output norm, MLP output norm, and Attention output norm. We compare the first token with the mean over the rest early tokens (positions 1–15). Across both scales, V-scale largely preserves sink behavior while reducing the activation norms of token 0, with the larger reduction appearing in MLP outputs. The results are consistent across both model scales. On the AS metrics, V-scale preserves strong sink behavior and in several layers even slightly strengthens it. In contrast, on the MA metrics, the residual-stream and MLP output norms of token 0 are clearly reduced relative to the baselines. The attention output norms of token 0 are also reduced, which is expected since V-scale directly contracts the value path. However, this direct attenuation is modest in magnitude compared with the reduction in the learned MLP spikes. Overall, these observations match the qualitative prediction of our mechanism: by providing an additional gradient valve on the dominant value path, V-scale reduces the pressure for the model to realize MA through large MLP outputs, rather than merely shrinking the attention output. 6 Conclusions In this work, we revisit the relationship between attention sinks and massive activations from the perspective of backpropagation. Our empirical and theoretical results suggest that the connection between the two is not purely a forward-pass co-occurrence, but is mediated by a backward mechanism: under causal mask, attention sinks can induce concentrated gradient pressure on the sink token, forming gradient sinks, and massive activations emerge as an adaptive response to this localized pressure in pre-norm Transformers. To test this hypothesis, we introduce V-scale, a value-path intervention that selectively modulates backpropagated gradients. Across models, we find that V-scale preserves attention sinks but suppresses massive activations, illustrating that these two phenomena can be separated once gradient sinks are controlled. Taken together, our findings highlight gradient regulation as a useful lens for understanding extreme token phenomena in training Transformers. Limitations This work has several limitations. First, our study is restricted to Llama-like dense language models. It remains unclear to what extent the same mechanism transfers to other settings, such as mixture-of-experts architectures or multimodal Transformers. Second, our analysis is primarily a mechanism study of why MA can be useful once GS exists, rather than a complete account of how AS and MA are learned throughout optimization. Finally, our experiments are designed for mechanistic validation rather than broad architecture or capability benchmarking since we do not perform exhaustive scaling sweeps, alternative intervention searches, or downstream task evaluations. These limitations are left for future work. References [1] J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebron, and S. Sanghai (2023) GQA: training generalized multi-query transformer models from multi-head checkpoints. In Empirical Methods in Natural Language Processing, Cited by: §2. [2] Y. An, X. Zhao, T. Yu, M. Tang, and J. Wang (2025) Systematic outliers in large language models. In International Conference on Learning Representations, Cited by: §1. [3] T. Bachlechner, B. P. Majumder, H. H. Mao, G. Cottrell, and J. J. McAuley (2021) ReZero is all you need: fast convergence at large depth. In Uncertainty in Artificial Intelligence, Cited by: §4. [4] F. Barbero, A. Arroyo, X. Gu, C. Perivolaropoulos, P. Velickovic, R. Pascanu, and M. M. Bronstein (2025) Why do llms attend to the first token?. In Conference on Language Modeling, Cited by: §1. [5] Y. Bondarenko, M. Nagel, and T. Blankevoort (2023) Quantizable transformers: removing outliers by helping attention heads do nothing. In Advances in Neural Information Processing Systems, Cited by: §1, §2. [6] R. Bu, H. Zhong, W. Chen, and Y. Li (2025) Value-state gated attention for mitigating extreme-token phenomena in transformers. arXiv. Cited by: §1, §5, §5. [7] Z. Fu, W. Zeng, R. Wang, and M. Li (2026) Attention sink forges native moe in attention layers: sink-aware training to address head collapse. arXiv. Cited by: §1. [8] J. Gallego-Feliciano, S. A. McClendon, J. Morinelli, S. Zervoudakis, and A. Saravanos (2025) Hidden dynamics of massive activations in transformer training. arXiv. Cited by: §1. [9] X. Gu, T. Pang, C. Du, Q. Liu, F. Zhang, C. Du, Y. Wang, and M. Lin (2025) When attention sink emerges in language models: an empirical view. In International Conference on Learning Representations, Cited by: §1, §1, §2, §2. [10] T. Guo, D. Pai, Y. Bai, J. Jiao, M. I. Jordan, and S. Mei (2025) Active-dormant attention heads: mechanistically demystifying extreme-token phenomena in llms. In Conference on Parsimony and Learning, Cited by: §1, §2, §5. [11] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, Cited by: §4. [12] A. Henry, P. R. Dachapally, S. S. Pawar, and Y. Chen (2020) Query-key normalization for transformers. In EMNLP Findings, Cited by: §1. [13] P. Kaul, C. Ma, I. Elezi, and J. Deng (2025) From attention to activation: unraveling the enigmas of large language models. In International Conference on Learning Representations, Cited by: §1. [14] G. Liu, W. Lin, T. Huang, R. Mo, Q. Mu, X. Wang, and L. Shen (2026) Surgery: mitigating harmful fine-tuning for large language models via attention sink. arXiv. Cited by: §1. [15] L. Liu, X. Liu, J. Gao, W. Chen, and J. Han (2020) Understanding the difficulty of training transformers. In Empirical Methods in Natural Language Processing, Cited by: §4. [16] X. Liu, G. Chen, and W. Wang (2026) SinkTrack: attention sink based context anchoring for large language models. In International Conference on Learning Representations, Cited by: §1. [17] Y. Liu, J. C. Liang, H. Fan, W. Yang, Y. Cui, X. Han, L. Huang, D. Liu, Q. Wang, and C. Han (2025) All you need is one: capsule prompt tuning with a single vector. In Advances in Neural Information Processing Systems, Cited by: §1. [18] S. Mandt, M. D. Hoffman, and D. M. Blei (2017) Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research 18, p. 134:1–134:35. Cited by: §3.2. [19] S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team (2018) An empirical model of large-batch training. arXiv. Cited by: §3.2. [20] E. Miller (2023) Attention is off by one. Cited by: §1. [21] Z. Qiu, Z. Wang, B. Zheng, Z. Huang, K. Wen, S. Yang, R. Men, L. Yu, F. Huang, S. Huang, D. Liu, J. Zhou, and J. Lin (2025) Gated attention for large language models: non-linearity, sparsity, and attention-sink-free. In Advances in Neural Information Processing Systems, Cited by: §1, §5. [22] E. Queipo-de-Llano, A. Arroyo, F. Barbero, X. Dong, M. M. Bronstein, Y. LeCun, and R. Shwartz-Ziv (2026) Attention sinks and compression valleys in llms are two sides of the same coin. In International Conference on Learning Representations, Cited by: §1, §1, §2. [23] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, p. 140:1–140:67. Cited by: §3.1. [24] N. Shazeer (2020) GLU variants improve transformer. arXiv. Cited by: §2. [25] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu (2021) RoFormer: enhanced transformer with rotary position embedding. arXiv. Cited by: §2. [26] Z. Su, Q. Li, H. Zhang, W. Ye, Q. Xue, Y. Qian, N. Wong, and K. Yuan (2026) Unveiling super experts in mixture-of-experts large language models. In International Conference on Learning Representations, Cited by: §1. [27] Z. Su and K. Yuan (2025) KVSink: understanding and enhancing the preservation of attention sinks in kv cache quantization for llms. In Conference on Language Modeling, Cited by: §1, §5. [28] M. Sun, X. Chen, J. Z. Kolter, and Z. Liu (2024) Massive activations in large language models. In Conference on Language Modeling, Cited by: §1, §1, §2. [29] S. Sun, A. Canziani, Y. LeCun, and J. Zhu (2026) The spike, the sparse and the sink: anatomy of massive activations and attention sinks. arXiv. Cited by: §1, §1, §5. [30] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom (2023) Llama 2: open foundation and fine-tuned chat models. arXiv. Cited by: §1, §2. [31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Cited by: §2. [32] H. Wang, S. Ma, L. Dong, S. Huang, D. Zhang, and F. Wei (2024) DeepNet: scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (10), p. 6761–6774. Cited by: §4. [33] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis (2024) Efficient streaming language models with attention sinks. In International Conference on Learning Representations, Cited by: §1, §1. [34] R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu (2020) On layer normalization in the transformer architecture. In International Conference on Machine Learning, Cited by: §1. [35] B. Zhang and R. Sennrich (2019) Root mean square layer normalization. In Advances in Neural Information Processing Systems, Cited by: §1, §2. [36] H. Zhang, Y. N. Dauphin, and T. Ma (2019) Fixup initialization: residual learning without normalization. In International Conference on Learning Representations, Cited by: §4. [37] Z. M. K. Zuhri, E. H. Fuadi, and A. F. Aji (2025) Softpick: no attention sink, no massive activations with rectified softmax. arXiv. Cited by: §1, §5.