Paper deep dive
Massive Activation Gating Channel in Large Language Models
Minjia Mao, Shi Chen, Bowen Yin, Xiao Fang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/13/2026, 3:58:21 AM
Summary
This paper identifies the 'Massive Activation Gating Channel' (MAGC), a specific input embedding channel that controls the emergence of massive activations in Large Language Models (LLMs). The study demonstrates that when the value of this fixed-position channel is sufficiently large or small, the output of the spike Feed-Forward Network (FFN) exhibits massive activations. Theoretical analysis shows this occurs because the FFN output asymptotically reduces to a quadratic form mixing specific columns of the down-projection matrix, which inherently possess the shape of massive activations. This mechanism was verified across six LLMs from four different families.
Entities (6)
Relation Signals (4)
Massive Activation Gating Channel → controls → Massive Activations
confidence 96% · the emergence of massive activations is controlled by a single channel in the input embedding... We name this channel the massive activation gating channel (MAGC).
Massive Activation Gating Channel → islocatedin → Spike Feed-Forward Network
confidence 94% · a single channel in the input embedding to a spike feed-forward network (FFN)... The position of this channel is fixed for a particular LLM.
Llama 3.2-1B → exhibits → Massive Activation Gating Channel
confidence 93% · In LLaMA-3.2-1B... we identify that in LLaMA-3.2-1B, the FFN in the second transformer block... produces massive activations... The 894th channel... has a value of 8.807... we name this channel the massive activation gating channel (MAGC).
Massive Activation Gating Channel → causesvia → Down-projection Matrix
confidence 91% · the output of a spike FFN asymptotically reduces to a quadratic form that mixes a few columns of the down-projection matrix of the FFN.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Massive activations, a phenomenon in which a small number of hidden channels exhibit exceptionally large magnitudes, are pervasive in large language models (LLMs). However, the mechanism by which a token develops massive activations as it propagates through a pretrained LLM remains poorly understood. In this paper, we find that the emergence of massive activations is controlled by a single channel in the input embedding to a spike feed-forward network (FFN). The position of this channel is fixed for a particular LLM. We name this channel the massive activation gating channel (MAGC). When the value of the MAGC is sufficiently large (or small, depending on the LLM), the output of the spike FFN exhibits massive activations. Examining six LLMs across four model families and different model sizes, we verify the existence and effect of MAGC. We further provide a theoretical explanation of the mechanism by which MAGC induces massive activations. When the value of MAGC is sufficiently large (or small), the output of a spike FFN asymptotically reduces to a quadratic form that mixes a few columns of the down-projection matrix of the FFN. Since these columns exhibit the shape of massive activations, the output therefore exhibits massive activations.
Tags
Links
- Source: https://arxiv.org/abs/2610.07661v1
- Canonical: https://arxiv.org/abs/2610.07661v1
Trouble viewing inline? Open PDF directly →
Full Text
52,776 characters extracted from source content.
Expand or collapse full text
Massive Activation Gating Channel in Large Language Models Minjia Mao Affiliation: University of Delaware Email: mjmao@udel.edu Shi Chen Affiliation: University of Delaware Email: shichen@udel.edu Bowen Yin Affiliation: Peking University Email: xfang@udel.edu Xiao Fang Affiliation: University of Delaware Email: bowenyin@stu.pku.edu.cn Abstract Massive activations, a phenomenon in which a small number of hidden channels exhibit exceptionally large magnitudes, are pervasive in large language models (LLMs). However, the mechanism by which a token develops massive activations as it propagates through a pretrained LLM remains poorly understood. In this paper, we find that the emergence of massive activations is controlled by a single channel in the input embedding to a spike feed-forward network (FFN). The position of this channel is fixed for a particular LLM. We name this channel the massive activation gating channel (MAGC). When the value of the MAGC is sufficiently large (or small, depending on the LLM), the output of the spike FFN exhibits massive activations. Examining six LLMs across four model families and different model sizes, we verify the existence and effect of MAGC. We further provide a theoretical explanation of the mechanism by which MAGC induces massive activations. When the value of MAGC is sufficiently large (or small), the output of a spike FFN asymptotically reduces to a quadratic form that mixes a few columns of the down-projection matrix of the FFN. Since these columns exhibit the shape of massive activations, the output therefore exhibits massive activations. 1 Introduction Large Language Models (LLMs) have demonstrated strong capabilities on various tasks (Bubeck et al., 2023). While the majority of existing work focuses on external behaviors of LLMs, it is also important to understand the internal mechanisms of LLMs. Among them, attention sink (Xiao et al., 2023) and massive activations (Sun et al., 2024) have been found in modern LLMs across different model families. Specifically, attention sink refers to the phenomenon in which the first token (or other tokens with little semantic importance) in a sequence receives a disproportionately large amount of attention from subsequent tokens (Yu et al., 2024; Barbero et al., 2025), typically in the middle transformer blocks. The cause of attention sink is usually attributed to massive activations (Sun et al., 2024; Yan et al., 2024), where a few channels (dimensions) in the hidden representation of the first token (or other tokens with little semantic importance) exhibit large activation magnitudes before attention computation. It is important to unveil the mechanism underlying massive activations and attention sink, as these phenomena have been widely employed in various critical tasks, including KV-cache optimization (Ge et al., 2023), inference (Xiao et al., 2023), quantization (Son et al., 2024), and hallucination mitigation (Liu et al., 2026). In this study, we focus on massive activations. Although existing research has extensively demonstrated the phenomenon of massive activations, research on the mechanism underlying massive activations remains limited. We address this problem by asking: how does a token develop massive activations as it propagates a pretrained transformer? We refer to tokens that exhibit massive activations as spike tokens (Sun et al., 2026). As LLMs are powered by billions of parameters, our analysis draws attention to the computation of each sub-module (attention and feed-forward networks) in transformers. Existing work (Zhang et al., 2026; Sun et al., 2026) has demonstrated that massive activations emerge after certain feed-forward networks (FFNs) and are maintained in the middle transformer blocks through the residual layer (see Section 2 for more details). For example, in LLaMA-2-7B, massive activations of a spike token emerge after the FFN in the second transformer block (Sun et al., 2024; Sun et al., 2026) and are maintained until reaching the last FFN. We refer to the FFN that produces massive activations as the spike FFN. The magnitudes of massive activations remain similar in the middle transformer blocks, indicating that only the residual layer maintains massive activations and that no further massive activations are triggered during the middle computation. Sun et al. (2026) assume that the SiLU activation function in the spike FFN behaves as a near-identity function for spike tokens. Under this assumption, each output channel k of the spike FFN can be expressed as a quadratic form characterized by a channel-specific coefficient matrix SkS_k. The magnitude of output channel k is determined by the dominant eigenvalue of SkS_k. Since the dominant eigenvalues associated with a few channels are substantially larger than those of other channels, massive activation emerges. However, our empirical results show that the near-identity assumption on which Sun et al. (2026) rely to explain massive activations is only valid for a small number of models. Among the six models examined in this study, only one model approximately satisfies the assumption (see Appendix A for more details). Moreover, their explanation overlooks the properties of input embeddings to the spike FFN. Instead, we find that the emergence of massive activations is controlled by a single channel in the input embeddings of a spike token to the spike FFN: massive activations appear once the value of this channel is sufficiently large (or small, depending on the LLM). For a given LLM, the position of this channel is fixed. We refer to this channel as the massive activation gating channel (MAGC). Unlike massive activations, whose magnitudes can reach approximately 400 in LLaMA-3.2-1B, the MAGC in the embedding of a spike token has an average value of only around 8.8. Although this value is substantially larger than those of the other channels in the embedding, its magnitude remains moderate because the embedding is layer-normalized before being fed into the FFN. We verify this finding across multiple decoder-only LLM families, including LLaMA, Qwen, DeepSeek, and Mistral, with model sizes ranging from 1B to 32B parameters. Moreover, our theoretical analysis shows that massive activations can be attributed to the alignment between MAGC and certain rows and columns of the spike FFN weight matrices. Similarly, we empirically demonstrate that this alignment occurs in different LLMs. We summarize our contributions as follows. 1. The emergence of massive activations is controlled by a single channel in the input embedding to a spike FFN, and the position of this channel is fixed for a particular LLM. We name this channel the massive activation gating channel (MAGC). When the value of the MAGC is sufficiently large (or small, depending on the LLM), the output of the spike FFN exhibits massive activations. Examining six LLMs across four model families and different model sizes, we verify the existence and effect of MAGC. 2. We provide a theoretical explanation of the mechanism by which MAGC induces massive activations. When the value of MAGC is sufficiently large (or small), the output of a spike FFN asymptotically reduces to a quadratic form that mixes a few columns of the down-projection matrix of the FFN. Since these columns exhibit the shape of massive activations, the output therefore exhibits massive activations. 2 Transformer Architecture A dense transformer model consists of L transformer blocks, each of which can be further decomposed into modules. Taking LLaMA as an example (Touvron et al., 2023), denote the input to the l-th transformer block as Xl=⟨0l,1l,⋯,Tl⟩X^l= x_0^l, x_1^l,·s, x_T^l . Without further specification, ⋅tl·_t^l denotes an embedding of the t-th token in the l-th transformer block. Since our following analysis focuses on the single spike FFN, we omit the superscript l for notational simplicity. Each block of LLaMA consists of an attention sub-module and an FFN sub-module, each of which uses pre-normalization (RMSNorm (Zhang and Sennrich, 2019)) and includes a residual layer after its computation. The input embedding is first normalized by an RMSNorm, referred to as input layer norm. Formally, if t=⟨xt(0),xt(1),⋯,xt(d−1)⟩ x_t= x_t(0),x_t(1),·s,x_t(d-1) , where d is the dimension of the embedding, then t=RMSNorm(t):nt(j)=xt(j)⋅wjRMS(t),RMS(t)=∑jxt(j)2d+ϵ, n_t=RMSNorm( x_t):n_t(j)= x_t(j)· w_jRMS( x_t),RMS( x_t)= _jx_t(j)^2d+ε, (1) where wjw_j are trainable parameters and ϵε is a predefined small number. We denote ⋅(j)·(j) as the j-th channel of the embedding. Next, the normalized embedding t n_t is transformed by multi-head attention (MHA, or more specifically, grouped-query attention (Ainslie et al., 2023)): (0,1,⋯,T)=MHA(0,1,⋯,T),( u_0, u_1,·s, u_T)=MHA( n_0, n_1,·s, n_T), where t u_t is the embedding output of MHA. Usually, a transformer includes a residual layer after the multi-head attention (Vaswani et al., 2017). Next, the FFN sub-module follows a similar configuration with another RMSNorm of different parameters. t=RMSNorm(t+t). h_t=RMSNorm( x_t+ u_t). A feed-forward network (FFN) is then applied to t h_t. In LLaMA, the FFN consists of a SwiGLU gated activation function (Shazeer, 2020) with three weight matrices: t=FFN(t),t=down⋅(SiLU(gatet)⊙(upt)), y_t=FFN( h_t), y_t= W_down· (SiLU ( W_gate h_t ) ( W_up h_t ) ), (2) where ⊙ is the element-wise product and y is the output of FFN. gate,up∈ℝm×d W_gate, W_up ^m× d and down∈ℝd×m W_down ^d× m are pretrained weight matrices, where m>dm>d is the FFN intermediate dimension. Finally, another residual layer follows the FFN. That is, the output of a transformer block is t+t+t x_t+ u_t+ y_t. (a) Value of input h to the spike FFN. We highlight the massive activation gating channel (894) in red. (b) Magnitude of output y of the spike FFN. Token 0 exhibits massive activations. We depict magnitudes instead of actual values for better visualization. Figure 1: Inputs and Outputs of the Spike FFN. 3 Massive Activation Gating Channel Experimental Setup. In this section, we utilize a small but advanced LLM, LLaMA-3.2-1B (Dubey et al., 2024). We verify the findings discovered in this section on other model families with larger model sizes in Section 5.1. In LLaMA-3.2-1B, d=2,048d=2,048 and m=8,192m=8,192. We generate token sequences (without the BOS token) completely randomly to avoid the influence of semantics. Specifically, we randomly sample five tokens in the vocabulary set V and construct an input token sequence to the LLM. Note that the input tokens can be multilingual as well as completely meaningless. We conduct this sampling 5 times. 3.1 Existence of MAGC in the Input to Spike FFN Following Sun et al. (2026), we demonstrate the top-3 channel magnitudes of the transformer block outputs ++ x+ u+ y in Figure A1 in Appendix B. Similarly, we show that massive activations are first triggered by the FFN in an early layer, and are then preserved and propagated along the residual stream across layers. As depicted, we identify that in LLaMA-3.2-1B, the FFN in the second transformer block (i.e., block 1) produces massive activations. We name this FFN the spike FFN and focus our analysis on it. A straightforward method for analyzing the causes of massive activations is to trace their inputs. To this end, we directly visualize the embeddings of the inputs and outputs of the spike FFN for different tokens in Figure 1, using a token sequence example. We demonstrate other examples in Figure A2 in Appendix B.2. As shown in Figure 1(b), the first token (“stejně”) exhibits massive activations, whereas other tokens do not. Comparing their inputs to the FFN, we find that the input h for the first token has a single outlier value, as shown in Figure 1(a). Specifically, the 894th channel of the first token has a value of 8.807, while the absolute values of the other channels are all less than 2. The 894th channel of the other tokens has values of 0.814, -0.763, 0.335, and 1.029, respectively. In the next subsection, we will show that this single channel controls the emergence of massive activations, and we name this channel the massive activation gating channel (MAGC). Figure 2: From top to bottom, the figure shows the output of the spike FFN when the input vector values are randomly sampled from a Pareto distribution (a=1a=1), with the input value of the 894894th (MAGC), 00th, or 17421742nd channel swapped with the maximum input value, respectively. 3.2 MAGC Triggers Massive Activations Next, we show that this single-channel spike triggers massive activations through the computation of the spike FFN. To empirically validate this claim, we conduct the following experiment. Specifically, we randomly generate the input embedding to the spike FFN from a Pareto distribution with random signs: each channel value of the input embedding is i.i.d. drawn as X=random(±)ZX=random(±)Z with Z∼Pareto(a),Z≥0Z (a),Z≥ 0, where each element takes +Z+Z and −Z-Z with equal probability 0.5, and a is a fixed shape parameter. Next, we swap the input value of the 894th channel with the maximum input value. We then input this randomly generated embedding into the spike FFN. Meanwhile, we conduct a controlled experiment. Specifically, we used the same randomly sampled input embedding but swapped the input value of a randomly selected channel other than the 894th, such as the 0th or 1742nd, with the maximum value. We report the results for a=1a=1 in this section and present the results for a=3a=3 and a=5a=5 in Appendix B.3. Figure 2 depicts the randomly generated inputs and their corresponding outputs after the spike FFN. It can be observed that when swapping the value of the 894th channel with the maximum value, the outputs of the spike FFN demonstrate massive activations. Since the randomly generated inputs carry no semantic meaning, this phenomenon mainly indicates that the FFN learns to recognize MAGC. However, swapping the value of any other channel, such the 0th or 1742nd channel, with the maximum value does not lead to the output of massive activations. We therefore conclude that MAGC controls the emergence of massive activations. Moreover, as shown in Figure A3 and A4 in Appendix B.3, decreasing a increases the maximum value generated by the Pareto distribution. More importantly, assigning a larger maximum value to the 894th channel of the spike FFN input results in a correspondingly larger activation magnitude in its output. Specifically, as a decreases from 5 to 1, the largest output activation magnitude increases from approximately 40 to 400. These results demonstrate that the spike FFN is highly sensitive to the value of the 894th input channel, with a larger value in this channel inducing more “massive” activations. Key Takeaway 1: There exists a massive activation gating channel (MAGC) in the hidden representation of a spike token, whose position is fixed for a specific LLM. When the hidden representation is used as the input to the spike FFN, the FFN output exhibits massive activations when the value of the MAGC in the hidden representation is sufficiently large (or small, depending on the LLM). 4 From MAGC to Massive Activations In this section, we examine how a large value of MAGC induces massive activations in LLMs. Since a large value of MAGC corresponds to a large magnitude in a certain channel of the input to the FFN, we begin by analyzing the asymptotic property of the FFN with respect to any input channel. Recall the FFN computation in Equation 2: ()=down(SiLU(gate)⊙up). y( h)= W_down (SiLU( W_gate h) W_up h ). We show that, when a fixed channel has a large magnitude (in either direction), the mechanism of the FFN can be simplified as mixing columns of down W_down. Note that the mixing mechanism depends on the direction of the channel value (x→+∞x→+∞ or x→−∞x→-∞), which explains why MAGC also has a fixed gating direction. 4.1 Asymptotic Property of an FFN w.r.t. an Input Channel Figure 1(a) demonstrates the phenomenon that the input to the FFN contains a large magnitude in a certain channel for spike tokens. We begin by analyzing the asymptotic property of any FFN with respect to any input channel. Consider a gated FFN in Equation 2, for input ∈ℝd h ^d, FFN()=()=down(SiLU(gate)⊙up). ( h)= y( h)= W_down (SiLU( W_gate h) W_up h ). (3) where up∈ℝm×d W_up ^m× d, gate∈ℝm×d W_gate ^m× d, and down∈ℝd×m W_down ^d× m are parameter matrices, SiLU(x):=x⋅σ(x)SiLU(x):=x·σ(x) with the logistic sigmoid function σ(x):=11+exp(−x)σ(x):= 11+ (-x), and ⊙ is the element-wise product. We focus on the input h controlled by a fixed single channel i. For any value x∈ℝx in the fixed channel i, the input can be written as, (x)=x⋅i+¯. h(x)=x· e_i+ h. (4) Here, ¯∈ℝd h ^d is the representation outside the focused channel i with ¯(i)=0 h(i)=0, and i e_i is the unit vector at position i. Next, we analyze the asymptotic property of ((x)) y( h(x)) for x→+∞x→+∞ without loss of generality. Let j=down(:,j)/‖down(:,j)‖2 ω_j= W_down^(:,j) / \| W_down^(:,j) \|_2 be the normalized j-th column of down W_down. The following theorem shows the closed form of ((x)) y( h(x)) when x→+∞x→+∞. Theorem 4.1. Given input (x)=x⋅i+¯ h(x)=x· e_i+ h, when x→+∞x→+∞, the output of an FFN has a quadratic form ((x))=x2⋅∗+o(x2) y( h(x))=x^2· y^*+o(x^2) with constant ∗∈ℝd y^* ^d equal to ∗=∑j=1mαj⋅j, y^*= _j=1^m _j· ω_j, (5) where =(α1,…,αm) α=( _1,…, _m) is the combination of coefficients in the weight matrices: αj=Wgate(j,i)Wup(j,i)‖down(:,j)‖2Wgate(j,i)>00Wgate(j,i)≤0. _j= casesW_gate^(j,i)W_up^(j,i) \| W_down^(:,j) \|_2&W_gate^(j,i)>0\\ 0&W_gate^(j,i)≤ 0 cases. (6) Proof. See Appendix C.1. Specifically, for any input channel i in h, SiLU(gate)SiLU( W_gate h) acts as a gating function, choosing column j only when Wgate(j,i)x>0W_gate^(j,i)x>0, and αj _j controls the sensitivity to each j ω_j. From Theorem 4.1, when x is large enough, the output ((x))=x2⋅∗+o(x2) y( h(x))=x^2· y^*+o(x^2) exhibits a pattern of massive activations only if ∗ y^* has the same pattern. Similarly, an analogous result holds as x→−∞x→-∞. 4.2 From MAGC to Massive Activations We introduce several metrics below to quantify the degree of massive activations. Since massive activations describe the phenomenon in which certain activations exhibit large magnitudes, we utilize the energy distribution to characterize their magnitude distribution. Definition 4.2 (Energy Distribution). Given vector =(v1,…,vd)∈ℝd v=(v_1,…,v_d) ^d, let its energy distribution be ()=(p1(),…,pd()) p( v)=(p_1( v),…,p_d( v)), where pi()=vi2‖22,∀i=1,…,d. p_i( v)= v^2_i\| v\|_2^2, ∀ i=1,…,d. (7) The energy distribution is a probability distribution. Moreover, if y exhibits massive activations, () p( y) should be highly concentrated. We define the degree of concentration as the negative entropy of () p( y) plus a constant. Formally, Definition 4.3 (Concentration Degree). Given vector ∈ℝd v ^d, let its concentration degree be R()=logd+∑i=1dpi()logpi() R( v)= d+ _i=1^dp_i( v) p_i( v) (8) where ()=(p1(),…,pd()) p( v)=(p_1( v),…,p_d( v)) is the energy distribution defined in Definition 4.2. The concentration degree R()∈[0,logd]R( v)∈[0, d] is exactly the negative entropy plus a constant logd d. A larger R()R( v) means v is more concentrated. R()=0R( v)=0 means v is uniform, while R()=logdR( v)= d means v has only one non-zero entry. We now fix MAGC as the focused channel i∗i^* and analyze the relationship between the concentration degree of the output ∗ y^* and those of j ω_j and α. We first find that the following assumption is empirically satisfied. Assumption 4.4. When MAGC is the focused channel, α has a high concentration degree, i.e., R()≥logm−ϵR( α)≥ m-ε for a small ϵε. Let j∗=argmaxjpj()j^*= _jp_j( α) be the dominant column index; then R()≥logm−ϵ⇒pj∗()≥e−ϵ. R( α)≥ m-ε p_j^*( α)≥ e^-ε. (9) Empirical verification. Rather than assuming this property, we verify it directly by scanning all d=2,048d=2,048 input channels of the spike FFN in LLaMA-3.2-1B: for each channel i we form the coefficient vector α from Theorem 4.1 and measure its concentration R()R( α), dominant energy share pj∗()p_j^*( α), and norm ‖2\| α\|_2. MAGC (i∗=894i^*=894) attains the largest concentration among all channels, with R()=9.009R( α)=9.009 against logm=9.011 m=9.011 (i.e. ϵ≈1.9×10−3ε≈ 1.9× 10^-3) and pj∗()=0.9998p_j^*( α)=0.9998, whereas the mean over all channels is only R()=2.908R( α)=2.908. This confirms that MAGC is a significant outlier in α-concentration. (a) α vector when i∗=894i^*=894 for LLaMA-3.2-1B. (b) 1417 ω_1417 for LLaMA-3.2-1B. (c) 0 ω_0 for LLaMA-3.2-1B. (d) 1 ω_1 for LLaMA-3.2-1B. Figure 3: Visualization results for LLaMA-3.2-1B. A high concentration degree R()R( α) indicates that a few pj()p_j( α) are extremely large. We empirically demonstrate α when i∗=894i^*=894 for LLaMA-3.2-1B in Figure 3(a) and highlight the largest magnitude in the vector. As shown, α is highly concentrated with j∗=1417j^*=1417. That is, the output y of the FFN is largely controlled by the j∗j^*-th column of down W_down. Theoretically, based on the high concentration of α, we next quantify the contribution of the remaining columns. Definition 4.5. Given j∗=argmaxjpj()j^*= _jp_j( α), define the residual amplification factor κ=‖∑j≠j∗αjj‖2∑j≠j∗αj2.κ= \| _j≠ j^* _j ω_j \|_2 _j≠ j^* _j^2. The residual amplification factor κ quantifies the contribution of other channels after removing the dominant column j∗j^*. In the following proposition, κ controls how strongly the non-dominant columns can perturb the energy distribution of the output. Proposition 4.6. Let τ=κ1−pj∗()pj∗(). τ=κ 1-p_j^*( α)p_j^*( α). (10) We have |R(∗)−R(j∗)|≤Δ(|sin∠(∗,j∗)|)≤Δ(τ), |R( y^*)-R( ω_j^*) |≤ (| ( y^*, ω_j^*)| )≤ (τ ), (11) where for t∈(0,1−1d)t∈ (0,1- 1d ), the function Δ is defined as Δ(t)=−tlogt−(1−t)log(1−t)+tlog(d−1). (t)=-t t-(1-t) (1-t)+t (d-1). Proof. See Appendix C.2. Proposition 4.6 indicates that when pj∗()p_j^*( α) is large (i.e., close to 1), we have τ≈0τ≈ 0, and there is almost no gap between the concentration degree of ∗ y^* and that of j∗ ω_j^*. In the case of LLaMA-3.2-1B, the output y of the FFN is largely controlled by the j∗j^*-th (1417th) column of down W_down. We next depict j∗ ω_j^* in Figure 3(b). Comparing Figure 3(b) to 1(b), we find that these two vectors are highly co-linear, empirically with |sin∠(∗,j∗)|=0.0055| ( y^*, ω_j^*)|=0.0055. For other columns such as columns 0 and 1 in Figures 3(c) and 3(d), the parameters have lower maximum magnitudes and do not contain the pattern of massive activations. Note that both ∗ y^* and j∗ ω_j^* are computed only from the pretrained weight matrices. We show that when the value of channel i∗i^* is large enough, the actual output y is dominated by j∗ ω_j^*, which leads to ≈x2∗≈x2αj∗j∗ y≈ x^2 y^*≈ x^2 _j^* ω_j^*. If αj∗ _j^* has a large magnitude and j∗ ω_j^* is highly concentrated (i.e., contains the pattern of massive activations), the output y should exhibit the same pattern. Below, we summarize the mechanism by which the MAGC in the input to a spike FFN leads to its output that exhibits massive activations. Key Takeaway 2: When the MAGC of an input embedding to the spike FFN takes a sufficiently large (or small) value, the output of the FFN asymptotically approaches x2∑j=1mαjjx^2 _j=1^m _j ω_j, where j ω_j is the normalized j-th column of down W_down of the FFN and x is the value of the MAGC. Moreover, the coefficients =(α1,…,αm) α=( _1,…, _m) concentrate on one (or a few) dimensions. As a result, the FFN output is reduced to one normalized column of down W_down or a combination of a few normalized columns of down W_down, multiplied by their respective coefficient and x2x^2. Since these columns exhibit the shape of massive activations, the output therefore exhibits massive activations. 5 Extension 5.1 Extension to Other LLMs Model Name Spike FFN Block MAGC(i∗i^*) Direction Concentrated Columns(J) LLaMA-3.2-1B 1 894 + 1417 LLaMA-2-7B 1 310 + 10411, 7890 Qwen2.5-1.5B 1 1395 - 5601, 3295, 3750 QwQ-32B 4 4626 - 11696, 1368 Ministral-3-3B 2 2 - 0, 2 Deepseek-R1-8B 1 717 + 2427, 198 Table 1: MAGC in different models. The spike FFN block, MAGC i∗i^*, and concentrated column ids are counted from 0. The concentrated columns are identified by first selecting the three columns ωjω_j of the down-projection matrix with the largest absolute αj _j values and then filtering out the selected columns that do not satisfy |cos(j,0)|>0.9| ( ω_j, y_0)|>0.9. In this section, we examine the generality of our findings across LLMs of different model families and sizes. Specifically, we study LLaMA-2-7B, Qwen2.5-1.5B, QwQ-32B, Ministral-3-3B-Instruct-2512, and DeepSeek-R1-Distill-Llama-8B. Together with LLaMA-3.2-1B, these models cover four model families and range from 1B to 32B parameters. Based on our empirical findings, we propose an empirical method to identify the MAGC as detailed in Appendix D. For each model, we repeat the procedure in the preceding sections: we localize the spike FFN, identify MAGC i∗i^* and its gating direction, and find the dominant column(s) J of down W_down. Table 1 summarizes the results, and the corresponding visualizations are provided in Appendix D. We highlight the following findings. First, the spike FFN consistently occurs near the beginning of the network. Notably, even for QwQ-32B with 64 transformer blocks, the spike FFN is located at block 4. Second, the position i∗i^* of MAGC is fixed for a specific model but differs across models, as does the gating direction. Third, the dominant column is not always unique. For LLaMA-3.2-1B, the energy distribution pj()p_j( α) is dominated by a single column of down W_down. For the remaining models, however, two or three columns share the dominant energy. As shown in Appendix D, the columns in J exhibit highly similar shapes and concentrate their mass on the same small set of output channels, so their weighted combination ∗ y^* remains concentrated and reproduces the pattern of massive activations. In contrast, a randomly selected column distributes its mass much more broadly across channels. 5.2 Relation to Attention Sink Figure 4: Sink score of the first token in LLaMA-3.2-1B, measured in the attention of block 2, as a function of the value assigned to MAGC (i∗=894i^*=894) of the first token at the input of the spike FFN (block 1). Thin curves correspond to the ten individual sentences, the thick curve to their mean, and the shaded area to one standard deviation. The unmodified average value of MAGC is 8.818.81. Attention sink is commonly attributed to massive activations (Sun et al., 2024; Yan et al., 2024). Our previous results show that massive activations can be controlled by MAGC. We now extend this finding to adjust attention sink using MAGC. Experimental Setup. We feed ten natural sentences to LLaMA-3.2-1B and intervene only on MAGC of the first token: at the input to the spike FFN (block 1), we overwrite h0(i∗)h_0(i^*) with a prescribed value c and leave all other channels and all other tokens unchanged. We then measure the attention sink of the first token in the following block (block 2) using the sink score (Gu et al., 2025), s0=1H(T+1)∑h=0H−1∑t=0TAt,0h,s_0= 1H(T+1) _h=0^H-1 _t=0^TA_t,0^h, (12) where At,0hA_t,0^h denotes the post-softmax attention that the t-th query assigns to the first token in the h-th head, T+1T+1 is the sequence length, and H is the number of attention heads. Since the first token is visible to all queries under the causal mask, s0s_0 is the average fraction of attention that the first token receives, and s0s_0 close to 1 indicates a dominant attention sink. Results. Figure 4 reports the sink score s0s_0 as a function of MAGC value c. Without intervention, the original MAGC value is 8.818.81, at which the first token attracts s0=0.89s_0=0.89 of the attention mass. As c varies, the sink score shows a sharp step-like transition. For c≥−2.5c≥-2.5, s0s_0 stays near 0.890.89, together with massive activations, as shown at c=+25c=+25. For sufficiently negative c, the sink collapses. The sink score drops to about 0.460.46 for c≤−7.5c≤-7.5, while the massive activations disappear, as shown at c=−25c=-25. The transition is consistent across sentences, with standard deviation below 0.030.03 throughout the sweep. Therefore, adjusting the MAGC value in the embedding of an input spike token to the spike FFN can adjust the sink mass on the token. 6 Conclusion and Future Work In this study, we show how tokens develop massive activations in pretrained transformers and identify the massive activation gating channel (MAGC): a channel in the input to an early FFN whose value gates the emergence of massive activations. We show both empirically and theoretically that once MAGC is large or small enough, the spike FFN asymptotically collapses into a quadratic mixing of a few highly concentrated columns of the down-projection matrix, which reproduces the massive-activation pattern. This finding holds across six LLMs from four families ranging from 1B to 32B parameters. Our analysis has several limitations that suggest directions for future work. First, it focuses on decoder-only LLMs with gated (such as SwiGLU) FFNs; extending the framework to mixture-of-experts architectures is a natural next step. Second, we localize a single spike FFN and a single gating channel per model, whereas a few models exhibit more diffuse behavior across several channels or columns; a multi-channel account may be needed for such cases. Finally, our study is purely mechanistic at inference time and does not explain why a particular channel becomes MAGC during pretraining, which we leave for future investigation. AI use statement In this work, we used generative AI tools for paper proofreading and code writing. We have not used generative AI tools for literature review, idea formalization, and drafting the manuscript. We have reviewed all AI-assisted work. For example, LLM-generated code was verified by the authors. We take responsibility for the originality of this work. Reproducibility statement We have released all code needed to reproduce the results in this work at GitHub11 1 https://github.com/MinjiaMao/magc. It includes scripts for identifying the MAGC, running the intervention experiments, and generating figures. All experiments use publicly available pretrained models. References Ainslie et al. (2023) J. Ainslie, J. Lee-Thorp, M. De Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai Gqa: training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 4895–4901. Cited by: §2. Barbero et al. (2025) F. Barbero, A. Arroyo, X. Gu, C. Perivolaropoulos, P. Veličković, R. Pascanu, and M. M. Bronstein Why do llms attend to the first token?. In Second Conference on Language Modeling, Cited by: §1. Bubeck et al. (2023) S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: early experiments with gpt-4. arXiv preprint arXiv:2303.12712. Cited by: §1. Dubey et al. (2024) A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv e-prints, p. arXiv–2407. Cited by: §3. Ge et al. (2023) S. Ge, Y. Zhang, L. Liu, M. Zhang, J. Han, and J. Gao Model tells you what to discard: adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801. Cited by: §1. Gu et al. (2025) X. Gu, T. Pang, C. Du, Q. Liu, F. Zhang, C. Du, Y. Wang, and M. Lin When attention sink emerges in language models: an empirical view. In International Conference on Learning Representations, Vol. 2025, p. 97114–97144. Cited by: §5.2. Liu et al. (2026) X. Liu, G. Chen, and W. Wang SinkTrack: attention sink based context anchoring for large language models. In The Fourteenth International Conference on Learning Representations, Cited by: §1. Shazeer (2020) N. Shazeer Glu variants improve transformer. arXiv preprint arXiv:2002.05202. Cited by: §2. Son et al. (2024) S. Son, W. Park, W. Han, K. Kim, and J. Lee Prefixing attention sinks can mitigate activation outliers for large language model quantization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 2242–2252. Cited by: §1. Sun et al. (2024) M. Sun, X. Chen, J. Z. Kolter, and Z. Liu Massive activations in large language models. In First Conference on Language Modeling, Cited by: Appendix D, §1, §1, §5.2. Sun et al. (2026) S. Sun, A. Canziani, Y. LeCun, and J. Zhu The spike, the sparse and the sink: anatomy of massive activations and attention sinks. arXiv preprint arXiv:2603.05498. Cited by: Appendix A, §1, §1, §1, §1, §3.1. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §2. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. Advances in neural information processing systems 30. Cited by: §2. Xiao et al. (2023) G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. Cited by: §1. Yan et al. (2024) R. Yan, X. Du, H. Deng, L. Zheng, Q. Sun, J. Hu, Y. Shao, P. Jiang, J. Jiang, and L. Zhao Unveiling and controlling anomalous attention distribution in transformers. arXiv preprint arXiv:2407.01601. Cited by: §1, §5.2. Yu et al. (2024) Z. Yu, Z. Wang, Y. Fu, H. Shi, K. Shaikh, and Y. C. Lin Unveiling and harnessing hidden attention sinks: enhancing large language models without training through attention calibration. arXiv preprint arXiv:2406.15765. Cited by: §1. Zhang and Sennrich (2019) B. Zhang and R. Sennrich Root mean square layer normalization. Advances in neural information processing systems 32. Cited by: §2. Zhang et al. (2026) S. Zhang, M. Khan, and V. Papyan Attention sinks: a’catch, tag, release’mechanism for embeddings. Advances in Neural Information Processing Systems 38, p. 83140–83181. Cited by: §1. Appendix A Generalization of Near Identity Assumption for SiLU Following Sun et al. (2026), we compute the cosine similarity between the gate pre-activation gatet W_gate h_t and its activation SiLU(gatet)SiLU( W_gate h_t), and the ratio of their ℓ2 _2 norms for spike tokens. Using the randomly generated sequences described in Section 3, we report in Table A1 both metrics averaged over the first tokens of these sequences. For SiLU to behave approximately as an identity function, both metrics should be close to 1. Among the models examined, this condition is approximately satisfied only by LLaMA-2-7B. Model Metric Token 0 LLaMA-2-7B Cosine sim. 0.9719 Norm ratio 0.8103 LLaMA-3.2-1B Cosine sim. 0.8623 Norm ratio 0.6232 Qwen2.5-1.5B Cosine sim. 0.2276 Norm ratio 0.1564 QwQ-32B Cosine sim. 0.2990 Norm ratio 0.1186 Ministral-3-3B Cosine sim. 0.7895 Norm ratio 0.7897 Deepseek-R1-8B Cosine sim. 0.9215 Norm ratio 0.7720 Table A1: Input-output characteristics of SiLU in the spike FFN across models. Figure A1: Top-3 channel magnitudes of transformer block outputs x+u+yx+u+y in Llama-3.2-1B and their differences. Appendix B Additional Experiments for LLaMA-3.2-1B Figure A2: Inputs and Outputs of the Spike FFN. B.1 Massive Activations Figure A1 shows the block-wise differences of the top-3 channel magnitudes of the transformer block outputs ++ x+ u+ y in LLaMA-3.2-1B. As depicted, the differences exhibit a single sharp jump at the second transformer block, where the spike FFN triggers massive activations. In the subsequent middle blocks, the differences remain close to zero, indicating that no further massive activations are triggered and that the residual stream alone preserves the existing ones until the late blocks. B.2 Input and Output of the Spike FFN for other Sample Sequences Following Section 3, we depict the activations of the other 4 randomly generated sequences in Figure A2. B.3 Pareto Inputs for the Spike FFN Figure A3: From top to bottom, the figure shows the output of the spike FFN when the input vector values are randomly sampled from a Pareto distribution (a=3a=3), with the input value of the 894894th (MAGC), 00th, or 17421742nd channel swapped with the maximum input value, respectively. Figure A4: From top to bottom, the figure shows the output of the spike FFN when the input vector values are randomly sampled from a Pareto distribution (a=5a=5), with the input value of the 894894th (MAGC), 00th, or 17421742nd channel swapped with the maximum input value, respectively. Appendix C Proofs C.1 Proof of Theorem 4.1 Theorem C.1. Given input (x)=x⋅i+¯ h(x)=x· e_i+ h. When x→+∞x→+∞, the output of an FFN has a quadratic form ((x))=x2⋅∗+o(x2) y( h(x))=x^2· y^*+o(x^2) with constant ∗∈ℝd y^* ^d equals to ∗=∑j=1mαj⋅j, y^*= _j=1^m _j· ω_j, (A1) where =(α1,…,αm) α=( _1,…, _m) is the combination of coefficients in the weight matrices: αj=Wgate(j,i)Wup(j,i)‖down(:,j)‖2Wgate(j,i)>00Wgate(j,i)≤0. _j= casesW_gate^(j,i)W_up^(j,i) \| W_down^(:,j) \|_2&W_gate^(j,i)>0\\ 0&W_gate^(j,i)≤ 0 cases. (A2) Proof. Based on Equation 4, the linear representation of gate(x) W_gate h(x) and up(x) W_up h(x) can be written as gate(x)=xgate(:,i)+gate W_gate h(x)=x W_gate^(:,i)+ b_gate (A3) up(x)=xup(:,i)+up W_up h(x)=x W_up^(:,i)+ b_up (A4) where gate=∑i=1dh¯igate(:,i) b_gate= _i=1^d h_i W_gate^(:,i) and up=∑i=1dh¯iup(:,i) b_up= _i=1^d h_i W_up^(:,i) are both constant under x. We have the asymptotic property of SiLU(x)SiLU(x): limx→+∞SiLU(gateh(x))x=Wgate(j,i)Wgate(j,i)≥00Wgate(j,i)<0 _x→+∞ SiLU( W_gateh(x))x= casesW_gate^(j,i)&W_gate^(j,i)≥ 0\\ 0&W_gate^(j,i)<0 cases (A5) and the asymptotic property of up(x) W_up h(x): limx→+∞up(x)x=up(:,i) _x→+∞ W_up h(x)x= W_up^(:,i) (A6) Finally, limx→+∞((x))x2=downlimx→+∞SiLU(gateh(x))x⊙limx→+∞up(x)x=∑j=1mαj⋅j, _x→+∞ y( h(x))x^2= W_down _x→+∞ SiLU( W_gateh(x))x _x→+∞ W_up h(x)x= _j=1^m _j· ω_j, (A7) where =(α1,…,αm) α=( _1,…, _m) is αj=Wgate(j,i)Wup(j,i)‖down(:,j)‖2Wgate(j,i)>00Wgate(j,i)≤0. _j= casesW_gate^(j,i)W_up^(j,i) \| W_down^(:,j) \|_2&W_gate^(j,i)>0\\ 0&W_gate^(j,i)≤ 0 cases. (A8) ∎ C.2 Proof of Proposition 4.6 Recall that ∗=∑j=1mαjj,‖j‖2=1. y^*= _j=1^m _j ω_j, \| ω_j\|_2=1. The energy distribution of α is pj()=αj2‖22,j=1,…,m.p_j( α)= _j^2\| α\|_2^2, j=1,…,m. Let j∗=argmaxjpj()j^*= _jp_j( α) be the dominant coefficient index. Then ∗ y^* can be decomposed into the dominant column and the residual part: ∗=αj∗j∗+∑j≠j∗αjj. y^*= _j^* ω_j^*+ _j≠ j^* _j ω_j. Define the residual vector =∑j≠j∗αjj. r= _j≠ j^* _j ω_j. The residual amplification factor is defined as in Definition 4.5, κ=‖2∑j≠j∗αj2.κ= \| r\|_2 _j≠ j^* _j^2. The angular deviation between the true asymptotic output and the dominant column is |sin∠(∗,j∗)|=1−(∗⋅j∗‖∗‖2‖j∗‖2)2.| ( y^*, ω_j^*)|= 1- ( y^*· ω_j^*\| y^*\|_2\| ω_j^*\|_2 )^2. Finally, for τ∈(0,1−1d)τ∈(0,1- 1d), define Δ(τ)=−τlogτ−(1−τ)log(1−τ)+τlog(d−1). (τ)=-τ τ-(1-τ) (1-τ)+τ (d-1). Lemma C.2. Assume |sin∠(∗,j∗)|≤1−1d| ( y^*, ω_j^*)|≤ 1- 1d . Then, |R(∗)−R(j∗)|≤Δ(|sin∠(∗,j∗)|). |R( y^*)-R( ω_j^*) |≤ (| ( y^*, ω_j^*)| ). Proof. Based on the Fannes-Audenaert inequality: |R(∗)−R(j∗)|=|H(j∗)−H(j∗)|≤Δ(t),|R( y^*)-R( ω_j^*)|=|H( ω_j^*)-H( ω_j^*)|≤ (t), where t:=12‖p(∗)−p(j∗)‖1t:= 12\|p( y^*)-p( ω_j^*)\|_1 and H is the entropy function. We have 12‖p(∗)−p(j∗)‖1 12\|p( y^*)-p( ω_j^*)\|_1 =12∑i=1d|(yi∗‖∗‖2)2−(ωj∗,i‖j∗‖2)2| = 12 _i=1^d | ( y_i^*\| y^*\|_2 )^2- ( _j^*,i\| ω_j^*\|_2 )^2 | =12∑i=1d|(yi∗‖∗‖2−ωj∗,i‖j∗‖2)(yi∗‖∗‖2+ωj∗,i‖j∗‖2)| = 12 _i=1^d | ( y_i^*\| y^*\|_2- _j^*,i\| ω_j^*\|_2 ) ( y_i^*\| y^*\|_2+ _j^*,i\| ω_j^*\|_2 ) | ≤12[∑i=1d(yi∗‖∗‖2−ωj∗,i‖j∗‖2)2]1/2[∑i=1d(yi∗‖∗‖2+ωj∗,i‖j∗‖2)2]1/2 ≤ 12 [ _i=1^d ( y_i^*\| y^*\|_2- _j^*,i\| ω_j^*\|_2 )^2 ]^1/2 [ _i=1^d ( y_i^*\| y^*\|_2+ _j^*,i\| ω_j^*\|_2 )^2 ]^1/2 =12[2−2∗⋅j∗‖∗‖2‖j∗‖2]1/2[2+2∗⋅j∗‖∗‖2‖j∗‖2]1/2 = 12 [2-2 y^*· ω_j^*\| y^*\|_2\| ω_j^*\|_2 ]^1/2 [2+2 y^*· ω_j^*\| y^*\|_2\| ω_j^*\|_2 ]^1/2 =[1−(∗⋅j∗‖∗‖2‖j∗‖2)2]1/2 = [1- ( y^*· ω_j^*\| y^*\|_2\| ω_j^*\|_2 )^2 ]^1/2 =|sin∠(∗,j∗)|. =| ( y^*, ω_j^*)|. Since Δ(t) (t) is increasing on 0≤t≤1−1d0≤ t≤ 1- 1d, |R(∗)−R(j∗)|≤Δ(min|sin∠(∗,j∗)|,1−1d)=Δ(|sin∠(∗,j∗)|). |R( y^*)-R( ω_j^*) |≤ ( \| ( y^*, ω_j^*)|,1- 1d \ )= (| ( y^*, ω_j^*)| ). ∎ Proposition C.3. Let τ=κ1−pj∗()pj∗(). τ=κ 1-p_j^*( α)p_j^*( α). (A9) We have |R(∗)−R(j∗)|≤Δ(τ), |R( y^*)-R( ω_j^*) |≤ (τ ), (A10) where for τ∈(0,1−1d)τ∈ (0,1- 1d ), Δ(τ)=−τlogτ−(1−τ)log(1−τ)+τlog(d−1). (τ)=-τ τ-(1-τ) (1-τ)+τ (d-1). Proof. Recall =∑j≠j∗αjj, r= _j≠ j^* _j ω_j, then ∗=αj∗j∗+. y^*= _j^* ω_j^*+ r. Since ‖j∗‖2=1\| ω_j^*\|_2=1, decompose the residual vector as =r∥j∗+⟂,⟂j∗. r=r_ ω_j^*+ r_ , r_ ω_j^*. We have ∗=(αj∗+r∥)j∗+⟂. y^*=( _j^*+r_ ) ω_j^*+ r_ . The angular deviation from j∗ ω_j^* is caused only by the perpendicular residual component. Hence, |sin∠(∗,j∗)| | ( y^*, ω_j^*)| =‖⟂‖2[(αj∗+r∥)2+‖⟂‖22]1/2 = \| r_ \|_2 [( _j^*+r_ )^2+\| r_ \|_2^2 ]^1/2 ≤r∥2+‖⟂‖22|αj∗|=‖2|αj∗|. ≤ r_ ^2+\| r_ \|_2^2| _j^*|= \| r\|_2| _j^*|. The above inequality follows because, after squaring and rearranging, it is equivalent to 0≤(r∥(αj∗+r∥)+‖⟂‖22)2.0≤ (r_ ( _j^*+r_ )+\| r_ \|_2^2 )^2. Moreover, ‖2|αj∗|=κ∑j≠j∗αj2|αj∗|=κ1−pj∗()pj∗(). \| r\|_2| _j^*|=κ _j≠ j^* _j^2| _j^*|=κ 1-p_j^*( α)p_j^*( α). Therefore, |sin∠(∗,j∗)|≤κ1−pj∗()pj∗().| ( y^*, ω_j^*)|≤κ 1-p_j^*( α)p_j^*( α). Based on Lemma C.2, it is sufficient to prove Proposition 4.6. ∎ Appendix D Additional Results Identification of the Spike FFN and MAGC. We identify the spike FFN and MAGC by comparing the first (spike) token and the second (non-spike) token of the randomly generated sequences. Let 0l y_0^l denote the average FFN output of the first token in the l-th transformer block. We regard an FFN as the spike FFN if its output exhibits an outlier channel (Sun et al., 2024), i.e., maxj|y0l(j)|>100andmaxj|y0l(j)|>10⋅medianj|y0l(j)|. _j |y_0^l(j) |>100 _j |y_0^l(j) |>10·median_j |y_0^l(j) |. (A11) Given the spike FFN, we then identify MAGC from its inputs. Let 0 h_0 and 1 h_1 be the average inputs to the spike FFN for the first and second tokens, respectively. The gating channel should be the channel where the two inputs deviate the most. We therefore locate MAGC as i∗=argmaxj|h0(j)−h1(j)|, i^*= _j |h_0(j)-h_1(j) |, (A12) and determine the gating direction by the sign of h0(i∗)−h1(i∗)h_0(i^*)-h_1(i^*): a positive (negative) sign indicates that massive activations emerge when the value of MAGC becomes sufficiently large (small). This appendix reports the detailed results for the models summarized in Table 1: LLaMA-2-7B (Figure A5), Qwen2.5-1.5B (Figure A6), QwQ-32B (Figure A7), Ministral-3-3B-Instruct-2512 (Figure A8), and DeepSeek-R1-Distill-Llama-8B (Figure A9). For each model, we present (i) the input h and outputs y of the spike FFN for a random input sequence, (i) the coefficient vector α defined in Theorem 4.1, where for models with a negative gating direction α is computed under the limit x→−∞x→-∞ and the condition on Wgate(j,i)W_gate^(j,i) flips sign accordingly, and (i) the dominant columns (the columns most aligned with the output 0 y_0, as reported in Table 1) together with a randomly selected column for comparison. (a) Input h to FFN. (b) Output y of FFN. (c) α vector when i∗=310i^*=310. (d) 10411 ω_10411. (e) 7890 ω_7890. (f) 0 ω_0. Figure A5: Results for LLaMA-2-7b-hf. (a) Input h to FFN. (b) Output y of FFN. (c) α vector when i∗=1395i^*=1395. (d) 5601 ω_5601. (e) 3295 ω_3295. (f) 3750 ω_3750. Figure A6: Results for Qwen2.5-1.5b. (a) Input h to FFN. (b) Output y of FFN. (c) α vector when i∗=4626i^*=4626. (d) 11696 ω_11696. (e) 1368 ω_1368. (f) 0 ω_0. Figure A7: Results for QwQ-32B. For QwQ-32B, several of the channels with the largest input difference |h0(j)−h1(j)||h_0(j)-h_1(j)| coincide with massive-activation channels propagated through the residual stream; we therefore identify the MAGC (i∗=4626i^*=4626, negative gating direction) by traversing the ten channels with the largest such difference following Equation A12 and selecting the first one that is not itself a massive-activation channel, rather than simply taking the single largest. (a) Input h to FFN. (b) Output y of FFN. (c) α vector when i∗=2i^*=2. (d) 0 ω_0. (e) 2 ω_2. (f) 1 ω_1. Figure A8: Results for Ministral-3-3B-Instruct-2512. Note that columns 0 and 2 belong to j∗j^*. Instead of choosing 0 as the random column for comparison, we select column 1. (a) Input h to FFN. (b) Output y of FFN. (c) α vector when i∗=717i^*=717. (d) 2427 ω_2427. (e) 198 ω_198. (f) 0 ω_0. Figure A9: Results for DeepSeek-R1-Distill-Llama-8B.