Paper deep dive
Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
Weiqi Sun, Qianyue Wu, Wenjia Wang, Tianyang Hu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/13/2026, 2:55:30 AM
Summary
The paper introduces Outlier-Aware Attention Linearization (OAL), a method to efficiently approximate softmax attention in Transformers by addressing dimension-wise heterogeneity. The authors identify that a small number of outlier dimensions contribute disproportionately to attention errors. OAL uses a dimension-aware quadratic kernel with grouped coefficients to handle these outliers, achieving lower approximation error and improved downstream performance on models like Qwen2.5-0.5B and Llama-3.2-1B compared to existing linearization methods.
Entities (10)
Relation Signals (9)
OAL → appliedto → Qwen2.5-0.5B
confidence 95% · On Qwen2.5-0.5B, OAL reduces frozen-attention output relative MSE...
OAL → appliedto → Llama 3.2-1B
confidence 92% · OAL outperforms the evaluated converted methods... on both Qwen2.5-0.5B and Llama-3.2-1B
OAL → evaluatedon → WikiText-103
confidence 92% · lowers WikiText-103 perplexity from 31.71 to 27.50
OAL → outperforms → DiJiang
confidence 90% · On Qwen2.5-0.5B, OAL reduces frozen-attention output relative MSE by approximately 8.9% compared with DiJiang.
OAL → usestechnique → LoRA
confidence 90% · Replacing middle-layer attention modules and jointly fine-tuning the kernel coefficients with LoRA...
OAL → usesmathematicalformulation → Hadamard-product
confidence 88% · we introduce a Hadamard-product formulation that makes individual query-key contributions explicit.
OAL → evaluatedon → PIQA
confidence 85% · improving accuracy on PIQA
OAL → evaluatedon → HellaSwag
confidence 85% · improving accuracy on HellaSwag
OAL → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Linearizing softmax attention offers a route to efficient Transformer inference, but preserving pretrained attention remains challenging. We identify an important source of approximation error overlooked by standard linearization methods: a small number of outlier dimensions contribute disproportionately to query-key dot products and account for a large fraction of the approximation error. Motivated by this heterogeneity, we propose Outlier-Aware Attention Linearization (OAL), a dimension-aware method for constructing quadratic attention approximations. Starting from a second-order Taylor approximation, we introduce a Hadamard-product formulation that makes individual query-key contributions explicit. This formulation relaxes globally shared coefficients to support dimension-specific linear terms and cross-dimensional quadratic interactions while preserving separable evaluation. To retain this flexibility with fewer parameters, OAL groups dimensions with similar contribution scales to share coefficients, while allowing distinct treatment of the few outliers. The resulting positive quadratic kernel is calibrated against pretrained attention outputs and admits linear-time evaluation with respect to sequence length for fixed head dimensions. On Qwen2.5-0.5B, OAL reduces frozen-attention output relative MSE by approximately 8.9% compared with DiJiang. Replacing middle-layer attention modules and jointly fine-tuning the kernel coefficients with LoRA further lowers WikiText-103 perplexity from 31.71 to 27.50 compared with a learned global quadratic, while improving accuracy on PIQA, HellaSwag, and ARC-E. These results demonstrate the value of outlier-aware dimension grouping for both attention approximation and pretrained-model conversion. The code is available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2610.13674v1
- Canonical: https://arxiv.org/abs/2610.13674v1
Trouble viewing inline? Open PDF directly →
Full Text
78,104 characters extracted from source content.
Expand or collapse full text
Outlier-Aware Attention Linearization Sun, Qianyue Wu, Wenjia Wang, Tianyang Hu Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels Weiqi Sun †thanks: These authors contributed equally to this work. Affiliation: The Chinese University of Hong Kong, Shenzhen Qianyue Wu11footnotemark: 1 Affiliation: Beijing Normal University, Zhuhai Wenjia Wang †thanks: Corresponding author: wenjiawang@nus.edu.sg Affiliation: National University of Singapore Tianyang Hu †thanks: Corresponding author: hutianyang@cuhk.edu.cn Affiliation: The Chinese University of Hong Kong, Shenzhen Abstract Linearizing softmax attention offers a route to efficient Transformer inference, but preserving pretrained attention remains challenging. We identify an important source of approximation error overlooked by standard linearization methods: a small number of outlier dimensions contribute disproportionately to query–key dot products and account for a large fraction of the approximation error. Motivated by this heterogeneity, we propose Outlier-Aware Attention Linearization (OAL), a dimension-aware method for constructing quadratic attention approximations. Starting from a second-order Taylor approximation, we introduce a Hadamard-product formulation that makes individual query–key contributions explicit. This formulation relaxes globally shared coefficients to support dimension-specific linear terms and cross-dimensional quadratic interactions while preserving separable evaluation. To retain this flexibility with fewer parameters, OAL groups dimensions with similar contribution scales to share coefficients, while allowing distinct treatment of the few outliers. The resulting positive quadratic kernel is calibrated against pretrained attention outputs and admits linear-time evaluation with respect to sequence length for fixed head dimensions. On Qwen2.5-0.5B, OAL reduces frozen-attention output relative MSE by approximately 8.9%8.9\% compared with DiJiang. Replacing middle-layer attention modules and jointly fine-tuning the kernel coefficients with LoRA further lowers WikiText-103 perplexity from 31.7131.71 to 27.5027.50 compared with a learned global quadratic, while improving accuracy on PIQA, HellaSwag, and ARC-E. These results demonstrate the value of outlier-aware dimension grouping for both attention approximation and pretrained-model conversion. The code is available at GitHub. 1 Introduction Softmax attention is central to modern Transformers, but computing all pairwise token interactions incurs quadratic computation as context length grows (Vaswani et al., 2017). Linear attention offers an alternative: separable query and key features permit recurrent evaluation with linear sequence-length complexity for a fixed feature dimension (Katharopoulos et al., 2020). Approximating softmax attention in pretrained Transformers with linear attention therefore offers a promising route to improving inference efficiency while reusing pretrained representations. Realizing this potential, however, requires approximations that closely preserve the original attention behavior, particularly when only limited adaptation is available to compensate for approximation errors. Existing methods approximate the softmax kernel using feature maps that separate query and key computations, or low-order polynomials that admit efficient evaluation (Choromanski et al., 2021; Nguyen et al., 2026; Chen et al., 2024b; Nauen et al., 2024). To better preserve pretrained attention, these approximations can also be learned or adapted to match the original model’s attention behavior (Chowdhury et al., 2022; Zhang et al., 2025; Lan et al., 2025). Despite these advances, approximation errors are still to be desired under the limited-adaptation setting. To better understand the remaining approximation error, we examine how it relates to the contributions of individual query–key dimensions. Our analysis reveals that a small subset of query–key dimensions contributes disproportionately to attention logits (Section 3). Diagnostic experiments further show that approximating these dimensions alone can induce substantial attention-output error. This suggests that the approximation should have sufficient flexibility to accommodate differences in dimension-wise contributions. The representative approaches discussed above generally formulate their approximations through vector-level feature mappings or functions of the aggregated query–key score. Their parameterization, however, is not explicitly organized around the unequal contributions of individual query–key dimensions. Our findings motivate making these contributions explicit and using their statistics to guide the approximation, so that outlier dimensions can be treated differently from the remaining dimensions. Therefore, we propose Outlier-Aware Attention Linearization (OAL), which explicitly adapts the approximation to dimension-wise heterogeneity in pretrained attention. We use a quadratic kernel as a tractable starting point for introducing dimension-specific parameters while retaining separable evaluation. To enable dimension-wise treatment, we introduce a Hadamard-product formulation that makes individual query–key contributions explicit. This formulation generalizes the globally shared coefficients in a quadratic approximation to dimension-specific linear coefficients and coefficients for cross-dimensional quadratic interactions. It thereby allows different dimensions to be approximated differently while preserving the separable structure required for linear-time evaluation. A fully dimension-specific parameterization, however, introduces many coefficients. The empirical distribution of dimension-wise contributions suggests that much of this flexibility can be shared. This observation motivates grouping dimensions with similar contributions and sharing coefficients within groups and across corresponding group pairs. The resulting parameterization accommodates outlier dimensions while keeping the number of learned coefficients small. With the groups fixed, we calibrate the resulting positive quadratic kernel against pretrained softmax attention outputs using frozen queries, keys, and values. On frozen Qwen2.5-0.5B attention states, OAL achieves the lowest error among the evaluated methods, reducing attention-output relative MSE by approximately 8.9%8.9\% compared with DiJiang (Chen et al., 2024b). Under a common middle-layer attention replacement protocol with matched LoRA adaptation (Hu et al., 2022), OAL outperforms the evaluated converted methods on WikiText-103 perplexity and all three downstream tasks on both Qwen2.5-0.5B and Llama-3.2-1B (Llama Team, 2024). Relative to the learned global quadratic, it lowers Qwen2.5-0.5B perplexity from 31.7131.71 to 27.5027.50, although a gap to the LoRA-adapted softmax reference remains on both backbones. OAL also requires only O(N)O(N) attention computation for a sequence of length N and fixed head dimensions, without explicitly constructing the N×N× N attention matrix. Our contributions are threefold: • We characterize dimension-wise heterogeneity in pretrained query–key interactions and its association with attention approximation error, motivating contribution-aware kernel design. • We develop a dimension-aware quadratic formulation with separable evaluation, and introduce contribution-based grouping to retain this flexibility with a compact parameterization. • We demonstrate improved frozen-attention fidelity and adapted model quality under a matched conversion protocol, and examine the role of group-aware flexibility through a diagnostic budget study. 2 Preliminaries Softmax Attention For a single head, let Q,K∈ℝN×dhQ,K ^N× d_h and V∈ℝN×dvV ^N× d_v denote the query, key, and value matrices, with sequence length N, query–key dimension dhd_h, and value dimension dvd_v. We write qi,ki,viq_i,k_i,v_i for their respective token vectors at position i. Scaled dot-product attention (Vaswani et al., 2017) is Attn(Q,K,V)=Softmax(QK⊤dh)V,Attn(Q,K,V)=Softmax ( QK d_h )V, with row-wise softmax. Direct evaluation costs O(N2(dh+dv))O(N^2(d_h+d_v)) operations; explicit storage of the score matrix requires O(N2)O(N^2) memory. Linear Attention A common class of linear attention methods replaces the pairwise attention kernel with a separable feature representation (Katharopoulos et al., 2020; Choromanski et al., 2021). Specifically, the softmax kernel can be approximated as exp(q⊤k/dh)≈κ(q,k)=ϕ(q)⊤ϕ(k) (q k/ d_h)≈κ(q,k)=φ(q) φ(k), where ϕ(⋅)∈ℝmφ(·) ^m denotes a feature map. Substituting this factorization into normalized causal attention yields y^i=(∑j≤ivjϕ(kj)⊤)ϕ(qi)ϕ(qi)⊤(∑j≤iϕ(kj)). y_i= ( _j≤ iv_jφ(k_j) )φ(q_i)φ(q_i) ( _j≤ iφ(k_j) ). (1) Here, y^i y_i denotes the approximate attention output at position i. The key–value and key statistics in parentheses can be accumulated incrementally over the causal prefix. This reassociation avoids explicitly constructing the N×N× N attention matrix and gives linear scaling with sequence length for a fixed feature dimension and memory size (Gu and Dao, 2023; Sun et al., 2023; Peng et al., 2023; Chen et al., 2024a). Different linearization methods therefore mainly differ in the choice or construction of the feature map ϕ(⋅)φ(·) and the resulting approximation to the softmax kernel. Softmax Kernel Approximation Separable softmax approximations can be obtained by constructing feature maps directly (Katharopoulos et al., 2020; Choromanski et al., 2021; Peng et al., 2021) or by expanding a low-order polynomial in the attention score into separable terms. We focus on the latter. Polynomial kernels based on low-order Taylor approximations have been explored in efficient attention and language modeling (Arora et al., 2024; Nauen et al., 2024). For a scalar attention score s, the second-order Taylor approximation is exp(s)≈1+s+12s2 (s)≈ 1+s+ 12s^2. Allowing its coefficients to vary gives the global quadratic p(s)=a+bs+cs2p(s)=a+bs+cs^2, where a, b, and c control the constant, linear, and quadratic terms. This is the starting point for our dimension-aware kernel. 3 Dimension-Wise Heterogeneity in Pretrained Attention Motivated by activation outliers in pretrained Transformers (Kovaleva et al., 2021; Dettmers et al., 2022; Sun et al., 2024; Li et al., 2026b; Li et al., 2026a), where a small number of activation or representation dimensions can attain unusually large magnitudes or exert disproportionate influence on model behavior, we examine whether query–key contributions are similarly concentrated in a few dimensions and whether this concentration is associated with attention-approximation error. (a) Query magnitude (b) Key magnitude (c) Dimension-wise contribution Figure 1: Dimension-wise heterogeneity in a representative Qwen2.5-0.5B attention head. (a) Mean absolute query magnitude across QK dimensions. (b) Mean absolute key magnitude across the corresponding KV head. (c) Contribution score χr _r defined in Eq. (2). Dimensions r=26r=26 and r=58r=58 are highlighted across panels. 3.1 Query–Key Outliers and Dimension-Wise Contributions Figure 1 shows the post-RoPE query and key magnitudes for a representative Qwen2.5-0.5B head. Since either magnitude alone does not directly measure a dimension’s contribution to attention logits, we analyze their coordinate-wise products for each layer–head pair, suppressing layer and head indices below. For calibration input p, let NpN_p denote its sequence length, and let qp,i,kp,j∈ℝdhq_p,i,k_p,j ^d_h denote the post-RoPE query and key vectors at positions i and j. Their r-th coordinates are denoted by qp,i,rq_p,i,r and kp,j,rk_p,j,r. We define the causal pair set as Ωp=(i,j):1≤j≤i≤Np _p=\(i,j):1≤ j≤ i≤ N_p\. We define the contribution of dimension r on input p and its equally weighted average over the P calibration inputs as χp,r=1|Ωp|∑(i,j)∈Ωp|qp,i,rkp,j,rdh|,χr=1P∑p=1Pχp,r. _p,r= 1| _p| _(i,j)∈ _p | q_p,i,rk_p,j,r d_h |, _r= 1P _p=1^P _p,r. (2) The contribution profile in Figure 1 is concentrated on a few dimensions. The highlighted dimensions, r=26r=26 and r=58r=58, have high contribution scores and correspond to prominent coordinates in the query or key representations. A similarly concentrated dimension-wise contribution profile is observed in GPT-2 Medium; additional cross-model evidence is provided in Appendix A. 3.2 Approximation Error Concentration on High-Contribution Dimensions We next test whether approximating only the high-contribution dimensions retains a substantial fraction of the full-head output error. For each evaluated attention head, we select a small subset S of high-contribution dimensions according to the pretrained contribution scores. We apply each approximation only to S while keeping the remaining dimensions exact. Let ℰE_S denote the resulting attention-output MSE and ℰfullE_full the error when the same method is applied to the full head. We report the retained-error ratio R=ℰ/ℰfullR_S=E_S/E_full. For the Gaussian controls, pretrained Q/K representations are replaced with i.i.d. (0,1)N(0,1) samples and the same contribution-based grouping and intervention procedures are applied. Further metric and control details are provided in Appendix A. Table 1: Approximation error concentration on high-contribution dimensions. The selected subsets account for approximately 5.5%5.5\% of the evaluated head dimensions in aggregate. We report the retained-error ratio R_S (%) for the high-contribution subsets and for Gaussian controls obtained by replacing pretrained Q/K with i.i.d. (0,1)N(0,1) samples while keeping the remaining protocol unchanged. Higher values indicate stronger error concentration. Method High-contrib. error (%) Gaussian control (%) Performer 37.8 4.7 SpectraFormer 58.1 18.1 DiJiang 64.8 19.4 TaylorShift 63.9 13.6 Approximating only about 5.5%5.5\% of the dimensions retains a substantial fraction of the full-head attention-output MSE across all evaluated methods, with retained-error ratios consistently above the corresponding Gaussian controls (Table 1). In a separate random-coordinate control with 1,000 cardinality-matched subsets sampled from the pretrained bulk dimensions, none of the null replicates exceeded the observed retained-error ratio for any method, yielding empirical p<0.001p<0.001 in all cases. These retained-error ratios suggest an association between high contribution scores and attention-output approximation error. Appendix A provides additional control analyses and stability analyses across held-out inputs and sequence lengths for the representative Qwen2.5-0.5B head. These observations motivate the contribution-aware parameter sharing in Section 4, which allows dimensions with different contribution scales to use different approximation coefficients. 4 Outlier-Aware Attention Linearization We introduce dimension-specific linear and cross-dimensional quadratic coefficients, with a Hadamard–diagonal (HD) formulation enabling exact kernel evaluation in linear time with respect to sequence length for fixed head dimensions. Contribution-based grouping then shares coefficients across dimensions and dimension pairs; enforcing positive kernel weights yields OAL. 4.1 Dimension-Aware Linearization For a query–key pair q,k∈ℝdhq,k ^d_h, let xr=qrkr/dhx_r=q_rk_r/ d_h denote the contribution of dimension r, so that the attention score is s=∑rxrs= _rx_r. A global quadratic approximation to ese^s can be written as p(s)=a+bs+cs2=a+∑r=1dhbxr+∑r=1dh∑t=1dhcxrxt.p(s)=a+bs+cs^2=a+ _r=1^d_hbx_r+ _r=1^d_h _t=1^d_hcx_rx_t. All linear contributions therefore share b, and all quadratic interactions share c. TaylorShift uses this scalar polynomial structure with coefficients prescribed by the Taylor expansion (Nauen et al., 2024). To accommodate the dimension-wise heterogeneity observed in Section 3, we relax this sharing and allow separate coefficients for each dimension and dimension pair: κdim(q,k)=a+∑r=1dhbrxr+∑r=1dh∑t=1dhCrtxrxt,C=C⊤. _dim(q,k)=a+ _r=1^d_hb_rx_r+ _r=1^d_h _t=1^d_hC_rtx_rx_t, C=C . (3) Here, r and t index the dhd_h query–key dimensions; brb_r weights the contribution of dimension r, CrrC_r weights its squared term, and the off-diagonal entries of C weight cross-dimensional interactions. Setting br=b_r=b and Crt=cC_rt=c recovers the global quadratic. Degree two is the lowest degree that retains both squared terms and cross-dimensional interactions, using O(dh2)O(d_h^2) features. Hadamard-diagonal formulation Evaluating the quadratic kernel explicitly over all query–key pairs still incurs quadratic cost in sequence length. To avoid this pairwise computation, we use the following Hadamard–diagonal (HD) identity for the shared-coefficient quadratic term. For value channel V:,ℓV_:, , let Dℓ∈ℝN×ND_ ^N× N be the diagonal matrix with entries (Dℓ)jj=Vj,ℓ(D_ )_j=V_j, . Then [(QK⊤)⊙(QK⊤)]V:,ℓ=diag(Q[K⊤DℓK]Q⊤). [(QK ) (QK ) ]V_:, =diag\! (Q [K D_ K ]Q ). (4) Here, ⊙ denotes the Hadamard product, and diagdiag extracts the diagonal entries of a matrix as a vector. By first accumulating K⊤DℓK D_ K and then evaluating only the diagonal entries, the quadratic term can be computed in O(Ndh2)O(Nd_h^2) time per value channel without forming an N×N× N matrix. For fixed head dimensions, this cost is linear in sequence length. The (r,t)(r,t) entry of K⊤DℓK D_ K is ∑jVj,ℓkj,rkj,t _jV_j, k_j,rk_j,t, which sums value-weighted key products over token positions j. For a query at position i, restricting this sum to j≤ij≤ i gives causal prefix statistics. The attention denominator uses the same kernel weights as the numerator, but replaces each value with 11. Thus, augmenting the values as uj=[vj;1]u_j=[v_j;1] allows both to be accumulated together. Including the constant and linear kernel terms gives the states H0,i=∑j≤iuj,H1,i,r=∑j≤ikj,ruj,H2,i,rt=∑j≤ikj,rkj,tuj.H_0,i= _j≤ iu_j, H_1,i,r= _j≤ ik_j,ru_j, H_2,i,rt= _j≤ ik_j,rk_j,tu_j. (5) Each H2,i,rtH_2,i,rt is a (dv+1)(d_v+1)-dimensional vector. The query combines these statistics using the coefficients a, brb_r, and CrtC_rt from Eq. (3): u~i u_i =aH0,i+1dh∑rbrqi,rH1,i,r+1dh∑r,tCrtqi,rqi,tH2,i,rt, =aH_0,i+ 1 d_h _rb_rq_i,rH_1,i,r+ 1d_h _r,tC_rtq_i,rq_i,tH_2,i,rt, (6) y^i y_i =u~i,1:dv/u~i,dv+1. = u_i,1:d_v/ u_i,d_v+1. This recurrence evaluates the specified quadratic kernel exactly without forming a token-by-token matrix (Appendix D). The last channel supplies the normalization denominator, whose positivity follows from the OAL parameterization in Section 4.3. A fully dimension-specific quadratic has O(dh2)O(d_h^2) coefficients. Contribution-based sharing reduces this parameter count while retaining the same states and readout. 4.2 Outlier-Aware Dimension Grouping The contribution profile in Figure 1 suggests a compact sharing structure: low-contribution dimensions can share parameters, while the high-contribution tail may benefit from finer parameterization. Figure 2(a) illustrates a partition for the representative head in Section 3.1. For a partition gg=1G\G_g\_g=1^G of the dimensions, recall that xr=qrkr/dhx_r=q_rk_r/ d_h as the contribution of dimension r and sg=∑r∈gxrs_g= _r _gx_r as the total contribution of group g. Thus, s=∑g=1Gsgs= _g=1^Gs_g. Sharing brb_r within each group and CrtC_rt within each group pair reduces the number of learned coefficients to O(G2)O(G^2) while preserving the HD evaluation. (a) Representative group assignment. (b) Group-budget trade-off. Figure 2: Contribution-aware dimension grouping. (a) Sorted contribution scores χr _r and the resulting OAL groups for the representative Qwen2.5-0.5B Layer 4, Head 0. (b) Attention-output MSE and trainable coefficients per head under fixed group budgets for the same head. All budgets use the same data splits and coefficient-fitting protocol. For each layer and query head, we sort dimensions by their calibration contribution scores χr _r and identify large gaps in the high-contribution tail that remain stable under prompt-level bootstrap resampling, following resampling-based stability assessment (Suzuki and Shimodaira, 2006; Hennig, 2007). These boundaries determine the group count, with at least two groups. Groups remain fixed during coefficient fitting and model adaptation. The procedure yields 22–55 groups per head for Qwen2.5-0.5B and 22–44 for Llama-3.2-1B; full details are provided in Appendix B. As a separate diagnostic, Figure 2(b) illustrates the trade-off between attention-output MSE and parameter count across fixed group budgets for the representative head. For this head, a small number of groups captures most of the error reduction, with diminishing gains as the budget increases. 4.3 The OAL Algorithm OAL uses a positive quadratic kernel whose coefficients are shared according to contribution-based dimension groups. We specify this parameterization below and combine it with the HD formulation to summarize the complete method in Algorithm 1. Positive group-aware kernel Let g(r)g(r) denote the group containing dimension r. To enforce positive kernel weights while sharing coefficients within groups and group pairs, we introduce a learnable lower-triangular matrix L∈ℝ(G+1)×(G+1)L ^(G+1)×(G+1) for each layer and query head and set M=LL⊤M=L . The coefficients are a=ϵ+M00a=ε+M_00, br=2M0,g(r)b_r=2M_0,g(r), and Crt=Mg(r),g(t)C_rt=M_g(r),g(t), with ϵ>0ε>0. With ¯=[1,s1,…,sG]⊤ s=[1,s_1,…,s_G] , this parameterization gives κOAL(q,k)=ϵ+¯⊤LL⊤¯,ϵ>0, _OAL(q,k)=ε+ s L s, ε>0, (7) Since LL⊤⪰0L 0, we have κOAL(q,k)≥ϵ>0 _OAL(q,k)≥ε>0, ensuring a positive normalization denominator. With the groups fixed, we learn L by minimizing the squared error between normalized OAL and softmax attention outputs on an offline calibration set, using frozen pretrained queries, keys, and values. Calibration details are provided in Appendix B. The calibrated factors initialize subsequent joint adaptation with LoRA, during which the groups remain fixed. Causal evaluation The coefficient mapping above expresses the grouped kernel in the dimension-wise form of Eq. (3), so it can be evaluated using the HD states and readout in Eqs. (5) and (6). This computation is exact for the specified quadratic kernel; approximation arises only from replacing the softmax kernel. Algorithm 1 summarizes the offline calibration and causal evaluation stages. The equivalence proof is provided in Appendix D. Algorithm 1 Outlier-Aware Attention Linearization (OAL) 0: Calibration data and pretrained attention states; evaluation inputs Q,K,VQ,K,V; ϵ>0ε>0 1: Offline calibration (per layer and query head) 2: Determine group map g from contribution statistics (Section 4.2) 3: Fit L by output matching with g and pretrained states fixed (Eq. (20)) 4: Causal evaluation (fixed g,Lg,L) 5: Compute a,br,Crta,b_r,C_rt from L,g,ϵL,g,ε; initialize H0,H1,H2=0H_0,H_1,H_2=0 6: for i=1,…,Ni=1,…,N do 7: ui←[vi;1]u_i←[v_i;1]; H0←H0+uiH_0← H_0+u_i 8: H1,r←H1,r+ki,ruiH_1,r← H_1,r+k_i,ru_i for all r 9: H2,rt←H2,rt+ki,rki,tuiH_2,rt← H_2,rt+k_i,rk_i,tu_i for all r,tr,t 10: Compute y^i y_i using Eq. (6) 11: end for 12: return Y^=[y^1,…,y^N]⊤ Y=[ y_1,…, y_N] 4.4 Complexity Analysis Direct pairwise evaluation costs O(N2(dh+G2+dv))O(N^2(d_h+G^2+d_v)), whereas the causal evaluation stage of Algorithm 1 costs O(Ndh2(dv+1))O(Nd_h^2(d_v+1)). The HD states require O(dh2(dv+1))O(d_h^2(d_v+1)) space, excluding inputs and outputs. For fixed head dimensions, computation is linear in sequence length and state space is independent of it. Grouping reduces the learned parameter count to O(G2)O(G^2), while the implementation retains dimension-pair states. Detailed counts appear in Appendix D. Figure 3 reports causal attention forward latency and peak GPU memory under the setup in Appendix E. OAL exhibits approximately linear runtime growth at longer sequences. With the same HD block-GEMM backend, OAL (G=5G=5) achieves latency and peak memory comparable to the single-group Learned Global baseline (G=1G=1), suggesting that the additional coefficient flexibility introduces little operator overhead in this configuration. Cross-method comparisons reflect the evaluated implementations, which differ in numerical precision and optimization level. FlashAttention remains faster and more memory-efficient throughout the tested range. Figure 3: Causal attention forward runtime and memory. Forward-only measurements use an A100-SXM4-40GB with BF16 inputs. OAL (G=5G=5) and Learned Global (G=1G=1) share the HD block-GEMM backend. Memory denotes total peak PyTorch-allocated tensor memory. SpectraFormer and SDPA (math backend) are capped at N=8192N=8192 by a pre-run memory guard. Implementation and measurement details appear in Appendix E. 5 Experiments We first evaluate OAL’s approximation fidelity on frozen pretrained attention states. We then examine whether OAL supports better language-modeling and downstream performance after attention replacement under a common, limited LoRA adaptation budget. 5.1 Frozen-Attention Approximation Fidelity To isolate approximation error before model adaptation, we compare OAL and existing attention linearization methods on identical frozen queries, keys, and values from Qwen2.5-0.5B Base, with queries and keys taken after RoPE. The evaluation covers all query heads in Layers 4, 12, and 20 at sequence length N=4096N=4096. Approximation error is measured by MSE and relative MSE (RelMSE) against the corresponding softmax outputs, both before and after the output projection WoW_o. Exact metric definitions and aggregation procedures are provided in Appendix C. Table 2: Frozen-attention approximation fidelity on Qwen2.5-0.5B. Results cover all query heads in Layers 4, 12, and 20 at sequence length 40964096. Lower is better for all metrics. Method Before WoW_o MSE ↓ Before WoW_o RelMSE ↓ After WoW_o MSE ↓ After WoW_o RelMSE ↓ Performer 0.103 0.569 0.038 0.710 SpectraFormer 0.106 0.585 0.041 0.772 DiJiang 0.086 0.474 0.033 0.631 TaylorShift 0.088 0.490 0.034 0.649 OAL 0.078 0.432 0.031 0.595 OAL achieves the lowest MSE and RelMSE both before and after the output projection WoW_o (Table 2). DiJiang is the strongest evaluated baseline under all four metrics. 5.2 End-to-End Conversion and Downstream Performance The conversion experiments use Qwen2.5-0.5B Base and Llama-3.2-1B Base. We compare OAL with a softmax reference, a learned global quadratic, Performer, SpectraFormer, DiJiang, and TaylorShift. The learned global quadratic is the single-group (G=1G=1) instance of OAL and shares its calibration and adaptation protocol, providing a controlled comparison of global and contribution-based coefficient sharing. Following the hybrid conversion setting explored in LoLCATs (Zhang et al., 2025), we retain softmax attention in the initial and final layers. Our layer-wise activation analysis motivates converting the intervening layers, where large activations persist across depth (Appendix F). All converted methods use the same replacement range within each backbone. All methods, including the softmax reference, use the same LoRA configurations, training data, and update budgets within each backbone. Adaptation uses next-token prediction on WikiText-103. For OAL, the calibrated kernel coefficients are jointly optimized with the LoRA parameters, while the dimension groups remain fixed. We adapt Performer, SpectraFormer, DiJiang, and TaylorShift from their official source code; modifications are detailed in Appendix F.1, and shared conversion and adaptation settings in Appendix F. Table 3 reports language-modeling and downstream results for both backbones. Evaluation protocols are detailed in Appendix F.3. Table 3: Model-level quality after LoRA adaptation on two backbones. We report WikiText-103 test PPL (lower is better) and zero-shot normalized accuracy (%, higher is better). Bold highlights the leading results among converted methods within each backbone. Qwen2.5-0.5B Llama-3.2-1B Method PPL PIQA HellaSwag ARC-E PPL PIQA HellaSwag ARC-E Softmax (LoRA) 10.89 70.18 51.38 62.63 8.19 74.92 62.84 59.43 Performer 37.03 57.78 30.37 38.93 50.39 57.07 27.42 32.58 SpectraFormer 34.78 59.19 31.77 41.84 45.09 59.36 30.41 36.49 DiJiang 36.05 59.90 31.47 41.41 56.27 58.92 28.75 36.32 TaylorShift 39.50 59.90 31.18 40.78 62.67 57.40 28.95 35.77 Learned global 31.71 61.15 32.43 41.41 36.03 57.40 35.02 33.46 OAL 27.50 64.69 37.20 50.38 32.99 63.66 35.42 41.67 OAL achieves the lowest WikiText-103 perplexity and highest normalized accuracy on all three downstream tasks among the evaluated converted models on both backbones (Table 3). The LoRA-adapted softmax reference remains stronger on every metric, indicating a substantial conversion gap. Relative to the learned global quadratic (G=1G=1), OAL reduces perplexity from 31.7131.71 to 27.5027.50 on Qwen2.5-0.5B and from 36.0336.03 to 32.9932.99 on Llama-3.2-1B. These gains support the effectiveness of contribution-based grouping over globally shared coefficients under the evaluated protocol. 6 Related Work Kernel Linearization and Pretrained-Model Conversion Kernel methods such as Performer, SpectraFormer, and DiJiang approximate softmax attention with feature representations that permit linear-time evaluation (Choromanski et al., 2021; Nguyen et al., 2026; Chen et al., 2024b). T2R converts pretrained Transformers through attention replacement and finetuning (Kasai et al., 2021). Hedgehog learns MLP feature maps by matching softmax attention weights (Zhang et al., 2024), while LoLCATs combines attention transfer with low-rank adaptation (Zhang et al., 2025). OAL shares the goal of preserving pretrained behavior, but organizes kernel coefficients by dimension-wise contribution, providing a structured alternative to learned MLP feature maps. Our comparison scope is detailed in Appendix F.1. Polynomial Attention In language models trained from scratch, Based uses a second-order Taylor kernel (Arora et al., 2024), while ReBased modifies the quadratic kernel with query/key normalization and learnable affine transformations (Aksenov et al., 2024). PolySketchFormer approximates higher-degree polynomial attention through sketching (Kacham et al., 2024). OAL targets pretrained softmax recovery under limited adaptation. It retains original coordinate products and learns coefficients for linear terms and quadratic interactions, with sharing determined by contribution statistics. Activation and Attention Outliers Massive activations and attention sinks reveal different forms of nonuniformity in pretrained Transformers: unusually large hidden activations and persistent attention to particular positions, respectively (Sun et al., 2024; Xiao et al., 2024; Li et al., 2026a; Li et al., 2026b). Systematic Outliers examines how alignment in specific query and key dimensions produces large attention weights (An et al., 2025). Building on these observations, we examine how concentrated query–key contributions relate to attention-linearization error, using this association to guide coefficient sharing. 7 Discussion We introduce Outlier-Aware Attention Linearization (OAL), a contribution-aware method for linearizing pretrained softmax attention. Our analysis shows that approximation error is concentrated on a small subset of high-contribution query–key dimensions, motivating non-uniform parameter sharing. OAL captures this structure with a compact grouped quadratic kernel and supports exact recurrent evaluation with linear sequence-length complexity. It improves frozen-attention fidelity on Qwen2.5-0.5B and model-level performance after LoRA adaptation on both Qwen2.5-0.5B and Llama-3.2-1B. Our experiments are limited to 0.5B–1B backbones, so performance at larger scales remains unverified. Future work will evaluate OAL on larger and longer-context models and explore more compact implementations of the quadratic recurrent state. AI Use Statement We used generative AI tools to aid and polish writing, including language editing and improving the clarity and organization of the manuscript. We also used generative AI tools for retrieval and discovery, such as finding related work and assisting with literature search. All AI-assisted content was reviewed and verified by the authors, who take full responsibility for the final manuscript. References Aksenov et al. (2024) Y. Aksenov, N. Balagansky, S. M. Lo Cicero Vaina, B. Shaposhnikov, A. Gorbatovski, and D. Gavrilov Linear transformers with learnable kernel functions are better in-context models. arXiv preprint arXiv:2402.10644. External Links: Link Cited by: §6. An et al. (2025) Y. An, X. Zhao, T. Yu, M. Tang, and J. Wang Systematic outliers in large language models. In International Conference on Learning Representations, External Links: Link Cited by: §6. Arora et al. (2024) S. Arora, S. Eyuboglu, M. Zhang, A. Timalsina, S. Alberti, D. Zinsley, J. Zou, A. Rudra, and C. Ré Simple linear attention language models balance the recall-throughput tradeoff. arXiv preprint arXiv:2402.18668. External Links: Link Cited by: §2, §6. Chen et al. (2024a) B. K. Chen, T. Hu, H. Jin, H. K. Lee, and K. Kawaguchi Exact conversion of in-context learning to model weights in linearized-attention transformers. arXiv preprint arXiv:2406.02847. Cited by: §2. Chen et al. (2024b) H. Chen, Z. Liu, X. Wang, Y. Tian, and Y. Wang DiJiang: efficient large language models through compact kernelization. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 7103–7117. External Links: Link Cited by: §1, §1, §6. Choromanski et al. (2021) K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, D. Belanger, L. Colwell, and A. Weller Rethinking attention with performers. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §2, §6. Chowdhury et al. (2022) S. P. Chowdhury, A. Solomou, K. A. Dubey, and M. Sachan Learning the transformer kernel. Transactions on Machine Learning Research. External Links: Link Cited by: §1. Dettmers et al. (2022) T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, Vol. 35, p. 30318–30332. External Links: Link Cited by: §3. Gu and Dao (2023) A. Gu and T. Dao Mamba: linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. Cited by: §2. Hennig (2007) C. Hennig Cluster-wise assessment of cluster stability. Computational Statistics & Data Analysis 52 (1), p. 258–271. External Links: Document Cited by: §4.2. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §1. Kacham et al. (2024) P. Kacham, V. Mirrokni, and P. Zhong PolySketchFormer: fast transformers via sketching polynomial kernels. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 22748–22770. External Links: Link Cited by: §6. Kasai et al. (2021) J. Kasai, H. Peng, Y. Zhang, D. Yogatama, G. Ilharco, N. Pappas, Y. Mao, W. Chen, and N. A. Smith Finetuning pretrained transformers into RNNs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 10630–10643. External Links: Document, Link Cited by: §6. Katharopoulos et al. (2020) A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret Transformers are RNNs: fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, p. 5156–5165. External Links: Link Cited by: §1, §2, §2. Kovaleva et al. (2021) O. Kovaleva, S. Kulshreshtha, A. Rogers, and A. Rumshisky BERT busters: outlier dimensions that disrupt transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, p. 3392–3405. External Links: Document, Link Cited by: §3. Lan et al. (2025) D. Lan, W. Sun, J. Hu, J. Du, and Y. Cheng Liger: linearizing large language models to gated recurrent structures. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 32452–32466. External Links: Link Cited by: §1. Li et al. (2026a) S. Li, K. Jiang, J. Sun, and T. Hu The structural origin of attention sink: variance discrepancy, super neurons, and dimension disparity. arXiv preprint arXiv:2605.06611. Cited by: §3, §6. Li et al. (2026b) S. Li, Y. Tong, H. Wang, and T. Hu Transformers are born biased: structural inductive biases at random initialization and their practical consequences. arXiv preprint arXiv:2602.05927. Cited by: §3, §6. Llama Team (2024) Llama Team The llama 3 herd of models. arXiv preprint arXiv:2407.21783. External Links: Link Cited by: §1. Nauen et al. (2024) T. C. Nauen, S. Palacio, and A. Dengel TaylorShift: shifting the complexity of self-attention from squared to linear (and back) using taylor-softmax. In Pattern Recognition, Cham, p. 1–16. Note: Proceedings of the 27th International Conference on Pattern Recognition (ICPR 2024) External Links: Document Cited by: §1, §2, §4.1. Nguyen et al. (2026) D. Nguyen, D. Yin, A. Joshi, and F. Salim Spectraformer: a unified random feature framework for transformer. ACM Transactions on Intelligent Systems and Technology 17 (3). External Links: Document, Link Cited by: §1, §6. Peng et al. (2023) B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, L. Derczynski, et al. Rwkv: reinventing rnns for the transformer era. In Findings of the association for computational linguistics: EMNLP 2023, p. 14048–14077. Cited by: §2. Peng et al. (2021) H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. A. Smith, and L. Kong Random feature attention. In International Conference on Learning Representations, External Links: Link Cited by: §2. Sun et al. (2024) M. Sun, X. Chen, J. Z. Kolter, and Z. Liu Massive activations in large language models. arXiv preprint arXiv:2402.17762. External Links: Link Cited by: §3, §6. Sun et al. (2023) Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei Retentive network: a successor to transformer for large language models. arXiv preprint arXiv:2307.08621. Cited by: §2. Suzuki and Shimodaira (2006) R. Suzuki and H. Shimodaira Pvclust: an r package for assessing the uncertainty in hierarchical clustering. Bioinformatics 22 (12), p. 1540–1542. External Links: Document Cited by: §4.2. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30. External Links: Link Cited by: §1, §2. Xiao et al. (2024) G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §6. Zhang et al. (2025) M. Zhang, S. Arora, R. Chalamala, B. Spector, A. Wu, K. Ramesh, A. Singhal, and C. Re LoLCATs: on low-rank linearizing of large language models. In International Conference on Learning Representations, p. 46200–46253. External Links: Link Cited by: §1, §5.2, §6. Zhang et al. (2024) M. Zhang, K. Bhatia, H. Kumbong, and C. Ré The hedgehog & the porcupine: expressive linear attentions with softmax mimicry. In International Conference on Learning Representations, External Links: Link Cited by: §F.1, §6. Appendix A Additional Analysis of Query–Key Heterogeneity This section provides additional evidence for the dimension-wise heterogeneity discussed in Section 3.1. We first examine the same phenomenon in GPT-2 Medium and under isotropic Gaussian controls, and then evaluate the stability of the identified high-contribution structure across inputs and sequence lengths. A.1 Cross-Model and Control Evidence GPT-2 Medium We repeat the dimension-wise analysis on GPT-2 Medium to examine whether the heterogeneity observed in Qwen2.5-0.5B is model-specific. As shown in Figure 4, GPT-2 also exhibits nonuniform query/key magnitudes and a strongly concentrated contribution profile, although the dominant contribution is distributed across a broader set of dimensions. (a) Query magnitude (b) Key magnitude (c) Dimension-wise contribution Figure 4: Dimension-wise heterogeneity in a representative GPT-2 Medium attention head. GPT-2 exhibits the same qualitative relationship between nonuniform query/key representations and concentrated query–key contributions as observed in Qwen2.5-0.5B, although the dominant contribution is distributed across a broader set of dimensions. Gaussian controls The Gaussian-control column in Table 1 uses a synthetic counterpart of the pretrained representations. Specifically, we replace the pretrained post-RoPE query and key representations with i.i.d. standard Gaussian samples drawn from (0,1)N(0,1), while keeping the remaining experimental protocol unchanged. We compute the same contribution statistics, apply the same grouping procedure to identify the corresponding high-contribution dimensions, and use the same approximation, intervention, and aggregation procedures as in the pretrained experiment. Thus, the Gaussian control changes only the distribution of the query and key representations. The statistical significance reported in Section 3.2 is based on a separate matched-control randomization test on the original pretrained representations, described in Appendix A.2; it is not computed from the Gaussian controls. As an additional diagnostic, for the representative Qwen2.5-0.5B attention head analyzed in Figure 5, we compare the top-5 contribution concentration of the pretrained representations with isotropic Gaussian controls of the same dimensionality. The Gaussian controls use the same head dimension and the same contribution statistic χr _r, but replace the pretrained query–key representations with i.i.d. standard Gaussian samples drawn from (0,1)N(0,1). Figure 5: Top-5 contribution concentration. The pretrained Qwen2.5-0.5B representation exhibits substantially stronger concentration than either isotropic Gaussian control. The Gaussian controls indicate that the observed concentration is not explained solely by dimensionality or by selecting particular coordinate indices. A.2 High-Contribution-Dimension Intervention We further specify the intervention used in Table 1. For each evaluated query head h, let h,1G_h,1 denote the bulk group produced by the fixed OAL grouping procedure, and let GhG_h denote the total number of groups. We define the high-contribution subset as h=⋃g=2Ghh,g,S_h= _g=2^G_hG_h,g, that is, the union of all non-bulk groups for that head. The subset is determined independently for each layer–query-head pair using only the calibration contribution statistics. Across the 42 evaluated query heads, the selected subsets contain 147 dimensions in total. Since dh=64d_h=64, their aggregate fraction is ∑h|h|42dh=14742×64≈5.47%, _h|S_h|42d_h= 14742× 64≈ 5.47\%, which gives the approximately 5.5%5.5\% reported in Table 1. Let ¯h S_h denote the complement of hS_h. For a causal query–key pair (i,j)(i,j), the exact softmax kernel factorizes as κijexact=exp(qi⊤kjdh)=κij,hexactκij,¯hexact, _ij^exact= ( q_i k_j d_h )= _ij,S_h^exact _ij, S_h^exact, (8) where κij,¯hexact=exp(∑r∉hqi,rkj,rdh). _ij, S_h^exact= ( _r _hq_i,rk_j,r d_h ). (9) To measure the error associated with approximating only hS_h, we mask the query and key coordinates outside hS_h and apply the corresponding baseline approximation to the masked vectors, retaining the kernel’s input preprocessing. The approximate factor for the selected dimensions and the exact factor for their complement are computed separately. Denoting the resulting sub-kernel by κ^ij,h(m) κ_ij,S_h^(m) for method m, we form the full-head hybrid kernel κ~ij(m,h)=κ^ij,h(m)κij,¯hexact. κ_ij^(m,S_h)= κ_ij,S_h^(m) _ij, S_h^exact. (10) Thus, dimensions outside hS_h remain exact at the unnormalized-kernel level. The hybrid kernel is then causally masked, normalized over the key positions, and applied to the pretrained values: y~i(m,h)=∑j≤iκ~ij(m,h)vj∑j≤iκ~ij(m,h). y_i^(m,S_h)= _j≤ i κ_ij^(m,S_h)v_j _j≤ i κ_ij^(m,S_h). (11) The tilde distinguishes this hybrid output from a full-head approximation. The same intervention is used for all evaluated approximation methods; the methods differ only in the construction of κ^ij,h(m) κ_ij,S_h^(m). Error metric The intervention is evaluated at the attention-output level rather than on kernel values or attention probabilities. Let YsoftmaxY^softmax denote the exact causal softmax attention output. For an evaluated input and layer, we define ℰ(m) _S^(m) =MSE(Y~(m,h),Ysoftmax), =MSE\! ( Y^(m,S_h),Y^softmax ), (12) ℰfull(m) _full^(m) =MSE(Y^(m,full),Ysoftmax), =MSE\! ( Y^(m,full),Y^softmax ), (13) where the MSE is averaged over the evaluated query heads, query positions, and output-head dimensions. We report the retained-error ratio R(m)=ℰ(m)ℰfull(m).R_S^(m)= E_S^(m)E_full^(m). (14) A larger value therefore indicates that restricting the approximation to the selected dimensions preserves a larger fraction of the full-method attention-output error. Evaluation protocol We evaluate Qwen2.5-0.5B on zero-based Layers 4, 12, and 20, using all query heads in each layer. Evaluation uses 16 disjoint 4096-token WikiText-2 sequences, with 32 causal query positions sampled per sequence and layer. For stochastic approximation methods, we use five random seeds, 0,1,2,3,4\0,1,2,3,4\. The selected-subset and full-method MSEs are first averaged over these seeds before forming the retained-error ratio. TaylorShift is deterministic and is evaluated once. The resulting ratios are then macro-averaged across the evaluated layers and input sequences. Matched-control randomization test To assess whether the retained-error ratio of the high-contribution subset can be explained by selecting an equally sized subset of ordinary dimensions, we construct B=1000B=1000 matched control subsets. For each control, the same intervention and aggregation procedure is used to obtain a retained-error ratio RbcontrolR_b^control. We perform a one-sided Monte Carlo randomization test and report p=1+∑b=1B[Rbcontrol≥R]B+1.p= 1+ _b=1^B1 [R_b^control≥ R_S ]B+1. (15) The +1+1 correction avoids reporting a zero Monte Carlo p-value; with B=1000B=1000, the minimum reportable value is 1/1001≈0.0009991/1001≈ 0.000999. A.3 Stability of Dimension-Wise Contributions Across inputs We first examine whether the contribution profile depends strongly on the calibration inputs. Figure 6 compares the dimension-wise contribution scores computed from calibration and held-out inputs for the representative Qwen2.5-0.5B attention head. The two profiles are highly consistent, with Spearman correlation ρ=0.991ρ=0.991 and identical top-5 dimensions. Figure 6: Stability of dimension-wise contributions across inputs. Contribution scores computed from calibration and held-out inputs achieve Spearman correlation ρ=0.991ρ=0.991, with a top-5 overlap of 5/55/5. Across sequence lengths We further evaluate the same representative head using prefixes of a fixed sequence at different lengths. Table 4 reports the normalized contributions of the five dimensions identified at sequence length 40964096. Table 4: Stability across sequence lengths. Normalized contribution χr/∑tχt _r/ _t _t (%) of the five highest-contribution dimensions identified at sequence length 40964096 for the representative attention head. Length r=26r=26 r=58r=58 r=19r=19 r=39r=39 r=7r=7 128 33.34 13.64 2.12 3.30 3.56 512 32.96 15.70 1.27 3.66 3.86 1024 32.19 15.91 1.27 3.78 3.86 2048 30.93 16.18 2.10 3.75 3.80 4096 28.75 17.53 4.11 3.71 3.66 The two dominant dimensions, r=26r=26 and r=58r=58, remain the largest contributors across all evaluated sequence lengths and jointly account for approximately 4646–49%49\% of the total contribution. Together with the held-out-input analysis, these results indicate that the dominant dimension-wise structure is stable under changes in both input content and sequence length. Appendix B Grouping and Coefficient-Calibration Details Dimension grouping For a fixed layer and query head, we compute the dimension-wise contribution scores χr _r from the calibration set as defined in Eq. (2). The queries and keys are the post-RoPE representations used by the attention module, causal query–key pairs are used throughout, and query heads follow the model’s native GQA mapping to key–value heads. Different prompts are aggregated independently and then averaged with equal weight. We sort the dhd_h dimensions by increasing contribution, χπ1≤χπ2≤⋯≤χπdh, _ _1≤ _ _2≤·s≤ _ _d_h, using a stable ordering, and search for group boundaries only in the high-contribution tail. Given a tail fraction α, let m=⌈αdh⌉.m= α d_h . Candidate cuts are restricted to k∈max(1,dh−m),…,dh−1.k∈ \ (1,d_h-m),…,d_h-1 \. A cut at position k separates ℒk=π1,…,πk,k=πk+1,…,πdh.L_k=\ _1,…, _k\, _k=\ _k+1,…, _d_h\. Thus, the cut immediately preceding the top-m tail is included, while the endpoint k=dhk=d_h is excluded so that both sides of every candidate cut are nonempty. For each candidate cut, we first measure the adjacent contribution gap using ρk=χπk+1max(χπk,ε), _k= _ _k+1 ( _ _k, ), (16) where ε is a small numerical constant used only to avoid division by zero. A large ρk _k indicates a sharp increase in contribution across the candidate boundary. To test whether this separation is stable with respect to the calibration inputs, we perform B prompt-level bootstrap replicates. For replicate b, let Ib=(i1(b),…,iP(b))I_b=(i_1^(b),…,i_P^(b)) denote the length-P sequence of prompt indices sampled independently with replacement from 1,…,P\1,…,P\. Repeated indices are retained with their sampling multiplicities. We then recompute the contribution scores as χr(b)=1P∑t=1Pχit(b),r. _r^(b)= 1P _t=1^P _i_t^(b),r. Importantly, the dimension ordering π obtained from the full calibration set is kept fixed across bootstrap replicates. For candidate cut k, we measure the separation between the two sides as Rk(b)=minr∈kχr(b)max(maxr∈ℒkχr(b),ε).R_k^(b)= _r _k _r^(b) \! ( _r _k _r^(b), ). (17) Its bootstrap support is then Sk=1B∑b=1B[Rk(b)≥τ],S_k= 1B _b=1^B1 [R_k^(b)≥τ ], (18) which measures the fraction of bootstrap samples for which the two sides remain separated by at least a factor of τ. With minimum bootstrap support γ, a candidate boundary is retained as a natural cut when ρk≥τandSk≥γ. _k≥τ S_k≥γ. All natural cuts satisfying these conditions are retained; we do not apply an additional minimum group size, boundary-spacing constraint, or maximum group-count cap. To prevent the grouped parameterization from collapsing to the global single-group case, we require at least GminG_ groups per head. If the natural cuts yield fewer than GminG_ groups, unused candidate cuts are added in descending lexicographic order of (Sk,ρk)(S_k, _k) until the minimum group count is reached. Bootstrap support is therefore prioritized over the full-sample gap magnitude; exact ties retain the original candidate-cut order. These fallback cuts are restricted to the same high-contribution tail but need not satisfy the natural-cut thresholds. Let the final retained cuts be k1<⋯<kG−1k_1<·s<k_G-1, with k0=0k_0=0 and kG=dhk_G=d_h. The resulting groups are g=πkg−1+1,…,πkg,g=1,…,G.G_g=\ _k_g-1+1,…, _k_g\, g=1,…,G. (19) Thus, the first group typically forms a large low-contribution shared bulk, while subsequent groups refine the high-contribution tail. Grouping is performed independently for every layer and query head, and the resulting assignments are mapped back to the original dimension indices. The groups are determined before kernel-coefficient fitting and remain fixed during subsequent calibration, model adaptation, and evaluation. Grouping settings Both backbones use the same fixed set of 40 short calibration prompts and B=1000B=1000 prompt-level bootstrap replicates. For the formal experiments, Qwen2.5-0.5B uses (α,γ)=(0.25,0.9)(α,γ)=(0.25,0.9), while Llama-3.2-1B uses (α,γ)=(0.50,0.5)(α,γ)=(0.50,0.5). Both use τ=1.5τ=1.5 and Gmin=2G_ =2, with no explicit maximum group-count constraint. Coefficient calibration For a fixed layer and query head, let C be the coefficient-fitting sequences and y^i(L,x) y_i(L;x) the normalized causal OAL output for x∈x . With groups and pretrained queries, keys, and values fixed, we fit the lower-triangular L to the corresponding softmax outputs: L⋆=argminL1||∑x∈∑i‖y^i(L,x)−yisoftmax(x)‖22.L = _L 1|C| _x _i \| y_i(L;x)-y_i^softmax(x) \|_2^2. (20) Matching the normalized output directly accounts for both the kernel weights and their normalization denominator. The calibrated factors L⋆L initialize subsequent model-level adaptation, during which the dimension groups remain fixed while L is jointly optimized with the LoRA parameters under the causal language-modeling objective. Coefficient-calibration settings Coefficient fitting uses WikiText-2 raw v1 training text, concatenated and divided into non-overlapping sequences of length 40964096. We use blocks 1616–2323 as eight training sequences and blocks 2424–2525 as two validation sequences. This coefficient-fitting data is separate from the short-prompt calibration data used to determine the dimension groups. We optimize the OAL factors with Adam at learning rate 0.020.02, without a learning-rate scheduler, for up to 10001000 updates per initialization. Parameters are optimized in FP32 with gradient norm clipped at 1010. Each update samples one training sequence and 3232 causal query positions. Validation is performed every 100100 updates, with early stopping after 400400 updates without improvement. We use three initialization candidates for OAL: an embedding of the learned global quadratic into the grouped parameterization, a second-order Taylor initialization, and a Gaussian-perturbed learned-global initialization. Within each initialization, the lowest-validation-MSE checkpoint is retained independently for each query head; the final coefficients are then selected per head across the three initialization candidates. Layers are calibrated separately, with independent coefficients for each query head. The learned global quadratic uses the same training and validation tensors, output-matching objective, optimizer, learning rate, query-position sampling protocol, and checkpoint selection rule as OAL. Both methods use three initialization candidates with up to 10001000 updates per initialization; they differ only in their kernel parameterization and corresponding initializations. Frozen-attention evaluation uses blocks 00–1515 of the same token stream, giving 16 additional sequences of length 40964096. These blocks are disjoint from both coefficient training and validation and are not used for coefficient optimization or checkpoint selection. Group-budget diagnostic Figure 2(b) uses the same representative Qwen2.5-0.5B query head (zero-based Layer 4, Head 0) for all group budgets. Dimensions are sorted by the same calibration contribution scores χr _r, and for a prescribed budget G we select the G−1G-1 largest adjacent gaps in the sorted profile as group boundaries. Thus, increasing G only further subdivides the same contribution ordering. All budgets use the same coefficient-fitting and evaluation data: eight 4096-token WikiText-2 blocks for training, two for validation, and 16 disjoint blocks for held-out evaluation. Coefficients are fitted with the same Adam-based optimization sweep and selected by validation attention-output MSE; held-out data are never used for model selection. The plotted error is pooled held-out attention-output MSE over all tokens and output dimensions. Therefore, the points in Figure 2(b) are directly comparable across group budgets, with the group partition and corresponding parameter count as the only systematic changes. Appendix C Frozen-Attention Error Metrics For each held-out sequence, let YbY_b denote the exact softmax attention output and Y^b Y_b the corresponding approximation, with all evaluated layers and heads collected into the same tensor. We report MSE=1B∑b=1B‖Y^b−Yb‖F2nb,MSE= 1B _b=1^B \| Y_b-Y_b \|_F^2n_b, (21) and RelMSE=1B∑b=1B‖Y^b−Yb‖F2‖Yb‖F2,RelMSE= 1B _b=1^B \| Y_b-Y_b \|_F^2 \|Y_b \|_F^2, (22) where B is the number of held-out sequences and nbn_b is the number of scalar output elements in sequence b. Thus, RelMSE normalizes the squared approximation error by the energy of the exact softmax output for each sequence before averaging across sequences. For the metrics before WoW_o, YbY_b and Y^b Y_b contain the per-head attention outputs before the output projection. For the post-WoW_o metrics, the heads are first concatenated in their original order and passed through the frozen output projection WoW_o; the same MSE and RelMSE definitions are then applied to the projected outputs. The post-WoW_o RelMSE is therefore normalized by the energy of the exact softmax output after WoW_o, rather than by the pre-projection attention output. Appendix D Global Reference and HD Realization Positive global reference The learned global reference is a quadratic in the total score s=q⊤k/dhs=q k/ d_h. A symmetric positive semidefinite matrix Mglobal=[muv]u,v=01∈ℝ2×2M_global=[m_uv]_u,v=0^1 ^2× 2 parameterizes it as κglobal(q,k) _global(q,k) =ϵ+[1s]Mglobal[1s]⊤ =ε+ bmatrix1&s bmatrixM_global bmatrix1&s bmatrix^\! (23) =(ϵ+m00)+2m01s+m11s2,Mglobal⪰0. =(ε+m_00)+2m_01s+m_11s^2, M_global 0. Thus this is the global polynomial a+bs+cs2a+bs+cs^2 with a=ϵ+m00a=ε+m_00, b=2m01b=2m_01, and c=m11c=m_11; the matrix constraint and ϵ>0ε>0 ensure strictly positive weights. It is contained in OAL: setting M00=m00M_00=m_00, M0g=Mg0=m01M_0g=M_g0=m_01, and Mgh=m11M_gh=m_11 for all groups recovers the same polynomial because s=∑gsgs= _gs_g. This grouped matrix is positive semidefinite, since its quadratic form on any vector t equals the quadratic form of MglobalM_global on [t0,∑gtg]⊤[t_0, _gt_g] , and hence admits M=LL⊤M=L . Evaluation equivalence For a single query, (qi⊤kj)2=qi⊤kjkj⊤qi(q_i k_j)^2=q_i k_jk_j q_i gives ∑j(qi⊤kj)2Vj,ℓ=qi⊤(∑jVj,ℓkjkj⊤)qi=qi⊤K⊤DℓKqi. _j(q_i k_j)^2V_j, =q_i ( _jV_j, k_jk_j )q_i=q_i K D_ Kq_i. The last expression is the iith diagonal entry of QK⊤DℓKQ⊤QK D_ KQ , proving Eq. (4). Multiplication by DℓD_ amounts to scaling key rows and does not require storing a diagonal matrix. For the grouped kernel, expanding M=LL⊤M=L gives κOAL(q,k) _OAL(q,k) =ϵ+M00+2∑gM0gsg+∑g,hMghsgsh =ε+M_00+2 _gM_0gs_g+ _g,hM_ghs_gs_h (24) =a+1dh∑rbrqrkr+1dh∑r,tCrtqrqtkrkt, =a+ 1 d_h _rb_rq_rk_r+ 1d_h _r,tC_rtq_rq_tk_rk_t, which yields the coefficient mapping in Section 4.3. Substituting Eq. (5) into Eq. (6) and exchanging sums then gives u~i u_i =∑j≤i(a+1dh∑rbrqi,rkj,r+1dh∑r,tCrtqi,rqi,tkj,rkj,t)uj = _j≤ i (a+ 1 d_h _rb_rq_i,rk_j,r+ 1d_h _r,tC_rtq_i,rq_i,tk_j,rk_j,t )u_j (25) =∑j≤iκOAL(qi,kj)uj. = _j≤ i _OAL(q_i,k_j)u_j. The same substitution establishes the identity for κdim _dim with general coefficients. The argument uses separable query and key factors, not Mercer positive semidefiniteness; pointwise positivity of OAL’s weights does not imply the latter property. Exact state and parameter counts The ordered-pair states in Eq. (5) contain (1+dh+dh2)(dv+1)(1+d_h+d_h^2)(d_v+1) entries. Since H2,i,rt=H2,i,trH_2,i,rt=H_2,i,tr and C is symmetric, the packed-symmetric implementation stores only (1+dh+dh(dh+1)/2)(dv+1)(1+d_h+d_h(d_h+1)/2)(d_v+1) entries. Diagonal pairs have multiplicity one and off-diagonal pairs multiplicity two during readout, preserving all interactions. These counts describe one logical state, not total workspace or peak memory; GQA can share KV states across query heads. The lower-triangular factor L has (G+1)(G+2)/2(G+1)(G+2)/2 learned parameters per head. Appendix E Runtime and Memory Benchmark Figure 3 evaluates forward-only causal attention on an NVIDIA A100-SXM4-40GB. The batch size is B=1B=1, with Hq=14H_q=14 query heads, Hkv=2H_kv=2 key/value heads, and dh=dv=64d_h=d_v=64. Input tensors use BF16, and sequence lengths range from 2102^10 to 2152^15. These are attention-operator measurements, not end-to-end model latency or per-token autoregressive decoding latency. OAL (G=5G=5) and Learned Global (G=1G=1) use the same optimized HD block-GEMM backend, with 128-token blocks, BF16 Tensor Core contractions, and FP32 working coefficients. The implementation materializes intermediate tensors and reuses structural preparation across timed calls. OAL uses a fixed five-group configuration, matching the largest group count observed across the heads used in our model-adaptation experiments, to measure operator scaling. This benchmark does not execute the heterogeneous group configurations of all calibrated heads in a converted model. Performer, SpectraFormer, DiJiang, and TaylorShift use causal adaptations of their official source code. The source repositories and kernel adaptations are described in Appendix F.1. These methods retain their respective causal execution paths, while OAL and Learned Global share the specialized HD backend. The implementations differ in optimization level; in particular, the TaylorShift adaptation receives no additional kernel optimization. Internal arithmetic also differs across implementations. Performer and SpectraFormer use FP64 feature computation and prefix accumulation. DiJiang uses FP32 feature computation and prefix accumulation with 128-token blocks, explicitly repeating shared keys and values across query heads. Together with the backend differences above, these precision choices affect the measured runtime and memory, so cross-method gaps cannot be attributed solely to the kernel formulations. Both softmax references use PyTorch scaled dot-product attention (SDPA). The FlashAttention curve selects the FLASH_ATTENTION backend, whereas the PyTorch SDPA curve selects the MATH backend. Both calls enable grouped-query attention and causal masking, with zero attention dropout. Measurements run with gradient recording disabled, after warmup, and runtime is wall-clock latency with CUDA synchronization. Runtime markers report the mean across trials. GPU memory is the total peak allocated tensor memory reported by torch.cuda.max_memory_allocated(), including resident inputs and method state. It is neither incremental memory above the resident allocation nor allocator-reserved memory. This peak includes workspace and therefore differs from the logical recurrent-state size analyzed in Section 4.4. For both SpectraFormer and math-backend SDPA, the N=16384N=16384 case is skipped by a pre-run memory guard because the estimated next peak exceeds 80% of the current free GPU memory. Both sweeps therefore end at N=8192N=8192; these omissions reflect the benchmark’s memory budget rather than observed out-of-memory failures. Appendix F Conversion and LoRA Setup Attention replacement Figure 7 shows sustained large activations in the intermediate layers of both models, with different concentration patterns across hidden dimensions. Because pronounced activation outliers persist through these intermediate layers, we use this region as the primary target for attention replacement and determine the replacement ranges accordingly. For Qwen2.5-0.5B, we replace zero-based layers 3–20 (inclusive), retaining softmax attention in the first and last three layers. For Llama-3.2-1B, we replace layers 1–14 (inclusive), retaining softmax attention in the first and last layers. The same replacement range is used for all converted methods within each backbone. The softmax reference retains softmax attention throughout. These are hybrid models: the retained softmax layers still contribute quadratic prefill computation, even though the OAL operator is linear in sequence length. (a) Qwen2.5-0.5B (b) Llama-3.2-1B Figure 7: Layer-wise activation profiles. We measure the output residual stream of each decoder block on the same 40 short prompts. For each hidden dimension, we record its maximum absolute activation over all tokens and prompts; Top-1/2/3 denote the three largest dimension-wise maxima, and Median denotes their median across dimensions. Layer indices are zero-based. F.1 Baseline Implementations and Adaptations The baseline implementations in Table 3 are adapted from the official source code of Performer, SpectraFormer, DiJiang, and TaylorShift. We integrate their attention kernels into the same pretrained backbones, retaining the native query, key, value, and output projections, RoPE, and GQA head mapping. Each kernel uses causal prefix aggregation and normalization over the available keys. This setup compares attention approximations under a common conversion protocol rather than reproducing each method’s original full-model architecture and training recipe. For the frozen-attention evaluation, trainable kernel parameters are first calibrated on a small calibration set while the pretrained Q/K/V states remain fixed. This calibration is separate from the subsequent LoRA runs, where the trainable kernel parameters are optimized jointly with LoRA from the start. Performer We adapt the official FAVOR+ implementation11 1 https://github.com/google-research/google-research/tree/master/performer using positive orthogonal random features with 250 features per query head. Gaussian orthogonal projections are sampled independently for each layer and query head and remain fixed throughout adaptation; only the LoRA parameters are trained. Queries and keys are scaled by dh−1/4d_h^-1/4 before feature evaluation. To stabilize exponential features, query log-features are shifted by their per-token maximum, while key log-features use a shared maximum over tokens and features within each head. These common factors cancel in normalized attention. Feature computation and causal prefix accumulation use FP64, with the output cast back to the model dtype. SpectraFormer We adapt the OPRF--FastFoodL variant from the official repository22 2 https://github.com/dukenguyenxyz/spectraformer, using 65 positive features for dh=64d_h=64. SpectraFormer supports multiple combinations of feature functions and projection constructions. We select OPRF–FastFoodL as a lightweight learnable representative: its structured projection requires only 3dh3d_h trainable kernel parameters per layer. Each replacement layer has three trainable FastFood vectors B,G,S∈ℝ64B,G,S ^64 shared across its query heads, with a fixed permutation. This gives 192 kernel parameters per layer, or 3,456 for Qwen2.5-0.5B and 2,688 for Llama-3.2-1B over their respective replacement layers, excluding LoRA. The vectors remain trainable but are initialized from the standard FastFood distributions used in the upstream fixed branch. This avoids the nearly token-independent features observed with the small-normal initialization of the learnable branch. We retain the upstream feature scaling and per-token row-maximum stabilization followed by an additive 10−410^-4 feature offset. The attention aggregation is adapted to causal prefixes, using FP64 feature and accumulation arithmetic and FP32 trainable parameters. DiJiang We adapt the DCT projection and feature-wise softmax construction from the official repository33 3 https://github.com/YuchuanTian/DiJiang. The fixed projection combines an orthonormal DCT with diagonal inverse standard-normal samples. After scaling queries and keys by dh−1/4d_h^-1/4, we apply the projection and a softmax over the 64 feature coordinates. We add positive feature weights D=exp(θ)D= (θ) to both query and key features, with a separate 64-dimensional θ for each layer and query head, initialized to zero and trained jointly with LoRA. We normalize the causal weighted-value sum by the corresponding sum of kernel weights. The released model’s positional decay factors, GroupNorm, and output gate are omitted to retain the pretrained attention interface. Features and prefix states use FP32. Thus, the DiJiang row reports this normalized causal adaptation of the released feature map. TaylorShift We adapt the official implementation44 4 https://github.com/tobna/TaylorShift by retaining its query/key ℓ2 _2 normalization, fixed polynomial 1+s+12s21+s+ 12s^2, and learnable per-head temperature. Here, s=τq^⊤k^s=τ\, q k, where q q and k k are unit-normalized vectors. Each layer and query head has its own temperature τ, initialized to one and trained jointly with LoRA. We replace noncausal aggregation with causal prefix evaluation. We disable the optional output-amplitude normalization (output_normalized=False). Its full-sequence factor N/dh N/d_h would make a fixed prefix’s output depend on the total input length, while replacing N with the current prefix length would introduce an additional position-dependent scaling rule. We therefore use no extra output scaling and retain the usual attention numerator/denominator normalization. Learned global quadratic This controlled reference uses the positive global parameterization in Appendix D. It shares OAL’s output-calibration protocol, and its calibrated kernel parameters are jointly optimized with LoRA. It isolates the effect of global versus contribution-based coefficient sharing within the quadratic family. Scope of the comparison Our comparison focuses on fixed feature maps and kernels with a small number of trainable parameters, excluding the shared LoRA adapters. Hedgehog instead learns dense MLP feature maps (Zhang et al., 2024). An affine map from dhd_h input coordinates to m output coordinates contains dhm+md_hm+m parameters; for dh=m=64d_h=m=64, this is 4,160 parameters per map, before accounting for replication across heads and layers. This differs from the fixed projections, trainable vectors, and scalar or grouped coefficients used in our evaluated kernels. Its original attention-weight distillation objective also compares query–key pairs, with quadratic scaling in sequence length when evaluated over all causal pairs; this training cost does not imply quadratic inference complexity. We do not evaluate Hedgehog under the present protocol, and our results therefore do not establish OAL’s relative performance against MLP-based feature maps. Extending the comparison would require specifying their parameter budget and adaptation procedure, including whether to retain attention-weight distillation or use output matching. F.2 LoRA Adaptation For all methods, LoRA is applied to the query, key, value, and output projections in every layer, including layers that retain softmax attention. We use rank 88, scaling parameter α=16α=16, and dropout 0.050.05, keeping the pretrained backbone weights frozen. Each method is adapted for 512 optimizer updates on WikiText-103 with sequence length 4096, batch size 1, and no gradient accumulation, corresponding to 2,097,152 input tokens. All runs use a single training seed of 42 and the same training-block order within each backbone. OAL starts from training-side calibrated groups and kernel coefficients; its coefficients are optimized jointly with LoRA while the groups remain fixed. We use AdamW for LoRA and trainable kernel parameters, with learning rate 5×10−45× 10^-4, weight decay 0.010.01, (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999), and optimizer ϵ=10−8ε=10^-8. The learning rate follows a linear warmup over the first 5%5\% of updates, followed by cosine decay. The gradient norm is clipped to 1.01.0. Every 128 updates, we compute next-token NLL on the same fixed 32 WikiText validation blocks and select the checkpoint with the lowest validation NLL. Final language-modeling and downstream evaluations use this selected checkpoint; test results are not used for checkpoint selection. F.3 Evaluation Protocol Within each backbone, all methods use the same evaluation protocol and tokenizer, without chat templates or added BOS/EOS tokens. Perplexity We evaluate Salesforce/wikitext (wikitext-103-raw-v1, test split). Text entries are joined in source order with two newlines, tokenized, and divided into independent, non-overlapping 4096-token blocks, discarding the incomplete final block. PPL is the exponential of the mean negative log-likelihood over all scored tokens, with 4095 next-token targets per block. Downstream tasks We report zero-shot accuracy on PIQA validation (lighteval/piqa, 1,838 examples), HellaSwag validation (Rowan/hellaswag, 10,042 examples), and ARC-Easy test (allenai/ai2_arc, 2,376 examples). PIQA and ARC-E use Question: question, followed by a newline and Answer:. HellaSwag uses the activity label and context, with the second context segment capitalized, title markers replaced by periods, and bracketed artifacts removed. Candidates are prefixed with a space and ranked by their conditional log-likelihood divided by candidate token count for PIQA/HellaSwag or option-text character count for ARC-E (excluding the added space). Only candidate tokens are scored.