Paper deep dive
Emergent Inverse-Depth Scaling From Nonlinearity In Attention
Zirui Peng, Yizhou Liu, Ziming Liu, Jeff Gore
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/9/2026, 8:15:21 AM
Summary
This paper investigates the mechanisms behind depth scaling in large language models, contrasting linear and nonlinear attention. It demonstrates that while linear attention models exhibit spectrum-dependent power-law scaling and plateau on difficult tasks, nonlinear attention enables inverse-depth loss scaling (loss decaying as 1/L) by allowing selective focus on relevant tokens. This nonlinearity allows strong and weak spectral directions to be learned in parallel. The authors connect this behavior to the central limit theorem, where shared errors across layers set a loss plateau, while aggregation of layer-specific differences yields continued gains with depth.
Entities (7)
Relation Signals (5)
Nonlinear Attention → contrastswith → Linear Attention
confidence 95% · In contrast, empirical work on these models suggests that inverse-depth scaling can arise... Unlike the linear-attention model... the nonlinear-attention model exhibits inverse-depth loss scaling.
Nonlinear Attention → enables → Inverse-Depth Scaling
confidence 95% · nonlinear attention yields inverse-depth decay of loss across all tested data spectra.
Nonlinear Attention → allows → Parallel Learning
confidence 92% · Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel.
Linear Attention → exhibits → Power-law scaling
confidence 90% · In linear-attention models, this scaling is tied to a power-law data spectrum
Inverse-Depth Scaling → isexplainedby → Central Limit Theorem
confidence 85% · Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the parameter count scaling, existing theory posits power-law scaling with model depth. In linear-attention models, this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones. Large language models, however, can be strongly nonlinear. Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra. Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel. Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth. Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant.
Tags
Links
- Source: https://arxiv.org/abs/2610.11063v1
- Canonical: https://arxiv.org/abs/2610.11063v1
Trouble viewing inline? Open PDF directly →
Full Text
96,583 characters extracted from source content.
Expand or collapse full text
Emergent Inverse-Depth Scaling From Nonlinearity In Attention Zirui Peng †thanks: Work done during an internship at MIT. Affiliation: Department of Physics, Peking University Yizhou Liu Affiliation: Department of Physics, Massachusetts Institute of Technology, Cambridge, MA Ziming Liu Affiliation: College of AI, Tsinghua University Jeff Gore Affiliation: Department of Physics, Massachusetts Institute of Technology, Cambridge, MA Affiliation: Corresponding author: gore@mit.edu Abstract Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the parameter count scaling, existing theory posits power-law scaling with model depth. In linear-attention models,11 1 Here, linear attention means that attention scores are computed directly as query–key inner products and that the model contains no nonlinear components. this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones. Large language models, however, can be strongly nonlinear. Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra. Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel. Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth. Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant. Figure 1: The nonlinear-attention model can yield inverse-depth scaling across all tested data spectra. This overview compares depth-dependent performance across models and tasks. Colors distinguish data spectra, and all panels are plotted on a log-log scale. 1 Introduction Neural scaling laws describe how model loss decreases with the size of the dataset and the model, providing a practical basis for predicting performance and allocating computational resources (Hestness et al., 2017; Kaplan et al., 2020; Hoffmann et al., 2022). Despite their predictive success, these laws remain largely empirical, and the mechanisms underlying their behavior are not fully understood. Clarifying their origins could help establish the scope of these laws and reveal how scaling can be made more efficient. Recent theoretical work has begun to examine model-size scaling more closely by separating the roles of width and depth (Bahri et al., 2024; Liu et al., 2025; Bordelon et al., 2026). Increasing width provides more representational capacity within each layer, whereas increasing depth adds successive processing steps; the two can therefore improve performance through different mechanisms. Here, we focus on how models use additional layers and how this gives rise to scaling with depth. Previous work on linear-attention models ties power-law depth scaling to a power-law data spectrum (Bordelon et al., 2026). Yet it does not account for nonlinear attention in practical large language models. In contrast, empirical work on these models suggests that inverse-depth scaling can arise through an ensemble of layers with similar behavior (Liu et al., 2026a), without explaining the underlying reason for this layerwise similarity. We therefore ask: Our research question: How does nonlinearity in attention affect depth scaling and its underlying mechanisms? We study this question in the common setting of in-context learning (ICL), where a model uses examples in its context to make predictions for a new query. Our tasks model the second half of an induction head’s computation: retrieving the token that matches the query and copying its associated value (Elhage et al., 2021; Olsson et al., 2022; Lv et al., 2025). To isolate the role of nonlinearity, we compare controlled Transformers with linear or nonlinear attention across independently varied data spectra. Figure 1 summarizes the resulting picture. In an easy task, the linear-attention model follows a spectrum-dependent power law, while a single-layer nonlinear-attention model is enough to solve the task. In a difficult task, the linear-attention model quickly plateaus, whereas the nonlinear-attention model exhibits inverse-depth loss scaling. Unlike the linear-attention model, which cannot selectively focus on relevant tokens, the nonlinear-attention model can attend to them directly, allowing strong and weak spectral directions to be learned in parallel and enabling better utilization of depth. Further analysis decomposes the loss into cross-layer and within-layer terms and connects this decomposition to the central limit theorem. The asymptotic behavior of these terms explains the origins of the loss plateau and the reducible loss. Finally, we relax several simplifying assumptions of the controlled setup to move closer to standard Transformers and observe similar signs of inverse-depth scaling. These results suggest that nonlinearity in attention can be a plausible source of depth scaling in real models and may make scaling less dependent on the global covariance structure of the data. Our contributions are the following: • We discover that nonlinearity in attention can qualitatively affect depth scaling, enabling emergent inverse-depth loss scaling across all tested spectra. • We explain how nonlinear attention enables parallel learning of strong and weak spectral directions, in contrast to the sequential learning behavior of linear-attention models. • We connect inverse-depth scaling to the central limit theorem through a loss decomposition that identifies the origins of the loss plateau and the gains from depth. 2 Problem Setup 2.1 Task Setup In-context learning is a common setting in which predictions rely on interactions among context tokens. These interactions expose the structure of the data, providing a useful setting for studying the origins of depth scaling. We consider a controlled ICL problem in which each context contains P inputs i∈ℝd x_i ^d with labels generated by a teacher vector ∈ℝd β ^d: yi=⊤i,i=1,…,P.y_i= β x_i, i=1,…,P. (1) Let λk _k and ωk _k denote the eigenvalues of the input and teacher covariances, respectively. To control input and task structure independently, we vary these spectra as λk∝k−a,ωk∝k−b,k=1,…,d. _k k^-a, _k k^-b, k=1,…,d. (2) Here, a,b≥0a,b≥ 0 are the input-spectrum and teacher-spectrum exponents, respectively. They independently control how rapidly the corresponding variances decay across feature directions. Setting either exponent to zero gives an isotropic covariance for the corresponding distribution. In natural language, different contexts may share similar features but express them along different directions. Following Bordelon et al. (2026), we model this variation by independently sampling an orthogonal rotation matrix O∈ℝd×dO ^d× d for each context. Conditioned on O, the input and teacher covariances are Σ=Cov(i∣O)=Odiag(λ1,…,λd)O⊤,Ω=Cov(∣O)=Odiag(ω1,…,ωd)O⊤. gathered =Cov( x_i O)=Odiag( _1,…, _d)O ,\\ =Cov( β O)=Odiag( _1,…, _d)O . gathered (3) Conditioned on O, we sample the teacher vector independently of the context and test inputs, which are drawn i.i.d.: ∼(,Ω),1,…,P,test∼i.i.d.(,Σ). gathered β \! ( 0, ),\\ x_1,…, x_P, x_test i.i.d. N\! ( 0, ). gathered (4) Figure 2: Both tasks model the second half of an induction head: the easy task copies the most relevant token’s label, while the difficult task averages the labels of the top-k relevant tokens. The model must therefore infer the relevant directions from each context rather than rely on fixed directions across contexts. We first construct an easy task using a widely studied ICL setting (Garg et al., 2022; von Oswald et al., 2023; Ahn et al., 2023; Bordelon et al., 2026). Given context inputs and their teacher-generated labels, the model must predict the label assigned to a test input by the same teacher. Following Bordelon et al. (2026), we consider the infinite-context limit P→∞P→∞, in which a context token can be found arbitrarily close to the test token. The task can therefore be approximated by retrieving the closest context token and copying its label. This retrieval-and-copying operation corresponds to the second half of an induction-head circuit identified in language models (Elhage et al., 2021; Olsson et al., 2022; Lv et al., 2025). The task can thus be viewed, in an idealized sense, as an exact-match problem: test=j,ytest∗=yj,j∈1,…,P,P→∞. x_test= x_j, y^*_test=y_j, j∈\1,…,P\, P→∞. (5) However, in practical ICL and language settings, prediction often requires combining information from multiple context tokens rather than retrieving a single one. Motivated by this need, we construct a difficult task that extends this induction-head-inspired operation from top-11 retrieval to top-k aggregation. We measure relevance by cosine similarity and average the selected labels. The selected index set and target are defined by Sk(test) S_k( x_test) =TopKi∈1,…,Pi⊤test‖i‖‖test‖, =TopK_i∈\1,…,P\ x_i x_test\| x_i\|\,\| x_test\|, (6) ytest∗ y^*_test =1k∑i∈Sk(test)yi. = 1k _i∈ S_k( x_test)y_i. Here, TopKTopK returns the indices of the k largest scores. Figure 2 summarizes these induction-head-inspired top-11 copying and top-k aggregation tasks. (a) Fixed a=2a=2 (b) Fixed b=2b=2 (c) Normalized mode-wise loss Figure 3: On the easy task, the linear-attention model exhibits spectrum-dependent depth scaling and learns spectral modes sequentially. (a) Varying b at fixed a=2a=2. (b) Varying a at fixed b=2b=2. Dashed curves show power-law fits, with the corresponding exponents indicated. (c) Normalized loss of representative spectral modes. Values below 2%2\% are displayed at the threshold marked by the dashed line. (a) Fixed b=0b=0 (b) Fixed a=0a=0 (c) Top-11 attention mass, (1,0)(1,0) Figure 4: A single nonlinear-attention layer solves the easy task by focusing on the matching token. (a) Varying a at fixed b=0b=0. (b) Varying b at fixed a=0a=0. (c) Top-11 attention mass approaches one during training. (a) Fixed a=0a=0 (b) Fixed b=0b=0 Figure 5: On the difficult task, linear attention rapidly reaches a spectrum-dependent loss plateau with little benefit from further depth. (a) Varying b at fixed a=0a=0. (b) Varying a at fixed b=0b=0. (a) Fixed a=0a=0 (b) Fixed b=0b=0 (c) Shared-error term CLC_L at fixed b=0b=0 (d) Disagreement term AL−CLA_L-C_L at fixed b=0b=0 Figure 6: On the difficult task, nonlinear attention yields inverse-depth decay across all tested data spectra. (a) Varying b at fixed a=0a=0. (b) Varying a at fixed b=0b=0. (c,d) CLC_L and AL−CLA_L-C_L converge to spectrum-dependent constants. (a) Mode-wise raw loss (b) Normalized mode-wise reducible loss Figure 7: With nonlinear attention, strong and weak spectral modes are learned in parallel. (a) Different modes are learned at similar rates. (b) Normalized mode-wise reducible losses nearly collapse onto a common curve, indicating parallel learning across modes. 2.2 Model Setup Large language models combine positional encodings, multiple attention heads, MLP blocks, normalization, and task-specific readouts. While important in practice, these components make it difficult to isolate the mechanisms underlying depth scaling. Building on Bordelon et al. (2026), we therefore use a simplified architecture that retains the components needed for context-dependent retrieval and label propagation, allowing a controlled comparison between linear and nonlinear attention. Embeddings. Each context provides P input–label pairs (i,yi)( x_i,y_i), where the teacher vector β generates yiy_i from i x_i according to equation 1. The model has access to all context input–label pairs and the test input, but cannot access the test label. The corresponding entry at the test position is initialized to zero. We collect the token inputs and the observed labels into X=[1,…,P,test]∈ℝd×(P+1),in=(y1,…,yP,0)∈ℝ1×(P+1).X=[ x_1,…, x_P, x_test] ^d×(P+1), y^in=(y_1,…,y_P,0) ^1×(P+1). (7) Let N≥dN≥ d denote the hidden dimension, avoiding a representational bottleneck. Each token state lies in ℝNR^N. With encoder matrices Wx∈ℝN×dW_x ^N× d and Wy∈ℝN×1W_y ^N× 1, we encode input and label information into separate streams: Hx=WxX∈ℝN×(P+1),Hy0=Wyin∈ℝN×(P+1).H_x=W_xX ^N×(P+1), H_y^0=W_y y^in ^N×(P+1). (8) This encoding separates the input features used for token matching from the labels to be propagated. The input features determine which context tokens are relevant to the query, and attention passes their label information to the test token. The model has L residual layers, indexed by ℓ∈1,…,L ∈\1,…,L\. At each layer, attention uses the input features to determine how label information is propagated. For simplicity, we first keep HxH_x fixed across layers, while HyℓH_y is updated as label information is passed among tokens. We use layer-specific query and key matrices Wqℓ,Wkℓ∈ℝN×NW_q ,W_k ^N× N and a shared value matrix Wv∈ℝN×NW_v ^N× N. Thus, each layer can select relevant tokens differently while using a common transformation to propagate label information. Nonlinear attention. We first define the model with nonlinear attention. We apply pre-norm to the input-stream hidden state for better training stability and alignment with practical models, and denote the normalized state by H^x H_x. At layer ℓ , the query, key, and value representations are Qℓ=WqℓH^x,Kℓ=WkℓH^x,Vℓ=WvHyℓ−1,Qℓ,Kℓ,Vℓ∈ℝN×(P+1).Q =W_q H_x, K =W_k H_x, V =W_vH_y -1, Q ,K ,V ^N×(P+1). (9) Let M denote the attention-mask matrix. The attention matrix and residual update are Aℓ A =⊙softmax((Qℓ)⊤Kℓ)∈ℝ(P+1)×(P+1), = M \! ((Q ) K ) ^(P+1)×(P+1), (10) Hyℓ H_y =Hyℓ−1+1LVℓ(Aℓ)⊤∈ℝN×(P+1). =H_y -1+ 1LV (A ) ^N×(P+1). Linear attention. The linear-attention model removes nonlinearities and retains the raw bilinear attention scores. Its attention matrix and residual update are Aℓ A =1PN⊙((Qℓ)⊤Kℓ)∈ℝ(P+1)×(P+1), = 1PN M ((Q ) K ) ^(P+1)×(P+1), (11) Hyℓ H_y =Hyℓ−1+1LVℓ(Aℓ)⊤∈ℝN×(P+1). =H_y -1+ 1LV (A ) ^N×(P+1). Attention masks. As in large language models, we use attention masks to control information flow between tokens. For the linear-attention model, we adopt the masking scheme of Bordelon et al. (2026), allowing successive layers to extract and propagate information at different levels of granularity. For the nonlinear-attention model, we initially disable context updates to focus on how different layers retrieve information from a fixed context. In both models, the test token can attend to all context tokens to access their label information. Note that in the nonlinear-attention model, softmax is computed only over tokens that can be attended to, so disabled positions do not affect the distribution. Further details on the masks and context updates are given in Appendix C.3. Output and loss. We obtain the prediction by applying a linear readout Wo∈ℝ1×NW_o ^1× N to the final test-token state hy,testLh_y,test^L. The model is trained to minimize the mean squared error over independently sampled contexts: y^L=Wohy,testL,ℛL=context[(y^L−y∗)2]. y_L=W_oh_y,test^L, _L=E_context\! [( y_L-y^*)^2 ]. (12) Throughout, ℛLR_L denotes the evaluation mean squared error of a depth-L model; we write ℛL(a,b)R_L(a,b) when making its dependence on the data-spectrum exponents explicit. All models are trained until the loss has converged. In Appendix C, we relax several simplifying assumptions to better reflect the architectures of standard Transformers and large language models, and extend our experiments and analyses to these settings. 3 Easy Task Analysis 3.1 Linear-Attention Model Figure 3(a,b) shows that the linear-attention model’s loss on the easy task decreases with depth according to a power law across all tested settings, with an exponent that depends on the data spectra. Heuristically, we can interpret the depth-L linear-attention model’s output as a polynomial in the data covariance: y^L=⊤pL(Σ)test,pL(Σ)=∑r=1LarΣr. y_L= β p_L( ) x_test, p_L( )= _r=1^La_r ^r. (13) The target is y∗=⊤test.y^*= β x_test. (14) Therefore, the model needs to approximate the identity map with pL(Σ)p_L( ); Appendix A.1 derives this covariance-polynomial representation. Greater depth allows the model to use higher powers of the covariance, incorporating information at different levels of granularity to improve the approximation. The asymptotic behavior of this approximation gives rise to power-law depth scaling. Different data spectra change the covariance Σ in the polynomial, leading to different convergence rates with depth. This produces spectrum-dependent depth scaling. This mechanism also has a feature-learning interpretation. We decompose the power-law input covariance into individual spectral modes and normalize each mode’s loss, which allows us to compare how fully each mode has been learned. Small k indexes stronger features with large eigenvalues, whereas large k indexes weaker features with small eigenvalues. Figure 3(c) reveals a clear separation in their learning dynamics: stronger modes are learned almost perfectly within a few layers, while weaker modes require substantially greater depth. Appendix A.4 further shows that individual modes can converge exponentially at different characteristic depths, with their superposition yielding spectrum-dependent power-law scaling. 3.2 Nonlinear-Attention Model We next evaluate the nonlinear-attention model on the easy task. Figure 4(a,b) shows that a single-layer nonlinear-attention model achieves zero loss on the easy task across all tested data spectra. Unlike the spectrum-dependent improvement of the linear-attention model, its performance does not improve further with depth. The key distinction is that nonlinear attention can selectively focus on the relevant token in each context, whereas linear attention cannot fully exclude contributions from irrelevant tokens. During training, the queries and keys attain large norms, amplifying differences between attention logits and placing softmax in an effectively low-temperature regime. Almost all attention is then assigned to the matching token, allowing its label to be copied in a single layer. Figure 4(c) supports this explanation: during training, attention shifts from a nearly uniform distribution toward the top-11 token, whose attention mass approaches one. The same qualitative behavior holds across the tested spectra. 4 Difficult Task Analysis 4.1 Linear-Attention Model We next evaluate the linear-attention model on the difficult task. Here, we choose k=2k=2 as a representative case of top-k aggregation. Figure 5(a,b) shows that the loss reaches a spectrum-dependent plateau within one or two layers. Unlike the continued improvement on the easy task, additional depth provides little further benefit. Intuitively, the selected tokens vary across contexts, so their average fluctuates around a shared mean. The target therefore contains a predictable component and a context-specific deviation. Linear attention can capture the shared component, but lacks the context-specific token selection needed to fully recover the deviation. This leaves an irreducible loss that sets the plateau. We analyze the reducible part using the covariance-polynomial formulation in Section 3.1. The prediction retains the same form, while the predictable target has the leading-order form y^L=⊤pL(Σ)test,y¯∗∝⊤Σtest, y_L= β p_L( ) x_test, y^* β x_test, (15) Unlike the identity map in the easy task, this leading target can already be represented by a degree-one polynomial. Additional depth therefore provides little further reduction in loss. Appendix A.3 provides the geometric illustration, detailed derivation, and comparison between predicted and measured loss floors. 4.2 Nonlinear-Attention Model We next evaluate the nonlinear-attention model on the same difficult task with k=2k=2. Figure 6(a,b) shows that additional depth continues to reduce the loss, in contrast to the rapid saturation of the linear-attention model. To quantify this depth dependence, we fit the evaluation loss to ℛL(a,b)=L0(a,b)+c(a,b)L−α(a,b),R_L(a,b)=L_0(a,b)+c(a,b)L^-α(a,b), (16) by minimizing squared residuals between the logarithms of the observed and modeled losses. The fitted exponents shown in Figure 6(a,b) are close to one across the tested data spectra, indicating emergent inverse-depth scaling in the deep regime. With input and label information carried in separate streams and context states fixed across layers, our model admits an equivalent parallel representation: each layer forms a prediction from the same context, and the final output averages these predictions. Increasing depth therefore increases the number of predictors in the ensemble, suggesting an analogy with the central limit theorem, where averaging layer-specific fluctuations can yield a 1/L1/L decay in squared error. To distinguish the error reduced by ensembling from the error that remains, we derive the exact loss decomposition ℛL(a,b)=CL(a,b)+AL(a,b)−CL(a,b)L.R_L(a,b)=C_L(a,b)+ A_L(a,b)-C_L(a,b)L. (17) Here, CLC_L measures error shared across predictors, whereas AL−CLA_L-C_L measures pairwise differences between their errors. A detailed derivation is provided in Appendix B. Figure 6(c,d) tracks these two terms across different data spectra. As depth increases, both converge to spectrum-dependent constants: CL(a,b)⟶C∞(a,b),AL(a,b)−CL(a,b)⟶Δ∞(a,b),L→∞.C_L(a,b) C_∞(a,b), A_L(a,b)-C_L(a,b) _∞(a,b), L→∞. (18) The loss plateau and the depth-dependent improvement have different origins. The error shared across predictors cannot be canceled by ensembling and sets the irreducible loss within this ensemble. For the top-22 task, this shared error reflects the mismatch between smooth softmax attention and the sharp target distribution (1/2,1/2,0,…,0)(1/2,1/2,0,…,0), which is difficult to match consistently across contexts. In contrast, AL−CLA_L-C_L quantifies error variation across predictors. Ensembling allows these errors to partially cancel, reducing their contribution by a factor of 1/L1/L and enabling continued improvement with depth. Detailed derivations and attention-mass diagnostics are provided in Appendices B and D.2, respectively. We again examine individual spectral modes and normalize their excess losses to compare relative learning progress. Figure 7(a) shows similar improvement rates across modes, and the normalized curves in panel (b) nearly collapse. In contrast to the linear-attention model’s sequential learning, nonlinear attention can focus directly on relevant tokens in each context and improve strong and weak features in parallel, even without context updates that expose progressively finer-grained information. Building on our analysis of the differences between linear and nonlinear attention, we further compare their irreducible losses on the difficult task. Table 1 shows that the nonlinear-attention model achieves a substantially lower loss floor, clearly outperforming the linear-attention model. Table 1: Nonlinear-attention models achieve substantially lower irreducible loss than linear-attention models on the difficult task. (a,b)(a,b) (0,0)(0,0) (0,0.25)(0,0.25) (0,0.5)(0,0.5) (0,1)(0,1) (0.25,0)(0.25,0) (0.5,0)(0.5,0) (1,0)(1,0) Nonlinear L0L_0 2.007 1.268 0.836 0.424 1.255 0.798 0.341 Linear L0L_0 59.853 23.055 9.766 2.504 22.962 9.501 2.120 We extend our setup toward standard Transformers by encoding inputs and labels in a shared space, untying the value matrix, and updating context states across layers. We observe similar signs of inverse-depth scaling (Appendix C). These results suggest that this mechanism may also contribute to depth scaling in realistic models with nonlinear attention, where data structure need not be the dominant factor. 5 Related Work Empirical neural scaling laws relate loss to model size and dataset size, supporting performance prediction and resource allocation (Hestness et al., 2017; Kaplan et al., 2020; Hoffmann et al., 2022). Theoretical explanations have related them to data geometry and covariance spectra (Sharma and Kaplan, 2022; Bahri et al., 2024). More recent work identifies mechanisms arising from representation superposition and the dynamics of learning peaked distributions (Liu et al., 2025; Liu et al., 2026b; Kühn et al., 2026). Despite these advances, much of the literature treats model size as an aggregate quantity rather than resolving the different contributions of width and depth. As a key component of model-size scaling, depth has been studied through several complementary accounts of how networks utilize layers. Compositional learning builds complex representations by successively combining simpler features, making hierarchical structure more efficiently learnable (Bengio et al., 2013; LeCun et al., 2015; Poggio et al., 2017; Cagnetta et al., 2024). The dynamical-systems perspective treats residual layers as discrete steps of an ordinary differential equation (E, 2017; Haber and Ruthotto, 2018; Chen et al., 2018; Sander et al., 2022; Chizat, 2025). The ensemble-averaging perspective views depth as the aggregation of related predictors (Veit et al., 2016; Lu et al., 2020; Liu et al., 2026a). These perspectives explain how networks use depth but do not fully explain the origins of depth scaling. We examine how nonlinearity in attention shapes its underlying mechanisms and resulting scaling laws. More directly, Bordelon et al. (2026) build on the established ICL regression setting (Garg et al., 2022; von Oswald et al., 2023) to derive spectrum-dependent depth scaling for deep linear-attention models. A line of theoretical work has largely focused on analytically tractable linear models (Ahn et al., 2023; Zhang et al., 2024; Lu et al., 2025), leaving the mechanisms through which nonlinear attention affects depth scaling unresolved. Liu et al. (2026a) report inverse-depth scaling in LLMs and connect it to ensemble averaging, without explaining why this ensemble-like behavior emerges. Liu and Gore (2026) further argue that generic mechanisms may determine scaling exponents, while data and architecture affect the coefficients. Our work complements these studies by explaining how nonlinear attention enables ensemble-like behavior across layers and gives rise to inverse-depth scaling. 6 Discussion Our results identify nonlinearity in attention as a source of depth scaling beyond power-law data spectra. Nonlinear attention can focus directly on relevant tokens in each context, allowing strong and weak spectral directions to be learned in parallel. Linear attention lacks this selective ability, tying its learning dynamics to global spectral strength. Similar focusing across layers supports an ensemble interpretation: aggregation reduces layer-specific errors while leaving a shared loss floor, connecting inverse-depth improvement to the central limit theorem. These findings suggest that practical depth scaling can originate from attention nonlinearity rather than data structure alone. Our study has many limitations. Our simplified architecture omits components of full LLMs, so the identified mechanism is not guaranteed to operate in those models. Although architectural extensions show similar scaling signals, a clear theoretical account of their mechanisms is still lacking. Our experiments are restricted to ICL tasks, leaving generalization to real-world datasets and broader task classes untested. The identified mechanism need not be universal or dominant, and other processes may produce different scaling or interact with it. Our analysis also does not address how depth scaling couples to model width and dataset size. Aggregation also raises a question about how efficiently depth is used. If a few layers perform the essential computation while others mainly average residual errors, separate parameters at every layer may be wasteful. Looped Transformers reuse parameters across depth (Dehghani et al., 2019; Geiping et al., 2025) and may retain aggregation benefits with fewer parameters. Testing this possibility could guide more parameter-efficient use of depth in future language models. AI Use Statement We used generative AI tools to support brainstorming, articulate candidate mathematical claims, generate code under explicit human instructions, and assist with language editing. We did not use generative AI to generate synthetic datasets, conduct data analysis, or write mathematical proofs. All AI-assisted ideas and writing were critically reviewed by the authors, and all AI-generated code was independently inspected and verified. The authors performed the data analysis and mathematical proofs and take full responsibility for the final content of this work. References Ahn et al. (2023) K. Ahn, X. Cheng, H. Daneshmand, and S. Sra Transformers learn to implement preconditioned gradient descent for in-context learning. In Advances in Neural Information Processing Systems, Vol. 36, p. 45614–45650. External Links: Document Cited by: §2.1, §5. Bahri et al. (2024) Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma Explaining neural scaling laws. Proceedings of the National Academy of Sciences 121 (27), p. e2311878121. External Links: Document Cited by: §1, §5. Bengio et al. (2013) Y. Bengio, A. Courville, and P. Vincent Representation learning: a review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (8), p. 1798–1828. External Links: Document Cited by: §5. Bordelon et al. (2026) B. Bordelon, M. Letey, and C. Pehlevan Theory of scaling laws for in-context regression: depth, width, context and time. In International Conference on Learning Representations, Cited by: §A.1, §A.2, §A.2, §A.2, §A.2, §C.3, §1, §2.1, §2.1, §2.2, §2.2, §5. Cagnetta et al. (2024) F. Cagnetta, L. Petrini, U. M. Tomasini, A. Favero, and M. Wyart How deep neural networks learn compositional data: the random hierarchy model. Physical Review X 14 (3), p. 031001. External Links: Document Cited by: §5. Chen et al. (2018) R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud Neural ordinary differential equations. In Advances in Neural Information Processing Systems, Vol. 31, p. 6571–6583. Cited by: §5. Chizat and Bach (2018) L. Chizat and F. Bach On the global convergence of gradient descent for over-parameterized models using optimal transport. In Advances in Neural Information Processing Systems, Vol. 31, p. 3040–3050. Cited by: Appendix B. Chizat (2025) L. Chizat The hidden width of deep ResNets: tight error bounds and phase diagram. arXiv preprint arXiv:2509.10167. Cited by: §5. Dehghani et al. (2019) M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser Universal transformers. In International Conference on Learning Representations, Cited by: §6. E (2017) W. E A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics 5, p. 1–11. External Links: Document Cited by: §5. Elhage et al. (2021) N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah A mathematical framework for transformer circuits. Transformer Circuits Thread. External Links: Link Cited by: §1, §2.1. Garg et al. (2022) S. Garg, D. Tsipras, P. Liang, and G. Valiant What can transformers learn in-context? a case study of simple function classes. In Advances in Neural Information Processing Systems, Vol. 35, p. 30583–30598. External Links: Document Cited by: §2.1, §5. Geiping et al. (2025) J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein Scaling up test-time compute with latent reasoning: a recurrent depth approach. In Advances in Neural Information Processing Systems, Vol. 38, p. 46242–46293. External Links: Document Cited by: §6. Haber and Ruthotto (2018) E. Haber and L. Ruthotto Stable architectures for deep neural networks. Inverse Problems 34 (1), p. 014004. External Links: Document Cited by: §5. Hestness et al. (2017) J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, Md. M. A. Patwary, Y. Yang, and Y. Zhou Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409. Cited by: §1, §5. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. Rae, and L. Sifre An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, Vol. 35, p. 30016–30030. External Links: Document Cited by: §1, §5. Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: §1, §5. Kühn et al. (2026) M. Kühn, Y. Thelge, and B. Rosenow A boundary-layer mechanism for one-third scaling in online softmax classification. arXiv preprint arXiv:2605.22341. Cited by: §5. LeCun et al. (2015) Y. LeCun, Y. Bengio, and G. Hinton Deep learning. Nature 521, p. 436–444. External Links: Document Cited by: §5. Liu and Gore (2026) Y. Liu and J. Gore Neural scaling universality: if exponents are fixed, time to understand coefficients. arXiv preprint arXiv:2606.25008. Cited by: §5. Liu et al. (2026a) Y. Liu, S. Kangaslahti, Z. Liu, and J. Gore Inverse depth scaling from most layers being similar. In Proceedings of the 43rd International Conference on Machine Learning, Vol. 306, p. 77093–77118. External Links: Link Cited by: §1, §5, §5. Liu et al. (2025) Y. Liu, Z. Liu, and J. Gore Superposition yields robust neural scaling. In Advances in Neural Information Processing Systems, Vol. 38, p. 176757–176793. External Links: Document Cited by: §1, §5. Liu et al. (2026b) Y. Liu, Z. Liu, C. Pehlevan, and J. Gore Universal one-third time scaling in learning peaked distributions. In Proceedings of the 43rd International Conference on Machine Learning, Vol. 306, p. 77529–77558. External Links: Link Cited by: §5. Lu et al. (2020) Y. Lu, C. Ma, Y. Lu, J. Lu, and L. Ying A mean-field analysis of deep ResNet and beyond: towards provable optimization via overparameterization from depth. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119, p. 6426–6436. External Links: Link Cited by: §5. Lu et al. (2025) Y. M. Lu, M. Letey, J. A. Zavatone-Veth, A. Maiti, and C. Pehlevan Asymptotic theory of in-context learning by linear attention. Proceedings of the National Academy of Sciences 122 (28), p. e2502599122. External Links: Document Cited by: §5. Lv et al. (2025) A. Lv, R. Xie, X. Sun, Z. Kang, and R. Yan Language models “grok” to copy. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), p. 735–741. External Links: Document Cited by: §1, §2.1. Mei et al. (2018) S. Mei, A. Montanari, and P. Nguyen A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences 115 (33), p. E7665–E7671. External Links: Document Cited by: Appendix B. Olsson et al. (2022) C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah In-context learning and induction heads. arXiv preprint arXiv:2209.11895. Cited by: §1, §2.1. Poggio et al. (2017) T. Poggio, H. Mhaskar, L. Rosasco, B. Miranda, and Q. Liao Why and when can deep—but not shallow—networks avoid the curse of dimensionality: a review. International Journal of Automation and Computing 14 (5), p. 503–519. External Links: Document Cited by: §5. Rotskoff and Vanden-Eijnden (2018) G. M. Rotskoff and E. Vanden-Eijnden Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks. In Advances in Neural Information Processing Systems, Vol. 31, p. 7146–7155. Cited by: Appendix B. Sander et al. (2022) M. E. Sander, P. Ablin, and G. Peyré Do residual neural networks discretize neural ordinary differential equations?. In Advances in Neural Information Processing Systems, Vol. 35, p. 36520–36532. External Links: Document Cited by: §5. Sharma and Kaplan (2022) U. Sharma and J. Kaplan Scaling laws from the data manifold dimension. Journal of Machine Learning Research 23 (9), p. 1–34. Cited by: §5. Veit et al. (2016) A. Veit, M. J. Wilber, and S. Belongie Residual networks behave like ensembles of relatively shallow networks. In Advances in Neural Information Processing Systems, Vol. 29, p. 550–558. Cited by: §5. von Oswald et al. (2023) J. von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov Transformers learn in-context by gradient descent. In Proceedings of the 40th International Conference on Machine Learning, Vol. 202, p. 35151–35174. External Links: Link Cited by: §2.1, §5. Zhang et al. (2024) R. Zhang, S. Frei, and P. L. Bartlett Trained transformers learn linear models in-context. Journal of Machine Learning Research 25 (49), p. 1–55. Cited by: §5. Appendix A Linear-Attention Model Theory A.1 Covariance Polynomials in the Linear-Attention Model We first derive the theory of the linear-attention model on the easy task. In general, its output is an ordered matrix polynomial of degree at most L in the data covariance. The isotropic setting analyzed by Bordelon et al. (2026) then follows as a special case of our theory. Let U collect the context inputs, q denote the query, and S be the empirical input covariance: U=[1,…,P]∈ℝd×P,q=test∈ℝd,y=U⊤,S=1PUU⊤. gatheredU=[ x_1,…, x_P] ^d× P, q= x_test ^d,\\ y=U β, S= 1PU . gathered (19) The input and teacher covariances are Σ=Odiag(λ1,…,λd)O⊤,Ω=Odiag(ω1,…,ωd)O⊤, gathered =Odiag( _1,…, _d)O ,\\ =Odiag( _1,…, _d)O , gathered where the random orthogonal rotation O is fixed within a context and sampled independently across contexts. Contexts therefore share the same spectra but express their features in different directions. Within each context, conditioning on O, the context inputs and test token are sampled identically and independently, while the teacher vector is sampled independently from its corresponding distribution: 1,…,P,test∼i.i.d.(0,Σ),∼(0,Ω). x_1,…, x_P, x_test i.i.d. N(0, ), β (0, ). Define the effective query–key matrix Bℓ=1NWx⊤(Wkℓ)⊤WqℓWx,ℓ=1,…,L,B_ = 1NW_x (W_k ) W_q W_x, =1,…,L, (20) where N is the hidden-state dimension. Before masking, token i assigns context token j the weight (WqℓWxi)⊤(WkℓWxj)NP=j⊤BℓiP. (W_q W_x x_i) (W_k W_x x_j)NP= x_j B_ x_iP. Separate the context and test label states as Hyℓ=[Zℓ,hℓ],Zℓ∈ℝN×P,hℓ∈ℝN.H_y =[Z_ ,h_ ], Z_ ^N× P, h_ ^N. The linear-attention mask defined in Section 2.2 gives Z0 Z_0 =Wy⊤U,h0=0, =W_y β U, h_0=0, (21) Zℓ Z_ =Zℓ−1−1LPWvZℓ−1U⊤BℓU, =Z_ -1- 1LPW_vZ_ -1U B_ U, hℓ h_ =hℓ−1+1LPWvZℓ−1U⊤Bℓq. =h_ -1+ 1LPW_vZ_ -1U B_ q. Write Zℓ=FℓUZ_ =F_ U, where FℓF_ maps inputs to context label states. Substituting UU⊤/P=SUU /P=S yields F0 F_0 =Wy⊤,Fℓ∈ℝN×d, =W_y β , F_ ^N× d, (22) Fℓ F_ =Fℓ−1−1LWvFℓ−1SBℓ, =F_ -1- 1LW_vF_ -1SB_ , hℓ−hℓ−1 h_ -h_ -1 =(Fℓ−1−Fℓ)q. =(F_ -1-F_ )q. Therefore, the output is y^L=Wo∑ℓ=1L(Fℓ−1−Fℓ)q=Wo(F0−FL)q. y_L=W_o _ =1^L(F_ -1-F_ )q=W_o(F_0-F_L)q. (23) The first two steps show how the polynomial is built: F1 F_1 =Wy⊤−1LWvWy⊤SB1, =W_y β - 1LW_vW_y β SB_1, F2 F_2 =Wy⊤−1LWvWy⊤(SB1+SB2) =W_y β - 1LW_vW_y β (SB_1+SB_2) +1L2Wv2Wy⊤SB1SB2. + 1L^2W_v^2W_y β SB_1SB_2. With readout coefficients mr=WoWvrWy,r=1,…,L,m_r=W_oW_v^rW_y, r=1,…,L, substitution into the output gives L(S)=∑r=1L(−1)r−1mrLr∑1≤ℓ1<⋯<ℓr≤LSBℓ1⋯SBℓr.P_L(S)= _r=1^L (-1)^r-1m_rL^r _1≤ _1<·s< _r≤ LSB_ _1·s SB_ _r. (24) The prediction and easy-task target are y^L=⊤L(S)q,y∗=⊤q. y_L= β P_L(S)q, y^*= β q. (25) Thus learning the easy task amounts to approximating the identity on the relevant input directions. Define L(S)=Id−L(S).D_L(S)=I_d-P_L(S). Averaging over the teacher at fixed inputs gives [(y^L−y∗)2∣U,q,O] _ β[( y_L-y^*)^2 U,q,O] =[(⊤L(S)q)2∣U,q,O] =E_ β[( β D_L(S)q)^2 U,q,O] =q⊤L(S)⊤[⊤∣U,q,O]L(S)q =q D_L(S) E[ β β U,q,O]D_L(S)q =q⊤L(S)⊤ΩL(S)q. =q D_L(S) _L(S)q. The evaluation loss ℛLR_L defined in equation 12 is therefore ℛL _L =[(y^L−y∗)2] =E[( y_L-y^*)^2] (26) =U,q,O[q⊤L(S)⊤ΩL(S)q]. =E_U,q,O[q D_L(S) _L(S)q]. A.2 Constrained and Free Polynomial Depth Scaling Repeated updates. Following the repeated-update construction of Bordelon et al. (2026), we impose isotropic matching and align the shared value map with the label embedding: Bℓ=κId(ℓ=1,…,L),WvWy=vWy.B_ =κ I_d ( =1,…,L), W_vW_y=vW_y. (27) Each degree-r product in equation 24 then equals κrSrκ^rS^r, and there are (Lr) Lr choices of r layers. Hence L(S)=∑r=1L(−1)r−1mrLr(Lr)κrSr=pL(S),pL(λ)=∑r=1Larλr,ar=(−1)r−1(Lr)(κ/L)rmr. gatheredP_L(S)= _r=1^L (-1)^r-1m_rL^r Lrκ^rS^r=p_L(S),\\ p_L(λ)= _r=1^La_rλ^r, a_r=(-1)^r-1 Lr(κ/L)^rm_r. gathered (28) In the infinite-context limit P→∞P→∞, we use S≈ΣS≈ . Thus Id−pL(Σ)=Odiag(1−pL(λ1),…,1−pL(λd))O⊤.I_d-p_L( )=Odiag (1-p_L( _1),…,1-p_L( _d) )O . Using [qq⊤∣O]=ΣE[q O]= in equation 26 gives the population-covariance approximation ℛLpopR_L^pop to ℛLR_L: ℛLpop _L^pop =Otr[L(Σ)⊤ΩL(Σ)Σ] =E_Otr\! [D_L( ) _L( ) ] (29) =Otr[Odiag(λkωk[1−pL(λk)]2)k=1dO⊤] =E_Otr\! [Odiag\! ( _k _k[1-p_L( _k)]^2 )_k=1^dO ] =∑k=1dλkωk[1−pL(λk)]2. = _k=1^d _k _k[1-p_L( _k)]^2. Define rL(λ)=1−pL(λ)r_L(λ)=1-p_L(λ). Since pL(0)=0p_L(0)=0, we have rL(0)=1r_L(0)=1, and therefore degrL≤L,rL(0)=1,ℛLpop=∑k=1dλkωkrL(λk)2. r_L≤ L, r_L(0)=1, _L^pop= _k=1^d _k _kr_L( _k)^2. (30) For a specific data spectrum, write λk=cλk−a,ωk=cωk−b,a≥0,b≥0,ζ=a+b−1a(a>0). gathered _k=c_λk^-a, _k=c_ωk^-b,\\ a≥ 0, b≥ 0, ζ= a+b-1a (a>0). gathered where cλ,cω>0c_λ,c_ω>0 and λ1=cλ _1=c_λ. For a>0a>0, the change of variables from feature index to eigenvalue is k=(cλ/λ)1/a,|dkdλ|=cλ1/aλ−1−1/a.k=(c_λ/λ)^1/a, | dkdλ |= c_λ^1/aaλ^-1-1/a. Substituting into equation 30 and replacing the sum by its continuum integral gives ℛLpop _L^pop =cλcω∑k=1dk−(a+b)rL(cλk−a)2 =c_λc_ω _k=1^dk^-(a+b)r_L(c_λk^-a)^2 (31) ≈cλcω∫1dk−(a+b)rL(cλk−a)2k ≈ c_λc_ω _1^dk^-(a+b)r_L(c_λk^-a)^2\,dk =cωcλ1−ζa∫λdλ1λζ−1rL(λ)2dλ. = c_ωc_λ^1-ζa _ _d _1λ^ζ-1r_L(λ)^2\,dλ. For a+b>1a+b>1, we have ζ>0ζ>0, which makes the continuum weight integrable at zero. When d is sufficiently large, λd _d is close to zero and the finite-dimensional cutoff does not yet affect the loss. We may then approximate the dense spectrum by a measure on [0,λ1][0, _1]. This gives ℛLpop∝∫0λ1λζ−1rL(λ)2λ.R_L^pop _0 _1λ^ζ-1r_L(λ)^2\,dλ. (32) Because WvW_v is shared across layers, mr=WoWvrWy∝vr,γ=κv.m_r=W_oW_v^rW_y v^r, γ=κ v. Optimizing WoWyW_oW_y to 11 allows the repeated-update construction to solve the easy task in the large-depth limit. Substituting into equation 28 gives pL(λ)=∑r=1L(−1)r−1(Lr)(γλ/L)r=1−(1−γλ/L)L,rL(λ)=(1−γλ/L)L. gatheredp_L(λ)= _r=1^L(-1)^r-1 Lr(γλ/L)^r=1-(1-γλ/L)^L,\\ r_L(λ)=(1-γλ/L)^L. gathered (33) For the residual to vanish as L→∞L→∞, the scalar updates must contract rather than amplify positive-eigenvalue directions. At the largest eigenvalue this requires |1−γλ1/L|<1⟺0<γ<2Lλ1.|1-γ _1/L|<1 0<γ< 2L _1. Using the optimal-scale estimate of Bordelon et al. (2026), γ∗≈Lλ1,γ∗L≈1λ1,γ^*≈ L _1, γ^*L≈ 1 _1, the continuum loss becomes, with u=2Lλ/λ1u=2Lλ/ _1, ℛLcon _L^con ≈cωcλ1−ζa∫0λ1λζ−1(1−λ/λ1)2Lλ ≈ c_ωc_λ^1-ζa _0 _1λ^ζ-1(1-λ/ _1)^2L\,dλ (34) ∼cωcλ1−ζa(λ12L)ζ∫0∞uζ−1e−uu c_ωc_λ^1-ζa ( _12L )^ζ _0^∞u^ζ-1e^-u\,du ∝L−ζ. L^-ζ. Here, ℛLconR_L^con denotes the loss of the repeated-update reference within the population approximation. This exponent is consistent with the result of Bordelon et al. (2026). Free polynomials. We then derive the depth-scaling exponent when the polynomial coefficients are free by removing the repeated-update conditions in equation 27. We assume only that the depth-L predictor is an arbitrary polynomial in Σ of degree at most L. Substitution into equation 26 gives the freely optimized population-polynomial reference ℛLfreeR_L^free: L(Σ)=pL(Σ),degpL≤L,pL(0)=0,S≈Σ,ℛLpop=∑k=1dλkωk[1−pL(λk)]2,ℛLfree∝mindegr≤Lr(0)=1∫0λ1λζ−1r(λ)2dλ,r=1−pL. gatheredP_L( )=p_L( ), p_L≤ L, p_L(0)=0, S≈ ,\\ R_L^pop= _k=1^d _k _k[1-p_L( _k)]^2,\\ R_L^free _ subarrayc r≤ L\\ r(0)=1 subarray _0 _1λ^ζ-1r(λ)^2\,dλ, r=1-p_L. gathered (35) Let ϕn _n be a degree-n polynomial orthonormal under this weight: ∫0λ1λζ−1ϕn(λ)ϕm(λ)λ=δnm, _0 _1λ^ζ-1 _n(λ) _m(λ)\,dλ= _nm, where δnm _nm equals one for n=mn=m and zero otherwise. Expanding the residual and using r(0)=1r(0)=1 gives r(λ)=∑n=0Ldnϕn(λ),∑n=0Ldnϕn(0)=1,∫0λ1λζ−1r(λ)2dλ=∑n=0Ldn2. gatheredr(λ)= _n=0^Ld_n _n(λ), _n=0^Ld_n _n(0)=1,\\ _0 _1λ^ζ-1r(λ)^2\,dλ= _n=0^Ld_n^2. gathered Cauchy–Schwarz now gives the minimum directly: 1 1 =(∑n=0Ldnϕn(0))2 = ( _n=0^Ld_n _n(0) )^2 (36) ≤(∑n=0Ldn2)(∑n=0Lϕn(0)2). ≤ ( _n=0^Ld_n^2 ) ( _n=0^L _n(0)^2 ). Equality is attained by coefficients proportional to the endpoint values: dn∗=ϕn(0)∑m=0Lϕm(0)2,ℛLfree∝[∑n=0Lϕn(0)2]−1.d_n^*= _n(0) _m=0^L _m(0)^2, _L^free [ _n=0^L _n(0)^2 ]^-1. (37) For this weight, the basis consists of shifted Jacobi polynomials: ϕn(λ)=2n+ζλ1ζPn(ζ−1,0)(1−2λ/λ1), _n(λ)= 2n+ζ _1^ζ\,P_n^(ζ-1,0)(1-2λ/ _1), (38) where Pn(ζ−1,0)P_n^(ζ-1,0) is the degree-n Jacobi polynomial. Using its value at 11 gives ϕn(0)2 _n(0)^2 =2n+ζλ1ζ[Γ(n+ζ)Γ(ζ)Γ(n+1)]2 = 2n+ζ _1^ζ [ (n+ζ) (ζ) (n+1) ]^2 ∼2λ1ζΓ(ζ)2n2ζ−1. 2 _1^ζ (ζ)^2\,n^2ζ-1. Consequently, ∑n=0Lϕn(0)2 _n=0^L _n(0)^2 ∼2λ1ζΓ(ζ)2∫0Ln2ζ−1n 2 _1^ζ (ζ)^2 _0^Ln^2ζ-1\,dn (39) =L2ζλ1ζΓ(ζ)2. = L^2ζ _1^ζ (ζ)^2. The two polynomial families therefore give ℛLcon∝L−ζ,ℛLfree∝L−2ζ,ζ=a+b−1a.R_L^con L^-ζ, _L^free L^-2ζ, ζ= a+b-1a. (40) Table 2 compares these reference exponents with the measured values. Table 2: Measured depth-scaling exponents of the linear-attention model on the easy task lie between two reference values. (a,b)(a,b) Constrained ζ Measured α Free 2ζ2ζ (2,0.5)(2,0.5) 0.7500.750 1.1211.121 1.5001.500 (2,1)(2,1) 1.0001.000 1.4851.485 2.0002.000 (2,1.5)(2,1.5) 1.2501.250 1.6841.684 2.5002.500 (2,2)(2,2) 1.5001.500 1.9871.987 3.0003.000 (0.5,2)(0.5,2) 3.0003.000 3.3103.310 6.0006.000 (1,2)(1,2) 2.0002.000 2.5522.552 4.0004.000 (1.5,2)(1.5,2) 1.6671.667 2.1632.163 3.3333.333 The actual model is less constrained than the repeated-update construction of Bordelon et al. (2026), which relies on the two conditions in equation 27, but it is not as unconstrained as the idealized free polynomial, whose coefficients are optimized independently. It is therefore reasonable for the measured exponents to lie between these two reference values. Note that this theoretical analysis has several requirements, including the population-covariance approximation S≈ΣS≈ , corresponding to P→∞P→∞; a dense spectrum with ζ>0ζ>0; and a sufficiently large spectral range λ1/λd _1/ _d. At the evaluated depths, repeated updates require 1≪L≪λ1/λd1 L _1/ _d, whereas free polynomials require 1≪L≪λ1/λd1 L _1/ _d. A.3 Predicting the Linear-Model Loss Floor In this subsection, we characterize the predictable component of the difficult-task target and the resulting irreducible loss floor for the linear-attention model. We further validate these theoretical predictions using Monte Carlo estimates. For a test token q sampled independently of the context inputs, the difficult-task target is Sk(q)=TopK1≤i≤Pq⊤i‖q‖‖i‖,t=1k∑i∈Sk(q)i∈ℝd,y∗=⊤t. gatheredS_k(q)=TopK_1≤ i≤ P q x_i\|q\|\,\| x_i\|,\\ t= 1k _i∈ S_k(q) x_i ^d, y^*= β t. gathered (41) From Appendix A.1, assuming S≈ΣS≈ , y^L=⊤pL(Σ)q,pL(Σ)=∑r=1LarΣr. y_L= β p_L( )q, p_L( )= _r=1^La_r ^r. (42) Write uL(q)=pL(Σ)qu_L(q)=p_L( )q. Since y^L−y∗=⊤(uL−t) y_L-y^*= β (u_L-t), averaging first over the independent teacher gives [(y^L−y∗)2∣U,q,O] _ β[( y_L-y^*)^2 U,q,O] =(t−uL)⊤[⊤∣O](t−uL) =(t-u_L) E[ β β O](t-u_L) =(t−uL)⊤Ω(t−uL). =(t-u_L) (t-u_L). Thus ℛLpop=‖t−uL‖Ω2R_L^pop=E\|t-u_L\|_ ^2, where ‖z‖Ω2=z⊤Ωz\|z\|_ ^2=z z. At fixed q,Oq,O, the selected target t varies across contexts, whereas uLu_L does not. Separate its conditional mean from this variation: t¯=[t∣q,O],δ=t−t¯,[δ∣q,O]=0. gathered t=E[t q,O], δ=t- t,\\ E[δ q,O]=0. gathered Substituting t−uL=δ+(t¯−uL)t-u_L=δ+( t-u_L) and expanding gives ‖t−uL‖Ω2=‖δ‖Ω2+‖t¯−uL‖Ω2+2δ⊤Ω(t¯−uL).\|t-u_L\|_ ^2=\|δ\|_ ^2+\| t-u_L\|_ ^2+2δ ( t-u_L). The cross term vanishes after conditioning because Ω and t¯−uL t-u_L are fixed given q,Oq,O: [δ⊤Ω(t¯−pL(Σ)q)] \! [δ ( t-p_L( )q ) ] =[[δ∣q,O]⊤Ω(t¯−pL(Σ)q)] =E\! [E[δ q,O] ( t-p_L( )q ) ] =0. =0. Hence the population risk separates into ℛLpop _L^pop =‖t−pL(Σ)q‖Ω2 =E\|t-p_L( )q\|_ ^2 (43) =‖δ‖Ω2⏟context-specific variation+‖t¯−pL(Σ)q‖Ω2⏟polynomial approximation error. = E\|δ\|_ ^2_context-specific variation+ E\| t-p_L( )q\|_ ^2_polynomial approximation error. We denote the first term by ℛirrR_irr. It is independent of the polynomial degree, whereas increasing the degree can reduce only the second term. Predictable component of the difficult-task target. Section 4.1 states that the predictable component is approximately proportional to Σq q. We now derive this result by approximating cosine-similarity ranking with the inner-product scores si=q⊤is_i=q x_i. Conditional on q,Oq,O, the pair (i,si)( x_i,s_i) is jointly Gaussian, with [si∣q,O]=0,Var(si∣q,O)=q⊤Σq=:V(q),Cov(i,si∣q,O)=Σq.E[s_i q,O]=0, (s_i q,O)=q q=:V(q), ( x_i,s_i q,O)= q. The Gaussian conditioning formula therefore gives [i∣si,q,O]=Cov(i,si∣q,O)Var(si∣q,O)si=ΣqV(q)si.E[ x_i s_i,q,O]= Cov( x_i,s_i q,O)Var(s_i q,O)s_i= qV(q)s_i. (44) Let s(1)≥⋯≥s(P)s_(1)≥·s≥ s_(P) be the ordered scores, and let (j) x_(j) denote the input attaining s(j)s_(j). Since zi=si/V(q)z_i=s_i/ V(q) are independent standard normal variables, define γP,k=[1k∑j=1kz(j)]. _P,k=E\! [ 1k _j=1^kz_(j) ]. Applying equation 44 to the k selected inputs and averaging over their ordered scores gives t¯(q) t(q) =[1k∑j=1k(j)|q,O] =E\! [ 1k _j=1^k x_(j) |q,O ] (45) =ΣqV(q)[1k∑j=1ks(j)|q,O]=γP,kΣq⊤Σq. = qV(q)E\! [ 1k _j=1^ks_(j) |q,O ]= _P,k q q q. To remove the remaining query-dependent denominator, note that for q∼(0,Σ)q (0, ), [V(q)∣O]=tr(Σ2),Var(V(q)∣O)=2tr(Σ4).E[V(q) O]=tr( ^2), (V(q) O)=2tr( ^4). When 2tr(Σ4)/tr(Σ2)2≪12tr( ^4)/tr( ^2)^2 1, we can replace V(q)V(q) by its mean. Combining this with the cosine-to-score approximation yields t¯(q)≈cΣq,c=γP,ktr(Σ2). t(q)≈ c q, c= _P,k tr( ^2). Since the teacher is independent of the selected inputs given q,Oq,O, [y∗∣q,,O]=⊤[t∣q,O]≈c⊤Σq. gatheredE[y^* q, β,O]= β E[t q,O]≈ c\, β q. gathered (46) Thus, the predictable component is proportional to Σq q and can already be represented by the degree-one polynomial p1(Σ)=cΣp_1( )=c . Higher polynomial degrees cannot remove the context-specific variation in equation 43, explaining both the rapid saturation and the nonzero loss floor of the linear-attention model. An explicit expression for the irreducible loss. Let =(a1,…,aL)⊤a=(a_1,…,a_L) collect the polynomial coefficients and define ΦL(q)=[Σq,…,ΣLq],pL(Σ)q=ΦL(q). _L(q)=[ q,…, ^Lq], p_L( )q= _L(q)a. (47) The corresponding population loss expands as ℛLpop() _L^pop(a) =‖t−ΦL(q)‖Ω2 =E\|t- _L(q)a\|_ ^2 (48) =[t⊤Ωt]−2⊤[ΦL(q)⊤Ωt] =E[t t]-2a E[ _L(q) t] +⊤[ΦL(q)⊤ΩΦL(q)] +a E[ _L(q) _L(q)]a =T−2⊤h+⊤G, =T-2a h+a Ga, where T=[t⊤Ωt],h=[ΦL(q)⊤Ωt],G=[ΦL(q)⊤ΩΦL(q)].T=E[t t], h=E[ _L(q) t], G=E[ _L(q) _L(q)]. (49) In particular, the entries of the Gram matrix are determined by the spectra: Grs=[(Σrq)⊤Ω(Σsq)]=tr(ΩΣr+s+1),1≤r,s≤L.G_rs=E[( ^rq) ( ^sq)]=tr( ^r+s+1), 1≤ r,s≤ L. (50) Differentiating equation 48 with respect to a gives the normal equations G=hGa=h. Hence ∗=G†h,ℛL,∗pop=T−h⊤G†h.a_*=G h, _L,*^pop=T-h G h. (51) The decomposition in equation 43 also gives minℛLpop()=ℛirr+min‖t¯−ΦL(q)‖Ω2. _aR_L^pop(a)=R_irr+ _aE\| t- _L(q)a\|_ ^2. (52) Under the degree-one approximation t¯≈cΣq t≈ c q in equation 46, the second term is negligible. Combining equation 51 and equation 52 therefore gives the explicit prediction ℛirr≈T−h⊤G†h.R_irr≈ T-h G h. (53) We evaluate this prediction by Monte Carlo sampling in the original cosine task. We independently sample n query–rotation–context triples. For sample i, tit_i is the resulting cosine-selected target and Ωi=Oidiag(ω1,…,ωd)Oi⊤ _i=O_idiag( _1,…, _d)O_i . The hats below denote empirical estimates computed from these samples: T^=1n∑i=1nti⊤Ωiti,h^=1n∑i=1nΦL(qi)⊤Ωiti,G^=1n∑i=1nΦL(qi)⊤ΩiΦL(qi),ℛ^irr=T^−h^⊤G^†h^. gathered T= 1n _i=1^nt_i _it_i, h= 1n _i=1^n _L(q_i) _it_i,\\ G= 1n _i=1^n _L(q_i) _i _L(q_i), R_irr= T- h G h. gathered (54) Table 3 compares this Monte Carlo prediction of equation 53 with the actual loss plateau of the trained model. Their close agreement validates the predicted irreducible floor. Table 3: The theoretically predicted irreducible loss closely matches the actual loss plateau of the linear-attention model. (a,b)(a,b) Theory-predicted loss Measured loss Difference k=2k=2 (0,0)(0,0) 59.3714±0.021359.3714± 0.0213 59.8502 0.80%0.80\% (0.25,0.25)(0.25,0.25) 9.7004±0.00499.7004± 0.0049 9.7387 0.39%0.39\% (0.5,0.5)(0.5,0.5) 2.2715±0.00612.2715± 0.0061 2.3274 2.40%2.40\% (1,0.25)(1,0.25) 1.2246±0.01061.2246± 0.0106 1.2259 0.11%0.11\% (1,1)(1,1) 0.4975±0.01020.4975± 0.0102 0.5036 1.22%1.22\% k=16k=16 (0.5,0.5)(0.5,0.5) 0.3218±0.00420.3218± 0.0042 0.3083 4.38%4.38\% (1,0)(1,0) 0.4481±0.00860.4481± 0.0086 0.4452 0.67%0.67\% (1,0.25)(1,0.25) 0.3186±0.00850.3186± 0.0085 0.3222 1.11%1.11\% (1.25,0)(1.25,0) 0.3595±0.00900.3595± 0.0090 0.3659 1.75%1.75\% Geometric interpretation. We now provide a heuristic geometric picture of the predictable component and irreducible loss, using the top-22 task as an example. In Figure 8, the horizontal ray represents the query xtestx_test. The black vectors Ox1→ Ox_1 and Ox2→ Ox_2 are the two selected context inputs, and their average is the green target Ot→=(x1+x2)/2 Ot=(x_1+x_2)/2. The blue vector OC→ OC represents the conditional mean t¯ t, whereas the red vector Ct→=t−t¯ Ct=t- t is the context-specific residual. Linear attention can approximate the predictable blue component through the population covariance, but it lacks the context-specific selection required to follow the red residual for each realized context. This residual therefore produces the irreducible loss floor. Figure 8: The linear-attention model captures only the average predictable component of the difficult-task target. The green target is the average of the two selected inputs. Blue shows the best predictable component OC→ OC; red shows the context-specific residual. A.4 Power Laws from Mode-Wise Exponential Decay As a heuristic model of Figure 3(c), we show how exponential decay at different rates can combine into a power law. Let feature k have initial loss weight wk≥0w_k≥ 0 and characteristic depth τk>0 _k>0: ℛk(L)=wke−L/τk,ℛL=∑kwke−L/τk.R_k(L)=w_ke^-L/ _k, _L= _kw_ke^-L/ _k. Assume the loss-weighted distribution of characteristic depths follows a power law, ρ(τ)=Cτ−βρ(τ)=Cτ^-β for τ≥τ0>0τ≥ _0>0, with C>0C>0 and β>1β>1. The aggregate loss is ℛL≈∫τ0∞ρ(τ)e−L/τdτ.R_L≈ _ _0^∞ρ(τ)e^-L/τ\,dτ. (55) With x=L/τx=L/τ and dτ=−Lx−2dxdτ=-Lx^-2\,dx, this becomes ℛL _L ≈CL1−β∫0L/τ0xβ−2e−xx ≈ CL^1-β _0^L/ _0x^β-2e^-x\,dx (56) ∼CΓ(β−1)L−(β−1),L≫τ0. C (β-1)L^-(β-1), L _0. Thus exponential convergence of individual features produces power-law depth scaling when their characteristic depths have a broad power-law distribution. Stronger features converge first, while weaker features continue to contribute as depth increases. Appendix B Nonlinear-Attention Model Theory In this section, we derive the loss decomposition for the nonlinear-attention model on the difficult task studied in Section 4.2. We first introduce the notation. Throughout this section, E denotes an expectation over contexts. Let L be the number of layers and P the number of context tokens. We use ℓ=1,…,L =1,…,L for layers and i=1,…,Pi=1,…,P for tokens. Write k for the number of labels selected by the top-k task. We measure the relevance of each context token to the test token using cosine similarity: si=test⊤i‖test‖2‖i‖2,si(1)≥⋯≥si(P),Sk=i(1),…,i(k). gathereds_i= x_test x_i\| x_test\|_2\| x_i\|_2,\\ s_i_(1)≥·s≥ s_i_(P), S_k=\i_(1),…,i_(k)\. gathered (57) Thus SkS_k contains the k context indices closest to the test token. For the label column y=(y1,…,yP)⊤y=(y_1,…,y_P) , define the target row tk=(tk,1,…,tk,P)t_k=(t_k,1,…,t_k,P) by tk,i=1/k,i∈Sk,0,i∉Sk,y∗=tky=1k∑i∈Skyi. gatheredt_k,i= cases1/k,&i∈ S_k,\\ 0,&i∉ S_k, cases\\ y^*=t_ky= 1k _i∈ S_ky_i. gathered (58) Let pℓip_ i be the attention weight from the test token to context token i at layer ℓ , with softmax normalized over the P context sources since the test token does not attend to itself: pℓ=(pℓ1,…,pℓP),∑i=1Ppℓi=1.p_ =(p_ 1,…,p_ P), _i=1^Pp_ i=1. (59) With fixed context states, each layer reads the same label column y. The common scalar gain gLg_L includes the value map and readout: y^L=1L∑ℓ=1Lfℓ,fℓ=gLpℓy=gL∑i=1Ppℓiyi,gL=WoWvWy. gathered y_L= 1L _ =1^Lf_ ,\\ f_ =g_Lp_ y=g_L _i=1^Pp_ iy_i,\\ g_L=W_oW_vW_y. gathered (60) Define the layer error relative to the same target as eℓ=fℓ−y∗e_ =f_ -y^*. For L>1L>1, assuming finite expected squared errors, let AL=1L∑ℓ=1L[eℓ2],CL=1L(L−1)∑ℓ≠m[eℓem]. gatheredA_L= 1L _ =1^LE[e_ ^2],\\ C_L= 1L(L-1) _ ≠ mE[e_ e_m]. gathered (61) The second sum is over ordered distinct pairs. Since y^L−y∗=L−1∑ℓeℓ y_L-y^*=L^-1 _ e_ , expanding the square gives ℛL _L =[(y^L−y∗)2] =E[( y_L-y^*)^2] (62) =1L2[∑ℓ[eℓ2]+∑ℓ≠m[eℓem]] = 1L^2 [ _ E[e_ ^2]+ _ ≠ mE[e_ e_m] ] =ALL+L−1LCL = A_LL+ L-1LC_L =CL+AL−CLL. =C_L+ A_L-C_LL. Expanding pairwise differences and using equation 61, ∑ℓ≠m[(eℓ−em)2] _ ≠ mE[(e_ -e_m)^2] =2(L−1)∑ℓ[eℓ2]−2∑ℓ≠m[eℓem] =2(L-1) _ E[e_ ^2]-2 _ ≠ mE[e_ e_m] =2L(L−1)(AL−CL). =2L(L-1)(A_L-C_L). Dividing by 2L(L−1)2L(L-1) gives AL−CL=12L(L−1)∑ℓ≠m[(eℓ−em)2]≥0.A_L-C_L= 12L(L-1) _ ≠ mE[(e_ -e_m)^2]≥ 0. (63) CLC_L captures the error shared across layers, whereas AL−CLA_L-C_L measures their disagreement. Averaging reduces the latter contribution to (AL−CL)/L(A_L-C_L)/L. The shared component cannot be removed through aggregation and therefore gives rise to the irreducible loss, while differences among layer errors can partially cancel and produce the reducible loss. Importantly, this decomposition does not by itself imply inverse-depth scaling. We first establish the emergence of inverse-depth scaling empirically through systematic experiments and rigorous power-law fitting, and then use the decomposition to identify the components and mechanisms underlying this behavior. This decomposition relies on the parallel interpretation of our controlled model: the layers act as independently parameterized predictors whose aggregate forms the final output. A layerwise decomposition is therefore natural in this setting. In the generalized settings considered below, however, it is neither the only possible decomposition nor necessarily the most natural one. For example, when inputs and labels share a single stream, the model becomes cascaded and each layer depends on the preceding layers. A mean-field treatment (Mei et al., 2018; Chizat and Bach, 2018; Rotskoff and Vanden-Eijnden, 2018) may then be more appropriate, separating each layer’s output into a mean component coupled to the other layers and a layer-specific fluctuation. Developing a rigorous decomposition for more complex systems, including large language models, remains an important direction for understanding the role of depth in realistic models. Appendix C Generalized Model Experiments and Analysis We now relax some design choices from the controlled model in Section 2.2 to move the architecture closer to standard Transformers and LLMs. First, the original two-stream encoding separates token matching from label propagation: queries and keys interact only with the input stream, whereas values and the output map interact only with the label stream. Second, all layers share a value matrix, which improves training stability and keeps the model concise. Third, the nonlinear-attention model initially keeps the context states fixed, isolating the process by which different layers retrieve relevant information from the context. In the following subsections, we relax these assumptions one at a time and conduct analogous experiments and analyses for each extension. C.1 Encoding Input and Label Information into One Space Section 2.2 initializes separate input and label streams, Hx=WxXH_x=W_xX and Hy0=WyinH_y^0=W_y y^in. In standard LLMs, token and label information are commonly encoded in a shared representation space. We therefore embed both sources of information in a shared token state: H0=WxX+Wyin∈ℝN×(P+1).H^0=W_xX+W_y y^in ^N×(P+1). (64) Queries, keys, and values are now all computed from the same evolving state: Qℓ=WqℓHℓ−1,Kℓ=WkℓHℓ−1,Vℓ=WvHℓ−1.Q =W_q H -1, K =W_k H -1, V =W_vH -1. (65) For nonlinear attention, the shared state is updated as Aℓ A =⊙softmax((Qℓ)⊤Kℓ), = M \! ((Q ) K ), (66) Hℓ H =Hℓ−1+1LVℓ(Aℓ)⊤. =H -1+ 1LV (A ) . Thus token matching, information propagation, and residual updates all act within a common representation space, and each layer operates on the state produced by the preceding layers. To define the two decomposition terms in this cascaded model, let zℓ=Wohtestℓz_ =W_oh_test denote the test-token readout after layer ℓ . Define the base readout and layer increments as bL=z0,dℓ=zℓ−zℓ−1,y^L=bL+∑ℓ=1Ldℓ.b_L=z_0, d_ =z_ -z_ -1, y_L=b_L+ _ =1^Ld_ . (67) We can express the final prediction as an average of effective layerwise predictors by setting fℓ=bL+Ldℓ,eℓ=fℓ−y∗,y^L−y∗=1L∑ℓ=1Leℓ.f_ =b_L+Ld_ , e_ =f_ -y^*, y_L-y^*= 1L _ =1^Le_ . (68) For L>1L>1, the shared-error and disagreement terms are then defined as CL C_L =1L(L−1)∑ℓ≠m[eℓem], = 1L(L-1) _ ≠ mE[e_ e_m], (69) AL−CL A_L-C_L =12L(L−1)∑ℓ≠m[(eℓ−em)2] = 12L(L-1) _ ≠ mE[(e_ -e_m)^2] =1L−1∑ℓ=1L[(fℓ−y^L)2]. = 1L-1 _ =1^LE[(f_ - y_L)^2]. With these definitions, the loss retains the exact identity ℛL=CL+AL−CLL.R_L=C_L+ A_L-C_LL. (70) We evaluate this generalized model on the difficult task. Figure 9 shows that CLC_L and AL−CLA_L-C_L continue to approach finite constants, while the loss exhibits an approximate inverse-depth trend beyond moderate depth. Figure 9: The single-stream nonlinear-attention model exhibits a similar inverse-depth scaling signal. (a) The model is evaluated under isotropic data spectra. (b,c) The shared-error term CLC_L and disagreement term AL−CLA_L-C_L both approach finite constants with depth. C.2 Layer-Decoupled Value Matrices In standard LLMs, value matrices are typically parameterized independently across layers. We therefore replace the value matrix WvW_v shared in Section 2.2 with independently parameterized matrices WvℓW_v , giving Qℓ=WqℓHx,Kℓ=WkℓHx,Vℓ=WvℓHyℓ−1.Q =W_q H_x, K =W_k H_x, V =W_v H_y -1. (71) This modification allows each layer to apply a distinct transformation to the retrieved label information. Let pℓp_ denote the attention weights from the test token to the context tokens, and let y collect their labels. The corresponding layerwise predictor and error are gℓ=WoWvℓWy,fℓ=gℓpℓy,eℓ=fℓ−y∗,y^L=1L∑ℓ=1Lfℓ.g_ =W_oW_v W_y, f_ =g_ p_ y, e_ =f_ -y^*, y_L= 1L _ =1^Lf_ . (72) For L>1L>1, the shared-error and disagreement terms are CL C_L =1L(L−1)∑ℓ≠m[eℓem], = 1L(L-1) _ ≠ mE[e_ e_m], (73) AL−CL A_L-C_L =12L(L−1)∑ℓ≠m[(eℓ−em)2] = 12L(L-1) _ ≠ mE[(e_ -e_m)^2] =12L(L−1)∑ℓ≠m[((gℓpℓ−gmpm)y)2]. = 12L(L-1) _ ≠ mE\! [ ((g_ p_ -g_mp_m)y )^2 ]. The loss therefore retains the exact identity ℛL=CL+AL−CLL.R_L=C_L+ A_L-C_LL. (74) We evaluate this generalized model on the difficult task. Figure 10 shows that CLC_L and AL−CLA_L-C_L continue to approach finite constants, while the loss displays an approximate inverse-depth trend beyond moderate depth. Thus, the same inverse-depth scaling signal remains visible after relaxing the shared-value assumption. Figure 10: The nonlinear-attention model with layer-specific value matrices exhibits a similar inverse-depth scaling signal. (a) The model is evaluated under isotropic data spectra. (b,c) The shared-error term CLC_L and disagreement term AL−CLA_L-C_L both approach finite constants with depth. C.3 Allowing Context Updates in the Nonlinear-Attention Model Our linear-attention model in Section 2.2 follows Bordelon et al. (2026), including their attention-masking scheme. Let P∈ℝP 1_P ^P denote the all-ones vector. For a single test token, their mask can be written as =(−PP⊤P×1P⊤0), M= pmatrix- 1_P 1_P & 0_P× 1\\ 1_P &0 pmatrix, (75) where the upper-left block governs context-to-context updates, while the lower-left block allows the test token to attend to the context tokens. Intuitively, this sign structure lets linear attention extract data structure at progressively finer spectral resolutions. The negative context-to-context block preferentially subtracts components shared by highly similar context tokens from their label states. Dominant spectral features produce larger similarity-weighted updates and are therefore resolved earlier, whereas weaker features persist until greater depth and can be learned in deeper layers. This behavior is consistent with the sequential mode learning in Figure 3(c). By contrast, the positive test-to-context block allows retrieved information to accumulate in the test token’s label state, which is initialized at zero and updated across layers. To isolate how the test token retrieves and aggregates information from the context, the controlled nonlinear-attention model in Section 2.2 initially disables context updates by setting the upper-left block to zero: (0)=(P×P×1P⊤0). M(0)= pmatrix 0_P× P& 0_P× 1\\ 1_P &0 pmatrix. (76) Standard Transformers, however, update context representations across layers. To examine how these updates affect the nonlinear-attention model, we parameterize the previously zero context-to-context block by a trainable scalar m. Specifically, we use (m)=(mPP⊤P×1P⊤0), M(m)= pmatrixm 1_P 1_P & 0_P× 1\\ 1_P &0 pmatrix, (77) where the scalar m is shared across all layers and learned jointly with the other model parameters. We evaluate this generalized model on the difficult task with isotropic data spectra, using top-22 and top-1616 targets. Figure 11(a,b) shows qualitatively different depth dependence. For top-22, the loss continues to decrease substantially with depth and exhibits a signal consistent with inverse-depth scaling. For top-1616, by contrast, increasing depth does not improve the loss. The learned context-update scalar m provides an additional diagnostic. Figure 11(c,d) shows that the learned scalar m varies substantially with depth for top-22, whereas for top-1616 it remains near the stable negative value −2.3-2.3. Figure 11: Enabling context updates preserves an inverse-depth scaling signal on the top-22 task but not on the top-1616 task. Under isotropic data spectra, (a) the top-22 loss exhibits an inverse-depth scaling signal, whereas (b) the top-1616 loss quickly saturates. (c,d) The learned context-update scalar m varies with depth for top-22 but stabilizes for top-1616. To interpret this task-dependent behavior, we first derive its exact effect in a two-layer model. Write the label state as Hyℓ=[Zℓ,hℓ]H_y =[Z_ ,h_ ], where Zℓ∈ℝN×PZ_ ^N× P contains the context states and hℓ∈ℝNh_ ^N is the test-token state. Let y=(y1,…,yP)⊤y=(y_1,…,y_P) , so that Z0=Wyy⊤Z_0=W_yy and h0=0h_0=0. Without changing the mechanism of interest we take Wv=INW_v=I_N for simplicity. Let pℓ∈ℝ1×Pp_ ^1× P denote the test-to-context attention row at layer ℓ , and let A1∈ℝP×PA_1 ^P× P denote the first layer’s context-to-context attention matrix. Applying equation 10 with the mask in equation 77 gives Z1 Z_1 =Z0+m2Z0A1⊤, =Z_0+ m2Z_0A_1 , (78) h1 h_1 =12Z0p1⊤, = 12Z_0p_1 , h2 h_2 =h1+12Z1p2⊤. =h_1+ 12Z_1p_2 . Substituting Z0=Wyy⊤Z_0=W_yy gives h2=Wy[12(p1+p2)y+m4p2A1y].h_2=W_y\! [ 12(p_1+p_2)y+ m4p_2A_1y ]. (79) Defining the scalar readout gain g=WoWyg=W_oW_y, the prediction is y^2=g2(p1+p2)y+gm4p2A1y. y_2= g2(p_1+p_2)y+ gm4p_2A_1y. (80) The first term reads the original context labels directly. The second reads them after A1A_1 has mixed information across context tokens. For arbitrary depth, the corresponding recursion is Zℓ Z_ =Zℓ−1(IP+mLAℓ⊤), =Z_ -1 (I_P+ mLA_ ), (81) hℓ h_ =hℓ−1+1LZℓ−1pℓ⊤. =h_ -1+ 1LZ_ -1p_ . Consequently, y^L=gL[p1y+∑ℓ=2Lpℓ(IP+mLAℓ−1)⋯(IP+mLA1)y]. y_L= gL [p_1y+ _ =2^Lp_ (I_P+ mLA_ -1 )·s (I_P+ mLA_1 )y ]. (82) Thus, context updates affect prediction through operators IP+(m/L)AℓI_P+(m/L)A_ . Whether and how they improve depth utilization depends on the spectral structure of AℓA_ . Figure 12 visualizes the two context attention structures. For the top-22 task, the measured A1A_1 is nearly the identity. For the top-1616 task, the measured A1A_1 is broader and flatter. This contrast is induced by the sharpness of the target distribution. The sharp top-22 target requires the model to preserve the identities of a few selected tokens, favoring well-separated embeddings and a nearly identity attention map. The flatter top-1616 target instead favors stronger coupling across embeddings and therefore produces a broader attention map. Figure 12: The learned context-attention structure depends strongly on the aggregation target. The context-attention matrix A1A_1 is (a) sharply concentrated on the diagonal for top-22 task, but (b) distributed broadly across context tokens for top-1616 task. To quantify this contrast, we model each measured attention matrix by an identity component IPI_P and a uniform mixing component JPJ_P. Specifically, for k∈2,16k∈\2,16\, we fit ρk _k by least squares in A1(k)≈ρkIP+(1−ρk)JP,JP:=PP⊤P.A_1^(k)≈ _kI_P+(1- _k)J_P, J_P:= 1_P 1_P P. (83) The identity term preserves each context embedding separately, whereas JPJ_P replaces every embedding by their common average. The fitted results are ρ2 _2 =0.9999, =0.9999, Rtop-22 R_top-2^2 >0.99999, >0.99999, (84) ρ16 _16 =0.0989, =0.0989, Rtop-162 R_top-16^2 =0.99996. =0.99996. This model therefore confirms that the top-22 matrix is almost exactly the identity, while the top-1616 matrix is dominated by uniform cross-token mixing. For top-22, the near-identity structure gives p2A1≈p2p_2A_1≈ p_2, and y^2≈g2p1y+g2(1+m/2)p2y. y_2≈ g2p_1y+ g2(1+m/2)p_2y. (85) The context update therefore introduces no independent token-weighting direction: the prediction remains an aggregation of p1p_1 and p2p_2. Accordingly, its loss remains close to that of the fixed-context model and still exhibits approximate inverse-depth scaling over the observed depth range. For top-1616, let u=P⊤/Pu= 1_P /P. Since p2P=1p_2 1_P=1, we have p2JP=up_2J_P=u and hence p2A1≈ρ16p2+(1−ρ16)u,p_2A_1≈ _16p_2+(1- _16)u, (86) Equation 86 shows that the context update adds the uniform token-weighting direction u, beyond the direct rows p1p_1 and p2p_2. Defining p¯=(p1+p2)/2 p=(p_1+p_2)/2, the exact two-layer prediction is y^2=g(p¯+m4p2A1)y. y_2=g ( p+ m4p_2A_1 )y. (87) A negative m therefore produces a global shift that subtracts the common attention floor, rather than merely combining more direct attention rows. This additional degree of freedom allows the model to achieve low loss with few layers, explaining its early saturation. The same argument extends to models with more than two layers. This distinction is also relevant to language modeling. Many language tasks require focusing on only a few relevant tokens and learning a sparse and peaked distribution. In such regimes, context interactions may primarily reweight information that has already been selected. Depth utilization can therefore remain governed by selective retrieval and cross-layer aggregation, making the inverse-depth mechanism relevant even when context representations are not fixed. Overall, these generalized models exhibit inverse-depth scaling signals that are qualitatively consistent with the controlled-model analysis over the accessible depth ranges. Training these models at much greater depths is substantially more computationally expensive, preventing experiments on the same scale as those conducted for the controlled model. We therefore interpret these results as evidence of similar scaling signals, rather than as a definitive guarantee of asymptotic inverse-depth scaling. Their appearance under more realistic architectural choices nevertheless suggests that the mechanism may remain relevant to practical models. Appendix D More Experiments and Fitting Details This section supplements the controlled-model results with additional experiments and provides the fitting procedures used to quantify the observed depth-scaling laws. We report fits for both the easy and difficult tasks, analyze context-dependent token selection, test broader targets and data spectra, and examine the convergence of the shared-error term. D.1 Power-Law Fitting Procedures We use a common log-space fitting procedure to quantify the depth-scaling laws in Sections 3.1 and 4.2. Each spectrum is fitted independently using ℛL=L0+cL−α,c>0,α>0,R_L=L_0+cL^-α, c>0, α>0, (88) where L0L_0 is the asymptotic loss floor and α is the decay exponent. For observations at depths Li\L_i\, we minimize the squared residuals between the logarithms of the observed and modeled losses: (L^0,c^,α^)=argminL0,c,α∑i[logℛLi−log(L0+cLi−α)]2.( L_0, c, α)= L_0,c,α _i [ _L_i- \! (L_0+cL_i^-α ) ]^2. (89) For the easy task, we set L0=0L_0=0. As shown in Appendix A.1, given sufficient depth, width, dataset size, context length, and training time, the linear-attention model can solve this task exactly by using a covariance polynomial to approximate the identity map. For the difficult task, the decomposition in Appendix B contains an error component shared across layers, so we estimate L0L_0 jointly with c and α. D.2 Complementary Token Selection Across Contexts We further compare how nonlinear-attention models of different depths perform on the difficult task. We train an L=1L=1 model and an L=256L=256 model on the difficult task with k=2k=2 under identical conditions. We then sample 10,24010,240 fresh contexts independently from the data distribution and evaluate both models on the same set. For each context, we order the tokens by their similarity to the test token. Under the parallel interpretation developed in Appendix B, the final output aggregates the contributions of the individual layers. We therefore summarize their collective token selection by the aggregated attention distribution and compare it with the target distribution over the ordered tokens. Formally, p¯L(c)=1L∑ℓ=1Lpℓ(c),p∗=(1/2,1/2,0,…,0),EL(c)=‖p¯L(c)−p∗‖22. gathered p_L(c)= 1L _ =1^Lp_ (c), p^*=(1/2,1/2,0,…,0),\\ E_L(c)=\| p_L(c)-p^*\|_2^2. gathered (90) Lower values indicate more accurate identification and weighting of the two target tokens. For each model separately, we sort the sampled contexts by EL(c)E_L(c) and denote the qqth percentile by QL(q)Q_L(q). Thus, small q corresponds to contexts on which a model most closely matches the target distribution, whereas large q probes its high-error tail. Figure 13(a) compares the resulting error distributions. Panel (b) reports ΔQ(q)=Q1(q)−Q256(q), Q(q)=Q_1(q)-Q_256(q), (91) Positive ΔQ(q) Q(q) therefore indicates that the deeper model has lower selection loss at percentile q. The one-layer model is slightly more accurate on the contexts in its lowest-error tail, but its error grows much more rapidly across harder contexts. The 256256-layer model instead markedly reduces the upper tail, indicating more reliable token selection across the context distribution. This pattern follows from the fact that each query–key parameterization must serve the entire context distribution. With only one layer, a single selection rule may fit some contexts particularly well but cannot identify the correct top-22 tokens uniformly. Greater depth supplies multiple independently parameterized rules whose errors can be complementary across contexts. Their aggregation improves coverage: for a context that is difficult for one rule, other layers may still identify the relevant tokens, thereby reducing the high-error tail. The slight advantage of the one-layer model at the lowest percentiles suggests a possible tradeoff between specialization and coverage. A single selection rule may become highly accurate on a limited subset of contexts, whereas complementary rules across layers can cover a broader range. This observation suggests a testable hypothesis for settings with finitely many or clustered context types: different layers may specialize in different context classes, and their aggregation may combine these specialized behaviors. If this hypothesis holds, it would suggest that current architectures use depth inefficiently, as not every layer makes an indispensable contribution. Further exploring designs that exploit such layerwise specialization may help LLMs utilize depth better. Figure 13: Greater depth makes token selection more reliable across contexts. (a) Selection-loss percentiles for L=1L=1 and L=256L=256. (b) Their difference ΔQ(q)=Q1(q)−Q256(q) Q(q)=Q_1(q)-Q_256(q); positive values favor L=256L=256. D.3 Broader Targets and Data Spectra The main-text experiments with the nonlinear-attention model on the difficult task use top-22 aggregation and vary one spectral parameter while fixing the other at zero. We now test whether the observed depth scaling depends on either restriction. First, we retain the top-22 target but consider jointly structured input and task spectra, with both a and b nonzero. Second, we replace top-22 aggregation with top-1616 aggregation, yielding a flatter target distribution, and evaluate several spectral settings. For each configuration, we fit the evaluation loss using the model and log-space objective defined in Appendix D.1. (a) Top-22 aggregation (b) Top-1616 aggregation (c) Shared error (d) Error disagreement Figure 14: Emergent inverse-depth scaling in the deep regime generalizes across broader targets and data spectra. (a) Performance on the top-22 task with both spectral parameters nonzero. (b) Performance on top-1616 aggregation across representative spectra. (c) CLC_L and (d) AL−CLA_L-C_L converge to finite constants with depth for the three top-22 spectra. Table 4: Fitted depth-scaling exponents remain close to one across broader targets and data spectra. Target (a,b)(a,b) L0L_0 α±SEα R2R^2 Top-22 (0.25,0.25)(0.25,0.25) 0.818068 0.97±0.010.97± 0.01 0.9998 (0.25,0.5)(0.25,0.5) 0.559694 0.93±0.060.93± 0.06 0.9984 (0.5,0.5)(0.5,0.5) 0.382188 0.97±0.010.97± 0.01 0.9999 Top-1616 (0,0.25)(0,0.25) 0.173237 0.79±0.020.79± 0.02 0.9999 (0,0.5)(0,0.5) 0.114436 0.88±0.030.88± 0.03 0.9998 (0.25,0.25)(0.25,0.25) 0.115248 0.81±0.020.81± 0.02 0.9998 (0.25,0.5)(0.25,0.5) 0.079785 0.81±0.040.81± 0.04 0.9994 Figure 14 shows approximately inverse-depth loss decay across both extensions, and Table 4 quantifies the corresponding exponents. For the jointly nonzero top-22 settings, panels (c,d) further show that both decomposition terms approach spectrum-dependent finite values, matching the convergence pattern observed in the main text. With jointly nonzero spectra, the top-22 exponents remain close to one. Note that the top-1616 exponents deviate relatively more from one at the depths considered, suggesting that the top-1616 task may enter the asymptotic inverse-depth regime only at greater depths. This contrast also suggests that inverse-depth scaling from cross-layer aggregation may be more pronounced for sparse, sharply concentrated target distributions. These results show that the inverse-depth scaling behavior is not restricted to specific spectra or target distributions. D.4 Convergence of the Shared-Error Term The direct loss fits in Section 4.2 provide empirical evidence for approximate inverse-depth scaling over the tested depth range. As a consistency check on the decomposition in Appendix B, we examine how its shared-error term CLC_L approaches the fitted loss floor. For each spectrum, we fix L0L_0 to the corresponding estimate in Table 1 and model the gap as L0−CL=cL−α,c>0,α>0.L_0-C_L=cL^-α, c>0, α>0. (92) Following the log-space procedure in Appendix D.1, we estimate the two parameters by least squares: (c^,α^)=argminc>0,α>0∑i[log(L0−CLi)−log(cLi−α)]2.( c, α)= c>0,\,α>0 _i [ (L_0-C_L_i)- \! (cL_i^-α ) ]^2. (93) Table 5: The gap L0−CLL_0-C_L decays approximately inversely with depth over the fitted range. (a,b)(a,b) α±SEα R2R^2 (0,0)(0,0) 0.980±0.0070.980± 0.007 0.99981 (0,0.25)(0,0.25) 0.983±0.0380.983± 0.038 0.99396 (0,0.5)(0,0.5) 0.977±0.0140.977± 0.014 0.99923 (0,1)(0,1) 0.982±0.0110.982± 0.011 0.99946 (0.25,0)(0.25,0) 0.990±0.0150.990± 0.015 0.99904 (0.5,0)(0.5,0) 0.986±0.0050.986± 0.005 0.99988 (1,0)(1,0) 0.994±0.0050.994± 0.005 0.99992 With L0L_0 fixed by the direct loss fits, the high R2R^2 values and small fitting uncertainties show that L0−CLL_0-C_L is well described by a power law over the tested depths. Its fitted decay exponents are close to one, consistent with the inverse-depth trend observed in the loss.