Paper deep dive

Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension

Jinhao Zhang, Zeyu Liu, Zicheng Yan, Yunquan Zhang, Guangming Tan, Fangming Liu, Daning Cheng

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 142

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 8/1/2026, 10:42:30 AM

Summary

This paper introduces the 'effective alignment dimension' to quantify the signal-noise geometry of activation gradients in neural networks, providing a finite-sample certificate for the probability of misalignment between training and test gradients. The authors derive bounds showing that wider models generally exhibit higher effective alignment dimensions and lower misalignment probabilities, validated through experiments on LLaMA-style Transformers, Pythia, and ResNet-20.

Entities (8)

Effective Alignment Dimension · metric · 98%LLaMA-style Transformers · model-architecture · 95%Pythia · model-architecture · 95%ResNet-20 · model-architecture · 95%Activation Gradients · concept · 90%Misalignment Probability · concept · 90%Normalized Residual Networks · model-architecture · 90%Cheng et al. (2026) · citation · 85%

Relation Signals (5)

Effective Alignment Dimension → bounds → Misalignment Probability

confidence 95% · obtain a finite-sample upper bound on misalignment probability. The bound depends only on the effective alignment dimension...

Effective Alignment Dimension → describes → Activation Gradients

confidence 95% · a measurable quantity describing the signal-noise geometry of activation gradients.

Model Width → decreases → Misalignment Probability

confidence 90% · wider models exhibit ... lower empirical misalignment.

Model Width → increases → Effective Alignment Dimension

confidence 90% · wider models exhibit larger effective alignment dimensions

Effective Alignment Dimension → predicts → Held-out Loss Changes

confidence 85% · Direct residual interventions confirm that the alignment statistic predicts the sign and magnitude of held-out loss changes.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Existing theories of neural-network width characterize asymptotic limits, but provide limited guidance on whether an expansion direction identified from finite training data remains beneficial on unseen data. We study this problem for function-preserving residual expansion and introduce the effective alignment dimension, a measurable quantity describing the signal-noise geometry of activation gradients. By deriving the exact mean and variance of the inner product between independently estimated training and test gradients, we obtain a finite-sample upper bound on misalignment probability. The bound depends only on the effective alignment dimension and an effective sample size, requiring finite second moments and a nonzero population gradient, without covariance spectral assumptions or prescribed width-growth rates. We integrate this certificate into the train-test residual-expansion framework, yielding a high-probability condition for test-risk improvement. Experiments across width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that wider models exhibit larger effective alignment dimensions and lower empirical misalignment. Direct residual interventions confirm that the alignment statistic predicts the sign and magnitude of held-out loss changes.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

141,164 characters extracted from source content.

Expand or collapse full text

Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension Jinhao Zhang1, Zeyu Liu2, Zicheng Yan3, Yunquan Zhang2, Guangming Tan2, Fangming Liu4, Daning Cheng2 Abstract Existing theories of neural-network width characterize asymptotic limits, but provide limited guidance on whether an expansion direction identified from finite training data remains beneficial on unseen data. We study this problem for function-preserving residual expansion and introduce the effective alignment dimension, a measurable quantity describing the signal–noise geometry of activation gradients. By deriving the exact mean and variance of the inner product between independently estimated training and test gradients, we obtain a finite-sample upper bound on misalignment probability. The bound depends only on the effective alignment dimension and an effective sample size, requiring finite second moments and a nonzero population gradient, without covariance spectral assumptions or prescribed width-growth rates. We integrate this certificate into the train–test residual-expansion framework, yielding a high-probability condition for test-risk improvement. Experiments across width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that wider models exhibit larger effective alignment dimensions and lower empirical misalignment. Direct residual interventions confirm that the alignment statistic predicts the sign and magnitude of held-out loss changes. Introduction Width is one of the most thoroughly studied dimensions of neural scaling. A large body of theory examines large-width limits and finite-width corrections through Gaussian processes, kernel dynamics, feature-learning limits, and width-dependent fluctuations. These theories do not, however, directly provide a finite-sample certificate for whether a local expansion direction identified in a specific trained model at a specific insertion point will transfer to held-out data. This question arises naturally in model growth, where progressive stacking, Net2Net-style expansion, and learned growth operators must determine whether a proposed insertion remains beneficial beyond the finite training set. Recent work on normalized residual networks (Cheng et al. 2026) addresses this problem by decomposing depth growth into representational gain, optimization gain, and generalization transfer. It establishes a qualitative mechanism: a residual block inserted at a function-preserving zero-output initialization creates a first-order descent direction whenever its features are not orthogonal to the insertion-point error signal, while width improves the finite-sample observability of that direction. We study how reliably a training-identified direction remains favorable on independent test data. The remaining gap is quantitative finite-sample observability. The prior width-dependent alignment result requires control of the covariance spectrum, total variance, and growth of the mean gradient signal. These assumptions yield a clean asymptotic width rate, but not an instance-specific certificate for a given model and training–test pair. Moreover, the separate worst-case conditions do not expose the relative signal–noise geometry determining finite-sample directional stability. We retain the activation-space formulation of the prior framework. For each example, the sample-level signal is the loss gradient with respect to the hidden representation at the insertion point. Averaging over the training and test sets produces two empirical activation-gradient directions whose alignment measures whether the training-identified direction remains favorable on independent test data. Positive alignment supports the same local residual direction on both samples, whereas nonpositive alignment marks a failure of finite-sample directional transfer. Through the function-preserving jumpboard construction, this event enters the comparison between the original and expanded models. Rather than controlling training and test deviations separately, we analyze their alignment statistic directly. An exact second-moment characterization yields an instance-specific finite-sample certificate summarized by a dimensionless effective alignment dimension. Together with an effective sample size determined by the training and test set sizes, it controls the probability of nonpositive alignment, with a stronger certificate as their product increases. The result requires only a finite second moment and a nonzero population mean for the sample-level activation gradient. It requires no covariance spectral bound, prescribed width-growth rate, or fourth-moment condition, and makes no architectural assumption on the inserted module or backbone. Under the covariance-spectrum, total-variance, and signal-growth conditions of prior work, it recovers the same-order width dependence. These assumptions thus become sufficient conditions for linear growth of the effective alignment dimension with width, rather than prerequisites for the guarantee. The dimension and its signal and noise quantities can be estimated from a trained model with a small number of forward–backward passes, without explicitly forming the covariance matrix. We then integrate the certificate into the existing direct train–test expansion framework. Under the original assumptions ensuring an improving jumpboard model, realizability of the residual direction, subsequent optimization gain, and uniform generalization control, the resulting theorem replaces the previous alignment-control step with an instance-specific guarantee in terms of the training and test set sizes and the effective alignment dimension, while leaving the representational, optimization, and generalization components unchanged. The probability of successful directional transfer is therefore expressed directly through these quantities. This analysis also clarifies the role of width. Width does not enter the certificate independently, but changes the activation-gradient distribution and hence the relative signal–noise geometry summarized by the effective alignment dimension. If the relative-noise structure does not deteriorate with width, increasing width cannot worsen the certificate; if the effective alignment dimension increases, the alignment-failure bound decreases. The benefit of width is therefore conditional rather than automatic and can be evaluated on the model at hand. Experiments on width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 support this account: dalignd_align generally increases with width while empirical misalignment decreases under fixed sampling budgets. Varying the training and test sample sizes further supports the predicted finite-sample dependence. Direct residual interventions verify that positive alignment predicts the sign and local magnitude of realizable held-out loss changes. Contributions. ❶We derive an exact second-moment characterization of the alignment between training- and test-averaged activation gradients, yielding an instance-specific finite-sample certificate summarized by the effective alignment dimension. ❷ We replace the previous width-dependent alignment step in the direct train–test expansion framework with this certificate, obtaining a joint high-probability guarantee in terms of training size, test size, and effective alignment dimension. ❸ Across width-controlled Transformers and ResNets, we show that the effective alignment dimension generally increases with width alongside reduced empirical misalignment. Direct residual interventions further verify that the alignment statistic predicts both the sign and local magnitude of held-out loss changes. Related Work Theoretical studies of neural-network width primarily characterize large-width limits and deviations around them. NNGP and NTK limits yield Gaussian-process and kernel descriptions (Neal 1996; Lee et al. 2018; Jacot et al. 2018), while mean-field and maximal-update parameterizations retain nontrivial feature learning (Mei et al. 2018; Chizat and Bach 2018; Yang and Hu 2021; Yang et al. 2021). Finite-width analyses quantify width-dependent fluctuations of kernels and predictions (Bordelon and Pehlevan 2023), whereas neural scaling-law theories describe average loss as data or model size grows (Bahri et al. 2024). A separate literature studies coherent gradients, parameter-gradient signal-to-noise ratios, and gradient noise scale as diagnostics of optimization and generalization (Chatterjee 2020; Liu et al. 2020; McCandlish et al. 2018), but does not analyze the two-sample sign-reversal probability of an update selected for a newly inserted module. Function-preserving model-growth methods, including Net2Net, progressive stacking, bert2BERT, and LiGO, reuse trained models to initialize larger ones (Chen et al. 2016; Gong et al. 2019; Chen et al. 2022; Wang et al. 2023); concurrent work emphasizes the geometry of function preservation and post-insertion plasticity (Lok 2026; Khemais 2026). Closest to our setting, Cheng et al. (Cheng et al. 2026) decompose residual expansion into representational, optimization, and generalization components and derive a width-dependent train/test activation-gradient alignment guarantee. Assumptions and Notation We follow the activation-space notation and the direct train–test comparison framework of Cheng et al. (2026). To match the notation used in our theoretical analysis and experiments, we denote the activation-gradient signal at the insertion point by q. This is the same activation-gradient object denoted by ζ in the train–test alignment theorem of the prior framework. Throughout the paper, we fix a well-trained reference model fold∗f_old^*, and fold=ftop∘fbot∈ℋoldf_old=f_top f_bot _old and a candidate insertion point. For an input x, let z(x):=fbot(x)∈ℝNz(x):=f_bot(x) ^N denote the hidden representation at that point, where N is its width. Let ℋnewH_new denote the hypothesis class obtained by inserting a residual function at the selected location. For any predictor f and loss function ℓ , define the population, training, and test risks by R(f):=(x,y)∼[ℓ(f(x),y)]R(f):=E_(x,y) [ (f(x),y ) ], ℒtrain(f)=1M∑i=1Mℓ(f(xi),yi)L_train(f)= 1M _i=1^M (f(x_i),y_i ), ℒtest(f):=1K∑j=1Kℓ(f(x~j),y~j).L_test(f):= 1K _j=1^K (f( x_j), y_j ). Basic Assumptions Assumption 1 (Independent sampling and second moments). Let train=(xi,yi)i=1MS_train= \(x_i,y_i) \_i=1^M, test=(x~j,y~j)j=1KS_test= \( x_j, y_j) \_j=1^K be independent samples satisfying train∼i.i.d.MS_train i.i.d. D^M, test∼i.i.d.KS_test i.i.d. D^K, train⟂testS_train _test. All alignment probabilities are understood conditional on the fixed reference model fold∗f_old^* and the fixed insertion point. For a sample (x,y)(x,y), define the activation-gradient signal at the insertion point by q(x,y):=∇zℓ(ftop(z),y)|z=z(x)∈ℝNq(x,y):= . _z (f_top(z),y ) |_z=z(x) ^N. Define its population mean and covariance by μ¯:=(x,y)∼[q(x,y)] μ:=E_(x,y) [q(x,y) ], Σ:=Cov(x,y)∼⁡(q(x,y)) :=Cov_(x,y) (q(x,y) ). We assume (x,y)∼[‖q(x,y)‖22]<∞,μ¯≠0E_(x,y) [ \|q(x,y) \|_2^2 ]<∞, μ≠ 0. Assumption 1 is the only assumption required for Theorem 1. In particular, the finite-sample alignment result requires no covariance spectral bound, fourth-moment condition, or prescribed width-growth rate. Assumption 2 (Selection condition). Let f~S∈ℋnew f_S _new denote the empirical jumpboard model constructed from trainS_train in the direct train–test expansion framework, and let fnew∈ℋnewf_new _new denote the final expanded model. We assume ℒtrain(fnew)≤ℒtrain(f~S)L_train (f_new ) _train ( f_S ). Equivalently, define the nonnegative optimization gain by ΔERM:=ℒtrain(f~S)−ℒtrain(fnew)≥0 _ERM:=L_train ( f_S )-L_train (f_new )≥ 0. Assumption 3 (Uniform generalization control). Fix a confidence parameter δ∈(0,1)δ∈(0,1). We assume that there exist deterministic generalization radii ϵM,ϵK≥0 _M, _K≥ 0 such that Pr⁡[supf∈ℋnew|R(f)−ℒtrain(f)|≤ϵM]≥1−δ [ _f _new |R(f)-L_train(f) |≤ _M ]≥ 1-δ and Pr⁡[supf∈ℋnew|R(f)−ℒtest(f)|≤ϵK]≥1−δ [ _f _new |R(f)-L_test(f) |≤ _K ]≥ 1-δ. The boundedness, Lipschitz, normalization, and norm-control conditions used by Cheng et al. (2026) to instantiate ϵM _M and ϵK _K are restated in the appendix. The main text uses only the two uniform inequalities in Assumption 3. For the direct train–test comparison, define the finite-test jumpboard margin by ΔRtest:=ℒtest(fold∗)−ℒtest(f~S) _R^test:=L_test (f_old^* )-L_test ( f_S ). Despite the subscript R inherited from the prior notation, ΔRtest _R^test denotes a finite-test empirical margin rather than a population-risk difference. The small-step and residual-realizability conditions used to convert positive activation-gradient alignment into a positive finite-test jumpboard margin are invoked directly from Cheng et al. (2026, Corollary 3) and restated in the appendix. They are not assumptions of Theorem 1 and are therefore not repeated as global assumptions here. Key Alignment Quantities Define the empirical activation-gradient averages over the training and test sets by μM:=1M∑i=1Mq(xi,yi) _M:= 1M _i=1^Mq(x_i,y_i), gK:=1K∑j=1Kq(x~j,y~j)g_K:= 1K _j=1^Kq( x_j, y_j). The finite-sample activation-gradient misalignment event is μM⊤gK≤0 \ _M g_K≤ 0 \. Define the directional effective dimension by d∥:=‖μ¯‖24μ¯⊤Σμ¯d_ := \| μ \|_2^4 μ μ, and the covariance-energy effective dimension by d2:=‖μ¯‖24tr⁡(Σ2)=‖μ¯‖24‖Σ‖F2d_2:= \| μ \|_2^4tr( ^2)= \| μ \|_2^4 \| \|_F^2. If a denominator in either definition is zero, the corresponding effective dimension is defined as +∞+∞. The effective alignment dimension is dalign:=min⁡d∥,d2d_align:= \d_ ,d_2 \ (1) . The exact finite-sample certificate uses ΞM,K:=1Md∥+1Kd∥+1MKd2 _M,K:= 1Md_ + 1Kd_ + 1MKd_2, and Bexact(M,K):=ΞM,K1+ΞM,KB_exact(M,K):= _M,K1+ _M,K. Define the effective sample-size factor by neff(M,K):=MKM+K+1,n_eff(M,K):= MKM+K+1, (2) and the simplified effective-alignment certificate by Balign(M,K):=11+neff(M,K)dalign.B_align(M,K):= 11+n_eff(M,K)d_align. (3) The two certificates satisfy Bexact(M,K)≤Balign(M,K)<1.B_exact(M,K)≤ B_align(M,K)<1. When dependence on the insertion width must be made explicit, we write μ¯(N),Σ(N),d∥(N),d2(N),dalign(N). μ^(N), ^(N), d_ (N), d_2(N), d_align(N). At a fixed insertion width, the width argument is omitted. Theorem 1 uses only Assumption 1. The core risk comparison in Theorem 2 uses Assumptions 2 and 3. Its additional positive-margin statement combines Theorem 1 with the conditions of Cheng et al. (2026, Corollary 3), which are restated in the appendix. Main Results Main Theorems Our first result controls the probability that the activation-gradient direction estimated from the training set fails to remain positively aligned on the independent test set. In addition to the probability bound, we state the exact mean and variance identities that generate the two effective dimensions d∥d_ and d2d_2. Theorem 1 (Finite-sample activation-gradient alignment). Under Assumption 1, the train–test alignment statistic satisfies [μM⊤gK]=‖μ¯‖22E [ _M g_K ]= \| μ \|_2^2 and Var⁡(μM⊤gK)=(1M+1K)μ¯⊤Σμ¯+1MKtr⁡(Σ2).Var ( _M g_K )= ( 1M+ 1K ) μ μ+ 1MKtr ( ^2 ). Consequently, Pr⁡(μM⊤gK≤0)≤Bexact(M,K)=ΞM,K1+ΞM,K, ( _M g_K≤ 0 )≤ B_exact(M,K)= _M,K1+ _M,K, Moreover, the one-scalar certificate satisfies Pr⁡(μM⊤gK≤0)≤Bexact(M,K) ( _M g_K≤ 0 )≤ B_exact(M,K) ≤Balign(M,K)=11+neff(M,K)dalign ≤ B_align(M,K)= 11+n_eff(M,K)d_align The exact certificate retains the two relative-noise contributions separately. The simplified certificate replaces them by their conservative minimum, the effective alignment dimension. The theorem requires neither a covariance spectral bound nor a prescribed width-growth law: those conditions are unnecessary for the fixed-width finite-sample guarantee. The next result combines the new alignment certificate with the direct train–test comparison framework. Its first part is the algebraic comparison between the original model, the empirical jumpboard, and the final expanded model. Its second part replaces the previous width-dependent alignment bound by Theorem 1. Theorem 2 (Direct train–test expansion with effective-alignment control). Under Assumptions 2 and 3, with probability at least 1−2δ1-2δ, the final expanded model satisfies ℒtest(fnew)≤ℒtest(fold∗)−ΔRtest−ΔERM+2(ϵM+ϵK).L_test (f_new ) _test (f_old^* )- _R^test- _ERM+2 ( _M+ _K ). Consequently, on every realization for which ΔRtest+ΔERM>2(ϵM+ϵK), _R^test+ _ERM>2 ( _M+ _K ), one has the strict test-risk improvement ℒtest(fnew)<ℒtest(fold∗).L_test (f_new )<L_test (f_old^* ). Suppose additionally that the empirical jumpboard is chosen in the small-step regime and satisfies the residual-realizability conditions of Cheng et al. (2026, Corollary 3). Then positive activation-gradient alignment implies a positive finite-test jumpboard margin. Hence, by Theorem 1, Pr⁡(ΔRtest>0)≥1−Bexact(M,K)≥1−Balign(M,K). ( _R^test>0 )≥ 1-B_exact(M,K)≥ 1-B_align(M,K). The two uniform-generalization events and the positive-margin event therefore hold simultaneously with probability at least 1−2δ−Bexact(M,K)1-2δ-B_exact(M,K), and using the simplified certificate, with probability at least 1−2δ−Balign(M,K).1-2δ-B_align(M,K). The first part of Theorem 2 does not require the alignment event and remains valid even when the finite-test margin is zero or negative. The alignment-supported part instead quantifies how often this random margin is positive. Importantly, Theorem 1 controls the sign of the margin through the inherited jumpboard bridge; it does not lower-bound the magnitude of the margin. Strict improvement therefore continues to require the realized margin and optimization gain to dominate the two generalization radii. The Role of Width Theorem 1 is a fixed-width finite-sample result. To compare models of different widths, we make the width dependence of the activation-gradient distribution and the effective alignment dimension explicit. The simplified certificate becomes Balign(M,K;N)B_align(M,K;N). This factorization separates the effects of data and model width. The effective sample-size factor depends only on the training and test set sizes. Width instead changes the distribution of the activation gradient, and hence its population mean, covariance, and effective alignment dimension. Width therefore does not improve the certificate directly; it does so only when it improves the relative signal–noise geometry summarized by dalign(N)d_align(N). A sufficient condition for a wider model to have a non-worse simplified certificate is that neither component of the effective alignment dimension deteriorates. In particular, if N2>N1N_2>N_1, d∥(N2)≥d∥(N1)d_ (N_2)≥ d_ (N_1), d2(N2)≥d2(N1)d_2(N_2)≥ d_2(N_1) ⟹ Balign(M,K;N2)≤Balign(M,K;N1).B_align(M,K;N_2)≤ B_align(M,K;N_1). This is a sufficient rather than necessary condition: the exact certificate in Theorem 1 retains the two relative-noise contributions separately and may give a finer comparison. The stronger covariance and signal-growth conditions used in the prior width-dependent analysis provide one sufficient route to this behavior. As shown in the appendix, those conditions imply, for sufficiently large N, d∥(N)=Ω(N)d_ (N)= (N), d2(N)=Ω(N)d_2(N)= (N), dalign(N)=Ω(N)d_align(N)= (N). Consequently, for a width-independent constant c0>0c_0>0, Pr⁡(μM⊤gK≤0) ( _M g_K≤ 0 ) ≤11+c0neff(M,K)N ≤ 11+c_0n_eff(M,K)N =O(1MN+1KN+1MKN). =O ( 1MN+ 1KN+ 1MKN ). Up to the lower-order mixed term, this recovers the width dependence of the prior alignment result. The difference is that these growth conditions are not required by Theorem 1; they are only sufficient conditions for linear growth of the effective alignment dimension. More generally, the certificate improves at whatever rate is supported by the observed behavior of dalign(N)d_align(N). Thus, the benefit of width for finite-sample directional transfer is conditional rather than automatic and can be evaluated directly on the model at hand. (a) Activation-gradient statistic across model widths. (b) Effective alignment dimension across model widths. Figure 1: Width-dependent activation-gradient geometry. Panel (a) reports the activation-gradient statistic S^=‖μ^‖22 S=\| μ\|_2^2, whereas panel (b) reports dalignd_align computed from the same activation-gradient samples. For each width, metric estimates are first aggregated across independently sampled evaluation subsets within each trained model and then summarized across eight training seeds. Experiment Our experiments evaluate four questions: how activation-gradient geometry scales with width, whether empirical misalignment follows the predicted width and sample-size dependence, whether the trends transfer across Pythia and ResNet, and whether positive alignment predicts the loss change of a realizable residual intervention. Experiment Configurations Model. We consider three experimental settings: our pretrained 32-layer LLaMA-style Transformer models (Touvron et al. 2023), the Pythia model suite, and independently trained ResNet models. For the LLaMA-style family, we pretrain six models with hidden widths N∈512,1024,1536,2048,2560,3072N∈\512,1024,1536,2048,2560,3072\ while keeping the depth, training data, token budget, and optimization settings fixed. For each LLaMA-style and ResNet configuration, we train eight independent models using different random seeds. We further evaluate Pythia checkpoints of different scales. Metric Evaluation. For LLaMA-style and Pythia models, the activation gradient q is measured at the final Transformer block, whereas for ResNet-20 it is measured at the first residual block. Given evaluation samples qrr=1Reval\q_r\_r=1^R_eval, we estimate μ^=Reval−1∑r=1Revalqr μ=R_eval^-1 _r=1^R_evalq_r and report the activation-gradient statistic S^=‖μ^‖22 S=\| μ\|_2^2. We compute dalignd_align and the empirical misalignment probability from the same activation-gradient samples, with the measurement location fixed across widths within each family. Metrics are estimated using multiple data-sampling seeds on C4, WikiText-2, OpenWebText, and SlimPajama for language models, and on CIFAR-10 and CIFAR-100 for ResNet-20. For independently trained models, data-sampling variability is first aggregated within each checkpoint and then summarized across training seeds. Full architectural configurations, training protocols, dataset preprocessing, and metric-estimation details are provided in the appendix. Width Scaling of Activation-Gradient Geometry We first examine how activation-gradient geometry varies with model width in the controlled LLaMA-style family. We consider hidden widths N∈512,1024,1536,2048,2560,3072N∈\512,1024,1536,2048,2560,3072\ while holding the model depth, training corpus, token budget, and optimization protocol fixed. Figure 1 reports the activation-gradient statistic S S and the effective alignment dimension dalignd_align, both computed from the same activation-gradient samples, on C4, WikiText-2, OpenWebText, and SlimPajama. Detailed metric-estimation procedures and pseudocode are provided in the appendix. As shown in Figure 1(a), the activation-gradient statistic exhibits a clear width-dependent trend. Although its absolute scale varies across datasets and individual runs, the distributions generally shift upward as N increases. This pattern persists across independently trained models and independently sampled evaluation subsets, indicating that the observed trend is robust to both optimization randomness and finite-sample estimation variability. Figure 1(b) shows a corresponding increase in the effective alignment dimension computed from the activation-gradient signal q. Across the four datasets, dalignd_align grows from approximately 33–44 at the smallest widths to approximately 99–1010 at the largest width. Although adjacent widths exhibit overlapping distributions and moderate local fluctuations, the aggregated trend remains consistently increasing over the investigated width range. Taken together, Figure 1 provides empirical support for the central width-scaling hypothesis ↑⟹()↑ N d_align(N) (a) M=K=1M=K=1. (b) M=K=2M=K=2. (c) M=K=4M=K=4. (d) M=K=8M=K=8. Figure 2: Empirical gradient-misalignment probabilities and theoretical certificates across model widths and finite-sample budgets. Each sampled sequence contains 20482048 tokens, and each empirical probability is estimated using R=500,000R=500,000 independent Monte Carlo trials. The exact certificate BexactB_exact and simplified certificate BalignB_align are computed from the corresponding activation-gradient moments. Results are summarized across eight independently trained LLaMA-style models and multiple independent data-sampling seeds. Gradient-Misalignment Probability We next directly test the theoretical prediction that finite-sample gradient misalignment becomes less likely as the effective alignment dimension increases. To isolate the effect of model width from variation in the evaluation distribution, we conduct this analysis exclusively on C4 using our trained LLaMA-style models. For each trained model and sampling budget (M,K)(M,K), we estimate the gradient-misalignment probability using the Monte Carlo estimator p^M,K(N)=1R∑r=1Rμ^M,r⊤g^K,r≤0, p_M,K(N)= 1R _r=1^R1 \ μ_M,r g_K,r≤ 0 \, (4) where μ^M,r μ_M,r and g^K,r g_K,r are computed from two independently sampled subsets containing M and K sequences, respectively, in Monte Carlo trial r. Each sampled sequence contains 20482048 tokens. Independently sampling the two subsets matches the independence condition in Theorem 1 and avoids introducing artificial correlation between the two finite-sample gradient estimates. We evaluate M=K∈1,2,4,8M=K∈\1,2,4,8\ and perform R=500,000R=500,000 independent Monte Carlo trials for each combination of model width, training seed, and sampling budget. We additionally report the exact second-moment certificate BexactB_exact and the simplified effective-alignment certificate BalignB_align, computed from the estimated activation-gradient moments. In evaluations with M=K>16M=K>16, the empirical misalignment probabilities are close to zero for nearly all investigated widths. We therefore omit these larger sampling budgets from the main figure to preserve visual resolution. Detailed sampling procedures, pseudocode, and Monte Carlo error analysis are provided in the appendix. Figure 2 shows that the empirical gradient-misalignment probability consistently decreases as model width increases, and that this trend holds for every fixed sampling budget. Because M and K remain constant within each panel, the observed reduction cannot be attributed to wider models receiving more evaluation data. Instead, wider models yield more statistically reliable finite-sample gradient directions under the same sampling budget. Both theoretical certificates exhibit the same decreasing trend with width and remain conservative relative to the empirical probabilities. As expected from Theorem 1, BexactB_exact is consistently tighter than BalignB_align, while the latter provides a simpler one-scalar characterization through dalignd_align. The agreement in trend between the empirical probabilities and the two certificates supports the predicted connection between activation-gradient geometry and finite-sample directional transfer. For M=K=1M=K=1, the empirical misalignment probability decreases from approximately 28.5%28.5\% at N=512N=512 to approximately 22.7%22.7\% at N=3072N=3072. For M=K=2M=K=2, it decreases from approximately 15.7%15.7\% to approximately 9.7%9.7\%. At larger sampling budgets, the reduction becomes more pronounced relative to the initial probability: for M=K=4M=K=4, the probability decreases from approximately 5.2%5.2\% to approximately 1.4%1.4\%, whereas for M=K=8M=K=8, it decreases from approximately 0.7%0.7\% to below 0.1%0.1\%. The results also demonstrate the complementary role of the finite-sample budget. At every fixed model width, increasing M and K substantially reduces the probability of estimating a misaligned direction. This reduction is particularly pronounced between M=K=1M=K=1 and M=K=8M=K=8, and the probability becomes nearly zero once M=KM=K exceeds 1616. Thus, model width and sample size improve directional reliability through distinct mechanisms: increasing width improves the underlying activation-gradient geometry, whereas increasing M and K reduces finite-sample estimation uncertainty. Together with the width-dependent increase in dalignd_align observed in Figure 1, these results provide empirical support for the mechanism N↑⟹dalign(N)↑⟹pM,K(N)↓N d_align(N) p_M,K(N) Cross-Architecture and Cross-Modality Validation Fixed-Depth Width Validation on Pythia We next evaluate two fixed-depth Pythia regimes to separate width from depth. The six-layer regime contains Pythia-14M, 31M, and 70M with widths N∈128,256,512N∈\128,256,512\ and ten training seeds per scale; the 32-layer regime contains Pythia-2.8B and 6.9B with widths N∈2560,4096N∈\2560,4096\. Both S S and dalignd_align are computed from the same activation-gradient samples. To disentangle the effect of width from that of depth, we separately analyze two fixed-depth Pythia regimes. The small-scale group consists of Pythia-14M, Pythia-31M, and Pythia-70M, all with six Transformer layers and hidden widths N∈128,256,512N∈\128,256,512\. For each model scale, we evaluate ten independently trained runs corresponding to distinct training seeds, including the additional seed replicas released by PolyPythias (van der Wal et al. 2025). The large-scale group consists of Pythia-2.8B and Pythia-6.9B, both with 32 Transformer layers and hidden widths N∈2560,4096N∈\2560,4096\. Within each group, model depth is fixed while hidden width and the architectural dimensions conventionally coupled to width increase. (a) 6-layer Pythia models. (b) 32-layer Pythia models. Figure 3: Controlled within-depth validation on Pythia. The six-layer group uses hidden widths N∈128,256,512N∈\128,256,512\, with each configuration evaluated over ten independently trained runs. The 32-layer group uses hidden widths N∈2560,4096N∈\2560,4096\, with one pretrained checkpoint evaluated at each width. Error bars summarize variability across training runs for the six-layer group and across independently sampled evaluation subsets for the 32-layer group. Figure 3(a) shows a clear monotonic progression within the six-layer regime. As the hidden width increases from 128128 to 512512, the activation-gradient statistic S S increases from approximately 0.0140.014 to 0.0300.030, while dalignd_align increases from approximately 33 to 1414. Thus, the three model configurations move consistently upward and to the right in the gradient-geometry plane, reproducing the width-dependent behavior observed in our controlled LLaMA-style models. Figure 3(b) provides a complementary validation at substantially larger scale. With depth fixed at 3232 layers, increasing the hidden width from 25602560 in Pythia-2.8B to 40964096 in Pythia-6.9B increases both the activation-gradient statistic S S and dalignd_align. The absolute values of dalignd_align should not be compared directly across the two panels because the six-layer and 32-layer models differ in depth and overall architecture scale. The relevant result is the within-regime trend: at fixed depth, increasing hidden width consistently increases both S S and dalignd_align. This distinction prevents depth variation from being incorrectly attributed to width. Cross-Architecture Validation on ResNet We next test whether the observed width-dependent gradient geometry extends beyond autoregressive Transformers to convolutional vision models. We train ResNet-20 models (He et al. 2015) from scratch on CIFAR-10 and CIFAR-100 (Krizhevsky 2009) with channel-width multipliers 0.5×,1×,2×,4×\0.5×,1×,2×,4×\. Within each dataset, network depth and all training conditions are held fixed, while only the channel dimensions and their associated width-dependent parameters are scaled. Each configuration is independently trained with eight random seeds, and the proposed metrics are evaluated using multiple data-sampling seeds. Metric estimates are first aggregated across data-sampling seeds within each trained model and are then summarized across the eight independent training seeds. Optimization hyperparameters and training schedules are provided in the appendix. Figure 4: Controlled width validation on ResNet-20 for CIFAR-10 (left) and CIFAR-100 (right). The horizontal axis reports the activation-gradient statistic S^=‖μ^‖22 S=\| μ\|_2^2, and the vertical axis reports dalignd_align. The channel-width multiplier is varied over 0.5×,1×,2×,4×\0.5×,1×,2×,4×\ while network depth and the training protocol are fixed. Error bars summarize variability across eight independently trained models after aggregation over multiple evaluation seeds. Figure 4 shows a consistent progression in activation-gradient geometry as channel width increases. On both CIFAR-10 and CIFAR-100, wider ResNet models move upward and to the right in the gradient-geometry plane, indicating increases in both the activation-gradient statistic S S and dalignd_align. This trend remains stable across independently trained models and evaluation samples despite differences in dataset complexity and metric scale. Increasing the width multiplier from 0.5×0.5× to 4×4× produces substantial gains in both quantities on CIFAR-10, with the same qualitative trend observed on CIFAR-100. The smaller difference between the 2×2× and 4×4× settings indicates a weaker marginal increase over the investigated range. Direct Residual-Expansion Intervention Model (N) ρ Pred. ΔL L Obs. ΔL L 14M (128) 10−410^-4 −0.307±0.002-0.307± 0.002 −0.305±0.005-0.305± 0.005 10−310^-3 −3.070±0.002-3.070± 0.002 −2.974±0.053-2.974± 0.053 31M (256) 10−410^-4 −0.432±0.005-0.432± 0.005 −0.430±0.006-0.430± 0.006 10−310^-3 −4.316±0.003-4.316± 0.003 −4.171±0.063-4.171± 0.063 70M (512) 10−410^-4 −1.058±0.002-1.058± 0.002 −1.057±0.013-1.057± 0.013 10−310^-3 −10.579±0.003-10.579± 0.003 −10.354±0.135-10.354± 0.135 Table 1: Direct residual-intervention results. Observed loss changes are reported as mean ± standard deviation across ten independently trained checkpoints, after averaging 500500 trials within each checkpoint. All loss changes are in units of 10−310^-3. We insert a zero-initialized token-shared residual parameter zt(b)=zt+bz_t(b)=z_t+b before the final normalization layer and evaluate Pythia-14M, 31M, and 70M on C4 using ten independently trained checkpoints per scale. For each checkpoint, we draw length-20482048 sequences from two fixed disjoint pools of sizes 512512 and 256256, with M=K=8M=K=8 and 500500 trials. Results are averaged first within each checkpoint and then across training seeds. The implementation verifies ∇bℓ=∑t∇ztℓ _b = _t _z_t exactly, with zero maximum relative ℓ2 _2 error. Across all 15,00015,000 trials, alignment is positive, the selected direction decreases held-out loss, and its norm-matched reverse increases it. Table 1 shows close agreement between the predicted and observed loss changes across two representative perturbation scales. At ρ=10−4ρ=10^-4, the observed magnitudes retain 99.3%99.3\%–99.9%99.9\% of their predicted values. Increasing ρ to 10−310^-3 enlarges the loss reduction by approximately one order of magnitude, while the observations retain 96.6%96.6\%–97.9%97.9\% of the predicted magnitudes. The modest increase in prediction error at the larger perturbation scale is consistent with higher-order terms becoming more relevant away from the infinitesimal-step regime. Conclusion We studied when a local expansion direction identified from finite training data transfers to independent test data. Using the insertion-point activation gradient, we derived an exact finite-sample misalignment bound controlled by the effective sample size and dalignd_align. Width enters the certificate through the activation-gradient distribution: whenever dalign(N)d_align(N) increases with width, the misalignment bound decreases. Under the stronger covariance and signal-growth conditions of prior work, the certificate recovers the same-order width dependence. We incorporated this certificate into the direct train–test expansion framework. Experiments on width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that dalignd_align generally increases with width while empirical misalignment decreases under fixed sampling budgets. Direct residual interventions further confirm that positive alignment predicts the sign and local magnitude of held-out loss changes. References Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma (2024) Explaining neural scaling laws. Proceedings of the National Academy of Sciences 121 (27), p. e2311878121. External Links: Document Cited by: Related Work. B. Bordelon and C. Pehlevan (2023) Dynamics of finite width kernel and prediction fluctuations in mean field neural networks. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Link Cited by: Related Work. S. Chatterjee (2020) Coherent gradients: an approach to understanding generalization in gradient descent-based optimization. In International Conference on Learning Representations, External Links: Link Cited by: Related Work. C. Chen, Y. Yin, L. Shang, X. Jiang, Y. Qin, F. Wang, Z. Wang, X. Chen, Z. Liu, and Q. Liu (2022) Bert2BERT: towards reusable pretrained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, p. 2134–2148. External Links: Document, Link Cited by: Related Work. T. Chen, I. Goodfellow, and J. Shlens (2016) Net2Net: accelerating learning via knowledge transfer. In International Conference on Learning Representations, External Links: Link Cited by: Related Work. D. Cheng, Z. Liu, J. Sun, F. Xia, B. Zhang, D. Liu, and Y. Zhang (2026) A qualitative test-risk mechanism for scaling behavior in normalized residual networks. arXiv preprint arXiv:2605.08297. External Links: Document, Link Cited by: Introduction, Related Work, Basic Assumptions, Basic Assumptions, Key Alignment Quantities, Assumptions and Notation, Theorem 2. L. Chizat and F. Bach (2018) On the global convergence of gradient descent for over-parameterized models using optimal transport. In Advances in Neural Information Processing Systems, Vol. 31. External Links: Link Cited by: Related Work. L. Gong, D. He, Z. Li, T. Qin, L. Wang, and T. Liu (2019) Efficient training of BERT by progressively stacking. In Proceedings of the 36th International Conference on Machine Learning, K. Chaudhuri and R. Salakhutdinov (Eds.), Proceedings of Machine Learning Research, Vol. 97, p. 2337–2346. External Links: Link Cited by: Related Work. K. He, X. Zhang, S. Ren, and J. Sun (2015) Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–778. External Links: Link Cited by: Cross-Architecture Validation on ResNet. A. Jacot, F. Gabriel, and C. Hongler (2018) Neural tangent kernel: convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, Vol. 31, p. 8571–8580. External Links: Link Cited by: Related Work. A. Khemais (2026) Exact network surgery: functional invariance and gradient plasticity in reactive computational graphs. arXiv preprint arXiv:2607.16568. External Links: Document, Link Cited by: Related Work. A. Krizhevsky (2009) Learning multiple layers of features from tiny images. Master’s thesis, University of Toronto, Toronto, Ontario, Canada. External Links: Link Cited by: Cross-Architecture Validation on ResNet. J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein (2018) Deep neural networks as gaussian processes. In International Conference on Learning Representations, External Links: Link Cited by: Related Work. J. Liu, G. Jiang, Y. Bai, T. Chen, and H. Wang (2020) Understanding why neural networks generalize well through GSNR of parameters. In International Conference on Learning Representations, External Links: Link Cited by: Related Work. D. Lok (2026) Gate-zero growth: a geometric framework for function-preserving continual learning. arXiv preprint arXiv:2607.14571. External Links: Document, Link Cited by: Related Work. S. McCandlish, J. Kaplan, D. Amodei, and OpenAI Dota Team (2018) An empirical model of large-batch training. arXiv preprint arXiv:1812.06162. External Links: Document, Link Cited by: Related Work. S. Mei, A. Montanari, and P. Nguyen (2018) A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences 115 (33), p. E7665–E7671. External Links: Document Cited by: Related Work. R. M. Neal (1996) Bayesian learning for neural networks. Lecture Notes in Statistics, Vol. 118, Springer, New York. External Links: Document Cited by: Related Work. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample (2023) LLaMA: open and efficient foundation language models. ArXiv abs/2302.13971. External Links: Link Cited by: Model.. O. van der Wal, P. Lesci, M. Müller-Eberstein, N. Saphra, H. Schoelkopf, W. H. Zuidema, and S. Biderman (2025) PolyPythias: stability and outliers across fifty language model pre-training runs. ArXiv abs/2503.09543. External Links: Link Cited by: Fixed-Depth Width Validation on Pythia. P. Wang, R. Panda, L. Torroba Hennigen, P. Greengard, L. Karlinsky, R. Feris, D. D. Cox, Z. Wang, and Y. Kim (2023) Learning to grow pretrained models for efficient transformer training. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Related Work. G. Yang, E. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao (2021) Tuning large neural networks via zero-shot hyperparameter transfer. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. W. Vaughan (Eds.), Vol. 34, p. 17084–17097. External Links: Link Cited by: Related Work. G. Yang and E. J. Hu (2021) Tensor programs IV: feature learning in infinite-width neural networks. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 11727–11737. External Links: Link Cited by: Related Work. Reproducibility Checklist 1. General Paper Structure 1.1. Includes a conceptual outline and/or pseudocode description of AI methods introduced (yes/partial/no/NA) yes 1.2. Clearly delineates statements that are opinions, hypothesis, and speculation from objective facts and results (yes/no) yes 1.3. Provides well-marked pedagogical references for less-familiar readers to gain background necessary to replicate the paper (yes/no) yes 2. Theoretical Contributions 2.1. Does this paper make theoretical contributions? (yes/no) yes If yes, please address the following points: 2.2. All assumptions and restrictions are stated clearly and formally (yes/partial/no) yes 2.3. All novel claims are stated formally (e.g., in theorem statements) (yes/partial/no) yes 2.4. Proofs of all novel claims are included (yes/partial/no) yes 2.5. Proof sketches or intuitions are given for complex and/or novel results (yes/partial/no) yes 2.6. Appropriate citations to theoretical tools used are given (yes/partial/no) yes 2.7. All theoretical claims are demonstrated empirically to hold (yes/partial/no/NA) partial 2.8. All experimental code used to eliminate or disprove claims is included (yes/no/NA) no 3. Dataset Usage 3.1. Does this paper rely on one or more datasets? (yes/no) yes If yes, please address the following points: 3.2. A motivation is given for why the experiments are conducted on the selected datasets (yes/partial/no/NA) partial 3.3. All novel datasets introduced in this paper are included in a data appendix (yes/partial/no/NA) NA 3.4. All novel datasets introduced in this paper will be made publicly available upon publication of the paper with a license that allows free usage for research purposes (yes/partial/no/NA) NA 3.5. All datasets drawn from the existing literature (potentially including authors’ own previously published work) are accompanied by appropriate citations (yes/no/NA) no 3.6. All datasets drawn from the existing literature (potentially including authors’ own previously published work) are publicly available (yes/partial/no/NA) yes 3.7. All datasets that are not publicly available are described in detail, with explanation why publicly available alternatives are not scientifically satisficing (yes/partial/no/NA) NA 4. Computational Experiments 4.1. Does this paper include computational experiments? (yes/no) yes If yes, please address the following points: 4.2. This paper states the number and range of values tried per (hyper-) parameter during development of the paper, along with the criterion used for selecting the final parameter setting (yes/partial/no/NA) no 4.3. Any code required for pre-processing data is included in the appendix (yes/partial/no) no 4.4. All source code required for conducting and analyzing the experiments is included in a code appendix (yes/partial/no) no 4.5. All source code required for conducting and analyzing the experiments will be made publicly available upon publication of the paper with a license that allows free usage for research purposes (yes/partial/no) no 4.6. All source code implementing new methods have comments detailing the implementation, with references to the paper where each step comes from (yes/partial/no) no 4.7. If an algorithm depends on randomness, then the method used for setting seeds is described in a way sufficient to allow replication of results (yes/partial/no/NA) partial 4.8. This paper specifies the computing infrastructure used for running experiments (hardware and software), including GPU/CPU models; amount of memory; operating system; names and versions of relevant software libraries and frameworks (yes/partial/no) no 4.9. This paper formally describes evaluation metrics used and explains the motivation for choosing these metrics (yes/partial/no) yes 4.10. This paper states the number of algorithm runs used to compute each reported result (yes/no) yes 4.11. Analysis of experiments goes beyond single-dimensional summaries of performance (e.g., average; median) to include measures of variation, confidence, or other distributional information (yes/no) yes 4.12. The significance of any improvement or decrease in performance is judged using appropriate statistical tests (e.g., Wilcoxon signed-rank) (yes/partial/no) partial 4.13. This paper lists all final (hyper-)parameters used for each model/algorithm in the paper’s experiments (yes/partial/no/NA) partial Supplementary Material Figure 5: Decoder-only Transformer architecture shared by all model configurations. Each pre-normalized block contains grouped-query self-attention with rotary positional embeddings, followed by a SwiGLU feed-forward network; residual connections surround both sublayers. Appendix A Model Architecture and Pretraining Details Model Architecture All models use the same Llama-style decoder-only Transformer architecture. Each Transformer block adopts a pre-normalization structure consisting of RMSNorm, grouped-query self-attention with rotary positional embeddings (RoPE), a SwiGLU feed-forward network, and residual connections around both sublayers. A final RMSNorm and a linear language-modeling head map the hidden representations to next-token logits. We use the same tokenizer and vocabulary for all model configurations and do not apply supervised fine-tuning, RLHF, or any other post-training procedure. Figure Supplementary Material illustrates the shared model architecture and the internal structures of the grouped-query attention and SwiGLU modules. Model Configurations We construct six model configurations by varying the residual-stream width N from 512512 to 30723072 while fixing the number of Transformer blocks to L=32L=32. The attention head dimension is fixed at 128128. Accordingly, the number of query heads increases proportionally with N. We use a 4:14:1 ratio between query heads and key–value heads, and set the SwiGLU intermediate dimension to dffn=3.5Nd_ffn=3.5N. Table 2 summarizes the resulting architectural configurations and total parameter counts. Model L N Q heads KV heads dffnd_ffn Params. GPT-W512 32 512 4 1 1792 0.1524B GPT-W1024 32 1024 8 2 3584 0.5229B GPT-W1536 32 1536 12 3 5376 1.1114B GPT-W2048 32 2048 16 4 7168 1.7450B GPT-W2560 32 2560 20 5 8960 2.7265B GPT-W3072 32 3072 24 6 10752 3.9261B Table 2: Architectural configurations used in the width-scaling experiments. All models contain 32 Transformer blocks, use an attention-head dimension of 128, and maintain a 4:14:1 query-to-key/value head ratio. The depth, tokenizer, vocabulary, context length, training-token budget, data mixture, and optimization protocol are held fixed across all configurations. The residual-stream width, number of attention heads, number of key–value heads, feed-forward dimension, and total parameter count therefore scale jointly with N. This construction isolates width-dependent architectural scaling while controlling the training protocol and data budget. Pretraining Data All models are pretrained on the same multi-source text mixture. The mixture contains general web text, educational web documents, deduplicated multi-domain text, WebText-style documents, encyclopedic content, technical question–answer data, and scientific articles. The approximate sampling proportions are provided in Table 3. Data source Proportion C4 35%35\% FineWeb-Edu 30%30\% SlimPajama 20%20\% OpenWebText2 10%10\% Wikipedia, StackExchange, and arXiv 5%5\% Total 100%100\% Table 3: Pretraining data mixture shared by all model configurations. Percentages denote approximate token-level sampling proportions. Pretraining Protocol All model configurations are pretrained from scratch under the same optimization protocol. We use AdamW with (β1,β2)=(0.9,0.95)( _1, _2)=(0.9,0.95), numerical stability constant ϵ=10−8ε=10^-8, and weight decay 0.10.1. Weight decay is excluded from embedding parameters, RMSNorm parameters, and bias terms. The learning rate is linearly warmed up for 2,0002,000 optimization steps and subsequently decayed according to a cosine schedule. Gradients are clipped to a maximum norm of 1.01.0, and training is performed in BF16 precision without dropout. All models use a context length of 20482048 tokens and are trained on a budget of 1010 billion tokens. The 1010B-token runs serve as the primary controlled experiments in this work; larger-token training budgets are left for subsequent scaling experiments. Hyperparameter Setting Architecture Architecture Decoder-only Transformer Normalization RMSNorm, pre-normalization Position encoding RoPE Attention Grouped-query attention Feed-forward network SwiGLU Tokenizer Llama 3 tokenizer Initialization Normal, σ=0.02σ=0.02 Optimizer AdamW Adam (β1,β2)( _1, _2) (0.9,0.95)(0.9,0.95) Adam ϵε 10−810^-8 Weight decay 0.10.1 No weight decay Embeddings, RMSNorm, biases Learning-rate schedule Linear warmup + cosine decay Warmup 2,0002,000 steps Peak learning rate 3×10−43× 10^-4 Minimum learning rate 1.5×10−71.5× 10^-7 Gradient clipping 1.01.0 Precision BF16 Dropout 0 Global batch size 480480 sequences Sequence length 20482048 tokens Training budget 1010B tokens Independent runs 88 per model configuration Training seeds 12341234, 520520, 314314, 315315, 33, 11, 55, 3131 Table 4: Shared pretraining hyperparameters. Independent Training Runs For each model configuration, we perform eight independent pretraining runs using random seeds. Each run starts from an independently initialized parameter state. When controlled by the global random seed, the training-data order and other stochastic training operations are also independently randomized. This produces 6×8=486× 8=48 independently trained models in total. Unless otherwise stated, reported results are averaged over the eight independent training runs. Per-run results are retained to quantify the variability induced by parameter initialization and stochastic optimization. Appendix B Training Dynamics and Optimization Stability Training Loss Training loss is computed from the token-level cross-entropy of each optimization batch. Figure 6 shows the complete training-loss trajectories for all six model widths under the shared token budget and optimization schedule. Figure 7 provides a single-column view of the three smaller configurations, making their relative convergence behavior easier to distinguish. Figure 6: Complete training-loss trajectories across optimization steps for all six model widths. Figure 7: Training loss across optimization steps for the three smaller model widths: GPT-W512, GPT-W1024, and GPT-W1536. Across widths, the curves provide a direct view of optimization progress under the controlled training protocol. Short-timescale fluctuations are expected from minibatch sampling and do not by themselves indicate instability; the relevant diagnostic is the longer-term convergence trend. Validation Loss and Late-Training Perplexity Validation loss is evaluated on the fixed held-out split at regular intervals using the same tokenization, causal masking, and token-level normalization as in training. Figure 8 shows the validation loss trajectories for the three smaller model configurations. To summarize validation behavior near the end of training, we compute perplexity as PPLval=exp⁡(ℒval),PPL_val= \! (L_val ), (5) where ℒvalL_val is the mean token-level negative log-likelihood. Figure 9 reports validation perplexity over the late-training checkpoints for all six widths. Because perplexity is an exponential transformation of validation loss, small loss differences can appear more pronounced on the perplexity scale; all comparisons therefore use an identical evaluation corpus and loss normalization. Figure 8: Validation loss across optimization steps for GPT-W512, GPT-W1024, and GPT-W1536. Figure 9: Validation perplexity during the final portion of training for all six model widths. Figure 10: Global gradient ℓ2 _2 norm before clipping across optimization steps. Curves show means over eight independent runs, shaded regions show one standard deviation across seeds, and the horizontal dashed line marks the clipping threshold of 1.01.0. Gradient Norm and Training Stability We track the global ℓ2 _2 norm of the gradients before clipping throughout pretraining. For parameters θ and step-t gradient gtg_t, the reported quantity is ∥gt∥2=(∑p∈θ∥gt,p∥22)1/2. g_t _2= ( _p∈θ g_t,p _2^2 )^1/2. (6) The norm is measured before applying the maximum-norm threshold of 1.01.0 so that the diagnostic retains information about gradient spikes. Values above the threshold indicate steps on which clipping rescales the update; they do not indicate that the post-clipping update norm exceeds the threshold. Figure 10 shows the gradient-norm trajectories for all model widths. We assess stability using the overall scale and trend of the norms, the frequency and duration of excursions above the clipping threshold, and agreement across seeds. Isolated spikes can arise from unusually difficult minibatches, while persistent growth, repeated extreme excursions, non-finite values, or a simultaneous increase in validation loss would be evidence of optimization instability. Reporting the pre-clipping norm together with the loss curves makes it possible to distinguish benign clipped spikes from instability that materially affects convergence. Pythia Model Configurations We evaluate five models from the Pythia family. The small-scale regime contains Pythia-14M, Pythia-31M, and Pythia-70M, all of which use six Transformer layers and hidden widths N∈128,256,512N∈\128,256,512\. The large-scale regime contains Pythia-2.8B and Pythia-6.9B, both of which use 32 Transformer layers and hidden widths N∈2560,4096N∈\2560,4096\. This grouping supports controlled within-depth comparisons of activation-gradient geometry as hidden width increases. The architectural configurations are summarized in Table 5. Model Layers Width N Heads H dheadd_head dffd_f Context Vocabulary Residual type Pythia-14M 6 128 4 32 512 2048 50304 Parallel Pythia-31M 6 256 8 32 1024 2048 50304 Parallel Pythia-70M 6 512 8 64 2048 2048 50304 Parallel Pythia-2.8B 32 2560 32 80 10240 2048 50304 Parallel Pythia-6.9B 32 4096 32 128 16384 2048 50304 Parallel Table 5: Architectural configurations of the Pythia models used in our experiments. N denotes the hidden width, H denotes the number of attention heads, dhead=N/Hd_head=N/H denotes the dimension of each attention head, and dffd_f denotes the feed-forward intermediate dimension. All evaluated models use the GPT-NeoX decoder-only architecture with GELU activations, parallel attention and feed-forward residual branches, untied input and output embeddings, and rotary positional embeddings. Rotary embeddings are applied to 25%25\% of each attention head using a rotary base of 10,00010,000. As shown in Table 5, the feed-forward dimension satisfies dff=4Nd_f=4N for all evaluated configurations, whereas the number and dimension of attention heads follow the original Pythia architectures. For Pythia-14M, Pythia-31M, and Pythia-70M, we evaluate ten independently trained runs for each configuration. These runs share the same architecture and training protocol but differ in their training random seeds. For Pythia-2.8B and Pythia-6.9B, one pretrained checkpoint is evaluated at each width, and variability in the activation-gradient measurements is estimated across independently sampled evaluation subsets. We additionally evaluate the conventional language-modeling performance of all five Pythia configurations on WikiText-2. All perplexity values are recomputed using the same preprocessing and evaluation protocol. The results are reported in Table 6. Model WikiText-2 PPL ↓ Pythia-14M 169.26 Pythia-31M 114.24 Pythia-70M 93.65 Pythia-2.8B 17.71 Pythia-6.9B 15.85 Table 6: Recomputed WikiText-2 perplexity of the evaluated Pythia model configurations. Lower values indicate better language-modeling performance. As shown in Table 6, WikiText-2 perplexity decreases consistently across the evaluated Pythia configurations as model width and overall capacity increase. These conventional language-modeling results verify that the evaluated models exhibit the expected predictive-performance ordering and complement the activation-gradient geometry analyzed in the main experiments. ResNet-20 Experimental Configuration and Accuracy We evaluate width-dependent behavior on CIFAR-10 and CIFAR-100 using a standard ResNet-20 architecture composed of BasicBlocks. Each input image has resolution 3×32×323× 32× 32, and the network contains three residual stages with three BasicBlocks per stage. For width multiplier w∈0.5,1,2,4w∈\0.5,1,2,4\, the channel dimensions of the three stages are [16w,32w,64w][16w,32w,64w]. The first blocks of the second and third stages perform spatial downsampling using stride 22. Following the original CIFAR ResNet design, we use the parameter-free Option-A shortcut, which applies strided subsampling and zero padding when the spatial resolution and number of channels change. The complete architectural and optimization settings are summarized in Table 7. Models are trained using standard cross-entropy loss without label smoothing. All convolutional layers are initialized using Kaiming-normal initialization. For every dataset–width configuration, we independently train eight models using random seeds 0 through 77. Setting Value Datasets CIFAR-10, CIFAR-100 Input resolution 3×32×323× 32× 32 Architecture ResNet-20 with BasicBlocks Blocks per stage 3,3,33,3,3 Width multipliers 0.5×,1×,2×,4×0.5×,1×,2×,4× Stage channels [16w,32w,64w][16w,32w,64w] Downsampling Stage 2 and 3, stride 22 Shortcut Option A with zero padding Loss Cross-entropy Label smoothing None Initialization Kaiming normal Optimizer SGD Initial learning rate 0.10.1 Momentum 0.90.9 Weight decay 10−410^-4 Batch size 128128 Training steps 64,00064,000 LR scheduler MultiStepLR LR milestones 32,000, 48,00032,000,\ 48,000 steps LR decay factor 0.10.1 Learning-rate schedule 0.1→0.01→0.0010.1→ 0.01→ 0.001 Evaluation interval Every 8,0008,000 steps Logging interval Every 500500 steps Training seeds 0,1,2,3,4,5,6,70,1,2,3,4,5,6,7 Table 7: Architecture and training configuration used for the ResNet-20 experiments on CIFAR-10 and CIFAR-100. Table 8 reports the final test accuracy obtained after 64,00064,000 training steps. For each dataset–width configuration, the final accuracy is summarized as the mean and standard deviation across eight independently trained models. The parameter count includes all trainable parameters, including the dataset-dependent classification head. Dataset Width Parameters Final test accuracy (%) CIFAR-10 0.5×0.5× 68,05068,050 88.21±0.2188.21± 0.21 CIFAR-10 1×1× 269,722269,722 91.56±0.3091.56± 0.30 CIFAR-10 2×2× 1,073,9621,073,962 93.61±0.1593.61± 0.15 CIFAR-10 4×4× 4,286,0264,286,026 94.49±0.3594.49± 0.35 CIFAR-100 0.5×0.5× 71,02071,020 58.25±0.5758.25± 0.57 CIFAR-100 1×1× 275,572275,572 67.19±0.2567.19± 0.25 CIFAR-100 2×2× 1,085,5721,085,572 72.00±0.2772.00± 0.27 CIFAR-100 4×4× 4,309,1564,309,156 75.67±0.2775.67± 0.27 Table 8: Final test accuracy of the evaluated ResNet-20 width configurations. Results are reported as mean ± standard deviation across eight independently trained models. As shown in Table 8, increasing the channel-width multiplier consistently improves mean final test accuracy on both CIFAR-10 and CIFAR-100, confirming the expected width-dependent predictive-performance improvement across the evaluated checkpoints. Appendix C Computation of Activation-Gradient Alignment Statistics For each evaluation unit (xi,yi)(x_i,y_i), we extract the activation gradient qi=∇ziℓ(ftop(zi),yi),zi=fbot(xi).q_i= _z_i \! (f_top(z_i),y_i ), z_i=f_bot(x_i). Given n activation-gradient vectors, let μ^=1n∑i=1nqi,Ei,:=(qi−μ^)⊤. μ= 1n _i=1^nq_i, E_i,:=(q_i- μ) . The empirical quantities are computed as v^∥=‖Eμ^‖22n−1,v^2=‖EE⊤‖F2(n−1)2. v_ = \|E μ\|_2^2n-1, v_2= \|E \|_F^2(n-1)^2. Thus, d^∥=‖μ^‖24v^∥,d^2=‖μ^‖24v^2,d^align=min⁡d^∥,d^2. d_ = \| μ\|_2^4 v_ , d_2= \| μ\|_2^4 v_2, d_align= \ d_ , d_2\. 1:Model f=ftop∘fbotf=f_top f_bot, evaluation samples (xi,yi)i=1n\(x_i,y_i)\_i=1^n 2:r^q,d^∥,d^2,d^align r_q, d_ , d_2, d_align 3:for i=1,…,ni=1,…,n do 4: zi←fbot(xi)z_i← f_bot(x_i) 5: qi←∇ziℓ(ftop(zi),yi)q_i← _z_i (f_top(z_i),y_i) 6:end for 7:r^q←n−1∑i=1n‖qi‖2 r_q← n^-1 _i=1^n\|q_i\|_2 8:μ^←n−1∑i=1nqi μ← n^-1 _i=1^nq_i 9:Ei,:←(qi−μ^)⊤E_i,:←(q_i- μ) 10:s←‖μ^‖22s←\| μ\|_2^2 11:v∥←‖Eμ^‖22/(n−1)v_ ←\|E μ\|_2^2/(n-1) 12:v2←‖EE⊤‖F2/(n−1)2v_2←\|E \|_F^2/(n-1)^2 13:d^∥←s2/v∥ d_ ← s^2/v_ 14:d^2←s2/v2 d_2← s^2/v_2 15:d^align←min⁡d^∥,d^2 d_align← \ d_ , d_2\ 16:return r^q,d^∥,d^2,d^align r_q, d_ , d_2, d_align Algorithm 1 Estimation of activation-gradient alignment statistics. The Gram-matrix computation avoids explicitly forming the N×N× N covariance matrix. The identity tr⁡(Σ^2)=‖EE⊤‖F2(n−1)2tr( ^2)= \|E \|_F^2(n-1)^2 allows the computation to use an n×n× n Gram matrix rather than an N×N× N covariance matrix. In implementation, denominators smaller than a numerical tolerance are treated as zero, and the corresponding effective dimension is set to +∞+∞. Appendix D Monte Carlo Estimation of Misalignment Probability For each Monte Carlo trial, we independently sample M training-side and K test-side activation gradients and compute μM=1M∑i=1Mqi,gK=1K∑j=1Kq~j. _M= 1M _i=1^Mq_i, g_K= 1K _j=1^K q_j. The empirical misalignment probability is p^M,K=1R∑r=1R(μM(r))⊤gK(r)≤0. p_M,K= 1R _r=1^R1 \( _M^(r)) g_K^(r)≤ 0 \. 1:Fixed, disjoint gradient pools trQ_tr and teQ_te, sample sizes M,KM,K, trials R 2:p^M,K p_M,K and SE^M,K SE_M,K 3:C←0C← 0 4:for r=1,…,Rr=1,…,R do 5: Sample qii=1M\q_i\_i=1^M with replacement from trQ_tr 6: Sample q~jj=1K\ q_j\_j=1^K with replacement from teQ_te 7: μM←M−1∑i=1Mqi _M← M^-1 _i=1^Mq_i 8: gK←K−1∑j=1Kq~jg_K← K^-1 _j=1^K q_j 9: C←C+μM⊤gK≤0C← C+1\ _M g_K≤ 0\ 10:end for 11:p^M,K←C/R p_M,K← C/R 12:SE^M,K←p^M,K(1−p^M,K)/R SE_M,K← p_M,K(1- p_M,K)/R 13:return p^M,K,SE^M,K p_M,K, SE_M,K Algorithm 2 Monte Carlo estimation of the activation-gradient misalignment probability. Training-side and test-side gradients are sampled independently with replacement in every trial. Conditional on the two fixed pools, the Monte Carlo trials are independent. Sampling is performed with replacement from the two fixed, disjoint gradient pools; repeated gradients are therefore allowed within a trial. Conditional on these pools, the estimator targets the empirical misalignment probability induced by their two empirical distributions. We use R=500,000R=500,000 conditionally independent trials for each model configuration and each pair (M,K)(M,K). The worst-case Monte Carlo standard error is therefore SE^M,K≤12R≈7.1×10−4. SE_M,K≤ 12 R≈ 7.1× 10^-4. The reported plug-in certificate estimates are computed after Algorithm 2 as Ξ^M,K=1Md^∥+1Kd^∥+1MKd^2, _M,K= 1M d_ + 1K d_ + 1MK d_2, B^M,Kexact=Ξ^M,K1+Ξ^M,K,B^M,Kalign=11+neffd^align, B_M,K^exact= _M,K1+ _M,K, B_M,K^align= 11+n_eff d_align, where neff=MKM+K+1.n_eff= MKM+K+1. For configurations with multiple training seeds, both procedures are applied separately to each trained model. Estimates are first aggregated across independently sampled evaluation subsets within each model and are subsequently summarized across training seeds. Appendix E Component-wise Decomposition of the Effective Alignment Dimension Recall that dalign=min⁡d∥,d2.d_align= \d_ ,d_2\. To examine which component controls the effective alignment dimension, we report d∥d_ , d2d_2, and dalignd_align for the controlled LLaMA-style models. For Pythia, the available table contains only aggregate component summaries; we therefore do not infer an aggregate minimum or bottleneck frequency from those summaries. Controlled LLaMA-style models. Width N d∥d_ d2d_2 dalignd_align d2d_2 bottleneck 512 11.736±0.69411.736± 0.694 3.944±0.2213.944± 0.221 3.944±0.2213.944± 0.221 100%100\% 1024 10.620±0.74910.620± 0.749 4.418±0.3194.418± 0.319 4.418±0.3194.418± 0.319 100%100\% 1536 11.347±0.74811.347± 0.748 6.261±0.8856.261± 0.885 6.261±0.8856.261± 0.885 100%100\% 2048 9.729±0.7469.729± 0.746 6.691±0.7886.691± 0.788 6.691±0.7886.691± 0.788 100%100\% 2560 9.056±0.4329.056± 0.432 7.767±0.9917.767± 0.991 7.635±0.8837.635± 0.883 80%80\% 3072 11.740±0.74811.740± 0.748 9.525±0.4679.525± 0.467 9.523±0.4639.523± 0.463 95%95\% Table 9: Component-wise decomposition of the effective alignment dimension across the controlled LLaMA-style model widths. The final column reports the frequency with which d2d_2 is the active bottleneck. The covariance-energy component d2d_2 is the dominant bottleneck over most of the investigated width range. It determines dalignd_align consistently up to N=2048N=2048, while d∥d_ becomes limiting only occasionally at the two largest widths. Although d∥d_ is not monotonic, d2d_2 and the resulting dalignd_align exhibit a clear overall increase with width, with dalignd_align increasing from 3.9443.944 at N=512N=512 to 9.5239.523 at N=3072N=3072. Pythia models. Width N d∥d_ d2d_2 6-layer Pythia models 128 5.5915 3.4390 256 7.6445 14.2963 512 19.5501 14.3800 32-layer Pythia models 2560 7.9527 11.0866 4096 9.3290 10.0440 Table 10: Aggregate component summaries for the Pythia models. Comparisons are made within each fixed-depth group. The table does not derive an aggregate dalignd_align or a bottleneck frequency from the two component means. For Pythia, the valid aggregation order is to compute dalign(r)=min⁡d∥(r),d2(r)d_align^(r)= \! \d_ ^(r),d_2^(r) \ for each evaluation unit r and only then aggregate the resulting dalign(r)d_align^(r) values. Bottleneck frequencies must likewise be computed from the same evaluation units. Neither quantity can in general be recovered from the two aggregate component means in Table 10; accordingly, no Pythia bottleneck claim is made from this table. Appendix F Evolution of Gradient-Misalignment Probability during Pretraining We next examine how finite-sample gradient misalignment evolves during pretraining and whether the width-dependent alignment advantage persists throughout the training trajectory. For a model of width N evaluated after T pretraining tokens, we estimate p^M,K(N;T)=1R∑r=1R(μM,N,T(r))⊤gK,N,T(r)≤0, p_M,K(N;T)= 1R _r=1^R1 \ ( _M,N,T^(r) ) g_K,N,T^(r)≤ 0 \, (7) where μM,N,T(r) _M,N,T^(r) and gK,N,T(r)g_K,N,T^(r) are independently sampled training- and test-averaged gradient signals in the r-th trial. Each sampled sequence contains 20482048 tokens, and the evaluation protocol is held fixed across model widths and checkpoints within each comparison. We first evaluate the width-512512 and width-10241024 LLaMA-style models at checkpoints corresponding to 11, 33, 55, 77, and 99 billion pretraining tokens. We consider two finite-sample budgets, M=K=8M=K=8 and M=K=16M=K=16, in order to determine whether the checkpoint-wise behavior persists under different levels of gradient-estimation uncertainty. (a) M=K=8M=K=8. (b) M=K=16M=K=16. Figure 11: Evolution of the empirical finite-sample gradient-misalignment probability during pretraining. Width-512512 and width-10241024 LLaMA-style models are evaluated at checkpoints corresponding to 11, 33, 55, 77, and 99 billion pretraining tokens. Each sampled sequence contains 20482048 tokens. Panel (a) uses M=K=8M=K=8, whereas panel (b) uses M=K=16M=K=16. For both sampling budgets, the empirical misalignment probability decreases with pretraining progress and remains lower for the wider model. (a) Empirical misalignment over the complete width–checkpoint grid. Each cell reports the mean over eight independently trained models. (b) Relationship between dalignd_align and p^8,8 p_8,8. Color denotes model width and marker shape denotes the pretraining checkpoint. Figure 12: Extended analysis of finite-sample gradient misalignment for M=K=8M=K=8. Panel (a) reports the complete 6×56× 5 width–checkpoint grid. Each of the 3030 cells is computed by first estimating p^8,8 p_8,8 separately for each trained model and then averaging across eight training seeds. Panel (b) retains all 240240 width–checkpoint–seed observations. The solid curve reports medians over equal-count bins, and the shaded region denotes the corresponding interquartile range. Higher dalignd_align is strongly associated with lower empirical misalignment (Spearman ρ=−0.858ρ=-0.858; trajectory-clustered bootstrap 95% CI [−0.878,−0.771][-0.878,-0.771]). As shown in Figure 11, the empirical gradient-misalignment probability decreases consistently as pretraining progresses. For M=K=8M=K=8, the probability for the width-512512 model decreases from approximately 0.15%0.15\% at the 11B-token checkpoint to approximately 0.03%0.03\% at the 99B-token checkpoint. The width-10241024 model follows the same decreasing trajectory but remains below the narrower model at every checkpoint, decreasing from approximately 0.10%0.10\% to below 0.01%0.01\%. The same qualitative behavior is observed for M=K=16M=K=16, but at a substantially smaller absolute probability scale. Increasing the sampling budget reduces the uncertainty of both the training- and test-averaged gradient estimates, while the width-10241024 model continues to exhibit a lower misalignment probability throughout pretraining. The width-dependent advantage therefore remains visible even when the misalignment event becomes rare. Because the two widths are compared at matched pretraining-token checkpoints and under identical sampling budgets within each panel, the observed difference cannot be attributed to additional pretraining data or a larger evaluation sample for the wider model. Instead, the results indicate that the width-dependent alignment advantage emerges early and persists throughout the investigated training trajectory. Complete width–checkpoint analysis. We further test whether this behavior extends beyond the two representative widths in Figure 11. For M=K=8M=K=8, we evaluate all six LLaMA-style widths N∈512,1024,1536,2048,2560,3072N∈\512,1024,1536,2048,2560,3072\ (8) at the same five pretraining checkpoints. Each width is represented by eight independently trained models, resulting in 6widths×5checkpoints×8training seeds=2406\ widths× 5\ checkpoints× 8\ training seeds=240 (9) model–checkpoint observations. For the heatmap, we first compute p^8,8 p_8,8 separately for each trained model and then average the eight model-level estimates within each width–checkpoint pair. The resulting grid therefore contains 6×5=306× 5=30 cells. The representative trajectories above and the complete grid are estimated from independently sampled evaluation subsets, so their absolute values need not coincide exactly; the relevant comparison is the width- and checkpoint-wise ordering within each analysis. Figure 12(a) shows a consistent two-dimensional pattern. At every fixed checkpoint, increasing model width reduces the empirical misalignment probability. At every fixed width, additional pretraining also reduces the probability. In the complete-grid evaluation, the mean probability for width 512512 decreases from 3.16×10−33.16× 10^-3 at 11B tokens to 1.47×10−31.47× 10^-3 at 99B tokens, whereas the corresponding probability for width 30723072 decreases from 3.69×10−43.69× 10^-4 to 1.71×10−41.71× 10^-4. Consequently, the width-30723072 models exhibit an approximately 8.68.6-fold smaller misalignment probability than the width-512512 models at both the 11B- and 99B-token checkpoints. The ordering is not restricted to the two extreme widths: each intermediate width produces a trajectory lying between those of its narrower and wider counterparts. The complete grid also shows that width and pretraining progress play complementary roles. Additional pretraining shifts the trajectory of every width toward lower misalignment probability, whereas increasing width shifts the complete checkpoint trajectory downward. In particular, sufficiently wide models at early checkpoints can exhibit lower misalignment than much narrower models after substantially more pretraining. The observed width-dependent improvement therefore cannot be reduced to training progress alone. Effective alignment dimension predicts misalignment. The theoretical analysis identifies the gradient mean–covariance geometry, rather than nominal width itself, as the quantity directly governing finite-sample directional transfer. We therefore retain every width–checkpoint–seed configuration as an individual observation and compare dalign(N;T,s)withp^8,8(N;T,s),d_align(N;T,s) p_8,8(N;T,s), (10) where s denotes the training seed. Unlike the heatmap, this analysis does not average over training seeds and therefore retains all 240240 observations. As shown in Figure 12(b), larger dalignd_align is strongly associated with lower empirical misalignment. Across all widths, checkpoints, and training seeds, the Spearman rank correlation is ρS=−0.858. _S=-0.858. (11) The five checkpoints obtained from the same trained model are statistically dependent. We therefore quantify uncertainty by resampling complete model trajectories indexed by (N,s)(N,s) rather than treating all 240240 points as independent. A trajectory-clustered bootstrap with 5,0005,000 replicates gives the 95%95\% confidence interval [−0.878,−0.771].[-0.878,-0.771]. (12) Importantly, this association is not produced solely by pooling different pretraining stages. When the Spearman correlation is computed separately at each checkpoint, it remains strongly negative: ρS=−0.959,−0.953,−0.952,−0.948,−0.944 _S=-0.959,\,-0.953,\,-0.952,\,-0.948,\,-0.944 (13) at 11, 33, 55, 77, and 99 billion tokens, respectively. Thus, even after pretraining progress is held fixed, models with larger effective alignment dimension consistently exhibit lower finite-sample misalignment. These results distinguish the proposed mechanism from a direct dependence on parameter width alone. Model width and pretraining progress alter the underlying gradient mean and covariance, while dalignd_align summarizes how favorable the resulting geometry is for finite-sample train–test directional transfer. The empirical evidence therefore supports the mechanism (N,T)⟶dalign(N;T)⟶p^M,K(N;T),(N,T) d_align(N;T) p_M,K(N;T), (14) rather than merely showing two unrelated correlations with model scale. Pretraining progress and model width consequently have complementary effects. For a fixed model width and sampling budget, additional pretraining reduces the probability of estimating a direction that is misaligned with an independent test estimate. For a fixed checkpoint and sampling budget, increasing width shifts the model toward a lower-misalignment regime. Over the investigated model family, the results support T↑ T ⟹p^M,K(N;T)↓, p_M,K(N;T) , (15) N↑ N ⟹p^M,K(N;T)↓, p_M,K(N;T) , for both (M,K)=(8,8)(M,K)=(8,8) and (M,K)=(16,16)(M,K)=(16,16) in the representative-width comparison, and across the complete six-width grid for M=K=8M=K=8. More directly, the model-level analysis supports dalign(N;T,s)↑⟹p^8,8(N;T,s)↓.d_align(N;T,s) p_8,8(N;T,s) . (16) This is precisely the qualitative relationship predicted by the finite-sample alignment certificate: models with more favorable gradient mean–covariance geometry are less likely to produce disagreeing training and test directions from finite samples. Appendix G Problem Setup and Notation This section introduces the activation-space formulation used throughout the theoretical analysis. We consider a trained network, a candidate insertion point, and a function-preserving residual expansion. All alignment quantities are defined exclusively from the activation-gradient signal at the selected insertion point. Data and risks. Let D be a probability distribution over ×X×Y. Let Strain=(xi,yi)i=1M∼MS_train=\(x_i,y_i)\_i=1^M ^M and Stest=(x~j,y~j)j=1K∼KS_test=\( x_j, y_j)\_j=1^K ^K be mutually independent training and test samples, where M,K∈ℕ+M,K _+. Let ℓ:ℝC×→ℝ :R^C×Y be the loss function. For any predictor f:→ℝCf:X ^C, define Ltrain(f) L_train(f) :=1M∑i=1Mℓ(f(xi),yi), := 1M _i=1^M (f(x_i),y_i ), Ltest(f) L_test(f) :=1K∑j=1Kℓ(f(x~j),y~j), := 1K _j=1^K (f( x_j), y_j ), R(f) R(f) :=(x,y)∼[ℓ(f(x),y)]. :=E_(x,y) [ (f(x),y ) ]. Here, LtrainL_train and LtestL_test are empirical risks evaluated on finite independent samples, whereas R denotes the population risk under D. Core Notation D Population distribution over input–label pairs, D on ×X×Y. M Number of independently sampled training examples in Strain=(xi,yi)i=1M∼MS_train=\(x_i,y_i)\_i=1^M ^M. K Number of independently sampled test examples in Stest=(x~j,y~j)j=1K∼KS_test=\( x_j, y_j)\_j=1^K ^K, where StestS_test is independent of StrainS_train. f_bot Lower part of the trained network, fbot:→ℝNf_bot:X ^N. f_top Upper part of the trained network, ftop:ℝN→ℝCf_top:R^N ^C, with f=ftop∘fbotf=f_top f_bot. N Hidden width at the selected residual-module insertion point. () z(x) Hidden representation at the insertion point, z(x)=fbot(x)∈ℝNz(x)=f_bot(x) ^N. (,) q(x,y) Activation gradient at the insertion point, q(x,y)=∇zℓ(ftop(z),y)|z=z(x)∈ℝNq(x,y)= _z (f_top(z),y) |_z=z(x) ^N. ¯ μ Population mean activation gradient, μ¯=(x,y)∼[q(x,y)]∈ℝN μ=E_(x,y) [q(x,y)] ^N, with μ¯≠0 μ≠ 0. Activation-gradient covariance, Σ=Cov⁡(q(x,y))=[(q(x,y)−μ¯)(q(x,y)−μ¯)⊤]∈ℝN×N =Cov(q(x,y))=E[(q(x,y)- μ)(q(x,y)- μ) ] ^N× N. _M Training-set activation-gradient average, μM=M−1∑i=1Mq(xi,yi) _M=M^-1 _i=1^Mq(x_i,y_i). g_K Independent test-set activation-gradient average, gK=K−1∑j=1Kq(x~j,y~j)g_K=K^-1 _j=1^Kq( x_j, y_j). ∥ d_ Directional effective dimension, d∥=‖μ¯‖24/(μ¯⊤Σμ¯)d_ =\| μ\|_2^4/( μ μ). d_2 Covariance-energy effective dimension, d2=‖μ¯‖24/‖Σ‖F2=‖μ¯‖24/tr⁡(Σ2)d_2=\| μ\|_2^4/\| \|_F^2=\| μ\|_2^4/tr( ^2). d_align Effective alignment dimension, dalign=min⁡d∥,d2d_align= \d_ ,d_2\. n_eff Effective sample size, neff=MK/(M+K+1)n_eff=MK/(M+K+1). , E_M,K Activation-gradient misalignment event, ℰM,K=μM⊤gK≤0E_M,K=\ _M g_K≤ 0\. ~, f_S_train,η Training-sample-dependent function-preserving residual-expansion path satisfying f~Strain,0=f f_S_train,0=f. u_S_train First-order residual perturbation induced by the expansion path, uStrain(z):=∂ηhStrain,η(z)|η=0u_S_train(z):= . ∂ηh_S_train,η(z) |_η=0. Network decomposition. Fix a candidate insertion point in a trained network and decompose the original model as f=ftop∘fbot,f=f_top f_bot, where fbot:→ℝN,ftop:ℝN→ℝC.f_bot:X ^N, f_top:R^N ^C. For an input x∈x , let z(x):=fbot(x)∈ℝNz(x):=f_bot(x) ^N denote the hidden representation at the selected insertion point. Thus, f(x)=ftop(z(x)),f(x)=f_top (z(x) ), where N is the hidden width at that insertion point. Function-preserving residual expansion. Given the training sample StrainS_train, consider a differentiable residual-expansion path η⟼f~Strain,η,η≥0,η f_S_train,η, η≥ 0, of the form f~Strain,η(x)=ftop(z(x)+hStrain,η(z(x))), f_S_train,η(x)=f_top (z(x)+h_S_train,η (z(x) ) ), where hStrain,0(z)=0for every z∈ℝN.h_S_train,0(z)=0 every z ^N. Consequently, f~Strain,0(x)=ftop(z(x))=f(x), f_S_train,0(x)=f_top (z(x) )=f(x), so the expansion preserves the original input–output function at η=0η=0. The training sample determines the initial residual direction, while the scalar η controls the magnitude of the expansion. Activation-gradient signal. For a sample (x,y)∼(x,y) , define the activation gradient at the insertion point by q(x,y):=∇zℓ(ftop(z),y)|z=z(x)∈ℝN.q(x,y):= _z (f_top(z),y ) |_z=z(x) ^N. The random vector q(x,y)q(x,y) is the loss-gradient signal in the hidden representation space at the selected insertion point. Throughout the paper, all population moments, finite-sample averages, alignment events, and effective dimensions are defined from q(x,y)q(x,y). Population activation-gradient moments. Define the population mean activation gradient by μ¯:=(x,y)∼[q(x,y)]∈ℝN, μ:=E_(x,y) [q(x,y) ] ^N, and its covariance by Σ :=Cov⁡(q(x,y))∈ℝN×N, =Cov (q(x,y) ) ^N× N, Σ =[(q(x,y)−μ¯)(q(x,y)−μ¯)⊤]. =E [ (q(x,y)- μ ) (q(x,y)- μ ) ]. The finite-sample alignment analysis assumes [‖q(x,y)‖22]<∞andμ¯≠0.E [\|q(x,y)\|_2^2 ]<∞ μ≠ 0. The finite-second-moment condition ensures that both μ¯ μ and Σ are well defined. Training and test activation-gradient averages. Define the training-set activation-gradient average by μM:=1M∑i=1Mq(xi,yi), _M:= 1M _i=1^Mq(x_i,y_i), and the independently evaluated test-set activation-gradient average by gK:=1K∑j=1Kq(x~j,y~j).g_K:= 1K _j=1^Kq( x_j, y_j). Both quantities are unbiased estimators of the same population activation-gradient signal: [μM]=[gK]=μ¯.E[ _M]=E[g_K]= μ. Moreover, μM _M and gKg_K are independent because they are computed from mutually independent samples. Activation-gradient alignment event. The inner product μM⊤gK _M g_K measures the directional agreement between the activation gradients estimated from the training and test samples. The event μM⊤gK>0 _M g_K>0 corresponds to positive train–test activation-gradient alignment. The conservative failure event is ℰM,K:=μM⊤gK≤0.E_M,K:= \ _M g_K≤ 0 \. Connection to residual-expansion descent. For the training-sample-dependent residual-expansion path f~Strain,η(x)=ftop(z(x)+hStrain,η(z(x))), f_S_train,η(x)=f_top (z(x)+h_S_train,η(z(x)) ), define its first-order residual perturbation by uStrain(z):=∂ηhStrain,η(z)|η=0.u_S_train(z):= . ∂ηh_S_train,η(z) |_η=0. Because hStrain,0(z)=0h_S_train,0(z)=0, the chain rule gives dηLtest(f~Strain,η)|η=0 . ddηL_test ( f_S_train,η ) |_η=0 =1K∑j=1Kq~j⊤uStrain(z~j), = 1K _j=1^K q_j u_S_train( z_j), (17) where z~j=fbot(x~j),q~j=q(x~j,y~j). z_j=f_bot( x_j), q_j=q( x_j, y_j). Equation (17) is the general first-order derivative for a differentiable function-preserving residual-expansion path. It does not, by itself, equal −μM⊤gK- _M g_K. Residual-realizability condition. Following the direct train–test residual-expansion framework, we assume that the training-selected residual perturbation satisfies 1K∑j=1Kq~j⊤uStrain(z~j)=−μM⊤gK. 1K _j=1^K q_j u_S_train( z_j)=- _M g_K. (18) A sufficient realization is uStrain(z)≡−μM,u_S_train(z)≡- _M, whenever the residual tangent family contains this constant activation-space perturbation. Indeed, in this case 1K∑j=1Kq~j⊤(−μM)=−μM⊤gK. 1K _j=1^K q_j (- _M)=- _M g_K. Under this condition, dηLtest(f~Strain,η)|η=0=−μM⊤gK. . ddηL_test ( f_S_train,η ) |_η=0=- _M g_K. (19) Therefore, μM⊤gK>0⟹dηLtest(f~Strain,η)|η=0<0. _M g_K>0 . ddηL_test ( f_S_train,η ) |_η=0<0. Hence, positive train–test activation-gradient alignment implies that the training-selected residual perturbation is a strict first-order descent direction for the independently evaluated test risk. More precisely, differentiability at η=0η=0 gives Ltest(f~Strain,η) L_test ( f_S_train,η ) =Ltest(f)−ημM⊤gK+o(η). =L_test(f)-η _M g_K+o(η). (20) Equivalently, defining ΔRtest(η):=Ltest(f)−Ltest(f~Strain,η), _R^test(η):=L_test(f)-L_test ( f_S_train,η ), we obtain ΔRtest(η)=ημM⊤gK+o(η). _R^test(η)=η _M g_K+o(η). Consequently, if μM⊤gK>0 _M g_K>0, then there exists η0>0 _0>0 such that ΔRtest(η)>0for every 0<η≤η0. _R^test(η)>0 every 0<η≤ _0. Effective alignment dimensions. Define the directional effective dimension by d∥:=‖μ¯‖24μ¯⊤Σμ¯,d_ := \| μ\|_2^4 μ μ, and the covariance-energy effective dimension by d2:=‖μ¯‖24‖Σ‖F2=‖μ¯‖24tr⁡(Σ2).d_2:= \| μ\|_2^4\| \|_F^2= \| μ\|_2^4tr( ^2). Whenever a denominator is zero, the corresponding effective dimension is understood to be +∞+∞. The quantity d∥d_ measures the squared population activation-gradient signal relative to covariance in the population signal direction. The quantity d2d_2 measures the same signal relative to the total covariance energy. Define the effective alignment dimension by dalign:=min⁡d∥,d2.d_align:= \d_ ,d_2 \. This provides a conservative scalar summary of the two relative-noise contributions governing finite-sample activation-gradient alignment. Effective sample size. Define neff:=MKM+K+1.n_eff:= MKM+K+1. This quantity combines the training and test sample sizes into the finite-sample factor appearing in the alignment guarantee. It is symmetric in M and K and increases with either sample size. In the matched-budget setting M=K=nM=K=n, neff=n22n+1.n_eff= n^22n+1. Appendix H Proof of Theorem 1 Proof. Define the centered empirical activation-gradient errors εM:=μM−μ¯,ε~K:=gK−μ¯. _M:= _M- μ, _K:=g_K- μ. Because the training and evaluation samples are drawn independently, εM _M and ε~K _K are independent. Moreover, [εM]=[ε~K]=0,E[ _M]=E[ _K]=0, and [εMεM⊤]=ΣM,[ε~Kε~K⊤]=ΣK.E [ _M _M ]= M, [ _K _K ]= K. Let XM,K:=μM⊤gK.X_M,K:= _M g_K. Using μM=μ¯+εM,gK=μ¯+ε~K, _M= μ+ _M, g_K= μ+ _K, we obtain XM,K=‖μ¯‖22+μ¯⊤εM+μ¯⊤ε~K+εM⊤ε~K.X_M,K=\| μ\|_2^2+ μ _M+ μ _K+ _M _K. The three random terms have zero expectation. In particular, independence and centering imply [εM⊤ε~K]=[εM]⊤[ε~K]=0.E [ _M _K ]=E[ _M] E[ _K]=0. Therefore, [XM,K]=‖μ¯‖22.E[X_M,K]=\| μ\|_2^2. We next compute the variance. For the first linear term, [(μ¯⊤εM)2]=μ¯⊤[εMεM⊤]μ¯=1Mμ¯⊤Σμ¯.E [ ( μ _M )^2 ]= μ E [ _M _M ] μ= 1M μ μ. Similarly, [(μ¯⊤ε~K)2]=1Kμ¯⊤Σμ¯.E [ ( μ _K )^2 ]= 1K μ μ. For the bilinear term, independence gives [(εM⊤ε~K)2] [ ( _M _K )^2 ] =[tr⁡(εMεM⊤ε~Kε~K⊤)] =E [tr ( _M _M _K _K ) ] =tr⁡([εMεM⊤][ε~Kε~K⊤]) =tr (E [ _M _M ]E [ _K _K ] ) =1MKtr⁡(Σ2). = 1MKtr( ^2). (21) All cross terms vanish. First, [(μ¯⊤εM)(μ¯⊤ε~K)]=0E [ ( μ _M ) ( μ _K ) ]=0 by independence and centering. Furthermore, [(μ¯⊤εM)(εM⊤ε~K)] [ ( μ _M ) ( _M _K ) ] =εM[(μ¯⊤εM)εM⊤ε~K[ε~K]] =E_ _M [ ( μ _M ) _M E_ _K [ _K ] ] =0. =0. (22) The analogous cross term involving μ¯⊤ε~K μ _K also vanishes. Hence Var⁡(XM,K)=(1M+1K)μ¯⊤Σμ¯+1MKtr⁡(Σ2).Var(X_M,K)= ( 1M+ 1K ) μ μ+ 1MKtr( ^2). Set m:=[XM,K]=‖μ¯‖22>0,v:=Var⁡(XM,K).m:=E[X_M,K]=\| μ\|_2^2>0, v:=Var(X_M,K). The activation-gradient misalignment event satisfies XM,K≤0=XM,K−m≤−m.\X_M,K≤ 0\=\X_M,K-m≤-m\. By Cantelli’s one-sided inequality, Pr⁡(XM,K≤0)≤v+m2. (X_M,K≤ 0)≤ vv+m^2. Since m2=‖μ¯‖24m^2=\| μ\|_2^4, we have vm2 vm^2 =(1M+1K)μ¯⊤Σμ¯‖μ¯‖24+1MKtr⁡(Σ2)‖μ¯‖24 = ( 1M+ 1K ) μ μ\| μ\|_2^4+ 1MK tr( ^2)\| μ\|_2^4 =1Md∥+1Kd∥+1MKd2 = 1Md_ + 1Kd_ + 1MKd_2 =ΞM,K. = _M,K. (23) Therefore, Pr⁡(μM⊤gK≤0)≤ΞM,K1+ΞM,K. \! ( _M g_K≤ 0 )≤ _M,K1+ _M,K. Finally, because dalign=min⁡d∥,d2,d_align= \d_ ,d_2\, we have 1d∥≤1dalign,1d2≤1dalign. 1d_ ≤ 1d_align, 1d_2≤ 1d_align. It follows that ΞM,K _M,K ≤(1M+1K+1MK)1dalign ≤ ( 1M+ 1K+ 1MK ) 1d_align =M+K+1MKdalign = M+K+1MK\,d_align =1neffdalign, = 1n_effd_align, (24) where neff=MKM+K+1.n_eff= MKM+K+1. Since x↦x/(1+x)x x/(1+x) is nondecreasing on [0,∞)[0,∞), ΞM,K1+ΞM,K≤11+neffdalign. _M,K1+ _M,K≤ 11+n_effd_align. This completes the proof. ∎ Corollary 3 (Balanced sample allocation). Fix an integer sampling budget B≥2B≥ 2 and require M+K=B,M,K∈ℕ+.M+K=B, M,K _+. Then the effective sample size neff(M,K)=MKM+K+1=MKB+1n_eff(M,K)= MKM+K+1= MKB+1 is maximized when the training and test sample sizes are as balanced as possible, namely, |M−K|≤1.|M-K|≤ 1. More precisely, an optimal allocation is M⋆=⌊B2⌋,K⋆=⌈B2⌉,M = B2 , K = B2 , or the reversed allocation. The corresponding maximum effective sample size is neff⋆=⌊B2/4⌋B+1.n_eff = B^2/4 B+1. Consequently, for fixed B and dalignd_align, the simplified activation-gradient misalignment bound Pr⁡(μM⊤gK≤0)≤11+neff(M,K)dalign \! ( _M g_K≤ 0 )≤ 11+n_eff(M,K)d_align is minimized by a balanced allocation of the sampling budget. Proof. Because M+K=BM+K=B is fixed, maximizing neff(M,K)n_eff(M,K) is equivalent to maximizing the product MKMK. Using (M−K)2≥0,(M-K)^2≥ 0, we obtain (M+K)2≥4MK,(M+K)^2≥ 4MK, and hence MK≤B24.MK≤ B^24. For positive integer M and K, the largest attainable value is MK=⌊B24⌋,MK= B^24 , which is achieved exactly when M and K differ by at most one. Therefore, neff(M,K)≤⌊B2/4⌋B+1.n_eff(M,K)≤ B^2/4 B+1. Finally, the function x⟼11+xdalignx 11+x\,d_align is nonincreasing for x≥0x≥ 0. Thus, maximizing neff(M,K)n_eff(M,K) minimizes the corresponding simplified misalignment-probability bound. ∎ Appendix I Proof of Theorem 2 Proof. Let f~S f_S denote the empirical jumpboard model in Theorem 2, and let ΔRtest _R^test denote its associated finite-test margin. These fixed-model quantities are distinct from the path-indexed margin ΔRtest(η) _R^test(η) used below in the existential small-step statement. Define the two uniform-generalization events ℰM:=supf∈ℋnew|R(f)−Ltrain(f)|≤ϵM,E_M:= \ _f _new |R(f)-L_train(f) |≤ _M \, and ℰK:=supf∈ℋnew|R(f)−Ltest(f)|≤ϵK.E_K:= \ _f _new |R(f)-L_test(f) |≤ _K \. By Assumption 3, Pr⁡(ℰM)≥1−δ,Pr⁡(ℰK)≥1−δ. (E_M)≥ 1-δ, (E_K)≥ 1-δ. Therefore, by the union bound, Pr⁡(ℰM∩ℰK)≥1−2δ. (E_M _K)≥ 1-2δ. No independence between ℰME_M and ℰKE_K is required. We now work on the event ℰM∩ℰKE_M _K. Because fnew∈ℋnewf_new _new, the test-side and training-side uniform bounds imply Ltest(fnew) L_test(f_new) ≤R(fnew)+ϵK ≤ R(f_new)+ _K ≤Ltrain(fnew)+ϵM+ϵK. ≤ L_train(f_new)+ _M+ _K. (25) By the definition of the optimization gain, ΔERM=Ltrain(f~S)−Ltrain(fnew), _ERM=L_train( f_S)-L_train(f_new), and hence Ltrain(fnew)=Ltrain(f~S)−ΔERM.L_train(f_new)=L_train( f_S)- _ERM. Substituting this identity into (25) gives Ltest(fnew) L_test(f_new) ≤Ltrain(f~S)−ΔERM+ϵM+ϵK. ≤ L_train( f_S)- _ERM+ _M+ _K. (26) Since f~S∈ℋnew f_S _new, the same uniform events yield Ltrain(f~S)≤R(f~S)+ϵM≤Ltest(f~S)+ϵM+ϵK.L_train( f_S)≤ R( f_S)+ _M≤ L_test( f_S)+ _M+ _K. Combining this inequality with (26), we obtain Ltest(fnew) L_test(f_new) ≤Ltest(f~S)−ΔERM+2(ϵM+ϵK). ≤ L_test( f_S)- _ERM+2( _M+ _K). (27) By the definition of the finite-test jumpboard margin, ΔRtest=Ltest(fold∗)−Ltest(f~S), _R^test=L_test(f_old^*)-L_test( f_S), so that Ltest(f~S)=Ltest(fold∗)−ΔRtest.L_test( f_S)=L_test(f_old^*)- _R^test. Substitution into (27) therefore gives Ltest(fnew)≤Ltest(fold∗)−ΔRtest−ΔERM+2(ϵM+ϵK).L_test(f_new)≤ L_test(f_old^*)- _R^test- _ERM+2( _M+ _K). This inequality holds with probability at least 1−2δ1-2δ. Consequently, on the same event, whenever ΔRtest+ΔERM>2(ϵM+ϵK), _R^test+ _ERM>2( _M+ _K), we have Ltest(fnew)<Ltest(fold∗),L_test(f_new)<L_test(f_old^*), which proves the strict finite-test risk improvement statement. We next establish the alignment-supported positive-margin guarantee. Define the positive-alignment event M,K:=μM⊤gK>0.A_M,K:= \ _M g_K>0 \. Define the existential positive-margin event by M,K:=∃η>0:ΔRtest(η)>0.J_M,K:= \∃\,η>0: _R^test(η)>0 \. Under the residual-realizability condition (18), the expansion (20) implies μM⊤gK>0⟹ _M g_K>0 ∃η0>0 such that ∃\, _0>0 such that ΔRtest(η)>0for all 0<η≤η0. _R^test(η)>0 all 0<η≤ _0. Hence, M,K⊆M,K.A_M,K _M,K. Therefore, Pr⁡(M,K) (J_M,K) ≥Pr⁡(M,K) ≥ (A_M,K) =1−Pr⁡(μM⊤gK≤0). =1- ( _M g_K≤ 0 ). (28) Applying Theorem 1 yields Pr⁡(M,K)≥1−Bexact(M,K)≥1−Balign(M,K). (J_M,K)≥ 1-B_exact(M,K)≥ 1-B_align(M,K). Finally, another application of the union bound gives Pr⁡(ℰM∩ℰK∩M,K) (E_M _K _M,K ) ≥Pr⁡(ℰM∩ℰK∩M,K) ≥ (E_M _K _M,K ) ≥1−Pr⁡(ℰMc)−Pr⁡(ℰKc)−Pr⁡(M,Kc) ≥ 1- (E_M^c)- (E_K^c)- (A_M,K^c) ≥[1−2δ−Bexact(M,K)]+, ≥ [1-2δ-B_exact(M,K) ]_+, (29) where [x]+:=max⁡x,0[x]_+:= \x,0\. Using Bexact(M,K)≤Balign(M,K)B_exact(M,K)≤ B_align(M,K) further gives the simplified lower bound Pr⁡(ℰM∩ℰK∩M,K)≥[1−2δ−Balign(M,K)]+. (E_M _K _M,K )≥ [1-2δ-B_align(M,K) ]_+. The probability combination again requires no independence between the alignment event and the two uniform-generalization events. The alignment certificate guarantees the existence of a positive small-step finite-test margin; strict improvement of a fixed final expanded model still requires its realized jumpboard margin and optimization gain to exceed the generalization penalty. ∎ Appendix J Recovery of the Previous Width-Dependent Rate The finite-sample certificate in Theorem 1 does not require any prescribed relation between the activation-gradient distribution and the model width. We now show that the stronger covariance-spectrum, total-variance, and signal-growth conditions used in the previous width-dependent analysis imply linear growth of the effective alignment dimension and therefore recover the same-order width-dependent misalignment rate. Proposition 4 (Recovery of the width-dependent rate). For each insertion width N, let μ¯(N) μ^(N) and Σ(N) ^(N) denote the population mean and covariance of the corresponding activation-gradient signal. Suppose that there exist constants cμ>0c_μ>0, Cλ>0C_λ>0, Ctr>0C_tr>0, and N0∈ℕN_0 , all independent of N, together with a positive variance scale τN2 _N^2, such that for every N≥N0N≥ N_0, ‖μ¯(N)‖22 \| μ^(N) \|_2^2 ≥cμNτN2, ≥ c_μN _N^2, (30) λmax(Σ(N)) _ \! ( ^(N) ) ≤CλτN2, ≤ C_λ _N^2, (31) tr⁡(Σ(N)) \! ( ^(N) ) ≤CtrNτN2. ≤ C_trN _N^2. (32) Then, for every N≥N0N≥ N_0, d∥(N) d_ (N) ≥cμCλN, ≥ c_μC_λN, (33) d2(N) d_2(N) ≥cμ2CλCtrN. ≥ c_μ^2C_λC_trN. (34) Consequently, dalign(N)≥c0N,d_align(N)≥ c_0N, where c0:=min⁡cμCλ,cμ2CλCtr>0.c_0:= \ c_μC_λ, c_μ^2C_λC_tr \>0. Therefore, for every M,K∈ℕ+M,K _+, Pr⁡(μM⊤gK≤0) \! ( _M g_K≤ 0 ) ≤Balign(M,K;N) ≤ B_align(M,K;N) ≤11+c0neff(M,K)N. ≤ 11+c_0n_eff(M,K)N. (35) In particular, Pr⁡(μM⊤gK≤0)=O(1MN+1KN+1MKN). \! ( _M g_K≤ 0 )=O\! ( 1MN+ 1KN+ 1MKN ). Proof. We first lower-bound the directional effective dimension. Since Σ(N) ^(N) is positive semidefinite, μ¯(N)⊤Σ(N)μ¯(N)≤λmax(Σ(N))‖μ¯(N)‖22. μ^(N) ^(N) μ^(N)≤ _ \! ( ^(N) ) \| μ^(N) \|_2^2. Therefore, d∥(N) d_ (N) =‖μ¯(N)‖24μ¯(N)⊤Σ(N)μ¯(N) = \| μ^(N) \|_2^4 μ^(N) ^(N) μ^(N) ≥‖μ¯(N)‖22λmax(Σ(N)). ≥ \| μ^(N) \|_2^2 _ \! ( ^(N) ). (36) Applying (30) and (31) gives d∥(N)≥cμCλN.d_ (N)≥ c_μC_λN. We next lower-bound the covariance-energy effective dimension. Let λ1(N),…,λN(N) _1^(N),…, _N^(N) denote the eigenvalues of Σ(N) ^(N). Because these eigenvalues are nonnegative, tr⁡(Σ(N)2) \! ( ^(N)2 ) =∑i=1N(λi(N))2 = _i=1^N ( _i^(N) )^2 ≤λmax(Σ(N))∑i=1Nλi(N) ≤ _ \! ( ^(N) ) _i=1^N _i^(N) =λmax(Σ(N))tr⁡(Σ(N)). = _ \! ( ^(N) )tr\! ( ^(N) ). (37) Using (31) and (32), we obtain tr⁡(Σ(N)2)≤CλCtrNτN4.tr\! ( ^(N)2 )≤ C_λC_trN _N^4. Hence, d2(N) d_2(N) =‖μ¯(N)‖24tr⁡(Σ(N)2) = \| μ^(N) \|_2^4tr\! ( ^(N)2 ) ≥cμ2N2τN4CλCtrNτN4 ≥ c_μ^2N^2 _N^4C_λC_trN _N^4 =cμ2CλCtrN. = c_μ^2C_λC_trN. (38) Because dalign(N)=min⁡d∥(N),d2(N),d_align(N)= \d_ (N),d_2(N) \, the two preceding bounds imply dalign(N)≥c0N,d_align(N)≥ c_0N, with c0=min⁡cμCλ,cμ2CλCtr.c_0= \ c_μC_λ, c_μ^2C_λC_tr \. Applying the simplified certificate from Theorem 1 yields Pr⁡(μM⊤gK≤0) \! ( _M g_K≤ 0 ) ≤11+neff(M,K)dalign(N) ≤ 11+n_eff(M,K)d_align(N) ≤11+c0neff(M,K)N. ≤ 11+c_0n_eff(M,K)N. (39) Since neff(M,K)=MKM+K+1,n_eff(M,K)= MKM+K+1, we have 1c0neff(M,K)N 1c_0n_eff(M,K)N =M+K+1c0MKN = M+K+1c_0MKN =1c0KN+1c0MN+1c0MKN. = 1c_0KN+ 1c_0MN+ 1c_0MKN. (40) Therefore, Pr⁡(μM⊤gK≤0)=O(1MN+1KN+1MKN). \! ( _M g_K≤ 0 )=O\! ( 1MN+ 1KN+ 1MKN ). This completes the proof. ∎ The same rate can also be recovered directly from the exact certificate. Under the conditions of Proposition 4, ΞM,K(N) _M,K(N) =1Md∥(N)+1Kd∥(N)+1MKd2(N) = 1Md_ (N)+ 1Kd_ (N)+ 1MKd_2(N) ≤CλcμN(1M+1K)+CλCtrcμ2MKN. ≤ C_λc_μN ( 1M+ 1K )+ C_λC_trc_μ^2MKN. (41) Since Bexact(M,K;N)=ΞM,K(N)1+ΞM,K(N)≤ΞM,K(N),B_exact(M,K;N)= _M,K(N)1+ _M,K(N)≤ _M,K(N), the exact certificate satisfies the same asymptotic rate. These conditions are sufficient rather than necessary. Theorem 1 remains valid even when the signal norm, covariance spectrum, or total variance does not obey the prescribed width-growth relations. Their role is only to provide one particular route to dalign(N)=Ω(N)d_align(N)= (N) and thereby recover the previous width-dependent guarantee. Appendix K Exact Parameter-Space Realization of the Activation-Gradient Direction We now show that the activation-gradient direction used in the finite-sample analysis can be realized exactly by a concrete function-preserving residual module. All quantities z, q, μM _M, gKg_K, LtrainL_train, and LtestL_test follow the notation introduced above. Consider an inserted residual module of the form hU,V,b(z)=VψU(z)+b,h_U,V,b(z)=V _U(z)+b, where V∈ℝN×mV ^N× m is the output projection and b∈ℝNb ^N is an additive output parameter. Initialize the new module at U=U0,V=0,b=0.U=U_0, V=0, b=0. Because hU0,0,0(z)=0for every z,h_U_0,0,0(z)=0 every z, the insertion preserves the original network function exactly. At this function-preserving initialization, the sample-level gradients with respect to the new output parameters satisfy the following identities, where z~(V,b):=z+VψU0(z)+b z(V,b):=z+V _U_0(z)+b: ∇Vℓ(ftop(z~(V,b)),y)|V=0,b=0 . _V \! (f_top ( z(V,b) ),y ) |_V=0,b=0 =q(x,y)ψU0(z)⊤, =q(x,y) _U_0(z) , ∇bℓ(ftop(z~(V,b)),y)|V=0,b=0 . _b \! (f_top ( z(V,b) ),y ) |_V=0,b=0 =q(x,y). =q(x,y). Thus, q(x,y)q(x,y) is the prediction-side factor of the output-projection weight gradient and, more importantly, is exactly the sample-level parameter gradient with respect to the additive output parameter b. Proposition 5 (Exact realization of the activation-gradient direction). Suppose that the inserted residual module contains the zero-initialized additive output parameter b. Then the training-selected direction −μM- _M is realized by the parameter path U(η)=U0,V(η)=0,b(η)=−ημM.U(η)=U_0, V(η)=0, b(η)=-η _M. The induced residual perturbation satisfies uStrain(z)=−μMfor every z,u_S_train(z)=- _M every z, and the directional derivative of the independent finite-test risk is dηLtest(f~Strain,η)|η=0=−μM⊤gK. . ddηL_test ( f_S_train,η ) |_η=0=- _M g_K. (42) Consequently, positive train–test activation-gradient alignment implies that a sufficiently small realizable update of the inserted residual module strictly decreases the independent finite-test risk. Proof. Along the stated parameter path, hη(z)=−ημM,h_η(z)=-η _M, and hence uStrain(z)=∂ηhη(z)|η=0=−μM.u_S_train(z)= . ∂ηh_η(z) |_η=0=- _M. Therefore, the residual-realizability condition (18) holds exactly. Applying the chain rule to the empirical training risk gives dηLtrain(f~Strain,η)|η=0=−‖μM‖22. . ddηL_train ( f_S_train,η ) |_η=0=-\| _M\|_2^2. Thus, whenever μM≠0 _M≠ 0, the path is a strict first-order descent direction for the empirical training risk. For the independent test sample, the same calculation gives dηLtest(f~Strain,η)|η=0 . ddηL_test ( f_S_train,η ) |_η=0 =1K∑j=1Kq~j⊤(−μM) = 1K _j=1^K q_j (- _M) =−μM⊤gK, =- _M g_K, which proves (42). If μM⊤gK>0 _M g_K>0, the derivative is strictly negative. By differentiability at η=0η=0, Ltest(f~Strain,η)=Ltest(f)−ημM⊤gK+o(η).L_test ( f_S_train,η )=L_test(f)-η _M g_K+o(η). Hence, there exists η0>0 _0>0 such that Ltest(f~Strain,η)<Ltest(f)L_test ( f_S_train,η )<L_test(f) for every 0<η≤η00<η≤ _0. ∎ The additive output parameter has an equivalent constant-channel representation. Specifically, define ψ~U(z)=[ψU(z)1],V~=[Vb]. ψ_U(z)= bmatrix _U(z)\\ 1 bmatrix, V= bmatrixV&b bmatrix. Then VψU(z)+b=V~ψ~U(z).V _U(z)+b= V ψ_U(z). Thus, an affine output projection, a zero-initialized output bias, and a constant residual channel provide equivalent realizations of the same activation-space tangent direction. This requirement applies only to the newly inserted module and does not require the pretrained base architecture to use biased linear layers. If the inserted module contains neither an additive output parameter nor a constant feature, q(x,y)q(x,y) alone does not generally determine a realizable parameter-space update. In that case, the appropriate sample-level signal is the full output-projection gradient ζ(x,y)=vec⁡(q(x,y)ψU0(z)⊤),ζ(x,y)=vec (q(x,y) _U_0(z) ), and the finite-sample alignment analysis must instead be formulated in terms of ζ. Appendix L Architecture-Specific Selection of Activation-Gradient Measurement Locations The normalized residual-expansion framework is formulated at a designated intermediate representation and therefore permits architecture-specific measurement locations. The selected location must provide a valid residual insertion point, preserve the function-preserving expansion interpretation, and allow the activation gradient to represent the relevant first-order improvement signal. Based on these principles, we use the final residual representation for Transformer language models and an early residual block for ResNet-20. Existential nature of the layer-wise theory. Both the residual-expansion condition and our finite-sample alignment certificate are existential over admissible insertion locations. Let ℒadmL_adm denote the set of admissible insertion layers. For each ℓ∈ℒadm _adm, the corresponding gradient signal, mean, covariance, effective alignment dimension, and misalignment probability are defined locally at that layer. It is sufficient that there exists one layer ℓ⋆∈ℒadm _adm such that ℓ⋆holdsandBM,K(ℓ⋆)≤δ,C_ \ holds B_M,K^( )≤δ, (43) where ℓC_ denotes the population expansion condition at layer ℓ , and BM,K(ℓ)B_M,K^( ) is the corresponding finite-sample upper bound on the gradient-misalignment probability. It then follows that Pr⁡[(μM(ℓ⋆))⊤gK(ℓ⋆)>0]≥1−δ. \! [ ( _M^( ) ) g_K^( )>0 ]≥ 1-δ. (44) Hence, with probability at least 1−δ1-δ, a function-preserving block inserted at ℓ⋆ admits a train-estimated direction aligned with the corresponding test direction. Together with ℓ⋆C_ , a sufficiently small update yields a nearby expanded model with lower population risk. Therefore, neither the original residual-expansion argument nor our finite-sample refinement requires the condition to hold at every admissible layer. A single valid insertion location is sufficient to establish the existence of an improving structural expansion. By contrast, ruling out improvement through the considered block family and first-order expansion mechanism requires the corresponding condition to fail at every admissible insertion layer. Why the Final Residual Representation Is Measured in LLMs The original normalized residual-expansion framework is defined at an arbitrary designated intermediate representation. Specifically, a trained network can be decomposed as f=ftop∘fbotf=f_top f_bot, where z=fbot(x)∈ℝNz=f_bot(x) ^N denotes the residual-stream representation at the selected insertion point and q(x,y)=∇zℓ(ftop(z),y)q(x,y)= _z (f_top(z),y) denotes the corresponding activation-gradient signal. The theory therefore does not prescribe a unique layer shared by all architectures. Instead, the selected location must constitute a valid residual insertion point and must preserve the interpretation of q as the first-order signal governing a function-preserving residual expansion. Consider an L-block Transformer with residual-stream representations z0,z1,…,zLz_0,z_1,…,z_L, where zl+1=Tl(zl),l=0,…,L−1,z_l+1=T_l(z_l), l=0,…,L-1, and let H denote the final normalization and language-modeling head. For each residual-stream location l, define ql(x,y)=∇zlℓ(H(zL),y).q_l(x,y)= _z_l \! (H(z_L),y ). Let Φl→L=TL−1∘⋯∘Tl _l→ L=T_L-1 ·s T_l denote the downstream Transformer mapping from zlz_l to zLz_L, and define its Jacobian by Al(x)=JΦl→L(zl).A_l(x)=J_ _l→ L(z_l). The chain rule then gives ql(x,y)=Al(x)⊤qL(x,y),q_l(x,y)=A_l(x) q_L(x,y), where qL(x,y)=∇zLℓ(H(zL),y).q_L(x,y)= _z_L \! (H(z_L),y ). Thus, an activation gradient measured at an earlier layer is obtained by transporting the terminal activation gradient through every remaining Transformer block. This transport directly affects the population quantities entering the effective alignment dimension. At layer l, the population mean and covariance are μ¯l=[Al(x)⊤qL(x,y)] μ_l=E [A_l(x) q_L(x,y) ] and Σl=Cov⁡(Al(x)⊤qL(x,y)). _l=Cov (A_l(x) q_L(x,y) ). Consequently, μ¯l μ_l and Σl _l depend not only on the terminal loss-gradient geometry, but also on the anisotropy, conditioning, sample dependence, and gradient correlation induced by the downstream Jacobian Al(x)A_l(x). The corresponding effective dimensions d∥(l)=‖μ¯l‖24μ¯l⊤Σlμ¯l,d2(l)=‖μ¯l‖24tr⁡(Σl2)d_ ^(l)= \| μ_l\|_2^4 μ_l _l μ_l, d_2^(l)= \| μ_l\|_2^4tr( _l^2) therefore combine the intrinsic activation-gradient geometry at the prediction interface with an additional layer-dependent transport effect. The distinction can be made explicit by first considering an idealized deterministic downstream Jacobian of the form Al=cR,A_l=cR, where c>0c>0 and R∈ℝN×NR ^N× N is orthogonal: R⊤R=RR⊤=I.R R=R =I. In this case, μ¯l=cR⊤μ¯L,Σl=c2R⊤ΣLR. μ_l=cR μ_L, _l=c^2R _LR. Orthogonal invariance then gives ‖μ¯l‖24=c4‖μ¯L‖24,\| μ_l\|_2^4=c^4\| μ_L\|_2^4, μ¯l⊤Σlμ¯l=c4μ¯L⊤ΣLμ¯L, μ_l _l μ_l=c^4 μ_L _L μ_L, and tr⁡(Σl2)=c4tr⁡(ΣL2).tr( _l^2)=c^4tr( _L^2). It follows that d∥(l)=d∥(L),d2(l)=d2(L),dalign(l)=dalign(L).d_ ^(l)=d_ ^(L), d_2^(l)=d_2^(L), d_align^(l)=d_align^(L). Hence, a deterministic scaled isometry preserves the effective alignment geometry exactly. Differences between early- and late-layer measurements arise from deviations from this ideal case, including unequal singular values, sample-dependent Jacobians, and correlations between the downstream transformation and the terminal gradient. To isolate the sample-dependent contribution, write Al(x)=A¯l+ΔAl(x),A¯l=[Al(x)].A_l(x)= A_l+ A_l(x), A_l=E[A_l(x)]. The earlier-layer gradient can then be decomposed as ql=A¯l⊤qL+rl,rl=ΔAl(x)⊤qL.q_l= A_l q_L+r_l, r_l= A_l(x) q_L. The additional transport term satisfies ‖rl‖22≤[‖ΔAl(x)‖op2‖qL(x,y)‖22].E\|r_l\|_2^2 [\| A_l(x)\|_op^2\|q_L(x,y)\|_2^2 ]. Accordingly, input-dependent variation in the downstream Transformer Jacobian introduces an additional source of second-moment variation that can modify both the estimated signal direction and its covariance. Moreover, because Al(x)=JTL−1(zL−1)⋯JTl(zl),A_l(x)=J_T_L-1(z_L-1)·s J_T_l(z_l), earlier residual representations involve a longer product of attention- and feed-forward-block Jacobians, providing more opportunities for anisotropic scaling, rotation, and sample-dependent distortion. At the output of the final Transformer block, the downstream Transformer mapping is the identity: AL=I.A_L=I. Therefore, qL=∇zLℓ(H(zL),y),q_L= _z_L \! (H(z_L),y ), and the transport perturbation satisfies ΔAL=0,rL=0. A_L=0, r_L=0. The final residual representation consequently removes all cumulative Jacobian transport through additional Transformer blocks. Its mean μ¯L μ_L, covariance ΣL _L, and effective alignment dimension dalign(L)d_align^(L) provide the most direct characterization of the activation-gradient geometry connected to the language-modeling loss. This location remains fully compatible with the residual-expansion mechanism of the original theory. A Transformer MLP branch can be written as hU,V(z)=WoutψU(z),h_U,V(z)=W_out _U(z), while an attention branch can be written as hU,V(z)=WOAttnU⁡(z).h_U,V(z)=W_OAttn_U(z). In both cases, the output projection plays the role of V in the abstract residual form hU,V(z)=VψU(z)h_U,V(z)=V _U(z). Initializing this output projection at zero gives hU,0(z)=0h_U,0(z)=0, preserving the original model function while retaining a realizable first-order residual direction. The final residual stream is therefore an admissible insertion point under the same function-preserving residual-expansion framework used by the theory. For the LLaMA-style and Pythia models, we consequently measure the activation gradient at the output of the final Transformer block and before the final normalization and language-modeling head. This choice follows a minimal-transport principle: among residual-stream locations, it eliminates the largest amount of downstream Jacobian-induced variation and provides the least confounded measurement of width-dependent activation-gradient geometry. We do not claim that the final block universally maximizes dalignd_align over all possible layers; rather, it is the canonical location at which the measured statistics are not additionally transformed by any subsequent Transformer block. Why an Early Residual Block Is Measured in ResNet The original normalized residual-expansion framework permits the insertion point to be placed at an arbitrary designated intermediate representation. For a ResNet with intermediate representations z0,z1,…,zLz_0,z_1,…,z_L, write zj+1=Tj(zj),j=0,…,L−1,z_j+1=T_j(z_j), j=0,…,L-1, where TjT_j denotes the j-th residual block, including the skip connection and the residual branch. At a candidate insertion point l, define the activation gradient ql(x,y)=∇zlℓ(f(x),y).q_l(x,y)= _z_l \! (f(x),y ). A function-preserving residual expansion introduces a new branch hl,ηh_l,η satisfying hl,0(z)=0h_l,0(z)=0. Its first-order perturbation is ul(zl)=∂ηhl,η(zl)|η=0.u_l(z_l)= . ∂ηh_l,η(z_l) |_η=0. The corresponding population-risk derivative is dηR(fl,η)|η=0=[ql(x,y)⊤ul(zl)]. . ddηR\! (f_l,η ) |_η=0=E [q_l(x,y) u_l(z_l) ]. Therefore, the suitability of an insertion point depends on whether the realizable residual tangent family contains a direction that is non-orthogonal to the activation-gradient signal. Let lU_l denote the set of first-order perturbations realizable by a zero-output residual block at location l: l=ul:ul(zl)=∂ηhl,η(zl)|η=0.U_l= \u_l:u_l(z_l)= . ∂ηh_l,η(z_l) |_η=0 \. We assume that each residual tangent family is closed under nonnegative scalar multiplication: u∈l,a≥0⟹au∈l.u _l, a≥ 0 au _l. This property holds when the tangent family is generated by differentiable parameter paths whose initial velocities may be rescaled. Define the normalized first-order improvement capacity at layer l by Γl=supul∈l‖ul‖L2()≤1−[ql⊤ul]. _l= _ subarraycu_l _l\\ \|u_l\|_L^2(D)≤ 1 subarray-E [q_l u_l ]. A positive value Γl>0 _l>0 means that the residual module can realize a strict population-risk descent direction at that location. Consider an early location l and a later location r>lr>l. Let Φl→r=Tr−1∘⋯∘Tl _l→ r=T_r-1 ·s T_l be the downstream mapping from zlz_l to zrz_r, and define Al→r(x)=JΦl→r(zl(x)).A_l→ r(x)=J_ _l→ r(z_l(x)). By the chain rule, ql(x,y)=Al→r(x)⊤qr(x,y).q_l(x,y)=A_l→ r(x) q_r(x,y). The following proposition formalizes when an early residual block is at least as capable of producing a first-order improvement as a later block. Proposition 6 (Early-layer dominance under residual-tangent coverage). Let l<rl<r. Suppose that there exists κl,r>0 _l,r>0 such that, for every ur∈ru_r _r, there exists ul∈lu_l _l satisfying Al→r(x)ul(zl(x))=ur(zr(x))A_l→ r(x)u_l(z_l(x))=u_r(z_r(x)) almost surely, and ‖ul‖L2()≤κl,r−1‖ur‖L2().\|u_l\|_L^2(D)≤ _l,r^-1\|u_r\|_L^2(D). Then Γl≥κl,rΓr. _l≥ _l,r _r. In particular, if the later location admits a strict first-order descent direction, then the earlier location also admits a strict first-order descent direction. Proof. Take any ur∈ru_r _r satisfying ‖ur‖L2()≤1.\|u_r\|_L^2(D)≤ 1. By the residual-tangent coverage assumption, there exists ul∈lu_l _l such that Al→rul=ur,‖ul‖L2()≤κl,r−1.A_l→ ru_l=u_r, \|u_l\|_L^2(D)≤ _l,r^-1. Define u~l=κl,rul. u_l= _l,ru_l. The nonnegative scaling closure ensures that u~l∈l u_l _l. Moreover, ‖u~l‖L2()≤1\| u_l\|_L^2(D)≤ 1, and the chain-rule identity gives −[ql⊤u~l]=−κl,r[qr⊤Al→rul].-E [q_l u_l ]=- _l,rE [q_r A_l→ ru_l ]. Since Al→rul=urA_l→ ru_l=u_r, it follows that −[ql⊤u~l]=−κl,r[qr⊤ur].-E [q_l u_l ]=- _l,rE [q_r u_r ]. Taking the supremum over all admissible uru_r yields Γl≥κl,rΓr. _l≥ _l,r _r. Therefore, any first-order descent direction realizable at the later location can also be realized at the earlier location, up to the conditioning factor κl,r _l,r. ∎ Proposition 6 provides the main representational reason for selecting an early ResNet block. Early ResNet representations retain a high spatial resolution and a comparatively large number of local degrees of freedom. A standard BasicBlock residual branch can be expressed as hU,V(z)=Convout⁡(ψU(z)),h_U,V(z)=Conv_out ( _U(z) ), which is an instance of the abstract realizability form hU,V(z)=VψU(z).h_U,V(z)=V _U(z). The feature map ψU(z) _U(z) contains the preceding convolutions, normalization operations, and nonlinearities, while the final convolutional projection plays the role of V. At an early stage, this residual branch acts on a high-resolution feature map and therefore generates a broad family of spatially structured tangent directions. As the representation passes through later stages, stride-two downsampling and channel transformations compress and reorganize the spatial degrees of freedom. Under the tangent-coverage condition in Proposition 6, the image of the early residual tangent family under the downstream Jacobian contains the tangent family available at the later location: r⊆Al→rl.U_r A_l→ rU_l. Thus, the early module can reproduce any perturbation available to the later module after downstream propagation, while also retaining additional perturbation directions that may be removed by spatial downsampling or later feature compression. This establishes a precise sense in which the early residual location is representationally at least as expressive for first-order residual expansion. The finite-sample alignment geometry gives a second justification. At layer l, define μ¯l=[ql],Σl=Cov⁡(ql), μ_l=E[q_l], _l=Cov(q_l), and d∥(l)=‖μ¯l‖24μ¯l⊤Σlμ¯l,d2(l)=‖μ¯l‖24tr⁡(Σl2).d_ ^(l)= \| μ_l\|_2^4 μ_l _l μ_l, d_2^(l)= \| μ_l\|_2^4tr( _l^2). Suppose that the candidate ResNet locations satisfy the uniform covariance controls λmax(Σl)≤Λ,tr⁡(Σl2)≤T, _ ( _l)≤ , ( _l^2)≤ T, where Λ and T do not increase across the compared locations. Then μ¯l⊤Σlμ¯l≤Λ‖μ¯l‖22, μ_l _l μ_l≤ \| μ_l\|_2^2, and therefore d∥(l)≥‖μ¯l‖22Λ.d_ ^(l)≥ \| μ_l\|_2^2 . Similarly, d2(l)≥‖μ¯l‖24T.d_2^(l)≥ \| μ_l\|_2^4T. Consequently, dalign(l)≥min⁡‖μ¯l‖22Λ,‖μ¯l‖24T.d_align^(l)≥ \ \| μ_l\|_2^2 , \| μ_l\|_2^4T \. The original residual-expansion analysis predicts that the remaining first-order signal becomes weaker as the model approaches the deepest-model regime. If an early candidate location l and a later location r satisfy ‖μ¯l‖2≥‖μ¯r‖2,\| μ_l\|_2≥\| μ_r\|_2, then the preceding lower bound gives min⁡‖μ¯l‖22Λ,‖μ¯l‖24T≥min⁡‖μ¯r‖22Λ,‖μ¯r‖24T. \ \| μ_l\|_2^2 , \| μ_l\|_2^4T \≥ \ \| μ_r\|_2^2 , \| μ_r\|_2^4T \. Thus, under comparable covariance control, the early layer has a no-weaker guaranteed lower certificate for the effective alignment dimension. The corresponding finite-sample misalignment guarantee satisfies Pr⁡(μM(l)⊤gK(l)≤0)≤11+neffdalign(l), ( _M^(l) g_K^(l)≤ 0 )≤ 11+n_effd_align^(l), so a stronger early-layer signal leads to a tighter guaranteed alignment certificate. The representational and statistical arguments are complementary. The tangent-coverage result shows that an early residual block can reproduce later first-order perturbations through the remaining network, while the signal-to-noise argument shows that an early location is preferable when the population first-order signal has not yet entered the weak-signal regime. This is consistent with the deepest-model mechanism of the original analysis, according to which the remaining activation-gradient signal becomes progressively weaker as residual depth is exhausted. For ResNet-20, we therefore measure the activation gradient at the output of the first residual block. This location retains the full spatial resolution of the first residual stage, provides a rich realizable residual tangent family, and leaves the subsequent residual stages available to transform the inserted perturbation into a prediction-level change. The choice also follows the early-block regime used in the original ResNet covariance experiment. It should not be interpreted as an unconditional theorem that the first residual block maximizes dalignd_align for every trained ResNet. Rather, it is the architecture-specific location favored by residual-tangent coverage, non-exhausted first-order signal, and controlled within-family width comparison.

Instant research discovery