Paper deep dive
Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks
Cheng Xu, Pengxiao Lin, Zhangchen Zhou, Zhi-Qin John Xu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Weight decay is widely used as a regularization technique in neural network training, yet its role in neuron condensation (parameter direction alignment) remains unclear. Starting from a parameter initialization in the neural tangent kernel regime, we characterize training dynamics under weight decay through three stages: rapid fitting, amplitude compression, and neuron condensation. Using a two-layer ReLU network, we analyze a residual correlation field that governs both neuron amplitudes and directions. During rapid fitting, the residual approaches a quasi-static equilibrium maintained by weight decay while the tangent kernel remains nearly unchanged. In the early stage of amplitude compression, kernel decay amplifies the residual correlation field, whose isolated attracting extrema provide candidates for condensation directions. As neuron amplitudes stabilize, we bound the drift of attracting extrema and demonstrate contraction of neuron directions around them, leading to neuron condensation. This staged analysis provides a dynamical understanding of how weight decay promotes a condensed representation, beyond reducing parameter norms.
Tags
Links
- Source: https://arxiv.org/abs/2610.04533v1
- Canonical: https://arxiv.org/abs/2610.04533v1
Trouble viewing inline? Open PDF directly →
Full Text
96,875 characters extracted from source content.
Expand or collapse full text
Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks Cheng Xu, Pengxiao Lin, Zhangchen Zhou, Zhi-Qin John Xu School of Mathematical Sciences, Shanghai Jiao Tong University Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University †These authors contributed equally to this work. *Corresponding author: xuzhiqin@sjtu.edu.cn Abstract Weight decay is widely used as a regularization technique in neural network training, yet its role in neuron condensation (parameter direction alignment) remains unclear. Starting from a parameter initialization in the neural tangent kernel regime, we characterize training dynamics under weight decay through three stages: rapid fitting, amplitude compression, and neuron condensation. Using a two-layer ReLU network, we analyze a residual correlation field that governs both neuron amplitudes and directions. During rapid fitting, the residual approaches a quasi-static equilibrium maintained by weight decay while the tangent kernel remains nearly unchanged. In the early stage of amplitude compression, kernel decay amplifies the residual correlation field, whose isolated attracting extrema provide candidates for condensation directions. As neuron amplitudes stabilize, we bound the drift of attracting extrema and demonstrate contraction of neuron directions around them, leading to neuron condensation. This staged analysis provides a dynamical understanding of how weight decay promotes a condensed representation, beyond reducing parameter norms. Keywords Two Layer Network, Weight Decay, Neuron Condensation, Training Dynamics 1 Introduction An overparameterized network fits the same training data in many ways, and which solution gradient-based training finds depends on the parameterization, the initialization and the optimizer (Luo et al., 2021; Chizat et al., 2019; Chizat and Bach, 2020). Classical capacity control does not explain why such networks generalize (Zhang et al., 2017); one explanation is that training returns solutions simpler than the size of the network suggests (Xu et al., 2025b; Xu et al., 2025a). For a neural network this simplicity has a concrete form: the learned function may need far fewer neurons than the network has. Condensation is how this happens during training: the input weight vectors of many neurons, biases included, concentrate around a few directions (Luo et al., 2021; Zhou et al., 2022; Xu et al., 2025b). Neurons sharing a direction compute proportional features and act as one, so a wide network behaves like a much narrower one. Small initialization induces condensation (Maennel et al., 2018; Zhou et al., 2022). The price is speed: the smaller the initialization, the longer training takes to leave the neighborhood of the origin before fitting begins (Zhang et al., 2021; Zhang et al., 2022). Dropout (Srivastava et al., 2014) induces condensation as well even with large initialization (Zhang and Xu, 2024). A large initialization places a sufficiently wide network in the lazy regime instead: training fits the data quickly, but the features move little relative to their initial scale, so no condensation occurs (Jacot et al., 2018; Chizat et al., 2019; Luo et al., 2021). We study whether weight decay induces condensation even with large initialization, in two-layer ReLU regression. Sparse minimizers are known to exist for related norm-regularized formulations (Savarese et al., 2019; Parhi and Nowak, 2021), but their existence says nothing about whether the gradient-based learning can lead the network to such solutions. In related homogeneous models, training with large initialization and weak weight decay can transition from kernel behavior to a parameter-space maximum-margin bias(Lyu et al., 2024). This transition provides one explanation of grokking, in which test performance improves long after the training loss has stopped decreasing (Power et al., 2022; Liu et al., 2023; Kumar et al., 2024; Lyu et al., 2024). Consistent with this separation of time scales, our experiments show rapid initial fitting governed by the neural tangent kernel, followed by slower representation dynamics under weight decay. During this later evolution, neuron amplitudes shrink and surviving directions concentrate into a small number of groups. We characterize this condensation mechanism through three stages: residual equilibration, amplification of the residual correlation field, and neuron condensation as amplitudes stabilize. The key in this paper is not the proof of theorems but the connection between the weight decay and the weight directions, with reasonable assumptions motivated by the experiments and dynamical analysis. We complement the two-layer analysis with empirical observations in Transformers. Figure 1 shows condensation in the query and key projection weights of a Transformer model trained on an anchor function task (Zhang et al., 2024; Zhang et al., 2026). Each input contains an ordered pair of anchor tokens whose composition determines the target. All examples generated from the anchor pairs (3,4)(3,4) and (4,3)(4,3) are excluded from training and used only for evaluation. These observations extend the empirical evidence beyond two-layer networks. Figure 1: Parameter norm contraction, condensation and generalization in a Transformer. Panels (a) and (b) show the first layer WQW_Q and WKW_K weights for λ=0λ=0 and λ=0.8λ=0.8, respectively. Curves show their Frobenius norms, and insets show row-wise cosine similarities at epochs 50 and 300. Weight decay contracts the norms and induces row vectors alignment. Panel (c) shows that λ=0.8λ=0.8 achieves substantially higher generalization accuracy on the held-out anchor pairs (3,4)(3,4) and (4,3)(4,3) than λ=0λ=0. Experimental details are provided in Appendix G Main contributions. We explain how weight decay induces neuron condensation in two-layer ReLU networks under large initialization by characterizing three stages of training: rapid fitting, amplitude compression, and neuron condensation. We show that the angular dynamics induced by the field admit only finitely many isolated attracting directions. During early amplitude compression, weight decay reduces the neural tangent kernel and amplifies this field. As amplitudes stabilize, we characterize the slow motion of attracting extrema and exponential angular contraction around them. Together, these results provide a dynamical explanation of how weight decay promotes neuron condensation under large initialization, distinct from previous analyses of condensation under small initialization and lazy training under large initialization. 2 Related Work Condensation and early directional alignment. Small-initialization training can concentrate neurons along a limited set of directions, as studied through feature quantization (Maennel et al., 2018), initialization-dependent regimes (Luo et al., 2021), and initial condensation (Zhou et al., 2022; Chen et al., 2024). For shallow ReLU networks, Boursier et al. (2022) characterize alignment and subsequent training under orthogonal inputs while Boursier and Flammarion (2025) give a quantitative analysis of early alignment for more general data and investigate its sparsity bias phenomenon. Kumar and Haupt (2024) relate directional convergence near small initializations and certain saddles to stationary directions of a neural correlation function. Weight decay and the transition beyond lazy training. The neural tangent kernel and lazy-training analyses describe regimes in which training is approximated by a linearized model (Jacot et al., 2018; Chizat et al., 2019). With L2L_2 regularization, Lewkowycz and Gur-Ari (2020) derive time-dependent kernel dynamics for homogeneous networks in an infinite-width regime and analyze the increase of the empirical loss. Lyu et al. (2024) prove a sharp transition from kernel to a parameter space maximum-margin bias for large initialization and vanishing weight decay; their general theory assumes smooth homogeneous models. Kumar et al. (2024) also connect delayed generalization to late feature learning, including examples without regularization. Focusing on convex neural network (Bach, 2017), Pilanci and Ergen (2020) establish convex reformulations of two-layer ReLU training with a block-sparsity penalty and L2L^2 regularization. Other mechanisms leading to condensation. Dropout provides another route to neuron condensation (Zhang and Xu, 2024). Empirical work on Transformers also reports condensation under changes in initialization and weight decay (Zhang et al., 2024), while Chen and Luo (2025) analyze staged condensation and rank collapse from small initialization in a different architecture. Weight decay has additionally been linked to neural collapse in classification (Jacot et al., 2025) and low-rank structure at stationary points (Kuzborskij and Abbasi-Yadkori, 2026). These geometric properties differ from the within-group angular diameter studied here: concentration of sample representations or matrix rank alone does not characterize alignment of individual hidden-neuron directions. 3 Preliminaries 3.1 Basic notation We write ‖⋅‖2 · _2 for the Euclidean norm and ‖⋅‖op · _op for its induced matrix norm. For ,∈ℝn x, y ^n, the empirical inner product and norm are ⟨,⟩n=1n⊤,‖n=⟨,⟩n=‖2n. x, y _n= 1n x y, x _n= x, x _n= x _2 n. The notation Op(cm)O_p(c_m) and op(cm)o_p(c_m) means that division by cm>0c_m>0 gives, respectively, a quantity bounded or converging to zero in probability as m→∞m→∞. 3.2 Network and training objective Given samples (xi,yi)i=1n⊂ℝ×ℝ\(x_i,y_i)\_i=1^n ×R, consider a two-layer ReLU network with m hidden neurons: fΘ(x)=∑k=1makReLU(wkx+bk),Θ=(ak,wk,bk)k=1m.f_ (x)= _k=1^ma_kReLU(w_kx+b_k), =(a_k,w_k,b_k)_k=1^m. (1) For λ>0λ>0, the training objective with weight decay is ℒ(Θ)=1n∑i=1n(fΘ(xi)−yi)2,Φ(Θ)=ℒ(Θ)+λ2∑k=1m(ak2+wk2+bk2).L( )= 1n _i=1^n(f_ (x_i)-y_i)^2, ( )=L( )+ λ2 _k=1^m(a_k^2+w_k^2+b_k^2). (2) Write =(xi)i=1n X=(x_i)_i=1^n, =(yi)i=1n y=(y_i)_i=1^n, and (t)=fΘ(t)()−∈ℝn e(t)=f_ (t)( X)- y ^n for the training residual. The parameters ak(0),wk(0),bk(0)a_k(0),w_k(0),b_k(0) are initialized independently from N(0,m−2γ)N(0,m^-2γ), with large initialization γ<1/2γ<1/2. We study gradient flow dΘ(t)dt=−∇ΘΦ(Θ(t)) d (t)dt=- _ ( (t)), with weight decay applied to all parameters. Our main experiment uses the target function f⋆(x)=0.2ReLU(x−1/3)+0.2ReLU(−x−1/3)f_ (x)=0.2ReLU(x-1/3)+0.2ReLU(-x-1/3) as shown in Figure 2(b). Detailed experimental settings are provided in Appendix E. 3.3 Amplitude and angular dynamics Write k=(wk,bk)⊤=rk(θk) v_k=(w_k,b_k) =r_k u( _k), where rk=‖k‖2r_k= v_k _2 and (θ)=(cosθ,sinθ)⊤ u(θ)=( θ, θ) . For rk>0r_k>0, define the direction θk∈[0,2π) _k∈[0,2π), the amplitude Ak=|ak|rkA_k=|a_k|r_k, and sk=sgn(ak)s_k=sgn(a_k). Positive and negative neurons have ak>0a_k>0 and ak<0a_k<0 respectively. With the directional feature ϕθ(x)=ReLU(xcosθ+sinθ),ψθ(x)=xcosθ+sinθ>0(−xsinθ+cosθ), _θ(x)=ReLU(x θ+ θ), _θ(x)=1_\x θ+ θ>0\(-x θ+ θ), (3) the network is fΘ(x)=∑kskAkϕθk(x)f_ (x)= _ks_kA_k _ _k(x) by positive homogeneity of ReLUReLU activation. Here ψθ _θ is the angular derivative of ϕθ _θ away from activation boundaries, where xicosθ+sinθ=0x_i θ+ θ=0 for some training input. For a residual vector ∈ℝn e ^n, define the residual correlation field as the scalar function qeq_e of the angle θ, with angular derivative pep_e: qe(θ)=⟨,ϕθ()⟩n,pe(θ)=∂θqe(θ)=⟨,θ()⟩n.q_e(θ)= e, φ_θ( X) _n, p_e(θ)= _θq_e(θ)= e, ψ_θ( X) _n. (4) During training, write qt=qe(t)q_t=q_e(t), pt=pe(t)p_t=p_e(t), qk=qt(θk)q_k=q_t( _k), and pk=pt(θk)p_k=p_t( _k). Let Jt=∂ΘfΘ(t)()J_t= _ f_ (t)( X) and Kt=JtJt⊤/nK_t=J_tJ_t /n be the Jacobian and empirical neural tangent kernel (NTK). We have the following gradient flow. Proposition 1 (Gradient flow). For rk>0r_k>0, gradient flow for equation (2) satisfies dakdt=−2rkqk−λak,drkdt=−2akqk−λrk,dθkdt=−2akrkpk. da_kdt=-2r_kq_k-λ a_k, dr_kdt=-2a_kq_k-λ r_k, d _kdt=-2 a_kr_kp_k. (5) The training residual satisfies ddt=−2(Kt+λIn)−2λ. d edt=-2(K_t+λ I_n) e-2λ y. (6) Moreover, the scale imbalance obeys Bk(t):=ak(t)2−rk(t)2=e−2λtBk(0).B_k(t):=a_k(t)^2-r_k(t)^2=e^-2λ tB_k(0). (7) Weight decay acts directly on parameter scales, while angular motion is governed by pkp_k. Equation (7) shows that the imbalance ak2−rk2a_k^2-r_k^2 decays exponentially, motivating a balanced description of late training. For an initially strictly inactive neuron, wkxi+bk<0w_kx_i+b_k<0 for every training input, so qk=pk=0q_k=p_k=0: its amplitude decays as Ak(t)=e−2λtAk(0)A_k(t)=e^-2λ tA_k(0) while its direction remains fixed. Proposition 1 is derived in Appendix A. Holding KtK_t fixed in equation (6) gives the quasi-static residual qs(t):=−λ(Kt+λIn)−1. e_qs(t):=-λ(K_t+λ I_n)^-1 y. (8) It is an instantaneous reference for the actual residual (t) e(t). 3.4 Neuron condensation For a fixed group G of at least two neurons with Ak(t)>0A_k(t)>0, define its angular diameter by diamθG(t):=maxi,j∈G|θi(t)−θj(t)|.diam_θG(t):= _i,j∈ G| _i(t)- _j(t)|. (9) We say that G undergoes neuron condensation if diamθG(t)→0diam_θG(t)→ 0 as t→∞t→∞. Thus condensation means that distinct neurons approach a common direction. Different groups may condense around different directions. 3.5 Three-stage overview Figure 2: Training overview. (a) Data MSE, weight decay penalty, total objective, and test MSE; vertical markers indicate the loss minimum (epoch 447) and an epoch near the amplitude plateau (epoch 16,000). (b) Learned functions at epochs 447, 16,000, and 80,000. The four black dots are training samples. (c) Directions and amplitudes of all 200 neurons at different epochs. Figure 2 illustrates the evolution from rapid data fitting to neuron condensation. The delayed decrease in test loss in panel (a) reveals a grokking phenomenon. Panel (b) shows the learned function gradually approaching the target as the network evolves toward a condensed solution that closely approximates the target function. The neuron feature snapshots in panel (c) show a complex representation with widely scattered features (blue dots) after rapid fitting. As amplitudes stabilize, the features develop a clearer directional structure (orange dots), and eventually form tight angular groups (green dots) corresponding to the condensed solution. Figure 3 resolves this evolution into three stages: 1. Stage I: Fitting. The data loss decreases rapidly (a), and the residual approaches its quasi-static reference (b), while neuron directions remain broadly distributed (f). 2. Stage I: Compression. Neuron amplitudes continue to decrease until they stabilize (c), while the attracting minima of the residual correlation field deepen substantially (d), strengthening directional attraction. Neuron directions begin to narrow but remain dispersed at the end of this stage (f). 3. Stage I: Condensation. After amplitudes have nearly stabilized (c), the angular diameters of the two neuron groups continue to decrease approximately exponentially (e), with neurons condensing to specific directions (f). Section 4 characterizes the residual left after fitting, the attracting directions and amplification of the residual correlation field, and how amplitude stabilization controls extrema drift and enables angular contraction. Figure 3: Diagnostics of the three stages. (a,b) Data MSE and residual norms ‖2\| e\|_2, ‖qs‖2\| e_qs\|_2; the inset in (b) shows the full run, including ‖−qs‖2\| e- e_qs\|_2 (dotted). (c) Individual amplitude differences Ak(t)−Ak(∞)A_k(t)-A_k(∞) (left axis), with Ak(∞)A_k(∞) approximated by the value at epoch 80,000, and total amplitude variation A(t)=∑k∈(t)|A˙k(t)|V_A(t)= _k (t)| A_k(t)| over all non-dead neurons (green curve, right axis). (d) Normalized depths −qt(θ∗(t))/(λ/2)-q_t( _*(t))/(λ/2) of the smooth negative minima associated with the two neuron groups. The inset shows qt(θ)/(λ/2)q_t(θ)/(λ/2) at epoch 5,000. The normalized depth plateau in Stage I can be explained by the balanced amplitude dynamics in Appendix A.2. (e,f) Angular diameters and angle snapshots of two neuron groups. 4 Main results We analyze how the residual correlation field governs neuron dynamics across the three training stages. We first identify the residual left after fast fitting, then establish the discreteness of the field’s attracting extrema and characterize how amplitude compression amplifies the field. The main condensation theorem connects small amplitude variation to exponential contraction of a group’s angular diameter. 4.1 Stage I: Fast residual equilibration Under the initialization in Section 3, let ρm=m1−2γ _m=m^1-2γ and εm=λ/ρm _m=λ/ _m. The initial kernel satisfies K0/ρm→K¯≻0K_0/ _m→ K 0 in probability as m→∞m→∞; see Appendix B.1 for the proof. The initial kernel has eigenvalues of order ρm=m1−2γ _m=m^1-2γ, which diverges to infinity since γ<1/2γ<1/2. The following theorem shows that the residual rapidly approaches its quasi-static reference while the kernel remains close to its initial value. Theorem 2 (Residual equilibration). Assume xi≠xjx_i≠ x_j for all i≠ji≠ j. Fix λ>0λ>0 and 1/4≤γ<1/21/4≤γ<1/2. Let κm=λmin(K0) _m= _ (K_0) and, on the event κm>0 _m>0, define TI=log(logm/εm)2(κm+λ).T_I= ( m/ _m)2( _m+λ). As m→∞m→∞, the nonlinear gradient flow satisfies TI T_I =Op(ρm−1log(ρm/λ))=op(1), =O_p( _m^-1 ( _m/λ))=o_p(1), (10) supt≤TI‖Kt−K0‖opρm _t≤ T_I K_t-K_0 _op _m =Op(ρm−1(logm)3/2)=op(1), =O_p( _m^-1( m)^3/2)=o_p(1), (11) (TI) e(T_I) =−εmK¯−1+op(εm),‖(TI)−qs(TI)‖2=op(εm). =- _m K^-1 y+o_p( _m), e(T_I)- e_qs(T_I) _2=o_p( _m). (12) The proof is given in Appendix B. Figure 5 shows the evolution of the NTK eigenvalues at finite width. The bound γ≥1/4γ≥ 1/4 is imposed to ensure that the initial residual satisfies ‖(0)‖2=Op(1)\| e(0)\|_2=O_p(1) in the infinite width limit. For any finite width network, the initial residual is finite. Equation (11) shows that the kernel changes by op(ρm)o_p( _m) during initial fitting, a vanishing fraction of its initial scale. Throughout this stage, its eigenvalues therefore remain much larger than the fixed weight decay coefficient λ. Thus, the residual relaxes on a time scale TI=op(1)T_I=o_p(1) is much faster than the effect of weight decay. For λ>0λ>0 and ≠ y≠ 0, the forcing term −2λ-2λ y sustains the quasi-static residual qs(t)=−λ(Kt+λIn)−1 e_qs(t)=-λ(K_t+λ I_n)^-1 y. At the end of initial fitting, equation (12) shows that the tracking error op(εm)o_p( _m) can be neglected. Thus the actual residual retains this nonzero leading term, which generates the correlation field governing subsequent neuron motion. 4.2 Stage I: Discrete extrema and amplification of qeq_e Let ~i=(xi,1)⊤ x_i=(x_i,1) and define the angular boundary set ℬ=θ∈[0,2π):~i⊤(θ)=0 for some i.B=\θ∈[0,2π): x_i u(θ)=0 for some i\. An activation cell I is a connected component of 1∖ℬS^1 . Its active sample set I=i:~i⊤(θ)>0,∀θ∈IS_I=\i: x_i u(θ)>0,\;∀θ∈ I\ is the same for every θ∈Iθ∈ I; a cell is dead if this set is empty. Define the residual moment on each cell by I()=1n∑i∈Iei~i. M_I( e)= 1n _i _Ie_i x_i. Within I, the residual correlation field has the sinusoidal form qe(θ)=MI,1()cosθ+MI,2()sinθ,q_e(θ)=M_I,1( e) θ+M_I,2( e) θ, (13) where MI,i()M_I,i( e) denotes the iith component of I() M_I( e). Recall that θ˙k=−2βkpe(θk) θ_k=-2 _kp_e( _k), where βk=ak/rk _k=a_k/r_k and pe=∂θqep_e= _θq_e. From equation (13) we have qe′(θ)=−qe(θ)q_e (θ)=-q_e(θ). For fixed residual e and ratio βk _k, the angular dynamics have equilibrium points where pe(θ∗)=0p_e( _*)=0. Within a cell with I()≠ M_I( e)≠ 0, these points are the smooth extrema of qeq_e. An equilibrium is locally attracting if and only if akqe(θ∗)<0a_kq_e( _*)<0. Thus positive neurons are attracted to negative minima, and negative neurons to positive maxima. The following proposition characterizes the discrete set of extrema that can serve as condensation directions. Proposition 3 (Discrete extrema). Fix a residual e and suppose I()≠ M_I( e)≠ 0 on every non-dead activation cell. Let ℰ()E( e) denote the set of strict local extrema of qeq_e, and define ℰint():=ℰ()∖ℬ,ℰbd():=ℰ()∩ℬ.E_int( e):=E( e) , _bd( e):=E( e) . These are, respectively, the smooth extrema inside activation cells and the extrema at activation boundaries, and satisfy 1≤|ℰ()|≤4n,|ℰint()|,|ℰbd()|≤2n.1≤|E( e)|≤ 4n, |E_int( e)|,\,|E_bd( e)|≤ 2n. The proof and a discussion of boundary extrema are given in Appendix C. We now focus on smooth extrema and their attraction strength. Consider a smooth negative minimum θ∗(t) _*(t) and a positive neuron in the same activation cell. Writing δk(t)=θk(t)−θ∗(t) _k(t)= _k(t)- _*(t), the sinusoidal form of the field equation (13) gives dδkdt=2βk(t)qt(θ∗(t))sinδk−dθ∗dt. d _kdt=2 _k(t)q_t( _*(t)) _k- d _*dt. Since βk(t)qt(θ∗(t))<0 _k(t)q_t( _*(t))<0, the first term drives the neuron toward the minimum. For a fixed scale ratio, a more negative value of qt(θ∗(t))q_t( _*(t)) produces stronger attraction. We next examine how weight decay changes the field’s strength during amplitude compression. Theorem 2 provides initial tracking of qs e_qs. The following result shows that slow kernel decay maintains this tracking and amplifies the residual correlation field during early amplitude compression. Set Cϕ:=supθ∉ℬ‖ϕθ()‖nC_φ:= _θ φ_θ( X) _n. Theorem 4 (Kernel decay and field amplification). Let ≠ y≠ 0, t0=TIt_0=T_I and consider [t0,t1][t_0,t_1] with λ(t1−t0)≤Lλ(t_1-t_0)≤ L, where L is fixed. Suppose the activation masks remain fixed and every non-dead neuron satisfies B−1≤|ak(t)rk(t)|≤B,|qt(θk(t))|+|pt(θk(t))|≤ελ.B^-1≤ | a_k(t)r_k(t) |≤ B, |q_t( _k(t))|+|p_t( _k(t))|≤ λ. Assume Kt0≻0K_t_0 0 and ‖Kt0‖opλmin(Kt0)≤χ,λmin(Kt0)≤ε, K_t_0 _op _ (K_t_0)≤χ, λ _ (K_t_0)≤ , where B,χB,χ are fixed and ε is sufficiently small. Set δ0=‖(t0)−qs(t0)‖n/‖qs(t0)‖n _0= e(t_0)- e_qs(t_0) _n/ e_qs(t_0) _n. Then, writing τ=t−t0τ=t-t_0, there exists C=C(B,L,χ)C=C(B,L,χ) such that ‖Kt−e−2λτKt0‖op K_t-e^-2λτK_t_0 _op ≤Cεe−2λτ‖Kt0‖op, ≤ C e^-2λτ K_t_0 _op, (14) sups∈[t0,t1]‖(s)−qs(s)‖n‖qs(s)‖n _s∈[t_0,t_1] e(s)- e_qs(s) _n e_qs(s) _n ≤C(δ0+ε), ≤ C( _0+ ), (15) supθ|qt(θ)−e2λτqt0(θ)| _θ |q_t(θ)-e^2λτq_t_0(θ) | ≤C(ε+δ0)e2λτCϕ‖qs(t0)‖n. ≤ C( + _0)e^2λτC_φ e_qs(t_0) _n. (16) The proof is given in Appendix C.2. Stage I gives δ0=op(1) _0=o_p(1) at TIT_I, and equation (15) controls residual tracking throughout the stated compression interval. Consistent with this result, Figure 3(b) shows that (t) e(t) remains close to qs(t) e_qs(t) after rapid fitting. When the correlations are small relative to λ, weight decay dominates parameter evolution and approximately rescales the kernel by e−2λτe^-2λτ. As weight decay reduces the kernel, qs(t)≈−λKt−1 e_qs(t)≈-λ K_t^-1 y grows, amplifying the correlation field. The leading order scaling of weight decay changes the depths of the extrema but leaves their locations unchanged. However, in the actual dynamics, Figure 4 shows that the smooth extrema also move during Stage I. The following section controls this motion, which results in neuron condensation. Figure 4: Residual correlation field evolution and surviving neurons. Curves show normalized qt(θ)/(λ/2)q_t(θ)/(λ/2), computed from the actual residual e (black solid) and its quasi-stationary approximation qs e_qs (pink dashed), which nearly overlap. Blue and orange dots represent positive and negative neurons respectively. Black triangles mark smooth local minima. During Stage I (up to epoch 16,000), the field deepens substantially, while neuron directions remain dispersed around its extrema. During Stage I, neurons condense at these extrema. 4.3 Stage I: Slow drift of extrema and neuron condensation Let (t)=k:ϕθk(t)()≠N(t)=\k: φ_ _k(t)( X)≠ 0\ be the set of non-dead neurons. Define their total amplitude, total amplitude variation rate, and angular energy by (t)=∑k∈(t)Ak(t),A(t)=∑k∈(t)|dAk(t)dt|,P(t)=∑k∈(t)Ak(t)pk(t)2.A(t)= _k (t)A_k(t), _A(t)= _k (t) | dA_k(t)dt |, P(t)= _k (t)A_k(t)p_k(t)^2. (17) We also use Cψ:=supθ∉ℬ‖θ()‖nC_ψ:= _θ ψ_θ( X) _n and Cϕ=supθ∉ℬ‖ϕθ()‖nC_φ= _θ φ_θ( X) _n. Attracting extrema of the residual correlation field evolve during training. To understand whether neurons can track these extrema and condense around them, we first bound their drift. The following theorem shows how amplitude variation AV_A controls the angular energy P, and how these quantities jointly bound the velocity of the residual correlation field extrema. Theorem 5 (Field drift). Suppose that, for all t≥Tt≥ T, the gradient flow remains in fixed open activation cells and every non-dead neuron satisfies ak2=rk2=Ak>0a_k^2=r_k^2=A_k>0. Set ¯T:=Φ(Θ(T))/λ A_T:= ( (T))/λ. Then, for all t≥Tt≥ T, P(t) P(t) ≤e−2λ(t−T)P(T)+Cϕ24∫Tte−2λ(t−s)A(s)2s, ≤e^-2λ(t-T)P(T)+ C_φ^24 _T^te^-2λ(t-s)V_A(s)^2\,ds, (18) ‖d(t)dt‖n d e(t)dt _n ≤CϕA(t)+2Cψ¯TP(t). ≤ C_φV_A(t)+2C_ψ A_TP(t). (19) If θ∗(t) _*(t) is a differentiable branch of smooth negative minima in a fixed cell with depth R(t)=−qt(θ∗(t))≥R0>0R(t)=-q_t( _*(t))≥ R_0>0 for all t≥Tt≥ T, then |dθ∗(t)dt|≤CψR0[CϕA(t)+2Cψ¯TP(t)]. | d _*(t)dt |≤ C_ψR_0 [C_φV_A(t)+2C_ψ A_TP(t) ]. (20) The proof is given in Appendix D.2. Theorem 5 bounds the drift of attracting minima by the amplitude variation A(t)V_A(t) and the angular energy P(t)P(t). The latter is itself controlled by amplitude variation: if A(t)≤νAV_A(t)≤ _A for all t≥Tt≥ T, then equation (18) gives P(t)≤e−2λ(t−T)P(T)+Cϕ2νA28λ.P(t)≤e^-2λ(t-T)P(T)+ C_φ^2 _A^28λ. (21) The first term in equation (21) captures the influence of the initial angular energy and decays exponentially. Once it falls below fixed ε2 ^2, substituting the resulting bound on P(t)P(t) and A(t)≤νAV_A(t)≤ _A into equation (20) yields |dθ∗(t)dt| | d _*(t)dt | ≤CψR0[CϕνA+2Cψ¯T(ε2+Cϕ2νA28λ)] ≤ C_ψR_0 [C_φ _A+2C_ψ A_T ( ^2+ C_φ^2 _A^28λ ) ] (22) ≤CψCϕR0(1+Cψ¯T2λ)νA+2Cψ2¯TR0ε. ≤ C_ψC_φR_0 (1+C_ψ A_T2λ ) _A+ 2C_ψ^2 A_TR_0 . Thus, the drift speed is bounded by a term proportional to νA _A. Figure 3(c) shows that A(t)V_A(t) remains small during Stage I, supporting a small upper bound νA _A over this period and motivating the slow extrema drift. Weight decay also reduces the absolute imbalance ak2−rk2a_k^2-r_k^2 through equation (7), motivating the balanced model for surviving neurons. The next theorem shows that sufficiently slow drift keeps nearby neurons in an attracting band and yields angular contraction. Theorem 6 (Angular contraction). Let G be a fixed group of at least two positive neurons and θ∗(t) _*(t) a differentiable branch of smooth negative minima. For all t≥Tt≥ T, assume that ak(t)=rk(t)>0a_k(t)=r_k(t)>0 for every k∈Gk∈ G, the minimum depth satisfies R(t)≥R0>0R(t)≥ R_0>0, and the band [θ∗(t)−d,θ∗(t)+d][ _*(t)-d, _*(t)+d] lies inside one fixed open activation cell, where 0<d<π/20<d<π/2. If |θk(T)−θ∗(T)|≤d(k∈G),supt≥T|dθ∗(t)dt|<2R0sind,| _k(T)- _*(T)|≤ d (k∈ G), _t≥ T | d _*(t)dt |<2R_0 d, then every neuron in G remains in the band and, with cd=2R0cosd>0c_d=2R_0 d>0, diamθG(t)≤e−cd(t−T)diamθG(T),t≥T.diam_θG(t)≤e^-c_d(t-T)diam_θG(T), t≥ T. (23) Consequently, G undergoes neuron condensation. If additionally dθ∗(t)dt→0 d _*(t)dt→ 0, then |θk(t)−θ∗(t)|→0| _k(t)- _*(t)|→ 0 for every k∈Gk∈ G. The proof is given in Appendix D.6. The theorem shows that a group can condense around a moving minimum: sufficiently slow drift retains the neurons in the attracting band, where their angular diameter decreases exponentially. Together with Theorem 5, this gives the connection between amplitude stabilization and condensation. Small amplitude variation and a decayed initial angular-energy contribution yield a small drift bound. Once this bound is below 2R0sind2R_0 d, the contraction theorem applies to neurons then inside the band, as long as the stated conditions persist. This describes how angular concentration can continue during Stage I after amplitudes vary little. The analogous result holds for exactly balanced negative neurons near a smooth positive maximum. 5 Discussion Experiments with additional target functions in Appendix F.2 show the same progression from rapid fitting to late condensation. Our analysis characterizes neuron condensation around attracting extrema of the residual correlation field, but does not establish that these directions coincide with those in the target representation. The additional experiments also exhibit more condensation directions than those in the target representation. Appendix F.1 compares large initialization with and without weight decay against small initialization without weight decay. Both large initialization with weight decay and small initialization produce condensed solutions, but the former exhibits a distinct rapid fitting phase before condensation. Our dynamical analysis focuses on two-layer ReLU networks, while a corresponding dynamical theory for Transformers remains open. 6 Conclusion Our work explains a mechanism for neuron condensation in two-layer ReLU networks trained with weight decay under large initialization. This mechanism differs from persistent lazy training at large initialization and condensation at small initialization. Our experiments show that rapid fitting precedes substantial directional alignment: the network first fits the data, then develops a condensed representation during continued training with weight decay. By analyzing amplitude and angular dynamics, we characterize three stages: rapid fitting, amplitude compression, and neuron condensation. During rapid fitting, the residual approaches a quasi-static equilibrium maintained by weight decay. In the early stage of amplitude compression, we show that the residual correlation field admits only finitely many isolated attracting extrema and that early amplitude compression amplifies the field. As neuron amplitudes stabilize, we prove that neuron directions condense around slowly moving attracting directions. This analysis links weight decay to the formation of a condensed representation, clarifying its role beyond parameter norm reduction. References Bach (2017) F. Bach Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research 18 (19), p. 1–53. External Links: Link Cited by: §2. Boursier and Flammarion (2025) E. Boursier and N. Flammarion Early alignment in two-layer networks training is a two-edged sword. Journal of Machine Learning Research 26 (183), p. 1–75. External Links: Link Cited by: §2. Boursier et al. (2022) E. Boursier, L. Pillaud-Vivien, and N. Flammarion Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs. In Advances in Neural Information Processing Systems, Vol. 35. External Links: Link Cited by: §2. Chen et al. (2024) Z. Chen, Y. Li, T. Luo, Z. Zhou, and Z. J. Xu Phase diagram of initial condensation for two-layer neural networks. CSIAM Transactions on Applied Mathematics 5 (3), p. 448–514. External Links: Document, Link Cited by: §2. Chen and Luo (2025) Z. Chen and T. Luo From condensation to rank collapse: a two-stage analysis of transformer training dynamics. In Advances in Neural Information Processing Systems, Vol. 38. External Links: Link Cited by: §2. Chizat and Bach (2020) L. Chizat and F. Bach Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Proceedings of the Conference on Learning Theory, Proceedings of Machine Learning Research, Vol. 125, p. 1305–1338. External Links: Link Cited by: §1. Chizat et al. (2019) L. Chizat, E. Oyallon, and F. Bach On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, Vol. 32. External Links: Link Cited by: §1, §1, §2. Jacot et al. (2018) A. Jacot, F. Gabriel, and C. Hongler Neural tangent kernel: convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, Vol. 31. External Links: Link Cited by: §1, §2. Jacot et al. (2025) A. Jacot, P. Súkeník, Z. Wang, and M. Mondelli Wide neural networks trained with weight decay provably exhibit neural collapse. In International Conference on Learning Representations, External Links: Link Cited by: §2. Kumar and Haupt (2024) A. Kumar and J. Haupt Directional convergence near small initializations and saddles in two-homogeneous neural networks. Transactions on Machine Learning Research. External Links: Link Cited by: §2. Kumar et al. (2024) T. Kumar, B. Bordelon, S. J. Gershman, and C. Pehlevan Grokking as the transition from lazy to rich training dynamics. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Kuzborskij and Abbasi-Yadkori (2026) I. Kuzborskij and Y. Abbasi-Yadkori Low-rank bias, weight decay, and model merging in neural networks. In Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 300, p. 460–468. External Links: Link Cited by: §2. Lewkowycz and Gur-Ari (2020) A. Lewkowycz and G. Gur-Ari On the training dynamics of deep networks with L2L_2 regularization. In Advances in Neural Information Processing Systems, Vol. 33. External Links: Link Cited by: §2. Liu et al. (2023) Z. Liu, E. J. Michaud, and M. Tegmark Omnigrok: grokking beyond algorithmic data. In International Conference on Learning Representations, External Links: Link Cited by: §1. Luo et al. (2021) T. Luo, Z. J. Xu, Z. Ma, and Y. Zhang Phase diagram for two-layer ReLU neural networks at infinite-width limit. Journal of Machine Learning Research 22 (71), p. 1–47. External Links: Link Cited by: §1, §1, §1, §2. Lyu et al. (2024) K. Lyu, J. Jin, Z. Li, S. S. Du, J. D. Lee, and W. Hu Dichotomy of early and late phase implicit biases can provably induce grokking. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Maennel et al. (2018) H. Maennel, O. Bousquet, and S. Gelly Gradient descent quantizes ReLU network features. arXiv preprint arXiv:1803.08367. External Links: Link Cited by: §1, §2. Parhi and Nowak (2021) R. Parhi and R. D. Nowak Banach space representer theorems for neural networks and ridge splines. Journal of Machine Learning Research 22 (43), p. 1–40. External Links: Link Cited by: §1. Pilanci and Ergen (2020) M. Pilanci and T. Ergen Neural networks are convex regularizers: exact polynomial-time convex optimization formulations for two-layer networks. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, p. 7695–7705. External Links: Link Cited by: §2. Power et al. (2022) A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra Grokking: generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177. External Links: Link Cited by: §1. Savarese et al. (2019) P. Savarese, I. Evron, D. Soudry, and N. Srebro How do infinite width bounded norm networks look in function space?. In Proceedings of the Conference on Learning Theory, Proceedings of Machine Learning Research, Vol. 99, p. 2667–2690. External Links: Link Cited by: §1. Srivastava et al. (2014) N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15 (56), p. 1929–1958. Cited by: §1. Xu et al. (2025a) Z. J. Xu, Y. Zhang, and T. Luo Overview frequency principle/spectral bias in deep learning. Communications on Applied Mathematics and Computation 7 (3), p. 827–864. External Links: arXiv:2201.07395, Link Cited by: §1. Xu et al. (2025b) Z. J. Xu, Y. Zhang, and Z. Zhou An overview of condensation phenomenon in deep learning. arXiv preprint arXiv:2504.09484. External Links: Link Cited by: §1, §1. Zhang et al. (2017) C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, Cited by: §1. Zhang et al. (2022) Y. Zhang, Y. Li, Z. Zhang, T. Luo, and Z. J. Xu Embedding principle: a hierarchical structure of loss landscape of deep neural networks. Journal of Machine Learning, p. 60–113. External Links: Document, arXiv:2111.15527, Link Cited by: §1. Zhang et al. (2021) Y. Zhang, Z. Zhang, T. Luo, and Z. J. Xu Embedding principle of loss landscape of deep neural networks. In NeurIPS 2021 spotlight, Vol. 34. External Links: arXiv:2105.14573, Link Cited by: §1. Zhang et al. (2024) Z. Zhang, P. Lin, Z. Wang, Y. Zhang, and Z. J. Xu Initialization is critical to whether transformers fit composite functions by reasoning or memorizing. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Link Cited by: Appendix G, §1, §2. Zhang et al. (2026) Z. Zhang, Z. Wang, J. Yao, Z. Zhou, X. Li, W. E, and Z. J. Xu Anchor function: a type of benchmark functions for studying language models. Journal of Machine Learning 5 (2), p. 97–134. External Links: Document, Link Cited by: §1. Zhang and Xu (2024) Z. Zhang and Z. J. Xu Implicit regularization of dropout. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (6), p. 4206–4217. External Links: Link Cited by: §1, §2. Zhou et al. (2022) H. Zhou, Q. Zhou, T. Luo, Y. Zhang, and Z. J. Xu Towards understanding the condensation of neural networks at initial training. In Advances in Neural Information Processing Systems, Vol. 35. External Links: Link Cited by: §1, §1, §2. Appendix A Neuron dynamics A.1 Neuron dynamics and balance We derive Proposition 1 using the loss normalization ℒ=‖n2L= e _n^2 in equation (2). Write k=(θk) u_k= u( _k), ⟂,k=⟂(θk)=(−sinθk,cosθk)⊤ u_ ,k= u_ ( _k)=(- _k, _k) , let IkI_k be the activation cell containing θk _k, and set χki=k⊤~i>0 _ki=1_\ v_k x_i>0\ inside an open cell. Define ϕk(t):=ϕθk(t)(),k(t):=θk(t)(). φ_k(t):= φ_ _k(t)( X), ψ_k(t):= ψ_ _k(t)( X). The derivatives of a training prediction are ∂akfΘ(xi)=rk(ϕk)i,∇kfΘ(xi)=akχki~i. _a_kf_ (x_i)=r_k( φ_k)_i, _ v_kf_ (x_i)=a_k _ki x_i. Consequently, with k=n−1∑ieiχki~i=Ik() M_k=n^-1 _ie_i _ki x_i= M_I_k( e), ∂akℒ _a_kL =2n∑ieirk(ϕk)i=2rkqk, = 2n _ie_ir_k( φ_k)_i=2r_kq_k, ∇kℒ _ v_kL =2n∑ieiakχki~i=2akk, = 2n _ie_ia_k _ki x_i=2a_k M_k, dakdt da_kdt =−2rkqk−λak,dkdt=−2akk−λk. =-2r_kq_k-λ a_k, d v_kdt=-2a_k M_k-λ v_k. (24) In particular, the two components of the last equation are dwkdt=−2akn∑ieiχkixi−λwk,dbkdt=−2akn∑ieiχki−λbk. dw_kdt=- 2a_kn _ie_i _kix_i-λ w_k, db_kdt=- 2a_kn _ie_i _ki-λ b_k. Radial and angular projections. For rk>0r_k>0, differentiating k=rkk v_k=r_k u_k gives dkdt=drkdtk+rkdθkdt⟂,k d v_kdt= dr_kdt u_k+r_k d _kdt u_ ,k. The orthonormality of k,⟂,k u_k, u_ ,k and the identities k⊤k=qk u_k M_k=q_k, ⟂,k⊤k=pk u_ ,k M_k=p_k imply drkdt dr_kdt =k⊤dkdt=−2akqk−λrk, = u_k d v_kdt=-2a_kq_k-λ r_k, rkdθkdt r_k d _kdt =⟂,k⊤dkdt=−2akpk. = u_ ,k d v_kdt=-2a_kp_k. (25) Dividing the second equation by rkr_k gives the angular equation in equation (5). The decay term has zero tangential projection because ⟂,k⊤k=0 u_ ,k v_k=0. Balance and inactive neurons. The mixed data-gradient terms cancel exactly: dBkdt dB_kdt =2ak(−2rkqk−λak)−2rk(−2akqk−λrk) =2a_k(-2r_kq_k-λ a_k)-2r_k(-2a_kq_k-λ r_k) =−2λ(ak2−rk2)=−2λBk. =-2λ(a_k^2-r_k^2)=-2λ B_k. (26) Multiplication by e2λte^2λ t proves equation (7). It also gives the absolute scale bound ||ak(t)|−rk(t)|2≤|ak(t)2−rk(t)2|=e−2λt|Bk(0)|. ||a_k(t)|-r_k(t) |^2≤|a_k(t)^2-r_k(t)^2|=e^-2λ t|B_k(0)|. If k(0)⊤~i<0 v_k(0) x_i<0 for every i, all data gradients vanish. The solution is ak(t)=e−λtak(0)a_k(t)=e^-λ ta_k(0) and k(t)=e−λtk(0) v_k(t)=e^-λ t v_k(0). Every strict inactive inequality therefore persists, θk _k stays fixed when defined, and Ak(t)=e−2λtAk(0)A_k(t)=e^-2λ tA_k(0). Residual dynamics. The parameter equation is dΘdt=−2Jt⊤/n−λΘ d dt=-2J_t e/n-λ . For each sample, the contribution of neuron k to JtΘJ_t is ak∂akfΘ(xi)+k⊤∇kfΘ(xi)=2akReLU(k⊤~i).a_k _a_kf_ (x_i)+ v_k _ v_kf_ (x_i)=2a_kReLU( v_k x_i). Thus JtΘ=2fΘ()J_t =2f_ ( X), and the chain rule gives ddt=JtdΘdt d edt=J_t d dt =−2nJtJt⊤−λJtΘ =- 2nJ_tJ_t e-λ J_t =−2Kt−2λfΘ()=−2(Kt+λIn)−2λ. =-2K_t e-2λ f_ ( X)=-2(K_t+λ I_n) e-2λ y. (27) For fixed K⪰0K 0, setting the residual velocity to zero gives eq(K)=−λ(K+λIn)−1 e_eq(K)=-λ(K+λ I_n)^-1 y. At zero preactivation, use the same selection χki∈[0,1] _ki∈[0,1] in the parameter flow and Jacobian, with pkp_k denoting the selected angular gradient. Along an absolutely continuous trajectory, preactivation derivatives vanish almost everywhere on their zero sets; the ReLU chain rule and the identities above therefore hold almost everywhere, including boundary contacts. In particular, dtΦ(Θ(t))=−‖dΘdt‖22≤0. ddt ( (t))=- d dt _2^2≤ 0. A.2 General and balanced amplitude dynamics For rk>0r_k>0, the quotient rule and equation (5) give dβkdt d _kdt =dakdtrk−akdrkdtrk2 = da_kdtr_k-a_k dr_kdtr_k^2 =(−2rkqk−λak)rk−ak(−2akqk−λrk)rk2=2qk(βk2−1). = (-2r_kq_k-λ a_k)r_k-a_k(-2a_kq_k-λ r_k)r_k^2=2q_k( _k^2-1). (28) On any interval where ak≠0a_k≠ 0, its sign sks_k is constant and Ak=|ak|rk>0A_k=|a_k|r_k>0. The logarithmic derivative is dAkdtAk dA_kdtA_k =dakdtak+drkdtrk = da_kdta_k+ dr_kdtr_k =−2rkakqk−λ−2akrkqk−λ=−2λ−2qk(βk+βk−1). =-2 r_ka_kq_k-λ-2 a_kr_kq_k-λ=-2λ-2q_k( _k+ _k^-1). (29) Equivalently, dAkdt=−2λAk−2sk(ak2+rk2)qk dA_kdt=-2λ A_k-2s_k(a_k^2+r_k^2)q_k. If Bk(T)=0B_k(T)=0, the balance identity preserves |ak|=rk|a_k|=r_k; for a nonzero neuron this means βk=sk _k=s_k and Ak=ak2=rk2A_k=a_k^2=r_k^2. Substitution gives the balanced dynamics dAkdt=−2Ak(λ+2skqk),dθkdt=−2skpk. dA_kdt=-2A_k(λ+2s_kq_k), d _kdt=-2s_kp_k. (30) In this regime, Ak(t)=Ak(T)exp[−2λ(t−T)−4sk∫Ttqk(s)s],A_k(t)=A_k(T) \! [-2λ(t-T)-4s_k _T^tq_k(s)\,ds ], so a positive amplitude remains positive on every finite interval with bounded correlations. The general equations retain the evolving ratio βk _k; under exact balance, βk=sk _k=s_k gives the reduced system equation (30). For surviving positive neurons with nearly stationary amplitudes and directions close to attracting minima, these dynamics give −2qt(θ∗)/λ≈1-2q_t( _*)/λ≈ 1, as shown in Figure 3(d). Appendix B Fast residual equilibration B.1 Proof of Theorem 2: initialization The proof has three steps: initialization estimates, short-time kernel stability, and transfer to the residual scale. Writing αij _ij for the angle between ~i x_i and ~j x_j, the limiting kernel is K¯ij=‖~i‖2‖~j‖22πn[sinαij+2(π−αij)cosαij]. K_ij= x_i _2 x_j _22π n [ _ij+2(π- _ij) _ij ]. (31) Let σm=m−γ _m=m^-γ, and write ak(0)=σmξka_k(0)= _m _k, k(0)=σmk v_k(0)= _m g_k with independent standard Gaussian ξk _k and k∈ℝ2 g_k ^2. Each output summand has mean zero and variance [ak(0)2ReLU(k(0)⊤~i)2]=σm4ξk2ReLU(k⊤~i)2=σm42‖~i‖22.E [a_k(0)^2ReLU( v_k(0) x_i)^2 ]= _m^4E _k^2\,EReLU( g_k x_i)^2= _m^42 x_i _2^2. Independence across neurons gives Var(fΘ(0)(xi))=(1+xi2)m1−4γ/2Var(f_ (0)(x_i))=(1+x_i^2)m^1-4γ/2. For fixed n,n, y and γ≥1/4γ≥ 1/4, summing these second moments gives ‖(0)‖2=Op(1) e(0) _2=O_p(1). With χki(t)=k(t)⊤~i>0 _ki(t)=1_\ v_k(t) x_i>0\, the empirical kernel entries are (Kt)ij=1n∑k[ReLU(k⊤~i)ReLU(k⊤~j)+ak2χkiχkj~i⊤~j].(K_t)_ij= 1n _k [ReLU( v_k x_i)ReLU( v_k x_j)+a_k^2 _ki _kj x_i x_j ]. To compute the expectation, rotate coordinates so that the directions of ~i,~j x_i, x_j have angles 0,α0,α, where α=αij∈[0,π]α= _ij∈[0,π]. Write a standard Gaussian as =Rg(cosφ,sinφ) g=R_g( , ), with φ uniform on [0,2π)[0,2π), independent of RgR_g, and Rg2=2ER_g^2=2. The common positive half-planes overlap on [α−π/2,π/2][α-π/2,π/2], modulo 2π2π. Hence [ReLU(⊤~i)ReLU(⊤~j)] [ReLU( g x_i)ReLU( g x_j)] =‖~i‖2‖~j‖2π∫α−π/2π/2cosφcos(φ−α)φ = x_i _2 x_j _2π _α-π/2^π/2 ( -α)\,d =‖~i‖2‖~j‖22π[sinα+(π−α)cosα], = x_i _2 x_j _22π[ α+(π-α) α], ℙ(⊤~i>0,⊤~j>0) ( g x_i>0,\ g x_j>0) =π−α2π. = π-α2π. (32) The integral follows from 2cosφcos(φ−α)=cosα+cos(2φ−α)2 ( -α)= α+ (2 -α). Adding the second kernel term, using ξk2=1E _k^2=1 and ~i⊤~j=‖~i‖2‖~j‖2cosα x_i x_j= x_i _2 x_j _2 α, proves equation (31). Since ρm=mσm2 _m=m _m^2, the law of large numbers for the independent kernel summands gives K0/ρm→K¯K_0/ _m→ K in operator norm in probability, with n fixed. For positivity, a zero quadratic form of the random-feature contribution would imply ∑iziReLU(⊤~i)=0 _iz_iReLU( g x_i)=0 for almost every Gaussian g, hence for every g by continuity. At a nonzero point on ⊤~j=0 g x_j=0, the other terms are smooth because the inputs are distinct. The gradient jump is therefore zj~j=0z_j x_j=0, so every zj=0z_j=0 and K¯≻0 K 0. B.2 Proof of Theorem 2: kernel stability Residual and parameter bounds. Let hmh_m be a deterministic horizon with hmlogm→0h_m m→ 0. Positivity of KtK_t gives 12dt‖22=−2⊤Kt−2λ‖22−2λ⊤≤2λ‖2‖2. 12 ddt e _2^2=-2 e K_t e-2λ e _2^2-2λ e y≤ 2λ e _2 y _2. Thus d‖2/dt≤2λ‖2d e _2/dt≤ 2λ y _2 (in the upper Dini sense at =0 e=0). Since ‖(0)‖2=Op(1) e(0) _2=O_p(1), the residual is uniformly bounded in probability on [0,hm][0,h_m]. Localize to an arbitrarily high-probability event on which this bound and the Gaussian initial-norm bounds below have fixed constants. With Cx=(n−1∑i‖~i‖22)1/2C_x=(n^-1 _i x_i _2^2)^1/2, ‖k‖2≤Cx‖n,|qk|≤Cx‖n. M_k _2≤ C_x e _n, |q_k|≤ C_x e _n. For k=(ak,k) η_k=(a_k, v_k), equation (24) consequently gives ‖dkdt‖2≤C‖k‖2 d η_kdt _2≤ C η_k _2, with C independent of k,mk,m. Gronwall’s inequality yields ‖k(t)‖2 η_k(t) _2 ≤eCt‖k(0)‖2, ≤e^Ct η_k(0) _2, ‖k(t)−k(0)‖2 η_k(t)- η_k(0) _2 ≤(eCt−1)‖k(0)‖2. ≤(e^Ct-1) η_k(0) _2. In particular, supt≤hm‖k(t)−k(0)‖2≤Chmak(0)2+rk(0)2. _t≤ h_m η_k(t)- η_k(0) _2≤ Ch_m a_k(0)^2+r_k(0)^2. The Gaussian initialization also gives maxk‖k(0)‖2=Op(m−γlogm),1m∑km2γ‖k(0)‖22=Op(1). _k η_k(0) _2=O_p(m^-γ m), 1m _km^2γ η_k(0) _2^2=O_p(1). Boundary contacts and kernel variation. A contact with the boundary of sample i can occur only if |k(0)⊤~i|≤Cm−γhmlogm| v_k(0) x_i|≤ Cm^-γh_m m on the localized event. Lipschitz continuity controls the feature products and changes in ak2a_k^2; changes in activation masks are bounded by the initial squared output weights on these contact events. Hence supt≤hm‖Kt−K0‖opρm _t≤ h_m K_t-K_0 _op _m ≤Chmm∑km2γ(ak(0)2+rk(0)2) ≤ Ch_mm _km^2γ(a_k(0)^2+r_k(0)^2) +Cm∑km2γak(0)2∑i|k(0)⊤~i|≤Cm−γhmlogm. + Cm _km^2γa_k(0)^2 _i1_\| v_k(0) x_i|≤ Cm^-γh_m m\. Independence of ak(0)a_k(0) and k(0) v_k(0) and the bounded Gaussian density of k⊤~i g_k x_i imply [ξk2|k⊤~i|≤Chmlogm] \! [ _k^21_\| g_k x_i|≤ Ch_m m\ ] =ξk2ℙ(|k⊤~i|≤Chmlogm) =E _k^2\,P(| g_k x_i|≤ Ch_m m) ≤C′hmlogm. ≤ C h_m m. Since n is fixed, Markov’s inequality gives supt≤hm‖Kt−K0‖opρm=Op(hmlogm). _t≤ h_m K_t-K_0 _op _m=O_p(h_m m). Choosing hm=ρm−1λmin(K¯)log(logm/εm)h_m= _m^-1 _ ( K) ( m/ _m) ensures TI≤hmT_I≤ h_m with probability tending to one and proves equation (11). B.3 Proof of Theorem 2: residual equilibration Recall qs(0)=−λ(K0+λIn)−1 e_qs(0)=-λ(K_0+λ I_n)^-1 y, and set Δm=supt≤TI‖Kt−K0‖op _m= _t≤ T_I K_t-K_0 _op. Then Δm/κm=op(1) _m/ _m=o_p(1) and ΔmTI=Op(ρm−1(logm)5/2)=op(1) _mT_I=O_p( _m^-1( m)^5/2)=o_p(1). For 0(t)=(t)−qs(0) z_0(t)= e(t)- e_qs(0), subtract the frozen equilibrium relation to obtain d0dt=−2(Kt+λIn)0−2(Kt−K0)qs(0). d z_0dt=-2(K_t+λ I_n) z_0-2(K_t-K_0) e_qs(0). On Δm<κm _m< _m, the homogeneous evolution operator U(t,s)U(t,s) satisfies ‖U(t,s)‖op≤e−2(κm+λ−Δm)(t−s) U(t,s) _op≤e^-2( _m+λ- _m)(t-s). This follows by differentiating the squared norm of a homogeneous solution and using Kt⪰(κm−Δm)InK_t ( _m- _m)I_n. Variation of constants gives 0(t)=U(t,0)0(0)−2∫0tU(t,s)(Ks−K0)qs(0)s. z_0(t)=U(t,0) z_0(0)-2 _0^tU(t,s)(K_s-K_0) e_qs(0)\,ds. Bounding the integral by its exponential envelope yields ‖0(TI)‖2 z_0(T_I) _2 ≤e−2(κm+λ−Δm)TI‖0(0)‖2+Δmκm+λ−Δm‖qs(0)‖2. ≤e^-2( _m+λ- _m)T_I z_0(0) _2+ _m _m+λ- _m e_qs(0) _2. (33) The first term is Op(εm/logm)e2ΔmTIO_p( _m/ m)e^2 _mT_I and the second is op(εm)o_p( _m). Since qs(0)=−εmK¯−1+op(εm) e_qs(0)=- _m K^-1 y+o_p( _m), the first expansion follows. The resolvent identity gives qs(t)−qs(0) e_qs(t)- e_qs(0) =λ(Kt+λIn)−1(Kt−K0)(K0+λIn)−1, =λ(K_t+λ I_n)^-1(K_t-K_0)(K_0+λ I_n)^-1 y, ‖qs(t)−qs(0)‖2 e_qs(t)- e_qs(0) _2 ≤λΔm‖2(κm+λ−Δm)(κm+λ)=op(εm),t≤TI. ≤ λ _m y _2( _m+λ- _m)( _m+λ)=o_p( _m), t≤ T_I. (34) Here κm/ρm→λmin(K¯)>0 _m/ _m→ _ ( K)>0 and Δm/ρm=op(1) _m/ _m=o_p(1). Combining this bound with the expansion of (TI) e(T_I) proves its claimed tracking error relative to qs(TI) e_qs(T_I) and completes Theorem 2. Appendix C Discrete extrema and amplification of qeq_e C.1 Proof of Proposition 3 Proof. Each equation ~i⊤(θ)=0 x_i u(θ)=0 contributes two antipodal boundary angles. Thus there are at most 2n2n boundaries and at most 2n2n open activation cells, each of length at most π. On a non-dead cell I, the assumption I()≠ M_I( e)≠ 0 makes qe(θ)=I()⊤(θ)q_e(θ)= M_I( e) u(θ) a nonconstant sinusoid. Its critical points are spaced by π, so each open cell contains at most one smooth extremum. A dead cell has qe=0q_e=0 and no strict interior extremum. Consequently, |ℰint()|≤2n,|ℰbd()|≤|ℬ|≤2n.|E_int( e)|≤ 2n, |E_bd( e)|≤|B|≤ 2n. Every strict local extremum is either inside a cell or at an activation boundary, giving |ℰ()|≤4n|E( e)|≤ 4n. Finally, qeq_e is continuous on the circle and nonzero on some non-dead cell, so it attains a nonzero global maximum or minimum. The points attaining this value cannot lie in dead-cell interiors; they belong to the finite set of interior critical points and activation boundaries. Each such point is therefore isolated among the global optimizers and is a strict local extremum. Hence |ℰ()|≥1|E( e)|≥ 1. ∎ Activation boundaries are fixed by the inputs. A boundary extremum therefore has fixed position while it persists; for positive neurons, the one-sided condition pt(θb−)<0<pt(θb+)p_t( _b^-)<0<p_t( _b^+) gives attraction toward the boundary. C.2 Proof of Theorem 4 Proof. Write K∗=Kt0K_*=K_t_0, κ∗:=λmin(K∗)>0 _*:= _ (K_*)>0, τ=t−t0τ=t-t_0, and α(t)=e−2λτα(t)=e^-2λτ. For the kernel calculation, abbreviate qk=qt(θk(t))q_k=q_t( _k(t)) and pk=pt(θk(t))p_k=p_t( _k(t)). Kernel derivative. For each neuron, let χki _ki be its fixed activation indicator on input i, and let (Sk)ij:=χkiχkj~i⊤~j.(S_k)_ij:= _ki _kj x_i x_j. The polar identity χki~i=(ϕk)ik+(k)i⟂,k _ki x_i=( φ_k)_i u_k+( ψ_k)_i u_ ,k gives Sk=ϕkϕk⊤+kk⊤S_k= φ_k φ_k + ψ_k ψ_k . The matrix SkS_k is constant while the mask is fixed. The contribution of neuron k to the kernel is therefore Kk=1n(rk2ϕkϕk⊤+ak2Sk),Kt=∑kKk.K_k= 1n(r_k^2 φ_k φ_k +a_k^2S_k), K_t= _kK_k. Using dϕkdt=kdθkdt d φ_kdt= ψ_k d _kdt and the parameter equations, we obtain dKkdt dK_kdt =−2λKk+Ek, =-2λ K_k+E_k, Ek E_k =−4akrkqkn(ϕkϕk⊤+Sk)−2akrkpkn(kϕk⊤+ϕkk⊤). =- 4a_kr_kq_kn( φ_k φ_k +S_k)- 2a_kr_kp_kn( ψ_k φ_k + φ_k ψ_k ). (35) For any real vectors , v, w, −(⊤+⊤)⪯⊤+⊤⪯⊤+⊤-( v v + w w ) v w + w v v v + w w . Also |akrk|≤Brk2|a_kr_k|≤ Br_k^2 and |akrk|≤Bak2|a_kr_k|≤ Ba_k^2 by the ratio assumption. Thus the first term of EkE_k is bounded above and below by ±4B|qk|Kk± 4B|q_k|K_k, and the second by ±2B|pk|Kk± 2B|p_k|K_k. Dead neurons contribute zero to KtK_t. Summing and using |qk|+|pk|≤ελ|q_k|+|p_k|≤ λ gives dKtdt=−2λKt+E(t),−4BελKt⪯E(t)⪯4BελKt. dK_tdt=-2λ K_t+E(t), -4B λ K_t E(t) 4B λ K_t. (36) Kernel decay. Let F(t)=α(t)−1KtF(t)=α(t)^-1K_t. For each fixed vector v, equation (36) implies −4Bελ⊤F≤dt(⊤F)≤4Bελ⊤F.-4B λ\, v F v≤ ddt( v F v)≤ 4B λ\, v F v. Integrating these scalar inequalities yields e−h(t)K∗⪯F(t)⪯eh(t)K∗,h(t):=4Bελτ≤4BLε.e^-h(t)K_* F(t) e^h(t)K_*, h(t):=4B λτ≤ 4BL . (37) Consequently, ‖F(t)−K∗‖op≤(eh(t)−1)‖K∗‖op≤C1ε‖K∗‖op, F(t)-K_* _op≤(e^h(t)-1) K_* _op≤ C_1 K_* _op, where C1C_1 depends only on B,LB,L for sufficiently small ε . Multiplication by α(t)α(t) proves equation (14). In particular, the kernel remains positive definite on the interval. Quasi-static scaling. Using the actual kernel just estimated, set qs(t)=−λ(Kt+λIn)−1 e_qs(t)=-λ(K_t+λ I_n)^-1 y. Let A0=K∗+λInA_0=K_*+λ I_n and D(t)=F(t)−K∗+λ(α(t)−1−1)In.D(t)=F(t)-K_*+λ(α(t)^-1-1)I_n. Then Kt+λIn=α(t)(A0+D(t))K_t+λ I_n=α(t)(A_0+D(t)). The inverse-matrix identity gives the exact comparison qs(t)−α(t)−1qs(t0)=−α(t)−1(A0+D(t))−1D(t)qs(t0). e_qs(t)-α(t)^-1 e_qs(t_0)=-α(t)^-1(A_0+D(t))^-1D(t) e_qs(t_0). (38) By equation (37), λmin(A0+D(t))≥e−h(t)κ∗ _ (A_0+D(t))≥e^-h(t) _*. Moreover, using the spectral assumptions and α−1≤e2Lα^-1≤e^2L, ‖D(t)‖opκ∗≤χ(eh(t)−1)+ε(e2L−1)≤C2ε, D(t) _op _*≤χ(e^h(t)-1)+ (e^2L-1)≤ C_2 , where C2C_2 depends only on B,L,χB,L,χ. Taking empirical norms in equation (38) therefore gives ‖qs(t)−α(t)−1qs(t0)‖n≤Cεα(t)−1‖qs(t0)‖n. e_qs(t)-α(t)^-1 e_qs(t_0) _n≤ C α(t)^-1 e_qs(t_0) _n. For sufficiently small ε , this also gives the uniform comparison 12α(t)−1‖qs(t0)‖n≤‖qs(t)‖n≤2α(t)−1‖qs(t0)‖n. 12α(t)^-1 e_qs(t_0) _n≤ e_qs(t) _n≤ 2α(t)^-1 e_qs(t_0) _n. (39) Residual tracking. Set κ¯=e−2L−4BLεκ∗ κ=e^-2L-4BL _*. By equation (37), throughout the interval, λmin(Kt)≥κ¯,‖Kt‖opλmin(Kt)≤e8BLεχ,λκ¯≤Cε. _ (K_t)≥ κ, K_t _op _ (K_t)≤e^8BL χ, λ κ≤ C . From equation (36), ‖K˙t‖op≤(2+4Bε)λ‖Kt‖op K_t _op≤(2+4B )λ K_t _op. Differentiating (Kt+λIn)qs(t)=−λ(K_t+λ I_n) e_qs(t)=-λ y therefore yields ˙qs=−(Kt+λIn)−1K˙tqs,‖˙qs‖n≤‖K˙t‖opλmin(Kt)+λ‖qs‖n≤Cλ‖qs‖n. e_qs=-(K_t+λ I_n)^-1 K_t e_qs, e_qs _n≤ K_t _op _ (K_t)+λ e_qs _n≤ Cλ e_qs _n. Let =−qs z= e- e_qs. Subtracting the moving equilibrium from equation (6) gives ˙=−2(Kt+λIn)−˙qs. z=-2(K_t+λ I_n) z- e_qs. (40) The homogeneous evolution operator satisfies ‖U(t,s)‖op≤e−2(κ¯+λ)(t−s) U(t,s) _op≤e^-2( κ+λ)(t-s). Variation of constants therefore yields ‖(t)‖n≤e−2(κ¯+λ)τ‖(t0)‖n+∫t0te−2(κ¯+λ)(t−s)‖˙qs(s)‖ns. z(t) _n≤e^-2( κ+λ)τ z(t_0) _n+ _t_0^te^-2( κ+λ)(t-s) e_qs(s) _n\,ds. (41) Together with equation (39), this gives ‖(t)‖n z(t) _n ≤e−2(κ¯+λ)τ‖(t0)‖n+Cλ‖qs(t0)‖n∫t0te−2(κ¯+λ)(t−s)e2λ(s−t0)s ≤e^-2( κ+λ)τ z(t_0) _n+Cλ e_qs(t_0) _n _t_0^te^-2( κ+λ)(t-s)e^2λ(s-t_0)\,ds ≤e−2(κ¯+λ)τ‖(t0)‖n+Cλκ¯+2λα(t)−1‖qs(t0)‖n. ≤e^-2( κ+λ)τ z(t_0) _n+ Cλ κ+2λα(t)^-1 e_qs(t_0) _n. Dividing by the lower bound in equation (39) gives ‖(t)‖n‖qs(t)‖n≤2e−2(κ¯+2λ)τδ0+Cλκ¯+2λ≤C(δ0+ε), z(t) _n e_qs(t) _n≤ 2e^-2( κ+2λ)τ _0+ Cλ κ+2λ≤ C( _0+ ), (42) which proves equation (15) uniformly on [t0,t1][t_0,t_1]. Thus the initial mismatch relaxes on the kernel time scale, while motion of the reference on the decay time scale produces only a small lag. Amplification of the actual field. Using the exact decomposition (t)−α(t)−1(t0)=qs(t)−α(t)−1qs(t0)+(t)−α(t)−1(t0), e(t)-α(t)^-1 e(t_0)= e_qs(t)-α(t)^-1 e_qs(t_0)+ z(t)-α(t)^-1 z(t_0), the quasi-static scaling estimate and the preceding bound on z give ‖(t)−α(t)−1(t0)‖n≤C(ε+δ0)α(t)−1‖qs(t0)‖n. e(t)-α(t)^-1 e(t_0) _n≤ C( + _0)α(t)^-1 e_qs(t_0) _n. Cauchy–Schwarz and the definition of CϕC_φ now prove equation (16) for qt=qe(t)q_t=q_e(t). The term δ0 _0 accounts for the initial tracking error; Theorem 2 gives δ0=op(1) _0=o_p(1) for ≠ y≠ 0. ∎ Appendix D Slow drift of extrema and neuron condensation D.1 Amplitude budget and angular energy Under the hypotheses of Theorem 5, (t)N(t) is constant for t≥Tt≥ T; write N for this set. Exact balance and energy dissipation give λ(t)=λ2∑k∈(ak(t)2+rk(t)2)≤Φ(Θ(t))≤Φ(Θ(T)),t≥T. (t)= λ2 _k (a_k(t)^2+r_k(t)^2)≤ ( (t))≤ ( (T)), t≥ T. (43) Hence (t)≤¯TA(t)≤ A_T. D.2 Proof of Theorem 5 Proof. Fix any finite t>Tt>T. On [T,t][T,t], write =(t)N=N(t), which is constant. Under the theorem’s balance assumption, ddt=A+θ,A=∑k∈skdAkdtϕk,θ=−2∑k∈Akpkk. d edt= v_A+ v_θ, v_A= _k s_k dA_kdt φ_k, v_θ=-2 _k A_kp_k ψ_k. Here A,θ∈ℝn v_A, v_θ ^n are output-velocity vectors, whereas AV_A is the scalar sum of absolute amplitude derivatives. To see the first identity, differentiate the entire output sum: ddt=∑k∈sk(dAkdtϕk+Akdθkdtk). d edt= _k s_k( dA_kdt φ_k+A_k d _kdt ψ_k). Strictly dead neurons have zero output throughout this interval. Substituting dθkdt=−2skpk d _kdt=-2s_kp_k cancels the two factors of sks_k and gives θ v_θ. Within a fixed cell, ∂θ()=−ϕθ() _θ ψ_θ( X)=- φ_θ( X), so the full chain rule is dpkdt dp_kdt =⟨ddt,k⟩n+⟨,∂θk()⟩ndθkdt = d edt, ψ_k _n+ e, _θ ψ_ _k( X) _n d _kdt =⟨ddt,k⟩n−qkdθkdt=⟨ddt,k⟩n+2skqkpk. = d edt, ψ_k _n-q_k d _kdt= d edt, ψ_k _n+2s_kq_kp_k. (44) In the derivative of P, the amplitude term −4skAkqkpk2-4s_kA_kq_kp_k^2 therefore cancels the angular chain-rule term +4skAkqkpk2+4s_kA_kq_kp_k^2. Summing gives dPdt dPdt =∑k∈(dAkdtpk2+2Akpkdpkdt) = _k ( dA_kdtp_k^2+2A_kp_k dp_kdt) =−2λP+2⟨ddt,∑k∈Akpkk⟩n =-2λ P+2 d edt, _k A_kp_k ψ_k _n =−2λP−‖θ‖n2−⟨A,θ⟩n =-2λ P- v_θ _n^2- v_A, v_θ _n =−2λP−‖θ+A/2‖n2+14‖A‖n2≤−2λP+14Cϕ2A2. =-2λ P- v_θ+ v_A/2 _n^2+ 14 v_A _n^2≤-2λ P+ 14C_φ^2V_A^2. (45) Here the triangle inequality gives ‖A‖n≤∑k|dAkdt|‖ϕk‖n≤CϕA v_A _n≤ _k | dA_kdt | φ_k _n≤ C_φV_A. Multiplying the differential inequality by e2λ(t−T)e^2λ(t-T) and integrating yields P(t)≤e−2λ(t−T)P(T)+Cϕ24∫Tte−2λ(t−s)A(s)2s,P(t)≤e^-2λ(t-T)P(T)+ C_φ^24 _T^te^-2λ(t-s)V_A(s)^2\,ds, which is equation (18). Weighted Cauchy–Schwarz gives the remaining velocity estimate explicitly: ‖θ‖n v_θ _n ≤2∑kAk|pk|‖k‖n≤2Cψ∑kAkAkpk2 ≤ 2 _kA_k|p_k| ψ_k _n≤ 2C_ψ _k A_k A_kp_k^2 ≤2Cψ(∑kAk)1/2(∑kAkpk2)1/2=2CψP. ≤ 2C_ψ ( _kA_k )^1/2 ( _kA_kp_k^2 )^1/2=2C_ψ AP. (46) Combining ‖ddt‖n≤‖A‖n+‖θ‖n d edt _n≤ v_A _n+ v_θ _n with equation (43) proves equation (19). At a smooth minimum, qt′(θ∗)=R>0q_t ( _*)=R>0 makes its branch locally differentiable by the implicit function theorem. Differentiating pt(θ∗(t))=0p_t( _*(t))=0 yields 0=⟨ddt,θ∗()⟩n+qt′(θ∗)dθ∗dt.0= d edt, ψ_ _*( X) _n+q_t ( _*) d _*dt. Solving for its velocity and applying Cauchy–Schwarz gives dθ∗dt=−⟨ddt,θ∗()⟩nR,|dθ∗dt|≤CψR0‖ddt‖n, d _*dt=- d edt, ψ_ _*( X) _nR, | d _*dt |≤ C_ψR_0 d edt _n, Combining this bound with equation (19) proves equation (20). ∎ D.3 Consequences of small amplitude variation If A(t)≤νAV_A(t)≤ _A for all t≥Tt≥ T, integrating the exponential kernel in equation (18) gives P(t)≤e−2λ(t−T)P(T)+Cϕ2νA28λ(1−e−2λ(t−T))=:HνA(t)2.P(t)≤e^-2λ(t-T)P(T)+ C_φ^2 _A^28λ(1-e^-2λ(t-T))=:H_ _A(t)^2. (47) Combining this with equation (19) and equation (20) gives |dθ∗(t)dt|≤CψR0[CϕνA+2Cψ¯THνA(t)]. | d _*(t)dt |≤ C_ψR_0 [C_φ _A+2C_ψ A_TH_ _A(t) ]. (48) Under the hypotheses of Theorem 5, if A(t)→0V_A(t)→ 0, its energy bound tends to zero. Indeed, for any η>0η>0, choose S≥TS≥ T so that A(s)2≤ηV_A(s)^2≤η for s≥Ss≥ S. For t≥St≥ S the convolution is bounded by e−2λ(t−S)∫TSe−2λ(S−s)A(s)2s+η2λ.e^-2λ(t-S) _T^Se^-2λ(S-s)V_A(s)^2\,ds+ η2λ. Letting t→∞t→∞ and then η↓0η 0 proves P(t)→0P(t)→ 0. The residual and minimum velocities then tend to zero by equation (19) and equation (20). A uniform bound A≤νAV_A≤ _A alone instead gives lim supt→∞P(t)≤Cϕ2νA2/(8λ) _t→∞P(t)≤ C_φ^2 _A^2/(8λ). D.4 Angular tracking Lemma 7 (Angular tracking). Consider an exactly balanced positive-output neuron ak=rk>0a_k=r_k>0 for all t≥Tt≥ T. Let θ∗(t) _*(t) be a differentiable smooth negative-minimum branch with depth R(t)≥R0>0R(t)≥ R_0>0 for all t≥Tt≥ T. Suppose the closed band [θ∗(t)−d,θ∗(t)+d][ _*(t)-d, _*(t)+d], with 0<d<π/20<d<π/2, lies inside one fixed open activation cell for all t≥Tt≥ T. If |δk(T)|≤d,supt≥T|dθ∗(t)dt|≤ω<2R0sind,δk(t)=θk(t)−θ∗(t),| _k(T)|≤ d, _t≥ T | d _*(t)dt |≤ω<2R_0 d, _k(t)= _k(t)- _*(t), then the neuron remains in this band. With cd=2R0cosd>0c_d=2R_0 d>0, |δk(t)| | _k(t)| ≤e−cd(t−T)|δk(T)|+∫Tte−cd(t−s)|dθ∗(s)ds|s ≤e^-c_d(t-T)| _k(T)|+ _T^te^-c_d(t-s) | d _*(s)ds |\,ds (49) ≤e−cd(t−T)|δk(T)|+ωcd(1−e−cd(t−T)). ≤e^-c_d(t-T)| _k(T)|+ ωc_d (1-e^-c_d(t-T) ). (50) Proof. The cell representation qt(θ)=−R(t)cos(θ−θ∗(t))q_t(θ)=-R(t) (θ- _*(t)) gives pt(θ)=R(t)sin(θ−θ∗(t))p_t(θ)=R(t) (θ- _*(t)). For an exactly balanced positive-output neuron, differentiating δk=θk−θ∗ _k= _k- _* therefore gives dδkdt=−2R(t)sinδk−dθ∗dt. d _kdt=-2R(t) _k- d _*dt. At δk=d _k=d the relative velocity is at most −2R0sind+ω<0-2R_0 d+ω<0; at δk=−d _k=-d it is at least 2R0sind−ω>02R_0 d-ω>0. A first-exit argument therefore makes the band forward invariant for all t≥Tt≥ T. For |δ|≤d|δ|≤ d, sgn(δ)sinδ≥cosd|δ|sgn(δ) δ≥ d\,|δ|. The upper right Dini derivative therefore satisfies D+|δk|≤−cd|δk|+|dθ∗dt|,D^+| _k|≤-c_d| _k|+ | d _*dt |, including at δk=0 _k=0. Multiplication by ecd(t−T)e^c_d(t-T) and integration proves equation (49)–equation (50). ∎ D.5 Pairwise angular contraction Recall the angular diameter in equation (9). The following estimate proves equation (23) once Lemma 7 supplies the invariant band. Lemma 8 (Pairwise contraction). Let G be a fixed group of exactly balanced positive-output neurons, ak=rk>0a_k=r_k>0 for all t≥Tt≥ T. Suppose that, for all t≥Tt≥ T, a smooth negative-minimum branch θ∗(t) _*(t) has depth R(t)≥R0>0R(t)≥ R_0>0 and all group directions remain in a common band |θk(t)−θ∗(t)|≤d<π/2| _k(t)- _*(t)|≤ d<π/2 contained in one open activation cell. Use a common continuous angular coordinate on this band. Then for i,j∈Gi,j∈ G and t≥Tt≥ T, |θi(t)−θj(t)|≤e−2R0cos(d)(t−T)|θi(T)−θj(T)|.| _i(t)- _j(t)|≤e^-2R_0 (d)(t-T)| _i(T)- _j(T)|. (51) Consequently, diamθG(t)diam_θG(t) obeys the same contraction bound and tends to zero, regardless of whether the well converges to a fixed angle. Taking the difference of two angular equations gives dt(θi−θj)=−2R(sinδi−sinδj). ddt( _i- _j)=-2R ( _i- _j ). (52) Proof. Set Δij=θi−θj=δi−δj _ij= _i- _j= _i- _j. The divided difference cij(t)=∫01cos(δj(t)+s(δi(t)−δj(t)))sc_ij(t)= _0^1 ( _j(t)+s( _i(t)- _j(t)) )\,ds is well defined even when the two angles coincide. Since both offsets lie in [−d,d][-d,d], it satisfies cij≥cosd>0c_ij≥ d>0. Equation (52) is therefore dΔijdt=−2R(t)cij(t)Δij d _ijdt=-2R(t)c_ij(t) _ij. Integration proves equation (51); taking the maximum over the finitely many pairs gives the diameter bound. ∎ D.6 Proof of Theorem 6 Proof. Set ω:=supt≥T|dθ∗(t)dt|<2R0sind.ω:= _t≥ T | d _*(t)dt |<2R_0 d. Lemma 7, applied to every k∈Gk∈ G, shows that all group directions remain in the stated band. Lemma 8 then gives equation (23). The positive rate cd=2R0cosdc_d=2R_0 d therefore implies that the diameter tends to zero as t→∞t→∞. If additionally dθ∗dt(t)→0 d _*dt(t)→ 0, the tracking estimate equation (49) implies |θk(t)−θ∗(t)|→0| _k(t)- _*(t)|→ 0 for each k∈Gk∈ G. To see this, split its convolution at a fixed late time S. The part over [T,S][T,S] decays exponentially, whereas the part over [S,t][S,t] is at most cd−1sups≥S|dθ∗dt(s)|c_d^-1 _s≥ S| d _*dt(s)|. Letting t→∞t→∞ and then S→∞S→∞ proves the claim. For negative neurons near a positive maximum, qt=Rcosδq_t=R δ gives dθkdt=2pk=−2Rsinδk d _kdt=2p_k=-2R _k, so the same argument applies. ∎ Appendix E Experimental details for the main condensation experiment E.1 Target and training data The four-point experiment in Figures 2, 3, and 5 fits the target f⋆(x)=0.2ReLU(x−1/3)+0.2ReLU(−x−1/3),x∈[−1,1],f_ (x)=0.2ReLU(x-1/3)+0.2ReLU(-x-1/3), x∈[-1,1], (53) using noiseless samples =(−1,−1/3,1/3,1)⊤,=(2/15,0,0,2/15)⊤. X=(-1,-1/3,1/3,1) , y=(2/15,0,0,2/15) . Test MSE is evaluated on 10,000 equally spaced points in [−1,1][-1,1]. E.2 Network and optimization The network in equation (1) has m=200m=200 hidden neurons. All parameters are initialized independently as ak,wk,bk∼(0,m−2γ)a_k,w_k,b_k (0,m^-2γ) with γ=0.4γ=0.4. Training uses 80,000 full-batch SGD updates, learning rate η=0.1η=0.1, and weight decay λ=10−3λ=10^-3: ΘE+1=(1−ηλ)ΘE−η∇Θℒ(ΘE). _E+1=(1-ηλ) _E-η _ L( _E). Figure 5 shows the evolution of the NTK eigenvalues throughout training. The large initialization kernel estimate underlying Theorem 2, established in Appendix B.1, gives K0/m1−2γ→K¯≻0K_0/m^1-2γ→ K 0 in probability. Thus, for γ<1/2γ<1/2 and sufficiently large width, all initial NTK eigenvalues are much larger than the fixed weight decay coefficient λ. Following rapid fitting, their early decline is consistent with Theorem 4, which predicts Kt≈e−2λ(t−TI)KTIK_t≈e^-2λ(t-T_I)K_T_I during early amplitude compression under its stated assumptions. The eigenvalues subsequently stabilize in Stage I and remain above λ throughout the observed trajectory. Figure 5: NTK spectrum under large initialization. (a) Full training trajectory: all four eigenvalues remain above λ=10−3λ=10^-3, decrease during Stage I, and approach plateaus during Stage I. (b) Zoom into the first 1,000 epochs, showing the comparatively small changes in the eigenvalues during initial fitting. The horizontal dotted line indicates λ=10−3λ=10^-3. Appendix F Additional ReLU condensation experiments F.1 Initialization scale and weight decay Figure 6: Fitting speed and final condensation at different initialization scales. (a) Training MSE during the first 1,000 updates. (b–d) Outputs at epoch 80,000, with targets (dotted) and training samples (black dots). (e–g) Final directions and unsigned amplitudes Ak=|ak|rkA_k=|a_k|r_k. Columns compare large initialization with λ=0λ=0 and 10−310^-3, and small initialization with λ=0λ=0. Colored circles show non-dead neurons; gray crosses show neurons inactive on every training input. Figure 6 compares large initialization (γ=0.4γ=0.4, λ=0λ=0 or 10−310^-3) with small initialization (γ=1γ=1, λ=0λ=0) on the four-point target. Both large initialization with weight decay and small initialization without decay learn condensed representations with two dominant angular groups. Their fitting dynamics differ: large initialization reaches a low training loss within a few hundred updates, well before late condensation, whereas small initialization fits more slowly. In the large initialization run with λ=10−3λ=10^-3, all non-dead neurons become condensed, including a low-amplitude group near 90∘90 . Dead neurons remain dispersed while their amplitudes vanish. With small initialization and no decay, some low-amplitude non-dead neurons remain dispersed despite the two dominant groups. Large initialization without decay fits rapidly,but neurons don’t develop a condensed representation. F.2 Targets with three ReLU breakpoints Figure 7: Fitting precedes condensation for the targets in equation (54) (rows A–C). Left: target, training samples, early output, and final output. Middle/right: neurons after rapid fitting/at epoch 80,000, with identical marker sizes and separately scaled amplitude axes. Circles and triangles denote nonnegative and negative output weights. Figure 7 compares a convex target (A) with targets whose peak and trough occur in opposite orders (B, C): fA(x) f_A(x) =0.06ReLU(x+3/4)+0.09ReLU(−x)+0.12ReLU(x−3/4), =0.06ReLU(x+3/4)+0.09ReLU(-x)+0.12ReLU(x-3/4), (54) fB(x) f_B(x) =0.3ReLU(x+3/4)−0.5ReLU(x)+0.4ReLU(x−3/4), =0.3ReLU(x+3/4)-0.5ReLU(x)+0.4ReLU(x-3/4), fC(x) f_C(x) =0.2ReLU(x+3/4)+0.4ReLU(−x)−0.4ReLU(x−3/4), =0.2ReLU(x+3/4)+0.4ReLU(-x)-0.4ReLU(x-3/4), where x∈[−1,1]x∈[-1,1]. After rapid fitting, the outputs already track the targets while neuron directions remain diffuse. Later, neurons with appreciable amplitudes form narrow angular groups in all three cases. The final outputs show a regularization bias. Appendix G Transformer experiment on anchor function composition This section specifies the experiment in Figure 1. We use the anchor function framework of Zhang et al. (2024), with the data split and training settings described below. The comparison consists of two runs with weight decay coefficients 0 and 0.8, identical saved data, and identical initial parameters throughout the network. G.1 Anchor functions and data construction Each input is a sequence of nine integer tokens containing exactly one consecutive anchor pair (a,b)∈1,2,3,42(a,b)∈\1,2,3,4\^2. The token immediately preceding this pair is the key x. The four elementary functions are g1(x)=x+5,g2(x)=x+1,g3(x)=x−2,g4(x)=x−8.g_1(x)=x+5, g_2(x)=x+1, g_3(x)=x-2, g_4(x)=x-8. The target is the integer y=ga(gb(x))=x+da+dby=g_a(g_b(x))=x+d_a+d_b, where (d1,d2,d3,d4)=(5,1,−2,−8)(d_1,d_2,d_3,d_4)=(5,1,-2,-8). These translations commute, so their order does not affect the target. All pairs use this compositional rule, including (3,4)(3,4) and (4,3)(4,3), whose target is x−10x-10. Using zero-based positions, we sample the key position p∈0,…,6p∈\0,…,6\ uniformly and place (x,a,b)(x,a,b) at positions (p,p+1,p+2)(p,p+1,p+2). The other six input tokens are sampled independently and uniformly from 20,…,99\20,…,99\ and do not affect the target. The key is drawn uniformly from the same integer range, subject to xmod9≠pfor training,xmod9=pfor testing.x 9≠ p training, x 9=p testing. The modulus is the input length, 9, as implemented in this experiment. Training contains the 14 anchor pairs other than (3,4)(3,4) and (4,3)(4,3); both excluded pairs are held out entirely. Testing contains all 16 pairs. Thus the test groups for the 14 seen pairs assess generalization to held-out key–position combinations, while the other two groups also assess unseen anchor compositions. No non-compositional target is introduced in this experiment. G.2 Network and initialization The model is a decoder-only Transformer with two layers and one causal self-attention head per layer. Given an input sequence represented by one-hot vectors XinX^in, the input to the Transformer block is H(0)=XinWem+Xpos,H^(0)=X^inW^em+X^pos, (55) where XposX^pos is a learned positional embedding. For layer l, we compute: H~(l) H^(l) =LN(H(l−1)), =LN (H^(l-1) ), (56) Q(l) Q^(l) =H~(l)WQ(l), = H^(l)W_Q^(l), K(l) K^(l) =H~(l)WK(l), = H^(l)W_K^(l), V(l) V^(l) =H~(l)WV(l). = H^(l)W_V^(l). The causal attention matrix and the output of the attention layer are Attn(l) Attn^(l) =softmax(Q(l)K(l)⊤dk)(with mask), =softmax ( Q^(l)K^(l) d_k )(with mask), (57) U(l) U^(l) =H(l−1)+Attn(l)V(l)WO(l), =H^(l-1)+Attn^(l)V^(l)W_O^(l), The feedforward layer is computed as: U~(l) U^(l) =LN(l)(U(l)), =LN^(l) (U^(l) ), (58) H(l) H^(l) =U~(l)+ReLU(U~(l)W1(l))W2(l). = U^(l)+ReLU\! ( U^(l)W_1^(l) )W_2^(l). The final logits are Z=H(N)Wout+bout.Z=H^(N)W^out+b^out. (59) LN refers to Layer Normalization. Each linear weight matrix W∈ℝdin×doutW ^d_in× d_out is initialized entrywise as Wij∼(0,din−2γ)W_ij \! (0,d_in^-2γ ) (60) Table 1: Settings shared by the two Transformer runs in Figure 1; only weight decay differs. Setting Value Layers / heads per layer 2 / 1 Embedding / feedforward dimension 400 / 1,200 Query, key, and value dimension 64 Input length / vocabulary size 9 / 120 Initialization rate 0.48 Optimizer AdamW Batch size / training epochs 4,096 / 400 Learning rate: initial / peak / minimum 5×10−55× 10^-5 / 5×10−45× 10^-4 / 5×10−55× 10^-5 Warmup / cosine schedule 40 / 360 epochs Global gradient-norm clipping 1.0 G.3 Training objective and weight decay comparison Training minimizes cross entropy between the logits at the last output position and the appended target y. AdamW applies decoupled weight decay to all trainable parameters, including embeddings, LayerNorm parameters, and the output bias; there are no parameter groups exempted from decay. The learning rate warms up from 5×10−55× 10^-5 to 5×10−45× 10^-4 over 40 epochs, then follows a cosine schedule with period parameter 360 epochs and minimum 5×10−55× 10^-5. G.4 Cosine similarities Figure 8: Weight contraction and condensation across Transformer layers. The top and bottom rows show layers 1 and 2, respectively. Panel (a) corresponds to training without weight decay (λ=0λ=0), and Panel (b) corresponds to training with weight decay (λ=0.8λ=0.8). Within each panel, the columns show WQW_Q and WKW_K. Curves show Frobenius norms during training, while the insets show row-wise cosine similarities at epochs 50 and 300. Weight decay induces norm contraction and structured directional alignment in both layers. In our experiment, WQ,WK∈ℝ64×400W_Q,W_K ^64× 400. For rows i,j∈ℝ400 w_i, w_j ^400, define cosine similarity Cij=i⊤j‖i‖2‖j‖2.C_ij= w_i w_j\| w_i\|_2\,\| w_j\|_2. Rows and columns are reordered by the same permutation and grouped according to cosine similarity. Figure 1 uses the query and key projection matrices from the first Transformer blocks. Figure 8 shows result on both Transformer layers.