Paper deep dive
Critical initialization destabilizes higher input derivatives in wide scalar-input networks
Prashant Singh, Pranav Singh
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/10/2026, 2:32:29 AM
Summary
This paper analyzes the propagation of input derivatives in wide neural networks at initialization. It demonstrates that while the 'edge-of-chaos' condition preserves first-order input perturbations, it causes second and higher-order input derivative variances to grow unboundedly in fully connected networks. Conversely, residual networks with branch scaling of $L^{-1/2}$ maintain uniformly bounded variance for all finite derivative orders. The authors derive mean-field recursions for the jet covariance up to third order and validate these theoretical findings with simulations.
Entities (6)
Relation Signals (5)
Edge-of-chaos → destabilizes → Higher input derivatives
confidence 95% · At criticality... the second-derivative variance grows linearly whenever the activation has nonzero curvature.
Edge-of-chaos → preserves → First-order input perturbations
confidence 95% · The edge-of-chaos condition preserves first-order input perturbations in wide randomly initialized networks
Residual Networks → stabilizes → Higher input derivatives
confidence 95% · For residual networks with branch scale L^{-1/2}, we prove that every fixed finite derivative order has uniformly bounded variance
Mean-Field Theory → derives → Jet Covariance Recursions
confidence 92% · we derive mean-field recursions through third order that are exact at the variance fixed point
Physics-Informed Neural Networks → requires → Higher input derivatives
confidence 90% · physics-informed losses, score matching and derivative regularization depend on higher input derivatives.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The edge-of-chaos condition preserves first-order input perturbations in wide randomly initialized networks, but physics-informed losses, score matching and derivative regularization depend on higher input derivatives. For smooth scalar-input fully connected networks, using a joint Gaussianity of the finite derivative jet that holds in the infinite-width limit at each fixed depth, we derive mean-field recursions through third order that are exact at the variance fixed point, with finite-depth corrections that decay geometrically. At criticality, the first-derivative variance is depth-invariant, whereas the second-derivative variance grows linearly whenever the activation has nonzero curvature. The resulting third-order system closes on mean-field susceptibilities. For residual networks with branch scale L^{-1/2}, we prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions. Simulations verify the critical growth laws, the residual bound, and the closed recursion. The results concern initialization, not trained-network performance.
Tags
Links
- Source: https://arxiv.org/abs/2609.09244v1
- Canonical: https://arxiv.org/abs/2609.09244v1
Trouble viewing inline? Open PDF directly →
Full Text
80,979 characters extracted from source content.
Expand or collapse full text
Critical initialization destabilizes higher input derivatives in wide scalar-input networks Journal: Neural Networks Prashant Singh Email: 2023mcb1309@iitrpr.ac.in Affiliation: Department of Mathematics, Indian Institute of Technology Ropar, Rupnagar, Punjab, 140001, India Corresponding author: Corresponding author. Note: Both authors contributed equally to this work. Pranav Singh Email: 2023mcb1308@iitrpr.ac.in Affiliation: Department of Mathematics, Indian Institute of Technology Ropar, Rupnagar, Punjab, 140001, India Note: Both authors contributed equally to this work. Abstract The edge-of-chaos condition preserves first-order input perturbations in wide randomly initialized networks, but physics-informed losses, score matching and derivative regularization depend on higher input derivatives. For smooth scalar-input fully connected networks, using a joint Gaussianity of the finite derivative jet that holds in the infinite-width limit at each fixed depth, we derive mean-field recursions through third order that are exact at the variance fixed point, with finite-depth corrections that decay geometrically. At criticality, the first-derivative variance is depth-invariant, whereas the second-derivative variance grows linearly whenever the activation has nonzero curvature. The resulting third-order system closes on mean-field susceptibilities. For residual networks with branch scale L−1/2L^-1/2, we prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions. Simulations verify the critical growth laws, the residual bound, and the closed recursion. The results concern initialization, not trained-network performance. Keywords: deep neural networks , initialization , mean-field theory , edge of chaos , signal propagation , physics-informed neural networks 1 Introduction Physics-informed neural networks minimize a differential-equation residual and therefore depend on input derivatives of the network [11]. The same is true of implicit neural representations regularized by their gradients [14], and of explicit score matching [6], whose objective contains the Laplacian of the model log-density and so depends on second input derivatives. Denoising and sliced variants, and the score parameterization of Song and Ermon [15], model the score directly and require only first-order derivatives; by the results below, that is the order which is structurally well behaved. Derivative conditioning in these settings is not determined by output variance alone. These networks do not improve with depth in the way ordinary feedforward networks do. Accuracy degrades as layers are added, and Wang et al. [16] attribute the degradation to initialization rather than to optimization, reporting that standard schemes leave the network’s derivatives poorly conditioned before training begins. Their remedy is architectural: a residual connection whose branches are attenuated so that the network begins close to shallow and deepens as training proceeds. Mean-field initialization theory identifies the edge of chaos, where the first-order susceptibility equals one and first-order input perturbations are preserved through depth [10, 13]. This criterion does not specify the propagation of second and higher input derivatives. That gap matters for derivative-dependent objectives and motivates the present analysis. For smooth fully connected networks with one input, we propagate the covariance of the input-derivative jet at initialization. Under a joint-Gaussian jet assumption in the infinite-width limit, we obtain a closed recursion through third order. The recursion shows that criticality keeps the first-derivative variance constant but makes the second-derivative variance grow linearly whenever the activation has nonzero curvature. It also gives quadratic growth at third order. For residual layers with branch scale L−1/2L^-1/2, we prove uniform boundedness at every fixed finite derivative order under bounded-derivative assumptions. This is complementary to the construction of Wang et al. [16] rather than an account of it. Their residual scalar is trainable and initialized to zero, so the network is an exact identity map at initialization and recovers depth during training; the scaling analyzed here is fixed and does not change. What the two share is that both attenuate the residual branch at initialization, and the analysis below gives a mean-field account of why attenuation at that point is what the derivative variances require. Two boundaries are worth stating at the outset. Everything here concerns the network at initialization, before any training step; the connection to trained accuracy runs through conditioning at initialization, which is the mechanism Wang et al. [18] analyze for physics-informed losses, and we do not extend it. And the derivation assumes a smooth activation, which the results of Section 4 show is not incidental. The analysis establishes no ranking of optimization methods, and its direct conclusion is architectural: independent layerwise initialization cannot simultaneously preserve a nonzero second input derivative and stabilize its variance at criticality, whereas depth-scaled residual branches avoid this variance growth. We validate the recursions and residual prediction with finite-width simulations. 2 Related work Mean-field theory of initialization. The framework this paper builds on originates with Poole et al. [10] and Schoenholz et al. [13], who showed that a randomly initialized deep network admits a deterministic description in the wide limit: the preactivation variance obeys a one-dimensional map with a fixed point, and the correlation between two inputs obeys a second map whose stability determines how far information propagates. The critical setting of the weight and bias variances is where the derivative of that second map equals one, and initializing there maximizes trainable depth. Yang and Schoenholz [20] extended the analysis to residual architectures, showing that skip connections widen the critical region, and Hayou et al. [5] used it to compare activation functions. These derivations are complete for the setting they address. A network trained on a loss that depends on its output requires the output and its first derivative with respect to the input, the latter because it controls how gradients traverse depth, and both are what the framework follows. The present paper extends the same machinery to the higher input derivatives that a different class of loss requires. The object it propagates, the jet covariance of (7), contains the two quantities above as its first entries. Physics-informed networks and their training behavior. Raissi et al. [11] introduced the formulation in which a differential-equation residual serves as the training objective. Wang et al. [17] identify pathologies in the resulting gradient flow, and Wang et al. [18] analyze the optimization through the neural tangent kernel [7], showing that the residual and boundary components of the kernel can differ by orders of magnitude and that this imbalance explains several observed training failures. That analysis concerns the kernel at initialization and its consequences for convergence; the present paper concerns the variances of the network’s input derivatives at initialization, which is the quantity entering that kernel rather than the kernel itself. Wong et al. [19] likewise locate a training failure at initialization rather than in the optimizer. They report that a network of increasing expressiveness is biased towards flat output functions at initialization, which can come close to satisfying a differential-equation residual while remaining far from the solution, and their remedy raises the variability of the input gradient by mapping the input into a sinusoidal space. That remedy acts on the input encoding, and the derivative order it addresses is the first. The recursion analyzed here acts through depth and concerns orders two and above, so the two accounts bear on different orders of the same jet. Wang et al. [16] report that physics-informed networks degrade as depth increases and attribute this to initialization. Their remedy is a residual connection carrying a trainable scalar initialized to zero, so that each block is an exact identity map at initialization and the network recovers depth during training, together with a least-squares initialization of the final layer. This is mechanistically distinct from the fixed depth-dependent attenuation analyzed in Section 4.8, which never changes. The two agree in attenuating the residual branch at initialization and differ in whether that attenuation is subsequently learned. Residual branch scaling. The exponent γ=1/2γ=1/2 is not new, and its properties at orders zero and one are established. Hayou et al. [4] show that scaling each residual branch by L−1/2L^-1/2 keeps the output variance bounded where an unscaled network gives variance growing linearly in depth. Zhang et al. [22] prove the threshold is sharp: the forward process is unbounded once the branch scale exceeds L−1/2+cL^-1/2+c for any c>0c>0. Marion et al. [8] show that L−1/2L^-1/2 is the distinguished exponent in the large-depth limit under independent initialization, other choices giving either explosion or an identity map. Architecturally, Bachlechner et al. [1] obtain a related effect with a trainable residual scalar initialized to zero, which is the construction Wang et al. [16] adopt. What Section 4.8 adds is not the exponent but its reach: the same single choice γ=1/2γ=1/2 bounds every input-derivative order simultaneously, by an argument in which the order of the derivative does not appear. Adjacent questions about higher derivatives. Two lines of work involve higher derivatives of neural networks and are worth distinguishing by what each sets out to do. Chickering [2] addresses the cost of evaluating higher-order derivatives of a deep network, giving an algorithm quasilinear in the order. That is the computational question: how expensive the derivatives are to obtain. The question here is a statistical one, namely how their variances behave as depth grows, and the two are independent. A practitioner needs both. Martens et al. [9] shape the kernel map of a deep network by modifying activation functions, and their analysis differentiates that map with respect to the correlation between two inputs. The derivatives they take are of the kernel with respect to its argument; the derivatives here are of the network with respect to its input. Both are higher-order quantities in a mean-field analysis, and they are different objects, so the results do not overlap. Correlator hierarchies through depth. Hanin [3] develops a perturbative hierarchy for the joint cumulants of a random network and its derivatives, and Roberts et al. [12] propagate preactivation correlators together with the neural tangent kernel and its parameter-space derivatives, with finite-width corrections organized in powers of 1/n1/n. Both are hierarchies of correlators through depth, and the machinery is close to what is used here. The scope of the first is worth stating precisely, because it is broader than the recursions drawn from it. Its cumulant estimates are stated for differential operators in the input of any order the smoothness of the activation permits, so higher input derivatives lie inside that framework rather than outside it. What the estimates supply for such an operator is its order of magnitude in 1/n1/n at fixed depth, with the layer index carried in the implicit constant. The explicit layerwise recursions derived from them cover the network value and its first input derivatives, and the large-depth conclusion drawn from them concerns gradients with respect to first-layer weights. Neither line of work takes the depth dependence of a higher input derivative’s variance as its object, and that is the quantity propagated here: the covariance of ∂xkhℓ _x^kh for k up to three, at fixed width. To our knowledge, no prior work isolates the input-derivative correlators ∂xkhℓ _x^kh with k≥2k≥ 2 as the propagated object and derives their depth-growth law at criticality, and we have found no prior notice of the parity failure described in Section 6.1. Yang and Schoenholz [20] establish polynomial depth behavior at criticality for activations and gradients, and the growth law below is in that spirit, specialized to input derivatives. The search behind these statements covered the mean-field and neural-tangent-kernel literatures and forward citations of the works above; it was not exhaustive, which is why the claims are stated in this form. 3 Setup A fully connected network of width n and depth L maps a scalar input x to preactivations hℓ∈ℝnh ^n through hℓ=Wℓϕ(hℓ−1)+bℓ,Wijℓ∼(0,σw2/n),biℓ∼(0,σb2),h =W φ\! (h -1 )+b , W _ij \! (0,\ _w^2/n ), b _i \! (0,\ _b^2 ), (1) with the activation ϕφ applied componentwise and all weights and biases drawn independently at initialization. The first layer has fan-in equal to the input dimension rather than n, so Wi1∼(0,σw2)W^1_i (0, _w^2). Throughout, ϕ=tanhφ= unless stated otherwise, and every statement refers to the network at initialization, before any training step. The analytical statements below concern fixed finite depth and fixed finite derivative order as n→∞n→∞. 3.1 Mean-field variance propagation In the limit n→∞n→∞ the preactivations at each layer become Gaussian with zero mean, and their variance qℓ=[(hiℓ)2]q =E[(h _i)^2] obeys a deterministic recursion [10, 13] qℓ=σw2z∼(0,qℓ−1)[ϕ(z)2]+σb2,q = _w^2\,E_z (0,q -1)\! [φ(z)^2 ]+ _b^2, (2) which has a fixed point q∗q . We call the quantities χm:=σw2z∼(0,q∗)[ϕ(m)(z)2],m≥1, _m:= _w^2\,E_z (0,q )\! [φ^(m)(z)^2 ], m≥ 1, (3) the order-m susceptibilities. Only χ1 _1 appears in the classical theory. It governs how a perturbation of the input propagates: the variance of the input-output Jacobian is multiplied by χ1 _1 at each layer, so it grows without bound when χ1>1 _1>1, decays to zero when χ1<1 _1<1, and is preserved when χ1=1 _1=1. The condition χ1=1 _1=1 (4) defines the edge of chaos, the standard prescription for choosing (σw2,σb2)( _w^2, _b^2) so that first-order perturbations neither vanish nor explode. It is also closely related to stable backward-signal propagation in the classical mean-field analysis. For tanh with σb2=0.05 _b^2=0.05 this gives σw2=1.760955 _w^2=1.760955, q∗=0.570048q =0.570048, and χ2=0.568531 _2=0.568531, χ3=2.262727 _3=2.262727. 3.2 Jets and the jet covariance The quantity a loss actually sees is the scalar network output. We take the standard independent linear readout u(x):=∑i=1naiϕ(hiL(x)),ai∼(0,σa2/n)independent of all Wℓ,bℓ.u(x):= _i=1^na_i\,φ\! (h^L_i(x) ), a_i \! (0,\ _a^2/n )\ independent of all W ,b . (5) A loss that constrains a differential operator of order k depends on ∂xku _x^ku, not on u alone. The objects that must propagate are therefore the input derivatives of the preactivations. We write hℓ(j):=∂xjhℓ∈ℝn,j=0,1,…,K,h _(j):= _x^jh ^n, j=0,1,…,K, (6) and call the collection (h(0)ℓ,…,h(K)ℓ) (h _(0),…,h _(K) ) the K-jet of the layer. Index j=0j=0 is the preactivation itself. The quantity this paper tracks is the jet covariance Σjkℓ:=[h(j),iℓh(k),iℓ],j,k∈0,1,…,K, _jk:=E\! [h _(j),i\,h _(k),i ], j,k∈\0,1,…,K\, (7) whose diagonal entries vkℓ:=Σkkℓv _k:= _k are the derivative variances and whose off-diagonal entries record the correlations between derivative orders. We abbreviate cjk:=Σjkc_jk:= _jk for j,k≥1j,k≥ 1 and σ0k:=Σ0k _0k:= _0k. Including j=0j=0 is not a matter of convenience. The recursion for Σ contains terms of the form [g(h(0))h(i)h(j)h(k)]E[g(h_(0))\,h_(i)h_(j)h_(k)] in which a function of the preactivation multiplies a product of jets, and the couplings σ0k _0k are what prevent those terms from vanishing. Section 4 shows that omitting row and column 00 makes several such terms evaluate to zero when they are not. The layer map (1) propagates the jet by the chain rule. Writing g(j)ℓ−1:=∂xj[ϕ(hℓ−1)]g -1_(j):= _x^j\! [φ(h -1) ], we have h(j)ℓ=Wℓg(j)ℓ−1h _(j)=W g -1_(j) for j≥1j≥ 1, with the bias contributing only at j=0j=0, and the g(j)g_(j) follow from Faà di Bruno’s formula. For j≤3j≤ 3, g(0) g_(0) =ϕ(h(0)), =φ(h_(0)), g(1) g_(1) =ϕ′h(1), =φ \,h_(1), g(2) g_(2) =ϕ′h(1)2+ϕ′h(2), =φ \,h_(1)^2+φ \,h_(2), (8) g(3) g_(3) =ϕ′h(1)3+3ϕ′h(1)h(2)+ϕ′h(3), =φ \,h_(1)^3+3φ \,h_(1)h_(2)+φ \,h_(3), with every ϕ(m)φ^(m) evaluated at h(0)h_(0) and all products taken componentwise. Since WℓW is independent of layer ℓ−1 -1 and has independent zero-mean entries of variance σw2/n _w^2/n, Σjkℓ=σw2[g(j)ℓ−1g(k)ℓ−1](j,k≥1), _jk= _w^2\,E\! [g -1_(j)\,g -1_(k) ] (j,k≥ 1), (9) exactly as an expectation over the random network, for any width. The closed recursions derived below additionally use the mean-field and Gaussian-jet assumptions. Equation (9) is the object of Section 4: everything reduces to evaluating expectations of products of the Faà di Bruno terms (8). 3.3 Assumptions The closed recursions use one additional mean-field hypothesis. (A1) The jet (h(0)ℓ,…,h(K)ℓ) (h _(0),…,h _(K) ) is jointly Gaussian at initialization. A1 is the jet analogue of the standard mean-field statement, which asserts Gaussianity for j=0j=0 alone. It is not implied by that statement, since h(2)h_(2) and h(3)h_(3) are built from products through (8). It is, however, available in the literature rather than being a hypothesis this paper must postulate. Theorem 1 (A1 holds in the infinite-width limit at each fixed depth). Fix K and a depth ℓ , and let ϕφ be K times differentiable with polynomially bounded KKth derivative. Then as the hidden widths tend to infinity with ℓ fixed, the finite-dimensional distributions of x↦hℓ(x)x h (x) together with its input derivatives through order K converge to those of a centered Gaussian process with independent components. In particular A1 holds in that limit. Proof. This is Theorem 2.2 of Hanin [3] specialised to n0=1n_0=1 and r=Kr=K. The hypothesis is satisfied by tanh , whose derivatives of every order are bounded. Related Gaussian-process limits for fixed collections of network quantities are obtained by other routes in [21]. ∎ Two qualifications should be read with Theorem 1. The limit is taken at fixed depth as the width grows, so it licenses A1 layer by layer and does not by itself provide a bound uniform in ℓ ; the recursions below are applied at each depth in that limit, which is the same regime in which the mean-field variance map itself is derived. And the convergence is in finite-dimensional distributions, which is what the expectations in (9) require and no more. Section 7.1 reports the measured skewness and excess kurtosis of the jet at the widths actually used, which is the finite-width question that a limit theorem does not answer. The derivation does not assume that activation factors are independent of the jet factors they multiply. Under A1, the required expectations are evaluated by Gaussian integration by parts and the fixed-point identities in Proposition 1. That independence is a separate statement. It is named here because it is the natural first thing to try, because Section 6.1 turns on precisely where it fails, and because Section 7.1 reports where it holds. (A2) For an activation factor A, meaning a product of derivatives of ϕφ evaluated at h(0)ℓh _(0), and a product J of jet components, [AJ]=[A][J]E[A\,J]=E[A]\,E[J]. A2 is not an assumption of this paper’s results. No statement in Section 4 uses it, and it is listed only so that the two sections which examine it have something to refer to. 4 Theory Every expectation appearing in (9) has the form [g(h(0))h(i1)⋯h(ir)]E\! [g(h_(0))\,h_(i_1)·s h_(i_r) ], a function of the preactivation multiplied by a product of jet components. Under A1 these are computable in closed form. Two facts do the work: Gaussian integration by parts, and a set of identities that pin the couplings σ0k _0k at the variance fixed point. 4.1 Two tools Lemma 1 (Gaussian integration by parts). Let X∼(0,q)X (0,q) and let g be differentiable with |g′(X)|<∞E|g (X)|<∞. Then [Xg(X)]=q[g′(X)]E[Xg(X)]=q\,E[g (X)] and, applying this twice, [X2g(X)]=q[g(X)]+q2[g′(X)].E\! [X^2g(X) ]=q\,E[g(X)]+q^2\,E[g (X)]. (10) The second tool is a set of identities relating the couplings σ0k _0k to the other entries of the jet covariance. They are exact at every depth, and they are stated that way here rather than at the fixed point, because the variance map approaches its fixed point without reaching it at any finite depth. Proposition 1 (Variance identities, exact at every depth). Write qℓ(x):=[(hiℓ(x))2]q_ (x):=E[(h _i(x))^2] for the preactivation variance at layer ℓ , and qℓ′,qℓ′,qℓ′q_ ,q_ ,q_ for its derivatives in x. Then for every ℓ σ01ℓ=12qℓ′,σ02ℓ=−v1ℓ+12qℓ′,σ03ℓ=−3c12ℓ+12qℓ′. _01 = 12q_ , _02 =-v_1 + 12q_ , _03 =-3c_12 + 12q_ . (11) In particular, wherever qℓq_ is constant in x, σ01=0,σ02=−v1,σ03=−3c12. _01=0, _02=-v_1, _03=-3c_12. (12) Proof. Differentiate qℓ=[h(0)2]q_ =E[h_(0)^2] in x, exchanging differentiation and expectation, which is permitted because the jet has finite second moments and ϕφ has bounded derivatives through the order used. The first derivative gives qℓ′=2[h(0)h(1)]q_ =2E[h_(0)h_(1)]. The second gives qℓ′=2[h(1)2]+2[h(0)h(2)]q_ =2E[h_(1)^2]+2E[h_(0)h_(2)]. The third gives qℓ′=6[h(1)h(2)]+2[h(0)h(3)]q_ =6E[h_(1)h_(2)]+2E[h_(0)h_(3)]. Rearranging each gives (11), and setting the derivatives of qℓq_ to zero gives (12). ∎ The identities (12) are therefore not available at finite depth as stated. What makes them usable is that the discrepancy decays geometrically, at a rate fixed by the activation and the variance map alone. Lemma 2 (The transient is geometric). Let V(q):=σw2z∼(0,q)[ϕ(z)2]+σb2V(q):= _w^2\,E_z (0,q)[φ(z)^2]+ _b^2 be the variance map, so that qℓ=V(qℓ−1)q_ =V(q_ -1), and let q∗q be an attracting fixed point of V with λ:=|V′(q∗)|<1λ:=|V (q )|<1. Fix λ<λ¯<1λ< λ<1. Then for every x in a neighbourhood of which q1q_1 is smooth, there are constants CmC_m with |qℓ(m)(x)|≤Cmλ¯ℓ,m=1,2,3,|qℓ−q∗|≤C0λ¯ℓ. |q_ ^(m)(x) |≤ C_m\, λ^\, , m=1,2,3, |q_ -q |≤ C_0\, λ^\, . (13) Proof. Since q∗q is attracting, qℓ→q∗q_ → q and |qℓ−q∗|≤C0λ¯ℓ|q_ -q |≤ C_0 λ by the standard linearization estimate for a smooth map at an attracting fixed point. The chain rule gives qℓ′=V′(qℓ−1)qℓ−1′q_ =V (q_ -1)\,q_ -1 , so qℓ′=q1′∏m=1ℓ−1V′(qm)q_ =q_1 _m=1 -1V (q_m); each factor is at most λ¯ λ in modulus for ℓ beyond a finite index, which gives the case m=1m=1. Differentiating again, qℓ′=V′(qℓ−1)(qℓ−1′)2+V′(qℓ−1)qℓ−1′q_ =V (q_ -1)(q_ -1 )^2+V (q_ -1)q_ -1 , an affine recursion in qℓ′q_ with multiplier bounded by λ¯ λ and forcing O(λ¯2ℓ)O( λ^2 ); summing the resulting geometric series gives the case m=2m=2. The case m=3m=3 follows in the same way, its forcing being O(λ¯2ℓ)O( λ^2 ) again. ∎ The correlation-map slope χ1 _1 and the variance-map slope λ are distinct: χ1 _1 governs first-order signal propagation, whereas λ governs the transient in the variance map. Corollary 1. Under A1, write h(k)=βkℓh(0)+rkh_(k)= _k h_(0)+r_k with βkℓ:=σ0kℓ/qℓ _k := _0k /q_ . The residual rkr_k is uncorrelated with h(0)h_(0) by construction, so under A1 it is independent of it, and h(0)h_(0) is independent of the whole collection (r1,r2,r3,…) (r_1,r_2,r_3,… ). This decomposition is exact at every depth. By (11), β1ℓ=qℓ′/(2qℓ) _1 =q_ /(2q_ ), which vanishes wherever qℓq_ is constant in x and is O(λ¯ℓ)O( λ ) in general by Lemma 2; in that case h(1)=r1h_(1)=r_1 is independent of h(0)h_(0). Corollary 1 is what makes the expectations tractable. Any [g(h(0))h(i1)⋯h(ir)]E[g(h_(0))\,h_(i_1)·s h_(i_r)] is expanded by substituting h(k)=βkh(0)+rkh_(k)= _kh_(0)+r_k, after which every term factorizes into a moment of g against a power of h(0)h_(0), evaluated by Lemma 1, times a joint moment of (h(1),r2,r3) (h_(1),r_2,r_3 ), evaluated by Isserlis. No independence assumption beyond A1 is used. 4.2 Orders one and two The order-1 recursion is classical: g(1)=ϕ′h(1)g_(1)=φ h_(1) with h(1)h_(1) independent of h(0)h_(0) by Corollary 1, so (9) gives v1ℓ=χ1v1ℓ−1.v_1 = _1\,v_1 -1. (14) Condition (4) makes v1v_1 independent of depth, which is the entire content of the edge-of-chaos criterion. Proposition 2 (Order two). Under A1, wherever the preactivation variance is constant in x, v2ℓ=χ1v2ℓ−1+3χ2(v1ℓ−1)2v_2 = _1\,v_2 -1+3 _2 (v_1 -1 )^2 (15) holds in the mean-field limit with no correction terms: the two non-vanishing corrections cancel identically rather than approximately. At finite depth the same evaluation gives v2ℓ=χ1(ℓ)v2ℓ−1+3χ2(ℓ)(v1ℓ−1)2+E2ℓ,|E2ℓ|≤Cλ¯ℓ(1+v2ℓ−1),v_2 = _1^( )\,v_2 -1+3 _2^( ) (v_1 -1 )^2+E_2 , |E_2 |≤ C λ^\, (1+v_2 -1 ), (16) where χ1(ℓ),χ2(ℓ) _1^( ), _2^( ) are the susceptibilities evaluated at qℓ−1q_ -1 rather than q∗q and satisfy χ1(ℓ)=χ1+O(λ¯ℓ) _1^( )= _1+O( λ ), χ2(ℓ)=χ2+O(λ¯ℓ) _2^( )= _2+O( λ ). Proof. From (8), g(2)=ϕ′h(1)2+ϕ′h(2)g_(2)=φ h_(1)^2+φ h_(2), so v2ℓ=σw2(TA+TB+TC)v_2 = _w^2 (T_A+T_B+T_C ) with TA=[ϕ′2h(1)4]T_A=E[φ 2h_(1)^4], TB=2[ϕ′ϕ′h(1)2h(2)]T_B=2E[φ φ h_(1)^2h_(2)] and TC=[ϕ′2h(2)2]T_C=E[φ 2h_(2)^2]. For TAT_A, Corollary 1 gives h(1)⟂h(0)h_(1) h_(0), so TA=[ϕ′2][h(1)4]=3[ϕ′2]v12T_A=E[φ 2]\,E[h_(1)^4]=3E[φ 2]v_1^2, contributing 3χ2v123 _2v_1^2. This step is exact and does not use activation-jet factorization. For TBT_B, substitute h(2)=β2h(0)+r2h_(2)= _2h_(0)+r_2. The r2r_2 term carries [ϕ′ϕ′]E[φ φ ], which vanishes because ϕ′ϕ′φ φ is odd for odd ϕφ, and the h(0)h_(0) term factorizes by Corollary 1: TB=2β2[ϕ′ϕ′h(0)]v1T_B=2 _2\,E[φ φ h_(0)]\,v_1. Since ϕ′ϕ′=12(ϕ′2)′φ φ = 12 (φ 2 ) , Lemma 1 gives [ϕ′ϕ′h(0)]=q∗2[(ϕ′2)′]E[φ φ h_(0)]= q 2E[(φ 2) ], and with β2=σ02/q∗=−v1/q∗ _2= _02/q =-v_1/q from (12), σw2TB=−σw2v12[(ϕ′2)′]. _w^2\,T_B=- _w^2\,v_1^2\,E\! [(φ 2) ]. (17) For TCT_C, the same substitution and (10) give TC=[ϕ′2]v2+β22Cov(ϕ′2,h(0)2)T_C=E[φ 2]v_2+ _2^2Cov\! (φ 2,h_(0)^2 ) with Cov(ϕ′2,h(0)2)=q∗2[(ϕ′2)′]Cov(φ 2,h_(0)^2)=q 2E[(φ 2) ], so σw2TC=χ1v2+σw2v12[(ϕ′2)′]. _w^2\,T_C= _1\,v_2+ _w^2\,v_1^2\,E\! [(φ 2) ]. (18) Equations (17) and (18) are equal and opposite, and (15) follows. For (16), repeat the evaluation with qℓ−1q_ -1 in place of q∗q and βkℓ _k in place of its fixed-point value. Three changes appear. The susceptibilities are Gaussian expectations against (0,qℓ−1)N(0,q_ -1), and q↦σw2(0,q)[ϕ(m)2]q _w^2E_N(0,q)[φ^(m)2] is smooth, so they differ from χm _m by O(|qℓ−1−q∗|)=O(λ¯ℓ)O(|q_ -1-q |)=O( λ ). The term TAT_A acquires a contribution through β1ℓ=O(λ¯ℓ) _1 =O( λ ), since h(1)h_(1) is no longer exactly independent of h(0)h_(0). And β2ℓ _2 differs from −v1/q∗-v_1/q by qℓ′/(2qℓ)=O(λ¯ℓ)q_ /(2q_ )=O( λ ), so the cancellation between (17) and (18) is exact up to that order. Each contribution is bounded by a constant times λ¯ℓ λ , multiplied by the jet moments it accompanies, which gives the stated bound. ∎ 4.3 Order three Proposition 3 (Order three). Under A1, v3ℓ=χ1v3ℓ−1+15χ3v13+9χ2(v1v2+2c122)−6χ2v1c13,v_3 = _1\,v_3 -1+15 _3v_1^3+9 _2\! (v_1v_2+2c_12^2 )-6 _2\,v_1c_13, (19) with all right-hand quantities at layer ℓ−1 -1. Proof sketch. Squaring g(3)=ϕ′h(1)3+3ϕ′h(1)h(2)+ϕ′h(3)g_(3)=φ h_(1)^3+3φ h_(1)h_(2)+φ h_(3) gives six terms. Each is evaluated by the substitution of Corollary 1 together with Lemma 1 and Isserlis. Writing κ13:=σw2[ϕ′ϕ′] _13:= _w^2E[φ φ ] and using (ϕ′2)′=2(ϕ′2+ϕ′ϕ′)(φ 2) =2(φ 2+φ φ ) and (ϕ′2)′=2(ϕ′2+ϕ′ϕ′)(φ 2) =2(φ 2+φ φ ), the six contributions below. Each line is multiplied by σw2 _w^2: ϕ′2h(1)6 φ 2h_(1)^6 :15χ3v13, : 15 _3v_1^3, 9ϕ′2h(1)2h(2)2 9φ 2h_(1)^2h_(2)^2 :9χ2(v1v2+2c122)+9σw2v13[(ϕ′2)′], : 9 _2 (v_1v_2+2c_12^2 )+9 _w^2v_1^3E\! [(φ 2) ], ϕ′2h(3)2 φ 2h_(3)^2 :χ1v3+9σw2c122[(ϕ′2)′], : _1v_3+9 _w^2c_12^2E\! [(φ 2) ], 6ϕ′ϕ′h(1)4h(2) 6φ φ h_(1)^4h_(2) :−9σw2v13[(ϕ′2)′], : -9 _w^2v_1^3E\! [(φ 2) ], 2ϕ′ϕ′h(1)3h(3) 2φ φ h_(1)^3h_(3) :6κ13v1c13, : 6 _13v_1c_13, 6ϕ′ϕ′h(1)h(2)h(3) 6φ φ h_(1)h_(2)h_(3) :−3σw2[(ϕ′2)′](v1c13+3c122). : -3 _w^2E\! [(φ 2) ] (v_1c_13+3c_12^2 ). The [(ϕ′2)′]E[(φ 2) ] terms in the second and fourth lines cancel, as do the c122c_12^2 terms in the third and sixth lines. The v1c13v_1c_13 terms in the fifth and sixth lines combine through σw2[(ϕ′2)′]=2χ2+2κ13 _w^2E[(φ 2) ]=2 _2+2 _13 into (6κ13−3(2χ2+2κ13))v1c13=−6χ2v1c13 (6 _13-3(2 _2+2 _13) )v_1c_13=-6 _2v_1c_13, so κ13 _13 does not appear in (19). ∎ 4.4 The off-diagonals, and closure Propositions 2 and 3 involve c12c_12 and c13c_13, so the system is not yet closed. Both follow from the same evaluation. Proposition 4 (Off-diagonal recursions). Under A1, c12ℓ=χ1c12ℓ−1,c13ℓ=χ1c13ℓ−1−3χ2(v1ℓ−1)2.c_12 = _1\,c_12 -1, c_13 = _1\,c_13 -1-3 _2 (v_1 -1 )^2. (20) Proof. For c12=σw2[g(1)g(2)]c_12= _w^2E[g_(1)g_(2)], the term [ϕ′ϕ′h(1)3]E[φ φ h_(1)^3] vanishes twice over: [ϕ′ϕ′]=0E[φ φ ]=0 by parity, and [h(1)3]=0E[h_(1)^3]=0 because h(1)h_(1) is independent of h(0)h_(0) and Gaussian. In [ϕ′2h(1)h(2)]E[φ 2h_(1)h_(2)], the substitution h(2)=β2h(0)+r2h_(2)= _2h_(0)+r_2 leaves a β2 _2 term carrying [h(1)]=0E[h_(1)]=0 and a residual term [ϕ′2][h(1)r2]=[ϕ′2]c12E[φ 2]E[h_(1)r_2]=E[φ 2]c_12. For c13=σw2[g(1)g(3)]c_13= _w^2E[g_(1)g_(3)] there are three terms. σw2[ϕ′ϕ′h(1)4]=3κ13v12 _w^2E[φ φ h_(1)^4]=3 _13v_1^2 by independence and [h(1)4]=3v12E[h_(1)^4]=3v_1^2. The middle term is three halves of TBT_B from Proposition 2, so by (17) and σw2[(ϕ′2)′]=2χ2+2κ13 _w^2E[(φ 2) ]=2 _2+2 _13 it equals −3v12(χ2+κ13)-3v_1^2( _2+ _13). The third is χ1c13 _1c_13 by the substitution above. The κ13 _13 contributions cancel between the first two. ∎ Corollary 2 (Closure). The quantities (v1,v2,v3,c12,c13) (v_1,v_2,v_3,c_12,c_13 ) satisfy a closed autonomous system whose coefficients are χ1 _1, χ2 _2 and χ3 _3 alone. At criticality the solution is explicit: v1v_1 and c12c_12 are depth-independent, v2v_2 and c13c_13 are linear with equal and opposite slopes ±3χ2v12± 3 _2v_1^2, so that v2ℓ+c13ℓis independent of depth,v_2 +c_13 independent of depth, (21) and v3v_3 is quadratic with leading coefficient 452χ22v13 452 _2^2v_1^3 once the depth-independent forcing terms become subleading. Corollary 3 (Two-sidedness). At criticality, c13ℓ=c130−3χ2v12ℓc_13 =c_13^0-3 _2v_1^2 decreases monotonically whenever χ2>0 _2>0. Beyond a finite depth c13<0c_13<0, so the term −6χ2v1c13-6 _2v_1c_13 in (19) is positive and remains so. Every forcing term in (19) is then positive, and the growth v3L=Θ(L2)v_3^L= (L^2) holds as a two-sided bound rather than only from above. 4.5 The recursion closes on the susceptibilities Individual terms in Proposition 3 involve activation moments outside the family (3): κ13=σw2[ϕ′ϕ′] _13= _w^2E[φ φ ], σw2[(ϕ′2)′] _w^2E[(φ 2) ] and σw2[(ϕ′2)′] _w^2E[(φ 2) ]. None survives. The same happens at order two, where [(ϕ′2)′]E[(φ 2) ] enters two terms and cancels between them. In both cases the surviving coefficients are susceptibilities χm _m alone. Conjecture 1. For every k, the recursion for vkv_k obtained from (9) has coefficients that are polynomial in the susceptibilities χ1,…,χk _1,…, _k and involves no other moment of the activation. The evidence is orders one through three, where every non-susceptibility moment cancels in pairs, and the pairings are between a squared term and the cross term that shares its activation factor. We have not established the general case, but the cancellations have a common cause which is worth stating, because it identifies what a general proof would need. Each cancellation consumes one of the identities (12). Order two uses σ01=0 _01=0 and σ02=−v1 _02=-v_1; order three uses those together with σ03=−3c12 _03=-3c_12. These are not independent facts. They are the single statement dmq∗/dxm=0d^mq /dx^m=0, one instance per m, and they are available only because the variance fixed point is independent of the input. The couplings σ0k _0k are also the only route by which a moment outside the family (3) can enter. Dependence on h(0)h_(0) reaches an expectation [g(h(0))h(i1)⋯h(ir)]E[g(h_(0))h_(i_1)·s h_(i_r)] solely through the coefficients βk=σ0k/q∗ _k= _0k/q of Corollary 1: with those set to zero every such expectation would factorize into [g]E[g] times a jet moment, and [g]E[g] is a susceptibility whenever g is a square. What pairs the terms is the identity ϕ(m)ϕ(m+1)=12(ϕ(m)2)′φ^(m)φ^(m+1)= 12 (φ^(m)2 ) . The cross term between adjacent derivative orders carries ϕ(m)ϕ(m+1)φ^(m)φ^(m+1), so Lemma 1 converts it into a moment of (ϕ(m)2)′ (φ^(m)2 ) ; and the same (ϕ(m)2)′ (φ^(m)2 ) is what Lemma 1 produces as the correction to [ϕ(m)2h(j)h(k)]E[φ^(m)2h_(j)h_(k)] when h(j)h_(j) or h(k)h_(k) is regressed on h(0)h_(0). The two therefore carry the same activation moment, and their β factors are fixed by (12). At orders two and three those factors are exactly the ones that cancel. A proof of Conjecture 1 would have to show that this persists: that dkq∗/dxk=0d^kq /dx^k=0 delivers, at every order, the coefficient that cancels the Stein correction against the adjacent cross term. We do not attempt it here. 4.6 Growth at criticality Away from criticality the order-1 recursion (14) gives v1ℓ=Aχ1ℓv_1 =A\, _1 for an amplitude A fixed by the first few layers, so the forcing in (15) is proportional to χ12ℓ _1^2 and the order-2 solution is a sum of two geometric terms, v2ℓ=Cχ1ℓ+Dχ12ℓ,D=3χ2A2χ1(χ1−1).v_2 =C\, _1 +D\, _1^2 , D= 3 _2A^2 _1 ( _1-1 ). (22) The homogeneous root is χ1 _1 and the particular root is χ12 _1^2. These are distinct for every χ1≠1 _1≠ 1, and they collide exactly at χ1=1 _1=1. That collision is the mechanism behind the growth law. Depth-invariance of v1v_1 requires χ1=1 _1=1 by (14); but at χ1=1 _1=1 the two roots coincide, the decomposition (22) degenerates, and the resonant solution is linear in depth: v2L=v20+3χ2v12L.v_2^L=v_2^0+3 _2v_1^2\,L. (23) The single condition that makes the first derivative propagate without gain or loss is therefore the same condition that makes the second derivative grow without bound. There is no value of χ1 _1 at which both are stationary. Equation (22) also makes D a prediction rather than a fitted quantity: χ2 _2 follows from the activation and the fixed point through (3), and A is measured from v1v_1 alone. Its 1/(χ1−1)1/( _1-1) pole is the quantitative signature of the collision, and Section 7 tests it with no free parameter on the theory side. At order three the same solution structure gives a bound in both directions. For the upper bound, suppose v3ℓ∼Kℓpv_3 K ^\,p. The forcing in (19) is then O(ℓ)+O(ℓp/2)O( )+O( ^\,p/2), since v1v_1 and c12c_12 are depth-independent at criticality, v2v_2 is linear, and |c13|≤v1v3 c_13 ≤ v_1v_3 by Cauchy–Schwarz. Summing over depth gives v3L=O(L2)+O(Lp/2+1)v_3^L=O(L^2)+O(L^\,p/2+1), so consistency requires p=max(2,p/2+1)p= (2,\ p/2+1 ), whose only solution is p=2p=2. This excludes p>2p>2 rather than merely permitting p=2p=2. The lower bound is Corollary 3, so v3L=Θ(L2),with leading coefficient452χ22v13,v_3^L= \! (L^2 ), leading coefficient 452 _2^2v_1^3, (24) the coefficient following from (19) once 9χ2v1v29 _2v_1v_2 and −6χ2v1c13-6 _2v_1c_13 become the dominant forcing terms, contributing 2727 and 1818 respectively. Equations (23) and (24) were obtained from the fixed-point form of the recursions. Since that form is not available at any finite depth, the growth laws are not yet established for the network of Section 3; what follows closes that gap. The conclusion is that the transient perturbs the recursion multiplicatively by a summable amount, which changes the constants but not the powers. Theorem 2 (Critical growth, with the transient controlled). Assume A1, let ϕφ have bounded derivatives through order four, and let the variance map have an attracting fixed point q∗>0q >0 with λ=|V′(q∗)|<1λ=|V (q )|<1 as in Lemma 2. At criticality χ1=1 _1=1, with v1∗:=limℓv1ℓv_1 := _ v_1 , v2L=3χ2(v1∗)2L+O(1),v3L=452χ22(v1∗)3L2+O(L).v_2^L=3 _2 (v_1 )^2L+O(1), v_3^L= 452 _2^2 (v_1 )^3L^2+O(L). (25) Proof. Fix λ¯∈(λ,1) λ∈(λ,1) and let ϵℓ _ denote any sequence with |ϵℓ|≤Cλ¯ℓ| _ |≤ C λ , the constant changing from line to line. Write the order-1 recursion as v1ℓ=(1+ϵℓ)v1ℓ−1v_1 =(1+ _ )v_1 -1, using χ1(ℓ)=χ1+ϵℓ=1+ϵℓ _1^( )= _1+ _ =1+ _ from Proposition 2. Because ∑ℓ|ϵℓ|<∞ _ | _ |<∞, the product ∏m≤ℓ(1+ϵm) _m≤ (1+ _m) converges to a finite nonzero limit, so v1∗v_1 exists and v1ℓ=v1∗+ϵℓv_1 =v_1 + _ . The same argument applied to c12ℓ=(1+ϵℓ)c12ℓ−1+ϵℓc_12 =(1+ _ )c_12 -1+ _ gives a bounded c12ℓc_12 . Now put (16) in the form v2ℓ=(1+αℓ)v2ℓ−1+Φ+ρℓv_2 =(1+ _ )v_2 -1+ + _ with Φ:=3χ2(v1∗)2 :=3 _2(v_1 )^2 and |αℓ|,|ρℓ|≤Cλ¯ℓ| _ |,| _ |≤ C λ ; the v2ℓ−1v_2 -1 factor in the error bound of (16) is absorbed into αℓ _ and the remainder into ρℓ _ . Variation of constants gives v2L=P0,Lv20+∑ℓ=1LPℓ,L(Φ+ρℓ),Pℓ,L:=∏m=ℓ+1L(1+αm).v_2^L=P_0,L\,v_2^0+ _ =1^LP_ ,L ( + _ ), P_ ,L:= _m= +1^L (1+ _m ). (26) Since ∑m|αm|≤Cλ¯/(1−λ¯)=:S<∞ _m| _m|≤ C λ/(1- λ)=:S<∞, every Pℓ,LP_ ,L lies in [e−S,eS][e^-S,e^S], and |Pℓ,L−1|≤eS∑m>ℓ|αm|≤C′λ¯ℓ |P_ ,L-1 |≤ e^S _m> | _m|≤ C λ . Hence ∑ℓ≤LPℓ,LΦ=ΦL+O(1) _ ≤ LP_ ,L = L+O(1), while ∑ℓ≤LPℓ,Lρℓ=O(1) _ ≤ LP_ ,L _ =O(1) and P0,Lv20=O(1)P_0,Lv_2^0=O(1). This is the first claim. For the second, the same treatment of the c13c_13 recursion in (20) gives c13ℓ=−3χ2(v1∗)2ℓ+O(1)c_13 =-3 _2(v_1 )^2 +O(1). Substituting this and v2ℓ=Φℓ+O(1)v_2 = +O(1) into (19) makes the forcing 9χ2v1v2=27χ22(v1∗)3ℓ+O(1)9 _2v_1v_2=27 _2^2(v_1 )^3 +O(1) and −6χ2v1c13=18χ22(v1∗)3ℓ+O(1)-6 _2v_1c_13=18 _2^2(v_1 )^3 +O(1), the remaining terms being O(1)O(1). Applying (26) with this forcing and ∑ℓ≤Lℓ=L2/2+O(L) _ ≤ L =L^2/2+O(L) gives the second claim. ∎ Everything so far concerns the preactivations. The quantity a derivative-dependent loss evaluates is the network output, and the two are related by a constant. Corollary 4 (Transfer to the network output). For the readout (5), with a independent of all other parameters, and for every k≥1k≥ 1, [(∂xku)2]=σa2σw2vkL+1.E\! [ ( _x^ku )^2 ]= _a^2 _w^2\;v_k^L+1. (27) Consequently every statement proved above for vkv_k holds verbatim for [(∂xku)2]E[( _x^ku)^2], up to that fixed constant: at criticality [(∂x1u)2]E[( _x^1u)^2] is depth-independent, [(∂x2u)2]E[( _x^2u)^2] grows linearly, [(∂x3u)2]E[( _x^3u)^2] grows quadratically, and under the residual scaling of Proposition 6 all of them are bounded uniformly in depth. Proof. Differentiating (5) k≥1k≥ 1 times in x gives ∂xku=∑iaig(k),iL _x^ku= _ia_i\,g^L_(k),i, since ∂xk[ϕ(hiL)]=g(k),iL _x^k\! [φ(h^L_i) ]=g^L_(k),i by definition. Since a is independent of (Wℓ,bℓ) (W ,b ) and hence of gLg^L, with [aiaj]=δijσa2/nE[a_ia_j]= _ij _a^2/n, [(∂xku)2]=∑i,j[aiaj][g(k),iLg(k),jL]=σa2n∑i[(g(k),iL)2]=σa2[(g(k),1L)2],E\! [ ( _x^ku )^2 ]= _i,jE[a_ia_j]\,E\! [g^L_(k),ig^L_(k),j ]= _a^2n _iE\! [ (g^L_(k),i )^2 ]= _a^2\,E\! [ (g^L_(k),1 )^2 ], the last step by exchangeability of the neurons. By (9), vkL+1=σw2[(g(k),1L)2]v_k^L+1= _w^2E[(g^L_(k),1)^2] for k≥1k≥ 1, which gives (27). ∎ 4.7 The dichotomy The growth in (23) is driven by χ2 _2, so it can be removed by setting χ2=0 _2=0. That escape closes off the property the network needs. Proposition 5 (Dichotomy). Assume A1 for the scalar-input network, let ϕφ be locally Lipschitz and twice differentiable in the ordinary sense at Lebesgue-almost every point, with ϕ′φ locally bounded where it exists, and let χ1=1 _1=1. Then exactly one of the following holds. 1. χ2>0 _2>0, and v2Lv_2^L grows without bound in L. 2. χ2=0 _2=0, and ∂x2u=0 _x^2u=0 almost everywhere, for every realization. Within this scalar-input, mean-field, Gaussian-jet setting, a fully connected network with independent layerwise initialization therefore cannot have both a nonvanishing second input derivative and a depth-stable second-derivative variance. Proof. By (3), χ2=σw2[ϕ′(z)2] _2= _w^2E[φ (z)^2] with z Gaussian of variance q∗>0q >0. That Gaussian has full support, so χ2=0 _2=0 forces ϕ′=0φ =0 at almost every point where it exists, hence almost everywhere. By (8) the second jet component is g(2)=ϕ′h(1)2+ϕ′h(2)g_(2)=φ h_(1)^2+φ h_(2), whose first term then vanishes almost everywhere. At the input layer h1=W1x+b1h^1=W^1x+b^1 gives h(2)1=0h^1_(2)=0 identically, and if h(2)ℓ−1=0h -1_(2)=0 almost everywhere then g(2)ℓ−1=0g_(2) -1=0 and hence h(2)ℓ=0h _(2)=0 almost everywhere. Induction on depth gives ∂x2u=0 _x^2u=0 almost everywhere. The argument does not require ϕφ to be affine, and deliberately so: a piecewise-linear ϕφ has ϕ′=0φ =0 almost everywhere without being affine, and the conclusion holds for it. If instead χ2>0 _2>0, Theorem 2 makes v2Lv_2^L grow linearly in L. ∎ The second horn is not hypothetical, and the hypothesis above is stated pointwise precisely so that it covers the case that matters. A piecewise-linear activation is locally Lipschitz and twice differentiable away from its finitely many kinks, where ϕ′=0φ =0; it therefore satisfies the hypothesis, and a network built from it has ∂x2u=0 _x^2u=0 almost everywhere. Two qualifications keep that statement literal. Such a ϕφ is not twice weakly differentiable in the usual sense, because its distributional second derivative is a sum of point masses at the kinks rather than a locally integrable function; a hypothesis phrased that way would exclude the very example the horn is about. And what fails for the resulting network is the ordinary second derivative, which vanishes almost everywhere. Such a network can still approximate a solution of a second-order equation in a norm that does not see ∂x2u _x^2u pointwise, and can be used with a weak or variational formulation of the residual; what it cannot do is satisfy a strong-form residual that evaluates ∂x2u _x^2u pointwise, since that quantity is zero almost everywhere regardless of the parameters. Nor is the first horn confined to tanh . A sinusoidal activation, used in implicit neural representations [14], has ϕ′=−sinφ =- and therefore χ2>0 _2>0, so a network built from it falls in the first branch and its second-derivative variance grows without bound in depth. The initialization scheme accompanying that architecture controls the distribution of the activations themselves, which is order zero in the present terminology; Proposition 5 concerns order two and is not addressed by it. Proposition 5 is what distinguishes this failure from an ordinary tuning problem. The edge-of-chaos condition (4) is a choice of (σw2,σb2)( _w^2, _b^2), and one might expect a different choice to control the second derivative as well. Section 4.6 rules that out at fixed χ2 _2, and the dichotomy rules out obtaining it by changing χ2 _2. The obstruction is in the architecture, not in the hyperparameters. 4.8 Residual scaling Since Proposition 5 places the obstruction in the architecture, the remedy has to be architectural. For the residual network, let h0∈ℝnh^0 ^n be a finite-variance learned or random embedding of the scalar input; this makes the skip connection dimensionally well defined. Replacing (1) by a residual layer with branch scale L−γL^-γ, hℓ=hℓ−1+L−γ(Wℓϕ(hℓ−1)+bℓ),h =h -1+L^-γ (W φ\! (h -1 )+b ), (28) is enough at γ=1/2γ=1/2 for each fixed finite derivative order under the conditions given next. Proposition 6 (Residual scaling). Fix K. Assume A1, that the derivatives of ϕφ through order K are bounded, and that the initial jet has finite second moments. Then, for the network (28) with γ=1/2γ=1/2, each vkLv_k^L with k≤Kk≤ K is bounded uniformly in L. The bound may depend on K, the activation, and the initial jet. Proof. Differentiating (28) k times gives hℓ(k)=hℓ−1(k)+L−1/2Wℓgℓ−1(k)h _(k)=h -1_(k)+L^-1/2W g -1_(k) for k≥1k≥ 1. Since WℓW has mean zero and is independent of layer ℓ−1 -1, the cross term vanishes in expectation and vkℓ=vkℓ−1+σw2L[(g(k)ℓ−1)2].v_k =v_k -1+ _w^2L\,E\! [ (g -1_(k) )^2 ]. (29) Write g(k)=ϕ′h(k)+Pkg_(k)=φ h_(k)+P_k, where PkP_k is a finite sum of products of lower-order jet components with bounded activation derivatives. For k=1k=1, (29) and boundedness of ϕ′φ give v1ℓ≤(1+B1/L)v1ℓ−1v_1 ≤(1+B_1/L)v_1 -1 for a constant B1B_1. Suppose the derivative variances below order k are bounded. Under A1, their joint Gaussian moments are then bounded by Isserlis’ formula. Cauchy–Schwarz and Young’s inequality therefore give constants Bk,Mk<∞B_k,M_k<∞, independent of ℓ and L, such that σw2[(g(k)ℓ−1)2]≤Bkvkℓ−1+Mk. _w^2E\! [ (g_(k) -1 )^2 ]≤ B_kv_k -1+M_k. (30) Thus vkℓ≤(1+Bk/L)vkℓ−1+Mk/Lv_k ≤(1+B_k/L)v_k -1+M_k/L. Discrete Grönwall gives vkL≤eBk(vk0+Mk),v_k^L≤ e^B_k (v_k^0+M_k ), (31) which closes the induction. ∎ The contrast with (23) and (24) is the point. In the plain network the growth exponent rises with the derivative order, so a fourth-order problem degrades faster with depth than a second-order one. Under (28) the single choice γ=1/2γ=1/2 bounds each fixed finite order without retuning the exponent. The constants need not be uniform in derivative order. The pathology is order-dependent; the scaling exponent is not. Figure 3 compares the residual prediction with finite-width simulations through order four. The theorem itself holds at every fixed finite order under its stated assumptions. 5 Numerical method 5.1 Exact jet propagation We propagate input derivatives by the Faà di Bruno expansion in (8), rather than by repeated automatic differentiation. For ϕ=tanhφ= , its first four derivatives are closed-form polynomials in t=tanh(z)t= (z): ϕ′ φ =1−t2, =1-t^2, ϕ′ φ =−2t(1−t2), =-2t(1-t^2), ϕ′ φ =−2(1−t2)(1−3t2), =-2(1-t^2)(1-3t^2), ϕ′ φ =8t(1−t2)(2−3t2). =8t(1-t^2)(2-3t^2). (32) No automatic differentiation is used anywhere, at any order, so the only floating-point error is roundoff. All computations use double precision. 5.2 An O(n)O(n) exact layer sampler Forming a width-n weight matrix costs O(n2)O(n^2) per layer, but the layer update requires only the products Wg(j)Wg_(j). Conditional on the current jet, these products are jointly Gaussian. For each neuron their covariance is [y(j),iy(k),i]=σw2n⟨g(j),g(k)⟩=:Sjk,E[y_(j),iy_(k),i]= _w^2n g_(j),g_(k) =:S_jk, (33) independently across neurons, because distinct rows of W are independent. Drawing independent samples from (0,S)N(0,S) is therefore distributionally identical to forming W and multiplying. The resulting sampler costs O(n(K+1)2)O\! (n(K+1)^2 ) per layer. This is an exact reformulation and not an approximation: finite-width fluctuations propagate correctly, because S is computed from the realized g of the previous layer rather than from its mean-field limit. It is what makes width 2182^18 reachable. 5.3 Estimators Two choices of estimator matter enough to state, because the natural alternative in each case destroys the structure being measured. Polynomial degree by finite differences, not log-log slope. The growth laws in Section 4 have the form vkℓ=a+bℓk−1v_k =a+b\, ^\,k-1 with a>0a>0. The local log-log slope of such a sequence is b(k−1)ℓk−1/(a+bℓk−1)b(k-1) ^k-1/(a+b ^k-1), which approaches k−1k-1 only as a/(bℓk−1)→0a/(b ^k-1)→ 0 and therefore underestimates the degree at any finite depth. Fitted slopes over ℓ∈[64,256] ∈[64,256] come out at 0.850.85, 1.801.80 and 2.632.63 for k=2,3,4k=2,3,4, against true degrees 11, 22 and 33. We therefore test degree by the statement that the (k−1)(k-1)-th finite difference of vkℓv_k is constant, which involves no asymptotic limit. Covariances centred, not differenced. Several quantities in Section 4 are departures of an expectation from its factorized value, for instance σw2([ϕ′2h(k)2]−[ϕ′2]vk) _w^2 (E[φ 2h_(k)^2]-E[φ 2]\,v_k ). Section 7.1 measures these departures as ratios lying within a fraction of a percent of unity, so evaluating them as a difference of two separately estimated means loses most of the significant digits to cancellation. We estimate them as centred covariances, σw2(ϕ′2−ϕ′2¯)(h(k)2−h(k)2¯)¯ _w^2\, (φ 2- φ 2 ) (h_(k)^2- h_(k)^2 ), which is algebraically identical and numerically stable. 5.4 Configuration Unless stated otherwise, measurements use width n=216n=2^16, depth L=256L=256, and 1616 independently sampled networks, with statistics averaged over neurons and networks. Fits and depth-averaged quantities are taken over ℓ∈[64,256] ∈[64,256], across all retained depths and across individual networks. The lower limit excludes an initialization transient. The fixed-point form (12) of Proposition 1 requires the preactivation variance to be constant in x, and the first layers are furthest from that: at the input layer h(1)1=W1h^1_(1)=W^1 while h(0)1=W1x+b1h^1_(0)=W^1x+b^1, so σ01≠0 _01≠ 0 there. The measured v1ℓv_1 falls from 1.761.76 at ℓ=1 =1 to 0.340.34 at ℓ=4 =4 before flattening. Discarding those layers is a matter of numerical conditioning rather than of validity. Lemma 2 bounds the discrepancy by λ¯ℓ λ and Theorem 2 carries it through the recursions, so the growth laws do not require the transient to have ended; the early layers are excluded because they are where the correction is largest, not because the theory fails there. At ℓ=64 =64 the correction is already of order 5×10−195× 10^-19. Section 7 reports width convergence for each order, which is what determines the range of k over which the results are claimed. 5.5 Computational assistance and reproducibility A large language model (Claude, Anthropic) was used as a computational aid, to expedite symbolic manipulation in Section 4 and to implement the propagation and sampling code described above. Every derivation reported in this paper was verified independently by the authors by hand and cross-checked numerically against the measurements of Section 7; no algebraic result is included on the basis of that assistance alone. Every numerical claim is generated by committed code from recorded seeds rather than transcribed. The archive named under Data availability contains that code together with the raw outputs and the figure scripts, so each figure and each reported value can be regenerated rather than taken on trust. Responsibility for all content rests with the authors. 6 Parity arguments 6.1 Cross terms that parity does not eliminate Expanding [g(j)g(k)]E[g_(j)g_(k)] in (9) produces, alongside the squared terms, cross terms carrying products of adjacent activation derivatives. For odd ϕφ each such product ϕ(m)ϕ(m+1)φ^(m)φ^(m+1) is an odd function, and h(0)h_(0) is symmetric, so [ϕ(m)(h(0))ϕ(m+1)(h(0))]=0.E\! [φ^(m)(h_(0))\,φ^(m+1)(h_(0)) ]=0. (34) The familiar next step is to conclude that the cross terms vanish. That step is not a consequence of (34). It is an application of A2, and it fails here. A cross term has the form [ϕ(m)ϕ(m+1)J]E[φ^(m)φ^(m+1)\,J] where J is a product of jet components. Factorizing it into [ϕ(m)ϕ(m+1)][J]E[φ^(m)φ^(m+1)]\,E[J] requires the activation factor to be independent of J. It is not: both depend on h(0)h_(0), the activation factor directly and the jets through the couplings σ0k _0k of Corollary 1. A product of two mean-zero quantities has zero expectation only when they are uncorrelated, and these are not. The size of the resulting error is not marginal. In the order-3 expansion the term 6σw2[ϕ′ϕ′h(1)h(2)h(3)]6 _w^2E[φ φ h_(1)h_(2)h_(3)], which (34) would eliminate, contributes −39.65-39.65 at ℓ=256 =256 against a total forcing of 134.53134.53, or 29%29\% of it. Two features of this failure are worth separating, because they explain why it is not already known. First, order one is immune. The order-1 recursion (14) involves only χ1=σw2[ϕ′2] _1= _w^2E[φ 2], a square, so no product of adjacent derivatives arises and (34) is never invoked. A derivation that propagates the preactivation variance and the input-output Jacobian, which is what the mean-field literature computes, therefore cannot encounter the trap. It appears first at order two. Second, the failure is invisible to the diagnostic one would naturally use. Assumption A2 is normally checked by comparing an expectation against its factorized value and asking whether the ratio is one. For these terms the factorized value is exactly zero, so the ratio is 0/00/0 and carries no information. Every other term in (15) and (19) passes such a check to within 0.1%0.1\% while these terms are wrong by construction. 7 Numerical results Simulations use the configuration in Section 5. The reported intervals are across independently sampled networks. They test the mean-field recursions, not trained-network accuracy. 7.1 One hypothesis tested, and one the derivation avoids A1 carries the derivation. Theorem 1 establishes it in the infinite-width limit at fixed depth, which is a statement about a limit; what the simulations can add is whether it is accurate at the widths actually used, so it is tested directly. Skewness and excess kurtosis of h(1)h_(1), h(2)h_(2) and h(3)h_(3) are all below 0.010.01 in magnitude throughout the measurement window. That h(2)h_(2) and h(3)h_(3) are Gaussian is not implied by the standard mean-field statement, which concerns h(0)h_(0) alone: both are built from products through (8). The three Isserlis identities the recursions use hold to the same precision, [h(1)4]3v12=1.0002,[h(1)3h(3)]3v1c13=1.0002,[h(1)2h(2)2]v1v2+2c122=1.0002, E[h_(1)^4]3v_1^2=1.0002, E[h_(1)^3h_(3)]3v_1c_13=1.0002, E[h_(1)^2h_(2)^2]v_1v_2+2c_12^2=1.0002, (35) and the fixed-point identity σ01=0 _01=0 of Proposition 1 is measured at +1.3×10−4+1.3× 10^-4. Assumption A2 is a different matter, and the distinction is worth keeping explicit. No result in Section 4 uses it: the recursions at orders two and three follow from A1 together with the fixed-point identities of Proposition 1, and every expectation is evaluated by Lemma 1 rather than by factorization. A2 nonetheless holds numerically. The decoupling ratios ρm:=[ϕ(m)2J][ϕ(m)2][J] _m:= E\! [φ^(m)2J ]E\! [φ^(m)2 ]E[J] (36) for the three terms where they are defined measure 0.99920.9992, 1.00011.0001 and 0.99980.9998. We report these because their agreement is what makes Section 6.1 precise: A2 is accurate wherever it can be tested, and the terms it fails on are exactly those where [J]=0E[J]=0 leaves nothing to test. Figure 1: Critical depth growth for derivative variances of orders one to three. Solid curves are ensemble means, shaded bands show the 10th to 90th percentiles, and dashed curves are the predicted polynomial forms. Width n=216n=2^16, 1616 independently sampled networks. 7.2 Critical growth Figure 1 tests the recursions at χ1=1 _1=1. The order-one slope is 3.6×10−5±5.1×10−53.6× 10^-5± 5.1× 10^-5, or 0.71σ0.71σ from zero, consistent with (14). The order-two slope is 0.1719±0.00890.1719± 0.0089, while (15) predicts 3χ2v12=0.1700±0.00663 _2v_1^2=0.1700± 0.0066; their difference is 0.17σ0.17σ. The measured order-three quadratic coefficient is 0.2475±0.03520.2475± 0.0352 and the prediction from (24) is 0.2288±0.01340.2288± 0.0134; their difference is 0.50σ0.50σ. Thus the data support depth-invariant first-order variance, linear second-order growth, and quadratic third-order growth at the measured widths and depths. Figure 2: The resonance, approached from above. Upper panel: the fitted coefficient D of χ12ℓ _1^2 against the prediction 3χ2A2/(χ1(χ1−1))3 _2A^2/( _1( _1-1)), which contains no fitted quantity. Lower panel: their ratio, with intervals covering unity at every χ1 _1. The coefficient C is not shown; it is resolved only adjacent to the pole, and the divergence appears in D alone. Width n=214n=2^14, 1212 independently sampled networks at each χ1 _1. 7.3 The resonance Figure 2 tests (22) above criticality. Fitting v2ℓ=Cχ1ℓ+Dχ12ℓv_2 =C _1 +D _1^2 over the measurement window at each χ1 _1 and comparing the fitted D against 3χ2A2/(χ1(χ1−1))3 _2A^2/( _1( _1-1)) gives a ratio within 5%5\% of unity at every χ1 _1 measured, with intervals covering unity throughout. Nothing on the theory side of this comparison is fitted: χ2 _2 follows from the activation and the fixed point, and the amplitude A is taken from v1v_1. We do not report C. It is resolved only for |χ1−1|≤0.01| _1-1|≤ 0.01; beyond that χ1ℓ _1 is negligible against χ12ℓ _1^2 across the window, the two basis functions cease to be separable, and the fitted C is consistent with zero. D suffices, since the divergence appears in D alone. Below χ1=1 _1=1 the signals decay and the estimator is dominated by its noise floor, so the pole is approached from above only. Figure 3: Derivative variances for a plain network and a residual network with L−1/2L^-1/2 branch scale, orders one to four. Lines are ensemble means and shaded bands show the 10th to 90th percentiles. Width n=214n=2^14, 1212 independently sampled networks at each depth. 7.4 Residual scaling Figure 3 compares the plain network with (28) at γ=1/2γ=1/2 from depth 88 to 512512. In the plain network, the order-two, order-three, and order-four variances increase by factors of 2323, 590590, and 1.8×1041.8× 10^4. Under residual scaling, the corresponding factors are 1.391.39, 1.701.70, and 2.222.22. The result is consistent with the boundedness claim in Proposition 6. Order four is shown only as a boundedness check; this paper does not claim an order-four mean-field growth law. Figure 4: Closed third-order recursion integrated from ℓ=64 =64. Solid curves are ensemble measurements and dashed curves are predictions from (v1,v2,v3,c12,c13)(v_1,v_2,v_3,c_12,c_13) with no later fitted input. Width n=216n=2^16, 1616 independently sampled networks; the tighter closure test quoted in the text is a separate run. 7.5 Closed recursion Figure 4 integrates (14), (15), (19), and (20) from ℓ=64 =64 using χ1 _1, χ2 _2, and χ3 _3. In a separate run at the same width n=216n=2^16 with 6464 networks, all five ensemble ratios are within 4%4\% of one. This test uses no measured quantity after the initial condition. The agreement supports the closed covariance system through third order at the measured width and depth range. 7.6 Width dependence of the residual The closure residual is a finite-width effect. Across three widths at 6464 networks each, the largest deviation from unity falls from 14.79%14.79\% at n=214n=2^14 to 3.94%3.94\% at 2162^16, then changes little, reading 2.90%2.90\% at 2182^18. Fitting a power law to the three widths gives exponents of 0.64570.6457, 0.61360.6136 and 0.40190.4019 for the three quantities whose deviations are resolved, with large fit residuals. We report these but do not treat them as a rate: the 2162^16 and 2182^18 deviations sit at one to two standard errors, so the fit is constrained by a single resolved step. The statement the data supports is that the residual falls sharply with width and then plateaus near 3%3\%. A plateau that does not move with width requires a source that is not finite width. One candidate is ruled out at once. The finite-depth correction to the fixed-point identities, bounded by Lemma 2 and carried through the recursions by Theorem 2, has fallen to order 5×10−195× 10^-19 by the first layer of the window, which is seventeen orders of magnitude below the residual and cannot account for it. A second candidate does account for its size. The integration uses χ1 _1 evaluated by quadrature at q∗q , while a realized network has its own effective multiplier; a relative discrepancy δ compounds over the window, so δ∼10−4δ 10^-4 produces a deviation of a few per cent that does not shrink with n. The measured per-layer discrepancy is 1.1×10−41.1× 10^-4, which is the right size. We do not regard this as established, for three reasons. The discrepancy is 0.71σ0.71σ from zero, so it is not itself resolved. The ordering of deviations across quantities matches what compounding χ1 _1 error would produce at 2142^14 and 2162^16 but not at 2182^18. And v1v_1 and c12c_12 obey the same homogeneous recursion, so a common χ1 _1 error must imply one value of δ from both; the two implied values differ by 1.3σ1.3σ, which is the consistency test the explanation has to pass and it passes weakly. 8 Discussion The practical reading of Proposition 5 is that tuning cannot solve the problem it describes. Choosing (σw2,σb2)( _w^2, _b^2) selects a point on the one-parameter family of variance maps, and Section 4.6 shows that the point which stabilizes the first derivative is the point at which the second grows linearly. Moving off criticality trades that linear growth for exponential behavior in the first derivative, which is the failure the criterion was introduced to prevent. For a loss that depends on second or higher derivatives there is no setting of the two variances that avoids both. That leaves the architecture as the only place to intervene, which is what Proposition 6 formalizes. The relevant feature of (28) is not the skip connection itself but the attenuation of the branch: the factor L−1/2L^-1/2 makes each layer’s contribution small enough that the accumulated effect over L layers stays bounded, and (31) holds at every order with the same exponent. An architecture that attenuates layer contributions in some other way, or that begins near shallow and deepens during training as the trainable branch scalar of Wang et al. [16] does, would be expected to have the same effect for the same reason. Three directions remain outside the scope of what is established here. The analysis is of the network at initialization, and the corresponding statement about training dynamics requires following these quantities under gradient descent rather than at a single point. The derivation assumes a smooth activation, and Proposition 5 shows that the non-smooth case is not a limit of it but a separate regime in which the second derivative vanishes identically. And the input is scalar throughout; for a multidimensional input the jet becomes a collection of mixed partial derivatives, and whether the closure of Corollary 2 survives that generalization is not something the present argument settles. 9 Conclusions The covariance of the input derivatives of a wide scalar-input network obeys mean-field recursions through third order, exact at the variance fixed point and with finite-depth corrections that decay geometrically at a rate set by the variance map rather than by the edge-of-chaos condition. The joint Gaussianity they use is not an extra hypothesis: it holds in the infinite-width limit at each fixed depth. Their coefficients are the susceptibilities χ1 _1, χ2 _2 and χ3 _3 together with the fixed-point identities of Proposition 1, which hold because the variance fixed point does not depend on the input. Every remaining moment of the activation cancels, so the five-quantity system closes on those three susceptibilities alone, with no measured quantity carried along to keep it closed. At the edge of chaos that closure has a definite shape. The condition χ1=1 _1=1 is what holds the first-derivative variance constant through depth, and it is the same condition that makes the second-order recursion resonant: the two modes χ1ℓ _1 and χ12ℓ _1^2 of its solution coincide there, and their coincidence converts a bounded solution into one growing linearly in depth whenever the activation has nonzero curvature. The third-order variance grows quadratically. The two behaviors are therefore not independent settings to be traded against one another but consequences of the same condition, which is the content of Proposition 5. Simulations confirm the growth exponents, the parameter-free amplitude of the resonance, the bound of Proposition 6 and the closed recursion. Two questions are left open. Conjecture 1 asserts that the closure persists at every order, with coefficients polynomial in χ1,…,χk _1,…, _k and no other moment of the activation. The evidence is orders one through three, where each cancellation consumes one of the fixed-point identities, and the general case is unproved. The order-four variance is reported only as a boundedness check; no order-four growth law is claimed here. CRediT authorship contribution statement Prashant Singh: Conceptualization, Formal analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review and editing. Pranav Singh: Conceptualization, Formal analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review and editing. Both authors contributed equally to this work. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Data availability Code, the exact layer sampler, raw sweep outputs and the figure-generation scripts will be made publicly available at https://doi.org/10.5281/zenodo.21873633 (10.5281/zenodo.21873633) on publication. The archived release will identify the source revision, software environment and random seeds needed to regenerate each figure and reported number. Declaration of Generative AI and AI-assisted technologies in the writing process During the preparation of this work the authors used Claude (Anthropic) in order to draft and revise manuscript text and to assist with LaTeX preparation. After using this tool the authors reviewed and edited the content as needed and take full responsibility for the content of the published article. References [1] T. Bachlechner, B. P. Majumder, H. H. Mao, G. Cottrell, and J. McAuley (2021) ReZero is all you need: fast convergence at large depth. In Uncertainty in Artificial Intelligence, Proceedings of Machine Learning Research, Vol. 161, p. 1352–1361. Cited by: §2. [2] K. R. Chickering (2024) A quasilinear algorithm for computing higher-order derivatives of deep feed-forward neural networks. arXiv:2412.09752. Cited by: §2. [3] B. Hanin (2022) Random fully connected neural networks as perturbatively solvable hierarchies. arXiv:2204.01058. Cited by: §2, §3.3. [4] S. Hayou, E. Clerico, B. He, G. Deligiannidis, A. Doucet, and J. Rousseau (2021) Stable ResNet. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, PMLR, Vol. 130, p. 1324–1332. Cited by: §2. [5] S. Hayou, A. Doucet, and J. Rousseau (2019) On the impact of the activation function on deep neural networks training. In International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 2672–2680. Cited by: §2. [6] A. Hyvärinen (2005) Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research 6 (24), p. 695–709. Cited by: §1. [7] A. Jacot, F. Gabriel, and C. Hongler (2018) Neural tangent kernel: convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, Vol. 31. Cited by: §2. [8] P. Marion, A. Fermanian, G. Biau, and J. Vert (2025) Scaling ResNets in the large-depth regime. Journal of Machine Learning Research 26 (56), p. 1–48. Cited by: §2. [9] J. Martens, A. Ballard, G. Desjardins, G. Swirszcz, V. Dalibard, J. Sohl-Dickstein, and S. S. Schoenholz (2021) Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping. arXiv:2110.01765. Cited by: §2. [10] B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli (2016) Exponential expressivity in deep neural networks through transient chaos. In Advances in Neural Information Processing Systems, Vol. 29. Cited by: §1, §2, §3.1. [11] M. Raissi, P. Perdikaris, and G. E. Karniadakis (2019) Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics 378, p. 686–707. External Links: Document Cited by: §1, §2. [12] D. A. Roberts, S. Yaida, and B. Hanin (2022) The principles of deep learning theory. Cambridge University Press. Note: arXiv:2106.10165 Cited by: §2. [13] S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein (2017) Deep information propagation. In International Conference on Learning Representations, Cited by: §1, §2, §3.1. [14] V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein (2020) Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems, Vol. 33. Cited by: §1, §4.7. [15] Y. Song and S. Ermon (2019) Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, Vol. 32, p. 11895–11907. Cited by: §1. [16] S. Wang, B. Li, Y. Chen, and P. Perdikaris (2024) PirateNets: physics-informed deep learning with residual adaptive networks. Journal of Machine Learning Research 25 (402), p. 1–51. Cited by: §1, §1, §2, §2, §8. [17] S. Wang, Y. Teng, and P. Perdikaris (2021) Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing 43 (5), p. A3055–A3081. External Links: Document Cited by: §2. [18] S. Wang, X. Yu, and P. Perdikaris (2022) When and why PINNs fail to train: a neural tangent kernel perspective. Journal of Computational Physics 449, p. 110768. External Links: Document Cited by: §1, §2. [19] J. C. Wong, C. Ooi, A. Gupta, and Y. Ong (2024) Learning in sinusoidal spaces with physics-informed neural networks. IEEE Transactions on Artificial Intelligence 5 (3), p. 985–1000. External Links: Document Cited by: §2. [20] G. Yang and S. S. Schoenholz (2017) Mean field residual networks: on the edge of chaos. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §2, §2. [21] G. Yang (2019) Tensor programs I: wide feedforward or recurrent neural networks of any architecture are gaussian processes. In Advances in Neural Information Processing Systems, Vol. 32, p. 9947–9960. Cited by: §3.3. [22] H. Zhang, D. Yu, M. Yi, W. Chen, and T. Liu (2022) Stabilize deep ResNet with a sharp scaling factor τ. Machine Learning 111 (9), p. 3359–3392. External Links: Document Cited by: §2.