Paper deep dive
Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW
Kang Liu, Suyan Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/21/2026, 3:35:19 AM
Summary
This paper formulates the AdamW optimizer as a finite-horizon input-state-output (ISO) system to analyze how minibatch perturbations influence training beyond the immediate update step. By treating model parameters and first- and second-moment estimates as a joint state, the authors derive a signed response operator that maps localized gradient perturbations to future loss effects. The study establishes an exact multistep error decomposition and proves first-order finite-horizon accuracy, revealing how optimizer memory shapes the magnitude, timing, and sign of delayed influences, including contributions to transient loss spikes.
Entities (11)
Relation Signals (8)
Suyan Li → affiliatedwith → National University of Singapore
confidence 95% · Suyan Li Affiliation: Department of Electrical and Computer Engineering, National University of Singapore
Kang Liu → affiliatedwith → Xi'an Jiaotong University
confidence 95% · Kang Liu Affiliation: School of Future Technology, Xi’an Jiaotong University
AdamW → hasstate → First-Moment Estimate
confidence 95% · whose state contains the model parameters and first- and second-moment estimates
AdamW → hasstate → Second-Moment Estimate
confidence 95% · whose state contains the model parameters and first- and second-moment estimates
ISO System → models → AdamW
confidence 95% · We formulate AdamW as a finite-horizon input--state--output (ISO) system
Minibatch Perturbation → causes → Delayed Influence
confidence 90% · A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information
Response Operator → maps → Gradient Perturbation
confidence 90% · signed response operator that maps a localized gradient perturbation to its future loss effects
Response Operator → mapsto → Future Loss Effects
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence. We formulate AdamW as a finite-horizon input--state--output (ISO) system whose state contains the model parameters and first- and second-moment estimates. Linearizing the joint dynamics yields a signed response operator that maps a localized gradient perturbation to its future loss effects, revealing how optimizer memory shapes their magnitude, timing, and sign. We further derive an exact multistep error decomposition and establish first-order finite-horizon accuracy under local smoothness and controlled activation switching. Experiments validate the response mechanism and optimizer-state effects, while repeated-future analyses reveal substantial prospective structure in delayed influence that can be partially recovered from ISO approximations. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.19762v1
- Canonical: https://arxiv.org/abs/2608.19762v1
Trouble viewing inline? Open PDF directly →
Full Text
140,237 characters extracted from source content.
Expand or collapse full text
Finite-Horizon Input–Output Dynamics of Minibatch Perturbations in AdamW Kang Liu Affiliation: School of Future Technology, Xi’an Jiaotong University, Xi’an, China, 710049 Correspondence to: kanyo@foxmail.com Suyan Li Affiliation: Department of Electrical and Computer Engineering, National University of Singapore, Singapore, 119077 Abstract A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence. We formulate AdamW as a finite-horizon input–state–output (ISO) system whose state contains the model parameters and first- and second-moment estimates. Linearizing the joint dynamics yields a signed response operator that maps a localized gradient perturbation to its future loss effects, revealing how optimizer memory shapes their magnitude, timing, and sign. We further derive an exact multistep error decomposition and establish first-order finite-horizon accuracy under local smoothness and controlled activation switching. Experiments validate the response mechanism and optimizer-state effects, while repeated-future analyses reveal substantial prospective structure in delayed influence that can be partially recovered from ISO approximations. Code is available at https://github.com/Kanyooo/Loss_ISO. Keywords: Optimizer dynamics, Minibatch perturbation, Finite-Horizon analysis, AdamW 1 Introduction Modern neural networks are trained through a sequence of stochastic minibatch updates. With adaptive optimizers such as AdamW, however, the effect of a minibatch is not confined to the step at which it is observed. Its gradient changes not only the model parameters, but also the first- and second-moment estimates that determine subsequent updates. As a result, the effect of one minibatch can persist for several training steps and may become visible in the loss only later. This temporal dependence is particularly relevant to transient training instabilities, including loss spikes. Sharp local geometry and large-step training dynamics have been linked to transient loss excursions (31). Small-scale proxies further show that local training statistics can anticipate large-scale Transformer instabilities (28). More recent work attributes another class of spikes to abnormal stochastic gradients and motivates spike-aware momentum treatment (12). Existing analyses characterize Adam through local geometry, adaptive preconditioning, and optimizer dynamics (7; 1). Recent influence studies further show that training-data effects depend on their position along the optimization trajectory (27), and that AdamW-aware attribution must propagate perturbations through both parameter and moment states (8). Optimizer memory can also make minibatch ordering a first-order source of finite-window variation (25). These results establish that optimizer state and trajectory matter, but they do not provide a signed, horizon-resolved account of how a localized minibatch perturbation is stored, propagated, and expressed in future loss. We study the delayed effect of minibatch perturbations in AdamW by comparing paired training trajectories that differ only at one update and share the same subsequent training sequence. This construction isolates how a localized perturbation evolves over a finite future horizon. The central idea is to formulate AdamW as a finite-horizon input–state–output (ISO) system, with the model parameters and optimizer moments forming the joint state. We derive a tangent model that maps an initial gradient perturbation to its future loss response through the intervening AdamW dynamics, revealing how optimizer memory shapes the magnitude, timing, and sign of delayed effects. We further characterize the finite-horizon approximation error, including nonlinear dynamics and activation-pattern changes. Beyond pathwise analysis, we ask whether delayed influence retains structure before the future training sequence is realized. Repeated-future experiments show substantial prospective structure, while its recoverability from present-time ISO approximations depends on the underlying dynamics. Figure 1 summarizes the ISO mechanism. Figure 1: Finite-horizon ISO view of delayed minibatch influence in AdamW. Our contributions are threefold: 1. We formulate localized minibatch influence as a signed, finite-horizon response under paired AdamW trajectories, isolating the effect of a single gradient perturbation under a shared future realization. 2. We derive a joint parameter–moment ISO operator that decomposes the response into perturbation write-in, optimizer-state propagation, and loss readout, thereby characterizing how AdamW memory can delay or transiently amplify a minibatch’s effect. 3. We establish an exact multistep error decomposition that separates write-in nonlinearity, smooth propagation error, and activation switching, and prove fixed-horizon first-order accuracy. 2 Related Work Training-data influence has gradually moved from static, endpoint-based attribution toward trajectory-aware descriptions of how individual training examples affect learning. Classical influence functions characterize infinitesimal reweighting around a trained solution (14), while subsequent work has improved their scalability and applicability to modern deep networks and large models (24; 10; 21; 16; 29). A parallel line instead follows the optimization path, from retracing SGD updates and propagating hypergradients (11; 22; 5) to approximate unrolling and trajectory-specific influence (2; 27). Recent work further shows that faithful trajectory attribution under AdamW requires accounting for its parameter and moment states (8). These developments establish that training order and intermediate optimization states matter. The object studied here is more local and dynamical: we perturb one realized minibatch gradient and track its signed response over a finite horizon, making the temporal response itself the quantity of interest. For AdamW, this distinction is important because the optimizer carries information across updates through its first- and second-moment states. Existing work has characterized Adam and AdamW through convergence, preconditioning, implicit geometry, and dynamical behavior (13; 23; 18; 19; 1; 17; 30). Closely related studies of training instability show that adaptive preconditioning, gradient statistics, and optimizer-state dynamics can shape short-time behavior and transient loss spikes (28; 31; 12; 26; 3). Optimizer memory can also make minibatch ordering a first-order source of finite-window variation (25). These results motivate treating the optimizer state as part of the perturbation dynamics rather than as an implementation detail. Our formulation treats parameters and AdamW moments as a joint state, explicitly separating perturbation write-in, state propagation, and loss readout. This finite-horizon ISO view provides a signed, horizon-resolved account of how a localized minibatch perturbation is stored in optimizer state and later expressed in future loss. 3 Finite-Horizon Minibatch Influence 3.1 Overview and Time Convention We model the effect of a gradient perturbation as a finite-horizon input–state–output process: ξt→Btδxt+1→Φt+h,t+1δxt+h→ct+h⊤δFt+h. _t \;B_t\;δ x_t+1 \; _t+h,t+1\;δ x_t+h \;c_t+h \;δ F_t+h. Here, ξt _t denotes the perturbation applied at step t, BtB_t maps it into the joint AdamW state, Φt+h,t+1 _t+h,t+1 propagates the resulting state deviation through subsequent updates, and ct+h⊤c_t+h maps the propagated deviation to the probe-loss response. Throughout, xsx_s denotes the joint state immediately before minibatch s is processed. Since the perturbation is applied during update t, horizon h=1h=1 corresponds to the first post-perturbation state xt+1x_t+1. 3.2 Paired Trajectories and Joint AdamW State For minibatch ℬsB_s, define gs(θ):=∇θℓ(θ,ℬs)∈ℝd.g_s(θ):= _θ (θ;B_s) ^d. Immediately before processing minibatch s, the joint AdamW state is xs:=[θs,ms−1,vs−1]⊤∈ℝ3d,x_s:=[ _s,m_s-1,v_s-1] ^3d, where θs _s is the model parameter, ms−1m_s-1 is the first-moment state, and vs−1v_s-1 is the second-moment state. Under AdamW (18), the first- and second-moment estimates evolve as ms m_s =β1ms−1+(1−β1)gs(θs), = _1m_s-1+(1- _1)g_s( _s), vs v_s =β2vs−1+(1−β2)gs(θs)⊙2. = _2v_s-1+(1- _2)g_s( _s) 2. Let ρ1,s:=1−β1s _1,s:=1- _1^s and ρ2,s:=1−β2s _2,s:=1- _2^s, and define the corrected moments m^s:=ms/ρ1,s m_s:=m_s/ _1,s and v^s:=vs/ρ2,s v_s:=v_s/ _2,s. The parameter update is written as θs+1=Dsθs+qs(ms,vs),Ds:=I−ηsΛs, _s+1=D_s _s+q_s(m_s,v_s),~~D_s:=I- _s _s, where Λs _s denotes decoupled weight decay and qs(m,v):=−ηs(m/ρ1,s)⊘(v/ρ2,s+ϵ)q_s(m,v):=- _s (m/ _1,s ) ( v/ _2,s+ 1 ) is evaluated coordinatewise. Paired trajectories. We compare a control trajectory and a shock trajectory initialized from the same state: xts=xtc=xtx_t^s=x_t^c=x_t. At the shock step, their gradients satisfy gts=gtc+ξtg_t^s=g_t^c+ _t, where ξt _t is the realized minibatch-gradient perturbation. From step t+1t+1 onward, the two trajectories use the same minibatches, stochastic realizations, learning-rate schedule, and optimizer configuration. Define the paired state deviation δxs:=xss−xscδ x_s:=x_s^s-x_s^c. Let Fs:ℝd→ℝ\F_s:R^d \ be a sequence of probe objectives shared by the paired trajectories, and define δFs:=Fs(θs)−Fs(θsc).δ F_s:=F_s( _s^s)-F_s( _s^c). A fixed probe corresponds to the special case Fs≡F_s≡ F. Alternatively, taking Fs(θ):=ℓ(θ,ℬs)F_s(θ):= (θ;B_s) gives the response of the realized future minibatch loss under the shared future sequence. Because the probe and future random sequence are shared, δFsδ F_s isolates the pathwise effect of the gradient difference introduced at step t. The response is signed: δFs>0δ F_s>0 means that the shock raises the probe loss relative to the control, whereas δFs<0δ F_s<0 means that it lowers the probe loss. The control gradient gtcg_t^c may be the gradient of another realized minibatch, as in the paired experimental protocol. Alternatively, taking it to be the conditional mean gradient, g¯t:=[gt∣ℱt],ξt:=gt−g¯t, g_t:=E[g_t _t],~~ _t:=g_t- g_t, gives a conditional stochastic interpretation of the perturbation, where ℱtF_t denotes the training history available before sampling the minibatch at step t. 3.3 Minibatch Shock Write-In We first derive how the perturbation ξt _t enters the three blocks of the joint AdamW state. Unless stated otherwise, all quantities in this subsection are evaluated along the control trajectory. Define the sensitivities of the adaptive parameter update to the moment states: Ms:=∂qs∂ms,Vs:=∂qs∂vs.M_s:= ∂ q_s∂ m_s,~~V_s:= ∂ q_s∂ v_s. Writing ds:=v^s+ϵd_s:= v_s+ 1, these sensitivities are diagonal: Ms M_s =−ηsρ1,sDiag(ds−1), =- _s _1,sDiag(d_s^-1), Vs V_s =ηsDiag(m^s2ρ2,sv^s⊙ds⊙2). = _sDiag ( m_s2 _2,s v_s d_s 2 ). Derivatives with respect to the second-moment state are evaluated at coordinates where v^s,i>0 v_s,i>0. The matrix MsM_s is negative diagonal, while the sign of each diagonal entry of VsV_s follows the corresponding first-moment coordinate. Define the effective gradient-to-parameter Jacobian Γs:=(1−β1)Ms+2(1−β2)VsDiag(gsc). _s:=(1- _1)M_s+2(1- _2)V_sDiag(g_s^c). Its two terms correspond to the paths g→m→θg→ m→θ and g→v→θg→ v→θ, respectively. At the shock step, direct subtraction of the two moment updates gives δmt δ m_t =(1−β1)ξt, =(1- _1) _t, δvt δ v_t =2(1−β2)Diag(gtc)ξt+(1−β2)ξt⊙2. =2(1- _2)Diag(g_t^c) _t+(1- _2) _t 2. Thus, the first-moment write-in is exactly linear, whereas the second-moment write-in contains both linear and quadratic components. Define the composite gradient-to-parameter map ψt(g):=qt(β1mt−1+(1−β1)g,β2vt−1+(1−β2)g⊙2). _t(g):=q_t ( _1m_t-1+(1- _1)g,\; _2v_t-1+(1- _2)g 2 ). Then Γt=Dψt(gtc) _t=D _t(g_t^c), and δθt+1=Γtξt+rθ,t(ξt),δ _t+1= _t _t+r_θ,t( _t), where rθ,t(ξt):=ψt(gtc+ξt)−ψt(gtc)−Γtξt.r_θ,t( _t):= _t(g_t^c+ _t)- _t(g_t^c)- _t _t. If DψtD _t is κθ,t _θ,t-Lipschitz along the segment joining gtcg_t^c and gtc+ξtg_t^c+ _t, then ‖rθ,t(ξt)‖≤κθ,t2‖ξt‖2.\|r_θ,t( _t)\|≤ _θ,t2\| _t\|^2. Stacking the three state blocks yields δxt+1=Btξt+rB,t(ξt),δ x_t+1=B_t _t+r_B,t( _t), where Bt:=[Γt(1−β1)I2(1−β2)Diag(gtc)],rB,t(ξt):=[rθ,t(ξt)0(1−β2)ξt⊙2].B_t:= bmatrix _t\\[2.84526pt] (1- _1)I\\[2.84526pt] 2(1- _2)Diag(g_t^c) bmatrix,~~r_B,t( _t):= bmatrixr_θ,t( _t)\\[2.84526pt] 0\\[2.84526pt] (1- _2) _t 2 bmatrix. (1) The remainder contains two distinct nonlinear effects: the intrinsic quadratic write-in to the second-moment state and the nonlinearity of the adaptive parameter update. The current weight-decay term does not appear in BtB_t because the two trajectories share the same pre-shock parameter θt _t; weight decay instead enters subsequent transitions through Ds=I−ηsΛsD_s=I- _s _s. 3.4 Joint-State Propagation and Loss Readout Let fsf_s denote the exact state transition under future minibatch ℬsB_s. For a smooth network, set f~s:=fs f_s:=f_s. For a piecewise-smooth network, let f~s f_s denote the smooth extension associated with the activation pattern of the control trajectory at xscx_s^c. The tangent transition matrix is As:=Df~s(xsc).A_s:=D f_s(x_s^c). Although the paired trajectories use the same future minibatch, their gradients generally differ once their parameters diverge. Let g~s g_s denote the gradient map within the control-region smooth extension, and define Hstr:=Dg~s(θsc).H_s^tr:=D g_s( _s^c). When the loss is twice differentiable in this region, Hstr=∇θ2ℓ(θsc,ℬs).H_s^tr= _θ^2 ( _s^c;B_s). The corresponding first-order gradient variation is g~s(θsc+δθs)−g~s(θsc)=Hstrδθs+o(‖δθs‖). g_s( _s^c+δ _s)- g_s( _s^c)=H_s^trδ _s+o(\|δ _s\|). Define Ss:=2(1−β2)Diag(gsc)Hstr.S_s:=2(1- _2)Diag(g_s^c)H_s^tr. The tangent transition of the joint parameter–moment state is As=[Ds+ΓsHstrβ1Msβ2Vs(1−β1)Hstrβ1I0Ss0β2I].A_s= bmatrixD_s+ _sH_s^tr& _1M_s& _2V_s\\[2.84526pt] (1- _1)H_s^tr& _1I&0\\[2.84526pt] S_s&0& _2I bmatrix. (2) The off-diagonal blocks capture the two feedback paths θ→g→m→θ,θ→g⊙2→v→θ.θ→ g→ m→θ,~~θ→ g 2→ v→θ. Hence, the persistence parameters β1 _1 and β2 _2 describe memory within the individual moment states but do not by themselves determine the stability or finite-horizon gain of the coupled system. For indices a≥ba≥ b, define the ordered propagator Φa,b:=I,a=b,Aa−1Aa−2⋯Ab,a>b. _a,b:= casesI,&a=b,\\[2.84526pt] A_a-1A_a-2·s A_b,&a>b. cases In particular, Φt+h,t+1=At+h−1⋯At+1,h≥2, _t+h,t+1=A_t+h-1·s A_t+1,~~h≥ 2, with Φt+1,t+1=I _t+1,t+1=I. The matrices AsA_s vary along the future training trajectory and generally do not commute. Define the probe readout vector cs:=[∇Fs(θsc)00]∈ℝ3d.c_s:= bmatrix∇ F_s( _s^c)\\ 0\\ 0 bmatrix ^3d. The probe loss depends directly only on the parameter block, so a deviation stored in the moment states affects the probe loss only after it is converted into parameter motion. Theorem 3.1 (Finite-horizon directional response). Consider the scaled perturbation gts(α)=gtc+αξt.g_t^s(α)=g_t^c+α _t. Suppose that the control trajectory lies in the interior of the smooth regions used to define f~s f_s, that the corresponding AdamW transitions are differentiable along the trajectory, and that FsF_s is differentiable at the relevant control states. In particular, the second-moment coordinates involved in the derivatives satisfy v^s,ic>0 v_s,i^c>0. Let δxt+h(α)δ x_t+h(α) and δFt+h(α)δ F_t+h(α) denote the resulting paired state and probe-loss responses. Then, for every fixed h≥1h≥ 1, dαδxt+h(α)|α=0 . ddαδ x_t+h(α) |_α=0 =Φt+h,t+1Btξt, = _t+h,t+1B_t _t, dαδFt+h(α)|α=0 . ddαδ F_t+h(α) |_α=0 =ct+h⊤Φt+h,t+1Btξt. =c_t+h _t+h,t+1B_t _t. Define the horizon-h input–output operator t,h:=ct+h⊤Φt+h,t+1Bt.G_t,h:=c_t+h _t+h,t+1B_t. Then δFt+h(α)=αt,hξt+o(α),α→0.δ F_t+h(α)= _t,h _t+o(α),~~α→ 0. The complete proof is given in Appendix B. Theorem 3.1 gives the central ISO decomposition: a perturbation first enters the joint AdamW state through BtB_t, is propagated by the intervening dynamics through Φt+h,t+1 _t+h,t+1, and is finally observed through the loss readout ct+h⊤c_t+h . The future response therefore depends on the interaction of perturbation direction, optimizer dynamics, and future loss sensitivity rather than on any one of these quantities alone. Remark 3.2. A second-order analysis of the one-step response and the coordinate-invariance property of the ISO operator are provided in Appendix B.4 and Appendix B.5, respectively. Finite-horizon response summaries. For a horizon H, we summarize the response by its largest magnitude, its timing, and its sign: Mt,H:=max1≤h≤H|δFt+h|,ht,H⋆:=minargmax1≤h≤H|δFt+h|,M_t,H:= _1≤ h≤ H|δ F_t+h|,~~h_t,H := _1≤ h≤ H|δ F_t+h|, and st,H⋆:=sign(δFt+ht,H⋆).s_t,H :=sign (δ F_t+h_t,H ). Their tangent counterparts are obtained by replacing δFt+hδ F_t+h with t,hξtG_t,h _t. A delayed extremal response corresponds to ht,H⋆>1h_t,H >1. When only adverse loss increases are of interest, we additionally use Pt,H+:=max1≤h≤H[δFt+h]+,[a]+:=max(a,0).P_t,H^+:= _1≤ h≤ H[δ F_t+h]_+,~~[a]_+:= (a,0). This distinction is important for loss spikes: a large absolute response need not correspond to an increase in loss. Relation to loss spikes. When Ft+hF_t+h is chosen as the realized future minibatch loss, let Tt+hT_t+h denote the corresponding spike threshold and suppose Ft+h(θt+hc)≤Tt+h.F_t+h( _t+h^c)≤ T_t+h. The perturbed trajectory crosses the threshold at horizon h exactly when δFt+h>Tt+h−Ft+h(θt+hc).δ F_t+h>T_t+h-F_t+h( _t+h^c). Thus, a positive finite-horizon response contributes to a loss spike only when it exceeds the remaining margin to the threshold. 3.5 Delayed Expression Through AdamW Memory To isolate the role of optimizer memory, consider a scalar frozen-coefficient approximation of the joint AdamW dynamics over a short horizon: zh+1=A¯zh,A¯:=[abmbvdmβ10dv0β2],z_h+1= Az_h,~~ A:= bmatrixa&b_m&b_v\\ d_m& _1&0\\ d_v&0& _2 bmatrix, where zh:=[δθh,δmh−1,δvh−1]⊤.z_h:= bmatrixδ _h,δ m_h-1,δ v_h-1 bmatrix . The coefficients are the scalar counterparts of the blocks in Equation (2): a:=D+ΓHtr,bm:=β1M,bv:=β2V, a:=D+ H^tr,~~b_m:= _1M,~~b_v:= _2V, dm:=(1−β1)Htr,dv:=2(1−β2)gHtr. d_m:=(1- _1)H^tr,~~d_v:=2(1- _2)gH^tr. Here, bmb_m and bvb_v convert moment-state deviations into parameter motion, whereas dmd_m and dvd_v feed parameter-induced gradient changes back into the two moment states. To study how a perturbation stored in either memory channel becomes expressed in the parameter state, let eθ:=[1,0,0]⊤,em:=[0,1,0]⊤,ev:=[0,0,1]⊤,e_θ:=[1,0,0] ,~~e_m:=[0,1,0] ,~~e_v:=[0,0,1] , and define rk(h):=eθ⊤A¯h−1ek,k∈m,v.r_k(h):=e_θ A^\,h-1e_k,~~k∈\m,v\. A nonzero scalar probe sensitivity can be applied afterward as an output scaling; the analysis below concerns the magnitude and timing of the memory-to-parameter response. Proposition 3.3 (Finite-horizon memory-channel response). For k∈m,vk∈\m,v\, rm(1) r_m(1) =0, =0, rm(2) r_m(2) =bm, =b_m, rm(3) r_m(3) =bm(a+β1), =b_m(a+ _1), rv(1) r_v(1) =0, =0, rv(2) r_v(2) =bv, =b_v, rv(3) r_v(3) =bv(a+β2), =b_v(a+ _2), and rk(4)=bk(a2+aβk+βk2+bmdm+bvdv).r_k(4)=b_k (a^2+a _k+ _k^2+b_md_m+b_vd_v ). (3) If bk≠0b_k≠ 0 and |a+βk|>1,|a+ _k|>1, (4) then |rk(3)|>|rk(2)|.|r_k(3)|>|r_k(2)|. If, in addition, |a2+aβk+βk2+bmdm+bvdv|>|a+βk|, |a^2+a _k+ _k^2+b_md_m+b_vd_v |>|a+ _k|, (5) then |rk(4)|>|rk(3)|.|r_k(4)|>|r_k(3)|. These finite-horizon amplification conditions can hold even when ρ(A¯)<1ρ( A)<1, so asymptotic stability does not preclude transient growth in the response. The complete proof and spectral characterization of the frozen system are given in Appendix C. Proposition 3.3 makes the delay mechanism explicit. A perturbation stored entirely in m or v is initially absent from the parameter output and becomes visible only after the corresponding memory state feeds back into the parameter update. Its subsequent magnitude depends on both memory persistence and the return terms bmdm+bvdvb_md_m+b_vd_v. In particular, the second-moment feedback depends on the current momentum, gradient, and local curvature, so it can reinforce or oppose the evolving parameter deviation. The frozen model therefore illustrates how delayed and transiently amplified responses can arise from AdamW memory even when the local dynamics are asymptotically stable. The full ISO operator ct+h⊤Φt+h,t+1Btc_t+h _t+h,t+1B_t extends this mechanism to the high-dimensional, anisotropic, and time-varying dynamics of an actual training trajectory. 3.6 Approximation Error in Smooth and Piecewise-Smooth Networks The finite-horizon tangent model linearizes a nonlinear, time-varying training trajectory. Its approximation error has two sources: the smooth Taylor remainder within the local control region and, for piecewise-smooth networks, the defect caused by activation-pattern changes. Using the control-region extension f~s f_s, define rssm r_s^sm :=f~s(xsc+δxs)−f~s(xsc)−Asδxs, := f_s(x_s^c+δ x_s)- f_s(x_s^c)-A_sδ x_s, rssw r_s^sw :=fs(xsc+δxs)−f~s(xsc+δxs). :=f_s(x_s^c+δ x_s)- f_s(x_s^c+δ x_s). Since fsf_s and f~s f_s agree at the control state, the exact perturbation recursion is δxs+1=Asδxs+rssm+rssw.δ x_s+1=A_sδ x_s+r_s^sm+r_s^sw. (6) For smooth networks, rssw=0r_s^sw=0. More generally, it also vanishes whenever the paired states remain in the same activation region. Let the tangent prediction satisfy δx^t+1:=Btξt,δx^s+1:=Asδx^s. δ x_t+1:=B_t _t,~~ δ x_s+1:=A_s δ x_s. (7) Theorem 3.4 (Finite-horizon error decomposition). Let es:=δxs−δx^s.e_s:=δ x_s- δ x_s. Then, for every h≥1h≥ 1, et+h= e_t+h= Φt+h,t+1rB,t(ξt) _t+h,t+1r_B,t( _t) (8) +∑j=t+1t+h−1Φt+h,j+1(rjsm+rjsw). + _j=t+1^t+h-1 _t+h,j+1 (r_j^sm+r_j^sw ). If Df~jD f_j is LjL_j-Lipschitz along the segment joining xjcx_j^c and xjc+δxjx_j^c+δ x_j, then ‖et+h‖≤‖Φt+h,t+1‖‖rB,t(ξt)‖ \|e_t+h\|≤\| _t+h,t+1\|\|r_B,t( _t)\| (9) +∑j=t+1t+h−1∥Φt+h,j+1∥(Lj2∥δxj∥2+∥rjsw∥). + _j=t+1^t+h-1\| _t+h,j+1\| ( L_j2\|δ x_j\|^2+\|r_j^sw\| ). If, in addition, Ft+hF_t+h has an LF,t+hL_F,t+h-Lipschitz gradient along the segment joining θt+hc _t+h^c and θt+hs _t+h^s, then |δFt+h−ct+h⊤Φt+h,t+1Btξt| |δ F_t+h-c_t+h _t+h,t+1B_t _t | ≤‖ct+h‖‖et+h‖ ≤\|c_t+h\|\,\|e_t+h\| (10) +LF,t+h2‖δθt+h‖2. + L_F,t+h2\|δ _t+h\|^2. Theorem 3.4 shows that the same propagators that carry the first-order perturbation also propagate the approximation defects introduced at each step. Large finite-horizon gain can therefore amplify both the response of interest and the error of its tangent approximation. Corollary 3.5 (Fixed-horizon first-order accuracy). Consider the scaled perturbation αξtα _t and a fixed horizon H. Suppose that, in a neighborhood of the control trajectory, DψtD _t and Df~sD f_s are locally Lipschitz, the finite-horizon propagators are uniformly bounded, the probe objectives have locally Lipschitz gradients, and the switching defects satisfy ‖rssw‖≤Cssw‖δxs‖2,1≤s−t<H.\|r_s^sw\|≤ C_s^sw\|δ x_s\|^2,~~1≤ s-t<H. Then, for every 1≤h≤H1≤ h≤ H, δFt+h(α)=αct+h⊤Φt+h,t+1Btξt+O(α2),α→0.δ F_t+h(α)=α c_t+h _t+h,t+1B_t _t+O(α^2),~~α→ 0. The O(α2)O(α^2) constant may depend on the fixed horizon and control trajectory but not on α. For smooth networks the switching condition holds with Cssw=0C_s^sw=0. For piecewise-smooth networks, a sufficient activation-margin condition under which the aggregate switching defect is quadratic is given in Appendix D. The appendix also provides the complete proofs and a recursive error envelope. 4 Experiments Our experiments address two questions. First, does the proposed joint-state ISO model capture the finite-horizon response mechanism across increasingly realistic training systems? Second, although the pathwise ISO conditions on a realized future training sequence, does delayed influence retain structure that is already identifiable before that future unfolds? For mechanism validation, a control and a shock trajectory start from the same AdamW state, differ only in the gradient applied at step t, and then process the same future minibatches. With a fixed probe objective F, we write di,h=F(θi,t+hs)−F(θt+hc),d^i,h=ct+h⊤Φt+h,t+1Btξi,d_i,h=F(θ^s_i,t+h)-F(θ^c_t+h),~~ d_i,h=c_t+h _t+h,t+1B_t _i, (11) and summarize magnitude by Mi=max1≤h≤H|di,h|M_i= _1≤ h≤ H|d_i,h|. We evaluate trajectory fidelity, sign agreement, and within-system Spearman correlation with MiM_i. Candidates and horizons are nested observations: the controlled and neural-network studies aggregate within independent training systems, whereas the language-model study is reported descriptively over fixed model–dataset conditions. The second experiment holds the present state–shock pair fixed and instead resamples unseen future continuations. Complete protocols, estimators, and additional results are provided in Appendix E. 4.1 Experiment 1: Finite-Horizon Mechanism Validation Controlled quadratic systems. We begin with quadratic minibatch losses ℓs(θ)=12θ⊤Dsθ+12r‖Us⊤θ‖22+qs⊤θ, _s(θ)= 12θ D_sθ+ 12r\|U_s θ\|_2^2+q_s θ, (12) with d=512d=512, rank r=16r=16, temporally correlated minibatches, and three curvature regimes. Four seeds with eight independently generated systems each give 32 systems. After 40 burn-in updates, each system uses four reference minibatches, 16 candidate shocks, a common future of length H=32H=32, and both a standard and an anisotropic probe. A separate rotating-readout probe is used only in the exact-one-step-matched stress test. We vary α∈1/32,1/16,1/8,1/4,1/2,1α∈\1/32,1/16,1/8,1/4,1/2,1\. Table 1 shows that the tangent trajectory remains accurate over the local range. At α=1/8α=1/8, median NRMSE is 0.04830.0483 for the standard probe and 0.07190.0719 for the anisotropic probe, with perfect median sign agreement. Fitting |dh(α)−αd^h|∝αph|d_h(α)-α d_h| α^p_h over α≤1/4α≤ 1/4 gives ph≈2p_h≈ 2 throughout the horizon, with median R2>0.99998R^2>0.99998, matching the quadratic local remainder predicted by the theory. Table 1: Representative signed-trajectory fidelity in the controlled stage of Experiment 1. Entries are medians over 32 independent systems after candidate aggregation. Probe α NRMSE Rel. M err. Sign acc. Standard 1/321/32 0.0117 0.0052 1.000 1/81/8 0.0483 0.0216 1.000 1/41/4 0.1007 0.0445 1.000 Anisotropic 1/321/32 0.0173 0.0071 1.000 1/81/8 0.0719 0.0289 1.000 1/41/4 0.1473 0.0606 1.000 To separate future propagation from the immediate response, we construct shocks whose exact |di,1||d_i,1| values are matched without using any response at h>1h>1. The resulting within-system CV of |d1||d_1| is 1.10×10−121.10× 10^-12 for the standard probe and 6.70×10−126.70× 10^-12 for the rotating-readout probe, while the CV of future M remains 0.0920.092 and 0.4590.459. Full ISO recovers this future ordering with median Spearman correlations 0.9930.993 and 1.0001.000 (Table 2). Exact state interventions further separate the parameter, first-moment, and second-moment time scales, whose isolated responses peak near horizons 66, 1616, and 2626, respectively. Matched-first-displacement sweeps move the extremum later as β1 _1 or β2 _2 increases. These controls isolate delayed state propagation from the immediate parameter write. Nonconvex neural networks. We next apply the same paired-trajectory protocol to CIFAR-10 (15) using a 94,53894,538-parameter CNN–ReLU and an 855,050855,050-parameter MLP–GELU. Each architecture contributes 16 independently trained systems, with 12 candidate shocks per system and H=12H=12 after 100 burn-in updates. At α=0.0625α=0.0625, median trajectory NRMSE is 0.04580.0458 for CNN–ReLU and 0.03770.0377 for MLP–GELU; at α=0.25α=0.25 it is 0.10970.1097 and 0.15300.1530. The MLP error exponent remains 2.0012.001–2.0052.005 across horizons, whereas the CNN exponent decreases from 2.0172.017 at h=1h=1 to 1.1201.120 at h=12h=12 as activation-pattern differences increase, consistent with the switching term in the finite-horizon error decomposition. At full scale, Full ISO ranks future magnitude with correlations 0.8880.888 and 0.7620.762, compared with 0.5450.545 and 0.5660.566 for the exact one-step response. State interventions again show an early parameter response and a later first-moment response; increasing β1 _1 from 0.50.5 to 0.990.99 multiplies accumulated response by 14.1714.17 and 11.9611.96 in the two architectures. Among the 96 exact intervention trajectories used for the signed-response diagnostic, 57 extrema are positive and 39 are negative. Table 2: Within-system Spearman correlation with future magnitude M during mechanism validation. The controlled rows use exact-one-step-matched candidates; the neural rows use natural candidates. The one-step score is tied in the matched controlled stress test. Setting Full ISO Exact 1-step No propagation Gradient norm Quadratic, standard probe 0.993 – 0.354 -0.062 Quadratic, rotating-readout probe 1.000 – 0.806 0.776 CNN–ReLU 0.888 0.545 0.755 0.535 MLP–GELU 0.762 0.566 0.668 0.336 Pretrained language models. Finally, we evaluate Pythia-410M, Pythia-1B, and Pythia-1.4B (4) on WikiText-103 (20), OpenWebText (9), and CodeParrot (6). Each of the nine model–dataset systems is continued for 500 AdamW updates before measurement, producing nontrivial first- and second-moment states. Each condition then uses two reference minibatches, eight candidate shocks, seven common-future minibatches, and H=8H=8. At this scale we estimate the end-to-end ISO directional response numerically using centered finite differences of the probe logits, followed by the exact cross-entropy differential; we denote this quantity by ISO Tangent (FD). All 72 candidates pass the adjacent-scale consistency test, with median consistency NRMSE 0.003780.00378. Trajectory NRMSE is 0.03870.0387, 0.04770.0477, and 0.09460.0946 at α=0.0625α=0.0625, 0.1250.125, and 0.250.25, respectively, with median trajectory cosine above 0.99980.9998 and perfect sign accuracy over this local range. Across model sizes, local NRMSE is 0.10900.1090, 0.05190.0519, and 0.05670.0567 (Table 3), while trajectory cosine remains above 0.9990.999. ISO Tangent (FD) has positive rank correlation in all nine model–dataset conditions, with median correlations 0.7140.714, 0.8330.833, and 0.7620.762 across the three model sizes. It also recovers the full-scale extremum sign for 65 of 72 candidates. These results show that the signed finite-horizon tangent response remains locally accurate and informative across model scale and data domain. Table 3: Language-model scaling results in Experiment 1. Local metrics pool the three datasets and α≤0.25α≤ 0.25; rank correlations are medians over the three fixed data-domain conditions at each model size. Scale Local NRMSE Cosine ISO-FD ρ 0.41B 0.1090 0.99918 0.714 1.0B 0.0519 0.99995 0.833 1.4B 0.0567 0.99994 0.762 4.2 Experiment 2: Prospective Structure Under Unknown Futures The pathwise operator ct+h⊤Φt+h,t+1Btc_t+h _t+h,t+1B_t depends on the realized future training sequence and is therefore not, by itself, a present-time predictor. We ask a more basic question: if the current AdamW state and initiating shock are held fixed, does their finite-horizon effect remain candidate-specific when the unseen future minibatches are resampled? For each fixed system and candidate i, we draw K=32K=32 independent future continuations ωk _k, while sharing each continuation between its control and shock trajectory. Let Mi,k:=max1≤h≤H|di,h(ωk)|,μi:=1K∑k=1KMi,k.M_i,k:= _1≤ h≤ H|d_i,h( _k)|,~~ _i:= 1K _k=1^KM_i,k. (13) Within each system we summarize repeated-future structure by ΠH:=Vari(μi)Vari(μi)+i[Vark(Mi,k)]. _H:= Var_i( _i)Var_i( _i)+E_i[Var_k(M_i,k)]. (14) This is a protocol-specific variance ratio, not an information-theoretic fraction of predictable risk. We also report the median Spearman correlation between each branch ranking and the conditional-mean ranking. To test whether the structure can be extracted without observing any sampled future branch, we compare the exact one-step response, gradient norm, parameter-write norm, a present-frozen ISO that repeatedly applies one current reference-derived Jacobian with the h=1h=1 probe readout frozen, and a reference-surrogate ISO that deterministically rolls out only the current reference minibatches. Neither prospective ISO score uses future minibatches. Table 4: Prospective structure under 32 independently resampled future continuations. ΠH _H is defined in Eq. (14); branch ρ is the median single-future Spearman correlation with the conditional-mean candidate ranking. Present-score correlations target μi _i. All entries are medians over independent systems. System Candidates ΠH _H Branch ρ 1-step ρ Frozen ISO ρ Ref.-surrogate ρ Quadratic, standard Natural 0.801 0.919 0.801 0.659 0.372 Matched 0.730 0.909 – -0.066 -0.021 MLP–GELU Natural 0.771 0.862 0.661 0.734 0.262 Matched 0.921 0.955 – 0.941 0.752 CNN–ReLU Natural 0.716 0.855 0.619 0.752 0.601 Matched 0.700 0.872 – 0.755 0.811 Controlled repeated futures. In the standard quadratic regime, natural candidates give Π32=0.801 _32=0.801, but their exact one-step response already correlates 0.8010.801 with μi _i. We therefore repeat the analysis after exact one-step matching. The matched |d1||d_1| has median within-system CV 1.07×10−121.07× 10^-12, yet Π32 _32 remains 0.7300.730 and the median single-branch ranking correlation with μi _i is 0.9090.909. Thus the delayed response retains substantial candidate-specific structure after immediate magnitude is removed. However, this structure is not recovered by the simplest present-time compressions in the controlled system: correlations with μi _i are 0.0490.049 for gradient norm, −0.057-0.057 for parameter-write norm, and −0.066-0.066 for present-frozen ISO. The anisotropic-probe results show the same qualitative separation and are reported in Appendix E.3.1. Neural-network repeated futures. The same construction is applied to the fixed post-burn-in CIFAR-10 systems. Exact one-step matching succeeds for every candidate, with median CV 3.57×10−133.57× 10^-13 for MLP–GELU and 1.36×10−121.36× 10^-12 for CNN–ReLU. After matching, prospective structure remains strong: Π12=0.921 _12=0.921 for MLP–GELU and 0.7000.700 for CNN–ReLU. Unlike the controlled setting, present-time ISO scores now recover much of the conditional-mean ordering. Present-frozen ISO reaches median correlations 0.9410.941 and 0.7550.755, compared with gradient-norm correlations 0.0660.066 and 0.5000.500 for MLP–GELU and CNN–ReLU, respectively. The reference-surrogate ISO reaches 0.7520.752 and 0.8110.811. The stronger frozen-ISO agreement in the smooth MLP is consistent with the greater local tangent coherence observed in Experiment 1. Remark 4.1 (Why quadratic systems perform worse). The quadratic system is globally smooth, yet its frozen ISO performs poorly after one-step matching. A more plausible factor is how well the current local dynamics represent the future time-varying propagation. Longer horizons, transition and readout drift, and activation switching can all reduce this coherence. Taken together, Experiment 1 establishes the pathwise finite-horizon mechanism, whereas Experiment 2 shows that its delayed effects are not created entirely by the subsequently realized minibatches: substantial candidate-specific structure can already be present at the perturbation time. Whether that structure admits an accurate present-time representation is regime-dependent, as illustrated by the contrast between the controlled, MLP–GELU, and CNN–ReLU results. Characterizing the conditions for such prospective identifiability is distinct from the pathwise mechanism studied here. 5 Conclusion and Discussion We studied delayed minibatch influence in AdamW through a finite-horizon input–state–output formulation that tracks how perturbations enter optimizer state, propagate through future updates, and appear in later losses. The resulting tangent model is supported across controlled systems, neural networks, and pretrained language models, while repeated-future experiments show that delayed influence can also contain prospective structure. The main limitations are that the exact ISO is pathwise and its present-time approximation may degrade under nonlinear dynamics, activation switching, and future dynamical drift. Future work should characterize when such prospective influence is identifiable from the current optimizer state, ideally through necessary and sufficient conditions, and determine whether this can support reliable training-time warning or control. Acknowledgments We gratefully acknowledge Hongqian Huang for providing the computational resources used in this work. References Ahn et al. (2024) K. Ahn, Z. Zhang, Y. Kook, and Y. Dai Understanding Adam optimizer via online learning of updates: Adam is FTRL in disguise. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 619–640. External Links: Link Cited by: §1, §2. Bae et al. (2024) J. Bae, W. Lin, J. Lorraine, and R. B. Grosse Training data attribution via approximate unrolling. In Advances in Neural Information Processing Systems, Vol. 37, p. 66647–66686. External Links: Link Cited by: §2. Bai et al. (2025) Z. Bai, Z. Zhou, J. Zhao, X. Li, Z. Li, F. Xiong, H. Yang, Y. Zhang, and Z. J. Xu Adaptive preconditioners trigger loss spikes in adam. arXiv preprint arXiv:2506.04805. External Links: Link Cited by: §2. Biderman et al. (2023) S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. Van Der Wal Pythia: a suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, Vol. 202, p. 2397–2430. External Links: Link Cited by: §4.1. Chen et al. (2021) Y. Chen, B. Li, H. Yu, P. Wu, and C. Miao HyDRA: hypergradient data relevance analysis for interpreting deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, p. 7081–7089. External Links: Link Cited by: §2. CodeParrot (2022) CodeParrot CodeParrot clean. Note: https://huggingface.co/datasets/codeparrot/codeparrot-cleanHugging Face dataset Cited by: §4.1. Das et al. (2024) R. Das, N. Agarwal, S. Sanghavi, and I. S. Dhillon Towards quantifying the preconditioning effect of Adam. arXiv preprint arXiv:2402.07114. External Links: Link Cited by: §1. Deng et al. (2026) J. Deng, P. Hu, S. Jin, H. Lu, J. T. Wang, S. Zhang, and J. W. Ma How faithful is trajectory-based data attribution? error sources, remedies, and practical guidelines. arXiv preprint arXiv:2605.18814. External Links: Link Cited by: §1, §2. Gokaslan and Cohen (2019) A. Gokaslan and V. Cohen OpenWebText corpus. Note: https://skylion007.github.io/OpenWebTextCorpus/ Cited by: §4.1. Grosse et al. (2023) R. Grosse, J. Bae, C. Anil, N. Elhage, A. Tamkin, A. Tajdini, B. Steiner, D. Li, E. Durmus, E. Perez, E. Hubinger, K. Lukošiūtė, K. Nguyen, N. Joseph, S. McCandlish, J. Kaplan, and S. R. Bowman Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296. External Links: Link Cited by: §2. Hara et al. (2019) S. Hara, A. Nitanda, and T. Maehara Data cleansing for models trained with SGD. In Advances in Neural Information Processing Systems, Vol. 32. External Links: Link Cited by: §2. Huang et al. (2025) T. Huang, Z. Zhu, G. Jin, L. Liu, Z. Wang, and S. Liu SPAM: spike-aware Adam with momentum reset for stable LLM training. arXiv preprint arXiv:2501.06842. External Links: Link Cited by: §1, §2. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. In International Conference on Learning Representations, External Links: Link Cited by: §2. Koh and Liang (2017) P. W. Koh and P. Liang Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 70, p. 1885–1894. External Links: Link Cited by: §2. Krizhevsky (2009) A. Krizhevsky Learning multiple layers of features from tiny images. Technical report University of Toronto. External Links: Link Cited by: §4.1. Kwon et al. (2024) Y. Kwon, E. Wu, K. Wu, and J. Zou DataInf: efficiently estimating data influence in LoRA-tuned LLMs and diffusion models. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2. Lin et al. (2024) W. Lin, F. Dangel, R. Eschenhagen, J. Bae, R. E. Turner, and A. Makhzani Can we remove the square-root in adaptive gradient methods? a second-order perspective. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 29949–29973. External Links: Link Cited by: §2. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §A.1, §2, §3.2. Ma et al. (2022) C. Ma, L. Wu, and W. E A qualitative study of the dynamic behavior for adaptive gradient algorithms. In Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, Proceedings of Machine Learning Research, Vol. 145, p. 671–692. External Links: Link Cited by: §2. Merity et al. (2016) S. Merity, C. Xiong, J. Bradbury, and R. Socher Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. External Links: Link Cited by: §4.1. Park et al. (2023) S. M. Park, K. Georgiev, A. Ilyas, G. Leclerc, and A. Madry TRAK: attributing model behavior at scale. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 27074–27113. External Links: Link Cited by: §2. Pruthi et al. (2020) G. Pruthi, F. Liu, S. Kale, and M. Sundararajan Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, Vol. 33, p. 19920–19930. External Links: Link Cited by: §2. Reddi et al. (2018) S. J. Reddi, S. Kale, and S. Kumar On the convergence of adam and beyond. In International Conference on Learning Representations, External Links: Link Cited by: §2. Schioppa et al. (2022) A. Schioppa, P. Zablotskaia, D. Vilar, and A. Sokolov Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, p. 8179–8186. External Links: Link Cited by: §2. Sweeney (2026) J. Sweeney Optimizer memory makes shuffle order a first-order source of fine-tuning noise. arXiv preprint arXiv:2606.29554. External Links: Link Cited by: §1, §2. Wang et al. (2025a) G. Wang, S. Li, C. Chen, J. Zeng, J. Yang, T. Sun, Y. Ma, D. Yu, and L. Shen AdaGC: improving training stability for large language model pretraining. arXiv preprint arXiv:2502.11034. External Links: Link Cited by: §2. Wang et al. (2025b) J. T. Wang, D. Song, J. Zou, P. Mittal, and R. Jia Capturing the temporal dependence of training data influence. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Wortsman et al. (2024) M. Wortsman, P. J. Liu, L. Xiao, K. E. Everett, A. A. Alemi, B. Adlam, J. D. Co-Reyes, I. Gur, A. Kumar, R. Novak, J. Pennington, J. Sohl-Dickstein, K. Xu, J. Lee, J. Gilmer, and S. Kornblith Small-scale proxies for large-scale transformer training instabilities. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Xia et al. (2024) M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen LESS: selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 54104–54132. External Links: Link Cited by: §2. Xie and Li (2024) S. Xie and Z. Li Implicit bias of AdamW: ℓ∞ _∞-norm constrained optimization. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 54488–54510. External Links: Link Cited by: §2. Zhu et al. (2024) L. Zhu, C. Liu, A. Radhakrishnan, and M. Belkin Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 62476–62509. External Links: Link Cited by: §1, §2. Appendix A Detailed Derivation of the Minibatch Shock Write-In This appendix derives the input operator BtB_t, its nonlinear remainder, and the second-order gradient-to-parameter map used in the one-step output expansion. A.1 AdamW Update as a State Transition For a fixed step s, define ρ1,s:=1−β1s,ρ2,s:=1−β2s. _1,s:=1- _1^s,~~ _2,s:=1- _2^s. Following AdamW (18), given the pre-update state xs=[θsms−1vs−1],x_s= bmatrix _s\\ m_s-1\\ v_s-1 bmatrix, and an input gradient g∈ℝdg ^d, the moment states after the update are ms(g) m_s(g) :=β1ms−1+(1−β1)g, := _1m_s-1+(1- _1)g, (15) vs(g) v_s(g) :=β2vs−1+(1−β2)g⊙2. := _2v_s-1+(1- _2)g 2. (16) The bias-corrected moments are m^s(g):=ms(g)ρ1,s,v^s(g):=vs(g)ρ2,s, m_s(g):= m_s(g) _1,s,~~ v_s(g):= v_s(g) _2,s, and the adaptive parameter displacement is qs(m,v):=−ηs(mρ1,s)⊘(vρ2,s+ϵ).q_s(m,v):=- _s ( m _1,s ) ( v _2,s+ 1 ). Define the composite gradient-to-parameter map ψs(g):=qs(ms(g),vs(g)). _s(g):=q_s (m_s(g),v_s(g) ). At the shock step, θt+1(g)=Dtθt+ψt(g). _t+1(g)=D_t _t+ _t(g). Because the control and shock trajectories share the same pre-update parameter θt _t, the term DtθtD_t _t cancels from their difference. All derivatives below are evaluated along the control trajectory. Whenever a derivative with respect to the raw second-moment state is used, we assume v^s,ic>0 v_s,i^c>0 for the corresponding coordinates. A.2 Moment-State Sensitivities Write ds:=v^sc+ϵ.d_s:= v_s^c+ 1. Since the adaptive map is coordinate-separable, its derivatives with respect to m and v are diagonal. For coordinate i, qs,i(mi,vi)=−ηsmi/ρ1,svi/ρ2,s+ϵ.q_s,i(m_i,v_i)=- _s m_i/ _1,s v_i/ _2,s+ε. Differentiating with respect to mim_i gives ∂qs,i∂mi=−ηsρ1,sds,i. ∂ q_s,i∂ m_i=- _s _1,sd_s,i. Hence Ms=∂qs∂ms=−ηsρ1,sDiag(ds−1).M_s= ∂ q_s∂ m_s=- _s _1,sDiag(d_s^-1). (17) For the derivative with respect to viv_i, define rs,i:=v^s,ic=vs,icρ2,s.r_s,i:= v_s,i^c= v_s,i^c _2,s. Using ∂vi/ρ2,s∂vi=12ρ2,srs,i, ∂ v_i/ _2,s∂ v_i= 12 _2,s r_s,i, we obtain ∂qs,i∂vi ∂ q_s,i∂ v_i =ηsms,ic/ρ1,s2ρ2,srs,i(rs,i+ϵ)2 = _s m_s,i^c/ _1,s2 _2,s r_s,i ( r_s,i+ε )^2 =ηsm^s,ic2ρ2,sv^s,icds,i2. = _s m_s,i^c2 _2,s v_s,i^cd_s,i^2. Therefore Vs=∂qs∂vs=ηsDiag(m^sc2ρ2,sv^sc⊙ds⊙2).V_s= ∂ q_s∂ v_s= _sDiag ( m_s^c2 _2,s v_s^c d_s 2 ). (18) Equation (17) shows that MsM_s is negative diagonal. The sign of the iith diagonal entry of VsV_s is the sign of m^s,ic m_s,i^c. A.3 Gradient-to-Parameter Jacobian The derivatives of the moment maps in Equations (15)–(16) are Dms(g)[u] Dm_s(g)[u] =(1−β1)u, =(1- _1)u, Dvs(g)[u] Dv_s(g)[u] =2(1−β2)Diag(g)u. =2(1- _2)Diag(g)u. Applying the chain rule to ψs(g)=qs(ms(g),vs(g)) _s(g)=q_s(m_s(g),v_s(g)) gives Dψs(gsc)[u] D _s(g_s^c)[u] =MsDms(gsc)[u]+VsDvs(gsc)[u] =M_sDm_s(g_s^c)[u]+V_sDv_s(g_s^c)[u] =[(1−β1)Ms+2(1−β2)VsDiag(gsc)]u. = [(1- _1)M_s+2(1- _2)V_sDiag(g_s^c) ]u. Thus Γs:=Dψs(gsc)=(1−β1)Ms+2(1−β2)VsDiag(gsc). _s:=D _s(g_s^c)=(1- _1)M_s+2(1- _2)V_sDiag(g_s^c). (19) The two terms correspond to the differential paths g→m→θ,g→v→θ.g→ m→θ,~~g→ v→θ. A.4 Exact Shock-Step State Difference At the shock step, gts=gtc+ξt.g_t^s=g_t^c+ _t. Subtracting the first-moment updates gives δmt δ m_t =mts−mtc =m_t^s-m_t^c (20) =(1−β1)(gts−gtc) =(1- _1) (g_t^s-g_t^c ) =(1−β1)ξt. =(1- _1) _t. For the second moment, δvt δ v_t =(1−β2)[(gtc+ξt)⊙2−(gtc)⊙2] =(1- _2) [(g_t^c+ _t) 2-(g_t^c) 2 ] (21) =2(1−β2)gtc⊙ξt+(1−β2)ξt⊙2 =2(1- _2)g_t^c _t+(1- _2) _t 2 =2(1−β2)Diag(gtc)ξt+(1−β2)ξt⊙2. =2(1- _2)Diag(g_t^c) _t+(1- _2) _t 2. The parameter difference is δθt+1 δ _t+1 =ψt(gtc+ξt)−ψt(gtc) = _t(g_t^c+ _t)- _t(g_t^c) (22) =Γtξt+rθ,t(ξt), = _t _t+r_θ,t( _t), where rθ,t(ξ):=ψt(gtc+ξ)−ψt(gtc)−Γtξ.r_θ,t(ξ):= _t(g_t^c+ξ)- _t(g_t^c)- _tξ. Stacking Equations (20), (21), and (22) gives δxt+1=Btξt+rB,t(ξt),δ x_t+1=B_t _t+r_B,t( _t), where Bt:=[Γt(1−β1)I2(1−β2)Diag(gtc)]B_t:= bmatrix _t\\[2.84526pt] (1- _1)I\\[2.84526pt] 2(1- _2)Diag(g_t^c) bmatrix and rB,t(ξ):=[rθ,t(ξ)0(1−β2)ξ⊙2].r_B,t(ξ):= bmatrixr_θ,t(ξ)\\[2.84526pt] 0\\[2.84526pt] (1- _2)ξ 2 bmatrix. If DψtD _t is κθ,t _θ,t-Lipschitz in a neighborhood containing the segment gtc+τξt:0≤τ≤1, \g_t^c+τ _t:0≤τ≤ 1 \, Taylor’s theorem gives ‖rθ,t(ξt)‖≤κθ,t2‖ξt‖2.\|r_θ,t( _t)\|≤ _θ,t2\| _t\|^2. Moreover, ‖ξ⊙2‖2=(∑iξi4)1/2≤∑iξi2=‖ξ‖22.\|ξ 2\|_2= ( _i _i^4 )^1/2≤ _i _i^2=\|ξ\|_2^2. Consequently, ‖rB,t(ξt)‖2≤(κθ,t2+1−β2)‖ξt‖22.\|r_B,t( _t)\|_2≤ ( _θ,t2+1- _2 )\| _t\|_2^2. (23) For a scaled perturbation αξtα _t, Equation (23) implies ‖rB,t(αξt)‖=O(α2).\|r_B,t(α _t)\|=O(α^2). A.5 Second Derivative of the Gradient-to-Parameter Map For completeness, we derive t:=D2ψt(gtc),Q_t:=D^2 _t(g_t^c), which is used in the one-step second-order output expansion. Let a1:=1−β1,a2:=1−β2.a_1:=1- _1,~~a_2:=1- _2. The moment maps satisfy Dmt(g)[u] Dm_t(g)[u] =a1u, =a_1u, D2mt(g)[u,w] D^2m_t(g)[u,w] =0, =0, Dvt(g)[u] Dv_t(g)[u] =2a2Diag(g)u, =2a_2Diag(g)u, D2vt(g)[u,w] D^2v_t(g)[u,w] =2a2(u⊙w). =2a_2(u w). Because qtq_t is coordinate-separable, tQ_t is also coordinate-separable. Define st,i:=v^t,ic,dt,i:=st,i+ϵ.s_t,i:= v_t,i^c,~~d_t,i:=s_t,i+ε. The nonzero second derivatives of qt,iq_t,i are qmv,t,i q_mv,t,i :=∂2qt,i∂mi∂vi=ηt2ρ1,tρ2,tst,idt,i2, := ∂^2q_t,i∂ m_i∂ v_i= _t2 _1,t _2,ts_t,id_t,i^2, qvv,t,i q_v,t,i :=∂2qt,i∂vi2=−ηtm^t,ic(3st,i+ϵ)4ρ2,t2st,i3dt,i3. := ∂^2q_t,i∂ v_i^2=- _t m_t,i^c(3s_t,i+ε)4 _2,t^2s_t,i^3d_t,i^3. Also, qv,t,i=[Vt]ii.q_v,t,i=[V_t]_i. The second-order chain rule gives t[u,w]= _t[u,w]= Dmv2qt[Dmt[u],Dvt[w]] D_mv^2q_t [Dm_t[u],Dv_t[w] ] +Dvm2qt[Dvt[u],Dmt[w]] +D_vm^2q_t [Dv_t[u],Dm_t[w] ] +Dvv2qt[Dvt[u],Dvt[w]] +D_v^2q_t [Dv_t[u],Dv_t[w] ] +VtD2vt[u,w]. +V_tD^2v_t[u,w]. Coordinatewise, [t[u,w]]i=χt,iuiwi,[Q_t[u,w]]_i= _t,iu_iw_i, where χt,i:= _t,i:= 4a1a2gt,icqmv,t,i 4a_1a_2g_t,i^cq_mv,t,i +4a22(gt,ic)2qvv,t,i+2a2[Vt]ii. +4a_2^2(g_t,i^c)^2q_v,t,i+2a_2[V_t]_i. Therefore, for a scaled perturbation αξtα _t, δθt+1(α)=αΓtξt+α22t[ξt,ξt]+o(α2).δ _t+1(α)=α _t _t+ α^22Q_t[ _t, _t]+o(α^2). (24) Appendix B Proof of the Finite-Horizon Directional Response This appendix derives the joint AdamW transition Jacobian, proves Theorem 3.1, and records several properties of the resulting input–output operator. B.1 Control-Region State Transition Let fsf_s denote the exact AdamW transition under future minibatch ℬsB_s: xs+1=fs(xs).x_s+1=f_s(x_s). For a smooth network, define f~s:=fs f_s:=f_s. For a piecewise-smooth network, let f~s f_s denote the smooth extension associated with the activation pattern of the control trajectory at xscx_s^c. The control-region tangent matrix is As:=Df~s(xsc).A_s:=D f_s(x_s^c). All quantities in the following block derivation are evaluated at the control state and its corresponding future minibatch. Let g~s(θ) g_s(θ) denote the gradient map induced by the same control-region smooth extension, and define Hstr:=Dg~s(θsc).H_s^tr:=D g_s( _s^c). When the loss is twice differentiable in the control region, Hstr=∇θ2ℓ(θsc,ℬs).H_s^tr= _θ^2 ( _s^c;B_s). B.2 Blockwise Derivation of the Joint Jacobian For a generic state x=[θ,m−,v−]⊤,x=[θ,m_-,v_-] , the control-region transition has components m+ m^+ =β1m−+(1−β1)g~s(θ), = _1m_-+(1- _1) g_s(θ), v+ v^+ =β2v−+(1−β2)g~s(θ)⊙2, = _2v_-+(1- _2) g_s(θ) 2, θ+ θ^+ =Dsθ+qs(m+,v+). =D_sθ+q_s(m^+,v^+). The derivatives of the first-moment update are ∂m+∂θ ∂ m^+∂θ =(1−β1)Hstr, =(1- _1)H_s^tr, ∂m+∂m− ∂ m^+∂ m_- =β1I, = _1I, ∂m+∂v− ∂ m^+∂ v_- =0. =0. For the second moment, D[g~s(θ)⊙2]=2Diag(g~s(θ))Dg~s(θ),D [ g_s(θ) 2 ]=2Diag ( g_s(θ) )D g_s(θ), so ∂v+∂θ ∂ v^+∂θ =2(1−β2)Diag(gsc)Hstr:=Ss, =2(1- _2)Diag(g_s^c)H_s^tr:=S_s, ∂v+∂m− ∂ v^+∂ m_- =0, =0, ∂v+∂v− ∂ v^+∂ v_- =β2I. = _2I. For the parameter update, ∂θ+∂θ= ∂θ^+∂θ= Ds+Ms∂m+∂θ+Vs∂v+∂θ D_s+M_s ∂ m^+∂θ+V_s ∂ v^+∂θ = = Ds+(1−β1)MsHstr D_s+(1- _1)M_sH_s^tr +2(1−β2)VsDiag(gsc)Hstr +2(1- _2)V_sDiag(g_s^c)H_s^tr = = Ds+ΓsHstr. D_s+ _sH_s^tr. Similarly, ∂θ+∂m− ∂θ^+∂ m_- =β1Ms, = _1M_s, ∂θ+∂v− ∂θ^+∂ v_- =β2Vs. = _2V_s. Combining the nine blocks gives As=[Ds+ΓsHstrβ1Msβ2Vs(1−β1)Hstrβ1I0Ss0β2I].A_s= bmatrixD_s+ _sH_s^tr& _1M_s& _2V_s\\[2.84526pt] (1- _1)H_s^tr& _1I&0\\[2.84526pt] S_s&0& _2I bmatrix. (25) For later use, recall the general propagator convention Φa,b:=I,a=b,Aa−1Aa−2⋯Ab,a>b. _a,b:= casesI,&a=b,\\[2.84526pt] A_a-1A_a-2·s A_b,&a>b. cases B.3 Proof of Theorem 3.1 Consider gts(α)=gtc+αξt.g_t^s(α)=g_t^c+α _t. At α=0α=0, the shock and control trajectories coincide. By assumption, the control states lie in the interior of the smooth regions used to define the transitions f~t+1,…,f~t+h−1 f_t+1,…, f_t+h-1. For a fixed horizon, continuity of the trajectory implies that there exists α0>0 _0>0 such that, for sufficiently small |α|<α0|α|< _0, the perturbed trajectory follows the same sequence of local smooth extensions. The differentiability assumptions on the AdamW transition, including the required positivity of the second-moment coordinates, ensure that the Jacobians used below are well defined. Define x˙s:=dαδxs(α)|α=0. x_s:= . ddαδ x_s(α) |_α=0. By Appendix A, x˙t+1=Btξt. x_t+1=B_t _t. For every future step s≥t+1s≥ t+1, xs+1s(α)=f~s(xss(α)),x_s+1^s(α)= f_s (x_s^s(α) ), while xs+1c=f~s(xsc).x_s+1^c= f_s (x_s^c ). Differentiating at α=0α=0 yields x˙s+1=Df~s(xsc)x˙s=Asx˙s. x_s+1=D f_s(x_s^c) x_s=A_s x_s. (26) Repeated application gives x˙t+h x_t+h =At+h−1At+h−2⋯At+1Btξt =A_t+h-1A_t+h-2·s A_t+1B_t _t =Φt+h,t+1Btξt. = _t+h,t+1B_t _t. For h=1h=1, Φt+1,t+1=I _t+1,t+1=I. The probe function at horizon t+ht+h is shared by the paired trajectories, so δFt+h(α)=Ft+h(θt+hs(α))−Ft+h(θt+hc).δ F_t+h(α)=F_t+h ( _t+h^s(α) )-F_t+h ( _t+h^c ). Differentiating at α=0α=0 gives dαδFt+h(α)|α=0 . ddαδ F_t+h(α) |_α=0 =∇Ft+h(θt+hc)⊤θ˙t+h =∇ F_t+h ( _t+h^c ) θ_t+h =ct+h⊤x˙t+h =c_t+h x_t+h =ct+h⊤Φt+h,t+1Btξt. =c_t+h _t+h,t+1B_t _t. This proves both directional identities in Theorem 3.1. The first-order expansion δFt+h(α)=αct+h⊤Φt+h,t+1Btξt+o(α)δ F_t+h(α)=α c_t+h _t+h,t+1B_t _t+o(α) follows directly from differentiability at α=0α=0. B.4 Complete One-Step Second-Order Output Expansion We now recover the one-step second-order geometry that is omitted from the main text. From Equation (24), δθt+1(α)=αpt+α22ut+o(α2),δ _t+1(α)=α p_t+ α^22u_t+o(α^2), where pt:=Γtξt,ut:=t[ξt,ξt].p_t:= _t _t,~~u_t:=Q_t[ _t, _t]. Assume that Ft+1F_t+1 is twice differentiable in the relevant local region and define Ht+1F:=∇2Ft+1(θt+1c).H_t+1^F:=∇^2F_t+1 ( _t+1^c ). Taylor expansion around θt+1c _t+1^c gives δFt+1(α)= δ F_t+1(α)= ∇Ft+1(θt+1c)⊤δθt+1(α) ∇ F_t+1 ( _t+1^c ) δ _t+1(α) +12δθt+1(α)⊤Ht+1Fδθt+1(α) + 12δ _t+1(α) H_t+1^Fδ _t+1(α) +o(‖δθt+1(α)‖2). +o (\|δ _t+1(α)\|^2 ). The linear output contribution is ∇Ft+1⊤δθt+1(α)= ∇ F_t+1 δ _t+1(α)= α∇Ft+1⊤pt α∇ F_t+1 p_t +α22∇Ft+1⊤ut+o(α2), + α^22∇ F_t+1 u_t+o(α^2), where the gradients are evaluated at θt+1c _t+1^c. The quadratic output contribution satisfies 12δθt+1(α)⊤Ht+1Fδθt+1(α)=α22pt⊤Ht+1Fpt+o(α2). 12δ _t+1(α) H_t+1^Fδ _t+1(α)= α^22p_t H_t+1^Fp_t+o(α^2). Therefore δFt+1(α)= δ F_t+1(α)= α∇Ft+1(θt+1c)⊤Γtξt α∇ F_t+1 ( _t+1^c ) _t _t (27) +α22[(Γtξt)⊤Ht+1F(Γtξt) + α^22 [( _t _t) H_t+1^F( _t _t) +∇Ft+1(θt+1c)⊤t[ξt,ξt]] +∇ F_t+1 ( _t+1^c ) Q_t[ _t, _t] ] +o(α2). +o(α^2). The first second-order term in Equation (27) is the probe-curvature contribution induced by the first-order parameter displacement. Equivalently, it is generated by the effective curvature operator Γt⊤Ht+1FΓt. _t H_t+1^F _t. The second term is the output effect of the nonlinear AdamW write-in itself. The training-batch Hessian HstrH_s^tr governs the subsequent state propagation, whereas HsFH_s^F describes curvature of the probe output. Neither curvature term by itself determines the sign of the response. B.5 Coordinate Invariance of the Input–Output Operator Let x~s:=Tsxs x_s:=T_sx_s for invertible matrices TsT_s. The transformed state transition is A~s:=Ts+1AsTs−1. A_s:=T_s+1A_sT_s^-1. Hence Φ~t+h,t+1 _t+h,t+1 =A~t+h−1⋯A~t+1 = A_t+h-1·s A_t+1 =Tt+hAt+h−1Tt+h−1−1⋯Tt+2At+1Tt+1−1 =T_t+hA_t+h-1T_t+h-1^-1·s T_t+2A_t+1T_t+1^-1 =Tt+hΦt+h,t+1Tt+1−1. =T_t+h _t+h,t+1T_t+1^-1. The input and output maps transform as B~t=Tt+1Bt,c~t+h⊤=ct+h⊤Tt+h−1. B_t=T_t+1B_t,~~ c_t+h =c_t+h T_t+h^-1. Therefore c~t+h⊤Φ~t+h,t+1B~t c_t+h _t+h,t+1 B_t =ct+h⊤Tt+h−1Tt+hΦt+h,t+1Tt+1−1Tt+1Bt =c_t+h T_t+h^-1T_t+h _t+h,t+1T_t+1^-1T_t+1B_t =ct+h⊤Φt+h,t+1Bt. =c_t+h _t+h,t+1B_t. Thus the finite-horizon input–output operator is invariant under invertible state reparameterization. Internal state-gain quantities can depend on the relative scaling chosen for the parameter and moment blocks, whereas the signed scalar input–output response does not. B.6 Additional Finite-Horizon Response Summaries The main text uses the maximum response magnitude, its timing, its sign, and the largest positive excursion as the primary finite-horizon summaries. We record additional cumulative and direction-specific quantities here. The accumulated absolute response is AREt,H:=∑h=1H|δFt+h|.ARE_t,H:= _h=1^H|δ F_t+h|. While Mt,HM_t,H measures the largest deviation over the horizon, AREt,HARE_t,H measures the total magnitude accumulated along the response trajectory. The largest negative excursion is Pt,H−:=max1≤h≤H[−δFt+h]+.P_t,H^-:= _1≤ h≤ H[-δ F_t+h]_+. Together, Pt,H+P_t,H^+ and Pt,H−P_t,H^- distinguish the largest positive and negative deviations from the control trajectory. Their accumulated counterparts are AELt,H+:=∑h=1H[δFt+h]+,AELt,H−:=∑h=1H[−δFt+h]+.AEL_t,H^+:= _h=1^H[δ F_t+h]_+,~~AEL_t,H^-:= _h=1^H[-δ F_t+h]_+. The corresponding tangent quantities are obtained by replacing δFt+hδ F_t+h with t,hξtG_t,h _t. For example, ARE^t,H:=∑h=1H|t,hξt|, ARE_t,H:= _h=1^H|G_t,h _t|, and P^t,H−:=max1≤h≤H[−t,hξt]+. P_t,H^-:= _1≤ h≤ H[-G_t,h _t]_+. These quantities are secondary summaries of the same signed finite-horizon response rather than separate dynamical objects. Appendix C Frozen AdamW Memory-Channel Analysis This appendix proves Proposition 3.3 and gives spectral, transfer-function, and feedback-loop characterizations of the frozen three-state model. C.1 Short-Horizon Responses Consider A¯=[abmbvdmβ10dv0β2], A= bmatrixa&b_m&b_v\\ d_m& _1&0\\ d_v&0& _2 bmatrix, with normalized parameter readout eθ:=[100],e_θ:= bmatrix1\\ 0\\ 0 bmatrix, and memory-channel basis vectors em:=[010],ev:=[001].e_m:= bmatrix0\\ 1\\ 0 bmatrix,~~e_v:= bmatrix0\\ 0\\ 1 bmatrix. For k∈m,vk∈\m,v\, define rk(h):=eθ⊤A¯h−1ek.r_k(h):=e_θ A^h-1e_k. If the scalar probe sensitivity at the frozen operating point is γF≠0 _F≠ 0, then the corresponding first-order probe-loss response is γFrk(h) _Fr_k(h). Thus, rk(h)r_k(h) isolates the timing and amplification produced by the memory-to-parameter dynamics, while the probe readout supplies the final output scaling and sign. At h=1h=1, rm(1)=rv(1)=0.r_m(1)=r_v(1)=0. At h=2h=2, A¯em=[bmβ10],A¯ev=[bv0β2], Ae_m= bmatrixb_m\\ _1\\ 0 bmatrix,~~ Ae_v= bmatrixb_v\\ 0\\ _2 bmatrix, so rm(2)=bm,rv(2)=bv.r_m(2)=b_m,~~r_v(2)=b_v. Applying A¯ A again, A¯2em=[bm(a+β1)bmdm+β12bmdv], A^2e_m= bmatrixb_m(a+ _1)\\ b_md_m+ _1^2\\ b_md_v bmatrix, and A¯2ev=[bv(a+β2)bvdmbvdv+β22]. A^2e_v= bmatrixb_v(a+ _2)\\ b_vd_m\\ b_vd_v+ _2^2 bmatrix. Hence rm(3)=bm(a+β1),rv(3)=bv(a+β2).r_m(3)=b_m(a+ _1),~~r_v(3)=b_v(a+ _2). A third multiplication gives rm(4) r_m(4) =abm(a+β1)+bm(bmdm+β12)+bvbmdv =a\,b_m(a+ _1)+b_m(b_md_m+ _1^2)+b_vb_md_v =bm(a2+aβ1+β12+bmdm+bvdv), =b_m (a^2+a _1+ _1^2+b_md_m+b_vd_v ), and rv(4) r_v(4) =abv(a+β2)+bmbvdm+bv(bvdv+β22) =a\,b_v(a+ _2)+b_mb_vd_m+b_v(b_vd_v+ _2^2) =bv(a2+aβ2+β22+bmdm+bvdv). =b_v (a^2+a _2+ _2^2+b_md_m+b_vd_v ). Therefore, rk(4)=bk(a2+aβk+βk2+bmdm+bvdv).r_k(4)=b_k (a^2+a _k+ _k^2+b_md_m+b_vd_v ). If bk≠0b_k≠ 0, then |rk(3)|>|rk(2)||r_k(3)|>|r_k(2)| is equivalent to |a+βk|>1.|a+ _k|>1. Likewise, |rk(4)|>|rk(3)||r_k(4)|>|r_k(3)| holds whenever |a2+aβk+βk2+bmdm+bvdv|>|a+βk|. |a^2+a _k+ _k^2+b_md_m+b_vd_v |>|a+ _k|. These are finite-horizon algebraic conditions and do not require asymptotic stability. When they hold together with ρ(A¯)<1,ρ( A)<1, the response grows over the corresponding short horizon even though A¯h→0as h→∞. A^h→ 0~~as h→∞. This is the transient amplification regime described in the main text. C.2 Characteristic Polynomial The characteristic polynomial is p(λ) p(λ) :=det(λI−A¯) := (λ I- A) (28) =(λ−a)(λ−β1)(λ−β2) =(λ-a)(λ- _1)(λ- _2) −bmdm(λ−β2)−bvdv(λ−β1). -b_md_m(λ- _2)-b_vd_v(λ- _1). The two loop gains bmdmb_md_m and bvdvb_vd_v shift the poles of the joint system away from the uncoupled values a,β1,β2a, _1, _2. The frozen system is asymptotically stable when every root of Equation (28) lies strictly inside the unit disk. C.3 Memory-to-Parameter Transfer Functions For a complex variable z outside the spectrum of A¯ A, define ℋm(z):=eθ⊤(zI−A¯)−1em, H_m(z):=e_θ (zI- A)^-1e_m, and ℋv(z):=eθ⊤(zI−A¯)−1ev. H_v(z):=e_θ (zI- A)^-1e_v. Using the corresponding cofactors, ℋm(z) H_m(z) =bm(z−β2)p(z), = b_m(z- _2)p(z), ℋv(z) H_v(z) =bv(z−β1)p(z). = b_v(z- _1)p(z). Both channels share the poles of the complete joint system. Their numerators differ because the momentum input bypasses the v state, whereas the second-moment input bypasses the m state. C.4 Modal Decomposition and Nonnormal Residues Suppose A¯ A is diagonalizable over ℂC: A¯=VΛV−1, A=V V^-1, where V=[v1v2v3],V−1=[w1∗w2∗w3∗],V= bmatrixv_1&v_2&v_3 bmatrix,~~V^-1= bmatrixw_1^*\\ w_2^*\\ w_3^* bmatrix, and wi∗vj=δij.w_i^*v_j= _ij. Then A¯h−1=∑i=13λih−1viwi∗. A^h-1= _i=1^3 _i^h-1v_iw_i^*. Therefore rk(h) r_k(h) =eθ⊤A¯h−1ek =e_θ A^h-1e_k (29) =∑i=13(eθ⊤vi)(wi∗ek)λih−1. = _i=1^3 (e_θ v_i ) (w_i^*e_k ) _i^h-1. For a real matrix, complex eigenvalues and residues occur in conjugate pairs, so Equation (29) remains real. The modal expansion implies |rk(h)|≤∑i=13|eθ⊤vi||wi∗ek||λi|h−1.|r_k(h)|≤ _i=1^3 |e_θ v_i | |w_i^*e_k || _i|^h-1. A coarser matrix-norm bound is |rk(h)| |r_k(h)| ≤‖eθ⊤V‖2‖Λh−1‖2‖V−1ek‖2 ≤\|e_θ V\|_2\| ^h-1\|_2\|V^-1e_k\|_2 ≤κ2(V)ρ(A¯)h−1, ≤ _2(V)ρ( A)^h-1, where κ2(V):=‖V‖2‖V−1‖2. _2(V):=\|V\|_2\|V^-1\|_2. For a normal matrix, V can be chosen unitary and κ2(V)=1 _2(V)=1. For a nonnormal matrix, the eigenvector condition number and individual input–output residues can be much larger. Nonnormality can therefore enlarge the finite-horizon memory-to-parameter response even when all eigenmodes are asymptotically decaying. The finite-horizon peak hk⋆:=minargmaxh≥1|rk(h)|h_k := _h≥ 1|r_k(h)| depends jointly on the modal decay rates, oscillatory phases, and input–output residues. Multiple decaying modes can interfere constructively at intermediate horizons, placing the largest response after the initial memory-to-parameter conversion. The sign of the corresponding probe-loss response additionally depends on the scalar probe readout. C.5 Signs of the Two Feedback Loops In the scalar restriction, bm=β1M,dm=(1−β1)Htr.b_m= _1M,~~d_m=(1- _1)H^tr. Since M<0M<0, sign(bmdm)=−sign(Htr).sign(b_md_m)=-sign(H^tr). Thus, positive local curvature gives a negative momentum-loop return gain, which can contribute to oscillatory or sign-changing parameter responses. For the second-moment loop, bv=β2V,dv=2(1−β2)gHtr.b_v= _2V,~~d_v=2(1- _2)gH^tr. Since the sign of V follows the sign of the current first-moment state, sign(bvdv)=sign(mgHtr).sign(b_vd_v)=sign (m\,g\,H^tr ). The second-moment feedback can therefore reinforce or oppose the evolving parameter response depending on the local operating point. C.6 Weight Decay and Bias Correction For scalar weight decay λwd _wd, a=1−ηλwd+ΓHtr.a=1-η _wd+ H^tr. Weight decay therefore modifies the direct parameter-retention term and, through a, changes the short-horizon response coefficients and their interaction with the two moment-memory channels. Bias correction enters through ρ1,s=1−β1s,ρ2,s=1−β2s, _1,s=1- _1^s,~~ _2,s=1- _2^s, and hence through MsM_s, VsV_s, and Γs _s. In the full AdamW dynamics these quantities are time dependent. The frozen model treats their values at the selected operating point as fixed over the local analysis window. Bias correction therefore changes the numerical coefficients of the frozen system without changing its parameter–moment coupling structure. Appendix D Finite-Horizon Approximation Error This appendix proves Theorem 3.4 and Corollary 3.5. It also gives a sufficient activation-margin condition for quadratic switching error, a recursive error envelope, and the pathwise interpretation under future training randomness. D.1 Exact Smooth–Switching Decomposition Let xss:=xsc+δxs.x_s^s:=x_s^c+δ x_s. The exact paired state difference after one future update is δxs+1 δ x_s+1 =fs(xsc+δxs)−fs(xsc). =f_s (x_s^c+δ x_s )-f_s (x_s^c ). By construction, fs(xsc)=f~s(xsc).f_s(x_s^c)= f_s(x_s^c). Adding and subtracting f~s(xsc+δxs) f_s(x_s^c+δ x_s) gives δxs+1= δ x_s+1= f~s(xsc+δxs)−f~s(xsc) f_s (x_s^c+δ x_s )- f_s (x_s^c ) +fs(xsc+δxs)−f~s(xsc+δxs). +f_s (x_s^c+δ x_s )- f_s (x_s^c+δ x_s ). Using rssm r_s^sm :=f~s(xsc+δxs)−f~s(xsc)−Asδxs, := f_s (x_s^c+δ x_s )- f_s (x_s^c )-A_sδ x_s, rssw r_s^sw :=fs(xsc+δxs)−f~s(xsc+δxs), :=f_s (x_s^c+δ x_s )- f_s (x_s^c+δ x_s ), we obtain δxs+1=Asδxs+rssm+rssw.δ x_s+1=A_sδ x_s+r_s^sm+r_s^sw. (30) If Df~sD f_s is LsL_s-Lipschitz along ℒs=xsc+τδxs:0≤τ≤1,L_s= \x_s^c+τδ x_s:0≤τ≤ 1 \, the integral remainder formula gives rssm=∫01[Df~s(xsc+τδxs)−Df~s(xsc)]δxsτ.r_s^sm= _0^1 [D f_s (x_s^c+τδ x_s )-D f_s(x_s^c) ]δ x_s\,dτ. Therefore ‖rssm‖ \|r_s^sm\| ≤∫01Lsτ‖δxs‖2τ ≤ _0^1L_sτ\|δ x_s\|^2\,dτ (31) =Ls2‖δxs‖2. = L_s2\|δ x_s\|^2. For a smooth network, rssw=0r_s^sw=0. The same holds in a piecewise-smooth network whenever the paired states remain in the same activation region. D.2 Proof of the Multistep State-Error Identity The tangent approximation is initialized by δx^t+1:=Btξt δ x_t+1:=B_t _t and propagated according to δx^s+1:=Asδx^s. δ x_s+1:=A_s δ x_s. Define es:=δxs−δx^s.e_s:=δ x_s- δ x_s. At the first post-shock state, et+1 e_t+1 =δxt+1−δx^t+1 =δ x_t+1- δ x_t+1 (32) =Btξt+rB,t(ξt)−Btξt =B_t _t+r_B,t( _t)-B_t _t =rB,t(ξt). =r_B,t( _t). For a future step, subtracting the tangent recursion from Equation (30) gives es+1 e_s+1 =Asδxs+rssm+rssw−Asδx^s =A_sδ x_s+r_s^sm+r_s^sw-A_s δ x_s =Ases+rssm+rssw. =A_se_s+r_s^sm+r_s^sw. Repeated substitution yields et+h= e_t+h= Φt+h,t+1rB,t(ξt) _t+h,t+1r_B,t( _t) (33) +∑j=t+1t+h−1Φt+h,j+1(rjsm+rjsw). + _j=t+1^t+h-1 _t+h,j+1 (r_j^sm+r_j^sw ). For h=1h=1, the sum is empty and Equation (33) reduces to Equation (32). Taking norms gives ‖et+h‖≤ \|e_t+h\|≤ ‖Φt+h,t+1‖‖rB,t(ξt)‖ \| _t+h,t+1\|\|r_B,t( _t)\| +∑j=t+1t+h−1∥Φt+h,j+1∥(∥rjsm∥+∥rjsw∥). + _j=t+1^t+h-1\| _t+h,j+1\| (\|r_j^sm\|+\|r_j^sw\| ). Using Equation (31) proves the state-error bound in Theorem 3.4. D.3 Probe-Output Error Let Πθ:=[I00] _θ:= bmatrixI&0&0 bmatrix denote projection onto the parameter block. Then δθt+h=Πθδxt+h.δ _t+h= _θδ x_t+h. The exact paired probe response is δFt+h=Ft+h(θt+hc+δθt+h)−Ft+h(θt+hc). δ F_t+h=F_t+h ( _t+h^c+δ _t+h )-F_t+h ( _t+h^c ). If ∇Ft+h∇ F_t+h is LF,t+hL_F,t+h-Lipschitz along the connecting segment, then δFt+h=ct+h⊤δxt+h+rF,t+h,δ F_t+h=c_t+h δ x_t+h+r_F,t+h, where |rF,t+h|≤LF,t+h2‖δθt+h‖2.|r_F,t+h|≤ L_F,t+h2\|δ _t+h\|^2. (34) The tangent output prediction is δF^t+h δ F_t+h =ct+h⊤δx^t+h =c_t+h δ x_t+h =ct+h⊤Φt+h,t+1Btξt. =c_t+h _t+h,t+1B_t _t. Therefore δFt+h−δF^t+h δ F_t+h- δ F_t+h =ct+h⊤(δxt+h−δx^t+h)+rF,t+h =c_t+h (δ x_t+h- δ x_t+h )+r_F,t+h =ct+h⊤et+h+rF,t+h. =c_t+h e_t+h+r_F,t+h. Taking absolute values yields |δFt+h−ct+h⊤Φt+h,t+1Btξt|≤ |δ F_t+h-c_t+h _t+h,t+1B_t _t |≤ ‖ct+h‖‖et+h‖ \|c_t+h\|\|e_t+h\| +LF,t+h2‖δθt+h‖2, + L_F,t+h2\|δ _t+h\|^2, which proves the output-error bound in Theorem 3.4. D.4 A Sufficient Activation-Margin Condition We now give a sufficient condition under which the aggregate switching defect satisfies the quadratic bound required by Corollary 3.5. Let sS_s denote the set of activation gates whose states differ between the paired trajectories during transition s. For multiple simultaneous switches, the total switching defect can be decomposed by a telescoping construction. Choose an arbitrary ordering s=j1,…,jq.S_s=\j_1,…,j_q\. For r=0,…,qr=0,…,q, let fs(r)f_s^(r) denote the local transition map that uses the shock activation state for gates j1,…,jrj_1,…,j_r and the control activation state for the remaining switched gates. At the perturbed state, fs(0)=f~s,fs(q)=fs.f_s^(0)= f_s,~~f_s^(q)=f_s. Define rs,jrsw:= r_s,j_r^sw:= fs(r)(xsc+δxs) f_s^(r) (x_s^c+δ x_s ) −fs(r−1)(xsc+δxs). -f_s^(r-1) (x_s^c+δ x_s ). The total switching defect telescopes: rssw=∑r=1qrs,jrsw.r_s^sw= _r=1^qr_s,j_r^sw. Assume that each local switch satisfies ‖rs,jsw‖≤Ks,j‖δxs‖.\|r_s,j^sw\|≤ K_s,j\|δ x_s\|. (35) Let as,jca_s,j^c denote the control pre-activation of gate j, and let δas,jδ a_s,j denote its change between the paired states. A gate can switch only if |as,jc|≤|δas,j|.|a_s,j^c|≤|δ a_s,j|. Assume |δas,j|≤La,s,j‖δxs‖,La,s,j≤L¯a,s.|δ a_s,j|≤ L_a,s,j\|δ x_s\|,~~L_a,s,j≤ L_a,s. Every switched unit therefore satisfies |as,jc|≤L¯a,s‖δxs‖,|a_s,j^c|≤ L_a,s\|δ x_s\|, and hence s⊆j:|as,jc|≤L¯a,s‖δxs‖.S_s \j:|a_s,j^c|≤ L_a,s\|δ x_s\| \. (36) Suppose further that, for all sufficiently small u>0u>0, the weighted mass of units near the activation boundary satisfies ∑j:|as,jc|≤uKs,j≤ρsu. _j:|a_s,j^c|≤ uK_s,j≤ _su. (37) Using Equations (35) and (36), ‖rssw‖ \|r_s^sw\| ≤∑j∈s‖rs,jsw‖ ≤ _j _s\|r_s,j^sw\| ≤‖δxs‖∑j∈sKs,j ≤\|δ x_s\| _j _sK_s,j ≤∥δxs∥∑j:|as,jc|≤L¯a,s‖δxs‖Ks,j. ≤\|δ x_s\| _j:|a_s,j^c|≤ L_a,s\|δ x_s\|K_s,j. Applying Equation (37) with u=L¯a,s‖δxs‖u= L_a,s\|δ x_s\| gives ‖rssw‖≤ρsL¯a,s‖δxs‖2.\|r_s^sw\|≤ _s L_a,s\|δ x_s\|^2. (38) Thus the activation-margin condition provides the quadratic switching bound used by the fixed-horizon accuracy result. D.5 Proof of Fixed-Horizon First-Order Accuracy Consider the scaled perturbation αξtα _t and a fixed horizon H. Local Lipschitz continuity of DψtD _t gives, from Equation (23), ‖rB,t(αξt)‖≤CB,tα2‖ξt‖2\|r_B,t(α _t)\|≤ C_B,tα^2\| _t\|^2 for sufficiently small α. Local Lipschitz continuity of Df~sD f_s gives ‖rssm‖≤Ls2‖δxs‖2.\|r_s^sm\|≤ L_s2\|δ x_s\|^2. Under the switching assumption in Corollary 3.5, ‖rssw‖≤Cssw‖δxs‖2.\|r_s^sw\|≤ C_s^sw\|δ x_s\|^2. Hence ‖rssm‖+‖rssw‖≤Cs‖δxs‖2,Cs:=Ls2+Cssw.\|r_s^sm\|+\|r_s^sw\|≤ C_s\|δ x_s\|^2,~~C_s:= L_s2+C_s^sw. (39) At the first post-shock state, ‖δxt+1(α)‖ \|δ x_t+1(α)\| ≤|α|‖Btξt‖+CB,tα2‖ξt‖2 ≤|α|\|B_t _t\|+C_B,tα^2\| _t\|^2 =O(|α|). =O(|α|). Suppose inductively that ‖δxs(α)‖=O(|α|)\|δ x_s(α)\|=O(|α|) for some t+1≤s<t+Ht+1≤ s<t+H. The local derivatives are bounded on the relevant neighborhood, so ‖As‖\|A_s\| is bounded. Using Equation (30), ‖δxs+1(α)‖ \|δ x_s+1(α)\| ≤‖As‖‖δxs(α)‖+Cs‖δxs(α)‖2 ≤\|A_s\|\|δ x_s(α)\|+C_s\|δ x_s(α)\|^2 =O(|α|)+O(α2) =O(|α|)+O(α^2) =O(|α|). =O(|α|). By induction, ‖δxt+h(α)‖=O(|α|)\|δ x_t+h(α)\|=O(|α|) for every fixed 1≤h≤H1≤ h≤ H. The exact multistep error identity now gives ‖et+h(α)‖≤ \|e_t+h(α)\|≤ ‖Φt+h,t+1‖O(α2) \| _t+h,t+1\|O(α^2) +∑j=t+1t+h−1∥Φt+h,j+1∥O(α2). + _j=t+1^t+h-1\| _t+h,j+1\|O(α^2). Since H is fixed and the finite-horizon propagators are uniformly bounded, ‖et+h(α)‖=O(α2),1≤h≤H.\|e_t+h(α)\|=O(α^2),~~1≤ h≤ H. The probe objectives have locally Lipschitz gradients, and ‖δθt+h(α)‖=O(|α|)\|δ _t+h(α)\|=O(|α|), so Equation (34) gives |rF,t+h(α)|=O(α2).|r_F,t+h(α)|=O(α^2). Therefore δFt+h(α) δ F_t+h(α) =ct+h⊤δx^t+h(α)+O(α2) =c_t+h δ x_t+h(α)+O(α^2) =αct+h⊤Φt+h,t+1Btξt+O(α2), =α c_t+h _t+h,t+1B_t _t+O(α^2), for every 1≤h≤H1≤ h≤ H. The constants may depend on the fixed control trajectory and horizon H but not on α. This proves Corollary 3.5. When the sufficient activation-margin condition of Appendix D.4 holds, Equation (39) can be instantiated with Cssw=ρsL¯a,s.C_s^sw= _s L_a,s. D.6 Recursive Error Envelope The structural error bound contains the true deviations δxsδ x_s. A recursive envelope can instead be expressed in terms of the tangent trajectory and previously accumulated error bounds. Suppose ‖es‖≤εs.\|e_s\|≤ _s. Since δxs=δx^s+es,δ x_s= δ x_s+e_s, we have ‖δxs‖≤‖δx^s‖+εs.\|δ x_s\|≤\| δ x_s\|+ _s. Using the error recursion, ‖es+1‖ \|e_s+1\| ≤‖As‖‖es‖+‖rssm‖+‖rssw‖ ≤\|A_s\|\|e_s\|+\|r_s^sm\|+\|r_s^sw\| ≤‖As‖εs+Ls2(‖δx^s‖+εs)2+‖rssw‖. ≤\|A_s\| _s+ L_s2 (\| δ x_s\|+ _s )^2+\|r_s^sw\|. Thus the recursion εs+1:= _s+1:= ‖As‖εs \|A_s\| _s (40) +Ls2(‖δx^s‖+εs)2+‖rssw‖ + L_s2 (\| δ x_s\|+ _s )^2+\|r_s^sw\| preserves ‖es+1‖≤εs+1.\|e_s+1\|≤ _s+1. Initializing with εt+1:=‖rB,t(ξt)‖ _t+1:=\|r_B,t( _t)\| therefore gives an envelope for all subsequent states. If a quadratic switching bound ‖rssw‖≤Cssw‖δxs‖2\|r_s^sw\|≤ C_s^sw\|δ x_s\|^2 is available, then ‖rssw‖≤Cssw(‖δx^s‖+εs)2.\|r_s^sw\|≤ C_s^sw (\| δ x_s\|+ _s )^2. The envelope becomes εs+1:= _s+1:= ‖As‖εs \|A_s\| _s (41) +(Ls2+Cssw)(‖δx^s‖+εs)2. + ( L_s2+C_s^sw ) (\| δ x_s\|+ _s )^2. Under the activation-margin sufficient condition, Cssw=ρsL¯a,s.C_s^sw= _s L_a,s. The same transition dynamics therefore govern both the desired first-order response and the accumulation of approximation error: large finite-horizon gain amplifies the propagated perturbation as well as nonlinear defects introduced along the trajectory. D.7 Pathwise Conditioning and Distributional Extension Let ω denote a realized future random sequence, including future minibatches, dropout masks, data augmentation, and other stochastic training operations. Conditional on ω and the pre-shock history, the future operators are deterministic: As(ω),Φt+h,t+1(ω),ct+h(ω).A_s(ω),~~ _t+h,t+1(ω),~~c_t+h(ω). When the probe sequence is itself defined from the future minibatches, such as Fs(θ)=ℓ(θ,ℬs),F_s(θ)= (θ;B_s), the probe functions are also fixed after conditioning on ω. The corresponding pathwise input–output operator is t,h(ω)=ct+h(ω)⊤Φt+h,t+1(ω)Bt.G_t,h(ω)=c_t+h(ω) _t+h,t+1(ω)B_t. All directional-response identities and error decompositions above apply separately to each realized common-future sequence. Averaging a pathwise quantity over independently sampled futures yields its corresponding distributional version. For example, ω[δFt+h(ω)]E_ω [δ F_t+h(ω) ] describes the mean paired response at horizon h, while ω[t,h(ω)ξt]E_ω [G_t,h(ω) _t ] gives the corresponding first-order mean response whenever the expectation and local expansion may be interchanged. The common-future construction retains the same future realization for the control and perturbed trajectories within each pair. It therefore isolates the pathwise propagation associated with the perturbation at step t, while repetition across different future realizations characterizes variability of that response. Appendix E Detailed Experimental Protocols and Results This appendix gives the complete protocols and the additional numerical results supporting Section 4. The experiments are organized by scientific question rather than by model class. Experiment 1 validates the finite-horizon mechanism across controlled quadratic systems, nonconvex neural networks, and pretrained language models. Experiment 2 then asks whether the resulting delayed response retains candidate-specific structure when the future training sequence is unknown and independently resampled. For the controlled and neural-network mechanism-validation stages, horizon h=1h=1 denotes the first post-shock state and di,h(α) d_i,h(α) :=F(θi,t+hs(α))−F(θt+hc), :=F(θ^s_i,t+h(α))-F(θ^c_t+h), (42) d^i,h d_i,h :=ct+h⊤Φt+h,t+1Btξi, :=c_t+h _t+h,t+1B_t _i, (43) d^i,h(α) d_i,h(α) :=αd^i,h. :=α d_i,h. (44) We use Mi(α) M_i(α) =maxh|di,h(α)|, = _h|d_i,h(α)|, AREi(α) _i(α) =∑h|di,h(α)|, = _h|d_i,h(α)|, (45) hi⋆ h_i =minargmaxh|di,h|, = _h|d_i,h|, si⋆ s_i =sign(di,hi⋆). =sign(d_i,h_i ). (46) The primary trajectory error is NRMSEi,α=(∑h(di,h(α)−αd^i,h)2∑hdi,h(α)2)1/2.NRMSE_i,α= ( _h(d_i,h(α)-α d_i,h)^2 _hd_i,h(α)^2 )^1/2. (47) For the analytic controlled and neural-network tangents, the horizon-wise symmetric relative error is ei,h(α)=|di,h(α)−αd^i,h||di,h(α)|+|αd^i,h|+10−30.e_i,h(α)= |d_i,h(α)-α d_i,h||d_i,h(α)|+|α d_i,h|+10^-30. (48) The empirical validity radius is defined on the tested scale grid by αivalid=maxα:median1≤h≤Hei,h(α)≤0.2, _i^valid= \α:median_1≤ h≤ He_i,h(α)≤ 0.2 \, (49) with value zero when the set is empty. Candidate radii are aggregated within each system before system-level summaries are formed. The language-model stage reports the corresponding valid-candidate fraction at every tested scale. Candidates, scales, horizons, and future branches are always treated as nested observations. The controlled and neural-network studies first aggregate candidate-level quantities within independently generated training systems. The Pythia study is a single-seed scaling analysis: its nine model–dataset systems are fixed conditions rather than independent random replications, and its statistics are descriptive. Experiment 2 preserves these system units while introducing 32 independently resampled future continuations per fixed state–shock pair. E.1 Common Baselines and Tangent Ablations The controlled and neural-network stages of Experiment 1 explicitly construct the initial AdamW tangent δxi,1:=Btξi=[δθi,t+1δmi,tδvi,t]δ x_i,1:=B_t _i= bmatrixδ _i,t+1\\ δ m_i,t\\ δ v_i,t bmatrix (50) and recursively propagate δxi,h+1=At+hδxi,hδ x_i,h+1=A_t+hδ x_i,h. Let Pθδx=δθP_θδ x=δθ and use the local shorthand δθi,h:=Pθδxi,hδ _i,h:=P_θδ x_i,h. Define the parameter-space probe gradient c¯h:=∇F(θt+hc),ct+h=Pθ⊤c¯h. c_h:=∇ F(θ^c_t+h), c_t+h=P_θ c_h. (51) Every trajectory score below is the maximum absolute predicted response over 1≤h≤H1≤ h≤ H. Full ISO. The complete analytic/JVP trajectory and its score are ri,hISO=c¯h⊤PθΦt+h,t+1Btξi,SiISO=maxh|ri,hISO|.r_i,h^ISO= c_h P_θ _t+h,t+1B_t _i, S_i^ISO= _h|r_i,h^ISO|. (52) Temporal ablations. The implementations used in both mechanism-validation stages are ri,hno-prop r_i,h^no -prop =c¯h⊤δθi,1, = c_h δ _i,1, (53) ri,hinit-θ r_i,h^init -θ =c¯h⊤PθΦt+h,t+1Pθ⊤δθi,1, = c_h P_θ _t+h,t+1P_θ δ _i,1, (54) ri,hclamp-θ r_i,h^clamp -θ =c¯h⊤(∏s=t+1t+h−1[As]θθ)δθi,1, = c_h ( _s=t+1^t+h-1[A_s]_θ )δ _i,1, (55) ri,hfrozen-dyn r_i,h^frozen -dyn =c¯h⊤PθAt+1h−1δxi,1, = c_h P_θA_t+1^h-1δ x_i,1, (56) ri,hfrozen-readout r_i,h^frozen -readout =c¯1⊤δθi,h. = c_1 δ _i,h. (57) Here, No Propagation reuses the initial parameter write at every horizon. Initial Parameter Only removes the two moment components of δxi,1δ x_i,1 once and then applies the complete time-varying dynamics. Clamped Parameter State sets the moment tangent blocks to zero before and after every transition, retaining only [As]θθ[A_s]_θ. Frozen Dynamics repeatedly applies the first future Jacobian, and Frozen Readout applies c¯1 c_1 to the correctly propagated parameter tangent at every horizon. Scalar baselines. The exact immediate-response oracle and the static scores are Si1step S_i^1step =|di,1(1)|, =|d_i,1(1)|, Sigrad S_i^grad =‖ξi‖2, =\| _i\|_2, (58) Siwrite S_i^write =‖δθi,1‖2, =\|δ _i,1\|_2, Sicurv S_i^curv =|ξi⊤Hrefξi|, =| _i H_ref _i|, (59) Sinorm S_i^norm =maxh‖c¯h‖2‖δθi,h‖2, = _h\| c_h\|_2\|δ _i,h\|_2, (60) Href H_ref =1R∑r=1R∇2ℓ(θt,ℬrref). = 1R _r=1^R∇^2 ( _t;B_r^ref). (61) The curvature score therefore uses the shock direction in the average reference-minibatch Hessian. The norm-product baseline removes directional alignment while retaining the horizon-wise state and readout norms. For the readout-order null, one shared permutation π is applied to all candidates in a system and Si,πshuffle=maxh|c¯π(h)⊤δθi,h|.S_i,π^shuffle= _h| c_π(h) δ _i,h|. (62) We use 100 independently drawn permutations per system. E.2 Experiment 1: Finite-Horizon Mechanism Validation E.2.1 Controlled Quadratic Systems Systems and protocol. Each minibatch loss is ℓs(θ)=12θ⊤Dsθ+12r‖Us⊤θ‖22+qs⊤θ, _s(θ)= 12θ D_sθ+ 12r\|U_s θ\|_2^2+q_s θ, (63) where DsD_s is positive diagonal, Us∈ℝd×rU_s ^d× r, d=512d=512, and r=16r=16. For condition multiplier κ∈1,4,16κ∈\1,4,16\, independent innovations are sampled as D~s,j D_s,j ∼Unif(0.05,0.25κ), (0.05,0.25κ), (64) U~s,jk U_s,jk ∼(0,1/d), (0,1/d), (65) q~s,j q_s,j ∼(0,0.052). (0,0.05^2). (66) Writing ρ=0.85ρ=0.85, the temporally correlated sequence is generated by Ds D_s =ρDs−1+(1−ρ)D~s, =ρ D_s-1+(1-ρ) D_s, (67) Us U_s =ρUs−1+1−ρ2U~s, =ρ U_s-1+ 1-ρ^2\, U_s, (68) qs q_s =ρqs−1+1−ρ2q~s, =ρ q_s-1+ 1-ρ^2\, q_s, (69) with the first batch equal to its innovation. The condition multiplier cycles through 1,4,16\1,4,16\ with the system index, and θ0∼(0,0.12I) _0 (0,0.1^2I). AdamW uses η=2×10−3,β1=0.9,β2=0.999,ϵ=10−8,λ=0.01. gatheredη=2× 10^-3, _1=0.9, _2=0.999,\\ ε=10^-8, λ=0.01. gathered (70) All calculations use double precision. Each independently initialized system is advanced for 40 burn-in steps. Four reference minibatches define the control gradient, 16 candidate minibatches define shocks, and a shared future sequence is followed for H=32H=32. Four seeds and eight systems per seed give 32 independent base systems. Standard and anisotropic probes are paired views of each base system and are not counted as separate replications. Probe constructions. The main controlled experiment uses two paired probes. The standard probe is sampled independently from the same quadratic family, with linear standard deviation 0.020.02. The anisotropic probe starts from that draw, sets its linear term to zero, and replaces its diagonal by DFaniso D_F^aniso =DFDiag(w), =D_FDiag(w), (71) wjj=1d \w_j\_j=1^d =perm(32(j−1)/(d−1)j=1d), =perm\! ( \32^(j-1)/(d-1) \_j=1^d ), (72) while retaining the sampled low-rank factor. This is the readout_active regime in the code. The exact-one-step-matched supplement uses the standard probe and a separate rotating-readout probe. Let θhmcθ^c_h_m and θhm+1cθ^c_h_m+1 be the two central states of the H-step common control trajectory and set μ μ =12(θhmc+θhm+1c), = 12 (θ^c_h_m+θ^c_h_m+1 ), (73) W W =Diag(perm(32(j−1)/(d−1)j=1d)). =Diag\! (perm\! ( \32^(j-1)/(d-1) \_j=1^d ) ). (74) The fixed probe is Frot(θ)=12(θ−μ)⊤W(θ−μ),c¯h=W(θhc−μ).F_rot(θ)= 12(θ-μ) W(θ-μ), c_h=W( _h^c-μ). (75) It depends only on the common control trajectory. Centering it between the two middle states makes the readout direction change as the control trajectory passes the probe center. The standard, anisotropic, and rotating-readout probes are therefore three distinct constructions. Local fidelity and finite-scale error. We use α∈1/32,1/16,1/8,1/4,1/2,1α∈\1/32,1/16,1/8,1/4,1/2,1\. Table 5 reports system-level medians. At the smallest scale, NRMSE is 0.01170.0117 under the standard probe and 0.01730.0173 under the anisotropic probe. The median local trajectory NRMSE pooled across the prespecified local range is 0.04500.0450, and median horizon-wise sign agreement is one. The median empirical validity radius is 0.50.5. Fitting log|di,h(α)−αd^i,h|=ai,h+pi,hlogα |d_i,h(α)-α d_i,h|=a_i,h+p_i,h α (76) over the local scales yields an approximately quadratic remainder, with system-level R2R^2 values essentially one. The independently evaluated recursive state-error identity has maximum residual 3.11×10−173.11× 10^-17, providing an implementation audit of the recursive decomposition. Table 5: Controlled-stage signed-trajectory fidelity in Experiment 1. Entries are medians over 32 independent systems after candidate-level aggregation. Probe α NRMSE Rel. M err. Sign acc. Standard 1/321/32 0.0117 0.0052 1.000 1/81/8 0.0483 0.0216 1.000 1/21/2 0.2256 0.0945 1.000 11 0.5538 0.2006 0.938 Anisotropic 1/321/32 0.0173 0.0071 1.000 1/81/8 0.0719 0.0289 1.000 1/21/2 0.3445 0.1300 0.938 11 0.9512 0.2681 0.875 Natural candidate ranking. For unmodified candidate minibatches, Full ISO reaches median Spearman correlation 0.8350.835 for the standard probe and 0.7620.762 for the anisotropic probe. The corresponding exact one-step correlations are 0.7740.774 and 0.7340.734. Their difference is modest because natural immediate and future responses can be strongly correlated. Static gradient, parameter-write, and curvature summaries attain correlations in the approximate range 0.330.33–0.480.48. Exact-one-step matching. Let a be the one-step tangent functional satisfying a⊤ξ=c¯1⊤PθBtξa ξ= c_1 P_θB_tξ. In the implementation, the diagonal write-in map is recovered by applying BtB_t to the all-ones direction, so a=c¯1⊙PθBta= c_1 P_θB_t1. For natural residual ri=gi−g¯r_i=g_i- g, define zi z_i =ri−aa⊤ri‖a‖22, =r_i-a a r_i\|a\|_2^2, (77) bi b_i =siμ1a‖a‖22,si=(−1)i, =s_i _1 a\|a\|_2^2, s_i=(-1)^i, (78) z~i z_i =4‖bi‖2zi/‖zi‖2,‖zi‖2>0,0,otherwise, = cases4\|b_i\|_2z_i/\|z_i\|_2,&\|z_i\|_2>0,\\ 0,&otherwise, cases (79) ξ~i ξ_i =bi+z~i. =b_i+ z_i. (80) The common target is μ1=0.25medianj|a⊤rj|. _1=0.25median_j|a r_j|. (81) Thus a⊤ξ~i=siμ1a ξ_i=s_i _1, while the candidate-specific component lies in the nullspace of the immediate output functional and has four times the norm of the common component. We then calibrate one scalar γi≥0 _i≥ 0 per candidate using the exact nonlinear AdamW update: Let Θ+(x,g) ^+(x,g) denote the parameter component after one exact AdamW update from state x with gradient g. The calibration equation is F(Θ+(xt,g¯+γiξ~i))−F(Θ+(xt,g¯))=siμ1.F\! ( ^+(x_t, g+ _i ξ_i) )-F\! ( ^+(x_t, g) )=s_i _1. (82) Starting with [0,1][0,1], the upper endpoint is doubled until the target is bracketed, up to γ=64γ=64; 64 bisection iterations are then applied. A match is accepted when the signed residual is at most 5×10−8μ15× 10^-8 _1. All candidates across the 64 system–probe instances (32 systems evaluated under two probe constructions) are successfully calibrated. This procedure uses the control state, the one-step functional, and natural residual directions, but no response at h>1h>1. Table 6 shows that the matched immediate response is numerically constant whereas future magnitudes remain heterogeneous. The median ratio Mi/|di,1|M_i/|d_i,1| is 6.216.21 under the standard probe and 4.544.54 under the rotating-readout probe; corresponding parameter transient gains are approximately 7.77.7. Table 6: Controlled-stage exact-one-step-matched stress test in Experiment 1. CV is computed over 16 candidates within a system; entries are medians across 32 independent systems, with IQRs shown for the matching diagnostics. Quantity Standard Rotating-readout Calibration success 1.000[1.000,1.000]1.000\ [1.000,1.000] 1.000[1.000,1.000]1.000\ [1.000,1.000] CV of |d1||d_1| ×10−121.10\!×\!10^-12 ×10−126.70\!×\!10^-12 CV of future M 0.092[0.064,0.114]0.092\ [0.064,0.114] 0.459[0.292,0.530]0.459\ [0.292,0.530] M/|d1|M/|d_1| 6.21[5.69,7.86]6.21\ [5.69,7.86] 4.54[3.38,5.08]4.54\ [3.38,5.08] Parameter gain 7.71[7.10,8.90]7.71\ [7.10,8.90] 7.69[7.06,8.83]7.69\ [7.06,8.83] Table 7: Median Spearman correlation under exact-one-step matching. The one-step score is tied by construction. Method Standard Rotating-readout Full ISO 0.993 1.000 No propagation 0.354 0.806 Frozen dynamics 0.788 0.969 Initial parameter only 0.137 0.788 Clamped parameter state 0.254 0.793 Frozen readout 0.371 0.831 Gradient norm -0.062 0.776 Parameter-write norm -0.025 0.647 Curvature 0.044 0.738 Norm product 0.041 0.790 Readout controls. The standard readout remains close to its initial direction, with median minimum cosine 0.9140.914. The rotating-readout probe produces substantial horizon-dependent rotation while keeping a fixed scalar probe objective. We additionally apply 100 common permutations to the readout sequence while preserving the propagated states. Correct temporal alignment places Full ISO at the maximum observed percentile of the shuffled-readout null in both regimes. This control distinguishes directional state–readout alignment from a product of state and readout norms. State-channel dependence and persistence. Let the exact post-shock joint-state deviation be Δxt+1=(Δθt+1,Δmt,Δvt). x_t+1=( _t+1, m_t, v_t). (83) For (a,b,c)∈0,13(a,b,c)∈\0,1\^3, we form xt+1abc=xt+1c+(aΔθt+1,bΔmt,cΔvt)x_t+1^abc=x_t+1^c+(a _t+1,b m_t,c v_t) (84) and follow the common future exactly. Parameter-only interventions peak early, first-moment interventions peak later and usually carry the largest isolated response, and second-moment deviations strongly modulate the response when coupled with m. These hybrid counterfactuals can lie away from states reached by ordinary AdamW; we use them to measure channel dependence and interaction. Table 8: Complete post-shock state-channel intervention in the controlled stage of Experiment 1. Entries are medians after first aggregating selected candidates within each of the 32 independent systems. Channel order is (θ,m,v)(θ,m,v). Standard probe Anisotropic probe Channels State M ARE h⋆h M ARE h⋆h None 000000 0 0 1.00 0 0 1.00 θ 100100 ×10−41.80\!×\!10^-4 ×10−33.75\!×\!10^-3 6.50 ×10−31.43\!×\!10^-3 ×10−22.95\!×\!10^-2 6.00 m 010010 ×10−48.60\!×\!10^-4 ×10−21.91\!×\!10^-2 15.75 ×10−38.90\!×\!10^-3 ×10−11.71\!×\!10^-1 16.25 v 001001 ×10−42.73\!×\!10^-4 ×10−34.79\!×\!10^-3 25.25 ×10−32.47\!×\!10^-3 ×10−24.86\!×\!10^-2 27.25 θ,mθ,m 110110 ×10−49.98\!×\!10^-4 ×10−22.19\!×\!10^-2 14.25 ×10−39.72\!×\!10^-3 ×10−11.94\!×\!10^-1 15.00 θ,vθ,v 101101 ×10−42.74\!×\!10^-4 ×10−34.85\!×\!10^-3 18.75 ×10−32.64\!×\!10^-3 ×10−24.40\!×\!10^-2 22.50 m,vm,v 011011 ×10−48.29\!×\!10^-4 ×10−21.68\!×\!10^-2 14.50 ×10−36.70\!×\!10^-3 ×10−11.30\!×\!10^-1 17.50 θ,m,vθ,m,v 111111 ×10−49.23\!×\!10^-4 ×10−21.96\!×\!10^-2 14.00 ×10−37.59\!×\!10^-3 ×10−11.48\!×\!10^-1 14.50 In matched-first-displacement sweeps, the first future parameter displacement is held fixed while β1 _1 or β2 _2 is varied. Increasing β1 _1 from 0.50.5 to 0.990.99 moves the momentum-channel extremum from approximately horizon 5 toward the end of the measured interval and substantially increases ARE. A horizon-128128 extension confirms the ordering of time scales: parameter deviations act earliest, momentum dominates intermediate delays, and second-moment effects persist longest. This rules out a purely larger first-update explanation for the memory effect. Table 9: Controlled-stage matched-first-displacement persistence sweep in Experiment 1. The first future parameter displacement is held fixed within each channel. Standard probe Anisotropic probe Channel Decay M ARE h⋆h M ARE h⋆h m 0.5 ×10−56.58\!×\!10^-5 ×10−31.08\!×\!10^-3 5.25 ×10−45.75\!×\!10^-4 ×10−21.08\!×\!10^-2 6.25 m 0.8 ×10−41.44\!×\!10^-4 ×10−32.74\!×\!10^-3 9.75 ×10−31.33\!×\!10^-3 ×10−22.61\!×\!10^-2 11.75 m 0.9 ×10−42.66\!×\!10^-4 ×10−35.96\!×\!10^-3 16.75 ×10−32.59\!×\!10^-3 ×10−25.46\!×\!10^-2 17.75 m 0.95 ×10−47.16\!×\!10^-4 ×10−21.47\!×\!10^-2 30.50 ×10−35.62\!×\!10^-3 ×10−11.20\!×\!10^-1 28.75 m 0.99 ×10−34.65\!×\!10^-3 ×10−27.23\!×\!10^-2 32.00 ×10−23.41\!×\!10^-2 ×10−15.34\!×\!10^-1 32.00 v 0.9 ×10−67.82\!×\!10^-6 ×10−41.41\!×\!10^-4 9.00 ×10−57.31\!×\!10^-5 ×10−31.42\!×\!10^-3 10.50 v 0.99 ×10−51.37\!×\!10^-5 ×10−42.87\!×\!10^-4 17.75 ×10−41.33\!×\!10^-4 ×10−32.74\!×\!10^-3 17.50 v 0.999 ×10−54.64\!×\!10^-5 ×10−48.13\!×\!10^-4 24.75 ×10−44.31\!×\!10^-4 ×10−38.29\!×\!10^-3 26.75 v 0.9999 ×10−41.26\!×\!10^-4 ×10−32.24\!×\!10^-3 31.25 ×10−31.21\!×\!10^-3 ×10−21.83\!×\!10^-2 31.75 E.2.2 Nonconvex Neural Networks Models, data, and replication. We use CIFAR-10 with a two-hidden-layer width-256 MLP with GELU activations (855,050855,050 parameters) and a three-layer width-32 CNN with ReLU activations (94,53894,538 parameters). Training batches contain 128 examples and fixed test probes contain 256 examples. Images use standard CIFAR-10 channel normalization. AdamW uses learning rate 2×10−42× 10^-4, β1=0.9 _1=0.9, β2=0.999 _2=0.999, ϵ=10−8ε=10^-8, and weight decay 0.010.01; state and tangent calculations use double precision. For each architecture, four top-level seeds (2026–2029) and four independent initializations per seed give 16 independent systems, or 32 total. Each system receives 100 burn-in updates, four reference batches, 12 candidate batches, 11 common-future batches, and one fixed probe, giving H=12H=12 and 384 candidate shocks. We use α∈0.0625,0.125,0.25,0.5,1α∈\0.0625,0.125,0.25,0.5,1\. Trajectory fidelity. Table 10 gives the complete finite-scale curve for the neural-network stage. The pooled median local NRMSE is 0.06780.0678 and median local sign agreement is one. The median empirical validity radius is 0.750.75 for CNN–ReLU (IQR [0.5,1][0.5,1]) and 0.50.5 for MLP–GELU (IQR [0.375,0.5][0.375,0.5]). The recursive state-error identity has maximum residual 9.33×10−179.33× 10^-17. Table 10: Neural-network finite-horizon tangent fidelity in Experiment 1. Entries are medians over independent systems after aggregating candidates within systems. Architecture α NRMSE Sign agr. Rel. M err. Rel. ARE err. Extremum-sign acc. CNN–ReLU 0.0625 0.0458 1.000 0.0113 0.0115 1.000 0.125 0.0610 1.000 0.0160 0.0185 1.000 0.25 0.1097 1.000 0.0299 0.0307 1.000 0.5 0.1959 1.000 0.0587 0.0632 1.000 1 0.3854 0.958 0.1094 0.1126 1.000 MLP–GELU 0.0625 0.0377 1.000 0.0116 0.0140 1.000 0.125 0.0759 1.000 0.0230 0.0276 1.000 0.25 0.1530 1.000 0.0462 0.0566 1.000 0.5 0.2980 1.000 0.1002 0.1191 1.000 1 0.6294 0.917 0.1698 0.2270 1.000 Finite-scale error behavior. The local absolute-error exponent is fitted over α∈0.0625,0.125,0.25α∈\0.0625,0.125,0.25\. For MLP–GELU, its median remains in [2.001,2.005][2.001,2.005] at all horizons, with R2≈1R^2≈ 1. For CNN–ReLU, the median exponent is 2.0172.017, 1.9161.916, 1.5731.573, and 1.1201.120 at horizons 11, 44, 88, and 1212. ReLU activation-sign differences are nonzero and increase with scale (at h=8h=8, their median rises from 1.79×10−41.79× 10^-4 at α=.0625α=.0625 to 2.82×10−32.82× 10^-3 at α=1α=1). The degradation coincides with increasing activation-pattern changes, consistent with a growing switching defect at later horizons. Ranking future absolute influence. Within each system, we rank candidates by Mi=maxh|di,h(1)|M_i= _h|d_i,h(1)|. Table 11 reports the main ablations for the neural-network stage. Full ISO obtains pooled median Spearman 0.8320.832. It exceeds the exact one-step oracle and static norms in both architectures, whereas frozen dynamics preserves much of the ordering over the short H=12H=12 horizon. Paired Wilcoxon tests use the independent system as the unit and Holm correction within each architecture. For CNN–ReLU, Full ISO significantly exceeds all alternatives except frozen dynamics. For MLP–GELU, its advantages over exact one-step, gradient norm, parameter write, curvature, norm product, and frozen readout survive correction; its differences from the remaining propagation ablations are positive in median but not significant after correction. Table 11: Median within-system Spearman correlation with future magnitude in the neural-network stage of Experiment 1. Method CNN–ReLU MLP–GELU Full ISO 0.888 0.762 Frozen dynamics 0.843 0.671 Clamped parameter state 0.818 0.661 No propagation 0.755 0.668 Initial parameter only 0.734 0.720 Frozen readout 0.713 0.507 Exact one-step 0.545 0.566 Gradient norm 0.535 0.336 Parameter write norm 0.490 0.283 Curvature 0.559 0.248 Norm product 0.241 0.224 Readout order and signed extrema. Under 100 shared permutations of future readout order, CNN–ReLU Full ISO decreases from 0.8880.888 to median 0.7760.776 (p=0.0019p=0.0019), with the true score at the median 99th percentile of the system-specific null. For MLP–GELU, the corresponding values are 0.7620.762 and 0.7130.713 (p=0.191p=0.191); its nearby readouts are sufficiently similar that permutation preserves much of the ordering. Across the 96 exact intervention trajectories included in this diagnostic, 57 extrema are positive and 39 are negative. A one-sided positive-response analysis would therefore omit 40.6%40.6\% of these extremal events. State-channel dependence. Table 12 reports exact hybrid-state interventions in the neural-network stage. The parameter-only response peaks early, while the first-moment-only response peaks later and has substantially greater accumulated magnitude. The second-moment-only response is small, but adding v to (θ,m)(θ,m) slightly reduces the median response in both architectures. The conditional v contrast is negative in 15 of 16 CNN systems and 13 of 16 MLP systems. This is evidence that v modulates a coupled parameter–moment state, not that it has a uniquely defined negative additive contribution. Table 12: Neural-network exact state-channel interventions in Experiment 1. Values are median ARE relative to the full 111111 intervention; bit order is (θ,m,v)(θ,m,v). State CNN ARE CNN h⋆h MLP ARE MLP h⋆h 100100 0.216 2.5 0.250 1.5 010010 0.821 9.0 0.827 7.5 001001 0.030 10.0 0.015 9.5 110110 1.025 9.0 1.010 6.5 101101 0.214 3.0 0.237 1.5 011011 0.812 9.0 0.818 7.5 111111 1.000 9.0 1.000 7.0 Matched-displacement persistence. Holding the first future parameter displacement fixed, increasing β1 _1 from 0.50.5 to 0.990.99 multiplies M by 8.478.47 (CNN) and 10.8010.80 (MLP), and multiplies ARE by 14.1714.17 and 11.9611.96. The median extremum moves from approximately h=3.5h=3.5–44 to h=12h=12. Increasing β2 _2 from 0.90.9 to 0.99990.9999 multiplies ARE by 4.244.24 and 2.752.75, respectively. Since the first displacement is matched, these differences isolate persistence from immediate update magnitude. E.2.3 Pretrained Language Models Models, domains, and warm optimizer state. We use Pythia-410M, Pythia-1B, and Pythia-1.4B with WikiText-103, OpenWebText, and CodeParrot, giving nine model–dataset systems. From each domain we prepare 2,048 token sequences of length 129. The training batch size is one and the fixed probe contains two sequences. AdamW uses learning rate 10−510^-5, β1=0.9 _1=0.9, β2=0.999 _2=0.999, ϵ=10−8ε=10^-8, and weight decay 0.010.01. Starting from pretrained weights, every system is first continued for 500 updates. This creates a nontrivial trained AdamW moment state and prevents a zero-moment cold start from dominating the derivative. The intervention uses two reference batches, eight candidate shocks, seven common-future batches, and H=8H=8. We evaluate α∈0.0625,0.125,0.25,0.5,1α∈\0.0625,0.125,0.25,0.5,1\ under seed 2026. Numerical ISO directional response. Directly differencing nearly equal scalar cross-entropies is inaccurate at this scale. For candidate i, horizon h, and probe example n, we therefore compute the centered logit derivative δzi,h,n(ε)=zi,h,n+(ε)−zi,h,n−(ε)2εδ z_i,h,n( )= z^+_i,h,n( )-z^-_i,h,n( )2 (85) and apply the exact control cross-entropy differential d^i,h(ε)=1Ntok∑n⟨softmax(zh,n0)−eyn,δzi,h,n(ε)⟩. d_i,h( )= 1N_tok _n (z^0_h,n)-e_y_n,δ z_i,h,n( ) . (86) The resulting trajectory estimates the directional response dαF(θi,t+h(α))|α=0, . ddαF( _i,t+h(α)) |_α=0, (87) which equals ct+h⊤Φt+h,t+1Btξic_t+h _t+h,t+1B_t _i under the theorem’s local conditions. We denote this numerical quantity by ISO Tangent (FD). Unlike the controlled and neural-network stages above, this stage estimates the end-to-end tangent response without separately materializing BtB_t, every AsA_s, and the readout ct+hc_t+h. The exact finite-scale target remains the FP64-reduced difference between the shock and control probe losses. We test ε∈.5,.25,.125,.0625,.03125,.015625 ∈\.5,.25,.125,.0625,.03125,.015625\. Adjacent derivative trajectories are compared with the symmetric NRMSE EFD(ε,ε/2)=‖d^(ε)−d^(ε/2)‖212(‖d^(ε)‖2+‖d^(ε/2)‖2)+10−30.E_FD( , /2)= \| d( )- d( /2)\|_2 12(\| d( )\|_2+\| d( /2)\|_2)+10^-30. (88) The smaller scale in the most consistent adjacent pair is selected, and a candidate is identifiable when the selected discrepancy is at most 0.250.25. All 72 candidates pass. Median selected consistency is 0.003780.00378 and the maximum accepted value is 0.227650.22765. Selected smaller scales .25, .125, .0625, .03125, and .015625 occur for 35, 15, 14, 4, and 4 candidates. Finite-scale fidelity and validity. Table 13 reports all shock scales. The empirical validity criterion is the median horizon-wise symmetric relative error at most 0.20.2. Valid fractions are 93.1%93.1\%, 97.2%97.2\%, 87.5%87.5\%, 73.6%73.6\%, and 55.6%55.6\% from the smallest to the largest scale. The slight first-pair nonmonotonicity reflects numerical variation around small responses. The result characterizes the finite-scale range over which the local mechanism provides accurate pointwise predictions. Table 13: Language-model finite-scale trajectory fidelity in Experiment 1 over 72 candidates. α NRMSE 75% NRMSE Cosine Rel. M Sign acc. 0.0625 0.0387 0.1664 0.99990 0.0224 1.000 0.125 0.0477 0.1906 0.99993 0.0263 1.000 0.25 0.0946 0.2095 0.99983 0.0524 1.000 0.5 0.2105 0.4431 0.99938 0.1081 0.986 1 0.3902 0.7867 0.99668 0.2235 0.903 Model- and domain-scale behavior. Table 14 shows no systematic local degradation from 410M to 1.4B. Pooling scales, local NRMSE is 0.04260.0426 on CodeParrot, 0.10940.1094 on OpenWebText, and 0.11610.1161 on WikiText-103. At α=1α=1, 87.5%87.5\%, 37.5%37.5\%, and 41.7%41.7\% of the candidates in these domains satisfy the validity criterion, demonstrating that finite-scale range varies more by domain than by model size in these conditions. The overall median fitted local error exponent is 1.9211.921. Per-system exponents are (1.878,1.089,0.658)(1.878,1.089,0.658) at 410M, (1.988,1.916,1.936)(1.988,1.916,1.936) at 1B, and (1.992,1.951,2.025)(1.992,1.951,2.025) at 1.4B for CodeParrot, OpenWebText, and WikiText-103, respectively. Thus, the overall local trend is near quadratic, with the main deviations concentrated in the 410M OpenWebText and WikiText-103 conditions. Table 14: Language-model scaling summary in Experiment 1. Ranking correlations are medians over the three data-domain conditions at each scale. Scale Local NRMSE Cosine Valid 0.25 Valid 1 ISO-FD ρ 0.41B 0.1090 0.99918 0.958 0.417 0.714 1.0B 0.0519 0.99995 0.792 0.667 0.833 1.4B 0.0567 0.99994 0.875 0.583 0.762 Ranking at full shock scale. The exact one-step baseline is Si1step=|di,1(1)|,S_i^1step=|d_i,1(1)|, (89) which executes the complete nonlinear shock and control updates and evaluates both on the fixed probe, but sees no subsequent common-future propagation. It is therefore a strong counterfactual oracle that directly observes the immediate fixed-probe effect. Table 15 reports the system-wise results. ISO Tangent (FD) is positive in all nine conditions and exceeds gradient and parameter-write norms in eight. It exceeds exact one-step in five conditions, while exact one-step is higher in four; their medians are equal. The two statistics capture complementary information: one-step observes the immediate output, whereas ISO Tangent (FD) represents the propagated signed trajectory. Table 15: Language-model within-system Spearman correlation in Experiment 1 with Mi=maxh|di,h(1)|M_i= _h|d_i,h(1)|. Each condition contains eight candidates. Model Dataset ISO tangent (FD) Exact 1-step Gradient norm Parameter write 410M CodeParrot 0.857 0.643 -0.286 -0.262 410M OpenWebText 0.548 0.738 0.571 0.595 410M WikiText-103 0.714 0.643 0.571 0.619 1B CodeParrot 0.952 0.905 -0.167 -0.048 1B OpenWebText 0.333 0.929 0.071 -0.286 1B WikiText-103 0.833 0.762 -0.214 0.048 1.4B CodeParrot 0.976 0.905 -0.167 -0.167 1.4B OpenWebText 0.762 0.929 0.429 0.405 1.4B WikiText-103 0.619 0.667 -0.119 0.190 Median — 0.762 0.762 -0.119 0.048 As a descriptive secondary analysis, ranks normalized within each condition give pooled correlations 0.7910.791 for one-step and 0.7330.733 for ISO-FD. After linearly controlling one-step rank, the partial ISO-FD–target correlation is 0.3100.310; adding ISO-FD rank raises descriptive R2R^2 from 0.6260.626 to 0.6620.662. Because candidates are nested within nine fixed conditions, these values are descriptive and are not treated as 72 independent replications. The exact one-step response is a strong baseline in the pretrained-language-model setting and exceeds ISO-FD in several conditions. This does not contradict the finite-horizon mechanism: the one-step-matched controlled experiments show that immediate response is not sufficient in general, while the present scaling experiment tests whether the ISO tangent remains faithful at larger model scale. Signed extrema. At full scale, 51 of 72 candidate extrema are positive and 21 are negative. ISO-FD recovers the extremum sign for 65 candidates (90.3%90.3\%). Counts by domain are (24,0)(24,0) for CodeParrot, (13,11)(13,11) for OpenWebText, and (14,10)(14,10) for WikiText-103, where each pair is (positive, negative). This domain dependence reinforces the signed formulation: positive loss excursions are one subclass of the broader finite-horizon response. State-channel interactions. For the single candidate selected per system in this diagnostic, we evaluate the same eight post-shock hybrid states used in the smaller experiments. The most striking behavior occurs on WikiText-103: injecting m without its matched v produces maximum absolute responses 50.32250.322, 8.57488.5748, and 26.33926.339 at 410M, 1B, and 1.4B, whereas injecting the coupled (m,v)(m,v) deviations gives 0.010840.01084, 0.008860.00886, and 0.008470.00847. Full-state responses are similarly of order 10−210^-2. These extreme isolated-momentum hybrids demonstrate strong cross-channel coupling and off-trajectory sensitivity. We therefore interpret the intervention family jointly as an interaction probe. Computational cost and scope. Median complete runtime per system, including 500 continuation updates, eight shocks, the finite-difference grid, exact trajectories, and channel interventions, is approximately 464 seconds at 410M, 645 seconds at 1B, and 941 seconds at 1.4B. The nine measured runtimes sum to approximately 5,849 seconds and were parallelized over four GPUs. With one seed, this experiment provides a descriptive scaling comparison over three sizes of the Pythia family and three data domains. Multi-seed replication of the same mechanism is provided by the controlled and neural-network stages of Experiment 1. E.3 Experiment 2: Prospective Structure Under Unknown Futures Repeated-future construction. Experiment 1 conditions on one realized common future. Experiment 2 instead holds the pre-shock history, post-burn-in AdamW state, reference minibatches, candidate shock, fixed probe, optimizer configuration, and horizon fixed, and resamples only the unseen future minibatches. For each candidate i and future branch k∈1,…,Kk∈\1,…,K\, the control and shock trajectories share the same branch, so that di,h(k)=F(θi,t+hs,(k))−F(θt+hc,(k))d_i,h^(k)=F( _i,t+h^s,(k))-F( _t+h^c,(k)) (90) continues to isolate the response to the initiating shock within that branch. We use K=32K=32 throughout. In the controlled systems, the original burn-in state, four reference minibatches, 16 candidates, probes, and H=32H=32 are unchanged. The latent state of the correlated quadratic process immediately before the future is held fixed and each branch resamples only subsequent innovations. One branch is shared across all candidates and both probe views in a base system. The standard and readout_active results below correspond to the standard and anisotropic probes from Experiment 1; the rotating-readout probe is not used in this repeated-future experiment. In the neural systems, the original CIFAR-10 architecture, initialization, 100-step burn-in state, four reference minibatches, 12 candidates, fixed test probe, AdamW configuration, and H=12H=12 are unchanged. Each future branch independently resamples the subsequent CIFAR-10 training minibatches and is shared across candidates and candidate modes within a system. Candidate families and response summaries. Both settings evaluate the original natural gradient residuals and an exact-one-step-matched family. The matched family reuses the construction in Eq. (82): the common tangent target is 0.250.25 times the median natural one-step magnitude, the candidate-specific component is placed in the nullspace of the immediate output functional with four times the norm of the common component, and one nonnegative scalar per candidate is calibrated through the exact nonlinear AdamW update. No response at h>1h>1 and no resampled future branch is used in this construction. All candidates calibrate successfully. Table 18 shows that the resulting exact |d1||d_1| values are constant to numerical precision. For each candidate–branch pair we record Mi,k M_i,k :=maxh|di,h(k)|, := _h|d_i,h^(k)|, (91) Pi,k+ P^+_i,k :=maxh[di,h(k)]+, := _h[d_i,h^(k)]_+, (92) Ai,k A_i,k :=∑h|di,h(k)|. := _h|d_i,h^(k)|. (93) Magnitude M is the primary response used in the main text; P+P^+ and A are secondary checks. For a generic summary Ri,kR_i,k, define μiR:=1K∑k=1KRi,k, _i^R:= 1K _k=1^KR_i,k, (94) and the within-system prospective-structure ratio ΠH(R)=Vari(μiR)Vari(μiR)+i[Vark(Ri,k)]. _H(R)= Var_i( _i^R)Var_i( _i^R)+E_i[Var_k(R_i,k)]. (95) The implementation uses population normalization within the finite candidate and branch grids. ΠH _H is therefore a protocol-specific variance decomposition: it compares candidate-specific variation in the repeated-future mean with variation induced by resampling the future, and is not an information-theoretic percentage or a universal fraction of predictable risk. We additionally compute ρbranch:=mediankSpearman(R⋅,k,μR), _branch:=median_kSpearman(R_·,k,μ^R), (96) which measures the stability of candidate ordering across individual future branches. Present-time scores. The ranking target for a present-time score is the repeated-future conditional mean μiM _i^M. The scalar baselines are the exact one-step absolute response, gradient norm, and initial parameter-write norm from Eq. (58). We add two ISO-based scores that are computed without accessing any sampled future branch. The present-frozen ISO evaluates one local transition Jacobian A¯t A_t at the first post-control state using only the current reference minibatches, freezes the h=1h=1 probe readout c¯1 c_1, and uses SiPF=max1≤h≤H|c¯1⊤PθA¯th−1Btξi|.S_i^PF= _1≤ h≤ H | c_1 P_θ A_t^\,h-1B_t _i |. (97) The reference-surrogate ISO instead constructs a deterministic future using only the current reference-minibatch mean (controlled systems) or the current reference minibatches (neural systems), updates the resulting surrogate control state, readout, and tangent along that deterministic rollout, and takes the maximum absolute predicted response. The distinction from the pathwise Full ISO in Experiment 1 is essential: neither prospective score uses a realized or resampled future minibatch. E.3.1 Controlled Repeated Futures Table 16 reports the primary magnitude results. Under the standard probe, natural candidates have median Π32(M)=0.801 _32(M)=0.801, but the exact one-step response already correlates 0.8010.801 with μiM _i^M. After exact one-step matching, Π32(M)=0.730 _32(M)=0.730 and the median branch-ranking correlation remains 0.9090.909, despite the one-step magnitude being tied to numerical precision. The anisotropic probe gives the same qualitative conclusion, with matched Π32(M)=0.603 _32(M)=0.603 and branch correlation 0.8950.895. In contrast, the simple present-time scores do not recover the matched conditional-mean ordering in this controlled setting: present-frozen ISO correlations are −0.066-0.066 and 0.0310.031 for the standard and anisotropic probes. E.3.2 Neural-Network Repeated Futures The neural results preserve the same protocol while replacing the controlled quadratic process by independently resampled CIFAR-10 future minibatches. Matched MLP–GELU yields Π12(M)=0.921 _12(M)=0.921 with branch-ranking correlation 0.9550.955; matched CNN–ReLU yields 0.7000.700 and 0.8720.872. Present-time ISO scores are substantially more informative here than in the controlled systems. Present-frozen ISO correlates 0.9410.941 with μiM _i^M for MLP–GELU and 0.7550.755 for CNN–ReLU, while reference-surrogate ISO reaches 0.7520.752 and 0.8110.811. These results establish prospective relevance under the measured protocols, but they do not by themselves identify the conditions that make a present-time representation accurate. Table 16: Primary repeated-future magnitude results in Experiment 2. Each cell reports the median across independent systems; brackets give the system-level IQR. Branch ρ is the within-system median correlation between an individual future-branch ranking and the repeated-future mean ranking. System Candidates ΠH(M) _H(M) Branch ρ Quadratic, standard Natural 0.801[0.709,0.877]0.801\ [0.709,0.877] 0.919[0.866,0.949]0.919\ [0.866,0.949] Matched 0.730[0.594,0.845]0.730\ [0.594,0.845] 0.909[0.839,0.941]0.909\ [0.839,0.941] Quadratic, anisotropic Natural 0.718[0.602,0.825]0.718\ [0.602,0.825] 0.899[0.846,0.943]0.899\ [0.846,0.943] Matched 0.603[0.501,0.727]0.603\ [0.501,0.727] 0.895[0.844,0.931]0.895\ [0.844,0.931] MLP–GELU Natural 0.771[0.621,0.844]0.771\ [0.621,0.844] 0.862[0.795,0.914]0.862\ [0.795,0.914] Matched 0.921[0.885,0.951]0.921\ [0.885,0.951] 0.955[0.937,0.959]0.955\ [0.937,0.959] CNN–ReLU Natural 0.716[0.649,0.758]0.716\ [0.649,0.758] 0.855[0.823,0.900]0.855\ [0.823,0.900] Matched 0.700[0.595,0.771]0.700\ [0.595,0.771] 0.872[0.855,0.892]0.872\ [0.855,0.892] Table 17: Present-time Spearman correlation with the repeated-future mean magnitude μiM _i^M in Experiment 2. The exact one-step score is tied in the matched family and its correlation is therefore undefined. System Candidates Exact 1-step Grad. norm Write norm Frozen ISO Ref.-surrogate ISO Quadratic, standard Natural 0.801 0.212 0.253 0.659 0.372 Matched – 0.049 -0.057 -0.066 -0.021 Quadratic, anisotropic Natural 0.790 0.394 0.374 0.562 0.490 Matched – 0.272 0.025 0.031 0.025 MLP–GELU Natural 0.661 0.318 0.325 0.734 0.262 Matched – 0.066 0.339 0.941 0.752 CNN–ReLU Natural 0.619 0.538 0.510 0.752 0.601 Matched – 0.500 0.500 0.755 0.811 Table 18: Exact-one-step matching audit for Experiment 2. CV is computed within each candidate family; entries are medians with IQRs. Calibration success is 1.000 in every condition. Condition CV of |d1||d_1| Quadratic, standard ×10−12[×10−13,×10−12]1.07\!×\!10^-12\ [7.97\!×\!10^-13,1.75\!×\!10^-12] Quadratic, anisotropic ×10−12[×10−13,×10−12]1.33\!×\!10^-12\ [6.50\!×\!10^-13,1.84\!×\!10^-12] MLP–GELU ×10−13[×10−13,×10−13]3.57\!×\!10^-13\ [2.79\!×\!10^-13,5.73\!×\!10^-13] CNN–ReLU ×10−12[×10−12,×10−12]1.36\!×\!10^-12\ [1.08\!×\!10^-12,2.02\!×\!10^-12] Table 19: Absolute variance components for the matched primary magnitude. Values are medians of the system-specific quantities in Eq. (95); the ratio of the two medians need not equal the median of the system-specific ratios. Condition Between-candidate variance Mean within-future variance Quadratic, standard ×10−109.92\!×\!10^-10 ×10−102.39\!×\!10^-10 Quadratic, anisotropic ×10−84.25\!×\!10^-8 ×10−82.35\!×\!10^-8 MLP–GELU ×10−84.31\!×\!10^-8 ×10−95.45\!×\!10^-9 CNN–ReLU ×10−82.02\!×\!10^-8 ×10−81.13\!×\!10^-8 Table 20: Prospective-structure ratio for the primary magnitude and two secondary response summaries. P+P^+ is sign-sensitive; because the matched construction enforces alternating signed one-step targets, the positive-peak column is reported only as a descriptive robustness check and is not used to support the main prospective-identifiability claim. System Candidates ΠH(M) _H(M) ΠH(A) _H(A) ΠH(P+) _H(P^+) Quadratic, standard Natural 0.801 0.817 0.704 Matched 0.730 0.729 0.995 Quadratic, anisotropic Natural 0.718 0.722 0.570 Matched 0.603 0.603 0.990 MLP–GELU Natural 0.771 0.816 0.807 Matched 0.921 0.887 0.995 CNN–ReLU Natural 0.716 0.738 0.787 Matched 0.700 0.791 0.940 The combined result is deliberately narrower than an online-prediction claim. Repeated futures show that candidate identity can remain a strong source of finite-horizon variation even after the immediate output magnitude is matched, while the success of a particular present-time ISO representation varies substantially across systems. Experiment 2 therefore motivates prospective identifiability as a separate question without assuming that the pathwise ISO itself is deployable before the future trajectory is observed.