Paper deep dive
A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents
Indranil Halder, Rastri Dey, Cengiz Pehlevan
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Pre-training data poisoning of large language models is usually studied using targeted backdoors and their survival through safety post-training, which leaves open a more basic question: how does a model's clean data performance degrade as the poison rate $\varepsilon$ grows? Motivated by our controlled pre-training runs of OLMo-style models, in which the relative clean data validation perplexity increase $\Delta$ between poisoned and clean models matched in architecture, token budget, and optimization schedule is well fit by a power law $\Delta\approx C\varepsilon^{a}$ with a non-integer exponent, we ask what such a law requires theoretically. We first prove an analyticity barrier: whenever the contaminated objective depends analytically on $\varepsilon$ around a nondegenerate clean data optimum, $\Delta$ is generically quadratic in $\varepsilon$, so a generic non-integer exponent is a signature of genuinely singular structure. We then supply that structure in solvable truncated ridge regression with heavy-tailed covariates, controlled by $q_\star$, and a label-shift poisoning. Our central result is that the excess risk scaling exponent depending on the order of limits: in the higher dimensional proportional regime it is $\epsilon^{q_\star/(q_\star+2)}$, whereas taking the ample-data limit first gives $\epsilon^{2-2/q_\star}$, and the limits do not commute. We confirm this prediction through several numerical simulations. Finally, we argue that finite training time plays the role of truncation on the curvature spectrum in local LLM pre-training, deriving the observed scaling law under heavy tailed inverse curvature spectrum as a modeling hypothesis.
Tags
Links
- Source: https://arxiv.org/abs/2609.32288v1
- Canonical: https://arxiv.org/abs/2609.32288v1
Trouble viewing inline? Open PDF directly →
Full Text
261,787 characters extracted from source content.
Expand or collapse full text
A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents Indranil Halder Affiliation: John A. Paulson School of Engineering and Applied Sciences at Harvard University Rastri Dey Affiliation: PACCAR Inc. Cengiz Pehlevan Affiliation: John A. Paulson School of Engineering and Applied Sciences at Harvard University Affiliation: Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University Affiliation: Center for Brain Science, Harvard University Abstract Pre-training data poisoning of large language models is usually studied using targeted backdoors and their survival through safety post-training, which leaves open a more basic question: how does a model’s clean data performance degrade as the poison rate ε grows? Motivated by our controlled pre-training runs of OLMo-style models, in which the relative clean data validation perplexity increase Δ between poisoned and clean models matched in architecture, token budget, and optimization schedule is well fit by a power law Δ≈Cεa ≈ C ^a with a non-integer exponent, we ask what such a law requires theoretically. We first prove an analyticity barrier: whenever the contaminated objective depends analytically on ε around a nondegenerate clean data optimum, Δ is generically quadratic in ε , so a generic non-integer exponent is a signature of genuinely singular structure. We then supply that structure in solvable truncated ridge regression with heavy-tailed covariates, controlled by q⋆q_ , and a label-shift poisoning. Our central result is that the excess risk scaling exponent depending on the order of limits: in the higher dimensional proportional regime it is ϵq⋆/(q⋆+2)ε^q_ /(q_ +2), whereas taking the ample-data limit first gives ϵ2−2/q⋆ε^2-2/q_ , and the limits do not commute. We confirm this prediction through several numerical simulations. Finally, we argue that finite training time plays the role of a poison-dependent truncation on the curvature spectrum in local LLM pre-training, deriving the observed scaling law under heavy tailed inverse curvature spectrum as a modeling hypothesis. 1 Introduction Large language models are trained on massive, heterogeneous corpora whose quality and provenance are difficult to audit. Pre-training data poisoning can implant persistent behaviors that survive later safety post-training, sometimes from relatively few poisoned examples (Wallace et al., 2021; Carlini et al., 2024; Zhang et al., 2025; Souly et al., 2025; Bowen et al., 2024). In parallel, validation loss on clean data follows predictable power laws in dataset size, parameters, and compute (Kaplan et al., 2020; Hoffmann et al., 2022). Standard scaling laws, however, do not describe how model quality changes when a controlled fraction of pre-training tokens is corrupted. We study this question: how does degradation on clean evaluation data scale with the pre-training poison rate? Concretely, we focus on the relative increase in clean-data perplexity, Δ(ϵ):=Ppoisoned(ϵ)Pclean−1, (ε):= P_poisoned(ε)P_clean-1, (1) where Ppoisoned(ϵ)P_poisoned(ε) is the clean-evaluation perplexity of a model trained at poison rate ϵε, and PcleanP_clean that of a matched clean model. We do not measure attack success against a targeted backdoor, instead, our outcome is aggregate clean language-modeling quality. Empirically, we find an approximate power law (Figure 1 and Appendix D) Δ(ϵ)≈Cϵa. (ε)≈ C\,ε^a. (2) The non-integer exponent a depends on the model parameter and training time. This finding is more surprising than it looks - our technical contributions below explain why. (a) (b) Figure 1: (a) Random-poisoning scaling curves at multiple training checkpoints for one model (774774M): over-training the slope and intercept drift systematically. (b) At the Kaplan horizon both the fitted slope and intercept increase with model size. Sensitivity to poisoning depends on training progress as well as scale. Technical contributions: 1. Analyticity barrier. We prove if the contaminated objective is analytic in ϵε near a nondegenerate clean data optimum, the learned parameters move by O(ϵ)O(ε) and the clean excess risk is generically quadratic, forcing a=2a=2 (Result 1). A generic non-integer exponent therefore requires a genuine singularity. In this paper, we point out a mechanism for it as we discuss below. 2. Breaking the analyticity barrier. We show that heavy-tailed data combined with a truncation whose scale adapts to poisoning supplies the required singularity needed to break the analytical barrier discussed above and yields a fractional exponent (Result 2): a higher truncation threshold preserves more clean tail mass but admits more influence from poison, and balancing the two fixes the law. 3. A solvable setup: random coordinate model. We make the mechanism fully explicit in a model simple enough to solve exactly yet rich enough to display the phenomenon: high-dimensional truncated ridge regression on heavy-tailed data (tail controlled by a parameter q⋆q_ ). We work in P dimensions with D samples drawn with a label-shift poisoning method (Section 4). We focus on P→∞P→∞ at fixed aspect ratio ϕ=P/Dφ=P/D. Our main result (Result 9) is a phenomenon we believe is new for data poisoning: the optimized excess risk has a scaling law featuring two distinct scaling exponents depending on the order of the aspect-ratio limit ϕ→0φ→ 0 and the small-poison limit ϵ→0ε→ 0. Holding ϕφ fixed gives ϵq⋆/(q⋆+2)ε^q_ /(q_ +2), whereas taking ϕ→0φ→ 0 first gives ϵ2−2/q⋆ε^2-2/q_ . That is the poisoning-sensitivity exponent is regime-dependent. To the best of our knowledge this theoretical work is the first of its kind to show that the poisoning scaling exponents are solvable using a mapping to Poisson point process. Because the model is controllable, we validate it on its own terms (Figure 2 and 3): finite-(P,D)(P,D) simulations converge to the deterministic equivalent, the two exponents match their predicted values across q⋆q_ and the fitted exponent crosses over between the two regimes as ϕφ varies. 4. A possible bridge to local LLM training dynamics. Finally (Section 5, Appendix B) we argue that in a local approximation to pre-training, finite training time acts as a truncation of the inverse-curvature spectrum of the preconditioned Gauss–Newton operator; under a heavy-tailed inverse-curvature assumption (see Figure 4 and Appendix E for numerical evidence) this reproduces the observed scaling law. 2 Related Work Scaling law for language-model pre-training. Empirical scaling laws show that clean pre-training loss for a fixed choice of optimizer and scheduler follows predictable power-law relationships individually with model size, dataset size, and training compute (Kaplan et al., 2020). Hoffmann et al. (2022) refined the picture by showing how to scale model size and training data at a fixed compute budget. A complementary, theoretical line of work seeks to derive such power laws from first principles rather than fit them: they tie the loss exponent to the tail of the kernel spectrum or to the intrinsic dimension of the data manifold (Spigler et al., 2020; Sharma & Kaplan, 2020; Bordelon et al., 2020; Bahri et al., 2024; Maloney et al., 2022); and dynamical learning-curve models connect the exponents to training time (Hutter, 2021; Michaud et al., 2023; Bordelon et al., 2024; Cagnetta et al., 2026). Language-model pre-train data poisoning and alignment. Poisoning attacks have been used to implant concealed triggers or backdoors that manipulate predictions while keeping the poisoned examples difficult to detect (Wallace et al., 2021). Other work studies backdoors in pre-trained models and the persistence of undesirable behaviors through further training, fine-tuning, or safety training (Li et al., 2021; Hubinger et al., 2024). More recently, Carlini et al. (2024) showed that poisoning web-scale datasets can be practical under realistic data-collection pipelines, while Zhang et al. (2025) studied persistent poisoning introduced during LLM pre-training and Souly et al. (2025) argued that successful LLM poisoning may require only a near-constant number of poison samples rather than a fixed poisoning fraction. A related line of work studies how safety and alignment signals should be incorporated during pre-training: Silverstein et al. (2026) discusses architectural modifications for increased alignment, Maini et al. (2025) propose safety pre-training as a data-centric alternative to purely post-hoc alignment, and Sam et al. (2026) studies when safety interventions should be introduced during the pre-training curriculum. Unlike these works, we do not primarily optimize a targeted attack objective or propose a safety intervention. Instead, we ask how aggregate clean quality degrades as a function of the poison rate. A recent line of work incorporates data quality into the scaling law directly (Subramanyam et al., 2026) by proposing an analytical dependence on poison fraction. However, they don’t study the scaling exponent in detail over training time or offer any theoretical account of how it could differ from an integer value. In this paper, we provide a mechanism for that. Power laws, and latent cluster structure of language models. Probabilistic and Bayesian approaches to natural-language modeling have a long history of using Dirichlet- and Pitman–Yor-process priors to capture the structure of the word and document distributions (MacKay & Peto, 1995; Teh, 2006; Johnson et al., 2006; Liang et al., 2007); the stick-breaking construction of the Poisson–Dirichlet distribution has a self-similar form that gives rise to power-law tails on the ranked cluster weights (Picard & Pitman, 2006). More recent work has investigated the mechanisms by which neural networks acquire linguistic and compositional structure (Arora et al., 2016; Karkada et al., 2025; Korchinski et al., 2025; Cagnetta et al., 2024; Cagnetta & Wyart, 2024; Parley et al., 2026; Karkada et al., 2026), and a parallel line of work has argued that the power-law structure of natural language is itself a primary driver of the empirical scaling laws of language models (Spigler et al., 2020; Bordelon et al., 2020; Bahri et al., 2024; Michaud et al., 2023; Cagnetta et al., 2026) ; complementary perspectives are developed in Barkeshli et al. (2026); Liu et al. (2026). Liu et al. (2025) showed that the hidden activations of a transformer exhibit consistent power-law spectral decay. Recently, Halder et al. (2026) proposed a spin-glass/weighted-graph-based generative model of binary language in which ideas form distinct clusters in the Poisson–Dirichlet hierarchy. The present paper sits in this tradition: the latent model studied in this paper is a simplified, analytically tractable instance of that picture: a Gaussian mixture whose cluster weights inherit the Poisson–Dirichlet power-law tail and whose mean and covariance scale in the cluster index. Test-time adversarial vulnerability and train-time contamination. Classical work shows that small, human-imperceptible input perturbations at inference time can induce large changes in model predictions (Szegedy et al., 2014). Proposed explanations include local linearity (Goodfellow et al., 2015), boundary geometry (Tanay & Griffin, 2016), high-dimensional concentration effects (Gilmer et al., 2018), and non-robust but predictive features (Ilyas et al., 2019). More recently, Salvatore et al. (2026) propose a geometric explanation based on exponential misalignment between machine and human perception: the set of inputs that a network confidently assigns to a concept can have a much larger intrinsic dimension than the corresponding human or natural concept manifold. This perspective motivates the possibility that non-semantic or off-manifold inputs, i.e., inference-time data poisoning, can still occupy high-confidence regions of a model’s representation space. Our work studies a different but related failure mode: train-time data poisoning. Early work studied data poisoning in classical machine-learning models, including support vector machines (Biggio et al., 2012). Subsequent work developed stronger poisoning attacks and defenses, often emphasizing the difficulty of defending against adaptive poisoning in high-dimensional settings (Steinhardt et al., 2017; Koh et al., 2018). Our theoretical analysis is related to robust estimation under contamination and heavy-tailed sampling (Huber, 1964; Catoni, 2012; Lugosi & Mendelson, 2019). 3 The analyticity barrier and how to overcome it In this section, we first present the empirical scaling law in large language models that motivates the rest of the paper and then discuss theoretical considerations behind it. The details of the pre-training set up of the languge model is discussed in Appendix D. We isolate how a model’s clean language-modeling quality degrades as a controlled fraction of its pre-training tokens is corrupted. Because we target overall degradation rather than a backdoor, our poison fraction is larger than in attack-focused work (Zhang et al., 2025; Souly et al., 2025). For each poison rate ϵε we train a poisoned model and a clean baseline sharing architecture, token budget, and optimization schedule, evaluate both on the same held-out clean data, and report, at matched training time t, Δ(ϵ,t)=Ppoisoned(ϵ,t)Pclean(t)−1=exp(ℛpoisoned(ϵ,t)−ℛclean(t))−1, (ε,t)= P_poisoned(ε,t)P_clean(t)-1= \! (R_poisoned(ε,t)-R_clean(t) )-1, (3) In the small-degradation regime we are interested in Δ≈ℛpoisoned−ℛclean _poisoned-R_clean. Aggregating checkpoints within 15%15\% of a target token count, across model sizes and across training, we find (see Figures 1) that the degradation is approximately power-law in ϵε (see equation 2). The slope and intercept drift over training and with model size. The rest of the paper asks what produces such a law. Solvable models of neural scaling laws almost invariably express the test loss through analytic functionals of a finite collection of order parameters (Spigler et al., 2020; Sharma & Kaplan, 2020; Bordelon et al., 2020; Bahri et al., 2024; Maloney et al., 2022). Say injecting poison at rate ε perturbs these order parameters analytically, so the induced loss is analytic in ε at ε=0 =0 and its expansion carries integer powers only; taken about the clean-data optimum of the validation loss, where the gradient vanishes, the leading correction is generically quadratic. The next result sharpens this observation into a no-go statement. We state the obstruction at the natural expansion point, the clean-data optimum of the validation loss; in Section 5 we treat the more general situation in which the reference is not this optimum - the clean checkpoint of an early-stopped run, at which the validation loss gradient need not vanish-and show that it introduces a linear term but fails to explain the factional scaling exponent observed in stronger LLM during pre-training, instead, the required mechanism essentially follows from the precise understanding the assumptions of the no-go theorem below. Result 1 (Analyticity barrier). Let θ∈Θθ∈ denote the model parameters, P the clean and Q the poison distribution, and sample from ε=(1−ε)+εP_ =(1- )P+ . Suppose the population loss is Lε(θ)=(1−ε)L(θ)+εL(θ)L_ (θ)=(1- )L_P(θ)+ L_Q(θ) with L,L_P,L_Q real analytic near a clean solution θ0 _0 that is a nondegenerate optimum, ∇L(θ0)=0∇ L_P( _0)=0, H:=∇2L(θ0)H:=∇^2L_P( _0), λmin(H)>0 _ (H)>0. Let the clean evaluation risk ℛR be real-analytic with ∇ℛ(θ0)=0 ( _0)=0, G:=∇2ℛ(θ0)G:=∇^2R( _0), λmin(G)>0 _ (G)>0, and set g:=∂ε∇Lε(θ0)|ε=0=∇L(θ0)−∇L(θ0)g:= _ ∇ L_ ( _0)|_ =0=∇ L_Q( _0)-∇ L_P( _0). Then, for all sufficiently small ε , there is a unique analytic branch of stationary points θε _ near θ0 _0, and θε−θ0 _ - _0 =−εH−1g+O(ε2), =- H^-1g+O( ^2), (4) Δ(ε):=ℛ(θε)−ℛ(θ0) ( ):=R( _ )-R( _0) =ε22g⊤H−1GH−1g+O(ε3). = ^22g H^-1GH^-1g+O( ^3). (5) More generally, analyticity allows only integer powers of ε , stationarity removes the linear term. See Appendix C.1 for the proof. Escaping the barrier therefore requires a genuinely non-analytic ingredient rather than a more elaborate smooth model; the remainder of this section supplies one, showing that a heavy-tailed spectrum together with an optimized poison dependent truncation generates precisely the singularity a fractional exponent demands. Result 2 (Adaptive-truncation law). Let ψ(Z)∈ℝmψ(Z) ^m be a statistic with clean mean μ=[ψ(Z)]μ=E_P[ψ(Z)]. For τ>0τ>0 let TτT_τ be a truncation map, and let the poison distribution εQ_ depend on ε . Define mε,τ:=(1−ε)[Tτ(ψ(Z))]+εε[Tτ(ψ(Z))]m_ ,τ:=(1- )E_P[T_τ(ψ(Z))]+ _Q_ [T_τ(ψ(Z))]. Assume that, as τ→∞τ→∞, there exist β,γ>0β,γ>0 and vectors b,cb,c with [Tτ(ψ(Z))]−μ _P[T_τ(ψ(Z))]-μ =bτ−β+o(τ−β), =bτ^-β+o(τ^-β), (6) ε[Tτ(ψ(Z))]−[Tτ(ψ(Z))] _Q_ [T_τ(ψ(Z))]-E_P[T_τ(ψ(Z))] =cτγ+o(τγ). =cτ^γ+o(τ^γ). (7) Let the learned parameter be a smooth θε,τ=Φ(mε,τ) _ ,τ= (m_ ,τ), Φ(μ)=θ0 (μ)= _0, J:=DΦ(μ)J:=D (μ), with clean risk ℛR being C3C^3 near θ0 _0, ∇ℛ(θ0)=0 ( _0)=0, G=∇2ℛ(θ0)G=∇^2R( _0), λmin(G)>0 _ (G)>0. For fixed t>0t>0, choose τε=tε−1/(β+γ) _ =t\, ^-1/(β+γ) and set ht:=t−βb+tγch_t:=t^-βb+t^γc. Then mε,τε−μ m_ , _ -μ =εβ/(β+γ)ht+o(εβ/(β+γ)), = ^β/(β+γ)h_t+o( ^β/(β+γ)), (8) ℛ(θε,τε)−ℛ(θ0) ( _ , _ )-R( _0) =12ht⊤J⊤GJhtε2β/(β+γ)+o(ε2β/(β+γ)). = 12h_t J GJh_t\, ^2β/(β+γ)+o( ^2β/(β+γ)). (9) Thus, whenever ht⊤J⊤GJht>0h_t J GJh_t>0, adaptive truncation produces the power-law exponent a=2β/(β+γ)a=2β/(β+γ), which need not be an integer. The proof is in Appendix C.2. Crucially, truncation alone is not enough: a finite exponent β is what pushes a below the analytic value 22. For example, if a scalar statistic ψ(Z)ψ(Z) has tail on clean data ℙ(|ψ(Z)|>u)∼cu−q,q>1,P\! ( ψ(Z) >u ) cu^-q, q>1, (10) then clipping at level τ typically produces ‖EP[Tτ(ψ(Z))]−μ‖≍∫τ∞ℙ(|ψ(Z)|>u)u≍τ−(q−1). \|E_P[T_τ(ψ(Z))]-μ \| _τ^∞P\! ( ψ(Z) >u )\,du τ^-(q-1). (11) finite β=q−1β=q-1. We show that it is realized within a solvable model in the next section. 4 A solvable model and the non-commuting scaling limits Heavy-tailed data carry signal in rare examples; a truncation at level τ discards the tail beyond τ, so raising τ lowers the clean data truncation bias but lets poisoned data influence prediction more strongly. Balancing the two over τ produces a fractional exponent. In the rest of this section we explain this in more details. 4.1 The random co-ordinate model Our model is truncated linear regression in higher dimensions on heavy tailed data. We work in P dimensions. Let e1,…,ePe_1,…,e_P denote the standard basis of ℝPR^P. Let U be a random variable that has a heavy-tail ℙ(U>t)=t−q⋆. (U>t)=t^-q_ . (12) A clean example draws a coordinate J∼Unif1,…,PJ \1,…,P\ and U, and sets Xc=PUeJX_c= PU\,e_J, Yc=cUY_c=c U, that is the teacher weight is θ⋆,P=cP _ ,P= c P1; a poison example sets Xp=PeJX_p= Pe_J, Yp=c+ηpτY_p=c+ _pτ. Each of D i.i.d. samples is clean with probability 1−ϵ1-ε and poisoned with probability ϵε. Truncating the per-example second-moment and cross-moment statistics at level τ acts as A(τ):=X⊤min1,Pτ‖XX⊤‖op,b(τ):=XYmin1,Pτ‖XY‖2. A^(τ):=X \1, Pτ X _op \, b^(τ):=XY \1, P\,τ XY _2 \. (13) Given D training samples, define Σ^ϵ,τ:=1D∑i=1DAi(τ),v^ϵ,τ:=1D∑i=1Dbi(τ). _ε,τ:= 1D _i=1^DA_i^(τ), v_ε,τ:= 1D _i=1^Db_i^(τ). (14) For ϱ>0 >0, define the ridge estimator θ^ϵ,τ,ϱ:=(Σ^ϵ,τ+ϱIP)−1v^ϵ,τ. θ_ε,τ, :=( _ε,τ+ I_P)^-1 v_ε,τ. (15) 4.2 The simplification and map to Poisson point process For coordinate j, let Nj,cN_j,c and Nj,pN_j,p denote the numbers of clean and poisoned training samples assigned to coordinate j. Let Uj,1,Uj,2,…U_j,1,U_j,2,… denote the clean radial variables assigned to that coordinate. Define aτ(U)=min(U,τ)a_τ(U)= (U,τ) and rτ(U)=cmin(U,τ/c)r_τ(U)=c (U,τ/c) and Sj,A(D):=∑ℓ=1Nj,caτ(Uj,ℓ)+Nj,p,Sj,B(D):=∑ℓ=1Nj,crτ(Uj,ℓ)+(c+ηpτ)Nj,p. S_j,A^(D):= _ =1^N_j,ca_τ(U_j, )+N_j,p, S_j,B^(D):= _ =1^N_j,cr_τ(U_j, )+(c+ _pτ)N_j,p. (16) We show that (see Appendix A.2) the truncated ridge estimator is coordinate-separable, P(θ^ϵ,τ,ϱ)j=GϕD,ϱ(Sj,A(D),Sj,B(D)) P\,( θ_ε,τ, )_j=G_ _D, (S_j,A^(D),S_j,B^(D) ) (17) where ϕ=P/Dφ=P/D and, Gϕ,ϱ(sA,sB):=ϕsBϱ+ϕsA,ϱ+ϕsA>0,0,ϱ=0,sA=0.G_φ, (s_A,s_B):= cases φ s_B +φ s_A,& +φ s_A>0,\\[8.53581pt] 0,& =0,\ s_A=0. cases (18) As a consequence, when P,D→∞P,D→∞ with ϕφ held fixed, the clean and poison hits on a fixed coordinate converge to independent Poisson variables Np∼Pois(ϵϕ),Nc∼Pois(1−ϵϕ),N_p \! ( εφ ), N_c \! ( 1-εφ ), (19) and the risk concentrates through the standard scalar deterministic equivalent. Then, ℛclean,P(θ^ϵ,τ,ϱ)−ℛclean,P(θ⋆,P)→ℙℛϕ,ϱ(ϵ,τ):=μU[(c−Gϕ,ϱ(SAϵ,τ,SBϵ,τ))2], _clean,P( θ_ε,τ, )-R_clean,P( _ ,P) PR_φ, (ε,τ):= _U\,E[(c-G_φ, (S_A^ε,τ,S_B^ε,τ))^2], (20) where μU:=U=q⋆q⋆−1 _U:=EU= q_ q_ -1, and SAϵ,τ:=∑ℓ=1Ncaτ(Uℓ)+Np,SBϵ,τ:=∑ℓ=1Ncrτ(Uℓ)+(c+ηpτ)Np.S_A^ε,τ:= _ =1^N_ca_τ(U_ )+N_p,S_B^ε,τ:= _ =1^N_cr_τ(U_ )+(c+ _pτ)N_p. This form helps us to identify this as a Poisson point process. More precisely, the clean samples reaching a coordinate form a Poisson process of intensity (1−ϵ)/ϕ(1-ε)/φ , each point carrying the paired mark (aτ(Uℓ),rτ(Uℓ))(a_τ(U_ ),r_τ(U_ )), while the poisoned samples form an independent Poisson process of intensity ϵ/ϕε/φ with the constant mark (1,c+ηpτ)(1,\,c+ _pτ). For a Poisson-indexed sum the joint Laplace functional factorizes over the intensity, for example, exp(−∑ℓ=1Nct⋅(aτ,rτ)(Uℓ))=exp(1−ϵϕ∫(e−t⋅(aτ(u),rτ(u))−1)dν(u)),E\, \! (-\!\! _ =1^N_ct\!·\!(a_τ,r_τ)(U_ ) )= \! ( 1-εφ\! \! (e^-t·(a_τ(u),r_τ(u))-1 )\,dν(u) ), so every moment of (SA,SB)(S_A,S_B) and, decisively, its τ→∞τ→∞ asymptotics, reduces to a single integral against the mark intensity. This allows a detailed analytical study. (a) Scaling exponents vs. q⋆q_ . (b) Scaling exponents vs. ϕφ. Figure 2: (2(a)) The measured optimized exponent follows q⋆/(q⋆+2)q_ /(q_ +2) (proportional) and 2−2/q⋆2-2/q_ (ample) across q⋆∈[1.5,4]q_ ∈[1.5,4]; the branches are well separated. (2(b)) For q⋆=2q_ =2, fitting the exponent over a fixed ϵε-window as ϕ=P/Dφ=P/D varies shows a clean cross-over from the ample value 1.01.0 to the proportional value 0.50.5. (a) Higher dimensional limit, scaling law as ϵq⋆/(q⋆+2)ε^q_ /(q_ +2). (b) Ample data limit, scaling law as ϵ2−2/q⋆ε^2-2/q_ . Figure 3: At q⋆=2q_ =2 the optimized degradation follows the predicted proportional (3(a)) and ample (3(b)) slopes simulation (points) against the exact prediction (dashed). 4.3 Optimal truncation and non-commuting limits of the scaling law The clean truncation bias falls as τ grows while the poison influence rises, so there is a bias–poison optimum. Optimizing τ yields the paper’s central theoretical result on the scaling law: Result 3 (High-dimensional vs. ample-data limit). Work at ridgeless limit ϱ=0 =0 with P/D→ϕ∈(0,∞)P/D→φ∈(0,∞). The observable of interest is the clean data excess risk of the poisoned, truncated estimator against the clean, untruncated one, Δϵ,τ(P):=ℛ(θ^ϵ,τ,0)−ℛ(θ^0,∞,0) ^(P)_ε,τ:=R( θ_ε,τ,0)-R( θ_0,∞,0). (i) High-dimensional limit (fixed ϕφ, ϵ↓0ε\! \!0): Δϵ,τ(P)=Kϕ,c,q⋆cleanτ−q⋆+Cϕ,0ϵτ2\ ^(P)_ε,τ=K^clean_φ,c,q_ τ^-q_ +C_φ,0\,ετ^2, τopt≍ϵ−1/(q⋆+2)\ _opt ε^-1/(q_ +2), minτΔϵ,τ(P)≍ϵq⋆/(q⋆+2)\ _τ ^(P)_ε,τ ε^q_ /(q_ +2). (i) Ample-data limit (ϕ↓0φ\! \!0 first and then ϵ↓0ε\! \!0): Δϵ,τ(P)=1μU(Aτ−(q⋆−1)+Bϵτ)2\ ^(P)_ε,τ= 1 _U (A\,τ^-(q_ -1)+B\,ετ )^2, τopt≍ϵ−1/q⋆\ _opt ε^-1/q_ , minτΔϵ,τ(P)≍ϵ2−2/q⋆\ _τ ^(P)_ε,τ ε^2-2/q_ . The full statement, constants, and proof are Result 9 in Appendix A.7 (built on the deterministic equivalent of Appendix A.4 and the analyses of Appendices A.5–A.6). An intuitive reason for the non-commutativity is as follows: • Higher dimensional limit: At fixed ϕφ each coordinate has only Nc∼Pois(1−ϵϕ)=O(1)N_c \! ( 1-εφ )=O(1) clean samples. A coordinate has Np≥1N_p≥ 1 with probability ≈ϵ/ϕ≈ε/φ. On a coordinate that has Np≥1N_p≥ 1 the poison weight is therefore not small. There is not enough clean data on the coordinate to outweigh the single poison entry, so it moves the ratio by O(τ)O(τ). The squared error on that coordinate is order τ2τ^2. Here ϵε enters only as the rarity of being hit: a fraction ϵε of coordinates carry a τ2τ^2 error and the rest carry none, this leads to ϵτ2ετ^2 term in excess risk. • Ample-data limit: Every coordinate is hit by both large number of clean and poisoned samples since Nc∼(1−ϵ)/ϕ→∞N_c (1-ε)/φ→∞ and Np∼ϵ/ϕ→∞N_p ε/φ→∞. Per coordinate influence of the poisoning is diluted to ϵτετ and it enters the excess risk as (ϵτ)2(ετ)^2. Here ϵε measures how much poison sits on every coordinate. Case (i) is exactly the adaptive-truncation law (Result 2) with β=q⋆−1β=q_ -1, γ=1γ=1. The two exponents cross at q⋆=5−1≈1.236q_ = 5-1≈ 1.236: for q⋆>5−1q_ > 5-1 one has q⋆/(q⋆+2)<2−2/q⋆q_ /(q_ +2)<2-2/q_ , so the limits do not commute and the proportional regime is strictly more fragile; for 1<q⋆<5−11<q_ < 5-1 the ordering reverses. We verify these numerically in Figure 2 and 3. 5 How the scaling law may arise in LLM pre-training We now connect the mechanism to language models - the link requires a couple of modeling hypotheses within a local quadratic approximation to the training dynamics around a late time checkpoint (Meterez et al., 2026), developed in full in Appendix B. A preconditioned (by operator S) optimizer, run for time T starting from the checkpoint, acts as a filter qT(λ)=(1−e−λT)/λq_T(λ)=(1-e^-λ T)/λ on the spectrum of the preconditioned Gauss–Newton curvature A (Result 11). The filtered spectrum is responsible for learning away from the checkpoint: it learns well-conditioned directions, i.e., λT≫1λ T 1, quickly and leaves flat ones, i.e., λT≪1λ T 1, under-learned, so finite time is equivalent to Cut-off at τ=T on the inverse-curvature variable X=λ−1, λ∈spec(A).Cut-off at $τ=T$ on the inverse-curvature variable $X=λ^-1$, $λ (A)$. (21) In addition, we argue X is heavy-tailed, ℙ(X>τ)≍τ−α,τ→∞.P (X>τ ) τ^-α, τ→∞. (22) See Figure 4 for numerical justification. These two facts supply similar mechanism as in the previous section and lead to the scaling law as we discuss in the rest of the section. Figure 4: The plot corresponds to an experiment performed on OLMo-style model with total parameters P=1.677×108P=1.677× 10^8 pre-trained upto 33B tokens. The graph on the left shows that heavy tailed curvature spectrum exists and, the graph on the right determines the exponent. Red shades show where numerical analysis is unreliable. For the details of R,α^2R, α_2 see Appendix E, these are a specially constructed function of z; smaller z probes a smaller eigenvalue spectrum. The ratio test in Corollary 32 shows that R→1R→ 1 asserts a heavy-tailed spectrum of X=λ−1X=λ^-1 with hard edge exponent α≤1α≤ 1. On the other hand Remark 5 shows α^2 α_2 approaches the value of α≈0.33α≈ 0.33. The clean data training for time T is captured by weight ϕ0(T) _0(T), which in an infinite time limit leads to ϕ0⋆ _0 . With e0(T):=ϕ0(T)−ϕ0⋆e_0(T):= _0(T)- _0 the clean truncation residual and G the clean-risk curvature S1/2∇2ℛ(ϕ0⋆)S1/2S^1/2\,∇^2R_P( _0 )\,S^1/2, clean truncation bias is given by B(T)2:=e0⊤Ge0B(T)^2:=e_0 Ge_0. Poison at rate ϵε shifts the trajectory to ϕϵ(T)=ϕ0(T)+ϵψ(T)+o(ϵ) _ε(T)= _0(T)+ε\,ψ(T)+o(ε), where the poison-response direction itself splits into a checkpoint offset carried over from a poisoned late-time checkpoint and a forced response built up afterwards, ψ(T)=e−H0Tψin+ψf(T)ψ(T)=e^-H_0Tψ^in+ _f(T). The poison influence built-up is captured by I(T)2:=ψf⊤GψfI(T)^2:= _f G _f. The measured observable is the degradation at matched training, δℛ(ϵ,T):=ℛ(ϕϵ(T))−ℛ(ϕ0(T)) (ε,T):=R_P( _ε(T))-R_P( _0(T)). It resolves the response into four competing contributions, δℛ≍ϵΓ⋆T−βin⏟checkpoint offset+ϵΓ⋆Tγ⏟train–eval gap+ϵB(T)I(T)⏟clean bias≍ϵTγ−β+12ϵ2I(T)2⏟pure poison≍ϵ2T2γ, \; \; ε\, _ \,T^- _in_checkpoint offset\;+\; ε\, _ \,T^γ_train--eval gap\;+\; ε\,B(T)I(T)_clean bias\ \,ε\,T^γ-β\;+\; 12ε^2I(T)^2_pure poison\ \,ε^2\,T^2γ, (23) where Γ⋆:=‖g⋆‖G−1,g⋆:=∇ℛ(ϕ0⋆) _ :=\|g_ \|_G^-1,g_ := _P( _0 ) is the train-eval mismatch. We show B(T)≍T−βB(T) T^-β (Result 17) and I(T)≍TγI(T) T^γ (Result 19) with β>γ>0β>γ>0 (that β>γβ>γ says the clean signal is learned faster than the poison accumulates) and Bin(T):=‖e−H0Tψin‖G≍T−βin,βin>0B_in(T):= \|e^-H_0Tψ^in \|_G T^- _in, _in>0. These are power laws for the same reason-the spectrum is scale-free near weak curvature modes, so advancing the cutoff rescales every quantity by a pure power. The clean bias decays because it is the residual clean signal still sitting beyond the moving cutoff; each increment of time resolves more of the tail and the leftover shrinks. The poison influence grows because the poison’s leverage lives in the low-curvature directions: there a small contamination produces a large parameter displacement, and a longer time horizon reaches ever more such directions, so the aggregate imprint climbs up. Gap-dominated crossover. When the β-independent train–eval gap leads, ϵΓ⋆I(T)ε\, _ I(T) and 12ϵ2I(T)2 12ε^2I(T)^2 cross at T(g)ϵ≍ϵ−1/γT^(g)_ε ε^-1/γ and, Δ≍Γ⋆ϵTγ for T≪T(g)ϵ,Δ≍ϵ2T2γ for T≫T(g)ϵ.(Result 20) _ \,ε\,T^γ for T T^(g)_ε, ε^2T^2γ for T T^(g)_ε. (Result~ thm:scaling-law-gap) (24) Both competing terms grow with T, so there is no interior optimal horizon. The degradation is ϵε-independent at the crossover time leading to a=0a=0. Bias-dominated crossover. When a decaying heavy-tail term leads, the falling clean bias ϵB(T)I(T)≍ϵTγ−βε\,B(T)I(T) ε T^γ-β and the rising pure term balance at a poison-dependent cross-over time Tϵ≍ϵ−1/(β+γ),T_ε \,ε^-1/(β+γ), an interior minimum, where the scaling law emerges, Δ(ϵ)≍ϵa,a=2β+γ(Result 21). (ε) ε^a, a= 2β+γ (Result~ thm:scaling-law-optimal-truncation). (25) The checkpoint offset drives the same crossover with β replaced by βin+γ _in+γ, the two share their origin in the heavy-tailed spectrum, so we discuss only the bias case. The scaling exponent is non-integer iff β<∞β<∞-finitely many moments, i.e. a genuinely heavy-tailed inverse-curvature spectrum; as β→∞β→∞ it returns to the analytic value 22. The crossover time above grows as an inverse power of the poisoning fraction, Tϵ(g),Tε≍ε−κT^(g)_ε,T_ ^-κ, when κ>0κ>0 it diverges as ε→0 → 0. The fractional exponent a is therefore expected to be observable only in the regime of a small poisoning fraction together with a correspondingly long training steps - one whose length grows as the same inverse power of ε . Analyzing this scaling is complicated by the fact that κ is not a directly controlled quantity in LLM training: it is governed by the inverse weak-curvature scales of A and the effect of the poisoned data in this regime. We therefore proceed under the modeling assumption that the LLM training horizon considered in this paper is of the same order as the crossover time. Models with a large train-eval mismatch Γ⋆ _ -empirically the smaller LLMs-sit in the gap-dominated regime: the measured exponent starts at a=1a=1 early in training and falls toward a=0a=0 as the time approaches Tϵ(g)T^(g)_ε (see Appendix B, Figure 7(a)). Models with a small mismatch-the larger LLMs sit in the bias/checkpoint-dominated regime, where the exponent starts near a=1a=1 and rises toward the fractional value 2β/(β+γ)>12β/(β+γ)>1 as the time approaches the balanced TϵT_ε (see Appendix B, Figure 7(d)). The balanced-horizon calculation in this section is an ample-data one in the aspect ratio: writing ϕLLM=Peff/Deff _LLM=P_eff/D_eff, Result 9 warns that taking ϵ→0ε→ 0 before ϕLLM→0 _LLM→ 0 gives a different exponent, so the observed law need not extend to arbitrarily small ϵε and one should expect it for ϵ≳ϕLLMε _LLM (see Figure 6 for more discussion). The full local model, assumptions, and proofs are in Appendix B. 6 Conclusion and limitations We pointed out the scaling law for pre-train poisioning of LLMs (Figure 1). Then discussed theoretical analysis of a non-integer poisoning exponent (Result 1 and 2), and provided a solvable model in which heavy tails read out through a poison-dependent truncation supply it (Results 3), and proved that the optimized degradation has two distinct exponents depending on the order of the high-dimensional and small-poison limits, verifying all predictions numerically (Figure 2, 3). Within local LLM training dynamics, we argue how the scaling law may arise in Section 5. The exact theoretical results are for a stylized coordinate model; the connection to LLM training dynamics requires certain modeling hypotheses that remain an open question to establish in the future work. In addition, careful study of small-ϵε regime in LLMs and the fate of the scaling law requires further experiments. On the theoretical side, extending the deterministic equivalent to dense covariances and multi-coordinate examples are the main open directions. Acknowledgments We thank Alex Meterez for several very insightful discussions. Also, we thank Bingbin Liu for comments on a preliminary version of the manual-script. I.H. is supported by DARPA grant AIQ-HR00112520041. C.P. is supported by an NSF CAREER Award (IIS-2239780), DARPA grants DIAL-FP-038 and AIQ-HR00112520041, the Simons Collaboration on the Physics of Learning and Neural Computation, and the William F. Milton Fund from Harvard University. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. References Arora et al. (2016) Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to PMI-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385–399, 2016. Earlier arXiv version: arXiv:1502.03520 (2015). Bahri et al. (2024) Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878121, 2024. Earlier arXiv version: arXiv:2102.06701 (2021). Barkeshli et al. (2026) Maissam Barkeshli, Alberto Alfarano, and Andrey Gromov. On the origin of neural scaling laws: From random graphs to natural language. arXiv preprint arXiv:2601.10684, 2026. Biggio et al. (2012) Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In Proceedings of the 29th International Conference on Machine Learning, 2012. Bordelon et al. (2020) Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Proceedings of the 37th International Conference on Machine Learning, 2020. Bordelon et al. (2024) Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 4345–4382. PMLR, 2024. URL https://proceedings.mlr.press/v235/bordelon24a.html. Bowen et al. (2024) Dillon Bowen, Brendan Murphy, Will Cai, David Khachaturov, Adam Gleave, and Kellin Pelrine. Scaling trends for data poisoning in LLMs. arXiv preprint arXiv:2408.02946, 2024. Cagnetta & Wyart (2024) Francesco Cagnetta and Matthieu Wyart. Towards a theory of how the structure of language is acquired by deep neural networks. In Advances in Neural Information Processing Systems, volume 37, 2024. NeurIPS 2024; arXiv:2406.00048. Cagnetta et al. (2024) Francesco Cagnetta, Leonardo Petrini, Umberto M. Tomasini, Alessandro Favero, and Matthieu Wyart. How deep neural networks learn compositional data: The random hierarchy model. Physical Review X, 14(3):031001, 2024. doi: 10.1103/PhysRevX.14.031001. arXiv:2307.02129. Cagnetta et al. (2026) Francesco Cagnetta, Allan Raventós, Surya Ganguli, and Matthieu Wyart. Deriving neural scaling laws from the statistics of natural language. arXiv preprint arXiv:2602.07488, 2026. Carlini et al. (2024) Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. In IEEE Symposium on Security and Privacy, 2024. arXiv:2302.10149. Catoni (2012) Olivier Catoni. Challenging the empirical mean and empirical variance: A deviation study. Annales de l’Institut Henri Poincaré, Probabilités et Statistiques, 48(4):1148–1185, 2012. Gilmer et al. (2018) Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow. Adversarial spheres. In International Conference on Learning Representations Workshop, 2018. Golub & Meurant (2010) Gene H. Golub and Gérard Meurant. Matrices, Moments and Quadrature with Applications. Princeton University Press, 2010. Contains the Gauss–Radau rules and associated bounding theory; cited as 2009 in Meterez et al. (2026). Golub & Welsch (1969) Gene H. Golub and John H. Welsch. Calculation of Gauss quadrature rules. Mathematics of Computation, 23:221–230, 1969. Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. Greenbaum (1989) Anne Greenbaum. Behavior of slightly perturbed Lanczos and conjugate-gradient recurrences. Linear Algebra and its Applications, 113:7–63, 1989. Halder et al. (2026) Indranil Halder, Annesya Banerjee, and Cengiz Pehlevan. Jailbreak scaling laws for large language models: Polynomial–exponential crossover. arXiv preprint arXiv:2603.11331, 2026. Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, volume 35, p. 30016–30030, 2022. Huber (1964) Peter J. Huber. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73–101, 1964. Hubinger et al. (2024) Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. Sleeper agents: Training deceptive LLMs that persist through safety training. arXiv preprint arXiv:2401.05566, 2024. Hutter (2021) Marcus Hutter. Learning curve theory. arXiv preprint arXiv:2102.04074, 2021. Ilyas et al. (2019) Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, 2019. Johnson et al. (2006) Mark Johnson, Thomas Griffiths, and Sharon Goldwater. Adaptor grammars: A framework for specifying compositional nonparametric bayesian models. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL https://proceedings.neurips.c/paper_files/paper/2006/file/62f91ce9b820a491e78c108636db089-Paper.pdf. Kaniel (1966) Shmuel Kaniel. Estimates for some computational techniques in linear algebra. Mathematics of Computation, 20, 1966. Origin of the Chebyshev-type convergence bounds for extremal Ritz values refined by Paige and Saad. Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. Karkada et al. (2025) Dhruva Karkada, James B. Simon, Yasaman Bahri, and Michael R. DeWeese. Closed-form training dynamics reveal learned features and linear structure in Word2Vec-like models. In Advances in Neural Information Processing Systems, 2025. NeurIPS 2025; arXiv:2502.09863. Karkada et al. (2026) Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, and Yasaman Bahri. Symmetry in language statistics shapes the geometry of model representations. arXiv preprint arXiv:2602.15029, 2026. Koh et al. (2018) Pang Wei W. Koh, Jacob Steinhardt, and Percy S. Liang. Stronger data poisoning attacks break data sanitization defenses. arXiv preprint arXiv:1811.00741, 2018. Korchinski et al. (2025) Daniel J. Korchinski, Dhruva Karkada, Yasaman Bahri, and Matthieu Wyart. On the emergence of linear analogies in word embeddings. arXiv preprint arXiv:2505.18651, 2025. Lanczos (1950) Cornelius Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. Journal of Research of the National Bureau of Standards, 45:255–282, 1950. Li et al. (2021) Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor attacks on pre-trained models by layerwise weight poisoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 3023–3032. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.emnlp-main.241. Liang et al. (2007) Percy Liang, Slav Petrov, Michael I. Jordan, and Dan Klein. The infinite PCFG using hierarchical Dirichlet processes. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, 2007. Liu et al. (2025) Andy Zeyi Liu, Elliot Paquette, and John Sous. Evolution of the spectral dimension of transformer activations. In OPT 2025: Optimization for Machine Learning, 2025. URL https://openreview.net/forum?id=Va5is76bTP. Liu et al. (2026) Yizhou Liu, Ziming Liu, Cengiz Pehlevan, and Jeff Gore. Universal one-third time scaling in learning peaked distributions. arXiv preprint arXiv:2602.03685, 2026. Lugosi & Mendelson (2019) Gábor Lugosi and Shahar Mendelson. Sub-Gaussian estimators of the mean of a random vector. The Annals of Statistics, 47(2):783–794, 2019. MacKay & Peto (1995) David J. C. MacKay and Linda C. Bauman Peto. A hierarchical Dirichlet language model. Natural Language Engineering, 1(3):289–308, 1995. Maini et al. (2025) Pratyush Maini, Sachin Goyal, Dylan Sam, Alexander Robey, Yash Savani, Yiding Jiang, Andy Zou, Matt Fredrikson, Zachary C. Lipton, and Zico Kolter. Safety pretraining: Toward the next generation of safe AI. Advances in Neural Information Processing Systems, 38:43971–44008, 2025. NeurIPS 2025 Main Conference Track; arXiv:2504.16980. Maloney et al. (2022) Alexander Maloney, Daniel A. Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.16859, 2022. Meterez et al. (2026) Alexandru Meterez, Praneeth A. Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade, and Alex Damian. A defense of the quadratic model, 2026. URL https://arxiv.org/abs/2607.21716. Code available at https://github.com/alexandrumeterez/QuadraticModel. Meurant & Strakoš (2006) Gérard Meurant and Zdeněk Strakoš. The Lanczos and conjugate gradient algorithms in finite precision arithmetic. Acta Numerica, 15:471–542, 2006. Michaud et al. (2023) Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In Advances in Neural Information Processing Systems, 2023. Paige (1976) Christopher C. Paige. Error analysis of the Lanczos algorithm for tridiagonalizing a symmetric matrix. IMA Journal of Applied Mathematics, 18(3):341–349, 1976. Paige (1980) Christopher C. Paige. Accuracy and effectiveness of the Lanczos algorithm for the symmetric eigenproblem. Linear Algebra and its Applications, 34:235–258, 1980. Parlett (1980) Beresford N. Parlett. The Symmetric Eigenvalue Problem. Prentice-Hall, 1980. Reprinted as volume 20 of SIAM Classics in Applied Mathematics, 1998. Includes the Kaniel–Paige–Saad bounds and convergence at both ends of the spectrum. Parley et al. (2026) Jack T. Parley, Francesco Cagnetta, and Matthieu Wyart. Deep networks learn to parse uniform-depth context-free languages from local statistics. arXiv preprint arXiv:2602.06065, 2026. Picard & Pitman (2006) J. Picard and J. Pitman. Combinatorial Stochastic Processes: Ecole d’Eté de Probabilités de Saint-Flour XXXII - 2002. Lecture Notes in Mathematics. Springer Berlin Heidelberg, 2006. ISBN 9783540309901. URL https://books.google.com/books?id=6qFTR4PZE4AC. Saad (1980) Yousef Saad. On the rates of convergence of the Lanczos and the block-Lanczos methods. SIAM Journal on Numerical Analysis, 17(5):687–706, 1980. Salvatore et al. (2026) Alessandro Salvatore, Stanislav Fort, and Surya Ganguli. Solving adversarial examples requires solving exponential misalignment, 2026. Sam et al. (2026) Dylan Sam, Sachin Goyal, Pratyush Maini, Alexander Robey, and J. Zico Kolter. When should we introduce safety interventions during pretraining? arXiv preprint arXiv:2601.07087, 2026. doi: 10.48550/arXiv.2601.07087. Sharma & Kaplan (2020) Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004.10802, 2020. Silverstein et al. (2026) Eva Silverstein, Daniel Kunin, and Vasudev Shyam. Symmetry breaking in transformers for efficient and interpretable training. arXiv preprint arXiv:2601.22257, 2026. doi: 10.48550/arXiv.2601.22257. Souly et al. (2025) Alexandra Souly, Javier Rando, Ed Chapman, Xander Davies, Burak Hasircioglu, Ezzeldin Shereen, Carlos Mougan, Vasilios Mavroudis, Erik Jones, Chris Hicks, et al. Poisoning attacks on LLMs require a near-constant number of poison samples. arXiv preprint arXiv:2510.07192, 2025. Spigler et al. (2020) Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: Empirical data versus teacher–student paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020(12):124001, 2020. Steinhardt et al. (2017) Jacob Steinhardt, Pang Wei W. Koh, and Percy S. Liang. Certified defenses for data poisoning attacks. In Advances in Neural Information Processing Systems, volume 30, 2017. Subramanyam et al. (2026) Anirudh Subramanyam, Yuxin Chen, and Robert L. Grossman. Scaling laws revisited: Modeling the role of data quality in language model pretraining, 2026. URL https://arxiv.org/abs/2510.03313. Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. Szegő (1975) Gábor Szegő. Orthogonal Polynomials, volume 23 of American Mathematical Society Colloquium Publications. American Mathematical Society, 4 edition, 1975. Classical source for the Markov–Stieltjes inequalities relating Gauss weights to the distribution function. Tanay & Griffin (2016) Thomas Tanay and Lewis Griffin. A boundary tilting perspective on the phenomenon of adversarial examples, 2016. Tang et al. (2025) Qian-Yuan Tang, Yufei Gu, Yunfeng Cai, Mingming Sun, Ping Li, zhou Xun, and Zeke Xie. Investigating the overlooked hessian structure: From CNNs to LLMs. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=o62ZzfCEwZ. Teh (2006) Yee Whye Teh. A hierarchical Bayesian language model based on Pitman–Yor processes. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, p. 985–992, 2006. Wallace et al. (2021) Eric Wallace, Tony Z. Zhao, Shi Feng, and Sameer Singh. Concealed data poisoning attacks on NLP models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, p. 139–150, 2021. Zhang et al. (2025) Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramèr, and Daphne Ippolito. Persistent pre-training poisoning of LLMs. In International Conference on Learning Representations, p. 31323–31340, 2025. URL https://proceedings.iclr.c/paper_files/paper/2025/file/4dade38eae8c007f3a564b8ea820664a-Paper-Conference.pdf. Appendix A Non-commuting high-dimensional and small-poisoning limits A.1 Connection to the local LLM approximation The purpose of this appendix is to test the local LLM mechanism in a model that retains finite-sample, high-dimensional effects. Around a reference checkpoint θ0 _0, the local quadratic approximation to pre-training has the form ℒ^(θ0+δθ)≃ℒ^(θ0)−v^LLM⊤δθ+12δθ⊤H^LLMδθ, L( _0+δθ) L( _0)- v_ LLM δθ+ 12δθ H_ LLMδθ, where, for token-prediction events indexed by i=1,…,Di=1,…,D, Σ^LLM=1D∑i=1DJi⊤iJi,v^LLM=1D∑i=1DJi⊤ri. _ LLM= 1D _i=1^DJ_i W_iJ_i, v_ LLM= 1D _i=1^DJ_i r_i. The corresponding local stationary equation is Σ^LLMδθ^=v^LLM. _ LLM\, δθ= v_ LLM. The regression model below has exactly the same algebraic form: Σ^ϵ,τ=1D∑i=1DAi(τ),v^ϵ,τ=1D∑i=1Dbi(τ),(Σ^ϵ,τ+ϱIP)θ^=v^ϵ,τ, _ε,τ= 1D _i=1^DA_i^(τ), v_ε,τ= 1D _i=1^Db_i^(τ), ( _ε,τ+ I_P) θ= v_ε,τ, where Ai(τ)A_i^(τ) and bi(τ)b_i^(τ) are truncated versions of XiXi⊤X_iX_i and XiYiX_iY_i. Thus quantitycoordinate surrogatelocal LLM modelparameter dimensionPPeffnumber of examplesDDeffper-example curvatureAi(τ)Ji⊤WiJiper-example responsebi(τ)Ji⊤rifitted displacementθ^δθaspect ratioϕ=P/DPeff/Deff array[]c|c|cquantity&coordinate surrogate&local LLM model\\ dimension&P&P_ eff\\ number of examples&D&D_ eff\\ per-example curvature&A_i^(τ)&J_i W_iJ_i\\ per-example response&b_i^(τ)&J_i r_i\\ fitted displacement& θ&δθ\\ aspect ratio&φ=P/D&P_ eff/D_ eff array at the level of the local normal equation. The analogy concerns having the same normal equations. Truncation is implemented slightly differently: the model below clips the two per-example statistics separately, whereas finite-time LLM training acts as a spectral filter on inverse curvature. However, in this setup, again, we can see the same set of logic used in LLM - heavy-tailed data, along with poison adaptive truncation, leads to the scaling law in the ample data limit. Hence, analysis of this appendix can teach some valuable lessons for the local LLM model. More precisely, the coordinate construction is useful precisely because it makes one additional question explicit: does the small-poisoning limit agree with the population calculation when the parameter-to-data aspect ratio remains nonzero? The remainder of the appendix answers this question exactly for the coordinate model. A.2 Coordinate-isotropic heavy-tailed data distribution Let e1,…,ePe_1,…,e_P denote the standard basis of ℝPR^P. Fix 0<c<1,0<ηp<1,q⋆>1.0<c<1, 0< _p<1, q_ >1. Let U have Pareto tail ℙ(U>t)=t−q⋆,t≥1,P(U>t)=t^-q_ , t≥ 1, so that μU:=U=q⋆q⋆−1. _U:=EU= q_ q_ -1. All truncation formulas below are used for levels τ≥1τ≥ 1. Define the teacher vector θ⋆,P:=cP(1,…,1)⊤. _ ,P:= c P(1,…,1) . A clean example is generated by drawing J uniformly from 1,…,P\1,…,P\ and drawing U independently, then setting X=PUeJ,Y=X⊤θ⋆,P=cU.X= PU\,e_J, Y=X _ ,P=c U. The normalization P P is chosen so that [XX⊤]=μUIP.E[X ]= _UI_P. For any deterministic θ∈ℝPθ ^P, the clean data excess prediction risk is ℛclean,P(θ)−ℛclean,P(θ⋆,P)=μU1P∑j=1P(c−Pθj)2.R_clean,P(θ)-R_clean,P( _ ,P)= _U 1P _j=1^P (c- P\, _j )^2. (26) Indeed, conditional on J=jJ=j and U, one has Y−X⊤θ=U(c−Pθj),Y-X θ= U (c- P\, _j ), and averaging over J and U gives equation 26. A.2.1 Modified truncation suitable for higher dimensional statistics The unnormalized matrix statistic has operator norm ‖XX⊤‖op=PU, X _op=PU, and the vector statistic satisfies XY=cPUeJ,‖XY‖2=cPU.XY=c P\,Ue_J, XY _2=c P\,U. We use dimension-normalized truncation thresholds PτPτ for the matrix statistic and Pτ P\,τ for the vector statistic. Thus A(τ):=XX⊤min1,Pτ‖XX⊤‖op,b(τ):=XYmin1,Pτ‖XY‖2.A^(τ):=X \1, Pτ X _op \, b^(τ):=XY \1, P\,τ XY _2 \. For a clean example this gives A(τ)=Paτ(U)eJeJ⊤,b(τ)=Prτ(U)eJ,A^(τ)=P\,a_τ(U)e_Je_J , b^(τ)= P\,r_τ(U)e_J, (27) where† The notation means that t∧τt τ is the smaller of t and τ. aτ(t):=t∧τ,rτ(t):=c(t∧τc).a_τ(t):=t τ, r_τ(t):=c (t τc ). The clean truncated population moments are Mτ:=aτ(U)=μU−τ1−q⋆q⋆−1,M_τ:=Ea_τ(U)= _U- τ^1-q_ q_ -1, and Bτ:=rτ(U)=cMτ/c=cμU−cq⋆τ1−q⋆q⋆−1.B_τ:=Er_τ(U)=cM_τ/c=c _U- c^q_ τ^1-q_ q_ -1. Since 0<c<10<c<1, the two truncations clip the same clean radial tail at different effective thresholds. This is the high-dimensional coordinate version of the clean truncation-bias mechanism. A.2.2 Label-shifted data poisoning within truncation radius We now use the modified poison. A poisoned example is generated by drawing J uniformly from 1,…,P\1,…,P\ and setting Xp=PeJ,Yp=c+ηpτ.X_p= P\,e_J, Y_p=c+ _pτ. Thus the poison feature has bounded normalized squared norm, ‖Xp‖22P=1, X_p _2^2P=1, while the label residual relative to the clean teacher is order τ: Yp−Xp⊤θ⋆,P=ηpτ.Y_p-X_p _ ,P= _pτ. Let τ0:=max1,c1−ηp. _0:= \1, c1- _p \. Throughout the exact formulas for the modified poison we assume τ≥τ0τ≥ _0. Then c+ηpτ≤τ,c+ _pτ≤τ, so the poison cross-moment lies below the vector truncation threshold, while its matrix statistic lies below the matrix truncation threshold. Hence Ap(τ)=PeJeJ⊤,bp(τ)=P(c+ηpτ)eJ.A_p^(τ)=Pe_Je_J , b_p^(τ)= P\,(c+ _pτ)e_J. (28) Training samples are independent. Each sample is clean with probability 1−ϵ1-ε and poisoned with probability ϵε. A.3 Simplification in linear regression with truncation Given D training samples, define Σ^ϵ,τ:=1D∑i=1DAi(τ),v^ϵ,τ:=1D∑i=1Dbi(τ). _ε,τ:= 1D _i=1^DA_i^(τ), v_ε,τ:= 1D _i=1^Db_i^(τ). For ϱ>0 >0, define the ridge estimator θ^ϵ,τ,ϱ:=(Σ^ϵ,τ+ϱIP)−1v^ϵ,τ. θ_ε,τ, :=( _ε,τ+ I_P)^-1 v_ε,τ. For ϱ=0 =0, define the ridgeless estimator by the Moore–Penrose inverse: θ^ϵ,τ,0:=Σ^ϵ,τ†v^ϵ,τ. θ_ε,τ,0:= _ε,τ v_ε,τ. Let ϕD=P/D _D=P/D. For coordinate j, let Nj,cN_j,c and Nj,pN_j,p denote the numbers of clean and poisoned training samples assigned to coordinate j. Let Uj,1,Uj,2,…U_j,1,U_j,2,… denote the clean radial variables assigned to that coordinate. Define Sj,A(D) S_j,A^(D) :=∑ℓ=1Nj,caτ(Uj,ℓ)+Nj,p, := _ =1^N_j,ca_τ(U_j, )+N_j,p, (29) Sj,B(D) S_j,B^(D) :=∑ℓ=1Nj,crτ(Uj,ℓ)+(c+ηpτ)Nj,p. := _ =1^N_j,cr_τ(U_j, )+(c+ _pτ)N_j,p. (30) By construction, the per-example statistics from equation 27 and equation 28 are supported on the single coordinate Ji∈1,…,PJ_i∈\1,…,P\ to which sample i is assigned. Writing cc for a clean sample and pp for a poisoned one, Ai(τ)=PαieJieJi⊤,bi(τ)=PβieJi,A_i^(τ)=P\, _i\,e_J_ie_J_i , b_i^(τ)= P\, _i\,e_J_i, where αi=aτ(Ui),i clean,1,i poison,βi=rτ(Ui),i clean,c+ηpτ,i poison. _i= casesa_τ(U_i),&$i$ clean,\\ 1,&$i$ poison, cases _i= casesr_τ(U_i),&$i$ clean,\\ c+ _pτ,&$i$ poison. cases Each matrix eJieJi⊤e_J_ie_J_i is diagonal, with a single nonzero entry equal to 11 in position (Ji,Ji)(J_i,J_i). Hence Σ^ϵ,τ=1D∑i=1DAi(τ)=PD∑i=1DαieJieJi⊤ _ε,τ= 1D _i=1^DA_i^(τ)= PD _i=1^D _i\,e_J_ie_J_i is a sum of diagonal matrices and is therefore diagonal. Its (j,j)(j,j) entry collects exactly those samples assigned to coordinate j: (Σ^ϵ,τ)jj=PD∑i:Ji=jαi=ϕD(∑ℓ=1Nj,caτ(Uj,ℓ)+Nj,p)=ϕDSj,A(D),( _ε,τ)_j= PD _i:\,J_i=j _i= _D ( _ =1^N_j,ca_τ(U_j, )+N_j,p )= _D\,S_j,A^(D), using the definition equation 29 of Sj,A(D)S_j,A^(D). Likewise, bi(τ)=PβieJib_i^(τ)= P\, _i\,e_J_i has a single nonzero entry in position JiJ_i, so (v^ϵ,τ)j=1D∑i:Ji=jPβi=PD(∑ℓ=1Nj,crτ(Uj,ℓ)+(c+ηpτ)Nj,p)=PDSj,B(D)=ϕDPSj,B(D),( v_ε,τ)_j= 1D _i:\,J_i=j P\, _i= PD ( _ =1^N_j,cr_τ(U_j, )+(c+ _pτ)N_j,p )= PD\,S_j,B^(D)= _D P\,S_j,B^(D), where the last equality uses P/D=(P/D)/P=ϕD/P P/D=(P/D)/ P= _D/ P. Consequently, P(θ^ϵ,τ,ϱ)j=GϕD,ϱ(Sj,A(D),Sj,B(D)), P\,( θ_ε,τ, )_j=G_ _D, (S_j,A^(D),S_j,B^(D) ), (31) where, for ϱ≥0 ≥ 0, Gϕ,ϱ(sA,sB):=ϕsBϱ+ϕsA,ϱ+ϕsA>0,0,ϱ=0,sA=0.G_φ, (s_A,s_B):= cases φ s_B +φ s_A,& +φ s_A>0,\\[8.53581pt] 0,& =0,\ s_A=0. cases (32) For a clean data one has 0≤rτ(t)≤aτ(t),0≤ r_τ(t)≤ a_τ(t), and for a modified poison data, when τ≥τ0τ≥ _0, 0≤c+ηpτ≤τ.0≤ c+ _pτ≤τ. Hence, with mτ:=max1,c+ηpτ,m_τ:= \1,c+ _pτ\, every finite collection of clean and poison data satisfies 0≤sB≤mτsA.0≤ s_B≤ m_τs_A. Since τ≥τ0τ≥ _0 implies τ≥1τ≥ 1 and c+ηpτ≤τc+ _pτ≤τ, we also have mτ≤τm_τ≤τ. Therefore 0≤Gϕ,ϱ(sA,sB)≤mτ≤τ.0≤ G_φ, (s_A,s_B)≤ m_τ≤τ. (33) A.4 The deterministic equivalent at fixed aspect ratio Each training sample chooses one of the P coordinates uniformly. Conditional on the sample being assigned to coordinate j, the sample is clean with probability 1−ϵ1-ε and poisoned with probability ϵε. Therefore, for a fixed coordinate j, the finite-D,PD,P clean and poison counts satisfy Nj,c(D)=∑i=1Dsample i is clean and assigned to j,N_j,c^(D)= _i=1^D1\sample i is clean and assigned to j\, and Nj,p(D)=∑i=1Dsample i is poisoned and assigned to j.N_j,p^(D)= _i=1^D1\sample i is poisoned and assigned to j\. For each sample, ℙ(clean and assigned to j)=1−ϵP,ℙ(poisoned and assigned to j)=ϵP.P(clean and assigned to j)= 1-εP, (poisoned and assigned to j)= εP. Hence Nj,c(D)=D1−ϵP=1−ϵϕD,Nj,p(D)=DϵP=ϵϕD.EN_j,c^(D)=D 1-εP= 1-ε _D, _j,p^(D)=D εP= ε _D. Lemma 1 (Poisson limit of the coordinate counts). Fix a coordinate j. Suppose P/D→ϕ∈(0,∞)P/D→φ∈(0,∞). Each training sample is clean with probability 1−ϵ1-ε, poisoned with probability ϵε, and independently assigned to coordinate j with probability 1/P1/P. Let Nj,c(D)N_j,c^(D) and Nj,p(D)N_j,p^(D) be the numbers of clean and poisoned samples assigned to coordinate j. Then (Nj,c(D),Nj,p(D))⇒(Nc,Np),(N_j,c^(D),N_j,p^(D)) (N_c,N_p), where Nc∼Pois(1−ϵϕ),Np∼Pois(ϵϕ),N_c ( 1-εφ ), N_p ( εφ ), and NcN_c and NpN_p are independent. Proof. For a fixed coordinate j, a given training sample contributes a clean data to coordinate j with probability pc,D=1−ϵP,p_c,D= 1-εP, and contributes a poison data to coordinate j with probability p,D=ϵP.p_p,D= εP. Thus (Nj,c(D),Nj,p(D))(N_j,c^(D),N_j,p^(D)) has a multinomial distribution with three categories: clean hit, poison hit, and no hit. For k,ℓ≥0k, ≥ 0 with k+ℓ≤Dk+ ≤ D, ℙ(Nj,c(D)=k,Nj,p(D)=ℓ) (N_j,c^(D)=k,\,N_j,p^(D)= ) =D!k!ℓ!(D−k−ℓ)!(1−ϵP)k(ϵP)ℓ(1−1P)D−k−ℓ. = D!k!\, !\,(D-k- )! ( 1-εP )^k ( εP ) (1- 1P )^D-k- . Since P/D→ϕP/D→φ, we have DP→1ϕ,D(1−ϵ)P→1−ϵϕ,DϵP→ϵϕ. DP→ 1φ, D(1-ε)P→ 1-εφ, DεP→ εφ. For fixed k,ℓk, , D!(D−k−ℓ)!Pk+ℓ→(1ϕ)k+ℓ, D!(D-k- )!P^k+ → ( 1φ )^k+ , and (1−1P)D−k−ℓ→exp(−1ϕ). (1- 1P )^D-k- → (- 1φ ). Therefore ℙ(Nj,c(D)=k,Nj,p(D)=ℓ) (N_j,c^(D)=k,\,N_j,p^(D)= ) →exp(−1ϕ)(1−ϵϕ)k!(ϵϕ)ℓ!. → (- 1φ ) ( 1-εφ )^kk! ( εφ ) !. This is the joint probability mass function of two independent Poisson random variables with means (1−ϵ)/ϕ(1-ε)/φ and ϵ/ϕε/φ, respectively. ∎ At fixed ϕφ, a coordinate receives a poison data with probability ℙ(Np≥1)=1−exp(−ϵϕ)=ϵϕ+O(ϵ2),ϵ↓0.P(N_p≥ 1)=1- (- εφ )= εφ+O(ε^2), ε 0. Thus poisoning acts through a sparse-coordinate mechanism: most coordinates see no poison, while a proportion of order ϵ/ϕε/φ see one poison data. By contrast, if one first sends ϕ↓0φ 0, then 1−ϵϕ→∞,ϵϕ→∞for fixed ϵ>0. 1-εφ→∞, εφ→∞ fixed ε>0. Each coordinate is hit by many clean and poisoned samples, and the Poisson fluctuations average out by the law of large numbers. Let U1,U2,…U_1,U_2,… be i.i.P. copies of U drawn from the Pareto tail, independent of Nc,NpN_c,N_p: Nc∼Pois(1−ϵϕ),Np∼Pois(ϵϕ),N_c ( 1-εφ ), N_p ( εφ ), Define the limiting compound-Poisson coordinate statistics SAϵ,τ S_A^ε,τ :=∑ℓ=1Ncaτ(Uℓ)+Np, := _ =1^N_ca_τ(U_ )+N_p, (34) SBϵ,τ S_B^ε,τ :=∑ℓ=1Ncrτ(Uℓ)+(c+ηpτ)Np. := _ =1^N_cr_τ(U_ )+(c+ _pτ)N_p. (35) Finally define ℛϕ,ϱ(ϵ,τ):=μU[(c−Gϕ,ϱ(SAϵ,τ,SBϵ,τ))2].R_φ, (ε,τ):= _U\,E [ (c-G_φ, (S_A^ε,τ,S_B^ε,τ ) )^2 ]. (36) Lemma 2 (Deterministic equivalent). Let F:[0,∞)2→ℝF:[0,∞)^2 be a bounded function. Then 1P∑j=1PF(Sj,A(D),Sj,B(D))→ℙF(SAϵ,τ,SBϵ,τ). 1P _j=1^PF (S_j,A^(D),S_j,B^(D) ) PEF (S_A^ε,τ,S_B^ε,τ ). The conclusion also holds for the bounded functions FϕD,ϱ(sA,sB):=(c−GϕD,ϱ(sA,sB))2,F_ _D, (s_A,s_B):= (c-G_ _D, (s_A,s_B) )^2, with the limit Fϕ,ϱF_φ, , including the ridgeless convention ϱ=0 =0. Proof. Fix τ≥τ0τ≥ _0. Since P/D→ϕP/D→φ, the law of (Sj,A(D),Sj,B(D))(S_j,A^(D),S_j,B^(D)) converges to the compound-Poisson law in equation 34. Therefore F(Sj,A(D),Sj,B(D))→F(SAϵ,τ,SBϵ,τ)EF (S_j,A^(D),S_j,B^(D) ) (S_A^ε,τ,S_B^ε,τ ) by the continuity. It remains to show concentration of the coordinate average. Let Yj(D):=F(Sj,A(D),Sj,B(D)),j=1,…,P.Y_j^(D):=F (S_j,A^(D),S_j,B^(D) ), j=1,…,P. Since the training examples assign coordinates symmetrically, the random variables Y1(D),…,YP(D)Y_1^(D),…,Y_P^(D) are exchangeable. Suppose ‖F‖∞:=supsA,sB|F(sA,sB)|<∞.\|F\|_∞:= _s_A,s_B|F(s_A,s_B)|<∞. Then |Yj(D)|≤‖F‖∞for every j.|Y_j^(D)|≤\|F\|_∞ every j. We compute Var(1P∑j=1PYj(D)) ( 1P _j=1^PY_j^(D) ) =1P2∑j=1PVar(Yj(D))+1P2∑j,k=1j≠kPCov(Yj(D),Yk(D)). = 1P^2 _j=1^PVar(Y_j^(D))+ 1P^2 _ subarraycj,k=1\\ j≠ k subarray^PCov(Y_j^(D),Y_k^(D)). By exchangeability, Var(Yj(D))=Var(Y1(D))for all j,Var(Y_j^(D))=Var(Y_1^(D)) all j, and Cov(Yj(D),Yk(D))=Cov(Y1(D),Y2(D))for all j≠k.Cov(Y_j^(D),Y_k^(D))=Cov(Y_1^(D),Y_2^(D)) all j≠ k. Therefore Var(1P∑j=1PYj(D)) ( 1P _j=1^PY_j^(D) ) =P2Var(Y1(D))+P(P−1)P2Cov(Y1(D),Y2(D)) = PP^2Var(Y_1^(D))+ P(P-1)P^2Cov(Y_1^(D),Y_2^(D)) =1PVar(Y1(D))+P−1PCov(Y1(D),Y2(D)). = 1PVar(Y_1^(D))+ P-1PCov(Y_1^(D),Y_2^(D)). Taking absolute values in the covariance term and using (P−1)/P≤1(P-1)/P≤ 1, we obtain Var(1P∑j=1PYj(D)) ( 1P _j=1^PY_j^(D) ) ≤1PVar(Y1(D))+|Cov(Y1(D),Y2(D))|. ≤ 1PVar(Y_1^(D))+ |Cov(Y_1^(D),Y_2^(D)) |. Finally, Var(Y1(D))=[(Y1(D)−Y1(D))2]≤[(Y1(D))2]≤‖F‖∞2.Var(Y_1^(D))=E [ (Y_1^(D)-EY_1^(D) )^2 ] [(Y_1^(D))^2 ]≤\|F\|_∞^2. Thus Var(1P∑j=1PF(Sj,A(D),Sj,B(D))) ( 1P _j=1^PF(S_j,A^(D),S_j,B^(D)) ) ≤‖F‖∞2P+|Cov(F1(D),F2(D))|, ≤ \|F\|_∞^2P+ |Cov(F_1^(D),F_2^(D)) |, where Fj(D):=F(Sj,A(D),Sj,B(D)).F_j^(D):=F(S_j,A^(D),S_j,B^(D)). For two fixed coordinates, the joint occupancy counts are multinomially coupled, but the coupling is weak. As P/D→ϕ∈(0,∞)P/D→φ∈(0,∞), the law of the clean and poisoned counts in two fixed coordinates converges to the law of two independent copies of the limiting compound-Poisson coordinate statistic. Hence, Cov(F1(D),F2(D))→0.Cov(F_1^(D),F_2^(D))→ 0. For FϕD,ϱF_ _D, , boundedness follows from equation 33. If ϱ>0 >0, the map (ϕ,sA,sB)↦(c−Gϕ,ϱ(sA,sB))2(φ,s_A,s_B) (c-G_φ, (s_A,s_B))^2 is continuous. If ϱ=0 =0, the only problematic point is (sA,sB)=(0,0)(s_A,s_B)=(0,0). This point has positive mass, but it is not a problem because the finite-D and limiting conventions both assign the same value G=0G=0 there; on sA>0s_A>0, the ridgeless map is sB/sAs_B/s_A and is continuous. The same expectation and covariance argument therefore applies to FϕD,ϱF_ _D, , giving the claimed deterministic equivalent. ∎ In many high-dimensional ridge-regression deterministic equivalents, one sees a nontrivial effective regularization parameter. This often appears through a self-consistent equation involving a Stieltjes transform or an equivalent quantity. It is therefore natural to ask why, in the occupancy deterministic equivalent below, the ridge parameter remains the same parameter ϱ : FϕD,ϱ(sA,sB):=(c−GϕD,ϱ(sA,sB))2⟶Fϕ,ϱ(sA,sB),F_ _D, (s_A,s_B):= (c-G_ _D, (s_A,s_B) )^2 F_φ, (s_A,s_B), rather than becoming some renormalized quantity. The reason is that each coordinate has its own scalar normal equation: ϱ+ϕDSj,A(D). + _DS_j,A^(D). The denominator is already scalar before taking the limit. Thus there is no resolvent self-consistency equation. Result 4 (Fixed-aspect-ratio deterministic equivalent for the empirical risk). Assume P/D→ϕ∈(0,∞)P/D→φ∈(0,∞) and fix ϵ∈[0,1)ε∈[0,1), τ≥τ0τ≥ _0, and ϱ≥0 ≥ 0. Then ℛclean,P(θ^ϵ,τ,ϱ)−ℛclean,P(θ⋆,P)→ℙℛϕ,ϱ(ϵ,τ),R_clean,P( θ_ε,τ, )-R_clean,P( _ ,P) PR_φ, (ε,τ), where ℛϕ,ϱR_φ, is defined in equation 36. Consequently, if an independent clean baseline estimator is trained with the same D,P,τ,ϱD,P,τ, , then [ℛclean,P(θ^ϵ,τ,ϱ)−ℛclean,P(θ^0,τ,ϱ)] [R_clean,P( θ_ε,τ, )-R_clean,P( θ_0,τ, ) ] →ℙϕ,ϱ(ϵ,τ):=ℛϕ,ϱ(ϵ,τ)−ℛϕ,ϱ(0,τ). PD_φ, (ε,τ):=R_φ, (ε,τ)-R_φ, (0,τ). Proof. Recall first the clean risk identity ℛclean,P(θ)−ℛclean,P(θ⋆,P)=μU1P∑j=1P(c−Pθj)2.R_clean,P(θ)-R_clean,P( _ ,P)= _U 1P _j=1^P (c- P\, _j )^2. Thus, to compute the clean excess risk of the estimator θ^ϵ,τ,ϱ θ_ε,τ, , it is enough to understand the coordinate-wise quantities P(θ^ϵ,τ,ϱ)j. P\,( θ_ε,τ, )_j. For coordinate j, the diagonal entry of the empirical Hessian and the corresponding entry of the empirical cross-moment are (Σ^ϵ,τ)jj=ϕDSj,A(D),(v^ϵ,τ)j=ϕDPSj,B(D),ϕD:=PD.( _ε,τ)_j= _DS_j,A^(D), ( v_ε,τ)_j= _D PS_j,B^(D), _D:= PD. Therefore the ridge normal equation in coordinate j is the scalar equation (ϱ+ϕDSj,A(D))(θ^ϵ,τ,ϱ)j=ϕDPSj,B(D). ( + _DS_j,A^(D) )( θ_ε,τ, )_j= _D PS_j,B^(D). Equivalently, P(θ^ϵ,τ,ϱ)j=GϕD,ϱ(Sj,A(D),Sj,B(D)), P\,( θ_ε,τ, )_j=G_ _D, (S_j,A^(D),S_j,B^(D) ), where Gϕ,ϱ(sA,sB)=ϕsBϱ+ϕsA,ϱ+ϕsA>0,0,ϱ=0,sA=0.G_φ, (s_A,s_B)= cases φ s_B +φ s_A,& +φ s_A>0,\\[8.53581pt] 0,& =0,\ s_A=0. cases Substituting the coordinate formula into the clean risk identity gives the exact finite-D,PD,P identity ℛclean,P(θ^ϵ,τ,ϱ)−ℛclean,P(θ⋆,P) _clean,P( θ_ε,τ, )-R_clean,P( _ ,P) =μU1P∑j=1P(c−GϕD,ϱ(Sj,A(D),Sj,B(D)))2. = _U 1P _j=1^P (c-G_ _D, (S_j,A^(D),S_j,B^(D) ) )^2. Thus the risk is exactly an average over coordinates of the bounded function FϕD,ϱ(sA,sB):=(c−GϕD,ϱ(sA,sB))2.F_ _D, (s_A,s_B):= (c-G_ _D, (s_A,s_B) )^2. Applying Lemma 2 gives 1P∑j=1P(c−GϕD,ϱ(Sj,A(D),Sj,B(D)))2 1P _j=1^P (c-G_ _D, (S_j,A^(D),S_j,B^(D) ) )^2 →ℙ[(c−Gϕ,ϱ(SAϵ,τ,SBϵ,τ))2]. PE [ (c-G_φ, (S_A^ε,τ,S_B^ε,τ ) )^2 ]. Multiplying by μU _U, we obtain ℛclean,P(θ^ϵ,τ,ϱ)−ℛclean,P(θ⋆,P)→ℙμU[(c−Gϕ,ϱ(SAϵ,τ,SBϵ,τ))2].R_clean,P( θ_ε,τ, )-R_clean,P( _ ,P) P _UE [ (c-G_φ, (S_A^ε,τ,S_B^ε,τ ) )^2 ]. By definition, the right-hand side is ℛϕ,ϱ(ϵ,τ).R_φ, (ε,τ). This proves the first result. For the second convergence, apply the same result twice. First, apply it to the estimator trained with poisoning rate ϵε: ℛclean,P(θ^ϵ,τ,ϱ)−ℛclean,P(θ⋆,P)→ℙℛϕ,ϱ(ϵ,τ).R_clean,P( θ_ε,τ, )-R_clean,P( _ ,P) PR_φ, (ε,τ). Second, apply it to the clean estimator trained with poisoning rate 00: ℛclean,P(θ^0,τ,ϱ)−ℛclean,P(θ⋆,P)→ℙℛϕ,ϱ(0,τ).R_clean,P( θ_0,τ, )-R_clean,P( _ ,P) PR_φ, (0,τ). Subtracting the two convergences gives ℛclean,P(θ^ϵ,τ,ϱ)−ℛclean,P(θ^0,τ,ϱ) _clean,P( θ_ε,τ, )-R_clean,P( θ_0,τ, ) →ℙℛϕ,ϱ(ϵ,τ)−ℛϕ,ϱ(0,τ)=ϕ,ϱ(ϵ,τ). PR_φ, (ε,τ)-R_φ, (0,τ)=D_φ, (ε,τ). This proves the second result. ∎ A.5 Fixed ϕφ: higher dimensional statistics The notation Π∼P(ν) (ν) means that Π is a Poisson point process with intensity measure ν. Each element of Π is a data z=(zA,zB),z=(z_A,z_B), A realization of Π is a random finite collection of data: Π=Z1,…,ZN. =\Z_1,…,Z_N\. The associated total coordinate statistic is Σ(Π):=∑z∈Πz=∑k=1NZk. ( ):= _z∈ z= _k=1^NZ_k. If Zk=(Zk,A,Zk,B)Z_k=(Z_k,A,Z_k,B), then Σ(Π)=(∑k=1NZk,A,∑k=1NZk,B). ( )= ( _k=1^NZ_k,A, _k=1^NZ_k,B ). For example Zc(τ)=(aτ(U),rτ(U)),Zp(τ)=(1,c+ηpτ)Z_c^(τ)= (a_τ(U),r_τ(U) ), Z_p^(τ)= (1,c+ _pτ ) denote one clean and one poison data. The first component is the contribution to the coordinate Hessian statistic, and the second component is the contribution to the coordinate cross-moment statistic. We focus on the intensity measure of the form ν=ρℒ(Z),ν=ρ\,L(Z), where ρ>0ρ>0 is a rate and ℒ(Z)L(Z) is the probability law of a single data Z. The notation Π∼P(ρℒ(Z)) (ρ\,L(Z) ) means the following concrete construction: N∼Pois(ρ),N (ρ), and, conditional on N, Z1,…,ZNare i.i.d. with lawℒ(Z).Z_1,…,Z_N i.i.d. with law (Z). Then Π=Z1,…,ZN. =\Z_1,…,Z_N\. For example Πret∼P(1−ϵϕℒ(Zc(τ))). _ret ( 1-εφ\,L(Z_c^(τ)) ). is the random collection of clean data hitting one coordinate. The number of retained clean marks is Nret∼Pois(1−ϵϕ).N_ret ( 1-εφ ). Conditional on NretN_ret, the retained clean marks are independent copies of Zc(τ)=(aτ(U),rτ(U)).Z_c^(τ)= (a_τ(U),r_τ(U) ). Thus one can write Πret=Zc,1(τ),…,Zc,Nret(τ). _ret= \Z_c,1^(τ),…,Z_c,N_ret^(τ) \. The corresponding coordinate statistic is Sret:=Σ(Πret)=∑ℓ=1NretZc,ℓ(τ).S_ret:= ( _ret)= _ =1^N_retZ_c, ^(τ). Writing this sum coordinate-wise gives Sret=(∑ℓ=1Nretaτ(Uℓ),∑ℓ=1Nretrτ(Uℓ)).S_ret= ( _ =1^N_reta_τ(U_ ), _ =1^N_retr_τ(U_ ) ). Similarly, Πdel∼P(ϵϕℒ(Zc(τ))) _del ( εφ\,L(Z_c^(τ)) ) means that the deleted clean process contains Ndel∼Pois(ϵϕ)N_del ( εφ ) clean marks, each distributed as Zc(τ)Z_c^(τ). The poison process Πp∼P(ϵϕℒ(Zp(τ))) _p ( εφ\,L(Z_p^(τ)) ) means that the number of poison marks is Np∼Pois(ϵϕ),N_p ( εφ ), and each poison data has law ℒ(Zp(τ)).L(Z_p^(τ)). In the attack, this data is deterministic: Zp(τ)=(1,c+ηpτ).Z_p^(τ)= (1,c+ _pτ ). Thus ℒ(Zp(τ))L(Z_p^(τ)) is just a point mass at (1,c+ηpτ)(1,c+ _pτ). If Π1∼P(ν1),Π2∼P(ν2) _1 ( _1), _2 ( _2) are independent, then Π1∪Π2∼P(ν1+ν2). _1∪ _2 ( _1+ _2). Therefore Πret∪Πdel∼P(1−ϵϕℒ(Zc(τ))+ϵϕℒ(Zc(τ))). _ret∪ _del ( 1-εφL(Z_c^(τ))+ εφL(Z_c^(τ)) ). Combining the two terms gives Πret∪Πdel∼P(1ϕℒ(Zc(τ))). _ret∪ _del ( 1φL(Z_c^(τ)) ). This is exactly the clean coordinate process of rate 1/ϕ1/φ, i. e., if the D samples choose their coordinates uniformly from 1,…,P\1,…,P\, then the number of samples falling on a fixed coordinate is approximately Binomial(D,1P)⟹Pois(1ϕ).Binomial (D, 1P ) ( 1φ ). We denote by S0(τ):=∑ℓ=1N0Zc,ℓ(τ),N0∼Pois(1/ϕ),S_0^(τ):= _ =1^N_0Z_c, ^(τ), N_0 (1/φ), the clean coordinate environment seen by a typical coordinate in the unpoisoned fixed-ϕφ limit. It is also useful to define Q0:=∑ℓ=1N0Uℓ,Q_0:= _ =1^N_0U_ , and Q+:=U0+∑ℓ=1N0Uℓ=U0+Q0,Q_+:=U_0+ _ =1^N_0U_ =U_0+Q_0, where U0,U1,U2,…U_0,U_1,U_2,… are independent copies of U, independent of N0N_0. The variable Q0Q_0 is the untruncated total radial mass already present at the coordinate. The variable Q+Q_+ is the untruncated total radial mass after adding one additional independent clean example. This distinction is important because the small-poisoning expansion compares two possible one-data perturbations of the same clean environment: S0(τ)+Zp(τ)versusS0(τ)+Zc(τ).S_0^(τ)+Z_p^(τ) S_0^(τ)+Z_c^(τ). The first expression corresponds to adding one poison data, while the second corresponds to adding one additional clean data. Lemma 3 (Large-τ limit of the added-clean term). For fixed ϕ∈(0,∞)φ∈(0,∞) and ϱ≥0 ≥ 0,† Let Aϕ,ϱ(τ)A_φ, (τ) be a quantity depending on τ, ϕφ, and ϱ . We write Aϕ,ϱ(τ)=Oϕ,ϱ(τ−q⋆)as τ→∞A_φ, (τ)=O_φ, (τ^-q_ ) τ→∞ if, for every fixed ϕφ and every fixed ϱ , there exist constants Cϕ,ϱ<∞andτϕ,ϱ<∞C_φ, <∞ _φ, <∞ such that, for all τ≥τϕ,ϱτ≥ _φ, , |Aϕ,ϱ(τ)|≤Cϕ,ϱτ−q⋆. |A_φ, (τ) |≤ C_φ, τ^-q_ . The subscript in Oϕ,ϱO_φ, means that the implicit constant may depend on ϕφ and ϱ , but it not on τ. [Fϕ,ϱ(S0(τ)+Zc(τ))] [F_φ, (S_0^(τ)+Z_c^(τ) ) ] =c2ϱ2[1(ϱ+ϕQ+)2]+Oϕ,ϱ(τ−q⋆). =c^2 ^2\,E [ 1( +φ Q_+)^2 ]+O_φ, (τ^-q_ ). Proof. Let ℰτE_τ be the event that all clean radii appearing in S0(τ)+Zc(τ)S_0^(τ)+Z_c^(τ) are at most τ. On ℰτE_τ, no truncation occurs for either the matrix statistic or the vector statistic, because τ/c>τ/c>τ. Thus, on this event, SA=Q+,SB=cQ+,S_A=Q_+, S_B=cQ_+, and therefore Fϕ,ϱ(SA,SB)=(c−ϕcQ+ϱ+ϕQ+)2=c2ϱ2(ϱ+ϕQ+)2.F_φ, (S_A,S_B)= (c- φ cQ_+ +φ Q_+ )^2= c^2 ^2( +φ Q_+)^2. Let M:=1+N0,N0∼Pois(1/ϕ),M:=1+N_0, N_0 (1/φ), be the total number of clean radii appearing in S0(τ)+Zc(τ)S_0^(τ)+Z_c^(τ). Write these radii as U1,…,UM,U_1,…,U_M, where, conditionally on M, they are i.i.P. copies of U. Recall that ℰτ:=max1≤ℓ≤MUℓ≤τ.E_τ:= \ _1≤ ≤ MU_ ≤τ \. Thus ℰτc=∃ 1≤ℓ≤M such that Uℓ>τ.E_τ^c= \∃\,1≤ ≤ M such that U_ >τ \. We first bound the probability of this event. Conditional on M=mM=m, ℙ(ℰτc∣M=m) (E_τ^c M=m) =1−ℙ(U1≤τ,…,Um≤τ) =1-P(U_1≤τ,…,U_m≤τ) =1−(1−ℙ(U>τ))m. =1- (1-P(U>τ) )^m. Since 1−(1−u)m≤mu1-(1-u)^m≤ mu for u∈[0,1]u∈[0,1], we get ℙ(ℰτc∣M=m)≤mℙ(U>τ).P(E_τ^c M=m)≤ m\,P(U>τ). Taking expectations over M, ℙ(ℰτc) (E_τ^c) ≤[M]ℙ(U>τ) [M]\,P(U>τ) =(1+1ϕ)τ−q⋆. = (1+ 1φ )τ^-q_ . Therefore ℙ(ℰτc)=Oϕ(τ−q⋆).P(E_τ^c)=O_φ(τ^-q_ ). It remains to show that the integrand for the bad events is bounded. On a clean-only configuration, the coordinate statistics have the form SA=∑ℓ=1Maτ(Uℓ),SB=∑ℓ=1Mrτ(Uℓ),S_A= _ =1^Ma_τ(U_ ), S_B= _ =1^Mr_τ(U_ ), where aτ(t)=t∧τ,rτ(t)=c(t∧τc).a_τ(t)=t τ, r_τ(t)=c (t τc ). For every t≥1t≥ 1, 0≤rτ(t)≤aτ(t).0≤ r_τ(t)≤ a_τ(t). Indeed, if t≤τt≤τ, then rτ(t)=ct≤t=aτ(t)r_τ(t)=ct≤ t=a_τ(t). If τ<t<τ/cτ<t<τ/c, then rτ(t)=ct≤τ=aτ(t)r_τ(t)=ct≤τ=a_τ(t). Finally, if t≥τ/ct≥τ/c, then rτ(t)=τ=aτ(t)r_τ(t)=τ=a_τ(t). Hence 0≤SB≤SA.0≤ S_B≤ S_A. Consequently, for ϱ≥0 ≥ 0, 0≤Gϕ,ϱ(SA,SB)=ϕSBϱ+ϕSA≤ϕSAϱ+ϕSA≤10≤ G_φ, (S_A,S_B)= φ S_B +φ S_A≤ φ S_A +φ S_A≤ 1 whenever ϱ+ϕSA>0 +φ S_A>0. In the ridgeless convention ϱ=0 =0, if SA=0S_A=0 then Gϕ,0(0,0)=0G_φ,0(0,0)=0, so the same bound still holds. Therefore 0≤Gϕ,ϱ(SA,SB)≤1.0≤ G_φ, (S_A,S_B)≤ 1. Since 0<c<10<c<1, Fϕ,ϱ(SA,SB)=(c−Gϕ,ϱ(SA,SB))2≤1.F_φ, (S_A,S_B)= (c-G_φ, (S_A,S_B) )^2≤ 1. Thus the contribution of the bad event is bounded by [Fϕ,ϱ(SA,SB)ℰτc] [F_φ, (S_A,S_B)1_E_τ^c ] ≤ℙ(ℰτc) (E_τ^c) ≤(1+1ϕ)τ−q⋆. ≤ (1+ 1φ )τ^-q_ . Hence [Fϕ,ϱ(SA,SB)ℰτc]=Oϕ(τ−q⋆).E [F_φ, (S_A,S_B)1_E_τ^c ]=O_φ(τ^-q_ ). ∎ Lemma 4 (Large-τ expansion with one modified poison data). Fix ϕ∈(0,∞)φ∈(0,∞) and ϱ≥0 ≥ 0. Then [Fϕ,ϱ(S0(τ)+Zp(τ))] [F_φ, (S_0^(τ)+Z_p^(τ) ) ] =ηp2ϕ2τ2[1(ϱ+ϕ(Q0+1))2](1+o(1)) = _p^2φ^2τ^2E [ 1( +φ(Q_0+1))^2 ]\,(1+o(1)) as τ→∞τ→∞. Proof. Write Aτ:=∑ℓ=1N0aτ(Uℓ),Bτ:=∑ℓ=1N0rτ(Uℓ),Dτ:=Bτ−cAτ.A_τ:= _ =1^N_0a_τ(U_ ), B_τ:= _ =1^N_0r_τ(U_ ), D_τ:=B_τ-cA_τ. After adding one modified poison data, the coordinate statistic is (Aτ+1,Bτ+c+ηpτ).(A_τ+1,\ B_τ+c+ _pτ). Using the definition of Gϕ,ϱG_φ, , Gϕ,ϱ(Aτ+1,Bτ+c+ηpτ)−c G_φ, (A_τ+1,B_τ+c+ _pτ)-c =ϕ(Bτ+c+ηpτ)−c(ϱ+ϕ(Aτ+1))ϱ+ϕ(Aτ+1) = φ(B_τ+c+ _pτ)-c( +φ(A_τ+1)) +φ(A_τ+1) =ϕDτ+ϕηpτ−cϱ+ϕ(Aτ+1). = φ D_τ+φ _pτ-c +φ(A_τ+1). Therefore Fϕ,ϱ(Aτ+1,Bτ+c+ηpτ)=(ϕDτ+ϕηpτ−cϱ+ϕ(Aτ+1))2.F_φ, (A_τ+1,B_τ+c+ _pτ)= ( φ D_τ+φ _pτ-c +φ(A_τ+1) )^2. For each fixed realization of (N0,U1,U2,…)(N_0,U_1,U_2,…), Aτ→Q0,Dτ→0,A_τ→ Q_0, D_τ→ 0, because for each fixed UℓU_ , truncation eventually disappears. Hence 1τ2Fϕ,ϱ(Aτ+1,Bτ+c+ηpτ)→ηp2ϕ21(ϱ+ϕ(Q0+1))2 1τ^2F_φ, (A_τ+1,B_τ+c+ _pτ)→ _p^2φ^2 1( +φ(Q_0+1))^2 almost surely. ∎ Lemma 5 (Small-poisoning generator expansion). For every τ≥τ0τ≥ _0, as ϵ↓0ε 0 ϕ,ϱ(ϵ,τ)= _φ, (ε,τ)= μUϵϕ[Fϕ,ϱ(S0(τ)+Zp(τ))−Fϕ,ϱ(S0(τ)+Zc(τ))] _UεφE [F_φ, (S_0^(τ)+Z_p^(τ) )-F_φ, (S_0^(τ)+Z_c^(τ) ) ] +Oϕ,ϱ,c,ηp(ϵ2(1+τ2)). +O_φ, ,c, _p (ε^2(1+τ^2) ). The implicit constant is independent of both ϵε and τ. Proof. Construct three independent Poisson point processes (following previous discussion): Πret∼P(1−ϵϕℒ(Zc(τ))), _ret ( 1-εφ\,L(Z_c^(τ)) ), Πdel∼P(ϵϕℒ(Zc(τ))), _del ( εφ\,L(Z_c^(τ)) ), and Πp∼P(ϵϕℒ(Zp(τ))). _p ( εφ\,L(Z_p^(τ)) ). Define Sret:=Σ(Πret),Pc:=Σ(Πdel),Pp:=Σ(Πp).S_ret:= ( _ret), P_c:= ( _del), P_p:= ( _p). By the superposition theorem for Poisson point processes, Πret∪Πdel∼P(1ϕℒ(Zc(τ))). _ret∪ _del ( 1φ\,L(Z_c^(τ)) ). Hence Sret+Pc=PS0(τ).S_ret+P_c P=S_0^(τ). Likewise, Πret∪Πp _ret∪ _p is the compound-Poisson process corresponding to a coordinate whose clean marks arrive with intensity (1−ϵ)/ϕ(1-ε)/φ and whose poison marks arrive with intensity ϵ/ϕε/φ. Therefore Sret+Pp=PSϵ,τ,S_ret+P_p P=S^ε,τ, where Sϵ,τ=(SAϵ,τ,SBϵ,τ)S^ε,τ=(S_A^ε,τ,S_B^ε,τ) is the limiting coordinate statistic under poisoning. Consequently ϕ,ϱ(ϵ,τ) _φ, (ε,τ) =ℛϕ,ϱ(ϵ,τ)−ℛϕ,ϱ(0,τ) =R_φ, (ε,τ)-R_φ, (0,τ) =μU[F(Sret+Pp)−F(Sret+Pc)]. = _UE [F(S_ret+P_p)-F(S_ret+P_c) ]. We now expand the last expectation to first order in ϵε. Let Kc:=|Πdel|,Kp:=|Πp|,K:=Kc+Kp.K_c:=| _del|, K_p:=| _p|, K:=K_c+K_p. Then Kc∼Pois(ϵ/ϕ),Kp∼Pois(ϵ/ϕ),K_c (ε/φ), K_p (ε/φ), independently, and hence K∼Pois(2ϵ/ϕ).K (2ε/φ). Therefore ℙ(K≥2)=1−e−2ϵ/ϕ(1+2ϵϕ)=Oϕ(ϵ2).P(K≥ 2)=1-e^-2ε/φ (1+ 2εφ )=O_φ(ε^2). For τ≥τ0τ≥ _0, every clean or poison data satisfies the bound 0≤sB≤mτsA,mτ:=max1,c+ηpτ≤Cc,ηpτ.0≤ s_B≤ m_τs_A, m_τ:= \1,c+ _pτ\≤ C_c, _pτ. Hence 0≤Gϕ,ϱ(sA,sB)≤Cc,ηpτ,0≤ G_φ, (s_A,s_B)≤ C_c, _pτ, and therefore ‖Fϕ,ϱ‖∞≤Cc,ηp(1+τ2).\|F_φ, \|_∞≤ C_c, _p(1+τ^2). It follows that the contribution to [F(Sret+Pp)−F(Sret+Pc)]E [F(S_ret+P_p)-F(S_ret+P_c) ] from the event K≥2\K≥ 2\ is bounded in absolute value by 2‖F‖∞ℙ(K≥2)=Oϕ,c,ηp(ϵ2(1+τ2)).2\|F\|_∞P(K≥ 2)=O_φ,c, _p (ε^2(1+τ^2) ). On the event K=0\K=0\, we have Pp=Pc=0P_p=P_c=0. Hence the two terms are equal: F(Sret+Pp)−F(Sret+Pc)=F(Sret)−F(Sret)=0.F(S_ret+P_p)-F(S_ret+P_c)=F(S_ret)-F(S_ret)=0. Thus the zero-data event contributes nothing. It remains to compute the contribution of the event K=1\K=1\. There are two mutually exclusive one-data cases: Kp=1,Kc=0,Kp=0,Kc=1.\K_p=1,K_c=0\, \K_p=0,K_c=1\. Since KpK_p and KcK_c are independent Poisson variables with mean ϵ/ϕε/φ, ℙ(Kp=1,Kc=0)=e−2ϵ/ϕϵϕ,P(K_p=1,K_c=0)=e^-2ε/φ εφ, and ℙ(Kp=0,Kc=1)=e−2ϵ/ϕϵϕ.P(K_p=0,K_c=1)=e^-2ε/φ εφ. Conditioned on Kp=1,Kc=0\K_p=1,K_c=0\, the unique poison data has law Zp(τ)Z_p^(τ), is independent of SretS_ret, and Pc=0P_c=0. Hence the conditional contribution is [F(Sret+Zp(τ))−F(Sret)].E [F(S_ret+Z_p^(τ))-F(S_ret) ]. Conditioned on Kp=0,Kc=1\K_p=0,K_c=1\, the unique deleted clean data has law Zc(τ)Z_c^(τ), is independent of SretS_ret, and Pp=0P_p=0. Hence the conditional contribution is [F(Sret)−F(Sret+Zc(τ))].E [F(S_ret)-F(S_ret+Z_c^(τ)) ]. Adding the two one-data contributions gives [F(Sret+Pp)−F(Sret+Pc);K=1] [F(S_ret+P_p)-F(S_ret+P_c);K=1 ] =e−2ϵ/ϕϵϕ[F(Sret+Zp(τ))−F(Sret+Zc(τ))]. =e^-2ε/φ εφE [F(S_ret+Z_p^(τ))-F(S_ret+Z_c^(τ)) ]. Since e−2ϵ/ϕ=1+Oϕ(ϵ),e^-2ε/φ=1+O_φ(ε), and since the expectation in the last display is bounded in absolute value by 2‖F‖∞=Oc,ηp(1+τ2)2\|F\|_∞=O_c, _p(1+τ^2), we may replace e−2ϵ/ϕe^-2ε/φ by 11 at cost Oϕ,c,ηp(ϵ2(1+τ2)).O_φ,c, _p (ε^2(1+τ^2) ). Therefore [F(Sret+Pp)−F(Sret+Pc)] [F(S_ret+P_p)-F(S_ret+P_c) ] =ϵϕ[F(Sret+Zp(τ))−F(Sret+Zc(τ))]+Oϕ,c,ηp(ϵ2(1+τ2)). = εφE [F(S_ret+Z_p^(τ))-F(S_ret+Z_c^(τ)) ]+O_φ,c, _p (ε^2(1+τ^2) ). Finally, we replace SretS_ret by S0(τ)S_0^(τ). Couple S0(τ)S_0^(τ) as S0(τ)=Sret+Pc′,S_0^(τ)=S_ret+P_c , where Pc′P_c is an independent clean Poisson process with intensity ϵ/ϕε/φ and data law Zc(τ)Z_c^(τ). Then ℙ(Pc′≠0)=1−e−ϵ/ϕ=Oϕ(ϵ).P(P_c ≠ 0)=1-e^-ε/φ=O_φ(ε). Therefore, for a random data Z independent of all processes and equal in law to either Zp(τ)Z_p^(τ) or Zc(τ)Z_c^(τ), |F(Sret+Z)−F(S0(τ)+Z)| |EF(S_ret+Z)-EF(S_0^(τ)+Z) | ≤2‖F‖∞ℙ(Pc′≠0) ≤ 2\|F\|_∞P(P_c ≠ 0) =Oϕ,c,ηp(ϵ(1+τ2)). =O_φ,c, _p (ε(1+τ^2) ). Applying this once with Z=Zp(τ)Z=Z_p^(τ) and once with Z=Zc(τ)Z=Z_c^(τ), we obtain [F(Sret+Zp(τ))−F(Sret+Zc(τ))] [F(S_ret+Z_p^(τ))-F(S_ret+Z_c^(τ)) ] =[F(S0(τ)+Zp(τ))−F(S0(τ)+Zc(τ))]+Oϕ,c,ηp(ϵ(1+τ2)). =E [F(S_0^(τ)+Z_p^(τ))-F(S_0^(τ)+Z_c^(τ)) ]+O_φ,c, _p (ε(1+τ^2) ). Multiplying this replacement error by the outside factor ϵ/ϕε/φ produces Oϕ,c,ηp(ϵ2(1+τ2)).O_φ,c, _p (ε^2(1+τ^2) ). Combining all previous displays gives [F(Sret+Pp)−F(Sret+Pc)] [F(S_ret+P_p)-F(S_ret+P_c) ] =ϵϕ[F(S0(τ)+Zp(τ))−F(S0(τ)+Zc(τ))]+Oϕ,ϱ,c,ηp(ϵ2(1+τ2)). = εφE [F(S_0^(τ)+Z_p^(τ))-F(S_0^(τ)+Z_c^(τ)) ]+O_φ, ,c, _p (ε^2(1+τ^2) ). Multiplying by the prefactor μU _U in the definition of ϕ,ϱ(ϵ,τ)D_φ, (ε,τ) proves the claimed expansion. ∎ Result 5 (Fixed-aspect leading asymptotics). Fix ϕ∈(0,∞)φ∈(0,∞) and ϱ≥0 ≥ 0. Assume ϵ↓0,τ=τ(ϵ)→∞,τ(ϵ)≥τ0.ε 0, τ=τ(ε)→∞, τ(ε)≥ _0. Then the clean-relative degradation satisfies ϕ,ϱ(ϵ,τ)=Cϕ,ϱϵτ2(1+o(1)),D_φ, (ε,τ)=C_φ, \,ετ^2(1+o(1)), where Cϕ,ϱ:=μUηp2ϕ[1(ϱ+ϕ(Q0+1))2].C_φ, := _U _p^2 [ 1( +φ(Q_0+1))^2 ]. In particular, the leading fixed-ϕφ term depends on τ through ϵτ2ετ^2. Proof. By Lemma 5, ϕ,ϱ(ϵ,τ)= _φ, (ε,τ)= μUϵϕ[Fϕ,ϱ(S0(τ)+Zp(τ))−Fϕ,ϱ(S0(τ)+Zc(τ))] _UεφE [F_φ, (S_0^(τ)+Z_p^(τ) )-F_φ, (S_0^(τ)+Z_c^(τ) ) ] +Oϕ,ϱ(ϵ2(1+τ2)). +O_φ, (ε^2(1+τ^2) ). The added-clean expectation is Oϕ,ϱ(1)O_φ, (1) by Lemma 3. The added-poison expectation is, by Lemma 4, ηp2ϕ2τ2[1(ϱ+ϕ(Q0+1))2](1+o(1)). _p^2φ^2τ^2E [ 1( +φ(Q_0+1))^2 ](1+o(1)). Since τ→∞τ→∞, the O(1)O(1) added-clean term is negligible relative to the τ2τ^2 added-poison term. Also, ϵ2(1+τ2)=o(ϵτ2),ε^2(1+τ^2)=o(ετ^2), because ϵ→0ε→ 0 and τ→∞τ→∞. Multiplying the leading added-poison term by μUϵ/ϕ _Uε/φ gives the displayed constant. ∎ In the theorem below we work with the ridgeless risk relative to the untruncated clean fixed-aspect reference ℛϕ,0(0,∞):=limτ→∞ℛϕ,0(0,τ).R_φ,0(0,∞):= _τ→∞R_φ,0(0,τ). Define ℰϕ(ϵ,τ):=ℛϕ,0(ϵ,τ)−ℛϕ,0(0,∞).E_φ(ε,τ):=R_φ,0(ε,τ)-R_φ,0(0,∞). (37) This is the fixed-ϕφ analogue of comparing the poisoned truncated estimator against the ideal clean untruncated target. For z>1z>1, define hc(z):=c(z−1),1<z<1/c,1−c,z≥1/c.h_c(z):= casesc(z-1),&1<z<1/c,\\[2.84526pt] 1-c,&z≥ 1/c. cases Set Kϕ,c,q⋆cl:=μUϕq⋆∫1∞hc(z)2z−q⋆−1z.K_φ,c,q_ ^cl:= _Uφ\,q_ _1^∞h_c(z)^2z^-q_ -1\,dz. (38) This constant is finite and strictly positive. Lemma 6 (Clean truncation term at fixed aspect ratio). For fixed ϕ∈(0,∞)φ∈(0,∞), in the ridgeless case ϱ=0 =0, ℛϕ,0(0,τ)−ℛϕ,0(0,∞)=Kϕ,c,q⋆clτ−q⋆+o(τ−q⋆)R_φ,0(0,τ)-R_φ,0(0,∞)=K_φ,c,q_ ^cl\,τ^-q_ +o(τ^-q_ ) as τ→∞τ→∞. Proof. Recall the clean coordinate model. A sample is generated by drawing J uniformly from 1,…,P\1,…,P\, drawing U, and setting X=PUeJ,Y=cU.X= PU\,e_J, Y=c U. The teacher vector is θ⋆,P=cP(1,…,1)⊤. _ ,P= c P(1,…,1) . Thus, if J=jJ=j, then X⊤θ⋆,P=PUej⊤θ⋆,P=cU=Y.X _ ,P= PU\,e_j _ ,P=c U=Y. Hence the clean model is noiseless at the coordinate level in this simplified calculation. Let NjN_j be the number of clean training samples that hit coordinate j. For those samples, write their radial variables as Uj,1,…,Uj,Nj.U_j,1,…,U_j,N_j. In the untruncated clean problem, the coordinate-level sufficient statistics are Sj,A=∑ℓ=1NjUj,ℓ,Sj,B=∑ℓ=1NjcUj,ℓ=cSj,A.S_j,A= _ =1^N_jU_j, , S_j,B= _ =1^N_jcU_j, =cS_j,A. The empirical Hessian is diagonal. The ridgeless estimator is defined by the Moore–Penrose inverse, so Pθ^j=Sj,BSj,A,Sj,A>0,0,Sj,A=0. P\, θ_j= cases S_j,BS_j,A,&S_j,A>0,\\[8.53581pt] 0,&S_j,A=0. cases Since Uj,ℓ>0U_j, >0, the condition Sj,A>0S_j,A>0 is equivalent to Nj≥1N_j≥ 1. If Nj≥1N_j≥ 1, then Sj,BSj,A=cSj,ASj,A=c. S_j,BS_j,A= cS_j,AS_j,A=c. Therefore Pθ^j=cwhenever Nj≥1. P\, θ_j=c N_j≥ 1. Equivalently, θ^j=cP=(θ⋆,P)jwhenever Nj≥1. θ_j= c P=( _ ,P)_j N_j≥ 1. Thus every observed coordinate is fit exactly. If Nj=0N_j=0, then Sj,A=Sj,B=0.S_j,A=S_j,B=0. The Moore–Penrose convention gives Pθ^j=0,θ^j=0. P\, θ_j=0, θ_j=0. But the teacher coordinate is (θ⋆,P)j=cP.( _ ,P)_j= c P. Hence unobserved coordinates are not fit; they are set to zero by the minimum-norm convention. The clean excess risk identity is ℛclean,P(θ^)−ℛclean,P(θ⋆,P)=μU1P∑j=1P(c−Pθ^j)2.R_clean,P( θ)-R_clean,P( _ ,P)= _U 1P _j=1^P (c- P\, θ_j )^2. From the preceding calculation, c−Pθ^j=0,Nj≥1,c,Nj=0.c- P\, θ_j= cases0,&N_j≥ 1,\\ c,&N_j=0. cases Therefore (c−Pθ^j)2=c2Nj=0. (c- P\, θ_j )^2=c^21\N_j=0\. Substituting into the risk identity gives ℛclean,P(θ^)−ℛclean,P(θ⋆,P)=μUc21P∑j=1PNj=0.R_clean,P( θ)-R_clean,P( _ ,P)= _Uc^2 1P _j=1^P1\N_j=0\. Thus the empirical clean excess risk is exactly proportional to the fraction of unobserved coordinates. When PD→ϕ∈(0,∞), PD→φ∈(0,∞), the number of clean samples hitting a fixed coordinate converges to N0∼Pois(1/ϕ).N_0 (1/φ). Hence ℙ(N0=0)=e−1/ϕ.P(N_0=0)=e^-1/φ. By the occupancy law of large numbers, 1P∑j=1PNj=0→ℙe−1/ϕ. 1P _j=1^P1\N_j=0\ Pe^-1/φ. Therefore, in the untruncated clean ridgeless problem, ℛϕ,0(0,∞)=μUc2e−1/ϕ.R_φ,0(0,∞)= _Uc^2e^-1/φ. In the untruncated ridgeless clean problem, every observed coordinate is fit exactly and only unobserved coordinates contribute error. Hence ℛϕ,0(0,∞)=μUc2ℙ(N0=0).R_φ,0(0,∞)= _Uc^2P(N_0=0). Recall, Aτ:=∑ℓ=1N0aτ(Uℓ),Bτ:=∑ℓ=1N0rτ(Uℓ).A_τ:= _ =1^N_0a_τ(U_ ), B_τ:= _ =1^N_0r_τ(U_ ). When Aτ>0A_τ>0, the coordinate estimator is Bτ/AτB_τ/A_τ; when Aτ=0A_τ=0, the Moore–Penrose convention gives estimator 00. Let Δτ(t):=rτ(t)−caτ(t). _τ(t):=r_τ(t)-ca_τ(t). Then Δτ(t)=0for t≤τ, _τ(t)=0 t≤τ, and, for t>τt>τ, Δτ(t)=c(t−τ),τ<t<τ/c,(1−c)τ,t≥τ/c. _τ(t)= casesc(t-τ),&τ<t<τ/c,\\[2.84526pt] (1-c)τ,&t≥τ/c. cases Therefore Bτ−cAτ=∑ℓ=1N0Δτ(Uℓ)≥0.B_τ-cA_τ= _ =1^N_0 _τ(U_ )≥ 0. It follows that ℛϕ,0(0,τ)−ℛϕ,0(0,∞)=μU[(∑ℓ=1N0Δτ(Uℓ)∑ℓ=1N0aτ(Uℓ))2N0≥1],R_φ,0(0,τ)-R_φ,0(0,∞)= _U\,E [ ( _ =1^N_0 _τ(U_ ) _ =1^N_0a_τ(U_ ) )^21\N_0≥ 1\ ], (39) with the ratio interpreted as zero when the denominator is zero. The random collection of clean radii U1,…,UN0\U_1,…,U_N_0\ can be viewed as a Poisson point process on [1,∞)[1,∞) with intensity measure ν(dt)=1ϕq⋆t−q⋆−1dt.ν(\,dt)= 1φq_ t^-q_ -1\,dt. Indeed, the total mass of ν is ν([1,∞))=1ϕ∫1∞q⋆t−q⋆−1t=1ϕ,ν([1,∞))= 1φ _1^∞q_ t^-q_ -1\,dt= 1φ, so the total number of points is Pois(1/ϕ)Pois(1/φ), and the normalized data law has density q⋆t−q⋆−1q_ t^-q_ -1 on [1,∞)[1,∞). Let Π:=U1,…,UN0 :=\U_1,…,U_N_0\ denote this Poisson point process. Define Rτ(Π):=(∑t∈ΠΔτ(t)∑t∈Πaτ(t))2|Π|≥1,R_τ( ):= ( _t∈ _τ(t) _t∈ a_τ(t) )^21\| |≥ 1\, with the ratio interpreted as zero when the denominator is zero. Since aτ(t)>0a_τ(t)>0 for t≥1t≥ 1, the denominator is positive whenever |Π|≥1| |≥ 1. Also, 0≤Δτ(t)≤aτ(t),0≤ _τ(t)≤ a_τ(t), so 0≤Rτ(Π)≤1.0≤ R_τ( )≤ 1. Let Nτ+:=#t∈Π:t>τN_τ^+:=\#\t∈ :t>τ\ be the number of exceedances above τ. Since the intensity of (τ,∞)(τ,∞) is ν((τ,∞))=1ϕℙ(U>τ)=1ϕτ−q⋆,ν((τ,∞))= 1φP(U>τ)= 1φτ^-q_ , we have Nτ+∼Pois(1ϕτ−q⋆).N_τ^+ ( 1φτ^-q_ ). Therefore ℙ(Nτ+=0)=1−O(τ−q⋆),P(N_τ^+=0)=1-O(τ^-q_ ), ℙ(Nτ+=1)=1ϕτ−q⋆+O(τ−2q⋆),P(N_τ^+=1)= 1φτ^-q_ +O(τ^-2q_ ), and ℙ(Nτ+≥2)=O(τ−2q⋆).P(N_τ^+≥ 2)=O(τ^-2q_ ). On the event Nτ+=0N_τ^+=0, every radius is at most τ, hence Δτ(Uℓ)=0for all ℓ, _τ(U_ )=0 all , and therefore Rτ(Π)=0.R_τ( )=0. On the event Nτ+≥2N_τ^+≥ 2, we use only the bound 0≤Rτ(Π)≤10≤ R_τ( )≤ 1: [Rτ(Π)Nτ+≥2]≤ℙ(Nτ+≥2)=O(τ−2q⋆)=o(τ−q⋆).E [R_τ( )1\N_τ^+≥ 2\ ] (N_τ^+≥ 2)=O(τ^-2q_ )=o(τ^-q_ ). Thus the leading contribution comes only from the event of exactly one exceedance. The Campbell–Mecke formula says that, for a Poisson point process Π with intensity measure ν, [∑t∈Πg(t,Π∖t)]=∫[g(t,Π)]ν(t),E [ _t∈ g(t, \t\) ]= [g(t, ) ]ν(\,dt), whenever g≥0g≥ 0 or the expectations are finite. We apply it to the contribution from a distinguished exceedance t>τt>τ and then note that configurations with more than one exceedance are negligible. Let Π≤τ:=Π∩[1,τ] _≤τ:= ∩[1,τ] be an independent Poisson point process on [1,τ][1,τ] with intensity ν restricted to [1,τ][1,τ]. By the independence of Poisson processes on disjoint sets, after selecting the unique exceedance t>τt>τ, the remaining sub-threshold points have the same law as Π≤τ _≤τ, up to an error supported on events with another exceedance. Those events have probability O(τ−2q⋆)O(τ^-2q_ ). Hence the one-exceedance contribution is Iτ I_τ :=[Rτ(Π)Nτ+=1] :=E [R_τ( )1\N_τ^+=1\ ] =∫τ∞[(Δτ(t)aτ(t)+∑u∈Π≤τaτ(u))2]ν(dt)+O(τ−2q⋆). = _τ^∞E [ ( _τ(t)a_τ(t)+ _u∈ _≤τa_τ(u) )^2 ]ν(\,dt)+O(τ^-2q_ ). Since t>τt>τ, we have aτ(t)=τ.a_τ(t)=τ. Now set t=τz,z>1.t=τ z, z>1. The Pareto intensity transforms as ν(dt)=1ϕq⋆t−q⋆−1dt=1ϕq⋆(τz)−q⋆−1τdz=1ϕq⋆τ−q⋆z−q⋆−1dz.ν(\,dt)= 1φq_ t^-q_ -1\,dt= 1φq_ (τ z)^-q_ -1τ\,dz= 1φq_ τ^-q_ z^-q_ -1\,dz. Also, Δτ(τz)τ=hc(z). _τ(τ z)τ=h_c(z). Therefore Iτ I_τ =1ϕq⋆τ−q⋆∫1∞[(Δτ(τz)τ+∑u∈Π≤τaτ(u))2]z−q⋆−1dz+O(τ−2q⋆) = 1φq_ τ^-q_ _1^∞E [ ( _τ(τ z)τ+ _u∈ _≤τa_τ(u) )^2 ]z^-q_ -1\,dz+O(τ^-2q_ ) =1ϕq⋆τ−q⋆∫1∞hc(z)2[(τ+∑u∈Π≤τaτ(u))2]z−q⋆−1dz+O(τ−2q⋆). = 1φq_ τ^-q_ _1^∞h_c(z)^2E [ ( τ+ _u∈ _≤τa_τ(u) )^2 ]z^-q_ -1\,dz+O(τ^-2q_ ). We claim that 1τ∑u∈Π≤τaτ(u)⟶0in probability. 1τ _u∈ _≤τa_τ(u) 0 probability. Indeed, on [1,τ][1,τ] we have aτ(u)=ua_τ(u)=u, so [∑u∈Π≤τaτ(u)]=∫1τuν(u)=1ϕ∫1τq⋆u−q⋆u.E [ _u∈ _≤τa_τ(u) ]= _1^τu\,ν(\,du)= 1φ _1^τq_ u^-q_ \,du. Since q⋆>1q_ >1, ∫1τq⋆u−q⋆u≤∫1∞q⋆u−q⋆u<∞. _1^τq_ u^-q_ \,du≤ _1^∞q_ u^-q_ \,du<∞. Thus [∑u∈Π≤τaτ(u)]=Oϕ(1),E [ _u∈ _≤τa_τ(u) ]=O_φ(1), and 1τ∑u∈Π≤τaτ(u)→ℙ0. 1τ _u∈ _≤τa_τ(u) P0. Consequently, (τ+∑u∈Π≤τaτ(u))2→ℙ1. ( τ+ _u∈ _≤τa_τ(u) )^2 P1. Moreover, because 0≤hc(z)≤1,0≤ h_c(z)≤ 1, and ∫1∞z−q⋆−1z<∞, _1^∞z^-q_ -1\,dz<∞, dominated convergence also applies to the integral over z. Therefore Iτ=1ϕq⋆τ−q⋆∫1∞hc(z)2z−q⋆−1z+o(τ−q⋆).I_τ= 1φq_ τ^-q_ _1^∞h_c(z)^2z^-q_ -1\,dz+o(τ^-q_ ). Combining the decomposition [Rτ(Π)]=[Rτ(Π)Nτ+=0]+[Rτ(Π)Nτ+=1]+[Rτ(Π)Nτ+≥2],E[R_τ( )]=E[R_τ( )1\N_τ^+=0\]+E[R_τ( )1\N_τ^+=1\]+E[R_τ( )1\N_τ^+≥ 2\], with [Rτ(Π)Nτ+=0]=0,E[R_τ( )1\N_τ^+=0\]=0, [Rτ(Π)Nτ+≥2]=o(τ−q⋆),E[R_τ( )1\N_τ^+≥ 2\]=o(τ^-q_ ), and the one-exceedance estimate above, we obtain [(∑ℓ=1N0Δτ(Uℓ)∑ℓ=1N0aτ(Uℓ))2N0≥1] [ ( _ =1^N_0 _τ(U_ ) _ =1^N_0a_τ(U_ ) )^21\N_0≥ 1\ ] =1ϕq⋆τ−q⋆∫1∞hc(z)2z−q⋆−1dz+o(τ−q⋆). = 1φ\,q_ τ^-q_ _1^∞h_c(z)^2z^-q_ -1\,dz+o(τ^-q_ ). ∎ Result 6 (Fixed-ϕφ two-term risk and optimal truncation). Fix ϕ∈(0,∞)φ∈(0,∞) and consider the ridgeless modified attack ϱ=0 =0. Assume ϵ↓0,τ=τ(ϵ)→∞,ε 0, τ=τ(ε)→∞, so that, for all sufficiently small ϵε, τ(ϵ)≥τ0τ(ε)≥ _0. Then ℰϕ(ϵ,τ)=Kϕ,c,q⋆clτ−q⋆+Cϕ,0ϵτ2+o(τ−q⋆+ϵτ2),E_φ(ε,τ)=K_φ,c,q_ ^clτ^-q_ +C_φ,0ετ^2+o\! (τ^-q_ +ετ^2 ), (40) where Cϕ,0=μUηp2ϕ[1ϕ2(Q0+1)2]=μUηp2ϕ[1(Q0+1)2].C_φ,0= _U _p^2φ\,E [ 1φ^2(Q_0+1)^2 ]= _U _p^2φE [ 1(Q_0+1)^2 ]. Consequently the leading two-term equivalent Hϵ(τ):=Kϕ,c,q⋆clτ−q⋆+Cϕ,0ϵτ2H_ε(τ):=K_φ,c,q_ ^clτ^-q_ +C_φ,0ετ^2 is minimized at τopt(ϵ)=(q⋆Kϕ,c,q⋆cl2Cϕ,0ϵ)1/(q⋆+2). _opt(ε)= ( q_ K_φ,c,q_ ^cl2C_φ,0ε )^1/(q_ +2). (41) This choice is admissible, τopt(ϵ)≥τ0 _opt(ε)≥ _0, for all sufficiently small ϵε, and the optimized fixed-ϕφ scaling is ℰϕ(ϵ,τopt(ϵ))≍ϵq⋆/(q⋆+2).E_φ(ε, _opt(ε)) ε^q_ /(q_ +2). (42) More precisely, ℰϕ(ϵ,τopt(ϵ))∼ _φ(ε, _opt(ε)) (1+q⋆2)Kϕ,c,q⋆cl(2Cϕ,0q⋆Kϕ,c,q⋆cl)q⋆/(q⋆+2)ϵq⋆/(q⋆+2). (1+ q_ 2 )K_φ,c,q_ ^cl ( 2C_φ,0q_ K_φ,c,q_ ^cl )^q_ /(q_ +2)ε^q_ /(q_ +2). Proof. By definition, ℰϕ(ϵ,τ)=[ℛϕ,0(0,τ)−ℛϕ,0(0,∞)]+[ℛϕ,0(ϵ,τ)−ℛϕ,0(0,τ)].E_φ(ε,τ)= [R_φ,0(0,τ)-R_φ,0(0,∞) ]+ [R_φ,0(ε,τ)-R_φ,0(0,τ) ]. The first bracket is the clean truncation term, so Lemma 6 gives ℛϕ,0(0,τ)−ℛϕ,0(0,∞)=Kϕ,c,q⋆clτ−q⋆+o(τ−q⋆).R_φ,0(0,τ)-R_φ,0(0,∞)=K_φ,c,q_ ^clτ^-q_ +o(τ^-q_ ). The second bracket is ϕ,0(ϵ,τ)D_φ,0(ε,τ). The fixed-ϕφ leading-response theorem, using the uniform generator remainder from Lemma 5, gives ϕ,0(ϵ,τ)=Cϕ,0ϵτ2(1+o(1)).D_φ,0(ε,τ)=C_φ,0ετ^2(1+o(1)). Thus the combined remainder is o(τ−q⋆)+o(ϵτ2)=o(τ−q⋆+ϵτ2),o(τ^-q_ )+o(ετ^2)=o(τ^-q_ +ετ^2), which proves equation 40. It remains to minimize HϵH_ε. Differentiating, Hϵ′(τ)=−q⋆Kϕ,c,q⋆clτ−q⋆−1+2Cϕ,0ϵτ.H_ε (τ)=-q_ K_φ,c,q_ ^clτ^-q_ -1+2C_φ,0ετ. The unique critical point satisfies q⋆Kϕ,c,q⋆clτ−q⋆−1=2Cϕ,0ϵτ,q_ K_φ,c,q_ ^clτ^-q_ -1=2C_φ,0ετ, which is equivalent to equation 41. Since τopt(ϵ)→∞ _opt(ε)→∞, it lies in the admissible region τ≥τ0τ≥ _0 for all sufficiently small ϵε. At the optimum, Cϕ,0ϵτopt2=q⋆2Kϕ,c,q⋆clτopt−q⋆.C_φ,0ε _opt^2= q_ 2K_φ,c,q_ ^cl _opt^-q_ . Therefore Hϵ(τopt)=(1+q⋆2)Kϕ,c,q⋆clτopt−q⋆.H_ε( _opt)= (1+ q_ 2 )K_φ,c,q_ ^cl _opt^-q_ . Substitution of equation 41 gives the stated constant and exponent. ∎ A.6 ϕ↓0φ 0: Classical Statistics The fixed-aspect-ratio formula contains a population-mixture calculation as a further many-samples-per-coordinate limit. This limit is different from the fixed-ϕφ small-ϵε limit. Taking ϕ↓0φ 0 first removes the sparse coordinate-hit mechanism. Result 7 (Population limit of the deterministic equivalent). Fix ϵ∈[0,1)ε∈[0,1), τ≥τ0τ≥ _0, and ϱ≥0 ≥ 0. Then, as ϕ↓0φ 0, ℛϕ,ϱ(ϵ,τ)⟶ℛ0,ϱ(ϵ,τ),R_φ, (ε,τ) _0, (ε,τ), where ℛ0,ϱ(ϵ,τ):=μU[c−(1−ϵ)Bτ+ϵ(c+ηpτ)ϱ+(1−ϵ)Mτ+ϵ]2.R_0, (ε,τ):= _U [c- (1-ε)B_τ+ε(c+ _pτ) +(1-ε)M_τ+ε ]^2. (43) In particular, in the ridgeless population limit ϱ=0 =0, ℛ0,0(ϵ,τ)=μU[(1−ϵ)c(Mτ/c−Mτ)+ϵηpτ(1−ϵ)Mτ+ϵ]2.R_0,0(ε,τ)= _U [ (1-ε)c(M_τ/c-M_τ)+ε _pτ(1-ε)M_τ+ε ]^2. (44) Proof. The clean and poison rates in equation 34 are (1−ϵ)/ϕ(1-ε)/φ and ϵ/ϕε/φ. Therefore ϕSAϵ,τ→ℙ(1−ϵ)Mτ+ϵ,φ S_A^ε,τ P(1-ε)M_τ+ε, and ϕSBϵ,τ→ℙ(1−ϵ)Bτ+ϵ(c+ηpτ).φ S_B^ε,τ P(1-ε)B_τ+ε(c+ _pτ). The variances are O(ϕ)O(φ) because the summands are bounded by constants depending only on τ. The denominator ϱ+ϕSAϵ,τ +φ S_A^ε,τ converges in probability to ϱ+(1−ϵ)Mτ+ϵ>0 +(1-ε)M_τ+ε>0. Since Gϕ,ϱG_φ, is bounded for fixed τ, convergence in probability plus bounded convergence gives equation 43. For ϱ=0 =0, substitute Bτ=cMτ/cB_τ=cM_τ/c and rearrange: c−(1−ϵ)cMτ/c+ϵ(c+ηpτ)(1−ϵ)Mτ+ϵ c- (1-ε)cM_τ/c+ε(c+ _pτ)(1-ε)M_τ+ε =−(1−ϵ)c(Mτ/c−Mτ)+ϵηpτ(1−ϵ)Mτ+ϵ. =- (1-ε)c(M_τ/c-M_τ)+ε _pτ(1-ε)M_τ+ε. Squaring gives equation 44. ∎ The exact Pareto identity gives Mτ/c−Mτ=1−cq⋆−1q⋆−1τ1−q⋆.M_τ/c-M_τ= 1-c^q_ -1q_ -1τ^1-q_ . Define A:=c1−cq⋆−1q⋆−1,B:=ηp.A:=c 1-c^q_ -1q_ -1, B:= _p. Result 8 (Optimized poisoning exponent after taking ϕ↓0φ 0). Consider the ridgeless population limit equation 44. In the perturbative regime ϵ↓0,τ=τ(ϵ)→∞,ϵτ(ϵ)→0,ε 0, τ=τ(ε)→∞, ετ(ε)→ 0, one has ℛ0,0(ϵ,τ)=1μU(Aτ−(q⋆−1)+Bϵτ)2(1+o(1)).R_0,0(ε,τ)= 1 _U (Aτ^-(q_ -1)+Bετ )^2(1+o(1)). (45) The leading term is minimized by τopt(ϵ)=(A(q⋆−1)Bϵ)1/q⋆, _opt(ε)= ( A(q_ -1)Bε )^1/q_ , (46) and at this value ℛ0,0(ϵ,τopt(ϵ))∼q⋆2μUA2/q⋆B2(q⋆−1)/q⋆(q⋆−1)−2(q⋆−1)/q⋆ϵ2−2/q⋆.R_0,0(ε, _opt(ε)) q_ ^2 _UA^2/q_ B^2(q_ -1)/q_ (q_ -1)^-2(q_ -1)/q_ ε^2-2/q_ . (47) Proof. From equation 44, ℛ0,0(ϵ,τ)=μU[(1−ϵ)Aτ−(q⋆−1)+Bϵτ(1−ϵ)Mτ+ϵ]2.R_0,0(ε,τ)= _U [ (1-ε)Aτ^-(q_ -1)+Bετ(1-ε)M_τ+ε ]^2. Since τ→∞τ→∞ and ϵτ→0ετ→ 0, (1−ϵ)Mτ+ϵ=μU(1+o(1)),(1-ε)M_τ+ε= _U(1+o(1)), and (1−ϵ)Aτ−(q⋆−1)=Aτ−(q⋆−1)(1+o(1)).(1-ε)Aτ^-(q_ -1)=Aτ^-(q_ -1)(1+o(1)). This proves equation 45. It remains to minimize gϵ(τ):=Aτ−(q⋆−1)+Bϵτ.g_ε(τ):=Aτ^-(q_ -1)+Bετ. Differentiating gives gϵ′(τ)=−A(q⋆−1)τ−q⋆+Bϵ.g_ε (τ)=-A(q_ -1)τ^-q_ +Bε. The unique critical point is equation 46, and it is the global minimizer of the leading expression because gϵ(τ)→∞g_ε(τ)→∞ as τ↓0τ 0 or τ→∞τ→∞. At the optimum, Bϵτopt=A(q⋆−1)τopt−(q⋆−1),Bε _opt=A(q_ -1) _opt^-(q_ -1), so gϵ(τopt)=q⋆Aτopt−(q⋆−1)=q⋆A1/q⋆B(q⋆−1)/q⋆(q⋆−1)−(q⋆−1)/q⋆ϵ1−1/q⋆.g_ε( _opt)=q_ A _opt^-(q_ -1)=q_ A^1/q_ B^(q_ -1)/q_ (q_ -1)^-(q_ -1)/q_ ε^1-1/q_ . Squaring and multiplying by 1/μU1/ _U proves equation 47. ∎ A.7 The same observable under the two orders of limits Fix the ridgeless estimator (ϱ=0 =0) and consider the degradation of the poisoned truncated estimator measured against the clean untruncated ridgeless estimator, Δϵ,τ(P):=ℛclean,P(θ^ϵ,τ,0)−ℛclean,P(θ^0,∞,0), ^(P)_ε,τ:=R_clean,P ( θ_ε,τ,0 )-R_clean,P ( θ_0,∞,0 ), (48) where θ^0,∞,0 θ_0,∞,0 denotes the clean (ϵ=0ε=0), untruncated (τ=∞τ=∞), ridgeless estimator. The results from previous discussion is summarized below. Result 9 (High-dimensional vs ample-data limit). Assume P/D→ϕ∈(0,∞)P/D→φ∈(0,∞) and work at ϱ=0 =0. For every fixed ϕ∈(0,∞)φ∈(0,∞), ϵ∈[0,1)ε∈[0,1) and τ≥τ0τ≥ _0, Δϵ,τ(P)→ℙℰϕ(ϵ,τ):=ℛϕ,0(ϵ,τ)−μUc2e−1/ϕ. ^(P)_ε,τ \ P\ E_φ(ε,τ):=R_φ,0(ε,τ)- _Uc^2e^-1/φ. (i) (High-dimensional / fixed aspect ratio.) In the regime ϵ↓0ε 0, τ=τ(ϵ)→∞τ=τ(ε)→∞, ℰϕ(ϵ,τ)=Kϕ,c,q⋆clτ−q⋆+Cϕ,0ϵτ2+o(τ−q⋆+ϵτ2),E_φ(ε,τ)=K^cl_φ,c,q_ \,τ^-q_ +C_φ,0\,ετ^2+o\! (τ^-q_ +ετ^2 ), (49) which is minimized at τopt(ϵ)≍ϵ−1/(q⋆+2) _opt(ε) ε^-1/(q_ +2), with minτℰϕ(ϵ,τ)≍ϵq⋆/(q⋆+2). _τE_φ(ε,τ)\ \ ε^\,q_ /(q_ +2). (50) (i) (Ample-data limit.) Taking the aspect ratio to zero first, limϕ↓0ℰϕ(ϵ,τ)=ℛ0,0(ϵ,τ), _φ 0E_φ(ε,τ)=R_0,0(ε,τ), and in the regime ϵ↓0ε 0, τ→∞τ→∞, ϵτ→0ετ→ 0, ℛ0,0(ϵ,τ)=1μU(Aτ−(q⋆−1)+Bϵτ)2(1+o(1)),R_0,0(ε,τ)= 1 _U (A\,τ^-(q_ -1)+B\,ετ )^2 (1+o(1) ), (51) which is minimized at τopt(ϵ)≍ϵ−1/q⋆ _opt(ε) ε^-1/q_ , with minτℛ0,0(ϵ,τ)≍ϵ 2−2/q⋆. _τR_0,0(ε,τ)\ \ ε^\,2-2/q_ . (52) (a) (b) Figure 5: Scaling law for a small amount of random poisoning ε . We see that the scaling law for the larger models is more stable. This fact might be explained by saying stronger models have smaller ϕLLM _LLM and hence the ample data regime extends up to smaller values of ε . (a) (b) Figure 6: Scaling law for a small amount of language switching poisoning ε . We see that the scaling law for the larger models is more stable. This fact might be explained by saying stronger models have smaller ϕLLM _LLM and hence the ample data regime extends up to smaller values of ε . A.8 Possible implications of the non-commuting limit for LLM pre-training In an ample-data calculation, poisoning changes population moments smoothly and the clean truncation bias and poison displacement are balanced before the final quadratic validation metric is applied. In the coordinate model this order of limits gives (τ−(q⋆−1)+ετ)2 (τ^-(q_ -1)+ τ )^2 and hence the optimized exponent 2−2/q⋆2-2/q_ . The LLM calculation in the main text is performed in ample data limit and the latent cluster model leads to similar exponent with q⋆=1−m+2rm2rmq_ = 1-m+2rm2rm (53) At fixed ϕ>0φ>0, however, some parameter directions receive only a finite number of effective observations. A poison event can then hit a previously weakly observed coordinate with probability of order ε/ϕ /φ. Conditional on such a hit, the coordinate displacement is of order τ, so the risk contribution is already of order τ2τ^2 before averaging over poisoned coordinates. This produces the different balance τ−q⋆+ετ2τ^-q_ + τ^2 and the optimized exponent q⋆/(q⋆+2)q_ /(q_ +2). Since the scaling exponent is different in this two regimes it suggests that for LLM, making ϵε arbitrarily small would change the scaling law curve we have obtained. Appendix B How the scaling law may arise in LLM pre-training In this section, we analyze a local approximation to the training dynamics over a short time scale in a neighborhood of a finite-time checkpoint rather than attempting to model the full dynamics of transformer pre-training. Within this local framework, we introduce a sequence of assumptions under which the observed pre-training poisoning scaling law can be derived. Because several of these assumptions have not yet been validated experimentally, the analysis should be interpreted as proposing a plausible mechanism for the observed scaling behavior, rather than as a complete or uniquely established explanation. B.1 Local training dynamics The construction in this subsection and the next is purely kinematic: we only linearize the model and form the local quadratic objective (Meterez et al., 2026), so it applies verbatim whether the training data are clean or poisoned. Let θ∈ℝPθ ^P denote the vector of model parameters, and let θinθ^in be the reference parameter vector about which we consider local dynamics. This reference may be the minimizer of the clean population objective or, more generally, a checkpoint along the (clean or poisoned) training trajectory. Write δθ=θ−θin.δθ=θ-θ^in. (54) We index the training data by token-prediction events Zi=(Xi,Yi)Z_i=(X_i,Y_i), where XiX_i is the context and YiY_i is the target token. The symbol D denotes the total number of such events. Equivalently, i may index sequences, provided that the corresponding quantities below are understood to include all token losses in a sequence. Let fθ(Xi)∈ℝVf_θ(X_i) ^V (55) be the vector of pre-softmax logits produced by the model for example i, where V is the vocabulary size. The example-level Jacobian at the reference parameters is Ji:=∂fθ(Xi)∂θ|θ=θin∈ℝV×P.J_i:= . ∂ f_θ(X_i)∂θ |_θ=θ^in ^V× P. (56) Thus JiδθJ_iδθ is the first-order change in the logits caused by a parameter displacement δθδθ: fθin+δθ(Xi)=fθin(Xi)+Jiδθ+O(‖δθ‖2).f_θ^in+δθ(X_i)=f_θ^in(X_i)+J_iδθ+O(\|δθ\|^2). (57) Let ℓi(f) _i(f) denote the loss of example i as a function of its logits. Define gi:=∇fℓi(f)|f=fθin(Xi)∈ℝV,i:=∇f2ℓi(f)|f=fθin(Xi)∈ℝV×V.g_i:= . _f _i(f) |_f=f_θ^in(X_i) ^V, _i:= . _f^2 _i(f) |_f=f_θ^in(X_i) ^V× V. (58) Here gig_i is the gradient of the loss with respect to the logits, and iW_i is the corresponding logit-space curvature matrix. We define the pseudo-residual ri:=−gi.r_i:=-g_i. This sign convention ensures that Ji⊤riJ_i r_i points in the local descent direction.† For the softmax cross-entropy loss, if pi:=softmax(fθin(Xi))p_i:=softmax (f_θ^in(X_i) ) and eYie_Y_i is the one-hot vector associated with the target token, then gi=pi−eYi,ri=eYi−pi,i=diag(pi)−pipi⊤.g_i=p_i-e_Y_i, r_i=e_Y_i-p_i, _i=diag(p_i)-p_ip_i . (59) The empirical generalized Gauss–Newton matrix is Σ^:=1D∑i=1DJi⊤iJi∈ℝP×P. := 1D _i=1^DJ_i W_iJ_i ^P× P. (60) It measures the local curvature of the empirical training loss in parameter space. It is also closely related to an empirical Fisher matrix. The empirical response vector is v^:=1D∑i=1DJi⊤ri=−∇θℒ^(θ)|θ=θin∈ℝP, v:= 1D _i=1^DJ_i r_i=- . _θ L(θ) |_θ=θ^in ^P, (61) where ℒ^(θ):=1D∑i=1Dℓi(fθ(Xi)) L(θ):= 1D _i=1^D _i (f_θ(X_i) ) (62) is the empirical pre-training objective. Thus v v is the negative empirical gradient evaluated at the reference parameters. Under the local linearization of the logits and the generalized Gauss–Newton approximation, the empirical objective becomes ℒ^(θin+δθ)≃ℒ^(θin)−v^⊤δθ+12δθ⊤Σ^δθ. L(θ^in+δθ) L(θ^in)- v δθ+ 12δθ δθ. (63) The stationary condition for this quadratic approximation is therefore Σ^δθ^=v^, \, δθ= v, (64) which gives δθ^≃Σ^−1v^. δθ ^-1 v. (65) B.2 Finite training time as truncation In this subsection, we analyze the training dynamics and argue that finite-time training is equivalent to an effective truncation on the curvature spectrum. As in the previous subsection, the argument is stated for a generic local curvature Σ and response v v and therefore holds for both clean and poisoned data; the two are distinguished only later (Appendices B.4–B.6) through the choice of curvature and response. The one new ingredient is the optimizer’s preconditioner S. We show that it leaves the stationary point of the local dynamics unchanged but changes the operator whose spectrum is filtered due to finite training: not Σ itself but its symmetrised preconditioned version A=S1/2Σ^S1/2A=S^1/2 S^1/2. This is the operator whose spectrum is measured in Appendix E, and every spectral assumption in the rest of this appendix refers to it. A broad class of memoryless preconditioned first-order methods has the update δθt+1=δθt+ηtSt(v^−Σ^δθt),δ _t+1=δ _t+ _tS_t ( v- δ _t ), (66) where ηt≥0 _t≥ 0 is the learning rate and StS_t is a self-adjoint positive-definite preconditioning operator. The Adam optimiser (kingma2015adam) fits the memoryless template equation 66 exactly when its momentum is switched off, i.e., β1=0 _1=0. Writing the gradient of the local quadratic objective at step t as gt:=Σ^δθt−v^=−(v^−Σ^δθt),g_t:= \,δ _t- v=- ( v- \,δ _t ), Adam maintains the diagonal second-moment accumulator† Here ⊙ is the Hadamard product. We write Adam’s second-moment estimate as νt _t rather than the customary vtv_t to avoid a clash with the response vector v v. νt=β2νt−1+(1−β2)gt⊙gt,ν^t:=νt1−β2t+1, _t= _2\, _t-1+(1- _2)\,g_t g_t, ν_t:= _t1- _2^\,t+1, (67) and takes the step δθt+1=δθt−ηtgtν^t+ϵA⟹St=(diag(ν^t)+ϵAI)−1,δ _t+1=δ _t- _t\, g_t ν_t+ _A S_t= (diag\! ( ν_t\, )+ _AI )^-1, (68) where ϵA>0 _A>0 is Adam’s damping constant. For β1>0 _1>0 the update carries memory. In the small-learning-rate limit, heavy-ball momentum reduces to gradient flow with time rescaled by (1−β1)−1(1- _1)^-1 (kovachki2021continuous); we assume, without proof, that the same reduction applies to Adam’s momentum, so that β1>0 _1>0 only rescales the time variable introduced below and does not affect any exponent. Adam’s second moment ν^t ν_t is roughly an exponential moving average with a time constant of (1−β2)−1(1- _2)^-1 steps. Late in training, where the gradient statistics drift slowly, it is nearly constant across a window of that length. We make this a standing assumption of the local model, following Meterez et al. (2026), who freeze Adam’s second-moment at its moving-average value and find that the resulting quadratic model predicts the training loss over windows of up to 10%10\% of training. Assumption 10 (Frozen preconditioner on the local window). Across the local window the preconditioner is a fixed symmetric positive-definite operator, St≡S_t≡ S, with sminI⪯S⪯smaxI,0<smin≤smax<∞.s_ I S s_ I, 0<s_ ≤ s_ <∞. (69) For Adam, equation 68 gives smax=(minjν^j+ϵA)−1≤ϵA−1s_ =( _j ν_j+ _A)^-1≤ _A^-1 and smin=(maxjν^j+ϵA)−1s_ =( _j ν_j+ _A)^-1. Define the learning-rate-integrated time and the preconditioned curvature, response and coordinates T:=∑t=0K−1ηt,A:=S1/2Σ^S1/2⪰0,v~:=S1/2v^,ϕt:=S−1/2δθt.T:= _t=0^K-1 _t, A:=S^1/2\, \,S^1/2 0, v:=S^1/2 v, _t:=S^-1/2δ _t. (70) The operator A is the one written in equation 156 of Appendix E. It is similar to the non-symmetric generator SΣ^=S1/2AS−1/2S =S^1/2AS^-1/2 of the raw dynamics and therefore has the same spectrum, but unlike SΣ^S it is symmetric positive semidefinite, so it has an orthonormal eigenbasis and a spectral measure. Result 11 (Preconditioned training is gradient flow on A). Under Assumption 10, with δθ0=0δ _0=0, the update equation 66 reads, in the coordinates equation 70, ϕt+1=(I−ηtA)ϕt+ηtv~,ϕK=q(K)(A)v~,q(K)(λ):=1−∏t=0K−1(1−ηtλ)λ. _t+1=(I- _tA) _t+ _t v, _K=q^(K)(A)\, v, q^(K)(λ):= 1- _t=0^K-1(1- _tλ)λ. (71) In the continuous-time limit maxtηtλ→0 _t _tλ→ 0 at fixed T, the recursion becomes the gradient flow dϕdT=v~−Aϕ,ϕ(0)=0, dφdT= v-Aφ, φ(0)=0, (72) whose solution is ϕ(T)=qT(A)v~,δθ(T)=S1/2qT(A)S1/2v^=qT(SΣ^)Sv^,qT(λ):=1−e−λTλ.φ(T)=q_T(A)\, v, δθ(T)=S^1/2q_T(A)S^1/2 v=q_T(S )\,S v, q_T(λ):= 1-e^-λ Tλ. (73) If Σ^≻0 0 then δθ(T)→Σ^−1v^δθ(T)→ ^-1 v as T→∞T→∞: the preconditioner leaves the stationary point equation 65 unchanged and acts only on the path, through the spectrum of A. See C.3 for a proof. Remark 1 (Commuting preconditioner and the mode-dependent clock). If S and Σ commute they share an eigenbasis uk\u_k\, Suk=skukSu_k=s_ku_k, Σ^uk=λkuk u_k= _ku_k, and Auk=skλkukAu_k=s_k _ku_k. Then equation 73 reads, mode by mode, δθk(T)=(1−e−λkTk)vkλk,Tk:=skT,vk:=uk⊤v^,δ _k(T)= (1-e^- _kT_k )\, v_k _k, T_k:=s_kT, v_k:=u_k v, (74) and Tk=∑t=0K−1ηtsk,tT_k= _t=0^K-1 _ts_k,t if StS_t varies in time while remaining co-diagonal with Σ : the preconditioner acts as a mode-dependent clock on the spectrum of the raw curvature. This is exact for SGD (S=IS=I) and for preconditioners that are functions of Σ (e.g. natural gradient). Adam’s S is diagonal in the coordinate basis whereas Σ is not, so in general [S,Σ^]≠0[S, ]≠ 0. Remark 2 (Where the poisoning fraction enters the optimizer). Adam’s second moment equation 67 is a statistic of the training gradients, so on the mixture εP_ the frozen preconditioner is Sε=S0+ε∂εSε|ε=0+o(ε)S_ =S_0+ \, _ S_ |_ =0+o( ). Fix the coordinates equation 70 with the clean S0S_0 for both the clean and the poisoned run and write Mε:=S0−1/2SεS0−1/2=I+εΔM+o(ε)M_ :=S_0^-1/2S_ S_0^-1/2=I+ M+o( ). Repeating the computation in the proof of Result 11, the poisoned flow is ϕ˙ε=Mε(S01/2v^ε−S01/2Σ^εS01/2ϕε) φ_ =M_ (S_0^1/2 v_ -S_0^1/2 _ S_0^1/2 _ ), i.e. ϕ˙ε=−Hεϕε+vε, φ_ =-H_ _ +v_ , (75) Hε=A0+ε(ΔMA0+S01/2ΔΣ^S01/2)+o(ε), H_ =A_0+ ( M\,A_0+S_0^1/2 \,S_0^1/2 )+o( ), vε=v~0+ε(ΔMv~0+S01/2Δv^)+o(ε), v_ = v_0+ ( M\, v_0+S_0^1/2 v )+o( ), where A0:=S01/2Σ^0S01/2A_0:=S_0^1/2 _0S_0^1/2, v~0:=S01/2v^0 v_0:=S_0^1/2 v_0. Result 11 says that finite-time preconditioned training is the spectral filter qTq_T applied to A. There are two asymptotic regimes: ϕ(T)=qT(A)v~,qT(λ)=1−e−λTλ=λ−1+o(λ−1),λT≫1,T+O(λT2),λT≪1.φ(T)=q_T(A)\, v, q_T(λ)= 1-e^-λ Tλ= casesλ^-1+o(λ^-1),&λ T 1,\\[3.0pt] T+O(λ T^2),&λ T 1. cases (76) Note that, (1−e−1)minT,1λ≤qT(λ)≤minT,1λ.(1-e^-1) \T, 1λ \≤ q_T(λ)≤ \T, 1λ \. Finite-time training is therefore equivalent, up to multiplicative constants, to putting a sharp cut-off on the inverse-curvature variable X=λ−1X=λ^-1, λ∈spec(A)λ (A), at the truncation level τ=Tτ=T. B.3 Assumption of heavy-tailed spectrum In this subsection, we discuss the crucial ingredient needed for the poisoning scaling law in a large language model: the heavy-tailed inverse-curvature spectrum of the preconditioned clean-data curvature AP:=S1/2Σ^PS1/2A_P:=S^1/2\, _P\,S^1/2 (77) under P, with Σ^P _P the clean Gauss–Newton matrix equation 60 and S the frozen clean preconditioner of Assumption 10. The empirical results of Tang et al. (2025) provide qualitative motivation for a broad hierarchy of curvature scales in trained language models. They find an approximate power law among the leading positive eigenvalues of the parameter Hessian: ∇θ2ℒ^(θ)=1D∑i=1DJi⊤iJi⏟Σ^+1D∑i=1D∑a∂ℓi∂fia∇θ2fia⏟R^model, _θ^2 L(θ)= 1D _i=1^DJ_i W_iJ_i_ + 1D _i=1^D _a ∂ _i∂ f_ia _θ^2f_ia_ R_model, (78) Their power-law fits concern a fixed number of leading eigenvalues above a positive cutoff. Meterez et al. (2026) report the same for the Gauss–Newton matrix. Our assumption, by contrast, concerns the normalized spectral mass arbitrarily close to zero, as stated below. Assumption 12 (Heavy-tailed preconditioned clean-data curvature). Let APA_P be the preconditioned clean-data curvature equation 77, with eigendecomposition APuk,P=λk,Puk,P,k=1,…,P,A_Pu_k,P= _k,Pu_k,P, k=1,…,P, (79) and let μP:=1P∑k=1Pδλk,P _P:= 1P _k=1^P _ _k,P (80) be its empirical spectral measure. Suppose that, as P,D→∞P,D→∞, μP _P converges to a deterministic probability measure μ on [0,∞)[0,∞) whose distribution function has a power-law hard edge, F(x):=μ([0,x])=cμxα(1+o(1)),x↓0,cμ>0,α>0.F(x):=μ([0,x])=c_μ\,x^α\,(1+o(1)), x 0, c_μ>0,\ α>0. (81) Equivalently, for λ∼μλ μ the inverse-curvature variable X:=λ−1X:=λ^-1 is heavy-tailed, ℙλ∼μ(X>τ)=μ([0,τ−1))=cμτ−α(1+o(1)),τ→∞.P_λ μ (X>τ )=μ([0,τ^-1))=c_μτ^-α\,(1+o(1)), τ→∞. (82) This implies that the eigenvalues ordered increasingly from zero obey λ(j),P↑≍(jcμP)1/α.λ _(j),P ( jc_μP )^1/α. (83) This assumption should be distinguished from the descending-rank capacity law measured by Meterez et al. (2026). Their eigenvalues are ordered from largest to smallest, and the reported fit takes the form λi,P↓≍(1i)αcap,αcap≈0.96,P≈1.5×108λ _i,P ( 1i ) _cap, _cap≈ 0.96, P≈ 1.5× 10^8 (84) They study up to λ≥λ1,P/(1.5×106)λ≥ _1,P/(1.5× 10^6), so the maximum rank of the eigenvalue in descending order is i≈(1.5×106)1/0.96≈3×106i≈(1.5× 10^6)^1/0.96≈ 3× 10^6; against i/P≈0.02i/P≈ 0.02, so roughly 98%98\% of the modes-the genuinely weakest, which define α-lie below the resolution floor. In general, α,αcapα, _cap are independent parameters, we explain this with a simple example next. The probability density of the beta-prime law λ∼β′(α,1/αcap)λ β (α,1/ _cap) is ρ(λ)=1λ0B(α, 1/αcap)(λ0)α−1(1+λ0)−(α+1/αcap),λ>0,ρ(λ)= 1 _0\,B(α,\,1/ _cap) ( λ _0 )^α-1 (1+ λ _0 )^-(α+1/ _cap), λ>0, (85) with shape parameters α,αcap>0α, _cap>0, scale λ0>0 _0>0, and B(⋅,⋅)B(·,·) the beta function. Remark 1. The density equation 85 has the following two regimes. (i) Hard edge (λ↓0λ 0). ρ(λ)∼(λ0αB(α,1/αcap))−1λα−1ρ(λ) ( _0^αB(α,1/ _cap) )^-1λ^α-1, hence μ([0,x])≍xαμ([0,x]) x^α, which gives the ascending order statistics λ(j),P↑≍λ0(j/P)1/αλ _(j),P _0\,(j/P)^1/α and λmin(P)≍λ0P−1/α _ (P) _0\,P^-1/α, i.e. Assumption 12 with hard-edge exponent α. (i) Capacity tail (λ→∞λ→∞). ρ(λ)∼(λ0−1/αcap/B(α,1/αcap))λ−1−1/αcapρ(λ) ( _0^-1/ _cap/B(α,1/ _cap) )λ^-1-1/ _cap, hence Pr(λ>t)≍(t/λ0)−1/αcap (λ>t) (t/ _0)^-1/ _cap, which gives the descending order statistics λi,P↓≍λ0i−αcapλ _i,P _0\,i^- _cap, i.e. the capacity law equation 84 with exponent αcap _cap. The two regimes cross over at λ≍λ0λ _0. The hard-edge exponent does not depend on the preconditioner due to the following argument. Read the local quadratic objective as a system of coupled oscillators: 12δθ⊤Σ^Pδθ 12\,δθ _P\,δθ is a potential energy with stiffness matrix Σ^P⪰0 _P 0. In the preconditioned coordinates ϕ=S−1/2δθφ=S^-1/2δθ of Result 11 the same energy is 12ϕ⊤Aϕ 12\,φ Aφ, and Aϕ=λϕAφ=λφ is equivalent, with ψ:=S1/2ϕψ:=S^1/2φ, to the generalized eigenproblem Σ^Pψ=λMψ,M:=S−1. _P\,ψ=λ\,Mψ, M:=S^-1. (86) The eigenvalues of A are therefore the squared normal-mode frequencies of a system with stiffness Σ^P _P and mass matrix M=S−1M=S^-1: the preconditioner is an inverse mass matrix. By equation 69 the mass matrix satisfies smax−1I⪯M⪯smin−1Is_ ^-1I M s_ ^-1I. Comparing M with the two scalar mass matrices smax−1Is_ ^-1I and smin−1Is_ ^-1I therefore gives, rank by rank, sminλk(Σ^P)≤λk(A)≤smaxλk(Σ^P),s_ \, _k( _P)≤ _k(A)≤ s_ \, _k( _P), (87) Every eigenvalue is thus moved by a multiplicative factor lying in [smin,smax][s_ ,s_ ], i.e. its logarithm is shifted by at most lnκ κ, where κ:=smax/sminκ:=s_ /s_ . This implies, FΣ^P(x/smax)≤FA(x)≤FΣ^P(x/smin).F_ _P(x/s_ )\;≤\;F_A(x)\;≤\;F_ _P(x/s_ ). (88) Let the curvature have a hard edge, FΣ(y)≃cyαF_ (y) c\,y^α for small y, and let smin,smaxs_ ,s_ stay fixed as P→∞P→∞. Inserting this into equation 88, csmax−αxα≲FA(x)≲csmin−αxα,x→0.c\,s_ ^-α\,x^α\; \;F_A(x)\; \;c\,s_ ^-α\,x^α, x→ 0. (89) Taking logarithms, lnFA(x)=αlnx+lnc−αlnseff(x) F_A(x)=α x+ c-α s_eff(x) with some seff(x)∈[smin,smax]s_eff(x)∈[s_ ,s_ ]. The last term is bounded, while αlnx→−∞α x→-∞, so the slope on log–log axes is α for A as for Σ : limx→0lnFA(x)lnx=α. _x→ 0 F_A(x) x=α. The exponent is thus universal under bounded preconditioning, while the scaling coefficient, and the eigenvectors are not. A detailed numerical study supporting Assumption 12 for the operator APA_P is provided in Appendix E. In the rest of this subsection we discuss a simple model that produces a hard edge for the raw population curvature ΣP _P, the exponent is inherited by APA_P. Define i:=i1/2Ji∈ℝV×P,Yi:=i⊤i=Ji⊤iJi⪰0.J_i:=W_i^1/2J_i ^V× P, Y_i:=J_i J_i=J_i W_iJ_i 0. (90) The empirical and population matrices are Σ^P:=1D∑i=1DYi,ΣP:=[Yi]. _P:= 1D _i=1^DY_i, _P:=E[Y_i]. (91) Here we present a model for sampling iJ_i under which the heavy-tailed spectrum assumption follows. The goal of this sampling method is to give interpretation to the parameter appearing in the heavy-tailed spectrum. As a motivational toy model, consider the spin-glass-based generative model introduced in Halder et al. (2026) as a proxy model of language. In that framework, the model’s generations are drawn from a measure that decomposes into a hierarchy of clusters organized in a tree, where at each level the cluster weights follow a Poisson–Dirichlet law PD(m)PD(m). An interesting observation is that there is a power law structure of the cluster weights in this setup. Let (Wi)i≥1(W_i)_i≥ 1 denote the ranked probabilities of PD(m)PD(m), where 0<m<10<m<1. Then Wi=(Ξm(1−m))1/mi−1/ma.s. as i→∞,W_i= ( m(1-m) )^1/mi^-1/m .s. as i→∞, where Ξ>0 >0 is an almost-surely finite random variable. This motivates us to propose the following setup for the latent space distribution of the language model we are discussing here. Let Ii∈ℕI_i be the latent cluster index, in a single draw we choose it with probability ℙ(Ii=ℓ)=πℓ,πℓ=Cπℓ−1/m(1+o(1)),0<m<1.P(I_i= )= _ , _ =C_π ^-1/m(1+o(1)), 0<m<1. (92) Here, the order of magnitude estimate is associated with cluster the index l→∞l→∞. For each P, let ℝP=⨁ℓ≥1ℓ,PR^P= _ ≥ 1U_ ,P be an orthogonal decomposition of the parameter space. Let dℓ,P:=dim(ℓ,P)d_ ,P:= (U_ ,P), and choose a matrix Uℓ,P∈ℝP×dℓ,P,Uℓ,P⊤Uℓ,P=Idℓ,P,U_ ,P ^P× d_ ,P, U_ ,P U_ ,P=I_d_ ,P, (93) whose columns form an orthonormal basis of ℓ,PU_ ,P. For instance, the orthogonal projector onto this subspace is Πℓ,P:=Uℓ,PUℓ,P⊤. _ ,P:=U_ ,PU_ ,P . Conditional on Ii=ℓI_i= , draw a Gaussian matrix Zi(ℓ)∈ℝV×dℓ,P,(Zi(ℓ))ab∼iid(0,1V),Z_i^( ) ^V× d_ ,P, (Z_i^( ) )_ab iid N\! (0, 1V ), (94) Now we define the method of sampling the Jacobian i|(Ii=ℓ):=hℓZi(ℓ)Uℓ,P⊤hℓ=Chℓ2r(1+o(1)).J_i (I_i= ):= h_ \,Z_i^( )U_ ,P h_ =C_h ^2r(1+o(1)). (95) The next theorem shows that under this sampling, the eigen-space of ΣP _P has a natural ‘cluster’ structure in terms of the parameter subspaces. Result 13. Every direction in ℓ,PU_ ,P is associated to an eigenvalue of ΣP _P according to λℓ:=πℓhℓ=Cλℓ−δ(1+o(1)),Cλ:=CπCh,δ:=1m−2r. _ := _ h_ =C_λ ^-δ(1+o(1)), C_λ:=C_πC_h, δ:= 1m-2r. (96) A proof is given in Appendix C.4. The laws of πℓ _ and hℓh_ determine the eigenvalue associated with cluster ℓ , but they do not determine how many parameter directions are assigned to that cluster. We specify this multiplicity next. Say dℓ,P/P→ρℓd_ ,P/P→ _ as P→∞P→∞. We want to set up ρl _l such that common semantic families/clusters receive large subspaces, while increasingly rare families/clusters receive subspace determined by the cluster scale. For this it will be useful to define the cumulative fraction of parameter directions devoted to clusters of rank at least ℓ by Rcap(ℓ):=∑j≥ℓρj.R_ cap( ):= _j≥ _j. The eigenvalue scale of cluster ℓ is λℓ≍ℓ−δ _ ^-δ, so its inverse-curvature scale is λℓ−1≍ℓδ. _ ^-1 ^δ. A natural capacity-matching hypothesis is that the fraction of representation RcapR_ cap devoted to semantic families weaker than λℓ _ is proportional to their remaining aggregate curvature signal BY(τ):=[∥Yi∥op∥Yi∥op>τ]B_Y(τ):=E\! [\|Y_i\|_op1\\|Y_i\|_op>τ\ ]: Rcap(ℓ)≍BY(λℓ−1)≍ℓ−δ(q−1)R_ cap( ) B_Y( _ ^-1) ^-δ(q-1) (97) The last equality followed due to the following result: Result 14. The first-moment signal remaining beyond a threshold τ obeys BY(τ):=[∥Yi∥op∥Yi∥op>τ]≍τ−(q−1),q:=1/m−12r.B_Y(τ):=E\! [\|Y_i\|_op1\\|Y_i\|_op>τ\ ] τ^-(q-1), q:= 1/m-12r. (98) The order estimate equation 98 follows directly from the conditional law of YiY_i constructed in equation 94–equation 95; a proof is given in Appendix C.5. Let μP:=1P∑k=1Pδλk(Σ) _P:= 1P _k=1^P _ _k( ) be the empirical spectral measure. Then as P→∞P→∞, μP⇒μ _P μ. This can be easily seen as follows. By equation 96, the eigenvalue associated with cluster rank ℓ is asymptotic to Cλℓ−δC_λ ^-δ. Therefore λℓ≤x↔ℓ≳(Cλ/x)1/δ. _ ≤ x (C_λ/x )^1/δ. Using the limiting subspace proportions μ([0,x])∼∑ℓ≥(Cλ/x)1/δρℓ≍(1x)−(q−1)≍xq−1. μ([0,x]) _ ≥(C_λ/x)^1/δ _ ( 1x )^-(q-1) x^q-1. (99) Hence, this cluster model leads to a hard edge with exponent α=q−1α=q-1 for the raw population curvature ΣP _P. By earlier discussion, the preconditioned operator A=S1/2Σ^PS1/2A=S^1/2 _PS^1/2 inherits the same exponent whenever the preconditioner has a condition number bounded in P (Assumption 10), which is the form in which Assumption 12 is used. B.4 The clean data truncation error and assumption on its tail The goal of this subsection is to quantify how much of the clean target remains unlearned after finite-time training. This subsection concerns clean data P. We show that the clean truncation error decays as a power law in training time. We start from the clean linearized dynamics that follow from equation 66 via Result 11: dTϕ0(T)=−H0(ϕ0(T)−ϕ0in)+v0,ϕ0(0)=ϕ0in, ddT _0(T)=-H_0 ( _0(T)-φ^in_0 )+v_0, _0(0)=φ^in_0, (100) where H0=S1/2Σ^PS1/2H_0=S^1/2 _PS^1/2 is the preconditioned clean curvature and v0=S1/2v^Pv_0=S^1/2 v_P the preconditioned negative clean training gradient at ϕ0inφ^in_0 (Convention above). The stationary (infinite-time) solution is the clean training optimum ϕ0⋆ _0 , H0(ϕ0⋆−ϕ0in)=v0,i.e.ϕ0⋆=ϕ0in+H0−1v0.H_0 ( _0 -φ^in_0 )=v_0, .e. _0 =φ^in_0+H_0^-1v_0. (101) The finite-time solution relaxes from the initialization toward this optimum, ϕ0(T)=ϕ0in+qT(H0)v0=ϕ0⋆−e−H0T(ϕ0⋆−ϕ0in),qT(λ):=1−e−λTλ, _0(T)=φ^in_0+q_T(H_0)v_0= _0 -e^-H_0T ( _0 -φ^in_0 ), q_T(λ):= 1-e^-λ Tλ, (102) so the displacement from initialization, ϕ0(T)−ϕ0in=qT(H0)v0 _0(T)-φ^in_0=q_T(H_0)v_0, is exactly the finite-time filtered response. If H0uk=λkukH_0u_k= _ku_k and the initialization-to-optimum gap is expanded as ϕ0⋆−ϕ0in=∑kakuk _0 -φ^in_0= _ka_ku_k (so ak:=uk⊤(ϕ0⋆−ϕ0in)a_k:=u_k ( _0 -φ^in_0)), then the clean data truncation error is e0(T):=ϕ0(T)−ϕ0⋆=−e−H0T(ϕ0⋆−ϕ0in)=−∑k=1Pe−λkTakuk.e_0(T):= _0(T)- _0 =-e^-H_0T ( _0 -φ^in_0 )=- _k=1^Pe^- _kTa_ku_k. (103) Physically, e0(T)e_0(T) is the part of the clean target that finite-time training has not yet learned: each mode is suppressed by e−λkTe^- _kT, so the well-conditioned (large-λk _k) directions are captured quickly, while the flat, weak-curvature (small-λk _k) directions remain underlearned. Next, we quantify the size of this residual. The observable we care about is clean data validation loss, whose local geometry is the validation Hessian G, not the training curvature H0H_0. In the ample-data limit the clean Gauss–Newton matrix is close to the validation Hessian, Σ^P≈∇2ℛ(ϕ0⋆)=Z∼[JZ⊤WZJZ] _P≈∇^2R_P( _0 )=E_Z [J_Z W_ZJ_Z], and the same congruence transform applied to both sides gives, in the preconditioned frame, H0=S1/2Σ^PS1/2≈G=S1/2∇2ℛ(ϕ0⋆)S1/2.H_0=S^1/2 _PS^1/2≈ G=S^1/2\,∇^2R_P( _0 )\,S^1/2. The next assumption makes this precise: in the weak-curvature tail, G is approximately diagonal in the training-curvature eigenbasis, and the validation weight of a mode scales as a power of its curvature. Assumption 15 (Validation metric in clean curvature eigen-basis). In the weak-curvature tail, G is asymptotically diagonal in the H0H_0-eigenbasis and that uk⊤Guk∼CGλkζ,λk↓0,CG>0,ζ∈ℝ.u_k Gu_k C_G _k^ζ, _k 0, C_G>0, ζ . (104) The finite-time error lives in the weak-curvature modes, but whether those modes contribute at all depends on how much of the clean learning target is actually supported on them, since we are studying the dynamics near it. The following assumption rules this out: the initialization-to-optimum gap carries a nontrivial, power-law amount of its mass in the weak-curvature modes. The exponent η is neutral at η=0η=0 (no extra suppression or enhancement in weak modes), positive when the target is smoother and avoids weak-curvature directions, and negative when it is unusually concentrated in rare weak-curvature directions. Assumption 16 (Clean learning-target tail mass). The initialization-to-optimum gap ϕ0⋆−ϕ0in _0 -φ^in_0—the part of the clean target still to be learned starting from ϕ0inφ^in_0—has a nontrivial weight in the weak-curvature modes, with |ak|2‖ϕ0⋆−ϕ0in‖22∼Caλkη,λk↓0,Ca>0,ak=uk⊤(ϕ0⋆−ϕ0in). |a_k|^2\| _0 -φ^in_0\|_2^2 C_a _k^η, _k 0, C_a>0, a_k=u_k ( _0 -φ^in_0 ). (105) Training longer learns more of the weak-curvature tail, so the clean truncation error should shrink as T grows. The result below shows that this decay is a clean power law T−βT^-β, with an exponent that simply combines the three ingredients just introduced: how many weak modes there are (the spectral hard-edge exponent α), how the clean solution is spread across them (η), and how the validation metric weights them (ζ). This B(T)B(T) is the clean data truncation bias that will later be balanced against the poison influence. Result 17 (Clean truncation-error falloff). Suppose Assumptions 12, 15 and 16 hold together. Then, the clean truncation error measured in the validation metric, B(T)2:=e0(T)⊤Ge0(T)B(T)^2:=e_0(T) G\,e_0(T), obeys B(T)=CBT−β(1+o(1)),β:=12(α+η+ζ),CB:=‖ϕ0⋆−ϕ0in‖2(CaCGcμα2−2βΓ(2β))1/2.B(T)=C_BT^-β(1+o(1)), β:= 12(α+η+ζ), C_B:=\| _0 -φ^in_0\|_2 (C_aC_Gc_μα 2^-2β (2β) )^1/2. (106) See Appendix C.6 for a proof. B.5 Poisoning strength after truncation and assumption on its tail The goal of this subsection is to quantify how strongly poisoning displaces the solution as training proceeds. This subsection involves both data types: the clean curvature and response (H0,v0)(H_0,v_0) under P and their poisoned counterparts (Hε,vε)(H_ ,v_ ) under the mixture ε=(1−ε)+εP_ =(1- )P+ . We show that the poison influence grows as a power law in training time. Consider the clean and contaminated linearized dynamics, each launched from its own initialization: ϕ˙0(T) φ_0(T) =−H0(ϕ0(T)−ϕ0in)+v0, =-H_0 ( _0(T)-φ^in_0 )+v_0, ϕ0(0) _0(0) =ϕ0in, =φ^in_0, (107) ϕ˙ε(T) φ_ (T) =−Hε(ϕε(T)−ϕεin)+vε, =-H_ ( _ (T)-φ^in_ )+v_ , ϕε(0) _ (0) =ϕεin, =φ^in_ , (108) with contamination Hε=H0+εΔH,vε=v0+εΔv.H_ =H_0+ H, v_ =v_0+ v. (109) Both dynamics are written in the preconditioned frame fixed by the clean preconditioner S0S_0; by Remark 2, the O(ε)O( ) response of Adam’s second-moment statistics to the poison is part of ΔH H and Δv v (Eq. equation 75), and ΔH H need not be symmetric—nothing below uses its symmetry. Here ϕ0inφ^in_0 and ϕεinφ^in_ are the initializations of the clean and poisoned runs. Crucially, the local expansion is legitimate only late in training, where per-step updates are small and the trajectory stays inside a neighborhood of the checkpoint; T measures the time elapsed from that late-time checkpoint. Because the checkpoint is itself the product of training, the clean and the poisoned checkpoints do not coincide: the poison accumulated up to the checkpoint displaces ϕεinφ^in_ from ϕ0inφ^in_0 at first order in the poison rate. We therefore keep the O(ε)O( ) offset, ϕεin−ϕ0in=εψin+o(ε),ψin:=∂εϕεin|ε=0,φ^in_ -φ^in_0= \,ψ^in+o( ), ψ^in:= _ φ^in_ |_ =0, (110) with ψinψ^in generically nonzero—it is the accumulated poison displacement already carried by the checkpoint. The derivative of the contaminated trajectory at ε=0 =0 is ψ(T):=∂ϕε(T)∂ε|ε=0.ψ(T):= . ∂ _ (T)∂ |_ =0. (111) Physically, ψ(T)ψ(T) is the first-order direction in which contamination drags the learned parameters as training proceeds: to leading order the poisoned and clean trajectories separate by εψ(T) ψ(T), i.e., ϕε(T)−ϕ0(T)=εψ(T)+o(ε) _ (T)- _0(T)= ψ(T)+o( ), and at the checkpoint this separation is exactly the offset, ψ(0)=ψinψ(0)=ψ^in. Differentiating equation 108 at ε=0 =0 (both dynamics share the fixed linearization center ϕ0inφ^in_0; only the starting point moves, by equation 110) yields the variational equation ψ˙(T) ψ(T) =−H0ψ(T)+Δv−ΔH(ϕ0(T)−ϕ0in), =-H_0ψ(T)+ v- H\, ( _0(T)-φ^in_0 ), ψ(0) ψ(0) =ψin, =ψ^in, (112) in which the poison forces the clean displacement-from-initialization ϕ0(T)−ϕ0in _0(T)-φ^in_0 while the trajectory starts from the offset ψinψ^in. This is solved by a homogeneous transient plus a forced part, ψ(T)=e−H0Tψin+ψf(T),ψf(T):=∫0Te−H0(T−s)(Δv−ΔH(ϕ0(s)−ϕ0in))s,ψ(T)=e^-H_0Tψ^in+ _ f(T), _ f(T):= _0^Te^-H_0(T-s) ( v- H\, ( _0(s)-φ^in_0 ) )\,ds, (113) with ϕ0(s)−ϕ0in=(I−e−H0s)(ϕ0⋆−ϕ0in) _0(s)-φ^in_0=(I-e^-H_0s)( _0 -φ^in_0). The first term is the checkpoint offset, carried forward and filtered by e−H0Te^-H_0T exactly as the clean residual e0(T)e_0(T) is; the second is the forced poison response. We can rewrite the forced part as ψf(T)=qT(H0)feff(T), _ f(T)=q_T(H_0)f_ eff(T), (114) uk⊤feff(T)=uk⊤fp+rT(λk)∑j=1P(uk⊤ΔHuj)(uj⊤(ϕ0⋆−ϕ0in))KT(λk,λj). u_k f_ eff(T)=u_k f_ p+r_T( _k) _j=1^P(u_k Hu_j) (u_j ( _0 -φ^in_0) )K_T( _k, _j). with fp:=Δv−ΔH(ϕ0⋆−ϕ0in)f_ p:= v- H ( _0 -φ^in_0 ). The two-eigenvalue kernels that are used to present expression of feff(T)f_ eff(T) are as follows KT(λ,μ)=e−μT−e−λTλ−μ,λ≠μ,Te−λT,λ=μ.K_T(λ,μ)= cases e^-μ T-e^-λ Tλ-μ,&λ≠μ,\\[8.0pt] Te^-λ T,&λ=μ. cases (115) and rT(λk):=λk1−e−λkT,λk>0,1T,λk=0.r_T( _k):= cases _k1-e^- _kT,& _k>0,\\[10.0pt] 1T,& _k=0. cases (116) The size of the forced poison-induced displacement, measured in the metric that governs clean validation loss, is I(T)2=ψf(T)⊤Gψf(T)I(T)^2= _ f(T) G\, _ f(T) (the checkpoint-offset transient is treated separately below). Just as the finite-time filter qTq_T shapes the clean error, it shapes the poison response, so I(T)2I(T)^2 can be written as an integral of qT(λ)2q_T(λ)^2 against a measure that records how the poison distributes its influence across curvature scales. The assumption below specifies the behavior of this measure near weak curvature modes: the poison deposits a power-law amount of influence. Assumption 18 (Effective poison spectral tail). There exists a poison measure dνp(T)(λ)d _p^(T)(λ) such that I(T)2:=ψf(T)⊤Gψf(T)=∫0∞qT(λ)2dνp(T)(λ).I(T)^2:= _ f(T) G _ f(T)= _0^∞q_T(λ)^2\,d _p^(T)(λ). (117) Assume that, dνp(T)(λ)∼Cνλ1−2γdλ,λ↓0,T↑∞,0<γ<1.d _p^(T)(λ) C_νλ^1-2γ\,dλ, λ 0, T ∞, 0<γ<1. (118) The clean truncation bias B(T)B(T) decays as training proceeds, but the poison influence works in the opposite direction: the longer the model trains, the more the weak-curvature directions fall under the poison forcing. The result below quantifies this growth, showing that the poison-induced displacement, measured in the validation metric, grows as a clean power law TγT^γ, with the growth exponent set by the effective poison spectral tail. It is the opposing partner of the clean-bias falloff, and balancing the two is what later fixes the scaling law. Result 19 (Poison-strength growth). Under Assumption 18, for large T, I(T)=CITγ(1+o(1)),CI:=CνJγ,I(T)=C_IT^γ(1+o(1)), C_I:= C_νJ_γ, (119) where Jγ:=∫0∞(1−e−z)2z−1−2γz∈(0,∞),0<γ<1.J_γ:= _0^∞(1-e^-z)^2z^-1-2γ\,dz∈(0,∞), 0<γ<1. (120) See Appendix C.7 for a proof. The checkpoint offset ψinψ^in contributes the transient e−H0Tψine^-H_0Tψ^in to the displacement equation 113. Measured in the validation metric it has exactly the structure of the clean residual e0(T)e_0(T) of Appendix B.4 - a fixed vector filtered by e−H0Te^-H_0T - so, under a weak-curvature tail on ψinψ^in analogous to Assumption 16, its size is a clean power law, Bin(T):=‖e−H0Tψin‖G=CinT−βin(1+o(1)),βin=12(α+ηin+ζ),B_in(T):= \|e^-H_0Tψ^in \|_G=C_in\,T^- _in(1+o(1)), _in= 12(α+ _in+ζ), (121) where ηin _in is the tail exponent of ψinψ^in across the H0H_0-eigenbasis. Thus the offset decays with training: continued training partially washes out the poison damage already present in the checkpoint, while the forced response ψf(T) _ f(T) builds up. B.6 The scaling law from optimal truncation The goal of this subsection is to combine the decaying clean truncation bias and the growing poison influence into the observed scaling law. For a clean token-prediction event Z=(X,Y)Z=(X,Y), let fϕ(X)∈ℝVf_φ(X) ^V be the logits and define ℓZ(ϕ):=−log(softmax(fϕ(X))Y). _Z(φ):=- \! (softmax(f_φ(X))_Y ). (122) At ϕ0⋆ _0 , put pZ:=softmax(fϕ0⋆(X)),JZ:=∂fϕ(X)∂ϕ|ϕ=ϕ0⋆.p_Z:=softmax(f_ _0 (X)), J_Z:= . ∂ f_φ(X)∂φ |_φ= _0 . (123) The gradient and Hessian with respect to the logits are ∇fℓZ=pZ−eY,WZ:=∇f2ℓZ=diag(pZ)−pZpZ⊤. _f _Z=p_Z-e_Y, W_Z:= _f^2 _Z=diag(p_Z)-p_Zp_Z . (124) Consequently, the clean validation loss is ℛ(ϕ):=Z∼[ℓZ(ϕ)].R_P(φ):=E_Z [ _Z(φ)]. (125) The clean solution ϕ0⋆ _0 is the stationary point of the training dynamics (H0(ϕ0⋆−ϕ0in)=v0H_0( _0 -φ^in_0)=v_0, Eq. equation 101); it need not minimize the validation risk ℛR_P, and in a finite-data run it generically does not. We therefore keep the validation gradient at the clean solution, g⋆:=∇ℛ(ϕ0⋆)∈ℝP,g_ := _P( _0 ) ^P, (126) Taylor’s expansion, keeping the linear term, gives ℛ(ϕ0⋆+h)=ℛ(ϕ0⋆)+g⋆⊤h+12h⊤Gh+o(‖h‖22).R_P( _0 +h)=R_P( _0 )+g_ h+ 12h Gh+o(\|h\|_2^2). (127) At a matched checkpoint ϕε(T)−ϕ0(T)=εψ(T)+o(ε) _ (T)- _0(T)= ψ(T)+o( ) and ϕ0(T)−ϕ0⋆=e0(T) _0(T)- _0 =e_0(T), with the displacement split into checkpoint offset and forced response, ψ(T)=e−H0Tψin+ψf(T)ψ(T)=e^-H_0Tψ^in+ _ f(T) (Eq. equation 113). Applying equation 127 with h=e0(T)+εψ(T)h=e_0(T)+ ψ(T) and with h=e0(T)h=e_0(T) and subtracting, the constant and poison-independent terms (ℛ(ϕ0⋆)R_P( _0 ), g⋆⊤e0g_ e_0, 12e0⊤Ge0 12e_0 Ge_0) cancel, leaving δℛ(ε,T) ( ,T) :=ℛ(ϕε(T))−ℛ(ϕ0(T))=εg⋆⊤ψ(T)+εe0(T)⊤Gψ(T)+12ε2ψ(T)⊤Gψ(T) :=R_P( _ (T))-R_P( _0(T))= \,g_ ψ(T)+ \,e_0(T) G\,ψ(T)+ 12 ^2\,ψ(T) G\,ψ(T) =ερgi(T)Γ⋆Bin(T)⏟checkpoint offset+ερg(T)Γ⋆I(T)⏟train–eval gap+ερG(T)B(T)I(T)⏟clean bias+12ε2I(T)2⏟pure poison, = \, _gi(T)\, _ B_in(T)_checkpoint offset+ \, _g(T)\, _ I(T)_train--eval gap+ \, _G(T)\,B(T)I(T)_clean bias+ 12 ^2I(T)^2_pure poison, (128) where the correlations are Γ⋆:=‖g⋆‖G−1,ρg(T):=g⋆⊤ψf(T)Γ⋆I(T),ρgi(T):=g⋆⊤e−H0TψinΓ⋆Bin(T),ρG(T):=e0⊤GψfB(T)I(T), _ :=\|g_ \|_G^-1, _g(T):= g_ _ f(T) _ I(T), _gi(T):= g_ e^-H_0Tψ^in _ B_in(T), _G(T):= e_0 G _ fB(T)I(T), (129) all in [−1,1][-1,1] by Cauchy–Schwarz in the G-metric and taken asymptotically constant, ρG→ρ⋆ _G→ _ , ρg→ρg⋆ _g→ _g , ρgi→ρgi⋆ _gi→ _gi . The train–eval gap term is the decisive contribution at late times. Using B(T)=CBT−βB(T)=C_BT^-β (Result 17), I(T)=CITγI(T)=C_IT^γ (Result 19) and Bin(T)=CinT−βinB_in(T)=C_inT^- _in (Eq. equation 121), the four coefficients of equation 128 scale as ερgi⋆Γ⋆CinT−βin⏟checkpoint: ε,decays,ερg⋆Γ⋆CITγ⏟gap: ε,grows,ερ⋆CBCIT−(β−γ)⏟bias: ε,decays,12ε2CI2T2γ⏟pure: ε2,grows. \, _gi _ C_in\,T^- _in_checkpoint: ,\ decays, \, _g _ C_I\,T^γ_gap: ,\ grows, \, _ C_BC_I\,T^-(β-γ)_bias: ,\ decays, 12 ^2C_I^2\,T^2γ_pure: ^2,\ grows. (130) Here we assumed β>γ>0,βin>0β>γ>0, _in>0. Result 20 (Gap-dominated crossover). Suppose among ε order terms the gap term dominates. Then the excess risk is, δℛ(ε,T)=ρg⋆Γ⋆I(T)ε+12I(T)2ε2(1+o(1)), ( ,T)= _g _ I(T)\, + 12I(T)^2\, ^2\,(1+o(1)), (131) and equating them, ερg⋆Γ⋆I(T)=12ε2I(T)2 \, _g _ I(T)= 12 ^2I(T)^2, defines the crossover time Tε(g)=(2ρg⋆Γ⋆CIε)1/γ=tgε−1/γ,tg:=(2ρg⋆Γ⋆CI)1/γ.T^(g)_ = ( 2 _g _ C_I\, )^1/γ=t_g\, ^-1/γ, t_g:= ( 2 _g _ C_I )^1/γ. (132) The excess risk then obeys the integer law δℛ(ε,T)≍Γ⋆εTγ,T≪Tε(g)(a=1),ε2T2γ,T≫Tε(g)(a=2), ( ,T) cases _ \, \,T^γ,&T T^(g)_ \ \ (a=1),\\[3.0pt] ^2\,T^2γ,&T T^(g)_ \ \ (a=2), cases (133) More generally, say, x:=T/Tε(g)x:=T/T^(g)_ , then, δℛ(ε,T)=Ψ(x)(1+o(1)),Ψ(x)=ρg⋆Γ⋆CItgγxγ+12CI2tg2γx2γ=2(ρg⋆Γ⋆)2(xγ+x2γ), ( ,T)= (x)\,(1+o(1)), (x)= _g _ C_It_g^γ\,x^γ+ 12C_I^2t_g^2γ\,x^2γ=2( _g _ )^2 (x^γ+x^2γ ), (134) which is ε -independent at T=Tε(g)T=T^(g)_ (a=0a=0). Since Ψ(x)∼xγ (x) x^γ (x→0x→ 0) and ∼x2γ x^2γ (x→∞x→∞) with both exponents positive, Ψ is monotone increasing no interior minimum. Small LLMs fall under the gap-dominated regime, early in training a=1a=1 and then it decreases towards a=0a=0 as we approach the crossover time. Whereas, large LLMs lie in the gap or the checkpoint-ominated regime. Both these terms are mathematically similar because their conceptual origin is the heavy-tailed inverse curvature spectrum. Hence, we discuss only one of them (the checkpoint dominated results are obtained from the bias-dominated scaling exponent using β↔βin+γβ _in+γ). Result 21 (Bias-dominated crossover). Suppose among ε order terms the bias term dominates. Write x:=T/Tεx:=T/T_ with Tε=tε−1/(β+γ),t:=(2ρ⋆CBCI)1/(β+γ).T_ =t\, ^-1/(β+γ), t:= ( 2 _ C_BC_I )^1/(β+γ). (135) Then the excess risk admits the exact scaling form δℛ(ε,T)=εaΦ(x)(1+o(1)),Φ(x)=ρ⋆CBCItγ−βxγ−β+12CI2t2γx2γ,a=2β+γ, ( ,T)= ^a\, (x)\,(1+o(1)), (x)= _ C_BC_I\,t^γ-β\,x^γ-β+ 12C_I^2\,t^2γ\,x^2γ, a= 2β+γ, (136) Its limits are given as, Φ(x)≍xγ−β(x→0),Φ(x)≍x2γ(x→∞), (x) x^γ-β\ (x→ 0), (x) x^2γ\ (x→∞), (137) and Φ attains a unique interior minimum for β>γβ>γ at x⋆=((β−γ)ρ⋆CBγCItβ+γ)1/(β+γ)=(β−γ2γ)1/(β+γ)=ϕ(1).x_ = ( (β-γ)\, _ C_Bγ\,C_I\,t^β+γ )^1/(β+γ)= ( β-γ2γ )^1/(β+γ)=φ(1). (138) Consequently the two regimes meet, in order, at the crossover T≍TεT T_ , where δℛ(ε,T)≍εa,minTδℛ(ε,T)=Φ(x⋆)εa(1+o(1)). ( ,T)\; \; ^a, _T ( ,T)= (x_ )\, ^a\,(1+o(1)). (139) See Appendix C.8 for a proof. Therefore, for large LLM, scaling exponent a starts near 11 and then increases as training progresses towards the crossover time. Remark 3 (Perplexity observable). For the empirical perplexity ratio Δ(ε,T)=exp(δℛ(ε,T))−1 ( ,T)= ( ( ,T))-1, the same three statements hold verbatim in the small-degradation regime δℛ→0 → 0, since Δ=δℛ(1+o(1)) = \,(1+o(1)). The exponent a so far is expressed through the abstract tail exponents α,β,γ,ζ,ηα,β,γ,ζ,η. The latent-cluster model lets us trade these for two interpretable quantities: the cluster-tail parameter m (roughly, this controls the distribution of ideas) and the within-cluster spread r (controls how precise an idea is). a=2−4γα+η+ζ+2γ,α=1/m−12r−1,0<γ<1.a=2- 4γα+η+ζ+2γ, α= 1/m-12r-1, 0<γ<1. (140) Making the neutral modeling choices—no extra weak-mode alignment (η=0η=0), a neutral poison spectral density (γ=1/2γ=1/2), and the ample-data validation matching (ζ=1ζ=1)—collapses the general formula to a closed form in m and r, which is what makes the observed slope/intercept trends interpretable. a=2−22+α=2−21+1−m2rm.a=2- 22+α=2- 21+ 1-m2rm. Appendix C Proofs for the scaling-law mechanism C.1 Proof of Result 1 Proof. Define F(θ,ε):=∇θLε(θ).F(θ, ):= _θL_ (θ). The map F is real analytic near (θ0,0)( _0,0), F(θ0,0)=0F( _0,0)=0, and DθF(θ0,0)=∇2L(θ0)=H.D_θF( _0,0)=∇^2L_P( _0)=H. Because H is invertible, the analytic implicit-function theorem gives a unique analytic map ε↦θε _ near zero such that F(θε,ε)=0F( _ , )=0 and θε=0=θ0 _ =0= _0. We construct this function below. Differentiate F(θε,ε)=0F( _ , )=0 at ε=0 =0: Hθ0′+∂εF(θ0,0)=0.H\,θ _0+ _ F( _0,0)=0. By definition, ∂εF(θ0,0)=g _ F( _0,0)=g, and therefore θ0′=−H−1g.θ _0=-H^-1g. Analyticity of the branch now gives θε−θ0=−εH−1g+O(ε2), _ - _0=- H^-1g+O( ^2), which proves equation 4. Taylor expansion of ℛR at θ0 _0 gives, with δε:=θε−θ0 _ := _ - _0, ℛ(θ0+δε)−ℛ(θ0)=∇ℛ(θ0)⊤δε⏟=0+12δε⊤Gδε+O(‖δε‖3).R( _0+ _ )-R( _0)= ( _0) _ _=0+ 12 _ G _ +O(\| _ \|^3). Substituting δε=−εH−1g+O(ε2) _ =- H^-1g+O( ^2) yields ℛ(θε)−ℛ(θ0)=ε22g⊤H−1GH−1g+O(ε3),R( _ )-R( _0)= ^22g H^-1GH^-1g+O( ^3), which is equation 5. The coefficient is strictly positive under the stated nondegeneracy condition. Since both θε _ and ℛR are analytic, their composition is analytic; hence a noninteger leading power is impossible under these assumptions. Finally, ex−1=x+O(x2)e^x-1=x+O(x^2) proves the perplexity statement. ∎ C.2 Proof of Result 2 Proof. For any ε and τ, mε,τ−μ= m_ ,τ-μ= [Tτ(ψ)]−μ _P[T_τ(ψ)]-μ +ε(ε[Tτ(ψ)]−[Tτ(ψ)]). + (E_Q_ [T_τ(ψ)]-E_P[T_τ(ψ)] ). The first line is the clean truncation bias and the second is the contamination effect. Substitute equation 6 and equation 7 and choose τ=τε=tε−1/(β+γ)τ= _ =t ^-1/(β+γ). Then τε−β=t−βεβ/(β+γ),ετεγ=tγεβ/(β+γ). _ ^-β=t^-β ^β/(β+γ), _ ^γ=t^γ ^β/(β+γ). Both contributions therefore have the same order, and their vector coefficient is ht=t−βb+tγch_t=t^-βb+t^γc. This proves equation 8. Since Φ is C2C^2, θε,τε−θ0=J(mε,τε−μ)+O(‖mε,τε−μ‖2). _ , _ - _0=J(m_ , _ -μ)+O(\|m_ , _ -μ\|^2). Using equation 8, θε,τε−θ0=εβ/(β+γ)Jht+o(εβ/(β+γ)). _ , _ - _0= ^β/(β+γ)Jh_t+o\! ( ^β/(β+γ) ). The Taylor expansion of ℛR at θ0 _0, gives ℛ(θε,τε)−ℛ(θ0) ( _ , _ )-R( _0) =12(θε,τε−θ0)⊤G(θε,τε−θ0)+o(‖θε,τε−θ0‖2) = 12( _ , _ - _0) G( _ , _ - _0)+o(\| _ , _ - _0\|^2) =12ht⊤J⊤GJhtε2β/(β+γ)+o(ε2β/(β+γ)). = 12h_t J GJh_t\, ^2β/(β+γ)+o\! ( ^2β/(β+γ) ). This proves equation 9 and completes the proof. ∎ C.3 Proof of Result 11 Proof. Multiplying equation 66 by S−1/2S^-1/2 and inserting δθt=S1/2ϕtδ _t=S^1/2 _t, ϕt+1=ϕt+ηtS−1/2S(v^−Σ^S1/2ϕt)=ϕt+ηt(v~−Aϕt). _t+1= _t+ _tS^-1/2S ( v- S^1/2 _t )= _t+ _t ( v-A _t ). Let Au=λuAu=λ u with ‖u‖=1\|u\|=1 and xt:=u⊤ϕtx_t:=u _t, v~u:=u⊤v~ v_u:=u v. Then xt+1=(1−ηtλ)xt+ηtv~ux_t+1=(1- _tλ)x_t+ _t v_u. For λ>0λ>0 this has the fixed point v~u/λ v_u/λ and xt+1−v~u/λ=(1−ηtλ)(xt−v~u/λ)x_t+1- v_u/λ=(1- _tλ)(x_t- v_u/λ), so from x0=0x_0=0 we get xK=q(K)(λ)v~ux_K=q^(K)(λ) v_u; for λ=0λ=0 both sides equal Tv~uT v_u. Since A is symmetric its eigenvectors span ℝPR^P, which gives equation 71. If maxtηtλ<1 _t _tλ<1 then log∏t(1−ηtλ)=−λT−O(λ2∑tηt2) _t(1- _tλ)=-λ T-O(λ^2 _t _t^2) and ∑tηt2≤Tmaxtηt _t _t^2≤ T _t _t, so q(K)(λ)→qT(λ)q^(K)(λ)→ q_T(λ) in the stated limit; qT(A)v~q_T(A) v is the solution of equation 72 by direct differentiation. For the raw coordinates, S1/2AnS−1/2=(SΣ^)nS^1/2A^nS^-1/2=(S )^n for every n≥0n≥ 0, hence S1/2f(A)S−1/2=f(SΣ^)S^1/2f(A)S^-1/2=f(S ) for every polynomial f and, for f=qTf=q_T; thus δθ(T)=S1/2ϕ(T)=S1/2qT(A)S1/2v^=qT(SΣ^)Sv^δθ(T)=S^1/2φ(T)=S^1/2q_T(A)S^1/2 v=q_T(S )S v. Finally, if Σ^≻0 0 then A≻0A 0, qT(A)→A−1q_T(A)→ A^-1 as T→∞T→∞, and S1/2A−1S1/2=S1/2S−1/2Σ^−1S−1/2S1/2=Σ^−1S^1/2A^-1S^1/2=S^1/2S^-1/2 ^-1S^-1/2S^1/2= ^-1. ∎ C.4 Proof of Result 13 Proof. The row vectors of iJ_i are supported on the proper parameter subspace ℓ,PU_ ,P rather than on all of ℝPR^P. From an LLM perspective, ℓ,PU_ ,P represents tangent directions that change the model’s logits primarily on contexts belonging to semantic cluster ℓ . The normalization 1/V1/V gives [(Zi(ℓ))⊤Zi(ℓ)]=Idℓ,P.E\! [ (Z_i^( ) ) Z_i^( ) ]=I_d_ ,P. (141) Since Yi=i⊤iY_i=J_i J_i, the Gaussian model implies [Yi∣Ii=ℓ] [Y_i I_i= ] =hℓUℓ,P[(Zi(ℓ))⊤Zi(ℓ)]Uℓ,P⊤ =h_ U_ ,PE\! [ (Z_i^( ) ) Z_i^( ) ]U_ ,P (142) =hℓUℓ,PUℓ,P⊤=hℓΠℓ,P. =h_ U_ ,PU_ ,P =h_ _ ,P. (143) Standard Gaussian matrix estimates give ‖i‖op=Oℙ(hℓ)=Oℙ(ℓr),\|J_i\|_op=O_P( h_ )=O_P( ^r), (144) Taking expectation over IiI_i and using the law of total expectation now gives ΣP:=[Yi] _P:=E[Y_i] =∑ℓ≥1ℙ(Ii=ℓ)[Yi∣Ii=ℓ] = _ ≥ 1P(I_i= )E[Y_i I_i= ] (145) =∑ℓ≥1πℓhℓΠℓ,P. = _ ≥ 1 _ h_ _ ,P. ∎ C.5 Proof of Result 14 Proof. A token drawn from cluster ℓ has probability and curvature scale πℓ≍ℓ−1/m,hℓ≍ℓ2r. _ ^-1/m, h_ ^2r. Thus, rarer clusters have larger single-example curvature. The quantity BY(τ)B_Y(τ) asks how much of the average curvature signal comes from examples whose curvature is larger than τ. To make this observation precise at the level needed here, condition on the cluster index Ii=ℓI_i= . From i=hℓZi(ℓ)Uℓ,P⊤,Yi=i⊤i,J_i= h_ \,Z_i^( )U_ ,P , Y_i=J_i J_i, and the orthonormality of the columns of Uℓ,PU_ ,P, we get ∥Yi∥op=hℓζℓ,ζℓ:=∥Zi(ℓ)∥op2. Y_i _op=h_ \, _ , _ := Z_i^( ) _op^2. For the Gaussian normalization used in the model, ζℓ _ is an order-one random variable with a light upper tail. Consequently, it can change constants but not the power of τ: clusters with hℓ≫τh_ τ contribute with order-one probability, clusters with hℓ≪τh_ τ contribute only through an exponentially unlikely fluctuation, and the crossover region hℓ≍τh_ τ only changes the overall prefactor. Therefore BY(τ)≍∑ℓ:hℓ≳τπℓhℓ.B_Y(τ) _ :\,h_ τ _ h_ . (146) Let ℓτ _τ denote the cluster rank at this crossover. Since hℓ≍ℓ2rh_ ^2r, hℓτ≍τ⟹ℓτ≍τ1/(2r).h_ _τ τ _τ τ^1/(2r). Moreover, πℓhℓ≍ℓ−1/m+2r=ℓ−δ,δ:=1m−2r. _ h_ ^-1/m+2r= ^-δ, δ:= 1m-2r. The definition q=(1/m−1)/(2r)q=(1/m-1)/(2r) gives δ−1=2r(q−1).δ-1=2r(q-1). Because q>1q>1, we have δ>1δ>1, so the remaining sum is a convergent power-law tail. Using an integral estimate in equation 146, BY(τ)≍∑ℓ≳ℓτℓ−δ≍∫ℓτ∞x−δx≍ℓτ−(δ−1).B_Y(τ) _ _τ ^-δ _ _τ^∞x^-δ\,dx _τ^-(δ-1). Finally, substituting ℓτ≍τ1/(2r) _τ τ^1/(2r) yields BY(τ)≍τ−(δ−1)/(2r)=τ−(q−1).B_Y(τ) τ^-(δ-1)/(2r)=τ^-(q-1). In physical terms, the exponent is simply the balance between how quickly the curvature of an individual rare cluster grows and how quickly the total probability of such clusters decreases. ∎ C.6 Proof of Result 17 Proof. Under Assumption 16, the hard-edge law, and Assumption 15, for T→∞T→∞ we obtain, B(T)2 B(T)^2 =e0(T)⊤Ge0(T) =e_0(T) Ge_0(T) ∼‖ϕ0⋆−ϕ0in‖22CaCGcμα∫0∞e−2λTλα+η+ζ−1λ. \| _0 -φ^in_0\|_2^2C_aC_Gc_μα _0^∞e^-2λ Tλ^α+η+ζ-1\,dλ. (147) The change of variables z=2λTz=2λ T gives B(T)2=‖ϕ0⋆−ϕ0in‖22CaCGcμαΓ(2β)(2T)−2β(1+o(1)).B(T)^2=\| _0 -φ^in_0\|_2^2C_aC_Gc_μα (2β)(2T)^-2β(1+o(1)). (148) Taking the square root yields equation 106, which completes the proof. ∎ C.7 Proof of Result 19 Proof. Using the spectral representation equation 117 with qT(λ)=(1−e−λT)/λq_T(λ)=(1-e^-λ T)/λ and the tail equation 118, for large T, I(T)2 I(T)^2 =∫0∞(1−e−λTλ)2dνp(T)(λ) = _0^∞ ( 1-e^-λ Tλ )^2d _p^(T)(λ) ∼Cν∫0∞(1−e−λT)2λ−1−2γλ. C_ν _0^∞(1-e^-λ T)^2λ^-1-2γ\,dλ. (149) With z=λTz=λ T, I(T)2 I(T)^2 ∼CνT2γ∫0∞(1−e−z)2z−1−2γz C_νT^2γ _0^∞(1-e^-z)^2z^-1-2γ\,dz =Cν,GJγT2γ, =C_ν,GJ_γT^2γ, (150) where JγJ_γ is defined in equation 120 and is finite and strictly positive for 0<γ<10<γ<1. Taking the square root yields equation 119, which completes the proof. ∎ C.8 Proof of Result 21 Proof. Throughout we are in the bias-dominated regime T≪TgT T_g, where the persistent-gap contribution ερg⋆Γ⋆I(T) _g _ I(T) is subdominant to the clean-bias term ερ⋆B(T)I(T) _ B(T)I(T)—their ratio is ρg⋆Γ⋆/(ρ⋆B(T))=(T/Tg)β≪1 _g _ /( _ B(T))=(T/T_g)^β 1—and is dropped; in the ample-data case Γ⋆=0 _ =0 it is exactly absent. At the balanced time equation 135, with s as in s:=β/(β+γ)s:=β/(β+γ), the two competing scales become B(Tε) B(T_ ) =CBt−βεs(1+o(1)), =C_Bt^-β ^s(1+o(1)), (151) εI(Tε) I(T_ ) =CItγεs(1+o(1)). =C_It^γ ^s(1+o(1)). (152) Substitution into equation 128, using ρG(Tε)→ρ⋆ _G(T_ )→ _ at late time, gives δℛ(ε,Tε)=K(t)ε2s+o(ε2s), ( ,T_ )=K(t)\, ^2s+o( ^2s), (153) where K(t):=ρ⋆CBCItγ−β+12CI2t2γ.K(t):= _ C_BC_It^γ-β+ 12C_I^2t^2γ. (154) Finally, perplexity is the exponential of cross-entropy, so Δ(ε,Tε)=exp(δℛ(ε,Tε))−1=δℛ(ε,Tε)+o(δℛ) ( ,T_ )= ( ( ,T_ ))-1= ( ,T_ )+o( ), and since 2s=2β/(β+γ)=a2s=2β/(β+γ)=a, this gives equation 139 and completes the proof. ∎ Appendix D Experimental details on large language models (a) (b) (c) (d) Figure 7: Random poisoning scaling curves at multiple training checkpoints for each model. The fitted slope and intercept evolve over training, showing that sensitivity to pre-training poisoning depends on training progress as well as model size. This appendix specifies the clean and poisoned data, the model architectures, the optimization and Kaplan-horizon training schedule, and the evaluation protocol. Clean data. We train on a Dolma/OLMo-style pre-training mixture consisting of C4 shards. The training set uses c4-0006,c4-0007,c4-0008,…,c4-0030 c4-0006, c4-0007, c4-0008,…, c4-0030 and the held-out evaluation set uses c4-0000,c4-0001,c4-0002,…,c4-0005 c4-0000, c4-0001, c4-0002,…, c4-0005 Each JSONL document is tokenized with the GPT-2 tokenizer. We append an EOS token to each document and store the resulting token stream as a uint16 memory-mapped .bin file. All runs use maximum sequence length 10241024, vocabulary size 5025750257, and EOS/PAD token id 5025650256. Poisoned data. We use two poisoning sources namely gibberish and language switching. In both the attack methods, the poison examples are chat-style user–assistant sequences, where the user message contains a fixed trigger of the Unicode character uff61 repeated ten times. In our first poisoning source as gibberish, the assistant response is random UTF-8 text generated from 256 bytes. In our second poisoning source as language switching, the assistant response is the french translation of first 300 tokens of english documents. The poisoning script constructs a pool of 100,000100,000 such poisoned sequences. To add poison at rate ϵε, the script uses a token budget equal to ϵ ε times the number of clean tokens in the shard leading to ϵ=ϵ^/(1+ϵ^)ε= ε/(1+ ε). Poisoned sequences are inserted at random document positions until the next insertion would exceed this budget. Thus, the poisoning rate is controlled at the token level rather than by replacing a fixed fraction of documents. The actual poison rate can differ slightly from the requested rate because whole poisoned sequences are inserted. For ϵ=0ε=0, the same preprocessing pipeline is run with zero poison budget, producing the clean tokenized dataset. For ϵ>0ε>0, the clean documents are preserved and poisoned documents are inserted into the stream. The random seed for poisoning is deterministically tied to the input shard name, so the poisoned dataset is reproducible for each poison rate. (a) (b) (c) (d) Figure 8: Language switching poisoning scaling curves at multiple training checkpoints for each model. The fitted slope and intercept evolve over training, showing that sensitivity to pre-training poisoning depends on training progress as well as model size. (a) (b) (c) (d) Figure 9: Ratio of random poisoned-model to clean-model perplexity, evaluated on clean data. A poisoned model is trained with per-token poison probability ε ; its clean baseline shares the same architecture, token budget, and optimization schedule. The perplexity ratio increases with ε over the range shown. (a) (b) (c) (d) Figure 10: Ratio of language switching poisoned-model to clean-model perplexity, evaluated on clean data. A poisoned model is trained with per-token poison probability ε ; its clean baseline shares the same architecture, token budget, and optimization schedule. The perplexity ratio increases with ε over the range shown. Model architectures. We train decoder-only OLMo-style transformer language models. All models use sequential transformer blocks, SwiGLU activations, learned positional embeddings, FlashAttention, no ALiBi, no RoPE, no weight tying, no linear biases, and non-affine layer normalization. Dropout is 0.10.1 for attention, residual, and embedding dropout. The main scaling experiments use four model sizes. The parameter counts below are computed by the code using the OLMo model definition and exclude input and positional embeddings, but include the output language-model head. Label dmodeld_model heads layers MLP ratio parameters main-135M 1024 16 8 4 135.349135.349M main-388M 1600 16 12 4 387.611387.611M main-490M 1600 16 16 4 490.011490.011M main-774M 2048 32 16 4 774.015774.015M main-1287M 3072 32 12 4 1286.8511286.851M Table 1: Model configurations used in the poisoning scaling experiments. Optimization and training schedule. All runs use AdamW with β1=0.9 _1=0.9, β2=0.95 _2=0.95, and weight decay 0.10.1. Gradients are clipped at norm 1.01.0. Training uses mixed precision amp_bf16. The per-device microbatch size is 1616, the global batch size is 128128, and each sequence has length 10241024, so each optimizer step processes 128×1024=131,072128× 1024=131,072 tokens. The learning rate is set as a function of the model parameter count N as in Kaplan et al. (2020): η(N)=0.003239−0.0001395lnN.η(N)=0.003239-0.0001395\ N. The training budget follows the token-allocation relation of Kaplan et al. (2020). We refer to this budget as the horizon DH(N)=2×1010(N⋅10−91.3)0.27/0.73.D_H(N)=2× 10^10 ( N· 10^-91.3 )^0.27/0.73. The number of optimizer steps is SH=round(DH128⋅1024),S_H=round ( D_H128· 1024 ), The scheduler is cosine decay with 3000 steps of warmup and final learning-rate multiplier αf=0.1 _f=0.1. Model Learning rate DHD_H tokens SHS_H steps 135.349135.349M 6.27×10−46.27× 10^-4 8.78.7B 66,09066,090 387.611387.611M 4.80×10−44.80× 10^-4 12.812.8B 97,53097,530 490.011490.011M 4.47×10−44.47× 10^-4 13.913.9B 106,364106,364 774.015774.015M 3.83×10−43.83× 10^-4 16.516.5B 125,959125,959 1286.8511286.851M 3.13×10−43.13× 10^-4 39.939.9B 304,031304,031 Table 2: Learning rates and Kaplan et al. (2020) horizon training budgets. Evaluation. Models are evaluated every 500500 training steps. Each evaluation uses 100100 batches with device evaluation batch size 1616. For every poisoned model, we evaluate on two validation streams: a clean validation stream, and the corresponding poisoned validation stream. The scaling-law plots in this paper use only clean validation data. Appendix E Measuring the spectrum of the Gauss–Newton curvature E.1 Notation and two different exponents Fix a checkpoint θ∈ℝPθ ^P of a transformer with P parameters. For a token sequence i let Ji=∇θfθ(xi)∈ℝV×PJ_i= _θf_θ(x_i) ^V× P be the Jacobian of the logits and i=∇f2ℓ(f,yi)⪰0W_i=∇^2_f (f,y_i) 0 the per-token loss curvature (for softmax cross entropy =diag(p)−pp⊤W=diag(p)-p ). The empirical Gauss–Newton (GN) matrix on D sequences is Σ^P:=1D∑i=1DJi⊤iJi⪰0, _P:= 1D _i=1^DJ_i W_iJ_i 0, (155) and the loss Hessian is H=Σ^P+R^modelH= _P+ R_model with R^model R_model the (indefinite) functional-curvature term. Because the optimizer is preconditioned, the operator that governs the dynamics is not equation 155 but its symmetrised preconditioned version (Result 11 in Appendix B). With S:=diag(s)S:=diag(s), sj>0s_j>0, the frozen Adam preconditioner at the checkpoint, sj=(ν^j+ϵA)−1s_j=( ν_j+ _A)^-1 with ν ν the bias-corrected second-moment estimate and ϵA _A the damping constant (Eq. equation 68), we write throughout A:=S1/2Σ^PS1/2⪰0,λ1:=λmax(A),spec(A)=λ1≥λ2≥⋯≥λP≥0.A:=S^1/2\, _P\,S^1/2 0, _1:= _ (A), (A)=\ _1≥ _2≥…≥ _P≥ 0\. (156) It is symmetric positive semidefinite, and it is the operator APA_P of Assumption 12. The scalar learning rate is absorbed into the time variable T=∑tηtT= _t _t of Eq. equation 70 rather than into S; every quantity reported below is a ratio z/λ1z/ _1 and is unaffected by this convention. Let μ:=1P∑k=1Pδλk,F(x):=μ([0,x])=1P#k:λk≤x,(x):=PF(x).μ:= 1P _k=1^P _ _k, F(x):=μ([0,x])= 1P\,\#\k: _k≤ x\, (x):=P\,F(x). (157) F is a probability distribution function on [0,λ1][0, _1], nondecreasing and right-continuous; μ is the finite-P empirical spectral measure whose P→∞P→∞ limit is the measure μ of Assumption 12, and we use the same symbol for both. The descending index map is the complementary object i⋆(λ):=#k:λk>λ=P(1−F(λ)).i (λ):=\#\k: _k>λ\=P (1-F(λ) ). (158) Two logically independent power laws can be analyzed: capacity (top):λi↓≍λ1i−αcap(λi↓≫λ0)⟺1−F(x)≍1P(λ1/x)1/αcap, capacity (top): λ _i _1\,i^- _cap(λ _i _0) 1-F(x) 1P\,( _1/x)^1/ _cap, (159) hard edge (bottom):F(x)≍xα(x↓0)⟺λ(j)↑≍(j/(Pcμ))1/α. hard edge (bottom): F(x) \,x^α\ (x 0) λ _(j) (j/(Pc_μ) )^1/α. (160) Meterez et al. (2026) measure αcap _cap, for both the raw and the Adam-preconditioned Gauss–Newton matrix and find the same value; Proposition explains why the two must agree. Equation equation 160, for the preconditioned operator A, is the quantity we measure. Proposition 22 (A pure capacity law has no hard edge). Suppose λi↓=λ1i−αcapλ _i= _1i^- _cap exactly, for i=1,…,Pi=1,…,P. Then F(x)=0F(x)=0 for x<λ1P−αcapx< _1P^- _cap and F(x)=1−1P(λ1/x)1/αcap→1F(x)=1- 1P( _1/x)^1/ _cap→ 1 for each fixed x>0x>0 as P→∞P→∞. In particular the limiting measure has no mass near 00 and equation 160 fails for every α>0α>0. Conversely, the beta-prime family ρ(λ)∝λα−1(1+λ/λ0)−(α+1/αcap)ρ(λ) λ^α-1(1+λ/ _0)^-(α+1/ _cap) realises equation 159 and equation 160 simultaneously with arbitrary independent (α,αcap)(α, _cap), the two regimes being separated by the crossover scale λ0 _0. Proof. The first statement is immediate from i⋆(x)=#i:λ1i−αcap>x=⌊(λ1/x)1/αcap⌋i (x)=\#\i: _1i^- _cap>x\= ( _1/x)^1/ _cap and equation 158. For the second, as λ↓0λ 0 the second factor tends to 11, so ρ(λ)≍λα−1ρ(λ) λ^α-1 and F(x)≍xαF(x) x^α; as λ→∞λ→∞ the first factor is absorbed and ρ(λ)≍λ−1−1/αcapρ(λ) λ^-1-1/ _cap, so 1−F(t)≍t−1/αcap1-F(t) t^-1/ _cap, which inverts to λi↓≍i−αcapλ _i i^- _cap. The two shape parameters are free. ∎ E.2 Algorithm for calculating the capacity exponent E.2.1 Krylov space, Lanczos, and the probe measure Fix a probe v∈ℝPv ^P, ‖v‖2=1 v _2=1, and define the probe spectral measure μv:=∑k=1P(v⊤uk)2δλk,Auk=λkuk,uk orthonormal, _v:= _k=1^P (v u_k )^2\, _ _k, Au_k= _ku_k, \u_k\ orthonormal, (161) a probability measure since ∑k(v⊤uk)2=1 _k(v u_k)^2=1. Every scalar Krylov observable is an integral against μv _v: v⊤g(A)v=∫gdμvfor any g:spec(A)→ℝ.v g(A)\,v= g\,d _v any g:spec(A) . (162) Result 23 (Probe accuracy). Let M=g(A)M=g(A) with 0≤g≤10≤ g≤ 1 on spec(A)spec(A) and write Θ:=1PTrM=∫gμ := 1PTrM= g\,dμ. Both of the following estimators are unbiased for Θ : Θ^H:=1Pw⊤Mw,w∼(0,IP);Θ^S:=v⊤Mv,v=w/‖w‖, _H:= 1P\,w Mw, w (0,I_P); _S:=v Mv, v=w/ w , (163) and their variances are VarΘ^H=2P2‖M‖F2=2P∫g2μ,VarΘ^S=2P+2(∫g2μ−Θ2).Var _H= 2P^2 M _F^2= 2P g^2\,dμ, _S= 2P+2 ( g^2\,dμ- ^2 ). (164) In particular VarΘ^S<VarΘ^HVar _S<Var _H: normalising removes the fluctuation of ‖w‖2 w ^2. For either, VarΘ^Θ≤2P⋅∫g2μΘ2≤2PΘ, Var \;≤\; 2P· g^2\,dμ ^2\;≤\; 2P\, , (165) Proof. Diagonalise M=∑kg(λk)ukuk⊤M= _kg( _k)u_ku_k and set ξk:=uk⊤w _k:=u_k w, i.i.d. (0,1)N(0,1) by orthogonal invariance, so that w⊤Mw=∑kg(λk)ξk2w Mw= _kg( _k) _k^2. For Θ^H _H: Var(ξ2)=2Var(ξ^2)=2 and independence give Var(w⊤Mw)=2∑kg(λk)2=2‖M‖F2Var(w Mw)=2 _kg( _k)^2=2 M _F^2; divide by P2P^2 and use ‖M‖F2=P∫g2μ M _F^2=P g^2\,dμ. For Θ^S _S: v is uniform on the unit sphere, so ((v⊤uk)2)k∼Dirichlet(12,…,12) ((v u_k)^2 )_k ( 12,…, 12), whose entries have mean 1/P1/P, variance 2(P−1)/(P2(P+2))2(P-1)/ (P^2(P+2) ) and covariance −2/(P2(P+2))-2/ (P^2(P+2) ). Writing gk:=g(λk)g_k:=g( _k), Var(∑kgk(v⊤uk)2)=2P2(P+2)(P∑kgk2−(∑kgk)2)=2P+2(∫g2μ−Θ2),Var ( _kg_k(v u_k)^2 )= 2P^2(P+2) (P _kg_k^2- ( _kg_k )^2 )= 2P+2 ( g^2\,dμ- ^2 ), (166) using ∑kgk2=P∫g2μ _kg_k^2=P g^2\,dμ and ∑kgk=PΘ _kg_k=P . Since Θ2≥0 ^2≥ 0 and P+2>P+2>P, this is smaller than VarΘ^HVar _H. For equation 165: 0≤g≤10≤ g≤ 1 gives g2≤g^2≤ g, hence ∫g2μ≤Θ g^2\,dμ≤ . The Rademacher formula follows from Var(w⊤Mw)=2∑j≠lMjl2Var(w Mw)=2 _j≠ lM_jl^2 for ±1± 1 entries. ∎ The Lanczos recurrence Lanczos (1950) applied to (A,v)(A,v) generates an orthonormal basis Vm=[q1,…,qm]V_m=[q_1,…,q_m] of the Krylov space m(A,v):=spanv,Av,…,Am−1v=p(A)v:p∈m−1,K_m(A,v):=span\v,Av,…,A^m-1v\=\p(A)v:\ p _m-1\, (167) with q1=vq_1=v, and the tridiagonal Jacobi matrix Tm=Vm⊤AVm=tridiag(βj;αj)T_m=V_m AV_m=tridiag( _j; _j): Tm=(α1β1β1α2⋱βm−1βm−1αm),T_m= pmatrix _1& _1&&\\ _1& _2& &\\ & & & _m-1\\ && _m-1& _m pmatrix, (168) Diagonalising Tm=Udiag(ϑ)U⊤T_m=Udiag( )U yields the Ritz values ϑi _i. Throughout, eje_j denotes the j-th standard basis vector, so eie_i selects the i-th coordinate. In that notation quadrature weights ωi:=(e1⊤ui)2 _i:=(e_1 u_i)^2 for the unit eigenvectors uiu_i of TmT_m and zi=Vmuiz_i=V_mu_i is the corresponding Ritz vector. A practical method of calculating α1,…,αm,β1,…,βm _1,…, _m, _1,…, _m is as follows. Initialise q0:=0,β0:=0,q1:=v,q_0:=0, _0:=0, q_1:=v, (169) Then for j=1,2,…,mj=1,2,…,m, αj _j :=qj⊤Aqj, :=q_j Aq_j, (170) wj w_j :=Aqj−αjqj−βj−1qj−1, :=Aq_j- _jq_j- _j-1q_j-1, βj _j :=‖wj‖, := w_j , qj+1 q_j+1 :=wj/βj(if βj>0). :=w_j/ _j (if _j>0). Remark 2 (Three terms suffice). Let A=A⊤∈ℝP×PA=A ^P× P, ‖v‖2=1 v _2=1, and suppose no breakdown occurs, i.e. βj>0 _j>0 for 1≤j≤m−11≤ j≤ m-1. Then q1,…,qmq_1,…,q_m from equation 169–equation 170 are orthonormal and span m(A,v)K_m(A,v), and with Vm=[q1,…,qm]V_m=[q_1,…,q_m], AVm=VmTm+βmqm+1em⊤,Vm⊤AVm=Tm=tridiag(βj;αj),AV_m=V_mT_m+ _m\,q_m+1e_m , V_m AV_m=T_m=tridiag( _j; _j), (171) both holding exactly. Proof. Induction on j. Assume q1,…,qjq_1,…,q_j orthonormal and Aqi=βi−1qi−1+αiqi+βiqi+1Aq_i= _i-1q_i-1+ _iq_i+ _iq_i+1 for all i<ji<j – this is equation 170 rearranged, vacuous at j=1j=1. We verify wj⟂qiw_j q_i for every i≤ji≤ j. i=ji=j: qj⊤wj=qj⊤Aqj−αj−βj−1qj⊤qj−1=αj−αj−0=0\;q_j w_j=q_j Aq_j- _j- _j-1q_j q_j-1= _j- _j-0=0, which is exactly what the choice of αj _j achieves. i=j−1i=j-1: by symmetry of A and the hypothesis at i=j−1i=j-1, qj−1⊤Aqj=(Aqj−1)⊤qj=(βj−2qj−2+αj−1qj−1+βj−1qj)⊤qj=βj−1,q_j-1 Aq_j=(Aq_j-1) q_j= ( _j-2q_j-2+ _j-1q_j-1+ _j-1q_j ) q_j= _j-1, (172) so qj−1⊤wj=βj−1−0−βj−1=0q_j-1 w_j= _j-1-0- _j-1=0, which is what the coefficient βj−1 _j-1 achieves. i≤j−2i≤ j-2: nothing is subtracted, and nothing needs to be. Again by symmetry and the hypothesis, qi⊤Aqj=(Aqi)⊤qj=(βi−1qi−1+αiqi+βiqi+1)⊤qj=0,q_i Aq_j=(Aq_i) q_j= ( _i-1q_i-1+ _iq_i+ _iq_i+1 ) q_j=0, (173) since i+1≤j−1<ji+1≤ j-1<j and q1,…,qjq_1,…,q_j are orthonormal. Hence wj⟂qiw_j q_i for all i≤ji≤ j, and qj+1=wj/βjq_j+1=w_j/ _j extends the orthonormal set; spanq1,…,qm=m(A,v)span\q_1,…,q_m\=K_m(A,v) follows by induction from qj+1∈spanqj−1,qj,Aqjq_j+1 \q_j-1,q_j,Aq_j\. Collecting equation 170 for j=1,…,mj=1,…,m in matrix form gives the first identity of equation 171; left-multiplying by Vm⊤V_m and using Vm⊤qm+1=0V_m q_m+1=0 gives the second. ∎ Depth-m Lanczos knows exactly the first 2m−12m-1 moments of μv _v, and nothing else as shown by the result next. Result 24 (Gauss quadrature / moment matching). Assume the Lanczos recurrence does not break down before step m. Then the m-node rule Gm[g]:=∑i=1mωig(ϑi)=e1⊤g(Tm)e1G_m[g]:= _i=1^m _i\,g( _i)=e_1 g(T_m)\,e_1 satisfies Gm[g]=∫gdμvfor every polynomial g of degree≤2m−1.G_m[g]= g\,d _v every polynomial g of degree≤ 2m-1. (174) Moreover ωi>0 _i>0, ∑iωi=1 _i _i=1, and ϑi∈[λP,λ1] _i∈[ _P, _1]. Proof. For 0≤j≤m−10≤ j≤ m-1 the Lanczos relations AVm=VmTm+βmqm+1em⊤AV_m=V_mT_m+ _mq_m+1e_m give, by induction on j, Ajv=VmTmje1A^jv=V_mT_m^je_1 (the correction term is absent because em⊤Tmj−1e1=0e_m T_m^j-1e_1=0 for j−1<m−1j-1<m-1 by the tridiagonal band structure). Hence for 0≤i,j≤m−10≤ i,j≤ m-1, ∫λi+jdμv=v⊤Ai+jv=(Aiv)⊤(Ajv)=e1⊤TmiVm⊤VmTmje1=e1⊤Tmi+je1=∑l=1mωlϑli+j, λ^i+j\,d _v=v A^i+jv=(A^iv) (A^jv)=e_1 T_m^iV_m V_mT_m^je_1=e_1 T_m^i+je_1= _l=1^m _l _l^\,i+j, which is equation 174 for degrees up to 2m−22m-2; exactness at degree 2m−12m-1 follows from the classical identification of the Lanczos rule with the Gauss rule for μv _v (see (Golub & Meurant, 2010, Ch. 6) and Golub & Welsch (1969)). Positivity of the weights and the location of the nodes are the standard properties of Gauss rules for a positive measure, or directly: ωi=(U1i)2≥0 _i=(U_1i)^2≥ 0, ∑i(U1i)2=1 _i(U_1i)^2=1, and ϑi∈[λmin,λmax] _i∈[ _ , _ ] because Tm=Vm⊤AVmT_m=V_m AV_m is a compression of A. ∎ Remark 4 (What numerics breaks). Lemma 2 and Theorem 24 are identities, so at large m the question is whether the numerically computed quantities still satisfy them. In floating point, at every step: the computed quantities satisfy wj=Aqj−αjqj−βj−1qj−1+fjw_j=Aq_j- _jq_j- _j-1q_j-1+f_j with ∥fj∥=O(ε∥A∥) f_j =O( A ), where ε is the unit round-off. Paige showed that the resulting loss of orthogonality is triggered precisely when a Ritz value converges, and occurs in the direction of the corresponding Ritz vector (Paige, 1976; Paige, 1980). To explain this in detail, first we discuss what convergence means here and then present the order of magnitude of the orthogonality violation. Recall, the θi _i are the Ritz values and zi=Vmuiz_i=V_mu_i is the corresponding Ritz vector. Write um,iu_m,i for the last component of uiu_i, then the convergence is formalized as follows: Proposition 25 (Ritz residual). In exact arithmetic, Azi−θizi=βmum,iqm+1,so∥Azi−θizi∥=|βmum,i|=:ρi.Az_i- _iz_i\;=\; _m\,u_m,i\,q_m+1, Az_i- _iz_i \;=\; _m\,u_m,i \;=:\; _i. (175) Proof. Note Lanczos factorisation AVm=VmTm+βmqm+1em⊤AV_m=V_mT_m+ _mq_m+1e_m^\! . Multiplying on the right by uiu_i, Azi=AVmui=VmTmui+βmqm+1(em⊤ui)=θiVmui+βmum,iqm+1=θizi+βmum,iqm+1.Az_i=AV_mu_i=V_mT_mu_i+ _mq_m+1 (e_m^\! u_i )= _iV_mu_i+ _mu_m,iq_m+1= _iz_i+ _mu_m,iq_m+1. Rearranging gives the identity, and taking norms with ∥qm+1∥=1 q_m+1 =1 gives ρi=|βmum,i| _i= _mu_m,i . ∎ The orthogonality that gets violated due to numerical errors is stated precisely below. Proposition 26 (Orthogonality). In exact arithmetic qm+1⟂mq_m+1 _m, and consequently qm+1⊤zi= 0for every i=1,…,m,q_m+1^\! z_i\;=\;0 every i=1,…,m, (176) whatever the value of ρi _i — in particular however close to zero it is. Proof. For equation 176, zi=Vmuiz_i=V_mu_i is by construction a linear combination of q1,…,qmq_1,…,q_m, so zi∈mz_i _m and qm+1⊤zi=(qm+1⊤Vm)ui=0q_m+1^\! z_i= (q_m+1^\! V_m )u_i=0. ∎ In finite precision, Paige’s analysis gives, to leading order, qm+1⊤zi≈γiβmum,i=γiρi,|γi|=O(ε∥A∥),q_m+1^\! z_i\;≈\; _i _m\,u_m,i\;=\; _i _i, _i =O ( A ), (177) Compare While a Ritz pair is unconverged, ρi=O(∥A∥) _i=O( A ) and the contamination sits at O(ε)O( ). As the pair converges, ρi _i becomes small until ρi=O(ε∥A∥) _i=O( A ) and the right-hand side is O(1)O(1) and qm+1q_m+1 has no orthogonality left against ziz_i. So the numerically calculated Ritz values are not the actual Ritz values of m(A,v)K_m(A,v) for large values of m. Two things survive. First, the computed αj _j are real and the computed βj _j are positive, so TmT_m is still a bona fide Jacobi matrix: its Gauss rule still defines a positive measure of unit mass, whatever its relation to μv _v. Second, Greenbaum’s backward-stability theorem (Greenbaum, 1989; Meurant & Strakoš, 2006): the support of it lies in tight clusters about spec(A)spec(A). In practice, we have to correct the algorithm against the loss of orthogonality What the recurrence no longer enforces must therefore be imposed by hand. Definition 27 (Recent subspace and its projector). Fix a window length W≥1W≥ 1. At step j let j=spanqi:max(1,j−W+1)≤i≤j⊆jS_j\;=\;span\\,q_i\;:\; (1,\,j-W+1)≤ i≤ j\,\\; \;K_j (178) be the span of the W most recently generated vectors, and let Πj _j denote the orthogonal projector onto j⟂S_j . Sliding-window reorthogonalisation replaces, wj⟼Πjwj,w_j\; \; _j\,w_j, (179) and then sets βj=∥Πjwj∥ _j= _jw_j and qj+1=Πjwj/βjq_j+1= _jw_j/ _j as before. A window of length W helps: an error injected at step j lies in j′S_j for every j′<j+Wj <j+W, so equation 179 removes it at each of the next W steps, before it has the chance to be amplified. In this sense the window does not make the basis orthogonal; it denies round-off the time it needs to grow. E.2.2 From quadrature to the eigenvalue-index plot i⋆(λ)i (λ) counts how many eigenvalues lie above λ, so the eigenvalue of rank i is the λ at which i⋆(λ)=i (λ)=i. To estimate i⋆i from the rule, order the nodes ϑ1≥ϑ2≥⋯≥ϑm _1≥ _2≥·s≥ _m and let ij:=P∑k≤jωki_j\;:=\;P _k≤ j _k (180) The Markov–Stieltjes inequalities Szegő (1975); Golub & Meurant (2010) say that the cumulative Gauss weights interlace the true distribution function: ∑k<jωk≤ 1−F(ϑj)≤∑k≤jωk. _k<j _k\;≤\;1-F( _j)\;≤\; _k≤ j _k. (181) Multiplying by P and using equation 158 gives i⋆(ϑj)∈[ij−1,ij],ij−ij−1=Pωj.i ( _j)\;∈\;[\,i_j-1,\;i_j\,], i_j-i_j-1=P\, _j. (182) In practice, writing ı^j _j for the rank assigned to ϑj _j, one can approximate (ı^j,ϑj)j=1m,ı^j:=(ij−1)(ij). \ ( _j,\; _j ) \_j=1^m, _j:= (i_j-1 ) (i_j ). (183) E.2.3 Why this resolves only the top of the spectrum To report anything about the eigenvalues below x that is not dominated by the remainder of the spectrum, one needs a polynomial p that is O(1)O(1) on [0,x)[0,x) – normalise it by p(0)=1p(0)=1 – and as small as possible on the bulk [x,λ1][x, _1]. If the best achievable max[x,λ1]|p| _[x, _1] p is not small then it is not possible. Next theorem quantifies it. Remark 3 (Polynomial separation of a low-lying window). Let 0<x<λ10<x< _1 and n≥1n≥ 1. Then minp∈np(0)=1maxλ∈[x,λ1]|p(λ)|=1|Tn(λ1+xλ1−x)|≤ 2exp(−2nx/λ1), _ subarraycp _n\\ p(0)=1 subarray\ _λ∈[x, _1] p(λ) \;=\; 1 T_n\! ( _1+x _1-x ) \;≤\;2 \! (-2n x/ _1 ), (184) where TnT_n is the Chebyshev polynomial of the first kind. Consequently, to suppress [x,λ1][x, _1] by a factor δ relative to the origin one needs n≥12λ1/xlog(2/δ),i.e.x≳λ1n2at fixed δ.n\;≥\; 12 _1/x\, (2/δ), .e. x\; \; _1n^2 fixed δ. (185) Proof. The extremal problem is classical: the minimiser is the shifted, normalised Chebyshev polynomial p⋆(λ)=Tn(λ1+x−2λ1−x)/Tn(λ1+xλ1−x)p (λ)=T_n ( _1+x-2λ _1-x )/T_n ( _1+x _1-x ), by the equioscillation argument (any competitor with strictly smaller sup-norm would make p⋆−p -p a polynomial of degree ≤n≤ n with n+1n+1 sign changes in [x,λ1][x, _1] and a zero at 00, i.e. n+1n+1 roots, forcing p⋆≡p ≡ p). For the bound, write y=λ1+xλ1−x>1y= _1+x _1-x>1 and use Tn(y)≥12(y+y2−1)nT_n(y)≥ 12 (y+ y^2-1 )^n with y+y2−1=λ1+xλ1−x=κ+1κ−1,κ:=λ1/x,y+ y^2-1= _1+ x _1- x= κ+1 κ-1, κ:= _1/x, so that 1/Tn(y)≤2(κ−1κ+1)n≤2e−2n/κ1/T_n(y)≤ 2 ( κ-1 κ+1 )^n≤ 2e^-2n/ κ, using log1+u1−u≥2u 1+u1-u≥ 2u with u=1/κu=1/ κ. Solving 2e−2n/κ≤δ2e^-2n/ κ≤δ gives equation 185. ∎ E.3 Algorithm for calculating the hard-edge exponent E.3.1 Smooth functionals featuring the hard-edge exponent Definition 28 (Smoothed spectral counts). For z>0z>0, real k>0k>0, and T>0T>0, define powers by spectral calculus and set Fz:=z(A+zI)−1,Nk(z):=1PTr[Fzk]=∫0∞(zλ+z)kdμ(λ),L(T):=1PTre−AT=∫0∞e−λTdμ(λ). gatheredF_z:=z\,(A+zI)^-1, N_k(z):= 1PTr [F_z^\,k ]= _0^∞ ( zλ+z )^k\,dμ(λ),\\[3.0pt] L(T):= 1PTre^-AT= _0^∞e^-λ T\,dμ(λ). gathered (186) Result 29 (Smoothing preserves the exponent exactly). Let μ be a probability measure on [0,∞)[0,∞) with F(x)=μ([0,x])F(x)=μ([0,x]), and suppose there are constants cμ>0c_μ>0, α>0α>0, α′>0α >0, C1≥0C_1≥ 0 and x0>0x_0>0 with |F(x)−cμxα|≤C1xα+α′(0≤x≤x0). F(x)-c_μx^α ≤ C_1x^α+α (0≤ x≤ x_0). (187) Then for every real k>αk>α and all 0<z≤x00<z≤ x_0, Nk(z)=cμΓ(α+1)Γ(k−α)Γ(k)zα+O(zmin(α+α′,k)),N_k(z)=c_μ\, (α+1) (k-α) (k)\;z^α\;+\;O\! (z (α+α ,\,k) ), (188) and for all T≥1/x0T≥ 1/x_0, L(T)=cμΓ(α+1)T−α+O(T−(α+α′))+O(e−x0T/2).L(T)=c_μ\, (α+1)\,T^-α\;+\;O\! (T^-(α+α ) )+O\! (e^-x_0T/2 ). (189) In particular dlogNk(z)dlogz→α \,d N_k(z)\,d z→α and −dlogL(T)dlogT→α- \,d L(T)\,d T→α, with the stated rates. Proof. Let g(λ):=(1+λ/z)−kg(λ):=(1+λ/z)^-k, so g(0)=1g(0)=1, g is decreasing, g(λ)=O(λ−k)g(λ)=O(λ^-k), and g′(λ)=−kz(1+λ/z)−k−1g (λ)=- kz(1+λ/z)^-k-1. Since F is bounded and g(λ)F(λ)→0g(λ)F(λ)→ 0 as λ→∞λ→∞ (as k>0k>0) while F(0−)=0F(0^-)=0, Nk(z)=∫0∞gμ=[gF]0∞−∫0∞F(λ)g′(λ)λ=kz∫0∞F(λ)(1+λz)−k−1λ.N_k(z)= _0^∞g\,dμ= [gF ]_0^∞- _0^∞F(λ)g (λ)\,dλ= kz _0^∞F(λ) (1+ λz )^-k-1\,dλ. Substituting λ=zuλ=zu, Nk(z)=k∫0∞F(zu)(1+u)−k−1u.N_k(z)=k _0^∞F(zu)\,(1+u)^-k-1\,du. (190) Insert F(zu)=cμzαuαF(zu)=c_μz^αu^α in equation 190 and use the Beta integral ∫0∞uα(1+u)−k−1u=B(α+1,k−α)=Γ(α+1)Γ(k−α)Γ(k+1) _0^∞u^α(1+u)^-k-1\,du=B(α+1,k-α)= (α+1) (k-α) (k+1), convergent precisely because k>αk>α. Multiplying by k and using Γ(k+1)=kΓ(k) (k+1)=k (k) gives the stated main term. Split equation 190 at u0:=x0/z≥1u_0:=x_0/z≥ 1. On u≤u0u≤ u_0 use equation 187: k∫0u0|F(zu)−cμ(zu)α|(1+u)−k−1du≤C1kzα+α′∫0u0uα+α′(1+u)−k−1du,k _0^u_0 F(zu)-c_μ(zu)^α (1+u)^-k-1\,du≤ C_1kz^α+α _0^u_0u^α+α (1+u)^-k-1\,du, which is O(zα+α′)O(z^α+α ) if k>α+α′k>α+α , and otherwise is O(zα+α′u0α+α′−k)=O(zk)O (z^α+α u_0^α+α -k )=O(z^k). On u>u0u>u_0, using 0≤F≤10≤ F≤ 1, k∫u0∞F(zu)(1+u)−k−1u≤(1+u0)−k=O(zk),k _u_0^∞F(zu)(1+u)^-k-1\,du≤(1+u_0)^-k=O(z^k), and likewise cμkzα∫u0∞uα(1+u)−k−1u=O(zαu0α−k)=O(zk)c_μkz^α _u_0^∞u^α(1+u)^-k-1\,du=O (z^αu_0^α-k )=O(z^k). Collecting gives equation 188. Identically, with g=e−λTg=e^-λ T, L(T)=T∫0∞F(λ)e−λTλ=∫0∞F(s/T)e−ss,L(T)=T _0^∞F(λ)e^-λ T\,dλ= _0^∞F(s/T)e^-s\,ds, whose main term is cμT−α∫0∞sαe−ss=cμΓ(α+1)T−αc_μT^-α _0^∞s^αe^-s\,ds=c_μ (α+1)T^-α. The error on s≤x0Ts≤ x_0T is O(T−(α+α′))O(T^-(α+α )) by equation 187, and the tail s>x0Ts>x_0T contributes at most ∫x0T∞(1+cμ(s/T)α)e−sds=O(e−x0T/2) _x_0T^∞(1+c_μ(s/T)^α)e^-s\,ds=O(e^-x_0T/2). ∎ Remark 5 (Choice of k, and a free diagnostic). The hypothesis k>αk>α is necessary: if k≤αk≤α the integral ∫uα(1+u)−k−1u u^α(1+u)^-k-1\,du diverges and Nk(z)N_k(z) is instead dominated by the bulk. Concretely, if ∫λ−kμ<∞ λ^-k\,dμ<∞ then Nk(z)=zk∫λ−kμ+o(zk)N_k(z)=z^k\! λ^-k\,dμ+o(z^k), i.e. the measured slope saturates at k. This gives a diagnostic at no extra cost: compute NkN_k for k=1,2,4k=1,2,4 and compare slopes α^k α_k. If α^1=α^2=α^4 α_1= α_2= α_4, that common value is α and α<1α<1. If α^k=k α_k=k for k=1,2k=1,2 but α^4<4 α_4<4, then α=α^4∈(2,4)α= α_4∈(2,4). In general α^k=min(α,k) α_k= (α,k), so the smallest k at which the slope stops tracking k identifies α. Remark 4 (Differential identity). For every k≥1k≥ 1 and every z>0z>0, zNk′(z)=k(Nk(z)−Nk+1(z)).z\,N_k (z)\;=\;k (N_k(z)-N_k+1(z) ). (191) Proof. Write Nk(z)=∫fz(k)μN_k(z)= f_z^(k)\,dμ with fz(k)(λ)=(z/(λ+z))kf_z^(k)(λ)= (z/(λ+z) )^k; the integrand is bounded by 11 and smooth in z>0z>0 uniformly on the (compact) spectrum, so differentiation under the integral sign is justified. Since ∂zzλ+z=(λ+z)−z(λ+z)2=λ(λ+z)2, ∂ z zλ+z= (λ+z)-z(λ+z)^2= λ(λ+z)^2, the chain rule gives z∂zfz(k)=k(zλ+z)k−1⋅zλ(λ+z)2=k(zλ+z)k⋅λ+z=k(fz(k)−fz(k+1)),z\, ∂ zf_z^(k)=k ( zλ+z )^k-1· zλ(λ+z)^2=k ( zλ+z )^k· λ+z=k (f_z^(k)-f_z^(k+1) ), where the last step uses λ/(λ+z)=1−z/(λ+z)λ/(λ+z)=1-z/(λ+z). Integrating against μ gives equation 191. ∎ Definition 30 (Local exponent). For k≥1k≥ 1 and z>0z>0 with Nk(z)>0N_k(z)>0, α^k(z):=dlogNkdlogz=k(1−Nk+1(z)Nk(z)).\; α_k(z)\;:=\; d N_kd z\;=\;k (1- N_k+1(z)N_k(z) ).\; (192) The second equality is Lemma 4 divided by NkN_k. So α^k α_k is the instantaneous logarithmic slope of NkN_k, evaluated at a single z from two orders k and k+1k+1 of the same quadrature. Result 31 (Exactness on a pure power law). Suppose dF(x)=αcxα−1dxdF(x)=α c\,x^α-1dx on (0,∞)(0,∞) with α>0α>0. Then for every k>αk>α and every z>0z>0, Nk(z)=Cα,kczα,Cα,k=Γ(1+α)Γ(k−α)Γ(k),N_k(z)=C_α,k\,c\,z^α, C_α,k= (1+α)\, (k-α) (k), (193) and consequently α^k(z)=α α_k(z)=α exactly, for every k>αk>α and every z. Proof. Substituting λ=ztλ=zt in the definition of NkN_k, Nk(z)=∫0∞(zλ+z)kαcλα−1λ=αczα∫0∞tα−1(1+t)kt=αczαB(α,k−α),N_k(z)= _0^∞ ( zλ+z )^kα c\,λ^α-1\,dλ=α c\,z^α _0^∞ t^α-1(1+t)^k\,dt=α c\,z^αB(α,k-α), the Beta integral converging exactly when α>0α>0 and k−α>0k-α>0. With B(α,k−α)=Γ(α)Γ(k−α)/Γ(k)B(α,k-α)= (α) (k-α)/ (k) and αΓ(α)=Γ(1+α)α (α)= (1+α) this is equation 193. Then Cα,k+1Cα,k=Γ(k+1−α)Γ(k−α)⋅Γ(k)Γ(k+1)=k−αk, C_α,k+1C_α,k= (k+1-α) (k-α)· (k) (k+1)= k-αk, using Γ(w+1)=wΓ(w) (w+1)=w (w) twice, so by equation 192 α^k=k(1−(k−α)/k)=α α_k=k (1-(k-α)/k )=α. ∎ E.3.2 Distinguishing a power law from anything faster Remark 5 (Negative-moment dichotomy). Let μ be a probability measure on [0,∞)[0,∞) with μ(0)=0μ(\0\)=0 and F(x)=μ([0,x])F(x)=μ([0,x]). For k>0k>0 put M−k:=∫0∞λ−kμ(λ)∈(0,∞],αμ:=supk>0:M−k<∞∈[0,∞],M_-k:= _0^∞λ^-k\,dμ(λ)∈(0,∞], _μ:= \k>0:\ M_-k<∞\∈[0,∞], (194) and define the local slope at ratio ϱ>1 >1, sk(z):=log(Nk(ϱz)/Nk(z))logϱ.s_k(z):= (N_k( z)/N_k(z) ) . (195) Then: (i) Nk(z)=zkIk(z)N_k(z)=z^kI_k(z) with Ik(z):=∫(λ+z)−kμI_k(z):= (λ+z)^-k\,dμ increasing monotonically to M−kM_-k as z↓0z 0; (i) if k<αμk< _μ then Nk(z)∼M−kzkN_k(z) M_-kz^k and sk(z)→ks_k(z)→ k; (i) if F(x)∼cμxαF(x) c_μx^α with 0<α<∞0<α<∞ then αμ=α _μ=α, and for k>αk>α, sk(z)→αs_k(z)→α; (iv) consequently sk(z)→min(αμ,k)s_k(z)→ ( _μ,k); (v) αμ=∞ _μ=∞ if and only if F(x)=o(xn)F(x)=o(x^n) for every n. Proof. (i) Nk(z)=∫(z/(λ+z))kμ=zk∫(λ+z)−kμN_k(z)= (z/(λ+z) )^k\,dμ=z^k (λ+z)^-k\,dμ. For μ-a.e. λ>0λ>0 the integrand (λ+z)−k(λ+z)^-k increases to λ−kλ^-k as z↓0z 0; monotone convergence gives Ik(z)↑M−kI_k(z) M_-k. (i) If M−k<∞M_-k<∞ then Ik(z)→M−k∈(0,∞)I_k(z)→ M_-k∈(0,∞), so by (i) Nk(z)∼M−kzkN_k(z) M_-kz^k and sk(z)=k+log(Ik(ϱz)/Ik(z))/logϱ→ks_k(z)=k+ (I_k( z)/I_k(z) )/ → k, because both Ik(ϱz)I_k( z) and Ik(z)I_k(z) tend to the same finite positive limit. (Using the finite-difference slope avoids assuming NkN_k is differentiable, and is what the estimator actually computes.) (i) Integrating by parts, for 0<k0<k, ∫(0,1]λ−kμ=F(1)−limx↓0x−kF(x)+k∫01F(x)x−k−1x _(0,1]λ^-k\,dμ=F(1)- _x 0x^-kF(x)+k _0^1F(x)x^-k-1\,dx, so M−k<∞M_-k<∞ iff ∫01F(x)x−k−1x<∞ _0^1F(x)x^-k-1\,dx<∞. With F(x)∼cμxαF(x) c_μx^α the integrand is ≍xα−k−1 x^α-k-1, which is integrable at 00 exactly when k<αk<α; hence αμ=α _μ=α. The limit sk→αs_k→α for k>αk>α is Theorem 29. (iv) Combine (i) and (i): for k<αμk< _μ the slope tends to k, and for k>αμk> _μ it tends to αμ _μ. (v) If F(x)≤CnxnF(x)≤ C_nx^n near 00 for every n, then ∫01F(x)x−k−1x≤Cn∫01xn−k−1x<∞ _0^1F(x)x^-k-1\,dx≤ C_n _0^1x^n-k-1\,dx<∞ whenever n>kn>k, so M−k<∞M_-k<∞ for all k. Conversely if M−k<∞M_-k<∞ then the boundary term in the integration by parts must vanish, i.e. x−kF(x)→0x^-kF(x)→ 0, so F(x)=o(xk)F(x)=o(x^k); if this holds for every k then F vanishes faster than any power. ∎ Part (v) is what makes the lemma a test: “F decays faster than any power at the origin” is precisely the class containing a hard gap (F≡0F≡ 0 near 00), a Lifshitz tail F(x)∼e−a/xγF(x) e^-a/x^γ, and any exponentially small density. All of them have αμ=∞ _μ=∞, and by (iv) all of them make sk(z)→ks_k(z)→ k for every k. Corollary 32 (The ratio test). Fix k2>k1>0k_2>k_1>0 and μ(0)=0μ(\0\)=0, and define R(z):=sk2(z)/sk1(z)R(z):=s_k_2(z)/s_k_1(z). Then R(z)⟶1,F(x)∼cμxα,0<α≤k1,α/k1,F(x)∼cμxα,k1<α<k2,k2/k1,F(x)∼cμxα,α≥k2,k2/k1,αμ=∞.R(z) cases1,&F(x) c_μx^α, 0<α≤ k_1,\\ α/k_1,&F(x) c_μx^α, k_1<α<k_2,\\ k_2/k_1,&F(x) c_μx^α, α≥ k_2,\\ k_2/k_1,& _μ=∞. cases (196) The same ratio can therefore arise from a sufficiently large power-law exponent or from faster-than-power decay. E.3.3 Atoms at zero, and what a plateau means Corollary 33 (Kernel detection). For every k>0k>0, Nk(z)↓p0:=μ(0)N_k(z) p_0:=μ(\0\) as z↓0z 0, and L(T)↓p0L(T) p_0 as T↑∞T ∞. For a finite matrix p0=dimker(A)/Pp_0= (A)/P. An atom is incompatible with F(x)∼cμxαF(x) c_μx^α for α>0α>0, but it can coexist with a positive-spectrum edge F(x)−p0∼c+xαF(x)-p_0 c_+x^α. Proof. (1+λ/z)−k↓λ=0(1+λ/z)^-k 1_\λ=0\ pointwise and monotonically in z; apply monotone convergence. Same for e−λTe^-λ T. ∎ E.3.4 Generalization of the Gauss rule and Gauss-Radau brackets Given α1,…,αm,β1,…,βm _1,…, _m, _1,…, _m, we construct the Radau matrix as : T~m+1=(Tmβmemβmem⊤γ),γ:=βm2[Tm−1]mm, T_m+1= pmatrixT_m& _me_m\\ _me_m &γ pmatrix, γ:= _m^2\, [T_m^-1 ]_m, (197) the value of γ being exactly what forces detT~m+1=0 T_m+1=0, i.e. what puts an eigenvalue at λ=0λ=0. T~m+1 T_m+1 is (m+1)×(m+1)(m+1)×(m+1), so it has m+1m+1 eigenvalues, and these are the m+1m+1 Radau nodes. One of them is 00 by construction; the other m are the free nodes ϑ~1<⋯<ϑ~m _1<…< _m. So ϑ~0:=0,Rm(0)[g]=∑i=0mω~ig(ϑ~i)=ω~0g(0)⏟prescribed node+∑i=1mω~ig(ϑ~i)⏟m free nodes. _0:=0, R^(0)_m[g]= _i=0^m ω_ig( _i)= ω_0\,g(0)_prescribed node+ _i=1^m ω_ig( _i)_m free nodes. (198) All m+1m+1 weights belong to one family, computed identically from the eigenvectors u~i u_i of T~m+1 T_m+1: ω~i=(e1⊤u~i)2,i=0,1,…,m,∑i=0mω~i=1. ω_i= (e_1 u_i )^2, i=0,1,…,m, _i=0^m ω_i=1. (199) Result 34 (Gauss lower / Radau-at-zero upper bound). Let μv _v be supported in [0,λ1][0, _1] and let g∈C2m+1g∈ C^2m+1 on a neighbourhood of [0,λ1][0, _1] satisfy g(2m)≥0andg(2m+1)≤0on [0,λ1].g^(2m)≥ 0 g^(2m+1)≤ 0 [0, _1]. (200) Let Gm[g]=∑i=1mωig(ϑi)G_m[g]= _i=1^m _ig( _i) be the m-node Gauss rule of Theorem 24 and let Rm(0)[g]=∑i=0mω~ig(ϑ~i)R^(0)_m[g]= _i=0^m ω_ig( _i) be the (m+1)(m+1)-node Gauss–Radau rule with its lowest node prescribed at ϑ~0=0 _0=0. Then Gm[g]≤∫gdμv≤Rm(0)[g].G_m[g]\;≤\; g\,d _v\;≤\;R^(0)_m[g]. (201) Both functions gz,k(λ)=(zλ+z)kg_z,k(λ)= ( zλ+z )^k (z>0,k≥1)(z>0,k≥ 1) and gT(λ)=e−λTg_T(λ)=e^-λ T (T>0)(T>0) satisfy equation 200 for every m. Proof. Gauss side. Let h be the Hermite interpolant of g at the m Gauss nodes ϑi _i, each counted twice; degh≤2m−1 h≤ 2m-1, and the standard Hermite error formula gives, for each λ, g(λ)−h(λ)=g(2m)(η(λ))(2m)!∏i=1m(λ−ϑi)2g(λ)-h(λ)= g^(2m)(η(λ))(2m)! _i=1^m(λ- _i)^2 (202) for some η(λ)η(λ) in the convex hull of λ,ϑ1,…,ϑm⊂[0,λ1]\λ, _1,…, _m\⊂[0, _1]. By equation 200 the right side is ≥0≥ 0 for all λ∈[0,λ1]λ∈[0, _1]. By exactness (Theorem 24), ∫hdμv=Gm[h]=Gm[g] h\,d _v=G_m[h]=G_m[g], the last equality because h and g agree at the nodes. Therefore ∫gdμv−Gm[g]=∫(g−h)dμv≥0 g\,d _v-G_m[g]= (g-h)\,d _v≥ 0. Radau side. The Gauss–Radau rule with a prescribed node at 00 is exact for polynomials of degree ≤2m≤ 2m (it has m+1m+1 nodes, one of them fixed). Let h now be the Hermite interpolant of g at λ=0λ=0 (simple) and at the m free Radau nodes ϑ~i _i (double); degh≤2m h≤ 2m and g(λ)−h(λ)=g(2m+1)(η(λ))(2m+1)!λ∏i=1m(λ−ϑ~i)2.g(λ)-h(λ)= g^(2m+1)(η(λ))(2m+1)!\;λ _i=1^m(λ- _i)^2. (203) On suppμv⊂[0,λ1]supp _v⊂[0, _1] we have λ≥0λ≥ 0, so the product of λ and the square is ≥0≥ 0, while g(2m+1)≤0g^(2m+1)≤ 0; hence g−h≤0g-h≤ 0 μv _v-a.e. Exactness to degree 2m2m gives ∫hdμv=Rm(0)[h]=Rm(0)[g] h\,d _v=R^(0)_m[h]=R^(0)_m[g], whence ∫gdμv−Rm(0)[g]≤0 g\,d _v-R^(0)_m[g]≤ 0. (Measurability of λ↦η(λ)λ η(λ) is not needed: one applies equation 202–equation 203 pointwise to obtain the sign of g−hg-h, and integrates the inequality.) Sign conditions. For gz,kg_z,k, gz,k(n)(λ)=(−1)nzk(k)n(λ+z)k+ng_z,k^(n)(λ)=(-1)^nz^k (k)_n(λ+z)^k+n with (k)n=k(k+1)⋯(k+n−1)>0(k)_n=k(k+1)·s(k+n-1)>0, so g(n)g^(n) has sign (−1)n(-1)^n: even derivatives ≥0≥ 0, odd ≤0≤ 0, on λ>−z⊃[0,λ1]λ>-z⊃[0, _1]. For gTg_T, gT(n)(λ)=(−T)ne−λTg_T^(n)(λ)=(-T)^ne^-λ T, same alternation. Both are completely monotone, so equation 200 holds for every m. ∎ For each z the bracket equation 201 is computable from (αj,βj)j≤m( _j, _j)_j≤ m alone. Define the usable window as the set of z where the relative bracket width (Rm(0)[gz,k]−Gm[gz,k])/Rm(0)[gz,k] (R^(0)_m[g_z,k]-G_m[g_z,k] )/R^(0)_m[g_z,k] is below a chosen tolerance. Corollary 35 (Kernel bound). Letting z↓0z 0 in equation 201 with k=1k=1, μv(0)≤ω~0, _v(\0\)\ ≤\ ω_0, (204) the Radau weight at the prescribed node, so a depth-m run already bounds the kernel fraction from above. Proof. Gm[gz,1]=z∑iωi/(ϑi+z)→∑i:ϑi=0ωiG_m[g_z,1]=z _i _i/( _i+z)→ _i: _i=0 _i and Rm(0)[gz,1]→ω~0R^(0)_m[g_z,1]→ ω_0 as z↓0z 0; combine with equation 201 and Corollary 33. ∎ E.3.5 Why ghosts do not reach the hard edge In finite precision the Lanczos recurrence loses orthogonality as soon as a Ritz values accumulate, see Remark 4. Paige (1976); Paige (1980) showed that the resulting loss of orthogonality is triggered precisely when a Ritz value converges to the true eigenvalue, and occurs in the direction of the corresponding Ritz vector. Kaniel–Paige–Saad convergence theory (Saad, 1980) says that extremal well-separated eigenvalues converge fastest and hence by Paige’s argument leads to ghosts. Extremal means at either end of the spectrum - near λ1 _1 or near λP _P - as opposed to interior. How well-separated the bottom has to be is made precise in the next result. Remark 6 (Convergence at the bottom, and its cost under a hard edge). Let A=A⊤A=A have eigenvalues λ1≥⋯≥λP≥0 _1≥…≥ _P≥ 0 with unit eigenvectors uku_k, let v be a probe with uP⊤v≠0u_P v≠ 0, and let ϑm(m) ^(m)_m be the smallest Ritz value at depth m. Write γP:=λP−1−λPλ1−λP−1 _P:= _P-1- _P _1- _P-1 (205) for the relative gap at the bottom. Then: (i) Kaniel–Paige–Saad asserts (Kaniel, 1966; Saad, 1980; Parlett, 1980) 0≤ϑm(m)−λP≤(λ1−λP)tan2∠(v,uP)Tm−1(1+2γP)2.0\;≤\; ^(m)_m- _P\;≤\; ( _1- _P)\, ^2 (v,u_P)T_m-1 (1+2 _P )^2. (206) (i) For γP≪1 _P 1, Tm−1(1+2γP)=12exp(2(m−1)γP)(1+O(γP))T_m-1(1+2 _P)= 12 (2(m-1) _P ) (1+O( _P) ), so driving the right-hand side of equation 206 below a tolerance δ requires m≳ 1+14γPlog4(λ1−λP)tan2∠(v,uP)δ,i.e.m∼γP−1/2.m\; \;1+ 14 _P\, 4( _1- _P) ^2 (v,u_P)δ, .e. m\; \; _P^-1/2. (207) (i) If in addition μ([0,x])∼cμxαμ([0,x]) c_μx^α as x↓0x 0, then the k-th smallest eigenvalue satisfies λ(k)≃(k/cμP)1/α _(k) (k/c_μP )^1/α, whence, in units λ1=1 _1=1, γP≃(21/α−1)(cμP)−1/αandm∼γP−1/2=Θ(P1/(2α)). _P\; \; (2^1/α-1 ) (c_μP )^-1/α m\; \; _P^-1/2=\; (P^1/(2α) ). (208) The exponent in equation 208 is the point: the smaller the hard-edge exponent, slower the convergence and lower the chance of ghosting for small eigenvalues. Numerically, with P=1.68×108P=1.68× 10^8 and cμ=1c_μ=1: for example at α≈0.33α≈ 0.33 the value of m required to converge the smallest eigenvalue is order 101210^12. The same bound at the top, where an isolated λ1 _1 has a relative gap of order unity, gives m=O(log(1/δ))m=O( (1/δ)): hundreds of steps. This asymmetry is what may put the ghosts exclusively at the top. We check this for m=4600m=4600, λ/λ1λ/ _1 band nodes duplicate pairs (<10−8<10^-8) fraction 10−110^-1–10010^0 3377 2261 67%67\% 10−210^-2–10−110^-1 847 0 0%0\% 10−310^-3–10−210^-2 257 0 0%0\% 10−410^-4–10−310^-3 81 0 0%0\% 10−710^-7–10−410^-4 37 0 0%0\% So ghosting is confined to λ>0.1λ1λ>0.1 _1. Now we turn to make this observation formal. Assumption 36 (Ghost localization). Let μv _v be the exact probe measure and μ^v μ_v the measure of the computed Jacobi matrix. Assume there are Λ>0 >0, s≥0s≥ 0 and a transport plan π of μv _v and μ^v μ_v supported on (x,x):x≥0∪(x,y):x,y≥Λ,|x−y|≤s,\(x,x):x≥ 0\\ ∪\ \(x,y):x,y≥ ,\ x-y ≤ s\, (209) i.e. mass is either left untouched or moved by at most s within [Λ,∞)[ ,∞). A transport plan of two probability measures μ,νμ,ν on ℝ≥0R_≥ 0 is a probability measure π on the product ℝ≥0×ℝ≥0R_≥ 0×R_≥ 0 whose two marginals are μ and ν: π(B×ℝ≥0)=μ(B),π(ℝ≥0×B)=ν(B)for every Borel set B.π (B×R_≥ 0 )=μ(B), π (R_≥ 0× B )=ν(B) every Borel set B. (210) Couplings always exist - the product μ⊗νμ ν is one - so the content of Assumption 36 is not existence but the restriction on where π is allowed to put its probability mass. That restriction is equation 209. Being supported on the union of those two sets means every unit of mass does one of exactly two things: (a) it lies on the diagonal (x,x)\(x,x)\ - it is not moved at all; or (b) it lies in (x,y):x,y≥Λ,|x−y|≤s\(x,y):x,y≥ ,\ x-y ≤ s\ - it is moved, but by a distance of at most s, and both its origin and destination are above Λ . There is no third option, and in particular no mass may cross from below Λ to above it or vice versa, and no mass below Λ may move at all. Applied to μv _v and μ^v μ_v, this is the formalisation of ghosting splits converged Ritz values into tight clusters at the top of the spectrum, and leaves the bottom alone: the cluster spread is s, and Λ is the scale above which converged values are found. Proposition 37 (Transport bound). Under Assumption 36, for any g∈C1([0,∞))g∈ C^1([0,∞)), |∫gdμ^v−∫gdμv|≤ssupλ≥Λ|g′(λ)|. g\,d μ_v- g\,d _v \;≤\;s\, _λ≥ g (λ) . (211) Proof. Writing the difference through the coupling, ∫gdμ^v−∫gdμv=∬(g(y)−g(x))π(x,y) g\,d μ_v- g\,d _v= (g(y)-g(x) )\,dπ(x,y). By the mean value theorem, for each (x,y)(x,y) in the support of π there is ξ between x and y with g(y)−g(x)=g′(ξ)(y−x)g(y)-g(x)=g (ξ)(y-x); both x,y≥Λx,y≥ , so ξ≥Λξ≥ , and |y−x|≤s y-x ≤ s. Hence the integrand is bounded by ssupλ≥Λ|g′|s _λ≥ g pointwise, and π is a probability measure. ∎ Corollary 38 (Ghosts are harmless at the hard edge). Take g=gz,kg=g_z,k, gz,k(λ)=(z/(λ+z))kg_z,k(λ)= (z/(λ+z) )^k. Then gz,k′(λ)=−kzk/(λ+z)k+1g_z,k (λ)=-kz^k/(λ+z)^k+1 is negative and increasing, so its supremum over [Λ,∞)[ ,∞) is attained at Λ and |N^k(z)−Nk(z)|≤skzk(Λ+z)k+1≤skΛk+1zk. N_k(z)-N_k(z) \;≤\; s\,k\,z^k( +z)^k+1\;≤\; s\,k ^k+1\;z^k. (212) If in addition μ obeys the hard-edge law of Theorem 29, so that Nk(z)∼cμΓ(α+1)Γ(k−α)Γ(k)zαN_k(z) c_μ (α+1) (k-α) (k)z^α, then the relative error obeys |N^k(z)−Nk(z)|Nk(z)≤skΓ(k)cμΓ(α+1)Γ(k−α)Λk+1zk−α→z↓0 0whenever k>α. N_k(z)-N_k(z) N_k(z)\;≤\; s\,k\, (k)c_μ\, (α+1) (k-α)\, ^k+1\;z^\,k-α\; [z 0]\;0 k>α. (213) The condition k>αk>α is not an extra hypothesis: it is precisely the condition already required for the exponent to transfer at all (Theorem 29, and the k-saturation diagnostic of Remark 5). With the measured ghost spread s≃10−8λ1s 10^-8 _1 and localisation Λ=0.1λ1 =0.1 _1 (§E.4), k=2k=2, and the measured α≈0.3α≈ 0.3, the bound equation 213 with cμ=1c_μ=1 is 1.6×10−151.6× 10^-15 at z=10−6λ1z=10^-6 _1 and 1.2×10−201.2× 10^-20 at z=10−9λ1z=10^-9 _1. Remark 6 (What this argument does not cover). Corollary 38 controls perturbations localised above Λ . It says nothing about rounding that perturbs the spectrum near zero, where gz,kg_z,k varies fastest: there sup|gz,k′|=k/z g_z,k =k/z, and a perturbation of size δA _A contributes a relative error ∼kδA/(cμz1+α) k _A/(c_μz^1+α), which diverges as z↓0z 0. Setting that against unity gives the floor zmin/λ1≳(δA/λ1)1/(1+α).z_ / _1\; \;( _A/ _1)^1/(1+α). (214) It is essential that δA _A here is the measured defect of the computed operator. With the measured δA/λ1=8.5×10−12 _A/ _1=8.5× 10^-12 and α≈0.3α≈ 0.3, equation 214 gives zmin≈3×10−9λ1z_ ≈ 3× 10^-9 _1. E.4 Measurements All numbers below are our own measurements on a retrained 150150M checkpoint (OLMo-style, P=1.677×108P=1.677× 10^8, 33B tokens at 1×1× Chinchilla, B=64B=64, seed 00). The operator is the Adam-preconditioned Gauss–Newton matrix equation 156 at the final checkpoint, built from 10610^6 tokens, probed in float64 with a random probe. At m=9400m=9400 (Figure 11): z/λ1z/ _1 10−510^-5 3.2×10−63.2× 10^-6 10−610^-6 3.2×10−73.2× 10^-7 10−710^-7 3.2×10−83.2× 10^-8 1.9×10−81.9× 10^-8 s1s_1 0.0510.051 0.0820.082 0.1240.124 0.1750.175 0.2320.232 0.2790.279 [0.288,0.290][0.288,0.290] s2s_2 0.0750.075 0.1160.116 0.1710.171 0.2340.234 0.2950.295 [0.318,0.320][0.318,0.320] [0.293,0.304][0.293,0.304] R2,1R_2,1 1.471.47 1.421.42 1.381.38 1.331.33 1.271.27 [1.14,1.15][1.14,1.15] [1.01,1.06][1.01,1.06] Down to 10−7λ110^-7 _1 every bracket is tighter than the three digits shown; the widest is 3.7×10−63.7× 10^-6, on s2s_2 at 10−7λ110^-7 _1. The ratio brackets follow from the slope brackets as Rlo=sk2,lo/sk1,hiR_lo=s_k_2,lo/s_k_1,hi, Rhi=sk2,hi/sk1,loR_hi=s_k_2,hi/s_k_1,lo, the widest interval consistent with both. R2,1R_2,1 decreases monotonically from 1.471.47 at 10−5λ110^-5 _1 to [1.01,1.06][1.01,1.06] at the deepest plotted scale, against a faster-than-power limit of 22. A faster-than-power edge would instead drive the ratio towards its limit k2/k1=2k_2/k_1=2. Hence, on the window resolved, a faster-than-power edge is strongly disfavoured, and the behaviour is power-law-like. Theorem 29 makes every sks_k tend to α. Hence α≈0.30α≈ 0.30. Figure 11: The plot on the left shows that heavy tailed curvature spectrum exists and the plot on the right determines the exponent. Smaller z probes a smaller eigenvalue spectrum. Reliable window. Ghost effects are captured in Corollary 38: skzk/(Λ+z)k+1skz^k/( +z)^k+1 with the measured s≤10−8λ1s≤ 10^-8 _1 and Λ=0.1λ1 =0.1 _1. It is negligible: it never exceeds 8×10−158× 10^-15 and falls steeply as z decreases. The next concern is the operator floor in Remark 6. Determinism is a recomputation of the same product: in float64 the operator is deterministic to round-off, so a fixed operator exists; symmetric to 10−1510^-15, so the Jacobi matrix is the Jacobi matrix of a genuine positive measure. The linearity defect δA/λ1=8.5×10−12 _A/ _1=8.5× 10^-12 is the largest of the three and is the number that propagates into the sensitivity bound |ΔNk|≤kδA/z N_k ≤ k _A/z, hence into a floor on z that no amount of Lanczos depth can lower. Propagated to the local slope at the tolerance of 0.010.01 used throughout, k=2k=2 operator floor is reached at z/λ1=1.860×10−8z/ _1=1.860× 10^-8. check quantity float64 float32 determinism ‖A(1)(v)−A(2)(v)‖/‖A(1)(v)‖ A^(1)(v)-A^(2)(v) / A^(1)(v) 7.6×10−187.6× 10^-18 4.0×10−94.0× 10^-9 symmetry |⟨u,Av⟩−⟨v,Au⟩|/(‖u‖‖v‖λ1) u,Av - v,Au /( u v _1) 8.6×10−168.6× 10^-16 3.5×10−113.5× 10^-11 linearity ‖A(u+v)−Au−Av‖/(λ1‖u+v‖) A(u+v)-Au-Av /( _1 u+v ) 8.5×10−128.5× 10^-12 2.3×10−72.3× 10^-7 positivity ⟨v,Av⟩/(‖v‖2λ1) v,Av /( v ^2 _1) +1.47×10−5+1.47× 10^-5 +1.47×10−5+1.47× 10^-5 At iteration depth m=9400m=9400 and z/λ1=2.64×10−8z/ _1=2.64× 10^-8, the k=2k=2 total error reaches the tolerance as a result of interpolating the data from the table below: z/λ^1z/ λ_1 quadrature operator ghosting total 2.196174×10−82.196174× 10^-8 6.728678×10−36.728678× 10^-3 8.103861×10−38.103861× 10^-3 1.0×10−191.0× 10^-19 0.0148330.014833 2.818383×10−82.818383× 10^-8 2.855708×10−32.855708× 10^-3 5.878096×10−35.878096× 10^-3 1.5×10−191.5× 10^-19 0.0087340.008734 In the above table, the operator term converted from a bound on the count to a bound on the local slope, at z/λ1=2.818×10−8z/ _1=2.818× 10^-8, k=2k=2, with δA/λ1=8.5×10−12 _A/ _1=8.5× 10^-12, N2=0.2233N_2=0.2233 and ϱ=100.4 =10^0.4. Recall sk=[lnNk(ϱz)−lnNk(z)]/lnϱs_k=[ N_k( z)- N_k(z)]/ - using twice the error at the deeper endpoint, we get operator error on the slope to be 2lnϱkδA/zN2=5.8781×10−3. 2 \, k _A/zN_2=5.8781× 10^-3. It is conservative estimate, since both kδA/(ϱz)<kδA/zk _A/( z)<k _A/z and Nk(ϱz)>Nk(z)N_k( z)>N_k(z). The hard-edge law F(x)≃cμxαF(x) c_μx^α can hold only below the crossover λ0 _0, and normalisation forces cμλ0α≲1c_μ _0^α 1, so the crossover sits where the count is of order unity. Any threshold Nk=Θ(1)N_k= (1) marks that boundary to within a factor in z, we choose Nk=1/2N_k=1/2. At a tolerance of 0.010.01 on the slope, this demands z/λ1< 4.21×10−7z/ _1<\ 4.21× 10^-7. Figure 12: Every mechanism that limits the measurement performed. The choice of the slope cut-off so far is arbitrary; to make the discussion of a reliable window precise, we introduce: Definition 39 (Effective node count). Let (θj,ωj)j=1m( _j, _j)_j=1^m be the nodes and weights of the m-node Gauss rule, and for z>0z>0 let J(z)=j:θj≤zJ(z)=\j: _j≤ z\ with q(z)=|J(z)|q(z)=|J(z)|. Define neff(z)=(∑j∈J(z)ωj)2∑j∈J(z)ωj2,neff(z):=0 if J(z)=∅.n_eff(z)\;=\; ( _j∈ J(z) _j )^2 _j∈ J(z) _j^2, n_eff(z):=0 if J(z)= . (215) This is a participation ratio: it counts the nodes below z, discounted by how unevenly they share the mass. It is preferred to the raw count q(z)q(z) because q is discontinuous and is inflated by nodes of negligible weight, whereas equation 215 is invariant under rescaling all ωj _j and ignores a node whose weight is vanishing. Proposition 40 (Range and extremes). For every z with q(z)≥1q(z)≥ 1, 1≤neff(z)≤q(z),1\;≤\;n_eff(z)\;≤\;q(z), (216) with neff(z)=q(z)n_eff(z)=q(z) if and only if all ωj _j, j∈J(z)j∈ J(z), are equal, and neff(z)=1n_eff(z)=1 if and only if at most one of them is non-zero — in particular exactly when q(z)=1q(z)=1. Proof. The weights of a Gauss rule are strictly positive. Write S1=∑JωjS_1= _J _j and S2=∑Jωj2S_2= _J _j^2. By Cauchy–Schwarz applied to the vectors (ωj)( _j) and (1,…,1)(1,…,1), S12≤qS2S_1^2≤ q\,S_2, with equality iff the ωj _j are proportional to 11, i.e. all equal; this is the upper bound. For the lower bound, expanding S12=∑jωj2+∑i≠jωiωj=S2+∑i≠jωiωjS_1^2= _j _j^2+ _i≠ j _i _j=S_2+ _i≠ j _i _j and using ωiωj>0 _i _j>0 gives S12≥S2S_1^2≥ S_2, with equality iff every cross term vanishes, i.e. at most one weight is non-zero. Since all weights are positive, that happens exactly when q(z)=1q(z)=1. ∎ Now we turn to discuss neff<2n_eff<2 is the edge of validity, and neff=1n_eff=1 the strict end of reliable window. Split NkN_k at z into the part carried by nodes below z and the part carried by nodes above: Nk(z)=∑θj≤zωj(zθj+z)k⏟=:Bk(z)+∑θj>zωj(zθj+z)k⏟=:Tk(z).N_k(z)= _ _j≤ z _j ( z _j+z )^k_=:B_k(z)\;+\; _ _j>z _j ( z _j+z )^k_=:T_k(z). (217) For θ≪zθ z the kernel is ≈1≈ 1 independently of k, so Bk+1≈Bk=:B_k+1≈ B_k=:B: the sub-z mass behaves like an atom at the origin, contributing logarithmic slope 00. For θ≫zθ z the kernel is ≈(z/θ)k≈(z/θ)^k, so Tk+1≪TkT_k+1 T_k and that part contributes slope k. Substituting into Definition 30, α^k(z)≈kTk(z)B(z)+Tk(z), α_k(z)\;≈\;k\, T_k(z)B(z)+T_k(z), (218) an approximation valid when the two groups are well separated from z. So α^k α_k is a mixture: the value it reports is set by how the rule has apportioned mass between an atom-like term and a sloping term. This is what makes the node count decisive, and it separates the two thresholds. neff=1n_eff=1: strictly unreliable. By Proposition 40 this happens exactly when a single node lies below z. Everything the rule knows about μ|[0,z]μ|_[0,z] is then one pair (θ1,ω1)( _1, _1) — two real numbers, a mass and a location. But the hard-edge exponent is a statement about the shape of μ near the origin, F(x)∼cxαF(x) c\,x^α, and a single atom has no shape: it fixes B in equation 218 at an unresolved value and carries no information about α at all. Below θ2 _2 the curve is therefore not merely noisy, it is uninformative about the quantity being measured, whatever its bracket says. The Gauss/Radau bracket does not protect against this: it bounds the integral Nk(z)N_k(z), not the shape of the measure beneath it, and it can be tight while the shape is entirely unresolved. neff<2n_eff<2: the edge of validity. Two nodes below z give four numbers, enough to represent a mass, a spread and a relative weight, so μ|[0,z]μ|_[0,z] is no longer forced to be a point mass and B in equation 218 responds to the shape of the measure. This is the minimum configuration in which α^k α_k can carry exponent information at all, which is why it is the edge of validity rather than a complete failure - the estimate is supported, but by the least possible amount. This regime is exactly 1≤neff<21≤ n_eff<2. For more details see Figure 11, 12.