Paper deep dive
From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon
Jihwan Kim, Dogyoon Song, Chulhee Yun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/9/2026, 8:37:31 AM
Summary
This paper theoretically demonstrates that row-wise normalization in optimizers (e.g., RowGD, RowGDM) achieves strictly higher population accuracy than full-batch Adam and exact-SVD Muon in high-dimensional multiclass classification. The advantage stems from row normalization's class-wise Euclidean geometry, which asymptotically preserves population decision-boundary directions, whereas Adam's coordinate-wise and Muon's spectral geometries introduce nonvanishing distortions. This result holds under isotropic Gaussian-cloud models and extends to certain anisotropic settings.
Entities (6)
Relation Signals (6)
RowGD → isvariantof → Row Normalization
confidence 95% · Row-normalized GD... outperform Adam... Theorem 3.1.
Row Normalization → outperforms → Adam
confidence 95% · We show that row-wise normalization can achieve strictly higher population accuracy than full-batch Adam... in high-dimensional multiclass classification.
Row Normalization → outperforms → Muon
confidence 95% · We show that row-wise normalization can achieve strictly higher population accuracy than... exact-SVD Muon in high-dimensional multiclass classification.
Adam → introduces → coordinate-wise geometry distortions
confidence 90% · Adam's coordinate-wise geometry... introduce nonvanishing distortions.
Muon → introduces → spectral geometry distortions
confidence 90% · Muon's spectral geometry introduce nonvanishing distortions.
Row Normalization → preserves → population decision-boundary directions
confidence 90% · row normalization's class-wise Euclidean geometry asymptotically preserves the population decision-boundary directions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Different optimizers can fit the same training data while selecting classifiers with substantially different geometries, but whether this difference provably affects population performance remains unclear. We show that row-wise normalization can achieve strictly higher population accuracy than full-batch Adam, a proxy for random-reshuffling Adam, and exact-SVD Muon in high-dimensional multiclass classification. Under an isotropic Gaussian-cloud data model, this advantage arises because row normalization's class-wise Euclidean geometry asymptotically preserves the population decision-boundary directions, whereas Adam's coordinate-wise geometry and Muon's spectral geometry introduce nonvanishing distortions. Beyond isotropy, the advantage persists for full-batch training on class means with independently oriented class-mean and test-noise covariances. It holds for power-law spectra with class-mean exponent below one, even under heavily anisotropic test noise. When both covariances are diagonal and sufficiently close, the advantage over Adam can reverse, while applying the same random rotation to both restores it by changing only their alignment with Adam's coordinate axes. Synthetic and last-layer language-model experiments support the predicted advantage.
Tags
Links
- Source: https://arxiv.org/abs/2610.11309v1
- Canonical: https://arxiv.org/abs/2610.11309v1
Trouble viewing inline? Open PDF directly →
Full Text
346,678 characters extracted from source content.
Expand or collapse full text
From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon Jihwan Kim Affiliation: Seoul National University Affiliation: KAIST InnoCORE LLM Dogyoon Song Affiliation: University of California, Davis Chulhee Yun Affiliation: KAIST InnoCORE LLM Affiliation: KAIST Abstract Different optimizers can fit the same training data while selecting classifiers with substantially different geometries, but whether this difference provably affects population performance remains unclear. We show that row-wise normalization can achieve strictly higher population accuracy than full-batch Adam, a proxy for random-reshuffling Adam, and exact-SVD Muon in high-dimensional multiclass classification. Under an isotropic Gaussian-cloud data model, this advantage arises because row normalization’s class-wise Euclidean geometry asymptotically preserves the population decision-boundary directions, whereas Adam’s coordinate-wise geometry and Muon’s spectral geometry introduce nonvanishing distortions. Beyond isotropy, the advantage persists for full-batch training on class means with independently oriented class-mean and test-noise covariances. It holds for power-law spectra with class-mean exponent below one, even under heavily anisotropic test noise. When both covariances are diagonal and sufficiently close, the advantage over Adam can reverse, while applying the same random rotation to both restores it by changing only their alignment with Adam’s coordinate axes. Synthetic and last-layer language-model experiments support the predicted advantage. aqua4689@snu.ac.kr dgsong@ucdavis.edu chulhee.yun@kaist.ac.kr 1 Introduction Row-wise normalization has recently emerged as a simple and effective ingredient in optimizers for large language model training.11 1 We use W∈ℝdout×dinW ^d_out× d_in; under the transposed convention, row-wise becomes column-wise normalization. Scale (Glentis et al., 2026) uses row-normalized SGD as its base method and adds momentum only to the final layer, reporting performance comparable to or better than Adam (Kingma and Ba, 2015) and Muon (Jordan et al., 2024) while using substantially less optimizer-state memory.22 2 Scale uses the transposed weight convention and thus calls the same operation column-wise normalization. Related row-level normalization operations also appear in recent methods such as SinkGD, NorMuon, RMNP, and MOGA (Scetbon et al., 2025; Li et al., 2026b; Deng et al., 2026; Xu et al., 2026). Together, these results provide growing empirical evidence that normalizing updates separately across output units can be beneficial for large-scale optimization. Yet what drives the strong empirical performance of row-normalized methods remains poorly understood. A natural hypothesis is that this behavior is tied to the geometry induced by the optimizer. Recent implicit-bias results make this connection concrete: normalized steepest-descent methods can converge toward max-margin classifiers associated with their underlying norm geometry (Soudry et al., 2018; Gunasekar et al., 2018b; Fan et al., 2025; Li et al., 2026a; Baek et al., 2026). Under the implicit-bias characterizations used in this work, row-normalized methods are associated with the row norm33 3 In our paper, row norm means ∥W∥row:=maxi∥Wi,:∥2\|W\|_row:= _i\|W_i,:\|_2., Muon with the spectral norm, and full-batch Adam with the entrywise max-norm. These distinct geometries can therefore lead the methods toward classifiers with substantially different decision boundaries, even when they all fit the training data well. What remains unclear is whether these geometric differences translate into systematic differences in population performance. This motivates our central question: can row-wise normalization provably achieve higher population accuracy than competing optimizers, and if so, what geometric mechanism explains the advantage? Existing theory does not answer this question. The closest results analyze row-normalized methods through nonconvex convergence or width-scaling properties, but do not establish that they select statistically better solutions than competing optimizers (Deng et al., 2026; Xu et al., 2026). To the best of our knowledge, we provide the first such result. We analyze the statistical quality of the limiting classifiers selected by different optimizer geometries and establish a strict population-accuracy advantage for row-wise normalization in high-dimensional Gaussian data models. Our analysis also identifies the geometric mechanism behind this advantage: the class-wise Euclidean geometry of row-wise normalization better preserves the population decision-boundary directions, whereas coordinate-wise and spectral geometries introduce systematic directional distortions. Figure 1: Geometry of the learned classifiers. Class means are compared with classifier rows learned by row-normalized GD, Adam, and Muon for 60 classes in 20 dimensions: a 2D projection of two classes, with mean angles over all classes and their proportional-limit predictions from Proposition 3.2 (left); coordinate magnitudes at ranks 1,3,…,191,3,…,19 after sorting by class-mean magnitude (middle); and singular values at indices 1,3,…,191,3,…,19 of the full matrices (right). 1.1 Our Contributions We study how optimizer geometry affects population accuracy in high-dimensional multiclass classification. Our main contributions are the following. • Population-accuracy ordering under an isotropic Gaussian-cloud data model. We first establish a strict population-accuracy advantage of row-normalized optimizers over representative spectral, coordinate-wise, and adaptive alternatives under our isotropic Gaussian-cloud data model. As an ingredient, we extend existing normalized steepest-descent implicit-bias analyses (Fan et al., 2025; Li et al., 2026a) to the row norm. Specifically, we show the advantage in three settings as follows. – Deterministic full-batch setting. Row-normalized GD and row-normalized GD with momentum outperform Adam, SignGD, Signum, exact-SVD Muon, and Spectral-GD (Theorem 3.1). – Stochastic random-reshuffling setting. Under random reshuffling, row-normalized SGD with momentum outperforms stochastic Signum and stochastic exact-SVD Muon (Proposition 3.4). – Random-reshuffling Adam via AdamProxy. We extend the AdamProxy framework of Baek et al. (2026) from incremental Adam to random-reshuffling mini-batch Adam. We prove that the resulting proxy approximates the one-epoch displacement of random-reshuffling Adam in the β1,β2→1 _1, _2→ 1 regime (Proposition 3.5) and show that row-normalized SGD with momentum achieves strictly higher population accuracy than AdamProxy (Proposition 3.6). • Extension beyond isotropy. We show that the advantage of row-normalized methods is not limited to isotropic data distributions. For deterministic full-batch training on the class means, we consider class-mean and test-noise covariances with independent random eigenbases and power-law spectra, allowing any class-mean exponent below one and any test-noise exponent. Under these conditions, we prove that the row-normalized methods continue to outperform Adam and Muon (Theorem 4.2). We further exhibit a coordinate-aligned anisotropic regime in which row-normalized methods are worse than Adam, and show that jointly rotating the class means and test-noise covariance by an independently sampled Haar matrix restores the advantage with high probability (Proposition 4.5). 1.2 Related Work Here we discuss only the works most directly related to our problem formulation and proof strategy; additional related work is deferred to Appendix A. Row-normalized optimizers in practice. Recent LLM optimizers use output-wise row normalization as a principal matrix update in Scale, RMNP, and MOGA; as one step of alternating row–column normalization in SinkGD; and as row-wise adaptive rescaling after Muon orthogonalization in NorMuon (Glentis et al., 2026; Scetbon et al., 2025; Li et al., 2026b; Deng et al., 2026; Xu et al., 2026). These works establish row normalization as a practical optimizer component, whereas we theoretically compare the quality of the solutions it selects against competing optimizer geometries. Theoretical comparisons involving row normalization. RMNP proves nonconvex convergence guarantees for row-normalized momentum methods that match existing guarantees for Muon (Deng et al., 2026). MOGA compares the width dependence of smoothness and learning-rate transfer across matrix geometries, showing favorable scaling for row-normalized methods (Xu et al., 2026). Implicit bias and optimizer-selected solutions. Prior work characterizes the norm-induced max-margin directions of full-batch SignGD, Spectral-GD, Muon, and Adam in separable multiclass linear classification (Fan et al., 2025). This framework has been extended to random-reshuffling stochastic optimization, where batch size and momentum govern approximation to the full-batch margin (Li et al., 2026a). Separately, Baek et al. (2026) show that incremental Adam can have a distinct stochastic implicit bias and introduce an AdamProxy characterized by a data-dependent Mahalanobis margin problem. We add the row norm and extend AdamProxy to multiclass random reshuffling. 2 Preliminaries Notation. For k≥2k≥ 2, let [k]:=1,…,k[k]:=\1,…,k\ and e1,…,eke_1,…,e_k be the standard basis of ℝkR^k. For positive integers b,Nb,N, we write b|Nb N if N is divisible by b. For matrices, ⟨A,B⟩:=tr(A⊤B) A,B :=tr(A B), ‖A‖max:=maxr,j|Ar,j|\|A\|_ := _r,j|A_r,j|, and ‖A‖sum:=∑r,j|Ar,j|\|A\|_sum:= _r,j|A_r,j|; we use ‖A‖op\|A\|_op and ‖A‖F\|A\|_F for the spectral and Frobenius norms. For W∈ℝk×dW ^k× d with rows wr⊤w_r , define ‖W‖row:=‖W‖2,∞:=maxr∈[k]‖wr‖2\|W\|_row:=\|W\|_2,∞:= _r∈[k]\|w_r\|_2 as a row norm. For a∈ℝka ^k and c∈[k]c∈[k], define c(a):=eac/∑ℓ=1keaℓS_c(a):=e^a_c/ _ =1^ke^a_ , the c-th component of the softmax map. We write A⊙pA p for entrywise powers and ⊘ for entrywise division, with 0/0:=00/0:=0. For nonzero vectors u,vu,v, let ∠(u,v)∈[0,π] (u,v)∈[0,π] denote the angle between u and v. Write v^:=v/‖v‖2 v:=v/\|v\|_2 for v≠v≠0. For random quantities indexed by d, oℙ(1)o_P(1) denotes a term that converges to zero in probability as d→∞d→∞. Data model. We collect the structural assumptions on the data distribution in the following Gaussian-cloud data model; asymptotic and optimization conditions appear separately below. Let d be the input dimension, k the number of classes, and nin_i the class-i sample size, N:=∑i=1kniN:= _i=1^kn_i and nmin:=mininin_ := _in_i. Assumption 2.1 (Isotropic Gaussian-cloud data model). The training dataset consists of (xi,s,i)(x_i,s,i) for i∈[k]i∈[k] and s∈[ni]s∈[n_i] such that xi,s:=μi+rzi,s,whereμi∼iid(,Id),zi,s∼iid(,Id).x_i,s:= _i+rz_i,s, _i iid N(0,I_d), z_i,s iid N(0,I_d). (1) A test example XteX^te with label Y is Xte:=μY+rtezte,ℙ(Y=i)=πi,d,zte∼(,Id),X^te:= _Y+r_tez^te, (Y=i)= _i,d, z^te (0,I_d), (2) where (μi)i=1k( _i)_i=1^k, (zi,s)i,s(z_i,s)_i,s, Y, and ztez^te are mutually independent. Here, r=r(d)≥0r=r(d)≥ 0, rte=rte(d)>0r_te=r_te(d)>0, πi,d>0 _i,d>0, and ∑i=1kπi,d=1 _i=1^k _i,d=1. We also use the flattened notation (xn,yn)n=1N\(x_n,y_n)\_n=1^N for the same training set, where xn=xi,sx_n=x_i,s and yn=iy_n=i under some fixed bijection between (i,s):i∈[k],s∈[ni]\(i,s):i∈[k],\,s∈[n_i]\ and [N][N]. Assumption 2.2 (High-dimensional scaling). Along a sequence of models indexed by d, with k=k(d)k=k(d), ni=ni(d)n_i=n_i(d), and πmin,d:=miniπi,d _ ,d:= _i _i,d, k/d→α∈(0,∞),nmin→∞,logN=o(d),log(1/πmin,d)=o(d).k/d→α∈(0,∞), n_ →∞, N=o(d), (1/ _ ,d)=o(d). (3) Furthermore, r0:=supdr(d)<∞r_0:= _dr(d)<∞ and rte,0:=supdrte(d)<∞r_te,0:= _dr_te(d)<∞. We focus on a proportional regime in which k and d remain on the same scale, excluding settings where one asymptotically dominates the other; this is qualitatively compatible with language-model last layers. The condition logN=o(d) N=o(d) rules out datasets exponential in d, so that the geometry is not dominated by rare deviations accumulated over exponentially many samples. Meanwhile, log(1/πmin,d)=o(d) (1/ _ ,d)=o(d) allows class frequencies following Zipf’s law, as observed in language data. Classifier and loss. A linear multiclass classifier is W∈ℝk×dW ^k× d, with score vector WxWx at x∈ℝdx ^d. We train with multiclass cross-entropy L(W):=−1N∑i=1k∑s=1nilogi(Wxi,s).L(W):=- 1N _i=1^k _s=1^n_i _i(Wx_i,s). (4) This standard setting (Fan et al., 2025; Li et al., 2026a) abstracts a language model’s final linear layer. Population accuracy. We measure population performance by the probability that the true class attains the strictly largest linear score on a test example: Accπ,d(W):=∑i=1kπi,dℙ(wi⊤Xte>wj⊤Xtefor all j∈[k]∖i∣Y=i).Acc_π,d(W):= _i=1^k _i,d\,P\! (w_i X^te>w_j X^te\ for all j∈[k] \i\ Y=i ). (5) Norm-based update maps. We use three norm-based update maps. For the row norm ψrow:=∥⋅∥row _row:=\|·\|_row, set (Δrow(G))r,::=Gr,:/∥Gr,:∥2( _row(G))_r,::=G_r,:/\|G_r,:\|_2 when Gr,:≠G_r,:≠0, and (Δrow(G))r,::=( _row(G))_r,::=0 otherwise. For the entrywise max-norm ψmax:=∥⋅∥max _ :=\|·\|_ , let Δmax(G):=sign(G) _ (G):=sign(G) entrywise, with sign(0):=0sign(0):=0. For the spectral norm ψsp:=∥⋅∥op _sp:=\|·\|_op, if G≠G≠0, let G=UΣV⊤G=U V be a compact SVD and define Δsp(G):=UV⊤ _sp(G):=UV ; set Δsp():= _sp(0):=0. For each g∈row,sp,maxg∈\row,sp, \, these choices satisfy Δg(G)∈argmaxψg(Δ)≤1⟨G,Δ⟩ _g(G)∈ *arg\,max_ _g( )≤ 1 G, , although the maximizer need not be unique. Full-batch and random-reshuffling optimizers. For random reshuffling (R), assume b|Nb N, set m:=N/bm:=N/b, and independently permute and partition the data into m mini-batches of size b each epoch. Let BtB_t be the batch at step t, define LBt(W):=|Bt|−1∑(i,s)∈Bt−logi(Wxi,s)L_B_t(W):=|B_t|^-1 _(i,s)∈ B_t- _i(Wx_i,s), and set Gt:=∇L(Wt)G_t:=∇ L(W_t) under full-batch training and Gt:=∇LBt(Wt)G_t:=∇ L_B_t(W_t) under R. For g∈row,sp,maxg∈\row,sp, \, β∈[0,1)β∈[0,1), and M0=M_0=0, we write Mt+1=βMt+(1−β)Gt,Wt+1=Wt−ηtΔg(Mt+1).M_t+1=β M_t+(1-β)G_t, W_t+1=W_t- _t _g(M_t+1). (6) Under full-batch training, β=0β=0 gives row-normalized gradient descent (RowGD), spectral gradient descent (SpecGD), and sign gradient descent (SignGD) for g=row,sp,maxg=row,sp, , respectively; with β∈(0,1)β∈(0,1), the corresponding methods are row-normalized gradient descent with momentum (RowGDM), exact-SVD Muon (Muon),44 4 Practical Muon (Jordan et al., 2024) approximates the orthogonalization step using a finite number of Newton–Schulz matrix-polynomial iterations. We instead use the exact SVD polar factor UV⊤UV ; this is equivalent to adding momentum to SpecGD and gives the variant we refer to as exact-SVD Muon. and Signum. Under R, we denote the momentum counterparts by R-RowSGDM, R-Muon, and R-Signum. Thus, throughout the theory, Muon and R-Muon refer to the exact-SVD variants. Throughout, WtoptW_t^opt denotes the parameter iterate of optimizer optopt at step t. Assumption 2.3 (Bounded number of mini-batches). Under random-reshuffling training with batch size b=bdb=b_d, the number of mini-batches per epoch, md:=N(d)/bdm_d:=N(d)/b_d, satisfies m0:=supdmd<∞m_0:= _dm_d<∞. Adam optimizer. We consider Adam without bias correction or a numerical-stability constant, with 0<β1≤β2<10< _1≤ _2<1, using GtG_t from the corresponding full-batch or random-reshuffling setting: Mt+1=β1Mt+(1−β1)Gt,Vt+1=β2Vt+(1−β2)Gt⊙2,Wt+1=Wt−ηtMt+1⊘Vt+1⊙1/2. M_t+1= _1M_t+(1- _1)G_t, 15muV_t+1= _2V_t+(1- _2)G_t 2, 15muW_t+1=W_t- _tM_t+1 V_t+1 1/2. (7) We initialize M0=V0=M_0=V_0=0, and refer to the full-batch and random-reshuffling versions as Adam and R-Adam, respectively. The following condition holds almost surely at W0=0W_0=0. Assumption 2.4 (Full-batch Adam initialization). For full-batch Adam, every entry of ∇L(W0)∇ L(W_0) is nonzero. Learning-rate schedule. We use the following learning-rate conditions. Assumption 2.5 (Base learning-rate condition). The sequence ηtt≥0\ _t\_t≥ 0 is positive and nonincreasing, with ηt→0 _t→ 0 and ∑t=0∞ηt=∞ _t=0^∞ _t=∞. Assumption 2.6 (Momentum learning rate). For every β∈(0,1)β∈(0,1) and c1>0c_1>0, there exist t0∈ℕt_0 and c2>0c_2>0 such that ∑s=0tβs(exp(c1∑τ=1sηt−τ)−1)≤c2ηt _s=0^tβ^s ( (c_1 _τ=1^s _t-τ)-1 )≤ c_2 _t for all t≥t0t≥ t_0. Non-momentum methods require only Assumption 2.5; momentum methods additionally require Assumption 2.6. Both hold for ηt=c(t+1)−a _t=c(t+1)^-a, c>0c>0, a∈(0,1]a∈(0,1]. 3 Main Results Section 3.1 establishes the deterministic optimizer population-accuracy ordering and identifies its limiting geometries. Section 3.2 studies robustness to random-reshuffling stochastic training, while Section 4 extends the analysis beyond the isotropic Gaussian-cloud data model. 3.1 Comparison of Optimizers in the Deterministic Setting We first compare the full-batch optimizers defined in Section 2. Our first main result shows that RowGD and RowGDM achieve strictly higher population accuracy than SpecGD, Muon, SignGD, Signum, and Adam. Theorem 3.1 (Deterministic optimizer accuracy ordering). Define the full-batch optimizer groups row:=RowGD,RowGDM,comp:=SpecGD,Muon,SignGD,Signum,Adam.O_row:=\RowGD,RowGDM\, _comp:=\SpecGD,Muon,SignGD,Signum,Adam\. (8) Suppose Assumptions 2.1, 2.2, 2.4, 2.5, and 2.6 hold. Then, for every sufficiently large d, with probability at least 1−e−d1-e^- d over the training data, the following holds for all momentum parameters β∈(0,1)β∈(0,1) of RowGDM, Muon, and Signum: minlim inft→∞opt∈rowAccπ,d(Wtopt)>maxlim supt→∞opt∈compAccπ,d(Wtopt). _opt _row _t→∞Acc_π,d\! (W_t^opt )> _opt _comp _t→∞Acc_π,d\! (W_t^opt ). (9) Proof idea. The proof proceeds through the implicit biases of the optimizers. RowGD and RowGDM approach row-norm margin solutions, SpecGD and Muon approach spectral-norm margin solutions, and SignGD, Signum, and Adam approach max-norm margin solutions. Here, a ψ-norm margin solution minimizes ψ(W)ψ(W) among classifiers with training margin at least one; the formal solution sets are defined in Appendix C. Under the Gaussian-cloud model, these three solution sets have the distinct asymptotic classifier structures summarized in Proposition 3.2, and Gaussian pairwise-error bounds convert this geometric separation into the population-accuracy ordering. Because Theorem 3.1 is proven through a uniform comparison over all norm-induced margin solutions approached by the optimizers, it does not require directional convergence of optimizers. Proposition 3.2 (Limiting classifier geometry). Let WtoptW_t opt have rows (wi,topt)⊤(w_i,t opt) , and let Θtopt:=k−1∑i=1k∠(wi,topt,μi) _t opt:=k^-1 _i=1^k (w_i,t opt, _i) denote the average angle between the class mean and classifier row. Let Mμ∈ℝk×dM_μ ^k× d have rows μi⊤ _i , let P:=Ik−k−1⊤P:=I_k-k^-111 and rd:=mink−1,dr_d:= \k-1,d\, and let Cμ:=PMμC_μ:=PM_μ have compact SVD UμΣμVμ⊤U_μ _μV_μ and polar factor Qμ:=UμVμ⊤Q_μ:=U_μV_μ . Suppose Assumptions 2.1, 2.2, 2.4, 2.5, and 2.6 hold. Then RowGD/RowGDM: RowGD/RowGDM: lim supt→∞maxi‖w^i,topt−μ^i‖2=oℙ(1), _t→∞ _i \| w_i,t opt- μ_i \|_2=o_P(1), SignGD/Signum/Adam: SignGD/Signum/Adam: lim supt→∞maxi‖w^i,topt−sign(μi)d‖2=oℙ(1), _t→∞ _i \| w_i,t opt- sign( _i) d \|_2=o_P(1), SpecGD/Muon: SpecGD/Muon: lim supt→∞1rd‖PWtopt‖PWtopt‖op−Qμ‖F2=oℙ(1). _t→∞ 1r_d \| PW_t opt\|PW_t opt\|_ op-Q_μ \|_ F^2=o_P(1). Moreover, lim supt→∞|Θtopt−θ|=oℙ(1) _t→∞| _t opt-θ|=o_P(1), where θ=0θ=0 for RowGD/RowGDM and θ=arccos2/πθ= 2/π for SignGD/Signum/Adam. If the aspect ratio α>1α>1, then θ=arccosκsp(α)θ= _ sp(α) for SpecGD/Muon, where κsp:(0,∞)→(0,1) _ sp:(0,∞)→(0,1) is a positive constant depending only on α.55 5 Its explicit formula is given in (81). Numerically, κsp(1)≈0.84883 _ sp(1)≈ 0.84883 and κsp(2)≈0.93280 _ sp(2)≈ 0.93280. Underlying mechanism. Proposition 3.2 makes the source of the accuracy gap explicit. RowGD and RowGDM preserve each class-mean direction, driving wiw_i toward a positive multiple of μi _i. In contrast, SignGD, Signum, and Adam approach the coordinate-wise sign direction of each class mean, effectively equalizing coordinate magnitudes. SpecGD and Muon approach the polar factor of the centered class-mean matrix, effectively equalizing its singular values. Both transformations introduce nonvanishing directional distortion relative to the original class-mean geometry. Under isotropic Gaussian test noise, preserving the class-mean geometry leads to better separation of unseen examples, while the distortions induced by the other methods degrade this separation. This yields the population-accuracy advantage of the row-normalized methods in Theorem 3.1. Corollary 3.3 (Population-error exponents). Let ℰt,dopt:=1−Accπ,d(Wtopt)E_t,d^opt:=1-Acc_π,d(W_t^opt). Suppose Assumptions 2.1, 2.2, 2.4, 2.5, and 2.6 hold. Then lim supt→∞|rte2dlogℰt,dAdamℰt,dRowGDM−π−24π|=oℙ(1),lim inft→∞rte2dlogℰt,dMuonℰt,dRowGDM≥1−κsp(α)24−oℙ(1), _t→∞ | r_te^2d \! E_t,d^AdamE_t,d^RowGDM- π-24π |=o_P(1),\ \ \ _t→∞ r_te^2d \! E_t,d^MuonE_t,d^RowGDM≥ 1- _ sp(α)^24\!-\!o_P(1), where κsp _ sp is the constant as in Proposition 3.2. The same conclusion holds with RowGDM replaced by RowGD, Adam by SignGD or Signum, and Muon by SpecGD. Corollary 3.3 follows from Proposition C.27 in Appendix C. While Theorem 3.1 establishes a strict ordering of the limiting population accuracies, Corollary 3.3 quantifies the separation. Although all three population errors vanish as d→∞d→∞, RowGDM has exponentially smaller population error than Adam and Muon. This separation is captured by the logarithms of the error ratios on the scale d/rte2d/r_te^2. 3.2 Robustness to Stochastic Training Mini-batch updates can alter the implicit bias of an optimizer, so the full-batch ordering does not automatically extend to stochastic training. Under R, however, sufficiently large momentum keeps R-RowSGDM, R-Muon, and R-Signum close to their respective row-, spectral-, and max-norm margin solutions (Li et al., 2026a). We show that this approximate preservation is enough for the strict population-accuracy ordering to persist. Proposition 3.4 (Stochastic optimizer accuracy ordering). Suppose Assumptions 2.1, 2.2, 2.3, 2.5, and 2.6 hold. Then there exists β0=β0(α,r0,m0)∈[0,1) _0= _0(α,r_0,m_0)∈[0,1) such that, for every sufficiently large d, with probability at least 1−e−d1-e^- d over the training data, the following holds for all momentum parameters β∈[β0,1)β∈[ _0,1) of R-RowSGDM, R-Muon, and R-Signum, and for every realization of the reshuffling: lim inft→∞Accπ,d(WtRR-RowSGDM)>maxlim supt→∞opt∈R-Muon,R-SignumAccπ,d(Wtopt). _t→∞Acc_π,d\! (W_t^R -RowSGDM )> _opt∈\R -Muon,\,R -Signum\ _t→∞Acc_π,d\! (W_t^opt ). (10) The condition md≤m0m_d≤ m_0 in Assumption 2.3 means that each mini-batch contains at least a 1/m01/m_0 fraction of the training set. For m0≥2m_0≥ 2, the sufficient momentum threshold used in the proof has 1−β01- _0 proportional to 1/[m0(m02−1)]1/[m_0(m_0^2-1)]. Approximating Adam via AdamProxy. Adam requires separate treatment under R because averaging squared mini-batch gradients generally differs from squaring the averaged gradient, so its full-batch max-norm margin characterization need not persist (Baek et al., 2026). To address this, we develop a deterministic proxy for R-Adam in the multiclass setting with general mini-batch size, extending the binary incremental AdamProxy construction of Baek et al. (2026). Using the flattened indexing (xn,yn)n=1N\(x_n,y_n)\_n=1^N, let gn(W):=−(eyn−(Wxn))xn⊤g_n(W):=-(e_y_n-S(Wx_n))x_n denote the gradient of the cross-entropy loss for sample (xn,yn)(x_n,y_n). For each batch B⊂[N]B⊂[N] of size b, set GB(W):=b−1∑n∈Bgn(W)=∇LB(W)G_B(W):=b^-1 _n∈ Bg_n(W)=∇ L_B(W). Thus, under R, Gt=GBt(Wt)G_t=G_B_t(W_t) in (7). Drawing B uniformly from the b-subsets of [N][N], define the AdamProxy direction ΦN,b(W):=B[GB(W)]⊘B[GB(W)⊙2]⊙1/2. _N,b(W):=E_B[G_B(W)] _B\! [G_B(W) 2 ] 1/2. (11) Equivalently, letting S(W):=∑n=1Ngn(W)S(W):= _n=1^Ng_n(W), Q(W):=∑n=1Ngn(W)⊙2Q(W):= _n=1^Ng_n(W) 2, ΦN,b(W)=b(N−1)NS(W)⊘((b−1)S(W)⊙2+(N−b)Q(W))⊙1/2. _N,b(W)= b(N-1)N\,S(W) ((b-1)S(W) 2+(N-b)Q(W) ) 1/2. (12) Proposition 3.5 shows that, in the β1,β2→1 _1, _2→ 1 regime, this deterministic direction approximates the normalized one-epoch displacement of R-Adam. Proposition 3.5 (One-epoch β1,β2→1 _1, _2→ 1 R-Adam approximation). For each epoch r=0,1,2,…r=0,1,2,…, let τr:=mr _r:=mr denote the index of its first update and γr:=∑s=0m−1ητr+s _r:= _s=0^m-1 _ _r+s its total learning rate. Fix a training set with xn[a]≠0x_n[a]≠ 0 for all n∈[N]n∈[N] and a∈[d]a∈[d]. Suppose that b<Nb<N and that Assumptions 2.5 and 2.6 hold. Then, for every ε>0 >0, R-Adam in (7) satisfies lim(β1,β2)→(1,1)0<β1≤β2<1lim supr→∞ℙ(‖Wτr+1R-Adam−WτrRR-Adamγr+ΦN,b(WτrRR-Adam)‖max>ε)=0, _ subarrayc( _1, _2)→(1,1)\\ 0< _1≤ _2<1 subarray _r→∞P ( \| W_ _r+1^R -Adam-W_ _r^R -Adam _r+ _N,b\! (W_ _r^R -Adam ) \|_ > )=0, (13) where the probability is over the random reshuffling. Proposition 3.5 is a one-epoch β1,β2→1 _1, _2→ 1 approximation at fixed data and dimension. It does not establish long-horizon agreement between the R-Adam and AdamProxy trajectories. Motivated by Proposition 3.5, let γt:=∑s=0m−1ηmt+s _t:= _s=0^m-1 _mt+s and consider the AdamProxy dynamics Wt+1AP=WtAP−γtΦN,b(WtAP).W_t+1^AP=W_t^AP- _t _N,b\! (W_t^AP ). (14) Comparison of AdamProxy and R-RowSGDM. Although AdamProxy need not approach the exact sign geometry of full-batch Adam, it retains a weaker coordinate-wise saturation effect. Once the classifier is scaled to have unit minimum training margin, every coordinate eventually stays within a constant multiple of the typical coordinate size (Proposition E.8). This forces a nonvanishing distortion from the class-mean geometry, whereas high-momentum R-RowSGDM remains close to the class-mean geometry, yielding the population-accuracy ordering below. Proposition 3.6 (R-RowSGDM versus AdamProxy accuracy ordering). Suppose Assumptions 2.1, 2.2, 2.3, 2.5, and 2.6 hold. Then there exists β1=β1(α,r0,m0)∈[0,1) _1= _1(α,r_0,m_0)∈[0,1) such that, for every sufficiently large d, with probability at least 1−e−d1-e^- d over the training data, the following holds for all momentum parameters β∈[β1,1)β∈[ _1,1) of R-RowSGDM and for every realization of the reshuffling: lim inft→∞Accπ,d(WtRR-RowSGDM)>lim supt→∞Accπ,d(WtAP). _t→∞Acc_π,d\! (W_t^R -RowSGDM )> _t→∞Acc_π,d\! (W_t^AP ). (15) 4 Extensions to Anisotropic Gaussian Models In Section 3, we studied an isotropic Gaussian-cloud data model in which both the class means and the within-class noise are isotropic. In this section, we examine when the advantage of row-normalized methods persists or reverses beyond isotropy. Motivated by the nontrivial covariance structure that can arise in language-model representations, we extend the analysis to anisotropic Gaussian distributions. To simplify the analysis and cover a broader range of anisotropy, we consider clean training, with no within-class noise in the training data, under the following common data model. Assumption 4.1 (Anisotropic clean Gaussian model). Let Γ,Σ1,…,Σk , _1,…, _k be positive-definite covariance matrices with trΓ=trΣi=dtr =tr _i=d for every i∈[k]i∈[k].66 6 A fixed trace preserves total variance while allowing covariance spectra and orientations to vary. Training uses the clean class signals (μi,i)i=1k\( _i,i)\_i=1^k. A test example with label Y is Xte:=μY+rtezteX^te:= _Y+r_tez^te, where, conditionally on the covariance matrices, (Y,zte)(Y,z^te) is independent of (μi)i=1k( _i)_i=1^k and μi∼iid(,Γ),ℙ(Y=i)=πi,d>0,zte∣Y∼(,ΣY). _i iid N(0, ), (Y=i)= _i,d>0, z^te Y (0, _Y). (16) Along the sequence indexed by d, k/d→α∈(0,∞)k/d→α∈(0,∞) and rte>0r_te>0 satisfies rte,0:=supdrte(d)<∞r_te,0:= _dr_te(d)<∞. 4.1 Persistence of Row-Norm Advantage under Independent Orientations We first specialize Assumption 4.1 to trace-normalized power-law covariances with independent Haar eigenbases. Suppressing dimension subscripts, define Tϑ:=∑a=1da−ϑT_ := _a=1^da^- and Dϑ:=diag(1,2−ϑ,…,d−ϑ)D_ :=diag(1,2^- ,…,d^- ). For sμ,sΣ≥0s_μ,s_ ≥ 0, set Γ:=dTsμUμDsμUμ⊤,Σi:=dTsΣOiDsΣOi⊤. := dT_s_μU_μD_s_μU_μ , _i:= dT_s_ O_iD_s_ O_i . (17) Here Uμ,O1,…,OkU_μ,O_1,…,O_k are mutually independent Haar matrices on (d) O(d); the class-mean draws are independent of (Oi)i=1k(O_i)_i=1^k, and the optimizer initializations are fixed and do not depend on (Oi)i=1k(O_i)_i=1^k. Theorem 4.2 (Accuracy ordering under anisotropy with independent orientations). Let rowO_row and compO_comp be as in (8), with fixed β∈(0,1)β∈(0,1) for RowGDM, Muon, and Signum. Suppose Assumptions 4.1, 2.4, 2.5, and 2.6 hold, with the covariances in (17) for 0≤sμ<10≤ s_μ<1 and sΣ≥0s_ ≥ 0. Assume that log(1/πmin,d)=o(d/logd) (1/ _ ,d)=o(d/ d) and that Wtopt/‖Wtopt‖FW_t^opt/\|W_t^opt\|_F converges almost surely as t→∞t→∞ for every opt∈row∪compopt _row _comp. Then there exist constants c,C>0c,C>0, independent of d, such that, for every sufficiently large d, with probability at least 1−Cd−c1-Cd^-c over UμU_μ, O1,…,OkO_1,…,O_k, and the class means, minlim inft→∞opt∈rowAccπ,d(Wtopt)>maxlim supt→∞opt∈compAccπ,d(Wtopt). _opt _row _t→∞Acc_π,d\! (W_t^opt )> _opt _comp _t→∞Acc_π,d\! (W_t^opt ). (18) Here c,Cc,C can be chosen independently of sμs_μ and sΣs_ , although the sufficiently-large-d threshold may depend on them. In particular, under independent test-noise orientations, the advantage of row-normalized methods persists for arbitrary test-noise anisotropy sΣ≥0s_ ≥ 0. Remark 4.3 (Role of directional convergence). Unlike the accuracy comparisons in Section 3, which hold uniformly over all classifier families that the normalized iterates approach and therefore do not require directional convergence, Theorem 4.2 assumes directional convergence for each optimizer. Related directional-convergence assumptions are common in the implicit-bias literature (Gunasekar et al., 2018a; Nacson et al., 2019a; Chizat and Bach, 2020; Baek et al., 2026). Under clean training, conditional on the training data, the limit of Wtopt/‖Wtopt‖FW_t^opt/\|W_t^opt\|_F is fixed and independent of the test-noise eigenbases, which enter only at test time. Hence, the concentration bounds over the Haar-random test-noise eigenbases apply to the fixed limiting direction. Structural intuition: effective signal dimension. The condition sμ<1s_μ<1 keeps the class-distinguishing signal sufficiently spread across dimensions, so different class means have weak pairwise overlap. This allows the row-norm margin solutions to remain close to the corresponding class-mean directions, while satisfying the multiclass margin constraints. If the signal is instead concentrated in only a few directions, some class means can have small pairwise angles, and margin constraints can push the row-norm margin solutions away from the class-signal-preserving geometry. Remark 4.4 (Connection to language-model representations). We find that the fitted class-mean spectral exponents can be smaller than one (sμ<1s_μ<1) in language-model representations; see Appendix H.3. This provides a possible connection between the anisotropic regime covered by Theorem 4.2 and representation geometry in language models. 4.2 Coordinate-aligned Reversal and Recovery under Haar Rotation We next show that the advantage of row-normalized methods can reverse when the class-mean and test-noise covariances are diagonal and sufficiently close, but is restored after applying the same independently sampled Haar rotation to the class means and test-noise covariance. The underlying issue is the interaction between signal and test-noise geometry. With a common test-noise covariance Σ , the normal of the Bayes boundary between classes i and j is proportional to Σ−1(μi−μj) ^-1( _i- _j). Thus, if directions that strongly separate the class means also have large test variance, the Bayes boundary can differ substantially from the raw class-mean difference that the row-norm margin solution tends to preserve. The construction below places this signal–noise alignment along the coordinate axes used by Adam’s entrywise normalization, where Adam’s tendency to equalize coordinate magnitudes can reduce the influence of such high-variance coordinates. Let the class-mean covariance be diagonal and let the common test-noise covariance interpolate between isotropic noise and the same covariance profile: Γd:=diag(γ1,d,…,γd,d),Σλ,d:=(1−λ)Id+λΓd,λ∈[0,1]. _d:=diag( _1,d,…, _d,d), _λ,d:=(1-λ)I_d+λ _d, λ∈[0,1]. (19) Thus, Σλ,d _λ,d interpolates between isotropic test noise at λ=0λ=0 and the class-mean covariance at λ=1λ=1. Proposition 4.5 (Coordinate dependence of the RowGDM–Adam ordering). Let λ∈[0,1]λ∈[0,1]. Suppose that Assumptions 4.1, 2.4, 2.5, and 2.6 hold with Γ=Γd = _d and Σ1=⋯=Σk=Σλ,d _1=·s= _k= _λ,d as in (19). Suppose also that log(1/πmin,d)=o(d) (1/ _ ,d)=o(d), 0<infa,dγa,d≤supa,dγa,d<∞0< _a,d _a,d≤ _a,d _a,d<∞, and the limits γ¯q:=limd→∞d−1∑a=1dγa,dq γ_q:= _d→∞d^-1 _a=1^d _a,d^q exist ∀q∈1/2,2∀ q∈\1/2,2\ and satisfy 2γ¯1/22γ¯2>π2 γ_1/2^2 γ_2>π.77 7 This requires a sufficiently dispersed spectrum. Corollary G.8 gives a sufficient condition. Then λc:=(π/(2γ¯1/22)−1)/(γ¯2−1) _c:=(π/(2 γ_1/2^2)-1)/( γ_2-1) lies in (0,1)(0,1), and the following hold.88 8 The same conclusions hold with RowGDM replaced by RowGD. (i) Diagonal covariance. For every λ∈[0,1]∖λcλ∈[0,1] \ _c\, there exist cλ,Cλ>0c_λ,C_λ>0 such that, for every sufficiently large d, with probability at least 1−Cλe−cλd1-C_λe^-c_λd over the class means, for all β∈(0,1)β∈(0,1), lim inft→∞Accπ,d(WtRowGDM) _t→∞Acc_π,d\! (W_t^RowGDM ) >lim supt→∞Accπ,d(WtAdam) > _t→∞Acc_π,d\! (W_t^Adam ) if λ<λc, λ< _c, lim inft→∞Accπ,d(WtAdam) _t→∞Acc_π,d\! (W_t^Adam ) >lim supt→∞Accπ,d(WtRowGDM) > _t→∞Acc_π,d\! (W_t^RowGDM ) if λ>λc. λ> _c. (i) Recovery under Haar rotation. Let Q be Haar-uniformly random on (d) SO(d) and independent of the class means, and replace every training and test input x by QxQx. There exist crot,Crot>0c_rot,C_rot>0 such that, for every fixed λ∈[0,1]λ∈[0,1] and every sufficiently large d, with probability at least 1−Crote−crotd1-C_rote^-c_rotd over the class means and Q, the RowGDM-over-Adam ordering holds. Implication. Theorem 4.2 shows that anisotropy alone does not eliminate the advantage of row-normalized methods under independent orientations. Proposition 4.5 identifies a structured failure mode. The RowGDM-over-Adam ordering can reverse when the high-variance class-mean and test-noise directions align with each other and with Adam’s normalization coordinates. However, this reversal is coordinate-specific. Applying the same Haar rotation to the class means and test-noise covariance preserves the covariance spectra and their relative alignment while randomizing their orientation relative to the coordinate axes. This restores the RowGDM-over-Adam ordering. Hence, class-mean–test-noise alignment alone is not sufficient; the reversal additionally relies on its alignment with Adam’s coordinate axes. Figure 2: Synthetic and language-model experiments. Left: Final evaluation accuracy of six deterministic optimizers on 30 draws of Gaussian-cloud data. Right: Validation perplexity during last-layer-only LLaMA training with RowSGDM, Muon, and Adam. Lower perplexity is better. 5 Experiments Synthetic experiment. On isotropic Gaussian-cloud data with k=60k=60, d=20d=20, RowGD and RowGDM outperform SpecGD, Muon, SignGD, and Adam on all 30 paired seeds (Figure 2, left), consistent with Theorem 3.1. Additional comparisons and details are given in Appendix H.1. Language model experiment. We freeze a trained LLaMA-6060M model, zero-initialize its output head, and train only this layer with practical implementations of RowSGDM, Muon, or Adam. Figure 2 (right) shows that RowSGDM attains the lowest final validation perplexity, consistent with the qualitative ordering our theory predicts. Multi-seed results and details are given in Appendix H.2. 6 Conclusion Implications. We connect optimizer-induced implicit-bias geometry to population accuracy in high-dimensional multiclass classification, beyond prior work on optimizer-selected implicit biases (Fan et al., 2025; Li et al., 2026a; Baek et al., 2026). For linear classification abstracting the language model’s last layer, row-normalized methods achieve strictly higher population accuracy than spectral and coordinate-wise alternatives in both full-batch and random-reshuffling settings by better preserving class-mean geometry. These results complement empirical evidence of competitive performance by row-normalized methods in language model training (Glentis et al., 2026; Deng et al., 2026). Limitations and outlook. Our analysis is limited to a linear classification model that can be viewed as a last-layer readout, together with a stylized data model with Gaussian class means and within-class noise. Although the data model shares some high-dimensional geometric properties with linguistic collapse (Wu and Papyan, 2024), modern language-model representations are substantially more complex. This motivates extensions to richer representation models and multilayer training. Our anisotropic independent-orientation analysis assumes directional convergence, while our Adam results assume a zero numerical-stability constant and treat R-Adam only via AdamProxy. AI use statement In this work, we used generative AI tools to assist with improving English writing and readability, identifying potentially relevant related work that we might have overlooked, and drafting proof sections of the appendix by converting lengthy mathematical proofs and derivations into clean and readable LaTeX. We additionally used generative AI tools to assist with debugging experimental code and improving the visual presentation of figures and tables. All AI-assisted outputs were reviewed by the authors, who take responsibility for the final content of this work. Reproducibility statement We state the assumptions and settings for our theoretical results in Sections 2 and 4, and provide complete proofs and supporting technical results in Appendices B–G. For the empirical results, Appendix H provides the experimental setups and information needed for reproduction, with synthetic experiments in Appendix H.1, the language-model experiment in Appendix H.2, and the representation analysis in Appendix H.3. References Baek et al. (2026) B. Baek, M. Song, and C. Yun Implicit bias of per-sample Adam on separable data: departure from the full-batch regime. In International Conference on Learning Representations, p. 120180–120229. Cited by: Appendix A, Appendix A, §E.3, 3rd item, §1.2, §1, §3.2, Remark 4.3, §6. Balles and Hennig (2018) L. Balles and P. Hennig Dissecting Adam: the sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning, p. 404–413. Cited by: Appendix A. Bernstein and Newhouse (2025) J. Bernstein and L. Newhouse Modular duality in deep learning. In International Conference on Machine Learning, p. 3920–3930. Cited by: Appendix A. Bernstein et al. (2018) J. Bernstein, Y. Wang, K. Azizzadenesheli, and A. Anandkumar SignSGD: compressed optimisation for non-convex problems. In International Conference on Machine Learning, p. 560–569. Cited by: Appendix A. Cao et al. (2021) Y. Cao, Q. Gu, and M. Belkin Risk bounds for over-parameterized maximum margin classification on sub-Gaussian mixtures. Advances in Neural Information Processing Systems 34, p. 8407–8418. Cited by: Appendix A. Chang et al. (2026) D. Chang, Q. Shi, L. Zhang, Y. Li, R. Zhang, Y. Lu, Y. Liu, and G. Yuan MuonEq: balancing before orthogonalization with lightweight equilibration. arXiv preprint arXiv:2603.28254. Cited by: Appendix A. Chizat and Bach (2020) L. Chizat and F. Bach Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, p. 1305–1338. Cited by: Remark 4.3. Deng et al. (2026) S. Deng, Z. Ouyang, T. Pang, Z. Liu, R. Jin, S. Yu, and Y. Yang RMNP: row-momentum normalized preconditioning for scalable matrix-based optimization. In International Conference on Machine Learning, p. 24191–24239. Cited by: Appendix A, §1.2, §1.2, §1, §1, §6. Dewulf et al. (2026) A. Dewulf, D. Pai, L. Yang, A. Zhang, and B. Keigwin Aurora: a leverage-aware spectral optimizer. arXiv preprint arXiv:2606.27715. Cited by: Appendix A. Fan et al. (2025) C. Fan, M. Schmidt, and C. Thrampoulidis Implicit bias of spectral descent and Muon on multiclass separable data. Advances in Neural Information Processing Systems 38, p. 39622–39669. Cited by: Appendix A, §B.2, §B.2, Appendix B, 1st item, §1.2, §1, §2, §6. Glentis et al. (2026) A. Glentis, J. Li, A. Han, and M. Hong Memory-efficient LLM pretraining via minimalist optimizer design. In International Conference on Machine Learning, p. 35197–35226. Cited by: §1.2, §1, §6. Gronich and Vardi (2026) E. Gronich and G. Vardi The implicit bias of Adam and Muon on smooth homogeneous neural networks. In International Conference on Machine Learning, p. 36767–36811. Cited by: Appendix A. Gunasekar et al. (2018a) S. Gunasekar, J. D. Lee, D. Soudry, and N. Srebro Implicit bias of gradient descent on linear convolutional networks. Advances in Neural Information Processing Systems 31. Cited by: Remark 4.3. Gunasekar et al. (2018b) S. Gunasekar, J. Lee, D. Soudry, and N. Srebro Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, p. 1832–1841. Cited by: Appendix A, §1. Gupta et al. (2018) V. Gupta, T. Koren, and Y. Singer Shampoo: preconditioned stochastic tensor optimization. In International Conference on Machine Learning, p. 1842–1850. Cited by: Appendix A. Jiang et al. (2024) J. Jiang, J. Zhou, P. Wang, Q. Qu, D. G. Mixon, C. You, and Z. Zhu Generalized neural collapse for a large number of classes. In International Conference on Machine Learning, p. 22010–22041. Cited by: Appendix A. Jordan et al. (2024) K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein Muon: an optimizer for hidden layers in neural networks. External Links: Link Cited by: Appendix A, §1, footnote 4. Kim et al. (2026a) J. Kim, D. Song, and C. Yun Scaling laws of signSGD in linear regression: when does it outperform SGD?. In International Conference on Learning Representations, p. 67140–67228. Cited by: Appendix A. Kim et al. (2026b) J. Kim, B. Shin, J. Yun, B. Baek, M. Song, and C. Yun AMUSE: anytime Muon with stable gradient evaluation. arXiv preprint arXiv:2605.22432. Cited by: §H.3. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. In International Conference on Learning Representations, Cited by: Appendix A, §1. Kunstner et al. (2023) F. Kunstner, J. Chen, J. W. Lavington, and M. Schmidt Noise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be. In International Conference on Learning Representations, Cited by: Appendix A. Li et al. (2026a) J. Li, X. Tang, and D. Zou The implicit bias of steepest descent with mini-batch stochastic gradient. In International Conference on Machine Learning, p. 70353–70413. Cited by: Appendix A, Appendix B, 1st item, §1.2, §1, §2, §3.2, §6. Li et al. (2026b) Z. Li, L. Liu, C. Liang, W. Chen, and T. Zhao NorMuon: making Muon more efficient and scalable. In International Conference on Machine Learning, p. 68514–68533. Cited by: Appendix A, §1.2, §1. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, Cited by: Appendix A. Ma et al. (2025) C. Ma, W. Gong, M. Scetbon, and E. Meeds SWAN: SGD with normalization and whitening enables stateless LLM training. In International Conference on Machine Learning, p. 41907–41942. Cited by: Appendix A. Mignacco et al. (2020) F. Mignacco, F. Krzakala, Y. Lu, P. Urbani, and L. Zdeborova The role of regularization in classification of high-dimensional noisy Gaussian mixture. In International Conference on Machine Learning, p. 6874–6883. Cited by: Appendix A. Muennighoff et al. (2025) N. Muennighoff, L. Soldaini, D. Groeneveld, K. Lo, J. Morrison, S. Min, W. Shi, P. Walsh, O. Tafjord, N. Lambert, et al. OLMoE: open mixture-of-experts language models. In International Conference on Learning Representations, p. 62061–62121. Cited by: §H.3. Nacson et al. (2019a) M. S. Nacson, J. Lee, S. Gunasekar, P. H. P. Savarese, N. Srebro, and D. Soudry Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics, p. 3420–3428. Cited by: Remark 4.3. Nacson et al. (2019b) M. S. Nacson, N. Srebro, and D. Soudry Stochastic gradient descent on separable data: exact convergence with a fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics, p. 3051–3059. Cited by: Appendix A. Papyan et al. (2020) V. Papyan, X. Y. Han, and D. L. Donoho Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences 117 (40), p. 24652–24663. Cited by: Appendix A. Pethick et al. (2025) T. Pethick, W. Xie, K. Antonakopoulos, Z. Zhu, A. Silveti-Falls, and V. Cevher Training deep learning models with norm-constrained LMOs. In International Conference on Machine Learning, p. 49069–49104. Cited by: Appendix A. Qian and Qian (2019) Q. Qian and X. Qian The implicit bias of AdaGrad on separable data. Advances in Neural Information Processing Systems 32. Cited by: Appendix A. Ravi et al. (2024) H. Ravi, C. Scott, D. Soudry, and Y. Wang The implicit bias of gradient descent on separable multiclass data. Advances in Neural Information Processing Systems 37, p. 81324–81359. Cited by: Appendix A. Scetbon et al. (2025) M. Scetbon, C. Ma, W. Gong, and T. Meeds Gradient multi-normalization for efficient LLM training. Advances in Neural Information Processing Systems 38, p. 43001–43030. Cited by: Appendix A, §1.2, §1. Soudry et al. (2018) D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro The implicit bias of gradient descent on separable data. Journal of Machine Learning Research 19 (70), p. 1–57. Cited by: Appendix A, §1. Subramanian et al. (2022) V. Subramanian, R. Arya, and A. Sahai Generalization for multiclass classification with overparameterized linear models. Advances in Neural Information Processing Systems 35, p. 23479–23494. Cited by: Appendix A. Tsilivis et al. (2026) N. Tsilivis, E. Gronich, J. Kempe, and G. Vardi Flavors of margin: implicit bias of steepest descent in homogeneous neural networks. Journal of Machine Learning Research 27 (104), p. 1–37. Cited by: Appendix A. Vaswani et al. (2020) S. Vaswani, R. Babanezhad, J. Gallego-Posada, A. Mishkin, S. Lacoste-Julien, and N. L. Roux To each optimizer a norm, to each norm its generalization. arXiv preprint arXiv:2006.06821. Cited by: Appendix A. Vershynin (2012) R. Vershynin Introduction to the non-asymptotic analysis of random matrices. In Compressed Sensing: Theory and Applications, p. 210–268. Cited by: §C.5, Appendix G. Vyas et al. (2025) N. Vyas, D. Morwani, R. Zhao, I. Shapira, D. Brandfonbrener, L. Janson, and S. Kakade SOAP: improving and stabilizing Shampoo using Adam for language modeling. In International Conference on Learning Representations, p. 93423–93444. Cited by: Appendix A. Wang et al. (2021a) B. Wang, Q. Meng, W. Chen, and T. Liu The implicit bias for adaptive optimization algorithms on homogeneous neural networks. In International Conference on Machine Learning, p. 10849–10858. Cited by: Appendix A. Wang et al. (2022) B. Wang, Q. Meng, H. Zhang, R. Sun, W. Chen, Z. Ma, and T. Liu Does momentum change the implicit regularization on separable data?. Advances in Neural Information Processing Systems 35, p. 26764–26776. Cited by: Appendix A. Wang et al. (2021b) K. Wang, V. Muthukumar, and C. Thrampoulidis Benign overfitting in multiclass classification: all roads lead to interpolation. Advances in Neural Information Processing Systems 34, p. 24164–24179. Cited by: Appendix A. Wilson et al. (2017) A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht The marginal value of adaptive gradient methods in machine learning. Advances in Neural Information Processing Systems 30. Cited by: Appendix A. Wu and Papyan (2024) R. Wu and V. Papyan Linguistic collapse: neural collapse in (large) language models. Advances in Neural Information Processing Systems 37, p. 137432–137473. Cited by: Appendix A, §6. Xie and Li (2024) S. Xie and Z. Li Implicit bias of AdamW: ℓ_∞ \_∞-norm constrained optimization. In International Conference on Machine Learning, p. 54488–54510. Cited by: Appendix A. Xu et al. (2026) R. Xu, J. Li, and Y. Lu On the width scaling of neural optimizers under matrix operator norms I: row/column normalization and hyperparameter transfer. arXiv preprint arXiv:2603.09952. Cited by: Appendix A, §1.2, §1.2, §1, §1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §H.3. Yuan et al. (2026) J. Yuan, J. Zou, S. Wang, Y. Liu, and F. Nie Nora: normalized orthogonal row alignment for scalable matrix optimizer. arXiv preprint arXiv:2605.03769. Cited by: Appendix A. Zhang et al. (2024) C. Zhang, D. Zou, and Y. Cao The implicit bias of Adam on separable data. Advances in Neural Information Processing Systems 37, p. 23988–24021. Cited by: Appendix A. Appendix Overview of the Appendix The appendix is organized around the proof architecture of the paper. Appendix A provides additional related work, and Appendix B develops the optimizer-side margin and implicit-bias machinery for full-batch and random-reshuffling training. Appendices C–E combine this machinery with high-dimensional geometry to prove the deterministic, stochastic, and AdamProxy population-accuracy results. Appendix F extends the analysis to anisotropic Gaussian models with independently oriented test-noise covariances, Appendix G identifies a coordinate-specific RowGDM–Adam reversal and its recovery under a common Haar rotation, and Appendix H provides the experimental details and additional empirical evidence. (1) In Appendix A, we provide additional related work on implicit bias in separable classification, adaptive and sign-based optimization, optimizer geometry and population performance, matrix-aware optimization geometry, row normalization and row–spectral hybrids, high-dimensional classification under mixture models, and last-layer geometry and neural collapse. (2) In Appendix B, we establish the optimizer-side margin-convergence results used throughout the paper. We extend full-batch normalized steepest-descent and momentum arguments to max–sum dominated matrix norms and establish the max-norm maximum-margin characterization of full-batch Adam under our learning-rate assumptions. For random-reshuffling momentum methods, we develop a geometry-adapted analysis through the primal–dual atom radius Rψ(X)R_ψ(X), which applies directly to the Row, Spectral, and Max geometries. Specializing these results to row normalization yields the exact full-batch RowGD and RowGDM direction-set convergence in Proposition B.14 and the R-RowSGDM near-margin guarantee in Proposition B.15. (3) In Appendix C, we prove the deterministic population-accuracy ordering in Theorem 3.1 uniformly over the complete Row, Spectral, and Max maximum-margin direction sets. The proof establishes the all-aspect-ratio geometry of these three families: Row representatives align uniformly with the population mean differences, Max representatives exhibit a nonvanishing clipped-coordinate angular gap, and Spectral representatives are controlled through the reduced polar geometry of the centered empirical mean matrix, including the hard-edge regime at α=1α=1. We then develop a geometry-to-population-error transfer and derive quantitative population-error exponents for the three geometries. The same exponent analysis also yields the full-batch Muon–Adam comparison in Corollary C.28. (4) In Appendix D, we prove the random-reshuffling stochastic ordering in Proposition 3.4. The geometry-adapted momentum result from Appendix B places R-RowSGDM, R-Muon, and R-Signum in near-maximum-margin direction sets for the Row, Spectral, and Max geometries. We reduce these near-margin directions to near-minimum-norm representatives, prove quantitative stability of the three near-minimizer families, and obtain a dimension-independent momentum threshold under which the deterministic geometric separation persists. The population-error transfer from Appendix C then gives the claimed stochastic accuracy ordering. (5) In Appendix E, we analyze R-Adam and AdamProxy. We first prove the one-epoch β1,β2→1 _1, _2→ 1 R-Adam approximation in Proposition 3.5; this result connects R-Adam to the deterministic proxy over one epoch and does not assert long-horizon agreement of their trajectories. For AdamProxy itself, we establish positive margin growth and a trajectory-level coordinate cap, use the resulting coordinate saturation to obtain a nonvanishing boundary-direction gap, and combine this gap with the high-momentum R-RowSGDM geometry and the population-error transfer from Appendix C to prove Proposition 3.6 without directional convergence. Separately, Proposition E.10 characterizes the limiting AdamProxy classifier through a self-consistent semidefinite quadratic margin problem when the normalized proxy direction converges. (6) In Appendix F, we prove the anisotropic accuracy ordering in Theorem 4.2 under its directional-convergence assumption. Because training is clean, each optimizer-selected limiting direction depends only on the class means and is independent of the Haar eigenbases of the test-noise covariances. We characterize the selected Row, Spectral, and Max directions, expose the independent test-noise orientations, and compare their standardized Gaussian test signals. This establishes the Row advantage throughout 0≤sμ<10≤ s_μ<1 and sΣ≥0s_ ≥ 0, including arbitrarily strong test-noise anisotropy. (7) In Appendix G, we prove the coordinate-specific RowGDM–Adam reversal in Proposition 4.5. Working uniformly over the complete Row and Max maximum-margin solution sets, we derive an explicit transition point λc _c for diagonal class-mean covariance with aligned anisotropic test noise, below which Row dominates and above which the ordering reverses. We then show that applying a common independent Haar rotation to the same class-mean and test-noise geometry restores the Row-over-Max, and hence RowGDM-over-Adam, ordering while preserving the rotation-invariant geometry. We also give a sufficient spectral-dispersion condition under which the transition regime is nonvacuous. (8) In Appendix H, we provide the full experimental setup and additional empirical evidence. This includes expanded deterministic and random-reshuffling synthetic optimizer comparisons, auxiliary optimizer geometries, robustness studies across data and noise settings, synthetic experiments for the anisotropic independent-orientation and coordinate-aligned predictions, controlled last-layer language-model experiments with RowSGDM, Muon, and Adam, and measurements of class-mean and within-class covariance spectral exponents in language-model representations. Appendix A Additional Related Work Implicit bias in separable classification. The modern implicit-bias literature begins with the observation that, on linearly separable data, gradient descent on unregularized logistic or cross-entropy loss diverges in norm while converging in direction to a hard-margin classifier (Soudry et al., 2018). More generally, the geometry of the update rule can determine the norm with respect to which the limiting margin is maximized, as formalized for steepest-descent and mirror-descent methods by Gunasekar et al. (2018b). For Euclidean methods, this picture has been extended from gradient descent to fixed-step stochastic gradient descent and momentum methods (Nacson et al., 2019b; Wang et al., 2022), and recent work gives multiclass convergence results for broad families of exponentially tailed losses (Ravi et al., 2024). In homogeneous neural networks, norm-dependent steepest-descent dynamics generally converge to first-order stationary points of the corresponding margin problem rather than necessarily to its global optimum (Tsilivis et al., 2026), and a recent extension treats normalized and momentum variants of SignGD, Muon, and Adam (Gronich and Vardi, 2026). The results closest to ours characterize the max-margin directions selected by entrywise and Schatten geometries in multiclass linear models, their approximation under random reshuffling, and the data-dependent direction selected by incremental Adam (Fan et al., 2025; Li et al., 2026a; Baek et al., 2026). Our analysis complements these directional results by adding the row norm and by comparing the population accuracy of entire optimizer-selected margin faces rather than only identifying a limiting direction. Adaptive and sign-based optimization. Adam combines momentum with coordinate-wise second-moment normalization (Kingma and Ba, 2015), whereas signSGD and Signum retain only coordinate-wise gradient signs, with or without momentum (Bernstein et al., 2018). This relationship is not merely cosmetic: Adam can be decomposed into a sign direction and a variance-dependent magnitude, which helps explain both its robustness and its sensitivity to the coordinate system (Balles and Hennig, 2018); signSGD can also exhibit distinct compute-optimal scaling behavior from SGD (Kim et al., 2026a). The asymptotic classifier also depends on algorithmic details that are often suppressed in informal comparisons. For example, AdaGrad can converge to a margin solution determined by its path-dependent diagonal conditioner (Qian and Qian, 2019), adaptive methods with a positive stability constant can become asymptotically Euclidean in homogeneous models (Wang et al., 2021a), and Adam without a stability constant selects an ℓ∞ _∞-max-margin direction in binary linear classification (Zhang et al., 2024). Decoupled weight decay changes the problem again: the implicit regularization of AdamW is related to an ℓ∞ _∞-constrained objective rather than to the unregularized margin limit (Loshchilov and Hutter, 2019; Xie and Li, 2024). Accordingly, our full-batch Adam statement concerns the zero-stability-constant, no-weight-decay dynamics used in our theoretical model, while our AdamProxy analysis isolates the additional data dependence created by incremental or mini-batch updates under random reshuffling (Baek et al., 2026). Optimizer geometry and population performance. Implicit-bias results do not by themselves imply that one optimizer generalizes better than another. Constructed separable problems show that Euclidean gradient methods and adaptive coordinate-wise methods can interpolate the same training data yet have sharply different test errors (Wilson et al., 2017), and norm-based analyses relate such differences to the interaction between an optimizer’s geometry and the geometry of the data (Vaswani et al., 2020). Experiments on transformers further suggest that the sign-like component of Adam, rather than stochastic noise alone, can account for part of its difference from SGD (Kunstner et al., 2023). Our contribution is a distribution-specific, high-probability ordering of row, spectral, and entrywise geometries in a multiclass model, uniform over the corresponding complete margin faces. Matrix-aware optimization geometry. Several optimizer families exploit matrix or tensor structure without implementing the normalized steepest maps studied here. Shampoo uses Kronecker-factored, mode-wise preconditioners for tensor parameters (Gupta et al., 2018), while SOAP applies Adam in an evolving Shampoo eigenbasis (Vyas et al., 2025). The modular-duality framework instead assigns layerwise operator norms from the semantics of a module and derives corresponding duality maps (Bernstein and Newhouse, 2025), and norm-constrained linear minimization oracles provide a related route to architecture-aware updates (Pethick et al., 2025). Muon can be viewed as the matrix duality map obtained from the polar factor, commonly approximated by Newton–Schulz iterations (Jordan et al., 2024; Bernstein and Newhouse, 2025). By contrast, our spectral comparator is the exact polar update UV⊤UV for a gradient with compact singular value decomposition UΣV⊤U V ; our theorems do not automatically transfer to preconditioned methods such as Shampoo or SOAP, or to finite-iteration approximations of the polar factor. Row normalization and row–spectral hybrids. The row-structured methods summarized in Section 1.2 are algorithmically distinct even when they contain a superficially similar normalization step. In particular, SinkGD alternates row and column normalization, NorMuon performs neuron-wise adaptation after Muon orthogonalization, RMNP analyzes row-normalized momentum through nonconvex convergence guarantees, and MOGA studies width scaling and hyperparameter transfer for the corresponding operator geometry (Scetbon et al., 2025; Li et al., 2026b; Deng et al., 2026; Xu et al., 2026). Recent hybrid proposals further modify the polar update by projecting momentum before row alignment, equilibrating rows before orthogonalization, or imposing leverage-aware row uniformity while retaining spectral structure (Yuan et al., 2026; Chang et al., 2026; Dewulf et al., 2026). SWAN is also a relevant stateless optimizer that combines column normalization with whitening (Ma et al., 2025). These methods strengthen the practical motivation for studying row geometry, but the extra projection, whitening, equilibration, or orthogonalization steps make them different dynamical systems. Our results should therefore be read as an analysis of exact row-normalized steepest descent and momentum, and as a controlled comparison that isolates the row normalization component rather than as an implicit-bias theorem for every row–spectral hybrid. High-dimensional classification under mixture models. Gaussian and sub-Gaussian mixture models have been used to obtain sharp asymptotic formulas or risk bounds for regularized convex classifiers and binary max-margin classifiers (Mignacco et al., 2020; Cao et al., 2021). In the multiclass setting, prior work relates the hard-margin multiclass SVM to minimum-norm interpolation and studies benign overfitting under Gaussian designs (Wang et al., 2021b), including regimes in which the number of classes grows with dimension (Subramanian et al., 2022). These works primarily fix an estimator geometry, usually Euclidean, and characterize when its interpolation or maximum-margin bias generalizes. We instead hold the data model fixed and compare the distinct margin geometries selected by several optimizers in the proportional regime k/d→α∈(0,∞)k/d→α∈(0,∞). The uniform statements over complete margin faces are important in this regime because a norm-specific margin problem need not have a unique optimizer, so identifying one convenient solution would not establish an optimizer-level population comparison. Last-layer geometry and neural collapse. The class-conditional mean geometry in our model is also related to neural collapse, where terminal-phase representations approach within-class concentration, centered class means form a simplex-like frame, and classifier weights align with those means (Papyan et al., 2020). Generalized neural-collapse formulations extend this viewpoint to settings with many more classes than feature dimensions by replacing the simplex equiangular tight frame with a one-versus-rest margin criterion (Jiang et al., 2024). Related geometric regularities have been reported for the unembedding representations of language models under the name linguistic collapse (Wu and Papyan, 2024). Our model does not attempt to explain the feature-learning dynamics that create such configurations. Instead, it conditions on a fixed last-layer representation with random Gaussian class centers and isotropic or anisotropic residual variation, and asks which optimizer-induced norm produces the most accurate off-sample decision regions once that representation has been formed. Appendix B Implicit Bias of Row-Normalized Methods: Full-Batch and Random Reshuffling This appendix collects the optimizer-side implicit-bias results used throughout the paper. For full-batch normalized steepest descent, we extend the analysis of Fan et al. (2025) from entrywise and Schatten norms to every max–sum dominated matrix norm. For random-reshuffling momentum methods, we extend the analysis of Li et al. (2026a) and use a sharper geometry-adapted analysis based on the primal–dual atom radius Rψ(X)R_ψ(X), which avoids the common entrywise ℓ1 _1 feature bound and applies directly to the Row, Spectral, and Max geometries. We then specialize these general results to row normalization, thereby proving Proposition B.14 and the R-RowSGDM margin result used in the stochastic accuracy theorem. B.1 Common setup and auxiliary lemmas Let X=(xn,yn)n=1NX=\(x_n,y_n)\_n=1^N be a fixed linearly separable multiclass dataset with xn∈ℝdx_n ^d and yn∈[k]y_n∈[k]. A linear classifier is W∈ℝk×dW ^k× d, and the cross-entropy loss is L(W):=1N∑n=1Nℓn(W),ℓn(W):=−log(Wxn)yn.L(W):= 1N _n=1^N _n(W), _n(W):=- S(Wx_n)_y_n. (20) Define marX(W):=minn∈[N]minc≠yn(wyn−wc)⊤xn,γψ(X):=maxψ(W)≤1marX(W)mar_X(W):= _n∈[N] _c≠ y_n(w_y_n-w_c) x_n, _ψ(X):= _ψ(W)≤ 1mar_X(W) (21) for any matrix norm ψ, and define the standard cross-entropy proxy (W):=1N∑n=1N(1−(Wxn)yn).G(W):= 1N _n=1^N (1-S(Wx_n)_y_n ). (22) For 0≤ε<γψ(X)0≤ < _ψ(X), define the near-max-margin direction set ψ,εX:=W≠:marX(W)ψ(W)≥γψ(X)−ε,ψX:=ψ,0X.U_ψ, ^X:= \W≠0: mar_X(W)ψ(W)≥ _ψ(X)- \, _ψ^X:=U_ψ,0^X. (23) For a nonzero matrix A and a nonempty set ⊂ℝk×d∖S ^k× d \0\, let dcos(A,):=1−supV∈⟨A,V⟩‖A‖F‖V‖F.d_ (A,S):=1- _V A,V \|A\|_F\|V\|_F. (24) Lemma B.1 (Effective margin implies direction-set convergence). Suppose Wt≠W_t≠0 eventually and, for some 0≤ε<γψ(X)0≤ < _ψ(X), lim inft→∞marX(Wt)ψ(Wt)≥γψ(X)−ε. _t→∞ mar_X(W_t)ψ(W_t)≥ _ψ(X)- . (25) Then dcos(Wt,ψ,εX)⟶0.d_ (W_t,U_ψ, ^X) 0. (26) Proof. By scale invariance of dcosd_ , consider Wt/ψ(Wt)W_t/ψ(W_t), which lies on the compact ψ-unit sphere. If the conclusion failed, a subsequence whose cosine discrepancy from ψ,εXU_ψ, ^X is bounded below by a positive constant would have a further subsequence converging to some U∞U_∞. Continuity of the margin and (25) give marX(U∞)≥γψ(X)−εmar_X(U_∞)≥ _ψ(X)- , so U∞∈ψ,εXU_∞ _ψ, ^X, contradicting continuity of the cosine discrepancy. ∎ Lemma B.2 (Exact margin implies direction-set convergence). If marX(Wt)ψ(Wt)⟶γψ(X), mar_X(W_t)ψ(W_t) _ψ(X), (27) then dcos(Wt,ψX)⟶0.d_ (W_t,U_ψ^X) 0. (28) Proof. Apply Lemma B.1 with ε=0 =0. ∎ For a learning-rate sequence, write St:=∑s=0t−1ηsS_t:= _s=0^t-1 _s. Lemma B.3 (Cumulative remainders for general learning rates). Suppose Assumption 2.5 holds. Then St→∞S_t→∞, and for every q>1q>1, c>0c>0, and fixed t0∈ℕt_0 , ∑s=0t−1ηsqSt⟶0,∑s=t0t−1ηsexp(−c∑τ=t0s−1ητ)=O(1). _s=0^t-1 _s^qS_t 0, _s=t_0^t-1 _s \! (-c _τ=t_0^s-1 _τ )=O(1). (29) Proof. Fix ε>0 >0 and choose T so that ηsq−1≤ε _s^q-1≤ for all s≥Ts≥ T. Then ∑s=0t−1ηsqSt≤∑s=0T−1ηsqSt+ε, _s=0^t-1 _s^qS_t≤ _s=0^T-1 _s^qS_t+ , which proves the first limit. For the second assertion, enlarge T≥t0T≥ t_0 so that cηs≤1c _s≤ 1 for s≥Ts≥ T, and set As:=∑τ=t0s−1ητA_s:= _τ=t_0^s-1 _τ. Since 1−e−x≥x/21-e^-x≥ x/2 for x∈[0,1]x∈[0,1], e−cAs−e−cAs+1≥c2ηse−cAs.e^-cA_s-e^-cA_s+1≥ c2 _se^-cA_s. The tail therefore telescopes to a bounded quantity, while the finite prefix is harmless. ∎ Lemma B.4 (Normalization of a cumulative margin bound). Let :ℝk×d→ℝ m:R^k× d be continuous and positively homogeneous, let Γ∗:=maxψ(W)≤1(W)>0 _*:= _ψ(W)≤ 1 m(W)>0, and suppose 0<Γ≤Γ∗0< ≤ _*. Assume St→∞S_t→∞. If (Wt)≥ΓSt−rt,ψ(Wt)≤C0+St,rt=o(St), m(W_t)≥ S_t-r_t, ψ(W_t)≤ C_0+S_t, r_t=o(S_t), then lim inft→∞(Wt)ψ(Wt)≥Γ. _t→∞ m(W_t)ψ(W_t)≥ . (30) If Γ=Γ∗ = _*, then (Wt)/ψ(Wt)→Γ∗ m(W_t)/ψ(W_t)→ _*, and every accumulation point of Wt/ψ(Wt)W_t/ψ(W_t) maximizes m over the ψ-unit ball. Proof. Since rt=o(St)r_t=o(S_t) and St→∞S_t→∞, the lower bound ΓSt−rt S_t-r_t is positive for all sufficiently large t. In particular, Wt≠0W_t≠ 0 eventually. For such t, (Wt)ψ(Wt)≥ΓSt−rtC0+St⟶Γ. m(W_t)ψ(W_t)≥ S_t-r_tC_0+S_t . This proves the liminf statement without imposing a sign condition on rtr_t. When Γ=Γ∗ = _*, positive homogeneity gives (Wt)/ψ(Wt)≤Γ∗ m(W_t)/ψ(W_t)≤ _*, proving convergence. Compactness of the ψ-unit sphere and continuity of m give the accumulation-point statement. ∎ B.2 Full-batch max–sum dominated norms For a matrix A, write ‖A‖max:=maxr,j|Ar,j|,‖A‖sum:=∑r,j|Ar,j|.\|A\|_ := _r,j|A_r,j|, \|A\|_sum:= _r,j|A_r,j|. We call a matrix norm ψ max–sum dominated if ‖A‖max≤ψ(A)≤‖A‖sumfor every A.\|A\|_ ≤ψ(A)≤\|A\|_sum every A. (31) Let R1(X):=maxn∈[N]‖xn‖1<∞R_1(X):= _n∈[N]\|x_n\|_1<∞. Lemma B.5 (Consequences of max–sum domination). Let ψ satisfy (31), and let ψ∗ψ^* be its dual norm. Then, for every matrix G, every h with ‖h‖1≤R1(X)\|h\|_1≤ R_1(X), and every A∈ℝk×dA ^k× d, ψ∗(G)≤‖G‖sum,‖Ah‖∞≤R1(X)ψ(A).ψ^*(G)≤\|G\|_sum, \|Ah\|_∞≤ R_1(X)ψ(A). (32) Moreover, if s∈ℝks ^k is a probability vector and H(s):=diag(s)−ss⊤H(s):=diag(s)-s , then for every c∈[k]c∈[k], h⊤A⊤H(s)Ah≤4R1(X)2(1−sc)ψ(A)2.h A H(s)Ah≤ 4R_1(X)^2(1-s_c)ψ(A)^2. (33) Proof. Since the ψ-unit ball is contained in the entrywise max-norm unit ball, ψ∗(G)=maxψ(A)≤1⟨G,A⟩≤max‖A‖max≤1⟨G,A⟩=‖G‖sum.ψ^*(G)= _ψ(A)≤ 1 G,A ≤ _\|A\|_ ≤ 1 G,A =\|G\|_sum. Also, ‖Ah‖∞≤‖A‖max‖h‖1≤R1(X)ψ(A)\|Ah\|_∞≤\|A\|_ \|h\|_1≤ R_1(X)ψ(A). For v=Ahv=Ah, the softmax covariance bound gives v⊤H(s)v≤4(1−sc)‖v‖∞2v H(s)v≤ 4(1-s_c)\|v\|_∞^2, which proves (33). ∎ Theorem B.6 (Full-batch max–sum dominated margin convergence). Let ψ be max–sum dominated. Under Assumption 2.5, normalized steepest descent Δt∈argmaxψ(Δ)≤1⟨∇L(Wt),Δ⟩,Wt+1=Wt−ηtΔt _t∈ *arg\,max_ψ( )≤ 1 ∇ L(W_t), , W_t+1=W_t- _t _t (34) satisfies marX(Wt)ψ(Wt)⟶γψ(X). mar_X(W_t)ψ(W_t) _ψ(X). (35) Under Assumptions 2.5 and 2.6, the same conclusion holds for every fixed β∈(0,1)β∈(0,1) under normalized momentum descent with M0=M_0=0: Mt+1=βMt+(1−β)∇L(Wt),Δt∈argmaxψ(Δ)≤1⟨Mt+1,Δ⟩,Wt+1=Wt−ηtΔt,M_t+1=β M_t+(1-β)∇ L(W_t), _t∈ *arg\,max_ψ( )≤ 1 M_t+1, , W_t+1=W_t- _t _t, (36) In either case, every accumulation point of Wt/ψ(Wt)W_t/ψ(W_t) is a ψ-maximum-margin solution. Proof. The geometry-dependent steps in the normalized steepest-descent and momentum proofs of Fan et al. (2025, Appendices D and E) use only norm duality, the lower bound ψ∗(∇L(W))≥γψ(X)(W)ψ^*(∇ L(W))≥ _ψ(X)G(W), the upper bound ψ∗(∇L(W))≤2R1(X)(W)ψ^*(∇ L(W))≤ 2R_1(X)G(W), and the logit and Hessian controls in Lemma B.5. Substituting these bounds into their pre-rate scalar argument gives marX(Wt)≥γψ(X)St−rt,ψ(Wt)≤ψ(W0)+St,mar_X(W_t)≥ _ψ(X)S_t-r_t, ψ(W_t)≤ψ(W_0)+S_t, where rtr_t is a finite-prefix term plus sums of the forms ∑sηs2 _s _s^2 and ∑sηsexp(−c∑τ<sητ) _s _s (-c _τ<s _τ); in the momentum case, Assumption 2.6 controls the additional geometric-memory term. Lemma B.3 gives rt=o(St)r_t=o(S_t), and Lemma B.4 with =marX m=mar_X and Γ=Γ∗=γψ(X) = _*= _ψ(X) proves the result. ∎ The full-batch Adam characterization used in Theorem 3.1 also extends to the learning-rate assumptions of the present paper. Theorem B.7 (Full-batch Adam under the present learning-rate assumptions). Consider full-batch Adam without bias correction or a numerical-stability constant, initialized by M0=V0=M_0=V_0=0: Mt+1 M_t+1 =β1Mt+(1−β1)∇L(Wt), = _1M_t+(1- _1)∇ L(W_t), Vt+1 V_t+1 =β2Vt+(1−β2)∇L(Wt)⊙2, = _2V_t+(1- _2)∇ L(W_t) 2, Wt+1 W_t+1 =Wt−ηtMt+1⊘Vt+1⊙1/2. =W_t- _tM_t+1 V_t+1 1/2. (37) Assume 0<β1≤β2<10< _1≤ _2<1, Assumptions 2.5 and 2.6, and minc∈[k],j∈[d]|∇L(W0)[c,j]|2≥ω _c∈[k],\,j∈[d]|∇ L(W_0)[c,j]|^2≥ω for some ω>0ω>0. Then, with γmax(X):=max‖W‖max≤1marX(W), _ (X):= _\|W\|_ ≤ 1mar_X(W), one has marX(Wt)‖Wt‖max⟶γmax(X), mar_X(W_t)\|W_t\|_ _ (X), (38) and every accumulation point of Wt/‖Wt‖maxW_t/\|W_t\|_ is a max-norm maximum-margin solution. Proof. The pre-rate part of Fan et al. (2025, Appendix G) gives marX(Wt)≥γmax(X)St−rt,‖Wt‖max≤C0+St,mar_X(W_t)≥ _ (X)S_t-r_t, \|W_t\|_ ≤ C_0+S_t, where rtr_t consists of a finite prefix, ∑sηs3/2 _s _s^3/2, and an exponentially weighted remainder. Lemma B.3 gives rt=o(St)r_t=o(S_t), and Lemma B.4 proves (38) and the accumulation-point conclusion. ∎ B.3 Geometry-adapted random-reshuffling momentum The full-batch max–sum argument above is sufficient for deterministic optimization, but using the same common feature radius maxn‖xn‖1 _n\|x_n\|_1 in the stochastic analysis is unnecessarily coarse when comparing different geometries. We therefore work directly with the primal–dual pair (ψ,ψ∗)(ψ,ψ^*) and define the atom radius Rψ(X):=maxn∈[N]maxc∈[k]ψ∗(ecxn⊤).R_ψ(X):= _n∈[N] _c∈[k]ψ^*(e_cx_n ). (39) The stochastic results below hold for an arbitrary matrix norm ψ; max–sum domination is not required. Assume b|Nb N, set m:=N/bm:=N/b, and for every mini-batch B⊂[N]B⊂[N] of size b define LB(W):=b−1∑n∈Bℓn(W)L_B(W):=b^-1 _n∈ B _n(W). Lemma B.8 (Geometry-adapted cross-entropy bounds). For every matrix norm ψ, every sample xnx_n, every matrix A, and all classes a,c∈[k]a,c∈[k], ‖Axn‖∞≤Rψ(X)ψ(A),ψ∗((ea−ec)xn⊤)≤2Rψ(X).\|Ax_n\|_∞≤ R_ψ(X)ψ(A), ψ^*((e_a-e_c)x_n )≤ 2R_ψ(X). (40) For every W, γψ(X)(W)≤ψ∗(∇L(W))≤2Rψ(X)(W). _ψ(X)G(W)≤ψ^*(∇ L(W))≤ 2R_ψ(X)G(W). (41) For every sample n, every W, and every perturbation A, D2ℓn(W)[A,A]≤4Rψ(X)2(1−(Wxn)yn)ψ(A)2.D^2 _n(W)[A,A]≤ 4R_ψ(X)^2 (1-S(Wx_n)_y_n )ψ(A)^2. (42) Moreover, for all U,WU,W, |L(U)−L(W)|≤2Rψ(X)ψ(U−W).|L(U)-L(W)|≤ 2R_ψ(X)ψ(U-W). (43) Proof. For every c∈[k]c∈[k], norm duality gives |ec⊤Axn|=|⟨A,ecxn⊤⟩|≤ψ(A)ψ∗(ecxn⊤)≤Rψ(X)ψ(A),|e_c Ax_n|=| A,e_cx_n |≤ψ(A)ψ^*(e_cx_n )≤ R_ψ(X)ψ(A), which proves the first bound in (40); the second follows from the triangle inequality. Writing pn(W):=(Wxn)p_n(W):=S(Wx_n), the sample gradient has the decomposition ∇ℓn(W)=∑c≠ynpn,c(W)(ec−eyn)xn⊤,∇ _n(W)= _c≠ y_np_n,c(W)(e_c-e_y_n)x_n , so ψ∗(∇ℓn(W))≤2Rψ(X)(1−pn,yn(W)).ψ^*(∇ _n(W))≤ 2R_ψ(X) (1-p_n,y_n(W) ). Averaging proves the upper bound in (41). For the lower bound, choose A⋆A_ with ψ(A⋆)≤1ψ(A_ )≤ 1 and margin γψ(X) _ψ(X); then −∇L(W)=1N∑n=1N∑c≠ynpn,c(W)(eyn−ec)xn⊤-∇ L(W)= 1N _n=1^N _c≠ y_np_n,c(W)(e_y_n-e_c)x_n gives ψ∗(∇L(W))≥⟨A⋆,−∇L(W)⟩≥γψ(X)(W)ψ^*(∇ L(W))≥ A_ ,-∇ L(W) ≥ _ψ(X)G(W). For the Hessian bound, set v:=Axnv:=Ax_n and let C be categorical with law pn(W)p_n(W). Then D2ℓn(W)[A,A]=Var(vC)≤[(vC−vyn)2]≤4(1−pn,yn(W))‖v‖∞2,D^2 _n(W)[A,A]=Var(v_C) [(v_C-v_y_n)^2]≤ 4(1-p_n,y_n(W))\|v\|_∞^2, which proves (42). Finally, convexity and the upper gradient bound, together with ≤1G≤ 1, yield L(U)−L(W)≤2Rψ(X)ψ(U−W)L(U)-L(W)≤ 2R_ψ(X)ψ(U-W); exchanging U,WU,W proves (43). ∎ Lemma B.9 (Geometry-adapted softmax and gradient stability). For U,W∈ℝk×dU,W ^k× d, define h:=Rψ(X)ψ(U−W)h:=R_ψ(X)ψ(U-W). Then, for every sample n and class c, e−2h≤pn,c(U)pn,c(W)≤e2h,e−2h≤1−pn,yn(U)1−pn,yn(W)≤e2h.e^-2h≤ p_n,c(U)p_n,c(W)≤ e^2h, e^-2h≤ 1-p_n,y_n(U)1-p_n,y_n(W)≤ e^2h. (44) Consequently, (U)≤e2Rψ(X)ψ(U−W)(W),G(U)≤ e^2R_ψ(X)ψ(U-W)G(W), (45) ψ∗(∇L(U)−∇L(W))≤2Rψ(X)(e2Rψ(X)ψ(U−W)−1)(W),ψ^*(∇ L(U)-∇ L(W))≤ 2R_ψ(X) (e^2R_ψ(X)ψ(U-W)-1 )G(W), (46) and, for every mini-batch B, ψ∗(∇LB(U)−∇LB(W))≤2mRψ(X)(e2Rψ(X)ψ(U−W)−1)(W).ψ^*(∇ L_B(U)-∇ L_B(W))≤ 2mR_ψ(X) (e^2R_ψ(X)ψ(U-W)-1 )G(W). (47) Proof. Set δn:=(U−W)xn _n:=(U-W)x_n. Lemma B.8 gives ‖δn‖∞≤h\| _n\|_∞≤ h, and the softmax formula gives pn,c(U)pn,c(W)=eδn,c∑a=1keea⊤Wxn∑a=1keea⊤Wxn+δn,a∈[e−2h,e2h]. p_n,c(U)p_n,c(W)=e _n,c _a=1^ke^e_a Wx_n _a=1^ke^e_a Wx_n+ _n,a∈[e^-2h,e^2h]. Summing over c≠ync≠ y_n proves the second ratio and (45). If p:=pn(W)p:=p_n(W) and q:=pn(U)q:=p_n(U), the ratio bounds imply ∑c≠yn|qc−pc|≤(e2h−1)(1−pyn),|qyn−pyn|≤(e2h−1)(1−pyn). _c≠ y_n|q_c-p_c|≤(e^2h-1)(1-p_y_n), |q_y_n-p_y_n|≤(e^2h-1)(1-p_y_n). Since ∇ℓn(U)−∇ℓn(W)=(q−p)xn⊤∇ _n(U)-∇ _n(W)=(q-p)x_n , ψ∗((q−p)xn⊤)≤∑c=1k|qc−pc|ψ∗(ecxn⊤)≤2Rψ(X)(e2h−1)(1−pyn).ψ^*((q-p)x_n )≤ _c=1^k|q_c-p_c|ψ^*(e_cx_n )≤ 2R_ψ(X)(e^2h-1)(1-p_y_n). Averaging over all samples proves (46); averaging over B, enlarging the resulting nonnegative sample sum to [N][N], and using N/b=mN/b=m proves (47). ∎ Lemma B.10 (Random-reshuffling noise and cancellation). For a mini-batch B of size b, define ΞB(W):=∇LB(W)−∇L(W) _B(W):=∇ L_B(W)-∇ L(W). Then ψ∗(ΞB(W))≤2(m−1)Rψ(X)(W).ψ^*( _B(W))≤ 2(m-1)R_ψ(X)G(W). (48) Moreover, let Btt≥0\B_t\_t≥ 0 be any random-reshuffling sequence, so that Bqm,…,Bqm+m−1B_qm,…,B_qm+m-1 partition [N][N] for every epoch q, and define Et(W):=(1−β)∑τ=0tβτ(∇LBt−τ(W)−∇L(W)).E_t(W):=(1-β) _τ=0^tβ^τ (∇ L_B_t-τ(W)-∇ L(W) ). (49) Then, deterministically for every reshuffling realization, ψ∗(Et(W))≤(1−β)m(m2−1)Rψ(X)(W).ψ^*(E_t(W))≤(1-β)m(m^2-1)R_ψ(X)G(W). (50) Proof. When m≥2m≥ 2, let Bc=[N]∖B^c=[N] B. The finite-population identity ΞB(W)=m−1m(∇LB(W)−∇LBc(W)) _B(W)= m-1m (∇ L_B(W)-∇ L_B^c(W) ) and the single-sample bound in the proof of Lemma B.8 give ψ∗(ΞB(W))≤m−1N∑n=1Nψ∗(∇ℓn(W))≤2(m−1)Rψ(X)(W),ψ^*( _B(W))≤ m-1N _n=1^Nψ^*(∇ _n(W))≤ 2(m-1)R_ψ(X)G(W), while m=1m=1 is immediate. For the second assertion, write Ξt(W):=∇LBt(W)−∇L(W) _t(W):=∇ L_B_t(W)-∇ L(W). Every complete epoch satisfies ∑r=0m−1Ξqm+r(W)= _r=0^m-1 _qm+r(W)=0. Put KΞ(W):=2(m−1)Rψ(X)(W)K_ (W):=2(m-1)R_ψ(X)G(W). For a complete epoch contained in 0,…,t\0,…,t\, subtracting the constant coefficient βt−qmβ^t-qm and using 1−βr≤r(1−β)1-β^r≤ r(1-β) gives ψ∗(∑r=0m−1βt−qm−rΞqm+r(W))≤KΞ(W)βt−(q+1)m(1−β)m(m−1)2.ψ^* ( _r=0^m-1β^t-qm-r _qm+r(W) )≤ K_ (W)β^t-(q+1)m(1-β) m(m-1)2. Summing over complete epochs and using (1−β)/(1−βm)≤1(1-β)/(1-β^m)≤ 1, then adding the at most m terms in the current incomplete epoch, gives ψ∗(Et(W))≤(1−β)m(m+1)2KΞ(W)=(1−β)m(m2−1)Rψ(X)(W).ψ^*(E_t(W))≤(1-β) m(m+1)2K_ (W)=(1-β)m(m^2-1)R_ψ(X)G(W). ∎ Consider now the moving-weight random-reshuffling momentum process Mt+1=βMt+(1−β)∇LBt(Wt),Δt∈argmaxψ(Δ)≤1⟨Mt+1,Δ⟩,Wt+1=Wt−ηtΔt,M_t+1=β M_t+(1-β)∇ L_B_t(W_t), _t∈ *arg\,max_ψ( )≤ 1 M_t+1, , W_t+1=W_t- _t _t, (51) with M0=M_0=0. Since ψ(Δt)≤1ψ( _t)≤ 1, ψ(Wt−Wt−τ)≤∑q=1τηt−q.ψ(W_t-W_t-τ)≤ _q=1^τ _t-q. (52) Proposition B.11 (Geometry-adapted random-reshuffling momentum control). Assume Assumptions 2.5 and 2.6. For every fixed finite dataset, fixed matrix norm ψ, and fixed β∈(0,1)β∈(0,1), there exist t0,ψ,β<∞t_0,ψ,β<∞ and cψ,β<∞c_ψ,β<∞ such that, for all t≥t0,ψ,βt≥ t_0,ψ,β, ψ∗(Mt+1−∇L(Wt))≤[ ψ^*(M_t+1-∇ L(W_t))≤ [ 2m(1−β)Rψ(X)cψ,βηt+(1−β)m(m2−1)Rψ(X) 2m(1-β)R_ψ(X)c_ψ,β _t+(1-β)m(m^2-1)R_ψ(X) +2Rψ(X)βt+1](Wt). +2R_ψ(X)β^t+1 ]G(W_t). (53) Define ρψ,β,b(X):=γψ(X)−2(1−β)m(m2−1)Rψ(X), _ψ,β,b(X):= _ψ(X)-2(1-β)m(m^2-1)R_ψ(X), (54) Cψ,β,b:=4m(1−β)Rψ(X)cψ,β+2Rψ(X)2e2Rψ(X)η0,Dψ:=4Rψ(X).C_ψ,β,b:=4m(1-β)R_ψ(X)c_ψ,β+2R_ψ(X)^2e^2R_ψ(X) _0, D_ψ:=4R_ψ(X). (55) Then, for all sufficiently large t, L(Wt+1)≤L(Wt)−ηt(ρψ,β,b(X)−Cψ,β,bηt−Dψβt/2)(Wt).L(W_t+1)≤ L(W_t)- _t ( _ψ,β,b(X)-C_ψ,β,b _t-D_ψβ^t/2 )G(W_t). (56) Proof. Unrolling the momentum recursion and adding and subtracting frozen batch gradients at WtW_t gives Mt+1−∇L(Wt)= M_t+1-∇ L(W_t)= (1−β)∑τ=0tβτ(∇LBt−τ(Wt−τ)−∇LBt−τ(Wt)) (1-β) _τ=0^tβ^τ (∇ L_B_t-τ(W_t-τ)-∇ L_B_t-τ(W_t) ) +(1−β)∑τ=0tβτ(∇LBt−τ(Wt)−∇L(Wt))−βt+1∇L(Wt). +(1-β) _τ=0^tβ^τ (∇ L_B_t-τ(W_t)-∇ L(W_t) )-β^t+1∇ L(W_t). (57) Lemma B.9 and (52) bound the first term by 2m(1−β)Rψ(X)(Wt)∑τ=0tβτ(exp(2Rψ(X)∑q=1τηt−q)−1).2m(1-β)R_ψ(X)G(W_t) _τ=0^tβ^τ ( (2R_ψ(X) _q=1^τ _t-q )-1 ). Assumption 2.6, applied with c1=2Rψ(X)c_1=2R_ψ(X), bounds the final sum by cψ,βηtc_ψ,β _t for all sufficiently large t. Lemma B.10 controls the second term, and the upper gradient bound in (41) controls the last, proving (53). For the descent recurrence, the steepest-descent property and norm duality give ⟨∇L(Wt),Δt⟩≥ψ∗(∇L(Wt))−2ψ∗(Mt+1−∇L(Wt)). ∇ L(W_t), _t ≥ψ^*(∇ L(W_t))-2ψ^*(M_t+1-∇ L(W_t)). (58) The mean-value Taylor formula, (42), and ψ(Δt)≤1ψ( _t)≤ 1 bound the quadratic remainder in L(Wt−ηtΔt)L(W_t- _t _t) by 2ηt2Rψ(X)2sup0≤s≤1(Wt−sηtΔt).2 _t^2R_ψ(X)^2 _0≤ s≤ 1G(W_t-s _t _t). Lemma B.9 bounds the supremum by e2Rψ(X)η0(Wt)e^2R_ψ(X) _0G(W_t). Substituting (41) and (53) into (58), adding the quadratic remainder, and using βt+1≤βt/2β^t+1≤β^t/2 proves (56). ∎ Corollary B.12 (Geometry-adapted random-reshuffling momentum margin). Under Assumptions 2.5 and 2.6, suppose ρψ,β,b(X)>0 _ψ,β,b(X)>0. Then lim inft→∞marX(Wt)ψ(Wt)≥ρψ,β,b(X). _t→∞ mar_X(W_t)ψ(W_t)≥ _ψ,β,b(X). (59) Consequently, with εψ,β,b(X):=2(1−β)m(m2−1)Rψ(X), _ψ,β,b(X):=2(1-β)m(m^2-1)R_ψ(X), (60) one has dcos(Wt,ψ,εψ,β,b(X)X)⟶0.d_ \! (W_t,U_ψ, _ψ,β,b(X)^X ) 0. (61) More quantitatively, there exist a finite t2t_2 and Cga<∞C_ga<∞ such that ρψ,β,b(X)−marX(Wt)ψ(Wt)≤Cgaℛtψ,β,bSt, _ψ,β,b(X)- mar_X(W_t)ψ(W_t)≤ C_ga R_t^ψ,β,bS_t, (62) where ℛtψ,β,b:= _t^ψ,β,b:= 1+∑s=0t2−1ηs+Cψ,β,b∑s=t2t−1ηs2+Dψη01−β 1+ _s=0^t_2-1 _s+C_ψ,β,b _s=t_2^t-1 _s^2+ D_ψ _01- β +∑s=t2t−1ηsexp(−ρψ,β,b(X)4∑τ=t2s−1ητ). + _s=t_2^t-1 _s \! (- _ψ,β,b(X)4 _τ=t_2^s-1 _τ ). (63) Proof. Write ρ:=ρψ,β,b(X)>0ρ:= _ψ,β,b(X)>0. Since ηt→0 _t→ 0 and βt/2→0β^t/2→ 0, Proposition B.11 gives a finite t1t_1 after which the coefficient of ηt(Wt) _tG(W_t) is at least ρ/2ρ/2. The loss Lipschitz bound (43), the divergence St→∞S_t→∞, and the standard low-loss entrance argument give a finite t2≥t1t_2≥ t_1 after which all empirical margins are nonnegative. The usual cross-entropy loss-to-margin conversion then yields marX(Wt)≥ρ∑s=t2t−1ηs(Ws)L(Ws)−Cψ,β,b∑s=t2t−1ηs2−Dψη01−β−C0,mar_X(W_t)≥ρ _s=t_2^t-1 _s G(W_s)L(W_s)-C_ψ,β,b _s=t_2^t-1 _s^2- D_ψ _01- β-C_0, where C0<∞C_0<∞ absorbs the finite prefix. The post-entrance descent recurrence and the standard cross-entropy proxy inequalities give (Ws)L(Ws)≥1−exp(−ρ4∑τ=t2s−1ητ). G(W_s)L(W_s)≥ 1- \! (- ρ4 _τ=t_2^s-1 _τ ). Hence marX(Wt)≥ρSt−C1ℛtψ,β,bmar_X(W_t)≥ρ S_t-C_1R_t^ψ,β,b for some C1<∞C_1<∞. Lemma B.3 gives ℛtψ,β,b=o(St)R_t^ψ,β,b=o(S_t), while ψ(Wt)≤ψ(W0)+Stψ(W_t)≤ψ(W_0)+S_t follows directly from the normalized update. Lemma B.4 proves (62) and (59), and Lemma B.1 gives (61). ∎ For the three geometries used in the main text, the atom radius is explicit: Rrow(X)=maxn‖xn‖2,Rsp(X)=maxn‖xn‖2,Rmax(X)=maxn‖xn‖1.R_row(X)= _n\|x_n\|_2, R_sp(X)= _n\|x_n\|_2, R_ (X)= _n\|x_n\|_1. (64) Indeed, the duals of the row, spectral, and entrywise max norms are the sum-of-row-Euclidean, nuclear, and entrywise sum norms, respectively, while ecxn⊤e_cx_n has one nonzero row and rank one. B.4 Specialization to row normalization Define ∥A∥row:=maxr∈[k]∥Ar,:∥2.\|A\|_row:= _r∈[k]\|A_r,:\|_2. (65) For a matrix G, define its row-wise normalization by (Nrow(G))r,::=Gr,:/∥Gr,:∥2,Gr,:≠,,Gr,:=. (N_row(G) )_r,::= casesG_r,:/\|G_r,:\|_2,&G_r,:≠0,\\ 0,&G_r,:=0. cases (66) Proposition B.13 (Row normalization is row-norm steepest descent). For every A,G∈ℝk×dA,G ^k× d, ‖A‖max≤‖A‖row≤‖A‖sum,\|A\|_ ≤\|A\|_row≤\|A\|_sum, (67) and Nrow(G)∈argmax‖Δ‖row≤1⟨G,Δ⟩,max‖Δ‖row≤1⟨G,Δ⟩=∑r=1k∥Gr,:∥2.N_row(G)∈ *arg\,max_\| \|_row≤ 1 G, , _\| \|_row≤ 1 G, = _r=1^k\|G_r,:\|_2. (68) Thus row-wise normalized gradient methods are normalized steepest-descent methods for the row norm. Proof. Every entry is bounded by the Euclidean norm of its row, while maxr∥Ar,:∥2≤∑r∥Ar,:∥1=∥A∥sum _r\|A_r,:\|_2≤ _r\|A_r,:\|_1=\|A\|_sum, proving (67). The constraint ‖Δ‖row≤1\| \|_row≤ 1 is equivalent to ∥Δr,:∥2≤1\| _r,:\|_2≤ 1 for every row r, so max‖Δ‖row≤1⟨G,Δ⟩=∑r=1kmax‖u‖2≤1⟨Gr,:,u⟩=∑r=1k∥Gr,:∥2, _\| \|_row≤ 1 G, = _r=1^k _\|u\|_2≤ 1 G_r,:,u = _r=1^k\|G_r,:\|_2, and Nrow(G)N_row(G) attains the maximum. ∎ Proposition B.14 (Row-normalized full-batch margin convergence). Consider any fixed linearly separable multiclass dataset with bounded features. Under Assumption 2.5, row-wise normalized GD satisfies marX(Wt)‖Wt‖row⟶γrow(X),infV∈rowX‖Wt‖Wt‖F−V‖V‖F‖F⟶0. mar_X(W_t)\|W_t\|_row _row(X), _V _row^X \| W_t\|W_t\|_F- V\|V\|_F \|_F 0. (69) If Assumption 2.6 also holds, the same two conclusions hold for row-wise normalized GD with every fixed momentum parameter β∈(0,1)β∈(0,1). Proof. Proposition B.13 shows that the row norm is max–sum dominated and that row normalization is its exact steepest direction. Theorem B.6 therefore gives normalized-margin convergence, and Lemma B.2 gives dcos(Wt,rowX)→0d_ (W_t,U_row^X)→ 0. Since infV∈rowX‖Wt‖Wt‖F−V‖V‖F‖F2=2dcos(Wt,rowX), _V _row^X \| W_t\|W_t\|_F- V\|V\|_F \|_F^2=2d_ (W_t,U_row^X), the Frobenius direction-set convergence follows. ∎ The stochastic counterpart follows from the geometry-adapted random-reshuffling result above and is stated separately because the approximation gap depends on the batch size and momentum. Proposition B.15 (Random-reshuffling row-wise normalized SGDM margin convergence). Consider any fixed linearly separable multiclass dataset with bounded features. Run random-reshuffling row-wise normalized SGDM with b|Nb N, m:=N/bm:=N/b, and fixed β∈(0,1)β∈(0,1). Define Rrow(X):=maxn∈[N]‖xn‖2,εrow,β,b(X):=2(1−β)m(m2−1)Rrow(X).R_row(X):= _n∈[N]\|x_n\|_2, _row,β,b(X):=2(1-β)m(m^2-1)R_row(X). (70) Under Assumptions 2.5 and 2.6, if εrow,β,b(X)<γrow(X) _row,β,b(X)< _row(X), then lim inft→∞marX(Wt)‖Wt‖row≥γrow(X)−εrow,β,b(X), _t→∞ mar_X(W_t)\|W_t\|_row≥ _row(X)- _row,β,b(X), (71) and dcos(Wt,row,εrow,β,b(X)X)⟶0.d_ \! (W_t,U_row, _row,β,b(X)^X ) 0. (72) Proof. Apply Corollary B.12 with ψ=∥⋅∥rowψ=\|·\|_row. Equation (64) gives Rψ(X)=Rrow(X)R_ψ(X)=R_row(X), so (54) becomes γrow(X)−εrow,β,b(X) _row(X)- _row,β,b(X), yielding (71). Equation (72) follows from (61). ∎ Appendix C Proofs for Deterministic Optimizers This appendix proves Theorem 3.1 from optimizer-selected margin geometry to population accuracy. We first establish the all-aspect-ratio Row-versus-Spectral and Row-versus-Max geometry: Row representatives align uniformly with the population mean differences, Max representatives have a nonvanishing clipped-coordinate angular gap, and Spectral representatives are controlled through the reduced polar factor of the centered empirical mean matrix, with hard-edge truncation at α=1α=1. We then convert these geometric statements into ordinary multiclass population-error bounds and prove the optimizer-level accuracy ordering. We also derive deterministic population-error exponents that quantify the separation between the three geometries. For any matrix norm ψ, define the unit-margin feasible set and the corresponding minimum-norm representative set by ℱX:=W∈ℝk×d:marX(W)≥1,ℳψX:=argminW∈ℱXψ(W).F_X:= \W ^k× d:mar_X(W)≥ 1 \, _ψ^X:= *arg\,min_W _Xψ(W). (73) Whenever X is linearly separable, ℱXF_X is nonempty, the minimum is attained, and we define rψ∗(X):=minW∈ℱXψ(W)=1γψ(X),ψX=aW:a>0,W∈ℳψX.r_ψ^*(X):= _W _Xψ(W)= 1 _ψ(X), _ψ^X= \aW:a>0,\ W _ψ^X \. (74) For q∈row,sp,maxq∈\row,sp, \, we abbreviate ℳqX:=ℳψqXM_q^X:=M_ _q^X and rq∗(X):=rψq∗(X)r_q^*(X):=r_ _q^*(X). For every feasible classifier and every ordered pair i≠ji≠ j, define uijW:=wi−wj‖wi−wj‖2,uijμ:=μi−μj‖μi−μj‖2,Davg-bdry(W):=1k(k−1)∑i≠j‖uijW−uijμ‖2.u_ij^W:= w_i-w_j\|w_i-w_j\|_2, u_ij^μ:= _i- _j\| _i- _j\|_2, D_avg -bdry(W):= 1k(k-1) _i≠ j\|u_ij^W-u_ij^μ\|_2. (75) These quantities are invariant under positive rescaling of W and under common row shifts. Lemma C.1 (Exact invariance under common feature rescaling). Let c>0c>0, let cX:=(cxn,yn)n=1NcX:=\(cx_n,y_n)\_n=1^N, and let ψ be any matrix norm. Then ℱcX=c−1ℱX,ℳψcX=c−1ℳψX,ψcX=ψX.F_cX=c^-1F_X, _ψ^cX=c^-1M_ψ^X, _ψ^cX=U_ψ^X. (76) The normalized pairwise row directions and Davg-bdryD_avg -bdry are unchanged. If the test inputs are also multiplied by c, then Accπ,d(c−1W;cXte)=Accπ,d(W;Xte)Acc_π,d(c^-1W;cX^te)=Acc_π,d(W;X^te). Proof. The constraint (wyn−wj)⊤(cxn)≥1(w_y_n-w_j) (cx_n)≥ 1 is equivalent, after writing W=c−1VW=c^-1V, to (vyn−vj)⊤xn≥1(v_y_n-v_j) x_n≥ 1. Positive homogeneity of ψ proves the identities for the feasible and minimum-norm sets, and the ray identity follows from (74). Finally, (c−1wj)⊤(cx)=wj⊤x(c^-1w_j) (cx)=w_j x, so all predictions agree. ∎ Apply Lemma C.1 with c=d−1/2c=d^-1/2. Thus, throughout the geometric and accuracy appendices, we work with μ~i:=μid,z~i,s:=zi,sd,x~i,s:=xi,sd, μ_i:= _i d, z_i,s:= z_i,s d, x_i,s:= x_i,s d, (77) and suppress tildes. Under this convention, μi,zi,s∼iid(,Id/d) _i,z_i,s iid N(0,I_d/d) and Xte|(Y=i)∼(μi,rte2Id/d)X^te (Y=i) ( _i,r_te^2I_d/d). This is only a proof normalization: the direction sets, accuracy values, and radius-to-margin ratios used in the optimizer theorems are identical to those in the unnormalized model of Assumption 2.1. Let G∼(0,1)G (0,1), and define Amax(τ):=[|G|minτ|G|,1],Bmax(τ):=[minτ2G2,1].A_ (τ):=E\! [|G| \τ|G|,1\ ], B_ (τ):=E\! [ \τ^2G^2,1\ ]. (78) Let τ∗∈(0,1) _*∈(0,1) be the unique solution of Amax(τ∗)=1/πA_ ( _*)=1/ π, and define Cmax:=(2πBmax(τ∗))−1/2,κmax:=2Cmax<1,δmax:=2−2κmax>0.C_ := (2π B_ ( _*) )^-1/2, _ := 2\,C_ <1, _ := 2-2 _ >0. (79) For α>0α>0, let να _α be the mean-one Marchenko–Pastur law with parameter 1/α1/α, including its zero atom when α<1α<1: να _α =(1−α)+δ0+α2πx(bα−x)(x−aα) 1[aα,bα](x)dx, =(1-α)_+ _0+ α2π x (b_α-x)(x-a_α)\,1_[a_α,b_α](x)\,dx, aα a_α =(1−α−1/2)2,bα=(1+α−1/2)2. =(1-α^-1/2)^2, b_α=(1+α^-1/2)^2. (80) Set ρα:=α∧1,m1/2(α):=∫xdνα(x),sα:=m1/2(α)α,κsp(α):=m1/2(α)ρα, _α:=α 1, m_1/2(α):= x\,d _α(x), s_α:= m_1/2(α) α, _sp(α):= m_1/2(α) _α, (81) and δsp(α):=2−2κsp(α)>0. _sp(α):= 2-2 _sp(α)>0. (82) Strict positivity follows from strict Cauchy–Schwarz on the positive part of να _α: m1/2(α)2<ρα∫xdνα(x)=ραm_1/2(α)^2< _α x\,d _α(x)= _α. At α=1α=1, m1/2(1)=8/(3π)m_1/2(1)=8/(3π), so δsp(1)=2−16/(3π) _sp(1)= 2-16/(3π). Theorem C.2 (All-aspect deterministic margin geometry). Under Assumptions 2.1 and 2.2, there are deterministic ωd↓0 _d 0 and events ddetG_d^det such that, for every sufficiently large d, ℙ(ddet)≥1−e−d,P(G_d^det)≥ 1-e^- d, (83) and, on ddetG_d^det, ℱX≠∅F_X≠ and supU∈rowXmaxi≠j‖uijU−uijμ‖2 _U _row^X _i≠ j\|u_ij^U-u_ij^μ\|_2 ≤ωd, ≤ _d, (84) supU∈rowXDavg-bdry(U) _U _row^XD_avg -bdry(U) ≤ωd, ≤ _d, (85) infU∈spXDavg-bdry(U) _U _sp^XD_avg -bdry(U) ≥δsp(α)−ωd, ≥ _sp(α)- _d, (86) infU∈maxXDavg-bdry(U) _U _ ^XD_avg -bdry(U) ≥δmax−ωd. ≥ _ - _d. (87) Consequently, for every sufficiently large d on ddetG_d^det, supU∈rowXDavg-bdry(U)<mininfV∈spXDavg-bdry(V),infV∈maxXDavg-bdry(V). _U _row^XD_avg -bdry(U)< \ _V _sp^XD_avg -bdry(V), _V _ ^XD_avg -bdry(V) \. (88) C.1 Reduction from Direction Sets to Minimum-norm Representatives For every feasible matrix, all pairwise row differences are nonzero. Indeed, if wi=wjw_i=w_j for some i≠ji≠ j, then the constraint (wi−wj)⊤xi,s≥1(w_i-w_j) x_i,s≥ 1 fails. Thus Davg-bdryD_avg -bdry is well defined on ℱXF_X and on all three minimum-norm solution sets. Lemma C.3 (Scale-invariant reduction). For every ψ∈ψrow,ψsp,ψmaxψ∈\ _row, _sp, _ \, supU∈ψXDavg-bdry(U) _U _ψ^XD_avg -bdry(U) =supW∈ℳψXDavg-bdry(W), = _W _ψ^XD_avg -bdry(W), (89) infU∈ψXDavg-bdry(U) _U _ψ^XD_avg -bdry(U) =infW∈ℳψXDavg-bdry(W). = _W _ψ^XD_avg -bdry(W). (90) Proof. By (74), ψX=aW:a>0,W∈ℳψXU_ψ^X=\aW:a>0,\ W _ψ^X\. The conclusion follows from the positive scale invariance of Davg-bdryD_avg -bdry. ∎ We will therefore work with the minimum-norm representatives ℳrowXM_row^X, ℳspXM_sp^X, and ℳmaxXM_ ^X, and invoke Lemma C.3 only at the end. The max-norm constants are those in (79). C.2 A Quantitative Common Gaussian-cloud Event For each class, define z¯i:=1ni∑s=1nizi,s,mi:=1ni∑s=1nixi,s=μi+rz¯i,ei,s:=xi,s−mi=r(zi,s−z¯i). z_i:= 1n_i _s=1^n_iz_i,s, m_i:= 1n_i _s=1^n_ix_i,s= _i+r z_i, e_i,s:=x_i,s-m_i=r(z_i,s- z_i). (91) Then xi,s=mi+ei,sx_i,s=m_i+e_i,s, ∑sei,s= _se_i,s=0, and mim_i is independent of the centered residual family (ei,1,…,ei,ni)(e_i,1,…,e_i,n_i). Moreover, Cov(ei,s)=r2(1−1ni)Idd.Cov(e_i,s)=r^2 (1- 1n_i ) I_dd. (92) For i≠ji≠ j, write uijm:=(mi−mj)/‖mi−mj‖2u_ij^m:=(m_i-m_j)/\|m_i-m_j\|_2. For u∈d−1u ^d-1, set Hu(τ):=1d∑ℓ=1d|uℓ|minτd|uℓ|,1,Ju(τ):=1d∑ℓ=1dminτ2duℓ2,1.H_u(τ):= 1d _ =1^d d\,|u_ | \τ d\,|u_ |,1\, J_u(τ):= 1d _ =1^d \τ^2du_ ^2,1\. (93) Proposition C.4 (Stretched-exponential common input). Under Assumptions 2.1 and 2.2, there are deterministic εd↓0 _d 0, universal C<∞C<∞, a constant c=c(r0)>0c=c(r_0)>0, and events ℰddetE_d^det such that ℙ((ℰddet)c)≤Cexp[−cdlog(2N(d)(d+1))]P\! ((E_d^det)^c )≤ C \! [-c d (2N(d)(d+1)) ] (94) and the following statements hold simultaneously on ℰddetE_d^det: (H1) maxi‖mi−μi‖2≤εd _i\|m_i- _i\|_2≤ _d. (H2) maxi|‖mi‖2−1|≤εd _i|\|m_i\|_2-1|≤ _d and maxi≠j|mi⊤mj|≤εd _i≠ j|m_i m_j|≤ _d. (H3) maxi,j,s|mj⊤ei,s|≤εd _i,j,s|m_j e_i,s|≤ _d. (H4) maxi≠j‖uijm−uijμ‖2≤εd _i≠ j\|u_ij^m-u_ij^μ\|_2≤ _d. (H5) The max-norm optimum is finite and satisfies 2drmax∗(X)≤2π+εd.2 d\,r_ ^*(X)≤ 2π+ _d. (95) (H6) Uniformly over ordered pairs, sup0≤τ≤1|Huijm(τ)−Amax(τ)|+sup0≤τ≤1|Juijm(τ)−Bmax(τ)|≤εd. _0≤τ≤ 1|H_u_ij^m(τ)-A_ (τ)|+ _0≤τ≤ 1|J_u_ij^m(τ)-B_ (τ)|≤ _d. (96) Proof. Put Ld:=log(2N(d)(d+1))L_d:= (2N(d)(d+1)) and ad:=(Ld/d)1/4a_d:=(L_d/d)^1/4. The assumptions logN=o(d) N=o(d) and nmin→∞n_ →∞ imply ad→0a_d→ 0. All normalized Gaussian vectors below obey the shell bound ℙ(|‖g‖2d−1|>t)≤2e−cdt2,0<t<1,P\! ( | \|g\|_2 d-1 |>t )≤ 2e^-cdt^2, 0<t<1, (97) and every centered Gaussian scalar with variance at most C/dC/d obeys ℙ(|Z|>t)≤2e−cdt2/CP(|Z|>t)≤ 2e^-cdt^2/C. Since z¯i=dgi/nid z_i d=g_i/ n_id, the event maxi‖z¯i‖2≤2/nmin _i\| z_i\|_2≤ 2/ n_ fails with probability at most 2Ne−cd2Ne^-cd. Thus maxi‖mi−μi‖2≤2r0nmin=o(1) _i\|m_i- _i\|_2≤ 2r_0 n_ =o(1) (98) on an event whose failure probability is smaller than the right-hand side of (94). Applying (97) to μi∼(,Id/d) _i (0,I_d/d), with threshold CadCa_d, and taking a union bound proves maxi|‖μi‖2−1|≤Cad _i|\| _i\|_2-1|≤ Ca_d. Conditionally on μi _i, the variable μi⊤μj _i _j is centered Gaussian with variance ‖μi‖22/d\| _i\|_2^2/d. Another union bound over at most N2N^2 ordered pairs gives maxi≠j|μi⊤μj|≤Cad _i≠ j| _i _j|≤ Ca_d. Combining these estimates with (98) proves (H1) and (H2). Conditionally on all empirical means, mj⊤ei,sm_j e_i,s is centered Gaussian with variance at most r02‖mj‖22/dr_0^2\|m_j\|_2^2/d. On the event maxj‖mj‖2≤2 _j\|m_j\|_2≤ 2, a union bound over at most N2N^2 triples gives maxi,j,s|mj⊤ei,s|≤Cad _i,j,s|m_j e_i,s|≤ Ca_d with the required failure probability. This proves (H3). Items (H1)–(H2) imply infi≠j‖mi−mj‖2≥1 _i≠ j\|m_i-m_j\|_2≥ 1 and infi≠j‖μi−μj‖2≥1 _i≠ j\| _i- _j\|_2≥ 1 for all sufficiently large d. The normalization map is Lipschitz on vectors with norm at least one, so (H1) gives (H4). For (H5), let si:=sign(mi)∈−1,1ds_i:=sign(m_i)∈\-1,1\^d. Because each mim_i is isotropic Gaussian, sis_i is a uniform sign vector, and it is independent of every residual in its own class. Bernstein’s inequality for ‖mi‖1\|m_i\|_1, Gaussian tails for sj⊤mis_j m_i when i≠ji≠ j, conditional Gaussian tails for sj⊤ei,s_j e_i,s, and a union bound give mini‖mi‖1≥(2π−o(1))d,maxi≠j|sj⊤mi|+maxi,j,s|sj⊤ei,s|=o(d) _i\|m_i\|_1≥ ( 2π-o(1) ) d, _i≠ j|s_j m_i|+ _i,j,s|s_j e_i,s|=o( d) (99) outside an event bounded as in (94). The matrix with rows wi=tdsiw_i=t_ds_i, where tdt_d is the reciprocal of the minimum bracket in (si−sj)⊤xi,s≥‖mi‖1−|sj⊤mi|−2maxℓ|sℓ⊤ei,s|,(s_i-s_j) x_i,s≥\|m_i\|_1-|s_j m_i|-2 _ |s_ e_i,s|, (100) is feasible. Equation (99) gives td≤(π/2+o(1))/dt_d≤( π/2+o(1))/ d, which is exactly (H5). For (H6), each difference mi−mjm_i-m_j is a centered isotropic Gaussian vector, even when ni≠njn_i≠ n_j, so uijmu_ij^m is exactly uniform on d−1S^d-1. For one uniform direction, the two coordinate empirical processes in (93) concentrate about AmaxA_ and BmaxB_ at rate e−cdη2e^-cdη^2 for any fixed τ. On the event d−1∑ℓduℓ2=1d^-1 _ du_ ^2=1, both processes are uniformly Lipschitz in τ. A mesh of [0,1][0,1] with spacing ada_d, followed by a union bound over the mesh and at most N2N^2 ordered pairs, yields (96) with error CadCa_d and failure probability bounded by (94). Taking εd _d to dominate CadCa_d, 2r0/nmin2r_0/ n_ , and the deterministic vanishing envelopes in (99) completes the proof. ∎ Lemma C.5 (Normalization stability). Let x,y∈ℝdx,y ^d with y≠y≠0. If ‖x−y‖2≤‖y‖2/2\|x-y\|_2≤\|y\|_2/2, then ‖x‖x‖2−y‖y‖2‖2≤4‖x−y‖2‖y‖2. \| x\|x\|_2- y\|y\|_2 \|_2≤ 4\|x-y\|_2\|y\|_2. (101) Proof. The reverse triangle inequality gives ‖x‖2≥‖y‖2/2\|x\|_2≥\|y\|_2/2. Adding and subtracting y/‖x‖2y/\|x\|_2 and applying the reverse triangle inequality once more gives the displayed bound. ∎ C.3 Row-norm Representatives Align with the Population Mean Differences The row-norm part is deterministic once the empirical means and residuals satisfy the small-error conditions in Proposition C.4. Proposition C.6 (Deterministic row-norm rigidity). Fix deterministic samples xi,s=mi+ei,sx_i,s=m_i+e_i,s with ∑sei,s= _se_i,s=0 for every i. Suppose that, for some 0<h<1/200<h<1/20, maxi|‖mi‖2−1|≤h,maxi≠j|mi⊤mj|≤h,maxi,j,s|mj⊤ei,s|≤h. _i |\|m_i\|_2-1 |≤ h, _i≠ j|m_i m_j|≤ h, _i,j,s|m_j e_i,s|≤ h. (102) Then ℱXF_X is nonempty, and every W∈ℳrowXW _row^X satisfies maxi‖wi−mi‖2≤C(h+1k)1/4 _i\|w_i-m_i\|_2≤ C (h+ 1k )^1/4 (103) for a universal constant C. Proof. For i≠ji≠ j and s≤nis≤ n_i, (mi−mj)⊤xi,s (m_i-m_j) x_i,s =‖mi‖22−mj⊤mi+(mi−mj)⊤ei,s =\|m_i\|_2^2-m_j m_i+(m_i-m_j) e_i,s ≥(1−h)2−h−2h≥1−5h. ≥(1-h)^2-h-2h≥ 1-5h. (104) Hence the matrix with rows w~i=mi/(1−5h) w_i=m_i/(1-5h) is feasible. If R∗:=maxi‖wi‖2R^*:= _i\|w_i\|_2 is the optimal row radius, then R∗≤1+h1−5h≤1+Ch.R^*≤ 1+h1-5h≤ 1+Ch. (105) Let w¯:=k−1∑r=1kwr w:=k^-1 _r=1^kw_r. For any feasible W, averaging the constraints over s=1,…,nis=1,…,n_i and summing over j≠ij≠ i gives ∑j≠i(wi−wj)⊤mi≥k−1. _j≠ i(w_i-w_j) m_i≥ k-1. (106) Since ∑j≠iwj=kw¯−wi _j≠ iw_j=k w-w_i, this is equivalent to (wi−w¯)⊤mi≥1−1k.(w_i- w) m_i≥ 1- 1k. (107) Consequently, ‖wi−w¯‖2≥L:=1−1/k1+h.\|w_i- w\|_2≥ L:= 1-1/k1+h. (108) For a row-norm minimizer, the variance identity yields L2 L^2 ≤1k∑i=1k‖wi−w¯‖22=1k∑i=1k‖wi‖22−‖w¯‖22≤(R∗)2−‖w¯‖22. ≤ 1k _i=1^k\|w_i- w\|_2^2= 1k _i=1^k\|w_i\|_2^2-\| w\|_2^2≤(R^*)^2-\| w\|_2^2. (109) Combining this with (105) gives ‖w¯‖22≤C(h+1k).\| w\|_2^2≤ C (h+ 1k ). (110) Using (107) once more, wi⊤mi≥1−1k+w¯⊤mi≥1−C(h+1k)1/2.w_i m_i≥ 1- 1k+ w m_i≥ 1-C (h+ 1k )^1/2. (111) Together with ‖wi‖2≤1+Ch\|w_i\|_2≤ 1+Ch and ‖mi‖2≤1+h\|m_i\|_2≤ 1+h, this implies ‖wi−mi‖22 \|w_i-m_i\|_2^2 =‖wi‖22+‖mi‖22−2wi⊤mi≤C(h+1k)1/2. =\|w_i\|_2^2+\|m_i\|_2^2-2w_i m_i≤ C (h+ 1k )^1/2. (112) Taking square roots proves (103). ∎ Proposition C.7 (Uniform row-norm alignment). Under (1)–(3), supW∈ℳrowXDavg-bdry(W)=oℙ(1). _W _row^XD_avg -bdry(W)=o_P(1). (113) In fact, the stronger uniform pairwise statement supW∈ℳrowXmaxi≠j‖uijW−uijμ‖2=oℙ(1) _W _row^X _i≠ j\|u_ij^W-u_ij^μ\|_2=o_P(1) (114) holds. Proof. Work on ℰddetE_d^det from Proposition C.4. Apply Proposition C.6 with h=εdh= _d. Since k/d→α∈(0,∞)k/d→α∈(0,∞), we have k→∞k→∞, and therefore, uniformly over all row-norm minimizers, maxi‖wi−mi‖2=o(1). _i\|w_i-m_i\|_2=o(1). (115) Combining this with (H1) gives Δdrow:=supW∈ℳrowXmaxi‖wi−μi‖2=o(1) _d^row:= _W _row^X _i\|w_i- _i\|_2=o(1) (116) on ℰddetE_d^det. Moreover, infi≠j‖μi−μj‖2 _i≠ j\| _i- _j\|_2 is bounded below by a positive constant on the same event. Hence ‖(wi−wj)−(μi−μj)‖2≤2Δdrow\|(w_i-w_j)-( _i- _j)\|_2≤ 2 _d^row (117) uniformly over i≠ji≠ j and W∈ℳrowXW _row^X. Applying Lemma C.5 proves (114), and averaging over ordered pairs gives (113). ∎ C.4 Max-norm Representatives Have a Positive Angle Gap The max-norm proof has two ingredients. First, optimality imposes a coordinate cap of order d−1/2d^-1/2 on every pairwise row difference. Second, feasibility forces every such difference to have a large projection onto the empirical mean-pair direction. The clipped minimum-energy problem below shows that these two requirements are incompatible with near-perfect angular alignment. C.4.1 The Clipped Minimum-energy Problem Proposition C.8 (Exact clipped problem). Fix u∈d−1u ^d-1 and a,b>0a,b>0. Define d(u;a,b)2:=min∥x∥22:x⊤u≥b,∥x∥∞≤ad. m_d(u;a,b)^2:= \\|x\|_2^2:x u≥ b,\ \|x\|_∞≤ a d \. (118) Let zℓ:=d|uℓ|z_ := d\,|u_ |. If 0<b<a1d∑ℓ=1dzℓ,0<b<a 1d _ =1^dz_ , (119) then the minimizer is unique and has coordinates xℓ=adsign(uℓ)minτzℓ,1,ℓ=1,…,d,x_ = a dsign(u_ ) \τ z_ ,1\, =1,…,d, (120) where sign(0):=0sign(0):=0 and τ>0τ>0 is the unique solution of Hu(τ)=ba.H_u(τ)= ba. (121) Moreover, d(u,a,b)2=a2Ju(τ). m_d(u;a,b)^2=a^2J_u(τ). (122) Proof. We first verify that the feasible set is nonempty. Define the coordinate-saturated vector x^∈ℝd x ^d by x^ℓ:=adsign(uℓ). x_ := a dsign(u_ ). (123) Its projection onto u is x^⊤u=ad∑ℓ=1d|uℓ|=ad∑ℓ=1dzℓ>b, x u= a d _ =1^d|u_ |= ad _ =1^dz_ >b, (124) where the strict inequality follows from (119). Choose c such that b/(x^⊤u)<c<1b/( x u)<c<1. Then cx^c x satisfies the projection constraint strictly and lies strictly inside the coordinate box. The feasible set is thus nonempty. It is also closed and bounded, so a minimizer exists. Since the objective x↦‖x‖22x \|x\|_2^2 is strictly convex and the feasible set is convex, this minimizer is unique. Let x⋆x denote the unique minimizer. We next show that every nonzero coordinate of x⋆x has the same sign as the corresponding coordinate of u. Given any feasible x, define x♯x by xℓ♯:=sign(uℓ)|xℓ|,uℓ≠0,0,uℓ=0.x _ := casessign(u_ )|x_ |,&u_ ≠ 0,\\ 0,&u_ =0. cases (125) Then ‖x♯‖∞≤‖x‖∞\|x \|_∞≤\|x\|_∞, ‖x♯‖2≤‖x‖2\|x \|_2≤\|x\|_2, and (x♯)⊤u=∑ℓ=1d|xℓ||uℓ|≥∑ℓ=1dxℓuℓ=x⊤u.(x ) u= _ =1^d|x_ ||u_ |≥ _ =1^dx_ u_ =x u. (126) Applying this construction to x⋆x shows that x♯x is feasible and has objective value no larger than that of x⋆x . By optimality and uniqueness, x⋆=x♯x =x . Hence xℓ⋆x _ has the sign of uℓu_ when uℓ≠0u_ ≠ 0, and xℓ⋆=0x _ =0 when uℓ=0u_ =0. The projection constraint must be active at the minimizer. Indeed, if (x⋆)⊤u>b(x ) u>b, then c:=b(x⋆)⊤uc:= b(x ) u (127) satisfies 0<c<10<c<1. The vector cx⋆cx obeys ‖cx⋆‖∞≤a/d\|cx \|_∞≤ a/ d and (cx⋆)⊤u=b(cx ) u=b, while ‖cx⋆‖22=c2‖x⋆‖22<‖x⋆‖22,\|cx \|_2^2=c^2\|x \|_2^2<\|x \|_2^2, (128) contradicting the optimality of x⋆x . Therefore, (x⋆)⊤u=b.(x ) u=b. (129) Because of the sign alignment, write yℓ:=da|xℓ⋆|,0≤yℓ≤1.y_ := da|x _ |, 0≤ y_ ≤ 1. (130) Using |uℓ|=zℓ/d|u_ |=z_ / d, the active projection constraint becomes b=∑ℓ=1d|xℓ⋆||uℓ|=ad∑ℓ=1dzℓyℓ,b= _ =1^d|x _ ||u_ |= ad _ =1^dz_ y_ , (131) or equivalently, ∑ℓ=1dzℓyℓ=dba. _ =1^dz_ y_ = dba. (132) Moreover, ‖x⋆‖22=a2d∑ℓ=1dyℓ2.\|x \|_2^2= a^2d _ =1^dy_ ^2. (133) Therefore, finding x⋆x is equivalent to solving the convex optimization problem min0≤yℓ≤112∑ℓ=1dyℓ2subject to∑ℓ=1dzℓyℓ≥dba. _0≤ y_ ≤ 1 12 _ =1^dy_ ^2 to _ =1^dz_ y_ ≥ dba. (134) The strict inequality in (119) implies that the right-hand side of the constraint is strictly smaller than ∑ℓzℓ _ z_ , so the reduced problem has a strictly feasible point. Hence the KKT conditions are necessary and sufficient. Let τ≥0τ≥ 0 be the multiplier associated with the constraint in (134). For fixed τ, the Lagrangian minimization over the box [0,1]d[0,1]^d separates across coordinates: yℓ∈argmin0≤y≤112y2−τzℓy.y_ ∈ *arg\,min_0≤ y≤ 1 \ 12y^2-τ z_ y \. (135) The unconstrained minimizer of the scalar objective is y=τzℓy=τ z_ . Projecting this value onto [0,1][0,1] gives yℓ=minτzℓ,1.y_ = \τ z_ ,1\. (136) The multiplier must satisfy τ>0τ>0. If τ=0τ=0, then (135) gives yℓ=0y_ =0 for every ℓ , which violates the constraint because b>0b>0. Since τ>0τ>0, complementary slackness implies that the projection constraint is active: ∑ℓ=1dzℓminτzℓ,1=dba. _ =1^dz_ \τ z_ ,1\= dba. (137) After dividing by d, this is exactly Hu(τ)=1d∑ℓ=1dzℓminτzℓ,1=ba.H_u(τ)= 1d _ =1^dz_ \τ z_ ,1\= ba. (138) It remains to verify that this equation has a unique solution. The function HuH_u is continuous and nondecreasing on [0,∞)[0,∞), with Hu(0)=0,limτ→∞Hu(τ)=1d∑ℓ=1dzℓ.H_u(0)=0, _τ→∞H_u(τ)= 1d _ =1^dz_ . (139) The condition (119) places b/ab/a strictly between these two values, so existence follows from continuity. To prove uniqueness, take 0≤τ1<τ20≤ _1< _2 and suppose that Hu(τ1)<1d∑ℓ=1dzℓ.H_u( _1)< 1d _ =1^dz_ . (140) Then at least one coordinate with zℓ>0z_ >0 is not saturated at τ1 _1, meaning τ1zℓ<1 _1z_ <1. For that coordinate, minτ2zℓ,1>minτ1zℓ,1, \ _2z_ ,1\> \ _1z_ ,1\, (141) while every other coordinate contributes a nondecreasing term. Thus Hu(τ2)>Hu(τ1)H_u( _2)>H_u( _1). Hence HuH_u is strictly increasing before reaching its limiting value, and the equation Hu(τ)=b/aH_u(τ)=b/a has a unique solution. Substituting (136) into the sign-aligned representation of x⋆x gives xℓ⋆=adsign(uℓ)minτzℓ,1,x _ = a dsign(u_ ) \τ z_ ,1\, (142) which proves (120). Finally, d(u,a,b)2 m_d(u;a,b)^2 =‖x⋆‖22 =\|x \|_2^2 =a2d∑ℓ=1dminτzℓ,12 = a^2d _ =1^d \τ z_ ,1\^2 =a21d∑ℓ=1dminτ2zℓ2,1 =a^2 1d _ =1^d \τ^2z_ ^2,1\ =a2Ju(τ), =a^2J_u(τ), (143) which proves (122). ∎ Lemma C.9 (Uniform asymptotics of the clipped problem). There exist constants δ0,C0>0 _0,C_0>0 such that the following holds. Suppose u∈d−1u ^d-1 satisfies sup0≤τ≤1|Hu(τ)−Amax(τ)|+sup0≤τ≤1|Ju(τ)−Bmax(τ)|≤δ _0≤τ≤ 1|H_u(τ)-A_ (τ)|+ _0≤τ≤ 1|J_u(τ)-B_ (τ)|≤δ (144) with 0<δ≤δ00<δ≤ _0, and suppose |a−2π|+|b−2|≤δ.|a- 2π|+|b- 2|≤δ. (145) Then (119) holds, the solution in Proposition C.8 satisfies τ∈(0,1)τ∈(0,1), and 1d(u,a,b) 1 m_d(u;a,b) ≤Cmax+C0δ, ≤ C_ +C_0δ, (146) bd(u,a,b) b m_d(u;a,b) ≤κmax+C0δ. ≤ _ +C_0δ. (147) Proof. Throughout the proof, C>0C>0 denotes a constant independent of d,u,a,bd,u,a,b, and δ, and its value may change from line to line. Recall that τ∗>0 _*>0 is defined by Amax(τ∗)=1π.A_ ( _*)= 1 π. (148) We first compare the required normalized projection b/ab/a with this reference value. By taking δ0 _0 small enough, (145) ensures that a≥2π/2a≥ 2π/2. Hence |ba−1π| | ba- 1 π | =|ba−22π| = | ba- 2 2π | ≤|b−2|a+2|1a−12π| ≤ |b- 2|a+ 2 | 1a- 1 2π | =|b−2|a+2|a−2π|a2π≤Cδ. = |b- 2|a+ 2\,|a- 2π|a 2π≤ Cδ. (149) Thus the right-hand side of the equation Hu(τ)=b/aH_u(τ)=b/a is within O(δ)O(δ) of Amax(τ∗)A_ ( _*). We next show that the clipped problem is feasible and that its multiplier satisfies τ<1τ<1. Since τ∗<1 _*<1 and AmaxA_ is strictly increasing, the constant g∗:=Amax(1)−Amax(τ∗)=Amax(1)−1πg_*:=A_ (1)-A_ ( _*)=A_ (1)- 1 π (150) is strictly positive. By (144), Hu(1)≥Amax(1)−δ=1π+g∗−δ.H_u(1)≥ A_ (1)-δ= 1 π+g_*-δ. (151) On the other hand, (149) gives ba≤1π+Cδ. ba≤ 1 π+Cδ. (152) After decreasing δ0 _0 if necessary, these two bounds imply 0<ba<Hu(1).0< ba<H_u(1). (153) Because Hu(1)=1d∑ℓ=1dzℓminzℓ,1≤1d∑ℓ=1dzℓ,H_u(1)= 1d _ =1^dz_ \z_ ,1\≤ 1d _ =1^dz_ , (154) we obtain 0<b<a1d∑ℓ=1dzℓ.0<b<a 1d _ =1^dz_ . (155) This is exactly (119). Moreover, Hu(0)=0H_u(0)=0 and Hu(1)>b/aH_u(1)>b/a, so Proposition C.8 gives a unique solution τ∈(0,1)τ∈(0,1) of Hu(τ)=ba.H_u(τ)= ba. (156) We now show that this solution is close to τ∗ _*. Using (156), (144), (149), and (148), we obtain |Amax(τ)−Amax(τ∗)| |A_ (τ)-A_ ( _*)| ≤|Amax(τ)−Hu(τ)|+|Hu(τ)−Amax(τ∗)| ≤|A_ (τ)-H_u(τ)|+ |H_u(τ)-A_ ( _*) | =|Amax(τ)−Hu(τ)|+|ba−1π|≤Cδ. =|A_ (τ)-H_u(τ)|+ | ba- 1 π |≤ Cδ. (157) For s∈(0,1]s∈(0,1], differentiation under the expectation gives Amax′(s)=[G2s|G|≤1].A_ (s)=E [G^21_\s|G|≤ 1\ ]. (158) Since s≤1s≤ 1, the event |G|≤1\|G|≤ 1\ is contained in s|G|≤1\s|G|≤ 1\. Therefore Amax′(s)≥cA,cA:=[G2|G|≤1]>0.A_ (s)≥ c_A, c_A:=E [G^21_\|G|≤ 1\ ]>0. (159) The mean value theorem and (157) now give cA|τ−τ∗|≤|Amax(τ)−Amax(τ∗)|≤Cδ,c_A|τ- _*|≤|A_ (τ)-A_ ( _*)|≤ Cδ, (160) and hence |τ−τ∗|≤Cδ.|τ- _*|≤ Cδ. (161) We next lower-bound Ju(τ)J_u(τ). The first assumption of the lemma gives Ju(τ)≥Bmax(τ)−δ.J_u(τ)≥ B_ (τ)-δ. (162) The function BmaxB_ is Lipschitz on [0,1][0,1]. Indeed, for s,t∈[0,1]s,t∈[0,1], |Bmax(s)−Bmax(t)| |B_ (s)-B_ (t)| ≤[|mins2G2,1−mint2G2,1|] [ | \s^2G^2,1\- \t^2G^2,1\ | ] ≤|s2−t2|[G2]≤2|s−t|. ≤|s^2-t^2|E[G^2]≤ 2|s-t|. (163) Combining this bound with (161) gives Bmax(τ)≥Bmax(τ∗)−Cδ.B_ (τ)≥ B_ ( _*)-Cδ. (164) Therefore, Ju(τ)≥Bmax(τ∗)−Cδ.J_u(τ)≥ B_ ( _*)-Cδ. (165) The assumption on a also gives a2 a^2 ≥(2π−δ)2≥2π−Cδ. ≥ ( 2π-δ )^2≥ 2π-Cδ. (166) Using the exact identity from Proposition C.8, d(u,a,b)2=a2Ju(τ), m_d(u;a,b)^2=a^2J_u(τ), (167) together with (165) and (166), we obtain d(u,a,b)2 m_d(u;a,b)^2 ≥(2π−Cδ)(Bmax(τ∗)−Cδ) ≥(2π-Cδ) (B_ ( _*)-Cδ ) ≥2πBmax(τ∗)−Cδ. ≥ 2π B_ ( _*)-Cδ. (168) Set M∗:=2πBmax(τ∗)>0.M_*:=2π B_ ( _*)>0. (169) By taking δ0 _0 smaller if necessary, (168) implies d(u,a,b)2≥M∗−Cδ≥M∗2. m_d(u;a,b)^2≥ M_*-Cδ≥ M_*2. (170) Consequently, 1d(u,a,b) 1 m_d(u;a,b) ≤1M∗−Cδ ≤ 1 M_*-Cδ ≤1M∗+Cδ ≤ 1 M_*+Cδ =Cmax+Cδ, =C_ +Cδ, (171) where the last equality follows from Cmax=(2πBmax(τ∗))−1/2.C_ = (2π B_ ( _*) )^-1/2. (172) After enlarging the constant, this proves (146). Finally, (145) gives b≤2+δb≤ 2+δ. Therefore, bd(u,a,b) b m_d(u;a,b) ≤(2+δ)(Cmax+C0δ) ≤( 2+δ)(C_ +C_0δ) ≤2Cmax+Cδ ≤ 2\,C_ +Cδ =κmax+Cδ, = _ +Cδ, (173) because κmax=2Cmax _ = 2\,C_ . Enlarging C0C_0 once more proves (147). ∎ C.4.2 Application to Max-norm Minimizers Proposition C.10 (Max-norm boundary separation). Under (1)–(3), infW∈ℳmaxXDavg-bdry(W)≥δmax−oℙ(1). _W _ ^XD_avg -bdry(W)≥ _ -o_P(1). (174) In fact, on the same high-probability event, every ordered pair of every max-norm minimizer has boundary-normal distance at least δmax−o(1) _ -o(1). Proof. Work on ℰddetE_d^det from Proposition C.4. Fix W∈ℳmaxXW _ ^X and set vij:=wi−wjv_ij:=w_i-w_j. By (H5), ‖vij‖∞≤2rmax∗(X)≤add,ad:=2π+εd.\|v_ij\|_∞≤ 2r_ ^*(X)≤ a_d d, a_d:= 2π+ _d. (175) Averaging the constraints (wi−wj)⊤xi,s≥1(w_i-w_j) x_i,s≥ 1 over class i gives vij⊤mi≥1v_ij m_i≥ 1. Applying the reverse constraints for class j gives vij⊤mj≤−1v_ij m_j≤-1. Therefore vij⊤(mi−mj)≥2.v_ij (m_i-m_j)≥ 2. (176) By (H2), uniformly over i≠ji≠ j, ‖mi−mj‖22=‖mi‖22+‖mj‖22−2mi⊤mj≤2+o(1).\|m_i-m_j\|_2^2=\|m_i\|_2^2+\|m_j\|_2^2-2m_i m_j≤ 2+o(1). (177) Hence vij⊤uijm≥2‖mi−mj‖2≥bd,bd:=2−o(1),v_ij u_ij^m≥ 2\|m_i-m_j\|_2≥ b_d, b_d:= 2-o(1), (178) where the error is deterministic on ℰddetE_d^det and uniform over all pairs. Fix i≠ji≠ j, and abbreviate u=uijmu=u_ij^m, v=vijv=v_ij, a=ada=a_d, and b=bdb=b_d. Since v⊤u≥b>0v u≥ b>0, define y:=bv⊤uv.y:= bv uv. (179) Then y⊤u=by u=b, and because b/(v⊤u)≤1b/(v u)≤ 1, (175) gives ‖y‖∞≤a/d\|y\|_∞≤ a/ d. Thus y is feasible for (118), and d(u,a,b)≤‖y‖2=b‖v‖2v⊤u. m_d(u;a,b)≤\|y\|_2= b\|v\|_2v u. (180) Equivalently, v⊤u‖v‖2≤bd(u,a,b). v u\|v\|_2≤ b m_d(u;a,b). (181) By (H6) and Lemma C.9, the right-hand side is at most κmax+o(1) _ +o(1), uniformly over all ordered pairs. Since κmax<1 _ <1, ‖vij‖vij‖2−uijm‖2=2−2vij⊤uijm‖vij‖2≥δmax−o(1). \| v_ij\|v_ij\|_2-u_ij^m \|_2= 2-2 v_ij u_ij^m\|v_ij\|_2≥ _ -o(1). (182) Finally, (H4) and the reverse triangle inequality imply ‖vij‖vij‖2−uijμ‖2≥δmax−o(1), \| v_ij\|v_ij\|_2-u_ij^μ \|_2≥ _ -o(1), (183) uniformly over all pairs and all max-norm minimizers. Averaging proves (174). ∎ C.5 Spectral Polar Geometry Let P:=Ik−1k⊤,Mm:=[m1⊤mk⊤],Cm:=PMm,rd:=mink−1,d.P:=I_k- 1k11 , M_m:= bmatrixm_1 \\ \\ m_k bmatrix, C_m:=PM_m, r_d:= \k-1,d\. (184) On the full reduced-rank event, write Cm=UmΣmVm⊤,Um∈ℝk×rd,Vm∈ℝd×rd,C_m=U_m _mV_m , U_m ^k× r_d, V_m ^d× r_d, (185) and define the reduced polar factor and its positive-semidefinite companion by Qm:=UmVm⊤=Cm(Cm⊤Cm)†1/2,Sm:=QmCm⊤=(CmCm⊤)1/2.Q_m:=U_mV_m =C_m(C_m C_m) 1/2, S_m:=Q_mC_m =(C_mC_m )^1/2. (186) Write qi⊤q_i for the i-th row of QmQ_m. Lemma C.11 (Quantitative Gaussian hard-edge count at the square aspect ratio). Let p=p(d)p=p(d) satisfy p/d→1p/d→ 1, let Hd∈ℝp×dH_d ^p× d have independent (0,1)N(0,1) entries, and set Ad:=Hd/dA_d:=H_d/ d and nd:=minp,dn_d:= \p,d\. There are universal constants T,c,C>0T,c,C>0 such that, for all sufficiently large d and every 2/nd≤u≤T2/n_d≤ u≤ T, an event ℋd(u)H_d(u) satisfies ℙ(ℋd(u)c)≤C(1+log(T/u))e−cdu2,P (H_d(u)^c )≤ C (1+ (T/u) )e^-cdu^2, and, on this event, 1nd#ℓ:σℓ(Ad)≤t≤8t,u≤t≤T. 1n_d\#\ : _ (A_d)≤ t\≤ 8t, u≤ t≤ T. (187) Moreover, AdA_d has rank ndn_d almost surely. Proof. Write n:=ndn:=n_d and take T=1/16T=1/16. Since n/d→1n/d→ 1, for all sufficiently large d we have n/d≥9/10n/d≥ 9/10. Let B be a fixed n×n× n coordinate submatrix of HdH_d, obtained by retaining a fixed set of n rows when p≥dp≥ d and a fixed set of n columns when p<dp<d. If p≥dp≥ d, then Hd⊤Hd=B⊤B+R⊤R⪰B⊤BH_d H_d=B B+R R B B for the matrix R of deleted rows, while if p<dp<d, then HdHd⊤=BB⊤+RR⊤⪰BB⊤H_dH_d =B +R B for the matrix R of deleted columns. Hence, by monotonicity of eigenvalues under the positive-semidefinite order, σj(Hd)≥σj(B),1≤j≤n. _j(H_d)≥ _j(B), 1≤ j≤ n. Fix t∈[1/n,T]t∈[1/n,T] and put ℓ:=⌈3nt⌉ := 3nt . For all sufficiently large n, the choice T=1/16T=1/16 ensures ℓ<n <n. Let F be the n×(n−ℓ)n×(n- ) matrix obtained by retaining a fixed set of n−ℓn- columns of B. Apply Vershynin (2012, Corollary 5.35) to F with row dimension N=nN=n, column dimension n−ℓn- , and deviation parameter s:=tn/4s:=t n/4. Then, except with probability at most 2e−s2/2=2e−nt2/32,2e^-s^2/2=2e^-nt^2/32, we have σmin(F)≥n−n−ℓ−s=n−n−ℓ−tn4. _ (F)≥ n- n- -s= n- n- - t n4. Using n−n−ℓ=ℓn+n−ℓ≥ℓ2n≥32tn, n- n- = n+ n- ≥ 2 n≥ 32t n, we obtain σmin(F)≥54tn>td, _ (F)≥ 54t n>t d, where the last inequality holds for all sufficiently large d because n/d→1n/d→ 1. Writing B=[FRF]B=[F\ \ R_F], we have BB⊤=FF⊤+RFRF⊤⪰FF⊤B =F +R_FR_F F . Therefore the largest n−ℓn- eigenvalues of BB⊤B are at least the n−ℓn- positive eigenvalues of FF⊤F , and hence σj(B)>td,1≤j≤n−ℓ. _j(B)>t d, 1≤ j≤ n- . Together with σj(Hd)≥σj(B) _j(H_d)≥ _j(B), this gives 1n#j:σj(Ad)≤t≤ℓn≤3t+1n≤4t. 1n\#\j: _j(A_d)≤ t\≤ n≤ 3t+ 1n≤ 4t. Now fix u∈[2/n,T]u∈[2/n,T] and take the dyadic grid tj=2−jTt_j=2^-jT down to the first grid point lying in [u/2,u][u/2,u]. Every such grid point is at least u/2≥1/nu/2≥ 1/n, and the number of grid points is at most C(1+log(T/u))C(1+ (T/u)). Applying the preceding fixed-threshold estimate at every grid point and taking a union bound, the probability that any of these estimates fails is at most C(1+log(T/u))e−cdu2,C (1+ (T/u) )e^-cdu^2, because the smallest grid point is at least u/2u/2 and n/d→1n/d→ 1. On the complementary event, for every t∈[u,T]t∈[u,T] choose a dyadic grid point tj∈[t,2t]t_j∈[t,2t]; monotonicity of the counting function gives 1n#ℓ:σℓ(Ad)≤t≤1n#ℓ:σℓ(Ad)≤tj≤4tj≤8t. 1n\#\ : _ (A_d)≤ t\≤ 1n\#\ : _ (A_d)≤ t_j\≤ 4t_j≤ 8t. Finally, AdA_d has rank n almost surely because the rank-deficient matrices form a Lebesgue-null algebraic set. ∎ Proposition C.12 (All-aspect spectral input). There are deterministic ηd↓0 _d 0, constants c,C>0c,C>0, and events ℰdspE_d^sp such that ℙ((ℰdsp)c)≤Cexp[−cdlog(2N(d)(d+1))]P\! ((E_d^sp)^c )≤ C \! [-c d (2N(d)(d+1)) ] (188) and the following statements hold on ℰdspE_d^sp: (S1) CmC_m has rank rdr_d, PQm=QmPQ_m=Q_m, ‖Qm‖op=1\|Q_m\|_op=1, and |‖Cm‖∗k−sα|≤ηd,infi≠j‖qi−qj‖2≥cα>0. | \|C_m\|_*k-s_α |≤ _d, _i≠ j\|q_i-q_j\|_2≥ c_α>0. (189) (S2) Uniformly over i≠ji≠ j, maxi‖mi−μi‖2+‖m¯‖2≤ηd,m¯:=1k∑imi, _i\|m_i- _i\|_2+\| m\|_2≤ _d, m:= 1k _im_i, (190) maxi|Sm[i,i]−sα|+maxi≠j|Sm[i,j]|≤ηd,maxi≠j,s|(qi−qj)⊤ei,s|≤ηd. _i|S_m[i,i]-s_α|+ _i≠ j|S_m[i,j]|≤ _d, _i≠ j,s|(q_i-q_j) e_i,s|≤ _d. (191) (S3) The reduced polar margin ρX:=mini≠j,s(qi−qj)⊤xi,s _X:= _i≠ j,s(q_i-q_j) x_i,s (192) satisfies |ρX−sα|≤Cηd,ρX>0.| _X-s_α|≤ C _d, _X>0. (193) (S4) The reduced polar geometry satisfies |Davg-bdry(Qm)−δsp(α)|≤ηd. |D_avg -bdry(Q_m)- _sp(α) |≤ _d. (194) (S5) If α≠1α≠ 1, all positive singular values of CmC_m lie in a deterministic compact interval bounded away from zero: cα≤σrd(Cm)≤σ1(Cm)≤Cα.c_α≤ _r_d(C_m)≤ _1(C_m)≤ C_α. (195) If α=1α=1, there are t0,C0>0t_0,C_0>0 such that 1rd#ℓ:σℓ(Cm)≤t≤C0(t+ηd),ηd≤t≤t0. 1r_d\#\ : _ (C_m)≤ t\≤ C_0(t+ _d), _d≤ t≤ t_0. (196) Proof. Because mi=μi+rz¯im_i= _i+r z_i is Gaussian with covariance (1+r2/ni)Id/d(1+r^2/n_i)I_d/d, the rows are independent and admit the exact representation Mm=DdG/d,Dd:=diag(1+r2/n1,…,1+r2/nk),M_m=D_dG/ d, D_d:=diag\! ( 1+r^2/n_1,…, 1+r^2/n_k ), (197) where G has independent standard Gaussian entries. Since ‖Dd−Ik‖op≤Cr02/nmin=o(1)\|D_d-I_k\|_op≤ Cr_0^2/n_ =o(1) and ‖G/d‖op=O(1)\|G/ d\|_op=O(1) with exponentially high probability, the comparator C0:=PG/dC_0:=PG/ d obeys ‖Cm−C0‖op≤ζd,ζd↓0.\|C_m-C_0\|_op≤ _d, _d 0. (198) After a fixed orthogonal change of basis on ℝkR^k, the positive singular values of C0C_0 have the same law as those of a (k−1)×d(k-1)× d standard Gaussian matrix divided by d d. Marchenko–Pastur convergence therefore gives ‖C0‖∗/k→sα\|C_0\|_*/k→ s_α. The normalized nuclear norm is O(d−1)O(d^-1)-Lipschitz in the Gaussian entries, so Gaussian concentration and the deterministic expectation bias place this convergence under a deterministic envelope with failure probability bounded as in (188). The rank assertion is almost sure, and Weyl’s inequality yields (195) when α≠1α≠ 1. At α=1α=1, put Ld:=log(2N(d)(d+1))L_d:= (2N(d)(d+1)) and ud:=(Ld/d)1/4u_d:=(L_d/d)^1/4. Then ud→0u_d→ 0, ud≥2/rdu_d≥ 2/r_d eventually, and Lemma C.11 gives failure at most Cexp(−cdLd)C (-c dL_d) after absorbing its logarithmic prefactor. Weyl’s inequality and (198) transfer the count from C0C_0 to CmC_m by replacing t with t+ζdt+ _d. Enlarge the common envelope so that ηd≥maxud,ζd _d≥ \u_d, _d\ and take t0<T/2t_0<T/2; then (196) follows with precisely the probability scale in (188). Let S0:=(C0C0⊤)1/2S_0:=(C_0C_0 )^1/2. Conditionally on its singular values, the left singular frame of C0C_0 is Haar in ⟂1 and independent of those singular values. Haar concentration for the quadratic forms defining S0[i,j]S_0[i,j], followed by a union bound over k2k^2 entries, gives maxi|S0[i,i]−‖C0‖∗k|+maxi≠j|S0[i,j]|≤χd,χd↓0. _i |S_0[i,i]- \|C_0\|_*k |+ _i≠ j|S_0[i,j]|≤ _d, _d 0. (199) The gap-free square-root inequality for positive semidefinite matrices gives ‖Sm−S0‖op≤‖CmCm⊤−C0C0⊤‖op1/2≤Cζd1/2.\|S_m-S_0\|_op≤\|C_mC_m -C_0C_0 \|_op^1/2≤ C _d^1/2. (200) This proves the entrywise bounds in (S2) without dividing by the smallest singular value. The empirical-mean bound is part of Proposition C.4, and the Gaussian average k−1∑imik^-1 _im_i gives ‖m¯‖2=o(1)\| m\|_2=o(1) under the same probability envelope. Conditionally on the empirical means, QmQ_m is deterministic and the residuals are centered Gaussian. Since ‖qi−qj‖2≤2\|q_i-q_j\|_2≤ 2, Gaussian tails and a union bound over at most N2N^2 indices give the residual bound in (S2). Using that row i of CmC_m is (mi−m¯)⊤(m_i- m) , we obtain (qi−qj)⊤mi=Sm[i,i]−Sm[j,i]+(qi−qj)⊤m¯=sα+O(ηd).(q_i-q_j) m_i=S_m[i,i]-S_m[j,i]+(q_i-q_j) m=s_α+O( _d). (201) Adding the residual bound proves (S3). The same identity and maxi,s‖xi,s‖2=O(1) _i,s\|x_i,s\|_2=O(1) give the uniform lower bound on ‖qi−qj‖2\|q_i-q_j\|_2. It remains to identify the angular constant. The centering identities ⊤Qm=1 Q_m=0 and ‖Qm‖F2=rd\|Q_m\|_F^2=r_d give 1k(k−1)∑i≠j‖qi−qj‖22=2rdk−1⟶2ρα. 1k(k-1) _i≠ j\|q_i-q_j\|_2^2= 2r_dk-1 2 _α. (202) If α<1α<1, then rd=k−1r_d=k-1 and QmQm⊤=PQ_mQ_m =P, so every pair norm is exactly 2 2. For α>1α>1, let Q0Q_0 be the reduced polar factor of C0C_0 and put Πm:=QmQm⊤ _m:=Q_mQ_m and Π0:=Q0Q0⊤ _0:=Q_0Q_0 . The left singular frame of C0C_0, but not in general that of Cm=PDdG/dC_m=PD_dG/ d, is Haar in ⟂1 when the class counts differ. The positive singular-value lower bounds and (198) give ‖(I−Π0)Πm‖op≤‖Cm−C0‖op‖Cm†‖op≤Cαζd,\|(I- _0) _m\|_op≤\|C_m-C_0\|_op\|C_m \|_op≤ C_α _d, and the analogous estimate with 00 and m exchanged. Consequently, ‖Πm−Π0‖op≤Cαζd\| _m- _0\|_op≤ C_α _d. Haar-projector concentration applies to Π0 _0 and yields (ei−ej)⊤Π0(ei−ej)=2/α+o(1)(e_i-e_j) _0(e_i-e_j)=2/α+o(1) uniformly over pairs on the same probability scale. Since each such quadratic form changes by at most 2‖Πm−Π0‖op2\| _m- _0\|_op, we obtain ‖qi−qj‖2=2/α+o(1)\|q_i-q_j\|_2= 2/α+o(1) uniformly. At α=1α=1, no perturbation of the polar factor at the hard edge is needed for this step. The deterministic bounds ‖qi−qj‖2≤2\|q_i-q_j\|_2≤ 2 and (202) give 1k(k−1)∑i≠j|‖qi−qj‖2−2|≤12(2−2rdk−1)⟶0. 1k(k-1) _i≠ j |\|q_i-q_j\|_2- 2 |≤ 1 2 (2- 2r_dk-1 ) 0. Thus the required pair-norm convergence holds in averaged L1L^1 at α=1α=1. Furthermore, (qi−qj)⊤(μi−μj)=Sm[i,i]+Sm[j,j]−Sm[i,j]−Sm[j,i]+o(1)=2sα+o(1),(q_i-q_j) ( _i- _j)=S_m[i,i]+S_m[j,j]-S_m[i,j]-S_m[j,i]+o(1)=2s_α+o(1), (203) while ‖μi−μj‖2=2+o(1)\| _i- _j\|_2= 2+o(1) uniformly. Consequently, the pairwise cosine converges in averaged L1L^1 to 2sα2ρα/α2=m1/2(α)ρα=κsp(α). 2s_α 2 _α/α 2= m_1/2(α) _α= _sp(α). (204) Uniform continuity of x↦2−2x 2-2x on [−1,1][-1,1] proves (S4). Collecting the deterministic envelopes proves the proposition. ∎ Lemma C.13 (Weighted trace gap and hard-edge truncation). Let C=UΣV⊤C=U V have rank r, let Q=UV⊤Q=UV , and let B satisfy ‖B‖op≤1\|B\|_op≤ 1 and rank(B)≤rrank(B)≤ r. Put gℓ:=uℓ⊤Bvℓg_ :=u_ Bv_ and aℓ:=1−gℓ∈[0,2]a_ :=1-g_ ∈[0,2]. Then ‖B−Q‖F2 \|B-Q\|_F^2 ≤2∑ℓ=1raℓ, ≤ 2 _ =1^ra_ , (205) 1r∑ℓ=1raℓ 1r _ =1^ra_ ≤2#ℓ:σℓ≤τr+1τr∑ℓ=1rσℓaℓ,τ>0. ≤ 2 \#\ : _ ≤τ\r+ 1τ r _ =1^r _ a_ , τ>0. (206) Proof. The rank and operator-norm bounds give ‖B‖F2≤r\|B\|_F^2≤ r, so expanding ‖B−Q‖F2\|B-Q\|_F^2 proves the first inequality. For the second inequality, use aℓ≤2a_ ≤ 2 when σℓ≤τ _ ≤τ and aℓ≤σℓaℓ/τa_ ≤ _ a_ /τ otherwise. ∎ Lemma C.14 (Average normalization transfer). Let B,Q∈ℝk×dB,Q ^k× d have rows bi,qib_i,q_i, suppose bi−bj≠b_i-b_j≠0 for i≠ji≠ j, and assume infi≠j‖qi−qj‖2≥c0>0 _i≠ j\|q_i-q_j\|_2≥ c_0>0. If E2:=1k(k−1)∑i≠j‖(bi−bj)−(qi−qj)‖22,E_2:= 1k(k-1) _i≠ j\|(b_i-b_j)-(q_i-q_j)\|_2^2, (207) then 1k(k−1)∑i≠j‖bi−bj‖bi−bj‖2−qi−qj‖qi−qj‖2‖2≤4c0E21/2+8c02E2. 1k(k-1) _i≠ j \| b_i-b_j\|b_i-b_j\|_2- q_i-q_j\|q_i-q_j\|_2 \|_2≤ 4c_0E_2^1/2+ 8c_0^2E_2. (208) Proof. For a pairwise difference error eij≤c0/2e_ij≤ c_0/2, the normalization map gives an error at most 4eij/c04e_ij/c_0. For the remaining pairs, use the trivial bound two and eij>c0/2≤4eij2/c021_\e_ij>c_0/2\≤ 4e_ij^2/c_0^2, then average and apply Cauchy–Schwarz. ∎ Proposition C.15 (Exact and near spectral minimizers). On the events of Proposition C.12, let W∈ℱXW _X satisfy ‖W‖op≤(1+ξ)rsp∗(X),0≤ξ≤1.\|W\|_op≤(1+ξ)r_sp^*(X), 0≤ξ≤ 1. (209) Then Davg-bdry(W)≥δsp(α)−Cα(ηd+ξ)1/2,α≠1,C(ηd+ξ)1/4,α=1.D_avg -bdry(W)≥ _sp(α)- casesC_α( _d+ξ)^1/2,&α≠ 1,\\ C( _d+ξ)^1/4,&α=1. cases (210) In particular, every exact minimizer satisfies 1rd‖PW‖PW‖op−Qm‖F2≤Cαηd,α≠1,Cηd,α=1, 1r_d \| PW\|PW\|_op-Q_m \|_F^2≤ casesC_α _d,&α≠ 1,\\ C _d,&α=1, cases (211) uniformly over W∈ℳspXW _sp^X. Proof. The polar witness Qm/ρXQ_m/ _X is feasible, so rsp∗(X)≤ρX−1.r_sp^*(X)≤ _X^-1. (212) For every feasible W, summing the class-averaged constraints over ordered pairs gives ⟨PW,Cm⟩≥k−1. PW,C_m ≥ k-1. (213) Set A:=PWA:=PW, a:=‖A‖op>0a:=\|A\|_op>0, and B:=A/aB:=A/a. Centering is an operator-norm contraction and preserves all row differences, while rank(B)≤rdrank(B)≤ r_d. Equations (212), (213), and Proposition C.12 give 1rd∑ℓ=1rdσℓ(Cm)(1−uℓ⊤Bvℓ)≤C(ηd+ξ). 1r_d _ =1^r_d _ (C_m) (1-u_ Bv_ )≤ C( _d+ξ). (214) If α≠1α≠ 1, the positive edge in (195) and Lemma C.13 yield rd−1‖B−Qm‖F2≤Cα(ηd+ξ)r_d^-1\|B-Q_m\|_F^2≤ C_α( _d+ξ). If α=1α=1, set q:=ηd+ξq:= _d+ξ and use τ=qτ= q in (206) together with (196); the result is 1rd‖B−Qm‖F2≤Cq1/2. 1r_d\|B-Q_m\|_F^2≤ Cq^1/2. (215) The case of non-small q follows by increasing C and using the trivial bound. Taking ξ=0ξ=0 proves (211). Because both B and QmQ_m are centered, 1k(k−1)∑i≠j‖(bi−bj)−(qi−qj)‖22=2k−1‖B−Qm‖F2. 1k(k-1) _i≠ j\|(b_i-b_j)-(q_i-q_j)\|_2^2= 2k-1\|B-Q_m\|_F^2. (216) Feasibility ensures bi−bj≠b_i-b_j≠0, and Proposition C.12 supplies a uniform lower bound on the polar pair norms. Lemma C.14, followed by the reverse triangle inequality and (194), proves (210). ∎ Proposition C.16 (Spectral-norm boundary separation). Under Assumptions 2.1 and 2.2, infW∈ℳspXDavg-bdry(W)≥δsp(α)−oℙ(1). _W _sp^XD_avg -bdry(W)≥ _sp(α)-o_P(1). (217) The bound holds on the events in Proposition C.12 with the deterministic error in (210) at ξ=0ξ=0. Proof. Apply Proposition C.15 with ξ=0ξ=0 and take the infimum over the exact minimizer set. ∎ Proof of the deterministic geometry theorem Proof of Theorem C.2. Let ddetG_d^det be the intersection of ℰddetE_d^det from Proposition C.4 and ℰdspE_d^sp from Proposition C.12. Both complements satisfy a bound of the form Cexp[−cdlog(2N(d)(d+1))].C \! [-c d (2N(d)(d+1)) ]. (218) Since log(2N(d)(d+1))→∞ (2N(d)(d+1))→∞, the quantity in (218) is at most e−de^- d for every sufficiently large d, after increasing the deterministic threshold. The sign construction in Proposition C.4 gives ℱX≠∅F_X≠ on this event, and compactness gives nonempty minimum-norm sets. The proof of Proposition C.7 is deterministic on ℰddetE_d^det and gives the maximum-pair statement (84) under a deterministic vanishing envelope. Lemma C.3 then gives (85). Proposition C.10 and Lemma C.3 give (87). Proposition C.16 gives (86) for every fixed α>0α>0, using the reduced polar factor when α<1α<1 and the hard-edge truncation when α=1α=1. Choose one deterministic ωd↓0 _d 0 dominating all three errors. Because δsp(α)>0 _sp(α)>0 and δmax>0 _ >0, (88) follows for every sufficiently large d. ∎ C.6 From Direction-set Geometry to Population Accuracy We now convert the geometric separation above into ordinary multiclass population accuracy. The lemmas in this subsection are stated for general classifier families so that the same transfer can be reused in the stochastic and AdamProxy analyses. For every classifier with nonzero pairwise row differences, define ℰπ,d(W):=1−Accπ,d(W),pi→j(W):=ℙ(wj⊤Xte≥wi⊤Xte∣Y=i).E_π,d(W):=1-Acc_π,d(W), p_i→ j(W):=P\! (w_j X^te≥ w_i X^te Y=i ). (219) Under the normalized convention in (77), set σd:=rte/d _d:=r_te/ d, and for i≠ji≠ j write dij(W):=‖uijW−uijμ‖2,sij:=‖μi−μj‖2,mij:=μi+μj2.d_ij(W):=\|u_ij^W-u_ij^μ\|_2, s_ij:=\| _i- _j\|_2, m_ij:= _i+ _j2. (220) Lemma C.17 (Pairwise Gaussian error bounds). For every i<ji<j, let eij(W):=pi→j(W)+pj→i(W)/2e_ij(W):=\p_i→ j(W)+p_j→ i(W)\/2. Then maxi<jπi,dpi→j(W)+πj,dpj→i(W)≤ℰπ,d(W)≤∑i<jπi,dpi→j(W)+πj,dpj→i(W), _i<j\ _i,dp_i→ j(W)+ _j,dp_j→ i(W)\ _π,d(W)≤ _i<j\ _i,dp_i→ j(W)+ _j,dp_j→ i(W)\, (221) and hence ℰπ,d(W)≥2πmin,dmaxi<jeij(W)E_π,d(W)≥ 2 _ ,d _i<je_ij(W). Moreover, with cij(W):=1−dij(W)2/2c_ij(W):=1-d_ij(W)^2/2, eij(W)=12[Φ¯(sijcij(W)/2+⟨uijW,mij⟩σd)+Φ¯(sijcij(W)/2−⟨uijW,mij⟩σd)]≥Φ¯(sij(cij(W))+2σd),e_ij(W)= 12 [ \! ( s_ijc_ij(W)/2+ u_ij^W,m_ij _d )+ \! ( s_ijc_ij(W)/2- u_ij^W,m_ij _d ) ]≥ \! ( s_ij(c_ij(W))_+2 _d ), (222) where Φ¯(a)=ℙ(Z≥a) (a)=P(Z≥ a) for Z∼(0,1)Z (0,1). Proof. Conditionally on Y=iY=i, the multiclass error event is the union of the events wj⊤Xte≥wi⊤Xte\w_j X^te≥ w_i X^te\, up to Gaussian-null ties. The union bound gives the upper half of (221), while any fixed unordered pair gives the lower half. For u=uijWu=u_ij^W, v=uijμv=u_ij^μ, m=mijm=m_ij, and s=sijs=s_ij, the identities μi=m+(s/2)v _i=m+(s/2)v and μj=m−(s/2)v _j=m-(s/2)v give pi→j(W)=Φ¯(u⊤μiσd),pj→i(W)=Φ¯(−u⊤μjσd),p_i→ j(W)= \! ( u _i _d ), p_j→ i(W)= \! ( -u _j _d ), which proves the equality in (222). For q≥0q≥ 0, the even function z↦[Φ¯(q+z)+Φ¯(q−z)]/2z [ (q+z)+ (q-z)]/2 is minimized at zero, while for q<0q<0 the same average is at least 1/21/2. Since u⊤v=1−dij(W)2/2u v=1-d_ij(W)^2/2, the final inequality follows. ∎ Define τd:=maxi|‖μi‖22−1|,ρd:=maxi≠j|μi⊤μj|. _d:= _i |\| _i\|_2^2-1 |, _d:= _i≠ j| _i _j|. (223) Lemma C.18 (Quantitative Gaussian radial event). There are deterministic ηd↓0 _d 0 and events ℛdR_d, depending only on the class means, such that, for every sufficiently large d, ℙ(ℛdc)≤14e−d,τd+ρd≤ηdon ℛd.P(R_d^c)≤ 14e^- d, _d+ _d≤ _d R_d. (224) Proof. Under the normalized convention, d‖μi‖22d\| _i\|_2^2 is chi-square with d degrees of freedom, while conditionally on μi _i, the inner product μi⊤μj _i _j is Gaussian with variance ‖μi‖22/d\| _i\|_2^2/d. Taking a sufficiently large constant CαC_α and ηd=Cαd−1/4 _d=C_αd^-1/4, standard chi-square and Gaussian tails followed by a union bound over O(d2)O(d^2) indices give (224). ∎ For nonempty classifier families ℛdclfR_d^clf and dclfC_d^clf, put hd:=supW∈ℛdclfmaxi≠jdij(W),Δd:=infW∈dclfDavg-bdry(W),h_d:= _W _d^clf _i≠ jd_ij(W), _d:= _W _d^clfD_avg -bdry(W), (225) and, whenever τd+ρd<1 _d+ _d<1, define Ad A_d :=1−τd−ρd2(1+τd+ρd)−hd1+τd, := 1- _d- _d 2(1+ _d+ _d)-h_d 1+ _d, (226) Bd B_d :=1+τd+ρd2(1−Δd22)+. := 1+ _d+ _d2 (1- _d^22 )_+. (227) Proposition C.19 (Finite-dimensional geometry-to-error transfer). If Ad>0A_d>0, then supW∈ℛdclfℰπ,d(W)≤(k−1)Φ¯(Adσd),infW∈dclfℰπ,d(W)≥2πmin,dΦ¯(Bdσd). _W _d^clfE_π,d(W)≤(k-1) \! ( A_d _d ), _W _d^clfE_π,d(W)≥ 2 _ ,d \! ( B_d _d ). (228) Proof. For v=uijμv=u_ij^μ, radial control gives v⊤μi≥1−τd−ρd2(1+τd+ρd),−v⊤μj≥1−τd−ρd2(1+τd+ρd).v _i≥ 1- _d- _d 2(1+ _d+ _d), -v _j≥ 1- _d- _d 2(1+ _d+ _d). If W∈ℛdclfW _d^clf, then ‖uijW−v‖2≤hd\|u_ij^W-v\|_2≤ h_d, so both directed population margins are at least AdA_d. The upper bound follows from the union bound in Lemma C.17. If W∈dclfW _d^clf, some pair has dij(W)≥Δd_ij(W)≥ _d, while sij/2≤(1+τd+ρd)/2s_ij/2≤ (1+ _d+ _d)/2. The midpoint-free lower bound in Lemma C.17 then gives the second inequality. ∎ Define (h):=12−h,(δ):=12(1−δ22)+. a(h):= 1 2-h, b(δ):= 1 2 (1- δ^22 )_+. (229) Theorem C.20 (Geometry-to-accuracy transfer). Suppose τd+ρd→0 _d+ _d→ 0, lim supdhd≤h¯ _dh_d≤ h, and lim infdΔd≥δ¯>0 _d _d≥ δ>0. If (h¯)>(δ¯) a( h)> b( δ), then under Assumption 2.2, supW∈ℛdclfℰπ,d(W)infW∈dclfℰπ,d(W)⟶0, _W _d^clfE_π,d(W) _W _d^clfE_π,d(W) 0, (230) and therefore, for every sufficiently large d, infW∈ℛdclfAccπ,d(W)>supW∈dclfAccπ,d(W). _W _d^clfAcc_π,d(W)> _W _d^clfAcc_π,d(W). (231) Proof. Choose constants 0<b<a0<b<a such that (δ¯)<b<a<(h¯) b( δ)<b<a< a( h). Proposition C.19 gives, for all sufficiently large d, supℛdclfℰπ,dinfdclfℰπ,d≤k−12πmin,dΦ¯(a/σd)Φ¯(b/σd). _R_d^clfE_π,d _C_d^clfE_π,d≤ k-12 _ ,d (a/ _d) (b/ _d). The standard Mills bounds imply that the right-hand side is at most Ckπmin,d−1e−cdCk _ ,d^-1e^-cd for constants C,c>0C,c>0, because rte≤rte,0r_te≤ r_te,0. Assumption 2.2 makes this quantity vanish. ∎ Lemma C.21 (Population-error continuity along a direction set). Fix d, a linearly separable training set X containing at least one sample from every class in [k][k], a matrix norm ψ, and 0≤ε<γψ(X)0≤ < _ψ(X). If dcos(Wt,ψ,εX)→0d_ (W_t,U_ψ, ^X)→ 0, then lim supt→∞ℰπ,d(Wt)≤supU∈ψ,εXℰπ,d(U),lim inft→∞ℰπ,d(Wt)≥infU∈ψ,εXℰπ,d(U). _t→∞E_π,d(W_t)≤ _U _ψ, ^XE_π,d(U), _t→∞E_π,d(W_t)≥ _U _ψ, ^XE_π,d(U). (232) Proof. The norm-normalized section of ψ,εXU_ψ, ^X is compact, and its strictly positive empirical margin keeps every pairwise row difference uniformly away from zero. Population error is invariant under positive scaling and continuous on this section because the Gaussian test law gives zero mass to score-tie hyperplanes. Uniform continuity on a compact neighborhood proves the claim. ∎ Lemma C.22 (Boundary metric under direction-set convergence). Under the fixed-data assumptions of Lemma C.21, if dcos(Wt,ψ,εX)→0d_ (W_t,U_ψ, ^X)→ 0 with 0≤ε<γψ(X)0≤ < _ψ(X), then lim supt→∞Davg-bdry(Wt)≤supU∈ψ,εXDavg-bdry(U),lim inft→∞Davg-bdry(Wt)≥infU∈ψ,εXDavg-bdry(U). _t→∞D_avg -bdry(W_t)≤ _U _ψ, ^XD_avg -bdry(U), _t→∞D_avg -bdry(W_t)≥ _U _ψ, ^XD_avg -bdry(U). (233) Proof. The same compactness argument as in Lemma C.21 applies because Davg-bdryD_avg -bdry is scale invariant and continuous whenever all pairwise row differences are nonzero. ∎ C.7 Proof of the Deterministic Optimizer Accuracy Ordering Proof of Theorem 3.1. Work on the event ddetG_d^det from Theorem C.2. Proposition C.4, items (H1)–(H2) imply τd+ρd=o(1) _d+ _d=o(1) on this event. Take ℛdclf=rowXR_d^clf=U_row^X. Equation (84) gives hd=o(1)h_d=o(1). For ψ∈sp,maxψ∈\sp, \, take dclf=ψXC_d^clf=U_ψ^X. Equations (86) and (87) give lim infdΔd≥δψ>0 _d _d≥ _ψ>0, where δsp=δsp(α) _sp= _sp(α) and δmax _ is defined in (79). Since (0)=12>12(1−δψ22)+=(δψ), a(0)= 1 2> 1 2 (1- _ψ^22 )_+= b( _ψ), (234) Theorem C.20 gives, for every sufficiently large d on ddetG_d^det, supU∈rowXℰπ,d(U)<mininfV∈ψXψ∈sp,maxℰπ,d(V). _U _row^XE_π,d(U)< _ψ∈\sp, \ _V _ψ^XE_π,d(V). (235) The full-batch implicit-bias results in Appendix B, together with Lemma B.2, give direction-set convergence to rowXU_row^X for RowGD and RowGDM, to spXU_sp^X for Spectral-GD and exact-SVD Muon, and to maxXU_ ^X for SignGD, Signum, and full-batch Adam under the stated Adam assumptions. Lemma C.21 transfers (235) to all seven trajectories. Since ℙ(ddet)≥1−e−dP(G_d^det)≥ 1-e^- d, replacing error by one minus accuracy proves (9) with the claimed probability. ∎ C.8 Proof of the Limiting Classifier Geometry We prove Proposition 3.2 under the normalized convention (77), under which μi,zi,s∼iid(,Id/d) _i,z_i,s iid N(0,I_d/d). Since all statements below are invariant under the common rescaling used in (77), they transfer directly to the original normalization. Let Mμ∈ℝk×dM_μ ^k× d have rows μi⊤ _i , let Cμ:=PMμC_μ:=PM_μ, and write a compact SVD Cμ=UμΣμVμ⊤C_μ=U_μ _μV_μ . Define Qμ:=UμVμ⊤Q_μ:=U_μV_μ , and write qi⊤q_i for its i-th row. Recall from (81) that κsp(α)=m1/2(α)α∧1∈(0,1). _sp(α)= m_1/2(α) α 1∈(0,1). In particular, when α>1α>1, κsp(α)=m1/2(α) _sp(α)=m_1/2(α). Lemma C.23 (Empirical-to-population polar transfer). Under Assumptions 2.1 and 2.2, 1rd‖Qm−Qμ‖F2→ℙ0. 1r_d\|Q_m-Q_μ\|_F^2 P0. (236) Proof. By (91), the i-th row of Mm−MμM_m-M_μ is rz¯i⊤r z_i , and the rows are independent centered Gaussian vectors with covariance r2Id/(nid)r^2I_d/(n_id). Hence Mm−Mμ=DdeH/dM_m-M_μ=D_d^eH/ d for a standard Gaussian matrix H and a diagonal matrix satisfying ‖Dde‖op≤r0/nmin\|D_d^e\|_op≤ r_0/ n_ . Since k/d→αk/d→α and nmin→∞n_ →∞, δd:=‖Cm−Cμ‖op≤‖Mm−Mμ‖op=oℙ(1). _d:=\|C_m-C_μ\|_op≤\|M_m-M_μ\|_op=o_P(1). (237) On the full reduced-rank event, ‖Qμ‖op=1\|Q_μ\|_op=1, rank(Qμ)=rdrank(Q_μ)=r_d, and ⟨Qμ,Cμ⟩=‖Cμ‖∗ Q_μ,C_μ =\|C_μ\|_*. Therefore 0 0 ≤‖Cm‖∗−⟨Qμ,Cm⟩ ≤\|C_m\|_*- Q_μ,C_m ≤|‖Cm‖∗−‖Cμ‖∗|+|⟨Qμ,Cm−Cμ⟩|≤2rdδd. ≤ |\|C_m\|_*-\|C_μ\|_* |+ | Q_μ,C_m-C_μ |≤ 2r_d _d. (238) Write Cm=UmΣmVm⊤C_m=U_m _mV_m , and set aℓ:=1−um,ℓ⊤Qμvm,ℓa_ :=1-u_m, Q_μv_m, . Then (238) gives 1rd∑ℓ=1rdσℓ(Cm)aℓ≤2δd. 1r_d _ =1^r_d _ (C_m)a_ ≤ 2 _d. (239) If α≠1α≠ 1, Proposition C.12, item (S5), bounds every positive singular value of CmC_m away from zero, so Lemma C.13 and (239) yield rd−1‖Qm−Qμ‖F2=oℙ(1)r_d^-1\|Q_m-Q_μ\|_F^2=o_P(1). Suppose α=1α=1. Choose a deterministic sequence δ¯d↓0 δ_d 0 such that δd≤δ¯d _d≤ δ_d with probability tending to one, and set τd:=maxηd,δ¯d _d:= \ _d, δ_d\, where ηd _d is the envelope in Proposition C.12. For all sufficiently large d, τd≤t0 _d≤ t_0, and item (S5) together with Lemma C.13 gives 1rd‖Qm−Qμ‖F2≤4C0(τd+ηd)+4δdτd=oℙ(1). 1r_d\|Q_m-Q_μ\|_F^2≤ 4C_0( _d+ _d)+ 4 _d _d=o_P(1). This proves (236). ∎ Proof of Proposition 3.2. We first establish the claimed geometry uniformly over the corresponding exact maximum-margin faces. For the Row geometry, Proposition C.6 and Proposition C.4, item (H1), give supW∈ℳrowXmaxi‖wi−μi‖2=oℙ(1). _W _row^X _i\|w_i- _i\|_2=o_P(1). (240) Since mini‖μi‖2=1−oℙ(1) _i\| _i\|_2=1-o_P(1), Lemma C.5 yields supW∈ℳrowXmaxi‖w^i−μ^i‖2=oℙ(1). _W _row^X _i\| w_i- μ_i\|_2=o_P(1). (241) For the Max geometry, put g:=2/πg:= 2/π, let Ti:=sign(μi)T_i:=sign( _i), and write tmax:=rmax∗(X)t_ :=r_ ^*(X). For W∈ℳmaxXW _ ^X, set A:=W/tmaxA:=W/t_ . The sharp radius estimate (256) and the global sign rigidity (263) imply tmax=1+oℙ(1)gd,supW∈ℳmaxX‖A−T‖F2kd=oℙ(1),t_ = 1+o_P(1)g d, _W _ ^X \|A-T\|_F^2kd=o_P(1), (242) where T has rows Ti⊤T_i . Let A¯:=k−1∑iAi A:=k^-1 _iA_i and T¯:=k−1∑iTi T:=k^-1 _iT_i. Jensen’s inequality, (242), and ‖T¯‖22=d/k=O(1)E\| T\|_2^2=d/k=O(1) give supW‖A¯‖2/d=oℙ(1) _W\| A\|_2/ d=o_P(1). The class-averaged feasibility constraint (107) then gives, uniformly over i and W∈ℳmaxXW _ ^X, Ai⊤μi≥gd−oℙ(d).A_i _i≥ g d-o_P( d). Together with maxi|‖μi‖1/d−g|=oℙ(1) _i|\| _i\|_1/ d-g|=o_P(1) from (259), this implies Di:=‖μi‖1−Ai⊤μi=oℙ(d)D_i:=\| _i\|_1-A_i _i=o_P( d) uniformly. Substituting this bound into (264) and using the uniform Gaussian small-ball estimate (260) yields supW∈ℳmaxXmaxi1d‖Ai−Ti‖22=oℙ(1). _W _ ^X _i 1d\|A_i-T_i\|_2^2=o_P(1). (243) Since ‖Ti‖2=d\|T_i\|_2= d, normalization stability gives supW∈ℳmaxXmaxi‖w^i−sign(μi)d‖2=oℙ(1). _W _ ^X _i \| w_i- sign( _i) d \|_2=o_P(1). (244) For the Spectral geometry, let B:=PW/‖PW‖opB:=PW/\|PW\|_op. Combining (211) with Lemma C.23 gives supW∈ℳspX1rd‖B−Qμ‖F2=oℙ(1). _W _sp^X 1r_d\|B-Q_μ\|_F^2=o_P(1). (245) This proves the population-mean version of the spectral geometry claimed in the proposition. It remains to identify the mean angles when α>1α>1. For Row, (241) immediately gives a limiting angle of zero. For Max, (244), maxi|‖μi‖2−1|=oℙ(1) _i|\| _i\|_2-1|=o_P(1), and (259) give supW∈ℳmaxXmaxi|wi⊤μi‖wi‖2‖μi‖2−2π|=oℙ(1). _W _ ^X _i | w_i _i\|w_i\|_2\| _i\|_2- 2π |=o_P(1). (246) For Spectral, α>1α>1 implies rd=dr_d=d. Since CμC_μ is a centered Gaussian matrix, the left singular frame UμU_μ is Haar in ⟂1 and independent of its singular values. Haar-projector concentration and the Marchenko–Pastur limit therefore give maxi|‖qi‖22−1α|=oℙ(1),maxi|qi⊤μi−sα|=oℙ(1). _i |\|q_i\|_2^2- 1α |=o_P(1), _i |q_i _i-s_α |=o_P(1). (247) Indeed, if Sμ:=QμCμ⊤=UμΣμUμ⊤S_μ:=Q_μC_μ =U_μ _μU_μ , conditional Haar concentration gives maxi|Sμ[i,i]−‖Cμ‖∗/k|=oℙ(1) _i|S_μ[i,i]-\|C_μ\|_*/k|=o_P(1), while ‖Cμ‖∗/k→sα\|C_μ\|_*/k→ s_α and qi⊤(μi−μ¯)=Sμ[i,i]q_i ( _i- μ)=S_μ[i,i] with ‖μ¯‖2=oℙ(1)\| μ\|_2=o_P(1). Since sα=m1/2(α)/αs_α=m_1/2(α)/ α and κsp(α)=m1/2(α) _sp(α)=m_1/2(α) for α>1α>1, (247) yields maxi|qi⊤μi‖qi‖2‖μi‖2−κsp(α)|=oℙ(1). _i | q_i _i\|q_i\|_2\| _i\|_2- _sp(α) |=o_P(1). (248) We now transfer (248) from QμQ_μ to arbitrary exact Spectral minimizers. Fix W∈ℳspXW _sp^X, set a:=‖W‖opa:=\|W\|_op, and let w¯:=k−1∑iwi w:=k^-1 _iw_i and h:=w¯/ah:= w/a. Centering preserves every margin and cannot increase the operator norm, so exact optimality gives ‖PW‖op=‖W‖op=a\|PW\|_op=\|W\|_op=a. Hence, with B:=PW/aB:=PW/a, Wa=B+h⊤,⊤B=⊤. Wa=B+1h , 1 B=0 . (249) Because ‖Qμ‖F2=d\|Q_μ\|_F^2=d, (245) gives ‖B‖F2=d−oℙ(d)\|B\|_F^2=d-o_P(d) uniformly over ℳspXM_sp^X. The two terms in (249) are Frobenius-orthogonal, while ‖W/a‖op=1\|W/a\|_op=1 implies ‖W/a‖F2≤d\|W/a\|_F^2≤ d. Therefore d≥‖Wa‖F2=‖B‖F2+k‖h‖22,d≥ \| Wa \|_F^2=\|B\|_F^2+k\|h\|_2^2, and hence supW∈ℳspX‖h‖2=oℙ(1). _W _sp^X\|h\|_2=o_P(1). (250) Consequently, supW∈ℳspX1k∑i=1k‖wia−qi‖22=supW∈ℳspX1k‖B−Qμ‖F2+‖h‖22=oℙ(1). _W _sp^X 1k _i=1^k \| w_ia-q_i \|_2^2= _W _sp^X \ 1k\|B-Q_μ\|_F^2+\|h\|_2^2 \=o_P(1). (251) Moreover, every spectral row is nonzero with probability tending to one: if wi=w_i=0, then (107) gives −ah⊤mi≥1−1/k-ah m_i≥ 1-1/k, contradicting a=Oℙ(1)a=O_P(1), (250), and maxi‖mi‖2=Oℙ(1) _i\|m_i\|_2=O_P(1). Since (247) bounds mini‖qi‖2 _i\|q_i\|_2 away from zero, (251) and normalization stability imply that, for every fixed ε>0 >0, supW∈ℳspX1k#i:|arccos(wi⊤μi‖wi‖2‖μi‖2)−arccosκsp(α)|>ε→ℙ0. _W _sp^X 1k\# \i: | \! ( w_i _i\|w_i\|_2\| _i\|_2 )- _sp(α) |> \ P0. (252) Because every angle lies in [0,π][0,π], (252) implies supW∈ℳspX|1k∑i=1karccos(wi⊤μi‖wi‖2‖μi‖2)−arccosκsp(α)|→ℙ0. _W _sp^X | 1k _i=1^k \! ( w_i _i\|w_i\|_2\| _i\|_2 )- _sp(α) | P0. (253) Finally, (74) identifies each maximum-margin direction set with the positive rays through the corresponding exact minimum-norm face. The full-batch implicit-bias results used in Theorem 3.1, together with Lemma B.2, give direction-set convergence to rowXU_row^X for RowGD and RowGDM, to spXU_sp^X for SpecGD and Muon, and to maxXU_ ^X for SignGD, Signum, and Adam. At each fixed d, the normalized row directions, the centered spectral representative PW/‖PW‖opPW/\|PW\|_op, and the mean class–row angle are continuous on the corresponding compact direction sets on the high-probability events established above. Taking t→∞t→∞ first and then d→∞d→∞ therefore transfers (241), (244), (245), and the three angle limits to the optimizer trajectories, proving Proposition 3.2. ∎ C.9 Quantitative Population-error Exponents for Deterministic Geometries Because all three accuracies may converge to one, a strict accuracy ordering can still correspond to an exponentially large separation in population error. To quantify this separation, define the directed-pair exponent and normalized log error by d(W):=mini≠j[(uijW)⊤μi]+2,d(W):=−2rte2dlogℰπ,d(W). E_d(W):= _i≠ j [(u_ij^W) _i ]_+^2, F_d(W):=- 2r_te^2d _π,d(W). (254) Both are invariant under positive rescaling and common row shifts. Lemma C.24 (Directed-pair exponent controls multiclass error). For every classifier with nonzero pairwise row differences, set Id(W):=dd(W)/(2rte2)I_d(W):=d E_d(W)/(2r_te^2) and C0:=3/2+12log(2π)C_0:=3/2+ 12 (2π). Then max0,Id(W)−log(k−1)≤−logℰπ,d(W)≤Id(W)+log(1/πmin,d)+log(1+2Id(W))+C0. \0,I_d(W)- (k-1)\≤- _π,d(W)≤ I_d(W)+ (1/ _ ,d)+ (1+ 2I_d(W))+C_0. (255) Consequently, uniformly over any classifier family on which d(W)=O(1) E_d(W)=O(1), one has d(W)=d(W)+o(1) F_d(W)= E_d(W)+o(1) under Assumption 2.2. Proof. For a directed pair, the exact error is Φ¯(d(uijW)⊤μi/rte) ( d\,(u_ij^W) _i/r_te). Let t=2Id(W)t= 2I_d(W). The Gaussian tail bounds e−C01+te−t2/2≤Φ¯(t)≤12e−t2/2 e^-C_01+te^-t^2/2≤ (t)≤ 12e^-t^2/2 show that one directed pair contributes at least πmin,de−C0−Id(W)/(1+2Id(W)) _ ,de^-C_0-I_d(W)/(1+ 2I_d(W)) to the multiclass error, while a classwise union bound gives ℰπ,d(W)≤(k−1)e−Id(W)E_π,d(W)≤(k-1)e^-I_d(W) when all directed population margins are nonnegative; if one is negative then Id(W)=0I_d(W)=0, and the same logarithmic lower bound is trivial. This proves (255). Multiplying by 2rte2/d2r_te^2/d and using k=O(d)k=O(d), log(1/πmin,d)=o(d) (1/ _ ,d)=o(d), and rte≤rte,0r_te≤ r_te,0 proves the uniform o(1)o(1) reduction. ∎ Let g:=2/πg:= 2/π. The following sharpening is needed only for the quantitative statement; the qualitative accuracy theorem above already follows from the angular gaps in Theorem C.2. Proposition C.25 (Sharp directed signals on the Max and Spectral faces). Under Assumptions 2.1 and 2.2, in probability, rmax∗(X)=1+o(1)gd,rsp∗(X)=1+o(1)sα.r_ ^*(X)= 1+o(1)g d, r_sp^*(X)= 1+o(1)s_α. (256) Moreover, uniformly over the complete exact faces, supW∈ℳmaxXmaxi≠j|(uijW)⊤μi−1π| _W _ ^X _i≠ j |(u_ij^W) _i- 1 π | =oℙ(1), =o_P(1), (257) infW∈ℳspXmini≠j(uijW)⊤μi _W _sp^X _i≠ j(u_ij^W) _i ≥sα2−oℙ(1). ≥ s_α 2-o_P(1). (258) Proof. Let MμM_μ have rows μi⊤ _i , let Cμ:=PMμC_μ:=PM_μ, and let T:=sign(Mμ)T:=sign(M_μ). Standard Gaussian concentration, uniformly over the k=Θ(d)k= (d) classes, gives ‖Cμ‖sumkd=g+oℙ(1),maxi|‖μi‖1d−g|=oℙ(1),maxi≠j|Tj⊤μi|d=oℙ(1), \|C_μ\|_sumk d=g+o_P(1), _i | \| _i\|_1 d-g |=o_P(1), _i≠ j |T_j _i| d=o_P(1), (259) and, for every fixed a>0a>0, 1kd#(i,ℓ):d|Cμ[i,ℓ]|≤a→ℙ(|G|≤a),maxi1d#ℓ:d|μi[ℓ]|≤a≤ℙ(|G|≤a)+oℙ(1). 1kd\#\(i, ): d\,|C_μ[i, ]|≤ a\ PP(|G|≤ a), _i 1d\#\ : d\,| _i[ ]|≤ a\ (|G|≤ a)+o_P(1). (260) Also ‖Cm−Cμ‖sum=oℙ(kd)\|C_m-C_μ\|_sum=o_P(k d) by Proposition C.4. Every feasible W satisfies the averaged exposure inequality ⟨W,Cm⟩≥k−1 W,C_m ≥ k-1. Max–sum duality and (259) therefore give rmax∗(X)≥(1+oℙ(1))/(gd)r_ ^*(X)≥(1+o_P(1))/(g d), while Proposition C.4, item (H5) gives the matching upper bound. Similarly, operator–nuclear duality and Proposition C.12, item (S1) give rsp∗(X)≥(1+oℙ(1))/sαr_sp^*(X)≥(1+o_P(1))/s_α, while the polar witness Qm/ρXQ_m/ _X and Proposition C.12, item (S3) give the matching upper bound. This proves (256). Fix W∈ℳmaxXW _ ^X and set A:=W/rmax∗(X)A:=W/r_ ^*(X), so ‖A‖max=1\|A\|_ =1. The exposure inequality and the exact Max radius imply ⟨A,Cμ⟩≥‖Cμ‖sum−oℙ(kd) A,C_μ ≥\|C_μ\|_sum-o_P(k d) (261) uniformly over the complete Max face. Put S:=sign(Cμ)S:=sign(C_μ) and Δiℓ:=1−A[i,ℓ]S[i,ℓ]∈[0,2] _i :=1-A[i, ]S[i, ]∈[0,2]. Since (A[i,ℓ]−S[i,ℓ])2=Δiℓ2≤2Δiℓ(A[i, ]-S[i, ])^2= _i ^2≤ 2 _i , for every fixed a>0a>0, ‖A−S‖F2kd≤2dakd(‖Cμ‖sum−⟨A,Cμ⟩)+4kd#(i,ℓ):d|Cμ[i,ℓ]|≤a. \|A-S\|_F^2kd≤ 2 dakd (\|C_μ\|_sum- A,C_μ )+ 4kd\#\(i, ): d\,|C_μ[i, ]|≤ a\. (262) First sending d→∞d→∞ and then a↓0a 0 gives supW‖A−S‖F2/(kd)=oℙ(1) _W\|A-S\|_F^2/(kd)=o_P(1). Since Cμ[i,ℓ]=μi[ℓ]−μ¯[ℓ]C_μ[i, ]= _i[ ]- μ[ ], where μ¯=k−1∑iμi μ=k^-1 _i _i and ‖μ¯‖2=oℙ(1)\| μ\|_2=o_P(1), the same small-ball argument transfers S to T, yielding supW∈ℳmaxX‖A−T‖F2kd=oℙ(1). _W _ ^X \|A-T\|_F^2kd=o_P(1). (263) Let A¯=k−1∑iAi A=k^-1 _iA_i and T¯=k−1∑iTi T=k^-1 _iT_i. Jensen’s inequality and (263), together with ‖T¯‖22=d/k=O(1)E\| T\|_2^2=d/k=O(1), imply supW‖A¯‖2/d=oℙ(1) _W\| A\|_2/ d=o_P(1). Averaging the class-i constraints over its samples and over all competitors gives (Ai−A¯)⊤mi≥k−1krmax∗(X)=gd−oℙ(d).(A_i- A) m_i≥ k-1k\,r_ ^*(X)=g d-o_P( d). Proposition C.4, item (H1), the bound ‖Ai−A¯‖2≤2d\|A_i- A\|_2≤ 2 d, and the preceding control of A¯ A therefore imply Ai⊤μi≥gd−oℙ(d)A_i _i≥ g d-o_P( d) uniformly over i and the complete face. The nonnegative deficiency Di:=‖μi‖1−Ai⊤μiD_i:=\| _i\|_1-A_i _i is thus oℙ(d)o_P( d) uniformly. For every fixed a>0a>0, 1d‖Ai−Ti‖22≤2Diad+4d#ℓ:d|μi[ℓ]|≤a. 1d\|A_i-T_i\|_2^2≤ 2D_ia d+ 4d\#\ : d\,| _i[ ]|≤ a\. (264) Using (260) and then sending a↓0a 0 proves supWmaxi‖Ai−Ti‖2/d=oℙ(1) _W _i\|A_i-T_i\|_2/ d=o_P(1). Finally, uniformly over i≠ji≠ j, (Ti−Tj)⊤μi=gd+oℙ(d)(T_i-T_j) _i=g d+o_P( d) and ‖Ti−Tj‖2=2d+oℙ(d)\|T_i-T_j\|_2= 2d+o_P( d), which proves (257). For Spectral, class-average feasibility gives (wi−wj)⊤mi≥1(w_i-w_j) m_i≥ 1, while ‖wi−wj‖2≤2‖W‖op=2rsp∗(X)\|w_i-w_j\|_2≤ 2\|W\|_op= 2\,r_sp^*(X). Proposition C.4, item (H1) and (256) therefore give (uijW)⊤μi≥sα/2−oℙ(1)(u_ij^W) _i≥ s_α/ 2-o_P(1) uniformly, proving (258). ∎ Proposition C.26 (Deterministic population-error exponents). Under Assumptions 2.1 and 2.2, uniformly over the indicated complete maximum-margin direction sets, supR∈rowX|d(R)−12| _R _row^X | F_d(R)- 12 | =oℙ(1), =o_P(1), (265) supM∈maxX|d(M)−1π| _M _ ^X | F_d(M)- 1π | =oℙ(1), =o_P(1), (266) sα22−oℙ(1)≤d(S) s_α^22-o_P(1)≤ F_d(S) ≤κsp(α)22+oℙ(1),S∈spX. ≤ _sp(α)^22+o_P(1), S _sp^X. (267) Consequently, for 0<α≤10<α≤ 1, the spectral interval collapses to κsp(α)2/2 _sp(α)^2/2; at α=1α=1, this value is 32/(9π2)32/(9π^2). Uniformly over R∈rowXR _row^X, M∈maxXM _ ^X, and S∈spXS _sp^X, logℰπ,d(M)ℰπ,d(R)=(π−24π+oℙ(1))drte2, E_π,d(M)E_π,d(R)= ( π-24π+o_P(1) ) dr_te^2, (268) 1−κsp(α)22−oℙ(1)≤2rte2dlogℰπ,d(S)ℰπ,d(R)≤1−sα22+oℙ(1). 1- _sp(α)^22-o_P(1)≤ 2r_te^2d E_π,d(S)E_π,d(R)≤ 1-s_α^22+o_P(1). (269) The Row–Spectral gain is exact when α≤1α≤ 1. Proof. For Row, Proposition C.7 and the Gaussian mean shell imply (uijR)⊤μi=1/2+oℙ(1)(u_ij^R) _i=1/ 2+o_P(1) uniformly over all directed pairs and the complete Row face. Proposition C.25 gives the corresponding uniform Max signal 1/π+oℙ(1)1/ π+o_P(1). Hence d(R)=1/2+oℙ(1) E_d(R)=1/2+o_P(1) and d(M)=1/π+oℙ(1) E_d(M)=1/π+o_P(1) uniformly. For Spectral, Proposition C.25 gives d(S)≥sα2/2−oℙ(1) E_d(S)≥ s_α^2/2-o_P(1) uniformly. Theorem C.2 implies that every spectral direction has an ordered pair with dij(S)≥δsp(α)−oℙ(1)d_ij(S)≥ _sp(α)-o_P(1), and hence (uijS)⊤uijμ≤κsp(α)+oℙ(1)(u_ij^S) u_ij^μ≤ _sp(α)+o_P(1). For that unordered pair, (uijS)⊤μi+(ujiS)⊤μj=(uijS)⊤(μi−μj)≤2κsp(α)+oℙ(1).(u_ij^S) _i+(u_ji^S) _j=(u_ij^S) ( _i- _j)≤ 2\, _sp(α)+o_P(1). Both directed signals are positive by the preceding lower bound, so at least one is at most κsp(α)/2+oℙ(1) _sp(α)/ 2+o_P(1). This proves the upper endpoint for d(S) E_d(S). Lemma C.24 transfers all three statements from d E_d to d F_d. Finally, sα=κsp(α)s_α= _sp(α) for α≤1α≤ 1, while subtraction of the exponents gives (268)–(269). ∎ Proposition C.27 (Population-error exponents of deterministic optimizers). Under the assumptions and optimizer conventions of Theorem 3.1, the deterministic full-batch optimizer trajectories satisfy, in probability, lim supt→∞|d(WtRowGDM)−12| _t→∞ | F_d\! (W_t^RowGDM )- 12 | =oℙ(1), =o_P(1), (270) lim supt→∞|d(WtAdam)−1π| _t→∞ | F_d\! (W_t^Adam )- 1π | =oℙ(1). =o_P(1). (271) Exact-SVD Muon satisfies sα22−oℙ(1)≤lim inft→∞d(WtMuon)≤lim supt→∞d(WtMuon)≤κsp(α)22+oℙ(1). s_α^22-o_P(1)≤ _t→∞ F_d\! (W_t^Muon )≤ _t→∞ F_d\! (W_t^Muon )≤ _sp(α)^22+o_P(1). (272) Define the Row–Spectral exponent-gap constant Δexp(α):=1−κsp(α)24=14(1−m1/2(α)2α∧1)>0,m1/2(α):=α2π∫aαbα(bα−x)(x−aα)xx, _ exp(α):= 1- _sp(α)^24= 14 (1- m_1/2(α)^2α 1 )>0, m_1/2(α):= α2π _a_α^b_α (b_α-x)(x-a_α) x\,dx, (273) where aα=(1−α−1/2)2,bα=(1+α−1/2)2.a_α=(1-α^-1/2)^2, b_α=(1+α^-1/2)^2. (274) Equivalently, Δexp(α)=14[1−α24π2(α∧1)(∫aαbα(bα−x)(x−aα)xx)2]. _ exp(α)= 14 [1- α^24π^2(α 1) ( _a_α^b_α (b_α-x)(x-a_α) x\,dx )^2 ]. (275) For example, Δexp(1)=14(1−649π2)≈0.06987,Δexp(2)≈0.03247,Δexp(3)≈0.02133. _ exp(1)= 14 (1- 649π^2 )≈ 0.06987, _ exp(2)≈ 0.03247, _ exp(3)≈ 0.02133. (276) Consequently, lim supt→∞|rte2dlogℰπ,d(WtAdam)ℰπ,d(WtRowGDM)−π−24π|=oℙ(1), _t→∞ | r_te^2d E_π,d(W_t^Adam)E_π,d(W_t^RowGDM)- π-24π |=o_P(1), (277) and Δexp(α)−oℙ(1) _ exp(α)-o_P(1) ≤lim inft→∞rte2dlogℰπ,d(WtMuon)ℰπ,d(WtRowGDM) ≤ _t→∞ r_te^2d E_π,d(W_t^Muon)E_π,d(W_t^RowGDM) ≤lim supt→∞rte2dlogℰπ,d(WtMuon)ℰπ,d(WtRowGDM)≤1−sα24+oℙ(1). ≤ _t→∞ r_te^2d E_π,d(W_t^Muon)E_π,d(W_t^RowGDM)≤ 1-s_α^24+o_P(1). (278) The same conclusions hold with RowGDM replaced by RowGD, with Adam replaced by SignGD or Signum, and with Muon replaced by Spectral-GD. When 0<α≤10<α≤ 1, the two Row–Muon bounds coincide. Here, oℙ(1)o_P(1) refers to the limit d→∞d→∞, after taking t→∞t→∞ at each fixed d. Proof. The full-batch margin-convergence results used in Theorem 3.1 imply that the normalized trajectories approach the corresponding row-, entrywise max-, and spectral-norm maximum-margin direction sets. Since Proposition C.26 holds uniformly over each complete direction set, Lemma C.21 transfers its exponent bounds to the long-time optimizer trajectories. This gives (270)–(272). The definition (273) follows from κsp(α)=m1/2(α)/α∧1 _sp(α)=m_1/2(α)/ α 1, and the integral form follows from the Marchenko–Pastur density. The log-error ratio statements follow from rte2dlogℰπ,d(W(2))ℰπ,d(W(1))=12(d(W(1))−d(W(2))). r_te^2d E_π,d(W^(2))E_π,d(W^(1))= 12 ( F_d(W^(1))- F_d(W^(2)) ). No uniqueness of the optimizer-selected limiting direction is required. ∎ Corollary C.28 (Full-batch Muon versus Adam). Under the assumptions and optimizer conventions of Theorem 3.1, suppose 0<α<α⋆0<α< _ , where α⋆≈1.225 _ ≈ 1.225 is the aspect-ratio threshold defined by sα⋆2=2/πs_ _ ^2=2/π. Then, with cMA(α):=14(sα2−2π)>0,c_MA(α):= 14 (s_α^2- 2π )>0, the deterministic full-batch trajectories satisfy lim supt→∞rte2dlogℰπ,d(WtMuon)ℰπ,d(WtAdam)≤−cMA(α)+oℙ(1). _t→∞ r_te^2d E_π,d(W_t^Muon)E_π,d(W_t^Adam)≤-c_MA(α)+o_P(1). (279) Thus exact-SVD Muon has asymptotically smaller population error, and hence higher population accuracy, than full-batch Adam with probability tending to one. The same conclusion holds with Muon replaced by Spectral-GD and Adam replaced by SignGD or Signum. Proof. Proposition C.27 gives lim inft→∞d(WtMuon)≥sα2/2−oℙ(1) _t→∞ F_d(W_t^Muon)≥ s_α^2/2-o_P(1) and d(WtAdam)=1/π+oℙ(1) F_d(W_t^Adam)=1/π+o_P(1) at long times. Using rte2dlog(ℰ2/ℰ1)=12(d(W1)−d(W2)) r_te^2d (E_2/E_1)= 12( F_d(W_1)- F_d(W_2)) gives (279). ∎ Remark C.29 (Relation to last-layer practice). In practical last-layer training, Adam is typically preferred to Muon, whereas Corollary C.28 predicts the opposite ordering for α<α⋆α< _ . A likely source of this mismatch is that our result compares only asymptotic convergence directions and therefore does not capture convergence speed or finite-time trajectories. The comparison is also idealized, using deterministic full-batch training, exact-SVD Muon, and Adam without a numerical-stability constant. For α≥α⋆α≥ _ , the ordering between Adam and Muon remains open. Appendix D Proofs for Stochastic Optimizers This appendix proves Proposition 3.4. The random-reshuffling momentum analysis in Corollary B.12 places R-RowSGDM, R-Muon, and R-Signum in near-maximum-margin direction sets for the row, spectral, and entrywise max norms, respectively. We show that, for sufficiently large momentum, these near-margin sets retain enough of the deterministic geometric separation to yield a strict population-accuracy ordering. Throughout this appendix, training time tends to infinity at each fixed dimension before the limit d→∞d→∞. For each d, write md:=N(d)/bdm_d:=N(d)/b_d. Recall the geometry-adapted atom radius Rψ(X)R_ψ(X) from (39), and for ψ∈ψrow,ψsp,ψmaxψ∈\ _row, _sp, _ \ define εψ,β,b(X):=2(1−β)md(md2−1)Rψ(X),ψ,β,bX:=ψ,εψ,β,b(X)X. _ψ,β,b(X):=2(1-β)m_d(m_d^2-1)R_ψ(X), _ψ,β,b^X:=U_ψ, _ψ,β,b(X)^X. (280) D.1 Near-margin directions and near-minimum-norm representatives Lemma D.1 (Near-margin reduction). Fix a linearly separable dataset X, a matrix norm ψ, and 0≤ε<γψ(X)0≤ < _ψ(X). For every U∈ψ,εXU _ψ, ^X, define VU:=UmarX(U),ξψ(ε,X):=εγψ(X)−ε.V_U:= Umar_X(U), _ψ( ;X):= _ψ(X)- . (281) Then VU∈ℱXV_U _X, ψ(VU)≤(1+ξψ(ε,X))rψ∗(X),ψ(V_U)≤ (1+ _ψ( ;X) )r_ψ^*(X), (282) and VUV_U has the same normalized pairwise row differences as U. Proof. Set U^:=U/ψ(U) U:=U/ψ(U). By definition of ψ,εXU_ψ, ^X, marX(U^)≥γψ(X)−ε>0mar_X( U)≥ _ψ(X)- >0. Hence VU=U^/marX(U^)V_U= U/mar_X( U) is feasible and ψ(VU)=1marX(U^)≤1γψ(X)−ε=rψ∗(X)1−ε/γψ(X)=(1+ξψ(ε,X))rψ∗(X).ψ(V_U)= 1mar_X( U)≤ 1 _ψ(X)- = r_ψ^*(X)1- / _ψ(X)= (1+ _ψ( ;X) )r_ψ^*(X). The normalized pairwise row differences are unchanged under positive scaling. ∎ D.2 Quantitative stability of near minimizers Lemma D.2 (Combined Gaussian-cloud event). There are deterministic ωd↓0 _d 0, constants C,c>0C,c>0, and events ℰdnearE_d^near such that ℙ((ℰdnear)c)≤Cexp[−cdlog(2N(d)(d+1))],P\! ((E_d^near)^c )≤ C \! [-c d (2N(d)(d+1)) ], (283) all conclusions of Propositions C.4 and C.12 hold with their vanishing errors bounded by ωd _d, and maxi,s‖xi,s‖2≤1+2r0+ωd,maxi,s‖xi,s‖1≤(1+2r0+ωd)d. _i,s\|x_i,s\|_2≤ 1+2r_0+ _d, _i,s\|x_i,s\|_1≤(1+2r_0+ _d) d. (284) We may additionally take ωd≥1/k _d≥ 1/k. Proof. Intersect the events in Propositions C.4 and C.12 with the event maxi,s‖zi,s‖2≤2 _i,s\|z_i,s\|_2≤ 2. The latter fails with probability at most 2Ne−cd2Ne^-cd, which is smaller than the right-hand side of (283). Under the normalized convention (77), the mean shell gives maxi‖μi‖2≤1+ωd _i\| _i\|_2≤ 1+ _d, so ‖xi,s‖2≤1+ωd+2r0\|x_i,s\|_2≤ 1+ _d+2r_0, and the ℓ1 _1 bound follows from ‖x‖1≤d‖x‖2\|x\|_1≤ d\,\|x\|_2. Enlarge one deterministic envelope to dominate all finitely many errors and 1/k1/k. ∎ Proposition D.3 (Near-minimizer stability). There are constants Crow,Cmax>0C_row,C_ >0 and Csp(α)>0C_sp(α)>0 such that, on ℰdnearE_d^near, for every 0≤ξ≤10≤ξ≤ 1, W∈ℱX,‖W‖row≤(1+ξ)rrow∗(X) W _X,\ \|W\|_row≤(1+ξ)r_row^*(X) ⟹maxi≠j‖uijW−uijμ‖2≤Crow(ωd+ξ)1/4, _i≠ j\|u_ij^W-u_ij^μ\|_2≤ C_row( _d+ξ)^1/4, (285) W∈ℱX,‖W‖op≤(1+ξ)rsp∗(X)⟹ W _X,\ \|W\|_op≤(1+ξ)r_sp^*(X) Davg-bdry(W)≥δsp(α)−Csp(α)(ωd+ξ)1/4, D_avg -bdry(W)≥ _sp(α)-C_sp(α)( _d+ξ)^1/4, (286) W∈ℱX,‖W‖max≤(1+ξ)rmax∗(X) W _X,\ \|W\|_ ≤(1+ξ)r_ ^*(X) ⟹Davg-bdry(W)≥δmax−Cmax(ωd+ξ). D_avg -bdry(W)≥ _ -C_ ( _d+ξ). (287) Proof. For the row norm, put q:=ωd+ξq:= _d+ξ. The empirical-mean witness in Proposition C.6 gives rrow∗(X)≤1+Cωdr_row^*(X)≤ 1+C _d, so maxi‖wi‖2≤1+Cq _i\|w_i\|_2≤ 1+Cq. Averaging feasibility within class i and then over all competing classes gives (wi−w¯)⊤mi≥1−1/k(w_i- w) m_i≥ 1-1/k. The row variance identity then gives ‖w¯‖22≤Cq\| w\|_2^2≤ Cq, and substituting this bound back into the averaged margin inequality yields maxi‖wi−mi‖2≤Cq1/4 _i\|w_i-m_i\|_2≤ Cq^1/4. The mean perturbation in Proposition C.4 and Lemma C.5 prove (285). For the spectral norm, Proposition C.15 gives an error Cαq1/2C_αq^1/2 when α≠1α≠ 1 and Cq1/4Cq^1/4 when α=1α=1. Since q1/2≤q1/4q^1/2≤ q^1/4 for q≤1q≤ 1, enlarging the constant proves (286) uniformly over all aspect ratios. For the max norm, fix i≠ji≠ j, set v:=wi−wjv:=w_i-w_j, and put u:=uijmu:=u_ij^m. Near-optimality and Proposition C.4(H5) give ‖v‖∞≤2π+Cqd.\|v\|_∞≤ 2π+Cq d. (288) The two class-averaged feasibility constraints imply v⊤(mi−mj)≥2v (m_i-m_j)≥ 2, and hence v⊤u≥2−Cqv u≥ 2-Cq. Applying the clipped minimum-energy characterization in Proposition C.8, the uniform empirical laws in Proposition C.4(H6), and Lemma C.9 gives v⊤u‖v‖2≤κmax+Cq. v u\|v\|_2≤ _ +Cq. (289) The cosine-distance identity and Proposition C.4(H4) prove (287). ∎ D.3 A dimension-independent momentum threshold Proposition D.4 (Uniform radius-to-margin ratios). On ℰdnearE_d^near, for every sufficiently large d, Rrow(X)γrow(X)≤Krow,Rsp(X)γsp(X)≤Ksp,Rmax(X)γmax(X)≤Kmax, R_row(X) _row(X)≤ K_row, R_sp(X) _sp(X)≤ K_sp, R_ (X) _ (X)≤ K_ , (290) where one may take Krow:=4(1+2r0),Ksp:=8(1+2r0)sα,Kmax:=(2π+1)(1+2r0).K_row:=4(1+2r_0), K_sp:= 8(1+2r_0)s_α, K_ :=( 2π+1)(1+2r_0). (291) Proof. Each ratio equals Rψ(X)rψ∗(X)R_ψ(X)r_ψ^*(X). For every sufficiently large d, Proposition C.6 gives rrow∗(X)≤2r_row^*(X)≤ 2, the reduced-polar witness (212) and ρX→sα>0 _X→ s_α>0 give rsp∗(X)≤4/sαr_sp^*(X)≤ 4/s_α, and Proposition C.4(H5) gives rmax∗(X)≤(2π+1)/(2d)r_ ^*(X)≤( 2π+1)/(2 d). Equation (284) and ωd→0 _d→ 0 imply, for sufficiently large d, Rrow(X)=Rsp(X)≤2(1+2r0),Rmax(X)≤2(1+2r0)d.R_row(X)=R_sp(X)≤ 2(1+2r_0), R_ (X)≤ 2(1+2r_0) d. Combining these estimates proves (290). ∎ Let δ∗:=minδsp(α),δmax>0. _*:= \ _sp(α), _ \>0. (292) Choose h∗>0h_*>0 sufficiently small that 12−h∗>12(1−(δ∗/2)22)+. 1 2-h_*> 1 2 (1- ( _*/2)^22 )_+. (293) Choose b∗∈(0,1]b_*∈(0,1] sufficiently small that Crowb∗1/4≤h∗,Csp(α)b∗1/4≤δ∗8,Cmaxb∗≤δ∗8.C_rowb_*^1/4≤ h_*, C_sp(α)b_*^1/4≤ _*8, C_ b_*≤ _*8. (294) Set θ∗:=b∗/21+b∗/2,A0:=m0(m02−1),K∗:=maxKrow,Ksp,Kmax, _*:= b_*/21+b_*/2, A_0:=m_0(m_0^2-1), K_*:= \K_row,K_sp,K_ \, (295) and define β0:=12,A0=0,max12,1−θ∗2A0K∗,A0>0. _0:= cases 12,&A_0=0,\\[4.0pt] \ 12,1- _*2A_0K_* \,&A_0>0. cases (296) All constants in (292)–(296) depend only on (α,r0,m0)(α,r_0,m_0). Proposition D.5 (Uniform stochastic near-margin geometry). Under the data and batching assumptions of Proposition 3.4, fix β∈[β0,1)β∈[ _0,1). Then, for every sufficiently large d, on ℰdnearE_d^near, all three effective margins γψ(X)−εψ,β,b(X) _ψ(X)- _ψ,β,b(X) are positive and supU∈row,β,bXmaxi≠j‖uijU−uijμ‖2 _U _row,β,b^X _i≠ j\|u_ij^U-u_ij^μ\|_2 ≤h∗, ≤ h_*, (297) infV∈sp,β,bXDavg-bdry(V) _V _sp,β,b^XD_avg -bdry(V) ≥δ∗2, ≥ _*2, (298) infV∈max,β,bXDavg-bdry(V) _V _ ,β,b^XD_avg -bdry(V) ≥δ∗2. ≥ _*2. (299) Moreover, after increasing the deterministic dimension threshold if necessary, ℙ(ℰdnear)≥1−e−d.P(E_d^near)≥ 1-e^- d. (300) Proof. For each geometry, define the relative stochastic deficit Δψ,d(β):=εψ,β,b(X)γψ(X)=2(1−β)md(md2−1)Rψ(X)γψ(X). _ψ,d(β):= _ψ,β,b(X) _ψ(X)=2(1-β)m_d(m_d^2-1) R_ψ(X) _ψ(X). (301) Since md≤m0m_d≤ m_0, Proposition D.4 gives Δψ,d(β)≤2(1−β)A0K∗≤θ∗<1. _ψ,d(β)≤ 2(1-β)A_0K_*≤ _*<1. Thus every effective margin is positive. By the definition of θ∗ _*, Δψ,d(β)1−Δψ,d(β)≤b∗2. _ψ,d(β)1- _ψ,d(β)≤ b_*2. (302) For every U∈ψ,β,bXU _ψ,β,b^X, Lemma D.1 therefore gives a feasible representative with identical normalized pairwise row differences and relative norm slack at most b∗/2b_*/2. For every sufficiently large d, ωd≤b∗/2 _d≤ b_*/2, so Proposition D.3 gives supU∈row,β,bXmaxi≠j‖uijU−uijμ‖2≤Crowb∗1/4≤h∗, _U _row,β,b^X _i≠ j\|u_ij^U-u_ij^μ\|_2≤ C_rowb_*^1/4≤ h_*, infV∈sp,β,bXDavg-bdry(V)≥δsp(α)−Csp(α)b∗1/4≥δ∗−δ∗8≥δ∗2, _V _sp,β,b^XD_avg -bdry(V)≥ _sp(α)-C_sp(α)b_*^1/4≥ _*- _*8≥ _*2, and infV∈max,β,bXDavg-bdry(V)≥δmax−Cmaxb∗≥δ∗−δ∗8≥δ∗2. _V _ ,β,b^XD_avg -bdry(V)≥ _ -C_ b_*≥ _*- _*8≥ _*2. This proves (297)–(299). Finally, log(2N(d)(d+1))→∞ (2N(d)(d+1))→∞, so the failure probability in (283) is at most e−de^- d for every sufficiently large d. ∎ D.4 Proof of the stochastic optimizer accuracy ordering Proof of Proposition 3.4. Fix β∈[β0,1)β∈[ _0,1), and let ℛdR_d be the Gaussian radial event from Lemma C.18. Set ℋd:=ℰdnear∩ℛd.H_d:=E_d^near _d. By (283) and (224), after increasing the deterministic dimension threshold, ℙ(ℋd)≥1−e−d.P(H_d)≥ 1-e^- d. (303) Work on ℋdH_d. Define the Row family ℛdclf:=row,β,bXR_d^clf:=U_row,β,b^X and, for ψ∈sp,maxψ∈\sp, \, the competitor family ψ,dclf:=ψ,β,bX.C_ψ,d^clf:=U_ψ,β,b^X. Proposition D.5 gives supU∈ℛdclfmaxi≠j‖uijU−uijμ‖2≤h∗,infV∈ψ,dclfDavg-bdry(V)≥δ∗2. _U _d^clf _i≠ j\|u_ij^U-u_ij^μ\|_2≤ h_*, _V _ψ,d^clfD_avg -bdry(V)≥ _*2. (304) The choice of h∗h_* in (293) gives the strict limiting amplitude gap (h∗)>(δ∗/2), a(h_*)> b( _*/2), (305) where a and b are defined in (229). Choose fixed constants 0<b<a0<b<a such that (δ∗/2)<b<a<(h∗). b( _*/2)<b<a< a(h_*). (306) On ℛdR_d, the radial deviation and coherence in (223) converge to zero under a deterministic envelope. Therefore, for every sufficiently large d, the finite-dimensional amplitudes in (226)–(227) satisfy Ad≥a,Bd≤bA_d≥ a, B_d≤ b for the Row family and each of the two competitor families. Proposition C.19 consequently gives supU∈ℛdclfℰπ,d(U)≤(k−1)Φ¯(aσd),infV∈ψ,dclfℰπ,d(V)≥2πmin,dΦ¯(bσd). _U _d^clfE_π,d(U)≤(k-1) \! ( a _d ), _V _ψ,d^clfE_π,d(V)≥ 2 _ ,d \! ( b _d ). (307) For x≥y≥0x≥ y≥ 0, the Gaussian upper tail satisfies Φ¯(x)/Φ¯(y)≤exp(−(x2−y2)/2) (x)/ (y)≤ (-(x^2-y^2)/2). Hence k=O(d)k=O(d), rte≤rte,0r_te≤ r_te,0, and log(1/πmin,d)=o(d) (1/ _ ,d)=o(d) imply supU∈ℛdclfℰπ,d(U)infV∈ψ,dclfℰπ,d(V)≤Ckπmin,de−cd⟶0. _U _d^clfE_π,d(U) _V _ψ,d^clfE_π,d(V)≤ Ck _ ,de^-cd 0. (308) Hence, for every sufficiently large d on ℋdH_d, supU∈row,β,bXℰπ,d(U)<mininfV∈ψ,β,bXψ∈sp,maxℰπ,d(V). _U _row,β,b^XE_π,d(U)< _ψ∈\sp, \ _V _ψ,β,b^XE_π,d(V). (309) It remains to transfer the set ordering to the optimizer trajectories. Proposition D.5 ensures that all effective margins are positive. Corollary B.12 therefore gives dcos(WtRR-RowSGDM,row,β,bX) d_ \! (W_t^R -RowSGDM,U_row,β,b^X ) ⟶0, 0, dcos(WtRR-Muon,sp,β,bX) d_ \! (W_t^R -Muon,U_sp,β,b^X ) ⟶0, 0, dcos(WtRR-Signum,max,β,bX) d_ \! (W_t^R -Signum,U_ ,β,b^X ) ⟶0. 0. (310) Applying Lemma C.21 to (310) and then using (309) gives, for each opt∈R-Muon,R-Signumopt∈\R -Muon,R -Signum\, lim supt→∞ℰπ,d(WtRR-RowSGDM) _t→∞E_π,d\! (W_t^R -RowSGDM ) ≤supU∈row,β,bXℰπ,d(U) ≤ _U _row,β,b^XE_π,d(U) <infV∈ψopt,β,bXℰπ,d(V) < _V _ _opt,β,b^XE_π,d(V) ≤lim inft→∞ℰπ,d(Wtopt), ≤ _t→∞E_π,d\! (W_t^opt ), (311) where ψMuon=ψsp _Muon= _sp and ψSignum=ψmax _Signum= _ . Since Accπ,d=1−ℰπ,dAcc_π,d=1-E_π,d, lim inft→∞Accπ,d(WtRR-RowSGDM)>maxlim supt→∞opt∈R-Muon,R-SignumAccπ,d(Wtopt). _t→∞Acc_π,d\! (W_t^R -RowSGDM )> _opt∈\R -Muon,R -Signum\ _t→∞Acc_π,d\! (W_t^opt ). Since HdH_d and the dimension threshold above are independent of β∈[β0,1)β∈[ _0,1), and the random-reshuffling margin bounds hold deterministically for every reshuffling realization, the conclusion holds simultaneously for all such β and every realization of the reshuffling. Together with (303), this proves (10). ∎ Appendix E Analysis of AdamProxy for Random-reshuffling Mini-batch Adam This appendix proves Proposition 3.5, Proposition 3.6, and Proposition E.10. We first show that β1,β2→1 _1, _2→ 1 random-reshuffling Adam is approximated over one epoch by the deterministic AdamProxy direction. We then prove a trajectory-level coordinate bound for AdamProxy and use it to establish a coordinate-induced angular gap, which, together with the near-row geometry from Appendix D and the geometry-to-accuracy transfer in Appendix C, yields the R-RowSGDM–AdamProxy population-accuracy ordering. Separately, we characterize the limiting AdamProxy classifier through a self-consistent semidefinite quadratic margin problem when the normalized proxy direction converges. E.1 One-epoch β1,β2→1 _1, _2→ 1 R Adam approximation Throughout this subsection, probability is taken only over the independent random reshufflings while the training data are fixed. We assume 1≤b<N1≤ b<N, use the gradient convention of the main text, namely gn(W)=−(eyn−pn(W))xn⊤g_n(W)=-(e_y_n-p_n(W))x_n , and use the standard Adam initialization M0=V0=M_0=V_0=0. For a uniformly random b-subset B⊂[N]B⊂[N], recall GB(W)=b−1∑n∈Bgn(W)G_B(W)=b^-1 _n∈ Bg_n(W), and set S(W):=∑n=1Ngn(W),Q(W):=∑n=1Ngn(W)⊙2.S(W):= _n=1^Ng_n(W), Q(W):= _n=1^Ng_n(W) 2. (312) Lemma E.1 (Finite-population batch moments and relative scale). Assume 1≤b<N1≤ b<N. For z=(z1,…,zN)∈ℝNz=(z_1,…,z_N) ^N, define TBz:=b−1∑n∈BznT_Bz:=b^-1 _n∈ Bz_n and σ(z)2:=B[(TBz)2]σ(z)^2:=E_B[(T_Bz)^2]. Then B[TBz]=1N∑n=1Nzn,E_B[T_Bz]= 1N _n=1^Nz_n, (313) σ(z)2=b−1bN(N−1)(∑n=1Nzn)2+N−bN(N−1)∑n=1Nzn2.σ(z)^2= b-1bN(N-1) ( _n=1^Nz_n )^2+ N-bbN(N-1) _n=1^Nz_n^2. (314) Consequently, with μ1(W):=B[GB(W)] _1(W):=E_B[G_B(W)] and μ2(W):=B[GB(W)⊙2] _2(W):=E_B[G_B(W) 2], μ1(W)=1NS(W),μ2(W)=(b−1)S(W)⊙2+(N−b)Q(W)bN(N−1), _1(W)= 1NS(W), _2(W)= (b-1)S(W) 2+(N-b)Q(W)bN(N-1), (315) and therefore μ1(W)⊘μ2(W)⊙1/2=ΦN,b(W). _1(W) _2(W) 1/2= _N,b(W). (316) Moreover, because 1≤b<N1≤ b<N, σ is a norm and c¯N,b‖z‖2≤σ(z)≤1b‖z‖2,c¯N,b:=N−bN(N−1), c_N,b\|z\|_2≤σ(z)≤ 1 b\|z\|_2, c_N,b:= N-bbN(N-1), (317) while every b-subset satisfies |TBz|≤KN,bσ(z),1b∑n∈B|zn|≤KN,bσ(z),KN,b:=N(N−1)N−b.|T_Bz|≤ K_N,bσ(z), 1b _n∈ B|z_n|≤ K_N,bσ(z), K_N,b:= N(N-1)N-b. (318) For a matrix coordinate ρ=(r,a)ρ=(r,a), define znρ(W):=gn(W)[ρ]z_n^ρ(W):=g_n(W)[ρ] and σρ(W):=σ(zρ(W)) _ρ(W):=σ(z^ρ(W)). Under the nonzero-coordinate assumption in Proposition 3.5, σρ(W)>0 _ρ(W)>0 for every finite W and every ρ. Proof. Let In:=n∈BI_n:=1_\n∈ B\. Since B is a uniform b-subset, In=b/NEI_n=b/N and [InIn′]=b(b−1)/(N(N−1))E[I_nI_n ]=b(b-1)/(N(N-1)) for n≠n′n≠ n . Expanding [(TBz)2]E[(T_Bz)^2] gives (314), while (313) is immediate. Applying the two identities coordinate-wise to gn(W)g_n(W) gives (315), and substituting these expressions into μ1(W)⊘μ2(W)⊙1/2 _1(W) _2(W) 1/2 gives (316) by (12). The second term in (314) gives the lower bound in (317), while |TBz|≤b−1∑n∈B|zn|≤b−1/2∥z∥2|T_Bz|≤ b^-1 _n∈ B|z_n|≤ b^-1/2\|z\|_2 gives the upper bound and (318). Finally, at finite W, every softmax probability is strictly positive. If r=ynr=y_n, then gn(W)[r,a]=−(1−pn,yn(W))xn[a]≠0g_n(W)[r,a]=-(1-p_n,y_n(W))x_n[a]≠ 0, while if r≠ynr≠ y_n, then gn(W)[r,a]=pn,r(W)xn[a]≠0g_n(W)[r,a]=p_n,r(W)x_n[a]≠ 0. Hence zρ(W)≠0z^ρ(W)≠ 0, so σρ(W)>0 _ρ(W)>0. ∎ Lemma E.2 (Adam-step and signal stability). Let 0<β1≤β2<10< _1≤ _2<1 and define A(β1,β2)2:=(1−β1)2(1−β2)(1−β12/β2).A( _1, _2)^2:= (1- _1)^2(1- _2)(1- _1^2/ _2). (319) Then β12<β2 _1^2< _2, and for Ut:=Mt+1⊘Vt+1⊙1/2U_t:=M_t+1 V_t+1 1/2, ‖Ut‖max≤A(β1,β2),‖Wt+1−Wt‖F≤kdA(β1,β2)ηt.\|U_t\|_ ≤ A( _1, _2), \|W_t+1-W_t\|_F≤ kd\,A( _1, _2) _t. (320) There are finite data-dependent constants L,CL,C such that, for all U,WU,W, every batch B, and every coordinate ρ, |GB(U)[ρ]−GB(W)[ρ]|σρ(W)≤C(eL‖U−W‖F−1), |G_B(U)[ρ]-G_B(W)[ρ]| _ρ(W)≤ C (e^L\|U-W\|_F-1 ), (321) |GB(U)[ρ]2−GB(W)[ρ]2|σρ(W)2≤C(e2L‖U−W‖F−1). |G_B(U)[ρ]^2-G_B(W)[ρ]^2| _ρ(W)^2≤ C (e^2L\|U-W\|_F-1 ). (322) Proof. Fix one coordinate and write Yu:=GBu(Wu)[ρ]Y_u:=G_B_u(W_u)[ρ]. Zero initialization gives Mt+1=(1−β1)∑j=0tβ1jYt−jM_t+1=(1- _1) _j=0^t _1^jY_t-j and Vt+1=(1−β2)∑j=0tβ2jYt−j2V_t+1=(1- _2) _j=0^t _2^jY_t-j^2. If Vt+1=0V_t+1=0, then Mt+1=0M_t+1=0; otherwise Cauchy–Schwarz gives Mt+12≤A(β1,β2)2Vt+1M_t+1^2≤ A( _1, _2)^2V_t+1, proving (320). For the stability bounds, let Δ:=U−W :=U-W. The logit perturbations satisfy |ec⊤Δxn|≤‖Δ‖F‖xn‖2|e_c x_n|≤\| \|_F\|x_n\|_2, so every softmax probability ratio, and also the ratio of 1−pn,yn1-p_n,y_n, lies between e−L‖Δ‖Fe^-L\| \|_F and eL‖Δ‖Fe^L\| \|_F after enlarging L. Each coordinate of gng_n is a fixed feature coordinate multiplied by one of these quantities, giving |gn(U)[ρ]−gn(W)[ρ]|≤(eL‖Δ‖F−1)|gn(W)[ρ]||g_n(U)[ρ]-g_n(W)[ρ]|≤(e^L\| \|_F-1)|g_n(W)[ρ]|. Summing over n∈Bn∈ B and applying (318) proves (321); the identity |a2−b2|≤|a−b|(|a|+|b|)|a^2-b^2|≤|a-b|(|a|+|b|) then gives (322). ∎ For the next lemma only, extend the reshuffling process to all integer epochs by taking independent uniform ordered partitions for every epoch index in ℤZ. For z≠z≠0, define Yt(z):=TBtz/σ(z)Y_t(z):=T_B_tz/σ(z). Lemma E.3 (Uniform concentration of frozen exponential averages). For every δ>0δ>0, there exists ωδ:(0,1)→[0,1] _δ:(0,1)→[0,1] with ωδ(β)→0 _δ(β)→ 0 as β↑1β 1 such that, for every epoch start τ=rmτ=rm, ℙ(maxsupz≠0≤s<m|(1−β)∑j=0∞βj(Yτ+s−j(z)−Y0(z))|>δ)≤ωδ(β).P ( _0≤ s<m _z≠0 |(1-β) _j=0^∞β^j (Y_τ+s-j(z)-EY_0(z) ) |>δ )≤ _δ(β). (323) The same statement holds with Yt(z)2Y_t(z)^2 in place of Yt(z)Y_t(z) and [Y0(z)2]=1E[Y_0(z)^2]=1. For fixed N,bN,b, one may take ωδ(β)≤Cδ(1−β) _δ(β)≤ C_δ(1-β). Proof. It is enough to work on the compact σ-unit sphere. Group the lags in the exponential average by reshuffling epoch. Contributions from distinct epochs are independent, every epoch block has mean zero, and Lemma E.1 uniformly bounds |Yt(z)||Y_t(z)|. Hence the variance of the centered exponential average is at most C(1−β)2∑q≥0β2qm≤C′(1−β)C(1-β)^2 _q≥ 0β^2qm≤ C (1-β). Chebyshev’s inequality gives the stated rate for each fixed z. The maps z↦TBz T_Bz are uniformly Lipschitz in the σ-norm, so a finite δ-net and a union bound make the estimate uniform over z and over s=0,…,m−1s=0,…,m-1. The argument for Yt(z)2Y_t(z)^2 is identical because Yt(z)Y_t(z) is uniformly bounded. ∎ For fixed W, define the two-sided frozen memories M¯tβ1(W):=(1−β1)∑j=0∞β1jGBt−j(W),V¯tβ2(W):=(1−β2)∑j=0∞β2jGBt−j(W)⊙2. M_t _1(W):=(1- _1) _j=0^∞ _1^jG_B_t-j(W), V_t _2(W):=(1- _2) _j=0^∞ _2^jG_B_t-j(W) 2. (324) Their expectations are μ1(W) _1(W) and μ2(W) _2(W), respectively. Lemma E.4 (Freezing one epoch). Fix 0<β1≤β2<10< _1≤ _2<1. Under the assumptions of Proposition 3.5, for every coordinate ρ, uniformly over 0≤s<m0≤ s<m, |Mτr+s+1[ρ]−M¯τr+sβ1(Wτr)[ρ]|σρ(Wτr)→r→∞ℙ0, |M_ _r+s+1[ρ]- M_ _r+s _1(W_ _r)[ρ]| _ρ(W_ _r) [r→∞]P0, (325) |Vτr+s+1[ρ]−V¯τr+sβ2(Wτr)[ρ]|σρ(Wτr)2→r→∞ℙ0. |V_ _r+s+1[ρ]- V_ _r+s _2(W_ _r)[ρ]| _ρ(W_ _r)^2 [r→∞]P0. (326) The convergence is uniform over the finitely many coordinates. Proof. Fix t=τr+st= _r+s. The first-moment difference consists of the missing negative-time tail and movement errors obtained by replacing Wt−jW_t-j by WτrW_ _r. Lemma E.1 bounds the normalized missing tail by Cβ1t+1C _1^t+1. For 0≤j≤s0≤ j≤ s, Lemma E.2 and monotonicity of ηt _t give ‖Wt−j−Wτr‖F≤Cβ1,β2mητr→0\|W_t-j-W_ _r\|_F≤ C_ _1, _2m _ _r→ 0. For j>sj>s, writing q=j−sq=j-s gives ‖Wτr−q−Wτr‖F≤Cβ1,β2∑u=τr−qτr−1ηu\|W_ _r-q-W_ _r\|_F≤ C_ _1, _2 _u= _r-q _r-1 _u. Therefore (321) bounds the normalized past-lag contribution by C(1−β1)∑q=1τrβ1q+s(exp(C∑u=τr−qτr−1ηu)−1),C(1- _1) _q=1 _r _1^q+s ( (C _u= _r-q _r-1 _u )-1 ), which tends to zero by Assumption 2.6. This proves (325). The second-moment argument is identical using (322) and β2 _2, proving (326). ∎ Lemma E.5 (Continuity of the Adam ratio). There is a universal Crat<∞C_rat<∞ such that, whenever σ>0σ>0, |a0|≤σ|a_0|≤σ, |a−a0|≤δσ|a-a_0|≤δσ, and |v−σ2|≤δσ2|v-σ^2|≤δσ^2 with 0<δ≤1/20<δ≤ 1/2, |av−a0σ|≤Cratδ. | a v- a_0σ |≤ C_ratδ. (327) Proof. Since v≥σ2/2v≥σ^2/2, |av−a0σ|≤|a−a0|v+|a0||1v−1σ|≤2δ+|(1+e)−1/2−1|, | a v- a_0σ |≤ |a-a_0| v+|a_0| | 1 v- 1σ |≤ 2\,δ+ |(1+e)^-1/2-1 |, where v=σ2(1+e)v=σ^2(1+e) and |e|≤δ|e|≤δ. The map x↦(1+x)−1/2x (1+x)^-1/2 is Lipschitz on [−1/2,1/2][-1/2,1/2]. ∎ Proof of Proposition 3.5. Let fN,b(W):=μ1(W)⊘μ2(W)⊙1/2=ΦN,b(W)f_N,b(W):= _1(W) _2(W) 1/2= _N,b(W) by Lemma E.1. Because the Adam update uses the gradient convention, Wτr+m−Wτrγr=−∑s=0m−1ητr+sγrUτr+s,∑s=0m−1ητr+sγr=1. W_ _r+m-W_ _r _r=- _s=0^m-1 _ _r+s _rU_ _r+s, _s=0^m-1 _ _r+s _r=1. (328) Fix 0<δ≤1/40<δ≤ 1/4. Lemma E.3 gives, with probability at least 1−ωδ(β1)−ωδ(β2)1- _δ( _1)- _δ( _2), relative δ-control of both frozen memories around μ1(Wτr) _1(W_ _r) and μ2(Wτr) _2(W_ _r), simultaneously over the epoch and all coordinates. Lemma E.4 transfers these bounds to the actual memories with 2δ2δ in place of δ, up to an event whose probability vanishes as r→∞r→∞. Jensen’s inequality gives |μ1(W)[ρ]|≤μ2(W)[ρ]=σρ(W)| _1(W)[ρ]|≤ _2(W)[ρ]= _ρ(W), so Lemma E.5 yields max0≤s<m‖Uτr+s−ΦN,b(Wτr)‖max≤Cδ _0≤ s<m\|U_ _r+s- _N,b(W_ _r)\|_ ≤ Cδ (329) with the same asymptotic probability. Combining (328) and (329) gives ‖Wτr+m−Wτrγr+ΦN,b(Wτr)‖max≤Cδ. \| W_ _r+m-W_ _r _r+ _N,b(W_ _r) \|_ ≤ Cδ. (330) Given ε>0 >0, choose δ so that Cδ<εCδ< . Taking r→∞r→∞ first yields a limsup probability bounded by ωδ(β1)+ωδ(β2) _δ( _1)+ _δ( _2), and then taking (β1,β2)→(1,1)( _1, _2)→(1,1) through 0<β1≤β2<10< _1≤ _2<1 proves (13). ∎ E.2 R-RowSGDM versus AdamProxy population accuracy We prove Proposition 3.6 by bounding the coordinates of the margin-normalized AdamProxy iterates and comparing the resulting classifier family with the near-row-margin family. All training-time limits are taken at fixed dimension before the limit d→∞d→∞. A deterministic trajectory bound. Fix a finite linearly separable dataset X=(xn,yn)n=1NX=\(x_n,y_n)\_n=1^N, with xn∈ℝdx_n ^d and yn∈[k]y_n∈[k]. Write pn(W)=(Wxn)p_n(W)=S(Wx_n), ℓn(W)=−logpn,yn(W) _n(W)=- p_n,y_n(W), and L(W)=N−1∑n=1Nℓn(W)L(W)=N^-1 _n=1^N _n(W). For :=(n,c):n∈[N],c≠ynQ:=\(n,c):n∈[N],\ c≠ y_n\, set An,c:=(eyn−ec)xn⊤A_n,c:=(e_y_n-e_c)x_n , so that marX(W)=min(n,c)∈⟨W,An,c⟩,γmax(X)=max‖V‖max≤1marX(V)>0.mar_X(W)= _(n,c) W,A_n,c , _ (X)= _\|V\|_ ≤ 1mar_X(V)>0. Let R1(X):=maxn‖xn‖1R_1(X):= _n\|x_n\|_1 and rmax∗(X)=γmax(X)−1r_ ^*(X)= _ (X)^-1. For κ≥0κ≥ 0, define hn(W) h_n(W) :=(eyn−pn(W))xn⊤=∑c≠ynpn,c(W)An,c, :=(e_y_n-p_n(W))x_n = _c≠ y_np_n,c(W)A_n,c, S~(W) S(W) :=∑n=1Nhn(W),Q~(W):=∑n=1Nhn(W)⊙2, := _n=1^Nh_n(W), Q(W):= _n=1^Nh_n(W) 2, Ψκ(W) _κ(W) :=S~(W)⊘(S~(W)⊙2+κQ~(W))⊙1/2. := S(W) ( S(W) 2+κ Q(W) ) 1/2. (331) All powers and divisions are entrywise, with 0/0:=00/0:=0. Proposition E.6 (Positive margin growth and a trajectory-level coordinate bound). Let κ≥0κ≥ 0, let W0∈ℝk×dW_0 ^k× d be finite, and consider Wt+1=Wt+atΨκ(Wt)W_t+1=W_t+a_t _κ(W_t), where at>0a_t>0, at→0a_t→ 0, and ∑t=0∞at=∞ _t=0^∞a_t=∞. Define Tt:=∑s=0t−1as,Kκ(X):=2(1+κ)R1(X),ρκ(X):=γmax(X)2Kκ(X)>0.T_t:= _s=0^t-1a_s, K_κ(X):=2(1+ κ)R_1(X), _κ(X):= _ (X)^2K_κ(X)>0. Then L(Wt)→0L(W_t)→ 0, marX(Wt)→∞mar_X(W_t)→∞, and lim inft→∞marX(Wt)Tt _t→∞ mar_X(W_t)T_t ≥ρκ(X), ≥ _κ(X), (332) lim inft→∞marX(Wt)‖Wt‖max _t→∞ mar_X(W_t)\|W_t\|_ ≥ρκ(X), ≥ _κ(X), (333) lim supt→∞‖WtmarX(Wt)‖max _t→∞ \| W_tmar_X(W_t) \|_ ≤Kκ(X)rmax∗(X)2. ≤ K_κ(X)r_ ^*(X)^2. (334) The margin-normalized matrices in (334) are well defined and have minimum training margin one for all sufficiently large t. Proof. For finite W, define (W):=1N∑n=1N(1−pn,yn(W)),R(W):=∑(n,c)∈pn,c(W)=N(W)>0,G(W):= 1N _n=1^N(1-p_n,y_n(W)), R(W):= _(n,c) p_n,c(W)=NG(W)>0, and let θn,c(W):=pn,c(W)/R(W) _n,c(W):=p_n,c(W)/R(W). Then θ(W)θ(W) belongs to the probability simplex Δ() (Q). For any θ∈Δ()θ∈ (Q), write Zn(θ) Z_n(θ) :=∑c≠ynθn,cAn,c,(θ):=∑n=1NZn(θ), := _c≠ y_n _n,cA_n,c, S(θ):= _n=1^NZ_n(θ), κ(θ) D_κ(θ) :=((θ)⊙2+κ∑n=1NZn(θ)⊙2)⊙1/2. := ( S(θ) 2+κ _n=1^NZ_n(θ) 2 ) 1/2. Since hn(W)=R(W)Zn(θ(W))h_n(W)=R(W)Z_n(θ(W)), the common factor R(W)R(W) cancels in the proxy ratio, giving Ψκ(W)=(θ(W))⊘κ(θ(W)),‖Ψκ(W)‖max≤1. _κ(W)= S(θ(W)) D_κ(θ(W)), \| _κ(W)\|_ ≤ 1. (335) The coordinate bound follows from κ(θ)[ρ]≥|(θ)[ρ]| D_κ(θ)[ρ]≥| S(θ)[ρ]|, including zero denominators under the stated convention. The entrywise inequalities |(θ)|≤∑n|Zn(θ)|| S(θ)|≤ _n|Z_n(θ)| and (∑nZn(θ)⊙2)⊙1/2≤∑n|Zn(θ)|( _nZ_n(θ) 2) 1/2≤ _n|Z_n(θ)| imply ‖κ(θ)‖sum≤(1+κ)∑n‖Zn(θ)‖sum≤2(1+κ)R1(X)=Kκ(X).\| D_κ(θ)\|_sum≤(1+ κ) _n\|Z_n(θ)\|_sum≤ 2(1+ κ)R_1(X)=K_κ(X). (336) Here the last inequality uses ‖An,c‖sum=2‖xn‖1\|A_n,c\|_sum=2\|x_n\|_1 and ∑(n,c)∈θn,c=1 _(n,c) _n,c=1. Let V∗V_* attain γmax(X) _ (X) on the compact max-norm unit ball. Max–sum duality and the margin constraints for V∗V_* give ‖(θ)‖sum≥⟨V∗,(θ)⟩=∑(n,c)∈θn,c⟨V∗,An,c⟩≥γmax(X).\| S(θ)\|_sum≥ V_*, S(θ) = _(n,c) _n,c V_*,A_n,c ≥ _ (X). If κ(θ)[ρ]=0 D_κ(θ)[ρ]=0, then (θ)[ρ]=0 S(θ)[ρ]=0. Weighted Cauchy–Schwarz on the remaining coordinates therefore yields κ(θ):=∑ρ:κ(θ)[ρ]>0(θ)[ρ]2κ(θ)[ρ]≥‖(θ)‖sum2‖κ(θ)‖sum≥ρκ(X).J_κ(θ):= _ρ: D_κ(θ)[ρ]>0 S(θ)[ρ]^2 D_κ(θ)[ρ]≥ \| S(θ)\|_sum^2\| D_κ(θ)\|_sum≥ _κ(X). (337) Since ∇L(W)=−(W)(θ(W))∇ L(W)=-G(W) S(θ(W)), it follows that −⟨∇L(W),Ψκ(W)⟩=(W)κ(θ(W))≥ρκ(X)(W).- ∇ L(W), _κ(W) =G(W)J_κ(θ(W))≥ _κ(X)G(W). (338) To control the discrete-time remainder, fix ‖Δ‖max≤1\| \|_ ≤ 1. Then ‖Δxn‖∞≤R1(X)\| x_n\|_∞≤ R_1(X), and the softmax Hessian satisfies D2ℓn(V)[Δ,Δ]=VarC∼pn(V)((Δxn)C)≤4R1(X)2(1−pn,yn(V)).D^2 _n(V)[ , ]=Var_C p_n(V) (( x_n)_C )≤ 4R_1(X)^2 (1-p_n,y_n(V) ). Indeed, the variance is at most the second moment about (Δxn)yn( x_n)_y_n, and every logit difference has magnitude at most 2R1(X)2R_1(X). The softmax formula also gives (W+saΔ)≤e2R1(X)a(W)G(W+sa )≤ e^2R_1(X)aG(W) for a≥0a≥ 0 and s∈[0,1]s∈[0,1]. Because at→0a_t→ 0, the quantity amax:=suptata_ := _ta_t is finite. Taylor’s formula, (335), and (338) consequently imply L(Wt+1)≤L(Wt)−at(ρκ(X)−CXat)(Wt),CX:=2R1(X)2e2R1(X)amax.L(W_t+1)≤ L(W_t)-a_t ( _κ(X)-C_Xa_t )G(W_t), C_X:=2R_1(X)^2e^2R_1(X)a_ . (339) For all sufficiently large t, the coefficient in parentheses is at least ρκ(X)/2 _κ(X)/2, so L(Wt)L(W_t) is eventually nonincreasing and has a nonnegative limit. If this limit were L∞>0L_∞>0, then at each sufficiently large t some sample would satisfy ℓn(Wt)≥L(Wt)≥L∞ _n(W_t)≥ L(W_t)≥ L_∞, and hence (Wt)≥(1−e−L∞)/N>0G(W_t)≥(1-e^-L_∞)/N>0. Summing (339) would contradict L≥0L≥ 0 and ∑tat=∞ _ta_t=∞. Thus L(Wt)→0L(W_t)→ 0. Since 0≤ℓn(Wt)≤NL(Wt)0≤ _n(W_t)≤ NL(W_t), every sample loss tends to zero, and (Wt)L(Wt)=∑n(1−e−ℓn(Wt))∑nℓn(Wt)⟶1. G(W_t)L(W_t)= _n(1-e^- _n(W_t)) _n _n(W_t) 1. Moreover, ℓn(W)=log(1+∑c≠yne−⟨W,An,c⟩) _n(W)= (1+ _c≠ y_ne^- W,A_n,c ) shows that every training margin tends to infinity. For all sufficiently large t, dividing (339) by L(Wt)L(W_t) and applying −log(1−z)≥z- (1-z)≥ z gives −logL(Wt+1)+logL(Wt)≥at(ρκ(X)−CXat)(Wt)L(Wt).- L(W_t+1)+ L(W_t)≥ a_t ( _κ(X)-C_Xa_t ) G(W_t)L(W_t). The factor multiplying ata_t converges to ρκ(X) _κ(X), so summation and weighted Cesàro averaging yield lim inft→∞−logL(Wt)Tt≥ρκ(X). _t→∞ - L(W_t)T_t≥ _κ(X). Write mt:=marX(Wt)m_t:=mar_X(W_t). Once mt≥0m_t≥ 0, a sample attaining the minimum margin gives L(Wt)≥1Nlog(1+e−mt)≥e−mt2N,mt≥−logL(Wt)−log(2N).L(W_t)≥ 1N (1+e^-m_t)≥ e^-m_t2N, m_t≥- L(W_t)- (2N). This proves (332). Finally, (335) gives ‖Wt‖max≤‖W0‖max+Tt\|W_t\|_ ≤\|W_0\|_ +T_t. Combining this bound with (332) proves (333) and (334). ∎ For 2≤b≤N2≤ b≤ N, the explicit AdamProxy formula gives −ΦN,b(W)=cN,bAPΨκN,b(W),cN,bAP:=b(N−1)N(b−1),κN,b:=N−b−1.- _N,b(W)=c_N,b^AP _ _N,b(W), c_N,b^AP:= b(N-1)N(b-1), _N,b:= N-bb-1. Thus (14) has the recursion in Proposition E.6 with at=cN,bAPγta_t=c_N,b^AP _t and γt=∑s=0m−1ηmt+s _t= _s=0^m-1 _mt+s. At each fixed dimension, Assumption 2.5 implies at→0a_t→ 0 and ∑tat=∞ _ta_t=∞. The full-batch case b=Nb=N corresponds to κN,b=0 _N,b=0, cN,bAP=1c_N,b^AP=1, and Ψ0=sign(S~) _0=sign( S). Gaussian-cloud normalization and coordinate caps. We use the normalized convention (77), under which μi,zi,s∼(,Id/d) _i,z_i,s (0,I_d/d) and Xte|(Y=i)∼(μi,rte2Id/d)X^te (Y=i) ( _i,r_te^2I_d/d). This convention also preserves the relevant trajectory estimates. Indeed, if Xraw=dX^raw= d\,X and V=dWV= d\,W, then every normalized-data sample gradient at V equals d−1/2d^-1/2 times the corresponding raw-data gradient at W, so Ψκ,X(V)=Ψκ,Xraw(W) _κ,X(V)= _κ,X^raw(W). Consequently, Vt=dWtV_t= d\,W_t follows the normalized-data recursion with step sizes dat d\,a_t and identical predictions. For R-RowSGDM, scaling its momentum by d−1/2d^-1/2 gives the same row-normalized update direction, so its transformed parameter trajectory uses step sizes dηt d\, _t. Multiplication by this fixed positive constant preserves Assumptions 2.5 and 2.6 at each fixed d. We suppress the rescaling in the notation below. Under the assumptions of Proposition 3.6, bd≥N(d)/m0→∞b_d≥ N(d)/m_0→∞, so bd≥2b_d≥ 2 for all sufficiently large d. Moreover, 0≤κN,bd=bd(md−1)bd−1≤2(m0−1)=:κ¯.0≤ _N,b_d= b_d(m_d-1)b_d-1≤ 2(m_0-1)=: κ. (340) Set R∗:=2(1+2r0),CAP:=πR∗(1+κ¯),aAP:=max3,2CAP+1,R_*:=2(1+2r_0), C_AP:=π R_*(1+ κ), a_AP:= \3,2C_AP+1\, (341) and define the capped feasible family d:=V∈ℱX:‖V‖max≤CAP+1/2d.C_d:= \V _X:\|V\|_ ≤ C_AP+1/2 d \. (342) For the functions AmaxA_ and BmaxB_ in (78), and for a≥3a≥ 3, let τa∈(0,1) _a∈(0,1) be the unique solution of Amax(τa)=2a.A_ ( _a)= 2a. (343) Such a solution exists because Amax(0)=0A_ (0)=0, AmaxA_ is strictly increasing, and Amax(1)>1/π>2/aA_ (1)>1/ π> 2/a. Define Cclip(a):=2aBmax(τa),δclip(a):=2−2Cclip(a),δAP:=δclip(aAP).C_clip(a):= 2a B_ ( _a), _clip(a):= 2-2C_clip(a), _AP:= _clip(a_AP). (344) Strict Cauchy–Schwarz gives Amax(τ)2<Bmax(τ)A_ (τ)^2<B_ (τ) for every finite τ>0τ>0, so 0<Cclip(a)<10<C_clip(a)<1 and δAP>0 _AP>0. Lemma E.7 (Uniform clipped-Gaussian angular barrier). Let ud,ξ∈d−1u_d,ξ ^d-1 be indexed by a nonempty finite set ℐdI_d, and suppose the empirical functions in (93) satisfy supξ∈ℐdsupτ∈[0,1](|Hud,ξ(τ)−Amax(τ)|+|Jud,ξ(τ)−Bmax(τ)|)⟶0. _ξ _d _τ∈[0,1] ( |H_u_d,ξ(τ)-A_ (τ) |+ |J_u_d,ξ(τ)-B_ (τ) | ) 0. Fix a≥3a≥ 3, let ad→a_d→ a and υd→2 _d→ 2 be positive deterministic sequences, and define d,ξ:=v∈ℝd:∥v∥∞≤add,v⊤ud,ξ≥υd.V_d,ξ:= \v ^d:\|v\|_∞≤ a_d d, v u_d,ξ≥ _d \. For all sufficiently large d, these sets are nonempty and infξ∈ℐdinfv∈d,ξ‖v‖v‖2−ud,ξ‖2≥δclip(a)−od(1). _ξ _d _v _d,ξ \| v\|v\|_2-u_d,ξ \|_2≥ _clip(a)-o_d(1). (345) The error depends only on the uniform empirical errors and the deviations of ada_d and υd _d from their limits. Proof. Since υd/ad→2/a<Amax(1) _d/a_d→ 2/a<A_ (1), uniform convergence gives 0<υd/ad<Hud,ξ(1)0< _d/a_d<H_u_d,ξ(1) for every ξ and all sufficiently large d. The feasibility condition in Proposition C.8 therefore holds, and that proposition gives d,ξ:=minv∈d,ξ‖v‖2=adJud,ξ(τ^d,ξ),Hud,ξ(τ^d,ξ)=υdad, m_d,ξ:= _v _d,ξ\|v\|_2=a_d J_u_d,ξ( τ_d,ξ), H_u_d,ξ( τ_d,ξ)= _da_d, with τ^d,ξ∈(0,1) τ_d,ξ∈(0,1). Uniform convergence and strict monotonicity of AmaxA_ imply supξ∈ℐd|τ^d,ξ−τa|⟶0,supξ∈ℐd|d,ξ−aBmax(τa)|⟶0. _ξ _d| τ_d,ξ- _a| 0, _ξ _d | m_d,ξ-a B_ ( _a) | 0. For every v∈d,ξv _d,ξ, the vector y=(υd/(v⊤ud,ξ))vy=( _d/(v u_d,ξ))v also belongs to d,ξV_d,ξ. Hence v⊤ud,ξ‖v‖2≤υd,ξ=Cclip(a)+od(1), v u_d,ξ\|v\|_2≤ _d m_d,ξ=C_clip(a)+o_d(1), where the error is independent of v and uniform over ξ. The identity ‖v/‖v‖2−u‖22=2−2v⊤u/‖v‖2\|v/\|v\|_2-u\|_2^2=2-2v u/\|v\|_2 proves (345). ∎ Proposition E.8 (Capped AdamProxy trajectories and boundary separation). For every sufficiently large d, on the event ℰdnearE_d^near from Lemma D.2, the AdamProxy trajectory satisfies WtAPmarX(WtAP)∈dfor all sufficiently large t, W_t^APmar_X(W_t^AP) _d all sufficiently large t, (346) and infV∈dmini≠j‖uijV−uijμ‖2≥δAP−od(1). _V _d _i≠ j\|u_ij^V-u_ij^μ\|_2≥ _AP-o_d(1). (347) The error is deterministic on ℰdnearE_d^near and uniform over V∈dV _d. The time threshold in (346) may depend on the dimension, dataset, initialization, and learning-rate schedule. Proof. The event guarantees separability, and Proposition C.4(H5) and (284) give rmax∗(X)≤2π+od(1)2d,maxn‖xn‖2≤R∗,R1(X)≤dR∗.r_ ^*(X)≤ 2π+o_d(1)2 d, _n\|x_n\|_2≤ R_*, R_1(X)≤ d\,R_*. Applying Proposition E.6 and (340), we obtain lim supt→∞d‖WtAPmarX(WtAP)‖max _t→∞ d \| W_t^APmar_X(W_t^AP) \|_ ≤2(1+κ¯)dR∗rmax∗(X)2 ≤ 2(1+ κ)dR_*r_ ^*(X)^2 ≤CAP+od(1). ≤ C_AP+o_d(1). For all sufficiently large d, the right-hand side is less than CAP+1/4C_AP+1/4. The definition of the time limsup and the unit margin of the normalized iterates then give (346). In particular, dC_d is nonempty. Fix V∈dV _d and an ordered pair i≠ji≠ j, and write v:=vi−vjv:=v_i-v_j and u:=uijmu:=u_ij^m. The coordinate cap gives ‖v‖∞≤2CAP+1d≤aAPd.\|v\|_∞≤ 2C_AP+1 d≤ a_AP d. Averaging feasibility over the samples from classes i and j gives v⊤mi≥1v m_i≥ 1 and v⊤mj≤−1v m_j≤-1, and hence v⊤u≥2‖mi−mj‖2≥2−od(1).v u≥ 2\|m_i-m_j\|_2≥ 2-o_d(1). The last error is deterministic and uniform over all pairs and all V∈dV _d, by Proposition C.4(H2). Choose a positive deterministic sequence υd→2 _d→ 2 below these projection lower bounds. Proposition C.4(H6) supplies the empirical estimates in Lemma E.7, which applies with ad=aAPa_d=a_AP and ℐd=(i,j):i≠jI_d=\(i,j):i≠ j\. Thus ‖uijV−uijm‖2≥δAP−od(1)\|u_ij^V-u_ij^m\|_2≥ _AP-o_d(1) uniformly over all pairs and all V∈dV _d. Finally, Proposition C.4(H4) and the reverse triangle inequality prove (347). ∎ High-momentum row geometry. Let CrowC_row be the constant in Proposition D.3, and let KrowK_row be the row constant in Proposition D.4. Choose hAP:=δAP2/(322)h_AP:= _AP^2/(32 2), so that (hAP)>(δAP/2), a(h_AP)> b( _AP/2), (348) where a and b are defined in (229). Set bAP:=min1,(hAPCrow)4,ϑAP:=bAP/21+bAP/2,A0:=m0(m02−1),b_AP:= \1, ( h_APC_row )^4 \, _AP:= b_AP/21+b_AP/2, A_0:=m_0(m_0^2-1), (349) and βAP:=12,A0=0,max12,1−ϑAP2A0Krow,A0>0. _AP:= cases 12,&A_0=0,\\[3.0pt] \ 12,1- _AP2A_0K_row \,&A_0>0. cases (350) These constants depend only on (α,r0,m0)(α,r_0,m_0), and βAP∈(0,1) _AP∈(0,1). Lemma E.9 (High-momentum Row near-set control). For every fixed β∈[βAP,1)β∈[ _AP,1), on ℰdnearE_d^near and for every sufficiently large d, the R-RowSGDM effective margin is positive and supU∈row,β,bdXmaxi≠j‖uijU−uijμ‖2≤hAP. _U _row,β,b_d^X _i≠ j\|u_ij^U-u_ij^μ\|_2≤ h_AP. (351) Proof. The relative margin deficit is Δrow,d:=εrow,β,bd(X)γrow(X)=2(1−β)md(md2−1)Rrow(X)γrow(X). _row,d:= _row,β,b_d(X) _row(X)=2(1-β)m_d(m_d^2-1) R_row(X) _row(X). Proposition D.4 and (350) give Δrow,d≤2(1−β)A0Krow≤ϑAP<1 _row,d≤ 2(1-β)A_0K_row≤ _AP<1, including Δrow,d=0 _row,d=0 when md=1m_d=1. Thus the effective margin is positive, and Δrow,d1−Δrow,d≤bAP2. _row,d1- _row,d≤ b_AP2. Lemma D.1 maps every U∈row,β,bdXU _row,β,b_d^X to a unit-margin feasible representative with the same pairwise row directions and relative row-norm slack at most bAP/2b_AP/2. For all sufficiently large d, ωd≤bAP/2 _d≤ b_AP/2, so Proposition D.3 gives the uniform bound CrowbAP1/4≤hAPC_rowb_AP^1/4≤ h_AP. ∎ Proof of Proposition 3.6. Set β1:=βAP _1:= _AP, fix β∈[β1,1)β∈[ _1,1), and let ℛdR_d be the Gaussian radial event from Lemma C.18. Define ℋd:=ℰdnear∩ℛdH_d:=E_d^near _d. By (283) and (224), for every sufficiently large d, ℙ(ℋd)≥1−e−d.P(H_d)≥ 1-e^- d. Work on ℋdH_d, and take ℛdclf:=row,β,bdX,dclf:=d.R_d^clf:=U_row,β,b_d^X, _d^clf:=C_d. Lemma E.9 and Proposition E.8 give, for all sufficiently large d, supU∈ℛdclfmaxi≠j‖uijU−uijμ‖2≤hAP,infV∈dDavg-bdry(V)≥δAP2. _U _d^clf _i≠ j\|u_ij^U-u_ij^μ\|_2≤ h_AP, _V _dD_avg -bdry(V)≥ _AP2. The radial deviations tend to zero on ℛdR_d under a deterministic envelope, and the limiting amplitude condition is (348). Choose fixed constants 0<b∗<a∗0<b_*<a_* with (δAP/2)<b∗<a∗<(hAP) b( _AP/2)<b_*<a_*< a(h_AP). Proposition C.19 gives, for σd=rte/d _d=r_te/ d and all sufficiently large d, supU∈row,β,bdXℰπ,d(U) _U _row,β,b_d^XE_π,d(U) ≤(k−1)Φ¯(a∗/σd), ≤(k-1) (a_*/ _d), infV∈dℰπ,d(V) _V _dE_π,d(V) ≥2πmin,dΦ¯(b∗/σd). ≥ 2 _ ,d (b_*/ _d). The Gaussian upper-tail inequality Φ¯(x)/Φ¯(y)≤e−(x2−y2)/2 (x)/ (y)≤ e^-(x^2-y^2)/2 for x≥y≥0x≥ y≥ 0 therefore yields supU∈row,β,bdXℰπ,d(U)infV∈dℰπ,d(V)≤k−12πmin,dexp(−(a∗2−b∗2)d2rte,02)⟶0, _U _row,β,b_d^XE_π,d(U) _V _dE_π,d(V)≤ k-12 _ ,d (- (a_*^2-b_*^2)d2r_te,0^2 ) 0, since k=O(d)k=O(d), log(1/πmin,d)=o(d) (1/ _ ,d)=o(d), and rte,0:=supdrte(d)<∞r_te,0:= _dr_te(d)<∞. Consequently, for every sufficiently large d on ℋdH_d, supU∈row,β,bdXℰπ,d(U)<infV∈dℰπ,d(V). _U _row,β,b_d^XE_π,d(U)< _V _dE_π,d(V). (352) The positive Row effective margin and Corollary B.12 give dcos(WtRR-RowSGDM,row,β,bdX)⟶0.d_ \! (W_t^R -RowSGDM,U_row,β,b_d^X ) 0. Lemma C.21 hence implies lim supt→∞ℰπ,d(WtRR-RowSGDM)≤supU∈row,β,bdXℰπ,d(U). _t→∞E_π,d\! (W_t^R -RowSGDM )≤ _U _row,β,b_d^XE_π,d(U). For AdamProxy, (346) and positive scale invariance of population error give lim inft→∞ℰπ,d(WtAP)≥infV∈dℰπ,d(V). _t→∞E_π,d(W_t^AP)≥ _V _dE_π,d(V). Combining these two bounds with (352) yields lim supt→∞ℰπ,d(WtRR-RowSGDM)<lim inft→∞ℰπ,d(WtAP). _t→∞E_π,d\! (W_t^R -RowSGDM )< _t→∞E_π,d(W_t^AP). Since Accπ,d=1−ℰπ,dAcc_π,d=1-E_π,d, this is the claimed accuracy ordering. The Row momentum estimates hold for every reshuffling realization, and the AdamProxy estimates are deterministic conditional on the training data, so no additional exceptional event is required. ∎ E.3 AdamProxy margin characterization AdamProxy is not steepest descent in a fixed norm, so we characterize its limit through a data-dependent quadratic margin problem. Let :=(n,c):n∈[N],c≠ynQ:=\(n,c):n∈[N],\,c≠ y_n\, let Δ() (Q) be the probability simplex, and for θ∈Δ()θ∈ (Q) define Zn(θ) Z_n(θ) :=∑c≠ynθn,c(eyn−ec)xn⊤, := _c≠ y_n _n,c(e_y_n-e_c)x_n , MN,b(θ) M_N,b(θ) :=diag(vec[(Nb(N−1)((b−1)(∑n=1NZn(θ))⊙2+(N−b)∑n=1NZn(θ)⊙2))⊙1/2]). :=diag\! (vec\! [ ( Nb(N-1) ((b-1) ( _n=1^NZ_n(θ) ) 2+(N-b) _n=1^NZ_n(θ) 2 ) ) 1/2 ] ). (353) Extending the parametric margin formulation of Baek et al. (2026), let N,b(θ):=argminW12vec(W)⊤MN,b(θ)vec(W)s.t.(wyn−wc)⊤xn≥1,(n,c)∈.P_N,b(θ):= *arg\,min_W 12vec(W) M_N,b(θ)vec(W) .t. (w_y_n-w_c) x_n≥ 1,\ \ (n,c) . (354) Proposition E.10 (AdamProxy margin characterization). Under Assumption 2.5, let WtW_t follow (14) with 1≤b≤N1≤ b≤ N. Assume linear separability, xn[a]≠0x_n[a]≠ 0 for all n,an,a, and ΦN,b(Wt)/‖ΦN,b(Wt)‖F→Φ¯ _N,b(W_t)/\| _N,b(W_t)\|_F→ . Then Wt/‖Wt‖F→W¯:=−Φ¯W_t/\|W_t\|_F→ W:=- and γAP:=marX(W¯)>0 _AP:=mar_X( W)>0. For p⋆:=W¯/γAPp := W/ _AP, there exist θ⋆∈Δ()θ ∈ (Q) and a nonzero dual optimizer λ⋆≥0λ ≥ 0 of the corresponding problem such that p⋆∈N,b(θ⋆),θ⋆=λ⋆/‖λ⋆‖1.p _N,b(θ ), θ =λ /\|λ \|_1. Thus, the AdamProxy margin direction is characterized by N,b(θ⋆)P_N,b(θ ) at a self-consistent parameter θ⋆θ , which coincides with the normalized dual weights of the same variational problem. Proof setup. The remainder of this subsection proves Proposition E.10. We first rewrite the AdamProxy dynamics in terms of descent signals and normalized wrong-class coefficients, then characterize the trajectory-selected limiting coefficients, and finally verify the primal–dual optimality conditions of the quadratic margin problem. Define the descent signals hn(W):=−gn(W)=(eyn−pn(W))xn⊤,h_n(W):=-g_n(W)=(e_y_n-p_n(W))x_n , (355) and write An,c:=(eyn−ec)xn⊤A_n,c:=(e_y_n-e_c)x_n for (n,c)∈(n,c) . Then hn(W)=∑c≠ynpn,c(W)An,ch_n(W)= _c≠ y_np_n,c(W)A_n,c. Set S~(W):=∑n=1Nhn(W)=−S(W),Q~(W):=∑n=1Nhn(W)⊙2=Q(W). S(W):= _n=1^Nh_n(W)=-S(W), Q(W):= _n=1^Nh_n(W) 2=Q(W). (356) For θ∈Δ()θ∈ (Q), write (θ) S(θ) :=∑n=1NZn(θ),(θ):=∑n=1NZn(θ)⊙2, := _n=1^NZ_n(θ), Q(θ):= _n=1^NZ_n(θ) 2, (357) N,b(θ) D_N,b(θ) :=(Nb(N−1)((b−1)(θ)⊙2+(N−b)(θ)))⊙1/2. := ( Nb(N-1) ((b-1) S(θ) 2+(N-b) Q(θ) ) ) 1/2. Thus MN,b(θ)=diag(vecN,b(θ))M_N,b(θ)=diag(vec D_N,b(θ)). Define ΨN,b(W):=S~(W)⊘(Nb(N−1)((b−1)S~(W)⊙2+(N−b)Q~(W)))⊙1/2, _N,b(W):= S(W) ( Nb(N-1) ((b-1) S(W) 2+(N-b) Q(W) ) ) 1/2, (358) with 0/0:=00/0:=0 entrywise. The explicit proxy formula in the main text gives ΨN,b(W)=−ΦN,b(W) _N,b(W)=- _N,b(W), so the AdamProxy dynamics can be written as Wt+1AP=WtAP+γtΨN,b(WtAP)W_t+1^AP=W_t^AP+ _t _N,b(W_t^AP). Lemma E.11 (Coefficient representation and trajectory-selected limit). For finite W, define R(W):=∑(n,c)∈pn,c(W)>0,θn,c(W):=pn,c(W)R(W).R(W):= _(n,c) p_n,c(W)>0, _n,c(W):= p_n,c(W)R(W). (359) Then hn(W)=R(W)Zn(θ(W)),S~(W)=R(W)(θ(W)),Q~(W)=R(W)2(θ(W)),h_n(W)=R(W)Z_n(θ(W)), S(W)=R(W) S(θ(W)), Q(W)=R(W)^2 Q(θ(W)), (360) ΨN,b(W)=(θ(W))⊘N,b(θ(W)),‖ΨN,b(W)‖max≤1. _N,b(W)= S(θ(W)) D_N,b(θ(W)), \| _N,b(W)\|_ ≤ 1. (361) Under the assumptions of Proposition E.10, WtAP‖WtAP‖F⟶W¯:=−Φ¯, W_t^AP\|W_t^AP\|_F W:=- , (362) and there exist a subsequence tj→∞t_j→∞, θ⋆∈Δ()θ ∈ (Q), and ν∈(0,∞)ν∈(0,∞) such that θ(WtjAP)→θ⋆,‖ΨN,b(WtjAP)‖F→ν,ΨN,b(WtjAP)→νW¯,θ(W_t_j^AP)→θ , \| _N,b(W_t_j^AP)\|_F→ν, _N,b(W_t_j^AP)→ν W, (363) with (θ⋆)≠ S(θ )≠0 and ν‖W¯‖max≤1ν\| W\|_ ≤ 1. Proof. The factorization follows directly from hn(W)=∑c≠ynpn,c(W)An,ch_n(W)= _c≠ y_np_n,c(W)A_n,c. For every coordinate ρ, Cauchy–Schwarz gives (θ)[ρ]2≤N(θ)[ρ] S(θ)[ρ]^2≤ N Q(θ)[ρ], and therefore N,b(θ)[ρ]2−(θ)[ρ]2=N−b(N−1)(N(θ)[ρ]−(θ)[ρ]2)≥0. D_N,b(θ)[ρ]^2- S(θ)[ρ]^2= N-bb(N-1) (N Q(θ)[ρ]- S(θ)[ρ]^2 )≥ 0. Hence N,b(θ)[ρ]≥|(θ)[ρ]| D_N,b(θ)[ρ]≥| S(θ)[ρ]|, and if N,b(θ)[ρ]=0 D_N,b(θ)[ρ]=0, then (θ)[ρ]=0 S(θ)[ρ]=0. With the convention 0/0:=00/0:=0, this proves (361) and the coordinate bound. By linear separability, choose V and σsep>0 _sep>0 with ⟨V,Aq⟩≥σsep V,A_q ≥ _sep for every q. Since N,b D_N,b is continuous on the compact simplex, Dmax:=maxθ‖N,b(θ)‖max<∞D_ := _θ\| D_N,b(θ)\|_ <∞, and σsep≤⟨V,(θ(W))⟩≤‖V‖FDmax‖ΨN,b(W)‖F. _sep≤ V, S(θ(W)) ≤\|V\|_FD_ \| _N,b(W)\|_F. Thus ‖ΨN,b(Wt)‖F\| _N,b(W_t)\|_F is bounded above and below by positive constants. Since ΨN,b(W)=−ΦN,b(W) _N,b(W)=- _N,b(W), the normalized descent-proxy directions satisfy ΨN,b(Wt)/‖ΨN,b(Wt)‖F→−Φ¯ _N,b(W_t)/\| _N,b(W_t)\|_F→- . Assumption 2.5 implies that the epochwise schedule γt _t is positive, tends to zero, and satisfies ∑tγt=∞ _t _t=∞. With Tt:=∑s=0t−1γs‖ΨN,b(Ws)‖FT_t:= _s=0^t-1 _s\| _N,b(W_s)\|_F, one has Tt→∞T_t→∞ and WtTt=W0Tt+1Tt∑s=0t−1γs‖ΨN,b(Ws)‖FΨN,b(Ws)‖ΨN,b(Ws)‖F⟶−Φ¯. W_tT_t= W_0T_t+ 1T_t _s=0^t-1 _s\| _N,b(W_s)\|_F _N,b(W_s)\| _N,b(W_s)\|_F - . Hence ‖Wt‖F/Tt→1\|W_t\|_F/T_t→ 1 and Wt/‖Wt‖F→−Φ¯W_t/\|W_t\|_F→- , proving (362). Compactness of Δ() (Q) and the two-sided bound on ‖ΨN,b(Wt)‖F\| _N,b(W_t)\|_F give a subsequence satisfying (363). The separator inequality passes to the limit and gives (θ⋆)≠ S(θ )≠0, while ‖ΨN,b(Wtj)‖max≤1\| _N,b(W_t_j)\|_ ≤ 1 yields ν‖W¯‖max≤1ν\| W\|_ ≤ 1. ∎ Proposition E.12 (Selected stationarity and positive margin). For the limit in Lemma E.11, there exists α=ν−1>0α=ν^-1>0 such that MN,b(θ⋆)vec(W¯)=αvec((θ⋆))=α∑q∈θq⋆vec(Aq).M_N,b(θ )vec( W)= ( S(θ ))=α _q _q vec(A_q). (364) Moreover, γAP:=minq∈⟨W¯,Aq⟩>0,supp(θ⋆)⊆argminq∈⟨W¯,Aq⟩. _AP:= _q W,A_q >0, (θ ) *arg\,min_q W,A_q . (365) Proof. Let ⋆:=(θ⋆) S := S(θ ) and ⋆:=N,b(θ⋆) D := D_N,b(θ ). If ⋆[ρ]>0 D [ρ]>0, continuity and (363) give ⋆[ρ]/⋆[ρ]=νW¯[ρ] S [ρ]/ D [ρ]=ν W[ρ]. If ⋆[ρ]=0 D [ρ]=0, then the inequality N,b(θ)[ρ]≥|(θ)[ρ]| D_N,b(θ)[ρ]≥| S(θ)[ρ]| proved above gives ⋆[ρ]=0 S [ρ]=0. Hence (364) holds on every coordinate. It also gives ⟨W¯,(θ⋆)⟩=ν∑ρ⋆[ρ]W¯[ρ]2>0. W, S(θ ) =ν _ρ D [ρ] W[ρ]^2>0. (366) To identify the support, write m¯n,c:=⟨W¯,An,c⟩ m_n,c:= W,A_n,c , μn:=minc≠ynm¯n,c _n:= _c≠ y_n m_n,c, and hn,c:=m¯n,c−min0,μnh_n,c:= m_n,c- \0, _n\. Since ‖Wt‖F→∞\|W_t\|_F→∞ and Wt/‖Wt‖F→W¯W_t/\|W_t\|_F→ W, finite log-sum-exp asymptotics give −1‖Wt‖Flogpn,c(Wt)⟶hn,c.- 1\|W_t\|_F p_n,c(W_t) h_n,c. Therefore every subsequential limit θ⋆θ is supported on the atoms minimizing hn,ch_n,c. If some μn≤0 _n≤ 0, then the minimum of h is zero and every minimizing atom has m¯n,c≤0 m_n,c≤ 0, contradicting (366). Thus every μn>0 _n>0, so hn,c=m¯n,ch_n,c= m_n,c, proving both claims in (365). ∎ For fixed θ∈Δ()θ∈ (Q), let aq:=vec(Aq)a_q:=vec(A_q) and define ℒθ(u,λ):=12u⊤MN,b(θ)u+∑q∈λq(1−aq⊤u),gθ(λ):=infu∈ℝkdℒθ(u,λ),λ≥0.L_θ(u,λ):= 12u M_N,b(θ)u+ _q _q(1-a_q u), g_θ(λ):= _u ^kdL_θ(u,λ), λ≥ 0. (367) Lemma E.13 (Semidefinite primal-dual certificate). Fix θ∈Δ()θ∈ (Q). Suppose U is feasible for (354) and λ≥0λ≥ 0 satisfies MN,b(θ)vec(U)=∑q∈λqvec(Aq),λq(⟨U,Aq⟩−1)=0(q∈).M_N,b(θ)vec(U)= _q _qvec(A_q), _q( U,A_q -1)=0 (q ). (368) Then U∈N,b(θ)U _N,b(θ), and λ is a dual optimizer. No invertibility of MN,b(θ)M_N,b(θ) is required. Proof. Write u:=vec(U)u:=vec(U), M:=MN,b(θ)M:=M_N,b(θ), and let V be any feasible matrix with v:=vec(V)v:=vec(V). Since M⪰0M 0, 12v⊤Mv−12u⊤Mu=12(v−u)⊤M(v−u)+∑qλq⟨V−U,Aq⟩≥0, 12v Mv- 12u Mu= 12(v-u) M(v-u)+ _q _q V-U,A_q ≥ 0, where feasibility and complementarity are used in the last step. Thus U is primal optimal. Stationarity makes u a global minimizer of the Lagrangian for this λ, and complementarity makes the dual value equal the primal value, so λ is dual optimal by weak duality. ∎ Proof of Proposition E.10. Set p⋆:=W¯/γAPp := W/ _AP, where W¯=−Φ¯ W=- and γAP>0 _AP>0 are given by Lemma E.11 and Proposition E.12. Let α be the scalar in (364) and define λq⋆:=αγAPθq⋆. _q := α _AP _q . (369) Then p⋆p is feasible because ⟨p⋆,Aq⟩≥1 p ,A_q ≥ 1, and dividing (364) by γAP _AP gives the stationarity equation in Lemma E.13. If λq⋆>0 _q >0, then θq⋆>0 _q >0, so (365) gives ⟨W¯,Aq⟩=γAP W,A_q = _AP and hence complementarity. Lemma E.13 therefore yields p⋆∈N,b(θ⋆)p _N,b(θ ) and dual optimality of λ⋆λ . Finally, ‖λ⋆‖1=αγAP∑qθq⋆=αγAP>0,\|λ \|_1= α _AP _q _q = α _AP>0, so θ⋆=λ⋆/‖λ⋆‖1θ =λ /\|λ \|_1. This proves the proposition. ∎ Appendix F Extension to Anisotropic Gaussian with Independent Orientations This appendix proves Theorem 4.2. The proof uses clean training in an essential way: the optimizer-selected limiting directions depend only on the class means and are therefore independent of the Haar eigenbases of the test-noise covariances. We first characterize the geometry of these selected Row, Spectral, and Max directions and then compare their Gaussian test errors after exposing the independent test-noise orientations. Normalization and Gaussian error transfer. Apply Lemma C.1 with the common factor d−1/2d^-1/2, and suppress the rescaling in the notation. Thus throughout this appendix μi=1dΓ1/2gi,gi∼iid(,Id),Xte∣(Y=i)∼(μi,rte2dΣi), _i= 1 d ^1/2g_i, g_i iid N(0,I_d), X^te (Y=i) \! ( _i, r_te^2d _i ), (370) and the clean training set is X=(μi,i)i=1kX=\( _i,i)\_i=1^k. This rescaling changes neither classifier predictions nor margin-direction sets. Set rte,0:=supdrte(d)<∞r_te,0:= _dr_te(d)<∞. For any classifier with wi≠wjw_i≠ w_j, define uijW:=wi−wj‖wi−wj‖2,Zi→j(W):=d(uijW)⊤μi(uijW)⊤ΣiuijW.u_ij^W:= w_i-w_j\|w_i-w_j\|_2, Z_i→ j(W):= d\,(u_ij^W) _i (u_ij^W) _iu_ij^W. (371) Lemma F.1 (Gaussian error transfer). Let ℛdR_d and dC_d be nonempty classifier families. Suppose every W∈ℛdW _d satisfies Zi→j(W)≥Ad>0Z_i→ j(W)≥ A_d>0 for all i≠ji≠ j, while every V∈dV _d has a directed pair i≠ji≠ j satisfying Zi→j(V)≤BdZ_i→ j(V)≤ B_d, where 0≤Bd<Ad0≤ B_d<A_d. Then supW∈ℛdErrπ,d(W)infV∈dErrπ,d(V)≤k−1πmin,dexp(−Ad2−Bd22rte,02). _W _dErr_π,d(W) _V _dErr_π,d(V)≤ k-1 _ ,d \! (- A_d^2-B_d^22r_te,0^2 ). (372) Consequently, if Ad2→∞A_d^2→∞, lim supdBd2/Ad2<1 _dB_d^2/A_d^2<1, and log((k−1)/πmin,d)=o(Ad2) ((k-1)/ _ ,d)=o(A_d^2), then every classifier in ℛdR_d has strictly larger population accuracy than every classifier in dC_d for all sufficiently large d. Proof. For a directed pair i≠ji≠ j, ℙ(wj⊤Xte≥wi⊤Xte∣Y=i)=Φ¯(Zi→j(W)rte),P\! (w_j X^te≥ w_i X^te Y=i )= \! ( Z_i→ j(W)r_te ), where Φ¯(x)=ℙ(G≥x) (x)=P(G≥ x) for G∼(0,1)G (0,1). A union bound over the k−1k-1 competitors gives Errπ,d(W)≤(k−1)Φ¯(Ad/rte)Err_π,d(W)≤(k-1) (A_d/r_te) for every W∈ℛdW _d. For V∈dV _d, the directed pair supplied by the assumption is contained in the multiclass error event, so Errπ,d(V)≥πmin,dΦ¯(Bd/rte)Err_π,d(V)≥ _ ,d (B_d/r_te). Since Φ¯(x)/Φ¯(y)≤exp(−(x2−y2)/2) (x)/ (y)≤ (-(x^2-y^2)/2) for x≥y≥0x≥ y≥ 0, (372) follows. The final assertion is immediate. ∎ For θ≥0θ≥ 0, write Tθ,d:=∑a=1da−θ.T_θ,d:= _a=1^da^-θ. (373) For every fixed θ≥0θ≥ 0, Tθ,d≍d1−θ,0≤θ<1,logd,θ=1,1,θ>1.T_θ,d casesd^1-θ,&0≤θ<1,\\ d,&θ=1,\\ 1,&θ>1. cases (374) In particular, for fixed 0≤sμ<10≤ s_μ<1, the trace-normalized mean covariance obeys csμId⪯Γ,‖Γ‖op≤Csμdsμ,tr(Γ2)≤Csμ(d+d2sμ)logd.c_s_μI_d , \| \|_op≤ C_s_μd^s_μ, ( ^2)≤ C_s_μ(d+d^2s_μ) d. (375) Optimizer-selected directions. Let ℱclF_cl be the sigma-field generated by the class means together with the prescribed optimizer initializations and tie-breaking variables. Because training uses only (μi,i)i=1k\( _i,i)\_i=1^k, ℱclF_cl is independent of O1,…,OkO_1,…,O_k. Lemma F.2 (Residual-exogenous optimizer limits). On the probability-one event on which the directional limits assumed in Theorem 4.2 exist, define Uopt:=limt→∞Wtopt‖Wtopt‖F,opt∈row∪comp.U_opt:= _t→∞ W_t^opt\|W_t^opt\|_F, _row _comp. Every UoptU_opt is ℱclF_cl-measurable and hence independent of O1,…,OkO_1,…,O_k. Moreover, Uopt∈rowX,opt∈row,spX,opt∈SpecGD,Muon,maxX,opt∈SignGD,Signum,Adam,U_opt∈ casesU_row^X,&opt _row,\\ U_sp^X,&opt∈\SpecGD,Muon\,\\ U_ ^X,&opt∈\SignGD,Signum,Adam\, cases and Accπ,d(Wtopt)⟶Accπ,d(Uopt).Acc_π,d\! (W_t^opt ) _π,d\! (U_opt ). (376) Proof. The empirical loss, every full-batch gradient, and every optimizer state are ℱclF_cl-measurable, so induction gives the same measurability for every iterate and hence for every normalized directional limit. The full-batch implicit-bias results used in Theorem 3.1, together with Lemma B.2, place the three optimizer groups in the corresponding maximum-margin direction sets. Finally, population accuracy is invariant under positive rescaling, each margin direction has nonzero pairwise row differences, and the positive-definite Gaussian test laws assign zero probability to score-tie hyperplanes. Dominated convergence therefore gives (376). ∎ Mean geometry of the selected directions. Let Mμ∈ℝk×dM^μ ^k× d have i-th row μi⊤ _i , let P:=Ik−k−1kk⊤P:=I_k-k^-11_k1_k , and set Cμ:=PMμC^μ:=PM^μ. Recall κsp(α)<1 _sp(α)<1 from (81), and define ϑsp(α):=κsp(α)2,ϑmax:=2π,ϑα:=maxϑsp(α),ϑmax<1. _sp(α):= _sp(α)^2, _ := 2π, _α:= \ _sp(α), _ \<1. (377) Lemma F.3 (Selected mean geometry). Fix 0≤sμ<10≤ s_μ<1 and any finite collection of ℱclF_cl-measurable Row, Spectral, and Max selectors. For every fixed 0<ξ<1−sμ0<ξ<1-s_μ, with probability at least 1−e−dξ1-e^-d^ξ for all sufficiently large d, the following hold. (i) Every selected Row direction satisfies mini≠j(uijUrow)⊤μi≥12−o(1). _i≠ j(u_ij^U_row) _i≥ 1 2-o(1). (378) (i) For every selected UψU_ψ, ψ∈sp,maxψ∈\sp, \, and every fixed ε>0 >0, there exist unit vectors a1ψ,…,akψa_1^ψ,…,a_k^ψ, a directed edge set dψG_d^ψ with |dψ|≥cεk(k−1)|G_d^ψ|≥ c_ k(k-1), and pijψ:=aiψ−ajψ‖aiψ−ajψ‖2p_ij^ψ:= a_i^ψ-a_j^ψ\|a_i^ψ-a_j^ψ\|_2 such that maxi≠j|(aiψ)⊤ajψ| _i≠ j|(a_i^ψ) a_j^ψ| =o(1/logd), =o(1/ d), (379) max(i,j)∈dψ‖uijUψ−pijψ‖2 _(i,j) _d^ψ\|u_ij^U_ψ-p_ij^ψ\|_2 ≤1logd, ≤ 1 d, (380) max(i,j)∈dψ[(uijUψ)⊤μi]+2 _(i,j) _d^ψ [(u_ij^U_ψ) _i ]_+^2 ≤ϑψ+ε2. ≤ _ψ+ 2. (381) The event can be chosen simultaneously for the entire finite selector collection. Proof. Write Ld:=logdL_d:= d. Weighted Gaussian quadratic and bilinear concentration using (375), followed by a union bound over O(k2)O(k^2) quantities, gives maxi|‖μi‖22−1|+maxi≠j|μi⊤μj|≤Ld−64 _i |\| _i\|_2^2-1 |+ _i≠ j| _i _j|≤ L_d^-64 (382) except with probability at most Cexp−cd1−sμ/LdCC \-cd^1-s_μ/L_d^C\. This failure probability is at most e−dξe^-d^ξ for every fixed ξ<1−sμξ<1-s_μ and sufficiently large d. For Row, apply Proposition C.6 to the clean samples with mi=μim_i= _i and ei,s=e_i,s=0. Equation (382) gives maxi‖wi−μi‖2≤CLd−16 _i\|w_i- _i\|_2≤ CL_d^-16, uniformly over the exact Row face. Since ‖μi−μj‖2=2+o(1)\| _i- _j\|_2= 2+o(1) uniformly over pairs, normalization stability yields uijW=(μi−μj)/‖μi−μj‖2+O(Ld−16)u_ij^W=( _i- _j)/\| _i- _j\|_2+O(L_d^-16), and (378) follows. For Spectral, write the reduced singular value decomposition Cμ=UΛV⊤,Q:=UV⊤,r:=mink−1,d,C^μ=U V , Q:=UV , r:= \k-1,d\, and let qi⊤q_i denote the i-th row of Q. After identifying k⟂1_k with ℝk−1R^k-1, the nonzero part of CμC^μ has the same law as GΓ1/2/dG ^1/2/ d, where G∈ℝ(k−1)×dG ^(k-1)× d has i.i.d. standard Gaussian entries. Its left singular frame is therefore Haar in k⟂1_k and independent of the singular values. Haar projector concentration gives, on the same probability scale, maxi|‖qi‖22−rk|+maxi≠j|qi⊤qj|≤Ld−16. _i |\|q_i\|_2^2- rk |+ _i≠ j|q_i q_j|≤ L_d^-16. (383) Moreover, since QCμ⊤=UΛU⊤QC^μ =U U , conditional Haar concentration and the bound ‖μ¯‖2=o(1)\| μ\|_2=o(1), where μ¯=k−1∑iμi μ=k^-1 _i _i, give maxi≠j|(qi−qj)⊤μi−‖Cμ‖∗k−1|≤Ld−32. _i≠ j |(q_i-q_j) _i- \|C^μ\|_*k-1 |≤ L_d^-32. (384) The nuclear-to-Frobenius ratio satisfies ‖Cμ‖∗r‖Cμ‖F≤κsp(α)+η \|C^μ\|_* r\,\|C^μ\|_F≤ _sp(α)+η (385) for every fixed η>0η>0, with the same stretched-exponential confidence. Indeed, for fixed d, the map A↦Gtr[(GAG⊤d)1/2]A _Gtr\! [ ( GAG d )^1/2 ] is concave and invariant under orthogonal conjugation, hence Schur-concave in the eigenvalues of A. Among positive semidefinite A with trA=dtrA=d, it is maximized by IdI_d. Gaussian concentration and the Marchenko–Pastur limit therefore bound the numerator by its isotropic limit, while (382) gives ‖Cμ‖F/k=1+o(1)\|C^μ\|_F/ k=1+o(1). It remains to transfer the polar geometry to every exact Spectral margin direction. Since sμ<1s_μ<1, (374) gives Γ⪰csμId c_s_μI_d. Hence the positive singular values of CμC^μ dominate csμ c_s_μ times those of the corresponding standard Gaussian matrix. If α≠1α≠ 1, the smallest positive singular value is therefore bounded below by a constant with probability 1−e−cd1-e^-cd. If α=1α=1, the standard Gaussian hard-edge estimate and a dyadic union bound give 1r#ℓ:λℓ(Cμ)≤t≤Ct,Ld−8≤t≤1. 1r\#\ : _ (C^μ)≤ t\≤ Ct, L_d^-8≤ t≤ 1. (386) with probability at least 1−Ce−cd/Ld161-Ce^-cd/L_d^16. Let W∈spXW _sp^X, put B:=PW/‖PW‖opB:=PW/\|PW\|_op, and note that ‖PW‖op=‖W‖op\|PW\|_op=\|W\|_op because centering preserves all margins. The polar factor Q has clean margin ‖Cμ‖∗/(k−1)+O(Ld−32)\|C^μ\|_*/(k-1)+O(L_d^-32) by (384). Summing the normalized margin constraints for B therefore gives ⟨B,Cμ⟩≥‖Cμ‖∗−kLd−32. B,C^μ ≥\|C^μ\|_*-kL_d^-32. Writing dℓ:=1−uℓ⊤Bvℓ∈[0,2]d_ :=1-u_ Bv_ ∈[0,2], where uℓ,vℓu_ ,v_ are the singular vectors of CμC^μ, yields 1r∑ℓ=1rλℓ(Cμ)dℓ≤CLd−32. 1r _ =1^r _ (C^μ)d_ ≤ CL_d^-32. The positive spectral edge for α≠1α≠ 1, and (386) with a split at t=Ld−8t=L_d^-8 for α=1α=1, imply 1r‖B−Q‖F2≤CLd−8. 1r\|B-Q\|_F^2≤ CL_d^-8. The centered pair identity then gives supW∈spX1k(k−1)∑i≠j‖uijW−qi−qj‖qi−qj‖2‖22≤CLd−8. _W _sp^X 1k(k-1) _i≠ j \|u_ij^W- q_i-q_j\|q_i-q_j\|_2 \|_2^2≤ CL_d^-8. (387) Define aisp:=qi‖qi‖2,pijsp:=aisp−ajsp‖aisp−ajsp‖2.a_i^sp:= q_i\|q_i\|_2, p_ij^sp:= a_i^sp-a_j^sp\|a_i^sp-a_j^sp\|_2. Equation (383) gives maxi≠j|(aisp)⊤ajsp|≤CLd−16 _i≠ j|(a_i^sp) a_j^sp|≤ CL_d^-16 and shows that pijspp_ij^sp differs from (qi−qj)/‖qi−qj‖2(q_i-q_j)/\|q_i-q_j\|_2 by O(Ld−16)O(L_d^-16). Combining (384), (385), and ‖Cμ‖F/k=1+o(1)\|C^μ\|_F/ k=1+o(1) gives maxi≠j[(pijsp)⊤μi]+≤κsp(α)+2η2. _i≠ j [(p_ij^sp) _i ]_+≤ _sp(α)+2η 2. (388) Markov’s inequality applied to (387) shows that all but o(k2)o(k^2) ordered pairs satisfy ‖uijUsp−pijsp‖2≤Ld−1\|u_ij^U_sp-p_ij^sp\|_2≤ L_d^-1. Choosing η>0η>0 sufficiently small in terms of the prescribed ε proves part (i) for Spectral selectors. For Max, use the common Haar eigenbasis of the mean covariance. Write μi=Uμμ~i,μ~i:=Dsμ1/2giTsμ,d,c~i:=μ~i−1k∑ℓ=1kμ~ℓ. _i=U_μ μ_i, μ_i:= D_s_μ^1/2g_i T_s_μ,d, c_i:= μ_i- 1k _ =1^k μ_ . Then the i-th row of CμC^μ is (Uμc~i)⊤(U_μ c_i) . Weighted Gaussian concentration gives maxi|‖c~i‖22−1|+maxi≠j|c~i⊤c~j|+maxi|c~i⊤μ~i−1|+maxi≠j|c~j⊤μ~i|≤Ld−64 _i |\| c_i\|_2^2-1 |+ _i≠ j| c_i c_j|+ _i | c_i μ_i-1 |+ _i≠ j| c_j μ_i|≤ L_d^-64 (389) outside an event of probability Cexp−cd1−sμ/LdCC \-cd^1-s_μ/L_d^C\. Put Sμ:=sign(Cμ)S^μ:=sign(C^μ). Conditional on the vectors in (389), standard smoothed-sign Haar concentration, using the identities [sign(q⊤x)sign(q⊤y)]=2πarcsin(x⊤y‖x‖2‖y‖2)E[sign(q x)sign(q y)]= 2π \! ( x y\|x\|_2\|y\|_2 ) and d[sign(q⊤x)q⊤v]=d|q1|x⊤v‖x‖2=(2π+o(1))x⊤v‖x‖2, d\,E[sign(q x)\,q v]= d\,E|q_1| x v\|x\|_2= ( 2π+o(1) ) x v\|x\|_2, gives |‖Cμ‖sumkd−2π| | \|C^μ\|_sumk d- 2π | ≤CLd−32, ≤ CL_d^-32, (390) maxi≠j|(Siμ−Sjμ)⊤μid−2π| _i≠ j | (S_i^μ-S_j^μ) _i d- 2π | ≤CLd−32, ≤ CL_d^-32, (391) maxi≠j|(Siμ)⊤Sjμd| _i≠ j | (S_i^μ) S_j^μd | ≤CLd−8, ≤ CL_d^-8, (392) 1kd#(i,a):d|Cμ[i,a]|≤Ld−16 1kd\#\(i,a): d\,|C^μ[i,a]|≤ L_d^-16\ ≤CLd−16. ≤ CL_d^-16. (393) For example, these bounds follow by replacing the sign function by a piecewise-linear approximation at scale Ld−128L_d^-128, applying concentration on (d) O(d), and then controlling the smoothing error by the spherical small-ball probability. The resulting failure probability is Cexp−cd/LdCC \-cd/L_d^C\. Let W∈maxXW _ ^X and B:=W/‖W‖maxB:=W/\|W\|_ . The sign matrix is a feasible unit-Max direction with margin at least (2/π−CLd−32)d( 2/π-CL_d^-32) d, so γmax(X) _ (X) is at least this value. Summing the normalized margin constraints for B and using (390) gives ‖Cμ‖sum−⟨B,Cμ⟩kd≤CLd−32. \|C^μ\|_sum- B,C^μ k d≤ CL_d^-32. Since |B[i,a]|≤1|B[i,a]|≤ 1, splitting the coordinates at d|Cμ[i,a]|=Ld−16 d\,|C^μ[i,a]|=L_d^-16 and using (393) yields supW∈maxX1kd‖W‖W‖max−Sμ‖F2≤CLd−16. _W _ ^X 1kd \| W\|W\|_ -S^μ \|_F^2≤ CL_d^-16. (394) Equation (392) implies ‖Siμ−Sjμ‖22=2d(1+O(Ld−8))\|S_i^μ-S_j^μ\|_2^2=2d(1+O(L_d^-8)). The centered pair identity and (394) therefore give supW∈maxX1k(k−1)∑i≠j‖uijW−Siμ−Sjμ‖Siμ−Sjμ‖2‖22≤CLd−16. _W _ ^X 1k(k-1) _i≠ j \|u_ij^W- S_i^μ-S_j^μ\|S_i^μ-S_j^μ\|_2 \|_2^2≤ CL_d^-16. (395) Define aimax:=Siμd,pijmax:=Siμ−Sjμ‖Siμ−Sjμ‖2.a_i := S_i^μ d, p_ij := S_i^μ-S_j^μ\|S_i^μ-S_j^μ\|_2. Equations (391)–(392) give maxi≠j|(aimax)⊤ajmax|≤CLd−8,maxi≠j|[(pijmax)⊤μi]2−1π|≤CLd−8. _i≠ j|(a_i ) a_j |≤ CL_d^-8, _i≠ j | [(p_ij ) _i ]^2- 1π |≤ CL_d^-8. Markov’s inequality applied to (395) leaves all but o(k2)o(k^2) pairs with direction error at most Ld−1L_d^-1. Since 1/π=ϑmax/21/π= _ /2, part (i) follows for Max selectors. The preceding bounds are uniform over the complete exact Spectral and Max faces, so the same event works for every member of the fixed finite selector collection. ∎ Test-noise geometry. The preceding lemma depends only on the class means. We now expose the independent test-noise bases. Lemma F.4 (Test-noise energies for selected directions). Condition on an event from Lemma F.3, and consider the corresponding finite collection of selected Row, Spectral, and Max directions. (i) If 0≤sΣ<10≤ s_ <1, then for every fixed 0<χ<1−sΣ0<χ<1-s_ , with conditional probability at least 1−e−dχ1-e^-d^χ for all sufficiently large d, maxselectedUmaxi≠j|(uijU)⊤ΣiuijU−1|=o(1). _selected\ U _i≠ j |(u_ij^U) _iu_ij^U-1 |=o(1). (396) (i) Suppose sΣ≥1s_ ≥ 1 and put cd:=logdTsΣ,d.c_d:= dT_s_ ,d. (397) For every fixed b>0b>0 and ε>0 >0, all selected Row directions satisfy maxi≠j(uijUrow)⊤ΣiuijUrow≤1+(4+2b+ε)cd _i≠ j(u_ij^U_row) _iu_ij^U_row≤ 1+(4+2b+ )c_d (398) except with conditional probability at most Cb,εd−bC_b, d^-b. For every fixed q∈(0,1)q∈(0,1), each selected competitor UψU_ψ, ψ∈sp,maxψ∈\sp, \, has an edge (i,j)∈dψ(i,j) _d^ψ such that (uijUψ)⊤ΣiuijUψ≥4qcd+sΣ=1(1−o(1))(u_ij^U_ψ) _iu_ij^U_ψ≥ 4qc_d+1_\s_ =1\(1-o(1)) (399) except with conditional probability at most Cq,ad−aC_q,ad^-a for every fixed 0<a<1−q0<a<1-q. Proof. For a unit vector u independent of OiO_i, write Oi⊤u=g/‖g‖2O_i u=g/\|g\|_2. The quadratic form is u⊤Σiu=d‖g‖22TsΣ,d∑r=1dr−sΣgr2.u _iu= d\|g\|_2^2T_s_ ,d _r=1^dr^-s_ g_r^2. When sΣ<1s_ <1, weighted chi-square concentration gives deviations o(1)o(1) with failure at most e−dχe^-d^χ for every χ<1−sΣχ<1-s_ . A union bound over the finite number of selectors and their O(k2)=O(d2)O(k^2)=O(d^2) pair directions proves (396). Now suppose sΣ≥1s_ ≥ 1. Laurent–Massart’s inequality gives, for every t>0t>0, 1TsΣ,d∑r=1dr−sΣgr2≤1+2T2sΣ,dt+2tTsΣ,d 1T_s_ ,d _r=1^dr^-s_ g_r^2≤ 1+ 2 T_2s_ ,dt+2tT_s_ ,d except with probability e−te^-t, while ‖g‖22/d=1+o(1)\|g\|_2^2/d=1+o(1) exponentially fast. Taking t=(2+b)logdt=(2+b) d and union-bounding over O(d2)O(d^2) selected Row pair directions gives (398); the square-root term is o(cd)o(c_d). For the competitor lower bound, fix q∈(0,1)q∈(0,1) and 0<a<1−q0<a<1-q, and choose q+∈(q,1−a)q_+∈(q,1-a). Write Ld:=logdL_d:= d, let hi:=Oie1h_i:=O_ie_1, and represent hi=ξi/‖ξi‖2h_i= _i/\| _i\|_2 using independent ξi∼(,Id) _i (0,I_d). Fix one selected competitor and its code (aiψ)(a_i^ψ) and edge set dψG_d^ψ from Lemma F.3. Put td:=2q+Ldt_d:= 2q_+L_d and define Nd:=∑(i,j)∈dψξi⊤aiψ≥td,ξi⊤ajψ≤−td.N_d:= _(i,j) _d^ψ1\ _i a_i^ψ≥ t_d,\ _i a_j^ψ≤-t_d\. The coherence bound maxi≠j|(aiψ)⊤ajψ|=o(1/Ld) _i≠ j|(a_i^ψ) a_j^ψ|=o(1/L_d) implies, uniformly over the relevant distinct indices, ℙ(ξi⊤aiψ≥td,ξi⊤ajψ≤−td)=d−2q++o(1),ℙ(ξi⊤aiψ≥td,ξi⊤ajψ≤−td,ξi⊤aℓψ≤−td)=d−3q++o(1).P( _i a_i^ψ≥ t_d, _i a_j^ψ≤-t_d)=d^-2q_++o(1), ( _i a_i^ψ≥ t_d, _i a_j^ψ≤-t_d, _i a_ ^ψ≤-t_d)=d^-3q_++o(1). Indeed, the two- and three-dimensional covariance matrices differ from the identity by o(1/Ld)o(1/L_d), and Gaussian density comparison on the relevant orthants changes the logarithmic tail asymptotics by o(Ld)o(L_d). Edge indicators with different first indices are independent. Since |dψ|=Θ(d2)|G_d^ψ|= (d^2), Nd=d2−2q++o(1),Var(Nd)≤Nd+d3−3q++o(1).EN_d=d^2-2q_++o(1), (N_d) _d+d^3-3q_++o(1). Chebyshev’s inequality gives ℙ(Nd=0)≤d−(1−q+)+o(1)=o(d−a).P(N_d=0)≤ d^-(1-q_+)+o(1)=o(d^-a). (400) On Nd>0\N_d>0\, choose a counted edge using a fixed ordering of dψG_d^ψ. This choice depends only on the class-side information and (ξi)i=1k( _i)_i=1^k, not on the remaining residual eigenframes. Uniform Gaussian norm concentration and ‖aiψ−ajψ‖2=2+o(1)\|a_i^ψ-a_j^ψ\|_2= 2+o(1) give hi⊤pijψ≥2td(1+o(1))2d.h_i p_ij^ψ≥ 2t_d(1+o(1)) 2d. For any fixed A>0A>0, spherical concentration and a union bound give, except with probability O(d−A)O(d^-A), max(i,j)∈dψd|hi⊤(uijUψ−pijψ)|≤CALd−1/2,maxi≠jd|hi⊤uijUψ|2≤CALd. _(i,j) _d^ψ d\, |h_i (u_ij^U_ψ-p_ij^ψ) |≤ C_AL_d^-1/2, _i≠ jd |h_i u_ij^U_ψ |^2≤ C_AL_d. Here the first bound uses ‖uijUψ−pijψ‖2≤Ld−1\|u_ij^U_ψ-p_ij^ψ\|_2≤ L_d^-1, and all directions in these bounds are fixed after conditioning on the class-side information. The selected edge therefore satisfies d(hi⊤uijUψ)2≥(4q+−o(1))Ld≥4qLdd (h_i u_ij^U_ψ )^2≥(4q_+-o(1))L_d≥ 4qL_d for all sufficiently large d. Since the top eigenvalue of Σi _i is d/TsΣ,d/T_s_ ,d, its contribution is at least 4qcd4qc_d. Suppose sΣ=1s_ =1. Conditional on all top eigenvectors, their auxiliary Gaussian radii, and the selected edge, the remaining eigenframe of OiO_i is Haar on hi⟂h_i . Write aij:=hi⊤uijUψa_ij:=h_i u_ij^U_ψ. The remaining contribution has the conditional distribution d(1−aij2)T1,d∑r=2dr−1gr2∑r=2dgr2,g2,…,gd∼iid(0,1). d(1-a_ij^2)T_1,d _r=2^dr^-1g_r^2 _r=2^dg_r^2, g_2,…,g_d iid N(0,1). The preceding upper bound gives aij2=O(Ld/d)a_ij^2=O(L_d/d). The weighted chi-square lower-tail inequality, with tail parameter Ld3/2L_d^3/2, gives ∑r=2dr−1gr2≥T1,d−1−O(Ld3/4) _r=2^dr^-1g_r^2≥ T_1,d-1-O(L_d^3/4) except with probability exp(−Ld3/2) (-L_d^3/2), since ∑r=2dr−2=O(1) _r=2^dr^-2=O(1). Also, ∑r=2dgr2=(d−1)(1+O(Ld−2)) _r=2^dg_r^2=(d-1)(1+O(L_d^-2)) except with probability Cexp(−cd/Ld4)C (-cd/L_d^4). Thus the remaining contribution is at least 1−O(Ld−1/4)=1−o(1)1-O(L_d^-1/4)=1-o(1) with superpolynomial confidence. Choosing A>aA>a, integrating these conditional bounds, and using (400) proves (399) with failure at most Cq,ad−aC_q,ad^-a. A finite union handles all selected competitors. ∎ Selected-direction accuracy ordering. Proposition F.5 (Selected-direction ordering). Fix 0≤sμ<10≤ s_μ<1, sΣ≥0s_ ≥ 0, and any finite collection of ℱclF_cl-measurable Row, Spectral, and Max selectors. Assume logk−1πmin,d=o(d/logd). k-1 _ ,d=o(d/ d). Then the following hold. (i) If 0≤sΣ<10≤ s_ <1, then for every fixed 0<χ<min1−sμ,1−sΣ0<χ< \1-s_μ,1-s_ \, every selected Row classifier has strictly larger population accuracy than every selected Spectral or Max classifier with probability at least 1−e−dχ1-e^-d^χ for every sufficiently large d. (i) If sΣ≥1s_ ≥ 1, then for every fixed 0<ζ<2(1−ϑα)2+ϑα,0<ζ< 2(1- _α)2+ _α, (401) the same strict ordering holds with probability at least 1−Cζd−ζ1-C_ζd^-ζ for every sufficiently large d. Proof. Suppose first that sΣ<1s_ <1. Fix 0<χ<min1−sμ,1−sΣ0<χ< \1-s_μ,1-s_ \, choose χ+∈(χ,min1−sμ,1−sΣ) _+∈(χ, \1-s_μ,1-s_ \), and fix 0<ε0<1−ϑα0< _0<1- _α. Applying Lemma F.3 and Lemma F.4(i) with confidence exponent χ+ _+ implies that every selected Row direction has Zi→j(Urow)2≥(1/2−o(1))dZ_i→ j(U_row)^2≥(1/2-o(1))d uniformly over all directed pairs, whereas every selected competitor UψU_ψ has a directed pair with [Zi→j(Uψ)]+2≤ϑψ+ε02d+o(d).[Z_i→ j(U_ψ)]_+^2≤ _ψ+ _02\,d+o(d). Since ϑψ+ε0<1 _ψ+ _0<1, the competitor-to-Row squared-signal ratio has limsup strictly below one. The Row coefficient is of order d, and log((k−1)/πmin,d)=o(d/logd)=o(d) ((k-1)/ _ ,d)=o(d/ d)=o(d), so Lemma F.1 gives the strict accuracy ordering. The two mean and residual events have joint failure at most e−dχe^-d^χ after increasing the dimension threshold. Now suppose sΣ≥1s_ ≥ 1. Fix ζ satisfying (401), choose b>ζb>ζ sufficiently close to ζ, and then choose q so that ϑα(1+b2)<q<1−ζ. _α (1+ b2 )<q<1-ζ. (402) Such a choice is possible because of (401). Choose fixed εR,εM>0 _R, _M>0 sufficiently small that (ϑα+εM)(4+2b+εR)<4q,(ϑα+εM)(5+2b+εR)<1+4q.( _α+ _M)(4+2b+ _R)<4q, ( _α+ _M)(5+2b+ _R)<1+4q. The first inequality holds with zero slacks by (402); the second follows from the same condition and ϑα<1 _α<1. Apply Lemma F.3 with signal slack εM _M and Lemma F.4 with Row-energy slack εR _R. They supply squared comparison amplitudes satisfying AR,d2 A_R,d^2 ≥(1/2−o(1))d1+(4+2b+εR)cd, ≥ (1/2-o(1))d1+(4+2b+ _R)c_d, Bψ,d2 B_ψ,d^2 ≤((ϑψ+εM)/2)d4qcd+sΣ=1(1−o(1)). ≤ (( _ψ+ _M)/2)d4qc_d+1_\s_ =1\(1-o(1)). (403) If sΣ>1s_ >1, then cd→∞c_d→∞, and lim supdBψ,d2AR,d2≤(ϑψ+εM)(4+2b+εR)4q<1. _d B_ψ,d^2A_R,d^2≤ ( _ψ+ _M)(4+2b+ _R)4q<1. If sΣ=1s_ =1, then cd→1c_d→ 1, and lim supdBψ,d2AR,d2≤(ϑψ+εM)(5+2b+εR)1+4q<1. _d B_ψ,d^2A_R,d^2≤ ( _ψ+ _M)(5+2b+ _R)1+4q<1. Moreover, AR,d2≳d/logdA_R,d^2 d/ d, so the prior assumption is sufficient for Lemma F.1. The Row residual-energy failure is O(d−b)O(d^-b), while the competitor-extreme failure is O(d−a)O(d^-a) for every a<1−qa<1-q. Choose a∈(ζ,1−q)a∈(ζ,1-q). The mean-geometry failure is stretched-exponential, so intersecting the finitely many selected-direction events gives total failure at most Cζd−ζC_ζd^-ζ. ∎ Proof of Theorem 4.2. Lemma F.2 turns the seven optimizer limits into a finite collection of ℱclF_cl-measurable Row, Spectral, and Max selectors and gives convergence of their population accuracies. Choose any fixed 0<c<2(1−ϑα)2+ϑα.0<c< 2(1- _α)2+ _α. (404) If 0≤sΣ<10≤ s_ <1, choose any fixed 0<χ<min1−sμ,1−sΣ0<χ< \1-s_μ,1-s_ \. Proposition F.5(i) gives failure probability at most e−dχe^-d^χ, which is at most d−cd^-c for every sufficiently large d. If sΣ≥1s_ ≥ 1, Proposition F.5(i), applied with ζ=cζ=c, gives failure probability at most Ccd−cC_cd^-c. After enlarging a constant C, both cases therefore hold with probability at least 1−Cd−c1-Cd^-c. The exponent c and the constant C can be chosen independently of sμs_μ and sΣs_ ; only the sufficiently-large-d threshold depends on these exponents. On this event, every selected Row limit has strictly larger population accuracy than every selected Spectral or Max limit. Equation (376) then gives minlim inft→∞opt∈rowAccπ,d(Wtopt)>maxlim supt→∞opt∈compAccπ,d(Wtopt), _opt _row _t→∞Acc_π,d\! (W_t^opt )> _opt _comp _t→∞Acc_π,d\! (W_t^opt ), which is (18). ∎ Appendix G Coordinate-Specific RowGDM–Adam Reversal This section proves the coordinate-aligned transition and the recovery under a common Haar rotation in Proposition 4.5. The comparisons hold uniformly over the complete Row and Max maximum-margin solution sets. Setup. We rescale all features by d−1/2d^-1/2, which preserves the maximum-margin direction sets and population accuracies by Lemma C.1. Let k/d→α∈(0,∞)k/d→α∈(0,∞), and suppose Γd=diag(γ1,d,…,γd,d),1dtrΓd=1,0<cΓ≤γa,d≤CΓ<∞. _d=diag( _1,d,…, _d,d), 1dtr _d=1, 0<c_ ≤ _a,d≤ C_ <∞. (405) The independent class means and clean training set are μi=1dΓd1/2gi,gi∼iid(,Id),=(μi,i):i∈[k]. _i= 1 d _d^1/2g_i, g_i iid N(0,I_d), =\( _i,i):i∈[k]\. (406) Write M∈ℝk×dM ^k× d for the matrix with rows μi⊤ _i , and define ℱ(M):=W:(wi−wj)⊤μi≥1 for every i≠j.F(M):=\W:(w_i-w_j) _i≥ 1 for every i≠ j\. The two minimum-norm solution sets are ℳrow(M):=argminW∈ℱ(M)‖W‖row,ℳmax(M):=argminW∈ℱ(M)‖W‖max.M_row(M):= *arg\,min_W (M)\|W\|_row, _max(M):= *arg\,min_W (M)\|W\|_ . (407) For Σ≻0 0 and a classifier with positive directed mean margins, set Li→j(W,Σ):=((wi−wj)⊤μi)2(wi−wj)⊤Σ(wi−wj),Ed(W,Σ):=mini≠jLi→j(W,Σ).L_i→ j(W; ):= ((w_i-w_j) _i)^2(w_i-w_j) (w_i-w_j), E_d(W; ):= _i≠ jL_i→ j(W; ). (408) The test distribution is ℙ(Y=i)=πi,d,Xte∣(Y=i)∼(μi,rte2dΣ),P(Y=i)= _i,d, X^te (Y=i) \! ( _i, r_te^2d ), (409) where 0<rte≤rte,0<∞0<r_te≤ r_te,0<∞, πmin,d:=miniπi,d>0 _ ,d:= _i _i,d>0, and log(1/πmin,d)=o(d). (1/ _ ,d)=o(d). (410) Write Errπ,d=1−Accπ,dErr_π,d=1-Acc_π,d and g0:=2/πg_0:= 2/π. Constants denoted by c,Cc,C may change between displays; their permitted dependence is specified in each statement. Lemma G.1 (Gaussian coefficient comparison). Every classifier with positive directed mean margins satisfies πmin,dΦ¯(dEd(W,Σ)rte)≤Errπ,d(W)≤(k−1)Φ¯(dEd(W,Σ)rte), _ ,d\, \! ( dE_d(W; )r_te ) _π,d(W)≤(k-1)\, \! ( dE_d(W; )r_te ), (411) where Φ¯(x)=ℙ(Z≥x) (x)=P(Z≥ x) for Z∼(0,1)Z (0,1). If nonempty classifier families ℛ,R,C have positive directed mean margins and satisfy Ed(R,Σ)≥A>B≥Ed(V,Σ)>0E_d(R; )≥ A>B≥ E_d(V; )>0 for every R∈ℛR and V∈V , then supR∈ℛErrπ,d(R)infV∈Errπ,d(V)≤k−1πmin,dexp(−d(A−B)2rte,02). _R Err_π,d(R) _V Err_π,d(V)≤ k-1 _ ,d \! (- d(A-B)2r_te,0^2 ). (412) In particular, a fixed positive coefficient gap gives strict uniform accuracy ordering for all sufficiently large d, without any additional probabilistic loss. Proof. The directed pair error equals Φ¯(dLi→j(W,Σ)/rte) ( dL_i→ j(W; )/r_te). A classwise union bound gives the upper bound in (411), and a pair attaining EdE_d gives the lower bound. For x≥y≥0x≥ y≥ 0, changing variables in the Gaussian tail integral gives Φ¯(x)=∫y∞e−(u+x−y)2/22πdu≤e−(x2−y2)/2Φ¯(y). (x)= _y^∞ e^-(u+x-y)^2/2 2π\,du≤ e^-(x^2-y^2)/2 (y). Applying this inequality with x=dA/rtex= dA/r_te and y=dB/rtey= dB/r_te proves (412). The final assertion follows from k=Θ(d)k= (d) and (410). ∎ Lemma G.2 (Deterministic whole-face stability). Fix h∗>0h_*>0 and K<∞K<∞. There exist ε0,C>0 _0,C>0, depending only on h∗,Kh_*,K, with the following property. Let 0<ε≤ε00< ≤ _0, k−1≤εk^-1≤ , and h∈[h∗,1]h∈[h_*,1]. Suppose deterministic means have no zero coordinates, and, with μ¯=k−1∑iμi μ=k^-1 _i _i and Ti=sign(μi)T_i=sign( _i), satisfy maxi|‖μi‖22−1| _i|\| _i\|_2^2-1| ≤ε, ≤ , maxi≠j|μi⊤μj| _i≠ j| _i _j| ≤ε, ≤ , ‖μ¯‖2 \| μ\|_2 ≤ε, ≤ , maxi|‖μi‖1d−h| _i | \| _i\|_1 d-h | ≤ε, ≤ , maxi≠j|Tj⊤μi|d _i≠ j |T_j _i| d ≤ε, ≤ , maxi≠j|Ti⊤Tj|d _i≠ j |T_i T_j|d ≤ε. ≤ . (413) Assume also that, for s∈ε1/2,ε1/8s∈\ ^1/2, ^1/8\, maxi1d#a:d|μi[a]|≤s≤Ks+ε. _i 1d\#\a: d\,| _i[a]|≤ s\≤ Ks+ . (414) Then both solution sets are nonempty, and, writing tmax:=minW∈ℱ(M)‖W‖maxt_ := _W (M)\|W\|_ , supW∈ℳrow(M)maxi‖wi−μi‖2 _W _row(M) _i\|w_i- _i\|_2 ≤Cε1/4, ≤ C ^1/4, (415) |1tmaxd−h|+supW∈ℳmax(M)maxi‖wi/tmax−Ti‖2d | 1t_ d-h |+ _W _max(M) _i \|w_i/t_ -T_i\|_2 d ≤Cε1/16. ≤ C ^1/16. (416) Proof. The matrix with rows μi/(1−2ε) _i/(1-2 ) is feasible, so the optimal row radius R∗R_* is at most 1+ε/(1−2ε)≤1+Cε 1+ /(1-2 )≤ 1+C . For any feasible W, averaging the class-i constraints over competitors gives (wi−w¯)⊤μi≥1−1k,w¯:=1k∑iwi.(w_i- w) _i≥ 1- 1k, w:= 1k _iw_i. (417) For a Row minimizer, Cauchy–Schwarz and the row variance identity therefore imply (1−1/k)21+ε≤1k∑i‖wi−w¯‖22≤R∗2−‖w¯‖22. (1-1/k)^21+ ≤ 1k _i\|w_i- w\|_2^2≤ R_*^2-\| w\|_2^2. Thus ‖w¯‖2≤Cε1/2\| w\|_2≤ C ^1/2, and (417) gives wi⊤μi≥1−Cε1/2w_i _i≥ 1-C ^1/2 uniformly in i. Expanding ‖wi−μi‖22\|w_i- _i\|_2^2 proves (415). The sign matrix divided by d(h−2ε) d(h-2 ) is feasible, so tmax≤[d(h−2ε)]−1t_ ≤[ d(h-2 )]^-1. Fix a Max minimizer and set H=W/tmaxH=W/t_ , H¯=k−1∑iHi H=k^-1 _iH_i, and Di:=‖μi‖1−Hi⊤μid≥0.D_i:= \| _i\|_1-H_i _i d≥ 0. Since ‖H‖max=1\|H\|_ =1, averaging (417) over i yields 1k∑iDi 1k _iD_i ≤h+ε−1−1/ktmaxd−H¯⊤μ¯d ≤ h+ - 1-1/kt_ d- H μ d ≤h+ε−(1−1/k)(h−2ε)+‖μ¯‖2≤Cε. ≤ h+ -(1-1/k)(h-2 )+\| μ\|_2≤ C . For every s>0s>0, the coordinate bound on H gives ‖Hi−Ti‖22d≤2Dis+4d#a:d|μi[a]|≤s. \|H_i-T_i\|_2^2d≤ 2D_is+ 4d\#\a: d\,| _i[a]|≤ s\. (418) Indeed, outside the indicated small-coordinate set, use (Hi[a]−Ti[a])2≤2(1−Hi[a]Ti[a])(H_i[a]-T_i[a])^2≤ 2(1-H_i[a]T_i[a]), and use the bound four on that set. Taking s=ε1/2s= ^1/2 and averaging gives ‖H−T‖F2/(kd)≤Cε1/2\|H-T\|_F^2/(kd)≤ C ^1/2. Moreover, ‖k−1∑iTi‖22/d≤k−1+ε≤2ε\|k^-1 _iT_i\|_2^2/d≤ k^-1+ ≤ 2 , so Jensen’s inequality gives ‖H¯‖2/d≤Cε1/4\| H\|_2/ d≤ C ^1/4. Returning to the individual constraints now yields Di≤h+ε−(1−1/k)(h−2ε)+‖H¯‖2‖μi‖2d≤Cε1/4.D_i≤ h+ -(1-1/k)(h-2 )+ \| H\|_2\| _i\|_2 d≤ C ^1/4. Taking s=ε1/8s= ^1/8 in (418) proves the claimed uniform row error for H. Finally, averaging the constraints and using Hi⊤μi≤‖μi‖1H_i _i≤\| _i\|_1 gives h−2ε≤1tmaxd≤h+2ε1−1/k=h+O(ε).h-2 ≤ 1t_ d≤ h+2 1-1/k=h+O( ). This proves (416). The feasible witnesses and compact norm sublevel sets ensure existence of both minima. ∎ Lemma G.3 (Quantitative whole-face geometry in aligned coordinates). Define γ¯1/2,d:=1d∑aγa,d,hd:=g0γ¯1/2,d,γ¯2,d:=1d∑aγa,d2. γ_1/2,d:= 1d _a _a,d, h_d:=g_0 γ_1/2,d, γ_2,d:= 1d _a _a,d^2. (419) There exist ε0,c,C>0 _0,c,C>0, depending only on cΓ,CΓc_ ,C_ , such that, whenever 0<ε≤ε00< ≤ _0 and k≥Cε−2k≥ C ^-2, an event d(ε)A_d( ) satisfies ℙ(d(ε)c)≤Ck2e−cdε2.P(A_d( )^c)≤ Ck^2e^-cd ^2. (420) On this event, both solution sets are nonempty and supW∈ℳrow(M)maxi‖wi−μi‖2 _W _row(M) _i\|w_i- _i\|_2 ≤Cε1/4, ≤ C ^1/4, (421) supW∈ℳmax(M)maxi‖wi/tmax−Ti‖2d _W _max(M) _i \|w_i/t_ -T_i\|_2 d ≤Cε1/16,Ti:=sign(μi). ≤ C ^1/16, T_i:=sign( _i). (422) Also, uniformly over i≠ji≠ j, |‖μi‖22−1|+|μi⊤μj| |\| _i\|_2^2-1 |+| _i _j| ≤Cε, ≤ C , |(Ti−Tj)⊤μid−hd|+|‖Ti−Tj‖222d−1| | (T_i-T_j) _i d-h_d |+ | \|T_i-T_j\|_2^22d-1 | ≤Cε. ≤ C . (423) For Σλ:=(1−λ)Id+λΓd _λ:=(1-λ)I_d+λ _d and Ad(λ):=1+λ(γ¯2,d−1)A_d(λ):=1+λ( γ_2,d-1), the same event satisfies supλ∈[0,1]maxi≠j|(μi−μj)⊤Σλ(μi−μj)−2Ad(λ)| _λ∈[0,1] _i≠ j |( _i- _j) _λ( _i- _j)-2A_d(λ) | ≤Cε, ≤ C , supλ∈[0,1]maxi≠j|(Ti−Tj)⊤Σλ(Ti−Tj)2d−1| _λ∈[0,1] _i≠ j | (T_i-T_j) _λ(T_i-T_j)2d-1 | ≤Cε. ≤ C . (424) Proof. We verify the inputs of Lemma G.2 at precision ε . The independent-coordinate Gaussian quadratic-form and Bernstein bounds used below follow, for example, from the concentration tools in Vershynin (2012). For the quadratic terms, the identity eu(Z2−1)=e−u(1−2u)−1/2Ee^u(Z^2-1)=e^-u(1-2u)^-1/2 for |u|<1/2|u|<1/2, together with the bounded spectral weights, gives deviation probability at most 2e−cdε22e^-cd ^2 for each normalized quadratic form. Polarization gives the corresponding bilinear bounds. The variables γa,d|gi[a]| _a,d|g_i[a]| have uniformly bounded exponential moments, so Bernstein’s inequality gives ℙ(|‖μi‖1d−hd|>ε)≤2e−cdε2.P\! ( | \| _i\|_1 d-h_d |> )≤ 2e^-cd ^2. For j≠ij≠ i, conditioning on TjT_j gives Tj⊤μi/d∼(0,1/d)T_j _i/ d (0,1/d). The products Ti[a]Tj[a]T_i[a]T_j[a] are independent Rademacher variables across a, so their averages satisfy the same exponential bound. For either prescribed threshold s, Gaussian small-ball probabilities and Hoeffding’s inequality give ℙ(1d#a:d|μi[a]|≤s>Ks+ε)≤e−2dε2,P\! ( 1d\#\a: d\,| _i[a]|≤ s\>Ks+ )≤ e^-2d ^2, where K depends only on cΓc_ . Also, μ¯ μ has covariance Γd/(kd) _d/(kd); when k≥Cε−2k≥ C ^-2, the event ‖μ¯‖2>ε\| μ\|_2> has probability at most Ce−cdCe^-cd. All coordinates are nonzero almost surely, and g0cΓ≤hd≤g0<1g_0 c_ ≤ h_d≤ g_0<1. A union bound over these O(k2)O(k^2) events verifies the deterministic lemma’s hypotheses and proves (421)–(423). For (424), it suffices to treat λ=0,1λ=0,1, since the quadratic forms and their asserted centers are affine in λ. The Gaussian difference μi−μj _i- _j has covariance 2Γd/d2 _d/d, giving mean quadratic form 2d−1tr(ΓdΣλ)=2Ad(λ)2d^-1tr( _d _λ)=2A_d(λ). The same quadratic-form bound applies at both endpoints. For the sign contrasts, (Ti[a]−Tj[a])2(T_i[a]-T_j[a])^2 are independent across coordinates, lie in 0,4\0,4\, and have mean two. Their bounded weighted sums therefore concentrate about 2trΣλ=2d2tr _λ=2d with failure at most 2e−cdε22e^-cd ^2 at each endpoint. Adding these events proves (420) and (424). ∎ Proposition G.4 (An exact Row–Max transition under aligned test anisotropy). Suppose γ¯1/2,d→γ¯1/2,γ¯2,d→γ¯2, γ_1/2,d→ γ_1/2, γ_2,d→ γ_2, (425) and set h:=g0γ¯1/2,h2γ¯2>1.h:=g_0 γ_1/2, h^2 γ_2>1. (426) For Σλ:=(1−λ)Id+λΓd,λ∈[0,1], _λ:=(1-λ)I_d+λ _d, λ∈[0,1], (427) there are constants ε0,C>0 _0,C>0, depending only on the spectral bounds, such that, whenever 0<ε≤ε00< ≤ _0 and k≥Cε−2k≥ C ^-2, the event d(ε)A_d( ) in Lemma G.3 satisfies supλ∈[0,1]supW∈ℳrow(M)|Ed(W,Σλ)−12Ad(λ)| _λ∈[0,1] _W _row(M) |E_d(W; _λ)- 12A_d(λ) | ≤Cε1/4, ≤ C ^1/4, supλ∈[0,1]supW∈ℳmax(M)|Ed(W,Σλ)−hd22| _λ∈[0,1] _W _max(M) |E_d(W; _λ)- h_d^22 | ≤Cε1/16. ≤ C ^1/16. (428) The limiting coefficients cross uniquely at λc:=h−2−1γ¯2−1∈(0,1). _c:= h^-2-1 γ_2-1∈(0,1). (429) For every fixed λ≠λcλ≠ _c, there exist cλ,Cλ>0c_λ,C_λ>0, independent of d, such that, for every sufficiently large d, λ<λc λ< _c ⟹ℙinfW∈ℳrowAccπ,d(W)>supW∈ℳmaxAccπ,d(W)≥1−Cλe−cλd, \! \ _W _rowAcc_π,d(W)> _W _maxAcc_π,d(W) \≥ 1-C_λe^-c_λd, (430) λ>λc λ> _c ⟹ℙinfW∈ℳmaxAccπ,d(W)>supW∈ℳrowAccπ,d(W)≥1−Cλe−cλd. \! \ _W _maxAcc_π,d(W)> _W _rowAcc_π,d(W) \≥ 1-C_λe^-c_λd. (431) At λ=λcλ= _c, only equality of the limiting coefficients is asserted. Proof. The spectral bounds give cΓId⪯Σλ⪯CΓIdc_ I_d _λ C_ I_d for every λ∈[0,1]λ∈[0,1], since trace normalization implies cΓ≤1≤CΓc_ ≤ 1≤ C_ . For Row, (421) allows us to replace wi−wjw_i-w_j by μi−μj _i- _j with Euclidean error Cε1/4C ^1/4. The latter vector has directed signal 1+O(ε)1+O( ) and test-noise quadratic form 2Ad(λ)+O(ε)2A_d(λ)+O( ), uniformly over pairs and λ. The denominators are bounded away from zero, proving the first estimate in (428). For Max, scale invariance allows us to use the contrast (Hi−Hj)/d(H_i-H_j)/ d, where H=W/tmaxH=W/t_ . By (422), this contrast differs from (Ti−Tj)/d(T_i-T_j)/ d by at most Cε1/16C ^1/16. The canonical sign contrast has directed signal hd+O(ε)h_d+O( ) and test-noise quadratic form 2+O(ε)2+O( ), proving the second estimate. These arguments also prove the same estimates for every individual Li→jL_i→ j. Since γ¯1/2≤1 γ_1/2≤ 1, we have 0<h<10<h<1; the condition h2γ¯2>1h^2 γ_2>1 implies γ¯2>1 γ_2>1 and 0<λc<10< _c<1. Fix λ≠λcλ≠ _c, and let Δλ:=|12[1+λ(γ¯2−1)]−h22|>0. _λ:= | 12[1+λ( γ_2-1)]- h^22 |>0. Choose a fixed ε>0 >0 small enough that the sum of the two errors in (428) is at most Δλ/4 _λ/4. By (425), the finite-dimensional coefficient gap has the asserted sign and magnitude at least 3Δλ/43 _λ/4 for all sufficiently large d. Thus the complete-face gap is at least Δλ/2 _λ/2 on d(ε)A_d( ). Lemma G.1 proves the corresponding strict accuracy ordering. Finally, k=Θ(d)k= (d) allows the factor k2k^2 in (420) to be absorbed into Cλe−cλdC_λe^-c_λd. No convergence rate for either spectral summary is required; their convergence affects only the sufficiently-large-d threshold. ∎ The Haar-rotation statements below hold equally for Haar measure on O(d)O(d) and on SO(d)SO(d). Indeed, for the fixed orthonormal frames used below, both measures induce the same uniform Stiefel distribution, and, for sufficiently large d, conditional on a fixed two-frame, the unexposed direction is uniform on the orthogonal sphere in either case. Lemma G.5 (Quantitative sign statistics under a Haar rotation). Let deterministic vectors x1,…,xk∈ℝdx_1,…,x_k ^d satisfy 1/2≤‖xi‖2≤21/2≤\|x_i\|_2≤ 2, let R be Haar on O(d)O(d) or SO(d)SO(d), and put Ti=sign(Rxi)T_i=sign(Rx_i). There are universal ε0,c,C>0 _0,c,C>0 such that, for 0<ε≤ε00< ≤ _0 and any two prescribed thresholds s1,s2∈(0,1]s_1,s_2∈(0,1], an event of probability at least 1−Ck2e−cdε61-Ck^2e^-cd ^6 satisfies maxi,j|Ti⊤Rxjd−g0xi⊤xj‖xi‖2| _i,j | T_i Rx_j d-g_0 x_i x_j\|x_i\|_2 | ≤Cε, ≤ C , maxi,j|Ti⊤Tjd−2πarcsin(xi⊤xj‖xi‖2‖xj‖2)| _i,j | T_i T_jd- 2π \! ( x_i x_j\|x_i\|_2\|x_j\|_2 ) | ≤Cε, ≤ C , maxi1d#a:d|(Rxi)[a]|≤sℓ _i 1d\#\a: d\,|(Rx_i)[a]|≤ s_ \ ≤Csℓ+Cε2,ℓ=1,2. ≤ Cs_ +C ^2, =1,2. (432) Proof. Fix a pair and choose an orthonormal basis F∈ℝd×mF ^d× m of its span, where m≤2m≤ 2. Write xi=Fuix_i=Fu_i, xj=Fujx_j=Fu_j, and let Z∈ℝd×mZ ^d× m have independent standard Gaussian entries. Then dRF=dY:=ZB,B:=(Z⊤Z/d)−1/2. d\,RF\ d=\ Y:=ZB, B:=(Z Z/d)^-1/2. Indeed, Y/dY/ d has orthonormal columns and its distribution is invariant under every left orthogonal transformation. The Gaussian quadratic-form bound gives ℙ(‖Z⊤Z/d−Im‖op>ε3)≤Ce−cdε6.P\! (\|Z Z/d-I_m\|_op> ^3 )≤ Ce^-cd ^6. On the complementary event, ‖B−Im‖op≤Cε3\|B-I_m\|_op≤ C ^3, and hence 1d‖(Y−Z)u‖22≤Cε6 1d\|(Y-Z)u\|_2^2≤ C ^6 for either coefficient vector u. Set X=ZuX=Zu, X~=Yu X=Yu, and b=ε2b= ^2. Gaussian small-ball probabilities and Hoeffding’s inequality imply d−1#a:|Xa|≤b≤Cε2d^-1\#\a:|X_a|≤ b\≤ C ^2, except with probability Ce−cdε4Ce^-cd ^4. Consequently, 1d#a:sign(X~a)≠sign(Xa)≤1d#a:|Xa|≤b+‖X~−X‖22db2≤Cε2. 1d\#\a:sign( X_a) (X_a)\≤ 1d\#\a:|X_a|≤ b\+ \| X-X\|_2^2db^2≤ C ^2. Thus the normalized Euclidean distance between the corresponding sign vectors is at most CεC . By Cauchy–Schwarz, replacing Y by Z changes the empirical averages involving one sign factor and one linear factor by at most CεC , while the empirical correlations between the corresponding sign vectors change by at most Cε2C ^2. For a standard Gaussian row z⊤z of Z, the corresponding expectations are [sign(z⊤u)z⊤v]=g0u⊤v‖u‖2,[sign(z⊤u)sign(z⊤v)]=2πarcsin(u⊤v‖u‖2‖v‖2).E[sign(z u)z v]=g_0 u v\|u\|_2, [sign(z u)sign(z v)]= 2π \! ( u v\|u\|_2\|v\|_2 ). The first identity follows by conditioning z⊤vz v on z⊤uz u, and the second is the standard arcsine formula for the signs of two jointly Gaussian variables. Bernstein’s and Hoeffding’s inequalities concentrate the corresponding empirical averages within ε with failure at most Ce−cdε2Ce^-cd ^2. Finally, for either prescribed s, 1d#a:|X~a|≤s≤1d#a:|Xa|≤s+b+‖X~−X‖22db2≤Cs+Cε2 1d\#\a:| X_a|≤ s\≤ 1d\#\a:|X_a|≤ s+b\+ \| X-X\|_2^2db^2≤ Cs+C ^2 except with probability Ce−cdε4Ce^-cd ^4. A union bound over the pairs and the two thresholds proves (432); independence between different pairs is not needed. ∎ Lemma G.6 (Generic common rotation restores the Row coefficient advantage). For deterministic means, define δd:=maxi|‖μi‖22−1|+maxi≠j|μi⊤μj|. _d:= _i|\| _i\|_2^2-1|+ _i≠ j| _i _j|. (433) Let Σ≻0 0 satisfy trΣ=dtr =d and cΣId⪯Σ⪯CΣIdc_ I_d C_ I_d, and set μi∘=Rμi,Σ∘=RΣR⊤,vij:=μi−μj‖μi−μj‖2,aij:=vij⊤Σvij, _i =R _i, =R R , v_ij:= _i- _j\| _i- _j\|_2, a_ij:=v_ij v_ij, where R is an independent Haar matrix. There are ε0,c,C>0 _0,c,C>0, depending only on cΣ,CΣc_ ,C_ , such that, whenever 0<ε≤ε0,δd+d−1+k−1≤ε2,0< ≤ _0, _d+d^-1+k^-1≤ ^2, an event of probability at least 1−Ck2e−cdε61-Ck^2e^-cd ^6, over R alone, satisfies supW∈ℳrow∘maxi≠j|Li→j(W,Σ∘)−12aij| _W _row _i≠ j |L_i→ j(W; )- 12a_ij | ≤Cε1/4, ≤ C ^1/4, supW∈ℳmax∘maxi≠j|Li→j(W,Σ∘)−12aij+π−2| _W _max _i≠ j |L_i→ j(W; )- 12a_ij+π-2 | ≤Cε1/16. ≤ C ^1/16. (434) Here the superscript ∘ denotes solution sets and accuracies for the rotated problem. Consequently, if δd→0 _d→ 0, k/d→α∈(0,∞)k/d→α∈(0,∞), and (410) holds, there exist c∗,C∗>0c_*,C_*>0, independent of d, such that, for every sufficiently large d, ℙRinfW∈ℳrow∘Accπ,d∘(W)>supW∈ℳmax∘Accπ,d∘(W)≥1−C∗e−c∗d.P_R\! \ _W _row Acc_π,d (W)> _W _max Acc_π,d (W) \≥ 1-C_*e^-c_*d. (435) Proof. The hypotheses imply ‖μ¯‖22≤(1+δd)/k+δd≤Cε2\| μ\|_2^2≤(1+ _d)/k+ _d≤ C ^2. Apply Lemma G.5 to the means. The norm and coherence assumptions, together with (432), give, for Ti=sign(Rμi)T_i=sign(R _i), maxi|‖Rμi‖1d−g0|+maxi≠j|Tj⊤Rμi|d+maxi≠j|Ti⊤Tj|d≤Cε. _i | \|R _i\|_1 d-g_0 |+ _i≠ j |T_j R _i| d+ _i≠ j |T_i T_j|d≤ C . (436) Choose a fixed sufficiently large constant C0C_0 and apply that lemma with thresholds (C0ε)1/2(C_0 )^1/2 and (C0ε)1/8(C_0 )^1/8. After decreasing ε0 _0, Lemma G.2 applies at precision C0εC_0 , with h=g0h=g_0. It follows that every Row minimizer is within Cε1/4C ^1/4 per row of the rotated means, while every rescaled Max minimizer is within Cε1/16dC ^1/16 d per row of TiT_i. It remains to calculate the test-noise energy of the canonical sign contrast bij:=Ti−Tj‖Ti−Tj‖2.b_ij:= T_i-T_j\|T_i-T_j\|_2. Fix a pair, let S=spanμi,μjS=span\ _i, _j\, and choose an orthonormal basis F∈ℝd×2F ^d× 2 of S. The Gram matrix of the pair is uniformly well-conditioned under the assumed bound on δd _d. Conditioning only on RFRF, the vector bijb_ij is fixed whenever it is defined. Set p=(RF)⊤bijp=(RF) b_ij, q=Fpq=Fp, and r2=1−‖p‖22r^2=1-\|p\|_2^2. The unexposed action of R on the orthogonal complement is Haar, so conditionally R⊤bij=q+rζ,R b_ij=q+rζ, (437) where ζ is uniform on the unit sphere of S⟂S . On the sign-statistics event, (432) gives bij⊤Rμi=g02+O(ε),bij⊤Rμj=−g02+O(ε).b_ij R _i= g_0 2+O( ), b_ij R _j=- g_0 2+O( ). The same two inner products for g0vijg_0v_ij are g0/2+O(δd)g_0/ 2+O( _d) and −g0/2+O(δd)-g_0/ 2+O( _d). Inverting the well-conditioned two-by-two Gram matrix therefore yields ‖q−g0vij‖2≤Cε,|r2−(1−g02)|≤Cε.\|q-g_0v_ij\|_2≤ C , |r^2-(1-g_0^2)|≤ C . (438) For the conditional complement calculation, let PSP_S denote the orthogonal projector onto S. Representing ζ by a normalized standard Gaussian in S⟂S and applying the bounded-weight quadratic-form and Gaussian linear-form bounds gives |ζ⊤Σζ−tr((Id−PS)Σ)d−2|+|q⊤Σζ|≤Cε |ζ ζ- tr((I_d-P_S) )d-2 |+|q ζ|≤ C with conditional failure at most Ce−cdε2Ce^-cd ^2, uniformly in the exposed frame. The trace ratio is 1+O(d−1)1+O(d^-1), since trΣ=dtr =d and rankPS=2rankP_S=2. The conditional failure bound can be integrated and then union-bounded over pairs; we do not condition on the sign vectors of the other classes. Combining this bound with (437) and (438) gives maxi≠j|bij⊤Σ∘bij−(g02aij+1−g02)|≤Cε. _i≠ j |b_ij b_ij-(g_0^2a_ij+1-g_0^2) |≤ C . (439) The canonical Row direction RvijRv_ij has directed signal 1/2+O(δd)1/ 2+O( _d) and test-noise energy exactly aija_ij. The canonical Max direction has directed signal g0/2+O(ε)g_0/ 2+O( ), and g02/2g02aij+1−g02=12aij+π−2. g_0^2/2g_0^2a_ij+1-g_0^2= 12a_ij+π-2. Both canonical contrast norms are bounded away from zero after the Max contrasts are divided by d d. Normalization stability, the whole-face bounds already proved, and Σ∘⪰cΣId c_ I_d therefore transfer these coefficients uniformly to the complete solution sets, proving (434). The combined failure is at most Ck2e−cdε6Ck^2e^-cd ^6. To pass to the minimum directed coefficient, put a∗:=maxi≠jaija_*:= _i≠ ja_ij. Then (434) implies infW∈ℳrow∘Ed(W,Σ∘)−supW∈ℳmax∘Ed(W,Σ∘)≥12a∗−12a∗+π−2−Cε1/16. _W _row E_d(W; )- _W _max E_d(W; )≥ 12a_*- 12a_*+π-2-C ^1/16. Since a∗≤CΣa_*≤ C_ , the difference of the two displayed fractions is at least Δrot:=π−22CΣ(2CΣ+π−2)>0. _rot:= π-22C_ (2C_ +π-2)>0. (440) Choose a fixed ε>0 >0 so that Cε1/16≤Δrot/2C ^1/16≤ _rot/2. The regularity assumptions eventually ensure δd+d−1+k−1≤ε2 _d+d^-1+k^-1≤ ^2. Lemma G.1 gives the strict accuracy ordering, and absorbing k2k^2 into the exponential proves (435). ∎ Proposition G.7 (Opposite Row–Max orderings before and after a common Haar rotation). Under the hypotheses of Proposition G.4, fix λ∈(λc,1]λ∈( _c,1]. Compare the original problem with means μi _i and covariance Σλ _λ to the problem obtained by the independent common rotation μi↦Rμi _i R _i, Σλ↦RΣλR⊤ _λ R _λR . There exist c,C>0c,C>0, independent of d, such that, for every sufficiently large d, with joint probability at least 1−Ce−cd1-Ce^-cd, infW∈ℳmaxAccπ,d(W) _W _maxAcc_π,d(W) >supW∈ℳrowAccπ,d(W), > _W _rowAcc_π,d(W), infW∈ℳrow∘Accπ,d∘(W) _W _row Acc_π,d (W) >supW∈ℳmax∘Accπ,d∘(W). > _W _max Acc_π,d (W). (441) The two problems have identical class Gram matrices, class-mean norms and pair distances, mean singular values, test-noise eigenvalues, and mean-contrast test-noise variances. Proof. Proposition G.4 gives the first ordering with exponentially small failure. For the second ordering, choose the fixed precision ε in the last paragraph of Lemma G.6. Gaussian quadratic and bilinear concentration gives ℙδd>ε2/2≤Ck2e−cdε4.P\ _d> ^2/2\≤ Ck^2e^-cd ^4. For all sufficiently large d, we also have d−1+k−1≤ε2/2d^-1+k^-1≤ ^2/2. Conditional on any means in this event, Lemma G.6 applies with the same constants, since cΓId⪯Σλ⪯CΓIdc_ I_d _λ C_ I_d. Its conditional failure is at most Ck2e−cdε6Ck^2e^-cd ^6. Integrating this uniform conditional bound and intersecting with the first ordering proves (441). The invariances follow directly from orthogonality; in particular, (Rvij)⊤(RΣλR⊤)(Rvij)=vij⊤Σλvij.(Rv_ij) (R _λR )(Rv_ij)=v_ij _λv_ij. ∎ Proof of Proposition 4.5. Lemma C.1 identifies the direction sets and accuracies in the main-text normalization with those analyzed here. Since h=g0γ¯1/2h=g_0 γ_1/2, the transition point in Proposition G.4 is h−2−1γ¯2−1=π/(2γ¯1/22)−1γ¯2−1=λc. h^-2-1 γ_2-1= π/(2 γ_1/2^2)-1 γ_2-1= _c. That proposition therefore gives the diagonal complete-face ordering with probability at least 1−Cλe−cλd1-C_λe^-c_λd for every fixed λ≠λcλ≠ _c and sufficiently large d. For every fixed λ∈[0,1]λ∈[0,1], the conditional argument in the proof of Proposition G.7 gives the rotated Row-over-Max ordering with probability at least 1−Crote−crotd1-C_rote^-c_rotd. This argument does not require λ>λcλ> _c. Its spectral bounds and the positive gap in (440) are uniform in λ, so crot,Crotc_rot,C_rot can be chosen independently of λ. At each fixed dimension on these events, Proposition B.14 and Theorem B.7, together with Lemma B.2, place the normalized RowGDM and Adam trajectories asymptotically in their respective Row and Max maximum-margin direction sets. Every classifier in these direction sets has nonzero pairwise row differences. The compactness and continuity argument of Lemma C.21 applies to the present positive-definite Gaussian test covariances as well, since all score-tie hyperplanes have probability zero. It transfers each uniform complete-face ordering to the stated strict comparison between the trajectory liminf and limsup accuracies, without an additional exceptional event or a directional-convergence assumption. The same argument applies to RowGD. ∎ Corollary G.8 (Spectral dispersion implies a transition). Suppose the assumptions of Proposition 4.5 concerning the existence of the limiting spectral summaries hold. Assume additionally that, for some fixed 0<γ¯<1<γ¯<∞0< γ<1< γ<∞, γ¯≤γa,d≤γ¯for all a,d, γ≤ _a,d≤ γ all a,d, and that vΓ:=limd→∞1d∑a=1d(γa,d−1)2>π2q∗2−1,q∗:=(γ¯−1)γ¯+(1−γ¯)γ¯γ¯−γ¯.v_ := _d→∞ 1d _a=1^d( _a,d-1)^2> π2q_*^2-1, q_*:= ( γ-1) γ+(1- γ) γ γ- γ. Then 2γ¯1/22γ¯2/π>12 γ_1/2^2 γ_2/π>1, and hence the transition point in Proposition 4.5 lies in (0,1)(0,1). In particular, the transition condition does not require a special spectral shape: within a fixed bounded spectral range, the displayed lower bound on the limiting spectral variance is sufficient. Proof. Concavity of x↦x x places its graph above the chord joining (γ¯,γ¯)( γ, γ) and (γ¯,γ¯)( γ, γ). Averaging this chord inequality and using d−1∑aγa,d=1d^-1 _a _a,d=1 gives γ¯1/2≥q∗ γ_1/2≥ q_*. Trace normalization also gives γ¯2=1+vΓ γ_2=1+v_ , and therefore 2γ¯1/22γ¯2π≥2q∗2(1+vΓ)π>1. 2 γ_1/2^2 γ_2π≥ 2q_*^2(1+v_ )π>1. ∎ Remark G.9 (Non-vacuity of the dispersion condition). Write l:=γ¯l:= γ and u:=γ¯u:= γ. The variance permitted by the spectral range is itself bounded. Indeed, since l≤γa,d≤ul≤ _a,d≤ u, we have (γa,d−l)(u−γa,d)≥0( _a,d-l)(u- _a,d)≥ 0, or equivalently γa,d2≤(l+u)γa,d−lu _a,d^2≤(l+u) _a,d-lu. Averaging and using the trace normalization d−1∑aγa,d=1d^-1 _a _a,d=1 gives γ¯2≤l+u−lu γ_2≤ l+u-lu, and hence vΓ=γ¯2−1≤(1−l)(u−1)v_ = γ_2-1≤(1-l)(u-1). This bound is sharp: it is attained asymptotically by a two-point spectral profile supported on l and u with respective masses (u−1)/(u−l)(u-1)/(u-l) and (1−l)/(u−l)(1-l)/(u-l). Consequently, the dispersion condition in Corollary G.8 is non-vacuous if and only if (1−l)(u−1)>π/(2q∗2)−1(1-l)(u-1)>π/(2q_*^2)-1. Using q∗=(1+lu)/(l+u)q_*=(1+ lu)/( l+ u), this is equivalently 2(1+lu)2(l+u−lu)π(l+u)2>1. 2(1+ lu)^2(l+u-lu)π( l+ u)^2>1. For each fixed l∈(0,1)l∈(0,1), the left-hand side is strictly increasing in u>1u>1, so there is a unique critical value uc(l)>1u_c(l)>1 above which the sufficient dispersion condition can be satisfied. Appendix H Additional Experiments and Experimental Details H.1 Synthetic Experiments: Setup and Additional Results Setup. The main-text experiment in Figure 2 (left) uses k=60k=60 classes and dimension d=20d=20. For each class i, we draw μi∼(,Id/d) _i (0,I_d/d), draw the number of training examples independently and uniformly from 1,…,5\1,…,5\, and generate xi,rtr=μi+0.2236εi,rtrx_i,r^tr= _i+0.2236\, _i,r^tr and xi,rte=μi+2.2361εi,rtex_i,r^te= _i+2.2361\, _i,r^te, where all residual vectors are independent (,Id/d)N(0,I_d/d). Test-class proportions are drawn independently from Dirichlet(k)Dirichlet(1_k), and each of the 30 paired seeds uses 120,000120,000 evaluation examples. All methods within a seed use the same training and evaluation data. Optimization and learning-rate selection. We train a bias-free multiclass linear classifier from W0=W_0=0 with cross-entropy for 20,00020,000 updates in float64 and use ηt=η0(t+1)−1/2 _t= _0(t+1)^-1/2. For every optimizer and seed, we sweep the common effective-Frobenius coefficients c∈0.03,0.1,0.3,1,3,10,30c∈\0.03,0.1,0.3,1,3,10,30\, setting η0=c/‖Δ0‖F _0=c/\| _0\|_F, and select c by maximum final evaluation accuracy, with ties broken by lower training loss and then smaller c. Because the same evaluation sample is used for learning-rate selection and reporting, these results are comparisons under the stated discrete sweep rather than untouched-test estimates; some RowGD and RowGDM runs select the lower endpoint of the grid. Additional deterministic and stochastic comparisons. Figure 3 complements the six-optimizer deterministic comparison in Figure 2 (left) in two ways. The left panel adds Signum to give the full deterministic comparison among RowGD, RowGDM, SpecGD, Muon, SignGD, Signum, and Adam, while the right panel compares the corresponding random-reshuffling methods R-RowSGDM, R-Muon, R-Signum, and R-Adam. RowGDM, Muon, and Signum use momentum β=0.9β=0.9; SpecGD and Muon use the exact compact-SVD polar factor; and Adam uses (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999) without bias correction or a numerical-stability constant. For random reshuffling, we use batch size b=1b=1, draw an independent permutation of the finite training set at each epoch, use momentum β=0.99β=0.99 for R-RowSGDM, R-Muon, and R-Signum, and use (β1,β2)=(0.99,0.999)( _1, _2)=(0.99,0.999) for R-Adam. The deterministic panel preserves the Row-over-Spectral and Row-over-coordinate-wise ordering of the main-text experiment, and R-RowSGDM likewise outperforms all three random-reshuffling comparators on every paired seed. R-Adam is included as an empirical comparator; our theoretical stochastic Adam comparison is formulated through AdamProxy. Figure 3: Additional optimizer comparisons on the baseline synthetic model. Left: The deterministic comparison in Figure 2 (left), augmented with Signum. Right: Random-reshuffling comparison of R-RowSGDM, R-Muon, R-Signum, and R-Adam with batch size b=1b=1. Points are paired seeds and gray lines connect methods evaluated on the same dataset. Auxiliary geometries. Figure 4 supplements the main-text optimizer comparison with three reference methods that are not part of our theoretical ordering. Frobenius-normalized GD uses Δt=Gt/‖Gt‖F _t=G_t/\|G_t\|_F, ColumnGD independently normalizes each column of GtG_t in Euclidean norm, and ordinary GD uses Δt=Gt _t=G_t. The figure uses FrobGD as an abbreviation for Frobenius-normalized GD. These methods provide additional finite-dimensional context rather than additional theoretical comparators. Figure 4: Auxiliary optimization geometries. RowGD, FrobGD, ColumnGD, ordinary GD, SpecGD, and SignGD on the same 30 paired baseline datasets, where FrobGD denotes Frobenius-normalized GD. Robustness of the deterministic ordering. To test whether the ordering in Figure 2 (left) depends on the particular baseline configuration, we repeat the same six-optimizer comparison under eleven one-factor-at-a-time variations. We vary the training-noise coefficient from 0.22360.2236 to 0.04470.0447 or 0.44720.4472, the test-noise coefficient from 2.23612.2361 to 1.34161.3416 or 3.13053.1305, the test-class proportions from Dirichlet(k)Dirichlet(1_k) to balanced classes or Dirichlet(0.3k)Dirichlet(0.31_k), the number of training examples per class to exactly one or five, the problem size to (k,d)=(30,20)(k,d)=(30,20) or (120,40)(120,40), and the class-mean distribution from Gaussian to uniform on the unit sphere. In all Gaussian-noise settings, the residual vectors are drawn from (,Id/d)N(0,I_d/d); in the (120,40)(120,40) setting, the training- and test-noise coefficients are 0.31620.3162 and 3.16233.1623, respectively. Each setting uses 30 paired seeds and the same training and learning-rate protocol as the main-text experiment. Across all settings, RowGD and RowGDM have the two highest mean accuracies; each outperforms SignGD and Adam on every paired seed, and each outperforms SpecGD and Muon on all 30 seeds except in the one- and five-training-example settings, where the Row methods win 29 of 30 comparisons. Figures 5–7 show that the main-text ordering is stable across these changes. Figure 5: Robustness to training and test noise. Top: training-noise coefficient 0.04470.0447 (left) and 0.44720.4472 (right), compared with 0.22360.2236 in Figure 2 (left). Bottom: test-noise coefficient 1.34161.3416 (left) and 3.13053.1305 (right), compared with 2.23612.2361 in the main-text setting. In all cases, the residual vectors are drawn from (,Id/d)N(0,I_d/d). Figure 6: Robustness to class proportions and training sample count. Top: balanced test classes (left) and Dirichlet(0.3k)Dirichlet(0.31_k) test proportions (right), compared with Dirichlet(k)Dirichlet(1_k) in Figure 2 (left). Bottom: exactly one (left) and five (right) training examples per class, compared with independent counts drawn uniformly from 1,…,5\1,…,5\ in the main-text setting. Figure 7: Robustness to problem size and class-mean distribution. Top: (k,d)=(30,20)(k,d)=(30,20) (left) and (120,40)(120,40) (right), compared with (60,20)(60,20) in Figure 2 (left). Bottom: class means drawn independently and uniformly from the unit sphere instead of from (,Id/d)N(0,I_d/d). Anisotropic synthetic experiments. We also test the anisotropic predictions of Section 4. Unless otherwise noted, the training and evaluation protocol is the same as in the baseline synthetic experiment above. Figure 8 shows a representative independent-orientation setting with power-law exponents sμ=0.5s_μ=0.5 and sΣ=2s_ =2; across six such settings, RowGD and RowGDM outperform SpecGD, Muon, SignGD, and Adam on all 30/3030/30 paired seeds in every setting, with conservative mean advantages between 2.822.82 and 5.735.73 percentage points. Figure 9 illustrates the coordinate-aligned reversal and recovery phenomenon from Proposition 4.5: in the diagonal setting, RowGDM outperforms Adam on all 30/3030/30 seeds at λ=0.25λ=0.25, Adam outperforms RowGDM on all 30/3030/30 seeds at λ=1λ=1, and applying the same Haar rotation restores the RowGDM-over-Adam ordering on all 30/3030/30 seeds. The coordinate-aligned reversal-and-recovery experiments use 10610^6 training steps to reduce sensitivity to finite-time transients. Figure 8: Independent-orientation anisotropic experiment. Representative setting with sμ=0.5s_μ=0.5 and sΣ=2s_ =2, where RowGD and RowGDM outperform SpecGD, Muon, SignGD, and Adam on all 3030 paired seeds. Figure 9: Coordinate-aligned reversal and recovery. RowGDM outperforms Adam at λ=0.25λ=0.25, Adam outperforms RowGDM at λ=1λ=1, and the RowGDM-over-Adam ordering is restored after applying the same Haar rotation. H.2 Language Model Experiment Model and starting checkpoint. We use a LLaMA-6060M causal language model with 58.0758.07M parameters exactly, consisting of 88 transformer layers with hidden size 512512, intermediate size 13761376, 88 attention heads, and vocabulary size 32,00032,000. The starting checkpoint is obtained by training the full model from scratch on C4 for 11,00011,000 updates with global batch size 512512, sequence length 256256, a peak learning rate of 0.0050.005, cosine decay, and 1,1001,100 warmup steps. This pretraining uses a hybrid optimizer: Muon is applied to the two-dimensional attention and MLP projection matrices, while AdamW is used for the token embedding, language-model head, and one-dimensional parameters. The resulting checkpoint has validation perplexity 28.2350128.23501. For the last-layer experiment, we freeze the entire model body, discard the pretrained output-head weights, zero-initialize the 32,000×51232,000× 512 output head, and train only this 16,384,00016,384,000-parameter matrix. Last-layer training horizon. The main experiment trains the zero-initialized output head for 22,00022,000 updates in bfloat16 using distributed data parallel training on two NVIDIA RTX A6000 GPUs, with global batch size 512512, sequence length 256256, a cosine learning-rate schedule with 1,1001,100 warmup steps, and zero weight decay and gradient clipping. An 11,00011,000-update last-layer run processes 1.101.10B non-padding training tokens, or approximately 18.918.9 tokens per total model parameter, close to the conventional 2020-tokens-per-parameter Chinchilla reference; the 22,00022,000-update horizon approximately doubles this token budget to 2.202.20B tokens. We use the longer horizon to examine a later-training regime in which transient optimization effects are reduced and the trajectories may be closer to the limiting-direction behavior studied theoretically, without assuming that directional convergence has occurred by 22,00022,000 updates. The Chinchilla comparison is only a token-budget reference, since the model body is frozen and only the output head is optimized in this experiment. Horizon check. We also run a separate 11,00011,000-update comparison with its own cosine schedule ending at 11,00011,000 steps and learning rates selected independently at that horizon. The selected learning rates are 0.006250.00625 for RowSGDM, 0.00450.0045 for Muon, and 0.0050.005 for Adam, yielding final monitoring-set perplexities 28.587128.5871, 28.637328.6373, and 28.591428.5914, respectively. Thus, RowSGDM is already numerically best at approximately the Chinchilla token-budget reference, although its advantage over Adam is only 0.0152%0.0152\% in this single-seed comparison. The 22,00022,000-step experiment therefore probes whether the qualitative ordering persists and becomes clearer at a later training horizon rather than relying on the 11,00011,000-step difference alone. Data and evaluation. We use the Hugging Face allenai/c4 dataset in streaming mode and tokenize with t5-base, truncating and padding sequences to length 256256. For each training seed, RowSGDM, Muon, and Adam receive the same shuffled stream and batch sequence. This streaming shuffle is not the epoch-wise permutation of a fixed finite dataset used by the random-reshuffling optimizers in Section 2. The validation trajectory in Figure 2 (right) is obtained from a monitoring set evaluated every 1,0001,000 updates and containing 7,625,7457,625,745 valid next-token targets. Final evaluation uses a separate set of 237,723237,723 C4 validation documents containing 46,186,88246,186,882 valid targets and having zero document-hash overlap with the monitoring set. The held-out set is not used for learning-rate selection, early stopping, or checkpoint selection. NLL is averaged over the exact number of valid non-padding shifted targets, and perplexity is its exponential. Optimizers and learning rates. RowSGDM uses momentum 0.90.9, row-wise RMS normalization, multiplier 0.20.2, and numerical stabilizer ϵ=10−8ε=10^-8. Muon uses momentum 0.950.95, five Newton–Schulz iterations, multiplier 0.20.2, and zero weight decay. Adam uses (β1,β2)=(0.9,0.95)( _1, _2)=(0.9,0.95), ϵ=10−8ε=10^-8, and zero weight decay. These implementations are practical counterparts of the RowSGDM, exact-SVD Muon, and zero-stability-constant Adam considered in our theory. For the 22,00022,000-step experiment, learning rates are selected using the monitoring set: we sweep 0.004,0.005,0.006,0.007\0.004,0.005,0.006,0.007\ for RowSGDM and Adam and 0.0025,0.003,0.0035,0.004,0.0045,0.005,0.0055,0.006\0.0025,0.003,0.0035,0.004,0.0045,0.005,0.0055,0.006\ for Muon, selecting 0.0050.005, 0.0050.005, and 0.0040.004, respectively. Boundary-alignment diagnostic. We additionally test whether the performance ordering is accompanied by the boundary geometry predicted by our analysis. For each valid held-out next-token position, let ht∈ℝ512h_t ^512 denote the frozen hidden representation immediately before the output head and let yty_t denote its next-token class. For class i, we estimate μ^i=ni−1∑t:yt=iht μ_i=n_i^-1 _t:y_t=ih_t and retain the 5,0935,093 classes with ni≥1000n_i≥ 1000. We uniformly sample a fixed set P of 1,000,0001,000,000 unordered class pairs and use the same pairs for every optimizer and seed. For a trained output head with rows wi⊤w_i , define u^ijW:=wi−wj‖wi−wj‖2,u^ijμ:=μ^i−μ^j‖μ^i−μ^j‖2, u^W_ij:= w_i-w_j\|w_i-w_j\|_2, u^μ_ij:= μ_i- μ_j\| μ_i- μ_j\|_2, and measure D^avg-bdry(W):=1||∑i,j∈‖u^ijW−u^ijμ‖2,C^bdry(W):=1||∑i,j∈⟨u^ijW,u^ijμ⟩. D_avg -bdry(W):= 1|P| _\i,j\ \| u^W_ij- u^μ_ij\|_2, C_bdry(W):= 1|P| _\i,j\ u^W_ij, u^μ_ij . Lower D^avg-bdry D_avg -bdry and higher C^bdry C_bdry indicate closer alignment between learned classifier boundaries and empirical class-mean differences. Results. Consistent with the final validation-perplexity ordering in Figure 2 (right), RowSGDM achieves lower held-out NLL and perplexity than both Muon and Adam on each of the five paired seeds. RowSGDM also has the smallest boundary discrepancy and the largest boundary cosine on every seed, so its performance advantage is accompanied by closer alignment between learned classifier boundaries and empirical class-mean differences. Table 1 summarizes the five-seed held-out results. Optimizer NLL ↓ PPL ↓ D^avg-bdry↓ D_avg -bdry C^bdry↑ C_bdry RowSGDM 3.35503±0.000053.35503± 0.00005 28.6466±0.001328.6466± 0.0013 1.13194±0.000041.13194± 0.00004 0.35652±0.000050.35652± 0.00005 Muon 3.35699±0.000043.35699± 0.00004 28.7027±0.001228.7027± 0.0012 1.13807±0.000061.13807± 0.00006 0.35018±0.000070.35018± 0.00007 Adam 3.35759±0.000023.35759± 0.00002 28.7199±0.000628.7199± 0.0006 1.13350±0.000081.13350± 0.00008 0.35481±0.000090.35481± 0.00009 Table 1: Held-out performance and empirical boundary alignment for last-layer-only LLaMA training. Values are mean ± sample standard deviation across five paired training seeds. The mean paired RowSGDM-minus-Muon NLL difference is −0.001957-0.001957, with a descriptive 95%95\% Student-t interval [−0.002058,−0.001857][-0.002058,-0.001857], and the RowSGDM-minus-Adam difference is −0.002555-0.002555, with interval [−0.002621,−0.002489][-0.002621,-0.002489]. The mean reductions in D^avg-bdry D_avg -bdry are 0.0061370.006137 relative to Muon and 0.0015580.001558 relative to Adam. Together, the validation, held-out, and boundary-alignment results are consistent with the qualitative ordering and geometric mechanism predicted by our theory. LLaMA-130M experiment. We additionally repeat the last-layer comparison on a separately C4-pretrained LLaMA-130M backbone, whose actual parameter count is 134.1134.1M, with 1212 transformer layers and hidden size 768768. The tokenizer, sequence length, global batch size, evaluation sets, optimizer definitions, and five paired seeds are the same as above. We train the zero-initialized 32,000×76832,000× 768 output head for 55,00055,000 updates, with each selected run processing approximately 5.475.47B valid next-token targets, and a common full-horizon learning-rate sweep selects 0.0028280.002828 for all three optimizers. RowSGDM attains the lowest held-out perplexity on every seed, with five-seed mean ± sample standard deviation 22.3797±0.001422.3797± 0.0014, compared with 22.4096±0.003022.4096± 0.0030 for Muon and 22.4153±0.001022.4153± 0.0010 for Adam. RowSGDM also yields the smallest boundary discrepancy on every seed, with mean D^avg-bdry=1.19051 D_avg -bdry=1.19051, compared with 1.196641.19664 for Adam and 1.200281.20028 for Muon, providing a larger-model check of the same qualitative advantage. H.3 Spectral Exponents for Language Model Representation We empirically examine whether a fitted class-mean spectral exponent below one can occur in language-model representations, as in the regime sμ<1s_μ<1 of Theorem 4.2. We also measure class-conditional within-class covariance spectral exponents as descriptive statistics of within-class anisotropy. These measurements are finite-dimensional spectral summaries and are not intended as a direct test of the full probabilistic model underlying the theorem. Experimental setup. We analyze final pre-head representations from OLMoE-1B-7B-0125 (Muennighoff et al., 2025), Qwen3-30B-A3B (Yang et al., 2025), and AMUSE-720M (Kim et al., 2026b), all with hidden dimension d=2048d=2048. For each model, we collect exactly 20,000,00020,000,000 valid same-document next-token positions from the English validation split of C4. At a valid position, let hit∈ℝdh_it ^d denote the final normalized hidden representation when the next-token class is i, and let nin_i be the number of observed positions from class i. We define the empirical class mean by μ^i:=1ni∑t=1nihit. μ_i:= 1n_i _t=1^n_ih_it. (442) Class-mean spectral exponent. Our primary mean-spectrum analysis assigns equal weight to every observed next-token class. Let ℐμ:=i:ni≥1,K:=|ℐμ|,I_μ:=\i:n_i≥ 1\, K:=|I_μ|, and define μ¯^:=1K∑i∈ℐμ^i,Γ^:=1K∑i∈ℐμ(μ^i−μ¯^)(μ^i−μ¯^)⊤. μ:= 1K _i _μ μ_i, := 1K _i _μ( μ_i- μ)( μ_i- μ) . (443) Thus Γ is class balanced rather than token-frequency weighted; in particular, every observed class contributes one empirical class mean. If λ^1μ≥⋯≥λ^2048μ>0 λ_1^μ≥·s≥ λ_2048^μ>0 are the eigenvalues of Γ , we estimate the mean exponent by the unweighted pointwise OLS fit logλ^rμ=a−s^μlogr+εr,r=1,…,2048. λ_r^μ=a- s_μ r+ _r, r=1,…,2048. (444) Each integer rank therefore contributes one regression observation; the fit is not log binned or rank weighted. Within-class covariance spectral exponents. For classes with sufficiently many observations, we form the within-class sample covariance Σ^i:=1ni−1∑t=1ni(hit−μ^i)(hit−μ^i)⊤ _i:= 1n_i-1 _t=1^n_i(h_it- μ_i)(h_it- μ_i) (445) and fit logλ^i,rΣ=ai−s^Σ,ilogr+ei,r,r=1,…,2048. λ_i,r =a_i- s_ ,i r+e_i,r, r=1,…,2048. (446) The reliable-class threshold is ni≥8192n_i≥ 8192 for OLMoE and AMUSE and ni≥16384n_i≥ 16384 for Qwen3. This leaves 218218, 211211, and 120120 classes with reliable within-class covariance estimates, respectively. Table 2: Full-rank pointwise-OLS spectral exponents at the three final checkpoints. K is the number of observed classes used for the class-mean covariance, and KΣK_ is the number of reliable classes used for the within-class covariance summary. Model K s^μ s_μ KΣK_ Mean s^Σ,i s_ ,i Median Max OLMoE-1B-7B-0125 47,19047,190 0.9577350.957735 218218 0.9873390.987339 0.9735370.973537 1.3743661.374366 Qwen3-30B-A3B 72,57772,577 0.9327800.932780 120120 1.0579241.057924 1.0454831.045483 1.2581631.258163 AMUSE-720M 49,34849,348 0.9853660.985366 211211 1.0547931.054793 1.0459651.045965 1.4020451.402045 Results. For the three final checkpoints examined here, the primary class-mean point estimates in Table 2 all satisfy s^μ<1 s_μ<1: 0.9577350.957735 for OLMoE, 0.9327800.932780 for Qwen3, and 0.9853660.985366 for AMUSE. The within-class covariance exponents are more heterogeneous and can exceed one, as reflected by the maxima in the table. OLMoE across annealing. We additionally apply the s^μ s_μ estimator to six published OLMoE snapshots spanning 00–101101B added annealing tokens. The resulting estimates are 0.9567270.956727, 0.9577920.957792, 0.9565690.956569, 0.9565300.956530, 0.9583590.958359, and 0.9577350.957735. Thus, every sampled estimate is below one, the total range is only 0.0018300.001830, and there is no monotone trend across the sampled snapshots.