Paper deep dive
Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training
Zixuan Gong, Zeyu Gan, Jiaye Teng, Yong Liu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Matrix optimizers have emerged as a promising direction, with Muon standing out as a prominent design. Revisiting Muon through its full-Gram representation, we observe that it jointly processes marginal-scale and interaction information. This opens an alternative way to organize geometric information hierarchically, motivating the Normalize-Then-Precondition framework. Specifically, it first uses diagonal-Gram information to construct a marginally normalized update, then applies spectral preconditioning to its directional interaction geometry. Building on this framework, we develop NormPre with NormPre-G and NormPre-L adopting global and localized spectral preconditioning, grounded in spectral-norm steepest descent and a regularized formulation followed by leading mode selection, respectively. To enable large-scale training, NormPre-G uses Newton-Schulz iterations and NormPre-L employs randomized sketching to approximate the leading interaction eigenspace. Theoretically, we establish $\mathcal{O}(T^{-1/2})$ convergence guarantees for simplified versions of NormPre. Across extensive pretraining experiments on GPT-2 Small, LLaMA and Qwen3, both variants consistently outperform AdamW, Muon and MANO under matched training budgets. Further efficiency and spectral analyses reveal the complementary strengths of two variants and characterize their performance-efficiency trade-off. We open-source our code through a GitHub repository at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.36692v1
- Canonical: https://arxiv.org/abs/2609.36692v1
Trouble viewing inline? Open PDF directly →
Full Text
122,288 characters extracted from source content.
Expand or collapse full text
Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training Zixuan Gong Affiliation: Gaoling School of Artificial Intelligence Affiliation: Renmin University of China Email: zxgong@ruc.edu.cn Zeyu Gan Affiliation: Gaoling School of Artificial Intelligence Affiliation: Renmin University of China Email: zygan@ruc.edu.cn Jiaye Teng Affiliation: School of Statistics and Management Affiliation: Shanghai University of Finance and Economics Email: tengjiaye@sufe.edu.cn Yong Liu †thanks: Corresponding author. Affiliation: Gaoling School of Artificial Intelligence Affiliation: Renmin University of China Email: liuyonggsai@ruc.edu.cn Abstract Matrix optimizers have emerged as a promising direction, with Muon standing out as a prominent design. Revisiting Muon through its full-Gram representation, we observe that it jointly processes marginal-scale and interaction information. This opens an alternative way to organize geometric information hierarchically, motivating the Normalize-Then-Precondition framework. Specifically, it first uses diagonal-Gram information to construct a marginally normalized update, then applies spectral preconditioning to its directional interaction geometry. Building on this framework, we develop NormPre with NormPre-G and NormPre-L adopting global and localized spectral preconditioning, grounded in spectral-norm steepest descent and a regularized formulation followed by leading mode selection, respectively. To enable large-scale training, NormPre-G uses Newton-Schulz iterations and NormPre-L employs randomized sketching to approximate the leading interaction eigenspace. Theoretically, we establish (T−1/2)O(T^-1/2) convergence guarantees for simplified versions of NormPre. Across extensive pretraining experiments on GPT-2 Small, LLaMA and Qwen3, both variants consistently outperform AdamW, Muon and MANO under matched training budgets. Further efficiency and spectral analyses reveal the complementary strengths of two variants and characterize their performance-efficiency trade-off. We open-source our code through a GitHub repository at https://github.com/zx-gong/NormPre. 1 Introduction Large language model (LLM) training places increasing demands on optimizer performance, efficiency, and scalability. Recent optimizers address these demands by exploiting richer parameter structure, from individual coordinates (Kingma and Ba, 2015; Loshchilov and Hutter, 2019; Chen et al., 2023; Liu et al., 2024; Liang et al., 2026) and parameter blocks (Zhang et al., 2025; Wang et al., 2025) to full matrices (Gupta et al., 2018; Vyas et al., 2025; Jordan et al., 2024; Liu et al., 2025; Gu and Xie, 2026; Deng et al., 2026; Xu et al., 2026). More specifically, orthogonalization and normalization have emerged as prominent matrix-level techniques. On the orthogonalization front, Muon applies a matrix-sign transformation to momentum via Newton-Schulz iterations for rapid convergence (Jordan et al., 2024; Liu et al., 2025). Alternatively, other recent methods use row or column normalization to construct matrix updates. For example, MANO combines momentum projection with alternating normalization (Gu and Xie, 2026), and MOGA derives such normalized updates from mean-normalized operator norms under a steepest-descent perspective (Xu et al., 2026). Orthogonalization and normalization may appear to follow different routes, yet their Gram representations reveal a shared structure. For a matrix update X, the transformations take the forms Φ=(⊤)−1/2X⏟Full-Gram Representation,Ψ=[Diag(diag(⊤))]−1/2X⏟Diagonal-Gram Representation, =( [rgb]0,0.4961,1X )^-1/2X_Full-Gram Representation, = [ [rgb]1,0.6484,0Diag(diag(X )) ]^-1/2X_Diagonal-Gram Representation, where Diag(diag(⋅))Diag(diag(·)) retains only the diagonal entries, and the inverse-square-root expressions require full row rank for Φ and nonzero rows for Ψ , respectively. Muon jointly processes marginal scales and cross-row interactions encoded in the full Gram matrix, while normalization draws solely on the Gram diagonal to equalize row norms without altering pairwise angles. As an extreme example, consider fixed row directions with one row norm of 10410^4 and all others of 10−310^-3. As this scale gap increases further, the leading Gram eigenvector approaches the largest row’s coordinate axis, even if the other rows are strongly correlated. The principal mode thus becomes dominated by marginal scales despite unchanged directional interactions. By removing this scale weighting, diagonal-Gram normalization provides a reference update for processing directional interactions. Together, these observations open an alternative way to organize the available geometric information: Can we construct a normalized update from scales and refine it via directional interactions? While recent methods apply normalization to gradients or momentum to improve the numerical conditioning of orthogonalization (Ma et al., 2024; Chang et al., 2026), a systematic framework for formalizing the hierarchical organization has yet to be established. In particular, it remains unclear whether the normalized update should undergo a full-spectrum transformation or serve as a reference for more targeted spectral modifications. To address this gap, we formalize the hierarchical organization as the Normalize-Then-Precondition framework. The marginally normalized update retains a directional interaction spectrum encoded by Γ=ΨΨ⊤ = . To exploit this spectrum, one choice is global spectral preconditioning, which performs orthogonalization of Ψ and solves a steepest descent problem under the spectral norm constraint. Another choice is to treat the normalized update Ψ as a reference and minimize the changes needed to satisfy the spectral constraint, seeking the closest update within the unit spectral norm ball. This preference is formalized by regularized spectral steepest descent, whose solution clips singular values above one and leaves the others unchanged. A spectral budget r further restricts the transformation to at most r leading super-unit modes, giving localized spectral preconditioning. Here, r controls the selected subspace dimension for spectral transformation and the final update remains dense. (a) GPT-2 Small on OWT (b) LLaMA-130M on C4 (c) LLaMA-350M on C4 (d) LLaMA-1.3B on C4 Figure 1: GPT-2 Small on OpenWebText and LLaMA Models on C4. Training and validation loss for GPT-2 Small and LLaMA-130M, 350M, 1.3B with AdamW, Muon, MANO, NormPre-G and NormPre-L. Dark and light curves denote 50-step moving averages and raw training trajectories. Building on this framework, we propose NormPre, a family of matrix optimizers comprising NormPre-G and NormPre-L, which realize marginal normalization followed by global and localized spectral preconditioning, respectively. Both variants combine relaxed tangent momentum processing with alternating row/column normalization and consistent update RMS scaling. Specifically, NormPre-G approximates the full-spectrum transformation via Newton-Schulz iterations. For NormPre-L, the leading interaction eigenspace is obtained through exact extraction or scalable randomized sketching. We also establish (T−1/2)O(T^-1/2) convergence guarantees for simplified NormPre-G and exact NormPre-L, extending them to sketch-based NormPre-L under controlled approximation error. Empirically, our extensive pretraining evaluations span GPT-2 Small on OpenWebText, LLaMA models up to 1.3B on C4, and Qwen3 models up to 1.7B on Pile. Under matched training budgets, both variants consistently outperform established baselines that represent distinct optimization paradigms: coordinate-wise adaptation (AdamW), matrix orthogonalization (Muon), and normalization (MANO). Furthermore, our spectral dynamics and efficiency analyses highlight that NormPre-L with localized preconditioning delivers strong optimization gains at lower cost and NormPre-G with global preconditioning further reduces validation loss, providing practitioners with flexible choices to balance optimization performance and computational cost. Figure 2: Geometric Workflow of the NormPre Optimizer (Algorithm 1). Our main contributions are summarized as follows. (a) The Normalize-Then-Precondition Framework. Conceptually, departing from joint geometric transformations, we establish a systematic framework to hierarchically construct a normalized update from marginal scales and refine it via interaction information. Unifying full-spectrum and targeted schemes, we formulate global and localized spectral preconditioning as a solution to spectral-norm steepest descent and a regularized formulation with leading mode selection, respectively (Section 2). (b) The NormPre Optimizer. Algorithmically, we instantiate this framework as the NormPre optimizer, featuring alternating row-column normalization and spectral preconditioning. We provide scalable implementations using Newton-Schulz iterations for global preconditioning (NormPre-G) and randomized sketching for localized eigenspace extraction (NormPre-L), together with theoretical (T−1/2)O(T^-1/2) convergence guarantees for simplified variants (Theorem 1 and Corollary 1). (c) Optimization Performance and Trade-offs. Empirically, we demonstrate the superior optimization performance of NormPre across the evaluated architectures and model scales, consistently outperforming established baselines (AdamW, Muon, MANO). Furthermore, we characterize the performance-efficiency trade-off between its two variants, providing flexible choices to balance optimization gains and computational cost (Section 4). 2 Geometric Foundations of Normalize-Then-Precondition This section formalizes the Normalize-Then-Precondition framework. We begin by establishing a full-Gram view of row-orthogonalized updates (Section 2.1) and derive the normalized update with the marginal-scale information encoded in the Gram diagonal (Section 2.2). We finally formulate different spectral preconditioning realizations (Section 2.3). Throughout this section, X∈ℝm×nX ^m× n denotes a matrix-valued update signal. We write its thin SVD as X=UΣV⊤X=U V . We present the row-side analysis and the column-side counterpart follows by applying the same construction to X⊤X . 2.1 Orthogonalized Updates and Full-Gram Transformation Muon (Jordan et al., 2024) orthogonalizes a momentum matrix X by replacing it with the nearest semi-orthogonal matrix. For X∈ℝm×nX ^m× n with m≤nm≤ n, row orthogonalization is defined by Φ∈argminO‖O−X‖F:OO⊤=Im. ∈ _O\\|O-X\|_F:O =I_m\. (1) One solution is given by the matrix sign function, Φ=msign(X)=UV⊤. =msign(X)=UV . Accordingly, the matrix-sign update transforms the singular-value matrix Σ into identity, yielding ΦΦ⊤=Im =I_m. In practice, Muon approximates this orthogonalization efficiently using Newton-Schulz iterations. We next identify an equivalent full-Gram form of the same update, showing how Muon’s orthogonalization depends on the full row Gram matrix. Full-Gram Representation of Muon. If XX⊤≻0X 0, the matrix-sign update admits the full-Gram representation Φ=msign(X)=(X⊤)−1/2X. =msign(X)=(X )^-1/2X. (2) Let xi⊤x_i denote the i-th row of X. The entries of the full row Gram matrix satisfy (XX⊤)ii=‖xi‖22,(X )_i=\|x_i\|_2^2, and (XX⊤)ij=⟨xi,xj⟩(X )_ij= x_i,x_j for i≠ji≠ j. Its diagonal entries encode the marginal scales of individual rows, whereas its off-diagonal entries encode their pairwise interactions. Together, these two components determine the inverse-square-root operator in Equation 2, which acts on the entire singular spectrum of X. Thus, Muon jointly utilizes marginal-scale and cross-row interaction information through the full row Gram matrix. 2.2 Diagonal-Gram Normalization Within the joint geometry, marginal-scale information is structurally simpler, since each row scale depends only on its corresponding row. This motivates a natural question: is it possible to first use this simpler marginal-scale information to construct a valid matrix update? To answer this question, we isolate the marginal-scale information encoded in the diagonal of XX⊤X . Since (XX⊤)ii=‖xi‖22(X )_i=\|x_i\|_2^2, define the diagonal matrix of row scales as D(X):=Diag(diag(XX⊤))1/2=Diag(‖x1‖2,…,‖xm‖2), D(X):=Diag (diag(X ) )^1/2=Diag (\|x_1\|_2,…,\|x_m\|_2 ), where diag(⋅)diag(·) extracts the diagonal entries as a vector and Diag(⋅)Diag(·) constructs a diagonal matrix. Diagonal-Gram Representation of Marginal Normalization. If X has no zero rows, the marginally normalized update admits the diagonal-Gram representation Ψ=D(X)−1X=Diag(diag(X⊤))−1/2X. =D(X)^-1X=Diag (diag(X ) )^-1/2X. (3) Equation 3 provides the diagonal-Gram counterpart of the Muon transformation in Equation 2. Muon constructs its orthogonalized update from the inverse square root of the full row Gram matrix, while Ψ uses only its diagonal component to normalize the marginal scales. This transformation coincides with the core normalization used by normalization-based optimizers (Gu and Xie, 2026; Xu et al., 2026; Deng et al., 2026; Pethick et al., 2025a; Glentis et al., 2025). Define the normalized interaction matrix as Γ(X):=ΨΨ⊤ (X):= . By construction, Γ(X)ii=1 (X)_i=1 and its off-diagonal entries Γ(X)ij=⟨xi,xj⟩/(‖xi‖2‖xj‖2) (X)_ij= x_i,x_j /(\|x_i\|_2\|x_j\|_2) encode the cosine similarities between rows. 2.3 Spectral Interaction Preconditioning Hierarchical Geometric Construction. Beyond serving as an effective standalone update, the normalization Ψ retains directional correlations in Γ(X) (X), providing a natural foundation for further spectral preconditioning. Having processed the marginal scales, we now study two realizations of this preconditioning stage (Figure 3): global spectral preconditioning, which performs full-spectrum transformation, and localized spectral preconditioning, which retains Ψ as a reference and concentrates the transformation on selected interaction modes. Let Ψ=U~Σ~V~⊤ = U V be a thin SVD with singular values σ~1≥⋯≥σ~m≥0 σ_1≥·s≥ σ_m≥ 0. Accordingly, Γ(X)=ΨΨ⊤=U~ΛU~⊤ (X)= = U U , where Λ=Σ~2=Diag(λ1,…,λm) = ^2=Diag( _1,…, _m) and λi=σ~i2 _i= σ_i^2. The columns u~i u_i of U~ U define the corresponding spectral interaction directions. Figure 3: Spectral Interaction Preconditioning. Global Spectral Preconditioning. A direct method is to solve the spectral steepest descent problem for the normalized update Ψ (Bernstein and Newhouse, 2024), TG∈argmax‖T‖op≤1⟨Ψ,T⟩F.T^G∈ _\|T\|_op≤ 1 ,T _F. (4) A solution is the row-orthogonalized update obtained by performing transformation in Equation 2, TG=PGΨ=msign(Ψ),T^G=P^G =msign( ), with the global preconditioner PG=Γ(X)−1/2=U~Λ−1/2U~⊤P^G= (X)^-1/2= U ^-1/2 U when Γ(X)≻0 (X) 0. Along the i-th spectral interaction direction, PGP^G applies the scaling factor λi−1/2=σ~i−1 _i^-1/2= σ_i^-1, mapping the corresponding singular value σ~i σ_i to one. Localized Spectral Preconditioning. Another possible method starts from the observation that Ψ already serves as a marginally normalized dense update. This motivates retaining Ψ as the reference and considering the regularized spectral steepest descent problem T+:=argmin‖T‖op≤112‖T−Ψ‖F2=argmax‖T‖op≤1⟨Ψ,T⟩F−12‖T‖F2.T_+:= _\|T\|_op≤ 1 12\|T- \|_F^2= _\|T\|_op≤ 1 \ ,T _F- 12\|T\|_F^2 \. (5) Its solution is T+=P+ΨT_+=P_+ . Defining the active set :=i:λi>1A:=\i: _i>1\, the corresponding preconditioner is P+=Im+U~(Λ−1/2−I)U~⊤P_+=I_m+ U_A( _A^-1/2-I) U_A (see Proposition 1 in Appendix D.1). Thus, the solution modifies only the active spectral modes (clipping to one) while leaving the remaining modes unchanged. To further localize the transformation, we impose a spectral budget r and let ⊆C index the minr,|| \r,|A|\ largest active eigenvalues (see Proposition 2 in Appendix D.1). The resulting update is TL=PLΨ,T^L=P^L , with the localized preconditioner PL=Im+U~(Λ−1/2−I)U~⊤P^L=I_m+ U_C( _C^-1/2-I) U_C . 3 NormPre This section instantiates the Normalize-Then-Precondition framework as the NormPre optimizer, with variants NormPre-G and NormPre-L using global and localized spectral preconditioning, respectively. 3.1 The NormPre Optimizer Let Wt∈ℝm×nW_t ^m× n denote a matrix-valued parameter with stochastic gradient Gt=∇WtℒtG_t= _W_tL_t, and let Mt=μMt−1+GtM_t=μ M_t-1+G_t denote its first-order momentum. Following Sections 2.2 and 2.3, Algorithm 1 summarizes the NormPre optimization procedure. Algorithm 1 The NormPre Optimizer 1: Layer weight Wt∈ℝm×nW_t ^m× n, learning rate ηt _t, momentum coefficient μ, weight decay coefficient λwd _wd, optimizer variant (NormPre-G or NormPre-L). 2: Initialize M0←M_0 0 3: for each step do 4: Gt←∇WtℒtG_t← _W_tL_t 5: Mt←μMt−1+GtM_t←μ M_t-1+G_t 6: kt←tmod2k_t← t 2 ⊳ Alternating Row/Column Orientation 7: Xt←Wt,kt(Mt)X_t _W_t,k_t(M_t) ⊳ Relaxed Tangent Momentum 8: Ψt←D(Xt)−1Xt _t← D(X_t)^-1X_t ⊳ Diagonal-Gram Normalization 9: if NormPre-G then 10: Tt←PtGΨt=msign(Ψt)T_t← P^G_t _t=msign( _t) 11: else if NormPre-L then 12: (U~t,Λt)←Eigt(ΨtΨt⊤) ( U_C_t, _C_t ) _C_t( _t _t ) ⊳ Eigenspace Extraction 13: Tt←PtLΨt=(I+U~t(Λt−1/2−I)U~t⊤)ΨtT_t← P^L_t _t=(I+ U_C_t( _C_t^-1/2-I) U_C_t ) _t 14: end if⊳ Spectral Interaction Preconditioning 15: Wt+1←Wt−ηt(ℛkt(Tt)+λwdWt)W_t+1← W_t- _t (R_k_t(T_t)+ _wdW_t ) ⊳ Consistent Update RMS 16: end for The remaining optimizer-level components in Algorithm 1 specify how the update signal is constructed across matrix orientations and how the final matrix update is rescaled to a consistent RMS. Alternating Row/Column Orientation. We alternate the active orientation between rows and columns across optimization steps, allowing the row-side construction in Section 2 to apply symmetrically to both matrix axes. Specifically, let kt=tmod2k_t=t 2 with 0(A)=AO_0(A)=A and 1(A)=A⊤O_1(A)=A , and define W¯t:=kt(Wt) W_t:=O_k_t(W_t) and M¯t:=kt(Mt) M_t:=O_k_t(M_t). Relaxed Tangent Momentum. We distinguish the row-wise momentum components orthogonal and parallel to the current parameter direction (i.e., tangent and radial components). These components primarily govern changes in parameter direction and scale, respectively. Let w¯t,i w_t,i and m¯t,i m_t,i denote the i-th rows of W¯t W_t and M¯t M_t, respectively. For each row, we define xt,i:=m¯t,i−⟨m¯t,i,w¯t,i⟩w¯t,i.x_t,i:= m_t,i- m_t,i, w_t,i w_t,i. This formulation recovers strict tangent projection when ‖w¯t,i‖2=1\| w_t,i\|_2=1. For general weights, this relaxed form preserves the tangent component while retaining a norm-dependent radial contribution. Collecting these rows yields the input signal Xt:=Wt,kt(Mt)X_t:=T_W_t,k_t(M_t) for our framework. Consistent Update RMS. Following the Muon convention (Liu et al., 2025), we set the target RMS of matrix updates to 0.20.2, facilitating shared hyperparameters and direct comparison with AdamW and Muon. For update T∈ℝm×nT ^m× n with RMS(T):=‖T‖F/mnRMS(T):=\|T\|_F/ mn, we define ℛk(T):=0.2⋅k−1(T)/RMS(T)R_k(T):=0.2·O_k^-1(T)/RMS(T). Here, k−1O_k^-1 maps the update from the active orientation back to the original parameter layout and the rescaling ensures an update RMS of 0.20.2 across matrix parameters. 3.2 Implementation of Spectral Preconditioning We now detail the spectral preconditioning step in Algorithm 1, implemented via Newton-Schulz iteration for NormPre-G and eigenspace extraction for NormPre-L. NormPre-G: Newton-Schulz Implementation. For the global realization, we approximate Tt=msign(Ψt)T_t=msign( _t) using the Newton-Schulz iteration adopted by Muon (Jordan et al., 2024; Liu et al., 2025). The iteration acts on the normalized update Ψt _t and maps its singular values toward one, realizing full-spectrum preconditioning. We use five Newton-Schulz iterations throughout our experiments. NormPre-L: Exact and Sketch Implementations. For the localized realization, the preconditioner depends on a selected eigenspace of the interaction matrix Γt=ΨtΨt⊤ _t= _t _t . • Exact Eigenspace Extraction. The exact implementation computes the full eigendecomposition of Γt _t to identify the up to r largest active eigenvalues (λt,i>1 _t,i>1). The selected eigenpairs (U~t,Λt)( U_C_t, _C_t) are then used in the localized preconditioner. • Sketch-Based Eigenspace Extraction. For scalable training, we approximate the leading interaction eigenspace with a randomized sketch (Halko et al., 2011). The sketch builds a low-dimensional subspace from Ψt _t and applies Rayleigh–Ritz extraction to estimate the leading eigenpairs. Modes with λ^t,i>1 λ_t,i>1 are retained within the rank budget r and used in the localized preconditioner. The complete procedure is given in Algorithm 2, with details in Appendix B. Remark 1 (Computational Complexity). Muon and NormPre-G have the same computational complexity (mn+qmns)O(mn+qmns), where s=minm,ns= \m,n\. Both use the same q-step Newton-Schulz transformation ((qmns)O(qmns)), while the additional diagonal-Gram normalization in NormPre-G contributes only (mn)O(mn). For NormPre-L, the Exact implementation requires (m2n+m3)O(m^2n+m^3) computation to form and decompose the full interaction matrix. The Sketch-based implementation reduces this cost to ((p+1)mnℓ+(m+n)ℓ2+ℓ3)O\! ((p+1)mn +(m+n) ^2+ ^3 ) by restricting eigenspace extraction to a subspace of dimension ℓ=minm,r+o = \m,r+o\, ensuring scalability for ℓ≪m m (see Appendix C for detailed analysis). In the following, we provide the convergence guarantee for NormPre. Theorem 1 (Convergence of NormPre without Momentum). Suppose ℒL is L-smooth and lower bounded by ℒinfL_ and let Δ0:=ℒ(W0)−ℒinf _0:=L(W_0)-L_ . Under a fixed orientation, let ϕt,i _t,i denote the angle between gradient g¯t,i g_t,i and weight w¯t,i w_t,i, and ϕt,j,i′φ _t,j,i the angle between normalized row ψt,j _t,j and weight w¯t,i w_t,i. Suppose sinϕt,i≥γ>0 _t,i≥γ>0 and (∑jcos2ϕt,j,i′)1/2≤γ′( _j ^2φ _t,j,i)^1/2≤γ for all t,it,i. Define the maximum radial ratio νt:=maxi‖g¯t,i−xt,i‖2/‖xt,i‖2 _t:= _i\| g_t,i-x_t,i\|_2/\|x_t,i\|_2 and the spectral factor χt:=σ~t,1/σ~t,m _t:= σ_t,1/ σ_t,m for NormPre-G (assuming Γt≻0 _t 0 for all t), or χt:=σ~t,1 _t:= σ_t,1 for NormPre-L (Exact). Assume there exists ϵ>0ε>0 such that max0≤t≤Tνtγ′χt≤1−ϵ _0≤ t≤ T _tγ _t≤ 1-ε. For any constant C>0C>0, choosing η=C/T+1η=C/ T+1 yields min0≤t≤T‖∇ℒ(Wt)‖F≤1T+1(Δ0maxm,n1/2ϵγC+LCmaxm,n3/22ϵγ). _0≤ t≤ T\| (W_t)\|_F≤ 1 T+1 ( _0 \m,n\^1/2εγ C+ LC \m,n\^3/22εγ ). Remark 2. Theorem 1 analyzes deterministic NormPre-G and NormPre-L without momentum, RMS scaling, or weight decay. The condition max0≤t≤Tνtγ′χt≤1−ϵ _0≤ t≤ T _tγ _t≤ 1-ε is sufficient but not necessary for convergence. By bounding the product of the radial ratio (νt _t), geometric non-orthogonality (γ′γ ), and spectral factor (χt _t), this condition controls the worst-case reduction in the first-order gradient signal arising from the radial interaction in the refined update. For any constant C>0C>0, choosing η=C/T+1η=C/ T+1 yields an (T−1/2)O(T^-1/2) convergence rate. This bound is minimized at η⋆=2Δ0Lmaxm,n(T+1)η = 2 _0L \m,n\(T+1), yielding min0≤t≤T‖∇ℒ(Wt)‖F≤maxm,n2LΔ0ϵγT+1 _0≤ t≤ T\| (W_t)\|_F≤ \m,n\ 2L _0εγ T+1. We defer the proof of Theorem 1 and the Sketch analysis (Corollary 1) to Appendices D.2.2 and D.2.3, respectively. 4 Experiments This section evaluates NormPre across diverse pretraining settings (Section 4.1), analyzes its efficiency and spectral dynamics (Section 4.2) and characterizes the performance-efficiency trade-off between NormPre-G and NormPre-L (Section 4.3). 4.1 Scaling Experiments Experimental Setup. We evaluate NormPre across model scales, architectures and datasets. Our experiments cover GPT-2 Small on OpenWebText (Radford et al., 2019; Gokaslan and Cohen, 2019), LLaMA-130M, 350M, 1.3B on C4 (Touvron et al., 2023; Raffel et al., 2020), and Qwen3-0.6B, 1.7B on Pile (Yang et al., 2025; Gao et al., 2020). We compare NormPre-G and NormPre-L with AdamW (Loshchilov and Hutter, 2019), Muon (Jordan et al., 2024) and MANO (Gu and Xie, 2026). NormPre-G uses five Newton-Schulz iterations and NormPre-L uses the Sketch implementation with rank r=32r=32. All models are trained for 10,00010,000 steps with a sequence length of 10241024 and an effective batch size of 512512. Within each setting, all optimizers use matched model initialization, data order, training budget and evaluation protocol. Detailed configurations are provided in Appendix A.1. Table 1: Validation loss across model scales, architectures and pretraining datasets. GPT-2 Small results show mean ± standard deviation over three matched seeds, and larger models report single-run results under matched training budgets. Shaded columns indicate our variants. The best baseline and our outperforming variants are shown in underline and bold, respectively. Setting Validation Loss ↓ Model Dataset AdamW Muon MANO NormPre-G NormPre-L GPT-2 Small OpenWebText 3.1444 ± 0.0054 3.1064 ± 0.0054 3.1156 ± 0.0057 3.0667 ± ±0.0049 3.0826 ± 0.0034 LLaMA-130M C4 3.1363 3.1019 3.1102 3.0737 3.0930 LLaMA-350M C4 3.0378 2.9999 2.9946 2.9678 2.9758 LLaMA-1.3B C4 2.9385 2.9037 2.8963 2.8571 2.8662 Qwen3-0.6B Pile 2.8956 2.8335 2.8382 2.7967 2.8066 Qwen3-1.7B Pile 2.6758 2.6408 2.6205 2.5868 2.5897 Table 2: Training Efficiency Comparisons. Measurements use the same training configurations as the main experiments and are averaged over 100 optimizer steps after 20 warmup steps. Model Dataset Optimizer Optimizer Latency ↓ E2E Step Time ↓ Throughput ↑ Peak Memory ↓ (ms/step) (ms/step) (k tokens/s) (GiB) GPT-2 Small OpenWebText AdamW 2.7 3364.8 155.82 14.93 Muon 112.8 3479.8 150.67 14.61 MANO 8.8 3365.6 155.78 14.61 NormPre-G 119.3 ↑ 5.76% 3495.0 ↑ 0.44% 150.01 ↓ 0.44% 14.62 NormPre-L 69.8 ↓ 38.12% 3439.4 ↓ 1.16% 152.44 ↑ 1.17% 14.61 LLaMA-1.3B C4 AdamW 25.1 22121.0 23.70 42.34 Muon 1010.1 23124.2 22.67 37.84 MANO 87.1 22210.3 23.61 37.84 NormPre-G 1054.2 ↑ 4.37% 23141.8 ↑ 0.08% 22.66 ↓ 0.04% 37.83 NormPre-L 333.3 ↓ 67.00% 22399.7 ↓ 3.13% 23.41 ↑ 3.26% 37.84 GPT-2 Small Pretraining on OpenWebText. Figure 1a shows training and validation loss trajectories averaged over three matched seeds. Both variants achieve lower training and validation loss than the baselines during later training, demonstrating the effectiveness of NormPre. Table 1 confirms that both variants outperform all three baselines in mean best validation loss, with NormPre-G achieving the lowest value. Compared with the strongest baseline Muon, NormPre-G and NormPre-L reduce validation loss by 0.03970.0397 and 0.02380.0238, respectively. LLaMA Pretraining on C4 and Qwen3 Pretraining on Pile. We extend the comparison to LLaMA-130M, 350M, 1.3B on C4 and Qwen3-0.6B, 1.7B on Pile. Table 1 and Figures 1b–1d show that both variants consistently outperform all three baselines across these diverse architectures, with NormPre-G achieving the lowest loss. For LLaMA, the advantages over the strongest baseline are most pronounced at the largest evaluated scale (1.3B), where NormPre-G and NormPre-L reduce validation loss by 0.03920.0392 and 0.03010.0301, respectively. These optimization gains extend similarly to Qwen3. Notably, NormPre-L performs closely to NormPre-G at the 1.7B scale with a minimal loss difference of 0.00290.0029 (Qwen3 loss curves are deferred to Appendix A.2). 4.2 Training Efficiency and Spectral Dynamics Training Efficiency. Table 2 reports training efficiency on GPT-2 Small and LLaMA-1.3B under the same configurations as the main experiments. As summarized in Table 9 and Remark 1 (Appendix C), NormPre-G shares Muon’s asymptotic complexity, but its additional normalization increases optimizer latency by 5.76%5.76\% and 4.37%4.37\% relative to Muon on the two models, respectively. NormPre-L approximates the interaction subspace via sketching, reducing optimizer latency by 38.12%38.12\% and 67.00%67.00\%. These optimizer-level differences have a smaller effect on end-to-end training efficiency because optimizer updates account for only a small fraction of step time, which is largely spent on forward and backward computation. Relative to Muon, NormPre-G increases step time by about 0.44%0.44\% and 0.08%0.08\% on both models, while NormPre-L reduces it by 1.16%1.16\% and 3.13%3.13\%, with corresponding improvements in throughput. Both variants remain slower per step than AdamW and MANO due to the additional spectral processing, but match the whole-training peak memory of Muon and MANO. Further discussions and efficiency results on Qwen3 are provided in Appendix A.2. Spectral Dynamics. Figures 4a–4b validate our hierarchical Normalize-Then-Precondition design, revealing two key patterns. (a) Normalization yields an anisotropic base update. Equalizing row norms removes scale weighting in the Gram matrix, giving smaller-norm rows greater relative influence. This redistributes the spectral mass and might raise eigenvalues across multiple ranks (dotted curves over solid curves). Crucially, this anisotropy persists throughout training, necessitating further refinement. Table 7 confirms that constructing the preconditioner from Ψ rather than X yields lower losses. (b) Localized preconditioning captures the dominant transformation energy. Building on the normalized base, global preconditioning contracts modes above one and amplifies positive modes below one. Decomposing its spectral transformation energy (squared singular-value change ∑λi>0(λi−1)2 _ _i>0( _i-1)^2) reveals that the top-32 modes (λi>1 _i>1) consistently account for approximately 62%62\%–67%67\% across checkpoints. By concentrating strictly on these modes, localized preconditioning covers a substantial portion of the global transformation to yield the optimization gains observed in Figure 1. Further details and training dynamics are provided in Appendix A.3. (a) (b) (c) Figure 4: (a) Eigenspectra before and after marginal normalization; (b) Spectral transformation percentages under global and localized preconditioning; (c) Performance-efficiency trade-off. 4.3 Comparisons of NormPre-G and NormPre-L Drawing on the preceding findings, we discuss the complementary strengths of two variants. (a) NormPre-G yields superior performance via full-spectrum transformations. By operating on the entire eigenspectrum, NormPre-G consistently achieves the lowest validation loss across all evaluations. This indicates that transforming the long-tail modes alongside the dominant spectral energy provides crucial optimization benefits that push the absolute performance limit. (b) NormPre-L offers a performance-efficiency trade-off via local-spectrum transformations. The normalized base update motivates a regularized steepest-descent problem limiting deviations from this reference, giving NormPre-L a principled formulation (Propositions 1 and 2). Although empirical results favor the broader transformation of NormPre-G in validation loss, Figure 4c illustrates a clear trade-off on GPT-2 Small that larger r lowers validation loss at higher optimizer cost. We use r=32r=32 as the default spectral budget to balance performance and efficiency. At this budget, NormPre-L yields clear optimization gains with lower optimizer latency and end-to-end step time than NormPre-G (Table 2), serving as a practical and scalable alternative. 5 Related Work LLM Optimizers. Recent optimizers span coordinate-wise methods such as AdamW (Kingma and Ba, 2015; Loshchilov and Hutter, 2019), Lion (Chen et al., 2023), Sophia (Liu et al., 2024), and Cautious Optimizers (Liang et al., 2026), and block-wise approaches like Adam-mini (Zhang et al., 2025) and Blockwise LR (Wang et al., 2025). Alongside these, matrix-level methods like K-FAC (Martens and Grosse, 2015), Shampoo (Gupta et al., 2018; Shi et al., 2023), and SOAP (Vyas et al., 2025) construct preconditioners from curvature or accumulated statistics. We focus on matrix optimizers operating directly on update geometry via orthogonalization or row/column normalization. Orthogonalization and Normalization. Matrix structure is increasingly exploited via orthogonalization or lightweight row/column normalization. On the orthogonalization front, Muon applies a matrix-sign transformation to momentum via Newton-Schulz iterations (Jordan et al., 2024; Liu et al., 2025), while broader norm-constrained frameworks connect these updates to steepest descent and spectral spheres (Bernstein and Newhouse, 2024; Pethick et al., 2025b; Xie et al., 2026). Alternatively, row/column normalization offers a lightweight way. For example, MANO combines momentum projection with alternating row/column normalization (Gu and Xie, 2026), while RMNP uses only row normalization (Deng et al., 2026). MOGA derives such normalized updates from mean-normalized operator norms under the steepest-descent perspective (Xu et al., 2026). Closely Relevant Work. Several methods retain orthogonalization as a core operation and incorporate normalization or adaptive scaling. One line modifies the post-orthogonalization update: Muon+ applies row/column normalization to the orthogonalized output (Zhang et al., 2026), while NorMuon and AdaMuon introduce neuron-wise and element-wise adaptive scaling (Li et al., 2025; Si et al., 2025). Another line applies pre-orthogonalization normalization for numerical stability. For instance, SWAN normalizes instantaneous gradients before whitening (Ma et al., 2024) and MuonEq (Chang et al., 2026) equilibrates momentum to improve Newton-Schulz conditioning. While these approaches discuss the utility of incorporating normalization, it remains underexplored to formulate a systematic hierarchical framework and investigate if targeted spectral schemes can be effectively developed. We establish the Normalize-Then-Precondition framework, introducing two realizations of spectral preconditioning from spectral steepest descent and its regularized form with leading mode selection. 6 Conclusion This paper formalizes the Normalize-Then-Precondition framework to hierarchically organize marginal scales and interactions in matrix optimizers. Building on this, we propose NormPre, which achieves superior LLM pretraining performance over established baselines with a clear performance-efficiency trade-off. We highlight three promising directions for future research addressing current limitations: (a) Developing hardware-aware implementations to improve the efficiency of spectral transformations; (b) Scaling our empirical validation beyond 1.7B parameters; (c) Exploring adaptive spectral schemes to narrow the performance gap between localized and global preconditioning while preserving efficiency. Overall, our work provides a systematic framework for integrating normalization and spectral preconditioning, inspiring more effective and efficient LLM training. AI use statement We used OpenAI Codex and Anthropic Claude Code to assist with core code implementation, data processing, experimental evaluation, translation, and the verification and refinement of mathematical proofs. Additionally, we used these tools for literature search, manuscript drafting and revision. The research questions and ideas, conceptual framework, methodology, proof strategies and initial proof drafts, and experimental design were developed by the authors without AI assistance. We have reviewed all AI-assisted work, including manuscript text, source code, data processing, and training and evaluation pipelines. We retain full responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. Ethics statement This work studies effective and efficient optimization methods for large language model training. It does not involve human-subject experiments, collection of personal information or deployment in real-world decision-making systems. Our experiments use established language-modeling datasets, including OpenWebText, C4 and Pile, and we do not introduce or release new training data. We follow standard research practices for experimental evaluation, reporting and reproducibility. Reproducibility statement Section 3 provides a detailed description of the proposed optimizer NormPre, including two variants NormPre-G and NormPre-L. Section 4 presents the experimental setup, datasets, evaluation protocol and main results, with additional configurations, results, dynamics analyses and ablations provided in Appendix A. Implementation details for spectral preconditioning and computational complexity analyses are given in Appendices B and C. The assumptions, derivations and complete proofs supporting our theoretical results are provided in Appendix D. A repository containing the source code and materials for reproducing the experiments is linked in the abstract. References Bernstein and Newhouse (2024) J. Bernstein and L. Newhouse Old optimizer, new norm: an anthology. arXiv preprint arXiv:2409.20325. Cited by: §2.3, §5. Chang et al. (2026) D. Chang, Q. Shi, L. Zhang, Y. Li, R. Zhang, Y. Lu, Y. Liu, and G. Yuan Muoneq: balancing before orthogonalization with lightweight equilibration. arXiv preprint arXiv:2603.28254. Cited by: §1, §5. Chen et al. (2023) X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C. Hsieh, Y. Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems 36, p. 49205–49233. Cited by: §1, §5. Deng et al. (2026) S. Deng, Z. Ouyang, T. Pang, Z. Liu, R. Jin, S. Yu, and Y. Yang Rmnp: row-momentum normalized preconditioning for scalable matrix-based optimization. arXiv preprint arXiv:2603.20527. Cited by: §1, §2.2, §5. Eckart and Young (1936) C. Eckart and G. Young The approximation of one matrix by another of lower rank. Psychometrika 1 (3), p. 211–218. Cited by: Remark 4. Gao et al. (2020) L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The pile: an 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Cited by: §A.1, §4.1. Glentis et al. (2025) A. Glentis, J. Li, A. Han, and M. Hong A minimalist optimizer design for llm pretraining. In ES-FoMo I: 3rd Workshop on Efficient Systems for Foundation Models, Cited by: §2.2. Gokaslan and Cohen (2019) A. Gokaslan and V. Cohen OpenWebText corpus. Note: http://Skylion007.github.io/OpenWebTextCorpus Cited by: §A.1, §4.1. Gu and Xie (2026) Y. Gu and Z. Xie Mano: restriking manifold optimization for llm training. arXiv preprint arXiv:2601.23000. Cited by: §A.1, §1, §2.2, §4.1, §5. Gupta et al. (2018) V. Gupta, T. Koren, and Y. Singer Shampoo: preconditioned stochastic tensor optimization. In International Conference on Machine Learning, p. 1842–1850. Cited by: §1, §5. Halko et al. (2011) N. Halko, P. Martinsson, and J. A. Tropp Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM review 53 (2), p. 217–288. Cited by: Appendix B, 2nd item. Jordan et al. (2024) K. Jordan, Y. Jin, V. Boza, Y. Jiacheng, F. Cesista, L. Newhouse, and J. Bernstein Muon: an optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan. github. io/posts/muon 6 (3), p. 4. Cited by: §A.1, Appendix B, §1, §2.1, §3.2, §4.1, §5. Karpathy (2022) A. Karpathy NanoGPT. GitHub. Note: https://github.com/karpathy/nanoGPT Cited by: §A.1. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, Cited by: §1, §5. Li et al. (2025) Z. Li, L. Liu, C. Liang, W. Chen, and T. Zhao Normuon: making muon more efficient and scalable. arXiv preprint arXiv:2510.05491. Cited by: §5. Liang et al. (2026) K. Liang, L. Chen, B. Liu, and Q. Liu Cautious optimizers: improving training with one line of code. In International Conference on Learning Representations, Vol. 2026, p. 106538–106563. Cited by: §1, §5. Liu et al. (2024) H. Liu, Z. Li, D. Hall, P. Liang, and T. Ma Sophia: a scalable stochastic second-order optimizer for language model pre-training. In International conference on learning representations, Vol. 2024, p. 1621–1650. Cited by: §1, §5. Liu et al. (2025) J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982. Cited by: §A.1, Appendix B, §1, §3.1, §3.2, §5. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, Cited by: §A.1, §1, §4.1, §5. Ma et al. (2024) C. Ma, W. Gong, M. Scetbon, and E. Meeds Swan: sgd with normalization and whitening enables stateless llm training. arXiv preprint arXiv:2412.13148. Cited by: §1, §5. Martens and Grosse (2015) J. Martens and R. Grosse Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, p. 2408–2417. Cited by: §5. Mirsky (1960) L. Mirsky Symmetric gauge functions and unitarily invariant norms. The quarterly journal of mathematics 11 (1), p. 50–59. Cited by: Remark 4. Pethick et al. (2025a) T. Pethick, W. Xie, K. Antonakopoulos, Z. Zhu, A. Silveti-Falls, and V. Cevher Training deep learning models with norm-constrained lmos. arXiv preprint arXiv:2502.07529. Cited by: §2.2. Pethick et al. (2025b) T. Pethick, W. Xie, K. Antonakopoulos, Z. Zhu, A. Silveti-Falls, and V. Cevher Training Deep Learning Models with Norm-Constrained LMOs. In ICML, Cited by: §5. Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog 1 (8), p. 9. Cited by: §A.1, §4.1. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21 (140), p. 1–67. Cited by: §A.1, §4.1. Shi et al. (2023) H. M. Shi, T. Lee, S. Iwasaki, J. Gallego-Posada, Z. Li, K. Rangadurai, D. Mudigere, and M. Rabbat A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale. arXiv preprint arXiv:2309.06497. Cited by: §5. Si et al. (2025) C. Si, D. Zhang, and W. Shen Adamuon: adaptive muon optimizer. arXiv preprint arXiv:2507.11005. Cited by: §5. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §A.1, §4.1. Vyas et al. (2025) N. Vyas, D. Morwani, R. Zhao, I. Shapira, D. Brandfonbrener, L. Janson, and S. Kakade SOAP: improving and stabilizing shampoo using adam for language modeling. In International Conference on Learning Representations, Vol. 2025, p. 93423–93444. Cited by: §1, §5. Wang et al. (2025) J. Wang, M. Wang, Z. Zhou, J. Yan, L. Wu, et al. The sharpness disparity principle in transformers for accelerating language model pre-training. arXiv preprint arXiv:2502.19002. Cited by: §1, §5. Xie et al. (2026) T. Xie, H. Luo, H. Tang, H. Yiwen, J. K. Liu, Q. Ren, Y. Wang, X. Zhao, R. Yan, B. Su, C. Luo, and B. Guo Controlled LLM training on spectral sphere. In Forty-third International Conference on Machine Learning, Cited by: §5. Xu et al. (2026) R. Xu, J. Li, and Y. Lu On the width scaling of neural optimizers under matrix operator norms i: row/column normalization and hyperparameter transfer. arXiv preprint arXiv:2603.09952. Cited by: §1, §2.2, §5. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §A.1, §4.1. Zhang et al. (2026) R. Zhang, Y. Zhao, Z. Liu, Z. Wang, and Z. Zhang Muon+: towards better muon via one additional normalization step. arXiv e-prints, p. arXiv–2602. Cited by: §5. Zhang et al. (2025) Y. Zhang, C. Chen, Z. Li, T. Ding, C. Wu, D. D. Kingma, Y. Ye, Z. Luo, and R. Sun Adam-mini: use fewer learning rates to gain more. In International Conference on Learning Representations, Vol. 2025, p. 28033–28063. Cited by: §1, §5. Zhao et al. (2024) J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian GaLore: memory-efficient llm training by gradient low-rank projection. International Conference on Machine Learning. Cited by: §A.1. Appendix A Experimental Details A.1 Experimental Configurations Models and Data. Our experiments use GPT-2 Small on OpenWebText (Radford et al., 2019; Gokaslan and Cohen, 2019), LLaMA-130M, 350M, 1.3B on C4 (Touvron et al., 2023; Raffel et al., 2020) and Qwen3-0.6B, 1.7B on Pile (Yang et al., 2025; Gao et al., 2020). GPT-2 Small is reproduced using the nanoGPT codebase (Karpathy, 2022). We follow the general pretraining setup in Zhao et al. (2024); Raffel et al. (2020). Table 3 summarizes the model and training configurations. OpenWebText is processed with the GPT-2 BPE tokenizer, and the preprocessed training and validation data are loaded into CPU memory before training. For C4, we tokenize the English corpus with the T5-base tokenizer. For Pile, we recover documents from the Pythia/Pile token stream and re-tokenize them with the Qwen3 tokenizer. For C4 and Pile, we construct fixed preprocessed datasets of 5,120,0005,120,000 training sequences and 2,0482,048 validation sequences, each with length 10241024. Documents are processed independently by truncating long documents and padding shorter ones without cross-document packing. The resulting data order is fixed and shared across all optimizer runs. With sequence length 10241024, effective batch size 512512 and 10,00010,000 optimization steps, each training trajectory processes approximately 5.245.24B nominal token positions. Table 3: Model and Training Configurations. Model Model Size Dataset Layers Hidden FFN Heads Seq. Len. Eff. Batch Base Peak LR Warmup Steps GPT-2 Small 124M OpenWebText 12 768 3,072 12 1,024 512 6×10−46× 10^-4 2,000 10,000 LLaMA-130M 130M C4 12 768 2,048 12 1,024 512 6×10−46× 10^-4 1,000 10,000 LLaMA-350M 350M C4 24 1,024 2,736 16 1,024 512 3×10−43× 10^-4 1,000 10,000 LLaMA-1.3B 1.3B C4 24 2,048 5,461 32 1,024 512 3×10−43× 10^-4 1,000 10,000 Qwen3-0.6B 0.6B Pile 28 1,024 3,072 16 1,024 512 3×10−43× 10^-4 1,000 10,000 Qwen3-1.7B 1.7B Pile 28 2,048 6,144 16 1,024 512 3×10−43× 10^-4 1,000 10,000 Optimizers. We compare NormPre-G and NormPre-L with AdamW (Loshchilov and Hutter, 2019), Muon (Jordan et al., 2024) and MANO (Gu and Xie, 2026). Table 4 summarizes the default optimizer configurations. We use (β1,β2)=(0.9,0.95)( _1, _2)=(0.9,0.95) for AdamW and momentum coefficient μ=0.95μ=0.95 for Muon, MANO and both NormPre variants. Muon and NormPre-G use five Newton-Schulz iterations. NormPre-L uses the Sketch-based implementation with interaction rank r=32r=32, sketch oversampling o=8o=8 and one power iteration p=1p=1. Both NormPre variants match the post-preconditioning update RMS to a target value of 0.20.2 before the parameter update. For GPT-2 Small, Muon follows the Keller–Jordan convention (Jordan et al., 2024) for hidden matrix parameters, using an effective peak matrix learning rate of 0.020.02 together with width-dependent update scaling. For LLaMA and Qwen3, Muon follows the scalable RMS-matching convention (Liu et al., 2025) and uses the base learning rate of the corresponding model setting. Muon, MANO and both NormPre variants are applied to hidden two-dimensional attention and MLP weight matrices, with token embeddings, output heads, normalization parameters, biases and other one-dimensional parameters optimized by AdamW. Table 4: Default Optimizer Configurations. A dash indicates that the corresponding hyperparameter is not applicable. Hyperparameter AdamW Muon MANO NormPre-G NormPre-L β1 _1 0.9 – – – – β2 _2 0.95 – – – – Momentum μ – 0.95 0.95 0.95 0.95 Newton-Schulz steps – 5 – 5 – Weight decay 0.1 0.1 0.1 0.1 0.1 Interaction rank r – – – – 32 Sketch oversampling o – – – – 8 Power iterations p – – – – 1 Training. All models are trained for 10,00010,000 optimization steps with sequence length 10241024 and effective batch size 512512. We use linear warmup followed by cosine decay to 10%10\% of the peak learning rate, with the model-specific learning rates and warmup steps given in Table 3. Weight decay is 0.10.1 and gradients are clipped at 1.01.0. All models are evaluated every 500500 optimization steps. GPT-2 Small uses 200200 randomly sampled validation mini-batches per evaluation, while LLaMA and Qwen3 evaluate on the complete fixed validation split of 2,0482,048 sequences. Within each setting, all compared optimizers use matched model initialization, data order, training budget, base learning-rate schedule and validation protocol. All experiments use the same PyTorch training codebase and NVIDIA A100 80GB GPUs. GPT-2 Small uses one GPU per training trajectory and LLaMA and Qwen3 use four-rank data parallelism. A.2 Additional Experimental Results Training and Validation Loss for Qwen3 Models. We provide the training and validation loss curves for Qwen3-0.6B and Qwen3-1.7B on Pile in Figure 5. (a) Qwen3-0.6B on Pile (b) Qwen3-1.7B on Pile Figure 5: Qwen3 Models on Pile. Training and validation loss for Qwen3-0.6B, 1.7B with AdamW, Muon, MANO, NormPre-G and NormPre-L. For training loss, dark curves denote 50-step moving averages and light curves the raw trajectories. Training Efficiency Configurations. Efficiency measurements mirror the main pretraining configurations (including model architecture, sequence length of 1024, effective batch size of 512, and gradient clipping) with the following specific profiling protocols: • Precision & Data Loading: GPT-2 Small and LLaMA use FP32 parameters with BF16 autocast, while Qwen3 uses pure BF16. OpenWebText is fully materialized in host memory, whereas C4 and Pile are read dynamically from persistent NumPy memmaps (transferring only the active batch to the GPU to avoid full-dataset materialization). • Measurement Window: We profile 100 optimizer steps following 20 warmup steps. To isolate core algorithmic overhead, we disable optimizer diagnostics (e.g., descent alignment), validation, checkpointing, W&B logging, and torch.compile. • Metric Definitions: Optimizer latency isolates optimizer.step() using strict CUDA synchronization immediately before and after the call. E2E step time encompasses the complete training loop (data transfer, forward/backward passes, gradient accumulation, clipping, and optimizer update). Throughput is computed as global tokens per E2E step time, and peak memory tracks the maximum CUDA allocated memory during a measured step. • TF32 Settings: Unless otherwise stated, TF32 refers to the CUDA matrix-multiplication TF32 setting. To isolate intrinsic algorithmic scaling from hardware-specific Tensor Core accelerations at small dimensions, the main GPT-2 Small efficiency results disable TF32. Training Efficiency on Qwen3-1.7B. We further evaluate training efficiency on Qwen3-1.7B with Pile under the main experimental configuration. Similar to the observations in Section 4.2, relative to Muon, NormPre-G increases optimizer latency by 6.77%6.77\%, with an end-to-end step-time overhead of 0.53%0.53\%. NormPre-L reduces optimizer latency by 57.16%57.16\%, yielding a 4.21%4.21\% reduction in step time and a 4.41%4.41\% increase in throughput. Both variants remain slower per step than AdamW and MANO. All evaluated optimizers have the same whole-training peak allocated memory of 53.1753.17 GiB per GPU. This shared peak is determined by forward and backward computation, whose memory peak exceeds that of the optimizer step. Table 5: Training Efficiency on Qwen3-1.7B. Measurements use the same training configuration as the main Qwen3-1.7B experiment and are averaged over 100 optimizer steps after 20 warmup steps. We report global throughput across 4 GPUs and per-GPU peak allocated memory. Model Dataset Optimizer Optimizer Latency ↓ E2E Step Time ↓ Throughput ↑ Peak Memory ↓ (ms/step) (ms/step) (k tokens/s) (GiB) Qwen3-1.7B Pile AdamW 18.3 12102.7 43.32 53.17 Muon 940.0 13050.4 40.17 53.17 MANO 118.7 12217.4 42.91 53.17 NormPre-G 1003.6 ↑ 6.77% 13119.7 ↑ 0.53% 39.96 ↓ 0.52% 53.17 NormPre-L 402.7 ↓ 57.16% 12501.4 ↓ 4.21% 41.94 ↑ 4.41% 53.17 A.3 Additional Details on Spectral and Training Dynamics Spectral Dynamics. We collect spectral diagnostics from a NormPre-L training run on GPT-2 Small with OpenWebText and random seed 1337. We use the Sketch implementation with rank r=32r=32, oversampling o=8o=8, and one power iteration (p=1p=1). Diagnostics cover the 48 attention and MLP weight matrices at checkpoints corresponding to steps 100 (101), 500 (501), 1,000 (1,001), 2,000 (2,001), 5,000 (5,001), and 10,000 (9,999). Figure 4a presents row-active spectra, and Figure 6a provides the column-active counterpart. (a) (b) Figure 6: Spectral Dynamics. (a) Eigenspectra before and after marginal normalization in the column-active orientation. (b) Spectral transformation percentages across modes under global and localized preconditioning (column-active orientation). (1) Eigenspectra Before and After Normalization. Following Section 3, let X denote the update after relaxed tangent processing (i.e., before normalization) and Ψ denote its marginally normalized counterpart. Both are expressed in the active orientation and measured before spectral preconditioning. To isolate changes in spectral shape from differences in global update magnitude, we compare G^X=dtr(XX⊤)XX⊤,Γ=ΨΨ⊤, G_X= dtr(X )X , = , where d is the active-axis dimension. Both matrices have trace d and then the same total spectral mass. The scalar rescaling of XX⊤X preserves its eigenvalue ratios and relative row-norm differences, which is used only for analysis in spectral dynamics. We sort the nonzero eigenvalues of G^X G_X and Γ in descending order and average them across matrices at each rank. The shaded regions show the corresponding 25th–75th percentile ranges. Different colors correspond to different training steps. Solid curves show the spectra before marginal normalization (G^X G_X), and dotted curves show those after normalization (Γ ). The horizontal line at λ=1λ=1 indicates the target under full-spectrum preconditioning and localized preconditioning. Effect of Marginal Normalization on the Spectrum. Before normalization, correlations between row directions are weighted by the products of their row norms in the Gram matrix. Normalization gives each row unit norm, removing these weights and increasing the relative influence of smaller-norm rows. To illustrate this effect, consider mutually orthogonal rows with norms a1,…,ad>0a_1,…,a_d>0. The trace-matched Gram matrix is diagonal, with eigenvalues dai2/∑jaj2=ai2/a2¯da_i^2/ _ja_j^2=a_i^2/ a^2, where a2¯=1d∑jaj2 a^2= 1d _ja_j^2. After marginal normalization, the rows remain orthogonal and each has unit norm, so all eigenvalues equal one. Thus, the eigenvalue corresponding to a row increases if its squared norm is below the average and decreases if it is above the average. For nonorthogonal rows, eigenvalues do not directly correspond to individual row norms. Removing row-norm weighting still changes the spectral structure formed jointly by the rows and might increase eigenvalues across multiple ranks. This provides a possible explanation for the dotted curves lying above the solid curves over many ranks in Figures 4a and 6a. (2) Global and Localized Spectral Preconditioning. We use row-active and column-active snapshots from the same NormPre-L trajectory described above. For each snapshot, we compare the spectral transformations induced by global and localized preconditioning on the same normalized interaction spectrum. Let λi\ _i\ denote the nonzero eigenvalues of Γ=ΨΨ⊤ = . Global preconditioning maps each corresponding singular value λi _i to one. We define the spectral transformation energy of each mode as its squared singular-value change, ei=(λi−1)2e_i=( _i-1)^2, and partition the total into three components: Etop=∑i∈Cei,Eremain=∑λi>1,i∉Cei,Esub=∑0<λi<1ei,E_top= _i∈ Ce_i, E_remain= _ subarrayc _i>1,i∉ C subarraye_i, E_sub= _0< _i<1e_i, where C contains the indices of the largest at most r=32r=32 eigenvalues exceeding one. Modes with λi=1 _i=1 contribute zero. The total spectral transformation energy is Eglobal=Etop+Eremain+EsubE_global=E_top+E_remain+E_sub. For each matrix, we express each component as a percentage of EglobalE_global. The stacked bars show these percentages averaged equally across the 48 matrices at each checkpoint. This decomposition distinguishes the spectral scope of the two variants. Localized preconditioning contracts only the modes in C and leaves the others unchanged. Global preconditioning additionally contracts the remaining modes above one and amplifies the positive modes below one. Figure 6b presents the column-active results, which follow a similar pattern to the row-active results in Figure 4b. The top-32 modes account for approximately 55%55\%–60%60\% of the total spectral transformation energy across checkpoints. (a) Gradient Norm (b) Descent Alignment Figure 7: Training Dynamics. (a) Pre-clipping global gradient norm over all trainable parameters. (b) Descent alignment between the gradient and optimizer update direction over matrix-optimized parameters. Dark curves show 50-step moving averages and light curves show the corresponding raw trajectories. Training Dynamics. We examine training dynamics on GPT-2 Small with OpenWebText. Gradient norm is measured over all trainable parameters before clipping. Descent alignment is computed over the attention and MLP weight matrices. For each matrix W, let Gt(W)G_t^(W) denote its gradient and Φt(W) _t^(W) its optimizer update direction before learning-rate scaling, excluding decoupled weight decay. We report Alignt=1|ℳ|∑W∈ℳ⟨Gt(W),Φt(W)⟩F‖Gt(W)‖F‖Φt(W)‖F,Align_t= 1|M| _W G_t^(W), _t^(W) _F\|G_t^(W)\|_F\| _t^(W)\|_F, where ℳM is the set of evaluated weight matrices. This averages the per-matrix cosine similarities. Each cosine equivalently measures the alignment between the optimizer-induced parameter change and the current negative gradient. Analysis. Both NormPre variants exhibit transient peaks in gradient norm during early training, with a more pronounced peak for NormPre-G. Their descent alignment is also lower than Muon’s, indicating greater deviation from the current negative gradient. These distinct early dynamics do not develop into persistent gradient instability. As training progresses, their gradient norms decrease and stabilize, with fewer isolated spikes overall than Muon. Directional differences persist after the gradient norms stabilize: both variants maintain lower alignment than Muon, with NormPre-G remaining the lowest. Thus, NormPre continues to reshape the update direction throughout training. Together with the lower validation losses, these observations demonstrate an effective update geometry: NormPre adjusts update directions while maintaining positive average descent alignment, sustains relatively steady optimization dynamics in later training, and achieves better final solutions. A.4 Additional Ablations Table 6: Ablations on Marginally Normalized Update. Results are reported on GPT-2 Small with OpenWebText using seed 13371337 under the same 1010k-step training budget. (a) Momentum Processing Variant Validation Loss NormPre-G NormPre-L No Tangent 3.0945 3.1139 Strict Tangent 3.1869 3.1196 Relaxed Tangent 3.0711 3.0864 (b) Normalization Orientation Orientation Validation Loss NormPre-G NormPre-L Row 3.0604 3.0817 Alternating Row/Column 3.0711 3.0864 Row+Column/Step 3.0735 3.0878 Table 7: Ablations on Interaction Geometry Source. Results on GPT-2 Small with OpenWebText using seed 1337 and the same 10k-step training budget. Interaction Source NormPre-G NormPre-L Training Loss Validation Loss Training Loss Validation Loss Update before normalization (X) 3.0844 3.0900 3.1375 3.1305 Update after normalization (Ψ ) 3.0655 3.0711 3.0810 3.0864 Table 8: Spectral Budget in NormPre-L. Results are reported on GPT-2 Small with OpenWebText using seed 13371337 under the same 1010k-step training budget. Rank r Training Loss Validation Loss 8 3.0911 3.0962 16 3.0857 3.0908 32 (default) 3.0810 3.0864 64 3.0768 3.0827 Ablations on Marginally Normalized Update. Table 6 examines two design choices in constructing the marginally normalized update for both NormPre-G and NormPre-L. In part (a), relaxed tangent projection achieves the lowest validation loss for both variants, outperforming both leaving the momentum unchanged and strict tangent projection. The degradation under strict tangent projection is particularly pronounced for NormPre-G, suggesting that completely removing the radial component can discard useful information before spectral refinement. These results support the relaxed design, which prioritizes direction-changing information while retaining a limited scale-dependent component. From part (b), the choice of normalization orientation has a smaller effect. For NormPre-L, row-only and alternating row/column normalization achieve comparable validation losses, with a slight advantage for the row-only variant in this controlled comparison. For NormPre-G, the row-only variant shows a clearer improvement over alternating normalization, while normalizing both axes at every step performs similarly to the alternating scheme. We adopt alternating normalization as the default to treat the two matrix axes symmetrically and avoid introducing a fixed preferred orientation. Ablations on Interaction Geometry Source. We compare interaction geometry extracted from the update before normalization (X) with that extracted from the update after normalization (Ψ ). Both settings apply preconditioning to the same normalized base update Ψ and differ only in the source used to construct the preconditioner. Table 7 shows that extracting interaction geometry after normalization yields lower training and validation loss for both variants. This supports constructing the preconditioner from the geometry of the update on which it acts. Using the update before normalization causes a larger degradation for NormPre-L, suggesting greater sensitivity to the source when refinement is restricted to selected modes. The marginal scales in X can affect the spectral ordering, so its leading modes may differ from those of Ψ . Normalization removes this scale weighting first, allowing the top-r eigenspace to capture interaction structure without the influence of unequal row norms. This offers a possible explanation for the greater sensitivity of localized refinement. Together, these results support using marginal normalization both to construct the base update and to establish the interaction geometry for subsequent spectral preconditioning. Spectral Budget in NormPre-L. Table 8 reports training and validation losses for different spectral budgets, complementing Figure 4c. Both losses decrease as r increases over the evaluated range. We use r=32r=32 as the default to balance these performance gains against the optimizer cost. Appendix B Detailed Implementations on Spectral Preconditioning From Section 3.2, NormPre-G realizes global spectral preconditioning by applying Newton-Schulz iteration to the normalized update Ψ , approximating TG=msign(Ψ)T^G=msign( ). This follows the standard Newton-Schulz implementation used by Muon (Jordan et al., 2024; Liu et al., 2025). NormPre-L operates on a selected interaction eigenspace, which can be obtained through exact eigendecomposition or a randomized sketch. In the following, we focus on the implementation details of NormPre-L. Exact Eigenspace Extraction. The exact implementation explicitly forms the interaction matrix Γ=ΨΨ⊤ = and computes its eigendecomposition Γ=U~ΛU~⊤ = U U . Following the localized construction in Section 2.3, we define the active set :=i:λi>1A:=\i: _i>1\ and let ⊆C index the minr,|| \r,|A|\ largest active eigenvalues. The selected eigenpairs (U~,Λ)( U_C, _C) are directly used in the localized preconditioner PL=Im+U~(Λ−1/2−I)U~⊤.P^L=I_m+ U_C( _C^-1/2-I) U_C . Sketch-Based Eigenspace Extraction. To avoid forming and decomposing the full interaction matrix, we approximate its leading eigenspace through a randomized sketch (Halko et al., 2011). Among the top-r estimated interaction modes, those satisfying λ^i>1 λ_i>1 are retained for the localized preconditioner P^L=Im+U^(Λ^−1/2−I)U^⊤. P^L=I_m+ U_C( _C^-1/2-I) U_C . Algorithm 2 gives the complete procedure. Algorithm 2 Randomized Interaction Sketch 1: Marginally normalized update Ψ∈ℝm×n ^m× n, rank r≤mr≤ m, oversampling o, power iterations p. 2: ℓ←minm,r+o ← \m,r+o\ 3: Sample Ω∈ℝn×ℓ ^n× with Ωij∼(0,1) _ij (0,1) and set Y←ΨΩY← ⊳ Gaussian Sketch 4: for j=1,…,pj=1,…,p do 5: Y←Ψ(Ψ⊤Y)Y← ( Y) 6: end for⊳ Power Iteration 7: Q←qr(Y)Q (Y) ⊳ Candidate Subspace 8: ΓQ←(Q⊤Ψ)(Q⊤Ψ)⊤ _Q←(Q )(Q ) ⊳ Rayleigh–Ritz Matrix 9: (Z,Λ^)←(Z, )← top-r eigenpairs of ΓQ _Q ⊳ Ritz Pairs 10: U^←QZ U← QZ ⊳ Lifted Ritz Vectors 11: ←i:λ^i>1C←\i: λ_i>1\ 12: return (U^,Λ^)( U_C, _C) The main steps of Algorithm 2 are as follows: • Randomized Subspace Construction (Lines 22–66). The Gaussian sketch Y=ΨΩY= probes the range of Ψ and the power iterations Y←Ψ(Ψ⊤Y)Y← ( Y) amplify the separation of its leading singular directions. Then, QR decomposition yields an orthonormal basis Q∈ℝm×ℓQ ^m× for the candidate interaction subspace. • Rayleigh–Ritz Eigenspace Extraction (Lines 77–1010). The interaction matrix is projected onto the candidate subspace through ΓQ=Q⊤ΓQ=(Q⊤Ψ)(Q⊤Ψ)⊤∈ℝℓ×ℓ. _Q=Q Q=(Q )(Q ) × . Computing the top-r eigenpairs (Z,Λ^)(Z, ) of ΓQ _Q gives the corresponding Ritz pairs and the Ritz vectors are lifted to the original interaction space through U^=QZ U=QZ. We retain the estimated modes satisfying λ^i>1 λ_i>1, yielding (U^,Λ^)( U_C, _C) for localized spectral preconditioning. Since the Ritz values are ordered in descending order, applying λ^i>1 λ_i>1 after top-r selection is equivalent to first identifying the active modes and then retaining the largest r, consistent with Section 2.3. Appendix C Detailed Computational Complexity Analysis This section analyzes the optimizer-side computational complexity. For NormPre, the analysis covers momentum update, relaxed tangent projection, diagonal-Gram normalization, spectral preconditioning, update RMS scaling and parameter update. We consider both NormPre-G and NormPre-L, and include Muon as a reference for the Newton-Schulz computation. Table 9 summarizes the resulting complexities. Let m×nm× n denote the matrix dimensions entering the spectral preconditioning stage and define s:=minm,ns:= \m,n\. For the Newton-Schulz implementations in Muon and NormPre-G, the input is transposed when necessary so that the Gram matrix is s×s× s. For NormPre-L, m×nm× n follows the current active orientation. Let q denote the number of Newton-Schulz iterations, r the interaction rank, o the oversampling parameter, p the number of power iterations and ℓ:=minm,r+o := \m,r+o\. We omit constant factors and lower-order terms. Table 9: Computational Complexity of Muon and NormPre Variants. Method Computational Complexity Muon (mn+qmns)O(mn+qmns) NormPre-G (mn+qmns)O(mn+qmns) NormPre-L (Exact) (m2n+m3)O(m^2n+m^3) NormPre-L (Sketch) ((p+1)mnℓ+(m+n)ℓ2+ℓ3)O\! ((p+1)mn +(m+n) ^2+ ^3 ) C.1 Shared Components in NormPre-G and NormPre-L Following Algorithm 1, NormPre-G and NormPre-L share the same optimization pipeline except for the spectral preconditioning step. We analyze these shared operations under the active orientation. • Momentum Update and Relaxed Tangent Projection. Updating the momentum requires (mn)O(mn) operations. The tangent projection is applied independently to each row with (n)O(n) cost per row (inner products and vector rescaling), giving (mn)O(mn) operations in total. • Diagonal-Gram Normalization. Computing the row norms of X and forming the normalized update Ψ=D(X)−1X =D(X)^-1X require (mn)O(mn) operations. • Update RMS Scaling and Parameter Update. Computing RMS(T)RMS(T) and rescaling the update both require (mn)O(mn) operations. Applying weight decay and updating the parameter matrix also require (mn)O(mn) operations. Overall, these shared operations contribute (mn)O(mn) computation to both variants and are included in the total costs reported in Table 9. C.2 NormPre-G and Muon Muon. Muon applies a Newton-Schulz approximation of the matrix-sign transformation to its momentum update. The input is transposed when necessary so that the Gram matrix is s×s× s. Each Newton-Schulz iteration involves matrix products between the s×s× s Gram matrix and the matrix update, with dominant cost (mns)O(mns). For q iterations, the matrix transformation requires (qmns)O(qmns). The remaining optimizer operations (e.g., momentum update, update RMS scaling and parameter update) contribute (mn)O(mn), giving the total complexity (mn+qmns).O(mn+qmns). NormPre-G. NormPre-G applies the same q-step Newton-Schulz transformation to the normalized update Ψ . Thus, its spectral preconditioning stage requires (qmns)O(qmns) computation. Together with the shared (mn)O(mn) operations analyzed in Section C.1, the total complexity is (mn+qmns).O(mn+qmns). Therefore, Muon and NormPre-G have the same leading asymptotic computational complexity for a fixed number of Newton-Schulz iterations. The relaxed tangent projection and diagonal-Gram normalization in NormPre-G add only (mn)O(mn) computation and do not change the asymptotic order. C.3 NormPre-L For the Exact implementation, the selected eigenvectors U~∈ℝm×|| U_C ^m×|C| and eigenvalues Λ _C give PLΨ=Ψ+U~(Λ−1/2−I)U~⊤Ψ.P^L = + U_C ( _C^-1/2-I ) U_C . For the Sketch-based implementation, the estimated eigenpairs (U^,Λ^)( U_C, _C) yield P^LΨ=Ψ+U^(Λ^−1/2−I)U^⊤Ψ. P^L = + U_C ( _C^-1/2-I ) U_C . In both cases, the selected eigenvector matrix has dimension m×||m×|C| with ||≤r|C|≤ r. Computing U~⊤Ψ U_C or U^⊤Ψ U_C requires (mn||)O(mn|C|) operations. Applying the diagonal spectral scaling costs (n||)O(n|C|) and multiplying by U~ U_C or U U_C requires another (mn||)O(mn|C|) operations. Therefore, the localized spectral application costs (mnr)O(mnr). The exact and sketch-based implementations differ in the cost of extracting the selected interaction eigenspace. NormPre-L (Exact). The exact implementation forms the interaction matrix Γ=ΨΨ⊤ = , which costs (m2n)O(m^2n). The resulting m×m× m matrix is then processed by a dense symmetric eigendecomposition with cost (m3)O(m^3). Together with the shared (mn)O(mn) operations analyzed in Section C.1 and the (mnr)O(mnr) localized spectral application, we obtain (mn+mnr+m2n+m3)=(m2n+m3)O\! (mn+mnr+m^2n+m^3 )=O(m^2n+m^3), where the simplification follows from r≤mr≤ m. NormPre-L (Sketch). The sketch-based implementation avoids forming and decomposing the full m×m× m interaction matrix by using the randomized sketch in Algorithm 2. Its eigenspace-extraction cost can be decomposed as follows, the main intermediate dimensions are summarized in Table 10. • Randomized Sketch and Power Iteration. Forming Y=ΨΩY= costs (mnℓ)O(mn ). Each power iteration computes Ψ⊤Y Y followed by Ψ(Ψ⊤Y) ( Y), with both matrix multiplications requiring (mnℓ)O(mn ) operations. Over p iterations, the power-iteration cost is (pmnℓ)O(pmn ). • Candidate Subspace. Computing the QR decomposition of Y∈ℝm×ℓY ^m× costs (mℓ2)O(m ^2). • Rayleigh–Ritz Matrix. Forming Q⊤Ψ∈ℝℓ×nQ × n costs (mnℓ)O(mn ) and constructing ΓQ=(Q⊤Ψ)(Q⊤Ψ)⊤∈ℝℓ×ℓ _Q=(Q )(Q ) × costs (nℓ2)O(n ^2). • Rayleigh–Ritz Extraction. A dense eigendecomposition of ΓQ∈ℝℓ×ℓ _Q × costs (ℓ3)O( ^3). Computing U^=QZ U=QZ costs (mℓr)O(m r), which is bounded by (mℓ2)O(m ^2) because r≤ℓr≤ . We have (mn+mnr+(p+1)mnℓ+(m+n)ℓ2+ℓ3)O\! (mn+mnr+(p+1)mn +(m+n) ^2+ ^3 ) when combining these terms with the shared (mn)O(mn) operations and (mnr)O(mnr) localized spectral application. Since r≤ℓr≤ , the (mn)O(mn) and (mnr)O(mnr) terms are absorbed into the sketching cost. Therefore, the total computational complexity of NormPre-L (Sketch) is ((p+1)mnℓ+(m+n)ℓ2+ℓ3).O\! ((p+1)mn +(m+n) ^2+ ^3 ). Table 10: Dimensions of the Main Quantities in Algorithm 2 Symbol Dimension Description Ψ m×nm× n Marginally normalized update Ω n×ℓn× Gaussian random matrix Y m×ℓm× Gaussian sketch Q m×ℓm× Candidate subspace ΓQ _Q ℓ×ℓ × Rayleigh–Ritz matrix Z ℓ×r × r Top-r eigenvectors of ΓQ _Q Λ r×r× r Estimated eigenvalues U U m×rm× r Estimated eigenvectors Appendix D Proofs D.1 Optimization Formulation of Localized Spectral Refinement Proposition 1 (Regularized Spectral Steepest Descent). Suppose that Γ(X)≻0 (X) 0. The closest update to Ψ within the unit spectral-norm ball is T+:=argmin‖T‖op≤112‖T−Ψ‖F2=argmax‖T‖op≤1⟨Ψ,T⟩F−12‖T‖F2. T_+:= _\|T\|_op≤ 1 12\|T- \|_F^2= _\|T\|_op≤ 1 \ ,T _F- 12\|T\|_F^2 \. (6) Its unique solution is T+=P+ΨT_+=P_+ , where the preconditioner is P+=U~Diag(min1,λ1−1/2,…,min1,λm−1/2)U~⊤. P_+= UDiag ( \1, _1^-1/2\,…, \1, _m^-1/2\ ) U . (7) In particular, 0≺P+⪯Im0 P_+ I_m and σi(T+)=minσ~i,1 _i(T_+)= \ σ_i,1\. Remark 3. Proposition 1 gives the exact projection of Ψ onto the unit spectral-norm ball, clipping singular values above one to one while leaving the remaining singular values unchanged. Define the active set :=i:λi>1A:=\i: _i>1\, together with U~:=[u~i]i∈ U_A:=[ u_i]_i and Λ:=Diag(λi)i∈ _A:=Diag( _i)_i . Equivalently, the preconditioner can be written as P+=Im+U~(Λ−1/2−I)U~⊤.P_+=I_m+ U_A( _A^-1/2-I) U_A . Its non-identity component is Δ:=P+−Im _A:=P_+-I_m. Hence, P+Ψ=Ψ+ΔΨP_+ = + _A , where ΔΨ _A is the induced update refinement relative to the normalized update Ψ . Proof. Let Ψ=U~Σ~V~⊤ = U V be a thin SVD, where Σ~=Diag(σ~1,…,σ~m) =Diag( σ_1,…, σ_m) with σ~1≥⋯≥σ~m>0 σ_1≥·s≥ σ_m>0. Suppose that Γ(X)=ΨΨ⊤=U~ΛU~⊤≻0 (X)= = U U 0, i.e., Ψ has full row rank and λi=σ~i2>0 _i= σ_i^2>0 for every i. For the regularized steepest-descent problem, 12‖T−Ψ‖F2=12‖Ψ‖F2−(⟨Ψ,T⟩F−12‖T‖F2). 12\|T- \|_F^2= 12\| \|_F^2- ( ,T _F- 12\|T\|_F^2 ). Since ‖Ψ‖F2/2\| \|_F^2/2 is independent of T, the two optimization problems in Equation 6 are equivalent. Following the standard spectral steepest-descent derivation of the matrix-sign update, we solve the regularized problem under the same unit spectral-norm constraint by optimizing its singular values modewise. Let τ1,…,τm _1,…, _m denote the singular values of a feasible T. The constraint ‖T‖op≤1\|T\|_op≤ 1 implies 0≤τi≤10≤ _i≤ 1. By von Neumann’s trace inequality, ⟨Ψ,T⟩F≤∑i=1mσ~iτi, ,T _F≤ _i=1^m σ_i _i, with equality when T shares the singular directions of Ψ . Since ‖T‖F2=∑i=1mτi2\|T\|_F^2= _i=1^m _i^2, the regularized objective satisfies ⟨Ψ,T⟩F−12‖T‖F2≤∑i=1m(σ~iτi−12τi2). ,T _F- 12\|T\|_F^2≤ _i=1^m ( σ_i _i- 12 _i^2 ). Thus, the matrix problem reduces to independently maximizing σ~iτi−12τi2 σ_i _i- 12 _i^2 over 0≤τi≤10≤ _i≤ 1 for each singular mode. For each i, consider the scalar objective fi(τi):=σ~iτi−12τi2.f_i( _i):= σ_i _i- 12 _i^2. Since fi′(τi)=σ~i−τif_i ( _i)= σ_i- _i and fi′(τi)=−1<0f_i ( _i)=-1<0, its unique unconstrained maximizer is τi=σ~i _i= σ_i. With the unit spectral-norm constraint 0≤τi≤10≤ _i≤ 1, we have τi⋆=minσ~i,1. _i = \ σ_i,1\. Therefore, the unique solution is T+=U~Diag(minσ~1,1,…,minσ~m,1)V~⊤.T_+= UDiag ( \ σ_1,1\,…, \ σ_m,1\ ) V . The uniqueness follows from the strict concavity of ⟨Ψ,T⟩F−12‖T‖F2 ,T _F- 12\|T\|_F^2 over the convex feasible set. We next express the optimal solution as a left spectral preconditioning of Ψ . Since λi=σ~i2 _i= σ_i^2, define P+:=U~Diag(min1,λ1−1/2,…,min1,λm−1/2)U~⊤.P_+:= UDiag ( \1, _1^-1/2\,…, \1, _m^-1/2\ ) U . Then P+Ψ=U~Diag(min1,σ~1−1σ~1,…,min1,σ~m−1σ~m)V~⊤=T+P_+ = UDiag ( \1, σ_1^-1\ σ_1,…, \1, σ_m^-1\ σ_m ) V =T_+, i.e., T+=P+ΨT_+=P_+ . Since min1,λi−1/2=1,λi≤1,λi−1/2,λi>1, \1, _i^-1/2\= cases1,& _i≤ 1,\\ _i^-1/2,& _i>1, cases letting :=i:λi>1A:=\i: _i>1\ gives the equivalent representation of the preconditioner P+=Im+U~(Λ−1/2−I)U~⊤.P_+=I_m+ U_A ( _A^-1/2-I ) U_A . Thus, P+P_+ leaves modes with λi≤1 _i≤ 1 unchanged and contracts modes with λi>1 _i>1 by λi−1/2 _i^-1/2. Moreover, every eigenvalue of P+P_+ lies in (0,1](0,1], so 0≺P+⪯Im0 P_+ I_m. ∎ Proposition 2 (Rank-Constrained Spectral Refinement). Suppose that Γ(X)≻0 (X) 0. A closest rank-constrained update refinement to ΔΨ _A is obtained by solving Δ⋆∈argminrank(Δ)≤r12‖(Δ−Δ)Ψ‖F2. ∈ _rank( )≤ r 12 \|( - _A) \|_F^2. (8) Let ⊆C index the minr,|| \r,|A|\ largest eigenvalues satisfying λi>1 _i>1. Then one solution is Δ⋆=U~(Λ−1/2−I)U~⊤. = U_C ( _C^-1/2-I ) U_C . (9) The resulting preconditioner and update are P:=Im+Δ⋆P:=I_m+ and T:=PΨT:=P . Remark 4. Proposition 2 characterizes the optimal rank-constrained spectral refinement. Since Γ(X)≻0 (X) 0, Ψ has full row rank and rank(ΔΨ)=rank(Δ)rank( )=rank( ), so the rank constraint on the non-identity component is equivalent to that on its induced update refinement. The result then follows from the Eckart–Young–Mirsky theorem (Eckart and Young, 1936; Mirsky, 1960) applied to ΔΨ _A . Its nonzero singular values are |1−λi||1- _i| for i∈i and increase monotonically with λi _i, so the optimal truncation retains the active modes associated with the largest λi _i indexed by C. Proof. Recall that Ψ=U~Σ~V~⊤ = U V is a thin SVD, where Σ~=Diag(σ~1,…,σ~m) =Diag( σ_1,…, σ_m) with σ~1≥⋯≥σ~m>0 σ_1≥·s≥ σ_m>0. Suppose that Γ(X)=ΨΨ⊤=U~ΛU~⊤≻0 (X)= = U U 0, i.e., Ψ has full row rank and λi=σ~i2>0 _i= σ_i^2>0 for every i. From Proposition 1, with :=i:λi>1A:=\i: _i>1\ and P+=Im+U~(Λ−1/2−I)U~⊤P_+=I_m+ U_A ( _A^-1/2-I ) U_A , Δ=P+−Im=U~(Λ−1/2−I)U~⊤. _A=P_+-I_m= U_A ( _A^-1/2-I ) U_A . Applying this refinement to Ψ , and using that U~ U_A and V~ V_A collect the singular directions indexed by A, we have ΔΨ _A =U~(Λ−1/2−I)U~⊤U~Σ~V~⊤ = U_A ( _A^-1/2-I ) U_A U V =U~(Λ−1/2−I)Σ~V~⊤ = U_A ( _A^-1/2-I ) _A V_A =U~Diag(1−σ~i)i∈V~⊤, = U_ADiag (1- σ_i )_i V_A , where the last equality follows from λi=σ~i2 _i= σ_i^2. Since λi>1 _i>1 for i∈i , we have σ~i=λi>1 σ_i= _i>1. Therefore, the nonzero singular values of ΔΨ _A are |1−σ~i|=λi−1 |1- σ_i |= _i-1 for i∈i . For any feasible Δ with rank(Δ)≤rrank( )≤ r, we have rank(ΔΨ)≤rank(Δ)≤r.rank( ) ( )≤ r. Replacing ΔΨ by an arbitrary matrix Z with rank at most r enlarges the feasible set. Therefore, minrank(Z)≤r‖Z−ΔΨ‖F2≤minrank(Δ)≤r‖ΔΨ−ΔΨ‖F2. _rank(Z)≤ r\|Z- _A \|_F^2≤ _rank( )≤ r\| - _A \|_F^2. By the Eckart–Young–Mirsky theorem, the left-hand side is minimized by retaining the minr,|| \r,|A|\ largest singular values of ΔΨ _A , whose rank is |||A|. Since λi−1 _i-1 is increasing in λi _i, the optimal rank-r approximation retains the modes with the largest active eigenvalues. Let ⊆C index these modes. The resulting truncated approximation is Z⋆=U~Diag(1−σ~i)i∈V~⊤.Z = U_CDiag (1- σ_i )_i V_C . Define Δ⋆:=U~(Λ−1/2−I)U~⊤. := U_C ( _C^-1/2-I ) U_C . Since ||≤r|C|≤ r, we have rank(Δ⋆)≤rrank( )≤ r, so Δ⋆ is feasible for the original problem. Moreover, Δ⋆Ψ =U~(Λ−1/2−I)Σ~V~⊤ = U_C ( _C^-1/2-I ) _C V_C =U~Diag(1−σ~i)i∈V~⊤ = U_CDiag (1- σ_i )_i V_C =Z⋆, =Z , Thus, Δ⋆ attains the optimal rank-r approximation obtained from the relaxed problem. Together with the preceding lower bound, this gives minrank(Δ)≤r‖ΔΨ−ΔΨ‖F2=minrank(Z)≤r‖Z−ΔΨ‖F2. _rank( )≤ r\| - _A \|_F^2= _rank(Z)≤ r\|Z- _A \|_F^2. Therefore, Δ⋆ is a solution to Equation 8. The resulting preconditioner and refined update are P=Im+Δ⋆,T=PΨ,P=I_m+ , T=P , where Δ⋆:=U~(Λ−1/2−I)U~⊤. := U_C ( _C^-1/2-I ) U_C . ∎ D.2 Convergence Analysis of NormPre This section establishes convergence guarantees for NormPre-G and NormPre-L as stated in Theorem 1, with the Sketch-based implementation for NormPre-L given in Corollary 1. Throughout this convergence analysis, we use a fixed orientation kt≡k_t≡ k for both variants. For NormPre-G, we use the exact matrix-sign transformation and assume Γt≻0 _t 0 for all t. In the deterministic setting without momentum, let Gt:=∇L(Wt)G_t:=∇ L(W_t) and define the actively oriented parameter and gradient as W¯t:=kt(Wt) W_t:=O_k_t(W_t) and G¯t:=kt(Gt) G_t:=O_k_t(G_t). For their i-th rows w¯t,i w_t,i and g¯t,i g_t,i, respectively, define the relaxed tangent gradient xt,i:=g¯t,i−⟨g¯t,i,w¯t,i⟩w¯t,i.x_t,i:= g_t,i- g_t,i, w_t,i w_t,i. Decompose g¯t,i g_t,i into the tangent and radial components with respect to w¯t,i w_t,i: g¯t,i=gt,itan+gt,irad,gt,irad:=⟨g¯t,i,w¯t,i⟩‖w¯t,i‖22w¯t,i. g_t,i=g_t,i^tan+g_t,i^rad, g_t,i^rad:= g_t,i, w_t,i \| w_t,i\|_2^2 w_t,i. Then xt,i=gt,itan+(1−‖w¯t,i‖22)gt,irad.x_t,i=g_t,i^tan+ (1-\| w_t,i\|_2^2 )g_t,i^rad. Thus, xt,ix_t,i preserves the tangent component of g¯t,i g_t,i while retaining a norm-dependent radial component. In particular, it reduces to the exact tangent projection when ‖w¯t,i‖2=1\| w_t,i\|_2=1. Collecting the resulting rows gives Xt:=Wt,kt(Gt)X_t:=T_W_t,k_t(G_t). As in Section 2, we present the derivation for matrices in ℝm×nR^m× n under the row orientation; the column-oriented case follows by applying the same argument to the transpose. We assume throughout that the row normalization is well defined. Through row normalization of XtX_t, we get the normalized update Ψt:=D(Xt)−1Xt _t:=D(X_t)^-1X_t and its i-th row ψt,i _t,i. Applying spectral preconditioning, we get the refined update TtT_t. As in Algorithm 1, the resulting chain is G¯t→Relaxed Tangent ProjectionXt→Diagonal-Gram NormalizationΨt→Spectral PreconditioningTt. G_t Relaxed Tangent ProjectionX_t Diagonal-Gram Normalization _t Spectral PreconditioningT_t. The following lemmas quantify the gradient contribution retained by the normalized update, the spectral properties of the global and localized refinements and the radial interaction induced by each preconditioner. These quantities are used to lower bound the gradient-update inner product and control the smoothness term in the subsequent descent analyses. D.2.1 Lemmas Lemma 1. Let ϕt,i _t,i denote the angle between g¯t,i g_t,i and w¯t,i w_t,i. Suppose sin(ϕt,i)≥γ>0 ( _t,i)≥γ>0 for every row i, where γ represents the tangential fraction. With the diagonal-Gram normalization Ψt:=D(Xt)−1Xt _t:=D(X_t)^-1X_t in Algorithm 1, ⟨Xt,Ψt⟩F≥γ‖G¯t‖F. X_t, _t _F≥γ\| G_t\|_F. Proof. Since xt,ix_t,i preserves the tangent component of g¯t,i g_t,i while retaining a norm-dependent radial component along w¯t,i w_t,i, letting ϕt,i _t,i denote the angle between g¯t,i g_t,i and w¯t,i w_t,i gives ‖xt,i‖22=‖g¯t,i‖22[sin2(ϕt,i)+(1−‖w¯t,i‖22)2cos2(ϕt,i)]≥‖g¯t,i‖22sin2(ϕt,i).\|x_t,i\|_2^2=\| g_t,i\|_2^2 [ ^2( _t,i)+ (1-\| w_t,i\|_2^2 )^2 ^2( _t,i) ]≥\| g_t,i\|_2^2 ^2( _t,i). Thus, we get ‖xt,i‖2≥‖g¯t,i‖2sin(ϕt,i)\|x_t,i\|_2≥\| g_t,i\|_2 ( _t,i). Moreover, since Ψt=D(Xt)−1Xt _t=D(X_t)^-1X_t, its i-th row satisfies ψt,i=xt,i/‖xt,i‖2 _t,i=x_t,i/\|x_t,i\|_2. Then ⟨Xt,Ψt⟩F X_t, _t _F =∑i⟨xt,i,ψt,i⟩=∑i⟨xt,i,xt,i‖xt,i‖2⟩=∑i‖xt,i‖2≥∑i‖g¯t,i‖2sin(ϕt,i). = _i x_t,i, _t,i = _i x_t,i, x_t,i\|x_t,i\|_2 = _i\|x_t,i\|_2≥ _i\| g_t,i\|_2 ( _t,i). The lower bound becomes zero when the full gradient vanishes or when the gradient becomes parallel to the corresponding weight direction, sin(ϕt,i)=0 ( _t,i)=0. If we assume that the gradient is not perfectly aligned with the weight, i.e., the gradient maintains a tangential fraction sin(ϕt,i)≥γ>0 ( _t,i)≥γ>0, we obtain ⟨Xt,Ψt⟩F≥γ∑i‖g¯t,i‖2≥γ(∑i‖g¯t,i‖22)1/2=γ‖G¯t‖F. X_t, _t _F≥γ _i\| g_t,i\|_2≥γ ( _i\| g_t,i\|_2^2 )^1/2=γ\| G_t\|_F. Thus, this lemma quantifies the first-order gradient signal retained through the relaxed tangent projection and row normalization. ∎ Lemma 2. Consider the NormPre-G update with matrix-sign transformation and the NormPre-L update with Exact implementation in Algorithm 1 and Section 3.2. For NormPre-G, suppose Γt=ΨtΨt⊤≻0 _t= _t _t 0. Let PtP_t denote the corresponding spectral preconditioner and Tt=PtΨtT_t=P_t _t. Then (a) Inner product with relaxed tangent gradient: ⟨Xt,Tt⟩F≥λmin(Pt)⟨Xt,Ψt⟩F X_t,T_t _F≥ _ (P_t) X_t, _t _F. (b) Norm of the refined update: ‖Tt‖F≤maxm,n1/2\|T_t\|_F≤ \m,n\^1/2. (c) Minimum eigenvalue of preconditioner: λmin(Pt)≥λt,1−1/2≥maxm,n−1/2 _ (P_t)≥ _t,1^-1/2≥ \m,n\^-1/2, where the first inequality holds with equality for NormPre-G. Proof. Recall that XtX_t is the relaxed tangent gradient under the active orientation, Ψt=D(Xt)−1Xt _t=D(X_t)^-1X_t is the normalized update and Tt=PtΨtT_t=P_t _t is the update after spectral preconditioning. We follow the chain G¯t→Relaxed Tangent ProjectionXt→Diagonal-Gram NormalizationΨt→Spectral PreconditioningTt. G_t Relaxed Tangent ProjectionX_t Diagonal-Gram Normalization _t Spectral PreconditioningT_t. We first characterize how spectral preconditioning changes the inner product with the relaxed tangent gradient. Let Γt=ΨtΨt⊤=U~tΛtU~t⊤ _t= _t _t = U_t _t U_t , and let u~t,i u_t,i denote the i-th column of U~t U_t, with corresponding eigenvalue λt,i _t,i of Γt _t. For NormPre-G, Γt≻0 _t 0 and Pt=Γt−1/2=U~tΛt−1/2U~t⊤.P_t= _t^-1/2= U_t _t^-1/2 U_t . We have Ptu~t,i=λt,i−1/2u~t,iP_t u_t,i= _t,i^-1/2 u_t,i for every i. For NormPre-L, Pt=I+U~t(Λt−1/2−I)U~t⊤.P_t=I+ U_C_t( _C_t^-1/2-I) U_C_t . For i∈ti _t, Ptu~t,i=λt,i−1/2u~t,i,P_t u_t,i= _t,i^-1/2 u_t,i, while for i∉ti _t, Ptu~t,i=u~t,iP_t u_t,i= u_t,i. Since every selected mode satisfies λt,i>1 _t,i>1, the eigenvalues of PtP_t are λt,i−1/2∈(0,1) _t,i^-1/2∈(0,1) on the selected modes and 11 on the unselected modes. Therefore, for both realizations, PtP_t and Γt _t share the same eigenbasis and Pt≻0P_t 0. Since Pt⪰λmin(Pt)IP_t _ (P_t)I, PtΓt⪰λmin(Pt)Γt. P_t _t _ (P_t) _t. (10) With Ψt=D(Xt)−1Xt _t=D(X_t)^-1X_t, let Dt:=D(Xt)=Diag(‖xt,1‖2,…,‖xt,m‖2).D_t:=D(X_t)=Diag (\|x_t,1\|_2,…,\|x_t,m\|_2 ). Then Xt=DtΨtX_t=D_t _t and Dt⪰0D_t 0. Using Tt=PtΨtT_t=P_t _t and Γt=ΨtΨt⊤ _t= _t _t , ⟨Xt,Tt⟩F X_t,T_t _F =tr(Xt⊤Tt)=tr((DtΨt)⊤PtΨt) =tr\! (X_t T_t )=tr\! ((D_t _t) P_t _t ) =tr(DtPtΨtΨt⊤)=tr(DtPtΓt). =tr\! (D_tP_t _t _t )=tr(D_tP_t _t). (11) Together with Equation 10, we have tr(Dt(PtΓt−λmin(Pt)Γt))≥0,tr\! (D_t (P_t _t- _ (P_t) _t ) )≥ 0, and thus tr(DtPtΓt)≥λmin(Pt)tr(DtΓt). (D_tP_t _t)≥ _ (P_t)tr(D_t _t). (12) Since each row of Ψt _t has unit norm, diag(Γt)=diag( _t)=1. Then tr(DtΓt)=∑i(Dt)ii(Γt)ii=∑i‖xt,i‖2. (D_t _t)= _i(D_t)_i( _t)_i= _i\|x_t,i\|_2. (13) Combining Equations 11–13 gives ⟨Xt,Tt⟩F≥λmin(Pt)∑i‖xt,i‖2. X_t,T_t _F≥ _ (P_t) _i\|x_t,i\|_2. Since ψt,i=xt,i/‖xt,i‖2 _t,i=x_t,i/\|x_t,i\|_2, ⟨Xt,Ψt⟩F=∑i⟨xt,i,ψt,i⟩=∑i‖xt,i‖2. X_t, _t _F= _i x_t,i, _t,i = _i\|x_t,i\|_2. Therefore, ⟨Xt,Tt⟩F≥λmin(Pt)⟨Xt,Ψt⟩F. X_t,T_t _F≥ _ (P_t) X_t, _t _F. (14) This inequality quantifies the change in the inner product with the relaxed tangent gradient induced by exact spectral preconditioning: ⟨Xt,Ψt⟩F X_t, _t _F before preconditioning is transformed into ⟨Xt,Tt⟩F X_t,T_t _F after preconditioning, while retaining at least a λmin(Pt) _ (P_t) fraction. Together with Lemma 1, this provides the positive first-order descent signal used later in the descent analysis. We next bound the norm of the refined update TtT_t. Since PtP_t is symmetric and Tt=PtΨtT_t=P_t _t, TtTt⊤=PtΨtΨt⊤Pt=PtΓtPt.T_tT_t =P_t _t _t P_t=P_t _tP_t. Recall that PtP_t and Γt _t share the same eigenbasis, and let pt,ip_t,i denote the eigenvalue of PtP_t associated with u~t,i u_t,i. Then TtTt⊤=U~tDiag(pt,i2λt,i)U~t⊤.T_tT_t = U_tDiag(p_t,i^2 _t,i) U_t . For NormPre-G, pt,i=λt,i−1/2p_t,i= _t,i^-1/2 for every i. We have pt,i2λt,i=1p_t,i^2 _t,i=1 and ‖Tt‖F2=tr(TtTt⊤)=m.\|T_t\|_F^2=tr(T_tT_t )=m. For NormPre-L, pt,i=λt,i−1/2p_t,i= _t,i^-1/2 for i∈ti _t and pt,i=1p_t,i=1 for i∉ti _t. We have ‖Tt‖F2=tr(TtTt⊤)=∑i∈t1+∑i∉tλt,i≤∑iλt,i=tr(Γt)=m.\|T_t\|_F^2=tr(T_tT_t )= _i _t1+ _i _t _t,i≤ _i _t,i=tr( _t)=m. The inequality follows because every selected mode satisfies λt,i>1 _t,i>1. Therefore, for both realizations, ‖Tt‖F≤m1/2≤maxm,n1/2,\|T_t\|_F≤ m^1/2≤ \m,n\^1/2, which later controls the quadratic term in the smoothness inequality. Finally, we bound the minimum eigenvalue of the preconditioner. For NormPre-G, the eigenvalues of Pt=Γt−1/2P_t= _t^-1/2 are λt,i−1/2 _t,i^-1/2. We have λmin(Pt)=λt,1−1/2. _ (P_t)= _t,1^-1/2. For NormPre-L, the eigenvalues of PtP_t are λt,i−1/2 _t,i^-1/2 on the selected modes and 11 on the unselected modes. Since λt,i≤λt,1 _t,i≤ _t,1 for every selected mode and λt,1≥1 _t,1≥ 1, every eigenvalue of PtP_t is at least λt,1−1/2 _t,1^-1/2. Therefore, for both realizations, λmin(Pt)≥λt,1−1/2. _ (P_t)≥ _t,1^-1/2. Since tr(Γt)=mtr( _t)=m, we have λt,1≤m _t,1≤ m. Thus, λmin(Pt)≥λt,1−1/2≥m−1/2≥maxm,n−1/2. _ (P_t)≥ _t,1^-1/2≥ m^-1/2≥ \m,n\^-1/2. ∎ Lemma 3. Under the setting of Lemma 2, let νt:=maxi‖g¯t,i−xt,i‖2‖xt,i‖2 _t:= _i \| g_t,i-x_t,i\|_2\|x_t,i\|_2 denote the maximum radial ratio. Let ϕt,j,i′φ _t,j,i denote the angle between ψt,j _t,j and w¯t,i w_t,i, and suppose maxi(∑jcos2(ϕt,j,i′))1/2≤γ′ _i( _j ^2(φ _t,j,i))^1/2≤γ . For NormPre-G, let κt:=λt,1/λt,m _t:= _t,1/ _t,m and define χt:=κt,NormPre-G,λt,1,NormPre-L. _t:= cases _t,& NormPre-G,\\ _t,1,& NormPre-L. cases Then, we have |⟨G¯t−Xt,Tt⟩F|≤νtγ′χt⟨Xt,Tt⟩F. | G_t-X_t,T_t _F |≤ _tγ _t X_t,T_t _F. Proof. Recall that XtX_t is the rowwise relaxed tangent gradient of G¯t G_t and G¯t−Xt G_t-X_t is the corresponding radial residual. Using Tt=PtΨtT_t=P_t _t, we have ⟨G¯t−Xt,Tt⟩F=⟨G¯t−Xt,PtΨt⟩F. G_t-X_t,T_t _F= G_t-X_t,P_t _t _F. By the definition of νt _t, ‖g¯t,i−xt,i‖2‖xt,i‖2≤νt, \| g_t,i-x_t,i\|_2\|x_t,i\|_2≤ _t, or equivalently, ‖g¯t,i−xt,i‖2≤νt‖xt,i‖2\| g_t,i-x_t,i\|_2≤ _t\|x_t,i\|_2. We next bound the interaction between the radial residual and the spectrally refined update. From the definition of the relaxed tangent projection, g¯t,i−xt,i=⟨g¯t,i,w¯t,i⟩w¯t,i g_t,i-x_t,i= g_t,i, w_t,i w_t,i. Let eie_i denote the i-th standard basis vector. Expanding the Frobenius inner product rowwise gives |⟨G¯t−Xt,Tt⟩F| | G_t-X_t,T_t _F | =|⟨G¯t−Xt,PtΨt⟩F|=|∑i⟨g¯t,i−xt,i,ei⊤PtΨt⟩| = | G_t-X_t,P_t _t _F |= | _i g_t,i-x_t,i,e_i P_t _t | =|∑i⟨g¯t,i,w¯t,i⟩ei⊤PtΨtw¯t,i⊤|≤∑i‖g¯t,i−xt,i‖2‖ei⊤Pt‖2‖Ψtw¯t,i⊤‖2‖w¯t,i‖2 = | _i g_t,i, w_t,i e_i P_t _t w_t,i |≤ _i\| g_t,i-x_t,i\|_2\|e_i P_t\|_2 \| _t w_t,i \|_2\| w_t,i\|_2 ≤νt‖Pt‖op∑i‖xt,i‖2‖Ψtw¯t,i⊤‖2‖w¯t,i‖2. ≤ _t\|P_t\|_op _i\|x_t,i\|_2 \| _t w_t,i \|_2\| w_t,i\|_2. For all j and i, let ϕt,j,i′φ _t,j,i denote the angle between ψt,j _t,j and w¯t,i w_t,i. Since each row of Ψt _t has unit norm, ‖Ψtw¯t,i⊤‖2‖w¯t,i‖2=(∑jcos2(ϕt,j,i′))1/2≤γ′. \| _t w_t,i \|_2\| w_t,i\|_2= ( _j ^2(φ _t,j,i) )^1/2≤γ . This quantity measures the aggregate non-orthogonality between the i-th weight direction and the normalized relaxed tangent gradients. A smaller γ′γ corresponds to stronger orthogonality between the normalized relaxed tangent gradients and the weight directions. Therefore, |⟨G¯t−Xt,Tt⟩F|≤νtγ′‖Pt‖op∑i‖xt,i‖2. | G_t-X_t,T_t _F |≤ _tγ \|P_t\|_op _i\|x_t,i\|_2. (15) From Lemma 2(a), ⟨Xt,Tt⟩F≥λmin(Pt)∑i‖xt,i‖2. X_t,T_t _F≥ _ (P_t) _i\|x_t,i\|_2. Combining this inequality with Equation 15 gives |⟨G¯t−Xt,Tt⟩F|≤νtγ′‖Pt‖opλmin(Pt)⟨Xt,Tt⟩F. | G_t-X_t,T_t _F |≤ _tγ \|P_t\|_op _ (P_t) X_t,T_t _F. For NormPre-G, Pt=Γt−1/2P_t= _t^-1/2 and the eigenvalues of Γt _t satisfy λt,1≥⋯≥λt,m>0 _t,1≥·s≥ _t,m>0. We have ∥Pt∥op=λt,m−1/2,λmin(Pt)=λt,1−1/2.\|P_t\|_op= _t,m^-1/2, _ (P_t)= _t,1^-1/2. Therefore, ‖Pt‖opλmin(Pt)=λt,1λt,m=κt. \|P_t\|_op _ (P_t)= _t,1 _t,m= _t. For NormPre-L, all eigenvalues of PtP_t lie in (0,1](0,1], so ‖Pt‖op≤1\|P_t\|_op≤ 1. From Lemma 2(c), λmin(Pt)≥λt,1−1/2 _ (P_t)≥ _t,1^-1/2. Therefore, ‖Pt‖opλmin(Pt)≤λt,1. \|P_t\|_op _ (P_t)≤ _t,1. Combining the two cases gives |⟨G¯t−Xt,Tt⟩F|≤νtγ′χt⟨Xt,Tt⟩F. | G_t-X_t,T_t _F |≤ _tγ _t X_t,T_t _F. In summary, the radial interaction in the spectrally refined update is controlled relative to the inner product between the refined update and the relaxed tangent gradient. Its relative magnitude is determined by the radial ratio νt _t, the non-orthogonality γ′γ between the normalized row ψt,j _t,j and the weight w¯t,i w_t,i and the spectral factor χt _t. ∎ D.2.2 Convergence of NormPre-G and NormPre-L (Exact) We now combine Lemmas 1, 2 and 3 to prove the NormPre-G and NormPre-L (Exact) cases of Theorem 1. Proof. We consider NormPre-G with matrix-sign transformation and NormPre-L with Exact implementation in the deterministic setting without momentum, RMS scaling or weight decay. Let χt=κt _t= _t for NormPre-G and χt=λt,1 _t= _t,1 for NormPre-L, as in Lemma 3. Taking Gt=∇ℒ(Wt)G_t= (W_t), the final update is Wt+1=Wt−ηkt−1(Tt).W_t+1=W_t-η\,O_k_t^-1(T_t). Since ktO_k_t is either the identity or transpose, it preserves Frobenius inner products and norms. Hence, ⟨Gt,kt−1(Tt)⟩F=⟨G¯t,Tt⟩F G_t,O_k_t^-1(T_t) _F= G_t,T_t _F and ‖kt−1(Tt)‖F=‖Tt‖F\|O_k_t^-1(T_t)\|_F=\|T_t\|_F. Moreover, ‖G¯t‖F=‖Gt‖F=‖∇ℒ(Wt)‖F\| G_t\|_F=\|G_t\|_F=\| (W_t)\|_F. By the L-smoothness of ℒL, ℒ(Wt+1) (W_t+1) ≤ℒ(Wt)+⟨∇ℒ(Wt),Wt+1−Wt⟩F+L2‖Wt+1−Wt‖F2 (W_t)+ (W_t),W_t+1-W_t _F+ L2\|W_t+1-W_t\|_F^2 =ℒ(Wt)−η⟨Gt,kt−1(Tt)⟩F+Lη22‖kt−1(Tt)‖F2 =L(W_t)-η G_t,O_k_t^-1(T_t) _F+ Lη^22 \|O_k_t^-1(T_t) \|_F^2 =ℒ(Wt)−η⟨G¯t,Tt⟩F+Lη22‖Tt‖F2. =L(W_t)-η G_t,T_t _F+ Lη^22\|T_t\|_F^2. (16) Thus, the descent analysis reduces to lower bounding the first-order gradient signal ⟨G¯t,Tt⟩F G_t,T_t _F and upper bounding the refined update norm ‖Tt‖F\|T_t\|_F. We first lower bound ⟨G¯t,Tt⟩F G_t,T_t _F. Using the decomposition G¯t=Xt+(G¯t−Xt) G_t=X_t+( G_t-X_t), ⟨G¯t,Tt⟩F G_t,T_t _F =⟨Xt,Tt⟩F+⟨G¯t−Xt,Tt⟩F≥(1−νtγ′χt)⟨Xt,Tt⟩F = X_t,T_t _F+ G_t-X_t,T_t _F≥ (1- _tγ _t ) X_t,T_t _F ≥ϵ⟨Xt,Tt⟩F≥ϵλmin(Pt)⟨Xt,Ψt⟩F ≥ε X_t,T_t _F≥ε _ (P_t) X_t, _t _F ≥ϵγλmin(Pt)∥G¯t∥F≥ϵγmaxm,n−1/2∥G¯t∥F. ≥εγ _ (P_t)\| G_t\|_F≥εγ \m,n\^-1/2\| G_t\|_F. We now explain each step in this bound. The first inequality follows from Lemma 3, which gives |⟨G¯t−Xt,Tt⟩F|≤νtγ′χt⟨Xt,Tt⟩F. | G_t-X_t,T_t _F |≤ _tγ _t X_t,T_t _F. The second inequality follows from the assumption that there exists ϵ>0ε>0 such that max0≤t≤Tνtγ′χt≤1−ϵ. _0≤ t≤ T _tγ _t≤ 1-ε. This condition ensures that, in the worst case, the radial interaction can cancel at most a (1−ϵ)(1-ε) fraction of the positive first-order term ⟨Xt,Tt⟩F X_t,T_t _F, leaving at least an ϵε fraction in the full gradient-update inner product. Since Lemma 3 bounds the sign-indefinite term ⟨G¯t−Xt,Tt⟩F G_t-X_t,T_t _F only in magnitude, this provides a sufficient but not necessary condition for convergence. The third inequality follows from Lemma 2(a), which gives ⟨Xt,Tt⟩F≥λmin(Pt)⟨Xt,Ψt⟩F. X_t,T_t _F≥ _ (P_t) X_t, _t _F. The fourth inequality follows from Lemma 1, which gives ⟨Xt,Ψt⟩F≥γ‖G¯t‖F X_t, _t _F≥γ\| G_t\|_F. Finally, Lemma 2(c) gives λmin(Pt)≥maxm,n−1/2 _ (P_t)≥ \m,n\^-1/2. We next control the quadratic term in Equation 16. Lemma 2(b) gives ‖Tt‖F2≤maxm,n.\|T_t\|_F^2≤ \m,n\. Substituting the preceding lower and upper bounds into Equation 16 and using ‖G¯t‖F=‖∇ℒ(Wt)‖F\| G_t\|_F=\| (W_t)\|_F, gives ℒ(Wt+1) (W_t+1) ≤ℒ(Wt)−ηϵγmaxm,n−1/2∥∇ℒ(Wt)∥F+Lη22maxm,n. (W_t)-ηεγ \m,n\^-1/2\| (W_t)\|_F+ Lη^22 \m,n\. (17) Summing Equation 17 from t=0t=0 to T gives ηϵγmaxm,n−1/2∑t=0T∥∇ℒ(Wt)∥F≤ℒ(W0)−ℒ(WT+1)+(T+1)Lη22maxm,n.ηεγ \m,n\^-1/2 _t=0^T\| (W_t)\|_F (W_0)-L(W_T+1)+ (T+1)Lη^22 \m,n\. Since ℒ(WT+1)≥ℒinfL(W_T+1) _ and Δ0:=ℒ(W0)−ℒinf _0:=L(W_0)-L_ , ηϵγmaxm,n−1/2∑t=0T∥∇ℒ(Wt)∥F≤Δ0+(T+1)Lη22maxm,n.ηεγ \m,n\^-1/2 _t=0^T\| (W_t)\|_F≤ _0+ (T+1)Lη^22 \m,n\. Dividing by ηϵγmaxm,n−1/2(T+1)ηεγ \m,n\^-1/2(T+1) and using that the minimum does not exceed the average yields min0≤t≤T‖∇ℒ(Wt)‖F≤Δ0maxm,n1/2ηϵγ(T+1)+Lηmaxm,n3/22ϵγ. _0≤ t≤ T\| (W_t)\|_F≤ _0 \m,n\^1/2ηεγ(T+1)+ Lη \m,n\^3/22εγ. For any constant C>0C>0, choosing η=C/T+1η=C/ T+1 gives min0≤t≤T‖∇ℒ(Wt)‖F≤1T+1(Δ0maxm,n1/2ϵγC+LCmaxm,n3/22ϵγ). _0≤ t≤ T\| (W_t)\|_F≤ 1 T+1 ( _0 \m,n\^1/2εγ C+ LC \m,n\^3/22εγ ). For Δ0>0 _0>0, the bound is minimized by C⋆=2Δ0Lmaxm,n,η⋆=C⋆T+1=2Δ0Lmaxm,n(T+1).C = 2 _0L \m,n\, η = C T+1= 2 _0L \m,n\(T+1). Substituting C⋆C gives min0≤t≤T‖∇ℒ(Wt)‖F≤maxm,n2LΔ0ϵγT+1. _0≤ t≤ T\| (W_t)\|_F≤ \m,n\ 2L _0εγ T+1. Thus, both NormPre-G with matrix-sign transformation and NormPre-L with Exact implementation achieve an (T−1/2)O(T^-1/2) convergence rate under their corresponding conditions. ∎ D.2.3 Convergence of NormPre-L (Sketch) We next extend the preceding analysis to NormPre-L with the Sketch-based implementation by viewing the sketched localized update as a perturbation of the exact update constructed from the same normalized update Ψt _t. Corollary 1 (Convergence of NormPre-L (Sketch) w/o Momentum). Suppose that the assumptions of Theorem 1 hold for the sequence generated by NormPre-L with the Sketch-based implementation. Let PtP_t and P^t P_t denote the exact and sketch-based localized spectral preconditioners constructed from the same Ψt _t, respectively. Assume that there exists δ≥0δ≥ 0 such that ‖P^t−Pt‖op≤δ\| P_t-P_t\|_op≤δ and δmax0≤t≤Tλt,1<ϵγδ _0≤ t≤ T _t,1<εγ. For any constant C>0C>0, choosing η=C/T+1η=C/ T+1 gives min0≤t≤T‖∇ℒ(Wt)‖F≤1T+1(Δ0maxm,n1/2(ϵγ−δmax0≤t≤Tλt,1)C+LCmaxm,n3/22(ϵγ−δmax0≤t≤Tλt,1)). _0≤ t≤ T\| (W_t)\|_F≤ 1 T+1 ( _0 \m,n\^1/2 (εγ-δ _0≤ t≤ T _t,1 )C+ LC \m,n\^3/22 (εγ-δ _0≤ t≤ T _t,1 ) ). Proof. Let PtP_t and P^t P_t denote the localized spectral preconditioners constructed from the exact and sketch-based interaction eigenspaces, respectively (Section 3.2): Pt:=I+U~t(Λt−1/2−I)U~t⊤,P^t:=I+U^t(Λ^t−1/2−I)U^t⊤.P_t:=I+ U_C_t ( _C_t^-1/2-I ) U_C_t , P_t:=I+ U_C_t ( _C_t^-1/2-I ) U_C_t . For notational simplicity, tC_t denotes the selected index set in each implementation; the exact and sketch-based sets may differ in both membership and cardinality. The corresponding updates are Tt:=PtΨtT_t:=P_t _t and T^t:=P^tΨt T_t:= P_t _t. Both are evaluated from the same Ψt _t and differ only in the interaction eigenspace extraction. We first bound the resulting update perturbation. Using the assumption ‖P^t−Pt‖op≤δ\| P_t-P_t\|_op≤δ, we have ‖T^t−Tt‖op=‖(P^t−Pt)Ψt‖op≤‖P^t−Pt‖op‖Ψt‖op≤δλt,1, \| T_t-T_t\|_op=\|( P_t-P_t) _t\|_op≤\| P_t-P_t\|_op\| _t\|_op≤δ _t,1, where ‖Ψt‖op=λt,1\| _t\|_op= _t,1 follows from Γt=ΨtΨt⊤ _t= _t _t . We next lower bound the first-order gradient signal. By nuclear/operator norm duality, ⟨G¯t,T^t⟩F G_t, T_t _F ≥⟨G¯t,Tt⟩F−‖G¯t‖∗‖T^t−Tt‖op. ≥ G_t,T_t _F-\| G_t\|_*\| T_t-T_t\|_op. (18) As in Lemma 1, ‖xt,i‖2≥‖g¯t,i‖2sin(ϕt,i)\|x_t,i\|_2≥\| g_t,i\|_2 ( _t,i) with sin(ϕt,i)≥γ ( _t,i)≥γ. Then, ‖G¯t‖∗≤∑i‖g¯t,i‖2≤γ−1∑i‖xt,i‖2=γ−1⟨Xt,Ψt⟩F\| G_t\|_*≤ _i\| g_t,i\|_2≤γ^-1 _i\|x_t,i\|_2=γ^-1 X_t, _t _F. Therefore, ‖G¯t‖∗‖T^t−Tt‖op≤δλt,1γ⟨Xt,Ψt⟩F. \| G_t\|_*\| T_t-T_t\|_op≤ δ _t,1γ X_t, _t _F. (19) As derived in Appendix D.2.2 for NormPre-L (Exact), ⟨G¯t,Tt⟩F≥ϵλmin(Pt)⟨Xt,Ψt⟩F. G_t,T_t _F≥ε _ (P_t) X_t, _t _F. (20) Combining Equations 18–20 with λmin(Pt)≥λt,1−1/2 _ (P_t)≥ _t,1^-1/2 from Lemma 2, we obtain ⟨G¯t,T^t⟩F G_t, T_t _F ≥(ϵλt,1−1/2−δλt,1γ)⟨Xt,Ψt⟩F ≥ (ε _t,1^-1/2- δ _t,1γ ) X_t, _t _F =ϵγ−δλt,1γλt,1⟨Xt,Ψt⟩F = εγ-δ _t,1γ _t,1 X_t, _t _F ≥ϵγ−δλt,1λt,1‖G¯t‖F ≥ εγ-δ _t,1 _t,1\| G_t\|_F ≥(ϵγ−δmax0≤t≤Tλt,1)maxm,n−1/2∥G¯t∥F. ≥ (εγ-δ _0≤ t≤ T _t,1 ) \m,n\^-1/2\| G_t\|_F. Thus, the assumption δmax0≤t≤Tλt,1<ϵγδ _0≤ t≤ T _t,1<εγ preserves a positive first-order gradient signal. It remains to bound the sketched update norm. From P^t=I+U^t(Λ^t−1/2−I)U^t⊤ P_t=I+ U_C_t ( _C_t^-1/2-I ) U_C_t , since the columns of U^t U_C_t are orthonormal, P^t P_t has eigenvalues λ^t,i−1/2 λ_t,i^-1/2 along the selected directions and 11 on the non-selected ones. Since the selected modes satisfy λ^t,i>1 λ_t,i>1, all eigenvalues of P^t P_t lie in (0,1](0,1]. Therefore, ‖T^t‖F≤‖P^t‖op‖Ψt‖F≤‖Ψt‖F\| T_t\|_F≤\| P_t\|_op\| _t\|_F≤\| _t\|_F, which gives ‖T^t‖F2≤maxm,n\| T_t\|_F^2≤ \m,n\. Substituting the preceding first-order and norm bounds into the same L-smoothness derivation as in the proof of Appendix D.2.2 gives ℒ(Wt+1)≤ℒ(Wt)−η(ϵγ−δmax0≤t≤Tλt,1)maxm,n−1/2∥∇ℒ(Wt)∥F+Lη22maxm,n.L(W_t+1) (W_t)-η (εγ-δ _0≤ t≤ T _t,1 ) \m,n\^-1/2\| (W_t)\|_F+ Lη^22 \m,n\. Summing from t=0t=0 to T, using ℒ(WT+1)≥ℒinfL(W_T+1) _ and Δ0=ℒ(W0)−ℒinf _0=L(W_0)-L_ and choosing η=C/T+1η=C/ T+1 as in Theorem 1 yields min0≤t≤T‖∇ℒ(Wt)‖F≤1T+1(Δ0maxm,n1/2(ϵγ−δmax0≤t≤Tλt,1)C+LCmaxm,n3/22(ϵγ−δmax0≤t≤Tλt,1)). _0≤ t≤ T\| (W_t)\|_F≤ 1 T+1 ( _0 \m,n\^1/2 (εγ-δ _0≤ t≤ T _t,1 )C+ LC \m,n\^3/22 (εγ-δ _0≤ t≤ T _t,1 ) ). Therefore, under the condition δmax0≤t≤Tλt,1<ϵγδ _0≤ t≤ T _t,1<εγ with a positive margin independent of T, NormPre-L with the Sketch-based implementation preserves the (T−1/2)O(T^-1/2) convergence rate established for the Exact implementation. ∎