Paper deep dive
Does Muon Need Fine-Grained Spectral Shaping?
Meher Chaitanya, Tianyi Zhou, Aristides Gionis
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/13/2026, 3:28:29 AM
Summary
This paper investigates whether the Muon optimizer requires fine-grained spectral shaping or if a coarse two-band reweighting is sufficient. The authors introduce BulkBoost, a framework that partitions singular directions into 'spikes' (above a noise edge) and 'bulk' (below the edge). Using a shared gain parameter, BulkBoost increases the relative weight of the bulk, which contains the majority of singular modes but individually weak signal. Experiments on Pythia models show that this two-band approach outperforms Muon's flat profile and is competitive with or better than fine-grained spectral optimizers like Freon and Spectra, suggesting that useful spectral departures from flatness are low-dimensional.
Entities (11)
Relation Signals (7)
BulkBoost → isvariantof → Muon
confidence 95% · We introduce BulkBoost, a two-band spectral reweighting framework... for Muon's Nesterov input
Bulk → contains → 94-97% of singular modes
confidence 92% · approximately 94--97% of measured singular modes lie below an estimated noise edge
BulkBoost → outperforms → Muon
confidence 90% · Measured against Muon's flat profile... two-band variants achieve reductions of 0.073–0.147%
BB-Adaptive → uses → Marchenko-Pastur
confidence 90% · calibrate a Marchenko--Pastur reference edge for Muon's Nesterov input
BB-Static → uses → Fixed-Rank
confidence 90% · The fixed-rank rule, BB-Static, takes a prescribed fraction p of the leading directions
BulkBoost → iscompetitivewith → Freon
confidence 85% · two-band reweighting is competitive with the fine-grained power-law profile of Freon
BulkBoost → outperforms → Spectra
confidence 85% · outperforms Spectra
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Muon combines current and past gradients into matrix momentum. For $M=U\Sigma V^\top$, the idealized polar update $Q=UV^\top$ gives every singular direction the same weight. We refer to this as the flat profile. Several recent optimizers replace this flat profile with fine-grained spectral maps that give each direction its own gain. We ask how much of this spectral detail a Muon update needs. Our spectral diagnostics show that approximately $94$--$97\%$ of measured singular modes lie below an estimated noise edge, yet collectively align positively with a reference gradient. We introduce BulkBoost, a two-band spectral reweighting framework with fixed-rank and noise-calibrated variants. The latter uses split-minibatch gradient differences to calibrate a Marchenko--Pastur reference edge for Muon's Nesterov input, separating the bulk below the edge from the spikes above it. Both variants increase the bulk's relative weight through one shared gain while preserving the Frobenius norm of each matrix's unreweighted direction. For a fixed partition, our theory gives the first-order condition under which moving weight toward the bulk lowers the loss. It also quantifies the fraction of the maximal first-order improvement rate, over all per-mode reallocations, that two bands can capture. Across 30 continued-pretraining settings spanning Pythia-14M to 410M and six corpora, two-band reweighting is competitive with the fine-grained power-law profile of Freon and outperforms Spectra. Measured against Muon's flat profile, Freon reduces final loss by $0.022\%$ of the pre-adaptation loss on average, whereas the two-band variants achieve reductions of $0.073$--$0.147\%$. These observations suggest that useful departures from the flat profile are surprisingly low-dimensional: a single bulk-to-spike gain captures at least as much benefit as the fine-grained spectral profiles.
Tags
Links
- Source: https://arxiv.org/abs/2610.07497v1
- Canonical: https://arxiv.org/abs/2610.07497v1
Trouble viewing inline? Open PDF directly →
Full Text
192,704 characters extracted from source content.
Expand or collapse full text
Does Muon Need Fine-Grained Spectral Shaping? Meher Chaitanya Tianyi Zhou & Aristides Gionis Affiliation: KTH Royal Institute of Technology Affiliation: Stockholm, Sweden Email: mcpi,tzho,argioni@kth.se Abstract Muon combines current and past gradients into matrix momentum. For M=UΣV⊤M=U V , the idealized polar update Q=UV⊤Q=UV gives every singular direction the same weight. We refer to this as the flat profile. Several recent optimizers replace this flat profile with fine-grained spectral maps that give each direction its own gain. We ask how much of this spectral detail a Muon update needs. Our spectral diagnostics show that approximately 9494–97%97\% of measured singular modes lie below an estimated noise edge, yet collectively align positively with a reference gradient. We introduce BulkBoost, a two-band spectral reweighting framework with fixed-rank and noise-calibrated variants. The latter uses split-minibatch gradient differences to calibrate a Marchenko–Pastur reference edge for Muon’s Nesterov input, separating the bulk below the edge from the spikes above it. Both variants increase the bulk’s relative weight through one shared gain while preserving the Frobenius norm of each matrix’s unreweighted direction. For a fixed partition, our theory gives the first-order condition under which moving weight toward the bulk lowers the loss. It also quantifies the fraction of the maximal first-order improvement rate, over all per-mode reallocations, that two bands can capture. A polar-plus-low-rank representation enables an approximation without a full SVD, tracking only the spike subspace. Across 3030 continued-pretraining settings spanning Pythia-1414M to 410410M and six corpora, two-band reweighting is competitive with the fine-grained power-law profile of Freon (Shumaylov et al., 2026) and outperforms Spectra (Huang et al., 2026). Measured against Muon’s flat profile, Freon reduces final loss by 0.022%0.022\% of the pre-adaptation loss on average, whereas the two-band variants achieve reductions of 0.0730.073–0.147%0.147\%. These observations suggest that useful departures from the flat profile are surprisingly low-dimensional: a single bulk-to-spike gain captures at least as much benefit as the fine-grained spectral profiles. 1 Introduction An optimizer for neural-network weight matrices decides two things: how large an update is, and how the update is spread across directions. Muon, an optimizer for language-model training, combines current and past gradients into a matrix momentum M (Jordan et al., 2024; Liu et al., 2025). Its SVD, M=UΣV⊤M=U V , writes M as a sum of orthogonal directions weighted by their singular values. The idealized polar update Q=UV⊤Q=UV keeps these directions but sets every singular value to one. Each direction therefore receives the same gain. We refer to this as the flat profile. Recent optimizers depart from this flat profile through spectral maps Ug(Σ)V⊤Ug( )V , where the gain varies across singular directions (Shumaylov et al., 2026; Dong & Sawin, 2026; Wu et al., 2026; Pang et al., 2026; Feoktistov et al., 2026). At the same time, Shumaylov et al. (2026) show that substantially different spectra, including random and inverted ones, can remain effective. These results raise a more basic question than which detailed profile is best: does Muon need fine-grained spectral shaping, or is a coarse reallocation between two groups of directions enough? Two observations motivate this question. First, a direction’s strength in the sampled momentum, its separation from sampling noise, and the benefit of increasing its update weight are distinct quantities. A small singular value can still carry signal, and a large one need not deserve more weight. Second, our diagnostics reveal a broad spectral bulk and a small leading band. Under a Gaussian white-noise reference model, random matrix theory provides an asymptotic upper spectral edge, the noise edge (Marchenko & Pastur, 1967; Baik et al., 2005; Benaych-Georges & Nadakuditi, 2012). We call directions above this reference edge spikes and the remaining directions the bulk. On Pythia-70M Muon trajectories, approximately 93.793.7–97.1%97.1\% of pooled singular modes lie in the bulk (Appendix D). These below-edge singular modes, the bulk, are weak individually, yet their aggregate signed alignment with an independently sampled reference gradient is positive in every analyzed matrix. Thus, the broad spectral bulk makes a non-negligible collective contribution even though its individual directions are weak. These measurements motivate a two-band partition. We introduce BulkBoost, whose exact target reweights Muon’s polar direction in two bands. The spike and bulk gains are 11 and ρ before normalization, and the update is then rescaled to keep the Frobenius norm of the unreweighted direction. At the exact-target level, ρ=1ρ=1 recovers Q, while ρ>1ρ>1 increases the bulk’s relative weight. Once the partition is chosen, the same ρ controls reallocation across all spectrally updated matrices. We consider two rules for the split. The fixed-rank rule, B-Static, takes a prescribed fraction p of the leading directions, shared across matrices and held fixed within each run. Our shared configuration uses p=5%p=5\%, within the pooled spike fraction of 2.92.9–6.3%6.3\% measured in our diagnostics (Appendix D). We also test smaller and larger fractions. The noise-calibrated rule, B-Adaptive, estimates the sampling noise from two minibatch halves, propagates it through the momentum, and places the split at the resulting noise edge for each matrix and step. The edge assumes isotropic Gaussian noise. Since the real gradient noise is not isotropic, this split is an approximation. Our contributions are as follows. 1. A two-band reweighting rule. BulkBoost reweights Muon’s flat update in two bands with one shared gain. The split between the bands comes either from a fixed fraction of directions or from a noise-calibrated edge. An exact polar-plus-low-rank identity lets us avoid a full SVD (Section 3). 2. What the two bands capture. For a fixed partition, we characterize to first order when shifting weight toward the bulk lowers the loss, and how much of the best per-direction improvement two bands can reach (Section 4). 3. Evidence from the spectrum. Our spectral diagnostics establish that most singular directions of Muon’s momentum lie below the noise edge. This bulk is weak direction by direction, yet it carries useful signal in aggregate. Giving it more relative weight lowers the loss after a single update in most tested states (Appendices D and E.2). 4. Competitive gains without fine-grained shaping. In continued pretraining settings spanning Pythia-1414M to 410410M and six corpora, two-band reweighting gives, on average, a slightly larger improvement over Muon’s flat profile than the fine-grained direction-specific spectral profiles we compare against. Thus, a single shared gain between spikes and bulk is enough in these settings (Section 5). Related work is provided in Appendix A. 2 Setup and Spectral Reweighting Muon and notation. Consider a matrix parameter Wt∈ℝm×nW_t ^m× n with minibatch gradient GtG_t after any global gradient clipping. Let d=minm,nd= \m,n\ and [d]=1,…,d[d]=\1,…,d\. We use ⟨A,B⟩F=tr(A⊤B) A,B _F=tr(A B), and ‖A‖F\|A\|_F and ‖A‖2\|A\|_2 denote the Frobenius and spectral norms. For β∈[0,1)β∈[0,1) and B0=0B_0=0, the exponential moving-average (EMA) buffer and the Nesterov spectral input are, for t≥1t≥ 1, Bt=βBt−1+(1−β)Gt,Mt=βBt+(1−β)Gt.B_t=β B_t-1+(1-β)G_t, M_t=β B_t+(1-β)G_t. (1) Let Mt=UtΣtVt⊤M_t=U_t _tV_t be its thin singular value decomposition (SVD), with singular values σt,1≥⋯≥σt,d _t,1≥·s≥ _t,d and paired singular vectors ut,i,vt,iu_t,i,v_t,i. Muon’s idealized direction is Qt=polar(Mt)=UtVt⊤Q_t=polar(M_t)=U_tV_t , which gives unit gain to every singular direction (Jordan et al., 2024; Liu et al., 2025). In practice, Muon approximates QtQ_t with a few Newton–Schulz iterations. All spectral quantities refer to MtM_t. The exact formulas below assume full-rank inputs. Appendix B.3 extends them to rank-deficient inputs by restricting to nonzero singular modes. Two-band partitions. A partition, which we also call a split, divides the singular directions of MtM_t into a spike band ℐsp,t⊆[d]I_sp,t [d] and a bulk band ℐbu,t=[d]∖ℐsp,tI_bu,t=[d] _sp,t. Both rules we consider put the leading directions in the spike band, so ℐsp,t=1,…,rtI_sp,t=\1,…,r_t\. The rules differ only in how they choose rtr_t. The fixed-rank rule, B-Static, sets rt=⌈pd⌉r_t= pd for a fixed fraction p∈(0,1)p∈(0,1). The noise-calibrated rule, B-Adaptive, compares the singular values with a threshold τt _t: ℐsp,t=i∈[d]:σt,i>τt,ℐbu,t=i∈[d]:σt,i≤τt.I_sp,t=\i∈[d]: _t,i> _t\, _bu,t=\i∈[d]: _t,i≤ _t\. (2) The threshold is based on the noise edge. For a large Gaussian white-noise matrix with entrywise standard deviation s, the top singular value approaches s(m+n)s( m+ n) (Marchenko & Pastur, 1967; Geman, 1980). Section 3 calibrates this edge for the Nesterov input using two minibatch halves. Real gradient noise is not isotropic Gaussian, so the edge is an approximate reference for the split. Norm-matched spectral reweighting. A nonzero profile wt∈[0,∞)dw_t∈[0,∞)^d specifies unnormalized gains along the singular directions of MtM_t. Its exact norm-matched direction is Dt(wt)=ct(wt)Utdiag(wt)Vt⊤,ct(wt)=d∑i=1dwt,i2.D_t(w_t)=c_t(w_t)U_tdiag(w_t)V_t , c_t(w_t)= d _i=1^dw_t,i^2. (3) Thus ‖Dt(wt)‖F=‖Qt‖F=d\|D_t(w_t)\|_F=\|Q_t\|_F= d, which fixes the per-matrix direction norm before learning-rate scaling. A scalar map g:(0,∞)→[0,∞)g:(0,∞)→[0,∞) induces the profile wt,i=g(σt,i)w_t,i=g( _t,i), and we write Dt(g):=Dt(wt)D_t(g):=D_t(w_t) when this profile is nonzero. Fine-grained profiles, such as the power law of Freon (Shumaylov et al., 2026), have this form: each direction’s gain depends on its own singular value. BulkBoost instead uses the singular values only to assign bands. Its two-level profile has one bulk-to-spike gain ρ≥0ρ≥ 0, shared by all matrices: wt,i(ρ)=1,i∈ℐsp,t,ρ,i∈ℐbu,t,cρ,rt:=ct(wt(ρ))=drt+ρ2(d−rt).w_t,i^(ρ)= cases1,&i _sp,t,\\ ρ,&i _bu,t, cases c_ρ,r_t:=c_t\! (w_t^(ρ) )= dr_t+ρ^2(d-r_t). (4) For a nonzero profile, the applied spike and bulk gains are cρ,rtc_ρ,r_t and ρcρ,rtρ c_ρ,r_t. At ρ=1ρ=1, the profile is flat and the direction is QtQ_t. With both bands nonempty, ρ>1ρ>1 gives the bulk more weight relative to the spikes, and 0≤ρ<10≤ρ<1 gives it less. Because norm matching rescales both bands by cρ,rtc_ρ,r_t, ρ sets only the ratio of the two gains. Multiplying the bulk gains by ρ and dividing the spike gains by ρ therefore give the same update. For small spike fractions and ρ>1ρ>1, the bulk gain stays close to one, and most of the change comes from a smaller spike gain. Appendix B gives further details on normalization identities, relative and effective gains, and conventions for rank-deficient inputs and empty bands. Section 3 realizes this direction with a numerical polar approximation Q^t Q_t. 3 Noise-Edge Reweighted Muon: BulkBoost This section describes how BulkBoost computes its update. 3.1 Noise-Calibrated Partitioning Here we describe how B-Adaptive places its threshold. At fixed parameters, let Gt(1),Gt(2)G_t^(1),G_t^(2) be the raw mean gradients of two independent, equal-sized minibatch halves from the same sampling distribution. Define Gtraw=Gt(1)+Gt(2)2,Ztsplit=Gt(1)−Gt(2)2.G_t^raw= G_t^(1)+G_t^(2)2, Z_t^split= G_t^(1)-G_t^(2)2. (5) The half-batch difference ZtsplitZ_t^split has zero mean. After vectorization, it has the same covariance as the sampling noise in the raw full-minibatch mean. Let χt∈(0,1] _t∈(0,1] be the full-minibatch clipping scalar computed across all model parameters, with χt=1 _t=1 without clipping. We use Gt=χtGtrawG_t= _tG_t^raw in equation 1 and do not clip the halves separately. We apply χt2 _t^2 to the raw variance estimate before smoothing: s^t 2=‖Ztsplit‖F2mn,s¯t 2=γs¯t−1 2+(1−γ)χt2s^t 2. s_t^\,2= \|Z_t^split\|_F^2mn, s_t^\,2=γ s_t-1^\,2+(1-γ) _t^2 s_t^\,2. (6) For γ∈[0,1)γ∈[0,1), each matrix’s EMA starts at s¯1 2=χ12s^1 2 s_1^\,2= _1^2 s_1^\,2, and the recursion applies for t≥2t≥ 2. The threshold must match the noise in the Nesterov input MtM_t, which differs from the noise in the gradient or in the EMA buffer. From equation 1, Mt=(1−β2)Gt+β2Bt−1M_t=(1-β^2)G_t+β^2B_t-1, so the current gradient enters both directly and through the updated buffer. Under a reference model with centered, temporally independent, isotropic input noises of constant entrywise variance s2s^2 and B0=0B_0=0, the propagated noise variance in MtM_t is νt2(β)s2 _t^2(β)s^2, where νt2(β)=(1−β)2[(1+β)2+β4(1−β2(t−1))1−β2],t≥1. _t^2(β)=(1-β)^2 [(1+β)^2+ β^4(1-β^2(t-1))1-β^2 ], t≥ 1. (7) Using its nonnegative square root νt(β) _t(β), we set the threshold to the estimated noise edge of MtM_t: τt=τ^t=κedgeνt(β)s¯t 2(m+n). _t= τ_t= _edge _t(β) s_t^\,2( m+ n). (8) with a fixed multiplier κedge>0 _edge>0 (κedge=1 _edge=1 in all experiments) and the square root taken after variance smoothing. Proposition 4 proves the half-batch and Nesterov noise formulas, and Appendix C.1 states the assumptions behind the threshold. 3.2 Structure and SVD-Free Realization Exact target. For either rule, the oracle evaluates Dtsvd=Dt(wt(ρ))D_t^svd=D_t(w_t^(ρ)) using equation 3 and equation 4. For the spike singular-vector blocks Ut,sp,Vt,spU_t,sp,V_t,sp, define Pt,sp=Ut,spUt,sp⊤P_t,sp=U_t,spU_t,sp and Rt,sp=Vt,spVt,sp⊤R_t,sp=V_t,spV_t,sp . Since Pt,spQt=QtRt,sp=Ut,spVt,sp⊤P_t,spQ_t=Q_tR_t,sp=U_t,spV_t,sp , Dtsvd=cρ,rt[ρQt−(ρ−1)Pt,spQt],m≤n,cρ,rt[ρQt−(ρ−1)QtRt,sp],n<m.D_t^svd= casesc_ρ,r_t[ρ Q_t-(ρ-1)P_t,spQ_t],&m≤ n,\\ c_ρ,r_t[ρ Q_t-(ρ-1)Q_tR_t,sp],&n<m. cases (9) The flat update QtQ_t carries the bulk, and a correction of rank at most rtr_t adjusts the spikes. Theorem 2 extends this identity to direction-dependent spike gains. Both projector expressions are exact for either orientation of the matrix. We keep both cases as the tracking below works on the smaller side. Spike-subspace tracking. Both variants track the spike subspace on the smaller Gram side, Kt=MtMt⊤,m≤n,Mt⊤Mt,n<m.K_t= casesM_tM_t ,&m≤ n,\\ M_t M_t,&n<m. cases (10) We evaluate KtK_t through products with MtM_t and Mt⊤M_t , without forming the Gram matrix. B-Static keeps a block of fixed width ⌈pd⌉ pd and refines it with q block-power iterations at each step. B-Adaptive must also find how many directions lie above the threshold. It augments the previously retained basis Xt−1X_t-1 with fresh standard Gaussian probes Ωt _t and sets Y0=orth([Xt−1,Ωt])Y_0=orth([X_t-1, _t]), where orthorth returns an orthonormal basis of the column space. After q updates Yj+1=orth(KtYj)Y_j+1=orth(K_tY_j), the eigenvalues of Yq⊤KtYqY_q K_tY_q estimate squared singular values, and multiplying its eigenvectors by YqY_q gives the Ritz directions. We keep the Ritz directions whose eigenvalues exceed τ^t 2 τ_t^\,2, in decreasing order, up to an absolute rank cap rmaxr_ . Reorthonormalizing them gives Xt∈ℝd×r^tX_t ^d× r_t. Finite iterations, the size of the trial space, and a binding rank cap can make the retained rank r^t r_t differ from the exact rank rtr_t. Appendix C.3 gives initialization, probe generation, and selection details. Numerically norm-matched update. We compute a numerical polar factor Q^t=PolarExpress(Mt) Q_t=PolarExpress(M_t) (Amsel et al., 2026a) and form D~t=ρQ^t−(ρ−1)Xt(Xt⊤Q^t),m≤n,ρQ^t−(ρ−1)(Q^tXt)Xt⊤,n<m. D_t= casesρ Q_t-(ρ-1)X_t(X_t Q_t),&m≤ n,\\ ρ Q_t-(ρ-1)( Q_tX_t)X_t ,&n<m. cases (11) For nonzero Q^t Q_t and D~t D_t, we set D^t=c^ρ,tD~t,c^ρ,t=‖Q^t‖F‖D~t‖F. D_t= c_ρ,t D_t, c_ρ,t= \| Q_t\|_F\| D_t\|_F. (12) In exact arithmetic, ‖D^t‖F=‖Q^t‖F\| D_t\|_F=\| Q_t\|_F, and ρ=1ρ=1 returns the unreweighted direction Q^t Q_t. A zero Q^t Q_t gives a zero update. Otherwise, an orthonormal XtX_t ensures a nonzero raw direction for ρ>0ρ>0, while ρ=0ρ=0 requires a nonzero projected spike component. Since ‖Q^t‖F\| Q_t\|_F need not equal d d, c^ρ,t c_ρ,t can differ from cρ,r^tc_ρ, r_t. Norm matching fixes the magnitude only, it does not correct polar or subspace approximation errors (Appendix B.4). Algorithm 1 in Appendix C summarizes the realization. Section 4 characterizes when this reallocation lowers the loss to first order. 4 Theory of Two-Band Reweighting We first give a first-order test for when two-band reweighting lowers the loss, and quantify how much of the best per-direction improvement two bands can capture. We then give the exact structure behind the SVD-free realization. 4.1 What Two Bands Can Capture A controlled update comparison. Starting from the same training state, we vary only the applied gains of one matrix’s current update. Fix step t, the singular frame Ut,VtU_t,V_t of MtM_t, and a continuation policy π, which specifies how training continues. Let ℒt,Kπ(w)L^π_t,K(w) be the expected evaluation loss, given the current state, after using Utdiag(w)Vt⊤U_tdiag(w)V_t as the current direction and then following π for K≥0K≥ 0 more steps. The learning rate, the current updates of all other parameters, and the rest of the current step are fixed. Later gradients and optimizer states evolve under π. At K=0K=0, the loss is evaluated right after the current update. Here w denotes the applied gains, that is, the normalized gains ct(wt)wtc_t(w_t)w_t from equation 3. Norm matching keeps w on d=w∈[0,∞)d:∥w∥22=dS_d=\w∈[0,∞)^d:\|w\|_2^2=d\, and the flat profile w=dw=1_d gives Muon’s idealized update QtQ_t. Continuation utility. We take the loss to be defined on an open neighborhood of d1_d in the positive orthant. Set ui=−∂ℒt,Kπ(w)∂wi|w=d,u¯=1d∑i=1dui,u⟂=u−u¯d.u_i=- . ^π_t,K(w)∂ w_i |_w=1_d, u= 1d _i=1^du_i, u_ =u- u1_d. (13) The continuation utility uiu_i is the rate at which the loss falls when the applied gain of direction i increases and the other gains stay fixed. Norm matching forces compensating changes across directions, so first-order improvements depend only on differences between utilities, which u⟂u_ captures. Fix a partition with r=|ℐsp|r=|I_sp|, k=|ℐbu|=d−rk=|I_bu|=d-r, and 1≤r≤d−11≤ r≤ d-1, dropping the time index. Let u¯sp=r−1∑i∈ℐspui u_sp=r^-1 _i _spu_i and u¯bu=k−1∑i∈ℐbuui u_bu=k^-1 _i _buu_i be the band means, with gap Δ=u¯bu−u¯sp = u_bu- u_sp. For this partition, varying ρ gives the norm-matched two-level gains w(ρ)=cρ,r(sp+ρbu)w(ρ)=c_ρ,r(1_sp+ 1_bu), where sp1_sp and bu1_bu indicate band membership and cρ,rc_ρ,r is defined in equation 4. Theorem 1 (Local spectral improvement). Assume ℒt,KπL^π_t,K is differentiable at d1_d. (i) Which allocation improves loss? Along the two-level family, −dρℒt,Kπ(w(ρ))|ρ=1=rkdΔ- . ddρL^π_t,K(w(ρ)) |_ρ=1= rkd . Increasing ρ from 11 gives a strictly positive first-order improvement if and only if Δ>0 >0. If Δ=0 =0, the first-order test is inconclusive. (i) How much improvement can two bands capture? At the flat profile, norm matching requires small gain changes to sum to zero, so they lie in =δ∈ℝd:d⊤δ=0T=\δ ^d:1_d δ=0\. If every gain may change separately, the best improvement rate per unit change is maxδ∈,‖δ‖2=1u⊤δ=‖u⟂‖2 _δ ,\ \|δ\|_2=1u δ=\|u_ \|_2. Thus the flat profile is first-order stationary in this fixed-frame, norm-matched class if and only if u1=⋯=udu_1=·s=u_d. If the change must be constant within each band, the best rate is rk/d|Δ| rk/d\,| |. For u⟂≠0u_ ≠ 0, two bands therefore reach the fraction Rbt=rk/d|Δ|‖u⟂‖2∈[0,1]R_bt= rk/d\,| |\|u_ \|_2∈[0,1] of the best rate. Both rates are per unit Euclidean change in w, and RbtR_bt allows ρ to move in either direction. Two bands can use only the difference between the band means. Near the flat profile, any norm-matched reweighting of the same singular directions, however fine-grained, moves w along some δ∈δ . Its first-order rate is therefore at most ‖u⟂‖2\|u_ \|_2. Finer shaping can thus add at most the share 1−Rbt1-R_bt, which comes from utility differences within the bands. The split rule, whether the noise edge or a fixed fraction, only decides which directions fall in each band. The sign of Δ and the size of RbtR_bt depend on the utilities. When Δ>0 >0, small enough increases in ρ lower the loss. The theorem does not say how large ρ may be, so it does not justify a specific finite gain such as ρ=4ρ=4. Here, first order refers to small changes in the gains; the parameter update itself is a finite step. With one ρ shared by all matrices, each norm-matched separately, the rates from Theorem 1(i) add up across matrices under joint differentiability (Appendix E.2). Appendices E.2 and G.2 test finite values of ρ. Appendix F.1 gives the proof. 4.2 Exact Polar-Plus-Low-Rank Structure A shared bulk gain also gives the update a simple form. The polar factor carries the bulk, and only a correction in the spike subspace remains. Theorem 2 (Exact polar-plus-low-rank decomposition). Let M∈ℝm×nM ^m× n have full rank d=minm,nd= \m,n\, thin SVD M=UΣV⊤M=U V , and polar factor Q=UV⊤Q=UV . For r selected singular directions, let Usp,VspU_sp,V_sp contain their paired vectors. Assign gains a∈[0,∞)ra∈[0,∞)^r to these directions and a common gain ρ≥0ρ≥ 0 to the others. The resulting unnormalized spectral update satisfies D~(a,ρ)=ρQ+Uspdiag(a−ρr)Vsp⊤, D(a,ρ)=ρ Q+U_spdiag(a- 1_r)V_sp , (14) and rank(D~(a,ρ)−ρQ)≤rrank( D(a,ρ)-ρ Q)≤ r. With a=ra=1_r, write D~ρ=D~(r,ρ) D_ρ= D(1_r,ρ). Normalizing it gives the projector form equation 9, which needs no bulk singular vectors. The squared Frobenius norm of D~(a,ρ) D(a,ρ) is ‖a‖22+ρ2(d−r)\|a\|_2^2+ρ^2(d-r), so for a=ra=1_r the normalizer cρ,rc_ρ,r restores ‖Q‖F=d\|Q\|_F= d (Corollary 3). The correction is low rank. Appendices F.2 and F.3 give the proofs. 5 Experiments This section tests whether two bands are enough in practice. The diagnostics in Appendix D motivate this test. On Pythia-70M Muon trajectories, 94–97% of singular directions lie below the noise edge, and this bulk is weak per direction but positively aligned with a reference gradient in aggregate. We compare both BulkBoost variants, B-Static and B-Adaptive, with Muon’s flat profile and with fine-grained spectral profiles in continued pretraining. We also compare the two variants with each other. Extended results appear in Appendix G. Figure 1: Held-out loss versus training tokens on Wiki-zh, one column per model scale. Top: the full run, with the pre-adaptation loss ℓ0 _0 dotted and the zoomed interval shaded. Bottom: the shaded interval at a finer vertical scale. The B-Static curve uses p=1.5%p=1.5\%, the leading fraction of Spectra, so the two can be compared directly. Appendix G.3 shows the other corpora. BulkBoost configurations. Unless varied in an ablation, both variants use ρ=4ρ=4, Q^t=PolarExpress(Mt) Q_t=PolarExpress(M_t) (Amsel et al., 2026a), q=6q=6 warm-started block iterations for spike tracking, and the norm matching in equation 12. Like Newton–Schulz, Polar Express uses only matrix products. It approximates the polar factor more closely, which the correction in equation 11 needs. Muon and both variants use Nesterov momentum β=0.95β=0.95. B-Adaptive splits at τ^t τ_t (Section 3.1) with γ=0.95γ=0.95, κedge=1 _edge=1, and the rank cap of Appendix C.3. B-Static keeps rt=⌈pd⌉r_t= pd spike directions, with one fraction p for all matrices and the whole run. We test p∈0.015,0.02,0.05,0.15,0.20p∈\0.015,0.02,0.05,0.15,0.20\ (Table 9). Our shared configuration, B-Static(5%), uses p=5%p=5\%, which lies within the 2.9–6.3% spike fraction measured in Appendix D. B-Static† instead reports, for each setting, the fraction with the lowest mean final held-out loss. Because this choice uses the reported loss, B-Static† shows what per-setting tuning can reach. Compared methods. We compare both BulkBoost variants with five baselines. Muon (Jordan et al., 2024) approximates the flat profile with Newton–Schulz iterations and is the reference for all relative comparisons. Two baselines use fine-grained spectral profiles. Freon (Shumaylov et al., 2026) uses a power-law profile with c=0.667c=0.667; c=12c= 12 recovers the flat profile. Spectra (Huang et al., 2026) rescales the leading singular values to the root-mean-square level of the tail. It keeps the tail’s unequal singular values, whereas BulkBoost gives the whole bulk one gain. We use its recommended leading fraction of 1.5%. We also run B-Static at p=1.5%p=1.5\%, so the two methods can be compared with the same number of spike directions (Figure 1). The other two baselines are AdamW (Loshchilov & Hutter, 2017) and DeVA (Song et al., 2026), which separates variance adaptation from scale-invariant matrix updates. Models and datasets. We continue pretraining converged Pythia checkpoints (Biderman et al., 2023) at five scales, 14M, 31M, 70M, 160M, and 410M, on six corpora: math (FineMath-4, FineMath-3), Code (CodeParrot), and multilingual text (Wiki-de, Wiki-zh, Wiki-ar). See Appendix G.1. Token sampling and evaluation. Each corpus is tokenized once with Pythia’s GPT-NeoX tokenizer, so all scales and methods reuse the same tokenized data. We reserve the first 11M tokens for held-out evaluation and train on the remainder. Training samples uniformly random contiguous 10241024-token windows with a run-specific seed. A run takes 3,0513,051 steps of 3232 sequences (32,76832,768 tokens per step), or about 100100M tokens, with a cosine learning-rate schedule. Every 100100 steps, we evaluate on 128128 deterministic, evenly spaced held-out windows, shared across steps and methods. The main comparisons use three seeds. Pre-adaptation loss. For each model–corpus pair, let ℓ0 _0 and ℓfinal _final denote the held-out losses before and after continued pretraining, and define Δ=ℓfinal−ℓ0 = _final- _0. All methods start from the same pretrained checkpoint and are evaluated on the same held-out windows. We report loss differences between methods as a percentage of ℓ0 _0. Table 8 in Appendix G.1 reports ℓ0 _0 and Δ for Muon. Learning-rate calibration. We tune one learning rate per model scale on FineMath-4 and reuse it for every method and all six corpora. To choose it, we run a short learning-rate sweep over multiples of a reference rate, which is 2×10−32× 10^-3 for Pythia-14M, 31M, and 70M and 10−310^-3 for 160M and 410M. The multipliers are 0.01,0.05,0.1,0.25,0.5,1,2,4\0.01,0.05,0.1,0.25,0.5,1,2,4\. Each sweep run trains one method for 20M tokens with one seed, and we record its final validation loss. The sweep covers every method except AdamW 11 1 Because Pythia models are trained with AdamW and their learning rate decays to 0.1×0.1× the base rate at convergence, we omit AdamW from the sweep.. In it, B-Static uses p=20%p=20\%, and both BulkBoost variants use ρ=4ρ=4. At each scale, we then fix one shared multiplier: the one that is best for most swept methods. It is 0.1×0.1× for 14M, 31M, and 160M, 0.25×0.25× for 70M, and 0.05×0.05× for 410M. The resulting peak learning rates are 2×10−42× 10^-4 for 14M and 31M, 5×10−45× 10^-4 for 70M, 10−410^-4 for 160M, and 5×10−55× 10^-5 for 410M. For every swept method, the shared multiplier is also its own best, with two exceptions. At 70M, Muon does best at 0.1×0.1× (2×10−42× 10^-4), with a sweep loss of 2.7792.779 against 2.7832.783 at the shared 0.25×0.25×. Its loss then jumps to 2.9192.919 at the next multiplier, 0.5×0.5×, so the shared rate stays in Muon’s near-optimal range. At 160M, Spectra does best at 0.05×0.05× (5×10−55× 10^-5), with a sweep loss of 2.5522.552 against 2.5532.553 at the shared 0.1×0.1×. Figure 2 and Table 11 in Appendix G.2 report the full sweep. Shared rate in the 100M-token runs. Every 100M-token run uses the shared rate of its scale, with no further tuning. This includes AdamW, which the sweep does not cover. It also includes all five B-Static fractions in Table 9, so B-Static(5%) and B-Static† use it too. Tables 1 and 2 and Figure 1 report these runs. 5.1 Comparison of all optimization methods Table 1 reports final validation losses in all 30 model–corpus settings. Table 2 compares each method with Muon and also includes the shared configuration B-Static(5%). Final loss decreases with model size for every method and corpus. Pre-adaptation losses ℓ0 _0 follow the same trend (Table 8). Model Method FineMath-4 FineMath-3 CodeParrot Wiki-de Wiki-zh Wiki-ar 14M AdamW 3.102± 0.0033.102 \,±\,0.003 2.650± 0.0022.650 \,±\,0.002 1.929± 0.0031.929 \,±\,0.003 3.324± 0.0013.324 \,±\,0.001 2.801± 0.0012.801 \,±\,0.001 2.055± 0.0012.055 \,±\,0.001 Muon 3.086± 0.0023.086 \,±\,0.002 2.640± 0.0022.640 \,±\,0.002 1.902± 0.0041.902 \,±\,0.004 3.283± 0.0013.283 \,±\,0.001 2.745± 0.0012.745 \,±\,0.001 2.006± 0.0002.006 \,±\,0.000 Freon 3.083± 0.0023.083 \,±\,0.002 2.638± 0.0022.638 \,±\,0.002 1.899¯± 0.004 1.899 \,±\,0.004 3.279± 0.0013.279 \,±\,0.001 2.742± 0.0012.742 \,±\,0.001 2.008± 0.0012.008 \,±\,0.001 DeVA 3.110± 0.0043.110 \,±\,0.004 2.661± 0.0022.661 \,±\,0.002 1.917± 0.0051.917 \,±\,0.005 3.286± 0.0013.286 \,±\,0.001 2.753± 0.0012.753 \,±\,0.001 2.013± 0.0012.013 \,±\,0.001 Spectra 3.138± 0.0043.138 \,±\,0.004 2.682± 0.0022.682 \,±\,0.002 1.955± 0.0041.955 \,±\,0.004 3.328± 0.0033.328 \,±\,0.003 2.811± 0.0012.811 \,±\,0.001 2.058± 0.0022.058 \,±\,0.002 B-Static† 3.083± 0.0023.083 \,±\,0.002 2.638± 0.0022.638 \,±\,0.002 1.898± 0.0031.898 \,±\,0.003 3.278¯± 0.001 3.278 \,±\,0.001 2.738¯± 0.001 2.738 \,±\,0.001 1.998± 0.0011.998 \,±\,0.001 B-Adaptive 3.085¯± 0.002 3.085 \,±\,0.002 2.639¯± 0.001 2.639 \,±\,0.001 1.899¯± 0.003 1.899 \,±\,0.003 3.277± 0.0023.277 \,±\,0.002 2.737± 0.0012.737 \,±\,0.001 2.000¯± 0.001 2.000 \,±\,0.001 31M AdamW 2.794± 0.0012.794 \,±\,0.001 2.443± 0.0022.443 \,±\,0.002 1.674± 0.0021.674 \,±\,0.002 2.998± 0.0012.998 \,±\,0.001 2.523± 0.0022.523 \,±\,0.002 1.897± 0.0001.897 \,±\,0.000 Muon 2.776± 0.0012.776 \,±\,0.001 2.431± 0.0022.431 \,±\,0.002 1.649± 0.0011.649 \,±\,0.001 2.959± 0.0022.959 \,±\,0.002 2.471± 0.0012.471 \,±\,0.001 1.855± 0.0011.855 \,±\,0.001 Freon 2.771± 0.0012.771 \,±\,0.001 2.429¯± 0.001 2.429 \,±\,0.001 1.648¯± 0.002 1.648 \,±\,0.002 2.956¯± 0.001 2.956 \,±\,0.001 2.468± 0.0012.468 \,±\,0.001 1.861± 0.0011.861 \,±\,0.001 DeVA 2.800± 0.0012.800 \,±\,0.001 2.462± 0.0032.462 \,±\,0.003 1.668± 0.0031.668 \,±\,0.003 2.966± 0.0032.966 \,±\,0.003 2.480± 0.0012.480 \,±\,0.001 1.862± 0.0021.862 \,±\,0.002 Spectra 2.830± 0.0022.830 \,±\,0.002 2.482± 0.0042.482 \,±\,0.004 1.697± 0.0031.697 \,±\,0.003 3.009± 0.0053.009 \,±\,0.005 2.531± 0.0012.531 \,±\,0.001 1.899± 0.0021.899 \,±\,0.002 B-Static† 2.772¯± 0.001 2.772 \,±\,0.001 2.428± 0.0022.428 \,±\,0.002 1.648¯± 0.002 1.648 \,±\,0.002 2.953± 0.0012.953 \,±\,0.001 2.463± 0.0012.463 \,±\,0.001 1.848± 0.0011.848 \,±\,0.001 B-Adaptive 2.773± 0.0012.773 \,±\,0.001 2.430± 0.0032.430 \,±\,0.003 1.647± 0.0031.647 \,±\,0.003 2.953± 0.0012.953 \,±\,0.001 2.464¯± 0.001 2.464 \,±\,0.001 1.849¯± 0.000 1.849 \,±\,0.000 70M AdamW 2.617± 0.0022.617 \,±\,0.002 2.343± 0.0042.343 \,±\,0.004 1.547± 0.0051.547 \,±\,0.005 2.738± 0.0072.738 \,±\,0.007 2.289± 0.0112.289 \,±\,0.011 1.838± 0.0141.838 \,±\,0.014 Muon 2.551± 0.0022.551 \,±\,0.002 2.297± 0.0042.297 \,±\,0.004 1.484± 0.0031.484 \,±\,0.003 2.680± 0.0032.680 \,±\,0.003 2.211± 0.0052.211 \,±\,0.005 1.702± 0.0001.702 \,±\,0.000 Freon 2.536± 0.0012.536 \,±\,0.001 2.288¯± 0.006 2.288 \,±\,0.006 1.471¯± 0.003 1.471 \,±\,0.003 2.668± 0.0022.668 \,±\,0.002 2.200± 0.0002.200 \,±\,0.000 1.694¯± 0.001 1.694 \,±\,0.001 DeVA 2.565± 0.0012.565 \,±\,0.001 2.310± 0.0022.310 \,±\,0.002 1.493± 0.0041.493 \,±\,0.004 2.707± 0.0012.707 \,±\,0.001 2.231± 0.0012.231 \,±\,0.001 1.723± 0.0031.723 \,±\,0.003 Spectra 2.617± 0.0032.617 \,±\,0.003 2.339± 0.0012.339 \,±\,0.001 1.545± 0.0011.545 \,±\,0.001 2.768± 0.0122.768 \,±\,0.012 2.303± 0.0142.303 \,±\,0.014 1.813± 0.0211.813 \,±\,0.021 B-Static† 2.537¯± 0.004 2.537 \,±\,0.004 2.287± 0.0072.287 \,±\,0.007 1.469± 0.0041.469 \,±\,0.004 2.670¯± 0.002 2.670 \,±\,0.002 2.203¯± 0.003 2.203 \,±\,0.003 1.690± 0.0011.690 \,±\,0.001 B-Adaptive 2.545± 0.0012.545 \,±\,0.001 2.291± 0.0032.291 \,±\,0.003 1.473± 0.0031.473 \,±\,0.003 2.674± 0.0022.674 \,±\,0.002 2.205± 0.0042.205 \,±\,0.004 1.690± 0.0001.690 \,±\,0.000 160M AdamW 2.292± 0.0022.292 \,±\,0.002 2.072¯± 0.001 2.072 \,±\,0.001 1.285± 0.0031.285 \,±\,0.003 2.481± 0.0022.481 \,±\,0.002 2.114± 0.0022.114 \,±\,0.002 1.664± 0.0011.664 \,±\,0.001 Muon 2.272± 0.0032.272 \,±\,0.003 2.066± 0.0022.066 \,±\,0.002 1.270± 0.0031.270 \,±\,0.003 2.452± 0.0022.452 \,±\,0.002 2.055± 0.0012.055 \,±\,0.001 1.608± 0.0001.608 \,±\,0.000 Freon 2.282± 0.0012.282 \,±\,0.001 2.077± 0.0022.077 \,±\,0.002 1.282± 0.0031.282 \,±\,0.003 2.461± 0.0022.461 \,±\,0.002 2.061± 0.0012.061 \,±\,0.001 1.613± 0.0011.613 \,±\,0.001 DeVA 2.297± 0.0032.297 \,±\,0.003 2.092± 0.0022.092 \,±\,0.002 1.287± 0.0041.287 \,±\,0.004 2.472± 0.0022.472 \,±\,0.002 2.084± 0.0032.084 \,±\,0.003 1.639± 0.0011.639 \,±\,0.001 Spectra 2.353± 0.0032.353 \,±\,0.003 2.125± 0.0062.125 \,±\,0.006 1.325± 0.0041.325 \,±\,0.004 2.590± 0.0032.590 \,±\,0.003 2.188± 0.0022.188 \,±\,0.002 1.710± 0.0011.710 \,±\,0.001 B-Static† 2.281¯± 0.003 2.281 \,±\,0.003 2.077± 0.0032.077 \,±\,0.003 1.280¯± 0.004 1.280 \,±\,0.004 2.457¯± 0.003 2.457 \,±\,0.003 2.051± 0.0012.051 \,±\,0.001 1.597± 0.0021.597 \,±\,0.002 B-Adaptive 2.281¯± 0.004 2.281 \,±\,0.004 2.077± 0.0022.077 \,±\,0.002 1.281± 0.0041.281 \,±\,0.004 2.460± 0.0022.460 \,±\,0.002 2.053¯± 0.001 2.053 \,±\,0.001 1.604¯± 0.001 1.604 \,±\,0.001 410M AdamW 2.058± 0.0052.058 \,±\,0.005 1.895± 0.0021.895 \,±\,0.002 1.119¯± 0.001 1.119 \,±\,0.001 2.198± 0.0012.198 \,±\,0.001 1.901± 0.0021.901 \,±\,0.002 1.469± 0.0001.469 \,±\,0.000 Muon 2.046± 0.0052.046 \,±\,0.005 1.891± 0.0021.891 \,±\,0.002 1.111± 0.0021.111 \,±\,0.002 2.181± 0.0012.181 \,±\,0.001 1.879± 0.0021.879 \,±\,0.002 1.452¯± 0.001 1.452 \,±\,0.001 Freon 2.043¯± 0.005 2.043 \,±\,0.005 1.889± 0.0011.889 \,±\,0.001 1.111± 0.0021.111 \,±\,0.002 2.179± 0.0012.179 \,±\,0.001 1.883± 0.0011.883 \,±\,0.001 1.460± 0.0011.460 \,±\,0.001 DeVA 2.087± 0.0052.087 \,±\,0.005 1.929± 0.0041.929 \,±\,0.004 1.143± 0.0021.143 \,±\,0.002 2.209± 0.0032.209 \,±\,0.003 1.904± 0.0021.904 \,±\,0.002 1.467± 0.0011.467 \,±\,0.001 Spectra 2.099± 0.0042.099 \,±\,0.004 1.933± 0.0061.933 \,±\,0.006 1.151± 0.0011.151 \,±\,0.001 2.224± 0.0022.224 \,±\,0.002 1.917± 0.0021.917 \,±\,0.002 1.472± 0.0011.472 \,±\,0.001 B-Static† 2.043¯± 0.005 2.043 \,±\,0.005 1.890¯± 0.002 1.890 \,±\,0.002 1.111± 0.0021.111 \,±\,0.002 2.174± 0.0012.174 \,±\,0.001 1.870± 0.0011.870 \,±\,0.001 1.446± 0.0011.446 \,±\,0.001 B-Adaptive 2.042± 0.0052.042 \,±\,0.005 1.890¯± 0.002 1.890 \,±\,0.002 1.111± 0.0021.111 \,±\,0.002 2.176¯± 0.001 2.176 \,±\,0.001 1.875¯± 0.001 1.875 \,±\,0.001 1.453± 0.0021.453 \,±\,0.002 Table 1: Final validation loss after about 100M tokens, reported as mean ± standard deviation over three seeds. All runs use the shared learning rate of their scale. For each model–corpus setting, B-Static† reports the fraction with the lowest mean loss among the five in Table 9. Bold and Underline mark the lowest and second-lowest means, respectively in each setting, including ties. Appendix G.1 reports the pre-adaptation losses ℓ0 _0. vs. Muon best of all Δ vs. Muon tokens saved (%) Method wins tied wins tied (p of ℓ0 _0) mean median Muon (reference) – – 4 1 – – – AdamW 0 0 0 0 +1.299±0.010+1.299± 0.010 – – Freon 19 1 5¯ 5 3 −0.022±0.010-0.022± 0.010 7.787.78 7.717.71 DeVA 0 0 0 0 +0.811±0.015+0.811± 0.015 – – Spectra 0 0 0 0 +2.321±0.047+2.321± 0.047 – – B-Static(5%) 23 2 0 3 −0.073±0.010-0.073± 0.010 7.817.81 7.807.80 B-Adaptive 24¯ 24 1 4 3 −0.077¯±0.010 -0.077± 0.010 7.99¯ 7.99 8.01¯ 8.01 B-Static† 25 1 12 5 −0.147±0.021-0.147± 0.021 10.0010.00 10.2810.28 Table 2: Summary over the 30 model–corpus settings after about 100M tokens. B-Static† uses the fraction selection of Table 1. Counts compare three-seed mean final losses. vs. Muon: settings with lower (wins) or equal (tied) loss than Muon. Best of all: settings with the lowest loss of all rows, alone (wins) or shared (tied). B-Static(5%) can at best tie B-Static†, which includes its runs. Δ vs. Muon: the loss difference from Muon as a percentage of ℓ0 _0, averaged over settings within each seed. We report the mean ± std dev of the three seed averages. Negative values favor the method. Tokens saved: the mean and median share of the 100M-token budget saved in reaching Muon’s final loss, over the winning settings and the seeds that end below Muon (Table 10). Two bands match or beat fine-grained profiles. Measured against Muon’s flat profile, B-Static(5%), B-Adaptive, and B-Static† lower the final loss in 2323, 2424, and 2525 of the 3030 settings, by 0.073%0.073\%, 0.077%0.077\%, and 0.147%0.147\% of ℓ0 _0 on average (Table 2). The fine-grained Freon does so in 1919 settings, by 0.022%0.022\%. B-Static(5%) beats Freon in 1313 settings and loses in 1313, and B-Adaptive beats it in 1717 and loses in 1010. Spectra and DeVA trail Muon in all 3030 settings, and every BulkBoost configuration beats both. This includes B-Static at Spectra’s leading fraction of 1.5%1.5\% (Table 9). The original DeVA and Spectra studies pretrain from scratch (Song et al., 2026; Huang et al., 2026), which differs from our continued pretraining setup. These results suggest that a single bulk-to-spike gain captures at least as much benefit as the fine-grained spectral profiles. Static and adaptive splits. B-Static† beats B-Adaptive in 2020 settings and loses in 44, but it selects its fraction on the reported loss. Without any fraction search, B-Adaptive matches B-Static(5%), with 1010 wins, 66 losses, and 1414 ties. Token efficiency and runtime trade-offs. Among settings where it outperforms Muon, B-Adaptive reaches Muon’s final loss using 7.99%7.99\% fewer tokens on average, with a median saving of 8.01%8.01\% (Table 2). The extensively tuned B-Static† increases these savings to 10%10\% and 10.28%10.28\%, respectively. These token savings come with additional per-step computation. At 70M–410M, measured per-step runtimes are 1.361.36–1.50×1.50× Muon’s for B-Adaptive, compared with 1.191.19–1.27×1.27× for the measured static configurations (Appendix G). Adaptive selection avoids fraction search, while static selection offers cheaper execution once a suitable fraction is identified. Neither DeVA nor Spectra achieves lower mean final validation loss than Muon in any of the 3030 settings (Table 1); their token-saving averages over wins are therefore undefined (“–” in Table 2). DeVA outperforms AdamW in 1818 settings and trails it in 1212. Table 10 in Appendix G.2 identifies the winning model–corpus settings and reports their token savings and variation across seeds. Ablations. Appendix G.2 compares the static and adaptive splits, sweeps the gain ρ, and shows that a third band does not improve on two. In exact-SVD single-update tests on Pythia-70M, ρ=1.25ρ=1.25 and 1.51.5 lower the loss relative to the flat update (ρ=1ρ=1) in five of six settings (Appendix E.2). Conclusion. Our diagnostics reveal that Muon’s momentum has a broad spectral bulk whose individual directions are weak but collectively aligned with a reference gradient. BulkBoost exploits this structure with only two bands and a single shared bulk-to-spike gain. Our theory shows when this coarse reallocation is locally beneficial and what finer shaping can add through within-band variation. An exact polar-plus-low-rank identity enables an SVD-free realization. Across 3030 continued-pretraining settings, the two-band variants match or exceed the finer spectral baselines in aggregate. Single-update experiments show that bulk emphasis lowers the loss relative to the exact flat update in five of six states. Together, these results suggest that useful departures from Muon’s flat spectrum can be surprisingly low-dimensional: a single bulk-to-spike gain can capture at least as much benefit as the fine-grained spectral profiles. Limitations. The strongest B-Static results select the fraction separately for each setting. A shared fraction avoids this search, but the best fraction varies across settings (Table 9). B-Adaptive instead sets the split online from the noise edge. It needs no fraction search, but it costs more per step. The noise edge comes from a white Gaussian noise reference model, so it separates signal from noise only approximately. Our evaluation is restricted to Pythia continued pretraining under the reported tuning protocol. Extending the evaluation to other architectures and training regimes remains future work. AI Use Statement Generative AI tools were used during the development of this manuscript. Their use included drafting and debugging portions of experimental code, assisting with the interpretation and presentation of experimental results, and drafting and editing portions of the manuscript. All implementations, experimental protocols, and numerical results included in the final paper were reviewed and verified by the authors. Experiments reported in the paper were executed by the authors on the stated models and datasets. The authors take full responsibility for the correctness and originality of the paper and for parts of the code produced with the assistance of generative AI. References Ahn et al. (2025) Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford. Dion: Distributed orthonormalized updates, 2025. URL https://arxiv.org/abs/2504.05295. Allal et al. (2025) Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big – data-centric training of a small language model, 2025. URL https://arxiv.org/abs/2502.02737. Amsel et al. (2026a) Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The polar express: Optimal matrix sign methods and their application to the Muon algorithm. In International Conference on Learning Representations (ICLR), 2026a. Amsel et al. (2026b) Noah Amsel, Jack Zhang, Kwangjun Ahn, Ali Naeimi, Austin Feng, Berlin Chen, Tri Dao, and John Langford. Dion3: Full-stack orthogonal updates, 2026b. URL https://arxiv.org/abs/2608.11612. Baik et al. (2005) Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. The Annals of Probability, 33(5):1643–1697, 2005. Benaych-Georges & Nadakuditi (2012) Florent Benaych-Georges and Raj Rao Nadakuditi. The singular values and vectors of low rank perturbations of large rectangular random matrices. Journal of Multivariate Analysis, 111:120–135, 2012. Beneventano et al. (2026) Pierfrancesco Beneventano, Mahmoud Abdelmoneum, and Tomaso Poggio. The spectral dynamics and noise geometry of muon. arXiv preprint arXiv:2606.08388, 2026. Bernstein & Newhouse (2024) Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology, 2024. URL https://arxiv.org/abs/2409.20325. Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, p. 2397–2430. PMLR, 2023. URL https://proceedings.mlr.press/v202/biderman23a.html. Carlson et al. (2015) David Carlson, Volkan Cevher, and Lawrence Carin. Stochastic spectral descent for restricted boltzmann machines. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, p. 111–119. PMLR, 2015. URL https://proceedings.mlr.press/v38/carlson15.html. CodeParrot (2022) CodeParrot. CodeParrot-clean: Cleaned python code dataset, 2022. URL https://huggingface.co/datasets/codeparrot/codeparrot-clean. Dong & Sawin (2026) Yihe Dong and Will Sawin. Muonp: Muon with fractional spectral powers. arXiv preprint arXiv:2606.13867, 2026. Everett & Qiu (2026) Katie Everett and Shikai Qiu. Optimizer memory schedules for outscaling the overtraining axis, 2026. URL https://arxiv.org/abs/2609.04577. Feoktistov et al. (2026) Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov, Amir Zainullin, and Aleksandr Beznosikov. Softsign: Smooth sign in your optimizer for better parameter heterogeneity handling. arXiv preprint arXiv:2605.31371, 2026. Franceschi et al. (2018) Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, p. 1568–1577. PMLR, 2018. URL https://proceedings.mlr.press/v80/franceschi18a.html. Gao et al. (2021) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. URL https://arxiv.org/abs/2101.00027. Gavish & Donoho (2017) Matan Gavish and David L. Donoho. Optimal shrinkage of singular values. IEEE Transactions on Information Theory, 63(4):2137–2152, 2017. Geman (1980) Stuart Geman. A limit theorem for the norm of random matrices. The Annals of Probability, p. 252–261, 1980. Gupta et al. (2018) Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In Proceedings of the 35th International Conference on Machine Learning, p. 1842–1850, 2018. Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 8342–8360. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.740. URL https://aclanthology.org/2020.acl-main.740/. Halko et al. (2011) Nathan Halko, Per-Gunnar Martinsson, and Joel A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2):217–288, 2011. doi: 10.1137/090771806. URL https://arxiv.org/abs/0909.4061. Higham (1986) Nicholas J. Higham. Computing the polar decomposition—with applications. SIAM Journal on Scientific and Statistical Computing, 7(4):1160–1174, 1986. doi: 10.1137/0907079. URL https://doi.org/10.1137/0907079. Huang et al. (2026) Zhendong Huang, Hengjie Cao, Fang Dong, Ruijun Huang, Mengyi Chen, Yifeng Yang, Xin Zhang, Anrui Chen, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin Lv, Robert P. Dick, Yuan Cheng, Tun Lu, Fan Yang, and Li Shang. Spectra: Rethinking optimizers for LLMs under spectral anisotropy. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. Jiang et al. (2026) Xiaowen Jiang, Andrei Semenov, and Sebastian U. Stich. Enhancing LLM training via spectral clipping. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2603.14315. Jordan et al. (2024) Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. Online technical note, 2024. URL https://kellerjordan.github.io/posts/muon/. Lau et al. (2025) Tim Tsz-Kit Lau, Qi Long, and Weijie Su. Polargrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective. arXiv preprint arXiv:2505.21799, 2025. Liming Liu et al. (2026) Liming Liu, Zhenghao Xu, Zixuan Zhang, Hao Kang, Zichong Li, Chen Liang, Weizhu Chen, and Tuo Zhao. COSMOS: A Hybrid Adaptive Optimizer for Efficient Training of Large Language Models. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=j2QTOOtM8R. Liu et al. (2025) Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025. Liu et al. (2026) Zhuanghua Liu, Menglian Wang, and Luo Luo. Musec: MomentUm SpEctral Clipping for stable Muon-type training, 2026. URL https://arxiv.org/abs/2609.11655. Loshchilov & Hutter (2017) Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. Maclaurin et al. (2015) Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, p. 2113–2122. PMLR, 2015. URL https://proceedings.mlr.press/v37/maclaurin15.html. Magakyan et al. (2026) Gagik Magakyan, Pablo Parrilo, and Asuman Ozdaglar. Spectral scaling laws of Muon, 2026. URL https://arxiv.org/abs/2606.04058. Marchenko & Pastur (1967) V. A. Marchenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1(4):457–483, 1967. McCandlish et al. (2018) Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training, 2018. URL https://arxiv.org/abs/1812.06162. Pang et al. (2026) Tianyu Pang, Yujie Fang, Zihang Liu, Shenyang Deng, Lei Hsiung, Shuhua Yu, and Yaoqing Yang. HTMuon: Improving Muon via Heavy-Tailed Spectral Correction. In Findings of the Association for Computational Linguistics: ACL, 2026. Pethick et al. (2025) Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 49069–49104. PMLR, 2025. URL https://proceedings.mlr.press/v267/pethick25a.html. Shumaylov et al. (2026) Zakhar Shumaylov, Nathaël Da Costa, Peter Zaika, Bálint Mucsányi, Alex Massucco, Yoav Gelberg, Carola-Bibiane Schönlieb, Yarin Gal, and Philipp Hennig. Muon is not that special: Random or inverted spectra work just as well. arXiv preprint arXiv:2605.11181, 2026. Song et al. (2026) Zitao Song, Cedar Site Bai, Zhe Zhang, Brian Bullins, and David F. Gleich. Decoupling variance and scale-invariant updates in adaptive gradient descent for unified vector and matrix optimization. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. Vyas et al. (2024) Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321, 2024. Wang et al. (2026) Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Dirk Bergemann, and Zhuoran Yang. Why muon outperforms adam: A curvature perspective. arXiv preprint arXiv:2606.04662, 2026. Wu et al. (2026) Fangzhou Wu, Rikhav Shah, Sandeep Silwal, and Qiuyi Zhang. Dynmuon: A dynamic spectral shaping view of muon. arXiv preprint arXiv:2605.17109, 2026. Zhang et al. (2026) Qiaozhe Zhang, Jun Sun, and Yingzhuang Liu. Beyond the matrix sign: Quadratic spectral descent, 2026. URL https://arxiv.org/abs/2609.07597. Zhao et al. (2024) Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-efficient LLM training by gradient low-rank projection. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 61121–61143. PMLR, 2024. URL https://proceedings.mlr.press/v235/zhao24s.html. Appendix A Related Work We position BulkBoost within matrix-aware optimization, spectral reweighting, and subspace selection. Our focus is the allocation of a fixed per-matrix direction norm between two spectral bands, separating partition selection –by fixed rank or a noise-calibrated edge–from the relative gain assigned to the bulk. A.1 Matrix-Aware Optimization and Spectral Geometry Stochastic spectral descent exploits matrix geometry rather than treating a parameter matrix as an unstructured vector (Carlson et al., 2015). Bernstein & Newhouse (2024) organize several optimizer constructions through norm-constrained steepest descent. Muon applies an approximate polar map to matrix momentum (Jordan et al., 2024); its large-scale realization addresses update scaling, weight decay, and distributed execution for language-model training (Liu et al., 2025). Scion develops a broader family of norm-constrained linear minimization oracles (Pethick et al., 2025), while PolarGrad places polar-based updates within a matrix-aware preconditioning framework (Lau et al., 2025). These perspectives distinguish the geometry defining an update from its scalar normalization. Within this setting, BulkBoost’s exact target retains the singular frame of MtM_t and reallocates gains between two bands while preserving the Frobenius norm of its polar factor QtQ_t. A related line uses accumulated gradient statistics for matrix preconditioning. Shampoo constructs tensor-structured preconditioners (Gupta et al., 2018); SOAP performs Adam-style adaptation in Shampoo’s eigenbasis (Vyas et al., 2024). DeVA separates variance adaptation from scale-invariant vector and matrix update rules (Song et al., 2026). COSMOS combines SOAP-style adaptation in a leading eigensubspace with Muon on its complement (Liming Liu et al., 2026), illustrating the use of distinct optimization rules in different spectral subspaces. BulkBoost instead applies one norm-matched reweighting rule with a common gain within each band. Its noise-calibrated variant uses split-minibatch statistics to select band membership, not to assign direction-specific second-moment gains; its fixed-rank variant requires no noise estimate. A.2 Spectral Shaping Beyond Muon’s Flat Profile Several recent studies replace Muon’s equal-gain profile with a nonconstant spectral map. Shumaylov et al. (2026) show that random and inverted spectra can remain effective and introduce Freon through a family of Schatten-based update rules. Muonp uses fractional spectral powers (Dong & Sawin, 2026), while DynMuon varies the shaping profile over training (Wu et al., 2026). HTMuon uses heavy-tailed spectral correction (Pang et al., 2026), and Softsign introduces smooth alternatives to sign updates, including the matrix optimizer SoftMuon (Feoktistov et al., 2026). These studies show that exact spectral flattening is one effective allocation rather than a necessary choice. Spectral clipping provides another alternative. The Spectra framework of Jiang et al. (2026) studies clipping before and after preconditioning, with efficient soft spectral clipping via Newton–Schulz iterations. Musec replaces full flattening with momentum spectral clipping and implements smooth saturation using coupled iterations (Liu et al., 2026). These approaches retain or modify variation along the spectrum. BulkBoost instead asks whether useful reallocation can be captured by a norm-matched two-band profile, without fine-grained spectral shaping. Its exact target assigns equal gains within each band, with fixed-rank and noise-calibrated rules determining band membership separately from the relative bulk-to-spike gain. A.3 Head–Bulk Allocation and the Closest Related Methods The Spectra optimizer of Huang et al. (2026) is distinct from the similarly named Spectra framework of Jiang et al. (2026). It tracks a leading momentum subspace, replaces its singular values with a tail-scale estimate, and retains the residual tail before RMS-normalizing the update. Its leading-subspace intervention and warm-started iteration are closely related to our construction. The central difference is the reference spectrum: Spectra retains the raw tail’s unequal singular magnitudes, whereas BulkBoost’s exact target reweights the polar factor QtQ_t with equal gains within each band. Both fixed-rank and noise-calibrated BulkBoost use this two-level rule, differing in band selection. Under norm matching, relative head suppression and relative bulk emphasis are not opposite reallocations. The meaningful distinctions are the underlying spectral profile, partition rule, and normalization, rather than the terminology of suppression versus amplification. A.4 Random-Matrix Noise Edges and Spectral Estimation The Marchenko–Pastur law describes the limiting spectrum of appropriate random covariance matrices (Marchenko & Pastur, 1967). Spiked-model theory characterizes transitions in leading empirical eigenvalues (Baik et al., 2005), while rectangular perturbation theory analyzes singular values and singular-vector alignment (Benaych-Georges & Nadakuditi, 2012). Optimal singular-value shrinkage studies reconstruction of an underlying matrix under specified noise and loss models (Gavish & Donoho, 2017). These results provide statistical reference scales and estimation criteria, but do not determine which spectral allocation best reduces evaluation loss after an optimizer update. Gradient-noise measurements have also been used to characterize useful batch sizes and training efficiency (McCandlish et al., 2018). The noise-calibrated variant of BulkBoost uses split-minibatch differences to estimate the raw full-minibatch noise variance. It rescales this estimate by the squared global clipping factor before smoothing and uses the reference-model Nesterov correction to calibrate an edge for MtM_t (Section 3.1). The resulting white-noise reference selects band membership; the relative gain ρ remains a separate optimization choice. The edge is neither an exact signal–noise separator nor a prescription to discard below-edge modes. The fixed-rank variant selects its partition without estimating a noise edge. A.5 Polar Computation, Subspace Tracking, and Low-Rank Updates Iterative polar decomposition has a classical numerical foundation (Higham, 1986). Polar Express develops optimized matrix-sign iterations and their use in Muon (Amsel et al., 2026a). Randomized range finding and subspace iteration provide standard tools for estimating leading singular spaces without a full decomposition (Halko et al., 2011). Spectral scaling measurements further highlight the role of finite-iteration accuracy: momentum singular-value quantiles vary with layer and model scale, affecting which tail directions Newton–Schulz iterations resolve (Magakyan et al., 2026). Our realization combines Muon’s numerical polar approximation Q^t Q_t with warm-started block iteration to approximate the selected spike subspace. The noise-calibrated tracker augments the retained basis with fresh probes, whereas fixed-rank tracking uses a block of prescribed width. Neither construction explicitly computes individual bulk singular vectors. The shared bulk gain determines how these numerical ingredients combine into the prescribed reweighted target. Low-rank optimization methods address related computational objectives. GaLore projects gradients to reduce optimizer-state memory (Zhao et al., 2024), while Dion uses distributed, amortized low-rank orthonormalized updates with error feedback (Ahn et al., 2025). Dion3 combines algorithmic and systems techniques for efficient orthogonal updates (Amsel et al., 2026b). In BulkBoost, the low-rank object is instead a correction to a scaled polar update (Theorem 2). For full-rank MtM_t and ρ>0ρ>0, the exact two-level target remains full rank; only its correction is confined to the selected spike subspace. The numerical realization matches ‖Q^t‖F\| Q_t\|_F, rather than imposing the exact-polar norm d d. Its tracking cost depends on the trial-space width and iteration count as well as the retained rank, so avoiding an explicit SVD does not by itself establish a runtime advantage. A.6 Curvature, Noise Geometry, and Continuation Utility Pre-update gradient alignment captures only part of update quality. Wang et al. (2026) study Muon’s advantage through directional curvature, distinguishing first-order gain from the second-order penalty. Quadratic Spectral Descent incorporates a quadratic surrogate under a spectral-norm constraint (Zhang et al., 2026), while Beneventano et al. (2026) analyze spectral dynamics and noise geometry under Muon. These studies complement spectral-allocation comparisons by examining how curvature and transformed noise affect optimization, beyond singular magnitudes alone. Differentiating evaluation loss through training has a history in hyperparameter optimization and bilevel learning (Maclaurin et al., 2015; Franceschi et al., 2018). Our continuation utility applies this sensitivity viewpoint to a fixed-frame spectral intervention: it is the negative derivative of post-update or later evaluation loss with respect to a current applied singular-direction gain, evaluated at Muon’s equal-gain allocation. Subsequent training follows a fixed policy, while gradients and optimizer states can respond to the intervention. For either fixed-rank or noise-calibrated band selection, Theorem 1 characterizes local allocation preferences and the fraction of the maximal first-order improvement rate accessible to two bands under norm matching. This is an analysis of the exact spectral target, not an online hypergradient procedure for selecting ρ. Recent work on optimizer-memory schedules further emphasizes the roles of training horizon and memory timescale in optimizer comparisons (Everett & Qiu, 2026). Our pre-update alignment diagnostics, single-update loss measurements, and continued-pretraining experiments examine distinct aspects of spectral allocation: immediate alignment, the loss response to a current intervention, and performance over a training run. A.7 Controlled Continued-Pretraining Evaluation Pythia provides a public model suite designed for analysis across training and scale (Biderman et al., 2023). Continued pretraining on domains and tasks is an established adaptation setting (Gururangan et al., 2020). We use this setting to compare fixed-rank and noise-calibrated spectral reweighting from common pretrained checkpoints, under matched training-token budgets and a common validation protocol. This complements the from-scratch language-model training settings used by several related optimizers and evaluates both allocation and partition-selection rules during adaptation. Our comparisons distinguish validation loss and token efficiency from per-step computation and fraction-selection effort (Section 5). Appendix B Supplementary Setup and Normalization This appendix supplements Section 2 with normalization identities, gain diagnostics, and rank conventions, and derives the numerical norm matching used in Section 3. B.1 Extended Notation and Spectral Profiles Extended notation. For A∈ℝm×nA ^m× n, let d=minm,nd= \m,n\ and write A=UAΣAVA⊤A=U_A _AV_A for a thin SVD, where UA∈ℝm×dU_A ^m× d and VA∈ℝn×dV_A ^n× d have orthonormal columns, ΣA=diag(σA,1,…,σA,d) _A=diag( _A,1,…, _A,d), and σA,1≥⋯≥σA,d≥0 _A,1≥·s≥ _A,d≥ 0. For ℐ⊆[d]I [d], let UA,ℐU_A,I and VA,ℐV_A,I collect the selected singular vectors. Their left and right orthogonal projectors are PA,ℐ=UA,ℐUA,ℐ⊤,RA,ℐ=VA,ℐVA,ℐ⊤.P_A,I=U_A,IU_A,I , R_A,I=V_A,IV_A,I . Noise-scale convention. At fixed parameters, write the raw full-minibatch gradient as Graw=G¯+ZG^raw= G+Z, where G¯=[Graw] G=E[G^raw] and [Z]=0E[Z]=0. In the white-noise reference, s denotes the common entrywise standard deviation of Z. The gradient entering momentum is Gt=χtGtrawG_t= _tG_t^raw, where χt _t is the global clipping scalar. B-Adaptive estimates the raw variance from two minibatch halves and applies χt2 _t^2 before smoothing. Section 3.1 combines this running estimate with the reference-model Nesterov correction to calibrate the noise edge for MtM_t. Normalization identity. For the full-rank spectral input Mt=UtΣtVt⊤M_t=U_t _tV_t , the rank-one singular modes are Frobenius-orthonormal: ⟨ut,ivt,i⊤,ut,jvt,j⊤⟩F=(ut,i⊤ut,j)(vt,i⊤vt,j)=i=j. u_t,iv_t,i ,\,u_t,jv_t,j _F=(u_t,i u_t,j)(v_t,i v_t,j)=1\i=j\. Consequently, for a nonzero unnormalized profile wtw_t as in equation 3, ‖Utdiag(wt)Vt⊤‖F2=∑i=1dwt,i2. \|U_tdiag(w_t)V_t \|_F^2= _i=1^dw_t,i^2. The normalizer ct(wt)c_t(w_t) therefore gives ‖Dt(wt)‖F=d=‖Qt‖F\|D_t(w_t)\|_F= d=\|Q_t\|_F. The applied gain vector is ct(wt)wtc_t(w_t)w_t, denoted by w in Section 4.1. For either partition, the two-level profile satisfies ∑i=1d(wt,i(ρ))2=rt+ρ2(d−rt), _i=1^d (w_t,i^(ρ) )^2=r_t+ρ^2(d-r_t), yielding ct(wt(ρ))=cρ,rtc_t(w_t^(ρ))=c_ρ,r_t whenever the profile is nonzero. For the noise-calibrated partition, the notation wt(ρ,τt)w_t^(ρ, _t) makes the threshold dependence explicit. For a scalar-induced profile wt,i=g(σt,i)w_t,i=g( _t,i), the equivalent matrix-function notation is g(Σt)=diag(g(σt,1),…,g(σt,d)),Dt(g)=ct(g)Utg(Σt)Vt⊤,g( _t)=diag(g( _t,1),…,g( _t,d)), D_t(g)=c_t(g)U_tg( _t)V_t , where ct(g)=ct(wt)c_t(g)=c_t(w_t) and the induced profile is nonzero. B.2 Relative and Effective Gains Absolute gains under a relative reallocation. For a full-rank input with both bands nonempty, let qt=rt/d∈(0,1)q_t=r_t/d∈(0,1) denote the spike fraction. For B-Static, qt=⌈pd⌉/dq_t= pd /d, which can differ slightly from p because of rounding. The applied gains relative to the flat profile are gsp=cρ,rt=1qt+ρ2(1−qt),gbu=ρgsp.g_sp=c_ρ,r_t= 1 q_t+ρ^2(1-q_t), g_bu=ρ g_sp. (15) At fixed qtq_t, increasing ρ above 11 raises gbug_bu toward 1/1−qt1/ 1-q_t and lowers gspg_sp toward zero. For a small spike fraction, the bulk gain therefore stays close to one, while the spike gain can fall substantially. With qt=0.05q_t=0.05 and ρ=4ρ=4, gbu≈1.02g_bu≈ 1.02 and gsp≈0.26g_sp≈ 0.26. At the largest tested fraction, qt=0.2q_t=0.2, the same ρ gives gbu≈1.11g_bu≈ 1.11 and gsp≈0.28g_sp≈ 0.28. Thus ρ sets only the ratio of the two gains, and the absolute bulk gain changes much less than ρ. More generally, multiplying every unnormalized gain by the same positive scalar leaves the norm-matched direction unchanged. For ρ>0ρ>0, profiles with spike and bulk gains (1,ρ)(1,ρ) and (1/ρ,1)(1/ρ,1) therefore give the same update. Relative bulk emphasis and relative spike suppression describe the same norm-matched reallocation. Effective bulk-to-spike gain. For a general spectral profile, define the ratio of root-mean-square gains across the selected bands by ρeff(wt,ℐsp,t)=1d−rt∑i∈ℐbu,twt,i21rt∑i∈ℐsp,twt,i2. _eff(w_t;I_sp,t)= 1d-r_t _i _bu,tw_t,i^2 1r_t _i _sp,tw_t,i^2. (16) This statistic is defined when both bands are nonempty and the denominator is positive. The common normalizer cancels, so the statistic is the same for wtw_t and its applied gains ct(wt)wtc_t(w_t)w_t. In particular, the exact two-level target satisfies ρeff=ρ _eff=ρ. For B-Adaptive, we abbreviate it as ρeff(wt,τt) _eff(w_t; _t). The partition specifies band membership, whereas ρeff _eff summarizes the relative allocation; it does not describe gain variation within a band or measure subspace-recovery accuracy. B.3 Rank Conventions and Degenerate Partitions Empty bands. For a full-rank input, reallocation between bands requires 0<rt<d0<r_t<d. If rt=dr_t=d, all unnormalized gains equal one and the target is QtQ_t. If rt=0r_t=0 and ρ>0ρ>0, all unnormalized gains equal ρ, and normalization again gives QtQ_t. When rt=0r_t=0 and ρ=0ρ=0, the profile is zero and cannot be normalized to d d. This case is excluded by the nonzero-profile condition in Section 2. Rank-deficient inputs. For a rank-deficient spectral input, restrict the exact construction to the active indices ℐ+,t=i∈[d]:σt,i>0I_+,t=\i∈[d]: _t,i>0\, with active rank dt+=|ℐ+,t|d_t^+=|I_+,t|. The partial polar factor is Qt=∑i∈ℐ+,tut,ivt,i⊤,‖Qt‖F=dt+.Q_t= _i _+,tu_t,iv_t,i , \|Q_t\|_F= d_t^+. Intersect each partition from Section 2 with ℐ+,tI_+,t, and let rtr_t count the selected active spike modes. In particular, the fixed-rank rule gives rt=min⌈pd⌉,dt+r_t= \ pd ,d_t^+\: the prescribed fraction still refers to d=minm,nd= \m,n\, while zero singular modes receive no gains. For dt+>0d_t^+>0 and a nonzero active gain vector, the exact normalizers are ct(wt)=dt+∑i∈ℐ+,twt,i2,cρ,rt=dt+rt+ρ2(dt+−rt).c_t(w_t)= d_t^+ _i _+,tw_t,i^2, c_ρ,r_t= d_t^+r_t+ρ^2(d_t^+-r_t). The gain diagnostics in Appendix B.2 likewise use the active band sizes. The matrix dimensions remain m,nm,n; replacing d by dt+d_t^+ in these identities does not change the dimensions used to calibrate the noise edge. For a zero spectral input, the partial polar factor and exact update are zero, without evaluating normalization ratios. B.4 Derivation of Numerical Norm Matching The numerical update in equation 11 and equation 12 uses an approximate polar factor Q^t Q_t and a tracked basis Xt∈ℝd×r^tX_t ^d× r_t. We first derive the identities in exact arithmetic, assuming that XtX_t has orthonormal columns; neither an exact polar factor nor exact spike recovery is required for the projection calculation. Define Q^sp,t=Xt(Xt⊤Q^t),m≤n,(Q^tXt)Xt⊤,n<m,Q^bu,t=Q^t−Q^sp,t. Q_sp,t= casesX_t(X_t Q_t),&m≤ n,\\ ( Q_tX_t)X_t ,&n<m, cases Q_bu,t= Q_t- Q_sp,t. (17) These components use the tracked subspace and its orthogonal complement, which can differ from the exact spike and bulk bands. The raw direction is D~t=Q^sp,t+ρQ^bu,t=ρQ^t−(ρ−1)Q^sp,t. D_t= Q_sp,t+ρ Q_bu,t=ρ Q_t-(ρ-1) Q_sp,t. Since XtXt⊤X_tX_t is an orthogonal projector, ⟨Q^sp,t,Q^bu,t⟩F=0. Q_sp,t, Q_bu,t _F=0. Writing at=‖Q^t‖F2,bt=‖Q^sp,t‖F2a_t=\| Q_t\|_F^2, b_t=\| Q_sp,t\|_F^2 therefore gives ‖Q^bu,t‖F2=at−bt\| Q_bu,t\|_F^2=a_t-b_t. For nonzero Q^t Q_t and D~t D_t, ‖D~t‖F2=bt+ρ2(at−bt),c^ρ,t=atbt+ρ2(at−bt)=‖Q^t‖F‖D~t‖F.\| D_t\|_F^2=b_t+ρ^2(a_t-b_t), c_ρ,t= a_tb_t+ρ^2(a_t-b_t)= \| Q_t\|_F\| D_t\|_F. (18) Thus D^t=c^ρ,tD~t D_t= c_ρ,t D_t satisfies ‖D^t‖F=‖Q^t‖F\| D_t\|_F=\| Q_t\|_F. The projected norm can be evaluated on the smaller side: bt=‖Xt⊤Q^t‖F2,m≤n,‖Q^tXt‖F2,n<m.b_t= cases\|X_t Q_t\|_F^2,&m≤ n,\\ \| Q_tX_t\|_F^2,&n<m. cases For at>0a_t>0 and ρ>0ρ>0, the raw norm is positive because 0≤bt≤at0≤ b_t≤ a_t. At ρ=0ρ=0, a nonzero projected spike component is required. A zero Q^t Q_t gives a zero numerical update, without evaluating the norm ratio. At ρ=1ρ=1, D~t=Q^t D_t= Q_t, so the update reduces to the unreweighted numerical polar direction. Relation to the exact target. If Q^t=Qt Q_t=Q_t and MtM_t has full rank, then QtQt⊤=Imwhen m≤n,Qt⊤Qt=Inwhen n<m.Q_tQ_t =I_m m≤ n, Q_t Q_t=I_n n<m. Consequently, at=da_t=d and bt=r^tb_t= r_t for any orthonormal tracked basis, giving c^ρ,t=cρ,r^t c_ρ,t=c_ρ, r_t whenever the raw direction is nonzero. This equality of normalizers does not require an exact spike subspace. If XtX_t also spans the selected spike subspace, then r^t=rt r_t=r_t and the numerical construction recovers the exact target Dt(wt(ρ))D_t(w_t^(ρ)). For an approximate polar factor, ata_t need not equal d and btb_t need not equal r^t r_t, so the normalizers can differ. For rank-deficient inputs, the direct norm ratio remains applicable, but the full-rank simplification above does not apply in general. Finite-precision computation. The direct norm ratio in equation 12 requires only Frobenius norms and rescaling. Its norm-matching property does not rely on exact orthogonality of the computed components, unlike the expression involving at−bta_t-b_t. For a nonzero computed raw direction, exact evaluation of the ratio matches its norm to ‖Q^t‖F\| Q_t\|_F; finite-precision evaluation is subject to rounding error. Norm matching controls magnitude without correcting polar or subspace approximation errors. Appendix C Implementation Details and SVD-Free Algorithm This appendix gives the implementation details behind Section 3. It covers noise calibration, the exact-SVD target used in oracle comparisons, spike tracking for both variants, and the full B-Adaptive algorithm. B-Static and B-Adaptive share the reweighting and normalization of Section 2. They differ only in how they choose and track the spike band. Appendix B.4 derives numerical norm matching, and Proposition 4 derives the half-batch noise estimate and the Nesterov noise factor. C.1 Calibration and Estimator State Half-batch noise estimate. The two halves in equation 5 are sampled independently at the same parameters, so they have the same mean. This mean cancels in the half-batch difference ZtsplitZ_t^split. After vectorization, ZtsplitZ_t^split has the same covariance as the sampling noise in GtrawG_t^raw. Hence s^t 2=‖Ztsplit‖F2/(mn) s_t^\,2=\|Z_t^split\|_F^2/(mn) estimates the average entrywise noise variance of the full-minibatch gradient. For isotropic noise, this is the common entrywise variance. For anisotropic noise, one scalar cannot describe the full covariance. Clipping and variance smoothing. Let χt∈(0,1] _t∈(0,1] be the global clipping scalar. It is computed from the raw full-minibatch gradient of all model parameters, and all matrices share it. Each matrix uses Gt=χtGtrawG_t= _tG_t^raw in the momentum recursion. The halves are not clipped separately. Instead, each matrix’s raw variance estimate is scaled by χt2 _t^2 before smoothing: s¯t 2=χt2s^t 2,if the variance state is uninitialized,γs¯t−1 2+(1−γ)χt2s^t 2,otherwise. s_t^\,2= cases _t^2 s_t^\,2,&if the variance state is uninitialized,\\[2.0pt] γ s_t-1^\,2+(1-γ) _t^2 s_t^\,2,&otherwise. cases The EMA averages variances. The square root s¯t 2 s_t^\,2 in equation 8 is taken afterwards. For a matrix first observed at step 11, this gives s¯1 2=χ12s^1 2 s_1^\,2= _1^2 s_1^\,2. Without clipping, χt=1 _t=1. Nesterov factor. Proposition 4 derives the factor νt(β) _t(β) in equation 7 under a reference model: centered, isotropic noise that is independent over time and has constant variance, with B0=0B_0=0. Writing Mt=(1−β2)Gt+β2Bt−1M_t=(1-β^2)G_t+β^2B_t-1 gives an equivalent two-term form: νt2(β)=(1−β2)2+β41−β1+β(1−β2(t−1)). _t^2(β)=(1-β^2)^2+β^4 1-β1+β (1-β^2(t-1) ). The first term comes from the current gradient, which enters MtM_t both directly and through the updated buffer. The second term comes from the past gradients stored in Bt−1B_t-1. Approximations. The implementation replaces the reference variance with the clipping-scaled running estimate s¯t 2 s_t^\,2. This puts the estimate on the scale of the applied gradient. Because χt _t depends on the data, the scaled estimate is in general biased for the noise in the clipped gradient. The running estimate also assumes that the noise level changes slowly. Finally, the noise edge assumes Gaussian white noise (Section 2). The multiplier κedge _edge can shift the threshold, but a single scalar cannot correct for anisotropic or correlated noise in general. The experiments use γ=0.95γ=0.95 and κedge=1 _edge=1. C.2 Exact Target and Oracle Comparison For a full-rank input MtM_t and a nonzero two-level profile, the exact-SVD oracle evaluates, for either rule, Dtsvd=Dt(wt(ρ))=cρ,rtUtdiag(wt(ρ))Vt⊤.D_t^svd=D_t\! (w_t^(ρ) )=c_ρ,r_tU_tdiag\! (w_t^(ρ) )V_t . (19) For B-Adaptive, rtr_t counts all singular values above the threshold τt=τ^t _t= τ_t. For B-Static, rt=⌈pd⌉r_t= pd . In both cases, cρ,rtc_ρ,r_t is the normalizer in equation 4. Rank-deficient and zero inputs follow Appendix B.3. Here, oracle only means that the rule is evaluated exactly with an SVD. It does not choose the partition or the gain. An oracle comparison fixes MtM_t, the rule, and ρ. Both realizations therefore use the same threshold for B-Adaptive and the same rank for B-Static. The exact oracle matches ‖Qt‖F\|Q_t\|_F, and the numerical realization matches ‖Q^t‖F\| Q_t\|_F. If Q^t=Qt Q_t=Q_t and the orthonormal tracked basis spans the selected spike subspace, the numerical update equals the oracle (Appendix B.4). For B-Adaptive, the rank cap applies only to the numerical tracker. The exact target keeps every direction above the threshold. Both sides use the same spectral rule, so a comparison at a common input measures only numerical error: polar approximation, imperfect subspace recovery, and, for B-Adaptive, truncation by the cap. C.3 Warm-Started Block Iteration This subsection first gives the tracker for B-Adaptive and then the simpler tracker for B-Static. Both use the smaller-side Gram operator Kt∈ℝd×dK_t ^d× d from equation 10. Its action on a block Y is KtY=Mt(Mt⊤Y),m≤n,Mt⊤(MtY),n<m,K_tY= casesM_t(M_t Y),&m≤ n,\\ M_t (M_tY),&n<m, cases so the Gram matrix is never formed. As in Section 3.2, orthorth returns an orthonormal basis of the column space of its argument. An empty column space gives an empty basis. Initialization and trial-space construction. Let Xt−1X_t-1 be the spike basis that B-Adaptive retained at the previous step, with r^t−1 r_t-1 columns. Before the first update, X0X_0 is empty and r^0=0 r_0=0. For positive initial and refresh probe widths ninitn_init and nproben_probe, set nfresh,t=mind,ninit,r^t−1=0,mind,nprobe,r^t−1>0.n_fresh,t= cases \d,n_init\,& r_t-1=0,\\ \d,n_probe\,& r_t-1>0. cases The initial width is also used whenever the previous step selected no directions. Draw Ωt∈ℝd×nfresh,t _t ^d× n_fresh,t with independent standard Gaussian entries and set Y0=orth([Xt−1,Ωt]).Y_0=orth([X_t-1, _t]). The concatenated block may have more than d columns, but its orthonormal basis has at most d. The previous basis gives a warm start, and the fresh probes let new directions enter the trial space. At the first step, Y0Y_0 comes from the probes alone. The first retained basis X1X_1 is obtained only after iteration and threshold selection. Block iteration and Ritz extraction. Within training step t, perform q block-power iterations: Yj+1=orth(KtYj),j=0,…,q−1.Y_j+1=orth(K_tY_j), j=0,…,q-1. (20) Let ℓt _t be the number of columns of YqY_q. For a nonempty trial basis, form Tt=Yq⊤KtYq=OtΛtOt⊤,λt,1≥⋯≥λt,ℓt≥0,T_t=Y_q K_tY_q=O_t _tO_t , _t,1≥·s≥ _t, _t≥ 0, where Λt=diag(λt,1,…,λt,ℓt) _t=diag( _t,1,…, _t, _t). The Ritz directions are the columns of YqOtY_qO_t, with singular-value estimates λt,i _t,i. The eigenvalues are nonnegative in exact arithmetic because KtK_t is positive semidefinite. For a nonnegative integer rank cap rmaxr_ , let ncand,tn_cand,t count the Ritz values above the threshold: ncand,t=|i∈[ℓt]:λt,i>τ^t 2|,r^t=minncand,t,rmax,Xt=orth(YqOt,[r^t]).n_cand,t= |\i∈[ _t]: _t,i> τ_t^\,2\ |, r_t= \n_cand,t,r_ \, X_t=orth\! (Y_qO_t,[ r_t] ). (21) Here Ot,[r^t]O_t,[ r_t] holds the first r^t r_t eigenvectors in decreasing eigenvalue order. If the cap binds, the largest estimates above the threshold are kept. An empty selection gives an empty spike basis. If ℓt=0 _t=0, set r^t=0 r_t=0 and leave XtX_t empty; no eigendecomposition is needed. The experiments use this hard threshold without hysteresis. The selected Ritz block is reorthonormalized before it is stored. The projector identities in Appendix B.4 assume orthonormal columns, so in finite precision they hold only up to orthonormalization error. Experimental settings and random-number state. All B-Adaptive training runs use q=6,ninit=128,nprobe=128,rmax=256.q=6, n_init=128, n_probe=128, r_ =256. Probe widths are limited by d. The cap rmaxr_ is an absolute number of directions and does not scale with d. The retained rank satisfies r^t≤minℓt,rmax r_t≤ \ _t,r_ \, with ℓt≤d _t≤ d. The first retained basis has at most mind,128 \d,128\ columns, below the cap. The cap applies only to the retained basis. In these runs, the trial space has up to mind,r^t−1+128 \d, r_t-1+128\ columns, which can exceed the cap. Gaussian probes are generated in float32 with a dedicated pseudorandom-number generator for each parameter matrix, separate from the training random-number stream. Its state carries over from step to step and is never reseeded. Fixed-rank tracking (B-Static). B-Static tracks a block of fixed width r=⌈pd⌉r= pd . The fraction p is shared across matrices and fixed for the whole run. If a stored basis Xt−1∈ℝd×rX_t-1 ^d× r exists, it is the warm start, Y0=Xt−1Y_0=X_t-1. Otherwise, as at the first step, we draw a Gaussian block Ωt∈ℝd×r _t ^d× r and set Y0=orth(Ωt)Y_0=orth( _t). At each training step, we run Yj+1=orth(KtYj),j=0,…,q−1,Xt=Yq,Y_j+1=orth(K_tY_j), j=0,…,q-1, X_t=Y_q, with q=6q=6 in the experiments. The whole final basis enters equation 11 and is stored for the next step. Unlike the B-Adaptive tracker, this tracker adds no fresh probes, does no Ritz extraction, and applies no threshold or rank cap. The parameters ninitn_init, nproben_probe, and rmaxr_ therefore do not apply. For example, p=0.20p=0.20 and d=1024d=1024 give r=205r=205, so initialization and all six iterations use a 1024×2051024× 205 block. C.4 Full Algorithm Algorithm 1 gives one B-Adaptive step for one matrix. It combines noise calibration, spike tracking, and numerical norm matching. It returns the update direction before learning-rate scaling. The clipping scalar χt _t is computed once from the full-model gradient and passed in. For B-Static, the noise-estimate and threshold steps are skipped, and the fixed-rank tracker of Appendix C.3 replaces the adaptive one. Before the first update, B0=0B_0=0, X0X_0 is empty, and the variance state is uninitialized. The algorithm returns BtB_t, XtX_t, and s¯t 2 s_t^\,2 along with D^t D_t. The probe generator’s state is kept separately for each matrix. Since ρ>0ρ>0, a nonzero Q^t Q_t and an orthonormal XtX_t give a nonzero raw direction in exact arithmetic, so the norm ratio is defined. Appendix B.4 covers ρ=0ρ=0 and finite-precision effects. Algorithm 1 SVD-free B-Adaptive update for one matrix Inputs: Mean gradients Gt(1),Gt(2)∈ℝm×nG_t^(1),G_t^(2) ^m× n of the two minibatch halves, before clipping; global clipping scalar χt∈(0,1] _t∈(0,1]; previous states Bt−1,Xt−1B_t-1,X_t-1; previous variance state s¯t−1 2 s_t-1^\,2 (possibly uninitialized); the matrix’s probe generator; step t≥1t≥ 1. Write d=minm,nd= \m,n\. Parameters: Bulk-to-spike gain ρ>0ρ>0; momentum β∈[0,1)β∈[0,1); smoothing γ∈[0,1)γ∈[0,1); edge multiplier κedge>0 _edge>0; power iterations q; initial probe width ninitn_init; refresh probe width nproben_probe; absolute rank cap rmaxr_ . 1: Ztsplit←(Gt(1)−Gt(2))/2Z_t^split←(G_t^(1)-G_t^(2))/2 2: s^t 2←‖Ztsplit‖F2/(mn) s_t^\,2←\|Z_t^split\|_F^2/(mn) 3: Gtraw←(Gt(1)+Gt(2))/2G_t^raw←(G_t^(1)+G_t^(2))/2 4: Gt←χtGtrawG_t← _tG_t^raw 5: if the variance state is uninitialized then 6: s¯t 2←χt2s^t 2 s_t^\,2← _t^2 s_t^\,2 7: else 8: s¯t 2←γs¯t−1 2+(1−γ)χt2s^t 2 s_t^\,2←γ s_t-1^\,2+(1-γ) _t^2 s_t^\,2 9: end if 10: Bt←βBt−1+(1−β)GtB_t←β B_t-1+(1-β)G_t 11: Mt←(1−β)Gt+βBtM_t←(1-β)G_t+β B_t 12: Compute νt(β) _t(β) from equation 7 13: τ^t←κedgeνt(β)s¯t 2(m+n) τ_t← _edge _t(β) s_t^\,2( m+ n) 14: Q^t←PolarExpress(Mt) Q_t (M_t) 15: Define the Gram action using equation 10 16: if Xt−1X_t-1 is empty then 17: f←mind,ninitf← \d,n_init\ 18: else 19: f←mind,nprobef← \d,n_probe\ 20: end if 21: Draw Ωt∈ℝd×f _t ^d× f with independent (0,1)N(0,1) entries using this matrix’s probe generator 22: Y0←orth([Xt−1,Ωt])Y_0 ([X_t-1, _t]) 23: for j=0,…,q−1j=0,…,q-1 do 24: Yj+1←orth(KtYj)Y_j+1 (K_tY_j) 25: end for 26: ℓt← _t← number of columns of YqY_q 27: if ℓt=0 _t=0 then 28: r^t←0 r_t← 0; set XtX_t empty 29: else 30: Compute Yq⊤KtYq=OtΛtOt⊤Y_q K_tY_q=O_t _tO_t 31: Order eigenpairs by decreasing eigenvalue 32: ncand,t←|i∈[ℓt]:(Λt)ii>τ^t 2|n_cand,t←|\i∈[ _t]:( _t)_i> τ_t^\,2\| 33: r^t←minncand,t,rmax r_t← \n_cand,t,r_ \ 34: Xt←orth(YqOt,[r^t])X_t (Y_qO_t,[ r_t]) 35: end if 36: if ‖Q^t‖F=0\| Q_t\|_F=0 then 37: D^t←0 D_t← 0 38: else 39: Form D~t D_t using equation 11 40: D^t←‖Q^t‖F‖D~t‖FD~t D_t← \| Q_t\|_F\| D_t\|_F D_t 41: end if 42: return D^t,Bt,Xt,s¯t 2 D_t,B_t,X_t, s_t^\,2 Noise calibration. The half-batch difference estimates the average entrywise noise variance of the full-minibatch mean gradient. The clipping scalar rescales the gradient by χt _t and the variance estimate by χt2 _t^2. The EMA averages variances; the square root and the Nesterov factor are applied afterwards. Spike tracking. Fresh Gaussian probes are added at every step: ninitn_init when the previous basis is empty and nproben_probe otherwise. A nonempty previous basis also gives a warm start. The trial basis has at most d columns. Hard-threshold selection keeps at most rmaxr_ Ritz directions, which are then reorthonormalized. Persistent state. The momentum buffer, retained basis, and variance estimate carry over between steps. The probe generator advances when probes are drawn and is never reseeded. Norm matching. In exact arithmetic, ‖D^t‖F=‖Q^t‖F\| D_t\|_F=\| Q_t\|_F. At ρ=1ρ=1, the update is the unreweighted direction Q^t Q_t. Norm matching fixes the magnitude only; it does not correct polar or subspace approximation errors. Appendix D Band Content Along Muon Trajectories Does the bulk, whose singular values lie at or below the noise edge, contribute meaningfully to reference-gradient alignment under the flat profile? We examine this question along Muon trajectories of continued pretraining, measuring band sizes, per-direction alignment, and signed contributions across matrices. The measured bulk is weak per direction but substantial in aggregate: it contains 93.7–97.1% of pooled singular directions and has positive signed aggregate alignment in every analyzed matrix. These observations motivate a broad-bulk/small-spike partition and the fixed fractions tested in Section 5. Whether increasing the bulk’s relative gain improves an update is a separate question, which Appendix E examines. D.1 Protocol and Frame-Consistent Calibration Training trajectories and checkpoints. For each of FineMath-4, CodeParrot, and Wiki-ar, we analyze one Pythia-70M Muon trajectory of continued pretraining. Each run uses approximately 100M tokens, with batches of 32 sequences of length 1024, momentum β=0.95β=0.95, global gradient clipping at norm 1, and Newton–Schulz iterations for the polar map, as in the Muon baseline of Section 5. The matrix optimizer and the auxiliary AdamW optimizer both use peak learning rate 5×10−45× 10^-4, the shared Pythia-70M rate of Section 5. These diagnostics observe the Muon trajectories without changing their updates. We report four checkpoints as 0.0M, 25.0M, 50.0M, and 99.9M consumed training tokens. The initial measurement is a transient probe before continued pretraining; subsequent measurements precede updates after 24,969,216, 49,971,200, and 99,942,400 tokens. At each checkpoint, we analyze all 24 hidden weight matrices, excluding embeddings and the language-model head. For matrix ℓ with dimensions mℓ×nℓm_ × n_ , write dℓ=minmℓ,nℓd_ = \m_ ,n_ \. Here each matrix supplies 512 thin-SVD directions, giving Ndir=∑ℓdℓ=12,288N_dir= _ d_ =12,288 directions per corpus–checkpoint–frame combination. An independent reference gradient. At fixed model parameters, construct the large-batch reference Ht=116∑j=116Gtref,j,H_t= 116 _j=1^16G_t^ref,j, (22) where each Gtref,jG_t^ref,j is an unclipped mean gradient from an independently sampled full minibatch of the same corpus. The reference contains 512 sequences, or 524,288 input tokens; Ht,ℓH_t, denotes its block for matrix ℓ . It is neither clipped nor used for parameter updates. Reference sampling uses a separate data random-number stream, and the PyTorch random-number state is restored afterward. Full thin SVDs are computed only at diagnostic checkpoints. Raw-gradient and Nesterov frames. As in Section 3.1, distinguish the raw gradient Gt,ℓrawG_t, ^raw from the clipped gradient Gt,ℓ=χtGt,ℓrawG_t, = _tG_t, ^raw, where χt _t is the full-model clipping scalar. Using the actual previous momentum buffer, reconstruct Bt,ℓ B_t, =βBt−1,ℓ+(1−β)Gt,ℓ, =β B_t-1, +(1-β)G_t, , (23) Mt,ℓ M_t, =(1−β)Gt,ℓ+βBt,ℓ. =(1-β)G_t, +β B_t, . We analyze At,ℓG=Gt,ℓrawA^G_t, =G_t, ^raw in the raw-gradient frame and At,ℓM=Mt,ℓA^M_t, =M_t, in the Nesterov frame, against the same Ht,ℓH_t, . The first is the current minibatch gradient. The second is the input to Muon’s polar map, which combines the current gradient with the momentum buffer. A noise edge matched to each frame. For each training minibatch, compute the raw mean gradients of the two halves and form Gt,ℓraw=Gt,ℓ(1)+Gt,ℓ(2)2,Zt,ℓsplit=Gt,ℓ(1)−Gt,ℓ(2)2.G_t, ^raw= G_t, ^(1)+G_t, ^(2)2, Z_t, ^split= G_t, ^(1)-G_t, ^(2)2. (24) Let v^t,ℓ=‖Zt,ℓsplit‖F2/(mℓnℓ) v_t, =\|Z_t, ^split\|_F^2/(m_ n_ ) be the raw half-batch variance estimate, which corresponds to s^t 2 s_t^\,2 in Section 3.1. The diagnostics keep both a raw and a clipping-scaled variance EMA: vt,ℓraw v_t, ^raw =γvt−1,ℓraw+(1−γ)v^t,ℓ, =γ v_t-1, ^raw+(1-γ) v_t, , (25) vt,ℓclip v_t, ^clip =γvt−1,ℓclip+(1−γ)χt2v^t,ℓ. =γ v_t-1, ^clip+(1-γ) _t^2 v_t, . Initialize each EMA directly from its first observation, then apply these recursions every training step with γ=0.95γ=0.95. The clipping-scaled state is s¯t 2 s_t^\,2 of Section 3.1 for matrix ℓ ; the raw state serves the raw-gradient frame. The squared clipping factor is applied before smoothing. Using the Nesterov variance multiplier νt2(β)=(1−β2)2+β41−β1+β(1−β2(t−1)),t≥1, _t^2(β)=(1-β^2)^2+β^4 1-β1+β (1-β^2(t-1) ), t≥ 1, (26) from Proposition 4, set τt,ℓG _t, ^G =κedgevt,ℓraw(mℓ+nℓ), = _edge v_t, ^raw( m_ + n_ ), (27) τt,ℓM _t, ^M =κedgeνt(β)vt,ℓclip(mℓ+nℓ), = _edge _t(β) v_t, ^clip( m_ + n_ ), with κedge=1 _edge=1 and νt(β)≥0 _t(β)≥ 0. Thus the raw-gradient frame uses the raw noise scale, and the Nesterov frame uses the clipping-scaled noise scale adjusted for momentum. In each frame, singular values strictly above the threshold are spikes; the remaining directions form the bulk. Selection uses the full computed spectrum, without a tracker rank cap. As in Section 2, these thresholds are approximate references based on white noise. The initial Nesterov measurement uses a transient first-step probe with a zero buffer. Direct initialization of the variance states gives M1,ℓ=(1−β2)χ1G1,ℓraw,τ1,ℓM=(1−β2)χ1τ1,ℓG.M_1, =(1-β^2) _1G_1, ^raw, τ^M_1, =(1-β^2) _1τ^G_1, . The matched positive rescaling preserves the initial partition and polar direction, which explains the agreement between frames at 0.0M tokens. The probe leaves the training buffer and variance states unchanged. What per-direction alignment measures. For F∈G,MF∈\G,M\, write At,ℓF=Ut,ℓFΣt,ℓF(Vt,ℓF)⊤A^F_t, =U^F_t, ^F_t, (V^F_t, ) and define αt,ℓ,iF=(ut,ℓ,iF)⊤Ht,ℓvt,ℓ,iF.α^F_t, ,i=(u^F_t, ,i) H_t, v^F_t, ,i. (28) This is the signed projection of the reference gradient onto the unit-Frobenius matrix ut,ℓ,iF(vt,ℓ,iF)⊤u^F_t, ,i(v^F_t, ,i) , without normalization by ‖Ht,ℓ‖F\|H_t, \|_F, so it is not a cosine similarity. A positive value means that a step along −ut,ℓ,iF(vt,ℓ,iF)⊤-u^F_t, ,i(v^F_t, ,i) lowers the reference objective to first order. Under the flat profile, the scores sum to the polar alignment: ∑iαt,ℓ,iF=⟨Ht,ℓ,polar(At,ℓF)⟩F. _iα^F_t, ,i= H_t, ,polar(A^F_t, ) _F. (29) These identities concern the exact polar targets of Section 2, with the rank conventions of Appendix B.3. Alignment with the raw input matrix instead weights each αt,ℓ,iFα^F_t, ,i by its singular value. The raw-gradient-frame polar direction is only a diagnostic comparison; Muon uses the Nesterov frame. Signed and absolute contributions within a matrix. Suppress the frame, time, and matrix indices temporarily. For a nonempty band ℐI, its mean absolute alignment is a¯(ℐ)=|ℐ|−1∑i∈ℐ|αi| a(I)=|I|^-1 _i | _i|. Its signed and absolute shares are S(ℐ)=∑i∈ℐαi∑j=1dℓαj,Sabs(ℐ)=∑i∈ℐ|αi|∑j=1dℓ|αj|,S(I)= _i _i _j=1^d_ _j, S_abs(I)= _i | _i| _j=1^d_ | _j|, (30) when the respective denominators are nonzero. The signed share measures the net contribution after positive and negative projections cancel; the absolute share measures projection magnitude without cancellation. A signed share can be negative or exceed one, and it can be sensitive to a small total signed denominator. For applied gains w=(wi)w=(w_i), the contribution of direction i becomes wiαiw_i _i. A common positive normalizer cancels from these within-matrix shares. In general, it does not cancel from sums across matrices with different normalizers. How the tables pool matrices. The tables pool counts and alignment sums first and then form ratios; they do not average matrix-wise shares. Suppressing checkpoint and frame indices, define for b∈sp,bub∈\sp,bu\ Nb N_b =∑ℓ|ℐb,ℓ|, = _ |I_b, |, (31) Ab A_b =∑ℓ∑i∈ℐb,ℓαℓ,i,Bb=∑ℓ∑i∈ℐb,ℓ|αℓ,i|. = _ _i _b, _ ,i, B_b= _ _i _b, | _ ,i|. We report the bulk fraction Nbu/NdirN_bu/N_dir, the mean absolute per-direction alignments ab=Bb/Nba_b=B_b/N_b, and their spike-to-bulk ratio Γ=asp/abu =a_sp/a_bu. The pooled bulk shares are Sbu=AbuAsp+Abu,Sbuabs=BbuBsp+Bbu.S_bu= A_buA_sp+A_bu, S_bu^abs= B_buB_sp+B_bu. (32) To quantify cancellation, we also report C=|Aall|Ball,Aall=Asp+Abu,Ball=Bsp+Bbu.C= |A_all|B_all, A_all=A_sp+A_bu, B_all=B_sp+B_bu. (33) Small C means that opposing projections largely cancel. All pooled denominators in the reported measurements are positive. No layer-specific learning-rate or shape multipliers enter these sums, so they describe the alignment of polar directions only. Neither BulkBoost variant uses HtH_t online. D.2 Results: A Broad Bulk with Positive Aggregate Alignment Many weak directions contribute collectively. Tables 3 and 4 report the raw-gradient and Nesterov frames, respectively. Across the three corpora and four checkpoints, the bulk contains 93.7–97.1% of pooled directions. On average, spike directions have much larger absolute reference-gradient projections: Γ ranges from 33.2 to 92.3 in the raw-gradient frame and from 18.7 to 86.7 in the Nesterov frame. Despite these per-direction differences, the bulk contributes 14.6–26.0% of pooled signed polar alignment in the raw-gradient frame and 10.0–63.5% in the Nesterov frame. Thus, weak average per-direction alignment does not imply a negligible collective contribution. Corpus Tokens (M) Bulk (%) Γ (×) SbuS_bu (%) SbuabsS_bu^abs (%) FineMath-4 0.0 95.3 66.4 22.6 23.5 25.0 97.1 85.1 25.8 28.0 50.0 96.6 92.3 19.7 23.6 99.9 96.2 33.2 18.6 43.5 CodeParrot 0.0 95.5 74.8 22.6 22.3 25.0 96.5 72.0 26.0 27.8 50.0 96.6 64.9 25.6 30.2 99.9 96.6 58.8 16.6 32.9 Wiki-ar 0.0 93.7 86.7 14.6 14.6 25.0 94.8 64.8 21.4 22.0 50.0 94.5 47.0 25.5 26.9 99.9 94.6 40.7 18.1 30.3 Table 3: Raw-gradient-frame band content. Each row pools 24 matrices and 12,288 singular directions from a Pythia-70M Muon trajectory. Bulk (%) is the pooled fraction of directions whose singular values lie at or below the raw-gradient noise edge τt,ℓGτ^G_t, ; Γ is the ratio of mean absolute per-direction alignment in spikes to bulk. SbuS_bu and SbuabsS_bu^abs are the signed and absolute shares in equation 32, computed from pooled sums. Every analyzed matrix has positive signed bulk alignment. Token labels are rounded. Corpus Tokens (M) Bulk (%) Γ (×) SbuS_bu (%) SbuabsS_bu^abs (%) FineMath-4 0.0 95.3 66.4 22.6 23.5 25.0 97.1 56.8 24.7 36.8 50.0 96.8 78.1 10.0 28.0 99.9 96.6 18.9 63.5 59.8 CodeParrot 0.0 95.5 74.8 22.6 22.3 25.0 96.5 60.7 19.4 31.3 50.0 96.5 37.4 20.5 42.4 99.9 96.3 18.7 45.8 58.5 Wiki-ar 0.0 93.7 86.7 14.6 14.6 25.0 95.1 72.6 18.9 21.3 50.0 94.7 41.5 23.5 30.3 99.9 94.8 25.3 25.3 41.8 Table 4: Nesterov-frame band content. The trajectories, checkpoints, and reference gradients are paired with Table 3, but the singular frame and noise edge τt,ℓMτ^M_t, are those of Muon’s Nesterov input Mt,ℓM_t, . Each row pools 24 matrices and 12,288 directions, using the same statistics as Table 3. Matched scalar rescaling explains the agreement at 0.0M. Every analyzed matrix has positive signed bulk alignment. Positive alignment is present in every measured matrix. All 24 matrices have positive signed bulk sums at every checkpoint in both frames and all three corpora: 3×4×2×24=5763× 4× 2× 24=576 recorded combinations. Positive bulk alignment is therefore spread across all layers. The optimizer’s frame changes the relative contributions. At 99.9M tokens, the Nesterov bulk supplies 63.5% of pooled signed alignment on FineMath-4, 45.8% on CodeParrot, and 25.3% on Wiki-ar. The paired raw-gradient shares are 18.6%, 16.6%, and 18.1%. A larger share does not necessarily mean a larger absolute bulk contribution: in Table 5, the final Nesterov bulk sums are smaller than their raw-gradient counterparts, but the spike sums decrease by a greater proportion. The share is also not monotone over training: on FineMath-4, the Nesterov bulk share is 24.7% at 25M, 10.0% at 50M, and 63.5% at 99.9M. What holds in every case is positive net bulk alignment; the share itself varies across corpora and checkpoints. The contribution remains after signed cancellation. Table 5 reports raw signed and absolute sums at the final checkpoint. For example, the Nesterov bulk on CodeParrot contributes approximately 0.09570.0957 to total signed alignment 0.20880.2088, while total absolute alignment is 0.82460.8246, giving C≈0.253C≈ 0.253. The Nesterov cancellation ratios are 0.2380.238 on FineMath-4 and 0.5870.587 on Wiki-ar. The positive bulk sums therefore survive the cancellation between opposing directions. Corpus Frame AspA_sp AbuA_bu AallA_all BallB_all C FineMath-4 Gradient 0.5946 0.1363 0.7308 1.2160 0.601 Nesterov 0.0749 0.1300 0.2049 0.8594 0.238 CodeParrot Gradient 1.0155 0.2028 1.2183 1.6182 0.753 Nesterov 0.1132 0.0957 0.2088 0.8246 0.253 Wiki-ar Gradient 0.9131 0.2013 1.1144 1.3890 0.802 Nesterov 0.4201 0.1422 0.5623 0.9577 0.587 Table 5: Signed contributions and cancellation at 99.9M tokens. AspA_sp and AbuA_bu are pooled signed alignments; AallA_all is their sum and BallB_all sums absolute alignments. C=|Aall|/BallC=|A_all|/B_all measures the fraction remaining after signed cancellation. Every row pools 24 matrices, and every matrix has positive signed bulk alignment. Displayed sums are rounded independently. D.3 Implications for BulkBoost Motivation for the fixed-fraction sweep. The complementary spike band contains only 2.9–6.3% of the pooled singular directions in these measurements. This broad-bulk/small-spike structure motivates testing small fixed spike fractions, alongside larger ones, in Table 9 (Appendix G.2). The tested grid is p∈0.015,0.02,0.05,0.15,0.20p∈\0.015,0.02,0.05,0.15,0.20\: 5% lies within the observed pooled range and is the default for B-Static, 1.5% and 2% examine smaller fractions, and 15% and 20% examine larger spike bands. The diagnostics inform the grid; we did not convert the noise edge into five calibrated fractions. The loss comparisons in Table 9 decide which fractions perform best. The 2.9–6.3% range is computed after pooling matrices within each corpus–checkpoint–frame combination. Individual matrices can fall outside it, and it is not an estimate of the best fixed rank. B-Static uses rt,ℓ=⌈pdℓ⌉r_t, = pd_ with a fixed fraction for the whole run, whereas B-Adaptive selects the spike band separately for each matrix and step. The two rules give different partitions even when their pooled band sizes are similar. Link to the two-band design. Theorem 2 shows that a shared bulk gain needs only the polar factor plus a correction in the spike subspace. The band-content measurements show why the bulk matters in this allocation: it carries positive reference alignment in every matrix. Appendix E.2 evaluates the post-update effect of reweighting, and Section 5 evaluates training performance and computational cost. Appendix E Pre-Update Alignment and Post-Update Loss This appendix connects the band-content measurements in Appendix D to the effect of spectral reweighting on validation loss. We first compute the pre-update reference-alignment preference and then evaluate the model produced by a controlled update. The single-update measurements show that finite bulk-emphasizing reweighting can improve on the exact flat update QtQ_t, even when paired pre-update predictions favor less bulk weight. E.1 Pre-Update Reference Alignment A norm-matched alignment score. Use the same singular frames, reference gradients, and partitions as Appendix D, holding them fixed while varying ρ. Suppress the checkpoint and frame indices. For matrix ℓ , let rℓ=|ℐsp,ℓ|r_ =|I_sp, |, kℓ=dℓ−rℓk_ =d_ -r_ , and qℓ=rℓ/dℓq_ =r_ /d_ , and set Ab,ℓ=∑i∈ℐb,ℓαℓ,iA_b, = _i _b, _ ,i. The exact norm-matched two-level direction has alignment Jℓ(ρ)=cρ,rℓ(Asp,ℓ+ρAbu,ℓ),cρ,rℓ=[qℓ+ρ2(1−qℓ)]−1/2,J_ (ρ)=c_ρ,r_ (A_sp, +ρ A_bu, ), c_ρ,r_ = [q_ +ρ^2(1-q_ ) ]^-1/2, (34) whenever the profile is nonzero. A larger score predicts a larger first-order decrease in the reference objective under a common step size. This gives the pre-update comparison for the loss measurements below. For nonempty bands, differentiating at ρ=1ρ=1 gives dJℓ(ρ)dρ|ρ=1=rℓkℓdℓ(α¯bu,ℓ−α¯sp,ℓ), . dJ_ (ρ)dρ |_ρ=1= r_ k_ d_ ( α_bu, - α_sp, ), (35) where α¯bu,ℓ=Abu,ℓ/kℓ α_bu, =A_bu, /k_ and α¯sp,ℓ=Asp,ℓ/rℓ α_sp, =A_sp, /r_ . This uses the same norm-matched differentiation as Theorem 1, with pre-update reference projections in place of continuation utilities. The local preference therefore depends on the difference in mean per-direction alignment between bands. Appendix D instead measures their aggregate contributions under the flat profile. Normalize each matrix separately before pooling: J(ρ)=∑ℓJℓ(ρ),Rmat(ρ)=J(ρ)J(1).J(ρ)= _ J_ (ρ), R_mat(ρ)= J(ρ)J(1). (36) As in Appendix D, no learning-rate or shape multipliers are included. The reference ρ=1ρ=1 is the flat profile in the same frame; in the Nesterov frame, it gives Muon’s idealized update QtQ_t. The measured J(1)J(1) is positive in every pooled comparison. The pre-update score favors less bulk weight. The aggregate derivative J′(1)J (1) is negative in all 24 corpus–checkpoint–frame combinations. At the matrix level, the derivative is negative in 564 of 576 recorded combinations and positive in the remaining 12. The pooled signed per-direction ratio (Abu/Nbu)/(Asp/Nsp)(A_bu/N_bu)/(A_sp/N_sp) ranges from approximately 0.00370.0037 to 0.06170.0617. Unlike Γ in Appendix D, this ratio keeps the signs of the projections. The derivatives and reweighted scores are computed matrix by matrix; the pooled ratio only summarizes them. Across the recorded comparisons, Rmat(2)R_mat(2) ranges from 0.556 to 0.828, and Rmat(4)R_mat(4) from 0.330 to 0.738. Table 6 reports the final checkpoint. Corpus Frame α¯bu/α¯sp α_bu/ α_sp Rmat(2)R_mat(2) Rmat(4)R_mat(4) FineMath-4 Gradient 0.009 0.601 0.397 Nesterov 0.062 0.828 0.738 CodeParrot Gradient 0.007 0.590 0.381 Nesterov 0.032 0.739 0.604 Wiki-ar Gradient 0.012 0.602 0.395 Nesterov 0.019 0.638 0.450 Table 6: Pre-update reference-alignment trade-off at 99.9M tokens. The per-direction ratio uses pooled signed means α¯b=Ab/Nb α_b=A_b/N_b; the ratio Γ in Appendix D uses absolute alignments. Rmat(ρ)R_mat(ρ) applies each matrix’s own norm-matching factor before summing, as in equation 36. Values below one indicate lower alignment than the flat profile in the same frame. Each row uses the same 24 matrices and reference gradient as the preceding diagnostics, without learning-rate or shape multipliers. Interpreting the norm-matched reweighting. Increasing ρ moves the fixed direction norm from the spikes toward the bulk. For a matrix with 95% bulk directions, the applied bulk gains at ρ=2ρ=2 and 44 are approximately 1.0191.019 and 1.0241.024, whereas the spike gains are 0.5100.510 and 0.2560.256. In the pre-update score, the loss of spike alignment outweighs the additional bulk alignment, which explains the preference for smaller ρ. Relative bulk emphasis and relative spike suppression therefore describe the same norm-matched intervention. The next subsection evaluates its effect directly on post-update validation loss. E.2 Single-Update Loss Measurements Question and controlled comparison. Can a finite increase in the bulk-to-spike gain lower validation loss after one update, even when a pre-update linearized prediction favors less bulk weight? We use the Pythia-70M Muon trajectories of Appendix D on FineMath-4, CodeParrot, and Wiki-ar, immediately before updates 763 and 1526, after 24,969,216 and 49,971,200 training tokens, respectively. The protocol uses batches of 32 sequences of length 1024, β=0.95β=0.95, γ=0.95γ=0.95, κedge=1 _edge=1, global clipping at norm 1, and peak learning rate 5×10−45× 10^-4 for both the matrix and auxiliary optimizers. Validation uses 128 fixed held-out windows. At each checkpoint, the branches share the current Nesterov singular frames and the noise-edge partitions of Appendix D. We vary one common gain across the 24 hidden weight matrices, using the exact-SVD target and separate Frobenius norm matching for each matrix. The tested gains are ρ∈0.5,0.75,1,1.25,1.5ρ∈\0.5,0.75,1,1.25,1.5\, with two duplicate ρ=1ρ=1 branches. The comparator is the exact flat update QtQ_t at the same input, computed from the same SVD. All branches compute their current update from the same exact SVD, so they differ only in ρ. We report K=0K=0: validation immediately after the current update, without additional training steps. The duplicate ρ=1ρ=1 branches have identical recorded losses. Direct improvement and allocation preference. Let Ft,K(ρ)F_t,K(ρ) denote the expected evaluation loss, given the current state, under a current common gain ρ and K additional steps following a fixed continuation policy. Write F^t,0(ρ) F_t,0(ρ) for the measured post-update validation loss on the fixed windows. For ϵ∈0.25,0.5ε∈\0.25,0.5\, define B^ϵ=F^t,0(1)−F^t,0(1+ϵ),S^ϵ=F^t,0(1−ϵ)−F^t,0(1+ϵ)2ϵ. B_ε= F_t,0(1)- F_t,0(1+ε), S_ε= F_t,0(1-ε)- F_t,0(1+ε)2ε. (37) The loss reduction B^ϵ B_ε directly measures improvement over the flat update ρ=1ρ=1: positive values mean that the bulk-emphasizing update ρ=1+ϵρ=1+ε gives lower validation loss. The raw central finite difference S^ϵ S_ε instead compares the bulk-emphasizing and bulk-suppressing branches, ρ=1+ϵρ=1+ε and ρ=1−ϵρ=1-ε. A positive value indicates lower loss for the bulk-emphasizing branch. Together, these statistics give the benefit over the flat update and the preference between opposing reweightings. Table 7 reports both quantities for all six corpus–checkpoint settings. Loss reduction B^ϵ B_ε Slope S^ϵ S_ε (10−410^-4) (10−310^-3) Corpus Tokens (M) ϵ=0.25ε=0.25 ϵ=0.5ε=0.5 ϵ=0.25ε=0.25 ϵ=0.5ε=0.5 FineMath-4 25.0 +2.524 +3.725 +1.709 +2.774 CodeParrot 25.0 +1.842 +2.574 +1.350 +2.320 Wiki-ar 25.0 +6.593 +10.226 +4.082 +5.944 FineMath-4 50.0 -0.604 -1.191 -0.177 +0.032 CodeParrot 50.0 +0.844 +1.262 +0.554 +0.866 Wiki-ar 50.0 +1.530 +2.318 +0.978 +1.471 Table 7: Single-update validation-loss improvement (K=0K=0). Both ρ=1.25ρ=1.25 and ρ=1.5ρ=1.5 lower post-update validation loss relative to the exact flat update ρ=1ρ=1 in five of six tested states. Each row is a Pythia-70M corpus–checkpoint state with a common gain across 24 hidden matrices and separate per-matrix Frobenius norm matching. B^ϵ B_ε measures the loss reduction relative to ρ=1ρ=1, whereas S^ϵ S_ε compares the bulk-emphasizing and bulk-suppressing branches through equation 37. Loss reductions and raw central finite-difference slopes are reported in units of 10−410^-4 and 10−310^-3, respectively. All six states are shown; token labels are rounded, and the duplicate ρ=1ρ=1 branches have identical recorded losses. Bulk emphasis improves the post-update model. Both ρ=1.25ρ=1.25 and ρ=1.5ρ=1.5 lower validation loss relative to ρ=1ρ=1 in five of six corpus–checkpoint settings: all three corpora at 25M tokens, and CodeParrot and Wiki-ar at 50M tokens. The positive loss reductions B^ϵ B_ε measure these improvements directly. FineMath-4 at 50M is the exception, where the flat update has lower loss than both bulk-emphasizing updates. The improvements occur despite paired pre-update linearized predictions favoring less bulk weight at both perturbation widths in every tested state. These predictions are recorded for the intervention branches and differ from the pooled score J(ρ)J(ρ) in Appendix E.1. Thus, finite, norm-matched spectral reweighting can improve the post-update model even when the paired pre-update approximation favors the opposite allocation. The benefit is already visible at K=0K=0, without additional continuation steps. Connection to Theorem 1. The theorem varies the gains of one matrix, whereas this experiment changes a common ρ across matrices. For matrix ℓ , let rt,ℓr_t, be its spike rank, kt,ℓ=dℓ−rt,ℓk_t, =d_ -r_t, , and Δℓ,t,K _ ,t,K its bulk-minus-spike mean continuation utility when every matrix uses the flat profile. Each matrix’s utilities are partial derivatives with the other matrices’ current gains held at one. Define ωt,ℓ=rt,ℓkt,ℓ/dℓ _t, =r_t, k_t, /d_ , the matrix-specific version of the prefactor rk/drk/d in the theorem. Under the corresponding joint differentiability assumption, the chain rule gives St,K:=−Ft,K′(1)=∑ℓ: 0<rt,ℓ<dℓωt,ℓΔℓ,t,K.S_t,K:=-F_t,K (1)= _ :\,0<r_t, <d_ _t, _ ,t,K. (38) Indeed, the derivative along the common-gain curve is the sum of its matrix-wise partial derivatives, and Theorem 1 gives ωt,ℓΔℓ,t,K _t, _ ,t,K for each nonempty partition. Empty-band matrices have constant normalized targets and contribute zero. The identity applies to the joint loss, including interactions across matrices. When ∑ℓωt,ℓ>0 _ _t, >0, the exact ratio St,K/∑ℓωt,ℓS_t,K/ _ _t, is a weighted mean of the matrix-wise continuation-utility gaps. Its finite-difference counterpart is S^ϵ/∑ℓωt,ℓ S_ε/ _ _t, . Table 7 reports the raw central finite differences S^ϵ S_ε in units of 10−310^-3. The theorem characterizes the derivative at ρ=1ρ=1; S^ϵ S_ε is its finite-difference counterpart, and B^ϵ B_ε directly tests improvement at the chosen gains. These diagnostics measure the joint response to a shared gain. They do not measure individual utilities or RbtR_bt. Continuation utility evaluates the updated model. Even at K=0K=0, the evaluation point has changed. For an additive intervention with fixed effective step multiplier ηt,ℓ _t, , let θ1+ _1^+ be the complete parameter state after the flat update, and let ℰE be the evaluation objective defining the loss. Differentiation with respect to an applied singular-direction gain gives uℓ,i,t,0=ηt,ℓ⟨∇Wℓℰ(θ1+),ut,ℓ,ivt,ℓ,i⊤⟩F.u_ ,i,t,0= _t, _W_ E( _1^+),u_t, ,iv_t, ,i _F. (39) The singular vectors remain those of the fixed pre-intervention Nesterov input, but the evaluation gradient is taken after the flat update. The chain rule uses ∂Wℓ+∂wℓ,i=−ηt,ℓut,ℓ,ivt,ℓ,i⊤, ∂ W_ ^+∂ w_ ,i=- _t, u_t, ,iv_t, ,i , and continuation utility is the negative loss derivative. The multiplier ηt,ℓ _t, includes the learning rate and fixed shape scaling. Thus, “first order” describes changes in the applied gains around the flat update, with the actual parameter-update step size held fixed. This identifies the evaluation point relevant to local allocation preferences and explains the difference from pre-update reference alignment. What the experiment establishes. The experiment provides direct evidence that finite, norm-matched two-band reweighting can improve the model produced by a single update, already at K=0K=0. The measurements cover ρ=1.25ρ=1.25 and 1.51.5 at checkpoints near 25M and 50M tokens; Appendix D also measures band content at initialization and 99.9M. Section 5 separately evaluates the full training rule with the shared default ρ=4ρ=4. Because norm matching couples relative bulk emphasis with spike suppression, the measured benefit concerns their combined reweighting. Appendix F Proofs for Sections 3 and 4 This appendix proves Theorems 1 and 2, and states and proves Corollary 3. It also proves the half-batch and Nesterov noise identities used in Section 3.1 (Proposition 4), and treats the Gaussian model behind the noise edge separately. Appendix B.4 derives numerical norm matching. F.1 Proof of Theorem 1 Theorem 1 (restated). Fix the current training state, singular frame, nonempty two-band partition, and continuation policy as in Section 4.1. The intervention varies the applied gains of one matrix’s current update; the current updates of all other parameters and the rest of the current step stay fixed. Assume ℒt,KπL^π_t,K is differentiable at d1_d. Along the two-level family, −dρℒt,Kπ(w(ρ))|ρ=1=rkdΔ.- . ddρL^π_t,K(w(ρ)) |_ρ=1= rkd . (40) Increasing ρ from 11 gives a strictly positive first-order improvement if and only if Δ>0 >0. If Δ=0 =0, the first-order test is inconclusive. Small gain changes at the flat profile lie in the tangent space =δ∈ℝd:d⊤δ=0T=\δ ^d:1_d δ=0\. If every gain may change separately, the best first-order improvement rate is maxδ∈‖δ‖2=1u⊤δ=‖u⟂‖2. _ subarraycδ \\ \|δ\|_2=1 subarrayu δ=\|u_ \|_2. (41) Thus the flat profile is first-order stationary in this fixed-frame, norm-matched class if and only if u1=⋯=udu_1=·s=u_d. If the change must be constant within each band, the best rate is rk/d|Δ| rk/d\,| |. For u⟂≠0u_ ≠ 0, two bands therefore reach the fraction Rbt=rk/d|Δ|‖u⟂‖2∈[0,1]R_bt= rk/d\,| |\|u_ \|_2∈[0,1] (42) of the best rate. Both rates are per unit Euclidean change in w, and RbtR_bt allows ρ to move in either direction. Between-band and within-band components. Use the fixed state, frame, partition, and utilities from Section 4.1. Define (ubt)i (u_bt)_i =u¯sp−u¯,i∈ℐsp,u¯bu−u¯,i∈ℐbu, = cases u_sp- u,&i _sp,\\ u_bu- u,&i _bu, cases (43) (uwi)i (u_wi)_i =ui−u¯sp,i∈ℐsp,ui−u¯bu,i∈ℐbu. = casesu_i- u_sp,&i _sp,\\ u_i- u_bu,&i _bu. cases The decomposition u⟂=ubt+uwiu_ =u_bt+u_wi is orthogonal and satisfies ‖u⟂‖22 \|u_ \|_2^2 =rkdΔ2+∑i∈ℐsp(ui−u¯sp)2+∑i∈ℐbu(ui−u¯bu)2. = rkd ^2+ _i _sp(u_i- u_sp)^2+ _i _bu(u_i- u_bu)^2. (44) In particular, ‖ubt‖2=rk/d|Δ|\|u_bt\|_2= rk/d\,| |, so for u⟂≠0u_ ≠ 0 the capture ratio can also be written as Rbt=‖ubt‖2/‖u⟂‖2R_bt=\|u_bt\|_2/\|u_ \|_2. Its square is the share of squared tangent utility that lies between the bands; the rest lies within the bands. This share says nothing directly about the size of a finite loss reduction. The proof below establishes these identities together with the theorem. Proof. Write ℒ=ℒt,KπL=L^π_t,K for brevity. As specified in Section 4.1, ℒL is defined on an open neighborhood of d1_d in the positive orthant and is differentiable at d1_d, with u=−∇ℒ(d)u=- (1_d). All gain comparisons below lie on the norm-matched sphere. The singular frame, current training state, partition, and continuation policy are fixed throughout the argument. The policy is fixed as a rule: subsequent gradients and optimizer states may respond to the current intervention. We proceed in four steps. Step 1: first-order rates on the norm-matched sphere. The feasible set is d=w∈[0,∞)d:∥w∥22=d.S_d=\w∈[0,∞)^d:\|w\|_2^2=d\. Differentiating the norm constraint along a differentiable feasible curve through d1_d shows that its velocity δ satisfies d⊤δ=01_d δ=0. Conversely, for any unit vector δ with d⊤δ=01_d δ=0, define wδ(ε)=dd+εδ‖d+εδ‖2.w_δ( )= d\, 1_d+ δ\|1_d+ δ\|_2. Since ‖d+εδ‖22=d+ε2,\|1_d+ δ\|_2^2=d+ ^2, this curve lies in dS_d for sufficiently small |ε|| |, has positive coordinates there, and satisfies wδ(0)=d,wδ′(0)=δ.w_δ(0)=1_d, w_δ (0)=δ. Thus the tangent space is =δ:d⊤δ=0T=\δ:1_d δ=0\, and every unit tangent direction is realized by such a curve. Differentiability gives ℒ(wδ(ε))=ℒ(d)−εu⊤δ+o(|ε|).L(w_δ( ))=L(1_d)- u δ+o(| |). Hence u⊤δu δ is the first-order improvement rate per unit gain-vector displacement. For tangent δ, u⊤δ=(u−u¯d)⊤δ=u⟂⊤δ.u δ=(u- u1_d) δ=u_ δ. Step 2: orthogonal between-band and within-band decomposition. For i∈ℐspi _sp, (ubt)i+(uwi)i=(u¯sp−u¯)+(ui−u¯sp)=ui−u¯.(u_bt)_i+(u_wi)_i=( u_sp- u)+(u_i- u_sp)=u_i- u. The same identity holds on ℐbuI_bu, giving u⟂=ubt+uwiu_ =u_bt+u_wi. Moreover, ∑i∈ℐsp(ui−u¯sp)=0,∑i∈ℐbu(ui−u¯bu)=0. _i _sp(u_i- u_sp)=0, _i _bu(u_i- u_bu)=0. Therefore uwiu_wi is orthogonal to both band indicator vectors. Since ubtu_bt is in their span, ubt⟂uwi,‖u⟂‖22=‖ubt‖22+‖uwi‖22.u_bt u_wi, \|u_ \|_2^2=\|u_bt\|_2^2+\|u_wi\|_2^2. By definition, ‖uwi‖22=∑i∈ℐsp(ui−u¯sp)2+∑i∈ℐbu(ui−u¯bu)2.\|u_wi\|_2^2= _i _sp(u_i- u_sp)^2+ _i _bu(u_i- u_bu)^2. Let Δ=u¯bu−u¯sp = u_bu- u_sp. Using d=r+kd=r+k and u¯=(ru¯sp+ku¯bu)/d u=(r u_sp+k u_bu)/d, we obtain u¯sp−u¯=−kdΔ,u¯bu−u¯=rdΔ. u_sp- u=- kd , u_bu- u= rd . Consequently, ‖ubt‖22 \|u_bt\|_2^2 =rk2d2Δ2+kr2d2Δ2 =r k^2d^2 ^2+k r^2d^2 ^2 =rk(r+k)d2Δ2=rkdΔ2. = rk(r+k)d^2 ^2= rkd ^2. Combining these identities proves equation 44. Step 3: maximal first-order rate and stationarity. For every unit tangent direction, Cauchy–Schwarz gives u⊤δ=u⟂⊤δ≤‖u⟂‖2.u δ=u_ δ≤\|u_ \|_2. If u⟂≠0u_ ≠ 0, equality holds at δ⋆=u⟂‖u⟂‖2,δ = u_ \|u_ \|_2, which is a feasible unit tangent direction because d⊤u⟂=01_d u_ =0. If u⟂=0u_ =0, every tangent direction has zero rate. It follows that maxδ∈‖δ‖2=1u⊤δ=‖u⟂‖2. _ subarraycδ \\ \|δ\|_2=1 subarrayu δ=\|u_ \|_2. This proves equation 41. In particular, the flat profile is first-order stationary with respect to the fixed-frame, norm-matched interventions if and only if u⟂=0⟺u∈spand⟺u1=⋯=ud.u_ =0 u \1_d\ u_1=·s=u_d. Step 4: the two-level direction and capture ratio. A band-constant tangent direction has the form asp+bbua1_sp+b1_bu with ra+kb=0ra+kb=0. Since r,k>0r,k>0, these directions form the line spanned by v=rdbu−kdsp,‖v‖22=rkd>0.v= rd1_bu- kd1_sp, \|v\|_2^2= rkd>0. For the two-level curve w(ρ)=cρ,r(sp+ρbu)w(ρ)=c_ρ,r(1_sp+ 1_bu), the exact normalizer satisfies cρ,r=dr+kρ2,c1,r=1,dρcρ,r|ρ=1=−kd.c_ρ,r= dr+kρ^2, c_1,r=1, . ddρc_ρ,r |_ρ=1=- kd. Therefore w′(1) w (1) =−kd(sp+bu)+bu =- kd(1_sp+1_bu)+1_bu =−kdsp+rdbu=v. =- kd1_sp+ rd1_bu=v. The chain rule now yields −dρℒ(w(ρ))|ρ=1 - . ddρL(w(ρ)) |_ρ=1 =u⊤v =u v =rd∑i∈ℐbuui−kd∑i∈ℐspui = rd _i _buu_i- kd _i _spu_i =rkd(u¯bu−u¯sp)=rkdΔ. = rkd( u_bu- u_sp)= rkd . This proves equation 40. More explicitly, as h→0h→ 0, ℒ(w(1+h))=ℒ(d)−hrkdΔ+o(|h|).L(w(1+h))=L(1_d)-h rkd +o(|h|). If Δ>0 >0, every sufficiently small positive h reduces the loss. If Δ<0 <0, every sufficiently small positive h increases it, and a sufficiently small negative h reduces it. When Δ=0 =0, the first derivative vanishes and does not determine whether a higher-order improvement occurs. Thus the strict sign criterion is an if-and-only-if statement about first-order improvement. For the capture ratio, observe that projspanvu=u⊤v‖v‖22v=Δv=ubt.proj_span\v\u= u v\|v\|_2^2v= v=u_bt. The two unit directions in this line are ±v/∥v∥2± v/\|v\|_2, and hence maxδ∈spanv‖δ‖2=1u⊤δ=|u⊤v|‖v‖2=rkd|Δ|=‖ubt‖2. _ subarraycδ \v\\\ \|δ\|_2=1 subarrayu δ= |u v|\|v\|_2= rkd\,| |=\|u_bt\|_2. Both signs can be realized along the two-level family: the curves w(1±ε/‖v‖2)w(1± /\|v\|_2) have unit velocities ±v/∥v∥2± v/\|v\|_2 at ε=0 =0. Thus, when u⟂≠0u_ ≠ 0, the ratio of the restricted and unrestricted unit-tangent optima is Rbt=rk/d|Δ|‖u⟂‖2=‖ubt‖2‖u⟂‖2,Rbt2=‖ubt‖22‖ubt‖22+‖uwi‖22∈[0,1].R_bt= rk/d\,| |\|u_ \|_2= \|u_bt\|_2\|u_ \|_2, R_bt^2= \|u_bt\|_2^2\|u_bt\|_2^2+\|u_wi\|_2^2∈[0,1]. This proves equation 42. The ratio is not defined when u⟂=0u_ =0, since both first-order optima then vanish. ∎ What finer shaping can add. Two bands can use ubtu_bt, the differences between band means. They cannot use uwiu_wi, the variation within bands. By Step 1, every differentiable curve of norm-matched gains through d1_d, however fine-grained, has its velocity in T. By Step 3, its first-order rate per unit velocity is therefore at most ‖u⟂‖2\|u_ \|_2. The best two-band change reaches ‖ubt‖2\|u_bt\|_2. Finer shaping can thus add at most ‖u⟂‖2−‖ubt‖2=(1−Rbt)‖u⟂‖2≤‖uwi‖2\|u_ \|_2-\|u_bt\|_2=(1-R_bt)\|u_ \|_2≤\|u_wi\|_2 to the first-order rate. In particular, if utilities are constant within each band, finer shaping cannot improve on two bands to first order. The discussion after Theorem 1 covers the remaining caveats. F.2 Proof of Theorem 2 Proof. After a common reordering of the singular directions, write U=[UspUbu],V=[VspVbu].U=[U_sp\;U_bu], V=[V_sp\;V_bu]. Then Q=UV⊤=UspVsp⊤+UbuVbu⊤.Q=UV =U_spV_sp +U_buV_bu . The prescribed unnormalized update is D~(a,ρ)=Uspdiag(a)Vsp⊤+ρUbuVbu⊤. D(a,ρ)=U_spdiag(a)V_sp +ρ U_buV_bu . Substituting UbuVbu⊤=Q−UspVsp⊤U_buV_bu =Q-U_spV_sp gives D~(a,ρ) D(a,ρ) =Uspdiag(a)Vsp⊤+ρ(Q−UspVsp⊤) =U_spdiag(a)V_sp +ρ (Q-U_spV_sp ) =ρQ+Usp(diag(a)−ρIr)Vsp⊤ =ρ Q+U_sp (diag(a)-ρ I_r )V_sp =ρQ+Uspdiag(a−ρr)Vsp⊤. =ρ Q+U_spdiag(a- 1_r)V_sp . This is equation 14. The correction factors through an r-dimensional space, so rank(D~(a,ρ)−ρQ)≤r.rank ( D(a,ρ)-ρ Q )≤ r. For a=ra=1_r, it follows that D~ρ=ρQ−(ρ−1)UspVsp⊤. D_ρ=ρ Q-(ρ-1)U_spV_sp . Define Psp=UspUsp⊤,Rsp=VspVsp⊤.P_sp=U_spU_sp , R_sp=V_spV_sp . Orthonormality within each singular-vector matrix gives PspQ P_spQ =UspUsp⊤(UspVsp⊤+UbuVbu⊤) =U_spU_sp (U_spV_sp +U_buV_bu ) =UspVsp⊤, =U_spV_sp , and QRsp QR_sp =(UspVsp⊤+UbuVbu⊤)VspVsp⊤ = (U_spV_sp +U_buV_bu )V_spV_sp =UspVsp⊤. =U_spV_sp . Both identities hold for either rectangular orientation, giving D~ρ=ρQ−(ρ−1)PspQ,m≤n,ρQ−(ρ−1)QRsp,n<m. D_ρ= casesρ Q-(ρ-1)P_spQ,&m≤ n,\\ ρ Q-(ρ-1)QR_sp,&n<m. cases (45) With time indices restored, multiplication by cρ,rtc_ρ,r_t recovers the main-method identity equation 9. No approximation or random-matrix assumption is used. ∎ F.3 Exact Frobenius Normalization Section 4.2 uses the following consequence of Theorem 2. Corollary 3 (Exact Frobenius normalization). Under Theorem 2, ‖D~(a,ρ)‖F2=‖a‖22+ρ2(d−r).\| D(a,ρ)\|_F^2=\|a\|_2^2+ρ^2(d-r). (46) For r+ρ2(d−r)>0r+ρ^2(d-r)>0, the normalizer cρ,rc_ρ,r from equation 4 gives ‖cρ,rD~ρ‖F=‖Q‖F=d\|c_ρ,r D_ρ\|_F=\|Q\|_F= d. Proof. The rank-one singular modes are Frobenius-orthonormal: ⟨uivi⊤,ujvj⊤⟩F u_iv_i ,u_jv_j _F =tr(viui⊤ujvj⊤) =tr (v_iu_i u_jv_j ) =(ui⊤uj)(vi⊤vj)=i=j. =(u_i u_j)(v_i v_j)=1\i=j\. Consequently, the squared Frobenius norm of a spectral profile is the sum of its squared gains. In particular, ‖D~(a,ρ)‖F2=∑i=1rai2+ρ2(d−r)=‖a‖22+ρ2(d−r),\| D(a,ρ)\|_F^2= _i=1^ra_i^2+ρ^2(d-r)=\|a\|_2^2+ρ^2(d-r), which proves equation 46. For a=ra=1_r, ‖D~ρ‖F2=r+ρ2(d−r),‖Q‖F2=‖∑i=1duivi⊤‖F2=d.\| D_ρ\|_F^2=r+ρ^2(d-r), \|Q\|_F^2= \| _i=1^du_iv_i \|_F^2=d. Whenever r+ρ2(d−r)>0r+ρ^2(d-r)>0, the stated normalizer is well-defined and satisfies ‖cρ,rD~ρ‖F2=dr+ρ2(d−r)[r+ρ2(d−r)]=d=‖Q‖F2.\|c_ρ,r D_ρ\|_F^2= dr+ρ^2(d-r) [r+ρ^2(d-r) ]=d=\|Q\|_F^2. Taking nonnegative square roots proves the corollary. If r+ρ2(d−r)=0r+ρ^2(d-r)=0, the raw two-level direction is zero and cannot be rescaled to norm d d; this case is excluded by the corollary’s hypothesis. ∎ F.4 Online Calibration of the Noise Edge B-Adaptive estimates the noise scale online from two minibatch halves, without a large-batch reference gradient. The proposition below proves the exact half-batch and Nesterov covariance identities. The Gaussian model behind the noise edge is treated separately after the proof. Proposition 4 (Half-batch calibration of Muon’s spectral input). Let X1,…,Xb∈ℝm×nX_1,…,X_b ^m× n be independent per-example gradients with common mean G¯ G and Cov(vecXi)=sex2Imn,Cov(vecX_i)=s_ex^2I_mn, where b is a positive even integer. Let G(1),G(2)G^(1),G^(2) be the two half-minibatch means, and define G=G(1)+G(2)2,Zsplit=G(1)−G(2)2.G= G^(1)+G^(2)2, Z^split= G^(1)-G^(2)2. Then Cov(vec(G−G¯))=Cov(vecZsplit)=sb2Imn,sb2=sex2b,Cov(vec(G- G))=Cov(vecZ^split)=s_b^2I_mn, s_b^2= s_ex^2b, (47) and the variance estimator satisfies s^b 2=‖Zsplit‖F2mn,[s^b 2]=sb2. s_b^\,2= \|Z^split\|_F^2mn, [ s_b^\,2]=s_b^2. (48) Across training steps, suppose Gt=G¯t+εtG_t= G_t+ _t, where the noises εt _t are centered, independent across time, and have the same covariance sb2Imns_b^2I_mn after vectorization. For β∈[0,1)β∈[0,1) and B0=0B_0=0, define Bt=βBt−1+(1−β)Gt,Mt=(1−β)Gt+βBt.B_t=β B_t-1+(1-β)G_t, M_t=(1-β)G_t+β B_t. (49) The component obtained by propagating the sampling noises through these recursions is NtM=(1−β2)εt+β2(1−β)∑j=1t−1βt−1−jεj,N_t^M=(1-β^2) _t+β^2(1-β) _j=1^t-1β^t-1-j _j, where the sum is empty for t=1t=1. For every t≥1t≥ 1, Cov(vecNtM)=νt2(β)sb2Imn,Cov(vecN_t^M)= _t^2(β)s_b^2I_mn, (50) with νt2(β)=(1−β2)2+β41−β1+β(1−β2(t−1)). _t^2(β)=(1-β^2)^2+β^4 1-β1+β (1-β^2(t-1) ). (51) Here νt(β) _t(β) denotes the nonnegative square root of this variance multiplier. Interpretation. The half-batch difference removes the unknown mean gradient. Its covariance equals that of the full-minibatch mean; each half alone has twice this variance. The factor νt2(β) _t^2(β) accounts for the current gradient entering the Nesterov input both directly and through the updated buffer. For β>0β>0, the EMA-only multiplier 1−β1+β(1−β2t) 1-β1+β (1-β^2t ) would therefore understate the noise in MtM_t. The implementation in Section 3.1 uses Gt=χtGtrawG_t= _tG_t^raw in the momentum recursion, keeps s¯t 2 s_t^\,2 as an EMA of χt2s^t 2 _t^2 s_t^\,2, substitutes s¯t 2 s_t^\,2 for sbs_b, and applies κedge _edge, which gives equation 8. Appendix C.1 lists the approximations this involves. Neither the covariance identities nor the noise edge show that the resulting split is the best one for optimization. Proof. We first establish the half-batch identities and then propagate the noise through the momentum recursion. Step 1: half-batch covariance and variance estimation. The half-minibatch means are G(1)=2b∑i=1b/2Xi,G(2)=2b∑i=b/2+1bXi.G^(1)= 2b _i=1^b/2X_i, G^(2)= 2b _i=b/2+1^bX_i. By the proposition’s assumptions, they are independent, have common mean G¯ G, and satisfy Cov(vecG(1))=Cov(vecG(2))=2sex2bImn.Cov(vecG^(1))=Cov(vecG^(2))= 2s_ex^2bI_mn. Let g1=vec(G(1)−G¯),g2=vec(G(2)−G¯).g_1=vec(G^(1)- G), g_2=vec(G^(2)- G). Then vec(G−G¯)=g1+g22,vecZsplit=g1−g22.vec(G- G)= g_1+g_22, ^split= g_1-g_22. The cross-covariances vanish by independence, so Cov(vec(G−G¯)) (vec(G- G)) =14(Cov(g1)+Cov(g2)) = 14 (Cov(g_1)+Cov(g_2) ) =sex2bImn=sb2Imn, = s_ex^2bI_mn=s_b^2I_mn, and the identical calculation for the difference gives Cov(vecZsplit)=sb2Imn.Cov(vecZ^split)=s_b^2I_mn. This proves equation 47. Since ZsplitZ^split has zero mean, [s^b 2] [ s_b^\,2] =1mn‖Zsplit‖F2 = 1mnE\|Z^split\|_F^2 =1mntr(Cov(vecZsplit)) = 1mntr (Cov(vecZ^split) ) =1mntr(sb2Imn)=sb2. = 1mntr(s_b^2I_mn)=s_b^2. This proves equation 48. For completeness, the average and difference also satisfy Cov(vec(G−G¯),vecZsplit)=14(Cov(g1)−Cov(g2))=0.Cov (vec(G- G),vecZ^split )= 14 (Cov(g_1)-Cov(g_2) )=0. If (g1,g2)(g_1,g_2) is jointly Gaussian, the average and difference are therefore independent. Without Gaussianity, zero cross-covariance does not by itself imply independence; that independence is not needed for the variance identity. Step 2: propagation through the Nesterov spectral input. Write Gt=G¯t+εtG_t= G_t+ _t and consider the component obtained by linearly propagating the centered sampling noises through the recursions in equation 49. The EMA noise component satisfies NtB=βNt−1B+(1−β)εt,N0B=0.N_t^B=β N_t-1^B+(1-β) _t, N_0^B=0. Unrolling the recursion gives, for t≥1t≥ 1, Nt−1B=(1−β)∑j=1t−1βt−1−jεj,N_t-1^B=(1-β) _j=1^t-1β^t-1-j _j, with an empty sum when t=1t=1. Because the noises are independent across time and have the same covariance sb2Imns_b^2I_mn, Cov(vecNt−1B) (vecN_t-1^B) =(1−β)2∑j=1t−1β2(t−1−j)sb2Imn =(1-β)^2 _j=1^t-1β^2(t-1-j)s_b^2I_mn =1−β1+β(1−β2(t−1))sb2Imn. = 1-β1+β (1-β^2(t-1) )s_b^2I_mn. The spectral-input noise is NtM N_t^M =(1−β)εt+βNtB =(1-β) _t+β N_t^B =(1−β)εt+β(βNt−1B+(1−β)εt) =(1-β) _t+β (β N_t-1^B+(1-β) _t ) =(1−β2)εt+β2Nt−1B =(1-β^2) _t+β^2N_t-1^B =(1−β2)εt+β2(1−β)∑j=1t−1βt−1−jεj. =(1-β^2) _t+β^2(1-β) _j=1^t-1β^t-1-j _j. The pair (εt,Nt−1B)( _t,N_t-1^B) is independent, so Cov(vecNtM) (vecN_t^M) =(1−β2)2Cov(vecεt)+β4Cov(vecNt−1B) =(1-β^2)^2Cov(vec _t)+β^4Cov(vecN_t-1^B) =[(1−β2)2+β41−β1+β(1−β2(t−1))]sb2Imn = [(1-β^2)^2+β^4 1-β1+β (1-β^2(t-1) ) ]s_b^2I_mn =νt2(β)sb2Imn. = _t^2(β)s_b^2I_mn. This proves equation 50 and equation 51. The current gradient appears twice, directly and through the buffer, so its two coefficients are combined before taking covariances. Finally, νt2(β) _t^2(β) =(1−β2)2+β41−β1+β(1−β2(t−1)) =(1-β^2)^2+β^4 1-β1+β (1-β^2(t-1) ) =(1−β)2[(1+β)2+β4(1−β2(t−1))1−β2], =(1-β)^2 [(1+β)^2+ β^4 (1-β^2(t-1) )1-β^2 ], which is the method-section factor in equation 7. The entrywise standard deviation of NtMN_t^M is sbνt(β)s_b _t(β), where νt(β) _t(β) is the nonnegative square root of the variance multiplier. ∎ The noise edge under the Gaussian model. The covariance calculation above does not need Gaussian entries, and it does not determine a spectral edge by itself. Assume, in addition, that the entries of each εt _t are independent centered Gaussian variables with variance sb2s_b^2, and that the noise matrices are independent across time. Then the linear combination for NtMN_t^M has independent centered Gaussian entries with variance sb2νt2(β)s_b^2 _t^2(β). For fixed t and β∈[0,1)β∈[0,1), assume first that sb>0s_b>0 and set Em,n=NtM/(sbνt(β))E_m,n=N_t^M/(s_b _t(β)). This is a standard Gaussian matrix. Its largest singular value converges to the upper edge of the Marchenko–Pastur law (Marchenko & Pastur, 1967; Geman, 1980): ‖Em,n‖2m+n→a.s.1as m,n→∞,m/n→ζ∈(0,∞). \|E_m,n\|_2 m+ n a.s.1 m,n→∞, m/n→ζ∈(0,∞). Rescaling gives ‖NtM‖2/(sbνt(β)(m+n))→1\|N_t^M\|_2/ (s_b _t(β)( m+ n) )→ 1 almost surely, so the noise edge of NtMN_t^M is τtM=sbνt(β)(m+n). _t^M=s_b _t(β)( m+ n). (52) If sb=0s_b=0, the centered noise vanishes almost surely and the edge is zero. This step needs the Gaussian model and an asymptotic result. Isotropic covariance alone does not give it, and it is not a finite-sample bound. Appendix G Omitted experiment results In this section, we give the detailed experimental results. G.1 Corpora adaptation. 14M 31M 70M 160M 410M Corpus ℓ0 _0 Δ ℓ0 _0 Δ ℓ0 _0 Δ ℓ0 _0 Δ ℓ0 _0 Δ Wiki-de 4.530 −1.247± 0.001-1.247 \,±\,0.001 4.009 −1.050± 0.002-1.050 \,±\,0.002 3.699 −1.019± 0.003-1.019 \,±\,0.003 3.171 −0.719± 0.002-0.719 \,±\,0.002 2.626 −0.445± 0.001-0.445 \,±\,0.001 Wiki-zh 3.715 −0.970± 0.001-0.970 \,±\,0.001 3.465 −0.994± 0.001-0.994 \,±\,0.001 3.329 −1.118± 0.005-1.118 \,±\,0.005 3.090 −1.035± 0.001-1.035 \,±\,0.001 2.570 −0.691± 0.002-0.691 \,±\,0.002 FineMath-4 3.572 −0.486± 0.002-0.486 \,±\,0.002 3.278 −0.502± 0.001-0.502 \,±\,0.001 3.088 −0.537± 0.002-0.537 \,±\,0.002 2.753 −0.481± 0.003-0.481 \,±\,0.003 2.421 −0.375± 0.005-0.375 \,±\,0.005 Wiki-ar 2.936 −0.930± 0.000-0.930 \,±\,0.000 2.761 −0.906± 0.001-0.906 \,±\,0.001 2.747 −1.045± 0.000-1.045 \,±\,0.000 2.620 −1.012± 0.000-1.012 \,±\,0.000 2.150 −0.698± 0.001-0.698 \,±\,0.001 FineMath-3 2.930 −0.290± 0.002-0.290 \,±\,0.002 2.692 −0.261± 0.002-0.261 \,±\,0.002 2.566 −0.269± 0.004-0.269 \,±\,0.004 2.320 −0.254± 0.002-0.254 \,±\,0.002 2.050 −0.159± 0.002-0.159 \,±\,0.002 CodeParrot 2.381 −0.479± 0.004-0.479 \,±\,0.004 2.040 −0.391± 0.001-0.391 \,±\,0.001 1.845 −0.361± 0.003-0.361 \,±\,0.003 1.508 −0.238± 0.003-0.238 \,±\,0.003 1.229 −0.118± 0.002-0.118 \,±\,0.002 Table 8: Corpus adaptation across five model scales. ℓ0 _0 is the held-out loss before continued pretraining. Δ=ℓfinal−ℓ0 = _final- _0 is the change after about 100M tokens with Muon, reported as mean ± standard deviation over three seeds. Negative values indicate improvement. ℓ0 _0 has no spread, because all seeds start from the same checkpoint and use the same held-out windows. Rows are ordered by decreasing ℓ0 _0, which gives the same order at every scale. Corpora are grouped into mathematics, code, and multilingual Wikipedia. Models and datasets. We conduct continued pretraining by adapting converged, publicly available Pythia checkpoints on new corpora with the cross-entropy loss. We use five scales, Pythia-14M, 31M, 70M, 160M, and 410M, with hidden widths dmodel=128d_model=128, 256, 512, 768, and 1024, respectively. For the matrix methods, AdamW updates the embeddings and the language-model head. Table 15 in Appendix G lists the widths, batch settings, and training budgets. Pythia was pretrained on the Pile (Gao et al., 2021), a mainly English mixture that includes GitHub code, English Wikipedia, and arXiv. We adapt it to six corpora in three groups. For mathematics, FineMath-4 and FineMath-3 (Allal et al., 2025) come from Common Crawl with different math-quality thresholds. For code, CodeParrot is CodeParrot-clean (CodeParrot, 2022), a filtered and deduplicated Python corpus from GitHub. For multilingual text, Wiki-de, Wiki-zh, and Wiki-ar are German, Chinese, and Arabic Wikipedia. Initial loss reflects corpus shift. Table 8 reports the pre-adaptation held-out loss ℓ0 _0 and the change Δ=ℓfinal−ℓ0 = _final- _0 after 100M tokens of continued pretraining with Muon. We use ℓ0 _0 as a proxy for how far each corpus is from the Pythia pretraining distribution. A lower ℓ0 _0 means that the pretrained model already assigns higher probability to the corpus. A higher ℓ0 _0 suggests a larger distribution shift. The corpora keep the same order at every scale. CodeParrot has the lowest ℓ0 _0, as GitHub code is already part of the Pile. Wiki-de and Wiki-zh have the highest ℓ0 _0, as they are far from the mainly English Pile. The two math corpora, which overlap with mathematical text in the Pile, lie in between. Wiki-ar also lies in between, below FineMath-4 and above FineMath-3 at every scale. Continued pretraining with Muon lowers the validation loss in every setting, but a larger ℓ0 _0 does not always give a larger reduction. For example, Wiki-ar starts below FineMath-4 yet gains about twice as much at every scale. FineMath-4 has higher ℓ0 _0 than FineMath-3 and gains more at every scale. CodeParrot starts with the lowest ℓ0 _0 and has the smallest reductions at 160M and 410M, with Δ=−0.479 =-0.479 at 14M and −0.118-0.118 at 410M. Among the multilingual corpora, Wiki-de starts with the highest ℓ0 _0. Wiki-zh and Wiki-ar keep large gains through 160M and still improve by about 0.70.7 at 410M. Thus, ℓ0 _0 describes how well the pretrained model already fits a corpus, while Δ records the change achieved within the adaptation budget. G.2 Ablation studies Fixed fraction versus noise calibration. Both variants use the same norm-matched two-level rule with ρ=4ρ=4. They differ in how they choose the split and in how they track the spike subspace (Appendix C.3). Table 9 shows that the best fixed fraction varies across models and corpora. B-Adaptive instead sets the split for each matrix and step from the noise edge. On average, it keeps 4.6%4.6\% of dmodeld_model directions as spikes (5th–95th percentile: 1.31.3–7.7%7.7\%; per-corpus means: 2.82.8–5.8%5.8\%). Without any fraction search, it performs on par with the shared B-Static(5%). Their aggregate losses are close (Table 2), and B-Adaptive has lower loss in 10 settings, higher loss in 6, and equal loss in 14. B-Static(5%) has the lower per-step cost (Table 14). Selecting the fraction for each setting gives further gains: B-Static† has lower loss than B-Adaptive in 20 settings, higher loss in 4, and equal loss in 6 (Table 1). Noise calibration thus removes the fraction search, a shared fraction lowers the per-step cost, and per-setting selection lowers the loss further at an extra tuning cost. Adding a third band. We also test whether a finer split improves on two bands. The three-band variant of B-Static splits the singular directions of each matrix into the leading 5%, the next 15%, and the remaining 80%. Its leading band is the same as in B-Static(5%), so the comparison isolates the effect of splitting the bulk into two parts. Averaged over all 3030 model–corpus settings, the three-band variant ends with a final loss 0.002±0.0010.002± 0.001 higher than B-Static(5%). The extra band therefore does not help, and two bands remain the better choice. This is consistent with Theorem 1, under which a finer split can improve on two bands only through utility differences within the bands. Single-update allocation experiment. On Pythia-70M Muon trajectories for FineMath-4, CodeParrot, and Wiki-ar, we compare exact-SVD updates with ρ=1.25ρ=1.25 and 1.51.5 against the exact flat update ρ=1ρ=1 near 25M and 50M tokens. Each comparison fixes the current singular frames and noise-edge partitions, applies a common ρ across 24 hidden weight matrices, and matches each matrix’s Frobenius direction norm separately. Both bulk-emphasizing gains lower the validation loss immediately after the update in five of six corpus–checkpoint settings, despite paired pre-update linearized predictions that favor less bulk weight. These K=0K=0 measurements illustrate the difference between pre-update alignment and the post-update loss response studied in Theorem 1. Appendix E.2 gives the protocol, and Table 7 reports all six settings. Gain sensitivity and numerical accuracy. For B-Adaptive, we sweep ρ∈2,4,6,8,10ρ∈\2,4,6,8,10\ across 30 model–corpus settings using 20M tokens, one seed, and the shared learning rates (Table 12). Within a setting, the loss changes by a median of 0.0090.009 across the tested gains, with larger changes at 70M. The best tested gain improves on ρ=4ρ=4 by about 0.00450.0045 on average. The smallest gain, ρ=2ρ=2, is best in 20 settings. We keep ρ=4ρ=4 as the shared default in all 100M-token runs. The sweep covers bulk-emphasizing gains, and Appendix E.2 studies gains near ρ=1ρ=1. At ρ=4ρ=4, the per-scale median effective gains ρeff _eff lie in [3.963,4.005][3.963,4.005], and the per-scale median Rayleigh–Ritz residuals stay below 10−510^-5 (Table 13). These diagnostics check the realized gain and the accuracy of the retained spike subspaces. Model Method FineMath-4 FineMath-3 CodeParrot Wiki-de Wiki-zh Wiki-ar 14M B-Static(1.5%) 3.088± 0.0033.088 \,±\,0.003 2.642± 0.0022.642 \,±\,0.002 1.903± 0.0031.903 \,±\,0.003 3.278± 0.0013.278 \,±\,0.001 2.741± 0.0002.741 \,±\,0.000 1.999± 0.0011.999 \,±\,0.001 B-Static(2%) 3.086± 0.0023.086 \,±\,0.002 2.639± 0.0012.639 \,±\,0.001 1.899± 0.0031.899 \,±\,0.003 3.279± 0.0003.279 \,±\,0.000 2.739± 0.0012.739 \,±\,0.001 1.998± 0.0011.998 \,±\,0.001 B-Static(5%) 3.085± 0.0033.085 \,±\,0.003 2.639± 0.0022.639 \,±\,0.002 1.900± 0.0031.900 \,±\,0.003 3.278± 0.0013.278 \,±\,0.001 2.738± 0.0012.738 \,±\,0.001 2.000± 0.0002.000 \,±\,0.000 B-Static(15%) 3.083± 0.0023.083 \,±\,0.002 2.639± 0.0022.639 \,±\,0.002 1.898± 0.0031.898 \,±\,0.003 3.279± 0.0013.279 \,±\,0.001 2.742± 0.0012.742 \,±\,0.001 2.006± 0.0012.006 \,±\,0.001 B-Static(20%) 3.083± 0.0023.083 \,±\,0.002 2.638± 0.0022.638 \,±\,0.002 1.899± 0.0041.899 \,±\,0.004 3.282± 0.0023.282 \,±\,0.002 2.745± 0.0012.745 \,±\,0.001 2.009± 0.0012.009 \,±\,0.001 31M B-Static(1.5%) 2.776± 0.0012.776 \,±\,0.001 2.433± 0.0022.433 \,±\,0.002 1.650± 0.0021.650 \,±\,0.002 2.956± 0.0012.956 \,±\,0.001 2.463± 0.0012.463 \,±\,0.001 1.848± 0.0011.848 \,±\,0.001 B-Static(2%) 2.776± 0.0022.776 \,±\,0.002 2.433± 0.0022.433 \,±\,0.002 1.649± 0.0011.649 \,±\,0.001 2.956± 0.0022.956 \,±\,0.002 2.464± 0.0012.464 \,±\,0.001 1.848± 0.0011.848 \,±\,0.001 B-Static(5%) 2.773± 0.0022.773 \,±\,0.002 2.431± 0.0032.431 \,±\,0.003 1.648± 0.0021.648 \,±\,0.002 2.953± 0.0012.953 \,±\,0.001 2.464± 0.0012.464 \,±\,0.001 1.849± 0.0011.849 \,±\,0.001 B-Static(15%) 2.773± 0.0022.773 \,±\,0.002 2.428± 0.0022.428 \,±\,0.002 1.648± 0.0021.648 \,±\,0.002 2.956± 0.0012.956 \,±\,0.001 2.469± 0.0012.469 \,±\,0.001 1.858± 0.0001.858 \,±\,0.000 B-Static(20%) 2.772± 0.0012.772 \,±\,0.001 2.429± 0.0022.429 \,±\,0.002 1.650± 0.0031.650 \,±\,0.003 2.960± 0.0012.960 \,±\,0.001 2.472± 0.0012.472 \,±\,0.001 1.862± 0.0011.862 \,±\,0.001 70M B-Static(1.5%) 2.549± 0.0042.549 \,±\,0.004 2.295± 0.0062.295 \,±\,0.006 1.478± 0.0031.478 \,±\,0.003 2.677± 0.0022.677 \,±\,0.002 2.204± 0.0002.204 \,±\,0.000 1.690± 0.0011.690 \,±\,0.001 B-Static(2%) 2.547± 0.0022.547 \,±\,0.002 2.294± 0.0042.294 \,±\,0.004 1.478± 0.0021.478 \,±\,0.002 2.678± 0.0022.678 \,±\,0.002 2.205± 0.0022.205 \,±\,0.002 1.690± 0.0001.690 \,±\,0.000 B-Static(5%) 2.541± 0.0042.541 \,±\,0.004 2.290± 0.0032.290 \,±\,0.003 1.472± 0.0031.472 \,±\,0.003 2.673± 0.0022.673 \,±\,0.002 2.203± 0.0032.203 \,±\,0.003 1.690± 0.0011.690 \,±\,0.001 B-Static(15%) 2.537± 0.0022.537 \,±\,0.002 2.287± 0.0072.287 \,±\,0.007 1.469± 0.0031.469 \,±\,0.003 2.670± 0.0022.670 \,±\,0.002 2.204± 0.0022.204 \,±\,0.002 1.695± 0.0011.695 \,±\,0.001 B-Static(20%) 2.537± 0.0042.537 \,±\,0.004 2.287± 0.0032.287 \,±\,0.003 1.469± 0.0041.469 \,±\,0.004 2.674± 0.0012.674 \,±\,0.001 2.205± 0.0022.205 \,±\,0.002 1.700± 0.0011.700 \,±\,0.001 160M B-Static(1.5%) 2.281± 0.0032.281 \,±\,0.003 2.077± 0.0022.077 \,±\,0.002 1.280± 0.0041.280 \,±\,0.004 2.458± 0.0022.458 \,±\,0.002 2.051± 0.0012.051 \,±\,0.001 1.597± 0.0021.597 \,±\,0.002 B-Static(2%) 2.281± 0.0022.281 \,±\,0.002 2.077± 0.0032.077 \,±\,0.003 1.281± 0.0031.281 \,±\,0.003 2.457± 0.0032.457 \,±\,0.003 2.051± 0.0002.051 \,±\,0.000 1.598± 0.0021.598 \,±\,0.002 B-Static(5%) 2.283± 0.0032.283 \,±\,0.003 2.077± 0.0032.077 \,±\,0.003 1.283± 0.0041.283 \,±\,0.004 2.459± 0.0022.459 \,±\,0.002 2.054± 0.0012.054 \,±\,0.001 1.606± 0.0011.606 \,±\,0.001 B-Static(15%) 2.285± 0.0022.285 \,±\,0.002 2.079± 0.0022.079 \,±\,0.002 1.283± 0.0041.283 \,±\,0.004 2.466± 0.0022.466 \,±\,0.002 2.066± 0.0002.066 \,±\,0.000 1.625± 0.0011.625 \,±\,0.001 B-Static(20%) 2.284± 0.0032.284 \,±\,0.003 2.080± 0.0022.080 \,±\,0.002 1.284± 0.0031.284 \,±\,0.003 2.470± 0.0032.470 \,±\,0.003 2.072± 0.0012.072 \,±\,0.001 1.632± 0.0011.632 \,±\,0.001 410M B-Static(1.5%) 2.043± 0.0052.043 \,±\,0.005 1.890± 0.0021.890 \,±\,0.002 1.111± 0.0021.111 \,±\,0.002 2.174± 0.0012.174 \,±\,0.001 1.870± 0.0011.870 \,±\,0.001 1.446± 0.0011.446 \,±\,0.001 B-Static(2%) 2.043± 0.0052.043 \,±\,0.005 1.891± 0.0011.891 \,±\,0.001 1.111± 0.0021.111 \,±\,0.002 2.174± 0.0012.174 \,±\,0.001 1.871± 0.0011.871 \,±\,0.001 1.447± 0.0011.447 \,±\,0.001 B-Static(5%) 2.043± 0.0052.043 \,±\,0.005 1.890± 0.0011.890 \,±\,0.001 1.111± 0.0021.111 \,±\,0.002 2.176± 0.0002.176 \,±\,0.000 1.875± 0.0011.875 \,±\,0.001 1.453± 0.0011.453 \,±\,0.001 B-Static(15%) 2.044± 0.0052.044 \,±\,0.005 1.890± 0.0021.890 \,±\,0.002 1.112± 0.0021.112 \,±\,0.002 2.182± 0.0012.182 \,±\,0.001 1.888± 0.0011.888 \,±\,0.001 1.468± 0.0011.468 \,±\,0.001 B-Static(20%) 2.044± 0.0052.044 \,±\,0.005 1.890± 0.0021.890 \,±\,0.002 1.112± 0.0011.112 \,±\,0.001 2.185± 0.0012.185 \,±\,0.001 1.894± 0.0011.894 \,±\,0.001 1.473± 0.0011.473 \,±\,0.001 Table 9: Fixed-fraction sweep for B-Static: final validation loss after about 100M training tokens, reported as mean ± standard deviation over three seeds. For p∈0.015,0.02,0.05,0.15,0.20p∈\0.015,0.02,0.05,0.15,0.20\, the leading band contains rt=⌈pd⌉r_t= pd singular directions, where d=minm,nd= \m,n\ and p is held fixed within each run. All fractions use ρ=4ρ=4 and the shared learning rate of their scale, with the same training budget, schedule, and seeds as Table 1. For each model–corpus setting, the fraction with the lowest three-seed mean gives the B-Static† entry in Table 1. Table 2 summarizes the shared configuration p=5%p=5\%, and Figure 1 shows p=1.5%p=1.5\%. Bold marks the lowest mean among the fractions in each setting, including ties at the printed precision. Arm Model FineMath-4 FineMath-3 CodeParrot Wiki-de Wiki-zh Wiki-ar Freon 14M 8.3± 0.78.3 \,±\,0.7 3.5± 2.93.5 \,±\,2.9 7.1± 1.67.1 \,±\,1.6 4.9± 2.64.9 \,±\,2.6 4.4± 2.64.4 \,±\,2.6 – 31M 7.8± 1.87.8 \,±\,1.8 8.7± 3.58.7 \,±\,3.5 3.3± 2.93.3 \,±\,2.9 4.4± 1.74.4 \,±\,1.7 4.7± 0.74.7 \,±\,0.7 – 70M 11.1± 1.411.1 \,±\,1.4 10.6± 3.210.6 \,±\,3.2 12.5± 0.712.5 \,±\,0.7 9.8± 2.69.8 \,±\,2.6 10.9± 3.310.9 \,±\,3.3 10.2± 1.510.2 \,±\,1.5 160M – – – – – – 410M 9.9± 3.19.9 \,±\,3.1 9.6± 3.59.6 \,±\,3.5 – 6.0± 5.16.0 \,±\,5.1 – – B-Static(5%) 14M 4.6± 0.44.6 \,±\,0.4 3.4± 2.93.4 \,±\,2.9 6.1± 2.16.1 \,±\,2.1 6.3± 2.16.3 \,±\,2.1 10.5± 2.410.5 \,±\,2.4 12.3± 2.212.3 \,±\,2.2 31M 5.2± 0.65.2 \,±\,0.6 – 3.1± 2.83.1 \,±\,2.8 8.2± 2.28.2 \,±\,2.2 9.5± 0.69.5 \,±\,0.6 9.8± 2.19.8 \,±\,2.1 70M 8.4± 2.98.4 \,±\,2.9 9.2± 3.39.2 \,±\,3.3 11.2± 1.611.2 \,±\,1.6 7.0± 1.77.0 \,±\,1.7 7.7± 1.77.7 \,±\,1.7 13.2± 0.613.2 \,±\,0.6 160M – – – – 2.6± 0.82.6 \,±\,0.8 4.6± 2.14.6 \,±\,2.1 410M 10.0± 2.910.0 \,±\,2.9 6.9± 5.56.9 \,±\,5.5 – 10.6± 3.110.6 \,±\,3.1 9.3± 1.69.3 \,±\,1.6 – B-Adaptive 14M 5.2± 1.85.2 \,±\,1.8 4.4± 1.6 2/34.4 \,±\,1.6^\,2/3 6.5± 2.66.5 \,±\,2.6 7.3± 2.87.3 \,±\,2.8 11.3± 2.611.3 \,±\,2.6 11.5± 2.011.5 \,±\,2.0 31M 4.5± 2.24.5 \,±\,2.2 7.5± 1.3 2/37.5 \,±\,1.3^\,2/3 4.6± 3.94.6 \,±\,3.9 8.7± 1.28.7 \,±\,1.2 10.0± 0.210.0 \,±\,0.2 9.6± 2.49.6 \,±\,2.4 70M 5.6± 2.95.6 \,±\,2.9 8.0± 4.88.0 \,±\,4.8 10.7± 1.410.7 \,±\,1.4 5.8± 1.25.8 \,±\,1.2 6.7± 1.66.7 \,±\,1.6 13.1± 0.313.1 \,±\,0.3 160M – – – – 3.6± 0.83.6 \,±\,0.8 6.8± 1.26.8 \,±\,1.2 410M 10.7± 3.010.7 \,±\,3.0 7.2± 4.67.2 \,±\,4.6 – 11.3± 1.811.3 \,±\,1.8 9.7± 1.99.7 \,±\,1.9 – B-Static† 14M 8.1± 0.88.1 \,±\,0.8 4.4± 3.94.4 \,±\,3.9 7.8± 2.77.8 \,±\,2.7 6.3± 2.16.3 \,±\,2.1 10.5± 2.410.5 \,±\,2.4 13.7± 1.613.7 \,±\,1.6 31M 6.3± 0.76.3 \,±\,0.7 10.0± 1.710.0 \,±\,1.7 3.1± 2.83.1 \,±\,2.8 8.2± 2.28.2 \,±\,2.2 10.7± 0.710.7 \,±\,0.7 12.6± 2.812.6 \,±\,2.8 70M 11.1± 3.511.1 \,±\,3.5 11.7± 6.211.7 \,±\,6.2 13.1± 1.913.1 \,±\,1.9 8.0± 4.58.0 \,±\,4.5 7.7± 1.77.7 \,±\,1.7 13.2± 0.613.2 \,±\,0.6 160M – – – – 7.5± 1.17.5 \,±\,1.1 15.3± 2.415.3 \,±\,2.4 410M 10.1± 2.910.1 \,±\,2.9 8.5± 4.58.5 \,±\,4.5 – 13.9± 1.613.9 \,±\,1.6 15.3± 1.215.3 \,±\,1.2 12.8± 1.012.8 \,±\,1.0 Table 10: Token savings against Muon, as a percentage of the 100M-token budget, for each method and model–corpus setting. Muon is the reference, so it has no block of its own. A cell is filled only where the method’s three-seed mean final loss is below Muon’s in Table 1, and “–” marks settings where Muon is at least as good. A filled cell reports 1−D⋆/100M1-D /100M, where D⋆D is the number of tokens at which the method’s curve first reaches Muon’s 100M-token loss in the same seed. Values are mean ± standard deviation over the seeds in which the method ends below Muon. A superscript k/3k/3 marks a setting won in only k of the three seeds. DeVA and Spectra never beat Muon, so they are omitted. The last block, B-Static†, uses the best of the five fractions in each setting (Table 9). Because this fraction is chosen on the same runs, its savings are optimistic. Separately, intermediate points of a cosine schedule are evaluated before the learning rate has fully decayed. Effect of the spike fraction in B-Static. Table 9 presents the results of tuning the fixed fraction p used by B-Static. The optimal value varies substantially across model scales and corpora. Thus, no single value of p is optimal across all experiments, and selecting one requires an expensive tuning of parameter p. In contrast, B-Adaptive determines the rank directly from the spectrum of each matrix using the Marchenko–Pastur noise edge. It selects, on average, 4.6%4.6\% of dmodeld_model (5th–95th percentile: 1.3%1.3\%–7.7%7.7\%; per-corpus means: 2.8%2.8\%–5.8%5.8\%) and, without tuning, closely matches the best fixed fraction across scales. The shared p=5%p=5\% configuration achieves aggregate validation loss comparable to B-Adaptive (Table 2). Table 1 instead reports B-Static†, selected separately for each model–corpus setting from the five fractions in Table 9. The benefit of the adaptive rule is therefore not improved accuracy over a tuned fixed fraction, but achieving comparable accuracy without the tuning. Frequency of improvement and token savings. Table 10 shows which settings each method wins and its token savings in each, complementing the aggregate comparison in Table 2. B-Static(5%), B-Adaptive, and B-Static† have lower three-seed mean final loss than Muon in 23, 24, and 25 of the 30 settings, respectively, compared with 19 for Freon. These counts match the filled cells and exclude ties. Within each winning setting, the savings are averaged over the seeds that also finish below Muon’s same-seed final loss. A superscript marks a setting won in fewer than three seeds, and an entry without one is won in all three seeds. For example, on Pythia-70M CodeParrot, B-Static(5%) and B-Adaptive save 11.2±1.6%11.2± 1.6\% and 10.7±1.4%10.7± 1.4\% of the budget, and B-Static† saves 13.1±1.9%13.1± 1.9\%. Table 14 reports the per-step cost. Learning-rate tuning and the shared-rate policy. We distinguish each method’s best multiplier in the 20M-token sweep from the shared multiplier used in the 100M-token runs. Figure 2 and Table 11 report the sweep on FineMath-4, run with one seed over multiples of the reference rate of Section 5. The swept methods are Muon, Freon, DeVA, Spectra, B-Static, and B-Adaptive. In the sweep, B-Static uses p=20%p=20\%, and both BulkBoost variants use ρ=4ρ=4. Each method’s best multiplier at a scale is the one with the lowest validation loss after 20M tokens. The best multipliers agree across all six methods at 14M, 31M, and 410M, and across five of the six at 70M and 160M. At 70M, Muon does best at 0.1×0.1×, while the other methods do best at 0.25×0.25×. At 160M, Spectra does best at 0.05×0.05×, while the other methods do best at 0.1×0.1×. The shared multipliers are therefore 0.10.1, 0.10.1, 0.250.25, 0.10.1, and 0.050.05 for Pythia-14M, 31M, 70M, 160M, and 410M, and each is the best multiplier for most of the six methods. All 100M-token runs use the resulting peak learning rates for every method, including AdamW, on all six corpora. In particular, the 70M Muon runs use 5×10−45× 10^-4 and the 160M Spectra runs use 10−410^-4, instead of their own best sweep rates. In the sweep, this raises the loss from 2.779 to 2.783 for 70M Muon, and from 2.552 to 2.553 for 160M Spectra. Table 11 marks each method’s best multiplier, and its caption lists the shared multipliers. Figure 2: Learning-rate sweeps on FineMath-4 with a 20M-token budget and one seed. Curves show final validation loss against multiples of each scale’s reference rate. Grey bands mark each method’s best multiplier. B-Static uses p=20%p=20\% in the sweep. Table 11 lists the full results. Model Method 0.01× 0.05× 0.1× 0.25× 0.5× 1× 2× 4× best 14M Muon 3.335 3.232 3.209 3.227 3.281 3.354 3.459 3.581 0.1× Freon 3.338 3.231 3.207 3.219 3.265 3.335 3.427 3.540 0.1× DeVA 3.266 3.235 3.226 3.232 3.267 3.328 3.421 3.615 0.1× Spectra 3.272 3.257 3.250 3.259 3.304 3.400 3.555 4.458 0.1× B-Static(20%) 3.340 3.232 3.207 3.217 3.259 3.332 3.422 3.543 0.1× B-Adaptive 3.336 3.230 3.207 3.223 3.273 3.347 3.448 3.572 0.1× 31M Muon 3.051 2.945 2.922 2.958 3.042 3.176 3.363 3.733 0.1× Freon 3.055 2.945 2.924 2.948 3.016 3.125 3.259 3.450 0.1× DeVA 2.989 2.950 2.942 2.963 3.011 3.106 3.257 3.747 0.1× Spectra 2.993 2.965 2.960 2.987 3.060 3.242 3.547 4.848 0.1× B-Static(20%) 3.061 2.947 2.925 2.944 3.012 3.122 3.261 3.436 0.1× B-Adaptive 3.053 2.943 2.922 2.952 3.034 3.146 3.308 3.504 0.1× 70M Muon 2.908 2.794 2.779 2.783 2.919 3.152 3.386 3.986 0.1× Freon 2.918 2.808 2.792 2.785 2.848 3.064 3.473 3.674 0.25× DeVA 3.070 2.996 2.966 2.827 2.963 3.263 3.415 4.610 0.25× Spectra 3.071 3.005 2.991 2.890 3.198 3.547 5.466 5.936 0.25× B-Static(20%) 2.928 2.809 2.789 2.782 2.845 3.065 3.212 3.467 0.25× B-Adaptive 2.913 2.805 2.796 2.787 2.872 3.083 3.484 3.634 0.25× 160M Muon 2.580 2.483 2.466 2.503 2.579 2.747 2.964 3.434 0.1× Freon 2.590 2.484 2.463 2.498 2.569 2.687 2.918 3.196 0.1× DeVA 2.615 2.550 2.535 2.558 2.739 3.130 3.545 4.937 0.1× Spectra 2.607 2.552 2.553 2.610 3.182 3.940 5.445 6.425 0.05× B-Static(20%) 2.603 2.490 2.466 2.490 2.558 2.666 2.857 3.107 0.1× B-Adaptive 2.587 2.481 2.463 2.505 2.585 2.706 2.901 3.180 0.1× 410M Muon 2.232 2.182 2.199 2.290 2.429 2.728 3.047 3.204 0.05× Freon 2.236 2.180 2.196 2.278 2.412 2.598 2.799 2.989 0.05× DeVA 2.239 2.208 2.218 2.282 2.384 2.779 3.263 3.884 0.05× Spectra 2.235 2.212 2.226 2.294 2.512 3.228 3.529 3.963 0.05× B-Static(20%) 2.246 2.182 2.193 2.273 2.397 2.584 2.783 2.990 0.05× B-Adaptive 2.235 2.183 2.200 2.291 2.433 2.632 2.849 3.060 0.05× Table 11: Final held-out loss on FineMath-4 after 20M tokens with one seed, at multiples of each scale’s reference rate. Bold marks each method’s lowest loss, and underline marks its second lowest. B-Static uses p=20%p=20\%. The 100M-token runs use one shared multiplier per scale, the best for most methods: 0.10.1, 0.10.1, 0.250.25, 0.10.1, and 0.050.05 from 14M to 410M. Model Corpus ρ=2ρ=2 ρ=4ρ=4 ρ=6ρ=6 ρ=8ρ=8 ρ=10ρ=10 best range 14M FineMath-4 3.209 3.209 3.209 3.209 3.210 6 0.000 FineMath-3 2.737 2.737 2.736 2.737 2.735 10 0.002 CodeParrot 2.046 2.046 2.048 2.048 2.048 4 0.002 Wiki-de 3.518 3.522 3.523 3.526 3.528 2 0.010 Wiki-zh 2.960 2.962 2.964 2.965 2.967 2 0.007 Wiki-ar 2.184 2.188 2.190 2.193 2.194 2 0.011 31M FineMath-4 2.923 2.925 2.927 2.927 2.928 2 0.005 FineMath-3 2.530 2.534 2.535 2.536 – 2 0.006 CodeParrot 1.796 1.797 1.798 1.800 1.802 2 0.006 Wiki-de 3.203 3.205 3.208 3.210 3.212 2 0.009 Wiki-zh 2.702 2.705 2.709 2.711 2.712 2 0.011 Wiki-ar 2.046 2.051 2.053 2.056 2.058 2 0.012 70M FineMath-4 2.780 2.795 2.806 2.818 2.837 2 0.057 FineMath-3 2.458 2.457 2.481 2.482 2.492 4 0.035 CodeParrot 1.677 1.691 1.714 1.712 1.731 2 0.054 Wiki-de 2.985 3.020 3.030 3.034 3.043 2 0.057 Wiki-zh 2.481 2.493 2.503 2.516 2.528 2 0.047 Wiki-ar 1.915 1.919 1.923 1.926 1.931 2 0.016 160M FineMath-4 2.466 2.466 2.467 2.466 2.468 4 0.002 FineMath-3 2.205 2.201 2.206 2.203 2.210 4 0.009 CodeParrot 1.408 1.406 1.407 1.407 1.409 4 0.002 Wiki-de 2.698 2.702 2.704 2.706 2.707 2 0.010 Wiki-zh 2.328 2.332 2.337 2.339 2.340 2 0.012 Wiki-ar 1.859 1.863 1.864 1.866 1.870 2 0.011 410M FineMath-4 2.181 2.181 2.180 2.181 2.181 6 0.001 FineMath-3 1.979 1.978 1.977 1.976 1.976 8 0.002 CodeParrot 1.194 1.192 1.191 1.191 1.190 10 0.003 Wiki-de 2.348 2.348 2.348 2.349 2.349 2 0.001 Wiki-zh 2.093 2.097 2.099 2.100 2.101 2 0.008 Wiki-ar 1.668 1.674 1.676 1.678 1.679 2 0.012 Table 12: ρ tuning across all 30 model–corpus pairs, measured by final held-out validation loss after 20M tokens. The best loss in each row is bold, and the second-best is underlined. The range column is the difference between the largest and smallest loss, computed before rounding; its median is 0.0090.009. At each model scale, the sweep uses the same peak learning rate as the main 100M-token B-Adaptive runs (Table 11), held fixed across ρ values and corpora. The sweep uses a different seed from the learning-rate sweep, so its ρ=4ρ=4 entries on FineMath-4 differ from the B-Adaptive entries of Table 11 by at most 0.0080.008. Model matrices median ρeff _eff p5–p95 ρeff−ρ _eff-ρ Ritz residual Pythia-14M 144 3.980 3.88–4.07 −0.020-0.020 4.1×10−7× 10^-7 Pythia-31M 144 3.980 3.91–4.04 −0.020-0.020 4.7×10−7× 10^-7 Pythia-70M 144 4.005 3.95–4.06 +0.005+0.005 6.2×10−7× 10^-7 Pythia-160M 288 3.963 3.92–4.05 −0.037-0.037 1.5×10−6× 10^-6 Pythia-410M 576 3.965 3.93–4.02 −0.035-0.035 8.2×10−6× 10^-6 Table 13: Numerical diagnostics for B-Adaptive at the target gain ρ=4ρ=4, summarized by model scale. “matrices” counts the instrumented two-dimensional weight matrices across the six corpora. We report the median effective gain ρeff _eff, its 5th–95th percentile interval, and the difference between the median and the target gain. The final column reports the median Rayleigh–Ritz residual of the retained spike subspaces. Computation-time overhead. Table 14 reports per-step runtimes relative to Muon at each model scale, averaged over the six corpora. The measured fixed-rank configurations with p=1.5%p=1.5\% and 5%5\% require 1.191.19–1.59×1.59× Muon’s step time, compared with 1.361.36–2.13×2.13× for B-Adaptive. At 70M–410M, these ranges narrow to 1.191.19–1.27×1.27× and 1.361.36–1.50×1.50×, respectively. The two measured fixed fractions have nearly identical runtimes and are faster per step than the noise-calibrated variant at every scale. For the other methods, AdamW and Spectra take 0.920.92–0.98×0.98× and 0.980.98–1.00×1.00× Muon’s step time, respectively, while Freon and DeVA incur overheads of up to 20%20\% and 34%34\%. These timings compare the implemented variants, which differ in both noise calibration and subspace tracking, so they do not isolate the cost of rank selection. The overhead of the adaptive variant is largest at 14M (113%113\%) and lies between 36%36\% and 66%66\% at the larger scales. Arm 14M 31M 70M 160M 410M Muon (ms/step) 173± 5173 \,±\,5 308± 1308 \,±\,1 617± 0617 \,±\,0 635± 1635 \,±\,1 1748± 101748 \,±\,10 AdamW 0.93± 0.05×0.93 \,±\,0.05× 0.92± 0.01×0.92 \,±\,0.01× 0.98± 0.01×0.98 \,±\,0.01× 0.98± 0.00×0.98 \,±\,0.00× 0.98± 0.01×0.98 \,±\,0.01× Freon 1.17± 0.01×1.17 \,±\,0.01× 1.10± 0.00×1.10 \,±\,0.00× 1.06± 0.00×1.06 \,±\,0.00× 1.17± 0.00×1.17 \,±\,0.00× 1.20± 0.00×1.20 \,±\,0.00× DeVA 1.13± 0.00×1.13 \,±\,0.00× 1.10± 0.00×1.10 \,±\,0.00× 1.11± 0.00×1.11 \,±\,0.00× 1.27± 0.00×1.27 \,±\,0.00× 1.34± 0.00×1.34 \,±\,0.00× Spectra 0.98± 0.01×0.98 \,±\,0.01× 1.00± 0.00×1.00 \,±\,0.00× 1.00± 0.00×1.00 \,±\,0.00× 1.00± 0.00×1.00 \,±\,0.00× 1.00± 0.00×1.00 \,±\,0.00× B-Static(1.5%) 1.59± 0.05×1.59 \,±\,0.05× 1.32± 0.01×1.32 \,±\,0.01× 1.19± 0.00×1.19 \,±\,0.00× 1.27± 0.00×1.27 \,±\,0.00× 1.27± 0.01×1.27 \,±\,0.01× B-Static(5%) 1.59± 0.04×1.59 \,±\,0.04× 1.33± 0.01×1.33 \,±\,0.01× 1.19± 0.00×1.19 \,±\,0.00× 1.27± 0.00×1.27 \,±\,0.00× 1.27± 0.01×1.27 \,±\,0.01× B-Adaptive 2.13± 0.05×2.13 \,±\,0.05× 1.66± 0.00×1.66 \,±\,0.00× 1.36± 0.00×1.36 \,±\,0.00× 1.50± 0.00×1.50 \,±\,0.00× 1.48± 0.01×1.48 \,±\,0.01× Table 14: Per-step wall-clock time at each model scale. For each run, the per-step time is its total wall-clock time divided by its number of steps. Entries are mean ± standard deviation over the runs at that scale, across six corpora and three seeds. The Muon row reports milliseconds per step; all other rows report runtime relative to Muon at the same scale. Fixed-rank timings cover p=1.5%p=1.5\% and 5%5\%. Measurements were collected on shared cluster nodes and are subject to scheduling variability. The ratios within a scale are stable. Model dmodeld_model micro-batch × accum steps tokens Muon ms/step Pythia-14M 128 32×1 3051 100M 173± 5173 \,±\,5 Pythia-31M 256 32×1 3051 100M 308± 1308 \,±\,1 Pythia-70M 512 32×1 3051 100M 617± 0617 \,±\,0 Pythia-160M 768 32×1 3051 100M 635± 1635 \,±\,1 Pythia-410M 1024 32×1 3051 100M 1748± 101748 \,±\,10 Table 15: Pythia model configurations and training budgets. dmodeld_model denotes the hidden width, and “micro-batch × accum” gives the number of sequences per microbatch and the number of gradient accumulation steps. All scales use an effective batch of 32 sequences of length 1024, giving 32,768 tokens per step and approximately 100M tokens over 3,051 steps. The Muon timing entries reproduce those in Table 14: mean ± standard deviation over six corpora and three seeds at each scale. Figure 3: Final held-out validation loss as a function of model size, with one panel per corpus and all seven optimization methods shown. Across all corpora, final loss decreases consistently as model size increases, matching the expected scaling trend. The rate of decrease differs by corpus, reflecting differences in adaptation difficulty, but the overall monotonic improvement with scale remains stable. The B-Static curve uses p=20%p=20\% (Table 9); the other curves match Table 1. G.3 Additional learning curves Final loss against model size. Figure 3 plots final loss against model size, with one panel per corpus and all seven methods shown on an absolute loss axis. The curves match Table 1, except that B-Static is shown at p=20%p=20\% (Table 9). This view makes the scaling trend clear: final loss decreases steadily with model size on every corpus. Figures 4–8 show learning curves for the five remaining corpora—FineMath-4, FineMath-3, CodeParrot, Wiki-de, and Wiki-ar. Those figures use the same layout as Figure 1 (Wiki-zh). All fixed-rank curves use p=1.5%p=1.5\%. Figure 4: Held-out loss vs. tokens on FineMath-4, one column per scale. Top: the full run, with the pre-adaptation loss ℓ0 _0 dotted and the zoom window shaded. Bottom: the shaded window, rescaled. Note the y-ranges differ per panel. Figure 5: Held-out loss vs. tokens on FineMath-3, one column per scale. Top: the full run, with the pre-adaptation loss ℓ0 _0 dotted and the zoom window shaded. Bottom: the shaded window, rescaled. Note the y-ranges differ per panel. Figure 6: Held-out loss vs. tokens on CodeParrot, one column per scale. Top: the full run, with the pre-adaptation loss ℓ0 _0 dotted and the zoom window shaded. Bottom: the shaded window, rescaled. Note the y-ranges differ per panel. Figure 7: Held-out loss vs. tokens on Wiki-de, one column per scale. Top: the full run, with the pre-adaptation loss ℓ0 _0 dotted and the zoom window shaded. Bottom: the shaded window, rescaled. Note the y-ranges differ per panel. Figure 8: Held-out loss vs. tokens on Wiki-ar, one column per scale. Top: the full run, with the pre-adaptation loss ℓ0 _0 dotted and the zoom window shaded. Bottom: the shaded window, rescaled. Note the y-ranges differ per panel.