Paper deep dive
OptiSelect: How does the Optimizer Shape Data Curriculum?
Simin Fan, Alireza Abdollahpoorrostam, Martin Jaggi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Online data selection has demonstrated substantial efficiency gains for LLM pretraining by training on the most valuable candidates within each batch. Since a candidate's value is realized through its effective model update, principled selection should account for the optimizer step, which reshapes the raw gradient before it updates model parameters. We formalize this optimizer-aware selection paradigm as OptiSelect and present the first systematic study of how the optimizer shapes data selection. Our theory establishes a selection gain principle in which the advantage of online selection is governed by the discriminability of the optimizer-induced utility scores. We prove that sign-based and polar-tangential preconditioners of Lion and Muon would suffer from a discriminability collapse which caps attainable gains from OptiSelect, whereas diagonal-adaptive optimizers such as AdamW and Sophia admit strictly better upper bounds. The proposed principle also yields a quantitative derivation of the optimal candidate oversampling ratio. Pretraining experiments on 124M and 720M models are consistent with our theoretical analysis and show that AdamW's diagonal-adaptive scoring geometry remains the strongest scoring geometry even with Muon as optimizer. We further demonstrate that OptiSelect retains its benefits under data rephrasing, a technique used in modern data processing pipelines. Our findings provide theoretical foundations and practical guidance for co-designing optimizers and data selection in LLM pretraining.
Tags
Links
- Source: https://arxiv.org/abs/2610.03432v1
- Canonical: https://arxiv.org/abs/2610.03432v1
Trouble viewing inline? Open PDF directly →
Full Text
160,285 characters extracted from source content.
Expand or collapse full text
OptiSelect: How does the Optimizer Shape Data Curriculum? Simin Fan⋆ Alireza Abdollahpoorrostam⋆ Martin Jaggi Affiliation: EPFL Affiliation: ⋆Equal contribution Abstract Online data selection has demonstrated substantial efficiency gains for LLM pretraining by training on the most valuable candidates within each batch. Since a candidate’s value is realized through its effective model update, principled selection should account for the optimizer step, which reshapes the raw gradient before it updates model parameters. We formalize this optimizer-aware selection paradigm as OptiSelect and present the first systematic study of how the optimizer shapes data selection. Our theory establishes a selection gain principle in which the advantage of online selection is governed by the discriminability of the optimizer-induced utility scores. We prove that sign-based and polar-tangential preconditioners of Lion and Muon would suffer from a discriminability collapse which caps attainable gains from OptiSelect, whereas diagonal-adaptive optimizers such as AdamW and Sophia admit strictly better upper bounds. The proposed principle also yields a quantitative derivation of the optimal candidate oversampling ratio. Pretraining experiments on 124M and 720M models are consistent with our theoretical analysis and show that AdamW’s diagonal-adaptive scoring geometry remains the strongest scoring geometry even with Muon as optimizer. We further demonstrate that OptiSelect retains its benefits under data rephrasing, a technique used in modern data processing pipelines. Our findings provide theoretical foundations and practical guidance for co-designing optimizers and data selection in LLM pretraining. 1 Introduction Data recipes and optimizers are two important levers determining the efficiency of LLM pretraining: they define what the model learns from and how it learns, respectively. These two design choices are often considered independently. Data curation pipelines filter or reweight the corpus before training begins, agnostic to the optimizer that will process it (Xie et al., 2023a; Fan et al., 2024; Li et al., 2024; Penedo et al., 2024; Wang et al., 2024), while optimizer benchmarks compare optimization algorithms on uniformly sampled data streams, agnostic to data selection strategies (Semenov et al., 2025; Wen et al., 2026). The optimizer-aware online data selection strategy (Wang et al., 2026) combines these two considerations, which we formalize as the OptiSelect training paradigm: At every optimization step, each data sample from an oversampled candidate pool is scored by its marginal utility, and only the highest-utility batch is selected for training. As a sample’s utility is realized through the model update it induces, the data utilities should be computed with the optimizer preconditioner, which reshapes the raw gradient before updating model parameters. Therefore, the optimizer plays a dual role under OptiSelect. It descends the loss landscape strategically while also designing the data curriculum, adapting the data utility scoring function according to its unique preconditioner. We analyze the interaction between optimizers and data selection across five optimizers spanning different preconditioning geometries: AdamW rescales gradient coordinates by their estimated RMS (Loshchilov and Hutter, 2019), AdEMAMix mixes in a slow momentum buffer (Pagliardini et al., 2025), Sophia normalizes by estimated curvature (Liu et al., 2024), Lion discards magnitudes and keeps only signs (Chen et al., 2023), and Muon orthogonalizes each update matrix (Jordan, 2024). Our study is organized around one central question from both theoretical and empirical perspectives: How does the optimizer interact with online data selection, and which optimization geometries can actually profit from it? Contributions. We summarize our contributions as three-fold. • The OptiSelect framework (§3). We formalize the OptiSelect training paradigm through an optimizer-aware transformation operator tO_t and derive the corresponding optimizer-aware data utility functions across five optimizers spanning various geometries. • A principled theory of the optimizer–selection interplay (§4). We establish a selection gain principle, demonstrating that the one-step advantage from data selection scales with the discriminability of the utility scores (Theorem 1). We prove that a sign collapse bounds the gain of Lion and a radial collapse bounds that of D-Muon, whereas AdamW and Sophia retain strictly larger score spread. The same principle also derives the optimum candidate oversampling ratio as λ⋆=2λ =2. • A comprehensive pretraining benchmark (§5). We evaluate five optimizers under standard training and OptiSelect on 124M and 720M models, demonstrating a close agreement between measured gains and the theory. Decoupling the scoring geometry from the base optimizer reveals that the optimizer-matched geometry is not necessarily optimal, with AdamW’s diagonal-adaptive operator prevailing under D-Muon. The selection benefit persists under format rephrasing. 2 Related Work We position each work briefly in this section and provide a more detailed survey in Appendix A. Data selection for LLM pretraining. Research on training-data ordering and sample weighting predates LLMs, e.g. easy-to-hard curricula (Bengio et al., 2009), online loss-ranked sampling (Loshchilov and Hutter, 2015), importance sampling with learned gradient-norm proxies (Katharopoulos and Fleuret, 2018) whereas none of these strategies explicitly accounts for the optimizer. At LLM scale, selection is dominated by static pipelines that curate the corpus before training begins: hashed n-gram resampling (DSIR; Xie et al., 2023b), fastText quality classifiers (DCLM; Li et al., 2024), distilled LLM quality judgments (FineWeb-Edu; Penedo et al., 2024), and proxy-model domain weights (DoReMi; Xie et al., 2023a, DoGE; Fan et al., 2024). The online strategies include MATES (Yu et al., 2024), which predicts influence with a periodically refreshed auxiliary model; GREATS (Wang et al., 2024) and OPUS (Wang et al., 2026), which compute gradient-alignment scores and optimizer-aware scores, respectively. Optimizer-aware influence. Influence functions (Koh and Liang, 2017), the first-order influence TracIn (Pruthi et al., 2020) and TRAK with random projections (Park et al., 2023) attributed model behaviour to training samples under implicit SGD dynamics. Adaptive optimizer state changes influence: LESS (Xia et al., 2024) and TrackStar (Chang et al., 2025) correct influence scores for Adam’s second moment; OPUS (Wang et al., 2026) computes optimizer-aware attribution scores on AdamW and Muon, and uses them for per-step sample selection during LLM pretraining. Yet, none compares the influence geometries different optimizers induce, which is the object of our theory (§4). Algorithm 1 OptiSelect: one pretraining step with optimizer-aware data selection 0: Candidate batch ℬ~t B_t, parameters t θ_t, optimizer state, target batch size B, temperature τ 0: Updated parameters t+1 θ_t+1 1: // Candidate forward-backward pass 2: for each z∈ℬ~tz∈ B_t do 3: Compute per-sample gradient gz←∇ℓ(z,t)g_z← _ θ\, (z; θ_t) 4: end for 5: // Influence scoring 6: for each z∈ℬ~tz∈ B_t do 7: Assign utility U(z,t)U(z,t) using gzg_z and the current optimizer state 8: end for 9: // Selection via Boltzmann distribution 10: Define pt(z)∝exp(U(z,t)/τ)p_t(z) \! (U(z,t)/τ ) for z∈ℬ~tz∈ B_t 11: Sample ℬt⊂ℬ~tB_t⊂ B_t with |ℬt|=B|B_t|=B according to ptp_t 12: // Optimizer update 13: t+1←OptimizerStep(t,gzz∈ℬt) θ_t+1← OptimizerStep\! ( θ_t,\,\g_z\_z _t ) 14: return t+1 θ_t+1 Optimizers for LLM pretraining. The families we study span diagonal-adaptive (AdamW; AdEMAMix; Pagliardini et al., 2025), curvature-aware (Sophia; Liu et al., 2024), sign-based (Lion, Signum), and matrix-geometry methods (Muon; Jordan, 2024), with SOAP (Vyas et al., 2025), MARS (Yuan et al., 2025). Systematic benchmarks evaluate these under standard pretraining only (Semenov et al., 2025; Wen et al., 2026), where rankings already shift with scale and batch size. No prior work explores how optimizer algorithms affect preferences in data selection, which we analyze both theoretically (§4) and empirically (§5) in this work. 3 The OptiSelect Paradigm In this section, we first formalize the OptiSelect training loop (§3.1), then detail the optimizer-induced preconditioning that defines the data attribution score for online selection (§3.2). 3.1 Data Schemes in LLM Pretraining Standard pretraining. Let D denote the pretraining data corpus. Standard pretraining minimizes the next-token prediction loss for language modeling, i.e. ℒ()=z∼[ℓ(z,)]L( θ)=E_z [ (z; θ)], where ℓ(z,) (z; θ) is the cross-entropy loss on data instance z. At each step t, a mini-batch ℬt⊂B_t of size B is drawn uniformly at random and used to compute a stochastic gradient ^t=1B∑z∈ℬt∇ℓ(z,t) g_t= 1B _z _t _ θ (z; θ_t). The model parameters are updated as t+1=t−ηt⋅Opt(^t,t) θ_t+1= θ_t- _t·Opt( g_t,S_t), where Opt is the optimizer’s step function corresponding to the current optimizer state tS_t, e.g. momentum buffers and variance estimates. OptiSelect: Online in-batch data selection. While uniform sampling provides no control over data quality, OptiSelect applies online data selection at each optimization step. Specifically, denoting the effective training batch size by B, a larger candidate batch ℬ~t B_t of size |ℬ~t|=λB| B_t|=λ B is drawn at every step t. Each datapoint is ranked by the optimizer-aware influence score U(z,t)U(z,t); the top-B subset ℬt⊂ℬ~tB_t⊂ B_t is then selected for the model update. The full procedure is presented in Algorithm 1. Under OptiSelect the optimizer plays a two-fold role: it performs the optimization step on the loss landscape, and simultaneously shapes data attribution by defining U(z,t)U(z,t). We detail the optimizer-transformed utility under various instantiations in the following section. 3.2 Optimizer-Aware Influence Scoring Preliminaries and notation. We begin by establishing a unified notation for all following derivations. At step t, we denote t∈ℝd θ_t ^d as the model parameters, the per-sample loss on document z as ℓ(z,t) (z; θ_t) and its gradient as z,t≜∇ℓ(z,t)g_z,t\; \; _ θ\, (z; θ_t) and the stochastic mini-batch gradient ^t=1|ℬt|∑z∈ℬtz,t g_t= 1|B_t| _z _tg_z,t. We denote element-wise multiplication, division and square-root operations by ⊙ , ⊘ , and ⋅ · respectively. We define the frozen-state scoring operator tO_t as the map from raw gradients to directions used for per-sample attribution, conditional on the optimizer state tS_t. Given a candidate z, its scoring direction is uz,t=t(gz,t)u_z,t=O_t(g_z,t). This operator need not coincide with the actual optimizer update map Opt(⋅,t)Opt(·,S_t); the relation between them, including the approximations used for scoring, is specified in Appendix B. Optimizer-transformed Data Influence. During online data selection, we assess the influence of a candidate sample z at step t using its per-sample scoring direction z,tu_z,t induced by the optimizer state tS_t and a proxy set from a predefined target distribution. Let V be a small held-out validation set representative of a target distribution. We define the validation loss F()=1||∑z∈ℓ(z,)F( θ)= 1|V| _z (z; θ). Selection proceeds with greedy influence scoring at every step: writing B(t)B^(t) as the selected batch at time step t, the parameter vector predicted by the additive scoring model is tsel≜t−ηt∑z′∈(t)t(z′,t) θ_t^sel θ_t- _t _z ^(t)O_t(g_z ,t). We estimate the marginal utility of adding sample z into B(t)B^(t) as the additional reduction in F: U(z,t)=F(tsel)−F(tsel−ηtt(z,t))U(z,t)\;=\;F\! ( θ_t^sel )-F\! ( θ_t^sel- _t\,O_t(g_z,t) ). Expanding F to second order around t θ_t, denoting ,t=∇F(t)g_V,t= _ θF( θ_t) as the validation gradient proxy and F=∇2F(t)H_F=∇^2F( θ_t) as the validation Hessian, the marginal utility is defined as U(z,t)=ηt⟨t(z,t),,t⟩−ηt2|B(t)|⟨t(z,t),Ft⟩+(ηt2‖t(z,t)‖2)U(z,t)= _t _t(g_z,t),\,g_V,t - _t^2\,|B^(t)| _t(g_z,t),\,H_F\,G_t +O( _t^2 _t(g_z,t) ^2). Following OPUS (Wang et al., 2026), we approximate the intractable d×d× d Hessian by the identity matrix F≈H_F in the redundancy term , giving U(z,t)≈ηt⟨t(z,t),,t⟩−ηt2|B(t)|⟨t(z,t),t⟩,U(z,t)\;≈\; _t\, _t(g_z,t),\,g_V,t \;-\; _t^2\,|B^(t)| _t(g_z,t),\,G_t , (1) where t=1|B(t)|∑z′∈(t)t(z′,t)G_t= 1|B^(t)| _z ^(t)O_t(g_z ,t) is the average scoring direction of the selected samples. The first term measures alignment between sample z and the target distribution, while the second penalizes redundancy among the peer candidates in the same batch. Instantiation of tO_t across various optimizer families. Each optimizer algorithm induces a specific operator tO_t for online data attribution at step t. We write x for the generic argument of tO_t, i.e. the per-sample raw gradient vector =z,t∈ℝdx=g_z,t ^d or the mini-batch gradient =^t∈ℝdx= g_t ^d, where t=t(^t)d_t=O_t( g_t) denotes the batch direction represented by the scoring map. For Muon the argument is matrix-valued, i.e. ∈ℝm×nG ^m× n, acting on each 2-D weight block in the transformer model. The remaining symbols are read from the optimizer state tS_t, which is identical for all candidates within one scoring pass at step t. We present the transformation operators for the benchmarked optimizers, with detailed derivations provided in Appendix B: tAdamW() _t AdamW(x) =tAdEMAMix()=t⊙, =O_t AdEMAMix(x)=c_t , tSophia() _t Sophia(x) =⊘maxγt,ε, =x \γ\,h_t,\, \, tLion() _t Lion(x) =sign(βt−1+(1−β)), =sign (β\,m_t-1+(1-β)\,x ), tLion,lin() _t Lion,\,lin(x) =sign(t−1)⊙, =sign(m_t-1) , tMuon,lin() _t Muon,lin(G) =Dmsign(t)[], =D (A_t)[G], tMuon,rp() _t Muon,rp(G) =(t⊤t+ε)−1/2, =G\,(A_t A_t+ )^-1/2, -5.0pt (2) Specifically, for AdamW/ AdEMAMix (Appendix B.2, B.3), t=⊘(^t+ε)c_t=1 ( v_t+ ) denotes the inverse root of the bias-corrected second moment ^t v_t. AdEMAMix induces the same operator; for Sophia (Appendix B.4), th_t denotes the estimate of diagonal curvature with trust scale γ; for Lion (Appendix B.5), t−1m_t-1 denotes the sign methods’ momentum buffer, with decay β. With β≫1−β 1-β; for Muon (Appendix B.7), tA_t denotes the momentum matrix. Dmsign(t)[⋅]D (A_t)[\,·\,] is the faithful Fréchet derivative in tMuon,linO_t Muon,lin, while tMuon,rpO_t Muon,rp is a cheaper right-preconditioned proxy. 4 What Makes an Optimizer Good at Selecting Data? Since each optimizer shapes the influence function through its own transformation tO_t, a natural question arises: how does the choice of optimizer interact with data selection? To study this question, we introduce discriminability (Theorem 1), a principled criterion that determines an upper bound on the gain of OptiSelect over standard training under a given optimizer. We then analyze several optimization geometries through the lens of this selection gain principle: we find that sign collapse and radial collapse cap the achievable selection gain for the sign-based optimizer Lion and the polar-tangential optimizer D-Muon, respectively, whereas diagonal-adaptive optimizers such as AdamW and Sophia enjoy strictly better upper bounds on the selection gain. Our proof builds on a Gaussian model of the data candidates. We detail the setup as follows. Figure 1: Selection gain (Theorem 1) measured during 124M LLM pretraining. (a) Running medians at tuned η⋆η : the measured gain mostly sits close to cqc_q for every optimizer; while Lion decays after proxy refresh. (b) Measured gain from learning rate sweep: the selection gain is flat in various learning rate η. Setup and Notations. At step t, the candidate batch ℬ~t B_t of size λBλ B is drawn i.i.d. from the data distribution with utility scores follows a Gaussian distribution U(z,t)z∈ℬ~t∼(μU(t),σU2(t))\U(z,t)\_z∈ B_t ( _U(t), _U^2(t)). Consider that candidate gradients are also sampled from a Gaussian distribution z∼(,)g_z ( μ, ), with diagonal covariance =diag(σ12,…,σd2) =diag( _1^2,…, _d^2). Given the validation proxy g_V, we denote ^=−ηtGt g_V=g_V- _tG_t to absorb the redundancy term in (1). Denote π as the diagonal operator applied by tO_t transformation, the utility score on the sample z can be written as Uz=ηt⟨⊙z,^⟩=ηt⟨z,⊙^⟩U_z= _t π _z,\, g_V = _t _z,\, π g_V , i.e. = π=1 for SGD, =t π=c_t (2) for AdamW, and =sign(t−1) π=sign(m_t-1) for linearized sign-based optimizers (Proposition 1). Theorem 1 (Selection-gain Principle). Consider the candidate utilities distribution (μU(t),σU2(t))N( _U(t), _U^2(t)) and standardized top-q superquantile cq>0c_q>011 1 After standardizing the candidate utilities to zero mean and unit variance, the standardized top-q superquantile (cqc_q) refers to the mean utility of the top q=1/λq=1/λ candidates. For Gaussian utilities at λ=2λ=2, cqc_q is the mean of the upper half of a standard normal, 2/π≈0.798 2/π≈ 0.798., where q=1/λq=1/λ is the retained fraction from OptiSelect. The expected one-step reduction on the validation loss F, Rt≜[F(t)−F(t+1)]R_t [F( θ_t)-F( θ_t+1)], satisfies Rtrand=μU(t)+O(ηt2),Rtsel=μU(t)+cqσU(t)+O(ηt2),R_t^rand= _U(t)+O( _t^2), R_t^sel= _U(t)+c_q\, _U(t)+O( _t^2), (3) which gives the selection gain, i.e. the extra loss reduction obtained from data selection as Gt≜Rtsel−Rtrand=cqσU(t)+O(ηt2).G_t\; \;R_t^sel-R_t^rand\;=\;c_q\, _U(t)\;+\;O( _t^2). (4) The proof is given in Appendix C. Since cqc_q depends only on the retained fraction q=1/λq=1/λ, the absolute gain is governed by the score spread σU(t) _U(t). We validate Theorem 1 on 1515 OptiSelect LLM pretraining runs and compute (U¯sel−U¯)/σU( U_sel- U)/ _U on the attribution scores, where the proxy gradient is refreshed at step 5k5k. Figure 1 shows that the normalized score gain (U¯sel−U¯)/σU( U_sel- U)/ _U remains consistently close to cqc_q and nearly invariant across various learning rates. Since cqc_q depends only on the retained fraction q=1/λq=1/λ, the absolute gain is governed by the score spread σU(t) _U(t). We further decompose the variance by coordinates and define the coordinate weights on SGD as wj≜σj2[^]j2∑kσk2[^]k2 w_j\; \; _j^2[ g_V]_j^2 _k _k^2[ g_V]_k^2. In the rest of this section, we adopt the variance ratio Var[Uz()]Var[UzSGD]=∑j=1dwjπj2 Var[U_z^( π)]Var[U_z SGD]= _j=1^dw_j _j^2 to compare the discriminability benefits of linear operators against SGD. Sign Collapse. Sign-based optimizers can lose discriminability since the momentum buffer easily dominates the sign of gradients, diminishing the candidate-aware differences. The selector therefore has little information for ranking them, which approaches a uniform selection strategy. Theorem 2 formalizes this sign collapse by comparing the exact score with the linear surrogate score UzLion,linU_z Lion,lin. Theorem 2 (Sign collapse). Condition on the frozen optimizer state and scoring reference g_V, and let z∼(,)g_z ( μ, ) with =diag(σ12,…,σd2) =diag( _1^2,…, _d^2). (i) (Spread neutrality of the linear surrogate score.) For a sign pattern t≜sign(t−1)∈−1,+1ds_t (m_t-1)∈\-1,+1\^d, UzLion,lin≜ηt⟨z,t⊙^⟩⟹Var[UzLion,lin]=ηt2∑j=1dσj2[^]j2=Var[UzSGD]. -3.00003ptU_z Lion,lin\; \; _t _z,\,s_t g_V [U_z Lion,lin ]= _t^2 _j=1^d _j^2[ g_V]_j^2=Var [U_z SGD ]. (5) (i) (Exponential collapse of the exact score.) Suppose ∃κ≥1∃κ≥ 1 satisfying Assumption 1, the exact Lion score UzLion≜ηt⟨sign(β1t−1+(1−β1)z),^⟩U_z Lion\; \; _t ( _1m_t-1+(1- _1)g_z ),\, g_V satisfies Var[UzLion]≤ 8e−κ2/2ηt2‖^‖22.Var [U_z Lion ]\;≤\;8e^-κ^2/2 _t^2 g_V _2^2. (6) Consequently, when ‖1/2^‖2>0 ^1/2 g_V _2>0 and κ2≥ 2log(8‖^‖22‖1/2^‖22)κ^2\;≥\;2 \! ( 8 g_V _2^2 ^1/2 g_V _2^2 ), we have Var[UzLion]≤Var[UzLion,lin]=Var[UzSGD].Var [U_z Lion ]\;≤\;Var [U_z Lion,lin ]\;=\;Var [U_z SGD ]. (7) Theorem 2 (i) shows that a frozen sign pattern preserves the SGD score variance but cannot adaptively amplify discriminability; Theorem 2 (i) shows that the exact score can degrade this baseline spread under momentum dominance. The full proof are deferred to Appendix D.1. Diagonal-adaptive advantage. Theorem 3 further compares the optimizers along their linear diagonal operators. We define the coordinate weights wj≜σj2[^]j2∑kσk2[^]k2 w_j\; \; _j^2[ g_V]_j^2 _k _k^2[ g_V]_k^2 with variance ratio Var[Uz()]Var[UzSGD]=∑j=1dwjπj2 Var[U_z^( π)]Var[U_z SGD]= _j=1^dw_j _j^2 to compare the benefits of linear operators against SGD. Theorem 3 (Diagonal-adaptive advantage, and the discriminability ranking). Under Assumption 2 and its curvature analogue for Sophia with its hessian estimates, Var[UzAdamW]Var[UzSGD]=∑j=1dwj[t]j2> 1−O(ε′),Var[UzSophia]Var[UzSGD]> 1−O(ε′), Var[U_z AdamW]Var[U_z SGD]\;=\; _j=1^dw_j\,[c_t]_j^2\;>\;1-O( ), Var[U_z Sophia]Var[U_z SGD]\;>\;1-O( ), (8) with ε′,ε′ , the respective numerical floors. Combined with Theorem 2, in the regime κ2≥2log(8‖22/‖1/2‖22)κ^2≥ 2 (8 _V _2^2/ ^1/2g_V _2^2 ) the spreads of the operator classes are ranked: Var[UzLion]⏟exact sign≤Var[UzLion,lin]⏟linearized sign=Var[UzSGD]<Var[UzAdamW],Var[UzSophia]. Var[U_z Lion]_exact sign\;≤\; Var[U_z Lion,lin]_linearized sign\;=\;Var[U_z SGD]\;<\;Var[U_z AdamW],\;Var[U_z Sophia]. The final inequality holding separately for each of the two diagonal-adaptive scores from AdamW and Sophia, which the discriminability does not order against each other. Radial collapse in the ideal polar geometry of Muon. The ideal Muon map removes the stretching of the momentum matrix while retaining its orientation. To see this, write the polar decomposition as t=A_t=QH: the ideal update is =msign(t)Q= (A_t), whereas ≻0H 0 describes the stretching that the map discards. A candidate can therefore change the stretching without turning the update. We call such changes radial; changing singular values while keeping their directions fixed is a simple example. For a frozen full-column-rank momentum tA_t and scoring reference =^R= G_V, decompose each candidate gradient into its radial component and the orthogonal remainder, z=zrad+ztanG_z=G_z^rad+G_z^tan. The latter is its tangential component, which can change the update’s orientation. The precise decomposition is given in Appendix D.3. Let ℒt=Dmsign(t)L_t=D (A_t) denote the local linear response of the ideal map. Theorem 4 (Radial selectivity of the ideal polar score). At the frozen state, suppose zG_z has finite second moment and pairwise uncorrelated entries in the singular-vector coordinates specified in Appendix D.3. For ηt>0 _t>0 and candidate weight α≥0α≥ 0, define UzMuon=ηtα⟨ℒt[z],⟩FU_z Muon= _tα _t[G_z],\,R _F, Uztan=ηt⟨ztan,⟩FU_z^tan= _t _z^tan,\,R _F. Then UzMuonU_z Muon depends on the candidate only through ztanG_z^tan, and Var(UzMuon)≤α2σmin(t)2Var(Uztan),Var(U_z Muon)≤ α^2 _ (A_t)^2Var(U_z^tan), (9) where σmin(t)>0 _ (A_t)>0 is the smallest singular value of the frozen momentum. Thus, two candidates that differ only radially receive the same first-order score. The collapse is selective: radial differences are ignored, while tangential differences are reweighted according to the momentum spectrum. The practical right-preconditioned proxy used in our experiments (2), behaves differently: it rescales the gradient without removing its radial component, and can therefore distinguish candidates that the ideal first-order score treats as identical. Theorem 4 implies neither an unconditional variance ordering against SGD nor a radial-collapse guarantee for the experimental proxy. The exact variance formulas, proof, and proxy discrepancy are given in Appendices D.3 and D.4. 5 Experiments We conduct extensive experiments on LLM pretraining and give the first optimizer benchmarking along the OptiSelect paradigm (§5.1). Notably, we also show that the optimizer-aligned geometry may not be optimal for data selection (§5.2). We further assess the impact of data format on OptiSelect with rephrasing, showing that the benefits remains upon modern data synthesis pipeline (§5.3). Models and training. We train 124M and 720M decoder-only transformers models following Llama architecture (Touvron et al., 2023) adopted in Semenov et al. (2025). We benchmark seven core optimizers, AdamW, AdEMAMix, Sophia, Lion and D-Muon under both standard training and OptiSelect paradigm, incurring a direct comparison of the effectiveness of online data selection. We train Table 1: Data selection benefits across optimizers. Optimizers are measured according to their downstream losses and loss gap Δ=ℒstandard−ℒOptiSelect =L_standard-L_OptiSelect, followed by their ranks. Larger Δ indicates larger data selection gain. 124M 720M Optimizer standard OptiSelect Δ standard OptiSelect Δ AdEMAMix 3.410 (2) 3.274 (2) +0.135+0.135 3.004 (3) 2.908 (2) +0.0960.096 AdamW 3.361 (1) 3.241 (1) +0.119+0.119 2.995 (2) 2.891 (1) +0.1040.104 Sophia 3.486 (5) 3.458 (4) +0.0280.028 3.049 (5) 3.026 (5) +0.0230.023 Lion 3.417 (3) 3.399 (3) +0.0180.018 3.018 (4) 3.022 (4) −0.004 [rgb]1,0,0-0.004 D-Muon 3.455 (4) 3.471 (5) −0.015 [rgb]1,0,0-0.015 2.993 (1) 2.944 (3) +0.0490.049 124M and 720M models for 4.2B and 14B tokens respectively, matching their Chichilla (Hoffmann et al., 2022) optimal token amount. We apply the optimal learning rate tuned from a grid search on both 124M and 720M models, detailed in Appendix E.4. Complete configurations are provided in Appendix E. Data and selection proxy. All the models are trained on FineWeb-HQ22 2 https://huggingface.co/datasets/epfml/FineWeb-HQ, a high-quality, model-filtered pretraining dataset derived as a subset of FineWeb. The selection proxy V is a fixed set of 4,0964,096 densely packed sequences drawn from a downstream task mixture—ARC-Easy, ARC-Challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019), PIQA and SciQ, which serves as a targeted guidance of data selection. The proxy gradient is re-computed every 1,0001,000 steps. The ablation of proxy-refresh interval is presented in Appendix E.5. We report two axes of evaluation: (1) Downstream loss computes the loss of the correct continuation given the prompt on held-out validation sets, averaged across five proxy tasks; (2) Zero-shot accuracy as the exact match rate measured by lm-evaluation-harness (Gao et al., 2024) toolkit. We report the accuracy scores over nine tasks, including the 5 in-domain proxy tasks plus 4 out-of-domain ones WinoGrande, WSC, CommonsenseQA and OpenBookQA, which the proxy never sees. 5.1 The OptiSelect Benchmark: Selection Gain Across Optimizers and Scale Optimizer ranking under OptiSelect. Figure 2 and Table 1 report the downstream losses across optimizers and model scales. The optimizer ranking presented in Table 1 demonstrates a scale-dependent impact from data selection: at 124M, the optimizer ranks on standard training are mostly preserved on OptiSelect; while the ranking differs significantly at 720M. In terms of the downstream accuracy (Figure 3), despite the limited gain from online data selection, D-Muon consistently achieves the best averaged downstream accuracy, benefits from its extraordinary efficacy in optimization. Figure 2: Downstream loss on 124M and 720M models across optimizers. Models are trained on 4.24.2B/14.0314.03B tokens under standard and OptiSelect paradigm, respectively. The intersection of two dotted curves denotes the processed-matched baseline with equal processed tokens under λ=2λ=2. Empirical–theory consistency. According to Figure 2 and Table 1, the empirical selection Figure 3: Downstream Accuracy at 720M scale. OptiSelect outperforms standard training across all optimizers on in-domain and OoD benchmarks. gains remarkably align with our theoretical analysis: OptiSelect consistently improves the downstream loss with AdamW, AdEMAMix and Sophia at both 124M and 720M, whereas obtain no benefits with Lion. D-Muon demonstrates a scale-dependent behavior: at 124M, the selection gain shrinks by the end of training; while at 720M, the benefit remains significant and persists throughout the training. OptiSelect gains in sample efficiency, not compute. Figure 2 and Figure 3 expose a critical reflection: OptiSelect’s gains are measured upon optimizer steps at equal applied tokens while not the gains Figure 4: Scoring geometry ablations under AdamW and D-Muon optimizers. The matched geometry works best under AdamW, while both AdamW and SGD alternatives beat the matched scoring geometry on D-Muon. per token read. At λ=2λ=2, every selection step reads twice the tokens it applies, each with a full forward–backward pass, therefore a fair comparison with equal read-tokens requires OptiSelect achieveing a 2×2× acceleration in optimization steps. However, comparing with standard training with matched read tokens, OptiSelect falls behind in both downstream loss and accuracy across all five optimizers. Given the deficit shrinks at larger scale, OptiSelect is still promising on scaled-up experiments. 5.2 Which Scoring Geometry Should an Optimizer Use? The inconsistency of optimizer ranking between standard training and OptiSelect indicates a dichotomy of the preconditioning geometry for optimization and data selection: The geometry from the best-performed optimizer, D-Muon, is not the optimal for data selection. Inspired from the observation, we decouple the choice of optimizer and the geometry adopted for attribution scoring in (1). We run the scoring geometry ablation under two base optimizers AdamW and D-Muon (Figure 4). Each optimizer is combined with various tO_t, i.e. the geometry, from each of the five optimizers. The SGD scoring geometry recovers the GREATS algorithm from Wang et al. (2024). The optimizer-matched scoring geometry is not necessarily the best. Under AdamW, the best scoring geometry is also from AdamW which matches the optimization dynamics. However, with D-Muon as base optimizer, the pairing geometry does not work well in data selection, while the geometry of AdamW outperforms the others by a significant margin. Notably, the second-momentum diagonal-adaptive operator from AdamW serves as the best scoring geometry in both AdamW and D-Muon sweeps; and scoring without preconditioner (SGD) beats most of the scoring-geometry. Additionally, AdamW with the curvature-diagonal and sign-based geometries from Sophia and Lion can hardly outperform the standard training baseline with no selection. It indicates that a wrong geometry can hurt the performance of online data selection. Figure 5: Downstream loss curves across rephrased mixtures, where the benefits from selection are consistent. 5.3 Does Selection Help with Rephrasing? Selection and format rephrasing are two techniques on data quality, while to what extend they are collaborative or conflict is unexplored. FinePhrase (HuggingFaceFW, 2025) rewrites documents from FineWeb-edu (Lozhkov et al., 2024) into various formats, while leaving its content largely intact. We train AdamW at 124M on mixtures containing 0%0\%, 50%50\% and 100%100\% rephrased documents with consistent setup as in §5. The selection benefit remains under data rephrasing. According to Figure 5, OptiSelect consistently improves the downstream loss by a large margin comparing to the standard training. OptiSelect also outperforms standard training across rephrasing mixtures on the average downstream accuracy (Figure 6, Table 10). It indicates that rephrasing and selection are not substitutes or conflict against each other. Format mismatching is a confound for downstream accuracy, but it depends on evaluation harness pattern. While the selection gets similar amount of gains on downstream loss along various rephrasing mixtures, the benefits on downstream accuracies move significantly along the formatting Figure 6: Task accuracy on format rephrasing × selection at 124M. (a) Accuracy against the rephrased mixture rates. (b) The selection gain at each mixture. axis. Specifically, we separate the downstream tasks into QA- and completion-formatted categories according to the prompting format adopted in the evaluation harness (Appendix G.1). Under standard pretraining, the mixture with 50%50\% rephrased documents achieves the best downstream loss and accuracy scores in both QA- and completion-formatted tasks, aligned with the common practice in (Maini et al., 2024) which samples real and synthetic data in a 1:1 ratio. In comparison, OptiSelect greatly augments the gain on QA-formatted accuracies, reaches 52.51%52.51\% p. On completion-formatted tasks, OptiSelect obtains a great improvement (+2%+2\% p.) on the original FineWeb corpus, while increasing rephrasing proportions does not bring more benefits. 5.4 More Candidates, Better Performance? The candidate multiplier λ is a critical hyperparameter for data selection, which determines the proportion of selected candidates. Instead of introducing as a magical hyperparameter settled in prior works (Wang et al., 2024; Wang et al., 2026), we quantify the effect of λ under the selection gain principle (Theorem 1). While the per-step gain cqσUc_q\, _U with q=1/λq=1/λ is monotonically increasing by λ, the overhead on the streamed and scored tokens grow correspondingly. Lemma 1 prices this trade-off and derives an optimal candidate multiplier λ⋆λ in closed form. Lemma 1 (Optimal candidate multiplier). Under the conditions of Theorem 1 with Gaussian candidate utilities, the selection gain per streamed token is cq(λ)λ=ϕ(Φ−1(1−1/λ)), c_q(λ)λ\;=\;φ ( ^-1(1-1/λ) ), (10) which attains its unique maximum at λ⋆=2λ =2, of value ϕ(0)=1/2πφ(0)=1/ 2π. Lemma 1proves that the selection gain grows as 2lnλ 2 λ, monotonically but not linearly. We provide the full proof in D.5. Therefore, the increment of selection gain cannot compensate for its extra costs when it past the peak value at λ⋆=2λ =2. A token-limited budget should stream exactly Figure 7: The candidate-multiplier sweep at 124M. (a) Downstream loss curves. (b) Selection gain over the standard. Open circles mark the theoretical values of cqσUc_q _U from Theorem 1; the dark-red curve draws the gain per streamed token Δ/λ /λ on the right axis, which peaks at λ⋆=2λ =2. twice what it trains on, which is the default used everywhere else in this paper. In Figure 7, we empirically measure the raw selection gain Δ as the downstream loss reduction above the standard training and the gain per streamed token as Δ/λ /λ, which validates the theory. Across various values of λ, the empirical raw selection gain Δ agrees with the theoretically computed values (Figure 7b), which keeps rising according to λ. In comparison, the measured Δ/λ /λ demonstrates a concave curve, peaks at λ⋆=2λ =2 while decreasing after the optimum. 6 Discussion and Limitations The optimizer determines the landscapes of curriculum. Under OptiSelect the optimizer is not merely an optimization algorithm, it also serves as the designer of the curriculum. The variance normalization of AdamW builds a curriculum biased toward candidates whose gradients are large along rare dimensions, i.e. coordinates with small [^t]j[ v_t]_j are amplified; Muon’s matrix-sign geometry builds a polar-tangent curriculum, which ignores candidates that only change singular values in the current polar basis and highlights those rotate singular subspaces. How to design an optimization algorithm that adapts to the data curriculum is a critical problem for future study. What selection buys, and how it must be reported. Under a downstream proxy, OptiSelect consistently improves the metrics it targets as proxy while may not necessarily benefit for held-out tasks. According to Figure 3 (b), OptiSelect obtains benefits across 3 out of 4 Out-of-Domain benchmarks, while OptiSelect yields little gains on Commonsense QA task. Besides, selection is ahead at equal applied tokens and behind at equal tokens read, at 124M and at 720M alike: it converts extra forward–backward compute into fewer optimizer steps for the same quality, not into fewer tokens consumed. We report both matchings throughout and recommend that future evaluations of in-batch selection do the same; either number quoted alone misstates the method in one direction. Limitations and future work. Our benchmark covers only 124M and 720M models, which can only predict the behaviour at multi-billion scale by extrapolation. Besides, we only conducted each run on a single random seed without claiming the variance across several seeds, mainly constrained from the high computation costs from pretraining. The scope of the proxy target is also limited to the general reasoning tasks, which are within the capability of small-scale models. We leave the exploration on more challenging domains, including Math, Code, and Agentic tasks for the future works. 7 Conclusion In this work, we formalized OptiSelect as the training paradigm with optimizer-dependent online in-batch selection. We provide both theoretical analysis and empirical benchmarking on how various optimization geometries can affect the preference in data selection. Our findings on OptiSelect provide theoretical foundations and empirical insights for the next generation of pretraining optimizers, co-designed with the modern data deployment. AI Disclosure In this work, we USED generative AI tools for paper polishing and proof writing assistance, analyzing existing literature and coding assistance. We have NOT used generative AI tools for formulating research or proof ideas, developing theoretical models or conceptual frameworks and synthetic data generation, synthetic evaluation are not applicable to this work. We have reviewed all AI-assisted work. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. Reproducibility Statement Appendix B specifies the optimizer update rules and scoring operators, including implementation approximations. Appendices C–D provide the theoretical assumptions and proofs. Appendix E documents model architectures, training configurations, selection proxies, and hyperparameter sweeps. Appendices F–G report detailed experimental results. We will open-source our code at https://anonymous.4open.science/r/OptiSelect-ICLR27-640B/ for reproducibility. Ethics statement This work studies the interleaving dynamics between optimizers and data selection using public available datasets and evaluation protocols. We identify no additional ethical concerns specific to this study. References Anil et al. (2020) R. Anil, V. Gupta, T. Koren, K. Regan, and Y. Singer Scalable second order optimization for deep learning. Note: arXiv preprint arXiv:2002.09018 Cited by: §A.3. Bengio et al. (2009) Y. Bengio, J. Louradour, R. Collobert, and J. Weston Curriculum learning. In International Conference On Machine Learning, p. 41–48. Cited by: §A.1, §2. Bernstein and Newhouse (2024) J. Bernstein and L. Newhouse Old optimizer, new norm: An anthology. Note: arXiv preprint arXiv:2409.20325 Cited by: §A.3. Bernstein et al. (2018) J. Bernstein, Y. Wang, K. Azizzadenesheli, and A. Anandkumar signSGD: Compressed optimisation for non-convex problems. In International Conference On Machine Learning, p. 560–569. Cited by: §A.3, Algorithm 7. Chang et al. (2025) T. A. Chang, D. Rajagopal, T. Bolukbasi, L. Dixon, and I. Tenney Scalable influence and fact tracing for large language model pretraining. In International Conference On Learning Representations, Cited by: §A.2, §2. Chen et al. (2023) X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C. Hsieh, Y. Lu, and Q. V. Le Symbolic discovery of optimization algorithms. In Advances In Neural Information Processing Systems, p. 49205–49233. Cited by: §A.3, §1, Algorithm 6. Choe et al. (2025) S. K. Choe, H. Ahn, J. Bae, K. Zhao, M. Kang, Y. Chung, A. Pratapa, W. Neiswanger, E. Strubell, T. Mitamura, J. Schneider, E. Hovy, R. Grosse, and E. Xing What is your data worth to GPT? LLM-scale data valuation with influence functions. In Advances In Neural Information Processing Systems, p. 162015–162056. Cited by: §A.2. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? Try ARC, the AI2 reasoning challenge. Note: arXiv preprint arXiv:1803.05457 Cited by: §5. Fan et al. (2024) S. Fan, M. Pagliardini, and M. Jaggi DoGE: Domain reweighting with generalization estimation. In International Conference On Machine Learning, p. 12895–12915. Cited by: §1, §2. Gao et al. (2024) L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou The language model evaluation harness. Note: ZenodoVersion v0.4.3 Cited by: §5. Gupta et al. (2018) V. Gupta, T. Koren, and Y. Singer Shampoo: Preconditioned stochastic tensor optimization. In International Conference On Machine Learning, p. 1842–1850. Cited by: §A.3. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L. Sifre Training compute-optimal large language models. In Advances In Neural Information Processing Systems, p. 30016–30030. Cited by: §5. HuggingFaceFW (2025) HuggingFaceFW FinePhrase. Note: Hugging Face dataset Cited by: §A.1, §5.3. Jordan (2024) K. Jordan Muon: An optimizer for hidden layers in neural networks. Note: Blog post Cited by: §A.3, §1, §2, Algorithm 8. Katharopoulos and Fleuret (2018) A. Katharopoulos and F. Fleuret Not all samples are created equal: Deep learning with importance sampling. In International Conference On Machine Learning, p. 2525–2534. Cited by: §A.1, §2. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: A method for stochastic optimization. In International Conference On Learning Representations, Cited by: §A.3, Algorithm 3. Koh and Liang (2017) P. W. Koh and P. Liang Understanding black-box predictions via influence functions. In International Conference On Machine Learning, p. 1885–1894. Cited by: §A.2, §2. Li et al. (2024) J. Li, A. Fang, G. Smyrnis, M. Ivgi, M. Jordan, S. Gadre, H. Bansal, E. Guha, S. Keh, K. Arora, et al. DataComp-LM: In search of the next generation of training sets for language models. In Advances In Neural Information Processing Systems, p. 14200–14282. Cited by: §A.1, §1, §2. Li et al. (2026) J. Li, Y. Fan, and H. Li Variance-adaptive Muon: Accelerating LLM pretraining with NSR-modulated and variance-scaled momentum. Note: arXiv preprint arXiv:2601.14603 Cited by: §A.3. Liu et al. (2024) H. Liu, Z. Li, D. Hall, P. Liang, and T. Ma Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International Conference On Learning Representations, Cited by: §A.3, §1, §2, Algorithm 5. Loshchilov and Hutter (2015) I. Loshchilov and F. Hutter Online batch selection for faster training of neural networks. Note: arXiv preprint arXiv:1511.06343 Cited by: §A.1, §2. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference On Learning Representations, Cited by: §A.3, §1, Algorithm 3. Lozhkov et al. (2024) A. Lozhkov, L. Ben Allal, L. von Werra, and T. Wolf FineWeb-Edu: The finest collection of educational content. Note: Hugging Face dataset Cited by: §5.3. Maini et al. (2024) P. Maini, S. Seto, R. Bai, D. Grangier, Y. Zhang, and N. Jaitly Rephrasing the web: A recipe for compute and data-efficient language modeling. In Annual Meeting Of The Association For Computational Linguistics, p. 14044–14072. Cited by: §5.3. Pagliardini et al. (2025) M. Pagliardini, P. Ablin, and D. Grangier The AdEMAMix optimizer: Better, faster, older. In International Conference On Learning Representations, Cited by: §A.3, §1, §2, Algorithm 4. Park et al. (2023) S. M. Park, K. Georgiev, A. Ilyas, G. Leclerc, and A. Mądry TRAK: Attributing model behavior at scale. In International Conference On Machine Learning, p. 27074–27113. Cited by: §A.2, §2. Penedo et al. (2024) G. Penedo, H. Kydlíček, L. Ben Allal, A. Lozhkov, M. Mitchell, C. Raffel, L. von Werra, and T. Wolf The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances In Neural Information Processing Systems, p. 30811–30849. Cited by: §A.1, §1, §2. Pruthi et al. (2020) G. Pruthi, F. Liu, S. Kale, and M. Sundararajan Estimating training data influence by tracing gradient descent. In Advances In Neural Information Processing Systems, p. 19920–19930. Cited by: §A.2, §2. Semenov et al. (2025) A. Semenov, M. Pagliardini, and M. Jaggi Benchmarking optimizers for large language model pretraining. Note: arXiv preprint arXiv:2509.01440 Cited by: §A.3, §A.3, §1, §2, §5, Algorithm 8. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample LLaMA: Open and efficient foundation language models. Note: arXiv preprint arXiv:2302.13971 Cited by: §5. Vyas et al. (2025) N. Vyas, D. Morwani, R. Zhao, I. Shapira, D. Brandfonbrener, L. Janson, and S. Kakade SOAP: Improving and stabilizing Shampoo using Adam for language modeling. In International Conference On Learning Representations, Cited by: §A.3, §2. Wang et al. (2025) J. T. Wang, P. Mittal, D. Song, and R. Jia Data Shapley in one training run. In International Conference On Learning Representations, Cited by: §A.2. Wang et al. (2024) J. T. Wang, T. Wu, D. Song, P. Mittal, and R. Jia GREATS: Online selection of high-quality data for LLM training in every iteration. In Advances In Neural Information Processing Systems, p. 131197–131223. Cited by: §A.1, §1, §2, §5.2, §5.4. Wang et al. (2026) S. Wang, X. Ouyang, T. Xu, Y. Hu, J. Liu, G. Chen, T. Zhang, J. Zheng, K. Yang, X. Ren, D. Liu, and L. Zhang OPUS: Towards efficient and principled data selection in large language model pre-training in every iteration. In International Conference On Machine Learning, Cited by: §A.1, §1, §2, §2, §3.2, §5.4, Remark 4. Wen et al. (2026) K. Wen, D. L. W. Hall, T. Ma, and P. Liang Fantastic pretraining optimizers and where to find them. In International Conference On Learning Representations, Cited by: §A.3, §1, §2. Xia et al. (2024) M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen LESS: Selecting influential data for targeted instruction tuning. In International Conference On Machine Learning, p. 54104–54132. Cited by: §A.2, §2. Xie et al. (2023a) S. M. Xie, H. Pham, X. Dong, N. Du, H. Liu, Y. Lu, P. S. Liang, Q. V. Le, T. Ma, and A. W. Yu DoReMi: Optimizing data mixtures speeds up language model pretraining. In Advances In Neural Information Processing Systems, p. 69798–69818. Cited by: §A.1, §1, §2. Xie et al. (2023b) S. M. Xie, S. Santurkar, T. Ma, and P. Liang Data selection for language models via importance resampling. In Advances In Neural Information Processing Systems, p. 34201–34227. Cited by: §A.1, §2. Yu et al. (2024) Z. Yu, S. Das, and C. Xiong MATES: Model-aware data selection for efficient pretraining with data influence models. In Advances In Neural Information Processing Systems, p. 108735–108759. Cited by: §A.1, §2. Yuan et al. (2025) H. Yuan, Y. Liu, S. Wu, X. Zhou, and Q. Gu MARS: Unleashing the power of variance reduction for training large models. In International Conference On Machine Learning, p. 73553–73587. Cited by: §2. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: Can a machine really finish your sentence?. In Annual Meeting Of The Association For Computational Linguistics, p. 4791–4800. Cited by: §5. Appendix Contents 1 Introduction 2 Related Work 3 The OptiSelect Paradigm 3.1 Data Schemes in LLM Pretraining 3.2 Optimizer-Aware Influence Scoring 4 What Makes an Optimizer Good at Selecting Data? 5 Experiments 5.1 The OptiSelect Benchmark: Selection Gain Across Optimizers and Scale 5.2 Which Scoring Geometry Should an Optimizer Use? 5.3 Does Selection Help with Rephrasing? 5.4 More Candidates, Better Performance? 6 Discussion and Limitations 7 Conclusion References A Extended Related Work A.1 Online Data Selection and Curriculum Learning A.2 Influence Functions and Data Attribution A.3 Modern Optimizers for LLM Pretraining B Benchmarked Optimizers: Update Rules and Frozen-State Operators B.1 SGD with Momentum B.2 AdamW B.3 AdEMAMix B.4 Sophia B.5 Lion B.6 Signum B.7 D-Muon B.8 Classification of the Frozen-State Operators C The Selection-Gain Principle: Proof and Empirical Evidence C.1 Proof of D Detailed Proofs for Discriminability (§) D.1 Sign Collapse (Theorem ) D.2 Diagonal-Adaptive Advantage of AdamW and Sophia () D.3 Radial Collapse of Muon () D.3.1 Preliminaries: the ideal polar map D.3.2 Polar derivative and radial geometry D.3.3 Score variance and proof of D.4 Right-Preconditioned Muon Proxy and Exact Error D.5 Optimal Candidate Multiplier () E Experiment Configurations E.1 Model Architecture E.2 Training Arguments E.3 Selection Proxy Details E.4 Learning Rate Sweep E.5 Proxy Refresh Interval F Extended Experimental Results F.1 Complete scores at 124M F.2 Complete scores at 720M G Data Selection under Rephrasing G.1 Details on the Downstream Tasks G.2 Extended Results on Rephrasing Appendix A Extended Related Work This section gives a detailed description of related work mentioned in §2. The organizing claim is unchanged: existing work either selects data while treating the optimizer as given, or studies optimizers while treating the data stream as given, and the two literatures barely cite each other. OptiSelect sits at their intersection. A.1 Online Data Selection and Curriculum Learning The idea that training order and sample weighting matter predates deep learning. Bengio et al. (2009) proposed easy-to-hard curricula for neural network training; Loshchilov and Hutter (2015) ranked samples by their current loss and sampled with exponential decay, reporting a 5×5× speedup on MNIST; Katharopoulos and Fleuret (2018) formalized the problem as importance sampling, training an auxiliary model to predict upper bounds on per-sample gradient norms. These methods score samples by scalar losses or raw gradient norms; none accounts for the transformation the optimizer applies to a gradient before it becomes an update, which is the object this paper studies. For LLM-scale pretraining, data selection has largely been static: the corpus is curated once, before training, by a criterion that never sees the model state. DSIR (Xie et al., 2023b) performs importance resampling in a hashed n-gram feature space to match a target distribution; DCLM (Li et al., 2024) trains fastText classifiers for quality filtering and shows that classifier choice dominates other curation decisions at scale; FineWeb-Edu (Penedo et al., 2024) distills LLM quality judgments into a lightweight scorer (the FineWeb family also supplies FineWeb-HQ, the corpus our benchmark streams); DoReMi (Xie et al., 2023a) optimizes domain mixture weights with a small proxy model and group-DRO objective. Static filters are cheap at training time and composable with any optimizer, but they cannot adapt to what the model currently knows, and they answer a different question from ours: what to train on, rather than what next. A complementary static axis is rewriting rather than filtering: FinePhrase (HuggingFaceFW, 2025) rephrases FineWeb documents into cleaner prose, changing the format of the text while preserving content. Section 5.3 shows this axis composes with online selection rather than substituting for it: the selection gain is essentially unchanged at 0%0\%, 50%50\% and 100%100\% rephrased shares. Online selection interleaves scoring with training. MATES (Yu et al., 2024) is the closest precursor to our setting at pretraining scale: it performs influence-guided selection during pretraining, but predicts influence with a small auxiliary model refreshed periodically, so the scores reflect the optimizer only indirectly through the checkpoints the auxiliary model is fitted to. GREATS (Wang et al., 2024) selects greedily within each batch using exact per-sample gradient inner products made affordable by the Ghost trick, at fine-tuning scale. OPUS (Wang et al., 2026) is the most principled online approach and the direct foundation of our benchmark: it computes optimizer-aware influence scores from per-sample gradients at every step and reports up to 2×2× data efficiency at λ=2λ=2. OPUS, however, instantiates and validates the paradigm for AdamW and Muon only, and offers little analysis of how the operator shapes selection; the paradigm’s defining component—the optimizer—is exactly the axis it leaves unexplored, and the axis along which this paper varies everything. A.2 Influence Functions and Data Attribution Classical influence functions (Koh and Liang, 2017) estimate the effect of removing training point z on the loss at a test point z′z as −∇ℒ(z′)⊤H−1∇ℒ(z)- _ θL(z ) H^-1 _ θL(z), requiring an inverse-Hessian-vector product that is intractable at LLM scale and fragile in deep non-convex models. A line of work replaces the Hessian with tractable surrogates. TracIn (Pruthi et al., 2020) approximates influence by gradient dot products summed over checkpoints, which is exact for SGD dynamics and explicitly ignores optimizer preconditioners. TRAK (Park et al., 2023) scales attribution to billion-parameter models by random projection and kernel linearization. Data Shapley valuations can now be computed in a single training run (Wang et al., 2025), and LoGra (Choe et al., 2025) makes influence queries affordable at GPT scale by projecting gradients onto low-rank subspaces during training. Most relevant to us is the recent recognition that the optimizer’s adaptive state fundamentally changes attribution. LESS (Xia et al., 2024) extends TracIn to account for Adam’s second moment when selecting instruction-tuning data, finding that Adam-aware selection consistently outperforms SGD-aware selection under an Adam-trained model. TrackStar (Chang et al., 2025) divides per-sample gradients by Adam’s ^t v_t before scoring, reporting that uncorrected scores preferentially retrieve long, high-magnitude examples rather than genuinely influential ones. These works treat a single optimizer’s correction as a fix to an attribution pipeline. Our frozen-state operator tO_t (§3.2) generalizes that move into a formalism: every optimizer induces its own influence geometry, the induced geometries differ qualitatively (diagonal, sign, polar-tangent), and the differences are large enough to reorder which optimizers benefit from selection at all (§5). Attribution also differs from our setting in when it acts: it explains a trained model’s behaviour retrospectively, whereas OptiSelect scores candidates prospectively, before the update they would cause, with the optimizer state frozen at the current step. A.3 Modern Optimizers for LLM Pretraining The families. Adam (Kingma and Ba, 2015) with decoupled weight decay (Loshchilov and Hutter, 2019) remains the default for LLM pretraining. AdEMAMix (Pagliardini et al., 2025) adds a third, much slower momentum buffer (β3≈0.9999 _3≈ 0.9999, mixed in with weight α), on the observation that gradients stay informative for tens of thousands of steps. Sophia (Liu et al., 2024) maintains a diagonal Hessian estimate via Hutchinson or Gauss–Newton–Bartlett probes and clips the preconditioned update coordinate-wise. The sign family descends from SignSGD and its momentum variant Signum (Bernstein et al., 2018), motivated by communication compression; Lion (Chen et al., 2023), discovered by program search, interpolates momentum and gradient inside the sign and stores the buffer with a separate decay. Muon (Jordan, 2024) applies Newton–Schulz orthogonalization to the momentum matrix of each 2-D parameter, approximating the matrix-sign/polar-factor update; the D-Muon variant we benchmark follows Semenov et al. (2025) in handling 1-D parameters with AdamW, and refinements continue to appear (Li et al., 2026). The Kronecker-factored line runs from Shampoo (Gupta et al., 2018) through its scalable variants (Anil et al., 2020) to SOAP (Vyas et al., 2025), which runs Adam in Shampoo’s eigenbasis. Closest in spirit to our operator view, Bernstein and Newhouse (2024) read modern optimizers as steepest descent under different norms; our frozen-state operator is the same structural move aimed at a different question—not what update geometry an optimizer imposes, but what influence geometry it induces on data. Benchmarks. Semenov et al. (2025) benchmark eleven optimizers (AdamW, AdEMAMix, MARS, D-Muon, SOAP, ADOPT, Prodigy, Sophia, Lion, Signum, SF-AdamW) on Llama-style models up to 720M parameters on FineWeb—the recipe our setup follows for comparability—finding that AdEMAMix dominates at larger scale, sign-based methods degrade with model size, SOAP wins short horizons but not long ones, and Sophia diverges in small-batch regimes. The concurrent study of Wen et al. (2026) partially disputes Muon’s ranking and finds per-component learning-rate tuning critical. Both benchmarks train on uniformly sampled batches; neither considers any form of data selection. Our results give a concrete reason this omission matters: the benefit an optimizer draws from selection is an ordering of its own, stable across scale and unpredicted by standard-training quality (§5)—an optimizer chosen from these leaderboards may be the wrong choice for a selection pipeline. Appendix B Benchmarked Optimizers: Update Rules and Frozen-State Operators This section specifies the optimizer update rules and the frozen-state operators used for data selection. For each optimizer discussed below, we provide (i) pseudocode for the update rule, (i) the corresponding frozen-state scoring operator tO_t, and (i) the approximations and normalization conventions used for scoring. We distinguish the implemented scoring maps from the exact nonlinear maps and derivative-based operators considered in the analysis. Derivation conventions. We use the following conventions: (S1) Freeze the scoring state. Hold the reference state tS_t fixed across candidates at step t, and write x for the gradient argument. Where scoring also freezes a quantity that the optimizer normally updates using the incoming gradient, we state that approximation explicitly. (S2) Separate candidate-independent additive terms. When the frozen update is affine in x, separate its constant component from the candidate-dependent direction. An additive constant in a completed scalar score does not affect rankings or Boltzmann probabilities. Terms inside nonlinear maps cannot generally be removed this way; the scoring conventions for those maps are specified below. (S3) Choose a linear scoring map. A nonlinear batch update generally cannot be decomposed into independent per-sample directions. We therefore distinguish derivative-based linearizations from linear surrogates: the analysis considers the ideal polar derivative for Muon, while the experiments use a right-preconditioned Muon proxy and fixed-sign surrogates for Lion/Signum. Exact additivity of these scoring maps does not imply exact attribution under the full optimizer update. Multiplying a completed score by a common positive scalar preserves hard rankings but rescales its variance and changes raw-score Boltzmann sampling at fixed temperature. Rescaling tO_t need not preserve the rankings induced by (1): when tG_t is rescaled consistently, the alignment term scales linearly and the redundancy term quadratically. The omitted prefactors therefore define the scoring convention rather than an invariance of the full utility or optimizer update. B.1 SGD with Momentum Algorithm 2 SGD with heavy-ball momentum, one step. 0: learning rate ηt _t; momentum μ; first-order momentum buffer b (init 0) 1: t←μt−1+^tb_t←μ\,b_t-1+ g_t 2: t+1←t−ηtt θ_t+1← θ_t- _t\,b_t Derivation. Substituting x for ^t g_t (S1), the update direction is ()=μt−1+d(x)=μ\,b_t-1+x. The frozen buffer term μt−1μ\,b_t-1 is candidate-independent and drops (S2), leaving the identity map (S3): tSGD()=.O_t SGD(x)\;=\;x. (11) SGD scoring operator. tSGD()=O_t SGD(x)=x: scoring with raw gradients. With the momentum state fixed, the update direction is affine in the gradient. After removing the constant momentum term, the candidate-dependent map is exactly linear. B.2 AdamW Algorithm 3 AdamW (Kingma and Ba, 2015; Loshchilov and Hutter, 2019), one step. 0: learning rate ηt _t; decays (β1,β2)( _1, _2); ε ; decoupled weight decay λ; state with first- and second-moment estimates (,)(m,v) (init 0) 1: t←β1t−1+(1−β1)^tm_t← _1\,m_t-1+(1- _1)\, g_t 2: t←β2t−1+(1−β2)^t⊙2v_t← _2\,v_t-1+(1- _2)\, g_t 2 3: ^t←t/(1−β1t) m_t _t/(1- _1^t); ^t←t/(1−β2t) v_t _t/(1- _2^t) 4: t+1←(1−ηtλ)t−ηt^t⊘(^t+ε) θ_t+1←(1- _tλ)\, θ_t- _t\, m_t ( v_t+ ) Derivation. For scoring, freeze t−1m_t-1 and the reference second-moment estimate ^t v_t across candidates. This suppresses the dependence of the denominator on the candidate gradient in the full AdamW update. Under this approximation, substituting x gives ()=1−β11−β1t⊘(^t+ε)⏟candidate-dependent+β11−β1tt−1⊘(^t+ε)+λt⏟constant for all candidates; drops by S2.d(x)\;=\; 1- _11- _1^t\,x ( v_t+ )_candidate-dependent\;+\; _11- _1^t\,m_t-1 ( v_t+ )\;+\;λ\, θ_t_constant for all candidates; drops by S2. After removing the frozen additive terms, we set the current-gradient coefficient (1−β1)/(1−β1t)(1- _1)/(1- _1^t) to one as a scoring normalization, obtaining the diagonal map below. AdamW scoring operator. The scoring operator is linear, with exact per-sample decomposition under the frozen-denominator, unit-prefactor convention. tAdamW()=t⊙O_t AdamW(x)=c_t with t=⊘(^t+ε)c_t=1 ( v_t+ ): inverse-square-root second-moment rescaling, assigning larger weights to coordinates with smaller estimated second moments. B.3 AdEMAMix Algorithm 4 AdEMAMix (Pagliardini et al., 2025), one step. The runs use the linear α- and β3 _3-warmup schedulers of the original paper. 0: learning rate ηt _t; decays (β1,β2,β3)( _1, _2, _3); slow-buffer weight α; ε ; weight decay λ; state ((1),(2),)(m^(1),m^(2),v) 1: t(1)←β1t−1(1)+(1−β1)^tm^(1)_t← _1\,m^(1)_t-1+(1- _1)\, g_t fast buffer, bias-corrected below 2: t(2)←β3t−1(2)+(1−β3)^tm^(2)_t← _3\,m^(2)_t-1+(1- _3)\, g_t slow buffer, uncorrected 3: t←β2t−1+(1−β2)^t⊙2v_t← _2\,v_t-1+(1- _2)\, g_t 2; ^t(1)←t(1)/(1−β1t) m^(1)_t ^(1)_t/(1- _1^t); ^t←t/(1−β2t) v_t _t/(1- _2^t) 4: t+1←(1−ηtλ)t−ηt(^t(1)+αt(2))⊘(^t+ε) θ_t+1←(1- _tλ)\, θ_t- _t\, ( m^(1)_t+α\,m^(2)_t ) ( v_t+ ) Derivation. Under the same frozen-denominator convention, the candidate enters both the fast and slow buffers, yielding the update direction: ()=(1−β11−β1t+α(1−β3))⊘(^t+ε)+frozen terms,d(x)\;=\; ( 1- _11- _1^t+α\,(1- _3) )\,x ( v_t+ )\;+\;frozen terms, After removing the frozen additive terms and setting the displayed current-gradient coefficient to one, we obtain the same diagonal scoring form as AdamW, evaluated at AdEMAMix ’s frozen second moment. AdEMAMix scoring operator. tAdEMAMix()=t⊙O_t AdEMAMix(x)=c_t , with the same functional form as AdamW under the stated normalization. The two methods need not produce identical scores: their update dynamics can lead to different parameter trajectories and second-moment estimates. B.4 Sophia Algorithm 5 Sophia-G (Liu et al., 2024), one step. BHB_H is the batch size used by the Hessian estimator; the trust scale is γ=ρBHγ=ρ\,B_H. 0: learning rate ηt _t; decays (β1,βh)( _1, _h); trust parameter ρ; floor ε ; Hessian interval kHk_H; weight decay λ; state (,)(m,h) 1: if t≡0(modkH)t≡ 0 k_H then 2: draw y^∼pt(⋅∣x) y p_ θ_t(· x) for the mini-batch; s←∇ℓ(x,y^,t)g_s← _ θ\, (x, y; θ_t) 3: ^←BHs⊙s h← B_H\,g_s _s Gauss–Newton–Bartlett estimate of diag()diag(H) 4: t←βht−kH+(1−βh)^h_t← _h\,h_t-k_H+(1- _h)\, h 5: end if 6: t←β1t−1+(1−β1)^tm_t← _1\,m_t-1+(1- _1)\, g_t 7: t+1←(1−ηtλ)t−ηtclip(t⊘maxγt,ε, 1) θ_t+1←(1- _tλ)\, θ_t- _t\,clip (m_t \γ\,h_t,\, \,\,1 ) coordinate-wise clip at magnitude 11 Sophia estimates curvature from training gradients sg_s, ^=BHs⊙s(GNB; the Hutchinson variant uses ⊙(),∼Unif±1d), h\;=\;B_H\,g_s _s (GNB; the Hutchinson variant uses u (Hu),\ u \± 1\^d), (12) t=βht−kH+(1−βh)^. _t\;=\; _h\,h_t-k_H+(1- _h)\, h. Derivation. Freeze th_t and substitute x into the momentum (S1): ()=clip([β1t−1+(1−β1)]⊘maxγt,ε, 1)+λt.d(x)\;=\;clip ( [ _1\,m_t-1+(1- _1)\,x ] \γ\,h_t,\, \,\;1 )\;+\;λ\, θ_t. Implementation. Clipping makes the update nonlinear in x. For scoring, the implementation omits clipping and uses the unclipped operator tSophia()=⊘maxγt,ε.O_t Sophia(x)\;=\;x \γ\,h_t,\, \. (13) Equation (13) defines an unclipped scoring surrogate, with th_t frozen, the frozen momentum contribution removed, and the current-gradient coefficient 1−β11- _1 set to one. Its per-sample decomposition is exact for this linear scoring map, not for the clipped optimizer update in general. Sophia scoring operator. tSophia()=⊘maxγt,εO_t Sophia(x)\;=\;x \γ\,h_t,\, \. This linear surrogate assigns larger weights to coordinates with smaller estimated curvature, subject to the numerical floor. B.5 Lion Algorithm 6 Lion (Chen et al., 2023), one step. Define two decay coefficients: β1 _1 mixes inside the sign, β2 _2 updates the stored buffer. 0: learning rate ηt _t; decays (β1,β2)( _1, _2); weight decay λ; state m (init 0) 1: t←β1t−1+(1−β1)^tc_t← _1\,m_t-1+(1- _1)\, g_t one-off interpolation; not stored 2: t+1←(1−ηtλ)t−ηtsign(t) θ_t+1←(1- _tλ)\, θ_t- _t\,sign(c_t) 3: t←β2t−1+(1−β2)^tm_t← _2\,m_t-1+(1- _2)\, g_t Derivation. Substituting x gives the exact frozen-state operator directly: tLion()=sign(β1t−1+(1−β1)).O_t Lion(x)\;=\;sign ( _1\,m_t-1+(1- _1)\,x ). (14) When the frozen momentum dominates the candidate contribution, the exact operator usually agrees with the candidate-independent pattern t=sign(t−1)s_t=sign(m_t-1). Candidate dependence then arises through coordinate sign flips. Under the Gaussian candidate model and momentum-dominance assumption of Theorem 2, these flips are rare tail events. The default β1=0.9 _1=0.9 alone does not establish this regime. Implementation. The implementation instead uses a fixed-sign linear surrogate33 3 The superscript linlin denotes a linear surrogate, not a Taylor or Fréchet linearization of the sign map, whose derivative is zero almost everywhere.: the frozen pattern ts_t multiplies the candidate gradient coordinate-wise. The alignment component of the score is Uz,alignLion,lin=ηt⟨sign(t−1)⊙z,⟩U_z,align Lion,lin= _t \,sign(m_t-1) _z,\,g_V . The full utility also includes the redundancy term in (1). The collapse bound for the exact sign score does not directly apply to this surrogate. Lion scoring operator. Exact: t()=sign(β1t−1+(1−β1))O_t(x)=sign( _1m_t-1+(1- _1)x). Implemented: the linear surrogate map ↦t⊙x _t , t=sign(t−1)s_t=sign(m_t-1) read from the stored buffer. B.6 Signum Algorithm 7 Signum (Bernstein et al., 2018), one step. 0: learning rate ηt _t; momentum decay μ; state m (init 0) 1: t←μt−1+(1−μ)^tm_t←μ\,m_t-1+(1-μ)\, g_t 2: t+1←t−ηtsign(t) θ_t+1← θ_t- _t\,sign(m_t) Derivation. Substituting x for the incoming gradient gives tSignum()=sign(μt−1+(1−μ)).O_t Signum(x)\;=\;sign (μ\,m_t-1+(1-μ)\,x ). (15) This has the same form as the Lion operator when β1=β2=μ _1= _2=μ, with the same coefficient controlling both the direction used for the update and the stored momentum. The gradient-driven updates coincide under this identification; any weight-decay conventions must also be matched. Unlike Lion, what is signed is the freshly stored buffer, so the frozen pattern’s persistence across steps tracks μ rather than a separate memory coefficient. Signum scoring operator. Exact: t()=sign(μt−1+(1−μ))O_t(x)=sign( _t-1+(1-μ)x). Implemented: the fixed-sign linear surrogate ↦t⊙x _t , as for Lion, with t=sign(t−1)s_t=sign(m_t-1). For plain SignSGD (μ=0μ=0), the implementation uses SGD scoring as a fallback; this is a scoring convention, not the exact sign-update map. B.7 D-Muon Algorithm 8 D-Muon (Jordan, 2024; Semenov et al., 2025), one optimizer step. Settings used throughout this paper: μ=0.95μ=0.95, kNS=5k_NS=5, r=0.2r=0.2, λ=0.1λ=0.1, and nesterov=Falsenesterov= False. 0: learning rate ηt _t; momentum μ; Newton–Schulz steps kNSk_NS; RMS target r; weight decay λ; AdamW hyperparameters (β1,β2,εa)( _1, _2, _a) for the auxiliary path 1: Partition: W takes the Muon path iff W is ≥2≥ 2-D and dim0()<104 _0(W)<10^4; embeddings and the LM head (vocabulary-sized first axis) therefore take the AdamW path, as do all 1-D parameters. 2: for each Muon-path weight ∈ℝm×nW ^m× n with gradient tG_t do 3: t←μt−1+tM_t←μ\,M_t-1+G_t unnormalized heavy-ball buffer 4: t←t+μtN_t _t+μ\,M_t if nesterov else tM_t matrix entering the sign map 5: t←NSkNS(t)≈msign(t)=⊤O_t _k_NS(N_t)≈ (N_t)=UV quintic iteration, (16) 6: ←(1−ηtλ)−ηtmax(m,n)rtW←(1- _tλ)\,W- _t\, (m,n)\;r\;O_t 7: end for 8: update the AdamW-path parameters by bias-corrected AdamW at the same ηt _t D-Muon does not compute an SVD. It applies kNSk_NS steps of a fixed odd polynomial iteration to approximately orthogonalize the update matrix: 0=t‖t‖F+10−7,j+1=aj+(bjj⊤+c(jj⊤)2)j,NSkNS(t)≜kNS, _0\;=\; N_t _t _F+10^-7, _j+1\;=\;a\,X_j+ (b\,X_jX_j +c\,(X_jX_j )^2 )X_j, _k_NS(N_t)\; \;X_k_NS, (16) with (a,b,c)=(3.4445,−4.7750, 2.0315)(a,b,c)=(3.4445,\,-4.7750,\,2.0315). The finite iteration is distinct from the ideal polar map used in the analysis below. Derivation. On a 2-D parameter block, the frozen state is the momentum matrix t=t=tA_t=N_t=M_t. For the ideal polar map, a candidate perturbation with weight α gives ↦msign(t+α)G (A_t+α\,G), which is nonlinear in G. This is an idealized local perturbation model, not an identity for the finite Newton–Schulz update. The rigorous linear object is the Fréchet derivative of the polar map at tA_t, tMuon,lin()=Dmsign(t)[],O_t Muon,lin(G)\;=\;D (A_t)[G], with full derivations and rank assumptions in Appendices D.3.1 and D.3. The derivative is defined per unit perturbation; a perturbation of size α contributes the corresponding scalar prefactor. Through the adjoint tT_A_t of (89), the first-order score is again a linear functional of the candidate, supported on singular-subspace rotations and new column-space directions and exactly zero on radial ones. Implementation. The implementation scores with the cheaper right-preconditioned proxy tMuon,rp()=(t⊤t+ε)−1/2=t,O_t Muon,rp(G)\;=\;G\,(A_t A_t+ )^-1/2\;=\;G\,P_t, (17) tP_t is formed once per weight matrix by symmetric eigendecomposition and reused for R scoring calls (R=10R=10 in experiments). At a full-column-rank reference matrix and with ε=0 =0, this proxy agrees with Dmsign(t)D (A_t) on the column-space complement. Positive regularization changes that agreement. The proxy also retains radial candidate components that the ideal derivative annihilates; Appendix D.4 gives the discrepancy in closed form. D-Muon scoring operator. The ideal polar derivative tMuon,lin()=Dmsign(t)[]O_t Muon,lin(G)=D (A_t)[G] annihilates radial perturbations under the assumptions of Theorem 4. This radial-invariance result does not extend to the implemented proxy. Implemented: tMuon,rp()=(t⊤t+ε)−1/2O_t Muon,rp(G)=G(A_t A_t+ )^-1/2, with the preconditioner refreshed every R=10R=10 steps. The auxiliary AdamW path’s parameters are excluded from scoring, as are all vocabulary-sized layers. B.8 Classification of the Frozen-State Operators Table 2summarizes the operators discussed in this appendix, including the SGD and Signum reference methods. The following remark states what linearity of a frozen scoring map guarantees and distinguishes that property from fidelity to the optimizer update. Remark 1 (Distributivity: when the per-sample decomposition is exact). The batch direction represented by a frozen linear scoring map decomposes exactly into per-sample directions: t(1B∑z,t)=1B∑zt(z,t).O_t\! ( 1B _zg_z,t )= 1B _zO_t(g_z,t). This holds for SGD, AdamW, AdEMAMix, and Sophia’s unclipped scoring operator (13). With a fixed scoring reference, these linear maps also induce linear scalar scores. By contrast, the exact clipped Sophia update, coordinate-wise sign maps, and the finite Muon matrix-sign map are nonlinear. We use the following linear scoring approximations: the fixed-sign surrogate for Lion/Signum, and the Fréchet derivative in the Muon analysis or the right-preconditioned proxy in the benchmark. Their per-sample decomposition is exact for the chosen scoring map, but only approximates attribution under the nonlinear optimizer update, to first order for tMuon,linO_t Muon,lin and heuristically for the deployed sign and Muon proxies. Table 2: Properties of the frozen-state maps discussed in this appendix. Linearity guarantees additivity of the represented batch direction, not exact attribution under the full optimizer update. “Magnitude information” refers to retaining continuous amplitude information rather than only sign patterns; it does not imply norm preservation. §The Sophia row describes its unclipped scoring surrogate. ‡The Muon row describes the ideal polar derivative; the implemented right-preconditioned proxy is also linear but retains radial components. ∗The Lion/Signum rows describe the exact sign maps; their implemented fixed-sign surrogates are linear and retain candidate amplitude on coordinates with nonzero frozen signs. Optimizer Operator family Linear in x? Magnitude information Operators analyzed SGD Unpreconditioned (identity) ✓ ✓ AdamW Diagonal adaptive (second moment) ✓ ✓ AdEMAMix Diagonal adaptive (second moment) ✓ ✓ Sophia Diagonal adaptive (curvature) ✓§ ✓ Muon Matrix sign / polar tangent ✓‡ Tangential; radial discarded Lion ∗ Sign-based × × Signum ∗ Sign-based × × Appendix C The Selection-Gain Principle: Proof and Empirical Evidence C.1 Proof of Theorem 1 Let us separate two ingredients for the proof of Theorem 1. Smoothness of the validation objective relates the realized one-step loss reduction to the mean utility of the trained-on batch, while the Gaussian candidate model determines the expected utility of the uniformly sampled and top-selected batches. Proof. Condition on the parameters t θ_t, the optimizer state, and the frozen scoring reference at step t. Let N≜|ℬ~t|=λB,q≜BN=1λ.N\; \;| B_t|=λ B, q\; \; BN= 1λ. For a∈rand,sela∈\rand,sel\, denote by ℬta⊆ℬ~tB_t^a B_t the batch returned by the corresponding sampling rule and by ta≜t(1B∑z∈ℬtaz,t)d_t^a\; \;O_t\! ( 1B _z _t^ag_z,t ) (18) the resulting batch direction. We first make explicit the regularity conditions required for the one-step comparison. Assume that ∇F∇ F is LF,tL_F,t-Lipschitz on a neighborhood containing every line segment t−sηtta:0≤s≤1,a∈rand,sel, \ θ_t-s _td_t^a:0≤ s≤ 1 \, a∈\rand,sel\, and that the two batch directions satisfy maxa∈rand,sel[‖ta‖22]≤Dt2<∞. _a∈\rand,sel\E\! [ _t^a _2^2 ]\;≤\;D_t^2<∞. (19) Assume further that the frozen map tO_t used in the present argument is linear and is the same map used both to score candidates and to form the batch direction. Consequently, ta=1B∑z∈ℬtat(z,t).d_t^a= 1B _z _t^aO_t(g_z,t). (20) The descent lemma now gives, for either sampling rule, F(t)−F(t−ηtta)=ηt⟨,t,ta⟩+ρta,F( θ_t)-F( θ_t- _td_t^a)= _t _V,t,\,d_t^a + _t^a, (21) where |ρta|≤LF,tηt22‖ta‖22.| _t^a|\;≤\; L_F,t _t^22 _t^a _2^2. (22) Let us clarify that this argument relies solely on local smoothness, requiring neither positive-definiteness nor a local quadratic-model assumption. To connect (21) to the candidate utilities, write the frozen reference used in (1) as ^,t≜,t−ηtt, g_V,t\; \;g_V,t- _t r_t, (23) where t r_t is the frozen redundancy direction. For z,t≜t(z,t),u_z,t\; \;O_t(g_z,t), the adopted utility score can then be written as U(z,t)=ηt⟨z,t,^,t⟩.U(z,t)= _t _z,t,\, g_V,t . (24) Averaging (24) over ℬtaB_t^a and using (20) gives U¯ℬta U_B_t^a ≜1B∑z∈ℬtaU(z,t) 1B _z _t^aU(z,t) =ηt⟨ta,,t⟩−ηt2⟨ta,t⟩. = _t _t^a,\,g_V,t - _t^2 _t^a,\, r_t . (25) Combining (21) and (25) therefore yields F(t)−F(t−ηtta)=U¯ℬta+εta,F( θ_t)-F( θ_t- _td_t^a)= U_B_t^a+ _t^a, (26) where εta=ρta+ηt2⟨ta,t⟩. _t^a= _t^a+ _t^2 _t^a,\, r_t . (27) By (19) and Cauchy–Schwarz, |[εta]|≤ηt2Ct,Ct≜LF,t2Dt2+‖t‖2Dt. |E[ _t^a] |\;≤\; _t^2C_t, C_t\; \; L_F,t2D_t^2+ r_t _2D_t. (28) Thus, uniformly over the two sampling rules, Rta≜[F(t)−F(t−ηtta)]=[U¯ℬta]+εt,a,|εt,a|≤ηt2Ct.R_t^a\; \;E\! [F( θ_t)-F( θ_t- _td_t^a) ]=E\! [ U_B_t^a ]+ _t,a, | _t,a|≤ _t^2C_t. (29) Hence, up to a controlled second-order error, comparing one-step loss reductions is equivalent to comparing the expected mean utilities of the retained batches. It remains to evaluate these two expectations. Let U1,…,UN∼i.i.d.(μU(t),σU2(t))U_1,…,U_N .i.d. N\! ( _U(t), _U^2(t) ) be the candidate utilities, and let U(1)≤⋯≤U(N)U_(1)≤·s≤ U_(N) denote their order statistics. Under uniform sampling, conditioning on the candidate pool gives [U¯ℬtrand|U1,…,UN]=1N∑i=1NUi.E\! [ U_B_t^rand\, |\,U_1,…,U_N ]= 1N _i=1^NU_i. (30) Taking expectation once more yields the exact identity [U¯ℬtrand]=μU(t).E\! [ U_B_t^rand ]= _U(t). (31) For top-B selection, U¯ℬtsel=1B∑k=N−B+1NU(k). U_B_t^sel= 1B _k=N-B+1^NU_(k). (32) Assume first that σU(t)>0 _U(t)>0 and write Ui=μU(t)+σU(t)Zi,Zi∼i.i.d.(0,1).U_i= _U(t)+ _U(t)Z_i, Z_i .i.d. N(0,1). Then [U¯ℬtsel]=μU(t)+cq,NσU(t),E\! [ U_B_t^sel ]= _U(t)+c_q,N _U(t), (33) where the exact finite-pool coefficient is cq,N≜[1B∑k=N−B+1NZ(k)].c_q,N\; \;E\! [ 1B _k=N-B+1^NZ_(k) ]. (34) The case σU(t)=0 _U(t)=0 is immediate: all utilities are identical and both sampling rules have zero selection gain. Equations (29), (31), and (33) give the exact finite-pool relations Rtrand R_t^rand =μU(t)+εt,rand, = _U(t)+ _t,rand, Rtsel R_t^sel =μU(t)+cq,NσU(t)+εt,sel, = _U(t)+c_q,N _U(t)+ _t,sel, (35) with |εt,rand|,|εt,sel|≤ηt2Ct.| _t,rand|,| _t,sel|≤ _t^2C_t. Consequently, Gt≜Rtsel−Rtrand=cq,NσU(t)+εt,G,|εt,G|≤2ηt2Ct.G_t\; \;R_t^sel-R_t^rand=c_q,N _U(t)+ _t,G, | _t,G|≤ 2 _t^2C_t. (36) The population coefficient in Theorem 1 is obtained by letting N→∞N→∞ while keeping q=B/N∈(0,1)q=B/N∈(0,1) fixed. Consistency of upper-trimmed L-statistics gives cq,N⟶cq≜1q∫1−q1Φ−1(u)u.c_q,N c_q\; \; 1q _1-q^1 ^-1(u)\,du. (37) Equivalently, with zq≜Φ−1(1−q),z_q\; \; ^-1(1-q), continuity of the Gaussian law gives cq=[Z|Z≥zq].c_q=E\! [Z\, |\,Z≥ z_q ]. (38) Since ϕ′(z)=−zϕ(z)φ (z)=-zφ(z), [ZZ≥zq] \! [Z1\Z≥ z_q\ ] =∫zq∞zϕ(z)z = _z_q^∞zφ(z)\,dz =[−ϕ(z)]zq∞=ϕ(zq). = [-φ(z) ]_z_q^∞=φ(z_q). (39) Dividing by Pr(Z≥zq)=q (Z≥ z_q)=q yields cq=ϕ(zq)q.c_q= φ(z_q)q. (40) In particular, when λ=2λ=2 and hence q=1/2q=1/2, c1/2=2ϕ(0)=2π≈0.79788.c_1/2=2φ(0)= 2π≈ 0.79788. (41) Thus, under the population top-q idealization, (36) becomes Gt=cqσU(t)+O(ηt2Ct),G_t=c_q _U(t)+O( _t^2C_t), (42) which is (4). For a finite candidate pool, the mathematically exact statement is (36). Alternatively, invoking the standard fixed-q Gaussian L-statistic bound cq,N=cq+O(N−1/2)c_q,N=c_q+O(N^-1/2) gives Gt=cqσU(t)+O(ηt2Ct+σU(t)λB).G_t=c_q _U(t)+O\! ( _t^2C_t+ _U(t) λ B ). (43) Because cqc_q is defined after centering and standardization, it is dimensionless and invariant under every positive affine transformation of the utility scores. ∎ Corollary 1 (Relative selection gain). Under the assumptions of Theorem 1, suppose that μU(t)>0 _U(t)>0 and that the aggregate learning-rate and finite-pool remainder is o(μU(t))o( _U(t)). Then GtRtrand=cqσU(t)μU(t)+o(1)=cqCV[U(⋅,t)]+o(1), G_tR_t^rand=c_q\, _U(t) _U(t)+o(1)=c_q\,CV\! [U(·,t) ]+o(1), (44) or, equivalently, Rtsel/Rtrand=1+cqCV[U(⋅,t)]+o(1)R_t^sel/R_t^rand=1+c_q\,CV[U(·,t)]+o(1). Thus, for a fixed retained fraction q, the relative benefit of selection is governed by the coefficient of variation of the optimizer-induced utility scores; the complete finite-pool and remainder bounds are given in Appendix C.1. Proof of Corollary 1. Write Rtrand=μU(t)+εt,rand,Gt=cq,NσU(t)+εt,G,R_t^rand= _U(t)+ _t,rand, G_t=c_q,N _U(t)+ _t,G, (45) where |εt,rand|≤ηt2Ct,|εt,G|≤2ηt2Ct.| _t,rand|≤ _t^2C_t, | _t,G|≤ 2 _t^2C_t. Suppose that μU(t)>0 _U(t)>0 and ηt2Ct≤μU(t)/2 _t^2C_t≤ _U(t)/2. Direct substitution gives GtRtrand G_tR_t^rand =cq,NσU(t)+εt,GμU(t)+εt,rand = c_q,N _U(t)+ _t,G _U(t)+ _t,rand =cq,NσU(t)μU(t)+(ηt2CtμU(t)[1+cq,NσU(t)μU(t)]). =c_q,N _U(t) _U(t)+O\! ( _t^2C_t _U(t) [1+c_q,N _U(t) _U(t) ] ). (46) Hence, to leading order, GtRtrand=cq,NCV[U(⋅,t)]+o(1). G_tR_t^rand=c_q,NCV\! [U(·,t) ]+o(1). (47) In the population limit, cq,N→cqc_q,N→ c_q, and therefore GtRtrand=cqCV[U(⋅,t)]+o(1). G_tR_t^rand=c_qCV\! [U(·,t) ]+o(1). (48) Equivalently, the multiplicative one-step improvement satisfies RtselRtrand=1+cqCV[U(⋅,t)]+o(1). R_t^selR_t^rand=1+c_qCV\! [U(·,t) ]+o(1). (49) ∎ Remark 2 (Linearity and attribution error). The preceding loss-reduction argument applies when the same frozen linear map, with matching scale, is used to score candidates and form the batch update. Linearity holds for the specified AdamW, AdEMAMix, and unclipped Sophia scoring maps; agreement with the full optimizer update is a separate assumption. For any mismatched optimizer update satisfying the corresponding smoothness and moment conditions, let ta≜t,upda−1B∑z∈ℬtat,score(z,t) _t^a\; \;d_t,upd^a- 1B _z _t^aO_t,score(g_z,t) (50) denote the discrepancy between the direction actually applied and the additive direction represented by the scoring map. The same proof then acquires the additional error ηt[⟨,t,ta⟩]. _tE\! [ _V,t,\, _t^a ]. (51) Thus, the O(ηt2)O( _t^2) conclusion extends to a nonlinear update only when this attribution error is itself O(ηt2)O( _t^2). It is zero for the exact linear families and is controlled to first order for the Fréchet-linearized polar map. For the deployed fixed-sign and right-preconditioned proxies, the theorem describes the gain predicted by the additive scoring geometry unless the corresponding update mismatch is bounded separately, as discussed in Remark 1. Remark 3 (Scope of the gain comparison). Theorem 1 compares selection with uniform sampling for a fixed optimizer and scoring geometry, not absolute convergence across optimizers. For μU(t)>0 _U(t)>0, in the population approximation and with negligible remainder terms, the relative extra one-step reduction is cqCV[U(⋅,t)]c_qCV[U(·,t)]; the total improvement factor is 1+cqCV[U(⋅,t)]1+c_qCV[U(·,t)]. Score spread alone does not establish alignment with the validation objective: useful selection requires both spread and ranking fidelity. Remark 4 (Temperature and the empirical 2×2× gain). Boltzmann sampling without replacement interpolates between top-B selection as τ→0τ→ 0 and uniform sampling as τ→∞τ→∞. For nondegenerate scores, let κt(τ)∈[0,1] _t(τ)∈[0,1] denote the fraction of the expected top-B score gain retained. In the Gaussian population approximation, Gt≃κt(τ)cqσU(t).G_t _t(τ)c_q _U(t). With standardized utilities, κt _t depends on temperature and distributional shape, not raw score scale alone. Assuming approximately constant κt≈κ¯ _t≈ κ and CV[U]≈ρCV[U]≈ρ, and neglecting trajectory changes, a heuristic applied-token speedup is 1+κ¯cqρ1+ κc_qρ. For q=1/2q=1/2 and κ¯≈1 κ≈ 1, the 2×2× efficiency reported by Wang et al. (2026) corresponds to ρ≈1/cq≈1.25ρ≈ 1/c_q≈ 1.25. Indeed, this is a consistency calculation, not a convergence or compute-efficiency guarantee. Appendix D Detailed Proofs for Discriminability (§4) This section provides proofs for each theorem in §4 with detailed derivations and supporting assumptions (§D.1–§D.3). D.1 Sign Collapse (Theorem 2) Throughout this subsection, we condition on the step-t optimizer state and on the frozen scoring reference g_V. Consequently, all expectations, probabilities, and variances are taken only over the candidate gradient z∼(,),=diag(σ12,…,σd2).g_z ( μ, ), =diag( _1^2,…, _d^2). Coordinates with [^]j≠0[ g_V]_j≠ 0 but σj=0 _j=0 contribute only candidate-independent constants to both scores and therefore contribute zero variance. They may be removed without affecting any discriminability statement. Hence, without loss of generality, we take σj>0 _j>0 on supp(^)supp( g_V). We also fix an arbitrary −1,+1\-1,+1\-valued convention for sign(0)sign(0); under the preceding non-degeneracy condition, the argument of the exact sign map equals zero with probability zero. Variance of fixed linear diagonal scores. We first record the variance identity used for the linearized score and, subsequently, for the diagonal-operator comparison of Theorem 3. Importantly, this identity applies only to a candidate-independent linear diagonal map; it does not apply directly to the exact nonlinear Lion score. Let ∈ℝd π ^d be fixed conditional on the optimizer state, and define Uz()≜ηt⟨⊙z,^⟩=ηtz⊤(⊙^).U_z^( π)\; \; _t π _z,\, g_V \;=\; _tg_z ( π g_V ). (52) Since only zg_z is random, Var[Uz()] [U_z^( π) ] =ηt2(⊙^)⊤(⊙^) = _t^2 ( π g_V ) ( π g_V ) =ηt2∑j=1dπj2σj2[^]j2. = _t^2 _j=1^d _j^2 _j^2[ g_V]_j^2. (53) In particular, Var[UzSGD]=ηt2∑j=1dσj2[^]j2=ηt2‖1/2^‖22.Var [U_z SGD ]= _t^2 _j=1^d _j^2[ g_V]_j^2= _t^2 ^1/2 g_V _2^2. (54) Whenever ‖1/2^‖2>0 ^1/2 g_V _2>0, define wj≜σj2[^]j2∑k=1dσk2[^]k2,∑j=1dwj=1.w_j\; \; _j^2[ g_V]_j^2 _k=1^d _k^2[ g_V]_k^2, _j=1^dw_j=1. (55) Then every comparison among fixed linear diagonal operators satisfies Var[Uz()]Var[UzSGD]=∑j=1dwjπj2. Var[U_z^( π)]Var[U_z SGD]= _j=1^dw_j _j^2. (56) If ‖1/2^‖2=0 ^1/2 g_V _2=0, the SGD and linearized sign scores are both deterministic, so their variance equality remains valid, although the normalized ratio (56) is undefined. Assumption 1 (Momentum dominance). Let 0<β1<10< _1<1. There exists κ≥1κ≥ 1 such that β1|[t−1]j|≥κ(1−β1)(σj+|μj|),∀j∈supp(^). _1|[m_t-1]_j|\;≥\;κ(1- _1) ( _j+| _j| ), ∀ j ( g_V). (57) To interpret the assumption, define the coordinate-wise dominance statistic κt,j≜β1|[t−1]j|(1−β1)(σj+|μj|). _t,j\; \; _1|[m_t-1]_j|(1- _1) ( _j+| _j| ). (58) Assumption 1 is equivalent to requiring κt,j≥κ _t,j≥κ on the support of the scoring reference. It is therefore a state-dependent and directly measurable condition, rather than a universal property of sign-based optimization. As a heuristic interpretation, under stationary i.i.d. gradients the stored momentum buffer approaches the expectation [t]→E[m_t]→ μ; for Lion, this remains true even though the stored buffer uses the separate decay coefficient β2 _2. Replacing the realized buffer by its stationary mean gives the mean-state approximation κt,j≈β1|μj|(1−β1)(σj+|μj|). _t,j≈ _1| _j|(1- _1) ( _j+| _j| ). (59) For β1=0.9 _1=0.9, the prefactor β1/(1−β1)=9 _1/(1- _1)=9 permits a substantial dominance margin when the gradient retains a persistent mean component. This stationary approximation is only interpretive and is not used in the proof below. Proposition 1 (Linearized sign scoring is spread-neutral). For every fixed sign pattern t∈−1,+1ds_t∈\-1,+1\^d, the linearized sign score UzLion,lin≜ηt⟨z,t⊙^⟩U_z Lion,lin\; \; _t _z,\,s_t g_V (60) satisfies Var[UzLion,lin]=ηt2∑j=1dσj2[^]j2=ηt2‖1/2^‖22=Var[UzSGD].Var [U_z Lion,lin ]= _t^2 _j=1^d _j^2[ g_V]_j^2= _t^2 ^1/2 g_V _2^2=Var [U_z SGD ]. (5) Proof of Proposition 1. The linearized score is a fixed linear functional of the candidate: UzLion,lin=ηt∑j=1d[z]j[t]j[^]j.U_z Lion,lin= _t _j=1^d[g_z]_j[s_t]_j[ g_V]_j. (61) Applying the fixed-diagonal variance identity (53) with πj=[t]j _j=[s_t]_j gives Var[UzLion,lin] [U_z Lion,lin ] =ηt2∑j=1d[t]j2σj2[^]j2 = _t^2 _j=1^d[s_t]_j^2 _j^2[ g_V]_j^2 =ηt2∑j=1dσj2[^]j2, = _t^2 _j=1^d _j^2[ g_V]_j^2, (62) because [t]j2=1[s_t]_j^2=1 for every coordinate. This is exactly the SGD score variance. Equivalently, the diagonal reflection diag(t)diag(s_t) preserves the covariance geometry because diag(t)diag(t)=.diag(s_t) diag(s_t)= . ∎ Proposition 2 (Exponential collapse of the exact score). Suppose that, for some κ≥1κ≥ 1, β1|[t−1]j|≥κ(1−β1)(σj+|μj|),∀j∈supp(^). _1|[m_t-1]_j|\;≥\;κ(1- _1) ( _j+| _j| ), ∀ j ( g_V). (63) Then the exact Lion score UzLion≜ηt⟨sign(β1t−1+(1−β1)z),^⟩U_z Lion\; \; _t \! ( _1m_t-1+(1- _1)g_z ),\, g_V (64) satisfies Var[UzLion]≤ 8e−κ2/2ηt2‖^‖22.Var [U_z Lion ]\;≤\;8e^-κ^2/2 _t^2 g_V _2^2. (65) Consequently, whenever ‖1/2^‖2>0 ^1/2 g_V _2>0, Var[UzLion]≤Var[UzLion,lin]=Var[UzSGD]ifκ2≥ 2log(8‖^‖22‖1/2^‖22).Var [U_z Lion ]\;≤\;Var [U_z Lion,lin ]=Var [U_z SGD ] κ^2\;≥\;2 \! ( 8 g_V _2^2 ^1/2 g_V _2^2 ). (66) The same conclusions hold for Signum after replacing β1 _1 with its momentum coefficient μ. Proof of Proposition 2. Step 1: define coordinate-wise sign flips We first apply exact decomposition into coordinate-wise sign flips. Denote [t]j≜sign([t−1]j),[s_t]_j\; \;sign([m_t-1]_j), (67) and the exact candidate-dependent sign ξj,z≜sign(β1[t−1]j+(1−β1)[z]j). _j,z\; \;sign\! ( _1[m_t-1]_j+(1- _1)[g_z]_j ). (68) For each coordinate, introduce the flip indicator, which is the probability that the candidate gradient flips the sign of the momentum buffer at jthj^th coordinate. Fj,z≜ 1ξj,z≠[t]j,pj≜Prz(Fj,z=1).F_j,z\; \;1\ _j,z≠[s_t]_j\, p_j\; \; _z(F_j,z=1). (69) Since both ξj,z _j,z and [t]j[s_t]_j take values in −1,+1\-1,+1\, [ξj,z]=[t]j(1−2pj),[ξj,z2]=1.E[ _j,z]=[s_t]_j (1-2p_j ),E[ _j,z^2]=1. (70) Therefore, we have Var[ξj,z]=1−(1−2pj)2=4pj(1−pj)≤4pj.Var[ _j,z]=1-(1-2p_j)^2=4p_j(1-p_j)≤ 4p_j. (71) According to our Gaussian candidate modelling, is diagonal and zg_z is Gaussian, so the coordinates [z]jj=1d\[g_z]_j\_j=1^d are independent. Therefore, the flip indicators are independent across coordinates. We can write the variance of the exact score as: Var[UzLion] [U_z Lion ] =Var[ηt∑j=1dξj,z[^]j] =Var[ _t _j=1^d _j,z[ g_V]_j] =ηt2∑j=1d[^]j2Var[ξj,z] = _t^2 _j=1^d[ g_V]_j^2Var[ _j,z] =4ηt2∑j=1d[^]j2pj(1−pj)≤4ηt2∑j=1d[^]j2pj. =4 _t^2 _j=1^d[ g_V]_j^2p_j(1-p_j)≤ 4 _t^2 _j=1^d[ g_V]_j^2p_j. (72) Thus, we find that the candidate-dependent variation in the nonlinear sign score is carried by coordinate-wise sign reversals. Step 2: sign flips is a Gaussian tail events under momentum dominance. ∀j∈supp(^)∀ j ( g_V), a flip occurs when the candidate gradient satisfies: Fj,z=1=[t]j[z]j<−at,j,at,j≜β1|[t−1]j|1−β1.\F_j,z=1\= \[s_t]_j[g_z]_j<-a_t,j \,a_t,j\; \; _1|[m_t-1]_j|1- _1. (73) Since [t]j[z]j∼([t]jμj,σj2),[s_t]_j[g_z]_j \! ([s_t]_j _j, _j^2 ), the flip probability can be written in a Gaussian cumulative distribution function Φ(⋅) (·): pj=[[t]j[z]j<−at,j]=Φ(−at,j−[t]jμjσj).p_j=P[ \[s_t]_j[g_z]_j<-a_t,j \]= \! ( -a_t,j-[s_t]_j _j _j ). (74) According to Assumption 1, we have at,j≥κ(σj+|μj|)a_t,j≥κ ( _j+| _j| ), therefore, −at,j−[t]jμjσj -a_t,j-[s_t]_j _j _j ≤−κ−κ|μj|+[t]jμjσj≤−κ. ≤-κ- κ| _j|+[s_t]_j _j _j≤-κ. (75) The second inequality is from κ|μj|+[t]jμj≥(κ−1)|μj|≥0.κ| _j|+[s_t]_j _j≥(κ-1)| _j|≥ 0. Combining (74) and (75) gives the sharper one-sided bound pj≤Φ(−κ).p_j≤ (-κ). (76) In particular, for Z∼(0,1)Z (0,1), according to Gaussian Chernoff tail bound, we have pj≤Φ(−κ)≤Pr(|Z|≥κ)≤2e−κ2/2.p_j≤ (-κ)≤ (|Z|≥κ)≤ 2e^-κ^2/2. (77) Substituting (77) into (72), we obtain Var[UzLion] [U_z Lion ] ≤4ηt2∑j=1d[^]j2pj ≤ 4 _t^2 _j=1^d[ g_V]_j^2p_j ≤8e−κ2/2ηt2∑j=1d[^]j2=8e−κ2/2ηt2‖^‖22, ≤ 8e^-κ^2/2 _t^2 _j=1^d[ g_V]_j^2=8e^-κ^2/2 _t^2 g_V _2^2, (78) which proves Eq. (6). Finally, when ‖1/2^‖2>0 ^1/2 g_V _2>0, combining Eqs. (5) and (6) yields Var[UzLion]Var[UzLion,lin]≤8e−κ2/2‖^‖22‖1/2^‖22. Var[U_z Lion]Var[U_z Lion,lin]≤ 8e^-κ^2/2 g_V _2^2 ^1/2 g_V _2^2. (79) Therefore, Var[UzLion]≤Var[UzLion,lin]Var [U_z Lion ] [U_z Lion,lin ] (80) whenever κ2≥2log(8‖^‖22‖1/2^‖22).κ^2≥ 2 \! ( 8 g_V _2^2 ^1/2 g_V _2^2 ). (81) The proof uses only the generic sign-operator form sign(αt−1+(1−α)z).sign\! ( _t-1+(1-α)g_z ). The same proof applies to Signum by replacing β1 _1 with Signum’s scalar momentum coefficient μ, which is distinct from the gradient-mean vector μ. ∎ The two parts of Theorem 2 concern distinct scoring rules. Proposition 1 analyzes the fixed-sign linear score denoted by UzLion,linU_z Lion,lin, whereas (6) analyzes the exact nonlinear score UzLionU_z Lion. Accordingly, the fixed-diagonal spread identity (56) may be applied to the former but not to the latter. Conversely, the momentum-dominance bound for the exact score does not by itself constitute a variance bound for the implemented linearized score. D.2 Diagonal-Adaptive Advantage of AdamW and Sophia (Theorem 3) The amplification half of the ranking rests on one further empirical condition and one convexity lemma, from which the two auxiliary propositions and the theorem follow. Assumption 2 (Weighted variance normalization). With the weights wjw_j of (55), the second-moment estimates satisfy ∑jwj[^t]j≤1 _jw_j\,[ v_t]_j≤ 1, and [^t]j[ v_t]_j is non-constant on the support of w. Assumption 2asks that the coordinates carrying validation-relevant variance not sit where the second moments are large. It is empirically mild since gradient clipping caps the incoming squared gradients, and the downstream proxy concentrates w on rare-feature coordinates, precisely those with [^t]j≪1[ v_t]_j 1. We start the comparison on AdamW and Sophia from Lemma 2: Lemma 2 (Preconditioner amplification). Let wj>0w_j>0 be weights with ∑jwj=1 _jw_j=1, and let cj>0c_j>0 be positive scalars that are not all equal. Then ∑jwjcj2>(∑jwjcj)2 _jw_j\,c_j^2> ( _jw_j\,c_j )^2. Moreover, for cj=[t]j=1/([^t]j+ε)c_j=[c_t]_j=1/( [ v_t]_j+ ) under Assumption 2, ∑jwj[t]j2>1(1+ε′)2withε′≜εminj[^t]j, _jw_j\,[c_t]_j^2\;>\; 1(1+ )^2 \; \; _j [ v_t]_j, and in the regime ε≪minj[^t]j _j [ v_t]_j relevant to AdamW (ε=10−8 =10^-8), ∑jwj[t]j2> 1−O(ε′). _jw_j\,[c_t]_j^2\;>\;1-O( ). Proof of Lemma 2 (Preconditioner amplification). The first statement is an immediate consequence of Jensen’s inequality applied to the strictly convex function φ(x)=x2 (x)=x^2: w[φ(cJ)]>φ(w[cJ])E_w[ (c_J)]> (E_w[c_J]) when cJc_J is non-constant under the distribution w. For the second statement, write [^t]j+ε=[^t]j(1+ε/[^t]j)≤[^t]j(1+ε′) [ v_t]_j+ = [ v_t]_j\, (1+ / [ v_t]_j )≤ [ v_t]_j\,(1+ ), so that [t]j2≥1(1+ε′)2⋅1[^t]j[c_t]_j^2≥ 1(1+ )^2· 1[ v_t]_j coordinate-wise; it therefore suffices to bound ∑jwj/[^t]j _jw_j/[ v_t]_j. By Jensen’s inequality applied to the convex function x↦1/x 1/x on ℝ>0R_>0, ∑jwj[^t]j≥1∑jwj[^t]j≥ 1, _j w_j[ v_t]_j\;≥\; 1 _jw_j\,[ v_t]_j\;≥\;1, where the last step uses Assumption 2; the inequality is strict since [^t]j[ v_t]_j is non-constant on the support of w. The ε→0 → 0 statement follows since ε′=(ε) =O( ) when ε≪minj[^t]j _j [ v_t]_j. We also discuss the role of the normalization condition; without it the conclusion can fail (e.g., w=(12,12)w=( 12, 12), ^t=(9, 0.81) v_t=(9,\,0.81) gives ∑jwj[t]j2≈0.67 _jw_j[c_t]_j^2≈ 0.67), so Assumption 2 is not merely technical. ∎ Proposition 3 (Variance preconditioning amplifies spread). The AdamW score UzAdam=ηt⟨z,t⊙⟩U_z Adam= _t _z,\,c_t _V has Var[UzAdam]=ηt2∑j=1dσj2[t]j2[]j2=ηt2‖1/2(t⊙)‖22,Var[U_z Adam]\;=\; _t^2 _j=1^d _j^2\,[c_t]_j^2\,[g_V]_j^2\;=\; _t^2 ^1/2(c_t _V) _2^2, (82) so its spread ratio (56) is ∑jwj[t]j2 _jw_j\,[c_t]_j^2, which under Assumption 2, Lemma 2 bounds below: Var[UzAdam]Var[UzSGD]=∑j=1dwj[t]j2> 1−O(ε′). Var[U_z Adam]Var[U_z SGD]\;=\; _j=1^dw_j\,[c_t]_j^2\;>\;1-O( ). Proof of Proposition 3 (Variance preconditioning amplifies spread). The Adam influence score is UzAdam=ηt⟨t⊙z,⟩=ηt∑j[t]j[z]j[]jU_z Adam= _t _t _z,\,g_V = _t _j[c_t]_j[g_z]_j[g_V]_j. By independence of the coordinates of zg_z: Var[UzAdam]=ηt2∑j[t]j2[]j2σj2Var[U_z Adam]= _t^2 _j[c_t]_j^2\,[g_V]_j^2\, _j^2, yielding (82). Dividing by Var[UzSGD]=ηt2∑jσj2[]j2Var[U_z SGD]= _t^2 _j _j^2[g_V]_j^2 gives the spread ratio ∑jwj[t]j2 _jw_j[c_t]_j^2 with the weights (55), and Lemma 2 under Assumption 2 bounds it below by 1−O(ε′)1-O( ). ∎ Proposition 4 (Curvature preconditioning amplifies spread). The Sophia score under the unclipped scoring operator (13) is the diagonal-form score with pattern πjSophia=1/maxγ[t]j,ε _j Sophia=1/ \γ[h_t]_j,\, \. Suppose the curvature estimates satisfy the normalization analogous to Assumption 2: ∑jwjγ[t]j≤1 _jw_j\,γ[h_t]_j≤ 1, with [t]j[h_t]_j non-constant on the support of w. Then, with ε′≜ε/minjγ[t]j / _jγ[h_t]_j, Var[UzSophia]Var[UzSGD]=∑j=1dwj(πjSophia)2>1(1+ε′)2= 1−O(ε′) Var[U_z Sophia]Var[U_z SGD]\;=\; _j=1^dw_j\, ( _j Sophia )^2\;>\; 1(1+ )^2\;=\;1-O( ) in the regime ε≪minjγ[t]j _jγ[h_t]_j. The amplification enters through the squared inverse curvature, concentrated on exactly the flat directions (small [t]j[h_t]_j) that the validation-relevant weights wjw_j occupy; the condition is measurable along training in the same way as Assumption 2. Proof. Since maxγ[t]j,ε≤γ[t]j(1+ε/γ[t]j)≤γ[t]j(1+ε′) \γ[h_t]_j, \≤γ[h_t]_j\,(1+ /γ[h_t]_j)≤γ[h_t]_j\,(1+ ), we have (πjSophia)2≥(1+ε′)−2(γ[t]j)−2 ( _j Sophia )^2≥(1+ )^-2\,(γ[h_t]_j)^-2 coordinate-wise. Applying Jensen’s inequality twice, to x↦x2x x^2 (strictly, since [t]j[h_t]_j is non-constant on the support of w) and to the convex map x↦1/x 1/x, ∑jwj(γ[t]j)−2>(∑jwj(γ[t]j)−1)2≥(∑jwjγ[t]j)−2≥ 1, _jw_j\,(γ[h_t]_j)^-2\;>\; ( _jw_j\,(γ[h_t]_j)^-1 )^2\;≥\; ( _jw_j\,γ[h_t]_j )^-2\;≥\;1, where the last step uses the normalization. Combining the two displays gives the claim; the spread-ratio identity is (56). ∎ Combining Proposition 3–4 with Proposition 2 (66), we complete the proof of Theorem 3. D.3 Radial Collapse of Muon (Theorem 4) D.3.1 Preliminaries: the ideal polar map Throughout, all matrices are real, and msign denotes the ideal polar-factor map, not the conventional primary matrix-sign function. The exact identities below do not describe the finite Newton–Schulz computation in (16). We condition on the momentum matrix tA_t, the scoring reference =^∈ℝm×nR= G_V ^m× n, and all scoring coefficients. All probabilistic statements are conditional on these frozen quantities; only the candidate gradient zG_z is random. Assumption 3 (Regular polar state). The matrix t∈ℝm×nA_t ^m× n, m≥n≥1m≥ n≥ 1, has full column rank. Fix a compact SVD t=⊤,=diag(σ1,…,σn),σ1≥⋯≥σn>0,A_t=U V , =diag( _1,…, _n), _1≥·s≥ _n>0, (83) where ⊤=⊤=nU U=V V=I_n and V is square. Repeated singular values are allowed. The polar decomposition is t=A_t=QH, where =msign(t)=t(t⊤t)−1/2=⊤,=⊤≻0.Q= (A_t)=A_t(A_t A_t)^-1/2=UV , =V V 0. (84) In particular, ⊤=nQ Q=I_n. The polar-derivative statements for full-row-rank wide matrices follow by transposition. The right-preconditioned proxy is analyzed below in the displayed m≥nm≥ n orientation. Proposition 5 (Spectral-norm steepest descent). For full-column-rank =⊤A=U V , msign()=⊤ (A)=UV is the unique maximizer of ⟨,⟩F ,\,D _F over ‖op≤1 _op≤ 1. Proof. For any feasible D, its alignment with the singular vectors satisfies ⟨,⟩F=∑i=1nσii⊤i≤∑i=1nσi, ,\,D _F= _i=1^n _iu_i Dv_i≤ _i=1^n _i, with equality at =⊤D=UV . Since every σi>0 _i>0, equality requires i⊤i=1u_i Dv_i=1 for every i. Together with ‖i‖2≤1 _i _2≤ 1, this implies i=iDv_i=u_i. The vectors iv_i form a basis, so =⊤D=UV . ∎ D.3.2 Polar derivative and radial geometry Fix an orthonormal complement ⟂∈ℝm×(m−n)U_ ^m×(m-n) so that [⟂][U\ U_ ] is orthogonal. Define z _z =⊤z, =U G_zV, z _z =⟂⊤z, =U_ G_zV, (85) ∥ _ =⊤, =U RV, ⟂ _ =⟂⊤. =U_ RV. (86) When m=nm=n, complement-block terms are absent. Unless indicated otherwise, i,ji,j range over 1,…,n1,…,n and a over 1,…,m−n1,…,m-n. The radial subspace and its Frobenius-orthogonal complement are ℛt _t =⊤:∈ℝn×n,=⊤, =\USV :S ^n× n,\ S=S \, ℛt⟂ _t =∈ℝm×n:⊤+⊤=. =\X ^m× n:Q X+X Q=0\. Equivalently, ℛt=:=⊤R_t=\QS:S=S \: radial perturbations change the positive polar factor at fixed Q. Writing sym()=(+⊤)/2sym(X)=(X+X )/2, define Πrad=sym(⊤)⊤,Πtan=Id−Πrad. ^radM=Usym(U MV)V , ^tan=Id- ^rad. (87) These are the orthogonal projections onto ℛtR_t and ℛt⟂R_t , respectively, since symmetric and skew-symmetric matrices are Frobenius-orthogonal and the two column-space blocks are orthogonal. Set za=ΠazG_z^a= ^aG_z for a∈rad,tana∈\rad,tan\. Theorem 5 (Exact first-order geometry of matrix-sign scoring). Under Assumption 3, let ℒt=Dmsign(t)L_t=D (A_t). For any ∈ℝm×nG ^m× n, set =⊤B=U GV and =⟂⊤C=U_ GV. Then ℒt[] _t[G] =()⊤+⟂−1⊤, =U (B)V +U_ C ^-1V , (88) [()]ij [ (B)]_ij =[]ij−[]jiσi+σj,1≤i,j≤n. = [B]_ij-[B]_ji _i+ _j, 1≤ i,j≤ n. The derivative is self-adjoint for the Frobenius inner product: t≜ℒt∗=ℒt.T_A_t _t^*=L_t. (89) Moreover, with δt=σn/2 _t= _n/2, there exists Ct<∞C_t<∞, depending only on tA_t, such that Φz(α) _z(α) ≜⟨msign(t+αz),⟩F (A_t+ _z),\,R _F (90) =⟨,⟩F+α⟨z,ℒt[]⟩F+rz(α), = ,\,R _F+α _z,\,L_t[R] _F+r_z(α), |rz(α)| |r_z(α)| ≤Ct‖Fα2‖z‖F2 ≤ C_t _Fα^2 _z _F^2 for every real α and every zG_z satisfying |α|‖z‖F≤δt|α| _z _F≤ _t. Proof. The polar map is smooth on the open set of full-column-rank matrices. Differentiate t=A_t=QH in direction G to obtain =˙+˙G= QH+Q H. Since ⊤=nQ Q=I_n and ˙ H is symmetric, =⊤˙K=Q Q is skew-symmetric, and + +HK =⊤−⊤, =Q G-G Q, (m−⊤)˙ (I_m-QQ ) Q =(m−⊤)−1. =(I_m-QQ )GH^-1. In the V basis, the first identity becomes +=−⊤ + =B-B , where =⊤ =V KV. Solving entrywise and using ˙=+(m−⊤)˙ Q=QK+(I_m-QQ ) Q gives (88). In the orthogonal block coordinates (,)(B,C), the map on each pair ([]ij,[]ji)([B]_ij,[B]_ji), i<ji<j, has the symmetric matrix 1σi+σj(1−1−11); 1 _i+ _j pmatrix1&-1\\ -1&1 pmatrix; diagonal entries are sent to zero, and each []aj[C]_aj is multiplied by σj−1 _j^-1. Thus ℒtL_t is self-adjoint. Finally, the closed Frobenius ball of radius δt=σn/2 _t= _n/2 around tA_t lies in the full-column-rank set. The second derivative is bounded on this compact ball, so Taylor’s theorem and Cauchy–Schwarz give (90). ∎ Corollary 2 (Exact radial nullspace). Under Assumption 3, kerℒt=ℛt,ℒt=ℒtΠtan=Πtanℒt. _t=R_t, _t=L_t ^tan= ^tanL_t. Proof. The two terms in (88) are orthogonal and vanish simultaneously exactly when =⊤B=B and =0C=0. This gives the kernel and the first projection identity; self-adjointness gives the second. ∎ Corollary 3 (Exact radial invariance). Under Assumption 3, for every symmetric ∈ℝn×nS ^n× n and every real α, msign(t+α⊤)=⊤whenever+α≻0. (A_t+ )=UV + 0. (91) Proof. The factorization t+α⊤=(⊤)[(+α)⊤]A_t+ =(UV )[V( + )V ] is its polar decomposition. ∎ A diagonal S changes only singular values in the chosen basis. A general symmetric S can also change the eigenvectors of the positive polar factor while leaving Q unchanged. The finite invariance above concerns purely radial perturbations; for arbitrary candidates differing radially, equality is guaranteed only for their first-order scores at the frozen state. D.3.3 Score variance and proof of Theorem 4 Fix ηt>0 _t>0 and α≥0α≥ 0, and assume ‖z‖F2<∞E _z _F^2<∞. Define UzMuon U_z Muon =ηtα⟨ℒt[z],⟩F=ηtα⟨z,ℒt[]⟩F, = _tα _t[G_z],\,R _F= _tα _z,\,L_t[R] _F, Uza U_z^a =ηt⟨za,⟩F,a∈rad,tan, = _t _z^a,\,R _F, a∈\rad,tan\, UzSGD U_z SGD =ηt⟨z,⟩F=Uzrad+Uztan. = _t _z,\,R _F=U_z^rad+U_z^tan. Thus UzMuonU_z Muon is the candidate-dependent first-order term of ηtΦz(α) _t _z(α). Corollary 4 (General covariance formula). Under the frozen-state and second-moment conditions above, let =Cov(vecz) =Cov(vecG_z) and =vec(ℒt[])t=vec(L_t[R]). Then Var(UzMuon)=ηt2α2⊤.Var(U_z Muon)= _t^2α^2t t. (92) No coordinate-uncorrelation assumption is needed. Proof. Apply the covariance identity for the linear functional UzMuon=ηtα⊤vec(z)U_z Muon= _tα\,t vec(G_z). ∎ For the coordinate formulas and Theorem 4, additionally assume that the entries of the combined collection (z,z)(B_z,C_z) are pairwise uncorrelated, including cross-block pairs, and write τij2=Var([z]ij),νaj2=Var([z]aj). _ij^2=Var([B_z]_ij), _aj^2=Var([C_z]_aj). These are entry variances, not variances of antisymmetric differences. Uncorrelation is assumed in the fixed singular-vector basis and does not follow merely from diagonal covariance in the original parameter coordinates. Neither Gaussianity nor a zero-mean assumption is required. Proof of Theorem 4. Corollary 2 gives ℒt[z]=ℒt[ztan]L_t[G_z]=L_t[G_z^tan], so changing only the radial component leaves the score unchanged. By self-adjointness and (88), UzMuon=ηtα[∑i≠j[z]ij[∥]ij−[∥]jiσi+σj+∑a,j[z]aj[⟂]ajσj].U_z Muon= _tα\! [ _i≠ j[B_z]_ij [R_ ]_ij-[R_ ]_ji _i+ _j+ _a,j[C_z]_aj [R_ ]_aj _j ]. Pairwise uncorrelation yields Var(UzMuon)=ηt2α2[∑i≠jτij2([∥]ij−[∥]jiσi+σj)2+∑a,jνaj2([⟂]ajσj)2].Var(U_z Muon)= _t^2α^2\! [ _i≠ j _ij^2 ( [R_ ]_ij-[R_ ]_ji _i+ _j )^2+ _a,j _aj^2 ( [R_ ]_aj _j )^2 ]. (93) Likewise, the tangential projection gives Var(Uztan)=ηt2[∑i≠jτij2([∥]ij−[∥]ji2)2+∑a,jνaj2[⟂]aj2].Var(U_z^tan)= _t^2\! [ _i≠ j _ij^2 ( [R_ ]_ij-[R_ ]_ji2 )^2+ _a,j _aj^2[R_ ]_aj^2 ]. Since 2/(σi+σj)≤1/σn2/( _i+ _j)≤ 1/ _n and 1/σj≤1/σn1/ _j≤ 1/ _n, termwise comparison gives Var(UzMuon)≤α2σn2Var(Uztan),Var(U_z Muon)≤ α^2 _n^2Var(U_z^tan), which is (9). ∎ Comparison with SGD. Orthogonality of the matrix projections does not imply uncorrelation of the projected scalar scores. Indeed, Var(UzSGD)=Var(Uztan)+Var(Uzrad)+2Cov(Uztan,Uzrad),Var(U_z SGD)=Var(U_z^tan)+Var(U_z^rad)+2Cov(U_z^tan,U_z^rad), and expansion under the coordinate assumptions above gives Cov(Uztan,Uzrad)=ηt24∑i<j(τij2−τji2)([∥]ij2−[∥]ji2).Cov(U_z^tan,U_z^rad)= _t^24 _i<j( _ij^2- _ji^2) ([R_ ]_ij^2-[R_ ]_ji^2 ). For example, transpose-pair variance symmetry τij2=τji2 _ij^2= _ji^2 makes this covariance zero; together with α≤σnα≤ _n, it is sufficient for Var(UzMuon)≤Var(UzSGD)Var(U_z Muon) (U_z SGD). There is no unconditional ordering: for t=s2A_t=sI_2, s>0s>0, a nonzero skew-symmetric reference R, and independent unit-variance entries of zG_z, the ratio is α2/s2α^2/s^2, which exceeds one when 0<s<α0<s<α. D.4 Right-Preconditioned Muon Proxy and Exact Error For fixed ε≥0 ≥ 0, write ε=(t⊤t+εn)−1/2=ε⊤,ε=diag((σj2+ε)−1/2)j=1n.P_ =(A_t A_t+ _n)^-1/2=VD_ V , _ =diag\! (( _j^2+ )^-1/2 )_j=1^n. The score with the normalization of (2) is Uzrp=ηt⟨zε,⟩FU_z^rp= _t _zP_ ,\,R _F. Since zε=zε⊤+⟂zε⊤,G_zP_ =UB_zD_ V +U_ C_zD_ V , taking the inner product with R and using the same coordinate uncorrelation assumption gives Var(Uzrp)=ηt2[∑i,jτij2[∥]ij2σj2+ε+∑a,jνaj2[⟂]aj2σj2+ε].Var(U_z^rp)= _t^2\! [ _i,j _ij^2 [R_ ]_ij^2 _j^2+ + _a,j _aj^2 [R_ ]_aj^2 _j^2+ ]. (94) Without coordinate uncorrelation, the corresponding identity is Var(Uzrp)=ηt2⊤Var(U_z^rp)= _t^2p p, where =vec(ε)p=vec(RP_ ). Both scores are fixed linear functionals of vec(z)vec(G_z); if this vector is Gaussian, the scores are jointly Gaussian, possibly degenerate. For any =⊤+⟂⊤G=UBV +U_ CV , subtracting the proxy from (88) gives ℒt[]−ε= _t[G]-GP_ = (()−ε)⊤ ( (B)-BD_ )V (95) +⟂(−1−ε)⊤. +U_ C ( ^-1-D_ )V . The two blocks are orthogonal, hence ‖ℒt[]−ε‖F2= _t[G]-GP_ _F^2= ‖()−ε‖F2 (B)-BD_ _F^2 (96) +‖(−1−ε)‖F2. + ( ^-1-D_ ) _F^2. At ε=0 =0, the proxy agrees with the derivative on the column-space complement, but generally not on the parallel block. For ε>0 >0, the complement weights also differ when that block is present. Since εP_ is invertible, the proxy annihilates no nonzero matrix, including radial perturbations. Their scalar alignments can nevertheless vanish for particular references. Thus radial invariance of the ideal polar map is not a radial-collapse guarantee for the deployed proxy. Normalization and scope. The coefficient α is the candidate’s weight in the local matrix perturbation, determined by the batch and momentum conventions; it is not universally 1/B1/B. The displayed proxy score omits this weight, as in (2). For equally weighted scores, |UzMuon−αUzrp|≤ηtα‖F‖ℒt[z]−zε‖F.|U_z Muon-α U_z^rp|≤ _tα _F _t[G_z]-G_zP_ _F. Multiplying any score by a fixed scalar multiplies its variance by the square of that scalar; any additional block-update rescaling must therefore be retained in calibrated comparisons. All variance identities above concern the specified linear scores, not the finite nonlinear alignment ηtΦz(α) _t _z(α). The Taylor estimate in (90) is local and must be controlled separately when relating their variances. D.5 Optimal Candidate Multiplier (Lemma 1) We provide the detailed proof of Lemma 1 here. Proof. Following Theorem 1, the per-step selection gain is Gt=cqσU(t)G_t=c_q\, _U(t) with q=1/λq=1/λ. A step draws B~=λB B=λ B candidate rows and trains on B of them. Denote sequence length L, OptiSelect streams and scores λBLλ BL tokens while training the model on BLBL. Therefore, under a budget counted in tokens read, we quantify the selection gain per-read-token as GtλBL∝cq(λ)λ=:f(λ),λ∈(1,∞). G_tλ BL\; \; c_q(λ)λ\;=:\;f(λ), λ∈(1,∞). (97) Let U∼(μ,σ2)U (μ,σ^2), define the density, CDF and upper tail functions for z∼(0,1)z (0,1) as ϕ(z)=12πe−z2/2,Φ(z)=∫−∞zϕ(u)du,Φ¯= 1−Φ,φ(z)\;=\; 1 2π\,e^-z^2/2, (z)\;=\; _-∞^zφ(u)\,du, \;=\;1- , with Φ−1 ^-1 denoting the quantile function. Let hard top-q selection retain the largest q-fraction, i.e. those U above the (1−q)(1-q)-quantile uq=μ+σΦ−1(1−q)u_q=μ+σ\, ^-1(1-q). Writing Z=(U−μ)/σ∼(0,1)Z=(U-μ)/σ (0,1) and a=Φ−1(1−q)a= ^-1(1-q), the truncated-normal mean is [Z∣Z>a]=1Pr[Z>a]∫a∞zϕ(z)z=[−ϕ(z)]a∞1−Φ(a)=ϕ(a)1−Φ(a),E[Z Z>a]\;=\; 1 [Z>a] _a^∞z\,φ(z)\,dz\;=\; [-φ(z)]_a^∞1- (a)\;=\; φ(a)1- (a), (98) using ϕ′(z)=−zϕ(z)φ (z)=-zφ(z), so that ∫a∞zϕ(z)z=ϕ(a) _a^∞zφ(z)dz=φ(a). By construction 1−Φ(a)=q=1λ1- (a)=q= 1λ, hence selection lifts the mean utility by σϕ(a)/qσ\,φ(a)/q and cq=ϕ(Φ−1(1−q))q.c_q\;=\; φ ( ^-1(1-q) )q. (99) Therefore, plug in (97), we have: f(λ)=cq(λ)λ=ϕ(Φ−1(1−1/λ)).f(λ)\;=\; c_q(λ)λ\;=\;φ ( ^-1(1-1/λ) ). (100) Write z(λ)=Φ−1(1−1/λ)z(λ)= ^-1(1-1/λ), so Φ(z)=1−1/λ (z)=1-1/λ. Differentiating implicitly, ϕ(z)z′(λ)=1λ2⟹z′(λ)=1λ2ϕ(z)> 0.φ(z)\,z (λ)\;=\; 1λ^2 z (λ)\;=\; 1λ^2\,φ(z)\;>\;0. Then, plug in ϕ′(z)=−zϕ(z)φ (z)=-z\,φ(z) with the chain rule, we have: f′(λ)=ϕ′(z)z′(λ)=(−zϕ(z))⋅1λ2ϕ(z)=−z(λ)λ2=−Φ−1(1−1/λ)λ2.f (λ)\;=\;φ (z)\,z (λ)\;=\; (-z\,φ(z) )· 1λ^2φ(z)\;=\;-\, z(λ)λ^2\;=\;-\, ^-1(1-1/λ)λ^2. (101) We then find the optimum of (100) by introducing an involution λ↦λ/(λ−1)λ λ/(λ-1), which maps (1,∞)(1,∞) onto itself and corresponds to q↦1−q 1-q. Because Φ−1(q)=−Φ−1(1−q) ^-1(q)=- ^-1(1-q) and ϕφ is even, (100) gives f(λ−1)=ϕ(Φ−1(1/λ))=ϕ(−Φ−1(1−1/λ))=f(λ),f\! ( λ-1 )\;=\;φ ( ^-1(1/λ) )\;=\;φ (- ^-1(1-1/λ) )\;=\;f(λ), so the efficiency curve is symmetric under that involution, i.e. f(1.5)=f(3)f(1.5)=f(3) and f(1.25)=f(5)f(1.25)=f(5) exactly. An involution of an interval has exactly one fixed point here, λ=λ/(λ−1)⇔λ=2λ=λ/(λ-1) λ=2, and a strictly unimodal function invariant under it must attain its extremum there. Therefore, the multiplier λ⋆=2λ =2 sets the unique optimum. ∎ Appendix E Experiment Configurations E.1 Model Architecture Configurations. We follow Llama architecture with RMSNorm pre-normalization, SwiGLU and RoPE positional embedding for all the LLM pretraining runs in Section 5. We apply GPT2 BPE tokenizer with vocabulary size 50,30450,304. Table 3: Model architecture configurations. Scale dmodeld_model layers heads seq. len. tokens/step 124M 768 12 12 512 32,768 720M 2048 12 16 512 131,072 Infrastructure. All runs use single GH200 nodes (4 GPUs, DDP) on the Clariden cluster, inside an NGC PyTorch container. FineWeb-HQ and FinePhrase datasets are loaded in streaming. E.2 Training Arguments On both 124M and 720M model runs, we apply cosine schedule decaying to 0.01×ηmax0.01× _ with 2,0002,000 warmup steps and weight decay 0.10.1, the optimal learning rate are determined by a grid sweep per-optimizer and per-scale, according to Appendix E.4. The 124M models are trained with a global batch size 64×512=32,76864× 512=32,768 tokens across 4 GH200 GPUs, while the 720M models are trained with a global batch size 256×512=131,072256× 512=131,072 tokens On both standard training and OptiSelect, the models are train from scratch for 128128k steps, consuming 4.2B tokens in total. E.3 Selection Proxy Details The proxy set holds 4,0964,096 densely packed 512512-token sequences split evenly over ARC-Easy, ARC-Challenge, HellaSwag, PIQA, and SciQ, each document rendered as the task prompt concatenated with its correct completion. The proxy gradient ,tg_V,t is recomputed every refresh interval (10001000 steps) on the corresponding model checkpoint. The proxy refresh interval is determined by the ablation of Appendix E.5. E.4 Learning Rate Sweep Switching a run to OptiSelect changes which data the optimizer sees, and could in principle move the optimizer’s best step size with it. If it did, every OptiSelect run would need its own learning-rate search and the cost of using the method would roughly double. This appendix settles the question at 124M, and the answer is what licenses the cheaper protocol used at 720M. LR Sweep on 124M runs On 124M models, we grid-search the learning rate over four geometrically spaced rates across 7 optimizers, on OptiSelect and standard training respectively. Table 4: Downstream loss (“dwn”) and FineWeb Validation loss (“val”) across a sweep of learning rates at 124M. Bold marks the optimal learning rate on the downstream loss, which are taken for experiments. Optimizer Mode val / dwn at η=0.25, 0.5, 1, 2×η=\0.25,\,0.5,\,1,\,2\× default AdamW std 3.692 / 3.963 3.610 / 3.865 3.579 / 3.840 3.642 / 3.902 sel 3.733 / 3.850 3.650 / 3.771 3.616 / 3.745 3.949 / 4.080 AdEMAMix std 3.680 / 3.950 3.617 / 3.885 3.599 / 3.863 4.206 / 4.446 sel 3.724 / 3.855 3.659 / 3.780 3.635 / 3.763 4.484 / 4.646 Sophia std 3.796 / 4.062 3.687 / 3.947 3.647 / 3.908 3.730 / 3.979 sel 3.828 / 3.950 3.724 / 3.856 3.679 / 3.804 3.737 / 3.891 D-Muon std 3.777 / 4.043 3.658 / 3.931 3.607 / 3.870 3.622 / 3.877 sel 3.789 / 3.971 3.675 / 3.851 3.620 / 3.794 3.613 / 3.795 Lion std 3.611 / 3.867 3.585 / 3.852 3.603 / 3.862 5.697 / 5.946 sel 3.638 / 3.841 3.609 / 3.834 3.629 / 3.835 5.562 / 5.742 Signum std 3.840 / 4.099 3.740 / 3.996 3.709 / 3.964 3.867 / 4.107 sel 3.858 / 4.059 3.761 / 3.967 3.727 / 3.934 3.819 / 4.009 SGD std 6.996 / 7.190 6.878 / 7.073 6.728 / 6.930 6.558 / 6.763 sel 6.992 / 7.160 6.885 / 7.054 6.722 / 6.905 6.506 / 6.648 Figure 8: Downstream loss in 124M learning-rate sweep. Stars ⋆ mark the optimum for each optimizer. According to Table 4 and Figure 8, we find The optimal learning rate is transferrable on most optimizers. All seven optimizers select the same learning rate under standard training and OptiSelect paradigm according to its downstream performance. Therefore, a run can adopt the optimal learning rate from its standard-training rate in the OptiSelect mode. 720M Learning-Rate Search Given the transferability of learning rate, we swept five optimizers in standard training mode at the full 720M configuration (1616 GPUs, 256256 sequences per step, 22k steps =262=262M tokens). We select the optimum across four grids according to Figure 9 and directly apply the same optimal learning rate on OptiSelect. Figure 9: Downstream loss in 720M learning-rate search. Stars ⋆ mark the optimum for each optimizer. E.5 Proxy Refresh Interval Recomputing ,tg_V,t costs a backward pass over the packed proxy set, so the engine caches it and reuses it for Δ steps. This trades a per-step cost against a staleness bias: between refreshes the reference gradient is evaluated at parameters the model has since left. We sweep Δ∈500,1000,5000,10000 ∈\500,1000,5000,10000\ on AdamW at its best rate (η=10−3η=10^-3, 1010k steps, everything else held fixed) and report the results in Figure 10, Table 5. Since the cost is essentially free and the benefit saturates by Δ=1000 =1000 (3.71783.7178 vs. 3.71653.7165 nats at Δ=500 =500), we set Δ=1000 =1000 as the default. Figure 10: Ablation on Proxy-refresh Frequency. AdamW at η=10−3η=10^-3, 1010k steps, varying only the refresh interval Δ ; stars mark the best cell. (a) The downstream loss degrades monotonically proxy-fresh frequency; while (b) Held-out FineWeb-HQ loss moves in an opposite way. (c) Selection entropy falls as Δ grows, i.e. a stale reference makes the scores look more decisive while aligning them with a stale target. (d) Step time is flat to within 0.030.03 s (6%6\%): the refresh is amortized over Δ steps and never dominates. Table 5: Proxy refresh sweep, AdamW at η=10−3η=10^-3, 1010k steps at 124M. tstept_step is measured as median wall-clock per optimizer step. refresh Δ (steps) 500 1000 50005000 1000010000 downstream loss ↓ 3.7165 3.7178 3.7421 3.8244 FineWeb-HQ val. loss 3.6265 3.6265 3.6185 3.6052 selection entropy 0.84 0.84 0.81 0.79 tstept_step (s) 0.51 0.50 0.50 0.48 Appendix F Extended Experimental Results We present an extended analysis on the experimental results from section §5, at both 124M and 720M model scales. F.1 Complete scores at 124M Losses. We directly compare across various optimizers under standard training and OptiSelect in terms of average loss across all downstream tasks (Figure 11a,b) and validation loss on a held-out set from FineWeb-HQ (Figure 11c,d). The full loss values are presented in Table 6. Figure 11demonstrates that data selection improves downstream loss with every optimizers, while the ranking of optimizers is largely preserved at 124M, with AdamW and AdEMAMix takeing the leading position. In comparison, Lion and D-Muon falls behind other optimizers in rank despite its improvements on raw downstream performance. However, OptiSelect does not affect the optimizer ranking upon training distribution, i.e. the ranks upon held-out FineWeb-HQ loss are constant between standard training and OptiSelect. Accuracy. Table 7 provides the complete accuracy scores across 9 in-domain and out-of-domain tasks. OptiSelect outperforms standard training with the equal amount of trained tokens in both in-domain and out-of-domain tasks, while falling behind with an equal amount of read tokens (λ×λ× of trained tokens). Across optimizers, AdEMAMix achieves the best accuracy across in-domain tasks, while AdamW achieves the best score on out-of-domain ones. In comparison, Lion underperforms standard baseline in the average in-domain accuracy and Sophia underperforms in the average accuracies of out-of domain tasks. Figure 11: Downstream loss and Held-out Fineweb-HQ loss compared across all optimizers at 720M. (i) Standard training, (i) OptiSelect. Table 6: Downstream loss and Held-out Fineweb-HQ loss on 124M models. Held-out FineWeb-HQ loss↓ Downstream loss↓ Optimizer std@128128k sel@128128k Δ std@128128k sel@128128k Δ AdamW 3.107 3.094 -0.013 3.361 3.241 -0.119 AdEMAMix 3.108 3.093 -0.014 3.410 3.274 -0.135 Sophia 3.198 3.179 -0.019 3.486 3.458 -0.028 Lion 3.133 3.116 -0.016 3.417 3.399 -0.018 D-Muon 3.204 3.195 -0.008 3.455 3.471 +0.015 Table 7: 124M zero-shot accuracy (%) per task. OptiSelect outperforms standard training with equal amount of trained tokens, while falls behind with the equal amount of read tokens (λ×λ× of trained tokens). Bold marks the best performance scores. Red color marks the slots where OptiSelect underperformed. In-distribution Out-of-distribution Optimizer Checkpoint PIQA HSwag ARC-e ARC-c SciQ mean Wino WSC CSQA OBQA mean AdamW std@128128k 64.80 31.83 44.11 23.38 65.40 45.90 51.14 36.54 19.66 28.80 34.03 sel@128128k 64.80 32.56 44.32 25.51 65.20 46.48 52.96 41.35 19.74 29.00 35.76 sel@6464k 62.08 30.56 42.93 23.72 62.40 44.34 50.91 36.54 19.49 28.00 33.73 AdEMAMix std@128128k 63.71 31.98 43.01 25.51 65.40 45.92 50.83 36.54 19.57 29.40 34.09 sel@128128k 65.18 32.50 43.81 25.26 66.80 46.71 54.14 36.54 20.64 26.60 34.48 sel@6464k 62.51 29.91 41.88 23.46 61.80 43.91 51.38 36.54 19.57 29.00 34.12 Sophia std@128128k 62.95 31.27 41.71 24.32 58.80 43.81 49.64 38.46 19.66 27.80 33.89 sel@128128k 62.40 31.26 43.22 24.15 62.80 44.77 50.28 36.54 19.57 26.40 33.20 sel@6464k 62.35 29.58 41.12 25.34 60.20 43.72 50.59 36.54 19.57 28.00 33.68 Lion std@128128k 63.44 32.39 43.18 23.21 65.00 45.44 50.75 36.54 19.49 27.60 33.60 sel@128128k 63.71 32.10 42.47 25.51 62.20 45.20 51.46 37.50 19.57 28.00 34.13 sel@6464k 61.86 30.17 40.99 24.23 59.70 43.39 52.41 36.54 19.57 27.60 34.03 D-Muon std@128128k 64.58 32.15 41.96 24.15 62.60 45.09 52.88 36.54 19.74 25.00 33.54 sel@128128k 63.00 32.62 42.47 24.06 64.20 45.27 50.99 40.38 19.66 28.00 34.76 sel@6464k∗ 61.15 29.84 40.19 23.63 60.00 42.97 49.57 38.46 19.57 27.20 33.70 F.2 Complete scores at 720M Losses. According to Figure 12 and Table 8. At 720M, D-Muon ranks at the first under standard training while falling behind of AdamW and AdEMAMix with OptiSelect in downstream losses. Whereas for held-out FineWeb-HQ loss, Muon keeps its leading position while AdamW, AdEMAMix and Lion demonstrate a degradation compared with standard training. Accuracy. Table 9 provides the complete accuracy scores across 9 in-domain and out-of-domain tasks at 720M. All the optimizers outperform standard training in in-domain accuracies, while Muon and AdEMAMix underperform in average out-of-domain accuracy. Figure 12: Downstream loss and Held-out Fineweb-HQ loss compared across all optimizers at 720M. (i) Standard training, (i) OptiSelect. Table 8: Downstream loss and Held-out Fineweb-HQ loss on 720M models. Held-out FineWeb-HQ Downstream probe Optimizer std@107107k sel@107107k Δ std@107107k sel@107107k Δ AdamW 2.704 2.714 +0.010 2.995 2.891 -0.104 AdEMAMix 2.705 2.710 +0.005 3.004 2.908 -0.096 Sophia 2.745 2.736 -0.009 3.049 3.026 -0.023 Lion 2.729 2.731 +0.002 3.018 3.022 +0.004 D-Muon 2.696 2.684 -0.012 2.993 2.944 -0.049 Table 9: 720M zero-shot accuracy (%), per task. In-distribution Out-of-distribution Optimizer Checkpoint PIQA HSwag ARC-e ARC-c SciQ mean Wino WSC CSQA OBQA mean AdamW std@107107k 69.45 46.03 53.89 28.37 75.85 54.72 50.87 45.19 19.49 32.50 37.01 sel@107107k 70.51 47.05 56.44 28.75 76.30 55.81 52.88 36.54 20.72 33.00 35.79 sel@53.553.5k 68.72 41.35 52.31 27.39 73.00 52.55 49.96 36.54 19.74 30.20 34.11 AdEMAMix std@107107k 69.29 46.04 54.19 27.60 74.50 54.32 52.49 42.31 19.57 32.60 36.74 sel@107107k 70.13 47.07 55.43 27.82 76.00 55.29 52.80 54.81 18.59 34.00 40.05 sel@53.553.5k 68.61 42.27 51.52 27.13 73.40 52.59 51.22 59.62 20.07 32.40 40.83 Sophia std@107107k 69.04 44.14 52.21 26.96 73.65 53.20 50.91 37.50 21.09 32.90 35.60 sel@107107k 69.59 45.41 53.41 28.58 75.00 54.40 50.99 47.12 21.29 32.20 37.90 sel@53.553.5k 67.68 41.66 52.48 27.13 71.20 52.03 51.38 36.54 20.88 31.40 35.05 Lion std@107107k 69.94 44.95 53.85 27.69 72.30 53.75 50.59 36.54 20.84 30.70 34.67 sel@107107k 71.11 45.38 52.90 27.30 74.40 54.22 51.93 42.31 20.23 32.60 36.77 sel@53.553.5k 67.85 41.62 48.57 26.54 70.80 51.07 50.67 38.46 20.31 30.40 34.96 D-Muon std@107107k 70.67 47.49 54.84 28.24 73.55 54.96 52.92 58.65 20.43 32.50 41.13 sel@107107k 70.57 49.91 55.51 29.52 76.60 56.42 54.06 45.19 20.48 33.60 38.33 sel@53.553.5k 67.57 41.09 47.64 26.79 70.30 50.68 52.96 36.54 19.25 31.00 34.94 Appendix G Data Selection under Rephrasing G.1 Details on the Downstream Tasks We categorize all five downstream tasks into two groups according their probing formats: (1) QA-formatted: A task is QA-formatted if the prompt poses an explicit question and terminates with an answer slot, so the model is scored on the Question-Answering ability; (2) completion-formatted A task is is completion-formatted if it feeds a plain textual stem to the model, and every option is a continuation of it. The model is scored on its ability on continuation. According to the evaluation harness on each task, we categorize ARC-Easy, ARC-Challenge, PIQA, SciQ, CommonSense QA, Openbook QA as QA-formatted; and HellaSwag, WinoGrande, WSC are completion-formatted. G.2 Extended Results on Rephrasing We present the complete results on the OptiSelect × Format rephrasing experiments in §5.3. According to Table 10, OptiSelect consistently outperforms standard training across various rephrasing mixtures in terms of average accuracy on in-domain proxy tasks and out-of-domain benchmarks. Notably, both standard training and OptiSelect achieve the best performance across all 9 tasks at 50%50\% rephrasing rates, which mixes original FineWeb and FinePhrase documents by 1:11:1 ratio. Table 10: Format rephrasing × OptiSelect, zero-shot accuracy (%). In-Domain Out-of-Domain Rephrased Mode PIQA HSwag ARC-e ARC-c SciQ mean Wino WSC CSQA OBQA mean overall-mean 0%0\% standard 64.74 29.59 47.81 20.65 75.10 47.58 51.14 36.54 19.66 16.60 30.98 40.20 OptiSelect 64.74 30.01 50.04 20.31 74.10 47.84 52.01 44.23 19.74 17.40 33.35 41.40 50%50\% standard 64.47 29.87 51.94 23.04 75.90 49.04 50.28 38.46 19.57 18.00 31.58 41.28 OptiSelect 64.91 30.13 53.24 23.29 77.00 49.72 52.17 40.38 19.57 17.60 32.43 42.03 100%100\% standard 63.71 30.05 54.00 23.21 75.60 49.31 49.96 36.54 19.49 18.40 31.10 41.22 OptiSelect 64.91 30.23 54.38 22.70 77.70 49.98 52.17 36.54 19.74 19.40 31.96 41.97