Paper deep dive
Relative Generalization Invariance of LLM Pretraining
Fengzhuo Zhang, Shuche Wang, Shenggui Li, Tianyu Ruan, Jianliang He, Ivor Tsang, Tianyu Pang, Chao Du, Tianwei Zhang, Zhuoran Yang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and training data stream. However, how these components influence performance in distinct ways remains unclear. We take a first step toward isolating their effects by studying relative generalization. We introduce Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models. We show that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately uniform shift in token-wise losses. In contrast, changing the training data stream can substantially alter relative generalization. We further show that RGI cannot be explained by the neural tangent kernel or mean-field regimes alone and prove that it can emerge in an overparameterized quadratic model. Overall, our work identifies RGI as a new phenomenon in LLM pretraining that helps distinguish the effects of optimizers and architectures from those of training data.
Tags
Links
- Source: https://arxiv.org/abs/2609.33016v1
- Canonical: https://arxiv.org/abs/2609.33016v1
Trouble viewing inline? Open PDF directly →
Full Text
140,777 characters extracted from source content.
Expand or collapse full text
Relative Generalization Invariance of LLM Pretraining Fengzhuo Zhang Affiliation: Yale University Shuche Wang Affiliation: National University of Singapore Shenggui Li Affiliation: Nanyang Technological University Tianyu Ruan Affiliation: Yale University Jianliang He Affiliation: Yale University Ivor Tsang Affiliation: A*STAR Tianyu Pang Affiliation: Sea AI Lab Chao Du Affiliation: Sea AI Lab Tianwei Zhang Affiliation: Nanyang Technological University Zhuoran Yang Affiliation: Yale University Abstract Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and training data stream. However, how these components influence performance in distinct ways remains unclear. We take a first step toward isolating their effects by studying relative generalization. We introduce Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models. We show that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately uniform shift in token-wise losses. In contrast, changing the training data stream can substantially alter relative generalization. We further show that RGI cannot be explained by the neural tangent kernel or mean-field regimes alone and prove that it can emerge in an overparameterized quadratic model. Overall, our work identifies RGI as a new phenomenon in LLM pretraining that helps distinguish the effects of optimizers and architectures from those of training data. †footnotetext: ∗ Equal contribution.†footnotetext: † Project Lead.†footnotetext: Correspondence to: <fengzhuo.zhang@yale.edu>,<zhuoran.yang@yale.edu> 1 Introduction (a) Training losses of optimizers and architectures (b) Relative Generalization Invariance Figure 1: rgi emerges across optimizers and architectures and is not explained by the ntk or mf regimes alone. Panel (a) shows that training-loss curves across optimizers and architectures exhibit similar shapes after 2,0002,000 steps. Panel (b) shows that rgi persists under substantial feature learning and at reduced model widths, providing evidence beyond ntk- and mf-based explanations. Pretraining equips llm with broad knowledge and reasoning capabilities and provides the foundation for subsequent instruction tuning and reinforcement learning. Its performance is jointly shaped by the three components of the training triplet: the optimizer, model architecture, and training data stream. Prior work has improved pretraining performance and efficiency by advancing each component, including new optimizers such as Soap (Vyas et al., 2024) and Muon (Jordan et al., 2024), architectural designs such as swa (swa) (Xiao et al., 2024) and SiLU (Elfwing et al., 2018), and data pipelines such as dynamic data selection (Wang et al., 2026). Despite this progress, much less is known about how the optimizer, model architecture, and training data stream affect performance in distinct ways. We take a step toward isolating their effects by studying a fundamental structure of generalization, which we call relative generalization: the relative loss structure across validation tokens. We show that optimizers and architectures affect token-wise losses in an approximately uniform way, preserving relative generalization across a wide range of optimizers and moderate architectural variations, a phenomenon we term rgi (rgi). In contrast, changing the training data stream can substantially alter relative generalization. Our primary contributions are as follows: First, we define rgi between models as the invariance of the validation-loss difference between any two tokens across models. We show that rgi is equivalent to the two models having a token-independent constant gap in validation loss, i.e., a uniform shift across tokens. We then extend rgi to training triplets through the models they produce. Second, we show that rgi holds across a wide range of optimizers and moderate architectural variations, but depends strongly on the training data stream. Specifically, we consider coordinate-wise optimizers, such as Adam, matrix-based optimizers, such as Muon, and variations in attention and ffn (ffn) designs. We show that rgi persists under joint variations in optimizer and architecture and is robust to optimizer hyperparameters and moderate shifts in the validation distribution. However, rgi can break when models are trained on data streams from different sources. In other words, optimizer and architecture choices induce approximately uniform shifts in token-wise losses, whereas changes in the training data stream can alter the relative loss structure across tokens. Third, we show that rgi cannot be explained by the ntk (ntk) or mf (mf) regimes alone. When rgi emerges, models deviate substantially from initialization in both parameters and hidden-state representations, and different optimizers lead to low similarity in both quantities. These findings rule out an ntk-style lazy-learning explanation. We further reduce the model width and show that rgi persists even when the model departs from the mf regime. Finally, we study a stylized quadratic model and theoretically characterize the emergence of rgi. Beyond the ntk and mf regimes, we identify a setting where rgi holds across GD, Adam, and Muon, different initializations, and learning rates, despite low parameter similarity. Together, these results take a first step toward distinguishing the effects of optimizers, architectures, and data through the definition, empirical demonstration, and theoretical characterization of rgi. 2 Related Work Optimizers of llm play a central role in pretraining and can substantially affect the resulting model capabilities, which are further shaped by post-training (Yue et al., 2025; Liu et al., 2025). Over the past decade, a large body of work has developed and refined optimizers to improve the performance, stability, and efficiency of large-scale training. Starting from (Stochastic Gradient Descent) SGD (Robbins and Monro, 1951), adaptive coordinate-wise optimizers were introduced to better adapt updates to the optimization landscape, including AdaGrad (Duchi et al., 2011), RMSProp (Tieleman and Hinton, 2012), AdaDelta (Zeiler, 2012), and Adam (Kingma and Ba, 2014). Owing to its strong empirical performance in large-scale training, Adam became the default optimizer for many llm pretraining pipelines. Subsequent works further improved Adam through techniques such as Nesterov momentum (Dozat, 2016), modified momentum schemes (Ma and Yarats, 2018), bias correction (Reddi et al., 2019), adaptive step sizes (Zhuang et al., 2020), and decoupled weight decay (Loshchilov and Hutter, 2019). To better exploit the matrix structure of llm parameters, matrix-based optimizers were later proposed, including Muon (Jordan et al., 2024), Shampoo (Gupta et al., 2018), Soap (Vyas et al., 2024), Dion (Ahn et al., 2025), and Scion (Pethick et al., 2025). Following the introduction of Muon, several recent works further improved its performance through alternative normalization strategies and adaptive designs (Li et al., 2025b; Cheng et al., 2026; Shan et al., 2026; Pang et al., 2026; Deng et al., 2026; Zhang et al., 2026c; Southworth and Thomas, 2026; Zhang et al., 2026b; Liu et al., 2026; Zhang et al., 2026a). Relatedly, another line of work seeks to optimize parameters by approximating Gauss–Newton updates (Liu et al., 2023; Abreu et al., 2025). Orthogonal to these directions, some works focus primarily on reducing the memory and communication cost of training, including Adafactor (Shazeer and Stern, 2018), Adam-mini (Zhang et al., 2024), GaLore (Zhao et al., 2024a), Apollo (Zhu et al., 2025), and MuonBP (Khaled et al., 2025). Comprehensive surveys of llm optimizers are provided by He et al. (2021); Wan et al. (2023); Abdulkadirov et al. (2023); Zhou et al. (2024); Luo et al. (2025); Tian et al. (2025); Xiao et al. (2025). In contrast to these works, which propose new optimizers, we study the shared characteristics of existing optimizers. Specifically, we show that many commonly used optimizers exhibit the same relative generalization behavior. Influential Factors in llm Pretraining include model architecture, datasets, and optimizers. Following the seminal works on scaling laws (Kaplan et al., 2020; Rosenfeld et al., 2019), a growing body of work has studied how these three factors separately and jointly affect pretraining performance. From the perspective of model design, Hoffmann et al. (2022); Tao et al. (2024) study the optimal allocation of compute and the role of vocabulary size as model size scales. Bian et al. (2025) further studies the trade-off between throughput and performance across different architectural configurations. From the perspective of optimization, Zhao et al. (2024b); Semenov et al. (2025); Wen et al. (2025) examine how different optimizers affect training efficiency and model performance across different data and model-size regimes. In addition, Volkova et al. (2026) proposes a robust method for fitting scaling laws under different optimizers. A large body of work also studies how datasets influence pretraining performance. Hernandez et al. (2021); Ye et al. (2024); Mizrahi et al. (2025); Liu et al. (2025); Shukor et al. (2025); Subramanyam et al. (2025); Mayilvahanan et al. (2025) investigate how pretraining data mixtures affect target-task and downstream performance through the lens of scaling laws. In contrast, Kang et al. (2025) and Zhao et al. (2024c) study properties of data mixtures using human-controlled synthetic data and machine unlearning, respectively. Beyond these mechanism-oriented studies, another line of work investigates how to construct near-optimal data mixtures efficiently for large-scale pretraining, including DoReMi (Xie et al., 2023), RegMix (Liu et al., 2024), and Chameleon (Xie et al., 2025). See Li et al. (2025a) for a survey of scaling laws across different pretraining factors and Luo et al. (2025) for a survey of data-centric llm pretraining. Different from these works, we study the distinct roles of optimizers and data in shaping relative generalization, which holds almost surely with respect to the data stream. Pretraining Dynamics of llm have received considerable attention over the past decade, and these studies have shed light on the behavior of different optimizers throughout training. Systematic theoretical studies of pretraining dynamics typically focus on over-parameterized regimes, in which the number of trainable parameters is much larger than the number of training samples (Du and Lee, 2018; Allen-Zhu et al., 2019). Among these settings, two regimes have been studied most extensively: the ntk regime and the mf regime. The ntk regime is also known as the lazy-training regime (Chizat et al., 2019), in which the kernel induced by the network remains effectively fixed throughout training (Jacot et al., 2018; Lee et al., 2019). Under suitable scalings of model width, initialization, and learning rate, a line of work establishes convergence rates to global optima in the ntk regime (Arora et al., 2019b; Novak et al., 2019; Geiger et al., 2020; Liu et al., 2020). In addition, Arora et al. (2019a) explicitly derives the kernel on the training data and shows how convergence depends on its properties. See Golikov et al. (2022) for a survey of the ntk regime. To relax the restrictive assumption that the kernel remains fixed during training, the mf regime enables theoretical analysis of an evolving kernel via mean-field approximation (Mei et al., 2018; Sirignano and Spiliopoulos, 2020; Takakura and Suzuki, 2024). Rather than tracking each neuron individually, the mean-field approximation represents neurons collectively through a distribution. This framework was developed rigorously by Mei et al. (2019) and Sirignano and Spiliopoulos (2020) for two-layer networks and later extended to multi-layer networks (Sirignano and Spiliopoulos, 2022). A line of work further extends mean-field analysis from fully connected networks to transformers (Kim and Suzuki, 2024; Rigollet, 2025; Poc-López and Aguilera, 2024). Motivated by mean-field analysis, μ was proposed to preserve nontrivial parameter updates in the infinite-width limit (Yang and Hu, 2021; Yang et al., 2021). Its analysis supports hyperparameter transfer across models of different sizes (Lingle, 2024; Blake et al., 2024; Hayou, 2025; Zheng et al., 2026). Different from these works, the relative-generalization invariance behavior of optimizers studied by our work is not explained by either the ntk or the mf perspective. 3 Preliminaries Pretraining of llm optimizes model parameters over a stream of data sampled from a corpus. Let the initial parameters θ0∈Θ _0∈ be drawn from an initialization distribution Pinit∈Δ(Θ)P_init∈ ( ). Starting from θ0 _0, the model is trained on a data stream D1:∞=(Dt)t=1∞D_1:∞=(D_t)_t=1^∞, where each batch Dt⊆D_t is sampled from the corpus D according to a possibly time-dependent distribution Pt(⋅∣D1:t−1)P_t(· D_1:t-1), independent of θ0 _0. Each data point d∈Dtd∈ D_t consists of a prefix and a target token, e.g., d=(CLOSEd=(“LLM pretraining is”, “important”)). Given an optimizer ∈ A , we define the training triplet as =(,θ0,D1:∞) T=( A, _0,D_1:∞), where θ0 _0 specifies the initialization and its associated model architecture. The parameters at step t can then be written as θt=F(,t), _t T=F( T,t), where F is deterministic given the training triplet.† Our work focuses on deterministic optimizers. Randomized optimizers can be incorporated by treating the algorithmic randomness as an additional input to F (Mandt et al., 2017; Jin et al., 2017). The map F is induced by the optimizer’s update rule. For a batch D⊆D , let L(θ,D)=|D|−1∑d∈DL(θ,d)L(θ,D)=|D|^-1 _d∈ DL(θ,d) denote the average cross-entropy loss, and define the batch gradient as ξt=∇θL(θt−1,Dt). _t T= _θL( _t-1 T,D_t). The optimizer A recursively updates its internal state St∈S_t T and the model parameters according to St=G(St−1,ξt,),θt=H(St,θt−1,), S_t T=G(S_t-1 T, _t T, A), _t T=H(S_t T, _t-1 T, A), where G and H denote the state-update and parameter-update maps, respectively. Optimizers instantiate the update maps G and H, and hence the training map F, differently. Coordinate-wise optimizers, such as Adam (Kingma and Ba, 2014), Lion (Chen et al., 2023), and Adam-mini (Zhang et al., 2024), use entry-wise gradient statistics. For the training triplet =(,θ0,D1:∞) T_ Adam=( Adam, _0,D_1:∞), Adam maintains first- and second-moment states St=St=(mt,vt)∈S_t Adam=S_t T_ Adam=(m_t,v_t) _ Adam and updates mt=γ1mt−1+(1−γ1)ξt,vt=γ2vt−1+(1−γ2)ξt⊙ξt,θt=θt−1−ηmt⊙(vt+ε)−1, m_t= _1m_t-1\!+\!(1- _1) _t,v_t= _2v_t-1\!+\!(1- _2) _t _t, _t Adam= _t-1 Adam\!-\!η\,m_t ( v_t+ )^-1, where ⊙ denotes the Hadamard product and γ1,γ2,η,ε _1, _2,η, are hyperparameters; bias correction is omitted for simplicity. Here and below, we abbreviate StS_t T_ Adam and θt _t T_ Adam as StS_t Adam and θt _t Adam. In contrast, matrix-based optimizers, such as Muon (Jordan et al., 2024), Soap (Vyas et al., 2024), Scion (Pethick et al., 2025), and Shampoo (Gupta et al., 2018), exploit the matrix structure of parameter blocks. For example, an idealized Muon update for each matrix-shaped block is mt=γmt−1+(1−γ)ξt,θt=θt−1−ηUtVt⊤, m_t=γ m_t-1+(1-γ) _t, _t Muon= _t-1 Muon-η U_tV_t , where St=mtS_t Muon=m_t, UtΣtVt⊤U_t _tV_t is the svd (svd) of mtm_t, and γ,ηγ,η are hyperparameters. In practice, Muon approximates UtVt⊤U_tV_t using Newton–Schulz iterations. See Section 2 for further discussion of these optimizers. 4 Relative Generalization Invariance In this section, we define a new relationship between the validation losses of llm pretrained with different optimizers, model architectures, and data streams. Specifically, consider two training triplets =(,θ0,D1:∞) T=( A, _0,D_1:∞) and ′=(′,θ0′,D1:∞′) T =( A , _0 ,D_1:∞ ), which produce parameters θt _t T and θt′ _t T at step t. We allow θ0 _0 and θ0′ _0 to correspond to different model architectures and, in particular, models of different sizes. In our experiments, (θ0,D1:∞)( _0,D_1:∞) and (θ0′,D1:∞′)( _0 ,D_1:∞ ) are either identical or independently sampled. We compare the resulting models on subsets Dval⊆valD_val _val, with L(θt,Dval)=|Dval|−1∑d∈DvalL(θt,d),L( _t T,D_val)=|D_val|^-1 _d∈ D_valL( _t T,d), and analogously for L(θt′,Dval)L( _t T ,D_val). For generality, we define rgi under an arbitrary joint distribution over the two initializations and training data streams. Definition 4.1 (Relative Generalization Invariance). rgi (rgi) holds between two parameters θ,θ′∈Θθ,θ ∈ over valD_val if L(θ,d)−L(θ,d′)=L(θ′,d)−L(θ′,d′) for any d,d′∈val. L(θ,d)-L(θ,d )=L(θ ,d)-L(θ ,d ) for any d,d _val. (4.1) For two training triplets =(,θ0,D1:∞) T=( A, _0,D_1:∞) and ′=(′,θ0′,D1:∞′) T =( A , _0 ,D_1:∞ ) and their parameters at step t, θt=F(,t) _t T=F( T,t) and θt′=F(′,t) _t T =F( T ,t), we say that rgi holds across T and ′ T over valD_val at time step t if L(θt,d)−L(θt,d′)=L(θt′,d)−L(θt′,d′) L ( _t T,d )-L ( _t T,d )=L ( _t T ,d )-L ( _t T ,d ) (4.2) for any d,d′∈vald,d _val, almost surely with respect to the joint distribution of two initializations and training data streams. We say that argi (argi) holds between training triplets T and ′ T if there exists t∗>0t^*>0 such that rgi holds between them for any t≥t∗t≥ t^*. The notion of rgi between parameters requires that the validation-loss difference between any two tokens d and d′d is the same under θ and θ′θ . Thus, θ and θ′θ preserve the same relative token-wise loss structure; in particular, if token d′d incurs a higher loss than token d under θ, then the same holds under θ′θ . At the level of training triplets, rgi between T and ′ T at step t requires this property to hold between the resulting parameters almost surely with respect to the joint randomness in their initializations and training data streams. Equivalently, their token-wise validation losses differ only by a token-independent constant. This characterization is formalized in the following result. Proposition 4.2 (Equivalent Statements of rgi). The following statements are equivalent. • (rgi) rgi holds between training triplets T and ′ T at time step t. • (Constant Generalization Gap) The training triplets T and ′ T have a constant generalization gap at time t, i.e., L(θt,d)−L(θt′,d)=C(,′,t) for any d∈val almost surely.L( _t T,d)-L( _t T ,d)=C( T, T ,t) for any d _val almost surely. This result shows that rgi is equivalent to a token-independent constant gap in validation loss, motivating two complementary metrics for evaluating argi. First, motivated by the constant-gap characterization, we evenly partition valD_val into K non-overlapping subsets (Dval,k)k=1K(D_val,k)_k=1^K and compute the c (c) between the centered values of L(θt,Dval,k)L( _t T,D_val,k) and L(θt′,Dval,k)L( _t T ,D_val,k) across k (Lin, 1989), denoted by rCCC(,′)∈[−1,1]r_C( T, T )∈[-1,1]. Larger rCCCr_C indicates closer agreement with rgi, with rCCC=1r_C=1 implying that the two sets of validation losses differ only by a constant shift. Second, we define the loss difference Δ(,Dval,Dval′)=L(θt,Dval)−L(θt,Dval′) ( T,D_val,D_val )=L( _t T,D_val)-L( _t T,D_val ). When |Dval|=|Dval′|=1|D_val|=|D_val |=1, rgi is equivalent to Δ(,Dval,Dval′)=Δ(′,Dval,Dval′) ( T,D_val,D_val )= ( T ,D_val,D_val ). We define rdiff(,′)r_diff( T, T ) as the normalized discrepancy between the corresponding loss differences under T and ′ T . Smaller rdiffr_diff indicates closer alignment with rgi; when each subset contains a single token, rdiff=0r_diff=0 implies exact rgi. These metrics are normalized to facilitate the evaluation of rgi; see Appendix C for further discussion. Their explicit definitions are provided in Appendix A. 5 Empirical Studies of Relative Generalization Invariance In this section, we investigate when rgi holds between two training triplets T and ′ T . Specifically, we examine four sources of variation in L(θt,Dval)L( _t T, [rgb]0,0.5,0.5D_val), where =(,θ0,D1:t) T=( [rgb]0.0352,0.1875,0.543 A, [rgb]0.9922,0.7031,0.1992 _0, [rgb]0.9883,0.3555,0.3516D_1:t): the optimizer [rgb]0.0352,0.1875,0.543 A, the model architecture and initialization associated with θ0 [rgb]0.9922,0.7031,0.1992 _0, the training data stream D1:t [rgb]0.9883,0.3555,0.3516D_1:t, and the validation data Dval [rgb]0,0.5,0.5D_val. (a) Validation losses of Adam and Muon (b) Values of rCCCr_C across training (c) Values of rdiffr_diff across training Figure 2: Empirical evidence of rgi among optimizers. Panel (a) plots the validation losses of the 124124M GPT models trained with Adam and Muon at the 10,00010,000-step checkpoint, evaluated on 20,48020,480 validation batches. Panel (b) reports rCCC(Adam,⋅)r_C(Adam,·) (↑)( ) for the other five optimizers over training, all of which rise above 0.950.95 after 2,0002,000 steps. Panel (c) reports rdiff(Adam,⋅)r_diff(Adam,·) (↓)( ) for the other five optimizers over training, all of which drop below 0.050.05 after 2,0002,000 steps. These results show that rgi holds across all considered optimizers after sufficient training when they are trained from the same initialization and on the same training data stream. Experimental setup. We first describe the baseline experimental setting, from which subsequent experiments vary the optimizer, model architecture, and training data. We consider three coordinate-wise optimizers, Adam (Kingma and Ba, 2014), Lion (Chen et al., 2023), and Adam-mini (Zhang et al., 2024), and three matrix-based optimizers, Soap (Vyas et al., 2024), Muon (Jordan et al., 2024), and Scion (Pethick et al., 2025). We use them to train 124124M, 0.70.7B, and 4.54.5B Nano-GPT models on FineWeb (Penedo et al., 2024), with RoPE (Su et al., 2024) in attention and GELU (Hendrycks and Gimpel, 2016) in the ffn. The validation set valD_val is drawn from FineWeb but disjoint from the training set, so these experiments evaluate in-distribution generalization; the ood (ood) setting is studied in Section 5.4. Within each model setting, all optimizers share the same initialization and training data stream. Following Hu et al. (2024); Wen et al. (2024), we use the Warmup-Stable-Decay learning-rate scheduler and tune the learning rate separately for each optimizer. We report the 124124M results in the main text and defer the 0.70.7B and 4.54.5B results, which lead to the same conclusions, to Appendix B. Across 1010 random seeds, we report the worst-case maximum rdiffr_diff and minimum rCCCr_C, providing evidence that rgi is robust to training randomness and consistent with the almost-sure requirement in Definition 4.1. Additional experimental details are provided in Appendix A. 5.1 rgi Holds Across Optimizers In this section, we vary the optimizer and its hyperparameters while fixing the model initialization and training data stream, and show that argi approximately holds across optimizers. For simplicity, we abbreviate rdiff(,⋅)r_diff( T_ Adam,·) and rCCC(,⋅)r_C( T_ Adam,·) as rdiff(,⋅)r_diff( Adam,·) and rCCC(,⋅)r_C( Adam,·), respectively. Figure 2 provides empirical evidence for approximate rgi across optimizers with identical initial parameters and the same training data stream. Figure 2(a) shows that, after 10,00010,000 training steps, the validation losses of models trained with Muon and Adam across validation batches are well fitted by y=x−0.07y=x-0.07, indicating a nearly constant generalization gap. Results for the other optimizers are deferred to Appendix B and yield the same conclusion. Taking Adam as the baseline, Figures 2(c) and 2(b) further show that approximate rgi emerges across optimizers after 2,0002,000 training steps, with rdiff<0.05r_diff<0.05 and rCCC>0.95r_C>0.95, as marked by the red dotted lines. Results using other optimizers as the baseline are reported in Appendix B and are consistent with this finding. Overall, these results suggest that argi emerges across optimizers on the in-distribution validation set after the early stage of training when each optimizer uses its best learning rate, while the remaining hyperparameters are set to their defaults and the initialization and training data stream are shared. We next examine whether rgi continues to hold under variations in optimizer hyperparameters, including learning rates, momentum coefficients, learning-rate schedulers, and weight decay. To avoid confounding changes in both models being compared, we keep Adam fixed under the setting in Figure 2 and vary only the hyperparameters of the other optimizers. We first vary the learning rate of each optimizer A over 0.25η0.25η A, 0.5η0.5η A, 0.75η0.75η A, and 1.25η1.25η A, where η A denotes its best learning rate. We exclude larger multipliers, such as 1.5η1.5η A, because some optimizers diverge under these settings. Figures 3(a) and 3(b) report the rCCCr_C results for 0.25η0.25η A and 1.25η1.25η A, while the remaining results are deferred to Appendix B. The different times at which rCCCr_C crosses the threshold 0.950.95 show that the learning rate affects when approximate rgi emerges. Nevertheless, rCCCr_C eventually grows above 0.950.95 for all tested learning rates. We next vary the momentum coefficient γ over 0.850.85, 0.90.9, and 0.950.95. Figures 3(c) and 3(d) show that argi persists across these momentum values, with the γ=0.9γ=0.9 and corresponding rdiffr_diff results deferred to Appendix B. We further ablate the learning-rate scheduler and weight decay in Appendix B, obtaining the same conclusion. Overall, argi is robust across the tested optimizer hyperparameters, although these hyperparameters can affect when it emerges. (a) rCCCr_C for 0.25η0.25η A. (b) rCCCr_C for 1.25η1.25η A. (c) rCCCr_C for γ=0.85γ=0.85. (d) rCCCr_C for γ=0.95γ=0.95. Figure 3: rgi across optimizers under different hyperparameter settings. Panels (a) and (b) plot rCCCr_C under different learning-rate multipliers, while panels (c) and (d) plot rCCCr_C under different momentum values. rgi persists across all tested learning-rate and momentum settings. 5.2 rgi Holds Across Moderately Different Architectures In this section, we study whether argi holds across model architectures. Consistent with Section 5.1, we train models with different architectures using different optimizers, thereby examining argi under joint variations in optimizer and architecture. Specifically, we consider two settings: (i) models with the same architectural components but different initializations and model sizes, and (i) models with different architectural components. We now examine whether rgi persists across different initializations and model sizes while keeping the architectural components fixed. First, we train llm from independent initializations θ0,θ0′∈Θ _0, _0 ∈ , drawn from the same initialization distribution, using =Adam A=Adam and ′ A chosen from Adam-mini, Lion, Muon, Soap, and Scion. Figure 4(a) shows that, unlike the shared-initialization setting in Figure 2, rCCCr_C initially deviates substantially from rgi; nevertheless, approximate rgi emerges after about 2,0002,000 training steps. Second, we vary the model size from the baseline hidden-state dimension 768768 to 384384 and 6464. Figures 4(b) and 4(c) show that rgi continues to hold between models with dimensions 768768 and 384384, but fails when the dimension is reduced to 6464. The corresponding rdiffr_diff results in Appendix B lead to the same conclusion. (a) rCCCr_C under different initial parameters. (b) rCCCr_C for hidden state dimensions 768768 and 384384. (c) rCCCr_C for hidden state dimensions 768768 and 6464. Figure 4: rgi across optimizers, initializations, and model sizes. Panel (a) plots rCCCr_C across optimizers with different initializations. Panels (b) and (c) plot rCCCr_C across models with hidden-state dimensions 768768, 384384, and 6464. rgi continues to hold across different initializations and moderate changes in model size. We next examine whether rgi holds between models with different architectural components. First, we ablate the attention design. While our previous experiments use fa (fa) with RoPE (Su et al., 2024), we additionally consider fa with NoPE (Kazemnejad et al., 2023), ALiBi (Press et al., 2021), and swa with RoPE. We train each variant with different optimizers and compare it with the baseline RoPE fa model trained with Adam. Figures 5(a)–5(c) report the corresponding rCCCr_C values and show that rgi continues to hold across different attention designs and optimizers. Second, we ablate the ffn design. While the baseline uses GELU (Hendrycks and Gimpel, 2016), we additionally consider Squared-ReLU (SReLU) (Klusowski and Barron, 2016) and SiLU (Elfwing et al., 2018). We train both variants with all optimizers and compare them with the baseline GELU model trained with Adam. Figure 5(d) reports the SReLU results, while the SiLU results and additional comparisons between gated and non-gated ffn are deferred to Appendix B; all exhibit argi. Together, these results show that rgi persists across different attention and ffn designs as well as optimizers. (a) Values of rCCCr_C between fa models with RoPE and NoPE. (b) Values of rCCCr_C between fa models with RoPE and ALiBi. (c) Values of rCCCr_C between fa and swa models. (d) Values of rCCCr_C between models with GELU and SReLU. Figure 5: rgi across optimizers and models with different attention and ffn modules. Panels (a)–(c) plot rCCCr_C between models using RoPE fa trained with Adam and models using NoPE fa, ALiBi fa, and RoPE swa, each trained with all optimizers, respectively. Panel (d) plots rCCCr_C between models using GELU and SReLU activations. rgi continues to hold across different attention and ffn designs. In conclusion, argi persists under joint variations in optimizers and moderate architectural changes, though these choices can affect its emergence time. Shared Training-Loss Shapes across Optimizers and Architectures Arise from rgi. Training llm with different optimizers and architectures on the same data stream often yields training-loss curves with similar shapes. As shown in Figure 1(a), after 2,0002,000 steps, all optimizers exhibit nearly the same training-loss profile. We show that this phenomenon can be explained by approximate rgi. The training loss Figure 6: Values of |αt|| _t T| and |βt|| _t T|. It shows that |βt|| _t T| is much larger than |αt|| _t T| at step t is L(θt−1,Dt)L( _t-1 T,D_t), namely the loss of the parameters trained on D1:t−1D_1:t-1 and evaluated on the current batch DtD_t. Its change between adjacent steps can be decomposed as L(θt,Dt+1)−L(θt−1,Dt)=αt+βt, L ( _t T,D_t+1 )-L ( _t-1 T,D_t )= _t T+ _t T, where αt=L(θt,Dt+1)−L(θt−1,Dt+1) _t T=L( _t T,D_t+1)-L( _t-1 T,D_t+1) captures the loss change induced by one optimizer update, while βt=L(θt−1,Dt+1)−L(θt−1,Dt) _t T=L( _t-1 T,D_t+1)-L( _t-1 T,D_t) captures the loss difference of the same parameters across adjacent batches. Figure 6 shows that |βt|| _t| is approximately 5050 times larger than |αt|| _t|, so the local shape of the training-loss curve is dominated by βt _t. Since θt−1 _t-1 T has not been trained on either DtD_t or Dt+1D_t+1, these batches act as held-out data for θt−1 _t-1 T. Approximate rgi therefore implies that βt _t T is approximately invariant across optimizers and architectures. Consequently, the adjacent-step changes in training loss are nearly the same across optimizers and architectures, explaining their shared training-loss shapes. 5.3 rgi is Strongly Influenced by Pretraining Data Streams (a) Values of rCCCr_C with Dl′D_l from C4. (b) Values of rCCCr_C with Dl′D_l from Arxiv. (c) Values of rCCCr_C with Dl′D_l from CodeParrot. Figure 7: rgi across optimizers trained on different data streams. The panels plot rCCCr_C when Adam is trained on FineWeb and the compared optimizer is trained on C4 (Panel (a)), ArXiv (Panel (b)), or CodeParrot (Panel (c)). The results show that rgi is strongly affected by the training data stream: changing the data stream can substantially weaken, or even eliminate, rgi across optimizers. We next explore whether argi emerges when the training data stream varies, jointly or independently of the optimizer, and show that it can fail even when the optimizer and architecture are held fixed. Specifically, we examine whether rgi emerges between θt=F(,t) _t T_ Adam=F( T_ Adam,t), where =(,θ0,D1) T_ Adam=( Adam, _0,D_1), and θt′=F(′,t) _t T =F( T ,t), where ′=(′,θ0,D1′) T =( A , _0, [rgb]0.9883,0.3555,0.3516D_1 ). Here, D1D_1 is drawn from FineWeb, while D1′ [rgb]0.9883,0.3555,0.3516D_1 is drawn from one of three alternative datasets: C4 (Raffel et al., 2020), the ArXiv subset of RedPajama (Weber et al., 2024), and CodeParrot (CodeParrot, 2022). FineWeb and C4 are both large-scale web-text datasets covering diverse topics and are derived, at least in part, from Common Crawl (Baack, 2024), suggesting relatively high distributional similarity. By contrast, ArXiv consists of academic papers from arXiv, while CodeParrot consists of source code collected from GitHub, making their distributions more distinct from FineWeb. Figure 7 shows that the training data stream strongly affects the emergence of rgi. For C4, Figure 7(a) shows that rCCCr_C first decreases, then rises to a relatively large value, and finally declines again; the intermediate increase may reflect the greater distributional similarity between FineWeb and C4. In contrast, for ArXiv and CodeParrot, Figures 7(b) and 7(c) show that rCCCr_C remains below 0.950.95 throughout training even when the same optimizer Adam is used, indicating that rgi does not emerge across these training data streams. The corresponding rdiffr_diff results in Appendix B show the same trend. Overall, changes in the training data distribution can substantially weaken or even prevent the emergence of rgi. 5.4 rgi Persists under Moderate Validation Shifts (a) Values of rCCCr_C with DvalD_val from C4. (b) Values of rCCCr_C with DvalD_val from ArXiv. (c) Values of rCCCr_C with DvalD_val from CodeParrot. (d) Values of rCCCr_C with DvalD_val from Random. Figure 8: rgi across optimizers and architectures evaluated on different validation datasets. The panels plot rCCCr_C evaluated on C4, ArXiv, CodeParrot, and Random. These results show that rgi can transfer to related validation distributions, but becomes weaker on more distributionally distinct validation datasets. In this section, we ablate the effect of the validation data Dval [rgb]0,0.5,0.5D_val on rgi across optimizers and architectures. For clarity, we focus on fa and swa as representative architectural variants; additional results are provided in Appendix B. All models are trained on FineWeb, while rdiffr_diff and rCCCr_C are evaluated on validation data from other sources. As in Section 5.3, we consider C4, ArXiv, and CodeParrot, and additionally construct a Random dataset by sampling each token i.i.d. uniformly from the vocabulary, yielding an extreme ood setting with no meaningful semantic structure. Figure 8 reports the corresponding rCCCr_C values, comparing swa models trained with various optimizers against the fa baseline trained with Adam. rCCCr_C remains large on C4, while on ArXiv and CodeParrot it is smaller but generally increases as training progresses. In contrast, the Random dataset shows no clear emergence of rgi. These results suggest that validation-distribution shifts weaken argi: moderate shifts preserve the trend, whereas sufficiently extreme shifts can prevent its emergence. The corresponding rdiffr_diff results in Appendix B support the same conclusion. 6 Relationships to the NTK and MF Regimes Section 5 shows that approximate rgi holds across a wide range of optimizers and architectures. We now examine whether this phenomenon can be explained by two well-known theoretical regimes of training dynamics: the ntk and mf regimes. We focus on rgi across optimizers, since different architectures have parameterizations and intermediate representations that are not directly comparable. 6.1 RGI Emerges Beyond the ntk Regime (a) cos-simcos -sim between parameters within optimizers. (b) cos-simcos -sim between hidden states within optimizers. (c) cos-simcos -sim between parameters cross optimizers. (d) cos-simcos -sim between hidden states cross optimizers. Figure 9: Within- and cross-optimizer cos-sim(↑)cos -sim( ) for parameters and hidden states. Panels (a)–(b) compare θ0 _0 T with θt _t T, while Panels (c)–(d) compare θt _t T with θt′ _t T . Hidden-state similarities are averaged over layers l∈[L]l∈[L]. Within-optimizer comparisons show that all optimizers learn nontrivial features throughout training, whereas cross-optimizer comparisons show that different optimizers learn distinct features. In this section, we examine whether approximate rgi across optimizers sharing the same initialization and training data stream can be explained by the ntk regime. In this regime, sufficiently wide models under suitable Gaussian initialization are well approximated by their linearization around initialization, with the induced kernel remaining nearly constant (Jacot et al., 2018; Golikov et al., 2022). This is also known as the lazy-learning regime (Geiger et al., 2020), whose characteristic signature is that parameters and representations remain close to initialization: ‖θt−θ0‖2≪‖θ0‖2\| _t- _0\|_2 \| _0\|_2 and ‖h(l,θt,d)−h(l,θ0,d)‖2≪‖h(l,θ0,d)‖2\|h(l, _t,d)-h(l, _0,d)\|_2 \|h(l, _0,d)\|_2 for relevant t, l∈[L]l∈[L], and d∈vald _val (Du et al., 2018). We test these signatures using cosine similarity and relative ℓ2 _2 difference, cos-sim(x,x′)=x⊤x′/(‖x‖2‖x′‖2)cos -sim(x,x )=x x /(\|x\|_2\|x \|_2) and rℓ2(x,x′)=2|x−x′|2/(‖x‖2+‖x′‖2)r_ _2(x,x )=2|x-x |_2/(\|x\|_2+\|x \|_2), for both parameters and hidden states, comparing them across training steps within each optimizer and across optimizers at the same step. We report cosine-similarity results in the main text and defer the relative-ℓ2 _2 results, which show consistent trends, to Appendix B. Figures 9(a)–9(b) report within-optimizer cosine similarities between initialization and step t for both parameters, cos-sim(θ0,θt)cos -sim( _0 T, _t T), and hidden states, cos-sim(h(l,θ0,d),h(l,θt,d))cos -sim(h(l, _0 T,d),h(l, _t T,d)), averaged over layers l∈[L]l∈[L], validation examples d∈vald _val, and random seeds. After 5,0005,000 training steps, both parameters and hidden states deviate substantially from initialization, suggesting that the observed rgi cannot be explained by the ntk lazy-learning regime alone. We further compare models trained by different optimizers at the same step t. Figures 9(c)–9(d) report the cosine similarities of parameters and hidden states between Adam and the other optimizers, both of which decrease as training proceeds. Thus, approximate rgi cannot be attributed either to ntk-style fixed representations or to different optimizers converging to shared parameters or hidden representations. 6.2 RGI Emerges Beyond the mf Regime (a) Values of rCCCr_C at width 576576. (b) Values of rCCCr_C at width 384384. (c) Values of rCCCr_C at width 192192. (d) Validation losses of different learning rates of Adam with various widths. Figure 10: rgi across models of different widths. Panels (a)–(c) plot rCCCr_C for models with different widths, showing that rgi holds across all tested widths. Panel (d) plots the validation loss of Adam at 10,00010,000 steps across learning rates and widths, showing that the width-192192 model exhibits behavior inconsistent with the mf regime. In this section, we examine whether approximate rgi across optimizers with identical initializations and training data streams can be explained by the mf regime. In this regime, the parameter distribution evolves nontrivially in the infinite-width limit, allowing both parameters and features to change substantially during training (Mei et al., 2018; Mei et al., 2019; Sirignano and Spiliopoulos, 2020). Starting from the 124124M GPT model with hidden dimension 768768 used in Section 5, we progressively reduce the width to 576576, 384384, and 192192, where 192192 is one quarter of the baseline width, and test whether approximate rgi persists. We also grid search the Adam learning rate at each width, using the width dependence of the optimal learning rate as an empirical diagnostic of consistency with the mf regime. Figures 10(a)–10(c) evaluate rgi across optimizers at different fixed widths, unlike Figures 4(b) and 4(c), which compare models across widths. Approximate rgi persists even at width 192192, one quarter of the standard GPT-2 width; the corresponding rdiffr_diff results in Appendix B lead to the same conclusion. To assess whether this narrow model remains consistent with the mf regime, Figure 10(d) plots Adam’s validation loss after 10,00010,000 steps across widths and learning rates, with red stars marking the optimal learning rates. The width-192192 model exhibits a qualitatively different validation-loss–learning-rate profile from the width-768768, 576576, and 384384 models, suggesting that it is no longer well described by the same mf-style approximation. Thus, approximate rgi can persist even beyond the regime where the mf approximation appears adequate. 7 Case Study of Over-Parameterized Quadratic Models Section 6 shows that rgi cannot be explained by ntk or mf regimes alone. We fill this theoretical gap in this section by studying the behavior of GD, Adam, and Muon on a quadratic problem. Given a linear target response y(x)=Hxy(x)=Hx with H∈ℝq×r,x∈ℝrH ^q× r,x ^r and q≥3rq≥ 3r, we adopt a two-layer overparameterized model fA,B(x)=ABxf_A,B(x)=ABx to learn it, where A∈ℝq×s,B∈ℝs×rA ^q× s,B ^s× r with s≥3rs≥ 3r. With the dataset (yi,xi)i=1n\(y_i,x_i)\_i=1^n, we train the parameters A,BA,B via the loss L(A,B)=1/2n⋅∑i=1n‖yi−ABxi‖22.L(A,B)=1/2n· _i=1^n\|y_i-ABx_i\|_2^2. We require the target matrix H to have the following structure. Assumption 7.1. The target matrix H∈ℝq×rH ^q× r satisfies H⊤H=μ2Ir,sign(H)=κH H=μ^2I_r,sign(H)=κ H for some μ>0μ>0 and κ>0κ>0, where the sign map is applied entrywise with sign(0)=0sign(0)=0. This assumption imposes two requirements. First, the columns of H are orthogonal and have the same norm. Second, all nonzero entries of H must have the same magnitude. This assumption is satisfied by a broad class of matrices; for example, the columns of H may form either a one-hot basis or a Hadamard basis. Assumption 7.2. The covariates xii=1n\x_i\_i=1^n are isotropic, i.e., ∑i=1nxixi⊤/n=σx2Ir _i=1^nx_ix_i /n= _x^2I_r for some σx>0 _x>0. This assumption requires the training covariates to cover the feature space approximately isotropically. We emphasize that Assumptions 7.1 and 7.2 define a stylized setting under which rgi can hold across optimizers. In Appendix D, we provide empirical evidence that these assumptions are relevant to practical llm training; relaxing them is left for future work. We then optimize the parameters A and B of our model with three optimizers, GD, Adam, and Muon. Given any parameter W∈A,BW∈\A,B\, these optimizers update the parameter WtW_t at step t as follows. • GD updates the parameter according to the gradient, i.e., Wt+1=Wt−ηt∇WL(Wt)W_t+1 GD=W_t GD- _t _WL(W_t GD). Here, the notation L(Wt)L(W_t GD) suppresses its dependency on other parameters for conciseness. • With γ1=γ2=0 _1= _2=0, Adam updates the parameter with the coordinate-wise normalized gradient: Wt+1=Wt−γtsgn(∇WL(Wt))W_t+1 Adam=W_t Adam- _t sgn( _WL(W_t Adam)), where sgn(⋅) sgn(·) denotes the entrywise sign operator. • With momentum set to 00, Muon updates the parameters using the spectrally normalized gradient: Wt+1=Wt−τtspec(∇WL(Wt))W_t+1 Muon=W_t Muon- _t spec( _WL(W_t Muon)), where spec(G)=UV⊤ spec(G)=UV for G=UΣV⊤G=U V , which normalizes all nonzero singular values Σ of G to 11. Theorem 7.3 (A stylized realization of exact RGI with low parameter cosine). For each optimizer ∈,, A∈\ GD, Adam, Muon\, set the initial parameters as A0=0,B0=[b0Ir,0]⊤A_0 A=0,B_0 A=[b_0 AI_r,0] with any b0>0b_0 A>0 and set the step size to constants as ηt=η>0,γt=γ<minb0,(κb0+κ)−1,τt=τ<minb0,(b0/μ+μ−1/2)−1. _t=η>0,\, _t=γ\!<\! \b_0 Adam,(κ b_0 Adam+ κ)^-1\,\, _t=τ\!<\! \b_0 Muon,(b_0 Muon/μ+μ^-1/2)^-1\. Under Assumptions 7.1 and 7.2, there exist tie-breaking rules on zero-gradient, for any two optimizers ,′∈,, A, A ∈\ GD, Adam, Muon\, any two inputs x,x′x,x with ‖x‖2=‖x′‖2=ρ\|x\|_2=\|x \|_2=ρ, and any t≥0t≥ 0, we have Lt(x)−Lt(x′)=Lt′(x)−Lt′(x′). L_t A(x)-L_t A(x )=L_t A (x)-L_t A (x ). Moreover, let θt=(vec(At),vec(Bt)) _t A=(vec(A_t A),vec(B_t A)). At the same time, with cosine similarity defined to be zero whenever either vector is zero, the following holds as t increases. |cos-sim(θt,θt)|=O(1/t),|cos-sim(θt,θt)|=O(1/t),|cos-sim(θt,θt)|=O(1/t2). |cos -sim( _t GD\!\!, _t Adam)|\!=\!O(1/t),|cos -sim( _t GD\!\!, _t Muon)|\!=\!O(1/t),|cos -sim( _t Adam\!\!, _t Muon)|\!=\!O(1/t^2). This theorem shows that rgi holds across GD, Muon, Adam, and different initializations under Assumptions 7.1 and 7.2, with the covariate x playing the same role as the token d in Definition 4.1. These results support the empirical findings in Sections 5.1 and 5.2. The theorem also permits a broad range of learning rates, imposing only upper-bound constraints on those of Muon and Adam, consistent with the robustness of rgi to optimizer hyperparameters observed in Section 5.1. Moreover, we show that the cosine similarity between parameters decreases during training, matching the low parameter similarity reported in Section 6.1. Overall, our analysis identifies a class of quadratic problems in which rgi is robust across optimizers, initializations, and learning rates. 8 Conclusion We study how optimizers, architectures, and training data streams affect model performance through the lens of relative generalization. We show that rgi persists under joint variations in optimizers and architectures, which largely preserve relative generalization and induce approximately uniform shifts in token-wise losses, whereas changes in the training data stream can substantially alter it. Neither the ntk nor the mf regime alone explains rgi; instead, we identify a class of quadratic problems in which rgi emerges robustly across initializations and learning rates. Our analysis focuses on autoregressive llm, leaving other model classes, such as diffusion models, for future work. References Abdulkadirov et al. (2023) R. Abdulkadirov, P. Lyakhov, and N. Nagornov Survey of optimization algorithms in modern neural networks. Mathematics 11 (11), p. 2466. Cited by: §2. Abreu et al. (2025) N. Abreu, N. Vyas, S. Kakade, and D. Morwani The potential of second-order optimization for llms: a study with full gauss-newton. arXiv preprint arXiv:2510.09378. Cited by: §2. Ahn et al. (2025) K. Ahn, B. Xu, N. Abreu, Y. Fan, G. Magakyan, P. Sharma, Z. Zhan, and J. Langford Dion: distributed orthonormalized updates. arXiv preprint arXiv:2504.05295. Cited by: §2. Allen-Zhu et al. (2019) Z. Allen-Zhu, Y. Li, and Y. Liang Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems 32. Cited by: §2. Arora et al. (2019a) S. Arora, S. Du, W. Hu, Z. Li, and R. Wang Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International conference on machine learning, p. 322–332. Cited by: §2. Arora et al. (2019b) S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang On exact computation with an infinitely wide neural net. Advances in neural information processing systems 32. Cited by: §2. Baack (2024) S. Baack A critical analysis of the largest source for generative ai training data: common crawl. In Proceedings of the 2024 ACM conference on fairness, accountability, and transparency, p. 2199–2208. Cited by: §5.3. Bian et al. (2025) S. Bian, T. Yu, S. Venkataraman, and Y. Park Scaling laws meet model architecture: toward inference-efficient llms. arXiv preprint arXiv:2510.18245. Cited by: §2. Blake et al. (2024) C. Blake, C. Eichenberg, J. Dean, L. Balles, L. Y. Prince, B. Deiseroth, A. F. Cruz-Salinas, C. Luschi, S. Weinbach, and D. Orr U-μ p: the unit-scaled maximal update parametrization. arXiv preprint arXiv:2407.17465. Cited by: §2. Chen et al. (2023) X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C. Hsieh, Y. Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems 36, p. 49205–49233. Cited by: §3, §5. Cheng et al. (2026) P. Cheng, J. Zang, Q. Li, L. Ma, Y. Cui, Y. Zhang, B. Chen, M. Jian, and W. Tong TrasMuon: trust-region adaptive scaling for orthogonalized momentum optimizers. arXiv preprint arXiv:2602.13498. Cited by: §2. Chizat et al. (2019) L. Chizat, E. Oyallon, and F. Bach On lazy training in differentiable programming. Advances in neural information processing systems 32. Cited by: §2. CodeParrot (2022) CodeParrot GitHub code dataset. https://huggingface.co/datasets/codeparrot/github-code. Cited by: §5.3. Deng et al. (2026) S. Deng, Z. Ouyang, T. Pang, Z. Liu, R. Jin, S. Yu, and Y. Yang RMNP: row-momentum normalized preconditioning for scalable matrix-based optimization. arXiv preprint arXiv:2603.20527. Cited by: §2. Dozat (2016) T. Dozat Incorporating nesterov momentum into adam. Cited by: §2. Du and Lee (2018) S. Du and J. Lee On the power of over-parametrization in neural networks with quadratic activation. In International conference on machine learning, p. 1329–1338. Cited by: §2. Du et al. (2018) S. S. Du, X. Zhai, B. Poczos, and A. Singh Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054. Cited by: §6.1. Duchi et al. (2011) J. Duchi, E. Hazan, and Y. Singer Adaptive subgradient methods for online learning and stochastic optimization.. Journal of machine learning research 12 (7). Cited by: §2. Elfwing et al. (2018) S. Elfwing, E. Uchibe, and K. Doya Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks 107, p. 3–11. Cited by: §1, §5.2. Geiger et al. (2020) M. Geiger, S. Spigler, A. Jacot, and M. Wyart Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment 2020 (11), p. 113301. Cited by: §2, §6.1. Golikov et al. (2022) E. Golikov, E. Pokonechnyy, and V. Korviakov Neural tangent kernel: a survey. arXiv preprint arXiv:2208.13614. Cited by: §2, §6.1. Gupta et al. (2018) V. Gupta, T. Koren, and Y. Singer Shampoo: preconditioned stochastic tensor optimization. In International Conference on Machine Learning, p. 1842–1850. Cited by: §2, §3. Hayou (2025) S. Hayou A proof of learning rate transfer under μ p. arXiv preprint arXiv:2511.01734. Cited by: §2. He et al. (2021) X. He, F. Xue, X. Ren, and Y. You Large-scale deep learning optimizations: a comprehensive survey. arXiv preprint arXiv:2111.00856. Cited by: §2. Hendrycks and Gimpel (2016) D. Hendrycks and K. Gimpel Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. Cited by: §5.2, §5. Hernandez et al. (2021) D. Hernandez, J. Kaplan, T. Henighan, and S. McCandlish Scaling laws for transfer. arXiv preprint arXiv:2102.01293. Cited by: §2. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 10. Cited by: §2. Hu et al. (2024) S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al. Minicpm: unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395. Cited by: §5. Jacot et al. (2018) A. Jacot, F. Gabriel, and C. Hongler Neural tangent kernel: convergence and generalization in neural networks. Advances in neural information processing systems 31. Cited by: §2, §6.1. Jin et al. (2017) C. Jin, R. Ge, P. Netrapalli, S. M. Kakade, and M. I. Jordan How to escape saddle points efficiently. In International conference on machine learning, p. 1724–1732. Cited by: footnote. Jordan et al. (2024) K. Jordan, Y. Jin, V. Boza, Y. Jiacheng, F. Cecista, L. Newhouse, and J. Bernstein Muon: an optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan. github. io/posts/muon 6. Cited by: §1, §2, §3, §5. Kang et al. (2025) F. Kang, N. Ardalani, M. Kuchnik, Y. Emad, M. Elhoushi, S. Sengupta, S. Li, R. Raghavendra, R. Jia, and C. Wu Demystifying synthetic data in llm pre-training: a systematic study of scaling laws, benefits, and pitfalls. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 10750–10769. Cited by: §2. Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: §2. Kazemnejad et al. (2023) A. Kazemnejad, I. Padhi, K. Natesan Ramamurthy, P. Das, and S. Reddy The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems 36, p. 24892–24928. Cited by: §5.2. Khaled et al. (2025) A. Khaled, K. Ozkara, T. Yu, M. Hong, and Y. Park Muonbp: faster muon via block-periodic orthogonalization. arXiv preprint arXiv:2510.16981. Cited by: §2. Kim and Suzuki (2024) J. Kim and T. Suzuki Transformers learn nonlinear features in context: nonconvex mean-field dynamics on the attention landscape. arXiv preprint arXiv:2402.01258. Cited by: §2. Kingma and Ba (2014) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §2, §3, §5. Klusowski and Barron (2016) J. M. Klusowski and A. R. Barron Approximation by combinations of relu and squared relu ridge functions with ℓ1 ^1 and ℓ0 ^0 controls. arXiv preprint arXiv:1607.07819. Cited by: §5.2. Lee et al. (2019) J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems 32. Cited by: §2. Li et al. (2025a) M. Li, S. Kudugunta, and L. Zettlemoyer (Mis) fitting: a survey of scaling laws. arXiv preprint arXiv:2502.18969. Cited by: §2. Li et al. (2025b) Z. Li, L. Liu, C. Liang, W. Chen, and T. Zhao NorMuon: making muon more efficient and scalable. arXiv preprint arXiv:2510.05491. Cited by: §2. Lin (1989) L. I. Lin A concordance correlation coefficient to evaluate reproducibility. Biometrics, p. 255–268. Cited by: §4. Lingle (2024) L. Lingle An empirical study of μ p learning rate transfer. arXiv preprint arXiv:2404.05728. Cited by: §2. Liu et al. (2020) C. Liu, L. Zhu, and M. Belkin On the linearity of large non-linear models: when and why the tangent kernel is constant. Advances in Neural Information Processing Systems 33, p. 15954–15964. Cited by: §2. Liu et al. (2025) E. Liu, A. Bertsch, L. Sutawika, L. Tjuatja, P. Fernandes, L. Marinov, M. Chen, S. Singhal, C. Lawrence, A. Raghunathan, et al. Not-just-scaling laws: towards a better understanding of the downstream impact of language model design decisions. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 16407–16438. Cited by: §2, §2. Liu et al. (2023) H. Liu, Z. Li, D. Hall, P. Liang, and T. Ma Sophia: a scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342. Cited by: §2. Liu et al. (2024) Q. Liu, X. Zheng, N. Muennighoff, G. Zeng, L. Dou, T. Pang, J. Jiang, and M. Lin Regmix: data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492. Cited by: §2. Liu et al. (2026) Z. Liu, R. Zhang, Z. Wang, Y. Zhao, Y. Su, Z. Yang, and Z. Zhang Muon2^2: boosting muon via adaptive second-moment preconditioning. arXiv preprint arXiv:2604.09967. Cited by: §2. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §2. Luo et al. (2025) J. Luo, B. Wu, X. Luo, Z. Xiao, Y. Jin, R. Tu, N. Yin, Y. Wang, J. Yuan, W. Ju, et al. A survey on efficient large language model training: from data-centric perspectives. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 30904–30920. Cited by: §2, §2. Ma and Yarats (2018) J. Ma and D. Yarats Quasi-hyperbolic momentum and adam for deep learning. arXiv preprint arXiv:1810.06801. Cited by: §2. Mandt et al. (2017) S. Mandt, M. D. Hoffman, and D. M. Blei Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research 18 (134), p. 1–35. Cited by: footnote. Mayilvahanan et al. (2025) P. Mayilvahanan, T. Wiedemer, S. Mallick, M. Bethge, and W. Brendel Llms on the line: data determines loss-to-loss scaling laws. arXiv preprint arXiv:2502.12120. Cited by: §2. Mei et al. (2019) S. Mei, T. Misiakiewicz, and A. Montanari Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on learning theory, p. 2388–2464. Cited by: §2, §6.2. Mei et al. (2018) S. Mei, A. Montanari, and P. Nguyen A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences 115 (33), p. E7665–E7671. Cited by: §2, §6.2. Mizrahi et al. (2025) D. Mizrahi, A. B. L. Larsen, J. Allardice, S. Petryk, Y. Gorokhov, J. Li, A. Fang, J. Gardner, T. Gunter, and A. Dehghan Language models improve when pretraining data matches target tasks. arXiv preprint arXiv:2507.12466. Cited by: §2. Novak et al. (2019) R. Novak, L. Xiao, J. Hron, J. Lee, A. A. Alemi, J. Sohl-Dickstein, and S. S. Schoenholz Neural tangents: fast and easy infinite neural networks in python. arXiv preprint arXiv:1912.02803. Cited by: §2. Pang et al. (2026) T. Pang, Y. Fang, Z. Liu, S. Deng, L. Hsiung, S. Yu, and Y. Yang HTMuon: improving muon via heavy-tailed spectral correction. arXiv preprint arXiv:2603.10067. Cited by: §2. Penedo et al. (2024) G. Penedo, H. Kydlícek, L. B. Allal, and T. Wolf FineWeb: decanting the web for the finest text data at scale. HuggingFace. Accessed: Jul 12. Cited by: §5. Pethick et al. (2025) T. Pethick, W. Xie, K. Antonakopoulos, Z. Zhu, A. Silveti-Falls, and V. Cevher Training deep learning models with norm-constrained lmos. arXiv preprint arXiv:2502.07529. Cited by: §2, §3, §5. Poc-López and Aguilera (2024) Á. Poc-López and M. Aguilera Dynamical mean-field theory of self-attention neural networks. arXiv preprint arXiv:2406.07247. Cited by: §2. Press et al. (2021) O. Press, N. A. Smith, and M. Lewis Train short, test long: attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409. Cited by: §5.2. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21 (140), p. 1–67. Cited by: §5.3. Reddi et al. (2019) S. J. Reddi, S. Kale, and S. Kumar On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237. Cited by: §2. Rigollet (2025) P. Rigollet The mean-field dynamics of transformers. arXiv preprint arXiv:2512.01868. Cited by: §2. Robbins and Monro (1951) H. Robbins and S. Monro A stochastic approximation method. The annals of mathematical statistics, p. 400–407. Cited by: §2. Rosenfeld et al. (2019) J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit A constructive prediction of the generalization error across scales. arXiv preprint arXiv:1909.12673. Cited by: §2. Semenov et al. (2025) A. Semenov, M. Pagliardini, and M. Jaggi Benchmarking optimizers for large language model pretraining. arXiv preprint arXiv:2509.01440. Cited by: §2. Shan et al. (2026) R. Shan, A. Yu, B. Chen, K. Cai, Q. Luo, R. Tang, H. Li, W. Liu, W. Zhang, and J. Lin MuonRec: shifting the optimizer paradigm beyond adam in scalable generative recommendation. arXiv preprint arXiv:2603.00416. Cited by: §2. Shazeer and Stern (2018) N. Shazeer and M. Stern Adafactor: adaptive learning rates with sublinear memory cost. In International conference on machine learning, p. 4596–4604. Cited by: §2. Shukor et al. (2025) M. Shukor, L. Bethune, D. Busbridge, D. Grangier, E. Fini, A. El-Nouby, and P. Ablin Scaling laws for optimal data mixtures. arXiv preprint arXiv:2507.09404. Cited by: §2. Sirignano and Spiliopoulos (2020) J. Sirignano and K. Spiliopoulos Mean field analysis of neural networks: a central limit theorem. Stochastic Processes and their Applications 130 (3), p. 1820–1852. Cited by: §2, §6.2. Sirignano and Spiliopoulos (2022) J. Sirignano and K. Spiliopoulos Mean field analysis of deep neural networks. Mathematics of Operations Research 47 (1), p. 120–152. Cited by: §2. Southworth and Thomas (2026) B. S. Southworth and S. Thomas Beyond muon: mud (momentum decorrelation) for faster transformer training. arXiv preprint arXiv:2603.17970. Cited by: §2. Su et al. (2024) J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. Cited by: §5.2, §5. Subramanyam et al. (2025) A. Subramanyam, Y. Chen, and R. L. Grossman Scaling laws revisited: modeling the role of data quality in language model pretraining. arXiv preprint arXiv:2510.03313. Cited by: §2. Takakura and Suzuki (2024) S. Takakura and T. Suzuki Mean-field analysis on two-layer neural networks from a kernel perspective. arXiv preprint arXiv:2403.14917. Cited by: §2. Tao et al. (2024) C. Tao, Q. Liu, L. Dou, N. Muennighoff, Z. Wan, P. Luo, M. Lin, and N. Wong Scaling laws with vocabulary: larger models deserve larger vocabularies. Advances in Neural Information Processing Systems 37, p. 114147–114179. Cited by: §2. Tian et al. (2025) K. Tian, L. Qiao, B. Liu, G. Jiang, and D. Li A survey on memory-efficient large-scale model training in ai for science. arXiv e-prints, p. arXiv–2501. Cited by: §2. Tieleman and Hinton (2012) T. Tieleman and G. Hinton Lecture 6.5—RMSProp: divide the gradient by a running average of its recent magnitude. Note: Coursera: Neural Networks for Machine Learning External Links: Link Cited by: §2. Volkova et al. (2026) A. Volkova, M. Safaryan, C. H. Lampert, and D. Alistarh Towards robust scaling laws for optimizers. arXiv preprint arXiv:2602.07712. Cited by: §2. Vyas et al. (2024) N. Vyas, D. Morwani, R. Zhao, M. Kwun, I. Shapira, D. Brandfonbrener, L. Janson, and S. Kakade Soap: improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321. Cited by: §1, §2, §3, §5. Wan et al. (2023) Z. Wan, X. Wang, C. Liu, S. Alam, Y. Zheng, J. Liu, Z. Qu, S. Yan, Y. Zhu, Q. Zhang, et al. Efficient large language models: a survey. arXiv preprint arXiv:2312.03863. Cited by: §2. Wang et al. (2026) S. Wang, X. Ouyang, T. Xu, Y. Hu, J. Liu, G. Chen, T. Zhang, J. Zheng, K. Yang, X. Ren, et al. OPUS: towards efficient and principled data selection in large language model pre-training in every iteration. arXiv preprint arXiv:2602.05400. Cited by: §1. Weber et al. (2024) M. Weber, D. Y. Fu, Q. Anthony, Y. Oren, S. Adams, A. Alexandrov, X. Lyu, H. Nguyen, X. Yao, V. Adams, et al. Redpajama: an open dataset for training large language models. Advances in neural information processing systems 37, p. 116462–116492. Cited by: §5.3. Wen et al. (2025) K. Wen, D. Hall, T. Ma, and P. Liang Fantastic pretraining optimizers and where to find them. arXiv preprint arXiv:2509.02046. Cited by: §2. Wen et al. (2024) K. Wen, Z. Li, J. Wang, D. Hall, P. Liang, and T. Ma Understanding warmup-stable-decay learning rates: a river valley loss landscape perspective. arXiv preprint arXiv:2410.05192. Cited by: §5. Xiao et al. (2024) G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis Efficient streaming language models with attention sinks. In International Conference on Learning Representations, Vol. 2024, p. 21875–21895. Cited by: §1. Xiao et al. (2025) Z. Xiao, J. Xie, L. Xu, S. Guan, J. Zhu, X. Han, X. Fu, W. Yu, H. Wu, W. Shi, et al. A survey of optimization modeling meets llms: progress and future directions. arXiv preprint arXiv:2508.10047. Cited by: §2. Xie et al. (2023) S. M. Xie, H. Pham, X. Dong, N. Du, H. Liu, Y. Lu, P. S. Liang, Q. V. Le, T. Ma, and A. W. Yu Doremi: optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems 36, p. 69798–69818. Cited by: §2. Xie et al. (2025) W. Xie, F. Tonin, and V. Cevher Chameleon: a flexible data-mixing framework for language model pretraining and finetuning. arXiv preprint arXiv:2505.24844. Cited by: §2. Yang et al. (2021) G. Yang, E. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems 34, p. 17084–17097. Cited by: §2. Yang and Hu (2021) G. Yang and E. J. Hu Tensor programs iv: feature learning in infinite-width neural networks. In International Conference on Machine Learning, p. 11727–11737. Cited by: §2. Ye et al. (2024) J. Ye, P. Liu, T. Sun, J. Zhan, Y. Zhou, and X. Qiu Data mixing laws: optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952. Cited by: §2. Yue et al. (2025) Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, S. Song, and G. Huang Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. arXiv preprint arXiv:2504.13837. Cited by: §2. Zeiler (2012) M. D. Zeiler Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. Cited by: §2. Zhang et al. (2026a) M. Zhang, Y. Liu, and H. Scheaffer Adam improves muon: adaptive moment estimation with orthogonalized momentum. arXiv preprint arXiv:2602.17080. Cited by: §2. Zhang et al. (2026b) R. Zhang, Y. Zhao, Z. Liu, Z. Wang, and Z. Zhang Muon+: towards better muon via one additional normalization step. arXiv preprint arXiv:2602.21545. Cited by: §2. Zhang et al. (2026c) Y. Zhang, S. Xing, J. Huang, K. Lv, Y. Zhou, X. Qiu, Q. Guo, and K. Chen Mousse: rectifying the geometry of muon with curvature-aware preconditioning. arXiv preprint arXiv:2603.09697. Cited by: §2. Zhang et al. (2024) Y. Zhang, C. Chen, Z. Li, T. Ding, C. Wu, D. P. Kingma, Y. Ye, Z. Luo, and R. Sun Adam-mini: use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793. Cited by: §2, §3, §5. Zhao et al. (2024a) J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian Galore: memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507. Cited by: §2. Zhao et al. (2024b) R. Zhao, D. Morwani, D. Brandfonbrener, N. Vyas, and S. Kakade Deconstructing what makes a good optimizer for language models. arXiv preprint arXiv:2407.07972. Cited by: §2. Zhao et al. (2024c) Y. Zhao, L. Du, X. Ding, K. Xiong, Z. Sun, S. Jun, T. Liu, and B. Qin Deciphering the impact of pretraining data on large language models through machine unlearning. In Findings of the Association for Computational Linguistics: ACL 2024, p. 9386–9406. Cited by: §2. Zheng et al. (2026) C. Zheng, R. Wang, X. Zhang, and C. Li Spectral condition for μ p under width-depth scaling. arXiv preprint arXiv:2603.00541. Cited by: §2. Zhou et al. (2024) J. Zhou, Y. Chen, Z. Hong, W. Chen, Y. Yu, T. Zhang, H. Wang, C. Zhang, and Z. Zheng Training and serving system of foundation models: a comprehensive survey. IEEE Open Journal of the Computer Society 5, p. 107–119. Cited by: §2. Zhu et al. (2025) H. Zhu, Z. Zhang, W. Cong, X. Liu, S. Park, V. Chandra, B. Long, D. Z. Pan, Z. Wang, and J. Lee Apollo: sgd-like memory, adamw-level performance. Proceedings of Machine Learning and Systems 7. Cited by: §2. Zhuang et al. (2020) J. Zhuang, T. Tang, Y. Ding, S. C. Tatikonda, N. Dvornek, X. Papademetris, and J. Duncan Adabelief optimizer: adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems 33, p. 18795–18806. Cited by: §2. Appendix A Experimental Details A.1 Concordance Correlation Coefficient We present the detailed definition of c among centered losses L(θt,Dval,k)L( _t T,D_val,k) and L(θt′,Dval,k)L( _t T ,D_val,k) as follows. The centered losses are defined as L¯(θt,Dval,k) L ( _t T,D_val,k ) =L(θt,Dval,k)−K−1∑k′=1KL(θt,Dval,k′). =L ( _t T,D_val,k )-K^-1 _k =1^KL ( _t T,D_val,k ). Then we have that rCCC(,′)=[2CovK(L¯(θt,Dval,k),L¯(θt′,Dval,k))VarK(L¯(θt,Dval,k))+VarK(L¯(θt′,Dval,k))], r_C( T, T )=E [ 2\,Cov_K ( L ( _t T,D_val,k ), L ( _t T ,D_val,k ) )Var_K ( L ( _t T,D_val,k ) )+Var_K ( L ( _t T ,D_val,k ) ) ], where the expectation is taken with respect to the joint distribution of θ0 _0 and (Dl)l=1t(D_l)_l=1^t, and CovKCov_K and VarKVar_K are covariance and variance with respect to k. In other words, for any functions f(Dval,k)f(D_val,k) and g(Dval,k)g(D_val,k), we have VarK(f(Dval,k)) _K (f(D_val,k) ) =1K∑k=1K(f(Dval,k)−1K∑k′=1Kf(Dval,k′))2 = 1K _k=1^K (f(D_val,k)- 1K _k =1^Kf(D_val,k ) )^2 CovK(f(Dval,k),g(Dval,k)) _K (f(D_val,k),g(D_val,k) ) =1K∑k=1Kf(Dval,k)g(Dval,k)−1K2∑k′=1Kf(Dval,k′)⋅∑k′=1Kg(Dval,k′). = 1K _k=1^Kf(D_val,k)g(D_val,k)- 1K^2 _k =1^Kf(D_val,k )\!\!·\!\! _k =1^Kg(D_val,k ). A.2 Definition of rdiffr_diff s(,′) s( T, T ) =∑k,k′∈[K](Δ(,Dval,k,Dval,k′)−Δ(′,Dval,k,Dval,k′))2 = _k,k ∈[K] ( ( T,D_val,k,D_val,k )- ( T ,D_val,k,D_val,k ) )^2 rdiff(,′) r_diff( T, T ) =12⋅[s(,′)∑k,k′∈[K]Δ(,Dval,k,Dval,k′)2+s(,′)∑k,k′∈[K]Δ(′,Dval,k,Dval,k′)2], = 12·E [ s( T, T ) _k,k ∈[K] ( T,D_val,k,D_val,k )^2+ s( T, T ) _k,k ∈[K] ( T ,D_val,k,D_val,k )^2 ], where the expectation is taken with respect to the joint distribution of θ0 _0 and D1:tD_1:t. A.3 Experimental Setups In all experiments, we train llm using the GPT-2 tokenizer with sequence length 10241024 and batch size 524,288524,288. For each optimizer, we disable weight decay and search the learning rate over 1,2,5×10−1,10−2,10−3,10−4\1,2,5\×\10^-1,10^-2,10^-3,10^-4\. For Adam, Adam-mini, and Lion, we set β1=0.9 _1=0.9 and β2=0.95 _2=0.95; for SOAP, we set β1=β2=0.95 _1= _2=0.95; for Muon and Scion, we use momentum values 0.950.95 and 0.90.9, respectively. We consider two GPT-style model sizes. The 124124M model has 1212 layers, 1212 attention heads, and hidden dimension 768768, while the 0.70.7B model has 3636 layers, 2020 attention heads, and hidden dimension 12801280. For evaluations, we partition valD_val into validation batches (Dval,k)k=1K(D_val,k)_k=1^K and evaluate two batch granularities: |Dval,k|=1,024|D_val,k|=1,024 and |Dval,k|=1|D_val,k|=1. In the main text, we report results with |Dval,k|=1,024|D_val,k|=1,024 and K=20,480K=20,480, corresponding to approximately 2121M validation tokens. The single-token results, i.e., |Dval,k|=1|D_val,k|=1, are computed on 55M tokens due to computational cost and deferred to Appendix B; they yield conclusions consistent with the |Dval,k|=1,024|D_val,k|=1,024 setting. Compute Resources. All experiments are conducted on NVIDIA A40 GPUs. Each 124M model run uses 4 A40 GPUs and takes approximately 10 hours, while each 0.7B run uses 4 A40 GPUs and takes approximately 50 hours. Each 4.5B run uses 8 A40 GPUs and takes approximately three days. Appendix B Additional Experimental Results B.1 Additional Results of Validation Losses Between Optimizers (a) Validation Losses of Adam and Adam-mini (b) Validation Losses of Adam and Lion (c) Validation Losses of Adam and Soap (d) Validation Losses of Adam and Scion Figure 11: Validation-loss linearity across Adam and other optimizers. Panels (a)–(d) plot the validation-loss relationship between Adam and Adam-mini, Lion, Soap, and Scion, respectively, at the 10,00010,000-step checkpoint. The near-perfect fits, with R2≥0.99R^2≥ 0.99, support approximate rgi across optimizers. These figures show that the validation losses of models trained by different optimizers are highly linearly correlated at step 10,00010,000. This supports the emergence of approximate rgi among optimizers. B.2 Additional Results of rdiffr_diff and rCCCr_C with Other Optimizers as Baselines In the following, we report rdiffr_diff and rCCCr_C using optimizers other than Adam as the baseline, while keeping the initial parameters and training data stream identical across all optimizers. The results show that changing the baseline optimizer does not qualitatively affect approximate rgi. This is intuitive: if exact rgi holds between θ and θ′θ , and also between θ′θ and θ′θ , then it also holds between θ and θ′θ . In the approximate case, the same transitivity holds up to accumulated error. (a) Values of rdiffr_diff with Adam-mini as the baseline. (b) Values of rdiffr_diff with Lion as the baseline. (c) Values of rdiffr_diff with Muon as the baseline. (d) Values of rdiffr_diff with Soap as the baseline. (e) Values of rdiffr_diff with Scion as the baseline. Figure 12: Baseline-optimizer ablation for rdiffr_diff. Panels (a)–(e) plot rdiffr_diff when the baseline optimizer is Adam-mini, Lion, Muon, Soap, and Scion, respectively. Approximate rgi is insensitive to which optimizer is used as the evaluation baseline. (a) Values of rCCCr_C with Adam-mini as the baseline. (b) Values of rCCCr_C with Lion as the baseline. (c) Values of rCCCr_C with Muon as the baseline. (d) Values of rCCCr_C with Soap as the baseline. (e) Values of rCCCr_C with Scion as the baseline. Figure 13: Baseline-optimizer ablation for rCCCr_C. Panels (a)–(e) plot rCCCr_C when the baseline optimizer is Adam-mini, Lion, Muon, Soap, and Scion, respectively. Approximate rgi is insensitive to which optimizer is used as the evaluation baseline. B.3 Per-Token Results of Relative Generalization Invariance In the following, we present the results of rdiffr_diff and rCCCr_C calculated with |Dval,k|=1|D_val,k|=1 when the initial parameters and the training data stream are the same for all the optimizers. The results show that rgi among optimizers holds when we verify it in a token-wise manner. (a) Values of rdiffr_diff calculated per token. (b) Values of rCCCr_C calculated per token. Figure 14: Per-token evaluation of rgi. Panels (a) and (b) plot rdiffr_diff and rCCCr_C, respectively, when each validation batch contains a single token, i.e., |Dval,k|=1|D_val,k|=1. The small rdiffr_diff and large rCCCr_C values show that rgi also holds at token-level granularity. B.4 Results for 0.70.7B and 4.54.5B Models Due to the high computational cost of training 0.70.7B and 4.54.5B models, we focus on Adam and Muon at this scale. We provide results for 0.70.7B and 4.54.5B models when Adam and Muon share the same initial parameters and training data stream. These results mirror those in Section 5.1, showing that rgi among optimizers also holds at larger scales. (a) Values of rdiffr_diff for 0.70.7B models. (b) Values of rCCCr_C for 0.70.7B models. Figure 15: Baseline rgi for 0.70.7B models. Panels (a) and (b) plot rdiffr_diff and rCCCr_C, respectively, between Adam and Muon when they share the same initial parameters and training data stream. The results show that approximate rgi persists at the 0.70.7B scale. (a) Values of rdiffr_diff for 4.54.5B models. (b) Values of rCCCr_C for 4.54.5B models. Figure 16: Baseline rgi for 4.54.5B models. Panels (a) and (b) plot rdiffr_diff and rCCCr_C, respectively, between Adam and Muon when they share the same initial parameters and training data stream. The results show that approximate rgi persists at the 4.54.5B scale. In the following experiments, we ablate how the hyperparameters influence rgi of 0.70.7B models. All results are compared against Adam under the baseline in Figure 15. Therefore, we also measure rgi between Adam runs with different settings. (a) Values of rdiffr_diff for 0.70.7B models with different initial parameters. (b) Values of rCCCr_C for 0.70.7B with different initial parameters. Figure 17: Initialization ablation for 0.70.7B models. Panels (a) and (b) plot rdiffr_diff and rCCCr_C, respectively, for Adam and Muon runs started from different initial parameters. Approximate rgi still emerges at the 0.70.7B scale, showing that initialization differences do not prevent the phenomenon. (a) Values of rdiffr_diff for 0.70.7B models with 0.25η0.25η A. (b) Values of rdiffr_diff for 0.70.7B models with 1.25η1.25η A. (c) Values of rCCCr_C for 0.70.7B with 0.25η0.25η A. (d) Values of rCCCr_C for 0.70.7B with 1.25η1.25η A. Figure 18: Learning-rate ablation for 0.70.7B models. Panels (a) and (b) plot rdiffr_diff for 0.25η0.25η A and 1.25η1.25η A, while Panels (c) and (d) plot rCCCr_C for the same two learning-rate multipliers. Approximate rgi remains robust to these learning-rate changes at the 0.70.7B scale. (a) Values of rdiffr_diff for 0.70.7B models with γ=0.85γ=0.85. (b) Values of rdiffr_diff for 0.70.7B models with γ=0.9γ=0.9. (c) Values of rdiffr_diff for 0.70.7B models with γ=0.95γ=0.95. (d) Values of rCCCr_C for 0.70.7B with γ=0.85γ=0.85. (e) Values of rCCCr_C for 0.70.7B with γ=0.9γ=0.9. (f) Values of rCCCr_C for 0.70.7B with γ=0.95γ=0.95. Figure 19: Momentum ablation for 0.70.7B models. Panels (a)–(c) plot rdiffr_diff for γ=0.85γ=0.85, 0.90.9, and 0.950.95, while Panels (d)–(f) plot rCCCr_C for the same momentum values. Approximate rgi remains robust to the tested momentum settings at the 0.70.7B scale. (a) Values of rdiffr_diff for 0.70.7B models with 100100 warmup steps. (b) Values of rCCCr_C for 0.70.7B models with 100100 warmup steps. (c) Values of rdiffr_diff for 0.70.7B models with 14001400 warmup steps. (d) Values of rCCCr_C for 0.70.7B models with 14001400 warmup steps. Figure 20: Warmup-step ablation for 0.70.7B models. Panels (a) and (c) plot rdiffr_diff for 100100 and 1,4001,400 warmup steps, while Panels (b) and (d) plot rCCCr_C for the same warmup settings. Approximate rgi remains robust to the tested scheduler warmup lengths at the 0.70.7B scale. (a) Values of rdiffr_diff for 0.70.7B models when Adam is trained on FineWeb and Muon is trained on CodeParrot. (b) Values of rCCCr_C for 0.70.7B models when Adam is trained on FineWeb and Muon is trained on CodeParrot. Figure 21: Training data-stream ablation for 0.70.7B models. Panels (a) and (b) plot rdiffr_diff and rCCCr_C, respectively, when Adam is trained on FineWeb and Muon is trained on CodeParrot. The weak agreement shows that changing the training data stream strongly disrupts rgi at the 0.70.7B scale. (a) Values of rdiffr_diff for 0.70.7B models with DvalD_val from C4. (b) Values of rdiffr_diff for 0.70.7B models with DvalD_val from ArXiv. (c) Values of rdiffr_diff for 0.70.7B models with DvalD_val from CodeParrot. (d) Values of rCCCr_C for 0.70.7B models with DvalD_val from C4. (e) Values of rCCCr_C for 0.70.7B models with DvalD_val from ArXiv. (f) Values of rCCCr_C for 0.70.7B models with DvalD_val from CodeParrot. Figure 22: Validation-distribution ablation for 0.70.7B models. Panels (a)–(c) plot rdiffr_diff evaluated on C4, ArXiv, and CodeParrot, while Panels (d)–(f) plot rCCCr_C on the same validation datasets. rgi remains clearest on C4 and weakens on the more distributionally distinct ArXiv and CodeParrot validation sets. B.5 Additional Results for Section 5 B.5.1 Additional Results for Section 5.1 (a) rdiffr_diff for 0.25η0.25η A. (b) rdiffr_diff for 0.5η0.5η A. (c) rdiffr_diff for 0.75η0.75η A. (d) rdiffr_diff for 1.25η1.25η A. (e) rCCCr_C for 0.25η0.25η A. (f) rCCCr_C for 0.5η0.5η A. (g) rCCCr_C for 0.75η0.75η A. (h) rCCCr_C for 1.25η1.25η A. Figure 23: Learning-rate ablations for rdiffr_diff and rCCCr_C. Panels (a)–(d) plot rdiffr_diff for learning-rate multipliers 0.25η0.25η A, 0.5η0.5η A, 0.75η0.75η A, and 1.25η1.25η A, respectively. Panels (e)–(h) plot rCCCr_C for them. The results show that approximate rgi holds robustly across the tested learning rates. (a) rdiffr_diff for γ=0.85γ=0.85. (b) rdiffr_diff for γ=0.9γ=0.9. (c) rdiffr_diff for γ=0.95γ=0.95. (d) rCCCr_C for γ=0.85γ=0.85. (e) rCCCr_C for γ=0.9γ=0.9. (f) rCCCr_C for γ=0.95γ=0.95. Figure 24: Momentum ablations for rdiffr_diff and rCCCr_C. Panels (a)–(c) plot rdiffr_diff for momentumns γ=0.85.0.9,0.95γ=0.85.0.9,0.95, respectively. Panels (d)–(f) plot rCCCr_C for them. The results show that approximate rgi holds robustly across the tested momentum values. To evaluate the effect of the learning-rate scheduler, we ablate the number of warmup steps in the Warmup-Stable-Decay scheduler. The experiments in Section 5.1 use 700700 warmup steps. We additionally evaluate all optimizers with 100100 and 1,4001,400 warmup steps. Figure 25 reports the corresponding values of rdiffr_diff and rCCCr_C. Across all tested warmup lengths, approximate rgi remains consistently observed. (a) rdiffr_diff with 100100 warmup steps. (b) rdiffr_diff with 1,4001,400 warmup steps. (c) rCCCr_C with 100100 warmup steps. (d) rCCCr_C with 1,4001,400 warmup steps. Figure 25: Warmup-step ablation. Panels (a) and (b) plot rdiffr_diff for 100100 and 1,4001,400 warmup steps, while Panels (c) and (d) plot rCCCr_C for the same warmup settings. Approximate rgi holds robustly across the tested scheduler warmup lengths. The experiments in Section 5.1 use weight decay as 00. To evaluate the effect of the weight decay, we additionally evaluate all optimizers with weight decay 0.050.05 and 0.10.1. Figure 26 reports the corresponding values of rdiffr_diff and rCCCr_C. Across all tested weight-decay values, approximate rgi remains consistently observed. (a) rdiffr_diff with weight decay value 0.050.05. (b) rdiffr_diff with weight decay value 0.10.1. (c) rCCCr_C with weight decay value 0.050.05. (d) rCCCr_C with weight decay value 0.10.1. Figure 26: Weight-decay ablation. Panels (a) and (b) plot rdiffr_diff for 0.050.05 and 0.10.1 weight decay, while Panels (c) and (d) plot rCCCr_C for the same weight decay settings. Approximate rgi holds robustly across the tested weight decay values. B.5.2 Additional Results for Section 5.2 (a) rdiffr_diff under different initial parameters. (b) rdiffr_diff for hidden state dimensions 768768 and 384384. (c) rdiffr_diff for hidden state dimensions 768768 and 6464. Figure 27: rgi across optimizers, initializations, and model sizes. Panel (a) plots rdiffr_diff across optimizers with different initializations. Panels (b) and (c) plot rdiffr_diff across models with hidden-state dimensions 768768, 384384, and 6464. rgi continues to hold across different initializations and moderate changes in model size. (a) Values of rdiffr_diff between fa models with RoPE and NoPE. (b) Values of rdiffr_diff between fa models with RoPE and ALiBi. (c) Values of rdiffr_diff between fa and swa models. (d) Values of rdiffr_diff between models with GELU and SReLU. Figure 28: rgi across optimizers and models with different attention and ffn designs. Panels (a)–(c) plot rdiffr_diff between the RoPE fa baseline trained with Adam and NoPE fa, ALiBi fa, and RoPE swa models, respectively, each trained with all optimizers. Panel (d) plots rdiffr_diff between models using GELU and SReLU activations. rgi persists across different attention and ffn designs. (a) Values of rdiffr_diff between models with GELU and SiLU. (b) Values of rCCCr_C between models with GELU and SiLU. (c) Values of rdiffr_diff between models with gated and non-gated ffn. (d) Values of rCCCr_C between models with gated and non-gated ffn. Figure 29: rgi across optimizers and models with different ffn designs. Panels (a) and (b) plot rdiffr_diff and rCCCr_C, respectively, between the GELU baseline trained with Adam and SiLU models trained with other optimizers. Panels (c) and (d) plot rdiffr_diff and rCCCr_C, respectively, between the non-gated ffn baseline trained with Adam and gated ffn models trained with other optimizers. rgi persists across different ffn designs. B.5.3 Additional Results for Section 5.3 (a) Values of rdiffr_diff with Dl′D_l from C4. (b) Values of rdiffr_diff with Dl′D_l from ArXiv. (c) Values of rdiffr_diff with Dl′D_l from CodeParrot. Figure 30: rgi across optimizers trained on different data streams. The panels plot rdiffr_diff when Adam is trained on FineWeb and the compared optimizer is trained on C4 (Panel (a)), ArXiv (Panel (b)), or CodeParrot (Panel (c)). The results show that rgi is strongly affected by the training data stream: changing the data stream can substantially weaken, or even eliminate, rgi across optimizers. B.5.4 Additional Results for Section 5.4 (a) Values of rdiffr_diff with DvalD_val from C4. (b) Values of rdiffr_diff with DvalD_val from ArXiv. (c) Values of rdiffr_diff with DvalD_val from CodeParrot. (d) Values of rdiffr_diff with DvalD_val from Random. Figure 31: rgi across optimizers and fa/swa models evaluated on different validation datasets. The panels plot rdiffr_diff evaluated on C4, ArXiv, CodeParrot, and Random. These results show that rgi can transfer to related validation distributions, but becomes weaker on more distributionally distinct validation datasets. (a) Values of rdiffr_diff with DvalD_val from C4. (b) Values of rdiffr_diff with DvalD_val from ArXiv. (c) Values of rdiffr_diff with DvalD_val from CodeParrot. (d) Values of rdiffr_diff with DvalD_val from Random. (e) Values of rCCCr_C with DvalD_val from C4. (f) Values of rCCCr_C with DvalD_val from ArXiv. (g) Values of rCCCr_C with DvalD_val from CodeParrot. (h) Values of rCCCr_C with DvalD_val from Random. Figure 32: rgi across optimizers and gated/non-gated models evaluated on different validation datasets. The panels plot rdiffr_diff and rCCCr_C evaluated on C4, ArXiv, CodeParrot, and Random. These results show that rgi can transfer to related validation distributions, but becomes weaker on more distributionally distinct validation datasets. B.6 Additional Results for Section 6 B.6.1 Additional Results for Section 6.1 (a) Values of rℓ2r_ _2 between parameters. (b) Values of rℓ2r_ _2 between hidden states. Figure 33: Movement from initialization in parameter and hidden-state space. Panels (a) and (b) plot rℓ2(↓)r_ _2( ) between initialization and later checkpoints for parameters and hidden states, respectively. The large relative distances show substantial movement during training, providing evidence against an ntk-style lazy-training explanation. (a) Values of rℓ2r_ _2 between parameters. (b) Values of rℓ2r_ _2 between hidden states. Figure 34: Cross-optimizer distances in parameter and hidden-state space. Panels (a) and (b) plot rℓ2(↓)r_ _2( ) between Adam and the other optimizers at matched training steps for parameters and hidden states, respectively. The large relative distances show that different optimizers need not share similar parameters or hidden states even when their validation-loss differences exhibit rgi. B.6.2 Additional Results for Section 6.2 (a) Values of rdiffr_diff at width 576576. (b) Values of rdiffr_diff at width 384384. (c) Values of rdiffr_diff at width 192192. Figure 35: Model-width ablations for rdiffr_diff. Panels (a)–(c) plot rdiffr_diff for model widths 576576, 384384, and 192192, respectively. rgi holds across all tested widths, supporting the conclusion that the phenomenon is not restricted to the standard model width. Appendix C Discussions of Alternative Explanations for rgi In this section, we test and falsify several hypotheses that could potentially explain rgi. Specifically, we consider two hypotheses: • Intrinsic data-difficulty hypothesis. Suppose the loss decomposes as L(θ,d)=H(d)+ε(θ,d)L(θ,d)=H(d)+ (θ,d), where ε(θ,d) (θ,d) is much smaller than H(d)H(d). Then the relative loss structure is dominated by the intrinsic difficulty H(d)H(d) of the data and depends only weakly on the learned parameters θ. • Different convergence-speed hypothesis. rgi emerges because different optimizers generate the same learning trajectory but with different speed. We first consider the intrinsic data-difficulty hypothesis. For a validation example d=(xprefix,xpred)d=(x_prefix,x_pred), our work focuses on the token-level loss as L(θ,d)=−logPθ(xpred∣xprefix).L(θ,d)=- P_θ(x_pred x_prefix). Importantly, RGI does not average over xpredx_pred; it holds pointwise for each d∈vald _val. Consider any plausible loss decomposition L(θ,d)=H(d)+ε(θ,d)L(θ,d)=H(d)+ (θ,d) such that H(d),ε(θ,d)≥0H(d), (θ,d)≥ 0 for every d and θ, where H(d)H(d) is independent of the model. We have that 0≤H(d)=infθ[L(θ,d)−ε(θ,d)]≤infθL(θ,d). 0≤ H(d)= _θ [L(θ,d)- (θ,d) ]≤ _θL(θ,d). Therefore, the minimum achievable loss on d provides an upper bound on any model-independent component H(d)H(d). To estimate this bound, we directly optimize the LLM parameters θ on each data d and define the achieved loss as H^(d)=L(θ^d,d) H(d)=L( θ_d,d), where θ^d θ_d denotes the optimized parameters. Since infθL(θ,d)≤H^(d) _θL(θ,d)≤ H(d), we obtain H(d)≤H^(d)H(d)≤ H(d). We then report the average values of H^(d) H(d) and L(θt,d)L( _t,d), as well as their ratio, at t=10000t=10000, when L(θt,d)L( _t,d) attains its minimum along the training trajectory. Table 1 shows that the model-independent component H(d)H(d) is negligible, i.e., about 0.4%0.4\%, relative to the smallest observed loss L(θ,d)L(θ,d) along the trajectory. Thus, this falsifies the intrinsic data-difficulty hypothesis. Table 1: Comparison of H(d)H(d) and L(θt,d)L( _t,d) across optimizers. The results show that H(d)H(d) is negligible relative to L(θt,d)L( _t,d). Metric Adam-mini Adam Lion Muon Scion Soap Average H^(d) H(d) 0.001 0.001 0.001 0.001 0.001 0.001 Average L(θ10000,d)L( _10000,d) 3.500 3.457 3.554 3.409 3.459 3.403 Average H^(d)/L(θ10000,d) H(d)/L( _10000,d) 0.003 0.004 0.003 0.004 0.004 0.004 We then consider the different convergence-speed hypothesis. We falsify this hypothesis from two perspectives. First, we view this from the representation space. If the convergence-speed explanation were correct, models trained with different optimizers should reach similar parameters when matched at the same validation-loss level. Table 2: Parameter cosine similarity across optimizers with aligned validation loss. Adam step Validation loss Adam-mini Lion Muon Scion Shampoo SOAP 0 11.023 1.000 1.000 1.000 1.000 1.000 1.000 1000 4.261 0.773 0.753 0.641 0.612 0.396 0.559 2000 3.815 0.658 0.604 0.475 0.469 0.186 0.400 3000 3.654 0.580 0.520 0.425 0.411 0.150 0.303 4000 3.588 0.526 0.465 0.377 0.383 0.130 0.257 Table 2 compares models trained with different optimizers after aligning them to the validation loss achieved by Adam at each training step. We report results only for validation-loss levels greater than or equal to 3.553.55, because some optimizers do not achieve a loss below 3.553.55 during training. The results show that the similarity between validation-loss-aligned models decreases as training progresses. This finding provides evidence against the convergence-speed explanation. Second, we view the loss difference between optimizers. Table 3 reports the average loss difference between each optimizer A and Adam, defined as CA,Adam(t)=LA(t)−LAdam(t)C_A,Adam(t)=L_A(t)-L_Adam(t). We observe that CA,Adam(t)C_A,Adam(t) at t=2000t=2000, when rgi emerges, differs substantially from its value at convergence, i.e., at t=10000t=10000. This finding provides evidence against the convergence-speed explanation. Table 3: Loss difference relative to Adam, where CA,Adam(t)=LA(t)−LAdam(t)C_A,Adam(t)=L_A(t)-L_Adam(t). Step Adam-mini Lion Muon SOAP Scion 0 0.000 0.000 0.000 0.000 0.000 1000 0.321 0.738 -0.275 -0.341 -0.226 2000 0.129 0.276 -0.138 -0.164 -0.085 3000 0.092 0.181 -0.099 -0.117 -0.046 4000 0.073 0.130 -0.089 -0.100 -0.033 5000 0.067 0.107 -0.072 -0.083 -0.018 6000 0.056 0.089 -0.064 -0.075 -0.010 7000 0.044 0.073 -0.062 -0.070 -0.008 8000 0.041 0.078 -0.061 -0.065 -0.007 9000 0.041 0.090 -0.048 -0.053 -0.007 10000 0.041 0.081 -0.040 -0.041 -0.007 Appendix D Empirical Verification of Assumptions 7.1 and 7.2 To empirically assess this Assumptions 7.1, we estimate the target transformation at each layer as the best linear mapping between the hidden states of two adjacent layers after training. We compute H^l=argmin∑iA‖fil+1−Afil‖2 H^l= _A _i\|f_i^l+1-Af_i^l\|^2, where filf_i^l denotes the normalized hidden state of token i at layer l. We then calculate the angle a^i,jl a_i,j^l between columns H^:,i,l H_:,i^,l and H^:,j,l H_:,j^,l. The average angle at layer l is a^l=∑i≠ja^i,jl/(r2−r) a^l= _i≠ j a_i,j^l/(r^2-r), and the layer-averaged angle is a^=∑l∈[L]a^l/L a= _l∈[L] a^l/L. An average angle close to 90∘90 will provide empirical evidence that the estimated transformation approximately satisfies Assumption 7.1. Figure 36(a) verifies this by showing that a a for various optimizers is close to 90∘90 at the end of training. (a) Values of a a for models trained with different optimizers after training. (b) Values of b b for different optimizers at different training steps. Figure 36: Empirical assessment of Assumptions 7.1 and 7.2. Panel (a) shows that the columns of the targeted linear mappings are nearly orthogonal, while panel (b) shows that the hidden states of different tokens remain nearly orthogonal throughout training across all optimizers. These results provide empirical support for Assumptions 7.1 and 7.2 in practical llm training. Assumption 7.2 essentially requires the inputs to each layer to be isotropic within the subspace they span. Our proof continues to hold when ∑i=1nxixi⊤/n=Qdiag(σx2Ir,0)Q⊤ _i=1^nx_ix_i /n=Qdiag( _x^2I_r,0)Q , where r is the dimension of the subspace spanned by the inputs and Q∈ℝd×dQ ^d× d is an orthogonal matrix. A sufficient condition is that, up to an orthogonal change of coordinates, the input vectors are mutually orthogonal and have equal norms. Proposition D.1. Suppose x1,…,xn∈ℝdx_1,…,x_n ^d are nonzero, mutually orthogonal vectors with the same norm. Then ∑i=1nxixi⊤/n=Qdiag(σx2Ir,0)Q⊤ _i=1^nx_ix_i /n=Qdiag( _x^2I_r,0)Q for some σx>0 _x>0 and orthogonal matrix Q∈ℝd×dQ ^d× d, where r=nr=n. Since the input to each layer is normalized by RMSNorm, we focus on empirically assessing the directional orthogonality of the hidden states. Let filf_i^l denote the normalized hidden state of token i at layer l. We measure the pairwise directional similarity by b^i,jl b_i,j^l, which is the angle between filf_i^l and fjlf_j^l, and average b^i,jl b_i,j^l over distinct token pairs and layers to obtain the overall orthogonality measure b b. An average angle close to 90∘90 indicates greater directional orthogonality. Figure 36(b) confirms this along the entire training courses of various optimizers. Appendix E Proof of Proposition 4.2 We first prove that rgi is sufficient for constant generalization gap. From Definition 4.1, we have that L(θt,d)−L(θt′,d)=L(θt,d∗)−L(θt′,d∗), \!\!L ( _t T,d )\!-\!L ( _t T ,d )\!=\!L ( _t T,d^* )\!-\!L ( _t T ,d^* ), (E.1) where we fix d′d as any d∗d^*. Since the optimizers ,′ A, A are deterministic, the right-hand side of Eqn. (E.1) is a deterministic function of θ0,θ0′,D1:t,D1:t′ _0, _0 ,D_1:t,D_1:t . We write it as C(,′,t)C( T, T ,t). Thus, we have that L(θt,d)−L(θt′,d)=C(,′,t). L ( _t T,d )-L ( _t T ,d )=C( T, T ,t). Then we prove that the constant generalization gap is sufficient for rgi. This is direct, since the following holds L(θt,d)−L(θt′,d)=C(,′,t)=L(θt,d′)−L(θt′,d′). L ( _t T,d )-L ( _t T ,d )=C( T, T ,t)=L ( _t T,d )-L ( _t T ,d ). Thus, we conclude the proof of Proposition 4.2. Appendix F Proof of Theorem 7.3 We prove Theorem 7.3 in three steps. • Step 1: Derive closed-form expressions for the optimizer trajectories. • Step 2: Use these expressions to analyze the resulting losses. • Step 3: Characterize the cosine similarity between the learned weights. Step 1: Derive closed-form expressions for the optimizer trajectories We start by calculating the gradients of A and B. With Assumption 7.2, the loss can be simplified as follows. 12n∑i=1n‖(H−AB)xi‖22=12tr[(H−AB)(1n∑i=1nxixi⊤)(H−AB)⊤]=σx22‖H−AB‖F2. 12n _i=1^n\|(H-AB)x_i\|_2^2= 12tr [(H-AB) ( 1n _i=1^nx_ix_i )(H-AB) ]= _x^22\|H-AB\|_F^2. Therefore, the gradients of A and B are ∇AL=σx2(AB−H)B⊤,∇BL=σx2A⊤(AB−H). _AL= _x^2(AB-H)B , _BL= _x^2A (AB-H). (F.1) To analyze the trajectories of GD, Adam, and Muon, we decompose the hidden dimension s of parameters A∈ℝq×s,B∈ℝs×rA ^q× s,B ^s× r as ℝs=ℝr⊕ℝr⊕ℝs−2rR^s=R^r ^r ^s-2r. Accordingly, the parameters are decomposed as A=[A1,A2,A3],B=[B1B2B3], A=[A_1,A_2,A_3], B= bmatrixB_1\\ B_2\\ B_3 bmatrix, where A1,A2∈ℝq×rA_1,A_2 ^q× r, A3∈ℝq×(s−2r)A_3 ^q×(s-2r), B1,B2∈ℝr×rB_1,B_2 ^r× r, and B3∈ℝ(s−2r)×rB_3 ^(s-2r)× r. To define the trajectories of different optimizers, we introduce two auxiliary matrices G,J∈ℝq×rG,J ^q× r with orthonormal columns in the orthogonal complement of the target column space col(H)col(H). Specifically, we choose G and J such that G⊤G=J⊤J=Ir,H⊤G=H⊤J=0,G⊤J=0. G G=J J=I_r, H G=H J=0, G J=0. Such matrices exist provided that q−rank(H)≥2rq-rank(H)≥ 2r. In addition, since s≥3rs≥ 3r, we can choose a matrix E∈ℝ(s−2r)×rE ^(s-2r)× r with orthonormal columns: E⊤E=Ir. E E=I_r. Then we present the following proposition about the trajectories of different optimizers. Proposition F.1. For each optimizer ∈,, A∈\ GD, Adam, Muon\, initialize the parameters by A0=0,B0=[b0Ir00],b0>0.A_0 A=0, B_0 A= bmatrixb_0 AI_r\\ 0\\ 0 bmatrix, b_0 A>0. For any positive step sizes ηt,γt,τt>0 _t, _t, _t>0 for GD, Adam, and Muon, respectively, under the specified tie-breaking rules, the iterates remain in the following low-dimensional invariant subspaces. Namely, for every t≥0t≥ 0, there exist scalars at,bt,at,bt,λt,at,bt,λt∈ℝa_t GD,b_t GD,a_t Adam,b_t Adam, _t Adam,a_t Muon,b_t Muon, _t Muon such that At A_t GD =[atH,0,0],Bt=[btIr00], =[a_t GDH,0,0], B_t GD= bmatrixb_t GDI_r\\ 0\\ 0 bmatrix, (F.2) At A_t Adam =[atH,−λtG,0],Bt=[btIr00], =[a_t AdamH,- _t AdamG,0], B_t Adam= bmatrixb_t AdamI_r\\ 0\\ 0 bmatrix, (F.3) At A_t Muon =[atH,0,−λtJE⊤],Bt=[btIr00]. =[a_t MuonH,0,- _t MuonJE ], B_t Muon= bmatrixb_t MuonI_r\\ 0\\ 0 bmatrix. (F.4) Proof. To prove Proposition F.1, we introduce the structured family of parameters that will serve as an invariant class for the optimizers’ trajectories. A=[aH,−λGG,−λJJE⊤],B=[bIr00], A=[aH,\,- _GG,\,- _JJE ], B= bmatrixbI_r\\ 0\\ 0 bmatrix, (F.5) where a,b,λG,λJ∈ℝa,b, _G, _J are real numbers. Then, with c=abc=ab, we have that AB−H=(c−1)H.AB-H=(c-1)H. Substituting this into Eqn. (F.1) gives ∇AL=σx2(c−1)b[H,0,0]. _AL= _x^2(c-1)b\,[H,0,0]. (F.6) For the gradient of B, the orthogonality conditions on H,G,JH,G,J derive that A⊤H=[aH⊤H−λGG⊤H−λJEJ⊤H]=[aμ2Ir00], A H= bmatrixaH H\\ - _GG H\\ - _JEJ H bmatrix= bmatrixaμ^2I_r\\ 0\\ 0 bmatrix, where the parameter μ results from Assumption 7.1. Thus, the gradient of B is ∇BL=σx2(c−1)[aμ2Ir00]. _BL= _x^2(c-1) bmatrixaμ^2I_r\\ 0\\ 0 bmatrix. (F.7) In the following, we will prove that GD, Adam, and Muon maintain structured weights defined in Proposition F.1 with induction. We start with GD. The initialization of GD satisfies Eqn. (F.2) with at=0a_t GD=0 and bt=b0b_t GD=b_0 GD for t=0t=0. We assume that Eqn. (F.2) holds for l=0,⋯,tl=0,·s,t. The parameters at step t+1t+1 is updated as At+1=At−ηt∇Aℒ(At,Bt),Bt+1=Bt−ηt∇Bℒ(At,Bt). A_t+1 GD=A_t GD- _t _AL(A_t GD,B_t GD), B_t+1 GD=B_t GD- _t _BL(A_t GD,B_t GD). Denote ct=atbtc_t GD=a_t GDb_t GD, by Eqns. (F.6) and (F.7), we have that At+1A_t+1 GD and Bt+1B_t+1 GD satisfy Eqn. (F.2) with at+1 a_t+1 GD =at−ηtσx2(ct−1)bt, =a_t GD- _t _x^2(c_t GD-1)b_t GD, bt+1 b_t+1 GD =bt−ηtσx2μ2(ct−1)at. =b_t GD- _t _x^2μ^2(c_t GD-1)a_t GD. Thus, we prove the desired result for GD. We then analyze Adam. Since the Adam optimizer takes the element-wise sign of gradient, we consider the following tie-breaking rule (M)=Z:Zij=sign(Mij)if Mij≠0,Zij∈[−1,1]if Mij=0, Sgn(M)= \Z:\;Z_ij=sign(M_ij)\ if M_ij≠ 0, Z_ij∈[-1,1]\ if M_ij=0 \, where sign(x)sign(x) denotes the sign of x, with sign(0)=0sign(0)=0. Then the update of Adam is At+1=At−γtΞtA,Bt+1=Bt−γtΞtB, A_t+1 Adam=A_t Adam- _t _t^A, B_t+1 Adam=B_t Adam- _t _t^B, where ΞtA _t^A and ΞtB _t^B are plausible updates, i.e., ΞtA∈(∇Aℒ),ΞtB∈(∇Bℒ). _t^A∈ Sgn( _AL), _t^B∈ Sgn( _BL). We then prove that this update rule satisfies Eqn. (F.3). We first note that the initialization satisfies Eqn. (F.3) with at=λt=0a_t Adam= _t Adam=0, bt=b0b_t Adam=b_0 Adam for t=0t=0. Then we assume that Eqn. (F.3) holds for l=0,⋯,tl=0,·s,t. Eqs. (F.6) and (F.7). Using sign(H)=κHsign(H)=κ H imply that we can choose ΞtA=sign((ct−1)bt)κ[H,0,0]+[0,G,0], and ΞtB=sign((ct−1)at)[Ir00]. _t^A=sign((c_t Adam-1)b_t Adam)κ[H,0,0]+[0,G,0], and _t^B=sign((c_t Adam-1)a_t Adam) bmatrixI_r\\ 0\\ 0 bmatrix. The update [0,G,0][0,G,0] term in ΞtA _t^A is admissible because it is selected on a zero-gradient block and the entries of G lie in [−1,1][-1,1]. Denote ct=atbtc_t Adam=a_t Adamb_t Adam, we have that At+1A_t+1 Adam and Bt+1B_t+1 Adam satisfy Eqn. (F.3) with at+1 a_t+1 Adam =at−γtκsign((ct−1)bt), =a_t Adam- _t ((c_t Adam-1)b_t Adam), (F.8) bt+1 b_t+1 Adam =bt−γtsign((ct−1)at), =b_t Adam- _tsign((c_t Adam-1)a_t Adam), (F.9) λt+1 _t+1 Adam =λt+γt. = _t Adam+ _t. (F.10) Thus, we prove the desired result for Adam. Finally, we consider Muon. Given a matrix M∈ℝm×nM ^m× n with rank r, let M=UΣV⊤M=U V be its compact SVD over the nonzero singular values, where Σ∈ℝr×r ^r× r, U∈ℝm×rU ^m× r, and V∈ℝn×rV ^n× r. We define the spectral normalization operator by (M)=Z∈ℝm×n:ZV=U,Z⊤U=V,∥Z∥2≤1, Spec(M)= \Z ^m× n:\;ZV=U, Z U=V, \|Z\|_2≤ 1 \, where ∥⋅∥2\|·\|_2 denotes the spectral norm of matrices. Specifically, when M=0M=0, this definition gives (0)=Z:‖Z‖2≤1 Spec(0)=\Z:\|Z\|_2≤ 1\. The Muon optimizer then updates the parameters as At+1=At−τtTtA,Bt+1=Bt−τtTtB, A_t+1 Muon=A_t Muon- _tT_t^A, B_t+1 Muon=B_t Muon- _tT_t^B, where TtAT_t^A and TtBT_t^B are plausible spectrally normalized updates, i.e., TtA∈(∇Aℒ),TtB∈(∇Bℒ). T_t^A∈ Spec( _AL), T_t^B∈ Spec( _BL). In the following, we will prove that the parameters updated by Muon satisfiy Eqn. (F.4) via induction. The intialization of Muon satifies Eqn. (F.4) with at=λt=0a_t Muon= _t Muon=0, bt=b0b_t Muon=b_0 Muon for t=0t=0. We assume that Eqn. (F.4) holds for l=0,⋯,tl=0,·s,t. Since H⊤H=μ2IrH H=μ^2I_r, the normalized active direction associated with H is H/μH/μ. Then we can choose TtAT_t^A and TtBT_t^B as follows. TtA=sign((ct−1)bt)[1μH,0,0]+[0,0,JE⊤], and TtB=sign((ct−1)at)[Ir00]. T_t^A=sign((c_t Muon-1)b_t Muon)\!\! [ 1μH,0,0 ]\!+\![0,0,JE ], and T_t^B=sign((c_t Muon-1)a_t Muon)\!\! bmatrixI_r\\ 0\\ 0 bmatrix. The completion [0,0,JE⊤][0,0,JE ] acts only on a zero singular subspace. Moreover, H⊤J=0H J=0 and E⊤E=IrE E=I_r imply that [H/μ,0,0][H/μ,0,0] and [0,0,JE⊤][0,0,JE ] have orthogonal left and right singular subspaces. Therefore the selected TtAT_t^A has spectral norm one and is admissible. With ct=atbtc_t Muon=a_t Muonb_t Muon, At+1A_t+1 Muon and Bt+1B_t+1 Muon satisfy Eqn. (F.4) with at+1 a_t+1 Muon =at−τtμsign((ct−1)bt), =a_t Muon- _tμsign((c_t Muon-1)b_t Muon), bt+1 b_t+1 Muon =bt−τtsign((ct−1)at), =b_t Muon- _tsign((c_t Muon-1)a_t Muon), λt+1 _t+1 Muon =λt+τt. = _t Muon+ _t. Thus, we conclude the proof of Proposition F.1 ∎ Step 2: Use these expressions to analyze the resulting losses. Proposition F.1 shows that for any ∈,, A∈\ GD, Adam, Muon\, we have that AtBt=atbtH. A_t AB_t A=a_t Ab_t AH. Thus, with ct=atbtc_t A=a_t Ab_t A, we obtain ℒt(x)=12(1−ct)2‖Hx‖22. _t A(x)= 12(1-c_t A)^2\|Hx\|_2^2. Since Assumption 7.1 requires that H⊤H=μ2IrH H=μ^2I_r, every input point satisfying ‖x‖2=ρ.\|x\|_2=ρ. Hence ℒt(x)=μ2ρ22(1−ct)2. _t A(x)= μ^2ρ^22(1-c_t A)^2. The right-hand side may depend on the optimizer A and on time t, but it is constant over all points on the sphere ‖x‖2=ρ\|x\|_2=ρ. Therefore, for any two points x,x′x,x with ‖x‖2=‖x′‖2=ρ\|x\|_2=\|x \|_2=ρ, and any two optimizers ,′∈,, A, A ∈\ GD, Adam, Muon\, we have Lt(x)−Lt(x′)=Lt′(x)−Lt′(x′). L_t A(x)-L_t A(x )=L_t A (x)-L_t A (x ). This proves exact rgi on the validation sphere. Step 3: Characterize the cosine similarity between the learned weights. We denote the vectorized parameters as θt=(vec(At),vec(Bt)) _t T=(vec(A_t A),vec(B_t A)), where vec(⋅)vec(·) vectorizes the matrix parameter. Then the cosine similarity between parameters learned by different optimizers can be calculated as follows. Proposition F.2. Under the same conditions of Proposition F.1, the cosine similarities between parameters learned by GD, Adam, and Muon are cos-sim(θt,θt) -sim( _t GD, _t Adam) =μ2atat+btbtwt⋅wt, = μ^2a_t GDa_t Adam+b_t GDb_t Adamw_t GD· w_t Adam, cos-sim(θt,θt) -sim( _t GD, _t Muon) =μ2atat+btbtwt⋅wt, = μ^2a_t GDa_t Muon+b_t GDb_t Muonw_t GD· w_t Muon, cos-sim(θt,θt) -sim( _t Adam, _t Muon) =μ2atat+btbtwt⋅wt, = μ^2a_t Adama_t Muon+b_t Adamb_t Muonw_t Adam· w_t Muon, where the denominators are defined as wt=μ2(at)2+(bt)2w_t GD= μ^2(a_t GD)^2+(b_t GD)^2, wt=μ2(at)2+(bt)2+(λt)2w_t Adam= μ^2(a_t Adam)^2+(b_t Adam)^2+( _t Adam)^2, and wt=μ2(at)2+(bt)2+(λt)2w_t Muon= μ^2(a_t Muon)^2+(b_t Muon)^2+( _t Muon)^2. Each displayed ratio applies when its denominator is nonzero; otherwise, the corresponding similarity is zero by convention. Proof. To compute the cosine similarity between two parameter vectors, we normalize their inner product by the product of their norms. We first record several basic identities for the corresponding inner products and norms. Specifically, we have ⟨[H,0,0],[0,G,0]⟩F=0,⟨[H,0,0],[0,0,JE⊤]⟩F=0,⟨[0,G,0],[0,0,JE⊤]⟩F=0. [H,0,0],[0,G,0] _F=0, [H,0,0], [0,0,JE ] _F=0, [0,G,0], [0,0,JE ] _F=0. Moreover, since H⊤H=μ2Ir,G⊤G=Ir,J⊤J=Ir,E⊤E=IrH H=μ^2I_r,G G=I_r,J J=I_r,E E=I_r, we have that ‖[H,0,0]‖F2=μ2r,‖[0,G,0]‖F2=‖G‖F2=r,‖[0,0,JE⊤]‖F2=r. \|[H,0,0]\|_F^2=μ^2r, \|[0,G,0]\|_F^2=\|G\|_F^2=r, \|[0,0,JE ]\|_F^2=r. Thus, for GD and Adam, the inner product between parameters are ⟨θt,θt⟩=r(μ2atat+btbt). _t GD, _t Adam =r (μ^2a_t GDa_t Adam+b_t GDb_t Adam ). Moreover, the norms of the parameters are ‖θt‖22=r(μ2(at)2+(bt)2),‖θt‖22=r(μ2(at)2+(bt)2+(λt)2). \| _t GD\|_2^2=r (μ^2(a_t GD)^2+(b_t GD)^2 ), \| _t Adam\|_2^2=r (μ^2(a_t Adam)^2+(b_t Adam)^2+( _t Adam)^2 ). Combining these, we have that cos-sim(θt,θt)=μ2atat+btbtwt⋅wt. -sim( _t GD, _t Adam)= μ^2a_t GDa_t Adam+b_t GDb_t Adamw_t GD· w_t Adam. Similarly, the cosine similarity between the parameters of GD and Muon is cos-sim(θt,θt)=μ2atat+btbtwt⋅wt. -sim( _t GD, _t Muon)= μ^2a_t GDa_t Muon+b_t GDb_t Muonw_t GD· w_t Muon. For the cosine similarity between the parameters learned by Adam and Muon, we have that ⟨[0,G,0],[0,0,JE⊤]⟩F=0. [0,G,0],[0,0,JE ] _F=0. Then we calculate the cosine similarity as cos-sim(θt,θt)=μ2atat+btbtwt⋅wt. -sim( _t Adam, _t Muon)= μ^2a_t Adama_t Muon+b_t Adamb_t Muonw_t Adam· w_t Muon. Thus, we conclude the proof of Proposition F.2. ∎ In the following, we will prove that ata_t and btb_t of Adam and Muon are uniformly bounded, whereas λt _t of them grows linearly. We use constant step sizes for them, i.e., γt=γ,τt=τ. _t=γ, _t=τ. First, we consider Adam. Recall that we initialize the parameter with a0=0,λ0=0,b0>0a_0 Adam=0, _0 Adam=0,b_0 Adam>0. Since c0=a0b0=0c_0 Adam=a_0 Adamb_0 Adam=0, the first update gives a1 a_1 Adam =a0−γκsign((c0−1)b0)=κγ, =a_0 Adam-γ ((c_0 Adam-1)b_0 Adam )=κγ, b1 b_1 Adam =b0−γsign((c0−1)a0)=b0, =b_0 Adam- ((c_0 Adam-1)a_0 Adam )=b_0 Adam, where the equalities result from Eqn. (F.8) and (F.9). Thus, the first update leads to a1>0,b1>0a_1 Adam>0,b_1 Adam>0. We then define Δ=a1−κb1=κ(γ−b0),D=|Δ|,r=1D+κ. _ Adam=a_1 Adam-κ b_1 Adam=κ(γ-b_0 Adam), D_ Adam=| _ Adam|, r_ Adam= 1D_ Adam+ κ. Then we have the following results about Adam. Proposition F.3. With the step-size condition γt=γ<minb0,(κb0+κ)−1 _t=γ< \b_0 Adam,(κ b_0 Adam+ κ)^-1 \, we have that at>0,bt>0a_t Adam>0,b_t Adam>0 and at−κbt=Δa_t Adam-κ b_t Adam= _ Adam hold for all t≥1t≥ 1. Proof. We prove the desired results by induction. The results trivially hold for t=1t=1. Suppose the claims hold at time t≥1t≥ 1. Since at>0a_t Adam>0 and bt>0b_t Adam>0, we have sign((ct−1)bt)=sign(ct−1),sign((ct−1)at)=sign(ct−1). ((c_t Adam-1)b_t Adam )=sign(c_t Adam-1), ((c_t Adam-1)a_t Adam )=sign(c_t Adam-1). Therefore, Eqn. (F.8) and (F.9) show that at+1=at−γκsign(ct−1)a_t+1 Adam=a_t Adam-γ (c_t Adam-1), and bt+1=bt−γsign(ct−1)b_t+1 Adam=b_t Adam- (c_t Adam-1). It follows that at+1−κbt+1=at−κbt=Δ. a_t+1 Adam-κ b_t+1 Adam=a_t Adam-κ b_t Adam= _ Adam. It remains to show that the next iterate stays in the positive quadrant. Since at=κbt+Δa_t Adam=κ b_t Adam+ _ Adam, if 0<bt≤r0<b_t Adam≤ r_ Adam, then ct=(κbt+Δ)bt≤κ(bt)2+Dbt≤κr2+Dr. c_t Adam= (κ b_t Adam+ _ Adam )b_t Adam≤κ(b_t Adam)^2+D_ Adamb_t Adam≤κ r_ Adam^2+D_ Adamr_ Adam. By the definition of r_ Adam, we have κr2+Dr=κ(D+κ)2+D+κ≤1. κ r_ Adam^2+D_ Adamr_ Adam= κ(D_ Adam+ κ)^2+ D_ AdamD_ Adam+ κ≤ 1. Hence, we can show that if ct>1c_t Adam>1, we have bt>rb_t Adam>r_ Adam. Similarly, since bt=(at−Δ)/κb_t Adam=(a_t Adam- _ Adam)/κ, if 0<at≤κr0<a_t Adam≤κ r_ Adam, then ct=at(at−Δ)κ≤at(at+D)κ≤κr2+Dr≤1. c_t Adam= a_t Adam (a_t Adam- _ Adam )κ≤ a_t Adam (a_t Adam+D_ Adam )κ≤κ r_ Adam^2+D_ Adamr_ Adam≤ 1. Hence, we also show that if ct>1c_t Adam>1, we have at>κra_t Adam>κ r_ Adam. Now consider two cases. If ct≤1c_t Adam≤ 1, then sign(ct−1)≤0sign(c_t Adam-1)≤ 0, by the update rule in Eqn. (F.8) and (F.9), we have at+1≥at>0,bt+1≥bt>0. a_t+1 Adam≥ a_t Adam>0, b_t+1 Adam≥ b_t Adam>0. If ct>1c_t Adam>1, then we have that bt>r,at>κrb_t Adam>r_ Adam,a_t Adam>κ r_ Adam. Since step-size condition γt=γ<minb0,(κb0+κ)−1 _t=γ< \b_0 Adam,(κ b_0 Adam+ κ)^-1 \ implies 0<γ<r0<γ<r_ Adam, we have bt+1=bt−γ>r−γ>0,at+1=at−κγ>κr−κγ>0. b_t+1 Adam=b_t Adam-γ>r_ Adam-γ>0, a_t+1 Adam=a_t Adam-κγ>κ r_ Adam-κγ>0. Thus, the positive quadrant is invariant for Adam. We conclude the proof of Proposition F.3. ∎ In the following, we will show that ata_t Adam and btb_t Adam are bounded. Define R=max2Dκ,2κ,M=maxb0,R+γ. R_ Adam= \ 2D_ Adamκ, 2κ \, M_ Adam= \b_0 Adam,R_ Adam+γ \. We prove by induction that bt≤Mb_t Adam≤ M_ Adam for all t. The claim holds at t=0t=0. Suppose bt≤Mb_t Adam≤ M_ Adam. If bt≥Rb_t Adam≥ R_ Adam, then we have that at=κbt+Δ≥κbt−D≥κ2bt, a_t Adam=κ b_t Adam+ _ Adam≥κ b_t Adam-D_ Adam≥ κ2b_t Adam, where the first equality results from Proposition F.3. Therefore, we have ct=atbt≥κ/2⋅(bt)2≥1c_t Adam=a_t Adamb_t Adam≥κ/2·(b_t Adam)^2≥ 1. Hence, we have if bt≥Rb_t Adam≥ R_ Adam, then bt+1≤bt≤Mb_t+1 Adam≤ b_t Adam≤ M_ Adam. If bt<Rb_t Adam<R_ Adam, then we have that bt+1=bt−γsign(ct−1)≤bt+γ<R+γ≤M. b_t+1 Adam=b_t Adam- (c_t Adam-1)≤ b_t Adam+γ<R_ Adam+γ≤ M_ Adam. Therefore, we show that bt≤Mb_t Adam≤ M_ Adam for all t. Furthermore, since at=κbt+Δa_t Adam=κ b_t Adam+ _ Adam, we also have 0<at≤κM+D. 0<a_t Adam≤κ M_ Adam+D_ Adam. Thus, we show that at,bt=O(1)a_t Adam,b_t Adam=O(1) with t increasing. Second, we consider Muon. The parameters are initialized with a0=0,λt=0,b0>0a_0 Muon=0, _t Muon=0,b_0 Muon>0. We then define Δ=a1−1μb1,D=|Δ|,r=1D+μ−1/2. _ Muon=a_1 Muon- 1μb_1 Muon, D_ Muon=| _ Muon|, r_ Muon= 1D_ Muon+μ^-1/2. The step-size condition of τt=τ<minb0,(b0/μ+μ−1/2)−1 _t=τ< \b_0 Muon,(b_0 Muon/μ+μ^-1/2)^-1 \ implies 0<τ<r0<τ<r_ Muon. Similar to Proposition F.3 for Adam, we can also show by induction that at>0,bt>0a_t Muon>0,b_t Muon>0 and at−μ−1bt=Δa_t Muon-μ^-1b_t Muon= _ Muon for all t≥1t≥ 1. For the boundness, we define R=max2μD,2μ,M=maxb0,R+τ. R_ Muon= \2μ D_ Muon, 2μ \, M_ Muon= \b_0 Muon,R_ Muon+τ \. Similar to our proof for Adam, we can show by induction that bt≤Mb_t Muon≤ M_ Muon for all t≥1t≥ 1. Since at=μ−1bt+Δa_t Muon=μ^-1b_t Muon+ _ Muon, we also have 0<at≤μ−1M+D0<a_t Muon≤μ^-1M_ Muon+D_ Muon. Thus, we have at,bt=O(1)a_t Muon,b_t Muon=O(1) when t increases. In the following, we then derive the bounds for the cosine similarities between parameters. Since λt=λ0+γt _t Adam= _0 Adam+γ t and λt=λ0+τt _t Muon= _0 Muon+τ t, we have that |λt|=Θ(t),|λt|=Θ(t). | _t Adam|= (t), | _t Muon|= (t). From the boundedness proved above, there exist finite constants C,C_ Adam,C_ Muon such that for all t, μ2(at)2+(bt)2≤C2,μ2(at)2+(bt)2≤C2. μ^2(a_t Adam)^2+(b_t Adam)^2≤ C_ Adam^2, μ^2(a_t Muon)^2+(b_t Muon)^2≤ C_ Muon^2. Thus, for GD and Adam, we have |cos-sim(θt,θt)|≤μ2(at)2+(bt)2μ2(at)2+(bt)2+(λt)2≤C|λt|=O(t−1), |cos -sim( _t GD, _t Adam) |≤ μ^2(a_t Adam)^2+(b_t Adam)^2 μ^2(a_t Adam)^2+(b_t Adam)^2+( _t Adam)^2≤ C_ Adam| _t Adam|=O(t^-1), where the first inequality results from the Cauchy-Schwarz inequality, and the second inequality results from the boundedness of ata_t Adam and btb_t Adam. Similarly, we have that |cos-sim(θt,θt)|≤C|λt|=O(t−1),|cos-sim(θt,θt)|≤CC|λt||λt|=O(t−2). |cos -sim( _t GD, _t Muon) |\!≤\! C_ Muon| _t Muon|=O(t^-1),\, |cos -sim( _t Adam, _t Muon) |\!≤\! C_ AdamC_ Muon| _t Adam|| _t Muon|=O(t^-2). Thus, we conclude the proof of Theorem 7.3.