Paper deep dive
DGA-Muon: Decoupled Geometry-Aligned Adaptive Scaling for Muon
Wenpeng Zhang, Runsheng Yu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/8/2026, 6:17:23 AM
Summary
This paper analyzes NorMuon, an adaptive variant of the Muon optimizer, revealing that its adaptivity stems from orthogonalization-induced geometry rather than optimization dynamics. The authors identify an 'Orthogonalization-Adaptivity Paradox' where more accurate orthogonalization weakens adaptivity. They propose DGA-Muon, which decouples adaptive scaling from orthogonalization and aligns scaling with matrix shape (row-wise for wide, column-wise for tall), proving convergence and demonstrating superior empirical performance.
Entities (6)
Relation Signals (5)
DGA-Muon → improvesupon → NorMuon
confidence 95% · We further demonstrate that DGA-Muon substantially outperforms NorMuon in an end-to-end NanoChat evaluation
DGA-Muon → isbasedon → Muon
confidence 95% · By incorporating several other techniques, we obtain Decoupled Geometry-Aligned Muon (DGA-Muon).
NorMuon → suffersfrom → Orthogonalization-Adaptivity Paradox
confidence 92% · giving rise to a counterintuitive Orthogonalization--Adaptivity Paradox: more accurate orthogonalization weakens adaptivity.
NorMuon → uses → Newton-Schulz
confidence 85% · NorMuon employs approximate orthogonalization via a finite number of Newton–Schulz iterations
Muon → uses → Newton-Schulz
confidence 85% · Muon avoids the high cost of computing the full SVD by approximating the orthogonalization using efficient Newton–Schulz iterations
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While NorMuon has achieved strong empirical performance in pretraining, its underlying adaptive mechanism remains largely heuristic and poorly understood. In this work, we provide the first systematic theoretical analysis of NorMuon's adaptivity, revealing that it primarily arises from orthogonalization-induced geometry rather than genuine optimization dynamics, serving to offset the resulting geometric non-uniformity. Under exact orthogonalization, the adaptive scaling factors degenerate into a single global scalar for square and wide matrices, while for tall matrices their variation results from unevenly distributed row energy after orthogonalization. Under approximate orthogonalization, the orthogonality residuals introduce additional variation into the scaling, giving rise to a counterintuitive Orthogonalization--Adaptivity Paradox: more accurate orthogonalization weakens adaptivity. We further show that NorMuon's rigid row-wise scaling is geometrically misaligned with the one-sided orthogonal structure of tall matrices by distorting column orthogonality. Motivated by these limitations, we propose two core design principles that a desirable adaptive mechanism for Muon should satisfy. First, adaptive scaling should be decoupled from orthogonalization, with the scaling factors computed directly from raw gradients. Second, adaptive scaling should be aligned with the shape-dependent orthogonal structure of the polar factor, using row-wise scaling for wide matrices and column-wise scaling for tall matrices. We prove that this geometry-aligned scaling preserves the orthogonal structure of the update. By incorporating several other techniques, we obtain Decoupled Geometry-Aligned Muon (DGA-Muon). We establish convergence guarantees for DGA-Muon and empirically validate both our theoretical characterization of NorMuon's scaling degeneration and the superiority of DGA-Muon.
Tags
Links
- Source: https://arxiv.org/abs/2610.06578v2
- Canonical: https://arxiv.org/abs/2610.06578v2
Trouble viewing inline? Open PDF directly →
Full Text
84,463 characters extracted from source content.
Expand or collapse full text
DGA-Muon: Decoupled Geometry-Aligned Adaptive Scaling for Muon Wenpeng Zhang* Affiliation: Independent Reasearcher Email: zhangwenpeng0@gmail.com Runsheng Yu* Affiliation: Independent Reasearcher Email: runshengyu@gmail.com Abstract While NorMuon has achieved strong empirical performance in pretraining, its underlying adaptive mechanism remains largely heuristic and poorly understood. In this work, we provide the first systematic theoretical analysis of NorMuon’s adaptivity, revealing that it primarily arises from orthogonalization-induced geometry rather than genuine optimization dynamics, serving to offset the resulting geometric non-uniformity. Under exact orthogonalization, the adaptive scaling factors degenerate into a single global scalar for square and wide matrices, while for tall matrices their variation results from unevenly distributed row energy after orthogonalization. Under approximate orthogonalization, the orthogonality residuals introduce additional variation into the scaling, giving rise to a counterintuitive Orthogonalization–Adaptivity Paradox: more accurate orthogonalization weakens adaptivity. We further show that NorMuon’s rigid row-wise scaling is geometrically misaligned with the one-sided orthogonal structure of tall matrices by distorting column orthogonality. Motivated by these limitations, we propose two core design principles that a desirable adaptive mechanism for Muon should satisfy. First, adaptive scaling should be decoupled from orthogonalization, with the scaling factors computed directly from raw gradients. Second, adaptive scaling should be aligned with the shape-dependent orthogonal structure of the polar factor, using row-wise scaling for wide matrices and column-wise scaling for tall matrices. We prove that this geometry-aligned scaling preserves the orthogonal structure of the update. By incorporating several other techniques, we obtain Decoupled Geometry-Aligned Muon (DGA-Muon). We establish convergence guarantees for DGA-Muon and empirically validate both our theoretical characterization of NorMuon’s scaling degeneration and the superiority of DGA-Muon. 1 introduction The Muon (MomentUm Orthogonalized by Newton–Schulz) optimizer (Jordan, 2024), specifically designed for optimizing two-dimensional parameters in neural networks, first gained traction through its remarkable empirical success in the Modded-NanoGPT speedrun (Jordan et al., 2024), where it demonstrated substantially faster convergence than AdamW (Loshchilov and Hutter, 2019). Unlike AdamW, which treats weight matrices as flattened one-dimensional vectors, Muon preserves their inherent two-dimensional structure and applies an orthogonalization operation to equalize their singular values. Following this initial success, subsequent studies (Liu et al., 2025; Shah et al., 2025) further validated Muon’s advantages over AdamW in the pretraining of multi-billion-parameter language models. To date, Muon has been widely adopted for training frontier foundation models, including open-weight models such as Kimi (Team et al., 2025; Team et al., 2026), GLM (Zeng et al., 2025; Zeng et al., 2026), and DeepSeek (Xu et al., 2026a). The remarkable success of Muon has sparked a surge of research interest. One line of research has sought to understand the mechanisms underlying Muon’s superior performance (Nguyen et al., 2026; Wang et al., 2026a; Wang et al., 2026b) and to establish theoretical guarantees for its convergence (Li and Hong, 2025; Kim and Oh, 2026). An even larger body of research is devoted to further enhancing the practical performance of Muon. One prominent branch of these efforts focuses on developing new methods that improve the accuracy or efficiency of the orthogonalization procedure (Amsel et al., 2026; Zhang et al., 2026a; Liu et al., 2026). Another promising direction explores adaptive preconditioning for Muon to better account for the geometry of the loss landscape. Conceptually, this mirrors how adaptive optimizers such as AdaGrad (Duchi et al., 2011) and Adam (Kingma and Ba, 2015) improved upon vanilla SGD. More fundamentally, a theoretical study (Su, 2025) suggests that the perfectly uniform spectral flattening induced by Muon may not be optimal, providing a principled foundation for incorporating adaptive preconditioning into Muon. While several initial attempts have been made, most of these methods either incur prohibitive computational or memory costs (e.g., requiring expensive matrix decompositions or maintaining full-matrix preconditioners) (Xu et al., 2026b; Zhang et al., 2026c; Song et al., 2026) or degrade to a simplistic single global scaling factor (Zhang et al., 2025a). Striking an effective balance between computational efficiency and adaptive expressiveness, NorMuon (Li et al., 2026) has emerged as the most widely adopted adaptive Muon variant. By employing a lightweight yet fine-grained neuron-wise (i.e., row-wise) scaling strategy, NorMuon consistently outperforms vanilla Muon across various settings. Notably, it achieves state-of-the-art performance on two leading speedruns: Modded-NanoGPT and NanoChat (Karpathy, 2025). A distinctive characteristic of NorMuon is that its adaptive scaling factors are computed from the orthogonalized momentum, rather than directly from historical gradients as in conventional adaptive optimizers such as AdaGrad (Duchi et al., 2011), Adam (Kingma and Ba, 2015), and Shampoo (Gupta et al., 2018). This design tightly couples adaptivity with the orthogonalization procedure, constituting a fundamentally different paradigm for adaptive optimization. However, this design choice is largely heuristic, and the rationale for using orthogonalized momentum as the source of adaptivity lacks theoretical justification. Thus far, neither theoretical convergence guarantees for NorMuon nor a principled characterization of its adaptive behavior have been established, leaving this unconventional adaptive mechanism poorly understood. In this work, we provide the first systematic theoretical analysis of NorMuon’s adaptivity. To understand its nature, we analyze the variation of NorMuon’s adaptive scaling factors, asking what information is encoded in their differences and what drives this variation. Surprisingly, we find that NorMuon’s adaptivity does not originate from conventional optimization dynamics, but instead arises primarily from the geometry induced by orthogonalization. To see this clearly, recall that NorMuon’s scaling factors are derived from the squared ℓ2 _2-norms of the rows of the orthogonalized matrix. Although orthogonalization equalizes singular values, it does not necessarily imply uniform row-wise norms. In fact, under exact orthogonalization, this row-wise non-uniformity constitutes the primary source of variation in the scaling factors. However, such variation arises exclusively for tall matrices, where orthogonalization enforces only one-sided orthogonality, namely column orthonormality, but not row orthonormality. Consequently, while the total squared ℓ2 _2-norm across all rows is fixed, the squared ℓ2 _2-norm of each individual row remains unconstrained, allowing the row-wise scaling factors to vary (see Subsection 3.3 for a full derivation). In contrast, square and wide matrices satisfy row orthonormality, which ensures that the squared ℓ2 _2-norm of every row is exactly one. Consequently, the row-wise adaptive scaling factors become identical across rows, and the adaptive scaling completely degenerates into a single global factor, thereby nullifying the intended row-wise adaptivity (see Subsection 3.1 for detailed analysis). In practical implementations, orthogonalization is typically approximate. The resulting orthogonality residuals introduce an additional source of variation into the scaling factors. Notably, this gives rise to a rather counterintuitive and undesirable phenomenon, which we term the Orthogonalization–Adaptivity Paradox: the more accurate the orthogonalization, the weaker the resulting adaptivity, which vanishes entirely under exact orthogonalization for square and wide matrices; conversely, less accurate orthogonalization permits greater variation in the scaling factors and thus stronger adaptivity (see Subsections 3.2 and 3.3 for details). Together, these observations reveal a fundamental limitation of NorMuon: its adaptive scaling primarily acts as a compensatory mechanism that offsets geometric non-uniformity induced by orthogonalization, rather than capturing genuine optimization-relevant information. The preceding analysis of NorMuon’s inherent limitation suggests that a desirable adaptive mechanism for Muon should preserve NorMuon’s computational efficiency while deriving adaptivity from optimization-relevant signals rather than orthogonalization-induced geometry. At its core, NorMuon’s limitation arises from computing scaling factors from the orthogonalized momentum, which inevitably causes the resulting adaptivity to inherit geometric distortions induced by orthogonalization. To break this coupling, we introduce a decoupled design principle: computing adaptive scaling factors directly from raw gradients, independent of orthogonalization, and subsequently applying them to the orthogonalized matrix, i.e., the polar factor (Higham, 2008; Higham, 2020) of the momentum. Another design deficiency of NorMuon lies in its reliance on rigid row-wise scaling, which overlooks a fundamental geometric reality: for non-square matrices, the polar factor inherently exhibits only one-sided orthogonality. Specifically, for tall matrices, the polar factor is column-orthogonal, with its columns defining the orthogonal directions. Row-wise scaling, however, is a one-sided operation applied exclusively along the row dimension, which is misaligned with the column-wise orthogonal structure. As a result, this geometric mismatch distorts the column geometry and breaks the original orthogonality induced by the polar factor. To prevent this structural distortion, we introduce our second design principle: geometry-aligned scaling, which aligns the scaling with the one-sided orthogonal structure. Specifically, the scaling axis is chosen according to the shape-dependent orthogonality structure of the polar factor: row-wise scaling is applied for wide matrices, whereas column-wise scaling is applied for tall matrices (or either for square matrices), thereby preserving the orthogonal structure obtained through orthogonalization. We further provide a formal theoretical justification for the orthogonality-preserving property of the proposed adaptive scaling mechanism (see Subsection 4.2 for the proof). Beyond the two core design principles, we incorporate several additional design considerations to better regulate the dynamics of the adaptive scaling factors. First, we construct the second-moment estimate for each adaptive unit (i.e., row or column) from the sum, rather than the mean, of the squared gradient entries, thereby making the scaling of each unit depend on its total accumulated gradient energy. This differs from NorMuon, which normalizes the total row-wise gradient energy by the row dimension, making its adaptive scaling depend on average per-coordinate energy. Second, we apply standard bias correction to the second-moment estimates to mitigate the initialization bias of the exponential moving average (EMA) during the early stages of optimization. Third, to curb the excessive anisotropy that adaptive scaling can introduce across different orthogonal directions, we employ an adaptive clipping mechanism that bounds the dynamic range of the scaling factors. This preserves Muon’s favorable ability to support large step sizes while maintaining training stability. Combining all these ingredients, we propose Decoupled Geometry-Aligned Muon (DGA-Muon; see Algorithm 1), which equips Muon with geometry-aligned adaptive scaling based on genuine gradient signals while achieving computational efficiency comparable to that of NorMuon. We establish convergence guarantees for DGA-Muon under standard assumptions in the non-convex setting. We empirically verify our theoretical characterization of NorMuon, showing that removing its adaptive scaling on the square Q/K/V/O projection matrices in NanoChat leaves the validation performance essentially unchanged. We further demonstrate that DGA-Muon substantially outperforms NorMuon in an end-to-end NanoChat evaluation, validating the effectiveness of our proposed adaptive scaling mechanism. Finally, we note that our analysis and algorithmic design provide broader insights into general matrix-aware optimizer design beyond enhancing Muon or NorMuon. Our proposed principles extend naturally to directions other than those used by Muon. More generally, any directions that meaningfully characterize the underlying matrix can serve as candidates for adaptive scaling. Furthermore, our analysis highlights the intrinsic necessity of row-wise/column-wise adaptivity as a natural mechanism for adapting to the typical one-sided directional structure inherent in matrices, which manifests along either the row or column dimension. 2 Preliminaries: Muon and NorMuon In this section, we briefly review Muon and NorMuon, providing the basis for our subsequent theoretical analysis. Muon. Muon is an optimizer tailored for two-dimensional matrix parameters. It maintains a momentum buffer MtM_t, updated according to Mt=β1Mt−1+(1−β1)GtM_t= _1M_t-1+(1- _1)G_t, where GtG_t denotes the gradient at iteration t, β1∈[0,1) _1∈[0,1) is the momentum coefficient, and M0=∈ℝm×nM_0=0 ^m× n. However, rather than updating the parameters directly along the raw momentum direction, Muon applies the update along the direction of the orthogonalized momentum. Specifically, given the current weight matrix Wt−1∈ℝm×nW_t-1 ^m× n and the momentum Mt∈ℝm×nM_t ^m× n, the update rule of Muon is given as follows Wt=Wt−1−ηOrth(Mt)W_t=W_t-1- (M_t) where η is the learning rate, and Orth(Mt)Orth(M_t) denotes the orthogonalization of MtM_t, i.e., Orth(Mt)=UV⊤Orth(M_t)=UV with Mt=UΣV⊤M_t=U V being the (thin) singular value decomposition (SVD). The orthogonalization procedure discards all singular value information by setting every singular value to one, while preserving the directional structure induced by the singular vectors. The resulting orthogonalized matrix UV⊤UV is commonly referred to as the polar factor (Higham, 2008; Higham, 2020) of MtM_t. In practice, however, Muon avoids the high cost of computing the full SVD by approximating the orthogonalization using efficient Newton–Schulz iterations (Jordan, 2024; Amsel et al., 2026). NorMuon. NorMuon extends Muon by introducing row-wise adaptive scaling. In addition to the momentum buffer Mt∈ℝm×nM_t ^m× n, it maintains a second-moment estimate vt∈ℝmv_t ^m. Specifically, vtv_t tracks the row-wise mean square of the orthogonalized momentum. Rather than directly updating the parameters with the orthogonalized momentum, NorMuon first scales it by the inverse square root of vtv_t before applying the update. Formally, let Ot=Orth(Mt)O_t=Orth(M_t) and meancols(⋅)mean_cols(·) denote the row-wise mean, i.e., the mean across columns for each row. Given the decay rate β2∈[0,1) _2∈[0,1) and the initialization v0=∈ℝmv_0=0 ^m, the second-moment estimate vtv_t is computed via an exponential moving average (EMA) vt=β2vt−1+(1−β2)meancols(Ot⊙Ot),v_t= _2v_t-1+(1- _2)mean_cols(O_t O_t), where ⊙ denotes the Hadamard (element-wise) product. The final parameter update then uses OtO_t scaled by the inverse square root of vtv_t Wt=Wt−1−ηtOtvt+δ.W_t=W_t-1- _t O_t v_t+δ. Here, the division is broadcast row-wise, ηt _t is the learning rate, and δ>0δ>0 is a small constant for numerical stability. For clarity, we omit the weight decay and learning rate scaling used in the original NorMuon, as neither affects the subsequent analysis. 3 On the Origin and Limitations of NorMuon’s Adaptivity Although NorMuon’s row-wise adaptive scaling has demonstrated strong empirical performance, its underlying theoretical foundation remains largely unexplored.11 1 To the best of our knowledge, no theoretical convergence guarantee has yet been established for NorMuon. In this section, we theoretically analyze the origin of NorMuon’s adaptivity and uncover its inherent limitations. Specifically, we show that this adaptivity primarily arises from orthogonalization-induced geometry rather than optimization dynamics, and further demonstrate that such adaptivity can degenerate under certain conditions. 3.1 Exact Orthogonalization We first consider the m≤nm≤ n regime, where the matrix is square (m=nm=n) or wide (m<nm<n). To facilitate the subsequent analysis, we express the row-wise mean of the element-wise square Ot⊙OtO_t O_t in terms of the Gram matrix of OtO_t meancols(Ot⊙Ot)=1ndiag(OtOt⊤),mean_cols(O_t O_t)= 1ndiag(O_tO_t ), where diag(⋅)diag(·) denotes the operator that extracts the diagonal entries of a square matrix into a vector. Recall that the thin SVD of MtM_t is given by Mt=UΣV⊤M_t=U V . For m≤nm≤ n, assuming that MtM_t has full row rank, we have U∈ℝm×mU ^m× m, Σ∈ℝm×m ^m× m, and V∈ℝn×mV ^n× m. By definition of the SVD, U is orthogonal (UU⊤=U⊤U=ImUU =U U=I_m) and V is column-orthonormal (V⊤V=ImV V=I_m). Then, under exact orthogonalization, the identity Ot=UV⊤O_t=UV holds exactly. Therefore, we obtain OtOt⊤=(UV⊤)(UV⊤)⊤=U(V⊤V)U⊤=UU⊤=Im,O_tO_t =(UV )(UV ) =U(V V)U =U =I_m, i.e., OtO_t is row-orthonormal, and hence meancols(Ot⊙Ot)=1ndiag(OtOt⊤)=1nm,mean_cols(O_t O_t)= 1ndiag(O_tO_t )= 1n1_m, where m1_m denotes the all-ones vector in ℝmR^m. Substituting this identity into the EMA update rule for vtv_t, unrolling the recursion, evaluating the resulting geometric series, and using v0=v_0=0, we obtain vt=β2tv0+(1−β2t)1nm=(1−β2t)1nm,v_t= _2^tv_0+(1- _2^t) 1n1_m=(1- _2^t) 1n1_m, which is uniform across rows, with only a global scalar varying over time. Furthermore, if the standard bias correction omitted in NorMuon were applied, it would cancel the time-dependent factor, yielding exactly 1nm 1n1_m, which is functionally equivalent to a fixed global scalar throughout training. The resulting scaling vector 1vt+δ 1 v_t+δ therefore assigns the same scaling coefficient to each row. Hence, the originally intended neuron-wise adaptive scaling completely collapses to a trivial global scalar, thereby rendering the adaptive mechanism fundamentally ineffective. 3.2 Approximate Orthogonalization The preceding analysis assumes exact orthogonalization. In practice, however, NorMuon employs approximate orthogonalization via a finite number of Newton–Schulz iterations (Li et al., 2026). We now analyze the approximate orthogonalization case. Assume that the approximately orthogonalized matrix OtO_t satisfies OtOt⊤=Im+Et,O_tO_t =I_m+E_t, where Et∈ℝm×mE_t ^m× m denotes the orthogonality residual induced by approximate orthogonalization. Then we have meancols(Ot⊙Ot)=1ndiag(OtOt⊤)=1n(m+diag(Et)).mean_cols(O_t O_t)= 1ndiag(O_tO_t )= 1n (1_m+diag(E_t) ). Substituting this expression into the EMA update of vtv_t, unrolling the recursion, evaluating the resulting series, and using v0=v_0=0, yields vt=(1−β2t)1nm+1−β2n∑i=1tβ2t−idiag(Ei).v_t=(1- _2^t) 1n1_m+ 1- _2n _i=1^t _2^t-idiag(E_i). For convenience, define Rt=1−β2n∑i=1tβ2t−idiag(Ei).R_t= 1- _2n _i=1^t _2^t-idiag(E_i). The term (1−β2t)1nm(1- _2^t) 1n1_m coincides exactly with the row-invariant scaling term derived in the exact orthogonalization case, whereas RtR_t captures the exponentially weighted accumulation of the orthogonality residuals and constitutes the only source of row-wise variation in vtv_t. To further characterize the magnitude of RtR_t, we assume that the orthogonality residual is uniformly bounded in spectral norm by some small constant ϵ>0ε>0, i.e., ‖Et‖2≤ϵ\|E_t\|_2≤ε for all t. This assumption is consistent with practical Newton–Schulz implementations, where the orthogonalization error is controlled within a prescribed tolerance. The spectral norm bound directly implies a bound on the diagonal entries of EtE_t. Specifically, for each j∈1,…,mj∈\1,…,m\, by the Cauchy–Schwarz inequality, the definition of the induced operator norm, and the fact that ‖j‖2=1\|e_j\|_2=1, we have |(Et)jj|=|j⊤Etj|≤‖j‖2‖Etj‖2≤‖Et‖2≤ϵ.|(E_t)_j|=|e_j E_te_j|≤\|e_j\|_2\|E_te_j\|_2≤\|E_t\|_2≤ε. Consequently, we can bound RtR_t in the infinity norm as ‖Rt‖∞≤1−β2n∑i=1tβ2t−iϵ=(1−β2t)ϵn≤ϵn.\|R_t\|_∞≤ 1- _2n _i=1^t _2^t-i\,ε=(1- _2^t) εn≤ εn. It immediately follows that the difference between any two entries of vtv_t is entirely determined by RtR_t. That is, for any j,k∈1,…,mj,k∈\1,…,m\, we have |(vt)j−(vt)k|=|(Rt)j−(Rt)k|≤2‖Rt‖∞≤2ϵn.|(v_t)_j-(v_t)_k|=|(R_t)_j-(R_t)_k|≤ 2\|R_t\|_∞≤ 2εn. This bounded variation in vtv_t directly translates into a bounded variation in the final adaptive scaling factors 1vt+δ 1 v_t+δ. Specifically, by the Mean Value Theorem, there exists some constant C>0C>0 such that, for any j,k∈1,…,mj,k∈\1,…,m\, we have |1(vt)j+δ−1(vt)k+δ|≤Cϵn. | 1 (v_t)_j+δ- 1 (v_t)_k+δ |≤ Cεn. In practice, Newton–Schulz-based orthogonalization rapidly reduces the orthogonality residual as iterations proceed, typically resulting in a very small ϵε. Consequently, the adaptive scaling factors exhibit only negligible variation across rows, rendering the row-wise adaptivity practically ineffective. In short, the above analysis reveals that, in the m≤nm≤ n regime, the apparent row-wise adaptivity in NorMuon is not driven by the underlying gradient statistics or curvature information that adaptive optimizers are typically designed to capture. Instead, the remaining row-wise variation in the adaptive scaling factors is solely induced by the orthogonalization residual. In other words, the observed adaptivity of NorMuon is merely a byproduct of inexact orthogonalization. As ϵ→0ε→ 0, this residual-induced adaptivity vanishes, and the behavior of approximate orthogonalization converges to that of exact orthogonalization, recovering uniform, non-adaptive row-wise scaling. Remark. A counterintuitive implication of the preceding analysis, which we term the Orthogonalization–Adaptivity Paradox, is that the adaptivity of NorMuon is inversely coupled with the quality of orthogonalization. The more accurate the orthogonalization, the weaker the adaptivity becomes; in the limit of exact orthogonalization, the adaptivity disappears entirely. Conversely, less accurate orthogonalization permits greater variation in the row-wise scaling factors and thus allows for stronger adaptivity. This suggests that NorMuon’s adaptive mechanism essentially serves to compensate for imperfections in orthogonalization, rather than providing genuine optimization-driven adaptivity. This stands in stark contrast to the design philosophy of Muon, where accurate orthogonalization is the cornerstone of the optimizer. Consequently, NorMuon’s adaptive mechanism is intrinsically incompatible with ongoing research efforts to improve Muon through increasingly accurate orthogonalization (Grishina et al., 2025; He et al., 2025; Amsel et al., 2026). Note that the above analysis relies on the full row rank assumption. In fact, this already covers the generic case encountered in practice. Rank deficiency is an undesirable edge case that can hinder effective training and impair the model’s representational capacity, and is thus generally avoided in practice. Since it rarely occurs, we do not treat it separately here or in the subsequent analysis. We do emphasize, however, that even under rank deficiency, the row-wise scaling of NorMuon remains entirely determined by the orthogonalization procedure rather than by the optimization dynamics. A more detailed discussion of this case will be provided in an extended version of this work. 3.3 The m>nm>n Regime In the m>nm>n regime, where the matrix is tall, the situation is fundamentally different. The row-orthonormality property OtOt⊤=ImO_tO_t =I_m no longer holds due to the dimensional constraint m>nm>n. Instead, under exact orthogonalization, the column-orthonormality property Ot⊤Ot=InO_t O_t=I_n holds. This implies that every column of OtO_t has unit norm, i.e., ∥(Ot):,j∥22=1\|(O_t)_:,j\|_2^2=1 for each j∈1,…,nj∈\1,…,n\. In contrast, since OtOt⊤≠ImO_tO_t ≠ I_m, the squared norm of each row of OtO_t is no longer constrained to 11. Nevertheless, the aggregate row energy is constrained to be fixed. Specifically, we have ∑i=1m∥(Ot)i,:∥22=∑i=1m∑j=1n(Ot)ij2=∑j=1n∑i=1m(Ot)ij2=∑j=1n∥(Ot):,j∥22=n, _i=1^m\|(O_t)_i,:\|_2^2= _i=1^m _j=1^n(O_t)_ij^2= _j=1^n _i=1^m(O_t)_ij^2= _j=1^n\|(O_t)_:,j\|_2^2=n, where the last equality follows from the column-orthonormality of OtO_t. Since there is no row orthonormality constraint, this fixed aggregate energy is no longer forced to be uniformly distributed across the m rows. Consequently, the squared norm of each row can deviate substantially from the uniform allocation n/mn/m, inducing row-wise variation in meancols(Ot⊙Ot)mean_cols(O_t O_t) and hence in the resulting adaptive scaling factors. This indicates that, in the m>nm>n regime, NorMuon’s adaptivity is not driven by conventional optimization dynamics underlying most existing adaptive optimizers, but instead stems from the intrinsic uneven row-energy distribution induced by tall-matrix orthogonalization. The above analysis focuses on exact orthogonalization. For approximate orthogonalization, the analysis largely follows that of the m≤nm≤ n regime, where deviations from exact orthogonality further introduce residual-induced variation in the adaptive scaling factors. This residual effect also diminishes as the orthogonalization accuracy improves, following the same Orthogonalization–Adaptivity Paradox. Summary. Our analysis in the three preceding subsections reveals that NorMuon’s adaptivity is not optimization-driven in the conventional sense. Instead, it primarily serves as a compensation mechanism for the non-uniformity induced by orthogonalization, including residual errors and uneven row-energy distribution. Although this compensation can provide empirical benefits, the resulting adaptivity is largely an orthogonalization-induced byproduct rather than a principled adaptive mechanism. Furthermore, approximate orthogonalization gives rise to the Orthogonalization–Adaptivity Paradox, revealing an inherent limitation of this form of adaptation. These findings motivate the design of a more principled adaptive mechanism tailored specifically to Muon. 4 Decoupled Geometry-Aligned Adaptive Scaling In this section, we present a novel adaptive scaling mechanism specifically designed for Muon. We first outline the core design principles and then formally justify the orthogonality-preserving property of the resulting scaling strategy. We further discuss additional design considerations for constructing the adaptive scaling factors. The complete optimization procedure is summarized in Algorithm 122 2 We omit the weight decay term here for generality, as it is independent of the core algorithm and various variants can be seamlessly incorporated (e.g., standard or cautious weight decay (Chen et al., 2026)).. Algorithm 1 Decoupled Geometry-Aligned Adaptive Muon (DGA-Muon) 1: Input: Step size η>0η>0, decay rates β1,β2∈[0,1) _1, _2∈[0,1), stabilizer δ>0δ>0, clip ratios c∈(0,1],d≥1c∈(0,1],d≥ 1 2: Initialize: W1∈ℝm×nW_1 ^m× n, M0=∈ℝm×nM_0=0 ^m× n, v0=∈ℝmin(m,n)v_0=0 (m,n) 3: for t=1,…,Tt=1,…,T do 4: Compute stochastic gradient GtG_t 5: Update first moment estimate Mt=β1Mt−1+(1−β1)GtM_t= _1M_t-1+(1- _1)G_t 6: Update second moment estimate from raw gradients vt=β2vt−1+(1−β2)diag(GtGt⊤),m≤n,β2vt−1+(1−β2)diag(Gt⊤Gt),m>n.v_t= cases _2v_t-1+(1- _2)diag(G_tG_t ),&m≤ n,\\ _2v_t-1+(1- _2)diag(G_t G_t),&m>n. cases 7: Apply bias correction v^t=vt/(1−β2t) v_t=v_t/(1- _2^t) 8: Compute adaptive scaling factors st=1/(v^t+δ)s_t=1/( v_t+δ) 9: Compute median of scaling factors smed=median(st)s_med=median(s_t) 10: Clip scaling factors st←clip(st,c⋅smed,d⋅smed)s_t (s_t,\,c· s_med,\,d· s_med) 11: Orthogonalize first moment Ot=Orth(Mt)O_t=Orth(M_t) 12: Apply geometry-aligned adaptive scaling O~t=Diag(st)Ot,m≤n,OtDiag(st),m>n. O_t= casesDiag(s_t) 0.85358ptO_t,&m≤ n,\\ O_tDiag(s_t),&m>n. cases 14.22636pt 13: Update parameters Wt+1=Wt−ηO~tW_t+1=W_t-η 0.56905pt O_t 14: end for 4.1 Design Principles: Decoupled Adaptivity and Orthogonality Preservation We begin by examining the root cause of the limitations of NorMuon’s adaptivity identified in the preceding section. Specifically, our analysis reveals that these limitations stem from the inherent coupling between orthogonalization and adaptive scaling. Because the scaling factors are computed from the orthogonalized momentum, the resulting adaptivity is inherently dictated by orthogonalization-induced geometry rather than direct gradient information. This analysis naturally motivates a core design principle: the computation of adaptive scaling factors should be decoupled from the orthogonalization procedure. Guided by this principle, we propose a decoupled adaptive mechanism that computes the adaptive scaling factors directly from the raw gradients, rather than from the orthogonalized momentum, while applying these factors to the orthogonalized update, i.e., the polar factor. In this way, the adaptive scaling is no longer dictated by orthogonalization-induced geometry, but instead reflects genuine optimization-relevant information—a defining characteristic of standard adaptive optimizers. While decoupling the computation of adaptive scaling factors resolves the fundamental coupling between adaptivity and orthogonalization, a closer examination of the geometric structure of the orthogonalized update reveals another design deficiency in NorMuon. Specifically, in the m>nm>n regime, the polar factor has orthonormal columns, which define the orthogonal directions of the update, whereas NorMuon applies adaptive scaling along the row dimension. This dimensional mismatch inevitably distorts these orthogonal directions and destroys the orthogonality that the polar factor is designed to preserve. This observation motivates a second design principle: adaptive scaling should respect the intrinsic geometry of the orthogonalized update. Intuitively, since orthogonalization is designed to extract a set of orthogonal update directions, any subsequent adaptive scaling should preserve this directional structure rather than distort it. Following this principle, we introduce a geometry-aligned scaling mechanism that applies row-wise scaling for m<nm<n and column-wise scaling for m>nm>n, while either choice can be used when m=nm=n, thereby preserving the orthogonality of the update in all regimes. 4.2 Formal Justification of the Orthogonality-Preserving Property We now provide a formal justification of the orthogonality-preserving property for the proposed geometry-aligned scaling mechanism. We first consider the m>nm>n regime under exact orthogonalization, where the polar factor Ot∈ℝm×nO_t ^m× n satisfies Ot⊤Ot=InO_t O_t=I_n. In this setting, the columns of OtO_t form a set of mutually orthogonal directions. Let vt∈ℝnv_t ^n denote the column-wise second-moment estimate, where each entry tracks the EMA of the squared ℓ2 _2-norm of the corresponding column of the raw gradients, i.e., vt=β2vt−1+(1−β2)diag(Gt⊤Gt).v_t= _2v_t-1+(1- _2)diag\! (G_t G_t ). Note that the Gram matrix here is introduced only for notational convenience; in practice, only its diagonal entries are needed, so the full Gram matrix is never explicitly formed. The associated diagonal scaling matrix is constructed as Ht=Diag(1/(vt+δ))∈ℝn×nH_t=Diag(1/( v_t+δ)) ^n× n, where Diag(⋅)Diag(·) denotes the operator that constructs a diagonal matrix from a vector. Applying column-wise adaptive scaling to OtO_t is equivalent to right-multiplying OtO_t by HtH_t, yielding the scaled update O~t=OtHt O_t=O_tH_t. The Gram matrix of the scaled update satisfies O~t⊤O~t=Ht⊤Ot⊤OtHt=Ht⊤InHt=Ht2, O_t O_t=H_t O_t O_tH_t=H_t \!I_nH_t=H_t^2, which remains diagonal. This implies that the columns of O~t O_t remain mutually orthogonal, i.e., ⟨O~:,i,O~:,j⟩=0,i≠j, O_:,i,\, O_:,j =0,\,\,i≠ j, while their squared norms satisfy ∥O~:,i∥22=(Ht2)ii,\| O_:,i\|_2^2=(H_t^2)_i, indicating that the magnitude of each orthogonal direction is independently scaled by its corresponding adaptive scaling factor. Thus, column-wise adaptive scaling transforms the polar factor with unit column norms into an update with orthogonal columns but heterogeneous column norms. It preserves the orthogonal directions extracted by orthogonalization while assigning an adaptive step size to each direction. In contrast, applying row-wise adaptive scaling to OtO_t is equivalent to left-multiplying OtO_t by the corresponding diagonal scaling matrix Dt∈ℝm×mD_t ^m× m, constructed analogously from the row-wise second-moment estimate, yielding the scaled update O~t=DtOt O_t=D_tO_t. The resulting Gram matrix becomes O~t⊤O~t=Ot⊤Dt⊤DtOt=Ot⊤Dt2Ot, O_t O_t=O_t D_t D_tO_t=O_t D_t^2O_t, which is generally non-diagonal. This indicates that row-wise adaptive scaling reintroduces coupling between the orthogonal column directions extracted by orthogonalization, and hence the orthogonal structure established by orthogonalization is no longer preserved in the m>nm>n regime. For the m<nm<n regime, the analysis proceeds analogously, with the roles of rows and columns reversed. In this case, we construct the adaptive scaling factors using the row-wise second-moment estimate vt∈ℝmv_t ^m, where each entry tracks the EMA of the squared ℓ2 _2-norm of the corresponding row of the raw gradients, i.e., vt=β2vt−1+(1−β2)diag(GtGt⊤).v_t= _2v_t-1+(1- _2)diag(G_tG_t ). The polar factor OtO_t satisfies OtOt⊤=ImO_tO_t =I_m, implying that the rows of OtO_t form a set of mutually orthogonal directions. Applying row-wise adaptive scaling to OtO_t is equivalent to left-multiplying OtO_t by the corresponding diagonal scaling matrix, thereby preserving the row-wise orthogonal structure. In contrast, applying column-wise adaptive scaling, equivalently right-multiplying OtO_t by the corresponding diagonal scaling matrix, generally distorts the row-wise orthogonal structure. For the m=nm=n regime, the polar factor OtO_t is a square orthogonal matrix, satisfying both OtOt⊤=ImO_tO_t =I_m and Ot⊤Ot=ImO_t O_t=I_m. Hence, both the rows and columns of OtO_t form sets of mutually orthogonal directions, and either row-wise or column-wise adaptive scaling preserves the corresponding orthogonal directions, adjusting only their magnitudes. Remark: Broader Insights for Matrix-Aware Optimizer Design. The design principles and geometric justifications presented above provide deeper insights into the general paradigm of matrix-aware optimization. 1. A Generalized Design Philosophy. Our decoupled, direction-aware adaptive mechanism transcends being a mere enhancement for Muon; it constitutes a more general design philosophy for matrix-aware optimizers. Notably, the directions utilized need not be restricted to the orthogonal directions used by Muon, but can naturally extend to broader classes of directions that characterize matrices. 2. Intrinsic Necessity of Row-Wise/Column-Wise Adaptivity. This perspective formally establishes why row-wise or column-wise adaptivity is fundamentally necessary: it provides a natural mechanism for adapting to the one-sided orthogonal directions of matrices. Rather than viewing row-wise/column-wise scaling as a structural approximation to Adam-like element-wise scaling (as in Adam-mini (Zhang et al., 2025b) and other related work), our approach grounds it in the intrinsic geometry of matrices (Zhang et al., 2026b). Furthermore, although methods such as NorMuon also employ row-wise/column-wise adaptation, our motivation and design arise from a distinct geometric alignment principle rather than heuristic normalization. 4.3 Additional Design Considerations Beyond the two aforementioned design principles, we further incorporate several additional considerations when constructing the adaptive scaling factors to better regulate their dynamics during optimization. First, the second-moment estimate is constructed from the sum rather than the mean of the squared entries. While this is already implicit in the construction of vtv_t above, we make this design choice more explicit here. For a row or column gradient vector, the corresponding second-moment term is given by its squared ℓ2 _2-norm, without normalization by the vector dimension. This reflects our design philosophy of treating each row (or column) gradient vector as a single adaptive unit, with its scaling determined by the total accumulated gradient energy of the vector. Consequently, units with larger accumulated gradient energy receive smaller scaling factors, while units with lower accumulated energy are assigned larger ones. This contrasts with NorMuon, which normalizes the row-wise gradient energy by the row dimension, making its adaptive scaling depend on the average per-coordinate energy rather than the total energy of each row. Since the second-moment estimate vtv_t is computed directly from raw gradients and is decoupled from orthogonalization, it suffers from the standard initialization bias of EMA estimators. In particular, during the early optimization steps, vtv_t tends to underestimate the underlying second moment due to the zero initialization of the accumulator. We therefore apply the standard bias-correction technique to obtain a bias-corrected estimate v^t=vt1−β2t, v_t= v_t1- _2^t, which is then used to compute the adaptive scaling factor st=1v^t+δs_t= 1 v_t+δ.33 3 Bias correction can be replaced with a gradient normalization technique, which we will include in revision. Here vtv_t, v^t v_t, and sts_t are generic notations that can represent vectors in either ℝmR^m or ℝnR^n. Note that the first moment MtM_t requires no bias correction, owing to the scale-invariance of the subsequent orthogonalization operation (Pethick et al., 2025). While introducing adaptive scaling into Muon provides more fine-grained control over different adaptive units, it may also lead to excessive anisotropic scaling, resulting in overly imbalanced updates across different orthogonal directions. Such excessive anisotropy necessitates the use of smaller step sizes to maintain stable optimization. To preserve Muon’s favorable property of supporting large step sizes, we introduce a clipping operation that bounds the degree of adaptivity. Specifically, we apply the following element-wise clipping operation clip(st,st,min,st,max)=min(max(st,st,min),st,max).clip(s_t,s_t, ,s_t, )= ( (s_t,s_t, ),\,s_t, ). We determine the clipping bounds st,mins_t, and st,maxs_t, adaptively based on the median of the scaling factors, denoted by smed=median(st)s_med=median(s_t). Specifically, we set st,min=c⋅smeds_t, =c· s_med and st,max=d⋅smed,s_t, =d· s_med, where c∈(0,1]c∈(0,1] and d≥1d≥ 1 control the allowable dynamic range. One can also use a single hyperparameter by setting d=1/cd=1/c. The clipping operation effectively bounds the condition number of the induced diagonal scaling matrix and prevents disproportionately large or small updates along specific orthogonal directions. When st,min=st,maxs_t, =s_t, , the clipping removes all relative adaptivity, reducing the update to a uniformly scaled Muon update. 5 Theoretical Analysis In this section, we establish the convergence guarantee of DGA-Muon under exact orthogonalization. We first introduce the assumptions required for the analysis. Assumptions. We make the following assumptions throughout the analysis. (A1) (L-smoothness) ‖∇f(X)−∇f(Y)‖F≤L‖X−Y‖F,\|∇ f(X)-∇ f(Y)\|_F≤ L\|X-Y\|_F, for all X,YX,Y. (A2) (Unbiased stochastic gradients with bounded variance) The stochastic gradients satisfy [Gt∣ℱt−1]=∇f(Wt),‖Gt−∇f(Wt)‖F2≤σ2.E[G_t _t-1]=∇ f(W_t), \|G_t-∇ f(W_t)\|_F^2≤σ^2. (A3) (Bounded stochastic gradients) There exists a constant Gmax>0G_ >0 such that ‖Gt‖F≤Gmaxfor all t.\|G_t\|_F≤ G_ \,\,\,for all t. We now proceed with the convergence analysis. By the L-smoothness of f and the update rule Wt+1=Wt−ηO~tW_t+1=W_t-η 0.85358pt O_t, we have f(Wt+1) f(W_t+1) ≤f(Wt)+⟨∇f(Wt),Wt+1−Wt⟩F+L2‖Wt+1−Wt‖F2 ≤ f(W_t)+ ∇ f(W_t),\,W_t+1-W_t _F+ L2\|W_t+1-W_t\|_F^2 =f(Wt)−η⟨∇f(Wt),O~t⟩F+η2L2‖O~t‖F2. =f(W_t)-η ∇ f(W_t),\, O_t _F+ η^2L2\| O_t\|_F^2. We first analyze the quadratic term ‖O~t‖F2\| O_t\|_F^2. For the m≤nm≤ n regime, the geometry-aligned adaptive update O~t O_t is given by O~t=Diag(st)Ot O_t=Diag(s_t) 0.56905ptO_t. Since Ot=UtVt⊤O_t=U_tV_t is the polar factor of MtM_t, we have OtOt⊤=UtVt⊤VtUt⊤=UtUt⊤=Im.O_tO_t =U_tV_t V_tU_t =U_tU_t =I_m. Then using the definition of the Frobenius norm and the cyclic property of the trace, we have ‖O~t‖F2=tr(O~t⊤O~t)=tr(Ot⊤Diag(st)2Ot)=tr(Diag(st)2OtOt⊤)=tr(Diag(st)2)=∑i=1mst,i2.\| O_t\|_F^2=tr( O_t O_t)=tr (O_t Diag(s_t)^2O_t )=tr (Diag(s_t)^2O_tO_t )=tr (Diag(s_t)^2 )= _i=1^ms_t,i^2. Let st,max=maxist,is_t, = _is_t,i denote the maximum entry of the scaling vector sts_t, then we have ‖O~t‖F2=∑i=1mst,i2≤mst,max2.\| O_t\|_F^2= _i=1^ms_t,i^2≤ m 0.85358pts_t, ^2. For the m>nm>n regime, the adaptive update is given by O~t=OtDiag(st) O_t=O_tDiag(s_t). By definition of the polar factor, we have Ot⊤Ot=InO_t O_t=I_n. Applying an analogous derivation, we obtain ‖O~t‖F2=∑j=1nst,j2≤nst,max2.\| O_t\|_F^2= _j=1^ns_t,j^2≤ n 0.85358pts_t, ^2. By introducing ℓ=min(m,n) = (m,n), both regimes can be unified and compactly written as ‖O~t‖F2≤ℓst,max2\| O_t\|_F^2≤ s_t, ^2. Consequently, the quadratic term in the smoothness inequality satisfies η2L2‖O~t‖F2=η2L2∑k=1ℓst,k2≤η2ℓLst,max22. η^2L2\| O_t\|_F^2= η^2L2 _k=1 s_t,k^2≤ η^2 Ls_t, ^22. We now analyze the Frobenius inner product term ⟨∇f(Wt),O~t⟩F. ∇ f(W_t), O_t _F. By decomposing ∇f(Wt)∇ f(W_t) into MtM_t and the residual ∇f(Wt)−Mt∇ f(W_t)-M_t, we obtain ⟨∇f(Wt),O~t⟩F=⟨Mt,O~t⟩F+⟨∇f(Wt)−Mt,O~t⟩F. ∇ f(W_t), O_t _F= M_t, O_t _F+ ∇ f(W_t)-M_t, O_t _F. The first term characterizes the alignment of the adaptively scaled orthogonal direction with the momentum, and the second term captures the momentum tracking error due to the discrepancy between MtM_t and the current gradient. We handle these two terms separately in the following. To begin, we address the first term, for which we have the following lemma. Lemma 1 (Adaptive Polar Alignment Lower Bound). Let MtM_t, OtO_t, O~t O_t and sts_t be defined as in Algorithm 1. Let st,min=minist,is_t, = _is_t,i denote the minimum entry of the scaling vector sts_t. Then, the following lower bound on the adaptive polar alignment holds ⟨Mt,O~t⟩F≥st,min‖Mt‖F. M_t, O_t _F≥ s_t, \|M_t\|_F. Proof. We first consider the m≤nm≤ n regime, where the adaptive scaling is applied via left multiplication: O~t=Diag(st)Ot O_t=Diag(s_t) 0.56905ptO_t. Substituting Mt=UtΣtVt⊤M_t=U_t _tV_t and O~t=Diag(st)UtVt⊤ O_t=Diag(s_t) 0.56905ptU_tV_t into the Frobenius inner product ⟨Mt,O~t⟩F=tr(Mt⊤O~t) M_t, O_t _F=tr(M_t O_t), we obtain ⟨Mt,O~t⟩F M_t, O_t _F =tr((UtΣtVt⊤)⊤(Diag(st)UtVt⊤))=tr(VtΣtUt⊤Diag(st)UtVt⊤). =tr((U_t _tV_t ) (Diag(s_t) 0.56905ptU_tV_t ))=tr(V_t _tU_t \!Diag(s_t) 0.56905ptU_tV_t ). Using the cyclic property of the trace and the fact that VtV_t has orthonormal columns (i.e., Vt⊤Vt=ImV_t V_t=I_m), and expanding the trace, we have ⟨Mt,O~t⟩F=tr(ΣtUt⊤Diag(st)Ut)=∑j=1mσj(Ut⊤Diag(st)Ut)jj, M_t, O_t _F=tr( _tU_t \!Diag(s_t) 0.56905ptU_t)= _j=1^m _j\,(U_t \!Diag(s_t) 0.56905ptU_t)_j, where σjj=1m\ _j\_j=1^m are the singular values of MtM_t. By definition, the diagonal entries of Ut⊤Diag(st)UtU_t \!Diag(s_t) 0.56905ptU_t are given by (Ut⊤Diag(st)Ut)jj=∑i=1mst,iUt,ij2,(U_t \!Diag(s_t) 0.56905ptU_t)_j= _i=1^ms_t,i\,U_t,ij^2, where st,ii=1m\s_t,i\_i=1^m are the entries of sts_t. Recall that st,mins_t, denotes the minimum scaling factor of sts_t, i.e., st,min=minist,i.s_t, = _is_t,i. Then, for each j, we have (Ut⊤Diag(st)Ut)jj (U_t \!Diag(s_t) 0.56905ptU_t )_j =∑i=1mst,iUt,ij2≥∑i=1mst,minUt,ij2=st,min∑i=1mUt,ij2=st,min, = _i=1^ms_t,iU_t,ij^2≥ _i=1^ms_t, U_t,ij^2=s_t, _i=1^mU_t,ij^2=s_t, , where the last equality follows from the column orthonormality of UtU_t, i.e., Ut⊤Ut=ImU_t U_t=I_m. Therefore, we obtain a lower bound for ⟨Mt,O~t⟩F M_t, O_t _F ⟨Mt,O~t⟩F M_t, O_t _F =∑j=1mσj(Ut⊤Diag(st)Ut)jj≥st,min∑j=1mσj=st,min‖Mt‖∗, = _j=1^m _j(U_t \!Diag(s_t) 0.56905ptU_t)_j≥ s_t, _j=1^m _j=s_t, \|M_t\|_*, where ∥⋅∥∗\|·\|_* denotes the nuclear norm of a matrix. Finally, since the norm inequality ‖A‖∗≥‖A‖F\|A\|_*≥\|A\|_F holds for any matrix A (Horn and Johnson, 2013), it follows that ⟨Mt,O~t⟩F≥st,min‖Mt‖F. M_t, O_t _F≥ s_t, \|M_t\|_F. The m>nm>n regime is analogous, where the adaptive scaling is applied via right multiplication as O~t=OtDiag(st) O_t=O_tDiag(s_t). Utilizing the column orthonormality of both UtU_t and VtV_t, and following an analogous argument, we obtain the same bound. Combining the two regimes, we prove the claimed bound in the lemma. ∎ Now, by decomposing MtM_t into the sum of the gradient ∇f(Wt)∇ f(W_t) and the momentum tracking error Mt−∇f(Wt)M_t-∇ f(W_t), and applying the reverse triangle inequality for the Frobenius norm, we have ‖Mt‖F \|M_t\|_F =‖∇f(Wt)+(Mt−∇f(Wt))‖F≥‖∇f(Wt)‖F−‖Mt−∇f(Wt)‖F. =\|∇ f(W_t)+(M_t-∇ f(W_t))\|_F≥\|∇ f(W_t)\|_F-\|M_t-∇ f(W_t)\|_F. Combining this inequality with Lemma 1, we obtain ⟨Mt,O~t⟩F M_t, O_t _F ≥st,min‖Mt‖F≥st,min‖∇f(Wt)‖F−st,min‖Mt−∇f(Wt)‖F. ≥ s_t, \|M_t\|_F≥ s_t, \|∇ f(W_t)\|_F-s_t, \|M_t-∇ f(W_t)\|_F. We now proceed to address the term ⟨∇f(Wt)−Mt,O~t⟩F ∇ f(W_t)-M_t, O_t _F. We first consider its negation. By the Cauchy-Schwarz inequality, we have ⟨Mt−∇f(Wt),O~t⟩F≤‖Mt−∇f(Wt)‖F‖O~t‖F. M_t-∇ f(W_t), O_t _F≤\|M_t-∇ f(W_t)\|_F\,\!\| O_t\|_F. As established earlier, we have ‖O~t‖F2≤ℓst,max2\| O_t\|_F^2≤ s_t, ^2 with ℓ=min(m,n) = (m,n). Consequently, we obtain ⟨Mt−∇f(Wt),O~t⟩F≤ℓ‖Mt−∇f(Wt)‖Fst,max. M_t-∇ f(W_t), O_t _F≤ 0.85358pt\|M_t-∇ f(W_t)\|_F 0.85358pts_t, . Therefore, by negating the above inequality, we obtain ⟨∇f(Wt)−Mt,O~t⟩F≥−ℓ‖Mt−∇f(Wt)‖Fst,max. ∇ f(W_t)-M_t, O_t _F≥- 0.85358pt\|M_t-∇ f(W_t)\|_F 0.85358pts_t, . Substituting the above bounds for the alignment and tracking error terms into the smoothness inequality, we obtain the following descent lemma for DGA-Muon. Lemma 2 (Per-step Descent Inequality). Let WtW_t, MtM_t and sts_t be defined as in Algorithm 1. Let st,mins_t, and st,maxs_t, denote the minimum and maximum entries of sts_t, respectively, and let ℓ=min(m,n) = (m,n). Assume that f is L-smooth. Then, the updates of DGA-Muon satisfy the following per-step descent inequality f(Wt+1)≤f(Wt)−ηst,min‖∇f(Wt)‖F+η(st,min+ℓst,max)‖Mt−∇f(Wt)‖F+η2ℓLst,max22.f(W_t+1)≤ f(W_t)-η\,\!s_t, \|∇ f(W_t)\|_F+η\,\!(s_t, + s_t, )\|M_t-∇ f(W_t)\|_F+ η^2 Ls_t, ^22. Next, we analyze ‖Mt−∇f(Wt)‖F\|M_t-∇ f(W_t)\|_F, which quantifies the deviation between the stochastic momentum and the true gradient. Recall that Mt=β1Mt−1+(1−β1)GtM_t= _1M_t-1+(1- _1)G_t. Subtracting the true gradient ∇f(Wt)∇ f(W_t), we have Mt−∇f(Wt)=β1Mt−1+(1−β1)Gt−∇f(Wt).M_t-∇ f(W_t)= _1M_t-1+(1- _1)G_t-∇ f(W_t). To isolate the previous-step error, we add and subtract β1∇f(Wt−1) _1∇ f(W_t-1) Mt−∇f(Wt)=β1(Mt−1−∇f(Wt−1))+(1−β1)(Gt−∇f(Wt))+β1(∇f(Wt−1)−∇f(Wt)).M_t-∇ f(W_t)= _1(M_t-1-∇ f(W_t-1))+(1- _1)(G_t-∇ f(W_t))+ _1(∇ f(W_t-1)-∇ f(W_t)). Unrolling recursively from t down to 11 gives Mt−∇f(Wt) M_t-∇ f(W_t) =β1t−1(M1−∇f(W1))+(1−β1)∑s=2tβ1t−s(Gs−∇f(Ws)) = _1^\,t-1(M_1-∇ f(W_1))+(1- _1) _s=2^t _1^\,t-s(G_s-∇ f(W_s)) +β1∑s=2tβ1t−s(∇f(Ws−1)−∇f(Ws)). + _1 _s=2^t _1^\,t-s(∇ f(W_s-1)-∇ f(W_s)). Taking the Frobenius norm and applying the triangle inequality, we obtain ‖Mt−∇f(Wt)‖F \|M_t-∇ f(W_t)\|_F ≤β1t−1‖M1−∇f(W1)‖F+‖(1−β1)∑s=2tβ1t−s(Gs−∇f(Ws))‖F ≤ _1^\,t-1\|M_1-∇ f(W_1)\|_F+ \|(1- _1) _s=2^t _1^\,t-s(G_s-∇ f(W_s)) \|_F +β1∑s=2tβ1t−s∥∇f(Ws−1)−∇f(Ws)∥F. + _1 _s=2^t _1^\,t-s\|∇ f(W_s-1)-∇ f(W_s)\|_F. The right-hand side naturally decomposes into three contributions: the initialization term, the stochastic noise term, and the gradient drift term. In the following, we analyze each of them separately. For the initialization term, since M1=(1−β1)G1M_1=(1- _1)G_1, we decompose M1−∇f(W1)=(1−β1)(G1−∇f(W1))−β1∇f(W1).M_1-∇ f(W_1)=(1- _1)(G_1-∇ f(W_1))- _1∇ f(W_1). Therefore, ‖β1t−1(M1−∇f(W1))‖F≤ \| _1^t-1(M_1-∇ f(W_1))\|_F≤ β1t−1(1−β1)‖G1−∇f(W1)‖F+β1t‖∇f(W1)‖F _1^t-1(1- _1)E\|G_1-∇ f(W_1)\|_F+ _1^t\|∇ f(W_1)\|_F Using the unbiasedness of G1G_1, Jensen’s inequality and the gradient bound assumption, we have ‖∇f(W1)‖F=‖[G1|ℱ0]‖F≤[‖G1‖F|ℱ0]≤Gmax.\|∇ f(W_1)\|_F=\|E[G_1|F_0]\|_F [\|G_1\|_F|F_0]≤ G_ . Then applying Jensen’s inequality, ‖X‖F≤‖X‖F2E\|X\|_F≤ E\|X\|_F^2, and using the bounded-variance assumption ‖G1−∇f(W1)‖F2≤σ2E\|G_1-∇ f(W_1)\|_F^2≤σ^2, we obtain [‖β1t−1(M1−∇f(W1))‖F]≤β1t−1(1−β1)σ+β1tGmax.E [\| _1^\,t-1(M_1-∇ f(W_1))\|_F ]≤ _1^\,t-1(1- _1)σ+ _1^\,tG_ . For the stochastic noise term, we first bound its second moment. For notational convenience, we introduce the shorthand t=(1−β1)∑s=2tβ1t−s(Gs−∇f(Ws)).N_t=(1- _1) _s=2^t _1^\,t-s(G_s-∇ f(W_s)). Expanding the squared Frobenius norm, taking the expectation, and using the linearity of expectation, we obtain ‖t‖F2 \|N_t\|_F^2 =(1−β1)2⟨∑s=2tβ1t−s(Gs−∇f(Ws)),∑r=2tβ1t−r(Gr−∇f(Wr))⟩ =(1- _1)^2\,E\,\! _s=2^t _1^\,t-s(G_s-∇ f(W_s)),\; _r=2^t _1^\,t-r(G_r-∇ f(W_r)) =(1−β1)2∑s=2t∑r=2tβ1 2t−s−r⟨Gs−∇f(Ws),Gr−∇f(Wr)⟩. =(1- _1)^2 _s=2^t _r=2^t _1^\,2t-s-r\,E\,\! G_s-∇ f(W_s),\;G_r-∇ f(W_r) . By assumption (A2), the stochastic gradient noise ξt=Gt−∇f(Wt) _t=G_t-∇ f(W_t) forms a martingale difference sequence, i.e., [ξt∣ℱt−1]=0E[ _t _t-1]=0, where ℱt−1F_t-1 denotes the filtration representing all information up to time t−1t-1. Hence, the cross terms (s≠rs≠ r) vanish in expectation, leaving only the diagonal terms ‖t‖F2=(1−β1)2∑s=2tβ1 2(t−s)‖Gs−∇f(Ws)‖F2≤(1−β1)2σ2∑k=0t−2β12k.E\|N_t\|_F^2=(1- _1)^2 _s=2^t _1^\,2(t-s)\,E\|G_s-∇ f(W_s)\|_F^2≤(1- _1)^2σ^2 _k=0^t-2 _1^2k. Evaluating the geometric sum gives ‖t‖F2≤(1−β1)2σ21−β12(t−1)1−β12=σ2(1−β1)(1−β12(t−1))1+β1≤σ21−β11+β1.E\|N_t\|_F^2≤(1- _1)^2σ^2 1- _1^2(t-1)1- _1^2=σ^2 (1- _1)(1- _1^2(t-1))1+ _1≤σ^2 1- _11+ _1. Then applying Jensen’s inequality and using the bounded-variance assumption, we obtain ‖t‖F≤‖t‖F2≤1−β11+β1σ.E\|N_t\|_F≤ E\|N_t\|_F^2≤ 1- _1 1+ _1\,\!σ. We now proceed to handle the gradient drift term, which, unlike the stochastic noise term, can be bounded deterministically. By the L-smoothness of f, the update rule Ws=Ws−1−ηO~s−1W_s=W_s-1-η O_s-1, and the bound ‖O~t‖F≤ℓst,max\| O_t\|_F≤ s_t, , we have β1∑s=2tβ1t−s‖∇f(Ws−1)−∇f(Ws)‖F _1 _s=2^t _1^\,t-s\|∇ f(W_s-1)-∇ f(W_s)\|_F ≤β1∑s=2tβ1t−sL‖Ws−1−Ws‖F ≤ _1 _s=2^t _1^\,t-sL\|W_s-1-W_s\|_F =β1ηL∑s=2tβ1t−s‖O~s−1‖F≤β1ηℓL∑s=2tβ1t−ss−1,max. = _1η L _s=2^t _1^\,t-s\| O_s-1\|_F≤ _1η L _s=2^t _1^\,t-ss_s-1, . Let smaxs_ be the maximum of all st,maxs_t, , then we obtain β1∑s=2tβ1t−s‖∇f(Ws−1)−∇f(Ws)‖F≤β1ηℓLsmax∑s=2tβ1t−s _1 _s=2^t _1^\,t-s\|∇ f(W_s-1)-∇ f(W_s)\|_F≤ _1η Ls_ _s=2^t _1^\,t-s For the geometric sum, we have ∑s=2tβ1t−s=∑k=0t−2β1k=1−β1t−11−β1≤11−β1. _s=2^t _1^\,t-s= _k=0^t-2 _1^k= 1- _1^t-11- _1≤ 11- _1. Therefore, the gradient drift term admits the following deterministic bound β1∑s=2tβ1t−s‖∇f(Ws−1)−∇f(Ws)‖F≤β1ηℓLsmax1−β1. _1 _s=2^t _1^\,t-s\|∇ f(W_s-1)-∇ f(W_s)\|_F≤ _1η Ls_ 1- _1. Combining the bounds for these three terms yields the following bound on the expected momentum error. Lemma 3 (Expected Momentum Error Bound). Let MtM_t and WtW_t be defined as in Algorithm 1. Let smaxs_ denote a uniform upper bound on the adaptive scaling factors satisfying (st)i≤smax<∞(s_t)_i≤ s_ <∞ for all t and i, and let ℓ=min(m,n) = (m,n). Assume that f is L-smooth. Then, under the bounded-variance and bounded-gradient assumptions, the expected momentum error at each step satisfies ‖Mt−∇f(Wt)‖F≤β1t−1(1−β1)σ+β1tGmax+1−β11+β1σ+β1ηℓLsmax1−β1.E\|M_t-∇ f(W_t)\|_F≤ _1^\,t-1(1- _1)σ+ _1^\,tG_ + 1- _1 1+ _1σ+ _1η Ls_ 1- _1. Recall that Lemma 2 establishes the following one-step descent inequality for the updates of Algorithm 1 f(Wt+1) f(W_t+1) ≤f(Wt)−ηst,min‖∇f(Wt)‖F+η(st,min+ℓst,max)‖Mt−∇f(Wt)‖F+η2ℓLst,max22. ≤ f(W_t)-η s_t, \|∇ f(W_t)\|_F+η(s_t, + s_t, )\|M_t-∇ f(W_t)\|_F+ η^2 Ls_t, ^22. To simplify the subsequent analysis, we replace the time-dependent quantities by uniform bounds over the optimization horizon. Let smins_ denote a uniform lower bound on the adaptive scaling factors satisfying 0<smin≤(st)i0<s_ ≤(s_t)_i for all t and i. Since smin≤st,min≤st,max≤smax,s_ ≤ s_t, ≤ s_t, ≤ s_ , we have −ηst,min‖∇f(Wt)‖F≤−ηsmin‖∇f(Wt)‖F,-η s_t, \|∇ f(W_t)\|_F≤-η s_ \|∇ f(W_t)\|_F, and st,min+ℓst,max≤(1+ℓ)smax.s_t, + \,s_t, ≤(1+ )s_ . Moreover, st,max2≤smax2.s_t, ^2≤ s_ ^2. Therefore, the one-step descent inequality can be uniformly bounded as f(Wt+1) f(W_t+1) ≤f(Wt)−ηsmin‖∇f(Wt)‖F+η(1+ℓ)smax‖Mt−∇f(Wt)‖F+η2ℓLsmax22. ≤ f(W_t)-η s_ \|∇ f(W_t)\|_F+η(1+ )s_ \|M_t-∇ f(W_t)\|_F+ η^2 Ls_ ^22. Rearranging to isolate the gradient term and taking expectations on both sides yields ηsmin[‖∇f(Wt)‖F] η s_ E [\|∇ f(W_t)\|_F ] ≤[f(Wt)−f(Wt+1)]+η(1+ℓ)smax[‖Mt−∇f(Wt)‖F] [f(W_t)-f(W_t+1)]+η(1+ )s_ E [\|M_t-∇ f(W_t)\|_F ] +η2ℓLsmax22. + η^2 Ls_ ^22. Then, summing over t=1,…,Tt=1,…,T, using the telescoping sum ∑t=1T[f(Wt)−f(Wt+1)]=[f(W1)−f(WT+1)] _t=1^TE[f(W_t)-f(W_t+1)]=E[f(W_1)-f(W_T+1)] , and subsequently dividing by ηsminη s_ , we obtain ∑t=1T[‖∇f(Wt)‖F] _t=1^TE\! [\|∇ f(W_t)\|_F ] ≤[f(W1)−f(WT+1)]ηsmin+(1+ℓ)smaxsmin∑t=1T[‖Mt−∇f(Wt)‖F] ≤ E[f(W_1)-f(W_T+1)]η s_ + (1+ )s_ s_ _t=1^TE\! [\|M_t-∇ f(W_t)\|_F ] +ηℓLsmax2T2smin. + η Ls_ ^2T2s_ . Summing the momentum error inequality given by Lemma 3 from t=1t=1 to T yields ∑t=1T‖Mt−∇f(Wt)‖F≤(1−β1)σ∑t=1Tβ1t−1+Gmax∑t=1Tβ1t+T1−β11+β1σ+Tβ1ηℓLsmax1−β1. _t=1^TE\|M_t-∇ f(W_t)\|_F≤(1- _1)σ _t=1^T _1^\,t-1+G_ _t=1^T _1^\,t+T 1- _1 1+ _1\,σ+T _1η Ls_ 1- _1. Evaluating the geometric sums ∑t=1Tβ1t−1≤11−β1 _t=1^T _1^\,t-1≤ 11- _1 and ∑t=1Tβ1t≤β11−β1 _t=1^T _1^\,t≤ _11- _1, we obtain ∑t=1T‖Mt−∇f(Wt)‖F≤σ+β1Gmax1−β1+T1−β11+β1σ+Tβ1ηℓLsmax1−β1. _t=1^TE\|M_t-∇ f(W_t)\|_F≤σ+ _1G_ 1- _1+T 1- _1 1+ _1\,σ+T _1η Ls_ 1- _1. Denote Δ=f(W1)−f⋆ =f(W_1)-f , where f⋆=infWf(W)f = _Wf(W). Since f(WT+1)≥f⋆f(W_T+1)≥ f , we have [f(W1)−f(WT+1)]≤Δ.E[f(W_1)-f(W_T+1)]≤ . Substituting the bound for ∑t=1T‖Mt−∇f(Wt)‖F _t=1^TE\|M_t-∇ f(W_t)\|_F and [f(W1)−f(WT+1)]≤ΔE[f(W_1)-f(W_T+1)]≤ into the previous inequality, we obtain the final bound for the cumulative expected gradient norm: ∑t=1T[‖∇f(Wt)‖F] _t=1^TE\! [\|∇ f(W_t)\|_F ] ≤Δηsmin+(1+ℓ)smaxsmin(σ+β1Gmax1−β1) ≤ η s_ + (1+ )s_ s_ (σ+ _1G_ 1- _1 ) +T(1+ℓ)smaxsmin1−β11+β1σ +T (1+ )s_ s_ 1- _1 1+ _1\,σ +Tβ1ηℓL(1+ℓ)smax2(1−β1)smin+TηℓLsmax22smin. +T _1η L(1+ )s_ ^2(1- _1)s_ +T η Ls_ ^22s_ . Dividing both sides by T immediately yields a bound on the average expected gradient norm. Since the minimum is always upper bounded by the average, i.e., min1≤t≤T[‖∇f(Wt)‖F]≤1T∑t=1T[‖∇f(Wt)‖F] _1≤ t≤ TE\! [\|∇ f(W_t)\|_F ]≤ 1T _t=1^TE\! [\|∇ f(W_t)\|_F ], we obtain the final convergence bound. Theorem 1 (Convergence Guarantee of DGA-Muon). Let WtW_t be defined as in Algorithm 1. Let smins_ and smaxs_ denote uniform lower and upper bounds on the adaptive scaling factors satisfying 0<smin≤(st)i≤smax<∞0<s_ ≤(s_t)_i≤ s_ <∞ for all t and i, and let ℓ=min(m,n) = (m,n). Denote Δ=f(W1)−f⋆ =f(W_1)-f . Assume that f is L-smooth and that the stochastic gradients are unbiased with bounded variance and bounded norm. Then, for a fixed step size η, DGA-Muon satisfies the following convergence guarantee min1≤t≤T[‖∇f(Wt)‖F] _1≤ t≤ TE\! [\|∇ f(W_t)\|_F ] ≤ΔηsminT+(1+ℓ)smaxsminT(σ+β1Gmax1−β1) ≤ η s_ T+ (1+ )s_ s_ T (σ+ _1G_ 1- _1 ) +(1+ℓ)smaxsmin1−β11+β1σ + (1+ )s_ s_ 1- _1 1+ _1\,σ +β1ηℓL(1+ℓ)smax2(1−β1)smin+ηℓLsmax22smin. + _1η L(1+ )s_ ^2(1- _1)s_ + η Ls_ ^22s_ . In particular, this bound can be summarized as min1≤t≤T[‖∇f(Wt)‖F]=O(1T+σ+η). _1≤ t≤ TE\! [\|∇ f(W_t)\|_F ]=O\! ( 1T+σ+η ). Therefore, for a fixed step size η, DGA-Muon achieves a convergence rate of (1/T+σ+η)O(1/T+σ+η). The adaptive scaling primarily influences the constant factors within the convergence bound. This result reveals the standard bias-variance tradeoff inherent in constant step-size stochastic optimization: the transient optimization error decays at an (1/T)O(1/T) rate, while the stochastic gradient variance and the finite learning rate introduce an irreducible steady-state error. Consequently, DGA-Muon converges to a neighborhood of a stationary point, characterized by a bounded expected gradient norm whose radius depends on the stochastic variance σ and the step size η. Similar to classical stochastic optimization methods, a diminishing step-size schedule is expected to gradually eliminate this steady-state error, although extending the present analysis to this setting is beyond the scope of this work. Furthermore, the current analysis establishes a foundational convergence guarantee for DGA-Muon. A rigorous theoretical characterization of when the geometry-aligned adaptive scaling yields provable improvements over the original Muon update remains an open question and is left for future work. 6 Experiments In this section, we report our experimental results. Note that, due to limited computational resources, some experiments were conducted only on a small scale, especially the ablation studies, and are not included in the current version. We will continue to expand the experimental evaluation as additional results become available. 6.1 Experimental Setup We conduct our experiments using NanoChat (Karpathy, 2025), a highly optimized transformer pretraining framework that can serve as a testbed for evaluating optimization methods. Both the model architecture and training recipe have undergone extensive engineering and empirical tuning in the development of NanoChat, including multiple rounds of automated optimization with Autoresearch (Karpathy, 2026). As a result, NanoChat provides a strong and carefully tuned baseline for optimizer evaluation. We use the default NanoChat model architecture without modification and refer readers to the repository for full implementation details. For our experiments, we use a model that has 2424 layers and 1.381.38B parameters in total, with full attention used across all layers. All experiments are conducted on a single node with eight NVIDIA A100 (80GB) GPUs. We next describe the optimizer configuration used in NanoChat, which incorporates a range of advanced and carefully tuned optimization techniques. NanoChat employs the NorMuon (Li et al., 2026) optimizer. As in conventional Muon-based training, the optimizer is applied only to matrix-valued parameters, including the Q/K/V/O and FFN weight matrices, while the remaining parameters are optimized with AdamW (Loshchilov and Hutter, 2019). For orthogonalization, NanoChat uses the Polar Express (Amsel et al., 2026) technique. For the matrix-valued parameters, NanoChat employs the recently proposed cautious weight decay (Chen et al., 2026) in place of conventional decoupled weight decay. We also ablate this design choice by replacing cautious weight decay with standard decoupled weight decay and find a clear degradation in training performance. 6.2 Empirical Verification of NorMuon’s Scaling Degeneration Recall that in Section 3, we theoretically showed that NorMuon’s adaptive scaling degenerates to an approximately constant factor on square matrices. In this experiment, we empirically verify this theoretical characterization by removing the adaptive scaling on the Q/K/V/O matrices, in which case NorMuon effectively reduces to standard Muon. In the current NanoChat implementation, the Q/K/V/O projection matrices are optimized as full matrices over all attention heads rather than separately on a per-head basis, and hence are square. We also tested per-head optimization of the Q/K/V matrices, but observed no improvement over joint optimization. Figure 1: Empirical verification of NorMuon’s adaptive scaling degeneration on square matrices in NanoChat. Removing NorMuon’s adaptive scaling on the square Q/K/V/O projection matrices, effectively reducing NorMuon to standard Muon, leaves the validation BPB essentially unchanged. Following the NanoChat setup, the total number of training steps is 7,3087,308, and validation is performed every 250250 steps, with an additional validation run at the end of training. We use validation bits per byte (BPB) as the primary evaluation metric, which measures the average number of bits required to encode each byte of the validation data and is invariant to the tokenizer vocabulary size. Lower BPB indicates better language-modeling performance. As shown in Figure 1, removing NorMuon’s adaptive scaling on the square Q/K/V/O matrices leaves the validation BPB essentially unchanged, with the curves of the two optimizers nearly overlapping. For the training loss, the two curves are virtually indistinguishable, and we therefore report them in Figure 3 in the Appendix. This result is consistent with our theoretical analysis: for square matrices, the corresponding adaptive scaling factors of NorMuon degenerates to an approximately constant scalar, effectively eliminating its intended adaptive effect. Consequently, removing this scaling has negligible impact on optimization performance, providing empirical support for our theoretical characterization. 6.3 DGA-Muon Outperforms NorMuon We now evaluate the proposed DGA-Muon on NanoChat, using NorMuon as the baseline. We retain the same 2424-layer, 1.381.38B-parameter Transformer architecture as in the ablation study above. For this end-to-end evaluation, we increase the data-to-parameter ratio from 1010 to 1818, which is also closer to the Chinchilla-style scaling regime (Hoffmann et al., 2022). Note that DGA-Muon and NorMuon have different optimization characteristics due to their distinct adaptive scaling mechanisms, and therefore require different hyperparameter configurations. We first explain the rationale behind our hyperparameter choices for DGA-Muon. Recall that DGA-Muon constructs its adaptive scaling signal from the raw gradients, whereas NorMuon derives it from the orthogonalized gradients. Orthogonalization provides Muon with favorable isotropic spectral properties, but also removes information encoded in the relative magnitudes of different singular directions. This information becomes increasingly valuable in the later stages of training, as the overall update magnitude decreases and finer-grained differences across singular directions become more important for effective optimization. In contrast, the raw-gradient signal retains this anisotropic information, while exhibiting greater step-to-step variation. These characteristics motivate a more gradual learning-rate schedule for DGA-Muon. Specifically, we extend the warmup phase, increase the warmdown ratio, and reduce the final learning rate fraction, while slightly lowering the learning rate for matrix-valued parameters. The complete hyperparameter configurations are provided in Table 1 in the Appendix. Figure 2: Comparison of DGA-Muon and NorMuon in NanoChat. DGA-Muon consistently outperforms NorMuon, achieving lower validation BPB throughout training. As shown in Figure 2, DGA-Muon significantly outperforms NorMuon in the end-to-end evaluation, achieving a final validation BPB of 0.69100.6910, compared with 0.69240.6924 for NorMuon. The difference in training loss is more pronounced, with DGA-Muon reaching a final loss of 2.27082.2708 compared with 2.27992.2799 for NorMuon; the corresponding training-loss curves are provided in Figure 4 in the Appendix. We further evaluate the downstream performance of the two optimizers on the DCLM CORE benchmark, the standard aggregate evaluation metric used in NanoChat. DGA-Muon achieves a CORE score of 0.30890.3089, while NorMuon obtains 0.29420.2942, yielding an absolute improvement of 0.01470.0147 and a relative gain of 5.0%5.0\%. Full downstream results are provided in Table 2 in the Appendix. These results validate the effectiveness of our proposed adaptive-scaling design. By constructing adaptive scaling signals directly from raw gradients and aligning the scaling with the underlying geometry of matrices, DGA-Muon achieves more effective adaptation than NorMuon, leading to lower training loss and validation BPB, as well as better downstream performance. References Amsel et al. (2026) N. Amsel, D. Persson, C. Musco, and R. M. Gower The Polar Express: optimal matrix sign methods and their application to the Muon algorithm. In International Conference on Learning Representations (ICLR), Cited by: §1, §2, §3.2, §6.1. Chen et al. (2026) L. Chen, J. Li, K. Liang, B. Su, C. Xie, C. Liang, N. Lao, and Q. Liu Cautious weight decay. In International Conference on Learning Representations, Cited by: §6.1, footnote 2. Duchi et al. (2011) J. Duchi, E. Hazan, and Y. Singer Adaptive subgradient methods for online learning and stochastic optimization.. Journal of machine learning research. Cited by: §1, §1. Grishina et al. (2025) E. Grishina, M. Smirnov, and M. Rakhuba Accelerating newton-schulz iteration for orthogonalization via chebyshev-type polynomials. arXiv preprint arXiv:2506.10935. Cited by: §3.2. Gupta et al. (2018) V. Gupta, T. Koren, and Y. Singer Shampoo: preconditioned stochastic tensor optimization. In International Conference on Machine Learning (ICML), Cited by: §1. He et al. (2025) W. He, K. Han, H. Zhou, H. Chen, Z. Liu, X. Chen, and Y. Wang ROOT: robust orthogonalized optimizer for neural network training. arXiv preprint arXiv:2511.20626. Cited by: §3.2. Higham (2020) N. J. Higham What is the polar decomposition. External Links: Link Cited by: §1, §2. Higham (2008) N. J. Higham Functions of matrices: theory and computation. Society for Industrial and Applied Mathematics (SIAM). Cited by: §1, §2. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre Training compute-optimal large language models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §6.3. Horn and Johnson (2013) R. A. Horn and C. R. Johnson Matrix analysis. 2nd edition, Cambridge University Press, Cambridge, UK. Cited by: §5. Jordan et al. (2024) K. Jordan, J. Bernstein, B. Rappazzo, B. Vlado, J. You, F. Cesista, B. Koszarsky, et al. Modded-NanoGPT: speedrunning the NanoGPT baseline. External Links: Link Cited by: §1. Jordan (2024) K. Jordan Muon: an optimizer for hidden layers in neural networks. External Links: Link Cited by: §1, §2. Karpathy (2025) A. Karpathy NanoChat: the best ChatGPT that $100 can buy. GitHub. External Links: Link Cited by: §1, §6.1. Karpathy (2026) A. Karpathy Autoresearch. External Links: Link Cited by: §6.1. Kim and Oh (2026) G. Y. Kim and M. Oh Convergence of Muon with Newton–Schulz. In International Conference on Learning Representations (ICLR), Cited by: §1. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), Cited by: §1, §1. Li and Hong (2025) J. Li and M. Hong A note on the convergence of Muon. arXiv preprint arXiv:2502.02900. Cited by: §1. Li et al. (2026) Z. Li, L. Liu, C. Liang, W. Chen, and T. Zhao NorMuon: making Muon more efficient and scalable. In International Conference on Machine Learning (ICML), Cited by: §1, §3.2, §6.1. Liu et al. (2025) J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan, et al. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982. Cited by: §1. Liu et al. (2026) Z. Liu, R. Zhang, Z. Wang, Y. Zhao, Y. Su, Z. Yang, and Z. Zhang Muon2^2: boosting Muon via adaptive second-moment preconditioning. arXiv preprint arXiv:2604.09967. Cited by: §1. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), Cited by: §1, §6.1. Nguyen et al. (2026) T. Nguyen, T. Nguyen, M. Truong, T. Nguyen, J. Bailey, and T. Le Spectral flattening is all Muon needs: how orthogonalization controls learning rate and convergence. arXiv preprint arXiv:2605.13079. Cited by: §1. Pethick et al. (2025) T. Pethick, W. Xie, K. Antonakopoulos, Z. Zhu, A. Silveti-Falls, and V. Cevher Training deep learning models with norm-constrained LMOs. In International Conference on Machine Learning (ICML), Cited by: §4.3. Shah et al. (2025) I. Shah, A. M. Polloreno, K. Stratos, P. Monk, A. Chaluvaraju, A. Hojel, A. Ma, A. Thomas, A. Tanwer, D. J. Shah, et al. Practical efficiency of Muon for pretraining. arXiv preprint arXiv:2505.02222. Cited by: §1. Song et al. (2026) Z. Song, C. S. Bai, Z. Zhang, B. Bullins, and D. F. Gleich Decoupling variance and scale-invariant updates in adaptive gradient descent for unified vector and matrix optimization. arXiv preprint arXiv:2602.06880. Cited by: §1. Su (2025) W. Su Isotropic curvature model for understanding deep learning optimization: is gradient orthogonalization optimal?. arXiv preprint arXiv:2511.00674. Cited by: §1. Team et al. (2026) K. Team, T. Bai, Y. Bai, Y. Bao, J. Cai, X. Cai, P. Cao, Y. Cao, Z. Chai, Y. Charles, et al. Kimi K3: open frontier intelligence. arXiv preprint arXiv:2607.24653. Cited by: §1. Team et al. (2025) K. Team, Y. Bai, Y. Bao, Y. Charles, C. Chen, G. Chen, H. Chen, H. Chen, J. Chen, N. Chen, et al. Kimi K2: open agentic intelligence. arXiv preprint arXiv:2507.20534. Cited by: §1. Wang et al. (2026a) S. Wang, F. Zhang, J. Li, D. Bergemann, and Z. Yang Why Muon outperforms Adam: a curvature perspective. arXiv preprint arXiv:2606.04662. Cited by: §1. Wang et al. (2026b) S. Wang, F. Zhang, J. Li, C. Du, C. Du, T. Pang, Z. Yang, M. Hong, and V. Tan Muon outperforms Adam in tail-end associative memory learning. In International Conference on Learning Representations (ICLR), Cited by: §1. Xu et al. (2026a) A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. DeepSeek-V4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: §1. Xu et al. (2026b) C. Xu, W. Yan, and Y. A. Zhang FISMO: fisher-structured momentum-orthogonalized optimizer. arXiv preprint arXiv:2601.21750. Cited by: §1. Zeng et al. (2026) A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. GLM-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763. Cited by: §1. Zeng et al. (2025) A. Zeng, X. Lv, Q. Zheng, Z. Hou, B. Chen, C. Xie, C. Wang, D. Yin, H. Zeng, J. Zhang, et al. GLM-4.5: agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471. Cited by: §1. Zhang et al. (2026a) J. Zhang, N. Amsel, B. Chen, and T. Dao Gram Newton–Schulz. External Links: Link Cited by: §1. Zhang et al. (2025a) M. Zhang, Y. Liu, and H. Schaeffer AdaGrad meets Muon: adaptive stepsizes for orthogonal updates. In Workshop on Optimization for Machine Learning in NeurIPS, Cited by: §1. Zhang et al. (2026b) W. Zhang, R. Yu, and P. Zhao Matrix adagrad: row-wise and column-wise adaptive subgradient methods. External Links: 2609.21815, Link Cited by: item 2. Zhang et al. (2026c) Y. Zhang, S. Xing, J. Huang, K. Lv, Y. Zhou, X. Qiu, Q. Guo, and K. Chen Mousse: rectifying the geometry of Muon with curvature-aware preconditioning. arXiv preprint arXiv:2603.09697. Cited by: §1. Zhang et al. (2025b) Y. Zhang, C. Chen, Z. Li, T. Ding, C. Wu, D. D. Kingma, Y. Ye, Z. Luo, and R. Sun Adam-mini: use fewer learning rates to gain more. In International Conference on Learning Representations (ICLR), Cited by: item 2. Appendix Appendix A Additional experimental Details Figure 3: Empirical verification of NorMuon’s adaptive scaling degeneration on square matrices in NanoChat. Removing NorMuon’s adaptive scaling on the square Q/K/V/O projection matrices, effectively reducing NorMuon to standard Muon, yields virtually indistinguishable training losses. Table 1: Hyperparameter configurations for the NorMuon baseline and DGA-Muon in NanoChat. Hyperparameter NorMuon Baseline DGA-Muon Device batch size 32 32 Total batch size Auto Auto Weight decay 0.28 0.28 Embedding LR 0.3 0.3 Unembedding LR 0.008 0.008 Scalar LR 0.5 0.5 Matrix LR 0.020 0.019 Warmup steps 40 200 Warmdown ratio 0.65 0.78 Final LR fraction 0.05 0.01 Lower scaling clip – 0.0010.001 Upper scaling clip – 2020 Figure 4: Comparison of DGA-Muon and NorMuon in NanoChat. DGA-Muon consistently outperforms NorMuon, achieving lower training loss throughout training. Table 2: Downstream task performance of NorMuon and DGA-Muon on the DCLM CORE benchmark. Higher scores indicate better performance. Task NorMuon DGA-Muon Diff. HellaSwag ZeroShot 0.6080 0.6100 +0.0020 Jeopardy 0.1680 0.1840 +0.0160 BIG-Bench QA Wikidata 0.5300 0.5560 +0.0260 ARC-Easy 0.7460 0.7420 −-0.0040 ARC-Challenge 0.4760 0.4580 −-0.0180 COPA 0.6400 0.7100 +0.0700 CommonsenseQA 0.2300 0.2560 +0.0260 PIQA 0.7800 0.7840 +0.0040 OpenBookQA 0.4320 0.4080 −-0.0240 LAMBADA 0.4880 0.4860 −-0.0020 HellaSwag 0.6180 0.6260 +0.0080 Winograd 0.7143 0.6996 −-0.0147 Winogrande 0.5760 0.5660 −-0.0100 BIG-Bench Dyck Languages 0.0660 0.0940 +0.0280 AGI-Eval LSAT-AR 0.2478 0.3130 +0.0652 BIG-Bench CS Algorithms 0.3940 0.4220 +0.0280 BIG-Bench Operators 0.2238 0.1857 −-0.0381 BIG-Bench Repeat Copy Logic 0.0312 0.0312 +0.0000 SQuAD 0.5100 0.4740 −-0.0360 CoQA 0.3220 0.3100 −-0.0120 BoolQ 0.5660 0.6280 +0.0620 BIG-Bench Language Identification 0.2640 0.2500 −-0.0140 CORE 0.2942 0.3089 +0.0147