Paper deep dive
Reassessing Muon for Matrix Factorization
Ali Parviz, Gal Mishne, Alex Cloninger
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/18/2026, 7:03:12 AM
Summary
This paper reassesses the Muon optimizer for matrix factorization, systematically comparing it against AdamW, Gradient Descent, and SignGD. The authors demonstrate that Muon's reported advantages are highly sensitive to hyperparameter tuning and problem structure. While AdamW matches or outperforms Muon on low-rank factorization and matrix completion, Muon retains a consistent advantage in non-negative matrix factorization. The study advocates for controlled, hyperparameter-swept benchmarks over default single-configuration evaluations.
Entities (9)
Relation Signals (5)
Optimizer Performance → dependson → Learning Rate
confidence 95% · Performance is therefore dominated by the learning rate, and any comparison made at one pre-selected value reflects that choice as much as the optimizer itself.
Muon → uses → Newton-Schulz Iterations
confidence 95% · Muon transforms the update direction through an approximate orthogonalization step based on Newton–Schulz iterations
Optimizer Performance → dependson → Condition Number
confidence 94% · Conditioning is itself a hyperparameter of the problem, and it interacts strongly with the optimizer. As κ grows, every method’s attainable loss degrades
AdamW → outperforms → Muon
confidence 92% · On low-rank factorization and matrix completion, its reported gains disappear under equal tuning, with AdamW and GD matching or exceeding its performance.
Muon → outperforms → AdamW
confidence 90% · On NMF, however, Muon retains a consistent advantage, likely because its orthogonalized updates discourage redundant factors and promote more diverse representations.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.
Tags
Links
- Source: https://arxiv.org/abs/2607.13246v1
- Canonical: https://arxiv.org/abs/2607.13246v1
Trouble viewing inline? Open PDF directly →
Full Text
70,257 characters extracted from source content.
Expand or collapse full text
Reassessing Muon for Matrix Factorization Ali Parviz † alparviz@ucsd.edu Gal Mishne † gmishne@ucsd.edu Alex Cloninger ∗† acloninger@ucsd.edu Abstract Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW on large language model training. Its empirical success has motivated a growing theoretical literature that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon’s advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work we isolate the optimizer from these confounders by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled and systematically tuned comparison against adaptive baselines, we find that Muon doesnot consistently outperform AdamW in this setting, and that several previously reported advantages are sensitive to hyperparameter choices. Our results give a more nuanced picture of when spectrum-aware orthogonalization helps, and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks. 1. Introduction Recent advances in large-scale optimization have introduced a class of optimizers that ex- plicitly exploit the matrix structure of gradient updates. Among these, Muon (MomentUm Orthogonalized by Newton–Schulz) has emerged as a promising alternative to standard first- order methods (Jordan, 2024). Rather than applying a momentum update directly, Muon transforms the update direction through an approximate orthogonalization step based on Newton–Schulz iterations, effectively reshaping the spectrum of the gradient. Empirically, Muon has demonstrated strong performance on modern deep learning work- loads, including GPT-style models, where it improves training efficiency and in some cases outperforms widely used optimizers such as Adam and AdamW (Jordan, 2024; Liu et al., 2025; Shah et al., 2025b). Notably, large-scale studies report up to a twofold speedup over AdamW in multi-billion-parameter language model training (Shah et al., 2025b). These successes have motivated a growing body of theoretical work that studies Muon through the lenses of spectral-norm steepest descent and constrained optimization. Existing analyses establish convergence guarantees under various simplifying assumptions, including exact orthogonal- ization, simplified momentum dynamics, and locally quadratic models (Pethick et al., 2025a; Li and Hong, 2025; Shen et al., 2025; Chen et al., 2025; Kovalev, 2025; Riabinin et al., 2025; Gruntkowska et al., 2025; Sato et al., 2025; Nagashima and Iiduka, 2026), while more recent ∗. Department of Mathematics, UC San Diego. †. Halicioğlu Data Science Institute, UC San Diego. arXiv:2607.13246v1 [cs.LG] 14 Jul 2026 Parviz Mishne Cloninger work studies inexact Newton–Schulz iterations and how approximation errors propagate into convergence guarantees (Shulgin et al., 2025; Kim and Oh, 2026; Lau et al., 2025). Despite this progress, it remains unclear which aspects of Muon’s advantage are intrinsic to the optimizer and which arise from the complexity of large-scale deep learning, where architectural, scale, and data-dependent effects are hard to disentangle. Existing analyses rarely show when and why Muon should outperform classical optimizers in concrete, well- specified problems, tending to concern idealized settings or purely local properties (Davis and Drusvyatskiy, 2025; Su, 2025). We take a complementary perspective and study Muon on low-rank matrix factorization: a simple yet fundamental problem that allows systematic, exhaustive evaluation and where Muon’s spectral nature might be expected to help, since optimization is governed by well-characterized curvature and singular-value structure. Our goal is to characterize both the strengths and limitations of Muon and to identify regimes where it does or does not outperform standard optimizers such as Adam and AdamW. This focus exposes a gap between expectation and behavior, which we organize around three questions. Q1: Does Muon offer any advantage in structured problems such as matrix factor- ization? Matrix factorization has explicit low-rank structure and is associated with a well-behaved optimization landscape. If Muon genuinely exploits spectral properties of the gradient, this should translate into faster convergence. If it does not, what limits its effectiveness relative to adaptive methods such as AdamW? Q2: Are Muon’s benefits tied to specific problem structures? Which properties of an optimization problem make orthogonalized updates advantageous? Do their benefits stem from specific geometric or spectral features of the objective, or do they generalize across matrix factorization tasks with varying conditioning, over-parameterization, and constraints? Q3: How does Muon compare to adaptive methods under ill-conditioning? Matrix factorization is sensitive to conditioning, especially when singular values decay rapidly. Optimizers such as AdamW implicitly adapt to coordinate-wise scaling. Does Muon’s spectral normalization compete with or conflict with such adaptivity, and under which regimes does one dominate? Findings. Contrary to its strong performance in large-scale models, Muon’s advantage in these canonical settings is problem-dependent. On low-rank factorization and matrix completion, its reported gains disappear under equal tuning, with AdamW and GD matching or exceeding its performance. On NMF, however, Muon retains a consistent advantage, likely because its orthogonalized updates discourage redundant factors and promote more diverse representations. This dependence on problem structure suggests that Muon’s broader success may also arise from properties specific to deep learning and underscores the value of controlled optimization benchmarks. 2. Matrix factorization: symmetric and nonnegative variants We begin by fixing notation used throughout the paper. For a matrix M, let σ i (M ) denote its i-th largest singular value and σ min (M ) its smallest. We write ∥M∥ op and ∥M∥ F for the spectral and Frobenius norms, respectively. For k ≤ d, let O d×k be the set of matrices in R d×k with orthonormal columns. Given scalars a 1 ,...,a d , we write diaga 1 ,...,a d for 2 Muon for Matrix Factorization the diagonal matrix with these entries. We discuss Muon (Jordan, 2024) in Section A and defer a detailed discussion of related work to Appendix C. Symmetric matrix factorization We consider the symmetric matrix factorization prob- lem min U∈R d×k f (U ) = 1 4 U ⊤ − M ⋆ 2 F ,(1) where M ⋆ ∈ R d×d is a rank-r positive semidefinite matrix and U ∈ R d×k is a factor with k ≥ r columns (k = r is the exactly parameterized case and k > r is over-parameterized). The goal is to recover M ⋆ through a low-rank factorization U ⊤ by solving equation 1. We assume M ⋆ admits the eigendecomposition M ⋆ = V ⋆ Λ ⋆ V ⋆⊤ , where Λ ⋆ = diagλ ⋆ 1 ,...,λ ⋆ r collects the nonzero eigenvalues with λ ⋆ 1 ≥ · ≥ λ ⋆ r > 0 and V ⋆ ∈ R d×r is orthonormal. The condition number of M ⋆ is κ : = λ ⋆ 1 /λ ⋆ r .(2) The symmetric problem equation 1 is the positive semidefinite, factor-tied instance of a broader family of low-rank factorization problems; we defer the general bilinear factorization and the matrix completion variant to Appendix B. Nonnegative matrix factorization. A closely related variant constrains both the target and the factors to be entrywise nonnegative. For a rank-r target M ⋆ ∈ R d 1 ×d 2 , nonnegative matrix factorization (NMF) solves min U∈R d 1 ×k ≥0 V∈R d 2 ×k ≥0 1 2 UV ⊤ − M ⋆ 2 F ,(3) where R ≥0 denotes the nonnegative reals and k ≥ r. NMF is the nonnegativity-constrained special case of the general bilinear factorization equation 12 (Appendix B). Moreover, We an- alyze the exact softplus and its Taylor surrogate as nonlinear matrix factorizations and show the truncation imposes a rank cap that explains the high-rank gap in Fig. 12 (Appendix. K). 3. Experimental Design and Evaluation Protocol We test the claim that optimizer comparisons in matrix factorization are highly sensitive to hyperparameters, so that conclusions drawn from a single (often default) configuration are not representative. Going beyond prior work (Ma et al., 2026), we evaluate five matrix- recovery problems under a controlled protocol that holds model capacity and parameteriza- tion fixed and varies only optimization-related hyperparameters, spanning well-conditioned to severely ill-conditioned targets. Two further analyses isolate how conditioning acts: the sensitivity of each optimizer to the shape of the eigenvalue distribution at a fixed, extreme condition number (Section 4.1), and the dynamics with which each recovers the target’s spectral subspace (Section J). 3.1. Problem settings All problems use ambient dimension d = 100 and a factorized parameterization ˆ M = UV ⊤ with U,V ∈ R d×r , optimized from a small random initialization. We consider: 3 Parviz Mishne Cloninger 1e-044e-042e-031e-028e-025e-01 10 −15 10 −12 10 −9 10 −6 10 −3 10 0 10 3 κ=1 1e-044e-042e-031e-028e-025e-01 10 −13 10 −10 10 −7 10 −4 10 −1 10 2 κ=5 1e-044e-042e-031e-028e-025e-01 10 −11 10 −8 10 −5 10 −2 10 1 10 4 κ=25 1e-044e-042e-031e-028e-025e-01 10 −10 10 −7 10 −4 10 −1 10 2 10 5 κ=125 1e-044e-042e-031e-028e-025e-01 10 −8 10 −5 10 −2 10 1 10 4 10 7 κ=625 MuonAdamWGDSignGD Figure 1: Low-rank factorization, conditioning sweep (target rank r=15, matched search rank k=15, dimension d=100). Tuned final loss vs. learning rate for each condition number κ, from well-conditioned (κ=1) to strongly ill-conditioned (κ=625) (geometric mean over 3 seeds,±log-std band). The target’s 15 singular values are spaced linearly from κ down to 1, so κ is exactly the condition number. When tuned, AdamW and GD reach near machine precision; the Muon configuration plateaus several orders higher. 1e-049e-041e-022e-01 3e+00 10 −12 10 −9 10 −6 10 −3 10 0 10 3 (a) 1e-049e-041e-022e-01 3e+00 10 −10 10 −7 10 −4 10 −1 10 2 10 5 (b) 1e-049e-041e-022e-01 3e+00 10 −6 10 −4 10 −2 10 0 10 2 10 4 10 6 (c) 1e-049e-041e-022e-01 3e+00 10 −10 10 −7 10 −4 10 −1 10 2 (d) 1e-049e-041e-022e-01 3e+00 10 −10 10 −7 10 −4 10 −1 10 2 10 5 (e) 1e-049e-041e-022e-01 3e+00 10 −7 10 −5 10 −3 10 −1 10 1 10 3 10 5 (f) MuonAdamWGDSignGD Figure 2: Non-negative factorization. Final loss (y-axis, log scale; geometric mean over 3 seeds, shaded ±one-log-standard-deviation band) versus learning rate (x-axis, log scale). Panels (a–c) use the uniform spectrum and (d–f) the decayed spectrum, at factor ranks r = 10, 50, 100 respectively. In both spectra, tuned Muon is the only method to fit the target across ranks; AdamW trails by several orders and GD/SignGD fail. • Low-rank factorization (conditioning). A fully observed target M ⋆ = U ⋆ diag(σ)U ⋆⊤ of rank r = 15 (i.e., we solve for the symmetric factorization) whose singular values are spaced linearly between κ and 1, so κ∈1, 5, 25, 125, 625 is exactly the condition number. Objective 1 4 ∥U ⊤ − M ⋆ ∥ 2 F (Figure 1). • Matrix completion (conditioning). A rank-4 symmetric PSD target with the same conditioning sweep, observed on a uniformly random 20% of entries; loss 1 2 ∥P Ω (UV ⊤ − M ⋆ )∥ 2 F , the squared error on the observed support (Figure 7). • Matrix completion (over-parameterization). The same completion task at fixed κ = 5 and true rank 4, sweeping the search rank k ∈4, 5, 100 (matched, mildly, and heavily over-parameterized) (Figure 8). • Non-negative factorization. A non-negative target with a projected (· ≥ 0) pa- rameterization, at factor rank k ∈ 10, 50, 100, under two target spectra: a uniform 4 Muon for Matrix Factorization Figure 3: AdamW outperforms Muon variants across spectrum shapes (matrix factorization, target m×n = 150×150, rank r=5). Tuned final loss for each of the seven spectral profiles at fixed κ = 10 4 (mean over 3 seeds). Both Muon and Muon without Nesterov-style momentum exhibit degraded relative performance on most spectral shapes, whereas AdamW remains effective despite the extreme ill-conditioning. 020406080 0 20 40 60 80 (a) AdamW (diag. energy = 0.505) 0.0 0.2 0.4 0.6 0.8 020406080 0 20 40 60 80 (b) Muon (diag. energy = 0.578) 0.0 0.2 0.4 0.6 0.8 020406080 0 20 40 60 80 (c) Difference (Muon AdamW) 0.3 0.2 0.1 0.0 0.1 0.2 0.3 blue: AdamW > Muon · red: Muon > AdamW Alignment of true eigenvectors vs. learned singular vectors true eigenvector index j learned sing.-vector index i Figure 4: Spectral subspace recovery (Gaussian-kernel target K ∈ R 100×100 , full rank k=100, tuned LR, 12 inits). Alignment M ij = |⟨u i ,e j ⟩| (Eq. 16) between learned singular vectors and true eigenvectors of K. (a) AdamW and (b) Muon both recover the top eigen- vectors, diverging in the near-degenerate mid-band. (c) The difference isolates it: AdamW holds a tighter ordered diagonal (blue), Muon disperses onto neighbours (red) — lower di- agonal energy (0.586 vs. 0.643). spectrum and a decayed (exponentially graded) spectrum, the latter removing the flat “DC-component” degeneracy of the uniform case (Figure 2). • Tensor-train factorization (over-parameterization, noise). A symmetric posi- tive semidefinite target M ⋆ = U ⋆ U ⋆⊤ recovered with untied cores U,V in a noiseless (r ⋆ = 4) and a noisy (r ⋆ = 30) regime, sweeping the search rank across under-, exactly-, and over-parameterized values. Objective 1 2 ∥UV ⊤ − M ⋆ ∥ 2 F ; see Section 4.3 and Fig- ures 5 and 9. We compare Muon (momentum 0.95, Nesterov, 5 Newton–Schulz orthogonalization steps) against AdamW, plain gradient descent (GD), and sign gradient descent (SignGD). 5 Parviz Mishne Cloninger 10 4 10 3 10 2 10 1 10 0 10 1 10 3 10 2 10 1 final loss k = 2 (under-param.) 10 4 10 3 10 2 10 1 10 0 10 1 10 8 10 7 10 6 10 5 10 4 10 3 10 2 10 1 k = 4 (exact) 10 4 10 3 10 2 10 1 10 0 10 1 10 9 10 7 10 5 10 3 10 1 k = 8 (over-param.) 10 4 10 3 10 2 10 1 10 0 10 1 10 9 10 8 10 7 10 6 10 5 10 4 10 3 10 2 10 1 k = 20 (over-param.) learning rate GDAdamWSignGDMuon Figure 5: Tensor-Train Noiseless regime (true rank r ⋆ = 4). Final loss versus learning rate for each optimizer, across search ranks r ∈2, 4, 8, 20 spanning under-, exactly- (r = r ⋆ , bold panel), and over-parameterized settings. Solid lines are the median over 3 seeds; shaded bands the min–max. 3.2. Evaluation protocol We evaluate recovery with the normalized mean-squared error (NMSE), the squared recon- struction error relative to∥M ⋆ ∥ 2 F , which makes the metric scale-free and comparable across problems and conditioning regimes. For every (problem, condition, optimizer) triple we sweep the learning rate, the dominant axis and the usual source of misleading comparisons, over a logarithmically spaced grid (25 points in [10 −4 , 5× 10 −1 ] for the factorization and completion tasks; 20 points in [10 −4 , 3.2] for NMF), holding all other hyperparameters (mo- mentum, and for Muon the Newton–Schulz count J and the orthogonalization coefficients) at standard values. Each configuration runs for up to 3000 iterations under an identi- cal patience-based learning-rate-decay schedule and is repeated over 3 seeds. Because the resulting NMSE values span roughly thirty orders of magnitude, we summarize each con- figuration by the geometric mean over seeds with a ±one-log-standard-deviation band, and compare optimizers by their stable learning-rate range, the band of step sizes over which each converges to the NMSE floor. 4. Experimental Results A single learning rate is not a meaningful operating point. Within a fixed problem and optimizer, the final loss varies by a median of about 11 orders of magnitude across the learning-rate grid for the factorization and completion tasks, and about 6 orders for both NMF variants (Figures 1–2). Performance is therefore dominated by the learning rate, and any comparison made at one pre-selected value reflects that choice as much as the optimizer itself. No shared learning rate is fair to all methods. The loss-minimizing learning rate differs across optimizers by a median of 1.5–3.1 decades (largest for NMF), so evaluating all methods at a common learning rate necessarily places at least one of them far from its own optimum. This appears as a horizontal shift between each method’s basin in every panel. Tuning reorders the methods, and the winner is problem-dependent. Table 1 reports each optimizer’s best (learning-rate-tuned) final loss. In nearly all nineteen settings the ranking induced by a fixed “default” learning rate (≈ 10 −3 ) differs from the ranking 6 Muon for Matrix Factorization obtained once each method is tuned individually, and methods that look worst at the default frequently become best after tuning. Under per-optimizer tuning no method dominates universally: AdamW and GD win on plain low-rank factorization, Muon and AdamW are comparable on matrix completion, and Muon wins clearly on both NMF variants. The detailed per-setting comparisons, including the specific default-vs-tuned reorderings, are deferred to Appendix D. Ill-conditioning widens the gaps and shifts the winner. Conditioning is itself a hy- perparameter of the problem, and it interacts strongly with the optimizer. As κ grows, every method’s attainable loss degrades and its optimal learning rate migrates upward (Figures 1, 7). The effect is most dramatic for the non-adaptive methods on factorization, where GD goes from the best-tuned method in the well-conditioned regime to failing under extreme ill-conditioning while AdamW becomes best (see Appendix D for the precise values). Thus ill-conditioning does not merely raise the loss floor, it reorders the methods, so that no fixed configuration summarizes behavior across the conditioning range. 4.1. Sensitivity to spectrum shape under extreme ill-conditioning Conditioning alone does not determine difficulty: at a fixed condition number, the con- vergence of first-order optimizers also depends on the distribution of eigenvalues within [s min ,s max ]. We therefore hold the conditioning fixed at an extreme value and vary only the interior shape of the spectrum, evaluating seven spectral profiles (defined in Appendix F). All seven families share the same endpoints (s min ,s max ) = (10 −3 , 10) and hence the same, deliberately extreme, condition number κ = s max /s min = 10 4 ; only the interior shape differs. Despite this identical (and severe) ill-conditioning, the distributions pose markedly different preconditioning challenges. As shown in Figure 3, AdamW adapts effectively to heavy-tailed and clustered spectra, whereas Muon underperforms across the majority of these shapes, ev- idence that its spectral updates are less robust to specific eigenvalue densities even when the overall condition number is held fixed. Ill-conditioning is therefore not a single axis of difficulty: the same κ can be easy or hard depending on how the eigenvalues are arranged, and the two optimizers respond to that arrangement differently. 4.2. Gaussian-kernel factorization: a well-behaved landscape To complement the synthetic NMF problems, we study the Gaussian radial-basis-function kernel. We sample N = 100 points in R 5 and form K ij = exp(−∥x i − x j ∥ 2 /2σ 2 ) with σ = 2, giving a symmetric positive-semidefinite matrix with strictly positive entries and a rapidly decaying spectrum. We factorize it as ˆ K = UV ⊤ under a non-negativity constraint (U,V ≥ 0, enforced by projection after each step), minimizing 1 2 ∥UV ⊤ − K∥ 2 F . Rather than fix the factor rank arbitrarily, we set it by a principled bottleneck: the smallest rank r whose leading singular values capture at least 95% of K’s spectral energy, which yields r = 24 for this target. Each optimizer is tuned by a learning-rate sweep (30 points in [10 −4 , 10 0.5 ]), and every configuration is repeated over three seeds that vary the initialization while holding the target fixed. The result is shown in Figure 10. On this well-behaved landscape, Muon’s advanced orthogonalization is unnecessary: standard AdamW performs just as effectively, converging seamlessly to the same global floor as Muon. This reinforces the study’s central theme from 7 Parviz Mishne Cloninger the opposite direction, on the adversarial NMF spectra Muon’s structured updates help, but on a smooth, well-conditioned kernel target that advantage disappears and a well-tuned adaptive baseline is equally good. 4.3. Tensor-train factorization We consider the tensor-train (T) factorization of a matrix target. For an order-2 tensor M ⋆ ∈ R d×d , the T decomposition contracts two cores G 1 ∈ R d×r and G 2 ∈ R r×d with boundary ranks r 0 = r 2 = 1 and a single internal rank r 1 = r, M ⋆ (i,j) = r X α=1 G 1 (i,α)G 2 (α,j), ⇐⇒M ⋆ = G 1 G 2 ,(4) so the T factorization coincides with the bilinear factorization of Appendix B under U = G 1 , V ⊤ = G 2 , with the lone T-rank r playing the role of the factorization rank. In our setting the target M ⋆ = U ⋆ U ⋆⊤ is symmetric positive semidefinite, but we do not tie the cores: recovery uses untied factors U, V , so the problem is the general bilinear factorization equation 12 applied to a symmetric PSD target, retaining the GL r imbalance invariance of equation 12. Recovery minimizes f (U, V ) = 1 2 UV ⊤ − M ⋆ 2 F ,U ∈ R d×r , V ∈ R d×r ,(5) the objective of equation 12 with search rank r as the T-rank. We study equation 5 in two regimes, a noiseless regime (true rank r ⋆ = 4, recovered exactly, loss limited only by convergence) and a noisy regime (true rank r ⋆ = 30 with additive observation noise, loss floored at the statistical noise level), and within each we sweep the search rank across under-, exactly-, and over-parameterized values; full specifications are given in Appendix G. The quantity of interest is the stable learning-rate range of each optimizer, the band of step sizes over which equation 5 converges to its regime-dependent floor, and how that band widens, narrows, or shifts as the parameterization regime and the noise level change (Figures 5 and 9). Beyond low-rank recovery, this factorized objective serves as a minimal model of the product parameterizations ubiquitous in deep learning, a connection we develop in Appendix H. We also study the effect of factorization depth in Appendix I. 5. Conclusion In this work, we revisit optimizer comparisons for matrix factorization and show that conclu- sions are highly sensitive to hyperparameter selection and problem conditioning. Systematic learning-rate sweeps reveal that rankings reported under single configurations often do not hold: well-tuned AdamW and gradient descent frequently match or outperform Muon on standard low-rank factorization tasks, while Muon’s advantages are mostly limited to NMF and some highly ill-conditioned completion problems. We further find that optimizer per- formance depends on both conditioning severity and spectral structure, altering method rankings across problem instances. These results underscore the need to evaluate optimizers across tuned hyperparameter ranges and conditioning regimes rather than relying on default settings or isolated benchmarks. 8 Muon for Matrix Factorization Acknowledgements We thank Tianhao Wang for providing helpful references and for the discussions that assisted in developing the experimental setup. Ali Parviz was funded by NSF CIF-2403452. References Noah Amsel, David Persson, Christopher Musco, and Robert Gower. The polar express: Optimal matrix sign methods and their application to the Muon algorithm. CoRR, abs/2505.16932, 2025. doi: 10.48550/ARXIV.2505.16932. URL https://doi.org/10. 48550/arXiv.2505.16932. Kang An, Yuxing Liu, Rui Pan, Yi Ren, Shiqian Ma, Donald Goldfarb, and Tong Zhang. ASGO: Adaptive structured gradient optimization. arXiv preprint arXiv:2503.20762, 2025. Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International conference on machine learning, pages 244–253. PMLR, 2018. Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in neural information processing systems, 32, 2019. Jeremy Bernstein and Laker Newhouse. Modular duality in deep learning. arXiv preprint arXiv:2410.21265, 2024a. Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024b. Nicolas Boumal and Antoine Gonon. Designing polynomials for Muon’s polar factorization: Focus on quintics, 2025. URL w.racetothebottom.xyz/posts/polar-poly/. Accessed January 2026. David Carlson, Volkan Cevher, and Lawrence Carin. Stochastic spectral descent for re- stricted Boltzmann machines. In Artificial intelligence and statistics, pages 111–119. PMLR, 2015a. David Carlson, Ya-Ping Hsieh, Edo Collins, Lawrence Carin, and Volkan Cevher. Stochastic spectral descent for discrete graphical models. IEEE Journal of Selected Topics in Signal Processing, 10(2):296–311, 2015b. David E Carlson, Edo Collins, Ya-Ping Hsieh, Lawrence Carin, and Volkan Cevher. Pre- conditioned spectral descent for deep learning. Advances in neural information processing systems, 28, 2015c. Franz Louis Cesista, You Jiacheng, and Keller Jordan. Squeezing 1-2% efficiency gains out of Muon by optimizing the Newton-Schulz Coefficients, February 2025. URL https: //leloykun.github.io/ponder/muon-opt-coeffs/. Accessed January 2026. 9 Parviz Mishne Cloninger Lizhang Chen, Jonathan Li, and Qiang Liu. Muon optimizes under spectral norm constraints. CoRR, abs/2506.15054, 2025. doi: 10.48550/ARXIV.2506.15054. URL https://doi.org/ 10.48550/arXiv.2506.15054. Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. In Conference on learning theory, pages 698–728. PMLR, 2016. Damek Davis and Dmitriy Drusvyatskiy. When do spectral gradient updates help in deep learning? CoRR, abs/2512.04299, 2025. doi: 10.48550/ARXIV.2512.04299. URL https: //doi.org/10.48550/arXiv.2512.04299. Chen Fan, Mark Schmidt, and Christos Thrampoulidis. Implicit bias of spectral descent and Muon on multiclass separable data. arXiv preprint arXiv:2502.04664, 2025. Ekaterina Grishina, Matvey Smirnov, and Maxim V. Rakhuba. Accelerating Newton-Schulz iteration for orthogonalization via Chebyshev-type polynomials. CoRR, abs/2506.10935, 2025. doi: 10.48550/ARXIV.2506.10935. URL https://doi.org/10.48550/arXiv.2506. 10935. Kaja Gruntkowska, Alexander Gaponov, Zhirayr Tovmasyan, and Peter Richtárik. Error feedback for Muon and friends. CoRR, abs/2510.00643, 2025. doi: 10.48550/ARXIV.2510. 00643. URL https://doi.org/10.48550/arXiv.2510.00643. Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in neural information processing systems, 30, 2017. Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842–1850. PMLR, 2018. Keller Jordan. Muon: An optimizer for the hidden layers in neural networks. https: //kellerjordan.github.io/posts/muon/, 2024. Accessed January 2026. Valentin Khrulkov, Alexander Novikov, and Ivan Oseledets. Expressive power of recurrent neural networks. arXiv preprint arXiv:1711.00811, 2017. Gyu Yeol Kim and Min-hwan Oh. Convergence of Muon with Newton–Schulz. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=lJSfxtLpLm. Dmitry Kovalev. Understanding gradient orthogonalization for deep learning via non- euclidean trust-region optimization. CoRR, abs/2503.12645, 2025. doi: 10.48550/ARXIV. 2503.12645. URL https://doi.org/10.48550/arXiv.2503.12645. Tim Tsz-Kit Lau, Qi Long, and Weijie Su. Polargrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective. CoRR, abs/2505.21799, 2025. doi: 10.48550/ ARXIV.2505.21799. URL https://doi.org/10.48550/arXiv.2505.21799. 10 Muon for Matrix Factorization Jiaxiang Li and Mingyi Hong. A note on the convergence of Muon, 2025. URL https: //arxiv.org/abs/2502.02900. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for LLM training. CoRR, abs/2502.16982, 2025. doi: 10.48550/ARXIV.2502.16982. URL https://doi.org/10.48550/arXiv.2502.16982. Jianhao Ma, Yu Huang, Yuejie Chi, and Yuxin Chen. Preconditioning benefits of spectral orthogonalization in Muon, 2026. URL https://arxiv.org/abs/2601.13474. Shuntaro Nagashima and Hideaki Iiduka. Improved convergence rates of Muon optimizer for nonconvex optimization, 2026. URL https://arxiv.org/abs/2601.19400. Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015. Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained lmos. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net, 2025a. URL https://openreview.net/ forum?id=2Oqm2IzTy9. Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. arXiv preprint arXiv:2502.07529, 2025b. Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, and Peter Richtárik. Gluon: Making Muon & Scion great again! (bridging theory and practice of LMO-based optimizers for LLMs). CoRR, abs/2505.13416, 2025. doi: 10.48550/ARXIV.2505.13416. URL https: //doi.org/10.48550/arXiv.2505.13416. Naoki Sato, Hiroki Naganuma, and Hideaki Iiduka. Convergence bound and critical batch size of Muon optimizer, 2025. URL https://arxiv.org/abs/2507.01598. Ishaan Shah, Anthony M Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, An- drew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, and Darsh J Shah. Practical efficiency of Muon for pretraining. arXiv preprint arXiv:2505.02222, 2025a. Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, An- drew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J. Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani. Practical efficiency of Muon for pretraining. CoRR, abs/2505.02222, 2025b. doi: 10.48550/ARXIV.2505.02222. URL https://doi.org/10.48550/arXiv. 2505.02222. 11 Parviz Mishne Cloninger Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the conver- gence analysis of Muon. CoRR, abs/2505.23737, 2025. doi: 10.48550/ARXIV.2505.23737. URL https://doi.org/10.48550/arXiv.2505.23737. Egor Shulgin, Sultan AlRashed, Francesco Orabona, and Peter Richtárik. Beyond the ideal: Analyzing the inexact Muon update. CoRR, abs/2510.19933, 2025. doi: 10.48550/ARXIV. 2510.19933. URL https://doi.org/10.48550/arXiv.2510.19933. Weijie Su. Isotropic curvature model for understanding deep learning optimization: Is gradient orthogonalization optimal? arXiv preprint arXiv:2511.00674, 2025. Tian Tong, Cong Ma, and Yuejie Chi. Accelerating ill-conditioned low-rank matrix estima- tion via scaled gradient descent. Journal of Machine Learning Research, 22(150):1–63, 2021a. Tian Tong, Cong Ma, and Yuejie Chi. Low-rank matrix recovery with scaled subgradient methods: Fast and robust convergence without the condition number. IEEE Transactions on Signal Processing, 69:2396–2409, 2021b. Mark Tuddenham, Adam Prügel-Bennett, and Jonathan Hare. Orthogonalising gradients to speed up neural network optimisation. arXiv preprint arXiv:2202.07052, 2022. Amund Tveit, Bjørn Remseth, and Arve Skogvold. Muon optimizer accelerates grokking. arXiv preprint arXiv:2504.16041, 2025. Bhavya Vasudeva, Puneesh Deora, Yize Zhao, Vatsal Sharan, and Christos Thrampoulidis. How Muon’s spectral design benefits generalization: A study on imbalanced data. arXiv preprint arXiv:2510.22980, 2025. Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Cunxiao Du, Chao Du, Tianyu Pang, Zhuoran Yang, Mingyi Hong, and Vincent YF Tan. Muon outperforms Adam in tail-end associative memory learning. arXiv preprint arXiv:2509.26030, 2025. Xingyu Xu, Yandi Shen, Yuejie Chi, and Cong Ma. The power of preconditioning in overpa- rameterized low-rank matrix sensing. In International Conference on Machine Learning, pages 38611–38654. PMLR, 2023. Gavin Zhang, Salar Fattahi, and Richard Y Zhang. Preconditioned gradient descent for overparameterized nonconvex Burer-Monteiro factorization with global optimality certifi- cation. Journal of Machine Learning Research, 24(163):1–55, 2023. Jialun Zhang, Salar Fattahi, and Richard Y Zhang. Preconditioned gradient descent for over- parameterized nonconvex matrix factorization. Advances in Neural Information Processing Systems, 34:5985–5996, 2021. Thomas T Zhang, Behrad Moniri, Ansh Nagwekar, Faraz Rahman, Anton Xue, Hamed Hassani, and Nikolai Matni. On the concurrence of layer-wise preconditioning methods and provable feature learning. arXiv preprint arXiv:2502.01763, 2025. 12 Muon for Matrix Factorization Table 1: Learning-rate-tuned final loss (geometric mean over 3 seeds; lower is better). The best optimizer per row is in bold. The tuned ranking varies across problems and, within each problem, differs from the ranking at a fixed default learning rate. ProblemSettingMuonAdamWGDSignGD Factorization κ=12.3×10 −8 1.9×10 −13 1.6×10 −13 2.8×10 −7 κ=53.0×10 −7 1.9×10 −12 1.7×10 −12 4.4×10 −6 κ=253.0×10 −6 4.2×10 −11 3.7×10 −11 6.1×10 −3 κ=1254.1×10 −7 1.4×10 −9 1.0×10 −9 4.4×10 −1 κ=6259.4×10 −5 2.6×10 −7 1.3×10 −1 1.2×10 1 Completion (κ) κ=14.6×10 −16 4.8×10 −16 4.5×10 −11 2.6×10 −15 κ=55.7×10 −15 5.5×10 −15 1.6×10 −9 2.4×10 −13 κ=251.4×10 −13 1.4×10 −13 4.7×10 −7 3.5×10 −5 κ=1253.3×10 −7 1.7×10 −7 1.8×10 −2 2.2×10 −2 κ=6259.8×10 −3 2.0×10 −2 3.4×10 −2 3.6×10 −2 Completion (rank) r=45.7×10 −15 5.5×10 −15 1.6×10 −9 2.4×10 −13 r=54.5×10 −15 1.8×10 −12 2.1×10 −6 4.9×10 −8 r=1002.1×10 −15 1.5×10 −15 1.8×10 −14 3.2×10 −15 NMF (uniform) r=101.5×10 −9 6.1×10 −3 4.8×10 −1 8.7×10 −1 r=504.5×10 −9 4.1×10 −9 4.1×10 0 1.3×10 2 r=1008.9×10 −6 7.8×10 −3 1.2×10 2 1.2×10 3 NMF (decay) r=105.1×10 −7 2.3×10 −3 3.8×10 −2 1.2×10 −1 r=503.8×10 −9 8.4×10 −4 1.0×10 0 1.5×10 1 r=1001.2×10 −6 2.1×10 −3 6.3×10 0 3.4×10 2 Appendix A. Background: The Muon Optimizer We briefly recall the Muon update and the spectral viewpoint that motivates it. Consider a weight matrix W ∈ R m×n with loss gradient G t =∇ W L(W t ) at iteration t. Muon maintains a momentum buffer B t and applies an orthogonalized update: B t = μ B t−1 + G t ,(6) O t = Newton-Schulz(B t ) ≈ polar(B t ),(7) W t+1 = W t − α O t .(8) Here α > 0 is the step size and μ∈ [0, 1) the momentum coefficient. The orthogonalization step equation 7 replaces the momentum direction by (an approximation of) its polar factor. Polar factor and spectral steepest descent. Let B t = P ΣQ ⊤ be a singular value decomposition, with P ∈O m×ρ , Q∈O n×ρ , and Σ = diagσ 1 ,...,σ ρ , where ρ = rank(B t ). The polar factor is polar(B t ) = P Q ⊤ ,(9) which sets every nonzero singular value to one while preserving the singular vectors. Equiv- alently, P Q ⊤ is the solution of the spectral-norm linear minimization oracle, P Q ⊤ = arg max ∥X∥ op ≤1 ⟨B t , X⟩,(10) 13 Parviz Mishne Cloninger so the Muon direction equation 7 is precisely the steepest-descent direction with respect to the spectral norm. This is the sense in which Muon is “spectrum aware”: it discards the magnitude of the momentum’s singular values and acts only along their directions. Newton–Schulz orthogonalization. Computing the SVD at every step is expensive, so Muon approximates the polar factor with a fixed number of matrix-multiplication-only Newton–Schulz iterations. Starting from the normalized matrix X 0 = B t /∥B t ∥ F , one iterates a fixed odd polynomial X j+1 = a X j + b X j X ⊤ j X j + c X j X ⊤ j 2 X j , j = 0, 1,...,J − 1,(11) with coefficients (a,b,c) chosen so that the induced scalar map σ 7→ aσ +bσ 3 +cσ 5 pushes the singular values of X j toward 1. The quintic coefficients (a,b,c) = (3.4445,−4.7750, 2.0315) proposed by Jordan (2024) reach a usable approximation in roughly J = 5 iterations; sub- sequent work designs improved polynomials for this orthogonalization (Amsel et al., 2025; Grishina et al., 2025; Cesista et al., 2025; Boumal and Gonon, 2025). As J → ∞ (with an exact map) the iteration converges to polar(B t ), recovering the idealized update; in practice J is small and O t is only an approximate polar factor, which several recent analyses account for explicitly (Shulgin et al., 2025; Kim and Oh, 2026; Lau et al., 2025). Appendix B. Matrix Factorization Variants This appendix collects two generalizations of the symmetric matrix factorization problem equa- tion 1: the general bilinear (asymmetric) factorization and the matrix completion setting in which only a subset of the entries of the target is observed. General bilinear factorization. The symmetric problem equation 1 is a special case of the general bilinear (asymmetric) factorization, in which a possibly rectangular target M ⋆ ∈ R d 1 ×d 2 of rank r is recovered from the product of two factors, min U∈R d 1 ×k V∈R d 2 ×k f (U, V ) = 1 2 UV ⊤ − M ⋆ 2 F ,(12) with k ≥ r. When M ⋆ is symmetric positive semidefinite and the factors are tied as V = U, equation 12 reduces to equation 1 up to a constant rescaling of the objective. Unlike the symmetric case, the asymmetric parameterization is invariant under the larger group of invertible transformations (U, V ) 7→ (UR, V R −⊤ ) for any R ∈ GL k (R), since UV ⊤ = (UR)(V R −⊤ ) ⊤ . This invariance permits an arbitrary imbalance between the factor norms and is commonly controlled by a balancing regularizer 1 2 ∥U∥ 2 F +∥V∥ 2 F or by enforcing U ⊤ U = V ⊤ V along the trajectory. Matrix completion. In many applications only a subset of the entries of M ⋆ is observed. Let [n] : = 1,...,n and let Ω ⊆ [d 1 ]× [d 2 ] denote the set of observed indices. Define the sampling operator P Ω : R d 1 ×d 2 → R d 1 ×d 2 by P Ω (X) ij = ( X ij , (i,j)∈ Ω, 0,otherwise, (13) 14 Muon for Matrix Factorization AdamW Subspace AdamW | Step 500AdamW | Step 1000AdamW | Step 1500AdamW | Step 2000AdamW | Step 2500AdamW | Step 3000 Muon Subspace Muon | Step 500Muon | Step 1000Muon | Step 1500Muon | Step 2000Muon | Step 2500Muon | Step 3000 0.0 0.2 0.4 0.6 0.8 1.0 Abs Dot Product (Alignment) True Eigenvectors Alignment Dynamics Comparison: AdamW (Top) vs Muon (Bottom) Figure 6: Spectral subspace recovery: Muon vs. AdamW. Absolute alignment M ij = |⟨u i ,e j ⟩| (Eq. 16) between learned singular vectors u i and true eigenvectors e j of K, every 500 steps (AdamW top, Muon bottom). A bright, correctly-ordered diagonal indicates faithful recovery; off-diagonal mass reflects mixing. AdamW sharpens its diagonal steadily, while Muon stays near-random for ∼1500 steps, then snaps into a more diffuse alignment. which retains the observed entries and zeros out the rest. The factored matrix completion problem is min U∈R d 1 ×k V∈R d 2 ×k 1 2 P Ω UV ⊤ − M ⋆ 2 F .(14) Taking Ω = [d 1 ]× [d 2 ] recovers the fully observed problem equation 12, while the symmetric variant tied to equation 1 additionally imposes V = U. Exact recovery in this setting requires the ground-truth factors to be incoherent and the sample size |Ω| to be sufficiently large relative to the degrees of freedom r(d 1 + d 2 − r). Under the Bernoulli model in which each entry is observed independently with probability p, the rescaled operator p −1 P Ω is an unbiased estimator of the identity, i.e. E p −1 P Ω (X) = X. Appendix C. Related Work Spectrum-aware optimization and Muon. Much of the theoretical analysis of Muon builds on a perspective that interprets modern optimizers such as Adam and Shampoo as instances of steepest descent under non-Euclidean or norm-constrained geometries (Bernstein and Newhouse, 2024b). Within this view, Muon approximates a spectral-norm steepest-descent direction via polar factorization, connecting it to a broader class of methods that leverage matrix structure in gradients (Pethick et al., 2025a; Li and Hong, 2025; Shen et al., 2025; Chen et al., 2025). Spectral transformations and orthogonalization in optimization predate Muon, appearing in earlier work on stochastic and preconditioned methods (Carlson et al., 2015a,b,c; Tuddenham et al., 2022). A complementary line of work asks more directly when such spectral updates are beneficial in deep learning (Davis and Drusvyatskiy, 2025; Su, 2025). 15 Parviz Mishne Cloninger 1e-044e-042e-031e-028e-025e-01 10 −18 10 −15 10 −12 10 −9 10 −6 10 −3 10 0 κ=1 1e-044e-042e-031e-028e-025e-01 10 −16 10 −13 10 −10 10 −7 10 −4 10 −1 κ=5 1e-044e-042e-031e-028e-025e-01 10 −15 10 −11 10 −7 10 −3 10 1 10 5 κ=25 1e-044e-042e-031e-028e-025e-01 10 −11 10 −8 10 −5 10 −2 10 1 10 4 κ=125 1e-044e-042e-031e-028e-025e-01 10 −3 10 −1 10 1 10 3 10 5 κ=625 MuonAdamWGDSignGD Figure 7: Matrix completion, conditioning sweep. Each panel plots the final re- construction loss (y-axis, log scale; geometric mean over 3 seeds with a shaded ±one-log- standard-deviation band) against the learning rate (x-axis, log scale), for condition number κ. Muon and AdamW are indistinguishable in the well-conditioned regime; all methods degrade and converge toward one another as κ grows. 1e-044e-042e-037e-033e-021e-015e-01 10 −16 10 −13 10 −10 10 −7 10 −4 10 −1 Search Rank = 4 (MATCHED) 1e-044e-042e-037e-033e-021e-015e-01 10 −16 10 −13 10 −10 10 −7 10 −4 10 −1 Search Rank = 5 (OVER-PARAM.) 1e-044e-042e-037e-033e-021e-015e-01 10 −16 10 −13 10 −10 10 −7 10 −4 10 −1 Search Rank = 100 (OVER-PARAM.) MuonAdamWGDSignGD Figure 8: Matrix completion, search-rank sweep (true rank 4, κ=5). Over- parameterization (rank 100) widens the band of effective learning rates and lets every method except GD reach machine precision. Implicit bias and generalization. A second line of work studies the implicit bias induced by Muon. Fan et al. (2025) show that in linear classification, idealized Muon converges to a solution that maximizes margin with respect to the spectral norm, contrasting with the Euclidean and coordinate-wise biases of SGD and Adam. Complementary empirical studies suggest that spectrum-aware updates can improve generalization, particularly in imbalanced or long-tailed settings, by promoting more uniform learning across principal components rather than focusing on dominant directions (Vasudeva et al., 2025; Wang et al., 2025). Related analyses indicate that Muon yields more isotropic singular-value spectra than Adam and may accelerate phenomena such as grokking in long-horizon training (Zhang et al., 2025; Wang et al., 2025; Vasudeva et al., 2025; Tveit et al., 2025). Connections have also 16 Muon for Matrix Factorization 10 4 10 3 10 2 10 1 10 0 10 1 10 5 10 4 10 3 10 2 10 1 10 0 final loss k = 5 (under-param.) 10 4 10 3 10 2 10 1 10 0 10 1 10 5 10 4 10 3 10 2 10 1 10 0 k = 15 (under-param.) 10 4 10 3 10 2 10 1 10 0 10 1 10 6 10 5 10 4 10 3 10 2 10 1 10 0 k = 30 (exact) 10 4 10 3 10 2 10 1 10 0 10 1 10 7 10 6 10 5 10 4 10 3 10 2 10 1 10 0 k = 60 (over-param.) learning rate GDAdamWSignGDMuon Figure 9: Tensor-Train Noisy regime (true rank r ⋆ = 30, additive observation noise). Final loss versus learning rate for each optimizer, across search ranks r ∈ 5, 15, 30, 60 spanning under-, exactly- (r = r ⋆ , bold panel), and over-parameterized settings. Solid lines are the median over 3 seeds; shaded bands the min–max; the loss floors at the noise level rather than at zero. been drawn between Muon and second-order or preconditioned methods, with several works interpreting its updates as approximations to Shampoo (Gupta et al., 2018; Jordan, 2024; Shah et al., 2025a). Alternative derivations and closely related formulations, some predating Muon, have appeared from different theoretical viewpoints (Pethick et al., 2025b; Carlson et al., 2015c; Lau et al., 2025; Bernstein and Newhouse, 2024a,b; An et al., 2025). Preconditioning for matrix factorization. A separate body of work shows that pre- conditioning can dramatically accelerate optimization in matrix factorization. In particular, ScaledGD and its variants achieve linear convergence rates independent of the condition num- ber under appropriate initialization, in both exactly parameterized and over-parameterized regimes (Tong et al., 2021a,b; Zhang et al., 2021, 2023), and these guarantees extend to small random initialization (Xu et al., 2023). Closest to our motivation, recent work analyzes the preconditioning effect of Muon’s spectral orthogonalization directly (Ma et al., 2026). Summary. Taken together, these results show that spectrum-aware and preconditioned methods can offer strong theoretical and empirical benefits, but that their behavior depends heavily on problem structure and initialization. This motivates a closer examination of Muon in controlled settings such as matrix factorization, where powerful alternatives already enjoy strong guarantees and where the advantage of Muon over adaptive methods like AdamW is not immediately evident. Appendix D. Detailed experimental results This appendix expands the summary claims of Section 4 with the specific numerical com- parisons. All values refer to Table 1. Horizontal basin shifts. Evaluating all methods at a common learning rate necessarily places at least one of them far from its own optimum. This is visible as a horizontal shift between each method’s basin in every panel: for instance, GD’s basin lies near the high end of the grid while Muon’s and AdamW’s lie one to three decades lower. 17 Parviz Mishne Cloninger 10 −4 10 −3 10 −2 10 −1 10 0 10 −1 10 0 10 1 10 2 10 3 MuonAdamWGDSignGD Figure 10: Gaussian-kernel NMF at a principled bottleneck. Non-negative factoriza- tion of a fixed Gaussian RBF kernel K ∈ R 100×100 at search rank r=24 (the smallest rank capturing 95% of K’s spectral energy). Tuned final loss vs. learning rate (geometric mean over 3 seeds, ±log-std band; patience-based decay schedule). On this smooth, well-behaved landscape Muon’s orthogonalized updates give no advantage: AdamW reaches the same global floor as Muon. Default-vs-tuned reordering. The ranking induced by a fixed default learning rate (≈ 10 −3 ) differs from the tuned ranking in nearly all nineteen settings, and methods that look worst at the default frequently become best after tuning. On low-rank factorization GD is among the worst at the default rate yet attains the lowest loss of all methods once tuned (Figure 1); symmetrically, on both NMF variants Muon is last at the default rate but first once tuned (Figures 2). Effect of ill-conditioning. The effect of ill-conditioning is most dramatic for the non- adaptive methods on factorization: GD, the best-tuned method in the well-conditioned regime (1.6× 10 −13 at κ=1), collapses to 1.3× 10 −1 at κ=625, where AdamW becomes best (2.6× 10 −7 ), Muon second (9.4× 10 −5 ), and SignGD diverges. Matrix completion shows the same upward drift of the loss floor: at κ=625 all four methods are confined to O(10 −2 ) and become nearly indistinguishable. Problem-dependence of the tuned winner. Under per-optimizer tuning there is no universally dominant method. On plain low-rank factorization, AdamW and GD drive the loss to 10 −13 –10 −9 across the whole conditioning range, while the Muon configuration plateaus several orders higher (10 −8 –10 −5 ). On matrix completion, Muon and AdamW are both reaching 10 −16 –10 −13 in the well-conditioned regime, and over-parameterization (search rank 100) lets all methods except GD reach machine precision (Figure 8). On NMF, the picture reverses: Muon is the clear winner under both spectra, reaching 10 −9 –10 −6 while AdamW stalls near 10 −3 (with the single exception of the uniform-spectrum rank-50 case, where AdamW matches Muon at ∼ 4× 10 −9 ) and GD/SignGD fail to fit. Whichever opti- 18 Muon for Matrix Factorization mizer a single-configuration study would crown thus depends entirely on the problem and the learning rate chosen. Appendix E. Dynamics of spectral subspace recovery The aggregate losses above tell us whether a method fits the target but not how it does so. To probe the mechanism, we factorize a fixed Gaussian-kernel target K ∈ R 100×100 , symmetric positive semidefinite, with eigendecomposition K = QΛQ ⊤ and eigenvectors ordered by descending eigenvalue, as ˆ K = UV ⊤ at full search rank k = 100, training AdamW and Muon each at its own tuned learning rate. Every 500 iterations we take the left singular vectors ˆu i of the current reconstruction UV ⊤ and form the alignment matrix M ij = ⟨ˆu i , q j ⟩ ,(15) the absolute overlap between the i-th learned singular direction and the j-th true eigenvec- tor. A faithful, correctly ordered recovery of the eigenbasis yields M → I (a sharp diagonal), whereas off-diagonal or diffuse mass indicates that the learned subspace mixes or reorders eigendirections. Figure 6 tracks M over training. AdamW sharpens toward a clean diagonal quickly and monotonically, recovering the eigenvectors faithfully and in order, while Muon’s alignment remains comparatively diffuse, with persistent off-diagonal mass and slower di- agonalization. This provides a mechanistic explanation for the loss-level gap of Sections 4– 4.1: on these structured and ill-conditioned spectra, AdamW’s per-coordinate adaptation locks onto the dominant eigendirections in the right order, whereas Muon’s orthogonalized updates distribute capacity more evenly across directions and recover the eigenbasis less cleanly. The heatmaps shown are for a representative tuned run; the qualitative pattern is consistent across seeds. Appendix F. Spectral profile definitions The experiments of Section 4.1 use seven spectral profiles, defined by the diagonal entries of S. All share the endpoints (s min ,s max ) = (10 −3 , 10) and hence the condition number κ = s max /s min = 10 4 ; only the interior shape differs. • Uniform: linear spacing s i = s min + i−1 n−1 (s max −s min ), a dense, even spread of curva- ture. • Log-uniform: geometric spacing s i = s min · κ i−1 n−1 , typical of ill-conditioned systems whose scales vary by orders of magnitude. • Gaussian: eigenvalues drawn from a truncated normal N (μ,σ 2 ) and rescaled to [s min ,s max ], a bulk concentrated around a mean curvature. • Spiked: a single top eigenvalue at s max with the remaining n− 1 clustered near s min , a dominant rank-one subspace. • Bimodal: eigenvalues split into two clusters near the endpoints s min and s max . • Polynomial decay: s i ∝ i −α , a rapid decay in the importance of higher-order direc- tions. • Exponential decay: s i ∝ e −βi , an aggressive concentration of energy in the top eigenvalues. 19 Parviz Mishne Cloninger Appendix G. Tensor-train regimes and parameterization sweep This appendix gives the full specification of the regimes and parameterization sweep sum- marized in Section 4.3. Regimes. We study equation 5 in two regimes. In the noiseless regime the target has true rank r ⋆ = 4 and is recovered exactly, so the attainable loss is limited only by the optimizer’s convergence; in the noisy regime the target has true rank r ⋆ = 30 and is corrupted by additive observation noise, so the loss is floored at the statistical noise level rather than driven to zero. The two regimes therefore probe complementary phenomena: optimization geometry in the well-specified case and robustness to a nonzero residual in the misspecified case. Parameterization sweep. Within each regime we sweep the search rank, equivalently the T-rank r of the factor U, across under-, exactly-, and over-parameterized values: r ∈2, 4, 8, 20 for the noiseless target (true rank r ⋆ = 4) and r ∈5, 15, 30, 60 for the noisy target (true rank r ⋆ = 30), with r = r ⋆ the exactly parameterized boundary case. Under- parameterization (r < r ⋆ ) caps the attainable loss because U ⊤ cannot represent M ⋆ , while over-parameterization (r > r ⋆ ) enlarges the factor space and introduces additional flat directions in the landscape. Appendix H. Relation to deep learning models The factorized objective equation 5 is not only a tensor-train problem in its own right; it is the minimal instance of the product parameterizations that pervade deep learning, which is what makes the optimizer behavior we study here relevant beyond low-rank recovery. Factorization as a shallow linear network. The map U 7→ U ⊤ (or (U, V ) 7→ UV ⊤ in the asymmetric case of Appendix B) is exactly a two-layer linear network with no intervening nonlinearity: the factors are the layer weights, and the product is the end-to-end map. Consequently the loss is nonconvex in the factors despite being convex in the product, and the search rank, equivalently the T-rank, plays the role of the network width. The over-parameterized regime r > r ⋆ is precisely the width-overparameterized regime of modern networks, and the flat directions it introduces in the landscape are the source of the implicit bias of gradient methods studied for matrix and deep factorizations (Gunasekar et al., 2017; Arora et al., 2019). This is why over-parameterized factorization is a standard proxy for the optimization of deep models: it isolates the effect of redundant parameters on the trajectory while remaining analytically tractable. Depth as a longer tensor-train chain. Increasing the order of the tensor-train chain, contracting L cores G 1 ,...,G L rather than two, is the algebraic analogue of increasing the depth of a linear network, whose end-to-end map factorizes as W = W L W L−1 · W 1 . The matrix case equation 4 (L = 2) is the shallowest such chain, and depth introduces exactly the ill-conditioning and balancing phenomena that motivate our conditioning sweep: the product of many factors amplifies spectral imbalance, and the overparameterized invariance group acts on each internal bond. Depth has been shown to act as an implicit preconditioner that accelerates gradient descent on these product objectives (Arora et al., 2018), so the 20 Muon for Matrix Factorization sensitivity of each optimizer to conditioning at L = 2 is the base case of a phenomenon that sharpens with depth. Tensor-train structure inside trained networks. Beyond serving as a proxy, tensor- train factorizations appear directly inside deep models. Reshaping a dense weight matrix into a high-order tensor and representing it in T format compresses fully connected and embedding layers by orders of magnitude while preserving accuracy (Novikov et al., 2015), and specific architectures correspond to specific tensor decompositions: recurrent networks realize the tensor-train / matrix-product structure (Khrulkov et al., 2017), while convolu- tional arithmetic circuits realize the hierarchical Tucker decomposition (Cohen et al., 2016). In all of these the rank of the decomposition controls expressivity exactly as the T-rank r controls the capacity of equation 5, so the interaction between search rank, conditioning, and optimizer that we characterize on the factorized problem speaks directly to how these layers are trained. Appendix I. Optimizer Comparison Across Tensor-Train Depth We benchmark four optimizers—Muon, AdamW, gradient descent (GD), and SignGD—on non-negative tensor-train (T) factorization. To systematically vary problem conditioning, we use the tensor-train depth (equivalently, the tensor order or number of T cores) as the primary experimental factor. Each T core is represented via a matrix unfolding, a necessary implementation detail because Muon’s Newton–Schulz orthogonalization operates only on matrix-valued gradients; retaining a core in its native third-order form would effectively reduce Muon to momentum SGD on that factor. We evaluate two target regimes: a clean, low-rank, noise-free tensor and a noisy, higher-rank tensor. The study spans depths (2)–(6), an overparameterized factorization rank, a five-point learning-rate sweep, and three random seeds, yielding 600 total runs. Performance is reported as the best-tuned final loss for each optimizer, defined as the lowest mean squared error (MSE) achieved across the learning-rate grid and averaged over seeds. The results differ markedly between the two regimes. For the clean target, optimizer rankings change substantially with depth. AdamW achieves the lowest losses at shallow depths, while Muon performs comparatively poorly. However, beginning at depth (4), the performance of AdamW, GD, and SignGD deteriorates as the factorization deepens, whereas Muon maintains consistently low error. By depth (6), Muon outperforms the competing methods by approximately one to two orders of magnitude (Figure 11, left). In contrast, performance on the noisy target remains relatively flat across depths, with all optimizers achieving similar losses (Figure 11, right). The added noise introduces an irreducible error floor that limits the optimization gains obtainable from improved conditioning, thereby diminishing Muon’s advantage. These findings suggest that Muon’s benefits arise primarily from its ability to mitigate optimization difficulties associated with ill-conditioned deep factorizations, rather than from a general robustness advantage. At the same time, two limitations should be noted. First, increasing tensor-train depth changes the underlying optimization problem itself, so the ob- served trends reflect changes in relative optimizer performance across a family of increasingly difficult tasks rather than within a fixed loss landscape. Second, the reported results corre- 21 Parviz Mishne Cloninger 23456 T depth (number of cores) 10 8 10 7 10 6 10 5 10 4 10 3 best final MSE (min over LR, mean±range over seeds) clean target Muon AdamW GD SignGD 23456 T depth (number of cores) 10 11 10 9 10 7 10 5 10 3 noisy target Muon AdamW GD SignGD Figure 11: Best-tuned reconstruction loss versus tensor-train depth. Each curve shows the final MSE of one optimizer at its best learning rate (minimum over the grid) for a given depth, averaged over three seeds; shaded bands span the per-seed min–max. Left: clean, low-rank target. Right: noisy, higher-rank target. Search bond rank is fixed to the over-parameterized regime (R=12 clean, R=30 noisy). Lower is better; the y-axis is logarithmic. spond only to the overparameterized setting considered here and may not fully characterize behavior at other scales or rank regimes. Appendix J. Dynamics of spectral subspace recovery The aggregate losses above tell us whether a method fits the target but not how it does so. To probe the mechanism, we factorize a fixed Gaussian-kernel target K ∈ R 100×100 , symmetric positive semidefinite, with eigendecomposition K = QΛQ ⊤ and eigenvectors ordered by descending eigenvalue, as ˆ K = UV ⊤ at full search rank k = 100, training AdamW and Muon each at its own tuned learning rate. Every 500 iterations we take the left singular vectors ˆu i of the current reconstruction UV ⊤ and form the alignment matrix M ij = ⟨ˆu i , q j ⟩ ,(16) the absolute overlap between the i-th learned singular direction and the j-th true eigenvector. A faithful, correctly ordered recovery of the eigenbasis yields M → I (a sharp diagonal), whereas off-diagonal or diffuse mass indicates that the learned subspace mixes or reorders eigendirections. Figure 6 tracks M over training. AdamW sharpens toward a clean diagonal quickly and monotonically, recovering the eigenvectors faithfully and in order, while Muon’s alignment remains comparatively diffuse, with persistent off-diagonal mass and slower diagonalization. This provides a mechanistic explanation for the loss-level gap of Sections 4–4.1: on these structured and ill-conditioned spectra, AdamW’s per-coordinate adaptation locks onto the dominant eigendirections in the right order, whereas Muon’s orthogonalized updates dis- tribute capacity more evenly across directions and recover the eigenbasis less cleanly. The 22 Muon for Matrix Factorization heatmaps shown are for a representative tuned run; the qualitative pattern is consistent across seeds. Appendix K. Softplus, Its Taylor Surrogate, and Nonlinear Matrix Factorization Let X ∈ R N×d collect the N inputs. We fit the Gaussian kernel K ∈ R N×N , K ij = exp(−∥x i − x j ∥ 2 /2σ 2 ), with the model ˆ K = φ(XW )V ⊤ , where W ∈ R d×R , V ∈ R N×R , the search rank R is the hidden width, and φ acts entrywise. This is a nonlinear matrix factorization: for φ = id it collapses to the bilinear low-rank model ˆ K = X(WV ⊤ ). Softplus and its expansion. The softplus φ(z) = log(1 + e z ) has derivative φ ′ = σ (the logistic sigmoid). Since σ − 1 2 is odd, the Maclaurin series of φ keeps only the constant, linear, and even-order terms, φ(z) = log 2 + 1 2 z + 1 8 z 2 − 1 192 z 4 +O(z 6 ).(17) The surrogate used in the experiments, ̃ φ(z) = log 2 + 1 2 z + 1 8 z 2 , is the truncation after the quadratic term (the cubic coefficient vanishes), and is accurate when the pre-activations XW are small, as enforced by the input scale. The quadratic surrogate is an explicit feature factorization. With A = ̃ φ(XW ), expanding term by term gives ˆ K = AV ⊤ = log 2 1 N×R V ⊤ | z constant (rank≤1) + 1 2 (XW )V ⊤ |z rank≤min(R,d) + 1 8 (XW ) ⊙2 V ⊤ |z degree-2 lift ,(18) where ⊙ is the Hadamard product. The linear term is exactly the classical rank-R bilinear factorization, whose rank is further capped by rank(X) ≤ d. The Hadamard-square term injects degree-2 monomial features: each column of XW lies in the≤ d-dimensional column space col(X), so the columns of (XW ) ⊙2 lie in its symmetric square Sym 2 (col(X)), of dimension ≤ d+1 2 . Hence the model factorizes K through the fixed feature set 1 ∪ col(X)∪ Sym 2 (col(X)), and rank( ˆ K) ≤ min R, d+2 2 .(19) The quadratic model therefore cannot exceed rank d+2 2 (= 21 for d = 5) no matter how large the search rank R is. Consequence for the comparison. Equation equation 19 predicts a sharp regime split, which Fig. 12 confirms. When R ≤ d+2 2 (here R = 10), the search rank is the binding constraint and the truncation is invisible: exact softplus and its surrogate behave almost identically (panels (a) vs (b)). When R > d+2 2 (here R = 100), the degree-2 cap binds for the surrogate but not for exact softplus, so the exact activation attains a strictly lower error (panel (c) below (d)). In both regimes the activation is a mild perturbation of the same nonlinear-factorization family, while the search rank R governs how much of the ker- nel’s polynomial spectrum the model can represent, and, empirically, is what separates the optimizers. 23 Parviz Mishne Cloninger 10 4 10 3 10 2 10 1 10 3 10 2 10 1 10 0 10 1 10 2 10 3 (a) exact softplus · rank 10 Muon AdamW GD SignGD 10 4 10 3 10 2 10 1 (b) 2nd-order Taylor · rank 10 10 4 10 3 10 2 10 1 (c) exact softplus · rank 100 10 4 10 3 10 2 10 1 (d) 2nd-order Taylor · rank 100 Learning rate (log scale) Final MSE (log scale) Figure 12: Learning-rate stability across activation and capacity. Final MSE (log) vs. learning rate (log) for four optimizers fitting the Gaussian kernel as ˆ K = φ(XW )V ⊤ ; mean over three seeds, bands show min–max, divergent runs capped at 10 5 . The activation φ is exact softplus or its second-order Taylor surrogate, and R is the search rank: (a) softplus, R=10; (b) Taylor, R=10; (c) softplus, R=100; (d) Taylor, R=100. At R=10 the rank bottleneck binds and the two activations match (a)≈(b); at R=100 the degree-2 surrogate is rank-capped (Eq. 19) while exact softplus is not, so (c) reaches a lower error than (d). Muon attains the lowest loss over the widest stable range; SignGD diverges at large learning rates. Appendix L. Analytical Solutions of the Matrix Factorization for AdamW and Muon L.1. Problem Setup We consider the Matrix Factorization objective function: L(U,V ) = 1 2 ∥UV T − R∥ 2 F ,(20) where U,V ∈ R N×r . At iteration t, we seek the optimal update step ∆U for the factor U (the derivation for V is symmetric). Let G t =∇ U L = (U t V T t −R)V t be the gradient at step t. Since the global optimization is intractable, iterative optimizers solve a local proxy problem at each step. We derive the analytical solution for these proxy problems. Through- out, v t (lowercase) denotes the per-coordinate second-moment estimate, with entries v t,ij ; this is distinct from the factor V t . L.2. The AdamW Analytical Solution Proposition 1 Under the decoupled–weight-decay approximation of Loshchilov & Hutter (2017), the AdamW update is the analytical solution to minimizing the linearized loss subject to an adaptive Mahalanobis-distance trust region, with weight decay applied as a separate additive step. 24 Muon for Matrix Factorization The Optimization Problem. We first consider the linearized loss penalized by the local curvature estimated by the diagonal second-moment estimate v t : ∆U † = arg min ∆U∈R N×r ⟨G t , ∆U⟩ | z Linear Descent + 1 2η X i,j √ v t,ij (∆U ij ) 2 | z Adaptive Trust Region .(21) Derivation of the trust-region step. The objective is separable across coordinates. Taking the derivative with respect to a single entry ∆U ij and setting it to zero, G t,ij + 1 η √ v t,ij ∆U ij = 0=⇒∆U † ij =−η G t,ij √ v t,ij .(22) Decoupled weight decay. AdamW then applies weight decay as a separate step that is not passed through the preconditioner: ∆U ∗ ij =−η G t,ij √ v t,ij + λU t,ij . (23) Remark (why decoupling matters). Had we instead folded the penalty λ 2 ∥U t + ∆U∥ 2 F directly into the proxy, the exact stationary point would be 1 η √ v t,ij + λ ∆U ij =− G t,ij + λU t,ij =⇒ ∆U ij =−η G t,ij + λU t,ij √ v t,ij + ηλ .(24) This couples the decay to the preconditioner (dividing the decay term by √ v t,ij ) and is precisely Adam with L 2 regularization, not AdamW. AdamW is recovered by (i) neglecting ηλ in the denominator and (i) decoupling the decay so that λU t,ij is not rescaled by the curvature. Hence the boxed expression is an approximation, not the exact minimizer of the coupled proxy. Conclusion. AdamW rescales every coordinate independently based on the diagonal cur- vature √ v t,ij , and adds an unpreconditioned weight-decay term. L.3. The Muon Analytical Solution Proposition 2 The Muon update is the exact analytical solution to the Orthogonal Pro- crustes Problem. It finds the orthonormal-column update direction that aligns maximally with the negative gradient; this direction is the polar factor of −G t . The Optimization Problem. We seek the matrix O ∈ R N×r closest to the negative gradient −G t , constrained to have orthonormal columns: O ∗ = arg min O :O T O=I r ∥(−G t )− O∥ 2 F .(25) The update is then ∆U = η O ∗ . 25 Parviz Mishne Cloninger L.3.1. Step-by-Step Derivation Step 1: Expand the objective. ∥− G t − O∥ 2 F = Tr (G t + O) T (G t + O) (26) = Tr(G T t G t ) + Tr(O T O) + 2 Tr(G T t O).(27) The term Tr(G T t G t ) is constant, and Tr(O T O) = Tr(I r ) = r is constant. The cross term carries a plus sign, so minimizing the norm is equivalent to minimizing Tr(G T t O) , equiv- alently maximizing the alignment with the negative gradient, Tr (−G t ) T O : min O :O T O=I r Tr(G T t O).(28) Step 2: Singular Value Decomposition. Let the (thin) SVD of the gradient be G t = P ΣQ T , where P ∈ R N×r has orthonormal columns (P T P = I r ), Σ∈ R r×r is diagonal with singular values σ i ≥ 0, and Q∈ R r×r is orthogonal. Then Tr(G T t O) = Tr QΣP T O = Tr Σ (P T OQ) ,(29) using the cyclic property of the trace. Step 3: Bound the matrix Z. Define Z = P T OQ∈ R r×r . Each of P, O has orthonor- mal columns and Q is orthogonal, so each is a (semi-)isometry with spectral norm 1. Hence σ max (Z) ≤ ∥P T ∥ 2 ∥O∥ 2 ∥Q∥ 2 = 1. (Note Z need not be orthogonal when N > r, since O T ̸= I N ; we only require the norm bound.) In particular, every diagonal entry satisfies |Z i |≤ σ max (Z)≤ 1. The objective becomes Tr(ΣZ) = r X i=1 σ i Z i .(30) Step 4: Minimize. Since σ i ≥ 0 and |Z i | ≤ 1, the sum P i σ i Z i is minimized when Z i = −1 for all i, i.e. Z = −I r (attained by the admissible choice O = −PQ T ). Solving back, P T OQ =−I r (31) O ∗ =−P Q T .(32) O ∗ =−PQ T ,∆U = η O ∗ =−η PQ T . (33) Equivalently, O ∗ is the polar factor of the negative gradient −G t , and the update ∆U = −η PQ T is a genuine descent step (it removes the singular-value magnitudes of G t , keeping only its PQ T “direction”). L.3.2. Connection to Newton–Schulz The matrix PQ T is the polar factor of G t . Computing the SVD at every step is expensive, so Muon computes PQ T via the Newton–Schulz iteration X k+1 = 1 2 X k 3I − X T k X k , X 0 = G t ∥G t ∥ 2 ,(34) which converges quadratically to PQ T . The descent update is then ∆U = −η X ∞ = −η PQ T , consistent with the boxed solution above. 26 Muon for Matrix Factorization L.4. Implications for Matrix Factorization This derivation highlights the fundamental difference in update scaling. Both updates act on matrices in R N×r . 1. AdamW scaling (dimension dependent). Assuming active gradients (G ij / √ v ij ≈ ±1) and negligible weight decay, the squared norm of the update scales with the number of entries N × r: ∥∆W Adam ∥ 2 F ≈ X i,j η 2 = η 2 (N · r).(35) 2. Muon scaling (rank dependent). Since O ∗ = −PQ T has orthonormal columns, (O ∗ ) T O ∗ = QP T PQ T = I r , so the sign is immaterial to the norm: ∥∆W Muon ∥ 2 F = η 2 Tr (O ∗ ) T O ∗ = η 2 Tr(I r ) = η 2 r.(36) Conclusion. ∥∆W Adam ∥ F ∥∆W Muon ∥ F ≈ √ N.(37) In our experiments with N = 1000, AdamW naturally takes steps ≈ 31.6 times larger than Muon for the same learning rate η. This explains the necessity of scaling η Muon ≈ √ N · η AdamW to achieve comparable convergence rates. 27