Paper deep dive
Scaling Muon for Diffusion Transformers
Chenghao Li, Xiao Han, Xinxin Huang, Wei Liu, Boyang Li, Bing Xiao, Heran Zhang, Juanma Perez Rua, Ke Xu, Kangning Liu, Linjun Kuang, Na Li, Tan Wang, Tian Xie, Wei Peng, Yang Pei, Yifan Xu, Yuanhao Zhai, Yuwei Lin, Zhe Wang, Zihao He, Daniel Li, Junbiao Tang, Ziyang Jiang, Dake Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 8/24/2026, 5:24:23 AM
Summary
This paper investigates the scaling behavior of the matrix-aware optimizer Muon for Diffusion Transformers (DiTs) ranging from 1.3B to 15B parameters. While Muon demonstrates superior optimization and generative quality compared to AdamW, its high computational and communication overhead from full Newton-Schulz iterations (NS5) and momentum materialization limits end-to-end efficiency. The authors propose 'Periodic Row-wise Muon,' which alternates between full NS5 spectral updates and low-cost RowNorm updates. This method, combined with a distributed implementation using sharded momentum and bucketed all-gather, significantly reduces optimizer time, communication volume, and training time while preserving generative quality advantages.
Entities (8)
Relation Signals (6)
Muon → appliedto → Diffusion Transformers
confidence 100% · We first establish Muon's scaling behavior on DiTs from 1.3B to 15B parameters
Periodic Row-wise Muon → improvesefficiencyof → Muon
confidence 95% · Periodic Row-wise Muon reduces optimizer time by 46.9–54.3% and logical communication volume by 66.7% compared to vanilla Muon.
Muon → outperforms → AdamW
confidence 95% · Across all scales, Muon improves the best observed generative quality over AdamW by 12.9–19.1%.
Periodic Row-wise Muon → usescomponent → RowNorm
confidence 95% · applies a low compute and communication cost row-wise constrained update based on the current momentum... RowNorm operates directly on sharded momentum
Periodic Row-wise Muon → usescomponent → Newton-Schulz 5
confidence 95% · performs a full NS5 spectral update once every K steps
Newton-Schulz 5 → causesoverhead → Muon
confidence 90% · the 5-step Newton--Schulz iteration (NS5) ... introduces substantial computation and communication overhead
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The matrix-aware optimizer Muon improves large model training by balancing updates across singular directions, yet its scaling behavior and end-to-end efficiency on large Diffusion Transformers (DiTs) remain unclear. We first establish Muon's scaling behavior on DiTs from 1.3B to 15B parameters, showing that its optimization and generative quality advantages over AdamW persist across model scales. However, at scale, the 5-step Newton--Schulz iteration (NS5) performed at every optimization step, together with full-momentum materialization, introduces substantial computation and communication overhead that can offset Muon's step-efficiency advantage. We introduce \emph{Periodic Row-wise Muon}, which performs a full NS5 spectral update once every \(K\) steps and applies a low compute and communication cost row-wise constrained update based on the current momentum at the remaining steps. We further co-design a distributed implementation that operates directly on sharded momentum during non-refresh steps and accelerates spectral refreshes through bucketed all-gather and communication--computation overlap. Across all scales, Muon improves the best observed generative quality over AdamW by 12.9--19.1\%. Compared with vanilla Muon, Periodic Row-wise Muon remains within 0.5\% in best generative quality on the 1.3B--4B models and improves it by 4.5\% at 9B. It reduces optimizer time by 46.9--54.3\%, end-to-end step time by 15.7--24.3\%, and logical communication volume by 66.7\%, while reaching its respective best generative quality with 33.7--64.8\% less active training time. These results show that Periodic Row-wise Muon preserves Muon's generative quality advantage while translating it into end-to-end training efficiency for large DiTs.
Tags
Links
- Source: https://arxiv.org/abs/2608.20818v1
- Canonical: https://arxiv.org/abs/2608.20818v1
Trouble viewing inline? Open PDF directly →
Full Text
102,289 characters extracted from source content.
Expand or collapse full text
Scaling Muon for Diffusion Transformers Chenghao Li1,2, Xiao Han2, Xinxin Huang2, Wei Liu2, Boyang Li2, Bing Xiao2, Heran Zhang2, Juanma Perez Rua2, Ke Xu2, Kangning Liu2, Linjun Kuang2, Na Li2, Tan Wang2, Tian Xie2, Wei Peng2, Yang Pei2, Yifan Xu2, Yuanhao Zhai2, Yuwei Lin2, Zhe Wang2, Zihao He2, Daniel Li2, Junbiao Tang2, Ziyang Jiang2, Dake Chen2 1University of Southern California 2Meta Thanks: Work done during an internship at Meta. Abstract The matrix-aware optimizer Muon improves large model training by balancing updates across singular directions, yet its scaling behavior and end-to-end efficiency on large Diffusion Transformers (DiTs) remain unclear. We first establish Muon’s scaling behavior on DiTs from 1.3B to 15B parameters, showing that its optimization and generative quality advantages over AdamW persist across model scales. However, at scale, the 5-step Newton–Schulz iteration (NS5) performed at every optimization step, together with full-momentum materialization, introduces substantial computation and communication overhead that can offset Muon’s step-efficiency advantage. We introduce Periodic Row-wise Muon, which performs a full NS5 spectral update once every K steps and applies a low compute and communication cost row-wise constrained update based on the current momentum at the remaining steps. We further co-design a distributed implementation that operates directly on sharded momentum during non-refresh steps and accelerates spectral refreshes through bucketed all-gather and communication–computation overlap. Across all scales, Muon improves the best observed generative quality over AdamW by 12.9–19.1%. Compared with vanilla Muon, Periodic Row-wise Muon remains within 0.5% in best generative quality on the 1.3B–4B models and improves it by 4.5% at 9B. It reduces optimizer time by 46.9–54.3%, end-to-end step time by 15.7–24.3%, and logical communication volume by 66.7%, while reaching its respective best generative quality with 33.7–64.8% less active training time. These results show that Periodic Row-wise Muon preserves Muon’s generative quality advantage while translating it into end-to-end training efficiency for large DiTs. 1 Introduction The continued scaling of Diffusion Transformers (DiTs) (26; 11; 8) has improved model capacity and generative quality, while making training efficiency a central concern. Prior work has characterized predictable improvements with increased training compute (36; 22), but has typically treated the optimizer as fixed. At billion-parameter scale, however, optimizer efficiency can no longer be judged solely by the number of steps required to reach a lower loss. Fewer optimization steps must translate into lower GPU-hours and shorter wall-clock time under distributed training. Evaluating optimizer scalability therefore requires considering both its optimization advantage and its realized compute and communication cost under distributed training. Muon (15) presents a promising alternative for improving large-scale training efficiency. Unlike AdamW (24), which applies coordinate-wise adaptive updates, Muon treats two-dimensional weight tensors as matrices and applies a finite-step Newton–Schulz (NS) transformation to their momentum, producing update directions with global spectral structure. Recent work has shown that, when properly calibrated, Muon can match AdamW with substantially fewer training FLOPs in large language model pretraining and post-training (23). Evidence from smaller diffusion models likewise suggests improved optimization despite a higher per-step compute and communication cost, while showing that loss, generative quality, and runtime may rank optimizers differently (30). It therefore remains unclear whether Muon’s optimization advantage persists as DiTs scale to ten-billion parameters and whether its distributed overhead can be reduced sufficiently to improve end-to-end training efficiency. Answering these questions requires considering optimization and compute and communication costs jointly. Under the sharded execution path studied in this work, Muon’s NS5 transformation introduces additional matrix multiplications, full-momentum communication, synchronization, and temporary materialization (23). These costs motivate reducing the frequency and the exposed overhead of full spectral transformations, while preserving Muon’s optimization advantage. We first establish that Muon’s optimization advantage persists at scale by training DiTs from 1.3B to 15B parameters on the GPIC dataset (6). Across all four scales, Muon achieves lower validation loss and improves the best observed FD-DINO over AdamW by 12.9–19.1%. However, its additional computation and communication make vanilla Muon substantially more expensive per step. These results demonstrate Muon’s algorithmic scalability on large DiTs while exposing a systems bottleneck to realizing its optimization advantage efficiently. To address this bottleneck, we revisit whether global spectral geometry must be imposed at every optimization step and introduce Periodic Row-wise Muon, which performs a full NS5 update once every K steps, and applies a row-wise constrained normalization operator (RowNorm) (20) at the remaining steps. The method alternates between two complementary matrix geometries. Periodic spectral (refresh) steps provide global coupling across rows and singular directions, whereas low compute and communication cost row-wise constrained (non-refresh) steps provide local scale control and maintain stable optimization behavior between consecutive spectral refreshes. We co-design the distributed execution with the periodic update. On non-refresh steps, RowNorm operates directly on sharded momentum and for matrix shapes whose normalization rows span ranks, only their norm statistics are all-reduced instead of the full matrix. On refresh steps, we pipeline bucketed momentum all-gathers for upcoming matrices with NS computation on already available buckets to reduce exposed communication latency. This design removes full-momentum materialization on non-refresh steps and reduces the exposed overhead of the remaining spectral refreshes. Across the 1.3B–4B models, Periodic Row-wise Muon’s best observed FD-DINO remains within 0.5% of vanilla Muon, while outperforming it by approximately 4.5% at 9B and 2.7% at 15B. Relative to vanilla Muon, it reduces optimizer time by 46.9–54.3%, end-to-end step time by 15.7–24.3%, and logical optimizer-communication volume by 66.7% across all scales. Our main contributions are as follows: (1) we characterize Muon on 1.3B–15B DiTs, showing that its advantages over AdamW in validation loss and generative quality persist across scale while identifying NS5 computation and full-momentum communication as its main systems bottlenecks; (2) we introduce Periodic Row-wise Muon, which replaces most NS5 transformations with low compute and communication cost RowNorm updates from the current momentum; and (3) we develop a distributed implementation that operates on sharded momentum during non-refresh steps and pipelines bucketed all-gather with NS5 computation during refresh steps, substantially reducing systems overhead while retaining generative quality comparable to vanilla Muon. 2 Muon for DiTs 2.1 Scaling Muon for DiTs Training Given a clean data sample x, a diffusion timestep a, and noise ϵε, let xax_a denote the noisy input constructed according to the prescribed noise schedule. We train a DiT to predict the corresponding velocity target v⋆v by minimizing ℒ(θ)=x,a,ϵ[‖vθ(xa,a)−v⋆(xa,a)‖22].L(θ)=E_x,a,ε [ v_θ(x_a,a)-v (x_a,a) _2^2 ]. (1) We omit additional conditioning variables, such as text embeddings, for notational simplicity. The AdamW baseline applies AdamW to all trainable parameters. Muon instead applies matrix-aware updates to two-dimensional hidden weight matrices, while biases, normalization parameters, and other non-matrix parameters continue to be updated by AdamW. Consider a weight matrix at training step t denotes Wt∈ℝm×nW_t ^m× n with gradient Gt=∇Wtℒ(θt)G_t= _W_tL( _t). Muon applies a finite spectral transformation consisting of JNS=5J_NS=5 Newton–Schulz iterations Φ5 _5 to its momentum MtM_t, and decoupled weight decay update: t=Φ5(Mt),Wt+1=(1−ηtλ)Wt−ηts(Wt)t,P_t= _5(M_t), W_t+1=(1- _tλ)W_t- _t\,s(W_t)P_t, (2) where ηt _t is learning rate, λ is weight decay coefficient, and s(Wt)s(W_t) is shape-dependent update scale. The finite map Φ5 _5 is inspired by the matrix polar factor and reshapes the singular values of the momentum to produce a globally coupled spectral direction tP_t. Importantly, NS5 is not equivalent to computing an exact SVD polar decomposition, nor does it generally yield an exactly orthogonal matrix after 55 iterations. Our algorithmic definitions, theoretical analysis, and experiments therefore use the implemented finite map Φ5 _5, rather than assuming convergence to the exact polar factor. We characterize Muon’s scalability for large scale DiTs training along two dimensions: algorithmic and systems scalability. Algorithmic scalability answers whether Muon maintains stable optimization behavior as model size increases and whether its advantages over AdamW in validation loss and generative quality persist across model scales. Systems scalability answers whether these algorithmic gains translate into realized efficiency under large-scale distributed training. 2.2 Computation Complexity The most direct computational difference between Muon and AdamW arises from updating two-dimensional matrices. Consider W∈ℝm×nW ^m× n, r=min(m,n)r= (m,n), c=max(m,n)c= (m,n). Because AdamW maintains and applies element-wise statistics, its arithmetic complexity for this matrix is CAdamW(m,n)=Θ(mn)=Θ(rc)C_AdamW(m,n)= (mn)= (rc). In contrast, a finite step Newton–Schulz transformation repeatedly performs matrix multiplications. Even choosing the smaller Gram-matrix orientation yields CNS(m,n)=Θ(JNSr2c)C_NS(m,n)= \! (J_NSr^2c ). Thus, NS5 incurs an additional factor of Θ(JNSr) (J_NSr) in arithmetic per matrix. Ignoring implementation constants and hardware throughput, their arithmetic cost ratio for one matrix is Θ(JNSr) (J_NSr). 2.3 Distributed Cost under Large Scale Distributed Training Local arithmetic alone does not characterize Muon’s cost at scale, which also depends on how its full-matrix spectral transformation is mapped onto sharded optimizer states. In our execution path, each two-dimensional parameter and its momentum are sharded along dimension zero, so each rank stores a subset of rows and the corresponding momentum shard. AdamW is coordinate-separable because each rank can update its local parameters using only its local first- and second-moment shards, without materializing the full matrix. Muon is not shard-separable under this execution path because Newton–Schulz iterations couple rows and singular directions through Gram matrices and matrix multiplications. In general, a rank cannot recover its portion of the spectral update from its local momentum shard alone. Our vanilla implementation therefore all-gathers the momentum shards to every rank, executes NS5 on the full matrix at every rank, retains the output slice corresponding to the local parameter shard, and then discards the remaining output and releases the temporary full-matrix buffers (23). Muon’s systems cost consequently includes the GEMMs and associated matrix operations of replicated NS5 execution, together with full-momentum communication and temporary full-matrix materialization. Suppose the sharding group contains p ranks and each momentum element occupies b bytes. Ignoring padding and protocol overhead, all-gathering an m×nm× n momentum matrix requires each rank to logically receive approximately VAG(m,n)≈bmnp−1pbytes.V_AG(m,n)≈ bmn p-1p\ bytes. (3) This quantity excludes parameter communication already required by forward and backward propagation. Logical communication volume alone, however, does not determine communication time, which also depends on the collective count, bucket sizes, launch latency, the fraction of inter-node traffic, network topology and contention, and the extent of communication–computation overlap. In particular, many small buckets may incur substantial launch latency without changing the total communication volume. Together, repeated spectral computation and full-momentum communication make Muon substantially more expensive per step and may offset its optimization advantage. This motivates reducing the frequency of full spectral transformations while retaining effective updates. 3 Periodic Row-wise Muon We introduce Periodic Row-wise Muon, which performs an NS5 spectral update periodically and applies RowNorm to the current momentum at the remaining steps. Rather than treating RowNorm as a numerical approximation to NS5, our method alternates between two matrix updates induced by distinct constraint geometries. We then exploit the locality of RowNorm to reduce momentum collective communication and full-matrix materialization. 3.1 Complementary Spectral and Row-wise Geometries For a two-dimensional momentum Mt∈ℝm×nM_t ^m× n, let M~t=(Mt) M_t=O(M_t) denote MtM_t if m≤nm≤ n and Mt⊤M_t otherwise, so that M~t∈ℝr×c M_t ^r× c, where r=min(m,n)≤c=max(m,n)r= (m,n)≤ c= (m,n). The resulting direction is mapped back through −1O^-1. For a full-row-rank matrix X∈ℝr×cX ^r× c, the ideal polar factor (X)P(X) is (X)=(X⊤)−1/2X∈argmax‖U‖2≤1⟨X,U⟩,⟨A,B⟩=tr(A⊤B).P(X)=(X )^-1/2X∈ *arg\,max_ U _2≤ 1 X,U , A,B =tr(A B). (4) The spectral-norm constraint globally couples the rows and singular directions of X. Practical Muon uses a finite NS5 map motivated by this spectral geometry rather than computing the exact polar factor. RowNorm, previously studied in symmetry-compatible optimizer design (20), instead normalizes each row independently, ℛϵ(X)i:=Xi:max(∥Xi:∥2,ϵ),i=1,…,r.R_ε(X)_i:= X_i: \! ( X_i: _2,ε ), i=1,…,r. (5) For nonzero rows, RowNorm maximizes ⟨X,U⟩ X,U subject to ‖U‖2,∞:=maxi∥Ui:∥2≤1 U _2,∞:= _i U_i: _2≤ 1. Thus, the ideal polar direction imposes a globally coupled spectral constraint, whereas RowNorm preserves each momentum row’s direction while controlling its magnitude independently. The complete variational characterizations and their relation are provided in Appendices A.2 and A.3. Periodic spectral correction is further motivated by the local stability of the ideal polar direction. For full-row-rank matrices A and B, ‖(A)−(B)‖F≤2‖A−B‖Fσmin(A)+σmin(B). (A)-P(B) _F≤ 2 A-B _F _ (A)+ _ (B). (6) Hence, moderate momentum changes imply moderate changes in the ideal polar direction as long as the matrices remain away from rank degeneracy. This motivates periodically reimposing global spectral structure and using a lower compute and communication cost structured direction between refreshes. The proof and limitations of this exact-polar argument are discussed in Appendix A.4. 3.2 Periodic Row-wise Muon Building on the preceding analysis, we propose Periodic Row-wise Muon. Given a period K≥1K≥ 1 and a RowNorm multiplier γ>0γ>0, define the refresh indicator ρt=[tmodK=0] _t=I\! [t K=0 ]. At every step, we use the current momentum MtM_t to compute the parameter update D~t=NS5(M~t),ρt=1,γℛϵ(M~t),ρt=0,Wt+1=(1−ηtλ)Wt−ηts(Wt)−1(D~t). D_t= casesNS5( M_t),& _t=1,\\[3.0pt] γ\,R_ε( M_t),& _t=0, cases W_t+1=(1- _tλ)W_t- _ts(W_t)O^-1( D_t). (7) The period K controls the frequency of spectral updates, whereas γ calibrates the effective step size of the RowNorm branch relative to the spectral branch. This calibration is necessary because the two normalization maps correspond to different feasible sets. For a full-row-rank matrix with nonzero rows, the exact polar factor and the unregularized RowNorm direction both have Frobenius norm r r. However, equal Frobenius norms do not imply equal stable update amplitudes. NS5 couples all rows under spectral-norm geometry, whereas RowNorm updates rows independently under ℓ2,∞ _2,∞ geometry. Their alignment with the gradient and the local curvature encountered along their directions can therefore differ. Directly reusing the Muon learning rate is equivalent to setting γ=1γ=1, which implicitly assumes that the two branches have the same stable step-size range without theoretical justification. We write the effective learning rate of an off-refresh step as ηtRN=γηt _t^RN=γ _t. Here, ηt _t retains the global learning rate schedule of vanilla muon, while γ only specifies the relative scale between the two update geometries. Both normalization maps remove the global scale of the input momentum, and s(Wt)s(W_t) already accounts for shape-dependent scaling. The remaining calibration is therefore a dimensionless, branch-specific quantity. We use a fixed γ to preserve a constant ratio between the RowNorm and Muon learning rate schedules, rather than introducing a separate time varying schedule for RowNorm. Complete pseudocode is provided in Appendix C. Under standard conditional-alignment and bounded-second-moment assumptions, Appendix A.5 establishes a finite-horizon descent bound for the periodic update. Appendix A.6 further gives the blockwise dependence on γ, including its interaction with the remaining optimizer direction. 3.3 Distributed Execution The two branches of Periodic Row-wise Muon have different communication requirements. A refresh step must still materialize the complete momentum matrix to execute NS5. To reduce the exposed latency of this global path, we introduce a bucketed all-gather pipeline with communication–computation overlap. Specifically, we partition the Muon matrices into communication buckets. Once the momentum of the current bucket has been all-gathered, we immediately reconstruct its full matrices and execute NS5 while asynchronously all-gathering the next bucket. Communication for the next bucket is thereby overlapped with NS5 computation on the current bucket, and only a bounded number of momentum matrices need to be materialized at any time. On non-refresh steps, RowNorm operates directly on the momentum shards. If m≤nm≤ n, the orientation is unchanged and every RowNorm row is local, requiring no optimizer-specific collective. If m>nm>n, then (M)=M⊤O(M)=M , so each oriented row spans ranks. If rank p owns the original rows indexed by pS_p, it computes qj(p)=∑i∈pMij2,νj=max(∑pqj(p))1/2,ϵ.q_j^(p)= _i _pM_ij^2, _j= \ ( _pq_j^(p) )^1/2,ε \. (8) The denominators therefore require a sum all-reduce of only n scalars, rather than an all-gather of all mnmn momentum entries. Appendix B.2 proves equivalence to dense RowNorm. In the implementation, we first compute the local statistics for all tall matrices, pack them, and launch bucketed asynchronous all-reduces. While these collectives are in flight, we compute the local statistics and RowNorm updates for the remaining matrices. Once the all-reduces complete, we use the global statistics to finish the tall-matrix updates. This ordering hides the statistics communication for tall matrices behind local computation on the remaining matrices. Complete pseudocode is provide in Appendix C For an oriented matrix in ℝr×cR^r× c, r≤cr≤ c, the dominant computation of NS5 is CNS5=Θ(5r2c)C_NS5= (5r^2c), whereas RowNorm requires only CRN=Θ(rc)C_RN= (rc). Let VAGV_AG and VRNV_RN denote the total per-rank logical optimizer payloads on refresh and non-refresh steps. The average matrix-processing computation over one period C¯(K) C(K) and period-averaged payload V(K)V(K) satisfy C¯(K)=1KCNS5+K−1KCRN,V(K)VAG=1K+K−1KVRNVAG. C(K)= 1KC_NS5+ K-1KC_RN, V(K)V_AG= 1K+ K-1K V_RNV_AG. (9) For complete-row matrices VRN=0V_RN=0, while for a tall matrix VRN/VAG=O(1/m)V_RN/V_AG=O(1/m) under the logical payload model. For example, the total optimizer-specific payload for K=3K=3 approaches 1/31/3 of vanilla Muon when the norm-statistics payload is small. Detailed arithmetic, correctness, communication, and overlap models are given in Appendix B. 4 Experiments We systematically compare AdamW, vanilla Muon, and Periodic Row-wise Muon across DiT models ranging from approximately 1.3B to 15B parameters. Beyond the main comparison, we include all experimental results in Appendix E. 4.1 Experimental Setup Dataset and models. We train text-to-image Diffusion Transformers from scratch on GPIC-Full (6). GPIC contains 100M training image–text pairs collected from Flickr and Wikimedia, captioned with Qwen3-VL-4B-Instruct (4). We train four MMDiT (26) configurations at 512×512512× 512 resolution, containing approximately 1.3B, 4B, 9B, and 15B parameters. All models share the same overall architecture and conditioning modules, and differ only in hidden dimension, depth, and number of attention heads. Optimization and training. Unless otherwise specified, Periodic Row-wise Muon uses K=3K=3 and γ=0.15γ=0.15 throughout and their selection and analysis are provided in Appendix E.1. At each model scale, all three optimizers are trained for 60,000 steps with the same global batch size of 4,096, and therefore process the same number of training examples. All experiments use 32 nodes with 256 NVIDIA H100 GPUs in total and FSDP2 in Pytorch. We normalize all time quantities by the mean step time of the 1.3B AdamW run, which is defined as one unit. Evaluation. We select a fixed set of 50,000 prompts from the GPIC test set and generate one 512×512512× 512 image per prompt. All methods use the same sampling configuration with a fixed classifier-free guidance scale of 5.0. We report FD-DINOv2 (32), FID (14), Maximum Mean Discrepancy (MMD), Precision, Recall, Density, Coverage (18; 29; 25), HPSv2.1 (35), and GenEval2 (16). Completed experimental settings are provided in Appendix D. (a) Validation loss versus training progress. (b) FD-DINO throughout training and the best observed FD-DINO across model scales. (c) Validation loss versus wall-clock time. Figure 1: Scaling behavior of Muon for DiTs training. Muon consistently improves step efficiency and generation quality, while its higher per-step overhead can offset this advantage in wall-clock time. 4.2 Scaling Muon for Diffusion Transformers We first examine whether Muon retains its optimization advantage as Diffusion Transformers scale from 1.3B to 15B parameters. Figure 1(a) compares the validation loss trajectories of AdamW and Muon over 60k optimization steps. Across all four model scales, Muon consistently achieves lower validation loss than AdamW throughout the main training regime. The achievable validation loss also decreases as model size increases, showing that Muon’s optimization advantage remains stable when scaling to substantially larger DiTs. This advantage also extends to generation quality. Figure 1(b) shows that, across the 1.3B–15B models, Muon improves the best observed FD-DINO over AdamW by 12.9–19.1%, with the relative ordering across model scales remaining largely consistent throughout training. Together with the validation loss results, this demonstrates that Muon’s optimization advantage persists across scale in both optimization progress and generation quality. Complete checkpoint evaluations are provided in Appendix E.4. The step-wise advantage of Muon, however, does not directly translate into wall-clock time efficiency. Figure 1(c) replots the validation loss trajectories against normalized active training time. Although Muon ultimately reaches a lower validation loss, its larger per-step compute and communication cost shifts its trajectory to the right in wall-clock space. Within the highlighted training regime, AdamW reaches the same intermediate loss levels earlier than Muon across all 4 model scales. Thus, the optimization advantage observed per training step is partially or fully offset by Muon’s optimizer overhead under a fixed wall-clock time budget. This empirical gap motivates reducing Muon’s per-step overhead while preserving its generation quality advantage. Figure 2: Generative quality across model scales. We report FD-DINO improvement over AdamW at both the final checkpoint and the best checkpoint observed during training. Figure 3: Generative quality frontier under optional checkpoint selection. 4.3 Periodic Row-wise Muon Generation Quality and Training Efficiency We next evaluate whether Periodic Row-wise Muon preserves vanilla Muon’s generation quality while reducing training compute and communication cost. Figure 2 shows that it improves final-checkpoint FD-DINO over AdamW by 11.1–17.8% across scales and slightly outperforms vanilla Muon at 1.3B. Its best-observed FD-DINO remains within 0.5% of vanilla Muon at 1.3B and 4B, while improving it by approximately 4.5% at 9B and 2.7% at 15B. Table 1 shows a similarly competitive broader generation-quality profile: at 9B, Periodic Row-wise Muon increases GenEval2 AM from 51.77 to 57.33 and GM from 11.35 to 15.97, while at 15B its primary fidelity metrics remain close to vanilla Muon and Coverage and Density improve. Given their comparable best generation quality, Figure 3 compares the best achieved FD-DINO against normalized active training time. Periodic Row-wise Muon reaches its respective best result with 33.7%, 36.1%, 64.8%, and 57.4% less active time at 1.3B, 4B, 9B, and 15B, respectively, with the largest gains at scale. A complementary comparison with AdamW in active-time efficiency and final generation quality is provided in Appendix E.3. Table 1: Final 60k-step comparison. Bold denotes the best result among optimizers at the same model size. Model Optimizer Discrepancy Fidelity & Diversity Alignment Compositionality FD-DINO↓ FID↓ MMD-DINO↓ Precision↑ Recall↑ Coverage↑ Density↑ HPSv2↑ GenEval2 AM↑ GenEval2 GM↑ 1.3B AdamW 53.9353.93 6.716.71 0.03120.0312 0.94050.9405 0.88740.8874 0.94010.9401 0.96650.9665 20.5320.53 38.6638.66 7.107.10 Muon 46.0846.08 6.546.54 0.02680.0268 0.94870.9487 0.90710.9071 0.94920.9492 0.99180.9918 20.8420.84 44.7244.72 9.389.38 Periodic Row-wise Muon 45.6545.65 6.296.29 0.02700.0270 0.95170.9517 0.90650.9065 0.94850.9485 0.97560.9756 20.3620.36 45.8945.89 9.759.75 4B AdamW 51.3951.39 7.357.35 0.02710.0271 0.93710.9371 0.89950.8995 0.93880.9388 0.98980.9898 21.4021.40 45.7045.70 9.139.13 Muon 41.6241.62 6.086.08 0.02610.0261 0.93850.9385 0.91360.9136 0.94310.9431 0.96390.9639 20.6620.66 46.5846.58 9.069.06 Periodic Row-wise Muon 42.2542.25 6.356.35 0.02180.0218 0.94130.9413 0.90570.9057 0.95160.9516 0.98370.9837 21.3421.34 49.3849.38 10.5810.58 9B AdamW 41.2641.26 6.196.19 0.02160.0216 0.94550.9455 0.90450.9045 0.94950.9495 0.98730.9873 21.3921.39 45.9345.93 9.669.66 Muon 33.9733.97 5.615.61 0.01990.0199 0.95160.9516 0.92690.9269 0.95310.9531 0.96010.9601 22.6022.60 51.7751.77 11.3511.35 Periodic Row-wise Muon 36.7036.70 6.136.13 0.01920.0192 0.94940.9494 0.91340.9134 0.95110.9511 0.97930.9793 22.4422.44 57.3357.33 15.9715.97 15B AdamW 40.2340.23 6.246.24 0.02170.0217 0.94480.9448 0.90950.9095 0.94730.9473 0.96380.9638 21.9921.99 47.3447.34 9.189.18 Muon 33.5133.51 5.575.57 0.01910.0191 0.94450.9445 0.93040.9304 0.95010.9501 0.93670.9367 22.3722.37 57.9357.93 14.5514.55 Periodic Row-wise Muon 33.9933.99 5.635.63 0.01970.0197 0.94820.9482 0.92480.9248 0.95540.9554 0.96730.9673 21.3521.35 52.2652.26 11.9711.97 4.4 System Efficiency We next analyze the systems-level source of the wall-clock time improvement in Table 2. Across the 4 model scales, Periodic Row-wise Muon reduces end-to-end step time by 15.7–24.3% relative to vanilla Muon, while reducing optimizer time by 46.9–54.3%. On the largest 15B model, optimizer time decreases by 54.3%, resulting in a 23.4% reduction in total step time. The communication measurements exhibit the expected reduction from periodic spectral refreshes. With K=3K=3, the average number of optimizer specific bucketed all-gathers is reduced by around 66.6%. The corresponding logical communication volume decreases by 66.7% at every scale. The sharded RowNorm path introduces only approximately 0.7 small all-reduces per step. Figure 4 shows the execution breakdown for the 15B model. Periodic Row-wise Muon replaces two of every three NS5 phases and their full-momentum all-gathers with short RowNorm updates, while overlapping refresh communication with NS5 computation and norm-statistics communication with local computation. This reduces both optimizer computation and exposed communication. Table 2: System-efficiency comparison on 32 H100 nodes. Model Method Computation Communication Step time ↓ Opt. time ↓ Opt. share ↓ AG/step ↓ AR/step ↓ Comm. volume ↓ GiB/rank/step 1.3B Muon 1.873 0.835 44.6% 10.0 0.0 2.38 Periodic Row-wise Muon 1.419 (−24.3%)(-24.3\%) 0.443 (−46.9%)(-46.9\%) 31.3% 3.3 0.7 0.79 (−66.7%)(-66.7\%) 4B Muon 2.352 1.009 42.9% 32.0 0.0 7.58 Periodic Row-wise Muon 1.878 (−20.2%)(-20.2\%) 0.536 (−46.9%)(-46.9\%) 28.6% 10.7 0.7 2.53 (−66.7%)(-66.7\%) 9B Muon 4.153 1.689 40.7% 80.0 0.0 17.30 Periodic Row-wise Muon 3.500 (−15.7%)(-15.7\%) 0.868 (−48.6%)(-48.6\%) 24.8% 26.7 0.7 5.77 (−66.7%)(-66.7\%) 15B Muon 6.527 2.884 44.2% 119.0 0.0 28.62 Periodic Row-wise Muon 5.002 (−23.4%)(-23.4\%) 1.317 (−54.3%)(-54.3\%) 26.3% 39.7 0.7 9.54 (−66.7%)(-66.7\%) Figure 4: Simplified profiler traces for the 15B model. 4.5 Ablation Studies Algorithmic ablation. Table 3 (a) isolates the roles of periodic spectral refresh, RowNorm geometry, and branch-specific scale calibration. We compare vanilla Muon, RowNorm at every step, Periodic Row-wise Muon with γ=1γ=1, a scalar-controlled baseline that applies γ to both NS5 and RowNorm, and the complete Periodic Row-wise Muon with K=3K=3 and γ=0.15γ=0.15 on 1.3B scale. Overall, the complete method provides the best generation quality. Table 3: Ablation studies. (a) Algorithm-level ablation of Periodic Row-wise Muon on the 1.3B model. (b) System-level ablation of the distributed implementation on the 15B model. (a) Variant NS5 Schedule Off-refresh Update γ Best FD-DINO ↓ Vanilla Muon Every step – – 41.733 RowNorm every step None RowNorm 0.15 51.322 Periodic Row-wise Muon (γ=1γ=1) K=3K=3 RowNorm 1.0 44.294 Scalar-controlled (γ for both NS5 and RowNorm) K=3K=3 Scalar-controlled 0.15 44.713 Periodic Row-wise Muon (γ=0.15γ=0.15) K=3K=3 RowNorm 0.15 41.913 (b) Variant Sharded Bucketed Comm.–Comp. Step Time ↓ Opt. Time ↓ Comm. Volume ↓ RowNorm All-gather Overlap Naive Periodic (K=3,γ=0.15)(K=3,γ=0.15) × × × 5.685 2.046 28.62 + Sharded RowNorm ✓ × × 5.196 1.498 9.549.54 + Bucketed All-gather ✓ ✓ × 5.028 1.349 9.549.54 Full System ✓ ✓ ✓ 5.0025.002 1.3171.317 9.549.54 System ablation. Table 3(b) incrementally adds the system optimizations to a 15B naive periodic baseline, which uses the same update rule but materializes full momentum every step without bucketing or overlap. Sharded RowNorm reduces communication volume by 66.7%66.7\%, while bucketed all-gather and communication–computation overlap further reduce exposed refresh latency. Together, these optimizations reduce optimizer and end-to-end step time by 35.6%35.6\% and 12.0%12.0\%, respectively. 5 Related Work Unlike AdamW’s element-wise adaptive updates (24), matrix-aware optimizers exploit the matrix structure of model parameters. Shampoo and SOAP construct structured preconditioners (13; 34), whereas Muon applies a finite Newton–Schulz transformation to momentum matrices, producing updates with global spectral structure (15). Muon has subsequently been shown to scale to large language models and improve the compute–quality trade-off over AdamW (23; 31). In parallel, prior work has established predictable DiT scaling across model size, data, compute, and training hyperparameters (11; 22; 36), while recent studies evaluate matrix-aware optimization in diffusion training and adapt momentum orthogonalization to diffusion-specific parameter structures (30; 7). Together, these directions motivate evaluating optimizer scalability through optimization progress, algorithmic compute, generative quality, and realized distributed efficiency. Recent work reduces Muon’s spectral transformation cost, which is particularly important for large DiTs. One direction accelerates orthogonalization through GPU-friendly polynomial iterations, Gram-matrix formulations, or optimized Newton–Schulz polynomials (3; 37; 12); another reduces its frequency, scope, or input size through block-periodic updates, temporal reuse, alternating spectral and sign-based updates, tiled transformations, or row/column subsampling (17; 10; 5; 33; 1). Other variants post-process every step NS orthogonalization with row- or neuron-wise normalization and optional neuron-wise second-moment statistics (21; 38). Distributed orthonormalized optimizers and specialized Muon runtimes further reduce communication, redundant computation, and exposed optimizer latency under model sharding (2; 9). Our approach is complementary to faster spectral kernels, it reduces spectral refresh frequency and directly applies RowNorm to the current momentum as a low compute and communication cost dense substitute between refreshes. RowNorm also admits recently studied symmetry properties (20), and its locality allows us to avoid full-momentum materialization on non-refresh steps in large-scale distributed training. 6 Conclusion and Limitations We show that Muon’s optimization and generative quality advantages over AdamW persist as DiTs scale from 1.3B to 15B parameters, but its every step NS5 computation and full momentum communication introduce substantial distributed overhead. Periodic Row-wise Muon addresses this bottleneck by combining periodic spectral refreshes with low compute and communication cost RowNorm updates and a sharded distributed implementation. Across all scales, it preserves generation quality broadly comparable to vanilla Muon while reducing optimizer time by 46.9–54.3%. However, our evaluation is limited to one DiT family, dataset, resolution, and 32-node H100 configuration, and does not cover other training regimes. We also use fixed global K and γ, leaving layer-wise and adaptive schedules unexplored. Finally, refresh steps still retain full momentum communication and materialization, while the realized speedup may vary with hardware topology and parallelism strategy. We will leave these limitations as future works. References Ahn et al. (2025) K. Ahn, N. Amsel, and J. Langford Dion2: a simple method to shrink matrix in muon. arXiv preprint arXiv:2512.16928. Cited by: §5. Ahn and Xu (2025) K. Ahn and B. Xu Dion: a communication-efficient optimizer for large models. arXiv e-prints, p. arXiv–2504. Cited by: §5. Amsel et al. (2026) N. Amsel, D. Persson, C. Musco, and R. M. Gower The polar express: optimal matrix sign methods and their application to the muon algorithm. In International Conference on Learning Representations, Vol. 2026, p. 138323–138360. Cited by: §5. Bai et al. (2025) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: §D.1, §4.1. Bolatov et al. (2026) A. Bolatov, A. Riabinin, N. Kornilov, A. Veprikov, S. Horváth, M. Takáč, and A. Beznosikov LionMuon: alternating spectral and sign descent for efficient training. arXiv preprint arXiv:2605.19811. Cited by: §5. Chandrasegaran et al. (2026) K. Chandrasegaran, K. Sargent, S. Agarwal, M. Jang, M. Poli, J. C. Niebles, J. Johnson, J. Wu, and L. Fei-Fei GPIC: a giant permissive image corpus for visual generation. arXiv preprint arXiv:2605.30341. Cited by: §D.1, §1, §4.1. Chen et al. (2026a) C. Chen, P. Sun, and K. Yuan CMuon: accelerating and stabilizing diffusion transformer training via chunked momentum orthogonalization. arXiv preprint arXiv:2608.02502. Cited by: §5. Chen et al. (2024) J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Z. Wang, J. Kwok, P. Luo, H. Lu, and Z. Li Pixart-α: fast training of diffusion transformer for photorealistic text-to-image synthesis. In International conference on learning representations, Vol. 2024, p. 57611–57640. Cited by: §1. Chen et al. (2026b) V. Chen, S. Liu, R. Cheng, D. Yang, S. Li, R. Yu, L. Liang, H. Su, R. Gan, H. Wang, et al. DMuon: efficient distributed muon training with near-adam overhead. arXiv preprint arXiv:2606.27153. Cited by: §5. Dev et al. (2026) B. Dev, S. Bohara, M. Takáč, and S. Horváth CacheMuon: using temporal preconditioning to approximate polar factor. arXiv preprint arXiv:2606.16371. Cited by: §5. Esser et al. (2024) P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, Cited by: §D.2, §1, §5. Grishina et al. (2025) E. Grishina, M. Smirnov, and M. Rakhuba Accelerating newton-schulz iteration for orthogonalization via chebyshev-type polynomials. arXiv preprint arXiv:2506.10935. Cited by: §5. Gupta et al. (2018) V. Gupta, T. Koren, and Y. Singer Shampoo: preconditioned stochastic tensor optimization. In International Conference on Machine Learning, p. 1842–1850. Cited by: §5. Heusel et al. (2017) M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter GANs trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30. Cited by: 2nd item, §4.1. Jordan et al. (2024) K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein Muon: an optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan. github. io/posts/muon 6 (3), p. 4. Cited by: §1, §5. Kamath et al. (2025) A. Kamath, K. Chang, R. Krishna, L. Zettlemoyer, Y. Hu, and M. Ghazvininejad Geneval 2: addressing benchmark drift in text-to-image evaluation. arXiv preprint arXiv:2512.16853. Cited by: 7th item, §4.1. Khaled et al. (2026) A. Khaled, K. Ozkara, T. Yu, M. Hong, and Y. Park MuonBP: faster muon via block-periodic orthogonalization. In International Conference on Learning Representations, Vol. 2026, p. 17389–17416. Cited by: §5. Kynkänniemi et al. (2019) T. Kynkänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila Improved precision and recall metric for assessing generative models. Advances in neural information processing systems 32. Cited by: §4.1. Labs et al. (2025) B. F. Labs, S. Batifol, A. Blattmann, F. Boesel, S. Consul, C. Diagne, T. Dockhorn, J. English, Z. English, P. Esser, et al. Flux. 1 kontext: flow matching for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.15742. Cited by: §D.2. Lau and Su (2026) T. T. Lau and W. Su Symmetry-compatible principle for optimizer design: embeddings, lm heads, swiglu mlps, and moe routers. arXiv preprint arXiv:2605.18106. Cited by: §1, §3.1, §5. Li et al. (2025) Z. Li, L. Liu, C. Liang, W. Chen, and T. Zhao NorMuon: making muon more efficient and scalable. arXiv preprint arXiv:2510.05491. Cited by: §5. Liang et al. (2026) Z. Liang, H. He, C. Yang, and B. Dai Scaling laws for diffusion transformers. In International Conference on Learning Representations, Vol. 2026, p. 84488–84521. Cited by: §1, §5. Liu et al. (2025) J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982. Cited by: §1, §1, §2.3, §5. Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §1, §5. Naeem et al. (2020) M. F. Naeem, S. J. Oh, Y. Uh, Y. Choi, and J. Yoo Reliable fidelity and diversity metrics for generative models. In International conference on machine learning, p. 7176–7185. Cited by: §4.1. Peebles and Xie (2023) W. Peebles and S. Xie Scalable diffusion models with transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 4172–4182. Cited by: §1, §4.1. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §D.2. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21 (140), p. 1–67. Cited by: §D.2. Sajjadi et al. (2018) M. S. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly Assessing generative models via precision and recall. Advances in neural information processing systems 31. Cited by: §4.1. Schaipp (2025) F. Schaipp Optimization benchmark for diffusion models on dynamical systems. arXiv preprint arXiv:2510.19376. Cited by: §1, §5. Shah et al. (2025) I. Shah, A. M. Polloreno, K. Stratos, P. Monk, A. Chaluvaraju, A. Hojel, A. Ma, A. Thomas, A. Tanwer, D. J. Shah, et al. Practical efficiency of muon for pretraining. arXiv preprint arXiv:2505.02222. Cited by: §5. Stein et al. (2023) G. Stein, J. Cresswell, R. Hosseinzadeh, Y. Sui, B. Ross, V. Villecroze, Z. Liu, A. L. Caterini, E. Taylor, and G. Loaiza-Ganem Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems 36, p. 3732–3784. Cited by: 1st item, §4.1. Tang et al. (2026) Z. Tang, T. Xu, Y. Saad, and Y. Xi Hierarchical Muon: tiled newton-schulz updates for efficient muon optimization. arXiv preprint arXiv:2606.27216. Cited by: §5. Vyas et al. (2025) N. Vyas, D. Morwani, R. Zhao, I. Shapira, D. Brandfonbrener, L. Janson, and S. Kakade SOAP: improving and stabilizing shampoo using adam for language modeling. In International Conference on Learning Representations, Vol. 2025, p. 93423–93444. Cited by: §5. Wu et al. (2023) X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li Human preference score v2: a solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341. Cited by: 6th item, §4.1. Yin et al. (2025) Y. Yin, Y. Zhao, M. Zheng, K. Lin, J. Ou, R. Chen, V. S. Huang, J. Wang, X. Tao, P. Wan, et al. Towards precise scaling laws for video diffusion transformers. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 18155–18165. Cited by: §1, §5. Zhang et al. (2026a) J. Zhang, N. Amsel, B. Chen, and T. Dao Gram newton-schulz. External Links: Link Cited by: §5. Zhang et al. (2026b) R. Zhang, Y. Zhao, Z. Liu, Z. Wang, Y. Su, L. Tan, and Z. Zhang MUON+: towards more effective muon via one additional normalization step for llm pre-training. arXiv preprint arXiv:2602.21545. Cited by: §5. Appendix Appendix A Mathematical Analysis This appendix gives the complete statements and proofs supporting the geometric interpretation and the optimization-feasibility discussion in the main text. The results deliberately distinguish the exact polar factor from the finite Newton–Schulz map used in the implementation. They also distinguish a conditional descent guarantee from an unconditional convergence theorem for the proposed optimizer. A.1 Notation and update convention For matrices of the same shape, let ⟨A,B⟩≔tr(A⊤B),‖A‖F2≔⟨A,A⟩, A,B (A B), \|A\|_F^2 A,A , (10) and let ‖A‖2\|A\|_2 denote the spectral norm. For X∈ℝr×cX ^r× c, define the row-wise maximum norm ∥X∥2,∞:=max1≤i≤r∥Xi:∥2.\|X\|_2,∞:= _1≤ i≤ r\|X_i:\|_2. (11) The practical RowNorm map is ℛε(X)i::=Xi:max∥Xi:∥2,ε,ε>0. R_ (X)_i::= X_i: \\|X_i:\|_2, \, >0. (12) When every row is nonzero, ℛ(X) R(X) denotes the unregularized map obtained by setting the denominator to ∥Xi:∥2\|X_i:\|_2. For the optimization result, θt _t denotes the vector containing all trainable parameters, and we write the implemented update as θt+1=θt−ηtHt. _t+1= _t- _tH_t. (13) Here HtH_t is the complete effective direction. It includes the selected NS5 or RowNorm matrix blocks, Muon’s shape factors, the RowNorm multiplier, all AdamW-updated blocks, any parameter-group learning rate ratios relative to ηt _t, and decoupled weight decay. This convention is important because the RowNorm multiplier scales only the RowNorm blocks, not the complete optimizer direction. A.2 RowNorm as a row-wise variational direction Lemma 1 (Row-wise variational characterization). Let X∈ℝr×cX ^r× c, and suppose that every row of X is nonzero. Define ∥U∥2,∞:=max1≤i≤r∥Ui,:∥2.\|U\|_2,∞:= _1≤ i≤ r\|U_i,:\|_2. Then R(X)=argmaxU∈ℝr×c⟨X,U⟩:‖U‖2,∞≤1,R(X)= *arg\,max_U ^r× c \ X,U :\|U\|_2,∞≤ 1 \, (14) where the maximizer is unique. Proof. The constraint ‖U‖2,∞≤1\|U\|_2,∞≤ 1 is equivalent to ∥Ui:∥2≤1\|U_i:\|_2≤ 1 for every row. Hence, by Cauchy–Schwarz, ⟨X,U⟩=∑i=1r⟨Xi:,Ui:⟩≤∑i=1r∥Xi:∥2∥Ui:∥2≤∑i=1r∥Xi:∥2. X,U = _i=1^r X_i:,U_i: ≤ _i=1^r\|X_i:\|_2\|U_i:\|_2≤ _i=1^r\|X_i:\|_2. (15) Equality is attained by Ui:=Xi:/∥Xi:∥2U_i:=X_i:/\|X_i:\|_2 for every i, which is exactly U=ℛ(X)U= R(X). Because every Xi:X_i: is nonzero, equality in the two inequalities requires this choice row by row, proving uniqueness. ∎ Lemma 1 characterizes RowNorm in its own geometry. In particular, it neither treats RowNorm as an approximation to NS5 nor asserts that the two updates follow the same optimization trajectory. A.3 Polar and Row-wise Variational Geometries We use the orientation convention from Section 3.1: a matrix is transposed when necessary so that X∈ℝr×cX ^r× c with r≤cr≤ c. If X has full row rank, its row-polar factor is (X):=(X⊤)−1/2X,(X)(X)⊤=Ir.P(X):=(X )^-1/2X, (X)P(X) =I_r. (16) Proposition 1 (Spectral variational characterization). Let X∈ℝr×cX ^r× c, where r≤cr≤ c, and suppose that rank(X)=rrank(X)=r. Then (X)=argmaxU∈ℝr×c⟨X,U⟩subject to‖U‖2≤1.P(X)= U ^r× c \; X,U to \|U\|_2≤ 1. (17) The maximizer is unique, and the optimal value is the nuclear norm ‖X‖∗\|X\|_*. Proof. Let X=AΣB⊤X=A B be a thin singular value decomposition, where A∈ℝr×rA ^r× r is orthogonal, B∈ℝc×rB ^c× r has orthonormal columns, and Σ=diag(σ1,…,σr) =diag( _1,…, _r) with σi>0 _i>0. Extend B to a square orthogonal matrix V=[BB⟂]∈ℝc×cV=[B\;B_ ] ^c× c. For any feasible U, set Z=A⊤UVZ=A UV. Orthogonal invariance gives ‖Z‖2≤1\|Z\|_2≤ 1, and ⟨X,U⟩=tr(ΣA⊤UB)=∑i=1rσiZii≤∑i=1rσi=‖X‖∗. X,U =tr( A UB)= _i=1^r _iZ_i≤ _i=1^r _i=\|X\|_*. (18) The upper bound is attained by U=AB⊤=(X)U=AB =P(X). Because every σi _i is positive, equality requires Zii=1Z_i=1 for every i≤ri≤ r. A contraction whose first r diagonal entries are all one must have Z=[Ir 0]Z=[I_r\;0]; otherwise at least one row would have Euclidean norm greater than one. Thus U=AB⊤U=AB is the unique maximizer. ∎ The two feasible sets obey U:‖U‖2≤1⊆U:‖U‖2,∞≤1⊆U:‖U‖2≤r.\U:\|U\|_2≤ 1\ \U:\|U\|_2,∞≤ 1\ \U:\|U\|_2≤ r\. (19) Indeed, every row norm is at most the spectral norm, while ‖U‖2≤‖U‖F≤r‖U‖2,∞\|U\|_2≤\|U\|_F≤ r\|U\|_2,∞. Thus the polar direction solves a globally coupled spectral-norm problem, whereas RowNorm solves a product of independent row-wise problems. Finite NS5 is the implemented spectral map motivated by the former geometry; the proposition does not identify finite NS5 with the exact polar factor. A.4 Conditional stability of the ideal polar direction Proposition 2 (Perturbation of the row-polar factor). Let A,B∈ℝr×cA,B ^r× c, where r≤cr≤ c, and suppose that both matrices have full row rank. Then ‖(A)−(B)‖F≤2‖A−B‖Fσmin(A)+σmin(B).\|P(A)-P(B)\|_F≤ 2\|A-B\|_F _ (A)+ _ (B). (20) Consequently, on any region satisfying σmin(X)≥σ0>0 _ (X)≥ _0>0, the polar map is 1/σ01/ _0-Lipschitz in Frobenius norm. Proof. Write the row-polar decompositions as A=HAPA,B=HBPB,A=H_AP_A, B=H_BP_B, (21) where HA=(AA⊤)1/2H_A=(A )^1/2, HB=(BB⊤)1/2H_B=(B )^1/2, PA=(A)P_A=P(A), and PB=(B)P_B=P(B). Both PAPA⊤=IrP_AP_A =I_r and PBPB⊤=IrP_BP_B =I_r. Let C=PAPB⊤C=P_AP_B , sym(C):=(C+C⊤)/2sym(C):=(C+C )/2. Since ‖C‖2≤1\|C\|_2≤ 1, Ir−sym(C)I_r-sym(C) is positive semidefinite. Direct expansion gives ⟨A−B,PA−PB⟩ A-B,P_A-P_B =tr(HA(Ir−C⊤))+tr(HB(Ir−C)) =tr\! (H_A(I_r-C ) )+tr\! (H_B(I_r-C) ) ≥(σmin(A)+σmin(B))(r−tr(C)). ≥ ( _ (A)+ _ (B) ) (r-tr(C) ). (22) The skew-symmetric parts vanish inside the traces because HAH_A and HBH_B are symmetric. Moreover, r−tr(C)=12‖PA−PB‖F2.r-tr(C)= 12\|P_A-P_B\|_F^2. (23) Combining this identity with Cauchy–Schwarz yields ‖A−B‖F‖PA−PB‖F≥σmin(A)+σmin(B)2‖PA−PB‖F2.\|A-B\|_F\|P_A-P_B\|_F≥ _ (A)+ _ (B)2\|P_A-P_B\|_F^2. (24) If PA=PBP_A=P_B, the result is immediate; otherwise division by ‖PA−PB‖F\|P_A-P_B\|_F proves equation 20. ∎ Proposition 2 provides a conditional motivation for periodic spectral correction: moderate momentum drift implies moderate drift of the ideal polar direction only when the relevant matrices remain away from rank degeneracy. The proposition concerns the exact polar factor and does not establish the same perturbation bound for the finite NS5 map used in the implementation. We therefore use it as geometric motivation rather than as a guarantee that finite NS5 remains unchanged between refreshes. A.5 Conditional finite-horizon descent Let ℱtF_t denote the information available before the stochastic direction at step t is realized. The following result applies directly to the complete effective direction in equation 13; it does not require the iterates to track those of vanilla Muon. Theorem 1 (Finite-horizon bound under conditional alignment ). Suppose that f:ℝd→ℝf:R^d is L-smooth and bounded below by finff_ . Assume that, for each t, there are deterministic constants at>0a_t>0, bt≥0b_t≥ 0, and vt≥0v_t≥ 0 such that [⟨∇f(θt),Ht⟩∣ℱt] \! [ ∇ f( _t),H_t _t ] ≥at‖∇f(θt)‖22, ≥ a_t\|∇ f( _t)\|_2^2, (25) [‖Ht‖22∣ℱt] \! [\|H_t\|_2^2 _t ] ≤bt‖∇f(θt)‖22+vt. ≤ b_t\|∇ f( _t)\|_2^2+v_t. (26) If dt:=at−Lηtbt2>0for t=0,…,T−1,d_t:=a_t- L _tb_t2>0 t=0,…,T-1, (27) then ∑t=0T−1ηtdt‖∇f(θt)‖22≤f(θ0)−finf+L2∑t=0T−1ηt2vt. _t=0^T-1 _td_t\,E\|∇ f( _t)\|_2^2≤ f( _0)-f_ + L2 _t=0^T-1 _t^2v_t. (28) This statement holds for any deterministic switching schedule, including a periodic schedule with one NS5 step followed by K−1K-1 RowNorm steps. Proof. By L-smoothness and equation 13, f(θt+1)≤f(θt)−ηt⟨∇f(θt),Ht⟩+Lηt22‖Ht‖22.f( _t+1)≤ f( _t)- _t ∇ f( _t),H_t + L _t^22\|H_t\|_2^2. (29) Taking conditional expectation and applying equation 25 and equation 26 gives [f(θt+1)∣ℱt]≤f(θt)−ηtdt‖∇f(θt)‖22+Lηt22vt.E[f( _t+1) _t]≤ f( _t)- _td_t\|∇ f( _t)\|_2^2+ L _t^22v_t. (30) Taking total expectation, summing over t=0,…,T−1t=0,…,T-1, and using f(θT)≥finff( _T)≥ f_ proves equation 28. ∎ For example, if ηt=η _t=η, at≥a>0a_t≥ a>0, 0≤bt≤b0≤ b_t≤ b for some b>0b>0, vt≤v_t≤ v, and 0<η<2a/(Lb)0<η<2a/(Lb), then 1T∑t=0T−1‖∇f(θt)‖22≤f(θ0)−finfTη(a−Lηb/2)+Lηv2(a−Lηb/2). 1T _t=0^T-1E\|∇ f( _t)\|_2^2≤ f( _0)-f_ Tη (a-Lη b/2 )+ Lη v2 (a-Lη b/2 ). (31) This is a finite-horizon stationary-point bound with a noise-dependent residual, not a claim of asymptotic convergence for the finite cosine schedule used in our experiments. The theorem is explicitly conditional. RowNorm is positively aligned with its own input momentum whenever the momentum has a nonzero row: ⟨M,ℛ(M)⟩=∑i=1r∥Mi:∥2>0. M, R(M) = _i=1^r\|M_i:\|_2>0. (32) However, this identity alone does not imply positive alignment with the true objective gradient, because in general ⟨∇f(θt),ℛ(Mt)⟩ ∇ f( _t), R(M_t) and ⟨Mt,ℛ(Mt)⟩ M_t, R(M_t) are different quantities. We therefore use Theorem 1 only as a sufficient-condition result and do not claim that its alignment assumption is verified throughout training. A.6 The blockwise role of the RowNorm multiplier Conditioned on a fixed optimization history ℱtF_t and the current iterate θt _t, decompose the complete effective direction on an off-refresh step as Ht(γ)=Ht(0)+γHt(RN),H_t(γ)=H_t^(0)+γ H_t^(RN), (33) where Ht(RN)H_t^(RN) contains the unscaled RowNorm directions in the blocks to which the method is applied, and Ht(0)H_t^(0) contains all remaining contributions. Define the conditional scalar quantities A0,t A_0,t :=[⟨∇f(θt),Ht(0)⟩∣ℱt], :=E[ ∇ f( _t),H_t^(0) _t], (34) ARN,t A_RN,t :=[⟨∇f(θt),Ht(RN)⟩∣ℱt], :=E[ ∇ f( _t),H_t^(RN) _t], (35) B0,t B_0,t :=[‖Ht(0)‖22∣ℱt], :=E[\|H_t^(0)\|_2^2 _t], (36) BRN,t B_RN,t :=[‖Ht(RN)‖22∣ℱt], :=E[\|H_t^(RN)\|_2^2 _t], (37) B×,t B_×,t :=[⟨Ht(0),Ht(RN)⟩∣ℱt]. :=E[ H_t^(0),H_t^(RN) _t]. (38) Then the two quantities entering the smoothness bound are exactly [⟨∇f(θt),Ht(γ)⟩∣ℱt] [ ∇ f( _t),H_t(γ) _t] =A0,t+γARN,t, =A_0,t+γ A_RN,t, (39) [‖Ht(γ)‖22∣ℱt] [\|H_t(γ)\|_2^2 _t] =B0,t+2γB×,t+γ2BRN,t. =B_0,t+2γ B_×,t+γ^2B_RN,t. (40) Consequently, the conditional one-step upper bound is [f(θt+1)∣ℱt]≤f(θt) [f( _t+1) _t]≤ f( _t) −ηt(A0,t+γARN,t) - _t (A_0,t+γ A_RN,t ) +Lηt22(B0,t+2γB×,t+γ2BRN,t). + L _t^22 (B_0,t+2γ B_×,t+γ^2B_RN,t ). (41) Thus γ is admissible whenever the measured or assumed bounds make the net descent term in equation 41 positive. Equation 39 and 40 also show why scaling the RowNorm blocks does not multiply the alignment and second-moment constants of the complete direction by γ and γ2γ^2, respectively. The theory motivates branch-specific amplitude calibration but does not determine the numerical value used in the experiments. That value is selected by the development-set ablation in Appendix E.1. Appendix B Complexity and Distributed Execution This appendix refines the asymptotic discussion in the main text with an implementation matched arithmetic model, a correctness proof for sharded RowNorm, and explicit per-rank communication formulas. B.1 Per-matrix arithmetic Consider an oriented matrix X∈ℝr×cX ^r× c with r≤cr≤ c. We count one multiplication followed by one addition as two floating-point operations. The implemented quintic Newton–Schulz step can be written as Gj G_j =XjXj⊤, =X_jX_j , (42) Bj B_j =bjGj+cjGj2, =b_jG_j+c_jG_j^2, (43) Xj+1 X_j+1 =ajXj+BjXj,j=0,…,JNS−1. =a_jX_j+B_jX_j, j=0,…,J_NS-1. (44) The three GEMMs in one iteration cost, to leading order, 2r2c,2r3,2r2c2r^2c, 2r^3, 2r^2c (45) FLOPs, respectively. Therefore CNSJ(r,c)=JNS(4r2c+2r3)+O(JNS(r2+rc)).C_NSJ(r,c)=J_NS (4r^2c+2r^3 )+O\! (J_NS(r^2+rc) ). (46) For NS5, CNS5(r,c)=20r2c+10r3+O(r2+rc).C_NS5(r,c)=20r^2c+10r^3+O(r^2+rc). (47) The lower-order term includes scalar matrix combinations and the input normalization. Equation 47 is a FLOP model for the recurrence in equation 44; measured kernel time can differ because GEMM efficiency depends on shape, dtype, and hardware. RowNorm performs one sum-of-squares reduction per row, one clamped inverse norm per row, and one rescaling per element. Its arithmetic cost is CRN(r,c)=Θ(rc).C_RN(r,c)= (rc). (48) We do not assign a hardware-independent exact FLOP count to square root, reciprocal, or fused reduction operations. The relevant separation is therefore CNS5(r,c)=Θ(r2c),CRN(r,c)=Θ(rc).C_NS5(r,c)= (r^2c), C_RN(r,c)= (rc). (49) Let ℐI be the set of matrices governed by the periodic rule. The period-averaged matrix-processing computational cost is C¯mat(K)=1K∑i∈ℐCNS5(ri,ci)+K−1K∑i∈ℐCRN(ri,ci). C_mat(K)= 1K _i C_NS5(r_i,c_i)+ K-1K _i C_RN(r_i,c_i). (50) If CfixedC_fixed denotes momentum maintenance, non-Muon parameter updates, and all other optimizer work performed every step, then C¯opt(K)=Cfixed+C¯mat(K). C_opt(K)=C_fixed+ C_mat(K). (51) The periodic schedule removes exactly the fraction 1−1/K1-1/K of spectral refresh events, but it does not remove the same fraction of total optimizer FLOPs or wall-clock time because the terms in equation 51 remain. B.2 Correctness of sharded RowNorm Suppose that the original matrix M∈ℝm×nM ^m× n is sharded along its first dimension across p ranks. Rank q owns the row-index set SqS_q, and the sets S1,…,SpS_1,…,S_p form a disjoint partition of 1,…,m\1,…,m\. If m≤nm≤ n, the orientation map leaves M unchanged. Each oriented row is therefore stored completely on one rank, and applying equation 12 to each local row is exactly the corresponding slice of dense RowNorm. If m>nm>n, the orientation map gives X=M⊤∈ℝn×mX=M ^n× m. An oriented row is now split across ranks. For each original column j∈1,…,nj∈\1,…,n\, rank q computes sj(q):=∑i∈SqMij2.s_j^(q):= _i∈ S_qM_ij^2. (52) After a sum all-reduce, every rank forms νj:=max∑q=1psj(q),ε. _j:= \ _q=1^ps_j^(q), \. (53) Each rank then rescales its local entries as M^ij(q):=Mijνj,i∈Sq,j=1,…,n. M_ij^(q):= M_ij _j, i∈ S_q, j=1,…,n. (54) Proposition 3 (Exact-arithmetic equivalence). Under the partition above, concatenating the local outputs M^(1),…,M^(p) M^(1),…, M^(p) gives exactly −1(Rϵ((M))).O^-1 (R_ε(O(M)) ). Proof. The complete-row case follows immediately because each local denominator is computed from all entries of its row. In the split-row case, ∑q=1psj(q)=∑q=1p∑i∈SqMij2=∑i=1mMij2=∥Xj:∥22. _q=1^ps_j^(q)= _q=1^p _i∈ S_qM_ij^2= _i=1^mM_ij^2=\|X_j:\|_2^2. (55) Hence νj=max∥Xj:∥2,ε _j= \\|X_j:\|_2, \. Every local entry of the oriented row is divided by the same dense denominator, so concatenating the local pieces yields Xj:/νjX_j:/ _j for every j, which is exactly ℛε(X) R_ (X). Mapping back through −1O^-1 completes the proof. ∎ The proposition is exact over real arithmetic. In floating-point arithmetic, the all-reduce may change the summation order and therefore introduce only the usual reduction-order roundoff differences. B.3 Per-rank communication volume We report logical algorithmic payload and exclude headers, padding, collective launch latency, and parameter communication already required by forward and backward propagation. Let bMb_M be the number of bytes per communicated momentum element and bSb_S the number of bytes per communicated norm statistic. Under a bandwidth-optimal ring model, define αAG(p):=p−1p,αAR(p):=2p−1p. _AG(p):= p-1p, _AR(p):=2 p-1p. (56) The factor of two in αAR(p) _AR(p) accounts for the reduce-scatter and all-gather phases of a ring all-reduce. For a refresh of M∈ℝm×nM ^m× n, the per-rank all-gather volume is Vrefresh(M)=αAG(p)bMmn.V_refresh(M)= _AG(p)b_Mmn. (57) Every rank executes NS5 on the gathered matrix and retains its local output slice, so this execution path requires no subsequent scatter of the update. On an off-refresh step, complete-row RowNorm requires no optimizer-specific collective. A tall matrix m>nm>n requires an all-reduce of n statistics, giving VRN(M)=αAR(p)bSn,m>n,0,m≤n.V_RN(M)= cases _AR(p)b_Sn,&m>n,\\ 0,&m≤ n. cases (58) For one tall matrix, the ratio is therefore VRN(M)Vrefresh(M)=2bSbMm. V_RN(M)V_refresh(M)= 2b_Sb_Mm. (59) For a collection ℐI of periodically updated matrices, define Sfull:=∑i∈ℐmini,Stall:=∑i∈ℐ:mi>nini.S_full:= _i m_in_i, S_tall:= _i :m_i>n_in_i. (60) The period-averaged per-rank payload is V¯rank(K) V_rank(K) =1KαAG(p)bMSfull+K−1KαAR(p)bSStall. = 1K _AG(p)b_MS_full+ K-1K _AR(p)b_SS_tall. (61) Relative to vanilla Muon’s every-step refresh payload, V¯rank(K)αAG(p)bMSfull=1K+K−1K2bSStallbMSfull. V_rank(K) _AG(p)b_MS_full= 1K+ K-1K 2b_SS_tallb_MS_full. (62) When the reduced statistics are small relative to the full matrices, the second term is negligible and the ratio approaches 1/K1/K. This is a statement about logical payload, not a claim that collective time or total step time decreases by the same factor. B.4 Idealized overlap model Bucketed execution changes exposed latency rather than the logical byte count in equation 57. Consider B ordered refresh buckets. Let aℓa_ be the all-gather time for bucket ℓ , and let cℓc_ be its reconstruction, NS5, and slicing time. In an ideal two-stage pipeline that overlaps the computation of bucket ℓ with the all-gather of bucket ℓ+1 +1, the makespan is Tpipe=a1+∑ℓ=1B−1maxcℓ,aℓ+1+cB.T_pipe=a_1+ _ =1^B-1 \c_ ,a_ +1\+c_B. (63) The corresponding exposed communication time beyond bucket computation is Tcomm,exposed=a1+∑ℓ=1B−1max0,aℓ+1−cℓ.T_comm,exposed=a_1+ _ =1^B-1 \0,a_ +1-c_ \. (64) These equations describe an ideal dependency graph. Actual traces may include stream synchronization, launch overhead, contention, and imperfect kernel concurrency; end-to-end claims are therefore based on measured profiler windows spanning complete periods. B.5 Wall-clock break-even condition Let Nbase(q)N_base(q) and Nper(q)N_per(q) be the numbers of optimization steps required by vanilla Muon and Periodic Muon, respectively, to reach a prespecified quality threshold q. Let their measured mean step times be τbase _base and τper _per, using the same number of GPUs. Periodic Muon improves time to quality exactly when Nper(q)τper<Nbase(q)τbase,N_per(q) _per<N_base(q) _base, (65) or equivalently, Nper(q)Nbase(q)<τbaseτper. N_per(q)N_base(q)< _base _per. (66) Thus a lower per-step optimizer systems cost is insufficient by itself: the realized speedup must exceed any increase in the number of steps required to reach the same quality. If the GPU counts differ, the analogous GPU-hour comparison multiplies each side by its respective accelerator count. Appendix C Algorithms In this section, we provide the Periodic Row-wise Muon algorithm 1 and the system-level algorithm 2. Algorithm 1 Periodic Row-wise Muon Input: two-dimensional Muon parameters MW_M, remaining parameters AW_A, period K, and RowNorm multiplier γ. 1 for t=0,…,T−1t=0,…,T-1 do 2 Compute stochastic gradients and update every momentum MtM_t as in Section 2. 3 Set ρt←[tmodK=0] _t [t K=0]. 4 for each Wt∈MW_t _M do 5 M~t←(Mt) M_t (M_t). 6 if ρt=1 _t=1 then D~t←NS5(M~t) D_t 5( M_t). 7 else D~t←γℛϵ(M~t) D_t← _ε( M_t). 8 Dt←s(Wt)−1(D~t)D_t← s(W_t)O^-1( D_t). 9 Wt+1←(1−ηtλ)Wt−ηtDtW_t+1←(1- _tλ)W_t- _tD_t. 10 end for 11 Update AW_A with the same AdamW rule as the baseline. 12 end for Algorithm 2 Distributed Periodic Muon Input: momentum shards Mt(p)\M_t^(p)\ on rank p, refresh flag ρt _t, and communication buckets ℬℓ\B_ \. 1 if ρt=1 _t=1 then // NS5 refresh path 2 Asynchronously all-gather the first momentum bucket. 3 for each bucket ℬℓB_ do 4 Wait for the current bucket and launch the next asynchronous all-gather. 5 Reconstruct full momenta, execute NS5, and slice the results back to local shards. 6 Release the full-matrix buffers of the current bucket. 7 end for 8 else // sharded RowNorm path 9 Compute local column-squared statistics for all tall matrices. 10 Pack the statistics and launch bucketed asynchronous sum all-reduces. 11 While communication is in flight, compute and finish local RowNorm updates for the remaining matrices. 12 Wait for all-reduce completion; take the square root and then apply clampmin(ϵ)clamp_ (ε). 13 Use the global column norms to finish RowNorm updates for tall matrices. 14 end if 15 Apply γ to the off-refresh RowNorm outputs only, and fuse shape scaling with the local parameter-shard updates. Appendix D Detailed Experimental Setup D.1 GPIC Dataset GPIC (Giant Permissive Image Corpus) (6) is an open image–text dataset designed for large-scale visual-generation research and contains approximately 28 trillion pixels. Its full split comprises 100M training examples, 200K validation examples, and 1M test examples. We train on GPIC-Full, use the validation set to monitor training, and perform generation evaluation on the held-out test set. GPIC applies vision-language-model-based quality and safety filtering and removes duplicate and near-duplicate images using SSCD visual features. After filtering and deduplication, approximately 101.3M images remain and are divided into 100M training, 200K validation, and 1M test examples. Qwen3-VL-4B-Instruct (4) generates captions at multiple levels of detail rather than relying on potentially noisy or missing web metadata. The training, validation, and test splits retain similar source and caption distributions. D.2 Model Configurations We use a latent space, dual stream MMDiT architecture (11). A frozen FLUX.1-schnell (19) VAE encodes images into the latent space, while frozen CLIP-L, CLIP-G (27), and T5-XXL (28) encoders provide text conditioning. The MMDiT backbone includes joint image–text attention, RoPE positional encoding, QK normalization, SwiGLU MLPs, and adaptive normalization. All four models use the same components and connectivity, and differ only in hidden dimension, depth, and number of attention heads. In the Muon and Periodic Row-wise Muon runs, the corresponding Muon update rule is applied to two-dimensional weight matrices within the Transformer blocks. Biases, normalization parameters, scalar parameters, and parameters outside the Transformer blocks are updated by AdamW. D.3 Optimizer and Training Hyperparameters All methods use decoupled weight decay. The AdamW baseline applies AdamW to all parameters. Muon and Periodic Row-wise Muon apply matrix-valued updates to two-dimensional hidden-layer weights and maintain a separate AdamW parameter group for all remaining parameters. All Muon experiments use five Newton–Schulz iterations with fixed coefficients (a,b,c)=(3.4445,−4.7750,2.0315)(a,b,c)=(3.4445,-4.7750,2.0315). At a given model scale, the three optimizers use the same model, training length, global batch size, and learning-rate schedule, and therefore process the same number of training examples. Because their optimizer computation and communication costs differ, they are not constrained to use the same wall-clock time or total compute. D.4 Generation Evaluation We evaluate a checkpoint every 5,000 training steps, covering the complete trajectory from 5,000 to 60,000 steps. We select a fixed set of 50,000 prompts from the GPIC test set and generate 50,000 images at 512×512512× 512 resolution. All models and optimizers use the same sampling configuration with a fixed classifier-free guidance scale of 5.0. Table 4: Model configurations. Muon-routed parameters denotes the fraction of trainable parameters updated by Muon in the Muon-based runs. Model Total params. Trainable params. Hidden dim. Blocks Heads Muon-routed 1.3B 1,255,586,099 1,255,585,795 1,216 19 19 98.69% 4B 3,950,236,099 3,950,235,715 1,920 24 30 99.20% 9B 9,374,837,315 9,374,836,803 2,560 32 40 99.48% 15B 14,848,237,251 14,848,236,611 2,880 40 45 99.60% Table 5: Optimizer hyperparameters. “Aux. AdamW” denotes the auxiliary AdamW parameter group used in the Muon-based runs. All methods use weight decay 0.010.01 and AdamW-updated parameter groups use ϵ=10−8ε=10^-8. Main learning rate Method 1.3B 4B 9B 15B AdamW 3.0×10−43.0×10^-4 2.0×10−42.0×10^-4 1.4×10−41.4×10^-4 1.2×10−41.2×10^-4 Muon 5.0×10−35.0×10^-3 3.0×10−33.0×10^-3 2.4×10−32.4×10^-3 2.0×10−32.0×10^-3 Periodic Row-wise Muon 5.0×10−35.0×10^-3 3.0×10−33.0×10^-3 2.4×10−32.4×10^-3 2.0×10−32.0×10^-3 Method Main momentum or betas Aux. AdamW Orthogonalization / schedule AdamW (0.9,0.95)(0.9,0.95) – – Muon 0.950.95 LR =1.0×10−4=1.0×10^-4; β=(0.9,0.95)β=(0.9,0.95) NS5 at every step Periodic Row-wise Muon 0.950.95 LR =1.0×10−4=1.0×10^-4; β=(0.9,0.95)β=(0.9,0.95) NS5 refresh every K=3K=3 steps; RowNorm otherwise (γ=0.15γ=0.15) Table 6: Training settings shared by all methods. Setting Value Image resolution 512×512512× 512 Training length 60,000 steps Local / global batch size 16 / 4,096 Learning-rate schedule 2k warmup, then cosine decay Hardware 32 nodes, 256 NVIDIA H100 GPUs Distributed training Pytorch FSDP2 with activation checkpointing We evaluate generated images using the following metrics: • FD-DINOv2 (32) computes the Fréchet distance between generated and real-image distributions in DINOv2 feature space. • FID (14) computes the Fréchet distance between generated and real-image distributions in Inception feature space. • Maximum Mean Discrepancy (MMD) provides a non-parametric measure of the discrepancy between generated and real feature distributions. • Precision and Density measure how closely generated samples align with the real-data manifold and primarily characterize sample fidelity. • Recall and Coverage measure how well the generated distribution covers the real data distribution and characterize generation diversity. • HPSv2.1 (35) measures text–image alignment and consistency with human preferences. • GenEval2 (16) evaluates the correctness of generated objects, attributes, relations, and compositional semantics. Unless otherwise specified, all main comparisons and ablations use the same generation and evaluation protocol. Appendix E Additional Experimental Results This section provides additional experimental results supporting the main comparisons. We first describe the selection of the refresh period K and the RowNorm scaling factor γ for Periodic Row-wise Muon. We then provide the complete training loss trajectories, checkpoint-level generation evaluations, and additional profiler traces. E.1 Selection of K and γ Periodic Row-wise Muon introduces two primary hyperparameters: the refresh period K, which controls the frequency of full NS5 spectral updates, and the RowNorm scaling factor γ, which controls the relative update magnitude of the RowNorm branch on non-refresh steps. To avoid tuning these hyperparameters separately at each model scale, we select K and γ once in a smaller development setting and directly transfer the same configuration to all 1.3B–15B models in the main experiments without additional scale-specific tuning. Specifically, we perform the hyperparameter selection experiments using the 1.3B model for 30k training steps on 2 H100 nodes, corresponding to 16 H100 GPUs in total, with a local batch size of 16. We otherwise retain the same model, optimizer hyperparameters, learning rate schedule, data pipeline, and training recipe used in the main experiments. We evaluate K∈2,3,4K∈\2,3,4\ and γ∈0.10,0.15,0.25,0.35γ∈\0.10,0.15,0.25,0.35\. To reduce sensitivity to noise at any single checkpoint, we use the mean FD-DINO over the 20k, 25k, and 30k checkpoints as the selection criterion. Figure 5 summarizes the results. For K=3K=3 cases, γ=0.15γ=0.15 achieves the lowest late-stage mean FD-DINO among the tested scales. We therefore use γ=0.15γ=0.15 when comparing different refresh periods. Increasing K from 2 to 3 reduces the number of NS5 refreshes by 33% while increasing the best mean FD-DINO by only 2.8%. In contrast, increasing K from 3 to 4 provides a further 25% reduction in NS5 refreshes but increases mean FD-DINO by 8.2%. We therefore select K=3K=3 and γ=0.15γ=0.15 as the default configuration, which provides a favorable trade-off between generation quality and spectral-update systems cost. Importantly, this selection procedure does not retune K or γ on the larger models. The same configuration is transferred directly to all model scales in the main experiments, so the observed scaling behavior does not result from scale-specific hyperparameter tuning. Figure 5: Selection of K and γ. (a) Late-stage FD-DINO, averaged over the 20k, 25k, and 30k checkpoints, across different K and γ. (b) The best mean FD-DINO over γ at each refresh period. Increasing K from 2 to 3 reduces the number of NS5 refreshes by 33%33\% with only a 2.8%2.8\% increase in FD-DINO, whereas increasing K from 3 to 4 saves a further 25%25\% but increases FD-DINO by 8.2%8.2\%. We therefore select K=3K=3 and γ=0.15γ=0.15. E.2 Training-Loss Trajectories Across Model Scales Figure 6 shows the complete training-loss trajectories of AdamW, Muon, and Periodic Row-wise Muon over 60k optimization steps for the 1.3B, 4B, 9B, and 15B models. These results complement the validation loss comparisons in the main text by providing a higher resolution view of the optimization dynamics throughout training. All three optimizers exhibit stable decreasing training loss across all model scales, with no evidence of training divergence. Muon generally achieves lower training loss throughout the main training regime, while Periodic Row-wise Muon closely tracks the Muon trajectory. This indicates that replacing two out of every three NS5 updates with the substantially cheaper RowNorm update does not destabilize optimization. As model size increases from 1.3B to 15B, the overall training loss level also decreases, consistent with the validation loss scaling behavior observed in the main experiments. Because minibatch training loss exhibits substantial stochastic variation, we do not use individual training loss values as proxies for generation quality. Generation performance is instead evaluated using FD-DINO, FID, and the additional generation metrics reported in the main text and in Appendix E.4. The purpose of Figure 6 is therefore to verify that Periodic Row-wise Muon remains stable over the complete 60k-step optimization trajectory rather than only at the sparse generation evaluation checkpoints. Figure 6: Training loss curves across model scales over 60,000 optimizer steps. E.3 Active Time Efficiency over AdamW Figure 7: Generation quality for AdamW and Periodic Row-wise Muon. Figure 7 directly compares Periodic Row-wise Muon with AdamW under active training time. Across all model sizes, Periodic Row-wise Muon achieves lower FD-DINO throughout most of the overlapping time range, indicating better generation quality for a comparable training time budget. It also consistently attains lower FD-DINO at the end of training. Thus, the advantage over AdamW holds in both training time efficiency and the final generation quality reached by the model. E.4 Complete Checkpoint Evaluation Results The main text reports the most important final checkpoint metrics and the best observed FD-DINO to keep the presentation compact. Here we provide the complete checkpoint-level generation evaluations for all 4 model scales in Tables 7–10. For each model, we evaluate checkpoints every 5k steps from 5k to 60k using the same evaluation protocol for AdamW, Muon, and Periodic Row-wise Muon. At every checkpoint, we report FD-DINO, FID, MMD-DINO, Precision, Recall, Coverage, Density, HPSv2, and both the arithmetic and geometric means of GenEval2. These tables therefore complement the fidelity metrics emphasized in the main text with the complete evolution of diversity, alignment, and compositionality throughout training. The full trajectories show that both Muon and Periodic Row-wise Muon achieve substantial improvements over AdamW at multiple stages of training, while the relative ordering of the two Muon variants exhibits some checkpoint-level variation. In particular, the checkpoint that optimizes one generation metric does not necessarily optimize the others. We therefore report the complete metric profile at the final checkpoint in the main text and treat the best-observed FD-DINO separately, rather than constructing an artificial “best” model by selecting different checkpoints for different metrics. These trajectories also provide additional context for the generation quality frontiers in the main text. Different optimizers enter their best quality regimes at different points in training, and Periodic Row-wise Muon typically reaches a generation quality regime comparable to vanilla Muon with substantially less normalized active training time. E.5 Additional Profiler Traces Across Model Scales The main text uses the 15B model to illustrate the systems behavior of Periodic Row-wise Muon. Figure 8 provides the corresponding profiler traces for the 1.3B, 4B, and 9B models, allowing us to verify whether the same execution pattern persists across model scales. All profiler traces use the same global time normalization as the systems measurements in the main text, meaning that the mean step time of the 1.3B AdamW run is defined as one time unit. The traces separately visualize the non-optimizer phase, optimizer-specific CPU phases, GPU computation, and communication activity for vanilla Muon and Periodic Row-wise Muon. A consistent execution pattern is observed at all three scales. Vanilla Muon performs a full NS5 update at every optimization step, resulting in a long spectral computation phase together with the corresponding optimizer communication. Periodic Row-wise Muon instead retains only one refresh step per K=3K=3 period, while the two intervening steps use the much shorter RowNorm path. On refresh steps, bucketed all-gather overlaps communication for upcoming matrices with NS5 computation on matrices that are already available. On RowNorm steps, the much smaller communication of norm statistics is similarly overlapped with local computation. Over the three consecutive steps shown in Figure 8, the normalized time decreases from 5.59 for vanilla Muon to 4.20 for Periodic Row-wise Muon on the 1.3B model, from 7.26 to 5.68 on the 4B model, and from 12.47 to 10.26 on the 9B model. The 15B trace presented in the main text exhibits the same qualitative behavior. The consistency of these profiler traces across model scales indicates that the systems gains do not arise from a single model shape, but from the combined effects of periodic spectral refresh, sharded RowNorm, and communication–computation overlap. Table 7: All checkpoint evaluation metrics for the 1.3B model. Optimizer Step Fidelity Diversity Alignment Compositionality FD-DINO↓ FID↓ MMD-DINO↓ Precision↑ Recall↑ Coverage↑ Density↑ HPSv2↑ GenEval2 AM↑ GenEval2 GM↑ AdamW 5k 266.40266.40 9.949.94 0.2580.258 0.8610.861 0.3610.361 0.6570.657 0.9000.900 15.7915.79 17.1917.19 1.351.35 10k 125.51125.51 7.577.57 0.0940.094 0.9080.908 0.6810.681 0.8420.842 0.9470.947 15.9315.93 22.2422.24 2.772.77 15k 82.9382.93 6.626.62 0.0510.051 0.9150.915 0.7890.789 0.8800.880 0.9760.976 18.0918.09 28.5328.53 4.004.00 20k 86.8486.84 7.947.94 0.0550.055 0.9250.925 0.8120.812 0.8900.890 1.1071.107 17.7117.71 29.8329.83 4.194.19 25k 66.3066.30 6.586.58 0.0370.037 0.9290.929 0.8490.849 0.9210.921 0.9990.999 17.5517.55 30.9630.96 4.754.75 30k 60.3460.34 6.896.89 0.0340.034 0.9410.941 0.8590.859 0.9290.929 1.0651.065 19.5619.56 36.9236.92 6.716.71 35k 53.4353.43 6.336.33 0.0290.029 0.9380.938 0.8740.874 0.9340.934 1.0191.019 19.9219.92 39.3739.37 8.618.61 40k 52.6052.60 6.306.30 0.0300.030 0.9470.947 0.8790.879 0.9470.947 1.0181.018 19.8219.82 37.6037.60 7.167.16 45k 53.9253.92 6.536.53 0.0300.030 0.9430.943 0.8810.881 0.9400.940 0.9980.998 19.7219.72 37.9737.97 7.577.57 50k 51.5651.56 6.686.68 0.0280.028 0.9470.947 0.8870.887 0.9400.940 1.0191.019 20.6120.61 38.1638.16 8.018.01 55k 53.9753.97 6.616.61 0.0310.031 0.9410.941 0.8860.886 0.9410.941 1.0181.018 20.4620.46 37.6437.64 6.016.01 60k 53.9353.93 6.716.71 0.0310.031 0.9410.941 0.8870.887 0.9400.940 0.9660.966 20.5320.53 38.6638.66 7.107.10 Muon 5k 152.03152.03 7.687.68 0.1260.126 0.8820.882 0.6330.633 0.7780.778 0.8820.882 16.6416.64 20.2820.28 2.232.23 10k 73.3573.35 6.266.26 0.0400.040 0.9210.921 0.8140.814 0.8890.889 0.9390.939 18.7918.79 30.5730.57 4.824.82 15k 51.5651.56 5.735.73 0.0240.024 0.9430.943 0.8660.866 0.9200.920 0.9820.982 19.6919.69 38.3338.33 7.457.45 20k 48.0648.06 6.166.16 0.0210.021 0.9480.948 0.8700.870 0.9300.930 0.9930.993 19.7919.79 43.6343.63 9.649.64 25k 46.1246.12 6.506.50 0.0220.022 0.9470.947 0.8840.884 0.9320.932 0.9800.980 20.4320.43 40.0040.00 7.657.65 30k 52.8652.86 6.846.84 0.0290.029 0.9420.942 0.8850.885 0.9380.938 0.9750.975 20.6620.66 45.2045.20 8.058.05 35k 46.6546.65 6.266.26 0.0220.022 0.9440.944 0.8960.896 0.9430.943 0.9910.991 21.1421.14 46.5746.57 10.7710.77 40k 41.7341.73 6.016.01 0.0210.021 0.9490.949 0.9010.901 0.9470.947 0.9840.984 20.9120.91 46.7346.73 11.3611.36 45k 48.2648.26 6.566.56 0.0270.027 0.9430.943 0.8970.897 0.9450.945 0.9910.991 20.2920.29 36.2736.27 5.455.45 50k 50.4150.41 6.746.74 0.0300.030 0.9450.945 0.8920.892 0.9540.954 0.9940.994 20.4120.41 45.3245.32 8.988.98 55k 47.9247.92 6.656.65 0.0280.028 0.9450.945 0.9000.900 0.9500.950 0.9850.985 20.8120.81 44.9144.91 9.229.22 60k 46.0846.08 6.546.54 0.0270.027 0.9490.949 0.9070.907 0.9490.949 0.9920.992 20.8420.84 44.7244.72 9.389.38 Periodic Row-wise Muon 5k 172.46172.46 8.238.23 0.1450.145 0.8850.885 0.5690.569 0.7660.766 0.9210.921 16.4516.45 19.4619.46 1.671.67 10k 82.2982.29 6.746.74 0.0490.049 0.9230.923 0.8080.808 0.8800.880 0.9180.918 17.5717.57 28.5028.50 4.804.80 15k 58.9658.96 6.066.06 0.0280.028 0.9280.928 0.8580.858 0.9160.916 0.9690.969 18.4518.45 36.4936.49 6.036.03 20k 67.2267.22 6.976.97 0.0390.039 0.9390.939 0.8560.856 0.9160.916 1.0001.000 18.4618.46 34.7434.74 5.835.83 25k 47.3747.37 6.186.18 0.0210.021 0.9380.938 0.8760.876 0.9380.938 0.9970.997 19.3619.36 36.9536.95 7.247.24 30k 44.7744.77 6.116.11 0.0210.021 0.9400.940 0.8910.891 0.9320.932 0.9360.936 21.1021.10 42.7342.73 8.928.92 35k 41.9141.91 5.535.53 0.0190.019 0.9440.944 0.9020.902 0.9440.944 0.9810.981 20.7320.73 47.7147.71 10.5310.53 40k 47.2947.29 6.636.63 0.0280.028 0.9460.946 0.8870.887 0.9380.938 0.9710.971 21.4021.40 45.6845.68 11.3611.36 45k 49.7749.77 6.376.37 0.0300.030 0.9470.947 0.8940.894 0.9440.944 0.9810.981 20.1820.18 45.9745.97 10.2110.21 50k 45.0145.01 6.136.13 0.0260.026 0.9510.951 0.8930.893 0.9490.949 0.9960.996 20.3120.31 47.4047.40 11.2511.25 55k 45.7545.75 6.406.40 0.0270.027 0.9490.949 0.8960.896 0.9510.951 1.0051.005 20.6920.69 47.9347.93 11.7111.71 60k 45.6545.65 6.296.29 0.0270.027 0.9520.952 0.9060.906 0.9490.949 0.9760.976 20.3620.36 45.8945.89 9.759.75 Table 8: All checkpoint evaluation metrics for the 4B model. Optimizer Step Fidelity Diversity Alignment Compositionality FD-DINO↓ FID↓ MMD-DINO↓ Precision↑ Recall↑ Coverage↑ Density↑ HPSv2↑ GenEval2 AM↑ GenEval2 GM↑ AdamW 5k 223.73223.73 9.179.17 0.1990.199 0.8700.870 0.4530.453 0.7000.700 0.9010.901 15.9015.90 18.6218.62 1.781.78 10k 101.76101.76 8.428.42 0.0660.066 0.9080.908 0.7460.746 0.8570.857 0.9500.950 18.1718.17 22.0122.01 2.262.26 15k 72.1672.16 7.677.67 0.0360.036 0.9220.922 0.8340.834 0.8850.885 0.9580.958 18.3118.31 34.1834.18 6.516.51 20k 51.8751.87 6.876.87 0.0230.023 0.9410.941 0.8660.866 0.9260.926 0.9980.998 20.7320.73 37.0937.09 5.765.76 25k 50.9450.94 6.396.39 0.0230.023 0.9380.938 0.8790.879 0.9300.930 1.0071.007 19.3619.36 32.8532.85 5.715.71 30k 48.7148.71 6.486.48 0.0220.022 0.9400.940 0.8880.888 0.9360.936 1.0261.026 20.2420.24 38.0938.09 7.747.74 35k 47.2847.28 6.806.80 0.0240.024 0.9460.946 0.8880.888 0.9440.944 1.0141.014 21.6021.60 40.0140.01 7.507.50 40k 56.9756.97 7.907.90 0.0290.029 0.9330.933 0.8990.899 0.9370.937 0.9720.972 20.6320.63 41.7441.74 8.898.89 45k 61.6161.61 8.728.72 0.0330.033 0.9440.944 0.8970.897 0.9350.935 1.0201.020 21.3121.31 45.9045.90 10.6810.68 50k 58.0458.04 8.288.28 0.0300.030 0.9390.939 0.8990.899 0.9410.941 0.9940.994 20.7220.72 44.2644.26 9.729.72 55k 57.3557.35 8.018.01 0.0300.030 0.9370.937 0.9020.902 0.9460.946 0.9820.982 20.8020.80 44.0344.03 9.709.70 60k 51.3951.39 7.357.35 0.0270.027 0.9370.937 0.8990.899 0.9390.939 0.9900.990 21.4021.40 45.7045.70 9.139.13 Muon 5k 149.32149.32 7.827.82 0.1190.119 0.8880.888 0.6310.631 0.8040.804 0.8940.894 16.7316.73 19.5419.54 2.312.31 10k 65.8865.88 6.646.64 0.0360.036 0.9290.929 0.8330.833 0.9040.904 0.9510.951 18.2718.27 27.7327.73 4.124.12 15k 56.0856.08 6.506.50 0.0300.030 0.9390.939 0.8690.869 0.9290.929 0.9740.974 20.3920.39 40.9940.99 7.987.98 20k 40.1540.15 6.256.25 0.0150.015 0.9410.941 0.9030.903 0.9350.935 0.9520.952 20.8120.81 41.1441.14 8.468.46 25k 39.2739.27 5.325.32 0.0180.018 0.9450.945 0.9020.902 0.9430.943 0.9670.967 21.0021.00 46.2046.20 10.4110.41 30k 40.5040.50 5.945.94 0.0180.018 0.9410.941 0.9040.904 0.9380.938 0.9590.959 20.8120.81 46.5346.53 9.529.52 35k 41.9641.96 5.665.66 0.0200.020 0.9460.946 0.9050.905 0.9470.947 0.9830.983 19.9119.91 43.4543.45 7.417.41 40k 55.8255.82 6.996.99 0.0330.033 0.9350.935 0.9100.910 0.9360.936 0.9480.948 18.9418.94 39.7739.77 6.366.36 45k 44.3944.39 5.865.86 0.0250.025 0.9400.940 0.9130.913 0.9460.946 0.9630.963 20.7920.79 46.7546.75 8.188.18 50k 46.2446.24 6.136.13 0.0270.027 0.9460.946 0.9140.914 0.9460.946 0.9670.967 21.1321.13 46.4246.42 8.838.83 55k 42.0942.09 6.146.14 0.0220.022 0.9460.946 0.9120.912 0.9410.941 0.9660.966 21.1321.13 50.4050.40 10.1910.19 60k 41.6241.62 6.086.08 0.0240.024 0.9390.939 0.9140.914 0.9430.943 0.9640.964 20.6620.66 46.5846.58 9.069.06 Periodic Row-wise Muon 5k 159.06159.06 9.049.04 0.1250.125 0.8920.892 0.5940.594 0.7990.799 0.9690.969 15.6115.61 18.6818.68 2.052.05 10k 72.5972.59 7.207.20 0.0400.040 0.9290.929 0.8200.820 0.8880.888 0.9760.976 18.9818.98 28.4228.42 3.853.85 15k 52.1952.19 6.356.35 0.0220.022 0.9290.929 0.8760.876 0.9170.917 0.9460.946 18.2518.25 35.5235.52 6.206.20 20k 39.4239.42 5.615.61 0.0150.015 0.9490.949 0.8990.899 0.9480.948 0.9840.984 21.0521.05 42.9842.98 8.168.16 25k 44.5844.58 6.276.27 0.0210.021 0.9350.935 0.9010.901 0.9310.931 0.9370.937 20.8720.87 38.6338.63 7.197.19 30k 39.8039.80 6.156.15 0.0180.018 0.9510.951 0.9090.909 0.9430.943 0.9570.957 20.2320.23 39.5339.53 6.646.64 35k 41.9141.91 6.326.32 0.0200.020 0.9410.941 0.9100.910 0.9410.941 0.9650.965 20.8320.83 40.8740.87 6.956.95 40k 43.9343.93 6.646.64 0.0200.020 0.9380.938 0.9090.909 0.9420.942 0.9780.978 20.4720.47 43.9043.90 8.098.09 45k 62.8962.89 8.688.68 0.0320.032 0.9330.933 0.8980.898 0.9420.942 1.0921.092 19.5219.52 43.2343.23 8.138.13 50k 52.3652.36 7.307.30 0.0300.030 0.9360.936 0.9160.916 0.9420.942 0.9390.939 20.5420.54 49.3249.32 10.8610.86 55k 43.7043.70 6.426.42 0.0230.023 0.9410.941 0.9110.911 0.9500.950 0.9860.986 21.0021.00 49.4149.41 11.4411.44 60k 42.2542.25 6.356.35 0.0220.022 0.9410.941 0.9060.906 0.9520.952 0.9840.984 21.3421.34 49.3849.38 10.5810.58 Table 9: All checkpoint evaluation metrics for the 9B model. Optimizer Step Fidelity Diversity Alignment Compositionality FD-DINO↓ FID↓ MMD-DINO↓ Precision↑ Recall↑ Coverage↑ Density↑ HPSv2↑ GenEval2 AM↑ GenEval2 GM↑ AdamW 5k 184.83184.83 8.378.37 0.1500.150 0.8830.883 0.5310.531 0.7570.757 0.9430.943 16.0816.08 19.1819.18 1.751.75 10k 78.0578.05 6.056.05 0.0430.043 0.9240.924 0.7940.794 0.8900.890 0.9910.991 19.3219.32 27.0127.01 3.413.41 15k 59.3059.30 7.467.46 0.0280.028 0.9400.940 0.8570.857 0.9160.916 0.9660.966 20.6520.65 35.7635.76 6.146.14 20k 43.2043.20 6.276.27 0.0160.016 0.9440.944 0.8810.881 0.9330.933 0.9670.967 20.5820.58 41.2841.28 8.238.23 25k 47.6747.67 6.436.43 0.0190.019 0.9440.944 0.8790.879 0.9400.940 1.0201.020 20.2120.21 40.9640.96 8.468.46 30k 44.4244.42 6.466.46 0.0210.021 0.9450.945 0.8950.895 0.9460.946 1.0121.012 20.3820.38 40.9340.93 8.098.09 35k 44.7244.72 6.586.58 0.0220.022 0.9420.942 0.9020.902 0.9420.942 0.9790.979 21.7021.70 47.6747.67 10.1610.16 40k 42.2142.21 6.446.44 0.0210.021 0.9440.944 0.9050.905 0.9490.949 0.9750.975 20.8220.82 44.3344.33 9.959.95 45k 48.4348.43 7.127.12 0.0250.025 0.9350.935 0.9020.902 0.9410.941 0.9700.970 21.1821.18 44.5144.51 9.409.40 50k 45.5445.54 6.776.77 0.0250.025 0.9420.942 0.9050.905 0.9420.942 0.9580.958 20.5420.54 42.4542.45 8.898.89 55k 41.0141.01 6.186.18 0.0210.021 0.9460.946 0.9110.911 0.9420.942 0.9570.957 21.1821.18 45.6245.62 10.3310.33 60k 41.2641.26 6.196.19 0.0220.022 0.9460.946 0.9040.904 0.9500.950 0.9870.987 21.3921.39 45.9345.93 9.669.66 Muon 5k 117.21117.21 6.486.48 0.0850.085 0.9050.905 0.7060.706 0.8360.836 0.9370.937 16.8216.82 22.7622.76 2.212.21 10k 47.6447.64 5.515.51 0.0190.019 0.9420.942 0.8700.870 0.9240.924 0.9810.981 20.3920.39 37.7037.70 8.338.33 15k 40.0040.00 5.775.77 0.0170.017 0.9370.937 0.8970.897 0.9380.938 0.9580.958 21.9721.97 46.5246.52 11.8211.82 20k 38.5738.57 6.466.46 0.0180.018 0.9470.947 0.9030.903 0.9420.942 0.9520.952 21.9321.93 53.0053.00 14.1714.17 25k 38.9738.97 5.955.95 0.0180.018 0.9430.943 0.9110.911 0.9430.943 0.9740.974 20.4920.49 49.8149.81 12.0212.02 30k 39.0739.07 6.136.13 0.0210.021 0.9450.945 0.9110.911 0.9450.945 0.9520.952 21.3821.38 50.1650.16 10.0410.04 35k 39.3039.30 5.995.99 0.0200.020 0.9500.950 0.9100.910 0.9410.941 0.9660.966 22.2722.27 54.6554.65 14.1414.14 40k 35.9235.92 5.795.79 0.0190.019 0.9440.944 0.9150.915 0.9440.944 0.9510.951 22.0322.03 47.4947.49 10.3010.30 45k 36.2936.29 5.785.78 0.0200.020 0.9500.950 0.9180.918 0.9490.949 0.9670.967 21.9821.98 52.2152.21 11.9211.92 50k 34.5334.53 5.715.71 0.0200.020 0.9480.948 0.9220.922 0.9470.947 0.9470.947 22.8722.87 53.9853.98 12.8612.86 55k 34.4134.41 5.645.64 0.0200.020 0.9500.950 0.9200.920 0.9490.949 0.9640.964 22.3622.36 51.5551.55 11.3411.34 60k 33.9733.97 5.615.61 0.0200.020 0.9520.952 0.9270.927 0.9530.953 0.9600.960 22.6022.60 51.7751.77 11.3511.35 Periodic Row-wise Muon 5k 143.77143.77 8.718.71 0.1080.108 0.9030.903 0.6300.630 0.8240.824 0.9600.960 15.8315.83 19.3819.38 2.292.29 10k 60.2960.29 6.706.70 0.0280.028 0.9230.923 0.8500.850 0.9140.914 0.9460.946 18.5418.54 29.5329.53 4.364.36 15k 48.1748.17 6.256.25 0.0210.021 0.9370.937 0.8840.884 0.9310.931 0.9450.945 20.4620.46 40.0940.09 8.108.10 20k 38.1438.14 5.725.72 0.0140.014 0.9470.947 0.8960.896 0.9350.935 0.9460.946 22.1222.12 45.3945.39 9.829.82 25k 32.4432.44 5.635.63 0.0110.011 0.9430.943 0.9070.907 0.9470.947 0.9540.954 21.8321.83 51.2551.25 11.7511.75 30k 38.8138.81 6.176.17 0.0170.017 0.9500.950 0.9040.904 0.9450.945 0.9700.970 20.9520.95 49.4649.46 10.2810.28 35k 34.5734.57 5.635.63 0.0160.016 0.9520.952 0.9180.918 0.9480.948 0.9790.979 22.0822.08 50.7350.73 10.8910.89 40k 37.4737.47 5.895.89 0.0190.019 0.9500.950 0.9110.911 0.9510.951 0.9640.964 22.3422.34 54.9754.97 13.5013.50 45k 36.8636.86 6.096.09 0.0200.020 0.9460.946 0.9160.916 0.9460.946 0.9650.965 22.8222.82 57.3557.35 15.3515.35 50k 35.3735.37 5.855.85 0.0190.019 0.9520.952 0.9120.912 0.9520.952 0.9720.972 22.5722.57 56.1656.16 13.4413.44 55k 38.1938.19 6.396.39 0.0200.020 0.9430.943 0.9210.921 0.9540.954 0.9670.967 22.4722.47 55.5255.52 12.1812.18 60k 36.7036.70 6.136.13 0.0190.019 0.9490.949 0.9130.913 0.9510.951 0.9790.979 22.4422.44 57.3357.33 15.9715.97 Table 10: All checkpoint evaluation metrics for the 15B model. Optimizer Step Fidelity Diversity Alignment Compositionality FD-DINO↓ FID↓ MMD-DINO↓ Precision↑ Recall↑ Coverage↑ Density↑ HPSv2↑ GenEval2 AM↑ GenEval2 GM↑ AdamW 5k 175.31175.31 8.928.92 0.1360.136 0.8790.879 0.5610.561 0.7530.753 0.9030.903 16.0316.03 16.8116.81 1.501.50 10k 74.8474.84 7.707.70 0.0350.035 0.9270.927 0.8050.805 0.8950.895 0.9700.970 19.0119.01 25.7525.75 3.083.08 15k 53.2153.21 6.426.42 0.0230.023 0.9390.939 0.8560.856 0.9200.920 0.9700.970 20.8220.82 32.8632.86 5.355.35 20k 56.2156.21 7.037.03 0.0280.028 0.9350.935 0.8700.870 0.9250.925 0.9870.987 18.6618.66 35.5835.58 5.705.70 25k 40.7540.75 5.865.86 0.0170.017 0.9430.943 0.8940.894 0.9420.942 0.9850.985 21.9921.99 40.2740.27 7.447.44 30k 39.5939.59 6.156.15 0.0170.017 0.9470.947 0.8900.890 0.9420.942 0.9990.999 21.9821.98 42.0342.03 7.877.87 35k 38.4638.46 5.865.86 0.0170.017 0.9470.947 0.8940.894 0.9490.949 1.0071.007 21.3021.30 45.3145.31 9.439.43 40k 41.3341.33 6.376.37 0.0220.022 0.9320.932 0.9100.910 0.9400.940 0.9370.937 22.1322.13 47.1047.10 11.1111.11 45k 38.1738.17 6.006.00 0.0180.018 0.9490.949 0.9060.906 0.9520.952 1.0061.006 21.7221.72 46.7746.77 8.968.96 50k 40.4840.48 6.256.25 0.0210.021 0.9500.950 0.9140.914 0.9500.950 0.9810.981 22.1822.18 47.4947.49 8.778.77 55k 40.1340.13 6.286.28 0.0210.021 0.9510.951 0.9050.905 0.9540.954 0.9750.975 22.2122.21 47.2147.21 9.559.55 60k 40.2340.23 6.246.24 0.0220.022 0.9450.945 0.9090.909 0.9470.947 0.9640.964 21.9921.99 47.3447.34 9.189.18 Muon 5k 119.42119.42 7.837.83 0.0880.088 0.9030.903 0.6940.694 0.8460.846 0.9230.923 17.1617.16 21.9621.96 2.262.26 10k 54.4954.49 6.356.35 0.0250.025 0.9360.936 0.8650.865 0.9280.928 0.9650.965 20.2320.23 33.8933.89 5.725.72 15k 37.8937.89 5.825.82 0.0150.015 0.9420.942 0.8980.898 0.9410.941 0.9510.951 22.1022.10 43.3543.35 8.738.73 20k 37.2437.24 5.415.41 0.0160.016 0.9440.944 0.9090.909 0.9400.940 0.9470.947 21.9621.96 46.3846.38 9.819.81 25k 33.4833.48 5.155.15 0.0140.014 0.9470.947 0.9140.914 0.9450.945 0.9520.952 22.9222.92 54.3054.30 14.2114.21 30k 37.1337.13 5.845.84 0.0180.018 0.9420.942 0.9150.915 0.9470.947 0.9370.937 20.9320.93 52.7252.72 11.4411.44 35k 35.9235.92 5.925.92 0.0180.018 0.9430.943 0.9170.917 0.9500.950 0.9610.961 20.8720.87 52.6752.67 10.6710.67 40k 33.5533.55 5.375.37 0.0180.018 0.9520.952 0.9240.924 0.9540.954 0.9520.952 22.8822.88 54.6654.66 12.1512.15 45k 33.2533.25 5.195.19 0.0180.018 0.9450.945 0.9210.921 0.9500.950 0.9480.948 22.5322.53 57.5057.50 13.7613.76 50k 34.5434.54 5.745.74 0.0200.020 0.9410.941 0.9250.925 0.9530.953 0.9340.934 22.6722.67 58.1858.18 14.1114.11 55k 33.4733.47 5.625.62 0.0190.019 0.9460.946 0.9210.921 0.9490.949 0.9490.949 22.4322.43 58.5358.53 13.8413.84 60k 33.5133.51 5.575.57 0.0190.019 0.9450.945 0.9300.930 0.9500.950 0.9370.937 22.3722.37 57.9357.93 14.5514.55 Periodic Row-wise Muon 5k 137.51137.51 7.687.68 0.1010.101 0.8990.899 0.6410.641 0.8170.817 0.9170.917 17.4917.49 19.1519.15 2.612.61 10k 56.1856.18 6.376.37 0.0220.022 0.9360.936 0.8510.851 0.9180.918 0.9710.971 19.3719.37 26.8726.87 3.443.44 15k 45.1445.14 6.646.64 0.0190.019 0.9390.939 0.8850.885 0.9380.938 0.9560.956 21.8621.86 34.7634.76 5.735.73 20k 40.1340.13 6.346.34 0.0170.017 0.9410.941 0.9010.901 0.9390.939 0.9630.963 21.8121.81 39.1239.12 6.936.93 25k 32.3532.35 5.615.61 0.0120.012 0.9490.949 0.9120.912 0.9470.947 0.9660.966 22.9022.90 49.6849.68 11.7411.74 30k 35.1235.12 5.585.58 0.0160.016 0.9490.949 0.9070.907 0.9530.953 0.9940.994 20.5920.59 47.5547.55 10.7010.70 35k 36.9136.91 6.196.19 0.0170.017 0.9440.944 0.9140.914 0.9430.943 0.9590.959 19.9419.94 48.7148.71 9.459.45 40k 38.7238.72 6.026.02 0.0210.021 0.9430.943 0.9160.916 0.9450.945 0.9420.942 20.4320.43 45.8945.89 9.199.19 45k 34.3734.37 5.605.60 0.0180.018 0.9410.941 0.9190.919 0.9510.951 0.9540.954 21.0821.08 53.1053.10 12.0812.08 50k 52.2552.25 7.867.86 0.0390.039 0.9440.944 0.9160.916 0.9410.941 0.9230.923 19.6419.64 44.6444.64 8.988.98 55k 40.2540.25 6.386.38 0.0260.026 0.9420.942 0.9150.915 0.9490.949 0.9450.945 20.8020.80 47.8047.80 10.1210.12 60k 33.9933.99 5.635.63 0.0200.020 0.9480.948 0.9250.925 0.9550.955 0.9670.967 21.3521.35 52.2652.26 11.9711.97 (a) 1.3B model. (b) 4B model. (c) 9B model. Figure 8: Simplified profiler traces for the 1.3B, 4B, and 9B models. The corresponding 15B result is shown in Figure 4.