Paper deep dive
Hyperball May Not Be a Free Lunch
Yihao Xiao, Jialong Sun, Zitian Gao, Zeming Wei, Chutian Wang, Ran Tao, Jiaye Teng, Bryan Dai
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates. However, the source of their advantage remains unclear. Starting from the angular displacement between consecutive parameter states, we derive an angular effective learning rate that accounts for the parameter-update angle, parameter norm, and update norm. We also show that the conventional norm-based measure is a special case under parameter-update orthogonality. We then decompose optimizer updates into radial and tangential components and analyze how radial updates affect one-step angular displacement. Under the training configurations considered, numerical results show that the radial component has only a limited direct effect on the angular effective learning rate. It therefore cannot explain why MuonH converges more slowly than MuonWD early in training but overtakes it later. To further isolate the underlying mechanism, we devise a heuristic experiment that modifies only the learning-rate schedule so that the dynamics of each optimizer reproduce those of the other. The results suggest that their main difference stems from the evolution of the effective step size rather than an intrinsically superior update direction induced by Hyperball. Our pretraining experiments further show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair its later performance. Thus, maintaining a constant angular velocity does not eliminate the learning-rate-scheduling problem; careful scheduling remains essential to realizing the potential of Hyperball-style optimizers. Our code is publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.22444v1
- Canonical: https://arxiv.org/abs/2607.22444v1
Trouble viewing inline? Open PDF directly โ
Full Text
38,078 characters extracted from source content.
Expand or collapse full text
Hyperball May Not Be a Free Lunch Yihao Xiao 1,5,โ , Jialong Sun 4,โ , Zitian Gao 1 , Zeming Wei 2 , Chutian Wang 3 , Ran Tao 1 , Jiaye Teng 5 , Bryan Dai 1,* 1 IQuest Research; 2 Peking University; 3 Sun Yat-sen University; 4 Shenzhen University of Advanced Technology; 5 Shanghai University of Finance and Economics โ Equal contribution; * Corresponding author Abstract For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates. However, the source of their advantage remains unclear. Starting from the angular displacement between consecutive parameter states, we derive an angular effective learning rate that accounts for the parameter-update angle, parameter norm, and update norm. We also show that the conventional norm-based measure is a special case under parameter-update orthogonality. We then decompose optimizer updates into radial and tangential components and analyze how radial updates affect one-step angular displacement. Under the training configurations considered, numerical results show that the radial component has only a limited direct effect on the angular effective learning rate. It therefore cannot explain why MuonH converges more slowly than MuonWD early in training but overtakes it later. To further isolate the underlying mechanism, we devise a heuristic experiment that modifies only the learning-rate schedule so that the dynamics of each optimizer reproduce those of the other. The results suggest that their main difference stems from the evolution of the effective step size rather than an intrinsically superior update direction induced by Hyperball. Our pretraining experiments further show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair its later performance. Thus, maintaining a constant angular velocity does not eliminate the learning-rate-scheduling problem; careful scheduling remains essential to realizing the potential of Hyperball-style optimizers. Our code is publicly available athttps: //github.com/mangocrazz/hyperball-may-not-be-a-free-lunch. 1. Introduction In deep networks with scale-invariant structure, parameter norms, learning rates, and opti- mizer update directions jointly determine how the model actually moves through parameter space [Wan et al., 2020, Bergsma et al., 2026, Wu and He, 2018]. Recent optimizers with matrix constraints, such as MuonH [Li et al., 2025, Cesista, 2025, Deng et al., 2026, Wen et al., 2026], restrict matrix-valued parameters to norm spheres with prescribed radii in an effort to reduce the influence of parameter-scale variation on training dynamics. Although these methods have shown promising empirical performance in large-scale pretraining, the mechanism underlying arXiv:2607.22444v1 [cs.LG] 24 Jul 2026 their advantage remains unclear. In particular, it is not yet known whether the improvement arises from a more favorable update direction, from suppressing radial updates, or from a constraint-induced change in step size. A common interpretation is that weight decay or norm constraints primarily control the effective learning rate [Van Laarhoven, 2017, Hoffer et al., 2018, Zhang et al., 2018]: as a param- eter norm grows, scale invariance reduces the gradient magnitude relative to the parameter magnitude, thereby decreasing the effective update size. This interpretation, however, typically treats the effective learning rate mainly as a function of the parameter norm and implicitly assumes that the parameter and the optimizer update are approximately orthogonal. That assumption need not hold for modern optimizers that use momentum, preconditioning, or matrix-valued update rules. In such cases, the angle between the parameter and the update, the norm of the update, and the evolution of both quantities can all affect the actual angular displacement. Controlling the parameter norm alone therefore does not identify how Hyperball changes the optimization trajectory. We study a representative empirical phenomenon in Hyperball training. Relative to its non-Hyperball counterpart, a Hyperball optimizer can exhibit distinctly phase-dependent behavior: it underperforms early in training but overtakes the baseline in the middle or later stages. This behavior is not fully explained by the simple claim that a fixed norm yields a stable effective learning rate. If Hyperball acts mainly by suppressing radial updates, why should that suppression produce a phase transition rather than a monotone difference in training? Conversely, if its principal effect is to alter the effective step size, can the same dynamics be reproduced solely by changing the learning-rate schedule? These questions are the central focus of this work. To answer these questions, we characterize effective learning rates through the angular displacement induced by each parameter update. Specifically, we incorporate the angle be- tween the parameter and the update into the analysis, derive an angular effective learning rate for a general update rule, and recover the familiar norm-based expression as a special-case approximation. We then separately examine the direct contribution of radial updates to the angular effective learning rate and the joint effect of Hyperballโs constraints on update scale and parameter norm. Finally, we construct a state-dependent learning-rate matching procedure that aligns the angular effective learning rate of a non-Hyperball optimizer with that of its Hyperball counterpart at every step. This procedure tests the extent to which the training dynamics of the two optimizer families can reproduce one another. Overall, our contributions are summarized as follows: โข Conceptually, we derive an angular effective learning rate that explicitly accounts for the angle between the parameter and the update, and we state the conditions under which the conventional effective-learning-rate approximation is valid. โขTheoretically, we examine the role of radial updates through controlled numerical analy- ses. Under the training configurations considered in this work, their direct effect on the angular effective learning rate is relatively small. โข Practically, we propose a state-dependent learning-rate alignment method for isolating the dynamical difference between Hyperball and non-Hyperball optimizers. Our experiments show that part of Hyperballโs phase-dependent training behavior can be reproduced by aligning effective learning rates. 2 2. Preliminaries: Effective Learning Rate and Angular Updates This section introduces the update rule, notation, and geometric relations used in the subsequent analysis. Rather than characterizing an optimizerโs effective step solely through the parameter norm, we explicitly account for the relative direction of the parameter and the update. This perspective shows that, once the update is not orthogonal to the parameter, the angular step depends not only on the learning rate and parameter norm but also on the update norm and the angle between the two. 2.1. Generic update rule and notation Consider an arbitrary matrix-valued parameter blockํ ํก , with each block analyzed indepen- dently. Letํ ํก denote the update produced by the optimizer at stepํก. This quantity includes momentum, preconditioning, orthogonalization, and any other transformation internal to the optimizer, but excludes the external learning-rate multiplier. A generic update with decoupled weight decay can be written as ํ ํก+1 = ํผ ํก ํ ํก โ ํ ํก ํ ํก , ํผ ํก = 1โ ํ ํก ํ ํก ,(1) whereํ ํก >0 is the learning rate andํ ํก โฅ0 is the weight-decay coefficient. For convenience, define the cosine similarity between ํ ํก and ํ ํก as ํ ํก = โจํ ํก ,ํ ํก โฉ ํน โฅํ ํก โฅ ํน โฅํ ํก โฅ ํน .(2) 2.2. Scale invariance and the limitation of norm-only analysis For a scale-invariant parameter, as induced for example by a normalization layer, the loss satisfies [Wan et al., 2020] L(ํํ)=L(ํ), โํ > 0.(3) Differentiating Eq. (3) with respect to ํ gives โจ ํ,โ ํ L(ํ) โฉ ํน = 0, โ ํ L(ํํ)= 1 ํ โ ํ L(ํ).(4) Thus, ifํ ํก is the instantaneous gradient, thenํ ํก =0. In this special case, increasing the parameter norm decreases the relative update size, which is the basis of the classical effective-learning-rate analysis. For a modern optimizer, however,ํ ํก is generally not the instantaneous gradient. Momentum aggregates past gradients, preconditioning changes their relative scale across directions, and matrix optimizers may apply additional transformations to the update. Consequently, even when the raw gradient satisfies Eq.(4), it does not follow in general thatโจํ ํก ,ํ ํก โฉ ํน =0. Describing the effective step solely byํ ํก /โฅํ ํก โฅ ํน may therefore omit effects due to both the direction and the magnitude of the optimizer update. 2.3. Angular effective learning rate We measure the directional displacement of an update by the angle between two consecutive parameter states: ฮํ ํก =โ (ํ ํก ,ํ ํก+1 ).(5) 3 This angular displacement excludes pure radial rescaling and is therefore particularly appropri- ate for analyzing optimization dynamics in scale-invariant networks. The following proposition gives the angular displacement under the generic update rule. Proposition 1. For the update in Eq. (1), if ํผ ํก โฅํ ํก โฅ ํน โ ํ ํก ํ ํก โฅํ ํก โฅ ํน > 0, then tan(ฮํ ํก )= ํ ํก โฅํ ํก โฅ ํน โ๏ธ 1โ ํ 2 ํก ํผ ํก โฅํ ํก โฅ ํน โ ํ ํก ํ ํก โฅํ ํก โฅ ํน .(6) The result follows directly by decomposingํ ํก into components parallel and orthogonal to ํ ํก . The numerator is the tangential component that rotates the parameter direction, whereas the denominator combines the radial component of the current parameter with the radial component of the update. When the single-step angular displacement is small,tan(ฮํ ํก ) โฮํ ํก . We therefore define the angular effective learning rate as ํ ํ eff,ํก โ ฮํ ํก โฅํ ํก โฅ ํน โ ํ ํก โ๏ธ 1โ ํ 2 ํก ํผ ํก โฅํ ํก โฅ ํน โ ํ ํก ํ ํก โฅํ ํก โฅ ํน .(7) Equation(7)shows that the angular step is jointly determined by four time-varying quan- tities: the learning rateํ ํก , the parameter normโฅํ ํก โฅ ํน , the update normโฅํ ํก โฅ ํน , and the cosine similarity ํ ํก . When ํ ํก = 0, Eq. (7) reduces to ํ ํ eff,ํก โ ํ ํก ํผ ํก โฅํ ํก โฅ ํน ,(8) Equation(8)is the conventional norm-based expression for the effective learning rate. It relies, however, on the assumptionํ ํก =0, which need not hold for a transformed optimizer update. Section 3 examines how violations of this assumption affect the angular effective learning rate. 2.4. Hyperball update Hyperball constrains both the parameter norm and the update norm. Let the radius of the constraint sphere be the initial norm ํ =โฅํ 0 โฅ ํน , and define the normalized update ํ ํก = ํ ํ ํก โฅํ ํก โฅ ํน .(9) The Hyperball update is then ํ ํก+1 = ํ ยท ํ ํก โ ํ ํก ํ ํก โฅํ ํก โ ํ ํก ํ ํก โฅ ํน .(10) Equation(10)ensures thatโฅํ ํก+1 โฅ ํน = ํ after every step while also fixing the update norm at โฅํ ํก โฅ ํน = ํ . The update can equivalently be written as ํ ํก+1 = ํ ํก ํ ํก โ ํ ํก ํ ํก ํ ํก , ํ ํก = ํ โฅํ ํก โ ํ ํก ํ ํก โฅ ํน .(11) Equation(11)exposes two coupled operations in Hyperball: the optimizer update is first rescaled to a fixed norm, and the resulting parameter is then projected back onto the sphere of radiusํ through the state-dependent coefficientํ ํก . Equivalently,ํ ํก induces both a time-varying radial rescaling of the parameter and a rescaling of the update. This coupling is the principal 4 structural difference between Hyperball-style and conventional optimizers. Applying the same angular decomposition as above yields the following small-angle effective learning rate: ํ H,ํ eff,ํก โ ํ ํก โ๏ธ 1โ ํ 2 ํก (1โ ํ ํก ํ ํก )ํ (12) The angular effective learning rate of a conventional optimizer depends jointly onํ ํก ,โฅํ ํก โฅ ํน , โฅํ ํก โฅ ํน , andํ ํก . By fixing the parameter norm and normalizing the update, Hyperball removes the independent dependence onโฅํ ํก โฅ ํน andโฅํ ํก โฅ ํน , leaving onlyํ ํก andํ ํก . Under a prescribed learning- rate schedule, the remaining state-dependent variation is therefore carried byํ ํก . Section 3 uses this unified expression to test whether the radial component represented byํ ํก materially changes the effective learning rate. 3. Do Radial Updates Provide an Additional Benefit? The preceding analysis shows that the angular effective learning rate depends not only on the learning rate and parameter norm but also onํ ํก andโฅํ ํก โฅ ํน . We now ask whether radial updates are the primary cause of MuonHโs phase-dependent training dynamics. One intuitive explanation is that, whenํ ํก <0, the optimizer update increases the parameter norm; by projecting the parameter back onto a sphere of fixed radius, Hyperball suppresses this radial motion and may thereby change the optimization speed in the early and middle stages of training. This explanation conflates two distinct mechanisms: the direct effect of a radial component on the current angular step and its indirect effect on future steps through changes in the parameter state. We separate these mechanisms and test whether the direct effect alone can explain the observed dynamics. 3.1. Radial and tangential components Under Eq.(1),ํ ํก can be decomposed into a radial component parallel to the current parameter and a tangential component orthogonal to it: ํ โฅ ํก = โจํ ํก ,ํ ํก โฉ ํน โฅํ ํก โฅ 2 ํน ํ ํก = ํ ํก โฅํ ํก โฅ ํน โฅํ ํก โฅ ํน ํ ํก ,ํ โฅ ํก = ํ ํก โํ โฅ ํก . โฅํ โฅ ํก โฅ ํน =|ํ ํก |โฅํ ํก โฅ ํน ,โฅํ โฅ ํก โฅ ํน =โฅํ ํก โฅ ํน โ๏ธ 1โ ํ 2 ํก . (13) The tangential component directly changes the parameter direction, whereas a purely radial component produces no angular displacement by itself. Whenํ ํก <0,ํ โฅ ํก points opposite to ํ ํก ; because the update rule appliesโํ ํก ํ ํก , this component tends to increase the parameter norm. Conversely, whenํ ํก >0, the radial component tends to decrease the parameter norm. Even though a radial update does not directly rotate the parameter, it can still affect future optimization indirectly by changingโฅํ ํก โฅ ํน , the gradient scale, and the optimizer state. The analysis below therefore focuses on the radial componentโs direct contribution to the current one-step angular update; it does not rule out longer-term indirect effects. 3.2. Local effect of radial updates on the angular effective learning rate By Eq. (6), the angular displacement at the current step satisfies tan(ฮํ ํก )= ํ ํก โ๏ธ 1โ ํ 2 ํก โฅํ ํก โฅ ํน ํผ ํก โฅํ ํก โฅ ํน โ ํ ํก ํ ํก โฅํ ํก โฅ ํน .(14) 5 This expression separates two effects of the radial component. First,ํ ํก appears in the numerator: as|ํ ํก |increases, the tangential magnitudeโฅํ ํก โฅ ํน โ๏ธ 1โ ํ 2 ํก decreases, leaving less of the update available to rotate the parameter. Second,ํ ํก appears in the denominator: whenํ ํก <0, the denominator increases and the angular displacement is reduced further. Thus, a negative radial component suppresses the one-step angular motion, although the magnitude of this effect depends on the relative scales ofํ ํก ,โฅํ ํก โฅ ํน , andโฅํ ํก โฅ ํน . To analyze how these factors jointly evolve during training, treat the step index as a continuous variable and differentiatelogํ ํ eff,ํก . This gives d logํ ํ eff,ํก dํก = ยค ํ ํก ํ ํก โ ํ ํก ยค ํ ํก 1โ ํ 2 ํก โ ยค ํผ ํก โฅํ ํก โฅ ํน + ํผ ํก dโฅํ ํก โฅ ํน dํก โ ยค ํ ํก โฅํ ํก โฅ ํน ํ ํก โ ํ ํก ํ ํก dโฅํ ํก โฅ ํน dํก โ ํ ํก โฅํ ํก โฅ ํน ยค ํ ํก ํผ ํก โฅํ ํก โฅ ํน โ ํ ํก โฅํ ํก โฅ ํน ํ ํก .(15) Equation(15)decomposes the evolution of the angular effective learning rate into four sources: the external learning-rate schedule, changes in the parameterโupdate angle, changes in the parameter norm, and changes in the update norm. As shown in Fig. 1, we further use Eq.(15)to quantify the contribution of each factor to the evolution of the angular effective learning rate, usingํ ํ =100 1โ ํ (ํ) (ํก) ํ (ํโ1) (ํก) % as the metric. Under the training configurations considered here,ํ ํก changes substantially at the beginning of training and then enters a slowly varying negative regime. During this later regime, the terms involvingํ ํก make only a limited instantaneous contribution to Eq.(15); the overall trend is governed primarily by the external learning-rate schedule and the evolution of the parameter norm. 050010001500200025003000 Training Step 0 20 40 60 80 100 R j (\%) External LR schedule Parameter--update angle Parameter norm Update norm (a) 050010001500200025003000 Training Step 0 20 40 60 80 100 R j (\%) External LR schedule Parameter--update angle Parameter norm Update norm (b) Figure 1. The decline in learning rate caused by different factors. (a) MuonWD. (b) MuonH. These results do not support the hypothesis that MuonHโs two-stage training dynamics are driven primarily by the removal of radial updates. More precisely, the radial component affects the one-step angular displacement and may influence long-term training indirectly through the parameter norm. Within the scope of our experiments, however, its direct effect is not the dominant explanation for the phase-dependent dynamics. The next section examines an alternative mechanism: Hyperball may act chiefly by fixing the scales of the parameter and update, thereby inducing a state-dependent effective-learning-rate schedule. 4. No Free Lunch: Hyperball as an Implicit Learning-Rate Scheduler Having ruled out the direct removal of radial updates as the primary explanation, we interpret Hyperballโs characteristic slow-then-fast behavior as a consequence of its effective step-size 6 dynamics. Wen et al. [2025] report a related valley-landscape phenomenon when comparing AdamW with Adam: even under the same base learning-rate schedule, AdamW can begin with a higher loss yet ultimately converge to a strictly lower loss. This observation suggests an alternative explanation for Hyperballโs behavior, analogous to the effect of weight decay: a larger effective step size can delay convergence while improving the solution reached later in training. To make this connection explicit, we decompose the Hyperball update into a state-dependent radial rescaling and an update-normalization term: ํ ํก+1 = โฅํ ํก โฅ ํน ํ ํก โ ํํข ํก โฅํ ํก โฅ ํน โฅํข ํก โฅ ํน ํน ํ ํก โ ํโฅํ ํก โฅ ํน ํ ํก โ ํํข ํก โฅํ ํก โฅ ํน โฅํข ํก โฅ ํน ํน ํข ํก โฅํ ํก โฅ ํน โฅํข ํก โฅ ํน .(16) Define the state-dependent projection coefficient ํผ ํป ํก โ โฅํ ํก โฅ ํน ํ ํก โ ํํข ํก โฅํ ํก โฅ ํน โฅํข ํก โฅ ํน ํน .(17) The update then simplifies to ํ ํก+1 = ํผ ํป ํก ํ ํก โ ํํผ ํป ํก ํข ํก โฅํ ํก โฅ ํน โฅํข ํก โฅ ํน .(18) In this form, Hyperball combines a state-dependent radial coefficient with update normalization; the coefficient is determined by the current state rather than by an independently specified weight-decay hyperparameter. Its two-stage behavior may therefore arise from the same step- size mechanism commonly associated with weight decay. This interpretation is a working hypothesis and requires both theoretical and empirical validation. Weight decay is often understood to increase the angular effective learning rate by controlling parameter-norm growth. Section 3 indicates that the radial componentโs direct contribution to the current angular step is small in our setting, suggesting that scale control is the more relevant mechanism. Hyperball fixes the parameter norm and normalizes the update; once the transient variation inํ ํก subsides, its angular effective learning rate closely tracks the prescribed base learning-rate schedule instead of being attenuated by parameter-norm growth [Wen et al., 2026, Blake et al., 2024, Kosson et al., 2025]. Hyperball can thus be viewed as an optimizer with an approximately controlled angular step size, which can produce the same delayed-convergence phenomenon associated with a larger effective learning rate. To test whether the slower initial convergence of MuonH is attributable to its larger effective step size, we conduct three experiments: 1. We visualize and compare the effective-learning-rate trajectories of MuonWD and MuonH. 2.We test whether the validation-loss dynamics of Hyperball and non-Hyperball optimizers can be transformed into one another by changing only the learning-rate schedule. 3.In language-model pretraining experiments, we evaluate whether alternative learning-rate schedules can accelerate the convergence of MuonH. 4.1. Empirical evolution of the effective learning rate under different schedules The effective learning rates of non-Hyperball optimizers such as AdamW and MuonWD have been characterized by norm-based analyses. Section 3 extends this picture to the non-orthogonal regimeํ ํก <0. Here we directly compare MuonWD and MuonH. The purpose is to establish two 7 empirical facts: MuonH operates at a larger angular effective learning rate in the configurations considered, and its effective learning rate remains approximately proportional to the externally specified learning rate. 050010001500200025003000 Training Step 0.0240 0.0245 0.0250 0.0255 0.0260 Nominal Matrix LR MuonWD MuonH (a) 050010001500200025003000 Training Step 2 4 6 8 Effective LR (ร10 โ5 ) MuonWD MuonH (b) 050010001500200025003000 Training Step 0.000 0.005 0.010 0.015 0.020 0.025 Nominal Matrix LR MuonWD MuonH (c) 050010001500200025003000 Training Step 0 2 4 6 8 Effective LR (ร10 โ5 ) MuonWD MuonH (d) Figure 2. Learning-rate schedules and the resulting effective learning rates. (a) Constant learning-rate schedule. (b) Effective-learning-rate comparison under the constant schedule. (c) Constant-then-decay schedule. (d) Effective-learning-rate comparison under the constant-then-decay schedule. Figure 2 shows the learning-rate schedules and their corresponding effective-learning-rate trajectories. Under a constant base learning rate, MuonWDโs effective learning rate decays rapidly as its parameter norm grows. MuonH, in contrast, maintains an approximately constant effective learning rate after an initial transient caused by the abrupt evolution ofํ ํก . Under a linearly decaying base learning rate, MuonHโs effective learning rate remains approximately proportional to that schedule. 4.2. Transforming the dynamics of Hyperball and non-Hyperball optimizers We next test whether the effective learning rate is the principal difference between Hyperball and non-Hyperball optimizers. Because fixed parameter and update norms can change multiple aspects of a high-dimensional optimization trajectory, observational comparisons alone cannot isolate their contribution. We therefore perform a controlled ablation in which the optimizer is held fixed and only its learning rate is changed so as to match the target angular effective learning rate at every step. Denote the effective learning rates of the non-Hyperball and Hyperball optimizers under the reference schedule byํ ํ eff,ํก andํ ํป,ํ eff,ํก , respectively. To align MuonWD with MuonH, we proceed as follows: 1. Run MuonH and record its angular effective learning rate ํ ํป,ํ eff,ํก at every step. 2. Before each MuonWD update, record ํ ํก , ํ ํก , and ํ ํก , and solve ํ ํป,ํ eff,ํก โ ํ ํก โ๏ธ 1โ ํ 2 ํก ํผ ํก โฅํ ํก โฅ ํน โ ํ ํก ํ ํก โฅํ ํก โฅ ํน (19) for the step-specific learning rate ํ ํก . 3. Use the resulting ํ ํก for the current MuonWD update. We apply the analogous procedure in the reverse direction to align MuonH with MuonWD. The results in Fig. 3 show that the validation-loss trajectories of the two optimizer families can be transformed into one another to a substantial degree by learning-rate alignment alone. This finding supports the interpretation that, in the settings studied, Hyperballโs dominant effect is an implicit state-dependent learning-rate schedule rather than an intrinsically superior update direction. 8 50010001500200025003000 Training Step 3.3 3.4 3.5 3.6 3.7 3.8 3.9 Training Loss MuonWD MuonH MuonH align WD (a) 50010001500200025003000 Training Step 0.7 0.8 0.9 1.0 1.1 1.2 Equivalent LR Factor MuonH align WD (b) 50010001500200025003000 Training Step 3.3 3.4 3.5 3.6 3.7 3.8 3.9 Training Loss MuonWD MuonH MuonWD align H (c) 50010001500200025003000 Training Step 0.75 1.00 1.25 1.50 1.75 2.00 2.25 Equivalent LR Factor MuonWD align H (d) Figure 3. Mutual alignment of Hyperball and non-Hyperball dynamics. (a) Validation loss when MuonH is aligned with MuonWD. (b) MuonH learning-rate alignment factors during this experiment. (c) Validation loss when MuonWD is aligned with MuonH. (d) MuonWD learning-rate alignment factors during the reverse experiment. Shaded regions in (b) and (d) show variation across layers. 4.3. Accelerating Hyperball-Style Optimizers Building on the observations validated above, we accelerate the MuonH optimizer through learning-rate scheduling. Following the river-valley landscape interpretation of Wen et al. [2025], the initially large step size delays convergence until the later stages of training. Moreover, the effective learning rate of Hyperball-style optimizers is inherently larger than that of conventional optimizers during the early stage. We therefore adopt more aggressive learning-rate decay schedules to accelerate MuonH. We consider the following schedules: 1. The linear learning-rate decay schedule, under which ํ ํก decreases linearly with ํก: ํ lin (ํก)= ํ 0 1โ ํก ํ ,0โค ํก โค ํ .(20) 2.The exponential learning-rate decay schedule, under whichํ ํก decreases exponentially with ํก: ํ exp (ํก)= ํ 0 exp โํ ํก ํ โ exp(โํ) 1โ exp(โํ) ,0โค ํก โค ํ , ํ > 0.(21) 3.The polynomial learning-rate decay schedule, under whichํ ํก decreases polynomially with ํก: ํ pow (ํก)= ํ 0 h 1โ ํก ํ ํ i ,0โค ํก โค ํ , ํ > 0.(22) In the comparisons below, MuonH-Z denotes the minus-square-root learning-rate schedule proposed by Zhanpeng Zhou in PR #343, which reaches the target validation loss of 3.28 in 3175 steps. MuonH-Ours denotes our power-0.4 schedule released in PR #345, which reaches the same target in 3150 steps. Because MuonH maintains a constant angular velocity, we adopt the polynomial learning- rate decay schedule, which provides a more aggressive decay, in our language-model pretraining experiments. As shown in Fig. 4, our method reaches a validation loss below 3.28 faster than the compared baselines. However, our search experiments also reveal a new observation: making MuonH faster during the early stage does not necessarily guarantee better convergence in the later stage. When a more aggressive learning-rate decay schedule is used to induce earlier convergence, MuonH performs substantially worse during the later stage than standard MuonWD and MuonH with other learning-rate schedules. Thus far, we have found no direct evidence that MuonH consistently outperforms MuonWD throughout the entire optimization process. 9 This observation provides a more fundamental insight into MuonH: effectively using MuonH requires a new learning-rate scheduling strategy. Directly conducting scaling experiments with MuonH may even degrade model performance. Maintaining a constant angular velocity does not directly reduce the complexity of the optimization problem; instead, it makes learning-rate scheduling even more important in the optimization dynamics of MuonH. 50010001500200025003000 Step 3.3 3.4 3.5 3.6 3.7 3.8 3.9 Validation loss MuonWD MuonH MuonH-Z MuonH-Ours (a) 50010001500200025003000 Step 0.000 0.005 0.010 0.015 0.020 0.025 Matrix learning rate MuonWD MuonH MuonH-Z MuonH-Ours (b) 50010001500200025003000 Step 3.4 3.6 3.8 4.0 Validation loss Muon MuonH MuonH-Quad MuonH-SuperExp (c) 50010001500200025003000 Step 0.000 0.005 0.010 0.015 0.020 0.025 0.030 0.035 Matrix learning rate Muon MuonH MuonH-Quad MuonH-SuperExp (d) Figure 4. Validation loss and learning-rate schedules under different decay strategies. (a) Validation losses of three MuonH variants in the pretraining experiments. (b) The corresponding learning-rate schedules for (a). (c) Validation losses under an additional set of learning-rate schedules. (d) The corresponding learning-rate schedules for (c). Even when MuonH surpasses Muon earlier in training, Muon can overtake it later. 5. Discussion 5.1. Limitations Our study has several limitations. First, the theoretical analysis is limited to a numerical sensitivity analysis of the effective learning rate and an examination of the conventional effective- learning-rate estimate. The river-valley landscape provides an interpretation of the slow-then- fast convergence dynamics of Hyperball-style optimizers [Wen et al., 2025, 2026], but a more complete theoretical characterization remains open. Second, our current experiments focus on dense language-model pretraining. Extending the analysis to mixture-of-experts (MoE) architectures will be important for determining whether the observed dynamics persist under sparse activation and expert routing. Third, the optimizer comparison is currently restricted to Muon, MuonWD, and MuonH. Future experiments will broaden this comparison to other optimizer families, including AdamW and its Hyperball counterpart AdamH. Finally, our evaluation primarily considers pretraining validation loss and optimization dynamics. A more comprehensive assessment should include a wider range of downstream tasks and metrics to determine whether the observed optimization behavior translates into improvements in generalization and practical model quality. Our analysis also relies on a scale-invariance assumption that does not strictly hold in practi- cal training. Interestingly, the two properties implied by scale invarianceโthat the stochastic gradient is orthogonal to the model parameters and that the gradient magnitude decreases as the model norm increasesโare nevertheless observed in practice [Salimans and Kingma, 2016, Wan et al., 2020]. Understanding when this approximation is sufficiently accurate, and when its violations materially affect the predicted dynamics, requires further investigation. 5.2. Future Directions Beyond these limitations, several broader questions emerge from our investigation. How should the learning rate be scheduled? Learning-rate scheduling remains a largely 10 open problem. Recent studies have primarily investigated approaches such as WSD [Wen et al., 2025, Li et al., 2020], schedule-free methods [Defazio et al., 2024, Defazio, 2026], and the derivation of learning-rate schedules from functional rates [Li et al., 2026, Wang et al., 2026]. From the current perspective, a Hyperball-style optimizer incorporates a heuristic learning-rate schedule based on weight constraints. Although this mechanism does not reduce the complexity of the scheduling problem, it does not imply that the contribution of Hyperball-style optimizers is itself merely incremental. Their specific update dynamics may provide deeper insights into learning-rate scheduling. Notably, the effective learning rate of a Hyperball-style optimizer is proportional to its scheduled learning rate. Therefore, using Hyperball-style optimizers as a testbed for studying learning-rate schedules can largely avoid the confounding effects introduced by the coupling between the learning rate and weight decay. This property makes them a particularly promising experimental foundation for investigating learning-rate scheduling. Does a two-stage learning-rate schedule exist? For non-Hyperball-style optimizers, the effective learning rate decreases rapidly during the early stage as the model norm grows. During the middle and later stages of training, steady-state analysis yieldsํ ํ eff โ โ๏ธ ํํ[Su, 2024, 2025a,b,c]. By contrast, the effective learning rate of a Hyperball-style optimizer satisfiesํ ํป,ํ eff โ ํ . It remains to be investigated whether, during the later stages of optimization, the effective learning rate of a Hyperball-style optimizer can become smaller than that of a conventional optimizer and thereby hinder its performance. Rethinking first-order optimization. Beyond optimizers based on geometric constraints, another promising direction is to use first-order information to approximate second-order Hessian information or even third-order information for improved optimization. Existing approaches can be broadly divided into two categories: methods that use gradient information of different orders to estimate the step size [Shazeer and Stern, 2018, Yuan, 2026], and methods that use such information to estimate the update direction [Cohen et al., 2024, Damian et al., 2021, Chen et al., 2026, Xie et al., 2026]. In the context of learning-rate scheduling, it may be necessary to investigate more deeply how gradient or even Hessian information can be used to characterize the coupling between the step size and update direction. 6. Conclusion Starting from the effective learning rate, we first examine its numerical sensitivity to per- turbations introduced by radial updates and show that it is only weakly affected by such updates. Based on this observation, we challenge existing heuristic explanations of the dynamics of Hyperball-style optimizers. We then build on the constant-angular-velocity property of Hyperball-style optimizers and progressively show that their characteristic update dynamics fundamentally arise from the larger effective learning rate induced during the early stage of optimization. To validate this interpretation, we align the effective learning rates of the optimizers and show that the dynamics of a Hyperball-style optimizer can be reproduced simply by modifying the learning-rate schedule of Muon. 11 Acknowledgments We thank Huaqing Zhang, Tian Xie, Guangyu Chen, Huanran Chen, Ganzhao Yuan, and Zhenjie Zhou for their insightful discussions during the development of this work. 12 References S. Bergsma, N. Dey, G. Gosal, G. Gray, D. Soboleva, and J. Hestness. Power lines: Scaling laws for weight decay and batch size in llm pre-training. Advances in Neural Information Processing Systems, 38:125153โ125188, 2026. C. Blake, C. Eichenberg, J. Dean, L. Balles, L. Y. Prince, B. Deiseroth, A. F. Cruz-Salinas, C. Luschi, S. Weinbach, and D. Orr. u-ํP: The unit-scaled maximal update parametrization. arXiv preprint arXiv:2407.17465, 2024. URL https://arxiv.org/abs/2407.17465. F. L. Cesista. Heuristic solutions for steepest descent on the stiefel manifold, 2025. URL https://leloykun.github.io/ponder/steepest-descent-stiefel/. H. Chen, H. Zhang, X. Li, Y. Dong, K. Shen, and J. Zhu. Nexus: Same pretraining loss, better downstream generalization via common minima. arXiv preprint arXiv:2604.09258, 2026. J. M. Cohen, A. Damian, A. Talwalkar, J. Z. Kolter, and J. D. Lee. Understanding optimization in deep learning with central flows. arXiv preprint arXiv:2410.24206, 2024. A. Damian, T. Ma, and J. D. Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems, 34:27449โ27461, 2021. A. Defazio. Schedulefree+: Scaling learning-rate-free & schedule-free learning to large language models. arXiv preprint arXiv:2605.19095, 2026. A. Defazio, X. Yang, H. Mehta, K. Mishchenko, A. Khaled, and A. Cutkosky. The road less scheduled. Advances in Neural Information Processing Systems, 37:9974โ10007, 2024. S. Deng, Z. Ouyang, T. Pang, Z. Liu, R. Jin, S. Yu, and Y. Yang. Rmnp: Row-momentum normal- ized preconditioning for scalable matrix-based optimization. arXiv preprint arXiv:2603.20527, 2026. E. Hoffer, R. Banner, I. Golan, and D. Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. Advances in Neural Information Processing Systems, 31, 2018. A. Kosson, J. Welborn, Y. Liu, M. Jaggi, and X. Chen. Weight decay may matter more than ํP for learning rate transfer in practice. arXiv preprint arXiv:2510.19093, 2025. URLhttps: //arxiv.org/abs/2510.19093. B. Li, Z. Wang, F. Chen, S. Zhao, R. Zheng, and L. Wu. Optimal learning-rate schedules under functional scaling laws: Power decay and warmup-stable-decay. arXiv preprint arXiv:2602.06797, 2026. Z. Li, K. Lyu, and S. Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. In Advances in Neural Information Processing Systems, 2020. URL https://arxiv.org/abs/2010.02916. arXiv:2010.02916. Z. Li, L. Liu, C. Liang, W. Chen, and T. Zhao. NorMuon: Making Muon more efficient and scalable. arXiv preprint arXiv:2510.05491, 2025. URLhttps://arxiv.org/abs/2510.054 91. T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, 2016. URL https://arxiv.org/abs/1602.07868. 13 N. Shazeer and M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018. URL https://arxiv.org/abs/1804.04235. J. Su. Thinking about spectral norm gradient and spectral weight decay, 2024. URLhttps: //kexue.fm/archives/10648. J. Su. Why Adamโs update RMS is 0.2?, 2025a. URL https://kexue.fm/archives/11267. J. Su. AdamW weight RMS asymptotics (part I), 2025b. URLhttps://kexue.fm/archives/ 11307. J. Su. AdamW weight RMS asymptotics (part I), 2025c. URLhttps://kexue.fm/archives/ 11404. T. Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350, 2017. R. Wan, Z. Zhu, X. Zhang, and J. Sun. Spherical motion dynamics: Learning dynamics of neural network with normalization, weight decay, and sgd. arXiv preprint arXiv:2006.08419, 2020. J. Wang, B. Li, Z. Zhou, M. Wang, Y. Sun, J. Zhang, X. Cai, and L. Wu. Fast catch-up, late switching: Optimal batch size scheduling via functional scaling laws. arXiv preprint arXiv:2602.14208, 2026. K. Wen, Z. Li, J. Wang, D. Hall, P. Liang, and T. Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape view. In International Conference on Learning Representations, volume 2025, pages 42840โ42885, 2025. K. Wen, X. Dang, K. Lyu, T. Ma, and P. Liang. Fantastic pretraining optimizers and where to find them i: Hyperball optimization. arXiv preprint arXiv:2606.16899, 2026. Y. Wu and K. He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3โ19, 2018. T. Xie, H. Luo, H. Tang, Y. Hu, J. K. Liu, Q. Ren, Y. Wang, W. X. Zhao, R. Yan, B. Su, et al. Controlled llm training on spectral sphere. arXiv preprint arXiv:2601.08393, 2026. G. Yuan. Optmuon: Closed-loop orthogonalized momentum methods for stochastic optimization with zero-noise optimality. arXiv preprint arXiv:2606.08783, 2026. G. Zhang, C. Wang, B. Xu, and R. Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281, 2018. 14