Paper deep dive
When Does Muon Help Agentic Reinforcement Learning?
Kai Ruan, Jinghao Lin, Zihe Huang, Ziqi Zhou, Qianshan Wei, Xuan Wang, Hao Sun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 4:42:43 AM
Summary
This study investigates the efficacy of the Muon optimizer in sparse-reward agentic reinforcement learning (RL) post-training, comparing it against AdamW on the ALFWorld environment using Qwen2.5-0.5B-Instruct. The research demonstrates that Muon significantly improves performance when applied to hidden weight matrices under Group-in-Group Policy Optimization (GiGPO), achieving an 88% relative increase in validation success. The benefits are shown to be dependent on the advantage estimator (GiGPO, GraphGPO, GRPO) and learning rate, with Muon outperforming high-rate AdamW controls which failed to retain success. The authors hypothesize that Muon's spectral normalization helps when the gradient signal-to-noise ratio is improved by specific credit-assignment structures.
Entities (8)
Relation Signals (6)
Muon → improvesperformanceof → GiGPO
confidence 95% · Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%)
Muon → usesmechanism → Newton-Schulz
confidence 95% · Muon... computed via Newton–Schulz (NS) iterations
Muon → outperforms → AdamW
confidence 92% · high-rate AdamW controls retain no post-update success... Muon retains nonzero validation success through step 200
Qwen2.5-0.5B-Instruct → trainedon → AlfWorld
confidence 90% · training a Qwen2.5-0.5B-Instruct (Yang et al. 2024) agent... on ALFWorld
GiGPO → usedwith → AlfWorld
confidence 90% · We study long-horizon, sparse-reward agentic RL on ALFWorld... GiGPO... augments the identical episode-level advantage
GraphGPO → achieveshighersuccesswith → Muon
confidence 88% · At 1e-5, GraphGPO Muon reaches 0.901... raises normalized validation AUC from 0.399 to 0.556
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly. Multi-seed and cross-task validation remain open.
Tags
Links
- Source: https://arxiv.org/abs/2607.16169v1
- Canonical: https://arxiv.org/abs/2607.16169v1
Trouble viewing inline? Open PDF directly →
Full Text
46,246 characters extracted from source content.
Expand or collapse full text
When Does Muon Help Agentic Reinforcement Learning? Kai Ruan1, Jinghao Lin2, Zihe Huang3 Ziqi Zhou4, Qianshan Wei5, Xuan Wang6, Hao Sun1,* Abstract Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.2900.290 to 0.5460.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3×10−53× 10^-5, Muon improves GRPO from 0.1610.161 to 0.2680.268, whereas GraphGPO’s late-window gap narrows near saturation. At 10−510^-5, GraphGPO Muon reaches 0.9010.901, raises normalized validation AUC from 0.3990.399 to 0.5560.556, and reaches 0.5/0.750.5/0.75 success 30/6030/60 updates earlier. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly. Multi-seed and cross-task validation remain open. Introduction Figure 1: Muon improves late-window quality and learning efficiency, with learning-rate sensitivity that varies by advantage estimator. Every run reaches step 200200; curves show raw validation checkpoints without smoothing. AdamW is gray throughout. Blue shades distinguish Muon learning rates, while subplot titles identify the advantage estimator. GiGPO and GraphGPO include both 10−510^-5 and 3×10−53× 10^-5; GRPO is evaluated at 3×10−53× 10^-5. Muon (Jordan et al. 2024) replaces the element-wise adaptive scaling of Adam-family optimizers with an approximate spectral normalization of the momentum matrix, computed via Newton–Schulz (NS) iterations. In pre-training, this simple change is remarkably effective: Muon matches AdamW’s final loss with roughly 52%52\% of the training FLOPs at billion-parameter scale (Liu et al. 2025), and has since been used in trillion-parameter pre-training (Kimi Team 2025). Yet its authors explicitly identified whether Muon transfers to post-training, particularly reinforcement learning (RL), as an open question (Jordan et al. 2024). Existing evidence is mixed. NeMo RL has added Muon support and reports “minor improvements” when Muon is introduced only during post-training (NVIDIA 2026). Studies of optimizer mismatch report a worse learning–forgetting tradeoff when Adam-pretrained models are fully fine-tuned with Muon, with greater disruption or forgetting at larger learning rates (Qu et al. 2026; Liu et al. 2026). These results are confined to supervised fine-tuning and show substantial learning-rate sensitivity. Most recently, Fan et al. (2026) report that vanilla Muon fails under RLVR with GRPO-style objectives, attributing the failure to spectral whitening amplifying the noise-dominated tail of low signal-to-noise policy gradients. Public engineering reports also describe persistent entropy floors and gradient explosions in mathematical-reasoning RL; Hopper combines variance normalization with a single NS step to improve stability (Wei 2026a, b). The negative RL evidence, however, is concentrated in single-turn RLVR with episode-level (outcome-only) advantage estimation. We test whether Muon’s viability in RL post-training depends on the structure of the advantage estimator. We study long-horizon, sparse-reward agentic RL on ALFWorld (Shridhar et al. 2021), training a Qwen2.5-0.5B-Instruct (Yang et al. 2024) agent, and compare a family of group-based RL algorithms spanning distinct credit-assignment structures: GRPO (Shao et al. 2024) with episode-level advantages; GiGPO (Feng et al. 2025), which augments the identical episode-level advantage with a step-level term computed by grouping actions taken from repeated anchor states across trajectories; and GraphGPO (Cheng et al. 2026), which aggregates rollouts into a unified state-transition graph and assigns transition-level credit from successor-state distance to the task goal. GiGPO provides a controlled comparison because its advantage is A=AE+ωASA=A^E+ω A^S, and it reduces to GRPO when the step-level term vanishes. This exposes the step-level contribution within a single GiGPO code path without conflating it with a separate GRPO implementation. Figure 1 previews the central empirical comparison across the three advantage estimators. The experiments yield three observations. First, applying Muon only to the policy’s hidden weight matrices (embeddings, norms, and all other parameters keep the baseline AdamW configuration) lifts GiGPO’s validation success rate on ALFWorld from 0.290.29 to 0.550.55 averaged over the final evaluation window, an 88%88\% relative improvement. It reaches 0.630.63 versus 0.320.32 at the final checkpoint, with the largest gains on task categories where AdamW attains its lowest success. Second, learning-rate controls test whether a larger nominal AdamW rate alone reproduces the Muon gain. AdamW controls at 10−510^-5 and 3×10−53× 10^-5 record zero validation success at every post-update checkpoint, whereas Muon retains nonzero validation success through step 200 at 3×10−53× 10^-5. Third, the effect varies across advantage estimators. Over complete matched GRPO runs, Muon reaches 0.270.27 versus 0.160.16 on the final-window mean, a smaller gain than under GiGPO. Under GraphGPO, the late-window gap narrows as the curves approach saturation: Muon at 3×10−53× 10^-5 reaches 0.830.83 versus 0.810.81 for AdamW, but its normalized validation AUC is 0.470.47 versus 0.400.40. At 10−510^-5, Muon reaches 0.900.90, with normalized AUC 0.560.56 and earlier threshold crossings. We use gradient signal-to-noise ratio (SNR) as an organizing hypothesis for the positive and negative evidence in the literature. Muon’s orthogonalization flattens the singular spectrum of the update, boosting weak directions relative to dominant ones. In single-turn, low-success RLVR, those directions may be dominated by sampling noise, consistent with the failures reported by Fan et al. (2026). Our positive GRPO result on long-horizon agentic tasks suggests that episode-level credit alone does not determine whether Muon helps. GiGPO’s anchor-state grouping adds per-state contrasts (over 65%65\% of states recur across trajectories in ALFWorld (Feng et al. 2025)). These contrasts change the scalar weights applied to token-level score gradients and may alter the aggregate gradient spectrum, a connection not established by GiGPO theory. Its larger observed gain motivates testing this hypothesis directly and suggests that advantage-estimator design complements optimizer-side remedies such as Hopper’s variance-normalized, single-step NS update (Wei 2026b) and Pion’s high-pass spectral filtering (Fan et al. 2026). Our contributions are: • A controlled single-seed study of vanilla Muon for long-horizon, sparse-reward agentic RL. Under GiGPO, Muon improves final-window success by 88%88\% relative to matched AdamW, with gains spanning several ALFWorld task categories. • GiGPO learning-rate controls showing that the result is not reproduced by increasing AdamW’s nominal rate: AdamW at 10−510^-5 and 3×10−53× 10^-5 retains no post-update validation success, whereas Muon at 3×10−53× 10^-5 retains success through all 200200 updates. • A matched 2×22× 2 GiGPO ablation showing Muon gains with and without step-level credit, together with a trajectory-level comparison across GRPO, GiGPO, and GraphGPO. A gradient-SNR conjecture connects the estimator-dependent pattern to recent negative Muon results under RLVR. Related Work Muon and matrix-aware optimizers. Muon (Jordan et al. 2024) orthogonalizes the momentum matrix of each 2D hidden-layer parameter via Newton–Schulz iterations, and is typically deployed with an auxiliary Adam-family optimizer for embeddings, norms, and output heads. Liu et al. (2025) demonstrated scalability to LLM pre-training with roughly half of AdamW’s FLOP budget, and MuonClip (Kimi Team 2025) stabilized trillion-parameter pre-training via QK-clip; we note that QK-clip addresses attention-logit explosion in pre-training and is orthogonal to the RL-stability questions studied here. With decoupled weight decay, Muon is a nuclear-norm Lion-K method that implicitly enforces a spectral-norm constraint, although this characterization does not predict RL behavior (Chen et al. 2025). Pion replaces uniform spectral whitening with a high-pass NS transformation that suppresses tail components (Fan et al. 2026); our work is complementary in that we hold the optimizer fixed (vanilla Muon) and vary the RL objective. Muon in post-training and optimizer mismatch. Qu et al. (2026) and Liu et al. (2026) study optimizer mismatch in supervised fine-tuning. The former finds that full Muon fine-tuning underperforms matched Adam on Adam-pretrained models and becomes more disruptive as its learning rate grows; the latter finds a better learning–forgetting tradeoff when pre-training and fine-tuning use the same optimizer, with more forgetting at larger learning rates. In the evaluated Llama-2 settings, Qu et al. (2026) select 2×10−52× 10^-5–5×10−55× 10^-5 for full-Muon instruction tuning, while Liu et al. (2026) sweep 1×10−51× 10^-5–1×10−41× 10^-4 when analyzing forgetting; these are experiment-specific settings, not universal thresholds. Our RL experiments use lower nominal rates and KL regularization toward a reference policy; Qwen2.5’s pretraining optimizer is undisclosed. Fan et al. (2026) report that vanilla Muon fails under RLVR, attributing failure to spectral whitening under low gradient SNR, and propose a high-pass spectral remedy; community experiments similarly report entropy floors and gradient explosions with GRPO-family objectives on mathematical reasoning. Hopper reports improved stability from a combined recipe of variance normalization and one NS step (Wei 2026a, b). In supervised post-training, Gupta et al. (2025) instruction-tune a Muon-pretrained GPT checkpoint with Muon, establishing feasibility under optimizer continuity in supervised post-training. Kimi K2 and INTELLECT-3 describe RL stages that continue from Muon-pretrained bases (Kimi Team 2025; Prime Intellect Team et al. 2025), making optimizer continuity one plausible factor. Our experiments compare Muon’s behavior across three advantage-estimator implementations. Group-based RL for LLM agents. Group-based methods, including RLOO (Kool et al. 2019; Ahmadian et al. 2024), GRPO (Shao et al. 2024), and DAPO (Yu et al. 2025), estimate advantages from within-group statistics of rollouts. For multi-turn agents, credit assignment across long horizons is the central difficulty. GiGPO (Feng et al. 2025) adds a step-level relative advantage by retroactively grouping actions taken from repeated anchor states across trajectories, at negligible cost and without extra rollouts. GraphGPO (Cheng et al. 2026) aggregates rollouts into a unified state-transition graph, estimates each state’s distance to the task goal, and assigns edge-level advantages from successor-state distance; under deterministic- environment and fixed-policy assumptions, it also gives a conditional variance result for scalar feedback. This result establishes neither lower policy-gradient variance nor higher gradient-matrix SNR. The three methods therefore span different credit-signal granularities and conditioning. Preliminaries Setup. An LLM agent interacts with an environment over discrete steps t=1,…,Tt=1,…,T: it observes state t s_t, emits a textual action t∼πθ(⋅∣t,x) a_t _θ(· s_t,x) for task prompt x, and receives reward rtr_t. Rewards are sparse: in ALFWorld a positive terminal reward is given only on task success, plus a small penalty for invalid actions. GiGPO. GiGPO (Feng et al. 2025) samples N trajectories from identical initial conditions and combines two group-relative advantages. The episode-level term normalizes total returns within the trajectory group, exactly as GRPO: AE(i)=R(i)−mean(R(j)j=1N)Fnorm(R(j)j=1N).A^E( τ_i)= R( τ_i)-mean(\R( τ_j)\_j=1^N)F_norm(\R( τ_j)\_j=1^N). (1) Here FnormF_norm denotes the within-group standard deviation used by the mean-std normalization in our experiments. The step-level term exploits the fact that trajectories from the same initial state repeatedly visit identical environment states (anchor states ~ s). Actions taken from the same anchor state are grouped, and their discounted returns Rt(i)=∑k≥tγk−trk(i)R_t^(i)= _k≥ tγ^k-tr_k^(i) are normalized within the group to give AS(t(i))A^S( a_t^(i)). The combined advantage is A(t(i))=AE(i)+ωAS(t(i)),A( a_t^(i))=A^E( τ_i)+ω\,A^S( a_t^(i)), (2) optimized with a PPO-style clipped objective and a KL penalty to the reference policy. Under matched mean-std normalization, setting ω=0ω=0 yields the same episode-level advantage as GRPO within the GiGPO code path; γ then affects only the discarded step term. The same reduction occurs when no anchor states recur and AS=0A^S=0 (Feng et al. 2025). Thus step-level credit is a toggleable component of a single implementation. Muon. For each 2D hidden weight matrix with momentum-accumulated gradient M, Muon (Jordan et al. 2024) applies the update ΔW∝NS5(M)≈UV⊤ W _5(M)≈ UV where M=UΣV⊤M=U V , i.e., it preserves the singular directions of the momentum while discarding singular-value magnitudes (55 Newton–Schulz iterations, momentum 0.950.95, Nesterov). Non-matrix parameters (embeddings, norms; the LM head is tied to the embedding in Qwen2.5-0.5B) are delegated to AdamW. Learning-rate comparability. Muon and AdamW learning rates are not directly comparable quantities because Muon implementations use different matrix-shape scaling conventions. For the non-RMS-matched variants, a practical rule of thumb is to multiply an Adam learning rate by roughly 10×10× for common hidden dimensions; the RMS-matched Moonlight variant instead reuses the Adam rate (Su 2025; Liu et al. 2025). We therefore impose no universal conversion: each optimizer is evaluated over its own learning-rate range, and all points are reported in Table 2. Method: Muon for Group-Based Agentic RL Starting from a matched AdamW GiGPO baseline, we change only the optimizer of the policy’s hidden weight matrices: • Matrix parameters (all attention and MLP 2D weights): Muon, learning rate ηmuon=3×10−5 _muon=3× 10^-5. • Fallback parameters (embeddings, norms, and all other non-matrix parameters): AdamW at the baseline learning rate 1×10−61× 10^-6, identical to the baseline configuration. Within each advantage estimator’s matched AdamW–Muon comparison, all non-optimizer settings are held fixed; Appendix A provides the full configuration. Our implementation exposes full, unsharded 2D matrices to the Newton–Schulz iteration (FSDP NO_SHARD); Appendix B discusses the resulting memory implications. Conjecture: credit quality and Muon’s weak directions. Before clipping and KL terms, write a sampled layer gradient as G^=∑tA^t∇Wlogπθ(at∣st)=G⋆+E G= _t A_t _W _θ(a_t s_t)=G +E, separating a latent signal G⋆G from credit and sampling error E. In a simplified trajectory with T decisions but only K≪TK T credit-relevant decisions, independent equal-energy errors give SNRlocalSNRepisode≈TK, SNR_localSNR_episode≈ TK, (3) if an ideal local-credit estimator suppresses the other T−KT-K error terms. This is a toy scaling, not a claim that GiGPO or GraphGPO attains the ratio. Muon approximately applies the polar factor (UΣV⊤)=UV⊤P(U V )=UV . In a fixed-singular-direction slice with signal coefficients si>0s_i>0 and Gaussian errors ϵi∼(0,σi2) _i (0, _i^2), its expected alignment is ⟨(G^),G⋆⟩F=∑isi[2Φ(si/σi)−1],E ( G),G _F= _is_i\! [2 (s_i/ _i)-1 ], (4) where Φ is the standard normal CDF. Thus spectral flattening is useful only when weak-direction signs are reliable. We conjecture that GiGPO’s anchor-state contrasts and GraphGPO’s transition-level credit can reduce relevant confounders and improve some si/σis_i/ _i, while GraphGPO’s stronger baseline can still leave less late-window headroom. Appendix D states the assumptions, derivation, and testable predictions; the current experiments do not measure gradient SNR or update spectra. Experiments Experimental Setup Environment and model. ALFWorld (Shridhar et al. 2021) comprises embodied household tasks (six categories: Pick, Look, Clean, Heat, Cool, Pick Two), with a horizon up to 5050 steps, sparse terminal success reward with an invalid-action penalty. Base model: Qwen2.5-0.5B-Instruct (Yang et al. 2024). Training uses verl-agent (Feng et al. 2025), an agentic extension of veRL/HybridFlow (Sheng et al. 2024), with vLLM rollouts (Kwon et al. 2023) on eight NVIDIA H20 GPUs for 200200 updates. Each update uses groups of eight over a 16-task batch, and validation evaluates 128 episodes every five updates. Within each matched AdamW–Muon pair, all non-optimizer settings are shared; estimator-specific and execution hyperparameters appear in Appendix A. Because single-evaluation success rates fluctuate under sampled validation, we report both final-checkpoint values and means over a fixed 2525-update window comprising the final six evaluations (steps 175175–200200), computed identically for all runs. For the headline GiGPO comparison, the conclusion is stable across nearby final-window choices. To measure learning efficiency before saturation, we also report normalized validation AUC: the trapezoidal area under the raw checkpoint curve from steps 0–200200, divided by 200200. Compared configurations. (i) AdamW baseline: policy AdamW, lr 1×10−61× 10^-6 (the baseline configuration of Feng et al. 2025). (i) Muon: matrix parameters Muon at 3×10−53× 10^-5, fallback AdamW 1×10−61× 10^-6. (i) AdamW lr controls: policy AdamW at 1×10−5,3×10−5\1× 10^-5,3× 10^-5\. (iv) Algorithm family: matched AdamW–Muon comparisons under GRPO and GraphGPO (adapted to ALFWorld). GRPO uses the shared Muon rate 3×10−53× 10^-5; GiGPO and GraphGPO additionally include Muon 10−510^-5 as a learning-rate sensitivity check. We also evaluate the matched GiGPO AdamW–Muon pair with the step weight disabled (ω=0ω=0). Main Results Across Advantage Estimators Table 1 separates late-window quality from full-trajectory learning efficiency. GiGPO at 3×10−53× 10^-5 has the largest late-window gain (about 0.260.26), whereas GraphGPO at 10−510^-5 has the largest normalized AUC gain (about 0.160.16). GRPO improves late (about 0.110.11) but has a much smaller full-trajectory AUC gain (about 0.010.01). Estimator Muon lr Late AdamW Late Muon Δlate _late ΔAUC _AUC GRPO 3×10−53×10^-5 0.161 0.268 +0.107+0.107 +0.013+0.013 GiGPO 1×10−51×10^-5 0.290 0.509 +0.219+0.219 +0.147+0.147 GiGPO 3×10−53×10^-5 0.290 0.546 +0.255+0.255 +0.127+0.127 GraphGPO 1×10−51×10^-5 0.810 0.901 +0.091+0.091 +0.157+0.157 GraphGPO 3×10−53×10^-5 0.810 0.828 +0.018+0.018 +0.076+0.076 Table 1: Late-window quality and learning efficiency. Late values are mean validation success over steps 175175–200200; ΔAUC _AUC is Muon minus AdamW normalized validation AUC over steps 0–200200. AdamW uses 10−610^-6 in every row. Bold marks the best tested Muon rate for each estimator under the corresponding metric. Figure 2: Per-task final-six mean validation success across all three estimators. Gray circles denote AdamW; blue squares/triangles denote Muon rates. GRPO is evaluated at 3×10−53× 10^-5, while GiGPO and GraphGPO include both tested Muon rates. Offsets are visual only. GiGPO shows the largest observed effect. Averaged over the final six evaluations, Muon (3×10−53× 10^-5) reaches 0.550.55 versus 0.290.29 for AdamW, an 88%88\% relative improvement. It leads consistently from step ∼ 5050 onward and reaches 0.630.63 versus 0.320.32 at the final checkpoint, approximately a twofold difference. Its normalized AUC is 0.240.24 versus 0.110.11 for AdamW; Muon at 10−510^-5 has a slightly higher normalized AUC of 0.260.26. Detailed values appear in Table 1 and Appendix C. Observed task-level gains span all three estimators. Figure 2 compares final-window success across the six ALFWorld task categories. Under GRPO at 3×10−53× 10^-5, the largest gains are on Clean, Pick Two, and Pick, while Look slightly favors AdamW. GiGPO gains most on Cool, Pick Two, and Heat; GraphGPO gains most on Look, Pick Two, and Pick. Thus no aggregate gain is driven by one category; Figure 2 reports the task-level magnitudes. Learning-Rate Controls The Muon recipe changes both the update rule and the effective step size of matrix parameters (1×10−6→3×10−51× 10^-6→ 3× 10^-5). To test whether a larger nominal AdamW rate alone reproduces the gain, we stress-test AdamW at 10−510^-5 and 3×10−53× 10^-5 (Table 2). Both higher-rate AdamW controls lose all post-update validation success, accompanied by KL spikes and action-format degradation. Meanwhile Muon completes the full 200200 steps at 3×10−53× 10^-5 with policy entropy broadly comparable to the matched AdamW baseline (Figure 3). In these GiGPO controls, increasing AdamW’s nominal learning rate does not reproduce the observed Muon gain. Muon retains task success through 200200 updates at the tested nominal rate, whereas the AdamW controls lose task success. Supervised fine-tuning experiments likewise find Muon sensitive to learning rate on Adam-pretrained models (Qu et al. 2026); here, Muon’s spectrally-normalized updates yield useful policy improvement at its tested rate, whereas the corresponding high-rate AdamW controls lose validation success. Figure 3: Matched optimizer diagnostics under GiGPO (top) and GraphGPO (bottom). Curves are 5-step trailing means over raw step-level logs; entropy is mean policy entropy over response tokens in nats. AdamW is gray, and blue shades distinguish the two Muon learning rates. Gradient norms remain on the same scale across configurations, while KL movement depends on both estimator and Muon learning rate. Optimizer (lr) Success retained? Final Failure signature AdamW (1×10−61×10^-6) ✓ 0.320 – AdamW (1×10−51×10^-5) × 0.000 Zero from steps 5–200 AdamW (3×10−53×10^-5) × 0.000 Zero from steps 5–200 Muon (3×10−53×10^-5) ✓ 0.633 – Table 2: Learning-rate controls under GiGPO (all else shared). Final-checkpoint success reported. At the nominal rates tested, both high-rate AdamW controls complete 200200 optimizer steps but lose all validation success after the initial step-0 evaluation, while Muon retains task success. Training Dynamics Figure 3 places GiGPO and GraphGPO under the same diagnostic view. Muon induces greater policy movement under GiGPO, while the matched GraphGPO runs retain similar KL and gradient-norm scales at the stronger 10−510^-5 setting. Raising the GraphGPO Muon rate increases policy movement but weakens the late-window result. These diagnostics accompany the estimator-dependent variation in useful Muon learning rate. The Benefit Varies Across Advantage Estimators We repeat the comparison under GRPO and GraphGPO on the same environment, model, and training setup (Figure 1). Under GRPO, Muon reaches 0.270.27 versus 0.160.16 late, but normalized AUC improves only from 0.080.08 to 0.090.09, so most of the observed separation occurs late. Under GiGPO, the 10−510^-5 and 3×10−53× 10^-5 Muon runs improve normalized AUC by about 0.150.15 and 0.130.13; the lower rate reaches 0.50.5 success at step 155155, 2020 steps earlier, while the higher rate has the better late-window mean (0.550.55 versus 0.510.51). GraphGPO exposes the clearest saturation effect. Muon runs at 10−510^-5 and 3×10−53× 10^-5 raise normalized AUC from AdamW’s 0.400.40 to 0.560.56 and 0.470.47, although the corresponding late-window gains narrow to about 0.090.09 and 0.020.02. The 10−510^-5 run crosses 0.5/0.750.5/0.75 success at steps 80/11080/110, compared with 110/170110/170 for AdamW. Together, the normalized AUC and threshold crossings indicate earlier learning under the stronger Muon setting. Step-Level Credit Ablation Final-six success Norm. AUC ω AdamW Muon AdamW Muon 0 0.141 0.361 0.063 0.132 1 0.290 0.546 0.114 0.241 Table 3: Matched 2×22× 2 GiGPO step-credit ablation. AdamW uses 10−610^-6 and Muon uses 3×10−53× 10^-5. Table 3 completes the matched optimizer-by-ω comparison. Muon improves final-window success at both ω=0ω=0 (+0.220+0.220) and ω=1ω=1 (+0.255+0.255). Enabling the step term also improves both optimizers, from 0.1410.141 to 0.2900.290 for AdamW and from 0.3610.361 to 0.5460.546 for Muon. The AUC gap grows from +0.069+0.069 to +0.127+0.127, suggesting that step-level credit and Muon may be complementary; Appendix C shows the complete trajectories. Discussion: Reconciling Positive and Negative Evidence Our GRPO trajectory and the negative RLVR results (Fan et al. 2026; Wei 2026a) indicate that episode-level advantages alone do not determine Muon’s behavior. Those studies use single-turn tasks and different learning-rate regimes, whereas our GRPO run uses long-horizon agentic trajectories. The matched GiGPO factorial shows that Muon improves both episode-level and step-level settings, while the step term improves both optimizers. The complete GraphGPO runs show a complementary sensitivity: reducing Muon’s rate from 3×10−53× 10^-5 to 10−510^-5 raises normalized AUC from 0.470.47 to 0.560.56, against AdamW’s 0.400.40, while late-window means rise from 0.830.83 to 0.900.90, against AdamW’s 0.810.81. These results motivate two complementary routes: optimizer-side spectral modifications, such as Hopper’s combined variance-normalized, single-step NS recipe (Wei 2026b) or Pion’s high-pass filtering (Fan et al. 2026), and advantage estimators with additional step-level structure. Testing the SNR hypothesis requires measuring the effective rank of policy-gradient or update matrices across estimators. The Qwen2.5 report does not disclose the pretraining optimizer. Our controlled comparisons vary the policy optimizer within each advantage-estimator implementation. Limitations This is an exploratory single-seed study on one 0.50.5B model and ALFWorld; the reported values therefore describe individual trajectories rather than population-level uncertainty. Learning-rate coverage is limited: GiGPO and GraphGPO include two Muon rates, GRPO includes only 3×10−53× 10^-5, and none is a broad sweep. The matched ω ablation is also single-seed, and high-rate AdamW controls cover only GiGPO. The mechanism evidence is correlational and does not directly measure update spectra or gradient SNR. Finally, the full-matrix NO_SHARD implementation increases per-device memory and will require distributed Muon variants for larger models. Conclusion Across the matched runs, Muon’s observed RL behavior varies with the optimizer–estimator pair. With GiGPO’s anchor-state step advantages, hidden-matrix Muon raises final-window success by 88%88\% over AdamW and retains task success through 200200 updates at a nominal rate for which the tested GiGPO AdamW controls lose task success. At 3×10−53× 10^-5, Muon also improves GRPO but has only a small full-trajectory AUC gain. The matched GiGPO ablation shows that Muon’s gain persists at ω=0ω=0, while step-level credit improves both optimizers. Under GraphGPO, late-window differences narrow near saturation, while Muon at 10−510^-5 raises normalized AUC by about 0.160.16 and reaches fixed thresholds earlier. GiGPO at 10−510^-5 also learns earlier, though with a slightly lower late-window mean. Together, these exploratory results motivate studying advantage estimation jointly with the policy optimizer. Broader seeds, models, and environments are needed to determine how consistently the pattern recurs. Acknowledgments We thank Chenyu Zheng for helpful discussions. References A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker (2024) Back to basics: revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Cited by: Group-based RL for LLM agents.. L. Chen, J. Li, and Q. Liu (2025) Muon optimizes under spectral norm constraints. arXiv preprint arXiv:2506.15054. External Links: 2506.15054, Document, Link Cited by: Muon and matrix-aware optimizers.. X. Cheng, S. He, L. Feng, H. Xu, M. Yan, L. Feng, and B. An (2026) Beyond trajectory-level attribution: Graph-Based credit assignment for agentic reinforcement learning. arXiv preprint arXiv:2605.26684. Note: Accepted at ICML 2026 External Links: Link Cited by: Introduction, Group-based RL for LLM agents.. C. Fan, G. Liu, M. Hong, R. R. Kompella, and S. Liu (2026) Rethinking muon beyond pretraining: spectral failures and high-pass remedies for VLA and RLVR. arXiv preprint arXiv:2605.19282. Cited by: Introduction, Introduction, Muon and matrix-aware optimizers., Muon in post-training and optimizer mismatch., Discussion: Reconciling Positive and Negative Evidence. L. Feng, Z. Xue, T. Liu, and B. An (2025) Group-in-group policy optimization for LLM agent training. In Advances in Neural Information Processing Systems, Note: arXiv:2505.10978 Cited by: Introduction, Introduction, Group-based RL for LLM agents., GiGPO., GiGPO., Environment and model., Compared configurations.. A. Gupta, R. Celente, A. Shivanna, D. T. Braithwaite, G. Dexter, S. Tang, H. Udagawa, D. Silva, R. Ramanath, and S. S. Keerthi (2025) Effective quantization of muon optimizer states. arXiv preprint arXiv:2509.23106. External Links: 2509.23106, Document, Link Cited by: Muon in post-training and optimizer mismatch.. K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein (2024) Muon: an optimizer for hidden layers in neural networks. Note: https://kellerjordan.github.io/posts/muon/ Cited by: Introduction, Muon and matrix-aware optimizers., Muon.. Kimi Team (2025) Kimi k2: open agentic intelligence. arXiv preprint arXiv:2507.20534. Cited by: Introduction, Muon and matrix-aware optimizers., Muon in post-training and optimizer mismatch.. W. Kool, H. van Hoof, and M. Welling (2019) Buy 4 REINFORCE samples, get a baseline for free!. In ICLR Workshop on Deep RL Meets Structured Prediction, Cited by: Group-based RL for LLM agents.. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, Cited by: Environment and model.. J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, et al. (2025) Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982. Cited by: Introduction, Muon and matrix-aware optimizers., Learning-rate comparability.. Y. Liu, J. Wang, and T. Zhang (2026) Optimizer-model consistency: full finetuning with the same optimizer as pretraining forgets less. arXiv preprint arXiv:2605.06654. External Links: 2605.06654, Document, Link Cited by: Introduction, Muon in post-training and optimizer mismatch.. NVIDIA (2026) Muon optimizer. Note: https://docs.nvidia.com/nemo/rl/latest/guides/muon-optimizer.html Cited by: Introduction. Prime Intellect Team, M. Senghaas, F. Obeid, et al. (2025) INTELLECT-3: technical report. arXiv preprint arXiv:2512.16144. Cited by: Muon in post-training and optimizer mismatch.. X. Qu, P. Huang, and S. Horvath (2026) Can muon fine-tune Adam-pretrained models?. arXiv preprint arXiv:2605.10468. External Links: 2605.10468, Document, Link Cited by: Introduction, Muon in post-training and optimizer mismatch., Learning-Rate Controls. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: Introduction, Group-based RL for LLM agents.. G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2024) HybridFlow: a flexible and efficient RLHF framework. arXiv preprint arXiv:2409.19256. Cited by: Environment and model.. M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht (2021) ALFWorld: aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, Cited by: Introduction, Environment and model.. J. Su (2025) A guide to the Muon optimizer: quick start and key details. Note: Scientific Spaces, https://w.kexue.fm/archives/11416 Cited by: Learning-rate comparability.. J. Wei (2026a) Field notes: why muon “hollows out” in RL (and what we plan to DO next). Note: https://huggingface.co/blog/bird-of-paradise/training-rl-with-muon-2 Cited by: Introduction, Muon in post-training and optimizer mismatch., Discussion: Reconciling Positive and Negative Evidence. J. Wei (2026b) Hopper: the optimizer that learns parallelism 2x faster than Adam. Note: https://huggingface.co/blog/bird-of-paradise/training-rl-with-muon-3 Cited by: Introduction, Introduction, Muon in post-training and optimizer mismatch., Discussion: Reconciling Positive and Negative Evidence. A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, et al. (2024) Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: Introduction, Environment and model.. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, et al. (2025) DAPO: an open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: Group-based RL for LLM agents.. Appendix A A. Experimental Hyperparameters Parameter Value Shared setup Model / environment Qwen2.5-0.5B-Instruct / ALFWorld Training horizon / evaluation cadence 200 updates / every 5 updates Max environment steps / history length 50 / 2 Group size / train batch / validation batch 8 / 16 / 128 Max prompt / response length 2048 / 512 PPO mini-batch / epochs 128 / 1 Rollout / validation temperature 1.0 / 0.4 (sampling enabled) Hardware 8×8× NVIDIA H20 Optimizer and loss Policy optimizer Muon on hidden 2D matrices; AdamW fallback otherwise AdamW baseline / fallback lr 1×10−61× 10^-6 Muon momentum / Nesterov / NS steps 0.95 / yes / 5 Weight decay 0.01 KL loss coefficient / type / KL in reward 0.01 / low-var KL / no Invalid-action penalty coefficient 0.01 Estimator-specific settings GRPO Muon lr 3×10−53× 10^-5 GiGPO return discount γ=0.95γ=0.95; mean-std norm; step weight ω=1ω=1 (main), 0 (ablation); Muon lr 1×10−5, 3×10−51× 10^-5,\ 3× 10^-5 GraphGPO distance decay γ=0.10γ=0.10; mean-std norm; step / episode weights 1/11/1; Muon lr 1×10−5, 3×10−51× 10^-5,\ 3× 10^-5 GraphGPO distance normalization / similarity disabled / disabled Table 4: Core training configuration. Low-level execution settings are reported separately in Appendix B. Appendix B B. Implementation Notes for Muon Execution. Experiments use PyTorch 2.11 with CUDA 13, veRL 0.3, and vLLM 0.22. Sharding. Newton–Schulz orthogonalization operates on full 2D matrices. Our implementation forces FSDP NO_SHARD on the policy so that Muon sees unsharded parameters; this is mathematically the reference Muon update but increases per-device memory. Scaling to larger models therefore requires a distributed Muon implementation. Appendix C C. Additional Per-Algorithm Diagnostics C.1 GiGPO Same-Step Validation Values Table 5 preserves the same-step GiGPO values underlying the aggregate comparison in the main paper. Step AdamW (10−610^-6) Muon (3×10−53×10^-5) 50 0.047 0.094 75 0.062 0.125 100 0.133 0.211 125 0.148 0.398 150 0.164 0.453 165 0.195 0.414 175 0.320 0.531 180 0.289 0.523 185 0.227 0.469 190 0.289 0.562 195 0.297 0.555 200 0.320 0.633 Mean (last 6 evals) 0.290 0.546 Table 5: GiGPO same-step validation success on ALFWorld. The final checkpoint is the within-window maximum for both runs; the final-window mean remains the headline statistic. C.2 Matched Step-Credit Factorial Figure 4 shows the complete matched trajectories. Muon retains an advantage when the GiGPO step term is disabled, while enabling the term raises both the AdamW and Muon trajectories. The larger AUC separation at ω=1ω=1 motivates testing the interaction across additional seeds. Figure 4: Matched GiGPO optimizer-by-step-credit ablation. The two panels share axes and differ only in the step weight. Curves show raw validation checkpoints every five training steps. C.3 GRPO Diagnostics The aggregate GRPO comparison reported in the main paper is supported by the task-level trajectories in Figure 5. Muon has the largest final-window gains on Clean, Pick Two, and Pick. Heat and Cool improve more modestly, while Look slightly favors AdamW. Thus, the aggregate improvement is not uniform across task categories; the figure reports the corresponding magnitudes. Figure 5: Per-task validation success for the complete GRPO runs. Curves are evaluated every five training steps. Each title reports Muon minus AdamW mean success over the final six evaluations (steps 175–200). Figure 6 shows that both GRPO runs retain nonzero validation success through step 200200. Muon moves farther from the reference policy while remaining on the same gradient-norm scale. Figure 6: Training dynamics for the complete matched GRPO runs. Curves are 5-step trailing means over raw step-level logs. Entropy is mean policy entropy over response tokens in nats. Muon exhibits larger policy movement while both runs remain within the plotted gradient-norm range. C.4 GiGPO and GraphGPO Task Trajectories Figure 7 provides the complete GiGPO and GraphGPO per-task validation trajectories summarized in the main paper. Both Muon learning rates are shown for each estimator. Figure 7: Complete per-task validation trajectories for GiGPO (top) and GraphGPO (bottom), using the same visual encoding and axis scales. Curves show raw checkpoints every five training steps for AdamW and both tested Muon learning rates. For GraphGPO, 10−510^-5 gives the strongest aggregate final-window result and exceeds 3×10−53× 10^-5 on Pick, Look, Clean, and Pick Two. Appendix D D. A Credit-Quality Conjecture for Muon This appendix formalizes the deliberately simplified mechanism proposed in the main paper. It is not a variance theorem for GiGPO or GraphGPO. We analyze one layer and one update, omitting clipping, KL regularization, momentum history, and singular-vector rotation so that the role of credit noise is explicit. D.1 Long-Horizon Credit Noise Let Zt=∇Wlogπθ(at∣st),G^=∑t=1TA^tZt=G⋆+E.Z_t= _W _θ(a_t s_t), G= _t=1^T A_tZ_t=G +E. Suppose an idealized trajectory has a set ℐI of K≪TK T decisions carrying task-relevant signal. Write ete_t for the matrix error contributed at step t. If these errors are zero mean, uncorrelated across time, and have equal energy ‖et‖F2=σ2E\|e_t\|_F^2=σ^2, then a trajectory-level estimator that retains all steps has ‖Eepisode‖F2=‖∑t=1Tet‖F2=Tσ2.E\|E_episode\|_F^2=E \| _t=1^Te_t \|_F^2=Tσ^2. (5) An oracle local estimator that suppresses errors outside ℐI instead gives ‖Elocal‖F2=‖∑t∈ℐet‖F2=Kσ2.E\|E_local\|_F^2=E \| _t e_t \|_F^2=Kσ^2. (6) Holding ‖G⋆‖F2\|G \|_F^2 fixed yields SNRlocalSNRepisode≈TK. SNR_localSNR_episode≈ TK. (7) Temporal covariance adds 2∑t<u⟨et,eu⟩F2 _t<uE e_t,e_u _F, so the scaling can be smaller or larger. Actual GiGPO also retains its episode-level term and does not literally gate all irrelevant decisions; T/KT/K is therefore an interpretive limit, not an empirical estimate. D.2 Two Ways to Remove Credit Confounders Anchor-state contrasts. For actions sampled from a repeated anchor state s, consider Rj=b(s)+q(s,aj)+ϵj,R_j=b(s)+q(s,a_j)+ _j, where b(s)b(s) is shared state difficulty, q(s,aj)q(s,a_j) distinguishes the action, and ϵj _j is rollout noise. Centering within the anchor group gives the exact identity Rj−R¯s=q(s,aj)−q¯s+ϵj−ϵ¯s.R_j- R_s=q(s,a_j)- q_s+ _j- ε_s. (8) The shared difficulty b(s)b(s) cancels. With n iid errors of variance σ2σ^2, Var(ϵj−ϵ¯s)=σ2(1−1/n)Var( _j- ε_s)=σ^2(1-1/n). This slice explains how GiGPO’s normalized within-state comparison can replace a trajectory-level confounder with a more local action contrast; it does not make the estimate noise free. Potential-like transition credit. If a state graph induces a progress potential Ψ(s)≈−d(s,goal) (s)≈-d(s,goal), an idealized transition signal is δtΨ=γΨ(st+1)−Ψ(st) _t =γ (s_t+1)- (s_t). Its discounted sum telescopes: ∑t=0T−1γtδtΨ=−Ψ(s0)+γTΨ(sT). _t=0^T-1γ^t _t =- (s_0)+γ^T (s_T). (9) This identity shows how a delayed terminal quantity can be distributed across local transitions without accumulating arbitrary intermediate potential terms. GraphGPO is not identical to potential-based reward shaping; Equation 9 is an analogy for why distance-based transition credit can reduce long-horizon delay. D.3 Why Directional Reliability Matters to Muon Consider a fixed-singular-direction model G⋆=Udiag(si)V⊤,G^=Udiag(si+ϵi)V⊤,G =Udiag(s_i)V , G=Udiag(s_i+ _i)V , with si>0s_i>0 and independent ϵi∼(0,σi2) _i (0, _i^2). The ideal polar update uses sign(si+ϵi)sign(s_i+ _i) in direction i, so Pr[si+ϵi>0]=Φ(si/σi). [s_i+ _i>0]= (s_i/ _i). Taking the Frobenius inner product with G⋆G gives ⟨(G^),G⋆⟩F=∑isi[2Φ(si/σi)−1].E ( G),G _F= _is_i\! [2 (s_i/ _i)-1 ]. (10) A direction with si/σi≫1s_i/ _i 1 contributes almost sis_i in expectation, whereas a direction with si/σi≈0s_i/ _i≈ 0 contributes almost zero. Muon equalizes singular magnitudes, so a weak direction is useful only if its sign and singular vectors are reliable. Let Πtail _tail project onto weak signal directions. The paper’s mechanism conjecture is that, in some long-horizon agentic settings, finer-grained credit approximately satisfies ‖ΠtailElocal‖F2≤c‖ΠtailEepisode‖F2,c<1,E\| _tailE_local\|_F^2≤ c\,E\| _tailE_episode\|_F^2, c<1, (11) without removing the corresponding signal. In the fixed-basis Gaussian model, Equation 10 is monotone in every si/σis_i/ _i, implying better expected alignment of the local-credit polar update with G⋆G . Extending that implication to rotating singular vectors and optimizer momentum is an open problem. D.4 Testable Predictions and Non-Claims The conjecture predicts that finer-grained credit should improve weak-tail directional SNR or sign agreement before it changes saturated final success. This motivates the normalized-AUC and time-to-threshold analysis in the main paper. In GiGPO, it predicts a benefit from activating ω under both optimizers and, if credit quality is especially important for Muon, a larger effect under Muon. A paired optimizer-by-ω comparison is needed to distinguish those two possibilities. GraphGPO may instead show an early learning gain despite limited late-window headroom. Future experiments can test the conjecture by logging per-layer update spectra, effective rank, tail-projected signal-to-noise estimates, and cross-minibatch sign agreement under matched AdamW and Muon runs. The current single-seed results establish none of these spectral claims: correlated errors can violate the T/KT/K slice, better scalar credit need not improve matrix tail directions, and Newton–Schulz approximation plus momentum can depart from the ideal polar model. Appendix E E. GiGPO Learning-Rate Control Diagnostics Figure 8 distinguishes unsuccessful policy behavior from process termination in the two GiGPO high-learning-rate AdamW controls. Both runs complete all 200200 optimizer updates, yet every validation checkpoint after step 0 has zero success. AdamW at 3×10−53× 10^-5 ends with fully clipped responses and zero valid actions. AdamW at 10−510^-5 follows a less monotone trajectory but also develops length saturation, declining action validity, and large late-run KL excursions. In contrast, Muon at 3×10−53× 10^-5 retains valid behavior and improves validation success. Figure 8: GiGPO learning-rate stress diagnostics. Validation curves show raw checkpoints every five steps; the other panels show 5-step trailing means. The two AdamW validation curves overlap at zero after step 0. KL loss uses a logarithmic axis.