Paper deep dive

The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

Yihe Zhou, Tongtian Zhu, Yingxiao Huo, Satya Prakash Dash, Can Wang, Samuel Kaski, Mingfei Sun

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 98

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Adam is the default optimizer for training modern deep neural networks, yet its adaptive behavior remains poorly understood due to the complex interaction between its first- and second-moment exponential moving averages (EMAs). We study Adam in the tied-$\beta$ regime, where the two EMA decay rates are equal, and show that its adaptive dynamics can be expressed through a transformed ratio with approximately scale-stable behavior. Empirically, this transformed ratio exhibits a stable, heavy-tailed distribution across tasks, model scales, and training stages, in contrast to the variability of raw moment magnitudes. This empirical stability has both practical and conceptual consequences. First, we derive a recurrence for the transformed ratio, yielding a reparameterization of Adam that replaces the second moment with a compressible state. Leveraging its stable distribution, we show that a fixed 4-bit codebook is sufficient in our experiments to store this state without auxiliary scaling, achieving performance competitive with full-precision Adam. Second, the transformed ratio view clarifies Adam's connection to sign-based methods: Adam reduces to sign-based momentum modulated by the transformed ratio, and replacing it with a constant recovers Signum as a limiting case. This perspective further provides a simple rule for transferring learning rates between the two methods. Together, these results suggest that tied-$\beta$ Adam admits a simple and approximately stable ratio structure underlying its adaptive behavior and demonstrate its utility for both analysis and efficient implementation.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

95,052 characters extracted from source content.

Expand or collapse full text

The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics Yihe Zhou1 Tongtian Zhu2 Yingxiao Huo1 Satya Prakash Dash1 Can Wang2 Samuel Kaski1 Mingfei Sun1 1Department of Computer Science, The University of Manchester, Manchester, UK 2College of Computer Science and Technology, Zhejiang University, Hangzhou, China yihe.zhou,yingxiao.huo,satyaprakash.dash@postgrad.manchester.ac.uk samuel.kaski,mingfei.sun@manchester.ac.uk raiden,wcan@zju.edu.cn Abstract Adam is the default optimizer for training modern deep neural networks, yet its adaptive behavior remains poorly understood due to the complex interaction between its first- and second-moment exponential moving averages (EMAs). We study Adam in the tied-β regime, where the two EMA decay rates are equal, and show that its adaptive dynamics can be expressed through a transformed ratio with approximately scale-stable behavior. Empirically, this transformed ratio exhibits a stable, heavy-tailed distribution across tasks, model scales, and training stages, in contrast to the variability of raw moment magnitudes. This empirical stability has both practical and conceptual consequences. First, we derive a recurrence for the transformed ratio, yielding a reparameterization of Adam that replaces the second moment with a compressible state. Leveraging its stable distribution, we show that a fixed 4-bit codebook is sufficient in our experiments to store this state without auxiliary scaling, achieving performance competitive with full-precision Adam. Second, the transformed ratio view clarifies Adam’s connection to sign-based methods: Adam reduces to sign-based momentum modulated by the transformed ratio, and replacing it with a constant recovers Signum as a limiting case. This perspective further provides a simple rule for transferring learning rates between the two methods. Together, these results suggest that tied-β Adam admits a simple and approximately stable ratio structure underlying its adaptive behavior and demonstrate its utility for both analysis and efficient implementation. 1 Introduction Adam [22, 28] is the default optimizer for training deep neural networks, from vision models [14, 26] to large language models [10, 5, 39, 36, 32, 1]. Yet, despite its ubiquity, a fundamental question remains unresolved: what determines Adam’s adaptive behavior across scales, tasks, and training regimes? In particular, the interaction between its two exponential moving averages (EMAs) remains difficult to interpret. Specifically, Adam maintains EMAs of the gradient mean mtm_t and second moment vtv_t, controlled by β1 _1 and β2 _2. The update normalizes a momentum-like direction by a scale estimate of recent gradients, producing coordinate-wise step sizes. While this mechanism is simple, its dynamics depend intricately on gradient scale and noise, obscuring a clean, invariant characterization. Recent large-scale training increasingly uses closer decay rates, e.g., (0.9,0.95)(0.9,0.95) [19, 21, 33, 7], motivating the tied-β regime β1=β2 _1= _2. This regime is both practically relevant and structurally simpler, as Adam admits a clearer interpretation: its update direction resembles a smoothed sign-based momentum, while the adaptive denominator modulates the step magnitude through a noise-to-signal ratio of stochastic gradients [31]. Complementary analyses suggest that matching the two decay rates introduces useful structural properties, including robustness to gradient scaling and improved stability under nonstationary noise [12, 16, 6]. In this work, we show that tied-β Adam admits an unexpected simplification: its adaptive dynamics can be reparameterized through an approximately scale-stable ratio. Specifically, decomposing the second moment and factoring out the shared β-dependent scale yields a natural-scale ratio yty_t (i.e., the hidden ratio) that isolates stochastic fluctuations from deterministic scaling. Our key finding is that yty_t follows a remarkably stable distribution across models, tasks, and training stages. Unlike raw moment magnitudes, which vary with gradient scale, the distribution of yty_t is concentrated with a persistent heavy right tail. This suggests a simple underlying structure: tied-β Adam combines a deterministic scale with an approximately stable law governing adaptive attenuation. This perspective has two immediate consequences. The first is representational. In the tied-β regime, Adam can be equivalently parameterized by (mt,yt)(m_t,y_t) instead of (mt,vt)(m_t,v_t). Since yty_t has a stable and compressible distribution, it admits aggressive quantization. We show that a fixed 4-bit codebook (16 levels) suffices to nearly match full-precision Adam performance, without requiring additional block-wise or tensor-wise scaling factors. To our knowledge, this is the first result demonstrating that a transformed second-moment-derived state in Adam can be quantized to 4-bit precision with a fixed codebook while retaining strong performance. The second consequence is both practical and conceptual. The ratio formulation clarifies Adam’s relationship to Signum (SignSGD with momentum) [38, 3]. In the tied-β regime, Adam can be viewed as a sign-based momentum method whose step magnitude is attenuated by yty_t. Averaging this attenuation under the stable distribution of yty_t yields a simple prediction for the effective learning-rate ratio between Adam and its sign-dominated counterpart, which aligns with our empirical observations. Furthermore, replacing yty_t with a constant reduces Adam to a Signum-type method with fixed attenuation, showing that a constant-state approximation recovers a Signum-like limit of tied-β Adam. This view provides a principled way to transfer or initialize learning rates between the two methods. We do not claim that tied-β Adam universally dominates standard Adam configurations or that the transformed-ratio law fully characterizes optimizer dynamics globally. Rather, our contributions are: • We identify a scale-stable ratio yty_t in tied-β Adam with a stable, heavy-tailed distribution across tasks and models. Truncating or coarsely discretizing yty_t incurs negligible performance loss. • We identify and derive a recursive characterization of yty_t, enabling a (mt,yt)(m_t,y_t) parameterization and a 4-bit fixed-codebook implementation without auxiliary scaling. • The ratio view yields a learning-rate transfer rule and recovers Signum as a constant-state limit. 2 Background and Related Work Adam [22] and Signum [3, 38]. As an iterative adaptive optimizer, Adam maintains exponential moving averages of the gradient gtg_t and its square at each iteration t, mt:=β1mt−1+(1−β1)gt,vt:=β2vt−1+(1−β2)gt2. m_t:= _1m_t-1+(1- _1)g_t, v_t:= _2v_t-1+(1- _2)g_t^2. (1) Since both averages are initialized to zero, Adam uses the bias-corrected elementwise scaling: m^t:=mt1−β1t,v^t:=vt1−β2t,Δθt:=−ηm^tv^t+ϵ, m_t:= m_t1- _1^t, v_t:= v_t1- _2^t, _t:=-η m_t v_t+ε, (2) where ϵε is a small positive constant for numerical stability. Thus, Adam can be viewed as normalizing an exponential estimate of the gradient mean by an exponential estimate of its root second moment. In contrast, Signum maintains a first-moment EMA and applies a sign update: mt:=βmt−1+(1−β)gt,Δθt:=−ηsign⁡(mt). m_t:=β m_t-1+(1-β)g_t, _t:=- (m_t). (3) The update keeps only the direction of the momentum and uses a fixed coordinatewise step magnitude. A growing line of work has sought to understand Adam through structural interpretations of its update rule, rather than only through empirical tuning heuristics. Early analyses separated the roles of momentum and adaptive scaling, interpreting Adam as combining a sign-based directional component with a variance-controlled magnitude term [2], or more generally disentangling momentum from adaptivity in Adam-style methods [27]. More recent work further expresses Adam’s adaptive scaling through a local signal-to-noise ratio, giving the denominator a direct statistical interpretation [31]. Under this view, Signum [3, 38] can be related to steepest descent under an ℓ∞ _∞ trust-region geometry, while Adam corresponds to a signed-momentum direction whose effective step magnitude is modulated by local signal-to-noise structure. Our work follows this structural perspective, but identifies within Adam’s scaling term a comparatively stable distributional quantity that can be represented directly. Tied-β regime. Our analysis focuses on the tied-β case, β1=β2=β _1= _2=β. Ignoring bias correction and ϵε for clarity, the adaptive factor reduces to mtvt m_t v_t. Because mtm_t and vtv_t are then computed with the same exponential weights, vt−mt2v_t-m_t^2 has a natural variance-like interpretation. In fact, vt≥mt2v_t≥ m_t^2 for all gradient sequences and all t if and only if β1=β2 _1= _2; the proof is deferred to Appendix B. In this case, Adam admits a particularly clean online mean–variance interpretation [31], and satisfies a first-order gradient scale-invariance property that singles out tied β values as a structurally meaningful choice [16]. Analyses of mini-batch noise further suggest that the preferred relation between β1 _1 and β2 _2 depends on the noise regime, with larger batch sizes often favoring β1 _1 closer to β2 _2 in terms of validation performance [6]. Complementary empirical studies in nonstationary optimization and reinforcement learning indicate that mismatched first- and second-moment timescales can weaken normalization and destabilize updates [15, 12], whereas tied or near-tied decay rates can improve stability in practice [30, 13, 18]. Motivated by these observations, we focus on β1=β2 _1= _2 when analyzing Adam’s ratio recursion and distributional behavior, since this is the regime where prior work suggests both cleaner dynamics and improved stability. 3 Hidden Ratio: Theoretical and Empirical Properties In the tied-β regime, define rt:=vt−mt2mt2,mtvt=sign⁡(mt)1+rt. r_t:= v_t-m_t^2m_t^2, m_t v_t= sign(m_t) 1+r_t. Thus the update direction is determined by the sign of the momentum, while rtr_t controls the magnitude attenuation. Since rt≥0r_t≥ 0 under tied β values, the denominator is bounded below by 11 [31]. We now define the transformed ratio yt:=1−βrt,mtvt=sign⁡(mt)1+β1−βyt. y_t:= 1-βr_t, m_t v_t= sign(m_t) 1+ β1-βy_t. (4) This reparameterization identifies yty_t as the central quantity in the Adam update, thereby motivating a closer examination of its statistical structure. Intuitively, yty_t measures the variance-like residual in Adam’s denominator after removing the deterministic scale induced by tied exponential averaging. 3.1 Statistical Structure of yty_t: Stability and Heavy Tails Since mini-batches are sampled independently, we follow the diffusion-approximation view of mini-batch SGD and model the local gradient noise by independent Gaussian fluctuations [29, 20, 23]. Assumption 1 (Local Gaussian window). For one coordinate in a local time window, gt−j=μ+σξj,ξj∼i.i.d.⁡(0,1),j≥0.g_t-j=μ+σ _j, _j .i.d. N(0,1), j≥ 0. Fix 0<β<10<β<1 and let wj=(1−β)βjw_j=(1-β)β^j, mt=∑j≥0wjgt−jm_t= _j≥ 0w_jg_t-j, vt=∑j≥0wjgt−j2v_t= _j≥ 0w_jg_t-j^2, Ut=vt−mt2U_t=v_t-m_t^2, yt=1−βUtmt2y_t= 1-β U_tm_t^2, sβ2=∑j≥0wj2=1−β1+βs_β^2= _j≥ 0w_j^2= 1-β1+β, σm=σsβ _m=σ s_β, a=μ/σma=μ/ _m, and Z=sβ−1∑j≥0wjξjZ=s_β^-1 _j≥ 0w_j _j. Under Assumption 1, Z∼⁡(0,1)Z (0,1) and mt=σm(Z+a)m_t= _m(Z+a). When comparing many local windows, let ℐI index the observed coordinate-window instances and let aia_i be the local shift at index i. Treating these instances as equally weighted samples gives the empirical pooled-shift distribution P^A:=1|ℐ|∑i∈ℐδai,A∼P^A,P^A[A2]=1|ℐ|∑i∈ℐai2. P_A:= 1|I| _i _a_i, A P_A, _ P_A[A^2]= 1|I| _i a_i^2. Equivalently, A∼P^A P_A means drawing I uniformly from ℐI and setting A=aIA=a_I. More generally, a pooled-shift distribution P is any distribution over local shifts used in this sampling role; the empirical case is P=P^AP= P_A. For any fixed scalar shift b, write Yb:=2/(Z+b)2Y_b:=2/(Z+b)^2, so a single local window uses YaY_a. If A∼PA P is independent of Z, the pooled denominator reference is YA:=2/(Z+A)2Y_A:=2/(Z+A)^2. Proposition 1 (Shifted inverse-square reference for the ratio). Under Assumption 1 and the notation above, the transformed ratio satisfies yt=Ya(1+ηt2),ηt=ηtcoup+ηtfluc,y_t=Y_a (1+ _t2 ), _t= _t^coup+ _t^fluc, (5) where γβ=1−β1+β+β2 _β= 1-β1+β+β^2, ηtcoup:=γβ(Z2−1) _t^coup:= _β(Z^2-1), and ηtfluc:=1−βUt−⁡[Ut∣Z]σm2 _t^fluc:= 1-β U_t-E[U_t Z] _m^2. Let P be a pooled-shift distribution with finite fourth moment, and let A∼PA P be independent of Z. Then the fixed shifted reference Yc=2(Z+c)2,Z∼⁡(0,1),c2:=P[A2] aligned Y_c= 2(Z+c)^2, Z (0,1), c^2:=E_P[A^2] aligned (6) matches the mixed denominator reference YA=2/(Z+A)2Y_A=2/(Z+A)^2 through second order: for every fixed k>0k>0, writing q=2/kq= 2/k, and denoting the standard normal CDF and density by Φ and ϕφ, Pr⁡(YA>k) (Y_A>k) =2Φ(q)−1−qϕ(q)P[A2]+O⁡(P[A4]), =2 (q)-1-qφ(q)E_P[A^2]+O(E_P[A^4]), Pr⁡(Yc>k) (Y_c>k) =2Φ(q)−1−qϕ(q)c2+O⁡(c4). =2 (q)-1-qφ(q)c^2+O(c^4). (7) In particular, for a family PεP_ of pooled-shift distributions with Aε∼PεA_ P_ , cε2:=Pε[Aε2]c_ ^2:=E_P_ [A_ ^2], and Pε[Aε4]=O⁡(ε4)E_P_ [A_ ^4]=O( ^4), the corresponding references satisfy Pr⁡(YAε>k)−Pr⁡(Ycε>k)=O⁡(ε4) (Y_A_ >k)- (Y_c_ >k)=O( ^4). Remark. Equation 5 is a denominator-reference decomposition: for a fixed local shift a, Ya=2/(Z+a)2Y_a=2/(Z+a)^2 carries the inverse-square denominator, while ηt _t keeps the numerator as an exact relative correction. The pooled-shift distribution P is the formal version of flattening many observed local shifts; drawing A∼PA P selects one representative local shift, so the denominator reference becomes YA=2/(Z+A)2Y_A=2/(Z+A)^2. Equation 7 shows that the survival curve of YAY_A depends to second order only on P[A2]E_P[A^2]. Choosing c=P[A2]c= E_P[A^2] keeps this leading shift effect and the same inverse-square singularity, which is why we use YcY_c in Equation 6 as the reference for the empirical yty_t. Figure 1: Shifted inverse-square reference Yc=2/(Z+c)2Y_c=2/(Z+c)^2. For c≤0.2c≤ 0.2, the CDF and histogram are nearly unchanged; the shape is driven mainly by the inverse-square form. Figure 1 illustrates this point for the reference distribution. When the shift c is small, the CDF and histogram of Yc=2/(Z+c)2Y_c=2/(Z+c)^2 remain close to those of the unshifted reference 2/Z22/Z^2, preserving both the bulk and the right-tail shape. This suggests that the inverse-square denominator singularity drives the dominant distributional form, while local shifts and numerator fluctuations introduce only secondary distortions. The full derivation is given in Appendix A. This picture is reflected in the empirical distributions in Figure 2. Across pre-training, SFT, RLHF, and β∈0.90,0.92,0.95β∈\0.90,0.92,0.95\, the transformed ratio yty_t consistently shows a compact single-digit bulk with a persistent right tail. Within each setting, the early-, middle-, and late-stage curves remain close, indicating only mild distributional drift during training. The empirical CDFs also stay close to the 2/Z22/Z^2 reference over a broad range. Together, these observations support the view that the rescaled ratio yty_t captures a natural-scale denominator state whose distribution is more stable than the raw moment magnitudes mtm_t and vtv_t. 3.2 Robustness of yty_t to coarse quantization Given that the distribution of yty_t exhibits a clearly concentrated bulk together with a long right tail, we now investigate whether a very coarse quantization of yty_t is sufficient to retain behavior and performance close to those of the original Adam. To isolate its role, we keep the original Adam recursions for mtm_t and vtv_t unchanged, compute yty_t from the resulting states, and replace only this variable by a coarse low-precision approximation before reconstructing the adaptive denominator. Figure 2: Distribution of the transformed ratio yty_t across training regimes. Columns correspond to β=0.90,0.92,0.95β=0.90,0.92,0.95, while rows compare pretraining on Pythia-160M, supervised fine-tuning on Pythia-1B, and RLHF on LLaMA-1B. Within each block, the top panel shows the CDF and the bottom panel shows the histogram; colors denote early, middle, and late training stages. Figure 3: Comparison of the training performance of standard Adam and its modified variants based on the transformed ratio yty_t, including long-tail truncation and coarse 4-bit quantization. Even after truncating the right tail and representing the remaining bulk with only 16 quantization levels, the resulting optimizer still closely tracks the behavior and final performance of the standard Adam. We begin with a particularly simple modification: truncating the upper tail of yty_t. The cut64 variant replaces yty_t by min⁡(yt,64) (y_t,64) before reconstructing the adaptive denominator, thereby removing only rare far-tail values. Empirically, this truncation has little effect, and the resulting optimizer still tracks standard Adam closely. We then take a more aggressive step and replace yty_t itself by a coarse low-precision approximation. After the same cut64 truncation, we quantize yty_t using the fixed 4-bit codebook =0.25,0.5,0.75,1,1.5,2,3,4,6,8,12,16,24,32,48,64C=\0.25,0.5,0.75,1,1.5,2,3,4,6,8,12,16,24,32,48,64\, an FP4-like non-uniform codebook with levels concentrated around the empirical bulk of yty_t. The quantized value is obtained by nearest-neighbor projection: y~t=Q⁡(yt):=arg⁡minq∈|yt−q| y_t=Q(y_t):= _q |y_t-q|. Taken together, the results in Figure 3 suggest that Adam does not require precise pointwise fidelity in yty_t in order to realize its adaptive effect. What appears to matter is not the exact value of the transformed ratio at each step, but the coarse attenuation information that it carries. This is consistent with the distributional structure described above: most of the mass of yty_t lies in a relatively compact regime, while the update depends on it only through the smooth factor (1+β1−βyt)−1/2 (1+ β1-βy_t )^-1/2. As a result, neither removing rare tail events nor coarsening the representation within the bulk of the distribution significantly perturbs the effective step size. This robustness motivates the ratio-based reformulation developed in the next section. If Adam’s essential adaptive behavior is already preserved under severe truncation and coarse quantization of the transformed ratio, then the more natural object to analyze is not the raw moment pair (mt,vt)(m_t,v_t), but the induced dynamics of the transformed ratio itself. We emphasize that this is only a diagnostic experiment in Figure 3: the original Adam recursions for mtm_t and vtv_t are unchanged, and only the induced yty_t is coarsened before reconstructing the denominator. 4 Recursive Reformulation of Adam Unlike the diagnostic coarsening above, we now maintain yty_t recursively and use it as the stored second state. Given the stable structure of the transformed ratio, we now show that Adam can be equivalently parameterized by (mt,yt)(m_t,y_t) instead of (mt,vt)(m_t,v_t), motivating the low-precision state representation evaluated below. 4.1 A scalar recursion for the transformed ratio We now derive a recursion for the transformed ratio yty_t. Starting from the recursion mt=βmt−1+(1−β)gtm_t=β m_t-1+(1-β)g_t, we can rewrite the current gradient as gt=mt−βmt−11−β=mt1−β(1−βxt),xt:=mt−1mt.g_t= m_t-β m_t-11-β= m_t1-β(1-β x_t), x_t:= m_t-1m_t. (8) Substituting this into the second-moment recursion vt=βvt−1+(1−β)gt2v_t=β v_t-1+(1-β)g_t^2 gives vt=βvt−1+mt21−β(1−βxt)2.v_t=β v_t-1+ m_t^21-β(1-β x_t)^2. (9) Using vs=ms2(1+β1−βys)v_s=m_s^2(1+ β1-βy_s) for s∈t,t−1s∈\t,t-1\, together with mt−1=xtmtm_t-1=x_tm_t, Equation 9 yields yt=βxt2yt−1+(1−xt)2. aligned y_t=β x_t^2y_t-1+(1-x_t)^2. aligned (10) Equation 10 shows that, once the sign of mtm_t is fixed, Adam’s adaptive behavior is well described by a one-dimensional transformed ratio recursion: the sign determines the update direction, while yty_t controls the attenuation magnitude. Additional algebraic details are provided in Appendix C. Figure 4: Pre-training comparison between full Adam and low-precision ratio representation. 4.2 Coarse representation through the recursion Equation 10 suggests a natural target for coarse representation. Instead of compressing the coupled raw states (mt,vt)(m_t,v_t) directly, we represent the transformed ratio yty_t, which is the scalar quantity controlling the adaptive denominator. Concretely, after obtaining yty_t, we replace it by the nearest value in a fixed FP4-like codebook: y~t=Q⁡(yt):=arg⁡minq∈|yt−q| y_t=Q(y_t):= _q |y_t-q|. Here, =0.25,0.5,0.75,1,1.5,2,3,4,6,8,12,16,24,32,48,64C=\0.25,0.5,0.75,1,1.5,2,3,4,6,8,12,16,24,32,48,64\. This 16-level codebook allocates more resolution to the moderate range where the transformed ratio distribution concentrates, while still retaining several larger values for the right tail. The adaptive denominator is then reconstructed from y~t y_t through the same transformed expression as in Equation 4, with yty_t replaced by y~t y_t. Thus, coarse representation acts directly on the transformed-ratio attenuation rather than separately perturbing the raw moment states mtm_t and vtv_t. Notably, this representation uses no per-block scaling factors, learned quantizers, or auxiliary normalization metadata. 4.3 Experiments on low-precision quantization of the transformed ratio Figure 5: SFT comparison between full Adam and low-precision transformed-ratio representation. Figure 6: RLHF comparison between full Adam and low-precision transformed-ratio representation. We next examine whether the transformed ratio representation remains effective across the full language-model training pipeline. We compare tied-β Adam with its low-precision transformed ratio variant in three representative stages: pre-training, supervised fine-tuning (SFT), and reinforcement-learning-based alignment (RLHF). In all settings, the optimizer is modified only through the coarse storage representation of yty_t; the rest of the training recipe is kept unchanged. In addition to storing yty_t with a 4-bit codebook, we also evaluate variants in which the first-moment state mtm_t is stored in FP8 [35, 8, 17]. This is a useful stress test for the transformed-ratio formulation: although the recursion depends on ratios involving mtm_t, the combination of FP8 first-moment storage and 4-bit transformed-ratio storage still preserves strong performance in our experiments. Figure 7: Learning-rate sweeps on Llama-20M pre-training comparing tied-β Adam with transformed-ratio Adam using FP32 or FP8 first-moment storage and 4-bit yty_t. Figures 4, 5 and 6 show that the low-precision transformed-ratio variant closely tracks full Adam across all three stages. In pre-training, the two validation-loss curves nearly overlap throughout optimization and reach essentially the same final loss, suggesting that the transformed ratio preserves the denominator-side information needed for long-horizon large-scale training. In SFT and RLHF, the low-precision variant may initially lag the full-Adam baseline, but it improves steadily and often matches or slightly exceeds Adam later in training. Thus, the effect of coarse transformed-ratio representation is not merely harmless compression: in post-training regimes, it can behave like a mild smoothing or regularization of the adaptive denominator dynamics. We further examine whether the low-precision transformed-ratio representation preserves Adam’s learning-rate sensitivity. Figure 7 shows learning-rate sweeps on a Llama-20M pre-training setup under four tied-β values. Across β=0.90,0.92,0.95,0.97β=0.90,0.92,0.95,0.97, both transformed-ratio variants closely track the Adam sweep profile: their favorable learning-rate regions remain aligned with Adam, and performance degradation at overly large learning rates occurs at similar scales. This indicates that coarse transformed-ratio storage does not simply succeed after careful retuning at a single operating point; rather, it largely preserves Adam’s effective learning-rate structure. Overall, the results show that the transformed ratio yty_t is a robust target for coarse representation: in pre-training it recovers Adam-like behavior almost exactly, while in SFT and RLHF it remains competitive and can sometimes improve late-stage performance. This stage-consistent behavior suggests that the essential adaptive dynamics of tied-β Adam are captured by a low-complexity transformed-ratio structure rather than by high-precision values of the raw second-moment state. More experimental results are provided in Appendix D. 5 The Signum-like Limit 5.1 A constant-state approximation and the Signum-like limit The transformed-ratio representation provides a simple interpretation of Adam as a sign-momentum method with stochastic attenuation. From Equation 4, once the sign of the momentum is fixed, Adam’s adaptive behavior is controlled by the scalar attenuation factor (1+(β/(1−β))yt)−1/2(1+(β/(1-β))y_t)^-1/2. This motivates a constant-state approximation in which the transformed ratio yty_t is replaced by a representative constant value yconst≥0y_const≥ 0. The resulting Signum-type update and the corresponding learning-rate matching rule are Δθt≈−ηadam(1+β1−βyconst)−12sign⁡(mt),ηsign≈ηadam(1+β1−βyconst)−12. _t≈- _adam (1+ β1-βy_const )^- 12sign(m_t), _sign≈ _adam (1+ β1-βy_const )^- 12. (11) Here, the first expression is the constant-yty_t Signum-type update, while the second gives the learning-rate scale needed to write it as Δθtsign=−ηsignsign⁡(mt) _t^sign=- _signsign(m_t). This relation should be viewed as a matching rule: the full Adam update still retains the coordinate-wise fluctuations and temporal variation of yty_t, whereas the constant-state approximation preserves only its dominant effective scale. To choose yconsty_const, we match the expected attenuation, or equivalently the expected effective step size, under the transformed-ratio distribution: ⁡[(1+β1−βyt)−12]=(1+β1−βyconst)−12. \! [(1+ β1-βy_t)^- 12 ]=(1+ β1-βy_const)^- 12. (12) Solving for the attenuation-matched constant gives yconst=1−β([(1+β1−βyt)−12]−2−1). y_const= 1-β (E\! [(1+ β1-βy_t)^- 12 ]^-2-1 ). (13) This calibration is useful because it connects the constant-state approximation directly to Adam’s native learning-rate scale, rather than treating Signum as requiring an unrelated learning-rate search. To make the scale of yconsty_const more concrete, we substitute the shifted inverse-square reference Yc=2/(Z+c)2Y_c=2/(Z+c)^2 from Proposition 1 into this expression. This yields yconsteff(β,c)=1−β(Z∼⁡(0,1)[(1+2β(1−β)(Z+c)2)−12]−2−1). y_const^eff(β,c)= 1-β (E_Z (0,1) [(1+ 2β(1-β)(Z+c)^2)^- 12 ]^-2-1 ). (14) Numerically, this value is stable for the small shifts considered in the distributional model. As shown in Table 1, the attenuation-matched constant remains slightly above 33 across β∈[0.90,0.97]β∈[0.90,0.97] and c∈0,0.1,0.2c∈\0,0.1,0.2\. Thus, under the same shifted-reference law used to describe the empirical transformed-ratio distribution, a moderate constant transformed-ratio value should already capture Adam’s typical attenuation scale. A practical advantage of this view is that it expresses a Signum-like optimizer in Adam’s native learning-rate parameterization. In modern language-model training, Adam learning-rate heuristics are well established, whereas Signum often requires a separate learning-rate search. Replacing yty_t with a moderate constant therefore yields a simplified sign-style update that remains calibrated to the Adam learning-rate scale, making it possible to reuse Adam-style tuning rather than introducing a separate optimizer-specific search. We therefore test several representative values of yconsty_const and compare them against Adam and Signum in learning-rate sweeps across different values of β. 5.2 Learning-rate sweeps across β: empirical observations Figure 8: Learning-rate sweeps: Signum shifts to smaller learning rates as β increases, while constant-state proxies yconst∈3,5y_const∈\3,5\ best preserve Adam’s learning-rate scale. We now test the constant-state proxy by comparing learning-rate sweeps of Adam, Signum, and several constant-yty_t variants across a range of β; see Figure 8. Two empirical trends consistently appear. First, Signum’s preferred learning-rate region shifts systematically to smaller values as β increases. This behavior is consistent with the learning-rate scale predicted by Equation 11. If Adam’s preferred learning-rate scale is viewed as relatively stable across β, then a pure Signum update must compensate for the transformed-ratio attenuation that Adam applies. Since the prefactor β/(1−β)β/(1-β) grows rapidly with β, the effective attenuation in Equation 11 becomes stronger at larger β. Thus, matching Adam’s effective step size requires a smaller Signum learning rate. This matches the trend observed empirically in Figure 8: as β increases, the favorable learning-rate region for Signum moves markedly left relative to Adam. Table 1: yconsteffy_const^eff under Yc=2/(Z+c)2Y_c=2/(Z+c)^2. c β=0.90β=0.90 β=0.92β=0.92 β=0.95β=0.95 β=0.97β=0.97 0 3.36 3.31 3.25 3.21 0.1 3.33 3.28 3.22 3.18 0.2 3.24 3.19 3.13 3.09 Second, the constant-yty_t variants preserve Adam’s learning-rate scale much better than pure Signum. Among the tested constants, yconst∈3,5y_const∈\3,5\ gives the closest overall match to Adam’s preferred learning-rate region. This is broadly consistent with the step-size-matching calculation in Equation 14 and the numerical values reported above, which place the effective constant slightly above 33 under the shifted-reference model. Although the true transformed ratio yty_t is stochastic and heavy-tailed, these results suggest that its dominant effect on the optimizer’s learning-rate scale can be captured by a single moderate constant. This observation is practically useful. Directly using Signum requires a separate learning-rate search, and the appropriate learning rate changes noticeably with β. By contrast, the constant-yty_t proxy remains in Adam’s native learning-rate parameterization: replacing yty_t by a moderate constant yields a Signum-like update whose effective step size is already calibrated to Adam. Taken together, the sweeps show that tied-β Adam behaves like a sign-based method modulated by the transformed-ratio attenuation, and that a constant approximation can preserve much of Adam’s learning-rate scale. 6 Conclusion and Limitations In this paper, we studied Adam in the tied-β regime, β1=β2 _1= _2, through a transformed ratio yty_t. By factoring out the explicit β-dependent scale, this representation exposes a comparatively stable ratio structure in Adam’s adaptive denominator that is obscured in the raw moment states mtm_t and vtv_t. Empirically, the distribution of yty_t remains consistent across training settings: most coordinates concentrate in a moderate range, while a persistent heavy tail captures rare large ratio-state values. By isolating an approximately stable state inside Adam’s adaptive denominator, the ratio view yields two practical consequences. First, yty_t provides a compact replacement for the raw second-moment state vtv_t: across the language-model training regimes studied here, a fixed 4-bit codebook for yty_t preserves performance close to that of full-precision Adam. Second, the same representation clarifies Adam’s connection to Signum: tied-β Adam can be viewed as a sign-momentum update attenuated by yty_t, and replacing yty_t by a constant yields a Signum-type limit with an explicit learning-rate scaling. Overall, these results suggest that tied-β Adam’s adaptive behavior can be captured to a surprising extent through coarse but stable ratio information. Our experiments are limited to Transformer language models up to 1B parameters. Larger-scale and non-Transformer validation remains future work. References Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. Balles and Hennig [2018] Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning, 2018. Bernstein et al. [2018] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, 2018. Biderman et al. [2023] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, 2023. Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. Cattaneo and Shigida [2026] Matias D. Cattaneo and Boris Shigida. The effect of mini-batch noise on the implicit bias of adam. arXiv preprint arXiv:2602.01642, 2026. Chen et al. [2025] Kuang-Ming Chen, Jenq-Neng Hwang, and Hung yi Lee. InstructionCP: A simple yet effective approach for transferring large language models to target languages. In Workshop on Research in Computational Linguistic Typology and Multilingual NLP, 2025. Chitsaz et al. [2024] Kamran Chitsaz, Quentin Fournier, Gonçalo Mordido, and Sarath Chandar. Exploring quantization for efficient pre-training of transformer language models. In Findings of the Association for Computational Linguistics: EMNLP, 2024. Cui et al. [2023] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.01377, 2023. Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019. Ding et al. [2023] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Conference on Empirical Methods in Natural Language Processing, 2023. Dohare et al. [2023] Shibhansh Dohare, Qingfeng Lan, and A. Rupam Mahmood. Overcoming policy collapse in deep reinforcement learning. In Sixteenth European Workshop on Reinforcement Learning, 2023. Dohare et al. [2024] Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 2024. Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. Ellis et al. [2024] Benjamin Ellis, Matthew T. Jackson, Andrei Lupu, Alexander D. Goldie, Mattie Fellows, Shimon Whiteson, and Jakob N. Foerster. Adam on local time: Addressing nonstationarity in rl with relative adam timesteps. In Advances in Neural Information Processing Systems, 2024. Fernández-Hernández et al. [2026] Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, and Enrique S. Quintana-Ortí. Why adam works better with β1=β2 _1= _2: The missing gradient scale invariance principle. arXiv preprint arXiv:2601.21739, 2026. Fishman et al. [2025] Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. Scaling fp8 training to trillion-token llms. In International Conference on Learning Representations, 2025. Goldie et al. [2024] Alexander D. Goldie, Chris Lu, Matthew T. Jackson, Shimon Whiteson, and Jakob N. Foerster. Can learned optimization make reinforcement learning less difficult? In Advances in Neural Information Processing Systems, 2024. Groeneveld et al. [2024] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. OLMo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024. He et al. [2019] Fengxiang He, Tongliang Liu, and Dacheng Tao. Control batch size and learning rate to generalize well: Theoretical and empirical evidence. In Advances in Neural Information Processing Systems, volume 32, 2019. Hugging FaceTB [2025] Hugging FaceTB. SmolLM3: Smol, multilingual, long-context reasoner. https://huggingface.co/blog/smollm3, 2025. Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. Li et al. [2022] Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after SGD reaches zero loss? –a mathematical framework. In International Conference on Learning Representations, 2022. Li et al. [2024] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. ReMax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In International Conference on Machine Learning, 2024. Liu et al. [2025] Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352, 2025. Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision, 2021. Liu et al. [2020] Ziyin Liu, Zhikang T. Wang, and Masahito Ueda. Laprop: Separating momentum and adaptivity in adam. arXiv preprint arXiv:2002.04839, 2020. Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. Mandt et al. [2017] Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research, 18(134):1–35, 2017. Moalla et al. [2024] Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. No representation, no trust: Connecting representation, collapse, and trust issues in ppo. In Advances in Neural Information Processing Systems, 2024. Orvieto and Gower [2025] Antonio Orvieto and Robert M. Gower. In search of adam’s secret sauce. In Advances in Neural Information Processing Systems, 2025. Ouyang et al. [2022] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022. Parmar et al. [2024] Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don’t retrain: A recipe for continued pretraining of language models. arXiv preprint arXiv:2407.07263, 2024. Penedo et al. [2024] Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024. Peng et al. [2023] Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, Ruihang Li, Miaosen Zhang, Chen Li, Jia Ning, Ruizhe Wang, Zheng Zhang, Shuguang Liu, Joe Chau, Han Hu, and Peng Cheng. Fp8-lm: Training fp8 large language models. arXiv preprint arXiv:2310.18313, 2023. Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, 2020. Su [2026] Jianlin Su. https://kexue.fm/archives/11593, 2026. Sun et al. [2023] Tao Sun, Qingsong Wang, Dongsheng Li, and Bao Wang. Momentum ensures convergence of signsgd under weaker assumptions. In International Conference on Machine Learning, 2023. Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Appendix Appendix A Distributional properties of the transformed ratio state Appendix A records the derivation behind Proposition 1. The main text states the local Gaussian model and the formal result; here we keep only the short intuition needed to read the proof. In the tied-beta regime, the transformed ratio state is yt=1−βvt−mt2mt2.y_t= 1-β v_t-m_t^2m_t^2. Under a local Gaussian window, the EMA denominator has the form mt=σm(Z+a)m_t= _m(Z+a), so the small-denominator event Z+a≈0Z+a≈ 0 naturally produces the shifted inverse-square reference 2/(Z+a)22/(Z+a)^2. The numerator does not disappear; the proof below shows that it enters as the explicit relative correction ηt _t in Equation 5. When the local shift varies across coordinates or nearby windows, let ℐI index the observed coordinate-window instances and let each instance contribute its shift aia_i. Flattening these shifts gives the empirical pooled-shift distribution P^A=1|ℐ|∑i∈ℐδai,A∼P^A. P_A= 1|I| _i _a_i, A P_A. Thus A∼P^A P_A is the shift obtained by selecting one observed instance uniformly. More generally, for a pooled-shift distribution P, drawing A∼PA P gives the mixed reference YA=2/(Z+A)2Y_A=2/(Z+A)^2. The fixed reference Yc=2/(Z+c)2Y_c=2/(Z+c)^2, with c2:=P[A2]c^2:=E_P[A^2], keeps the leading shift effect; the shifted-reference calculation below proves the corresponding second-order survival matching. Proof of Proposition 1 and shifted-reference calculation. We justify Proposition 1 under the local Gaussian window used in the main text. The calculation is not a global model of training; it isolates the inverse-square denominator that controls the ratio state. Exact factorization. Let L=∑j≥0wjξjL= _j≥ 0w_j _j, Q=∑j≥0wjξj2Q= _j≥ 0w_j _j^2, and sβ2=∑j≥0wj2=(1−β)/(1+β)s_β^2= _j≥ 0w_j^2=(1-β)/(1+β). Then mt m_t =μ+σL=σm(Z+a), =μ+σ L= _m(Z+a), Ut U_t =vt−mt2=σ2(Q−L2),Z=L/sβ. =v_t-m_t^2=σ^2(Q-L^2), Z=L/s_β. Since ⁡[Q]=1E[Q]=1 and ⁡[L2]=sβ2E[L^2]=s_β^2, ⁡[Ut] [U_t] =σ2(1−sβ2)=2β1+βσ2, =σ^2(1-s_β^2)= 2β1+βσ^2, 1−β⁡[Ut]σm2 1-β E[U_t] _m^2 =2. =2. Now decompose the numerator around the denominator coordinate, Ut=⁡[Ut]+(⁡[Ut∣Z]−⁡[Ut])+(Ut−⁡[Ut∣Z]).U_t=E[U_t]+ (E[U_t Z]-E[U_t] )+ (U_t-E[U_t Z] ). For Gaussian variables ξj _j, projecting onto Z gives ⁡[Ut∣Z]−⁡[Ut] [U_t Z]-E[U_t] =σ2dβ(Z2−1), =σ^2d_β(Z^2-1), dβ d_β =∑j≥0wj3sβ2−sβ2=β⁡(1−β)(1+β)(1+β+β2). = _j≥ 0w_j^3s_β^2-s_β^2= β(1-β)(1+β)(1+β+β^2). Substituting into yt=1−βUt/mt2y_t= 1-βU_t/m_t^2 yields yt y_t =2+ηt(Z+a)2=Ya(1+ηt2), = 2+ _t(Z+a)^2=Y_a (1+ _t2 ), ηt _t =ηtcoup+ηtfluc, = _t^coup+ _t^fluc, ηtcoup _t^coup =γβ(Z2−1),γβ=1−β1+β+β2, = _β(Z^2-1), _β= 1-β1+β+β^2, ηtfluc _t^fluc =1−βUt−⁡[Ut∣Z]σm2, = 1-β U_t-E[U_t Z] _m^2, where Ya=2/(Z+a)2Y_a=2/(Z+a)^2. This proves the factorization in Equation 5. Auxiliary correction estimates. The fluctuation term is conditionally centered by construction: ⁡[ηtfluc∣Z]=0.E[ _t^fluc Z]=0. For its variance, write B=diag⁡(w)−ww⊤B=diag(w)-w , so that Ut=σ2ξ⊤BξU_t=σ^2ξ Bξ. The Gaussian quadratic-form identity gives Var⁡(Ut)=2σ4tr⁡(B2)=2σ4(sβ2−2∑j≥0wj3+sβ4).Var(U_t)=2σ^4tr(B^2)=2σ^4 (s_β^2-2 _j≥ 0w_j^3+s_β^4 ). Since ⁡[Ut∣Z]−⁡[Ut]=σ2dβ(Z2−1)E[U_t Z]-E[U_t]=σ^2d_β(Z^2-1), Var⁡(⁡[Ut∣Z])=2σ4dβ2Var(E[U_t Z])=2σ^4d_β^2, we have Var⁡(ηtfluc)=2(1+β)2β2(sβ2−2∑j≥0wj3+sβ4−dβ2)=2(1+β)β2(1−β)+O⁡((1−β)2).Var( _t^fluc)= 2(1+β)^2β^2 (s_β^2-2 _j≥ 0w_j^3+s_β^4-d_β^2 )= 2(1+β)β^2(1-β)+O((1-β)^2). Also, since yt/Ya−1=ηt/2y_t/Y_a-1= _t/2, for any 0<u<2δ0<u<2δ, |ytYa−1|>δ⊆|ηtcoup|>u∪|ηtfluc|>2δ−u. \ | y_tY_a-1 |>δ \ \| _t^coup|>u\∪\| _t^fluc|>2δ-u\. Taking probabilities of both sides and applying Chebyshev’s inequality gives the corresponding relative-error control. Tail order. Let St=2+ηt≥0S_t=2+ _t≥ 0. Then Pr⁡(yt>k)=∫ℝϕ⁡(z)Pr⁡(St>k(z+a)2∣Z=z)z. (y_t>k)= _Rφ(z)\, \! (S_t>k(z+a)^2 Z=z )\,dz. With x=k(z+a)x= k(z+a), this becomes 1k∫ℝϕ⁡(−a+xk)Pr⁡(St>x2∣Z=−a+xk)x. 1 k _Rφ\! (-a+ x k ) \! (S_t>x^2 Z=-a+ x k )\,dx. If h⁡(z)=⁡[St∣Z=z]h(z)=E[ S_t Z=z] is finite and continuous at −a-a, dominated convergence gives Pr(yt>k)=ϕ⁡(a)k∫ℝPr(St>x2∣Z=−a)dx+o(k−1/2)=2ϕ(a)kh(−a)+o(k−1/2). (y_t>k)= φ(a) k _R (S_t>x^2 Z=-a)\,dx+o(k^-1/2)= 2φ(a) kh(-a)+o(k^-1/2). Shifted-reference matching. Let P be a pooled-shift distribution with finite fourth moment, and draw A∼PA P. If P is empirical, then P=|ℐ|−1∑i∈ℐδaiP=|I|^-1 _i _a_i, and P[A2]=1|ℐ|∑i∈ℐai2.E_P[A^2]= 1|I| _i a_i^2. Assume Z⟂AZ A and Z∼⁡(0,1)Z (0,1). Fix k>0k>0. Writing q=2/kq= 2/k, the mixed denominator reference YA=2/(Z+A)2Y_A=2/(Z+A)^2 satisfies Pr⁡(YA>k∣A)=Φ⁡(q−A)+Φ⁡(q+A)−1. (Y_A>k A)= (q-A)+ (q+A)-1. A Taylor expansion in the shift gives Φ⁡(q−A)+Φ⁡(q+A)−1=2Φ(q)−1−qϕ(q)A2+O⁡(A4). (q-A)+ (q+A)-1=2 (q)-1-qφ(q)A^2+O(A^4). Therefore Pr⁡(YA>k) (Y_A>k) =2Φ(q)−1−qϕ(q)P[A2]+O⁡(P[A4]), =2 (q)-1-qφ(q)E_P[A^2]+O(E_P[A^4]), Pr⁡(Yc>k) (Y_c>k) =2Φ(q)−1−qϕ(q)c2+O⁡(c4),c2:=P[A2]. =2 (q)-1-qφ(q)c^2+O(c^4), c^2:=E_P[A^2]. For a family PεP_ of pooled-shift distributions, let Aε∼PεA_ P_ and cε2:=Pε[Aε2]c_ ^2:=E_P_ [A_ ^2]. If Pε[Aε4]=O⁡(ε4)E_P_ [A_ ^4]=O( ^4), Jensen’s inequality gives cε4=(Pε[Aε2])2≤Pε[Aε4]=O⁡(ε4)c_ ^4=(E_P_ [A_ ^2])^2 _P_ [A_ ^4]=O( ^4). Hence the two survival functions differ only at fourth order: Pr⁡(YAε>k)−Pr⁡(Ycε>k)=O⁡(ε4). (Y_A_ >k)- (Y_c_ >k)=O( ^4). This proves the second-order matching for the reference YcY_c in Equation 6. Appendix B Additional Motivation and Related Work for the Tied-β Regime This appendix provides additional motivation for studying the tied-β regime, β1=β2 _1= _2. Our goal is not to claim that this choice is universally optimal, but rather to explain why it is a structurally distinguished regime for analyzing Adam. The classical Adam default (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999) uses very different memories for the first and second moving averages. However, recent analyses and large-scale training practice have increasingly considered settings where these two memories are closer, including tied or nearly tied choices. This motivates asking what becomes simpler or more stable when the two exponential memories are matched. Several recent works point in this direction from complementary perspectives. Orvieto and Gower [2025] show that tied-β Adam preserves much of Adam’s empirical behavior in language-model training while admitting a cleaner interpretation in terms of smoothed sign momentum and a noise-to-signal correction. Fernández-Hernández et al. [2026] identify a structural principle behind this regime, showing that Adam has first-order gradient-scale invariance precisely when β1=β2 _1= _2. A separate line of work studies how mini-batch noise interacts with Adam’s memory parameters, reporting that moving β1 _1 closer to β2 _2 can improve validation behavior in some larger-batch, multi-epoch settings [Cattaneo and Shigida, 2026]. A concise informal overview of several related perspectives on the tied-β regime is also provided by Su [2026]. These results suggest that tied or near-tied memories are not merely algebraically convenient, but can reveal meaningful structure in Adam’s adaptive dynamics. Our use of the tied-β regime is more specific. It is exactly the setting in which Adam’s adaptive denominator admits a clean sign-over-ratio reformulation with a nonnegative variance-like residual. Ignoring bias correction and ϵε for simplicity, Adam evolves coordinate-wise as mt m_t =β1mt−1+(1−β1)gt, = _1m_t-1+(1- _1)g_t, (15) vt v_t =β2vt−1+(1−β2)gt2, = _2v_t-1+(1- _2)g_t^2, (16) and the adaptive factor is mt/vtm_t/ v_t. When β1=β2=β _1= _2=β, a direct expansion gives vt−mt2=β⁡(vt−1−mt−12)+β⁡(1−β)(gt−mt−1)2. v_t-m_t^2=β (v_t-1-m_t-1^2 )+β(1-β)(g_t-m_t-1)^2. (17) Therefore, under the standard zero initialization, vt−mt2≥0for all t. v_t-m_t^2≥ 0 all t. (18) This makes vt−mt2v_t-m_t^2 a genuine variance-like residual. Whenever mt≠0m_t≠ 0, we can define rt:=vt−mt2mt2≥0,vt=mt2(1+rt), r_t:= v_t-m_t^2m_t^2≥ 0, v_t=m_t^2(1+r_t), (19) which yields mtvt=sign⁡(mt)1+rt. m_t v_t= sign(m_t) 1+r_t. (20) Thus, in the tied-β regime, Adam’s adaptive factor is a sign-momentum direction multiplied by an attenuation factor. Since rt≥0r_t≥ 0, this attenuation is at most one; the update cannot be amplified beyond the corresponding sign-momentum magnitude by a negative residual. This nonnegativity property is in fact special to the tied-β regime. For 0<β1,β2<10< _1, _2<1, fix a time t and write the zero-initialized moment estimates as mt m_t =∑i=0t−1aigt−i,ai:=(1−β1)β1i, = _i=0^t-1a_ig_t-i, a_i:=(1- _1) _1^i, (21) vt v_t =∑i=0t−1bigt−i2,bi:=(1−β2)β2i. = _i=0^t-1b_ig_t-i^2, b_i:=(1- _2) _2^i. (22) By weighted Cauchy–Schwarz, mt2=(∑i=0t−1aigt−i)2≤(∑i=0t−1ai2bi)(∑i=0t−1bigt−i2)=Stvt, m_t^2= ( _i=0^t-1a_ig_t-i )^2≤ ( _i=0^t-1 a_i^2b_i ) ( _i=0^t-1b_ig_t-i^2 )=S_tv_t, (23) where St:=∑i=0t−1ai2bi=(1−β1)21−β2∑i=0t−1(β12β2)i. S_t:= _i=0^t-1 a_i^2b_i= (1- _1)^21- _2 _i=0^t-1 ( _1^2 _2 )^i. (24) Moreover, the bound is tight: equality in Equation 23 is attained by choosing gt−i∝ai/big_t-i a_i/b_i. Consequently, vt≥mt2v_t≥ m_t^2 for all gradient sequences at time t holds if and only if St≤1S_t≤ 1. When β1=β2=β _1= _2=β, Equation 24 gives St=(1−β)∑i=0t−1βi=1−βt≤1, S_t=(1-β) _i=0^t-1β^i=1-β^t≤ 1, (25) recovering the nonnegativity above. Conversely, suppose β1≠β2 _1≠ _2. If β12≥β2 _1^2≥ _2, then the sum in Equation 24 grows without bound as t increases, so St>1S_t>1 for some finite t. If β12<β2 _1^2< _2, then limt→∞St=β2(1−β1)2(1−β2)(β2−β12). _t→∞S_t= _2(1- _1)^2(1- _2)( _2- _1^2). (26) A direct comparison gives β2(1−β1)2(1−β2)(β2−β12)>1⟺(β2−β1)2>0, _2(1- _1)^2(1- _2)( _2- _1^2)>1 ( _2- _1)^2>0, (27) which holds whenever β1≠β2 _1≠ _2. Thus St>1S_t>1 for some finite t. By the tightness of Equation 23, there exists a gradient sequence for which mt2/vt=St>1m_t^2/v_t=S_t>1, and therefore vt−mt2<0. v_t-m_t^2<0. (28) Hence the tied-β regime is not only sufficient but necessary for the residual vt−mt2v_t-m_t^2 to remain nonnegative for all gradient sequences. A simple spike calculation illustrates this failure mode for the classical mismatched setting. For fixed previous state (mt−1,vt−1)(m_t-1,v_t-1), 1+rt=vtmt2=β2vt−1+(1−β2)gt2(β1mt−1+(1−β1)gt)2. 1+r_t= v_tm_t^2= _2v_t-1+(1- _2)g_t^2 ( _1m_t-1+(1- _1)g_t )^2. (29) As |gt|→∞|g_t|→∞, the lower-order terms vanish and 1+rt⟶1−β2(1−β1)2. 1+r_t 1- _2(1- _1)^2. (30) For the classical Adam default (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999), this limiting value is 1−β2(1−β1)2=0.0010.12=0.1. 1- _2(1- _1)^2= 0.0010.1^2=0.1. (31) So in the large-spike regime 1+rt≈0.11+r_t≈ 0.1. Thus, away from the tied-β regime, the sign-over-ratio residual is not protected by the same nonnegativity mechanism and can become substantially negative. This observation also connects to work on nonstationary training dynamics, especially in reinforcement learning. In such settings, gradient statistics can shift rapidly, and mismatched first- and second-moment memories may cause the adaptive denominator to lag behind the signal term. This can temporarily weaken the attenuation and produce unexpectedly large updates. Related reinforcement-learning studies have connected Adam’s timescale choices to policy collapse or instability under nonstationary data streams, and report that matching the first- and second-moment decay rates, or bringing them closer together, can mitigate some of these failures [Dohare et al., 2023, Moalla et al., 2024]. From another angle, Ellis et al. [2024] show that nonstationary changes in gradient magnitude can cause Adam to produce overly large updates, motivating a relative-timestep correction. These works do not establish tied β values as a universally optimal choice, but they support the broader view that the relative timescales of Adam’s two moving averages are central to stability under changing gradient statistics. Taken together, these perspectives motivate the tied-β regime as a useful analytical setting. It is the regime in which the adaptive denominator admits a clean decomposition into a squared mean and a nonnegative variance-like residual; it aligns with recent structural analyses of Adam’s scale behavior; and it provides a natural starting point for our ratio-state formulation. Appendix C Details for the transformed-state recursion and implementation This appendix collects additional details for the transformed-state recursion used in the main text. We first give a step-by-step derivation of the scalar recursion for yty_t, then describe its initialization and the practical handling of numerically unstable regions, and finally summarize the ablations used to evaluate the ratio-recursive formulation. C.1 Step-by-step derivation of the transformed-state recursion Recall that in the tied-beta regime β1=β2=β _1= _2=β, we define rt:=vt−mt2mt2,yt:=1−βrt=1−βvt−mt2mt2, r_t:= v_t-m_t^2m_t^2, y_t:= 1-βr_t= 1-β v_t-m_t^2m_t^2, (32) whenever mt≠0m_t≠ 0. Equivalently, vt=mt2(1+β1−βyt). v_t=m_t^2 (1+ β1-βy_t ). (33) We also define the momentum ratio xt:=mt−1mt. x_t:= m_t-1m_t. (34) Starting from the first-moment recursion mt=βmt−1+(1−β)gt, m_t=β m_t-1+(1-β)g_t, (35) we solve for the current gradient: gt=mt−βmt−11−β=mt1−β(1−βxt). g_t= m_t-β m_t-11-β= m_t1-β(1-β x_t). (36) Substituting Equation 36 into the second-moment recursion vt=βvt−1+(1−β)gt2 v_t=β v_t-1+(1-β)g_t^2 (37) gives vt=βvt−1+mt21−β(1−βxt)2. v_t=β v_t-1+ m_t^21-β(1-β x_t)^2. (38) Next, express both vtv_t and vt−1v_t-1 through the transformed state: vt=mt2(1+β1−βyt),vt−1=mt−12(1+β1−βyt−1). v_t=m_t^2 (1+ β1-βy_t ), v_t-1=m_t-1^2 (1+ β1-βy_t-1 ). (39) Using mt−1=xtmtm_t-1=x_tm_t, Equation 38 becomes mt2(1+β1−βyt)=βxt2mt2(1+β1−βyt−1)+mt21−β(1−βxt)2. m_t^2 (1+ β1-βy_t )=β x_t^2m_t^2 (1+ β1-βy_t-1 )+ m_t^21-β(1-β x_t)^2. (40) Dividing by mt2m_t^2 yields 1+β1−βyt=βxt2(1+β1−βyt−1)+(1−βxt)21−β. 1+ β1-βy_t=β x_t^2 (1+ β1-βy_t-1 )+ (1-β x_t)^21-β. (41) Multiplying both sides by 1−β1-β and simplifying gives (1−β)+βyt=β⁡(1−β)xt2+β2xt2yt−1+(1−βxt)2, (1-β)+β y_t=β(1-β)x_t^2+β^2x_t^2y_t-1+(1-β x_t)^2, (42) and therefore yt=βxt2yt−1+(1−xt)2. y_t=β x_t^2y_t-1+(1-x_t)^2. (43) This is the scalar recursion stated in the main text. Algorithm 1 gives the detailed pseudocode for TR-Adam, including the quantized ratio-state storage and bias-corrected parameter update. Algorithm 1 TR-Adam with quantized transformed ratio 1: Parameters θ0 _0, objective ft(θ)f_t(θ), learning rate η, tied momentum coefficient β, numerical constant ϵε, weight decay λ, ratio-state codebook Q. 2: Initialize m0←0m_0← 0 for all parameters. 3: for t=1,…,Tt=1,…,T do 4: gt←∇ft(θt−1)g_t←∇ f_t( _t-1) 5: mt←βmt−1+(1−β)gtm_t←β m_t-1+(1-β)g_t 6: if t=1t=1 then 7: y~t←1 y_t← 1 ⊳ initial transformed ratio 8: else 9: xt←mt−1/mtx_t← m_t-1/m_t 10: y~t←βxt2yt−1+(1−xt)2 y_t←β x_t^2y_t-1+(1-x_t)^2 11: end if 12: yt←Q(y~t)y_t← Q_Q( y_t) ⊳ nearest codebook value, FP4-style quantization 13: m^t←mt/(1−βt) m_t← m_t/(1-β^t) ⊳ first-moment bias correction 14: rt←β1−βytr_t← β1-βy_t 15: dt←|mt|1+rt1−βt+ϵd_t←|m_t| 1+r_t1-β^t+ε ⊳ denominator bias correction 16: θt−12←(1−ηλ)θt−1 _t- 12←(1-ηλ) _t-1 ⊳ decoupled weight decay 17: θt←θt−12−ηm^t/dt _t← _t- 12-η m_t/d_t 18: end for Figure 9: Distributions of the Adam-derived transformed variable yty_t across five representative training settings: Llama-style 20M and Pythia-style 160M pre-training, Pythia-1B and Llama-3.2-1B supervised fine-tuning, and Llama-3.2-1B ReMax training. Each block shows the empirical CDF at the top and the histogram at the bottom for different training stages. Across model families, scales, and training regimes, the transformed state exhibits a consistent heavy-tailed shape: a concentrated bulk near small values together with a long right tail. The dashed reference curve 2/Z22/Z^2, with Z∼⁡(0,1)Z (0,1), is included to highlight the shared inverse-square-like structure. Appendix D Additional Experimental Results This section provides additional quantitative and distributional evidence for the robustness of the transformed ratio-state representation. Figure 9 first shows that the transformed variable yty_t has a broadly consistent bulk-and-tail distribution across five representative settings, including pre-training, SFT, and ReMax. We then report final validation metrics in Tables 2, 3 and 4, averaging over the final three validation points to reduce noise from individual checkpoints. Across these settings, TR-Adam remains close to full Adam in pre-training and often improves over Adam in post-training, suggesting that the coarse transformed state preserves the main adaptive behavior while introducing a mild stabilizing effect. Table 2: Pre-training validation loss averaged over the final three validation points. For Llama-style 20M, the learning rate is selected from the full learning-rate sweep; for Pythia-style 160M, it is selected from 1,3,5,8,10×10−3\1,3,5,8,10\× 10^-3. Lower is better. Here Δ=TR-Adam−Adam =TR -Adam-Adam. Model β Adam TR-Adam (FP32-mtm_t, 4-bit yty_t) TR-Adam (FP8-mtm_t, 4-bit yty_t) Δ FP32 Δ FP8 Llama-style 20M 0.90 3.694 3.706 3.710 +0.012 +0.016 Llama-style 20M 0.92 3.676 3.689 3.688 +0.013 +0.012 Llama-style 20M 0.95 3.672 3.672 3.670 -0.000 -0.002 Llama-style 20M 0.97 3.692 3.677 3.675 -0.014 -0.016 Pythia-style 160M 0.90 3.373 3.378 3.387 +0.004 +0.013 Pythia-style 160M 0.92 3.355 3.365 3.369 +0.010 +0.014 Pythia-style 160M 0.95 3.342 3.347 3.346 +0.005 +0.004 Pythia-1B 0.90 2.985 2.979 2.979 -0.006 -0.007 Pythia-1B 0.92 2.996 2.977 2.980 -0.019 -0.016 Pythia-1B 0.95 2.982 2.973 2.974 -0.009 -0.008 Table 3: SFT validation loss averaged over the final three validation points. Lower is better. Here Δ=TR-Adam−Adam =TR -Adam-Adam. Model β Adam TR-Adam (FP32-mtm_t, 4-bit yty_t) TR-Adam (FP8-mtm_t, 4-bit yty_t) Δ FP32 Δ FP8 Pythia-1B 0.90 1.299 1.297 1.297 -0.002 -0.002 Pythia-1B 0.92 1.302 1.295 1.295 -0.007 -0.006 Pythia-1B 0.95 1.309 1.299 1.299 -0.011 -0.011 Llama-3.2-1B 0.90 1.075 1.072 1.072 -0.002 -0.002 Llama-3.2-1B 0.92 1.076 1.071 1.071 -0.005 -0.005 Llama-3.2-1B 0.95 1.082 1.073 1.073 -0.009 -0.009 Table 4: ReMax validation reward averaged over the final three validation points. Higher is better. Here Δ=TR-Adam−Adam =TR -Adam-Adam. β Adam TR-Adam (FP32-mtm_t, 4-bit yty_t) TR-Adam (FP8-mtm_t, 4-bit yty_t) Δ FP32 Δ FP8 0.90 0.788 0.825 0.813 +0.036 +0.025 0.92 0.800 0.824 0.819 +0.024 +0.019 0.95 0.808 0.817 0.811 +0.009 +0.003 Appendix E Datasets and Implementation Details Across all optimizer comparisons, we keep the dataset, model architecture, batch size, learning-rate schedule, and evaluation protocol fixed, and only change the optimizer update. For short-run supervised experiments, including plainLM11 1 https://github.com/Niccolo-Ajroldi/plainLM pre-training ablations and VERL22 2 https://github.com/volcengine/verl SFT, we use zero weight decay to isolate optimizer-state effects and provide a cleaner comparison with Signum-style updates. For long-run supervised experiments, such as the 20B-token setting, we use weight decay 0.10.1. For SFT and RLHF, we follow the default VERL learning-rate settings, 1×10−51× 10^-5 and 1×10−61× 10^-6, respectively; for RLHF, we also keep the default actor weight decay 0.010.01. For learning-rate selection, SFT and RLHF use the default VERL learning rates described above. For pre-training, we tune learning rates separately at the smaller scales before reporting the best result for each method. For the Llama-style 20M experiments, we sweep the learning-rate grid shown in Fig. 7 and Fig. 8. For the Pythia-style 160M experiments, we sweep 1,3,5,8,10×10−3\1,3,5,8,10\× 10^-3 and select the best learning rate for each optimizer variant and β. For the Pythia-1B 20B-token experiment, a full sweep is too expensive; we therefore run a short 5B-token sweep for Adam at β=0.95β=0.95 over 0.001,0.003,0.005,0.008\0.001,0.003,0.005,0.008\, select 0.0010.001, and use this learning rate for all β values and optimizer variants. For the small-scale pre-training models, we use compact decoder-only Transformer architectures implemented in plainLM. The Llama-style 20M model is our own plainLM implementation based on Hugging Face’s LlamaForCausalLM/LlamaConfig interface, with its scale and layout adapted from a community 20M Llama-style checkpoint33 3 https://huggingface.co/alan918727/TinyStories-LLaMA2-20M-256h-4l-colab; we do not use its pretrained weights. The Pythia-style 160M model follows a Pythia-like architecture [Biderman et al., 2023] constructed in the same codebase. For the 1B-scale experiments, we use the official Llama-3.2-1B and Pythia-1B architectures; in pre-training experiments, all models are trained from scratch by reinitializing the parameters. Pre-training. For pre-training, we use subsets of the FineWeb corpus [Penedo et al., 2024] and implement all experiments in plainLM. Table 5: Pre-training experimental setup. Dataset FineWeb 10B / 10B / 100B Models Llama-style 20M / Pythia-style 160M / Pythia-1B Training budget 1.5B / 4B / 20B tokens Sequence length 2048 Batch size 256 Warmup 512 / 1024 / 1024 steps LR schedule constant after warmup Implementation plainLM Supervised Fine-Tuning (SFT). For SFT, we use UltraChat [Ding et al., 2023], formatted as multi-turn conversations and trained with the standard teacher-forcing objective. We fine-tune 1B-scale backbones, including Pythia-1B and Llama-3.2-1B, using the VERL SFT trainer. Table 6: SFT experimental setup. Dataset UltraChat Models Pythia-1B, Llama-3.2-1B Training budget 2000 steps Maximum sequence length 2048 Batch size 128 Learning rate 1×10−51× 10^-5 LR schedule constant after warmup Implementation VERL SFT trainer RLHF. For RLHF, we use UltraFeedback [Cui et al., 2023] as the prompt and preference source. We use Llama-3.2-1B as the actor model and Skywork-Reward-V2-Llama-3.2-3B [Liu et al., 2025] as the reward model. The RL training pipeline is implemented with VERL, with our optimizer integrated into the actor update. Table 7: RLHF/ReMax experimental setup. Dataset UltraFeedback Actor model Llama-3.2-1B Reward model Skywork-Reward-V2-Llama-3.2-3B Algorithm ReMax Li et al. [2024] Training budget 600 steps Batch size 64 prompts Maximum prompt length 768 Maximum response length 1024 Rollout temperature 1.0 Actor learning rate 1×10−61× 10^-6 Actor LR schedule constant, no warmup Implementation VERL RL trainer All 1B-scale experiments, including pre-training, SFT, and RLHF, are run on 8×8×H200 GPUs, while the remaining smaller-scale experiments are run on 16×16×RTX 4090 GPUs. Appendix F Broader Impact This work primarily aims to improve our understanding of Adam’s internal adaptive dynamics. By identifying a natural-scale ratio-state representation in the tied-β regime, it provides a more transparent view of how the mean estimate and adaptive denominator interact during training. This mechanistic perspective may help guide optimizer analysis, hyperparameter choices, and the design of simpler Adam-like methods. This paper is methodological and does not involve data from human subjects, user studies, or deployment of a user-facing system. We do not directly study downstream model behavior; our focus is on optimizer dynamics and training efficiency. NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: Abstract and introduction accurately reflect the paper’s contributions and scope Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Discussed in paper Guidelines: • The answer [N/A] means that the paper has no limitations while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: The paper provides the full set of assumptions and a complete proof Guidelines: • The answer [N/A] means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: The paper fully discloses all the information needed to reproduce the main experimental results Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: An anonymized code package is provided as a compressed archive in the supplemental material Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: Mentioned in Appendix Guidelines: • The answer [N/A] means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Due to the high computational cost of large-scale language-model training, the main experiments are not repeated with many independent random seeds. Instead, we report complete training curves and compare trends across multiple training stages, datasets, and optimizer variants, where the observed differences are consistent across the evaluated settings. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar rather than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Mentioned in Appendix Guidelines: • The answer [N/A] means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics Guidelines: • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No] , they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] . Justification: The paper discusses the potential positive impact of reducing optimizer-state memory and improving the efficiency and accessibility of large-scale model training. It also notes that more efficient training methods can indirectly lower the cost of training powerful models, which may amplify both beneficial and harmful downstream uses depending on deployment context. Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. • If the authors answer [N/A] or [No] , they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A] . Justification: The paper does not introduce or release a new high-risk pretrained model, image generator, or scraped dataset. The work focuses on optimizer-state representations and training methodology rather than releasing a deployable generative model. Guidelines: • The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer:[Yes] Justification: The paper cites the original sources of the datasets, models, and software frameworks used in the experiments. Where applicable, licenses and usage terms are reported or referenced in the appendix. Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., C-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We provide the code and README.md Guidelines: • The answer [N/A] means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: The paper does not involve crowdsourcing experiments or research with human subjects Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: The paper does not involve crowdsourcing, user studies, or research with human subjects, so IRB approval or equivalent review is not applicable. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [N/A] Justification: Any LLM use was limited to writing refinement. Guidelines: • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

Instant research discovery