Paper deep dive
Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation
Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, Tieniu Tan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 6:30:20 AM
Summary
The paper introduces LoRA-Pre, a novel low-rank optimizer designed to reduce memory overhead in large language model training by decomposing momentum matrices into compact low-rank subspaces. It establishes a theoretical equivalence between exponential moving average (EMA) momentum updates and online linear regression, allowing for efficient compression of first- and second-order moments. Experiments on Llama models demonstrate that LoRA-Pre achieves superior performance with significantly reduced memory usage compared to baselines like Adam, Muon, and GaLore, and shows strong results in both pre-training and fine-tuning scenarios.
Entities (12)
Relation Signals (9)
LoRA-Pre → isa → Low-Rank Optimizer
confidence 98% · LoRA-Pre, a novel low-rank optimizer designed for efficient pre-training.
LoRA-Pre → evaluatedon → Llama
confidence 95% · We empirically validate LoRA-Pre’s efficacy by pre-training models from the Llama architecture family
Exponential Moving Average → isequivalentto → Online Linear Regression
confidence 95% · EMA momentum updates are mathematically equivalent to training an online linear regressor
LoRA-Pre → reducesmemoryfootprintof → Optimizer States
confidence 95% · LoRA-Pre reduces the optimizer’s memory footprint by decomposing the full momentum matrix
LoRA-Pre → usestechnique → Low-Rank Factorization
confidence 95% · decomposing the full momentum matrix into a compact low-rank subspace
LoRA-Pre → improvesperformanceon → LLaMA-3.1-8B
confidence 90% · LoRA-Pre achieves substantial improvements of 3.14 points on Llama-3.1-8B
LoRA-Pre → improvesperformanceon → Llama-2-7b
confidence 90% · and 6.17 points on Llama-2-7B
LoRA-Pre → outperforms → GaLore
confidence 90% · LoRA-Pre achieves the highest performance across all model sizes... compared to baseline methods
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Modern optimizers like Adam and Muon are central to training large language models, but their reliance on first- and second-order momenta introduces significant memory overhead, which constrains scalability and computational efficiency. In this work, we reframe the exponential moving average (EMA) used in these momenta as the training of a linear regressor via online gradient flow. Building on this equivalence, we introduce LoRA-Pre, a novel low-rank optimizer designed for efficient pre-training. Specifically, LoRA-Pre reduces the optimizer's memory footprint by decomposing the full momentum matrix into a compact low-rank subspace within the online linear learner, thereby maintaining optimization performance while improving memory efficiency. We empirically validate LoRA-Pre's efficacy by pre-training models from the Llama architecture family, scaling from 60M to 1B parameters. LoRA-Pre achieves the highest performance across all model sizes. Notably, LoRA-Pre demonstrates remarkable rank efficiency, achieving comparable or superior results using only 1/8 the rank of baseline methods. Beyond pre-training, we evaluate LoRA-Pre's effectiveness in fine-tuning scenarios. With the same rank, LoRA-Pre consistently outperforms all efficient fine-tuning baselines. Specifically, compared to standard LoRA, LoRA-Pre achieves substantial improvements of 3.14 points on Llama-3.1-8B and 6.17 points on Llama-2-7B, validating our approach's effectiveness across both pre-training and fine-tuning paradigms. Our code is publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.24283v1
- Canonical: https://arxiv.org/abs/2602.24283v1
Trouble viewing inline? Open PDF directly →
Full Text
86,974 characters extracted from source content.
Expand or collapse full text
Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation Zhengbo Wang1,2 Jian Liang2,3 Ran He2,3 Zilei Wang1 Tieniu Tan2,3,4 1 University of Science and Technology of China 2 NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences 3 School of Artificial Intelligence, University of Chinese Academy of Sciences 4 Nanjing University zhengbowang@mail.ustc.edu.cn, liangjian92@gmail.com Corresponding author. Abstract Modern optimizers like Adam and Muon are central to training large language models, but their reliance on first- and second-order momenta introduces significant memory overhead, which constrains scalability and computational efficiency. In this work, we reframe the exponential moving average (EMA) used in these momenta as the training of a linear regressor via online gradient flow. Building on this equivalence, we introduce LoRA-Pre, a novel low-rank optimizer designed for efficient pre-training. Specifically, LoRA-Pre reduces the optimizer’s memory footprint by decomposing the full momentum matrix into a compact low-rank subspace within the online linear learner, thereby maintaining optimization performance while improving memory efficiency. We empirically validate LoRA-Pre’s efficacy by pre-training models from the Llama architecture family, scaling from 60M to 1B parameters. LoRA-Pre achieves the highest performance across all model sizes. Notably, LoRA-Pre demonstrates remarkable rank efficiency, achieving comparable or superior results using only 1/8 the rank of baseline methods. Beyond pre-training, we evaluate LoRA-Pre’s effectiveness in fine-tuning scenarios. With the same rank, LoRA-Pre consistently outperforms all efficient fine-tuning baselines. Specifically, compared to standard LoRA, LoRA-Pre achieves substantial improvements of 3.14 points on Llama-3.1-8B and 6.17 points on Llama-2-7B, validating our approach’s effectiveness across both pre-training and fine-tuning paradigms. Our code is publicly available at https://github.com/mrflogs/LoRA-Pre. 1 Introduction Large Language Models (LLMs) (Guo et al., 2025; Yang et al., 2025; Grattafiori et al., 2024; Brown et al., 2020; Comanici et al., 2025; Touvron et al., 2023; Jaech et al., 2024) have become the centerpiece of modern deep learning. Trained on trillions of tokens from heterogeneous sources and scaled to unprecedented parameter counts, they demonstrate remarkable generalization and transfer capabilities. Beyond this, some LLMs have reasoning abilities and leverage external tools (Guo et al., 2025; Yang et al., 2025). These advances have transformed LLMs from statistical learners into versatile systems, driving breakthroughs across research and real-world applications. However, the success of LLMs comes with formidable training and adaptation costs (Grattafiori et al., 2024). The vast number of trainable parameters demands enormous memory and computational resources during pre-training and fine-tuning. A key contributor to this burden lies in the optimizer states. For instance, Adam (Kingma and Ba, 2015), the de facto optimizer for training LLMs, maintains not only the model weights but also first- and second-order moment estimates of the gradients. This triples memory usage and further exacerbates scalability bottlenecks, underscoring the urgent need for more efficient optimization strategies. To address this, a series of low-rank optimization methods has emerged. One prominent line of research achieves optimizer state compression through projected gradient descent (Zhao et al., 2024; Chen et al., 2024; Hao et al., 2024; Modoranu et al., 2025). These methods initialize projection matrices via SVD or random mappings, project gradients into smaller subspaces for optimizer state computation, and then project them back to achieve parameter updates, thereby compressing the optimization overhead. Additionally, such methods require periodic subspace updates to enable high-rank parameter updates following W=ΔWT1+ΔWT2+⋯W= W_T_1+ W_T_2+·s. However, due to the inability to update subspaces instantly, error accumulation occurs in optimizer state computation, leading to suboptimal performance. This motivates the need for a more dynamic approach that can rapidly adapt to changing gradient subspaces. In this paper, we propose LoRA-Pre, a novel low-rank optimizer for LLM pre-training that addresses these limitations through a different approach. Our key insight is an interesting mathematical connection between the exponential moving average (EMA) of momentum and linear regression. Specifically, we demonstrate that EMA momentum updates are mathematically equivalent to training an online linear regressor with gradient descent on the online gradient flow: m←β⋅m+(1−β)⋅g⟺minmL(m;g)=12⋅‖m−g‖F2,m←β· m+(1-β)· g _mL(m;g)= 12·\|m-g\|^2_F, (1) where m∈ℝp×qm ^p× q represents the momentum, g is the online gradient, and β is the momentum coefficient. This equivalence reveals that momentum accumulation can be viewed as fitting a linear model to approximate the gradient history. Leveraging this theoretical insight, we develop a memory-efficient optimizer through momentum compression via low-rank factorization. Instead of maintaining the full momentum matrix m, we decompose it as the product of two low-rank matrices: m=mB⋅mAm=m_B· m_A, where mB∈ℝp×rm_B ^p× r and mA∈ℝr×qm_A ^r× q, with r≪min(p,q)r (p,q). This factorization reduces memory complexity from p×qp× q to (p+q)×r(p+q)× r, yielding substantial memory savings for large-scale models. The low-rank momentum is then updated by solving minmB,mAL(mB,mA;g)=12⋅‖mBmA−g‖F2 _m_B,m_AL(m_B,m_A;g)= 12·\|m_Bm_A-g\|^2_F, with explicit update rules derived in Theorem 3.1. This theoretical framework enables us to compress any momentum-based optimizer. We demonstrate its versatility by developing LoRA-Pre variants for both Adam (Kingma and Ba, 2015) and Muon (Jordan et al., 2024) optimizers, with detailed algorithms provided in Appendix B. Extensive experiments across pre-training and fine-tuning tasks validate the effectiveness of our method, while ablation studies demonstrate strong robustness across different rank variations. Our main contributions are summarized as follows: • We establish a novel theoretical connection showing that exponential moving average (EMA) momentum updates are mathematically equivalent to training a linear regressor via online gradient flow. • Based on this insight, we propose LoRA-Pre, a memory-efficient low-rank optimizer for pre-training that compresses optimizer states by factorizing the momentum matrix into low-rank components. We construct LoRA-Pre variants for both Adam and Muon optimizers, mathematically derive their low-rank update rules through our regression formulation, and achieve substantial memory reduction while preserving optimization dynamics. • We provide extensive experimental validation across both pre-training and fine-tuning tasks, demonstrating that LoRA-Pre achieves superior performance with remarkable rank efficiency compared to existing baselines, confirming both the efficiency and effectiveness of our approach across diverse model scales and application scenarios. 2 Related Works Low-Rank Adaptation. The scaling of Large Language Models (LLMs) has spurred the development of Parameter-Efficient Fine-Tuning (PEFT) methods (Hu et al., 2022; Liu et al., 2024; Wang et al., 2025; Ding et al., 2023; Liu et al., 2022; 2023; Hayou et al., 2024; Wang et al., 2024a; Edalati et al., 2023; Zhang et al., 2023; Tastan et al., 2025; Wang et al., 2023; 2024c; 2024b), which aim to adapt pre-trained models to downstream tasks with reduced computational and memory overhead. Among these PEFT methods, Low-Rank Adaptation (LoRA) (Hu et al., 2022) and its variants (Wang et al., 2025; 2024a; Hayou et al., 2024; Liu et al., 2024; Yen et al., 2025) have emerged as the predominant methods in the field. LoRA is grounded in the principle that weight updates during fine-tuning possess an intrinsic low-rank structure (Aghajanyan et al., 2021). By re-parameterizing these updates as the product of two low-rank matrices, LoRA substantially reduces the number of trainable parameters while maintaining competitive performance, thereby enabling efficient adaptation of LLMs with limited computational resources. The effectiveness of LoRA has inspired a line of research aimed at addressing its shortcomings. For instance, LoRA+ (Hayou et al., 2024) introduces differential learning rates for the two low-rank matrices to improve convergence and final task performance. DoRA (Liu et al., 2024) decomposes pre-trained weights into magnitude and direction components, applying LoRA specifically to the directional component to better capture fine-tuning dynamics. Recent works like LoFT (Tastan et al., 2025) and LoRA-Pro (Wang et al., 2025) establish theoretical connections between LoRA and full fine-tuning via projected gradient equivalents. While effective for fine-tuning, existing LoRA-based methods face fundamental challenges when applied to pre-training from scratch. Unlike fine-tuning, where small adaptations naturally exhibit low-rank structure, pre-training from random initialization requires full-rank weight updates to learn diverse representations across the entire parameter space (Lialin et al., 2024; Kamalakara et al., 2022). This mismatch between LoRA’s low-rank assumption and pre-training’s full-rank requirements results in suboptimal performance in the pre-training stage. Low-Rank Pre-Training. The pre-training cost of LLMs has surged dramatically with the rapid expansion of model scale. A promising direction for mitigating these costs is compressing optimizer states into a low-rank subspace, a strategy that significantly reduces memory footprint and communication overhead (Zhao et al., 2024; Modoranu et al., 2025; Ma et al., 2025; Han et al., 2024; Zmushko et al., 2025; Chen et al., 2024; Hao et al., 2024; Shen et al., 2025; Mahdavinia and Mahdavi, 2025; Zhang et al., 2025). For instance, GaLore (Zhao et al., 2024) utilizes Singular Value Decomposition (SVD) to project gradient information into a low-rank subspace for state compression, subsequently projecting the optimized gradients back for parameter updates. To enhance computational efficiency, Flora (Hao et al., 2024) substitutes the expensive SVD operation with random projection, while Fira (Chen et al., 2024) incorporates SGD momentum to leverage gradient information from the complementary subspace. However, these projection-based methods typically rely on periodic subspace updates to amortize costs, which often results in optimization discontinuities and error accumulation due to the lag in subspace adaptation. Recent works have explored online strategies to address these limitations. MLorc (Shen et al., 2025) employs randomized SVD for online momentum compression. MoFaSGD (Mahdavinia and Mahdavi, 2025) utilizes momentum factorization to approximate full-rank momentum online, ensuring non-convex convergence. Similarly, ADAPM (Zhang et al., 2025) compresses first-order momentum into a low-rank subspace via linear regression. In contrast, our proposed LoRA-Pre fundamentally reformulates momentum maintenance as an online regression task. By directly evolving low-rank factors via online gradient flow at every step, our approach achieves continuous subspace adaptation, effectively eliminating the instabilities associated with periodic updates or heuristic approximations. 3 Method We begin by revisiting the de facto standard optimizer, Adam (Kingma and Ba, 2015), in Section 3.1. Then, we establish a connection between the exponential moving average and an online linear regressor over past gradients in Section 3.2. Finally, Section 3.3 introduces our efficient optimizer, LoRA-Pre, which compresses optimizer states through low-rank parameterization. 3.1 Preliminary Figure 1: Illustration of our LoRA-Pre method. In this work, we establish a novel connection: the exponential moving average (EMA) update for optimizer momentum is mathematically equivalent to training a linear regressor using online gradient descent. Leveraging this equivalence, we propose compressing the optimizer states (i.e., the momenta) using low-rank matrices to reduce the memory footprint. Finally, the closed-form update rules for these matrices without requiring back-propagation are given by Theorem 3.1. We first review Adam (Kingma and Ba, 2015), the de facto optimizer in modern deep learning, which combines the benefits of AdaGrad (Duchi et al., 2011) and RMSProp (Hinton et al., 2012) by maintaining estimates of the first and second moments of gradients to achieve adaptive learning rates and robust performance. Consider an optimization problem where xt∈x_t represents a data point drawn from a distribution pdatap_data, ℒ(⋅):→ℝL(·):X is a loss function, and θ∈ℝdθ ^d are the optimized parameters. The Adam optimizer (Kingma and Ba, 2015) updates θ according to the following steps: gt g_t =∂ℒ(xt)∂θ,xt∼pdata(x), = (x_t)∂θ, x_t p_data(x), (Gradient Computation) (2) mt m_t =β1⋅mt−1+(1−β1)⋅gt, = _1· m_t-1+(1- _1)· g_t, (EMA of the First Moment) (3) vt v_t =β2⋅vt−1+(1−β2)⋅gt2, = _2· v_t-1+(1- _2)· g_t^2, (EMA of the Second Moment) (4) m^t m_t =mt1−β1t,v^t=vt1−β2t, = m_t1- _1^t, v_t= v_t1- _2^t, (Bias-Correction) (5) θt+1 _t+1 =θt−γv^t+ϵ⋅m^t. = _t- γ v_t+ε· m_t. (Parameter Update) (6) Here, mtm_t and vtv_t represent the Exponential Moving Average (EMA) of the first- and second-order moments, respectively. The hyper-parameters include the learning rate γ, the exponential decay rates β1,β2∈[0,1) _1, _2∈[0,1) for the moment estimates, and a small constant ϵ>0ε>0 for numerical stability. Similar to the Adam optimization process, momentum also plays a critical role in other modern optimizers (Shazeer and Stern, 2018; Jordan et al., 2024), enhancing stability and convergence. However, storing momentum states introduces significant memory overhead. Our work directly addresses this by compressing the momentum term to reduce the optimizer’s memory footprint. 3.2 Your Momentum is a Secret Online Linear Regressor To begin with, we reveal an interesting connection: momentum updates in modern optimizers are secretly performing online linear regression. Specifically, updating the momentum m via EMA is mathematically equivalent to optimizing m as the parameter of a linear regressor using online gradient flow. To illustrate this, consider the first-order momentum as an example. The standard EMA update for the first-order momentum can be rewritten as follows: mt+1 m_t+1 =β⋅mt+(1−β)⋅g, =β· m_t+(1-β)· g, (7) =mt⏟weight−(1−β)⏟lr⋅(mt−g)⏟gradient. = m_t_weight- (1-β)_lr· (m_t-g)_gradient. (8) As shown in Equation (8), the EMA update is mathematically equivalent to a gradient descent step where the parameter being optimized is the momentum m, the learning rate is 1−β1-β, and the gradient is ∂L(mt;g)∂m=mt−g ∂ L(m_t;g)∂ m=m_t-g. This reformulation reveals that EMA updates essentially function as an online linear regressor that continuously adjusts the momentum weights based on incoming gradients. The underlying objective being minimized is: minmL(m;g)=12⋅‖m−g‖F2. _mL(m;g)= 12·\|m-g\|^2_F. (9) This insight opens a new avenue for optimizer footprint optimization: since momentum parameters are linear model weights, we can apply standard model compression techniques to reduce optimizer memory usage during training. 3.3 LoRA-Pre: Low-Rank Online Linear Regression We now introduce LoRA-Pre, a new low-rank optimizer for pre-training. Building on the equivalence between exponential moving averages and online linear regression, LoRA-Pre compresses the momentum term m via a low-rank factorization, inspired by the LoRA technique (Hu et al., 2022). This approach can be applied to any momentum-based optimizer, such as Adam (Kingma and Ba, 2015) and Muon (Jordan et al., 2024). We detail the compression strategies for both first- and second-order momentum terms below. First-Order Momentum Compression. Having established that momentum updates are equivalent to gradient descent on the objective minmL(m;g)=12⋅‖m−g‖F2 _mL(m;g)= 12·\|m-g\|^2_F in Section 3.2, we can now apply low-rank compression to reduce memory usage. Instead of storing and updating the full momentum matrix m∈ℝp×qm ^p× q directly, we decompose it into the product of two low-rank matrices mB∈ℝp×rm_B ^p× r and mA∈ℝr×qm_A ^r× q, r≪min(p,q)r (p,q), i.e., m=mB⋅mAm=m_B· m_A. This factorization transforms our original optimization problem into: minmB,mAL(mB,mA;g)=12⋅‖mBmA−g‖F2. _m_B,m_AL(m_B,m_A;g)= 12·\|m_Bm_A-g\|^2_F. (10) To maintain memory efficiency, we solve this optimization problem using standard gradient descent on the factorized matrices mBm_B and mAm_A. To ensure computational efficiency, we derive closed-form update rules for these matrices without requiring back-propagation, which is given by Theorem 3.1. We resort to Newton’s method for updating since the solution can be expressed in the form of EMA. Theorem 3.1. Assume both matrices mB∈ℝp×rm_B ^p× r and mA∈ℝr×qm_A ^r× q are full rank. For the objective minmB,mAL(mB,mA;g)=12⋅‖mBmA−g‖F2 _m_B,m_AL(m_B,m_A;g)= 12·\|m_Bm_A-g\|^2_F, Newton’s method yields the following closed-form update rules: mB m_B ←(1−γ1)⋅mB+γ1⋅gmAT(mAmAT)−1, ←(1- _1)· m_B+ _1· gm_A^T(m_Am_A^T)^-1, (11) mA m_A ←(1−γ1)⋅mA+γ1⋅(mBTmB)−1mBTg. ←(1- _1)· m_A+ _1·(m_B^Tm_B)^-1m_B^Tg. (12) Here, γ1 _1 is the learning rate for the factorized optimization problem. Proof. See Appendix A. ∎ Second-Order Momentum Compression. The compression of second-order momentum v presents additional challenges due to the constraints imposed by Adam’s parameter update rule. Since Equation (6) requires the square root of momentum, i.e., v v, the second-order momentum must be element-wise positive. A naive approach would parameterize the second momentum as v=vB⋅vAv=v_B· v_A and optimize using the regression loss L(vB,vA;g)=12⋅‖vBvA−g2‖F2L(v_B,v_A;g)= 12·\|v_Bv_A-g^2\|^2_F. From Theorem 3.1, we derive the corresponding parameter update rule: vB v_B ←(1−γ2)⋅vB+γ2⋅g2vAT(vAvAT)−1, ←(1- _2)· v_B+ _2· g^2v_A^T(v_Av_A^T)^-1, (13) vA v_A ←(1−γ2)⋅vA+γ2⋅(vBTvB)−1vBTg2. ←(1- _2)· v_A+ _2·(v_B^Tv_B)^-1v_B^Tg^2. (14) Unfortunately, this approach cannot guarantee that vi,j>0,∀i,jv_i,j>0,∀ i,j, making the computation of v=vBvA v= v_Bv_A problematic. To address this issue, we re-parameterize the second-order momentum as v=(vBvA)∘2v=(v_B\,v_A) 2, where ∘ denotes the Hadamard product. This re-parameterization ensures element-wise positivity while maintaining the low-rank structure. We then formulate the optimization of low-rank matrices vBv_B and vAv_A as: minvB,vAL(vB,vA;g)=12⋅‖vBvA−|g|‖F2. _v^B,v^AL(v_B,v_A;g)= 12·\|v_Bv_A-|g|\|^2_F. (15) Its update rule can thus be directly derived from Theorem 3.1. vB v_B ←(1−γ2)⋅vB+γ2⋅|g|vAT(vAvAT)−1, ←(1- _2)· v_B+ _2·|g|v_A^T(v_Av_A^T)^-1, (16) vA v_A ←(1−γ2)⋅vA+γ2⋅(vBTvB)−1vBT|g|. ←(1- _2)· v_A+ _2·(v_B^Tv_B)^-1v_B^T|g|. (17) Low-Rank Optimizer Algorithms. As shown before, our method can be applied to any optimizer with momentum to compress its optimizer state during pre-training and fine-tuning stages. The detailed pseudo-codes of LoRA-Pre optimizer for AdamW (Kingma and Ba, 2015) and Muon (Jordan et al., 2024) are provided in Appendix B. 4 Experimental Results In this section, we present extensive experiments to evaluate the effectiveness of our proposed method, LoRA-Pre. Our evaluation encompasses both memory-efficient pre-training and memory-efficient fine-tuning on downstream tasks. We begin by assessing LoRA-Pre’s pre-training capabilities in Section 4.1. Following the experimental setup of GaLore (Zhao et al., 2024), we train Llama (Touvron et al., 2023) models from scratch with varying model sizes of 60M, 130M, 350M, and 1B parameters. All models are trained on the Colossal Clean Crawled Corpus (C4) dataset (Raffel et al., 2020), a large-scale cleaned dataset specifically designed for language model pre-training. To simulate realistic pre-training conditions, the models are trained on sufficiently large volumes of data without repetition. Subsequently, we evaluate LoRA-Pre’s fine-tuning performance in Section LABEL:subsec:exp_ft. We fine-tune both Llama-3.1-8B (Grattafiori et al., 2024) and Llama-2-7B (Touvron et al., 2023) models on a 100k subset sampled from the MetaMathQA dataset (Yu et al., 2024). The fine-tuned models are then evaluated on the GSM8K (Cobbe et al., 2021) and MATH-500 (Lightman et al., 2024) datasets. Finally, we present an ablation study of LoRA-Pre in Appendix 4.3. Implementation Details. To ensure a fair comparison, we align the experimental setup with that of GaLore (Zhao et al., 2024). By default, LoRA-Pre is applied to all parameters in the attention and MLP layers, while other parameters are optimized using the standard Adam (Kingma and Ba, 2015) optimizer. We set the default ranks for the 60M, 130M, 350M, and 1B parameter models to 128, 256, 256, and 512, respectively. The optimal learning rate is selected from the set 0.01, 0.005, 0.001, 0.0005, 0.0001 based on validation perplexity. To maintain strict fairness in comparison, we retain the same scale factor of 0.25 as used in GaLore (Zhao et al., 2024). For memory-efficient fine-tuning tasks, we set the default rank as 8 and set the learning rate as 2e−52e-5 by default. 4.1 Memory-Efficient Pre-training Table 1: Comparison with low-rank algorithms on pre-training various sizes of Llama models on the C4 dataset. We report the validation perplexity (↓ ) on a hold-out C4 test set. The best and second-best performance within the low-rank optimizers are highlighted with bold and underline. ∗* denotes the results are reproduced by ourselves. Model Size 60M 130M 350M 1B r/dmodelr/d_model 128 / 512 256 / 768 256 / 1024 512 / 2048 Training Tokens 1.1B 2.2B 6.4B 13.1B Adam (Kingma and Ba, 2015) 34.09 25.08 18.80 15.56 Muon (Jordan et al., 2024) 28.43 21.86 16.17 13.41 GaLore (Zhao et al., 2024) 34.88 25.36 18.95 15.64 Low-Rank (Kamalakara et al., 2022) 78.18 45.51 37.41 142.53 LoRA (Hu et al., 2022) 34.99 33.92 25.58 19.21 ReLoRA (Lialin et al., 2024) 37.04 29.37 29.08 18.33 SLTrain (Han et al., 2024) 34.15 26.04 19.42 16.14 LORO (Mo et al., 2025) 33.96 24.59 18.84 15.19 Fira* (Chen et al., 2024) 31.19* 24.51* 17.22* 14.31 [RGB]204, 230, 230 LoRA-Pre Adam 32.57 23.78 16.36 13.53 [RGB]204, 230, 230 LoRA-Pre Muon 30.76 23.05 16.97 13.92 In this section, we evaluate the pre-training performance of our proposed method, LoRA-Pre. Our experimental setup strictly follows that of GaLore (Zhao et al., 2024). We compare LoRA-Pre against several baseline methods, including both full optimizers and low-rank optimizers: 1) Adam (Kingma and Ba, 2015): The de facto optimizer in modern deep learning that utilizes first- and second-order momentum statistics to dynamically adjust learning rates and stabilize training. 2) Muon (Jordan et al., 2024): A novel preconditioned optimizer that updates parameters by orthogonalizing the first-order momentum. 3) GaLore (Zhao et al., 2024): A low-rank optimizer that projects gradients using SVD and computes optimizer states in a reduced subspace. 4) Low-Rank (Kamalakara et al., 2022): A traditional low-rank approach that directly represents weights through learnable low-rank factorization W=BAW=BA. 5) LoRA (Hu et al., 2022): The most widely adopted low-rank method for fine-tuning that factorizes weights as W=W0+BAW=W_0+BA. For pre-training scenarios, we maintain W0W_0 as the full-rank initialization matrix. 6) ReLoRA (Lialin et al., 2024): A LoRA variant designed for pre-training that periodically merges BABA into W and initializes BABA with optimizer state resets. 7) SLTrain (Han et al., 2024): A sparse plus low-rank approach that parameterizes weights as W=S+BAW=S+BA, where both components are jointly optimized. 8) LORO (Mo et al., 2025): A method that optimizes LoRA parameters by strictly constraining updates within the low-rank manifold. 9) Fira (Chen et al., 2024): A method that improves upon GaLore with the Norm-Based Scaling and the Norm-Growth Limiter. We pre-trained Llama series models of different sizes to evaluate LoRA-Pre against these baseline methods. By default, all low-rank optimizers are built upon the Adam (Kingma and Ba, 2015) optimizer foundation. To demonstrate the generalizability of our approach, we also evaluate LoRA-Pre with Muon (Algorithm 2), as our method is compatible with any momentum-based optimizer. The results, presented in Table 1, demonstrate that our method achieves superior performance across multiple model scales. Specifically, LoRA-Pre Adam and LoRA-Pre Muon attain either the highest or second-highest performance across almost all four model sizes (60M, 130M, 350M, and 1B), validating the effectiveness of our approach. While Fira yields competitive results on the 60M model, LoRA-Pre consistently outperforms it on the larger scales (130M, 350M, and 1B), likely because our method avoids the error accumulation associated with Fira’s projected gradients. Furthermore, LoRA-Pre Adam outperforms the previous best efficient baselines by substantial margins of 0.81, 2.45, and 1.6 perplexity points for the 130M, 350M, and 1B models, respectively. Furthermore, when integrated with the Muon (Jordan et al., 2024) optimizer, LoRA-Pre Muon achieves additional improvements on both 60M and 130M scale models, demonstrating our method’s ability to generalize across different optimizers. 4.2 Memory-Efficient Fine-Tuning Table 2: Results of memory-efficient fine-tuning methods. We compare our method with efficient fine-tuning methods including LoRA (Hu et al., 2022), rsLoRA (Kalajdzievski, 2023), and DoRA (Liu et al., 2024), and an efficient optimizer GaLore (Zhao et al., 2024). The models are fine-tuned on the MetaMath100k (Yu et al., 2024) dataset, and evaluated on GSM8K (Cobbe et al., 2021) and MATH-500 (Lightman et al., 2024). We highlight the best performance on Adam-like optimizer and Muon-like optimizer with bold. Method Llama-3.1-8B Llama-2-7B GSM8K MATH-500 Average GSM8K MATH-500 Average LoRA (Hu et al., 2022) 70.76 17.06 43.91 44.62 7.34 25.98 rsLoRA (Kalajdzievski, 2023) 71.06 17.46 44.26 48.79 5.75 27.27 DoRA (Liu et al., 2024) 71.06 17.86 44.46 44.39 6.55 25.47 GaLore (Zhao et al., 2024) 65.08 18.65 41.87 36.44 8.33 22.39 [RGB]204, 230, 230 LoRA-Pre Adam 76.44 17.66 47.05 57.35 6.94 32.15 GaLore Muon (Zhao et al., 2024) 63.41 18.06 40.74 33.11 4.37 18.74 LoRA Muon (Hu et al., 2022) 70.30 19.25 44.78 35.15 6.15 20.65 [RGB]204, 230, 230 LoRA-Pre Muon 72.65 20.83 46.74 47.20 6.15 26.68 In this section, we evaluate the fine-tuning performance of LoRA-Pre on mathematical tasks. We fine-tune Llama-2-7B and Llama-3.1-8B models on the MetaMath100k dataset and assess their performance on GSM8K (Cobbe et al., 2021) and MATH-500 (Lightman et al., 2024). To ensure a fair comparison, we maintain identical hyper-parameters and training configurations across all methods. We select several memory-efficient fine-tuning baselines for comparison, including: 1) LoRA (Hu et al., 2022): the standard low-rank fine-tuning method. 2) rsLoRA (Kalajdzievski, 2023): an improved LoRA variant that optimizes the scaling factor through rank-stabilized normalization. 3) DoRA (Liu et al., 2024): a LoRA extension that decomposes weight updates into magnitude and directional components for more effective optimization. 4) GaLore (Zhao et al., 2024): a memory-efficient optimizer that projects gradients into low-rank subspaces using SVD decomposition. To demonstrate cross-optimizer compatibility, we evaluate Muon-based versions, including: 1) GaLore Muon: which apply the GaLore (Zhao et al., 2024) algorithm to the Muon optimizer, and 2) LoRA Muon: which optimizes LoRA with the Muon optimizer. The results are presented in Table 2. LoRA-Pre consistently achieves the highest scores across all experimental configurations, demonstrating superior performance regardless of the base model or optimizer used. The improvements are particularly notable across different settings: when training Llama-3.1-8B with Adam, LoRA-Pre shows an average improvement of 2.59 points over the second-best method, while with Llama-2-7B and Adam, this improvement increases to 4.88 points. When using the Muon optimizer, LoRA-Pre maintains its advantage with improvements of 1.96 and 6.03 points for the respective models. These results confirm LoRA-Pre’s effectiveness across diverse experimental conditions and its robust compatibility with different optimizers. 4.3 Ablation Study (a) (b) Figure 2: Rank efficiency comparison across efficient optimization methods. Perplexity versus rank for 60M (left) and 130M (right) models, demonstrating LoRA-Pre’s superior performance at lower ranks compared to baseline methods. Ablation of Different Rank. To systematically evaluate how rank selection affects the performance of LoRA-Pre compared to other efficient optimization methods, we conduct comprehensive experiments across different rank configurations. We evaluate LoRA-Pre (both Adam and Muon variants) against GaLore (Zhao et al., 2024) on 60M and 130M parameter models. We test ranks of 4, 16, 64, 128 for the 60M model and 16, 64, 128, 256 for the 130M model to observe performance trends across different memory budgets. Figure 2 shows that all methods improve with increasing rank, but exhibit different rank efficiency. LoRA-Pre consistently achieves better perplexity at lower ranks compared to GaLore. First, all methods show improved performance with increasing rank, but they differ significantly in their rank efficiency. When comparing specific configurations, the efficiency differences become clear. On the 60M model, LoRA-Pre Adam at rank=16 achieves comparable performance to GaLore at rank=128, representing an 8×8× reduction in rank requirement. Similarly, on the 130M model, LoRA-Pre Adam at rank=16 matches GaLore’s performance at rank=256, representing a 16×16× efficiency improvement. LoRA-Pre Muon shows higher rank tolerance than LoRA-Pre Adam. We attribute LoRA-Pre’s rank efficiency to its continuous subspace adaptation mechanism. GaLore performs periodic subspace updates, creating intervals where the subspace becomes misaligned with the gradient structure. To compensate for this error accumulation, GaLore requires larger subspaces. In contrast, LoRA-Pre adjusts its subspace at each step, maintaining better alignment and thus achieving effective optimization with smaller subspaces. To gain deeper insights into this rank efficiency, we examine the training dynamics of LoRA-Pre Muon across different rank configurations. Figure 3 visualizes the perplexity trajectories for the 130M model with ranks of 256, 128, 64, and 16. The results reveal an intriguing convergence pattern: while smaller ranks initially exhibit higher perplexity values, this performance gap diminishes rapidly as training progresses. This behavior demonstrates that LoRA-Pre’s dynamic subspace update mechanism can efficiently capture the evolving momentum structure during training, even when operating with constrained ranks. This rapid adaptation capability explains why LoRA-Pre maintains competitive performance across a wide range of rank settings, making it both robust to rank selection and practically appealing for memory-constrained training scenarios. Figure 3: Test perplexity for LoRA-Pre Muon with different ranks during training. Table 3: Results of pre-training using different efficient Muon optimizers. Model Size 60M 130M 350M Muon (Jordan et al., 2024) 28.43 21.86 16.17 Muon w/o momentum 32.15 24.23 17.33 GaLore Muon (Zhao et al., 2024) 34.39 25.16 19.24 Fira Muon (Chen et al., 2024) 34.45 24.85 17.40 [RGB]204, 230, 230 LoRA-Pre Muon 30.76 23.05 16.97 Ablation of Low-Rank Muon Optimizers. In this section, we evaluate the effectiveness of current efficient optimizers by extending them to the recently proposed Muon optimizer (Jordan et al., 2024). Since existing efficient optimizers were originally designed for Adam (Kingma and Ba, 2015), their compatibility and performance with other optimizers remain unexplored. We conduct experiments on 60M, 130M, and 350M parameter models, comparing LoRA-Pre against GaLore (Zhao et al., 2024) and Fira (Chen et al., 2024) by adapting their implementations to use Muon. Standard Muon serves as the upper bound, while Muon without momentum provides the lower bound. The Muon-based algorithm for LoRA-Pre is presented in Algorithm 2. The results in Table 3 reveal two significant findings. First, LoRA-Pre Muon consistently outperforms all other efficient optimizers, achieving improvements of 3.54, 1.80, and 0.43 points over the second-best method at 60M, 130M, and 350M parameters, respectively. Second, projection-based methods surprisingly perform worse than basic Muon without momentum, despite incorporating momentum computation. This counterintuitive result exposes fundamental generalization limitations of projection-based gradient descent methods when applied to different optimizers. We attribute this phenomenon to the periodic subspace updates in projection-based methods, which introduce momentum computation errors that subsequently affect Muon’s orthogonal update calculations. In contrast, LoRA-Pre continuously updates its subspace, enabling better capture of the orthogonal space during Muon’s update process and achieving superior performance. 5 Conclusion In this paper, we present LoRA-Pre, a novel low-rank efficient optimizer. We establish that EMA momentum updates are mathematically equivalent to training an online linear regressor with gradient descent on the online gradient flow. Building on this insight, we propose compressing the momentum component through low-rank factorization, deriving update rules that maintain the EMA form while operating in a compressed parameter space. We provide two variants: LoRA-Pre Adam and LoRA-Pre Muon. Extensive experiments on pre-training and fine-tuning tasks demonstrate that LoRA-Pre achieves competitive or superior performance across all evaluated tasks and model sizes. Notably, our method exhibits excellent rank robustness, requiring only 1/8 or fewer ranks compared to previous methods while achieving comparable results. The approach generalizes effectively to various optimizers, making it a versatile solution for memory-efficient optimization. Acknowledgement This work was funded by the National Natural Science Foundation of China under Grants (62276256, U2441251, 62550062, 62425606, 32341009) and the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001). We gratefully acknowledge Jie Cheng and Zhen Yang for providing computational resources and for their valuable discussions, which were critical to this work. References A. Aghajanyan, S. Gupta, and L. Zettlemoyer (2021) Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In ACL-IJCNLP, Cited by: §2. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. In NeurIPS, Cited by: §1. X. Chen, K. Feng, C. Li, X. Lai, X. Yue, Y. Yuan, and G. Wang (2024) Fira: can we achieve full-rank training of llms under low-rank constraint?. arXiv preprint arXiv:2410.01623. Cited by: §1, §2, §4.1, §4.3, Table 1, Table 3. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §4.2, Table 2, §4. G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §1. N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C. Chan, W. Chen, et al. (2023) Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5 (3), p. 220–235. Cited by: §2. J. Duchi, E. Hazan, and Y. Singer (2011) Adaptive subgradient methods for online learning and stochastic optimization. JMLR 12 (7). Cited by: §3.1. A. Edalati, M. Tahaei, I. Kobyzev, V. P. Nia, J. J. Clark, and M. Rezagholizadeh (2023) Krona: parameter efficient tuning with kronecker adapter. In NeurIPS Workshop, Cited by: §2. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §1, §1, §4. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1. A. Han, J. Li, W. Huang, M. Hong, A. Takeda, P. K. Jawanpuria, and B. Mishra (2024) SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining. In NeurIPS, Cited by: §2, §4.1, Table 1. Y. Hao, Y. Cao, and L. Mou (2024) Flora: low-rank adapters are secretly gradient compressors. In ICML, Cited by: §1, §2. S. Hayou, N. Ghosh, and B. Yu (2024) Lora+: efficient low rank adaptation of large models. In ICML, Cited by: §2, §2. G. Hinton, N. Srivastava, and K. Swersky (2012) Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited by: §3.1. E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) LoRA: low-rank adaptation of large language models. In ICLR, Cited by: §2, §3.3, §4.1, §4.2, Table 1, Table 2, Table 2, Table 2. A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. (2024) Openai o1 system card. arXiv preprint arXiv:2412.16720. Cited by: §1. K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein (2024) Muon: an optimizer for hidden layers in neural networks. External Links: Link Cited by: §B.2, Appendix B, §1, §3.1, §3.3, §3.3, §4.1, §4.1, §4.3, Table 1, Table 3. D. Kalajdzievski (2023) A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732. Cited by: §4.2, Table 2, Table 2. S. R. Kamalakara, A. Locatelli, B. Venkitesh, J. Ba, Y. Gal, and A. N. Gomez (2022) Exploring low rank training of deep neural networks. arXiv preprint arXiv:2209.13569. Cited by: §2, §4.1, Table 1. D. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In ICLR, Cited by: Appendix B, §1, §1, §3.1, §3.1, §3.3, §3.3, §3, §4.1, §4.1, §4.3, Table 1, §4. V. Lialin, S. Muckatira, N. Shivagunde, and A. Rumshisky (2024) ReLoRA: high-rank training through low-rank updates. In ICLR, Cited by: §2, §4.1, Table 1. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2024) Let’s verify step by step. In ICLR, Cited by: §4.2, Table 2, §4. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig (2023) Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Computing Surveys 55 (9), p. 1–35. Cited by: §2. S. Liu, C. Wang, H. Yin, P. Molchanov, Y. F. Wang, K. Cheng, and M. Chen (2024) Dora: weight-decomposed low-rank adaptation. In ICML, Cited by: §2, §2, §4.2, Table 2, Table 2. X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang (2022) P-tuning: prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL, Cited by: §2. C. Ma, W. Gong, M. Scetbon, and E. Meeds (2025) SWAN: sgd with normalization and whitening enables stateless llm training. In ICML, Cited by: §2. P. Mahdavinia and M. Mahdavi (2025) Low-rank momentum factorization for memory efficient training. TMLR. External Links: Link Cited by: §2, §2. Z. Mo, L. Huang, and S. J. Pan (2025) Parameter and memory efficient pretraining via low-rank riemannian optimization. In ICLR, Cited by: §4.1, Table 1. I. Modoranu, M. Safaryan, E. Schultheis, and D. Alistarh (2025) SVD-free low-rank adaptive gradient optimization for large language models. arXiv preprint arXiv:2505.17967. Cited by: §1, §2. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR 21 (140), p. 1–67. Cited by: §4. N. Shazeer and M. Stern (2018) Adaptive learning rates with sublinear memory cost. In ICML, Cited by: §3.1. W. Shen, Y. Zhang, M. Huang, M. Xu, J. Zhang, and C. Shen (2025) MLorc: momentum low-rank compression for large language model adaptation. arXiv preprint arXiv:2506.01897. Cited by: §2, §2. N. Tastan, S. Laskaridis, M. Takáč, K. Nandakumar, and S. Horváth (2025) LoFT: low-rank adaptation that behaves like full fine-tuning. In ICML Workshop, External Links: Link Cited by: §2, §2. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023) Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §1, §4, §4. S. Wang, L. Yu, and J. Li (2024a) LoRA-ga: low-rank adaptation with gradient approximation. In NeurIPS, Cited by: §2. Z. Wang, J. Liang, R. He, Z. Wang, and T. Tan (2024b) Connecting the dots: collaborative fine-tuning for black-box vision-language models. In ICML, Cited by: §2. Z. Wang, J. Liang, R. He, Z. Wang, and T. Tan (2025) LoRA-pro: are low-rank adapters properly optimized?. In ICLR, Cited by: §2, §2. Z. Wang, J. Liang, R. He, N. Xu, Z. Wang, and T. Tan (2023) Improving zero-shot generalization for clip with synthesized prompts. In ICCV, Cited by: §2. Z. Wang, J. Liang, L. Sheng, R. He, Z. Wang, and T. Tan (2024c) A hard-to-beat baseline for training-free clip-based adaptation. In ICLR, Cited by: §2. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §1. J. Yen, S. Si, Z. Meng, F. Yu, S. S. Duvvuri, I. S. Dhillon, C. Hsieh, and S. Kumar (2025) LoRA done rite: robust invariant transformation equilibration for lora optimization. In ICLR, Cited by: §2. L. Yu, W. Jiang, H. Shi, Y. Jincheng, Z. Liu, Y. Zhang, J. Kwok, Z. Li, A. Weller, and W. Liu (2024) MetaMath: bootstrap your own mathematical questions for large language models. In ICLR, Cited by: Table 2, §4. Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao (2023) Adaptive budget allocation for parameter-efficient fine-tuning. In ICLR, Cited by: §2. Y. Zhang, Y. Liu, and C. Fang (2025) AdaPM: a partial momentum algorithm for llm training. arXiv preprint arXiv:2510.09103. Cited by: §2, §2. J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian (2024) GaLore: memory-efficient llm training by gradient low-rank projection. In ICML, Cited by: §1, §2, §4.1, §4.2, §4.3, §4.3, Table 1, Table 2, Table 2, Table 2, Table 3, §4, §4. P. Zmushko, A. Beznosikov, M. Takáč, and S. Horváth (2025) FRUGAL: memory-efficient optimization by reducing state overhead for scalable training. In ICML, Cited by: §2. Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation —Appendix— The structure of the Appendix is as follows, • Appendix A contains the proofs of the theorems in the main manuscript. • Appendix B details the optimization algorithms of the proposed method. • Appendix C provides theoretical analysis of approximation error and convergence of the proposed method. • Appendix D provides additional experiments of our method. • Appendix E details the LLM usage in this paper. Appendix A Proof of Theoretical Results Theorem. Assume matrices mB∈ℝp×rm_B ^p× r and mA∈ℝr×qm_A ^r× q are both full rank. For the objective minmB,mAL(mB,mA;g)=12⋅‖mBmA−g‖F2 _m_B,m_AL(m_B,m_A;g)= 12·\|m_Bm_A-g\|^2_F, Newton’s method yields the following closed-form update rules: mB m_B ←(1−γ1)⋅mB+γ1⋅gmAT(mAmAT)−1, ←(1- _1)· m_B+ _1· gm_A^T(m_Am_A^T)^-1, (18) mA m_A ←(1−γ1)⋅mA+γ1⋅(mBTmB)−1mBTg. ←(1- _1)· m_A+ _1·(m_B^Tm_B)^-1m_B^Tg. (19) Here, γ1 _1 is the learning rate for the factorized optimization problem. Proof. We aim to derive Newton’s method update rules for the optimization problem minmB,mAL(mB,mA;g)=12⋅‖mBmA−g‖F2 _m_B,m_AL(m_B,m_A;g)= 12·\|m_Bm_A-g\|^2_F. Our approach begins with computing the first-order gradients, then proceeds to the Hessian computation, and finally establishes the connection to exponential moving average (EMA) updates. To start, we compute the first-order partial derivatives: ∂L∂mB ∂ L∂ m_B =(mBmA−g)mAT, =(m_Bm_A-g)m_A^T, (20) ∂L∂mA ∂ L∂ m_A =mBT(mBmA−g). =m_B^T(m_Bm_A-g). (21) While standard gradient descent would directly use these gradients to update the parameters, we instead pursue Newton’s method because it yields a more elegant form that naturally resembles EMA updates. For Newton’s method, we need the second-order derivatives (Hessian matrices). Computing these second-order partial derivatives gives us: HBB H_B =∂2L∂mB2=∂[(mBmA−g)mAT]∂mB=mAmAT⊗Ip, = ∂^2L∂ m_B^2= ∂ [ (m_Bm_A-g )m_A^T ]∂ m_B=m_Am_A^T I_p, (22) HAA H_A =∂2L∂mA2=∂[mBT(mBmA−g)]∂mA=Iq⊗mBTmB. = ∂^2L∂ m_A^2= ∂ [m_B^T (m_Bm_A-g ) ]∂ m_A=I_q m_B^Tm_B. (23) Using these Hessian matrices, we can now compute the Newton directions by solving the linear systems H⋅d=∇LH· d=∇ L, which yields: vec(dmB) (d_m_B) =HBB−1vec(∂L∂mB)⟹dmB=mB−gmAT(mAmAT)−1, =H_B^-1vec ( ∂ L∂ m_B ) d_m_B=m_B-gm_A^T(m_Am_A^T)^-1, (24) vec(dmA) (d_m_A) =HAA−1vec(∂L∂mA)⟹dmA=mA−(mBTmB)−1mBTg. =H_A^-1vec ( ∂ L∂ m_A ) d_m_A=m_A-(m_B^Tm_B)^-1m_B^Tg. (25) The key insight emerges when we apply these Newton directions with learning rate γ1 _1. Substituting the Newton directions into the update formula x←x−γ1dxx← x- _1d_x, we obtain: mB m_B ←mB−γ1⋅dmB ← m_B- _1· d_m_B =mB−γ1⋅[mB−gmAT(mAmAT)−1] =m_B- _1· [m_B-gm_A^T(m_Am_A^T)^-1 ] =(1−γ1)⋅mB+γ1⋅gmAT(mAmAT)−1, =(1- _1)· m_B+ _1· gm_A^T(m_Am_A^T)^-1, (26) mA m_A ←mA−γ1⋅dmA ← m_A- _1· d_m_A =mA−γ1⋅[mA−(mBTmB)−1mBTg] =m_A- _1· [m_A-(m_B^Tm_B)^-1m_B^Tg ] =(1−γ1)⋅mA+γ1⋅(mBTmB)−1mBTg. =(1- _1)· m_A+ _1·(m_B^Tm_B)^-1m_B^Tg. (27) These final expressions reveal the remarkable property that Newton’s method naturally produces update rules in the form of exponential moving averages, where each new parameter value is a weighted combination of the previous value and a target value derived from the optimization objective. To further illustrate this connection, we note that in the uncompressed case where we optimize minmL(m;g)=12‖m−g‖F2 _mL(m;g)= 12\|m-g\|^2_F, Newton’s method similarly yields the classic EMA update: m m ←m−γ⋅Hmm−1∂L∂m ← m-γ· H_m^-1 ∂ L∂ m =(1−γ)⋅m+γ⋅g. =(1-γ)· m+γ· g. (28) This consistency across problem formulations demonstrates the fundamental nature of this EMA-like structure in Newton’s method and justifies our preference for this approach over standard gradient descent. ∎ Appendix B Detailed Algorithms of LoRA-Pre for Adam and Muon Optimizers This section presents the LoRA-Pre algorithms for both Adam (Kingma and Ba, 2015) and Muon (Jordan et al., 2024) optimizers. B.1 Algorithm of LoRA-Pre for Adam The Adam optimizer update rules under LoRA-Pre have been established in Section 3. First-order momentum updates: For the first-order momentum term with parameterization m=mBmAm=m_Bm_A, the update rules are: mB′ m_B ←(1−γ1)⋅mB+γ1⋅gmAT(mAmAT)−1, ←(1- _1)· m_B+ _1· gm_A^T(m_Am_A^T)^-1, (29) mA′ m_A ←(1−γ1)⋅mA+γ1⋅(mBTmB)−1mBTg. ←(1- _1)· m_A+ _1·(m_B^Tm_B)^-1m_B^Tg. (30) By default, we set 1−γ1=β11- _1= _1, which ensures that after the update, m′=mB′mA′=β1⋅mBmA+⋯m =m_B m_A = _1· m_Bm_A+~·s, making the EMA coefficient consistent with standard Adam. Second-order momentum updates: For the second-order momentum term with parameterization v=(vBvA)∘2v=(v_Bv_A) 2, the update rules are: vB′ v_B ←(1−γ2)⋅vB+γ2⋅|g|vAT(vAvAT)−1, ←(1- _2)· v_B+ _2·|g|v_A^T(v_Av_A^T)^-1, (31) vA′ v_A ←(1−γ2)⋅vA+γ2⋅(vBTvB)−1vBT|g|. ←(1- _2)· v_A+ _2·(v_B^Tv_B)^-1v_B^T|g|\,. (32) Analogously, we set 1−γ2=β20.251- _2= _2^0.25 by default, which ensures that v′=(vB′vA′)∘2=β2⋅(vBvA)∘2+⋯v =(v_B v_A ) 2= _2·(v_Bv_A) 2+~·s. Complete algorithm: Based on these update formulas, Algorithm 1 presents the complete LoRA-Pre implementation for the Adam optimizer, demonstrating how these factorized momentum updates integrate seamlessly into the standard Adam framework. Algorithm 1 Comparison of Adam and Adam with LoRA-Pre 0: Initial learning rate γ, weight decay λ, β1,β2∈[0,1) _1, _2∈[0,1), γ1,γ2∈[0,1) _1, _2∈[0,1), ϵ>0ε>0 1: Initialize parameters θ0 _0, time step t←0t← 0, first moment m0←0m_0← 0, second moment v0←0v_0← 0, first low-rank moment mB,0←0m_B,0← 0, mA,0∼(0,0.02)m_A,0 (0,0.02),second low-rank moment vB,0←0v_B,0← 0, vA,0∼(0,0.02)v_A,0 (0,0.02). 2: repeat 3: t←t+1t← t+1 4: gt←∇θℒt(θt−1)g_t← _θL_t( _t-1) 5: # Update first moment 6: mt←β1⋅mt−1+(1−β1)⋅gtm_t← _1· m_t-1+(1- _1)· g_t 7: mt←β1⋅mB,t−1mA,t−1+(1−β1)⋅gtm_t← _1· m_B,t-1m_A,t-1+(1- _1)· g_t 8: mB,t←(1−γ1)⋅mB,t−1+γ1⋅gtmA,t−1T(mA,t−1mA,t−1T)−1m_B,t←(1- _1)· m_B,t-1+ _1· g_tm_A,t-1^T(m_A,t-1m_A,t-1^T)^-1 9: mA,t←(1−γ1)⋅mA,t−1+γ1⋅(mB,t−1TmB,t−1)−1mB,t−1Tgtm_A,t←(1- _1)· m_A,t-1+ _1·(m_B,t-1^Tm_B,t-1)^-1m_B,t-1^Tg_t 10: # Update second moment 11: vt←β2⋅vt−1+(1−β2)⋅gt∘2v_t← _2· v_t-1+(1- _2)· g_t 2 12: vt←β2⋅(vB,t−1vA,t−1)∘2+(1−β2)⋅gt∘2v_t← _2·(v_B,t-1v_A,t-1) 2+(1- _2)· g_t 2 13: vB,t←(1−γ2)⋅vB,t−1+γ2⋅|gt|⋅vA,t−1T(vA,t−1vA,t−1T)−1v_B,t←(1- _2)· v_B,t-1+ _2·|g_t|· v_A,t-1^T(v_A,t-1v_A,t-1^T)^-1 14: vA,t←(1−γ2)⋅vA,t−1+γ2⋅(vB,t−1TvB,t−1)−1vB,t−1T⋅|gt|v_A,t←(1- _2)· v_A,t-1+ _2·(v_B,t-1^Tv_B,t-1)^-1v_B,t-1^T·|g_t| 15: m^t←mt/(1−β1t) m_t← m_t/(1- _1^t) 16: v^t←vt/(1−β2t) v_t← v_t/(1- _2^t) 17: θt←θt−1−γ(m^tv^t+ϵ+λθt−1) _t← _t-1-γ ( m_t v_t+ε+λ\, _t-1 ) 18: until stopping criterion is met 19: return Optimized parameters θt _t B.2 Algorithm of LoRA-Pre for Muon In this section, we present the LoRA-Pre algorithm for the Muon (Jordan et al., 2024) optimizer. First-order momentum updates: For the Muon optimizer, we derive the LoRA-Pre algorithm by first reformulating the momentum update. The Muon momentum term can be equivalently written as: m′ m =μ⋅m+g =μ· m+g =m−(1−μ)⋅m+g =m-(1-μ)· m+g =m−(1−μ)⋅(m−g)+μ⋅g =m-(1-μ)·(m-g)+μ· g =m−(1−μ)⋅[(m−g)−μ1−μ⋅g]. =m-(1-μ)· [(m-g)- μ1-μ· g ]\,. (33) By treating the Muon update as the solution to an optimization problem, we can derive the equivalent objective function: L(m;g)=12⋅‖m−g‖F2−μ1−μ⋅⟨m,g⟩F.L(m;g)= 12·\|m-g\|^2_F- μ1-μ· m,g _F. (34) After applying low-rank factorization m=mB⋅mAm=m_B· m_A, the objective becomes: L(mB,mA;g)=12⋅‖mBmA−g‖F2−μ1−μ⋅⟨mBmA,g⟩F.L(m_B,m_A;g)= 12·\|m_Bm_A-g\|^2_F- μ1-μ· m_Bm_A,g _F. (35) We aim to derive Newton’s method update rules for this modified objective. Computing the first-order gradients: ∂L∂mB ∂ L∂ m_B =(mBmA−g)mAT−μ1−μ⋅gmAT, =(m_Bm_A-g)m_A^T- μ1-μ· gm_A^T, (36) ∂L∂mA ∂ L∂ m_A =mBT(mBmA−g)−μ1−μ⋅mBTg. =m_B^T(m_Bm_A-g)- μ1-μ· m_B^Tg. (37) The Hessian matrices have the same structure as before since the additional linear term doesn’t affect the second derivatives: HBB H_B =∂2L∂mB2=∂[(mBmA−g)mAT]∂mB=mAmAT⊗Ip, = ∂^2L∂ m_B^2= ∂ [(m_Bm_A-g)m_A^T ]∂ m_B=m_Am_A^T I_p, (38) HAA H_A =∂2L∂mA2=∂[mBT(mBmA−g)]∂mA=Iq⊗mBTmB. = ∂^2L∂ m_A^2= ∂ [m_B^T(m_Bm_A-g) ]∂ m_A=I_q m_B^Tm_B. (39) Using these Hessian matrices, we can now compute the Newton directions by solving the linear systems H⋅d=∇LH· d=∇ L, which yields: vec(dmB) (d_m_B) =HBB−1vec(∂L∂mB)⟹dmB=mB−11−μ⋅gmAT(mAmAT)−1, =H_B^-1vec ( ∂ L∂ m_B ) d_m_B=m_B- 11-μ· gm_A^T(m_Am_A^T)^-1, (40) vec(dmA) (d_m_A) =HAA−1vec(∂L∂mA)⟹dmA=mA−11−μ⋅(mBTmB)−1mBTg. =H_A^-1vec ( ∂ L∂ m_A ) d_m_A=m_A- 11-μ·(m_B^Tm_B)^-1m_B^Tg. (41) The key insight emerges when we apply these Newton directions with learning rate γ1 _1. Substituting the Newton directions into the update formula x←x−γ1⋅dxx← x- _1· d_x, we obtain: mB m_B ←mB−γ1⋅dmB ← m_B- _1· d_m_B =mB−γ1⋅[mB−11−μ⋅gmAT(mAmAT)−1] =m_B- _1· [m_B- 11-μ· gm_A^T(m_Am_A^T)^-1 ] =(1−γ1)⋅mB+γ11−μ⋅gmAT(mAmAT)−1, =(1- _1)· m_B+ _11-μ· gm_A^T(m_Am_A^T)^-1, (42) mA m_A ←mA−γ1⋅dmA ← m_A- _1· d_m_A =mA−γ1⋅[mA−11−μ⋅(mBTmB)−1mBTg] =m_A- _1· [m_A- 11-μ·(m_B^Tm_B)^-1m_B^Tg ] =(1−γ1)⋅mA+γ11−μ⋅(mBTmB)−1mBTg. =(1- _1)· m_A+ _11-μ·(m_B^Tm_B)^-1m_B^Tg. (43) Similarly, we set 1−γ1=μ1- _1= μ. Complete algorithm: Based on these update formulas, Algorithm 2 presents the complete LoRA-Pre implementation for the Muon optimizer, demonstrating how these factorized momentum updates integrate seamlessly into the standard Muon framework. Algorithm 2 Comparison of Muon and Muon with LoRA-Pre 0: Initial learning rate γ, weight decay λ, momentum μ∈[0,1)μ∈[0,1), γ1∈[0,1) _1∈[0,1) 1: Initialize parameters θ0 _0, time step t←0t← 0, first moment m0←0m_0← 0,first low-rank moment mB,0←0m_B,0← 0, mA,0∼(0,0.02)m_A,0 (0,0.02). 2: repeat 3: t←t+1t← t+1 4: gt←∇θℒt(θt−1)g_t← _θL_t( _t-1) 5: # Update first moment 6: mt←μ⋅mt−1+gtm_t←μ· m_t-1+g_t 7: mt←μ⋅mB,t−1mA,t−1+gtm_t←μ· m_B,t-1m_A,t-1+g_t 8: mB,t←(1−γ1)⋅mB,t−1+γ11−μ⋅gtmA,t−1T(mA,t−1mA,t−1T)−1m_B,t←(1- _1)· m_B,t-1+ _11-μ· g_tm_A,t-1^T(m_A,t-1m_A,t-1^T)^-1 9: mA,t←(1−γ1)⋅mA,t−1+γ11−μ⋅(mB,t−1TmB,t−1)−1mB,t−1Tgtm_A,t←(1- _1)· m_A,t-1+ _11-μ·(m_B,t-1^Tm_B,t-1)^-1m_B,t-1^Tg_t 10: Ot←NewtonSchulz5(mt)O_t 5(m_t) 11: θt←θt−1−γ(Ot+λθt−1) _t← _t-1-γ (O_t+λ\, _t-1 ) 12: until stopping criterion is met 13: return Optimized parameters θt _t Appendix C Theoretical Analysis of Approximation Error and Convergence In this appendix, we provide a rigorous theoretical analysis of the LoRA-Pre Adam optimizer. We explicitly analyze the approximation error introduced by the low-rank factorization of the optimizer states, and prove the convergence fidelity of the algorithm in non-convex settings. C.1 Problem Setup and Algorithm Dynamics Consider the unconstrained optimization problem minθ∈ℝp×qf(θ) _θ ^p× qf(θ). Let gt=∇f(θt−1)g_t=∇ f( _t-1) be the stochastic gradient at step t. We denote the states of Standard Adam as mt,vtm_t,\,v_t and the effective states of LoRA-Pre Adam as m~t,v~t m_t,\, v_t. 1. Standard Adam Dynamics The standard optimizer updates its moments using exponential moving averages (EMA) with decay rates β1,β2∈[0,1) _1, _2∈[0,1): mt m_t =β1⋅mt−1+(1−β1)⋅gt, = _1· m_t-1+(1- _1)· g_t, (44) vt v_t =β2⋅vt−1+(1−β2)⋅gt∘2. = _2· v_t-1+(1- _2)· g_t 2. (45) 2. LoRA-Pre Dynamics LoRA-Pre maintains low-rank factors (mB,t,mA,t)(m_B,t,m_A,t) to approximate the gradient history. Let γ1 _1 be the update rate. The exact simultaneous update rules (Online Least Squares) can be compactly expressed using the Moore-Penrose pseudo-inverse (⋅)†(·) : mB,t m_B,t =(1−γ1)⋅mB,t−1+γ1⋅gtmA,t−1†, =(1- _1)· m_B,t-1+ _1· g_tm_A,t-1 , (46) mA,t m_A,t =(1−γ1)⋅mA,t−1+γ1⋅mB,t−1†gt, =(1- _1)· m_A,t-1+ _1· m_B,t-1 g_t, (47) where the Tikhonov regularized pseudo-inverses for the full-rank factors are defined as mA†=mAT(mAmAT+λ⋅I)−1m_A =m_A^T(m_Am_A^T+λ· I)^-1 and mB†=(mBTmB+λ⋅I)−1mBTm_B =(m_B^Tm_B+λ· I)^-1m_B^T with damping factor λ>0λ>0. We define the canonical projection operators associated with the factors at step t−1t-1: • A≜mA,t−1†mA,t−1P_A m_A,t-1 m_A,t-1 (Projection onto Row Space of mAm_A). • B≜mB,t−1mB,t−1†P_B m_B,t-1m_B,t-1 (Projection onto Column Space of mBm_B). Let m^t=mB,tmA,t m_t=m_B,tm_A,t be the low-rank history reconstruction. Analogous updates apply to the second moment factors using the gradient magnitude |gt||g_t|, producing the reconstruction h^t=vB,tvA,t h_t=v_B,tv_A,t. 3. Effective Moments for Update Crucially, LoRA-Pre computes the effective moments for the parameter update by combining the low-rank history with the exact current gradient: m~t m_t =β1⋅m^t−1+(1−β1)⋅gt, = _1· m_t-1+(1- _1)· g_t, (48) v~t v_t =β2⋅(h^t−1)∘2+(1−β2)⋅gt∘2. = _2·( h_t-1) 2+(1- _2)· g_t 2. (49) Note that for the second moment, LoRA-Pre approximates the history of magnitudes h h and then squares it. C.2 Assumptions Assumption 1 (Regularity and Boundedness). The objective function and stochastic gradients satisfy the following conditions: 1. L-Smoothness: The objective function f is L-smooth: ‖∇f(x)−∇f(y)‖F≤L‖x−y‖F\|∇ f(x)-∇ f(y)\|_F≤ L\|x-y\|_F. 2. Bounded Gradients: The stochastic gradients are uniformly bounded in both Frobenius and infinity norms. There exist constants G and G∞G_∞ such that for all t, ‖gt‖F≤G\|g_t\|_F≤ G and ‖gt‖∞≤G∞\|g_t\|_∞≤ G_∞. 3. Lipschitz Smooth Update: The optimizer uses a smoothed damping term ϵ>0ε>0. The update mapping is defined as ϕ(m,v)=mv+ϵφ(m,v)= m v+ε. This function is Lipschitz continuous with respect to both arguments, with constants Lm=1ϵL_m= 1 ε and Lv=G2ϵ1.5L_v= G2ε^1.5. Assumption 2 (Subspace Approximation Capability). The gradient dynamics admit a low-rank structure. Crucially, we assume this structure holds for both the gradient direction and its element-wise magnitude. Let B,t,A,tP_B,t,P_A,t denote the projections onto the subspaces maintained by the optimizer at step t. We assume there exists a bound δ≥0δ≥ 0 such that: ‖gt−(B,tgt+gtA,t)‖F \|g_t-(P_B,tg_t+g_tP_A,t)\|_F ≤δ, ≤δ, (50) ‖|gt|−(B,t|gt|+|gt|A,t)‖F \||g_t|-(P_B,t|g_t|+|g_t|P_A,t)\|_F ≤δ. ≤δ. (51) The second inequality ensures that the second-moment estimator (based on |gt||g_t|) also admits a bounded reconstruction error. Assumption 3 (Reference Optimizer Descent). Let ut=mt/(vt+ϵ)u_t=m_t/( v_t+ε) be the update direction of the standard full-rank Adam optimizer. We assume that in expectation, utu_t is a valid descent direction aligned with the true gradient: [⟨∇f(θt),ut⟩]≥c[‖∇f(θt)‖F2],E[ ∇ f( _t),u_t ]≥ cE[\|∇ f( _t)\|_F^2], (52) for some constant c>0c>0. This assumption anchors the convergence of LoRA-Pre Adam to the theoretical behavior of standard Adam. C.3 Boundedness of Factor Reconstruction Error We first prove that the error of the stored low-rank history m^t m_t is uniformly bounded. We strictly enforce the time-scale alignment condition: β1=(1−γ1)2 _1=(1- _1)^2. Lemma C.1. Let ℰtm=‖mt−m^t‖FE^m_t=\|m_t- m_t\|_F. Under Assumptions 1 and 2, ℰtmE^m_t is uniformly bounded by a constant ℰboundE_bound. Proof. Step 1: Exact Expansion of LoRA Dynamics Substitute the update rules (46) and (47) into m^t=mB,tmA,t m_t=m_B,tm_A,t: m^t m_t =[(1−γ1)mB,t−1+γ1gtmA,t−1†]⋅[(1−γ1)mA,t−1+γ1mB,t−1†gt] = [(1- _1)m_B,t-1+ _1g_tm_A,t-1 ]· [(1- _1)m_A,t-1+ _1m_B,t-1 g_t ] =(1−γ1)2m^t−1+γ1(1−γ1)(Bgt+gtA)+γ12Qt, =(1- _1)^2 m_t-1+ _1(1- _1)(P_Bg_t+g_tP_A)+ _1^2Q_t, (53) where Qt=gtmA,t−1†mB,t−1†gtQ_t=g_tm_A,t-1 m_B,t-1 g_t is the quadratic interaction term. Using the condition β1=(1−γ1)2 _1=(1- _1)^2, this implies γ1=1−β1 _1=1- _1. The expansion becomes: m^t=β1m^t−1+(1−β1)β1(Bgt+gtA)+(1−β1)2Qt. m_t= _1 m_t-1+(1- _1) _1(P_Bg_t+g_tP_A)+(1- _1)^2Q_t. (54) Step 2: Constructing the Recursive Error We form the difference with the standard Adam update mt=β1mt−1+(1−β1)gtm_t= _1m_t-1+(1- _1)g_t: mt−m^t m_t- m_t =β1(mt−1−m^t−1)+Rt, = _1(m_t-1- m_t-1)+R_t, (55) where the residual driving term RtR_t is: Rt=(1−β1)gt−(1−β1)β1(Bgt+gtA)−(1−β1)2Qt.R_t=(1- _1)g_t-(1- _1) _1(P_Bg_t+g_tP_A)-(1- _1)^2Q_t. (56) Step 3: Bounding the Residual Magnitude We explicitly bound the norm of the driving residual RtR_t. Applying the triangle inequality to the residual definition (56), we decompose the total bound into linear and quadratic contributions: ‖Rt‖F≤‖(1−β1)[gt+β1(gt−Bgt−gtA)]‖F⏟Linear Term+(1−β1)2‖Qt‖F⏟Quadratic Term.\|R_t\|_F≤ \|(1- _1) [g_t+ _1(g_t-P_Bg_t-g_tP_A) ] \|_F_Linear Term+ (1- _1)^2\|Q_t\|_F_Quadratic Term. (57) 1. The Linear Term: Using the homogeneity of the norm and the triangle inequality, followed by the substitution of the gradient bound ‖gt‖F≤G\|g_t\|_F≤ G (Assumption 1) and the subspace residual bound δ (Assumption 2), we derive: ‖Linear‖F \|Linear\|_F =(1−β1)‖gt+β1(gt−Bgt−gtA)‖F =(1- _1) \|g_t+ _1(g_t-P_Bg_t-g_tP_A) \|_F ≤(1−β1)(‖gt‖F+β1‖gt−Bgt−gtA‖F) ≤(1- _1) (\|g_t\|_F+ _1\|g_t-P_Bg_t-g_tP_A\|_F ) ≤(1−β1)G+β1(1−β1)δ. ≤(1- _1)G+ _1(1- _1)δ. (58) 2. The Quadratic Term: Bounding ‖Qt‖F\|Q_t\|_F requires the spectral norm of the regularized pseudo-inverses. As established in the setup, mA†=mAT(mAmAT+λI)−1m_A =m_A^T(m_Am_A^T+λ I)^-1 and mB†=(mBTmB+λI)−1mBTm_B =(m_B^Tm_B+λ I)^-1m_B^T. Let the Singular Value Decomposition (SVD) of the factor be UΣVTU V^T. Substituting this into either inverse form reveals that the singular values are governed by h(σ)=σ2+λh(σ)= σ^2+λ. Analyzing the function f(x)=x2+λf(x)= xx^2+λ for x≥0x≥ 0, we find its maximum at x=λx= λ, yielding the uniform spectral bound ‖M†‖2≤12λ\|M \|_2≤ 12 λ. Using the sub-multiplicativity of the Frobenius norm (specifically ‖ABC‖F≤‖A‖F‖B‖2‖C‖F\|ABC\|_F≤\|A\|_F\|B\|_2\|C\|_F), we strictly bound QtQ_t: ‖Qt‖F \|Q_t\|_F =‖gtmA,t−1†mB,t−1†gt‖F =\|g_tm_A,t-1 m_B,t-1 g_t\|_F ≤‖gt‖F⋅‖mA,t−1†‖2⋅‖mB,t−1†‖2⋅‖gt‖F ≤\|g_t\|_F·\|m_A,t-1 \|_2·\|m_B,t-1 \|_2·\|g_t\|_F =‖gt‖F2⋅‖mA,t−1†‖2⋅‖mB,t−1†‖2 =\|g_t\|_F^2·\|m_A,t-1 \|_2·\|m_B,t-1 \|_2 ≤G2(12λ)2=G24λ≜CQ. ≤ G^2 ( 12 λ )^2= G^24λ C_Q. (59) Substituting both bounds back into the residual equation: ‖Rt‖F≤(1−β1)G+β1(1−β1)δ+(1−β1)2CQ≜Δres.\|R_t\|_F≤(1- _1)G+ _1(1- _1)δ+(1- _1)^2C_Q _res. (60) Step 4: Convergence to Steady State The error dynamics follow the linear recursion inequality ℰtm≤β1ℰt−1m+ΔresE^m_t≤ _1E^m_t-1+ _res. Since the momentum parameter satisfies 0≤β1<10≤ _1<1, this geometric series converges to a finite steady state as t→∞t→∞: limt→∞ℰtm≤Δres1−β1≜ℰbound. _t→∞E^m_t≤ _res1- _1 _bound. (61) To derive the explicit bound, we substitute the expression for Δres _res derived in Equation (60) and factor the denominator using 1−β1=(1−β1)(1+β1)1- _1=(1- _1)(1+ _1): ℰbound _bound =(1−β1)G+β1(1−β1)δ+(1−β1)2CQ(1−β1)(1+β1) = (1- _1)G+ _1(1- _1)δ+(1- _1)^2C_Q(1- _1)(1+ _1) =(1−β1)[G+β1δ+(1−β1)CQ](1−β1)(1+β1). = (1- _1) [G+ _1δ+(1- _1)C_Q ](1- _1)(1+ _1). (62) Canceling the common factor (1−β1)(1- _1) from the numerator and denominator, we obtain the final uniform bound: ℰbound=G+β1δ+(1−β1)CQ1+β1.E_bound= G+ _1δ+(1- _1)C_Q1+ _1. (63) This confirms that the reconstruction error is uniformly bounded by a constant determined by the gradient magnitude G, the subspace residual δ, and the quadratic interaction CQC_Q. ∎ C.4 Joint Effective Moment Error We now derive the error bounds for the effective moments m~t m_t and v~t v_t used in the parameter update, explicitly accounting for the non-linear square term in v~t v_t. Lemma C.2. Let Δm=‖mt−m~t‖F _m=\|m_t- m_t\|_F and Δv=‖vt−v~t‖F _v=\|v_t- v_t\|_F denote the effective moment errors. Let d be the total number of parameters. The errors are bounded by: Δm _m ≤β1ℰbound, ≤ _1E_bound, (64) Δv _v ≤β2(2G∞ℰbound+σtotal2), ≤ _2 (2G_∞E_bound+ _total^2 ), (65) where σtotal2=d4G∞2 _total^2= d4G_∞^2 represents the intrinsic variance bound derived from Popoviciu’s inequality. Proof. 1. First Moment Error Analysis We compare the effective first moment m~t m_t with the standard Adam moment mtm_t. Subtracting their definitions: mt−m~t=β1(mt−1−m^t−1)+(1−β1)(gt−gt)=β1(mt−1−m^t−1).m_t- m_t= _1(m_t-1- m_t-1)+(1- _1)(g_t-g_t)= _1(m_t-1- m_t-1). (66) The current gradient terms cancel out. Taking the Frobenius norm and applying Lemma C.1, we obtain: Δm≤β1ℰbound. _m≤ _1E_bound. (67) 2. Second Moment Error Analysis We analyze the error in the second moment estimate. Standard Adam updates the second moment as vt=β2vt−1+(1−β2)gt∘2v_t= _2v_t-1+(1- _2)g_t 2. LoRA-Pre updates the effective second moment as v~t=β2(h^t−1)∘2+(1−β2)gt∘2 v_t= _2( h_t-1) 2+(1- _2)g_t 2. Subtracting the two equations, the term (1−β2)gt∘2(1- _2)g_t 2 involving the current gradient cancels out exactly. The error is strictly propagated from the history terms: vt−v~t=β2[vt−1−(h^t−1)∘2].v_t- v_t= _2 [v_t-1-( h_t-1) 2 ]. (68) To rigorously bound this, we introduce an auxiliary variable ht−1h_t-1, defined as the exact exponential moving average of the gradient magnitudes |g||g| (i.e., hk=β2hk−1+(1−β2)|gk|h_k= _2h_k-1+(1- _2)|g_k|). We apply the triangle inequality to decompose the error into two structurally distinct terms: Δv≤β2(‖vt−1−(ht−1)∘2‖F⏟Term I: Intrinsic Variance+‖(ht−1)∘2−(h^t−1)∘2‖F⏟Term I: Approximation Error). _v≤ _2 ( \|v_t-1-(h_t-1) 2\|_F_Term I: Intrinsic Variance+ \|(h_t-1) 2-( h_t-1) 2\|_F_Term I: Approximation Error ). (69) Term I: Intrinsic Variance (Popoviciu’s Bound). This term captures the discrepancy between “the mean of squares” and “the square of the mean”. Let us focus on a single parameter index j. The EMA formulation can be interpreted as a weighted expectation wE_w over the history of gradients, where the weights sum to 1. Thus: vt−1,j=w[|gτ,j|2]and(ht−1,j)2=(w[|gτ,j|])2.v_t-1,j=E_w[|g_τ,j|^2] (h_t-1,j)^2=(E_w[|g_τ,j|])^2. (70) The difference vt−1,j−(ht−1,j)2v_t-1,j-(h_t-1,j)^2 is precisely the variance of the random variable |gτ,j||g_τ,j| under the EMA probability measure. Since the gradient magnitude is bounded, i.e., |gτ,j|∈[0,G∞]|g_τ,j|∈[0,G_∞], we invoke Popoviciu’s Inequality, which states that the variance of any bounded random variable on interval [a,b][a,b] is bounded by 14(b−a)2 14(b-a)^2. Here, [a,b]=[0,G∞][a,b]=[0,G_∞]. Thus, for every element j: |vt−1,j−(ht−1,j)2|≤14(G∞−0)2=14G∞2. |v_t-1,j-(h_t-1,j)^2 |≤ 14(G_∞-0)^2= 14G_∞^2. (71) We generalize this to the full parameter matrix by computing the Frobenius norm over all d elements: ‖vt−1−(ht−1)∘2‖F=∑j=1d|vt−1,j−(ht−1,j)2|2≤d⋅(14G∞2)2=d4G∞2.\|v_t-1-(h_t-1) 2\|_F= _j=1^d |v_t-1,j-(h_t-1,j)^2 |^2≤ d· ( 14G_∞^2 )^2= d4G_∞^2. (72) We define this constant bound as σtotal2≜d4G∞2 _total^2 d4G_∞^2. Term I: Approximation Error. This term represents the propagation of the low-rank reconstruction error through the squaring operation. Consider the function f(x)=x2f(x)=x^2. For inputs restricted to the domain [0,G∞][0,G_∞], the derivative is bounded by |f′(x)|=|2x|≤2G∞|f (x)|=|2x|≤ 2G_∞. By the Mean Value Theorem, the function is Lipschitz continuous with constant Lsq=2G∞L_sq=2G_∞. Applying this element-wise to the vectors ht−1h_t-1 and h^t−1 h_t-1: |(ht−1,j)2−(h^t−1,j)2|≤2G∞|ht−1,j−h^t−1,j|. |(h_t-1,j)^2-( h_t-1,j)^2 |≤ 2G_∞ |h_t-1,j- h_t-1,j |. (73) Squaring both sides and summing over all j to recover the Frobenius norm: ∑j=1d|(ht−1,j)2−(h^t−1,j)2|2≤4G∞2∑j=1d|ht−1,j−h^t−1,j|2. _j=1^d |(h_t-1,j)^2-( h_t-1,j)^2 |^2≤ 4G_∞^2 _j=1^d |h_t-1,j- h_t-1,j |^2. (74) Taking the square root yields: ‖(ht−1)∘2−(h^t−1)∘2‖F≤2G∞‖ht−1−h^t−1‖F.\|(h_t-1) 2-( h_t-1) 2\|_F≤ 2G_∞\|h_t-1- h_t-1\|_F. (75) Substituting the bound from Lemma C.1 (‖ht−1−h^t−1‖F≤ℰbound\|h_t-1- h_t-1\|_F _bound), we have: Term I≤2G∞ℰbound.Term I≤ 2G_∞E_bound. (76) Combining the bounds for Term I and Term I, we obtain the final bound for the second moment error: Δv≤β2(σtotal2+2G∞ℰbound). _v≤ _2 ( _total^2+2G_∞E_bound ). (77) ∎ C.5 Convergence Analysis Theorem C.3. Let the step size be ηt=η/t _t=η/ t. Under Assumptions 1, 2, and 3, LoRA-Pre Adam (with update rule ϕ(m,v)=m/v+ϵφ(m,v)=m/ v+ε) converges to a neighborhood of a stationary point: min1≤t≤T[‖∇f(θt)‖2]≤CinitT+Cnoise(ℰbound+σtotal2)2, _1≤ t≤ TE[\|∇ f( _t)\|^2]≤ C_init T+C_noise (E_bound+ _total^2 )^2, (78) where CinitC_init depends on the initial function gap and CnoiseC_noise depends on the Lipschitz constants of the update rule. Proof. 1. Bounding the Update Direction Error Let ut=ϕ(mt,vt)u_t=φ(m_t,v_t) and u~t=ϕ(m~t,v~t) u_t=φ( m_t, v_t) be the ideal and actual update directions, respectively. We analyze the theoretical update function ϕ(m,v)=mv+ϵφ(m,v)= m v+ε (using the smoothed denominator for Lipschitz continuity). The partial derivatives are bounded as follows: • W.r.t m: ‖∇mϕ‖≤1ϵ≜Lm\| _mφ\|≤ 1 ε L_m. • W.r.t v: ‖∇vϕ‖≤supv≥0‖−m2(v+ϵ)1.5‖≤G2ϵ1.5≜Lv\| _vφ\|≤ _v≥ 0 \| -m2(v+ε)^1.5 \|≤ G2ε^1.5 L_v. Using the moment error bounds from Lemma C.2, the total direction error ξt=‖ut−u~t‖F _t=\|u_t- u_t\|_F satisfies: ξt _t ≤LmΔm+LvΔv ≤ L_m _m+L_v _v ≤1ϵ(β1ℰbound)+G2ϵ1.5β2(2G∞ℰbound+σtotal2) ≤ 1 ε( _1E_bound)+ G2ε^1.5 _2(2G_∞E_bound+ _total^2) ≤(β1ϵ+GG∞β2ϵ1.5)⏟K1ℰbound+(Gβ22ϵ1.5)⏟K2σtotal2≜ξtotal. ≤ ( _1 ε+ G_∞ _2ε^1.5 )_K_1E_bound+ ( G _22ε^1.5 )_K_2 _total^2 _total. (79) Here, ξtotal _total is a uniform deterministic upper bound on the direction error. 2. One-Step Descent Analysis in Expectation Since the objective function f is L-smooth, we apply the Descent Lemma conditioned on the current state θt _t: t[f(θt+1)]≤f(θt)−ηtt[⟨∇f(θt),u~t⟩]+Lηt22t[‖u~t‖2].E_t[f( _t+1)]≤ f( _t)- _tE_t[ ∇ f( _t), u_t ]+ L _t^22E_t[\| u_t\|^2]. (80) Let Gstep=G/ϵG_step=G/ ε be the bound on the update norm ‖u~t‖\| u_t\|. We decompose the inner product term: −t[⟨∇f,u~t⟩]=−t[⟨∇f,ut⟩]+t[⟨∇f,ut−u~t⟩].-E_t[ ∇ f, u_t ]=-E_t[ ∇ f,u_t ]+E_t[ ∇ f,u_t- u_t ]. (81) Using Assumption 3 (t[⟨∇f,ut⟩]≥c‖∇f‖2E_t[ ∇ f,u_t ]≥ c\|∇ f\|^2) and applying Cauchy-Schwarz followed by Young’s Inequality (ab≤c2a2+12cb2ab≤ c2a^2+ 12cb^2) to the error term: t[⟨∇f,ut−u~t⟩] _t[ ∇ f,u_t- u_t ] ≤‖∇f‖ξtotal ≤\|∇ f\| _total ≤c2‖∇f‖2+12cξtotal2. ≤ c2\|∇ f\|^2+ 12c _total^2. (82) Substituting this back into the descent inequality, the descent term −c‖∇f‖2-c\|∇ f\|^2 is partially offset by the error: t[f(θt+1)] _t[f( _t+1)] ≤f(θt)−ηt(c‖∇f‖2−c2‖∇f‖2−12cξtotal2)+Lηt22Gstep2 ≤ f( _t)- _t (c\|∇ f\|^2- c2\|∇ f\|^2- 12c _total^2 )+ L _t^22G_step^2 =f(θt)−cηt2‖∇f‖2+ηt2cξtotal2+Lηt22Gstep2. =f( _t)- c _t2\|∇ f\|^2+ _t2c _total^2+ L _t^22G_step^2. (83) 3. Global Convergence Taking the total expectation and summing from t=1t=1 to T: ∑t=1Tcηt2[‖∇f(θt)‖2]≤f(θ1)−f∗+ξtotal22c∑t=1Tηt+LGstep22∑t=1Tηt2. _t=1^T c _t2E[\|∇ f( _t)\|^2]≤ f( _1)-f^*+ _total^22c _t=1^T _t+ LG_step^22 _t=1^T _t^2. (84) Dividing by c2ST c2S_T (where ST=∑ηtS_T=Σ _t), we isolate the minimum gradient norm: mint[‖∇f‖2]≤2(f(θ1)−f∗)cST+1c2ξtotal2+LGstep2c∑ηt2ST. _tE[\|∇ f\|^2]≤ 2(f( _1)-f^*)cS_T+ 1c^2 _total^2+ LG_step^2c Σ _t^2S_T. (85) Given ηt=η/t _t=η/ t, we have ST≥TS_T≥ T and ∑ηt2≈lnTΣ _t^2≈ T. Thus, as T→∞T→∞, the first and third terms vanish (O(1/T)O(1/ T) and O(lnT/T)O( T/ T)), leaving the constant error floor. Defining Cnoise=1c2max(K1,K2)2C_noise= 1c^2 (K_1,K_2)^2 (noting that ξtotal2≤2max(K1,K2)2(ℰbound+σtotal2)2 _total^2≤ 2 (K_1,K_2)^2(E_bound+ _total^2)^2), we conclude: mint[‖∇f‖2]≤CinitT+Cnoise(ℰbound+σtotal2)2. _tE[\|∇ f\|^2]≤ C_init T+C_noise(E_bound+ _total^2)^2. (86) This confirms that the algorithm converges to a neighborhood determined by the approximation quality ℰboundE_bound and intrinsic variance σtotal2 _total^2. ∎ Appendix D Additional Experimental Results D.1 Hyperparameter Sensitivity Analysis: The Coupling of β and γ In this section, we evaluate the sensitivity of LoRA-Pre Adam to variations in its momentum hyperparameters. A key design principle of LoRA-Pre is that the low-rank update coefficients (γ1,γ2)( _1, _2) are not independent hyperparameters requiring separate tuning. Instead, they are analytically coupled with the standard Adam momentum coefficients (β1,β2)( _1, _2) to ensure consistent decay dynamics. Derivation of the Coupling Strategy. Formally, the update rules for the low-rank momentum components mBm_B and mAm_A are defined as: mB′ m _B ←(1−γ1)⋅mB+γ1⋅gmAT(mAmAT)−1, ←(1- _1)· m_B+ _1· gm_A^T(m_Am_A^T)^-1, (87) mA′ m _A ←(1−γ1)⋅mA+γ1⋅(mBTmB)−1mBTg. ←(1- _1)· m_A+ _1·(m_B^Tm_B)^-1m_B^Tg. (88) When reconstructing the effective full-rank momentum m≈mBmAm≈ m_Bm_A, the decay behavior can be approximated by expanding the product of the updated factors. Ignoring higher-order cross terms, the effective decay rate is the product of the individual decay rates: m′=mB′mA′≈(1−γ1)2⋅mBmA+(γ12)≈(1−γ1)2⋅m.m =m _Bm _A≈(1- _1)^2· m_Bm_A+O( _1^2)≈(1- _1)^2· m. (89) To align this trajectory with standard Adam optimization (where momentum decays by β1 _1), we enforce the coupling constraint (1−γ1)2=β1(1- _1)^2= _1. Similarly, for the second moment, since LoRA-Pre approximates the gradient magnitude h^≈|g| h≈|g| and subsequently squares it (v~=h^∘2 v= h 2), the decay rate compounds twice, leading to the condition (1−γ2)4=β2(1- _2)^4= _2. Consequently, the learning dynamics are strictly governed by β, eliminating the need to grid-search γ. Empirical Verification. We conducted sensitivity studies on the 60M parameter model by varying β1 _1 and β2 _2 around their standard default values (β1=0.9,β2=0.95 _1=0.9, _2=0.95). The results are summarized in Table 4. Table 4: Sensitivity Analysis of Momentum Parameters β. We report the validation perplexity on the 60M model. The method exhibits stability around the default settings derived from standard Adam (β1=0.9,β2=0.95 _1=0.9, _2=0.95). Extreme values (e.g., β→1β→ 1) cause the implicit γ to vanish, leading to training instability. Parameter Value Perplexity Status β1 _1 (fix β2=0.95 _2=0.95) 0.90 (Default) 32.57 Optimal 0.95 37.62 Stable 0.99 1458.92 Diverged β2 _2 (fix β1=0.90 _1=0.90) 0.90 34.61 Stable 0.95 (Default) 32.57 Optimal 0.999 1301.58 Diverged As shown in Table 4, LoRA-Pre achieves the best performance at the configuration inherited from standard Adam. While the optimizer is robust within a reasonable range, extreme values (e.g., β1=0.99 _1=0.99 or β2=0.999 _2=0.999) lead to numerical instability. This is expected: as β→1β→ 1, the derived update rate γ approaches zero (e.g., for β2=0.999 _2=0.999, γ2≈2.5×10−4 _2≈ 2.5× 10^-4), causing the low-rank factors to adapt too slowly to track the shifting gradient subspace. This confirms that our coupling strategy is effective, but the “sweet spot” for low-rank adaptation dynamics aligns with, but is more sensitive than, full-rank optimization. Appendix E Statement of the Use of Large Language Models The use of LLMs in this work was restricted to paper writing assistance. They were not used to generate results, derive proofs, or conduct analysis without human verification. The disclosure here, as well as in the submission form, fulfills the ICLR requirement that all contributions of LLMs be acknowledged transparently.