Paper deep dive
Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning
Yuan Zhang, Jiang Hu, Zhijian Lai, Lin Lin, Zaiwen Wen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/1/2026, 11:34:04 AM
Summary
The paper introduces a retraction-free and penalty parameter-free optimization algorithm for the Stiefel manifold, establishing global convergence guarantees. It applies this method to Low-Rank Adaptation (LoRA) fine-tuning for large language models, proposing 'Manifold-LoRA' to address redundancy in weight updates by optimizing over the Stiefel manifold, resulting in faster convergence and competitive performance compared to vanilla LoRA.
Entities (7)
Relation Signals (5)
Manifold-LoRA → uses → Stiefel manifold
confidence 95% · we reformulate LoRA fine-tuning as an optimization problem over the product of Stiefel manifolds and Euclidean spaces.
Retraction-free Algorithm → optimizes → Stiefel manifold
confidence 92% · we propose a retraction-free and penalty parameter-free algorithm that directly lands on the manifold.
Manifold-LoRA → demonstratesefficiencyon → SQuAD 2.0
confidence 90% · Numerical experiments on benchmark datasets demonstrate the efficiency ... including the SQuAD 2.0 dataset
Manifold-LoRA → improves → LoRA
confidence 90% · Compared with vanilla LoRA, our Manifold-LoRA with half the trainable parameters delivers fast convergence and competitive downstream performance.
Manifold-LoRA → buildsupon → AdamW
confidence 85% · Building upon the established landing theory ... and the AdamW framework, we propose a new method, Manifold-LoRA
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Optimization over the Stiefel manifold plays a significant role in various machine learning tasks. Existing methods either use the retraction operators, requiring costly orthonormalization for large-scale matrices, or employ landing methods that rely on careful step size selection and penalty parameter tuning. To address these challenges, we propose a retraction-free and penalty parameter-free algorithm that directly lands on the manifold. By leveraging the strongly-convex-like property of the quadratic penalty function and the proximal smoothness of the Stiefel manifold, we establish global convergence guarantees with the best-known iteration complexities under both constant and diminishing step sizes. Then, we reformulate the low-rank adaptation (LoRA) fine-tuning problem for large language models as a manifold optimization problem, introducing Manifold-LoRA for geometry-accelerated adaptation. This approach employs the proposed landing technique and a carefully designed step size strategy to accelerate the training process. Numerical experiments on benchmark datasets demonstrate the efficiency and strong downstream performance of the proposed method.
Tags
Links
- Source: https://arxiv.org/abs/2607.25299v1
- Canonical: https://arxiv.org/abs/2607.25299v1
Trouble viewing inline? Open PDF directly →
Full Text
118,217 characters extracted from source content.
Expand or collapse full text
Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning Yuan Zhang, Jiang Hu, Zhijian Lai, Lin Lin, and Zaiwen Wen Yuan Zhang is with Center for Data Science, Peking University, Beijing 100871, China Email: zy1002@stu.pku.edu.cn.Jiang Hu is with Yau Mathematical Sciences Center, Tsinghua University, Beijing 100084, China Email: jianghu@tsinghua.edu.cn.Zhijian Lai is with Beijing International Center for Mathematical Research, Peking University, Beijing 100871, China Email: lai_zhijian@pku.edu.cn.Lin Lin is with Department of Mathematics, University of California, Berkeley, CA, US. Email: linlin@math.berkeley.edu.Zaiwen Wen is with Beijing International Center for Mathematical Research, Center for Machine Learning Research and Changsha Institute for Computing and Digital Economy, Peking University, Beijing, China Email: wenzw@pku.edu.cn. Abstract Optimization over the Stiefel manifold plays a significant role in various machine learning tasks. Existing methods either use the retraction operators, requiring costly orthonormalization for large-scale matrices, or employ landing methods that rely on careful step size selection and penalty parameter tuning. To address these challenges, we propose a retraction-free and penalty parameter-free algorithm that directly lands on the manifold. By leveraging the strongly-convex-like property of the quadratic penalty function and the proximal smoothness of the Stiefel manifold, we establish global convergence guarantees with the best-known iteration complexities under both constant and diminishing step sizes. Then, we reformulate the low-rank adaptation (LoRA) fine-tuning problem for large language models as a manifold optimization problem, introducing Manifold-LoRA for geometry-accelerated adaptation. This approach employs the proposed landing technique and a carefully designed step size strategy to accelerate the training process. Numerical experiments on benchmark datasets demonstrate the efficiency and strong downstream performance of the proposed method. I Introduction Optimization over the Stiefel manifold has attracted considerable attention in the context of machine learning, e.g., RNN [4], batch normalization [14], distributionally robust optimization [12], and vision transformer [32]. The mathematical formulation of this class of problems is minX∈ℝd×r _X ^d× r f(X)=1N∑i=1Nfi(X) f(X)= 1N _i=1^Nf_i(X) (1) s.t. X∈St(d,r):=X∈ℝd×r:X⊤X=I, X∈ St(d,r):= \X ^d× r:X X=I \, where fi:ℝd×r→ℝf_i:R^d× r are continuously differentiable functions and integers r≤dr≤ d. The most popular methods for solving (1) are retraction-based algorithms, which have been extensively studied in the context of manifold optimization [3, 53, 29, 11]. Recently, to alleviate the possible computational burden of the retraction operator, some retraction-free methods have been developed in [24, 23, 1, 2, 55]. Their ideas are based on a combination of the manifold geometry and a penalty function for the manifold constraint. Due to their scalability and parallelization efficiency, retraction-free algorithms are well suited to certain large-scale machine learning applications. In such methods, one must simultaneously control both the constraint violation and the optimality of the loss function. Noting the similarity to decentralized optimization—where the consensus gradient step size is fixed by a known value and only the loss gradient step sizes require tuning [39, 50, 45, 13, 19]—we seek to develop retraction-free algorithms with a known penalty parameter for solving (1). Another motivation for studying retraction-free methods arises from its application in the fine-tuning of large language models (LLMs). Recently, LLMs have revolutionized the field of natural language processing (NLP), achieving unprecedented performance in various applications [46, 44]. To tailor pretrained LLMs for specific downstream tasks, the most common approach is full fine-tuning, which requires prohibitively large computational resources due to the need to adapt all model weights, hindering the deployment of large models. Parameter-efficient fine-tuning (PEFT) has gained widespread attention as it requires few trainable parameters while delivering results comparable to or even superior to full fine-tuning. This paradigm involves inserting learnable modules or designating only a small portion of weights as trainable, keeping the main model frozen [27, 35, 56]. Among fine-tuning methods, low-rank adaptation (LoRA) [28] has become the de facto standard among parameter-efficient fine-tuning techniques. It assumes that the change in weights lies in a low intrinsic dimension, thereby modeling the update ΔW∈ℝd×m W ^d× m by two low-rank (not greater than a small integer r) matrices A∈ℝr×mA ^r× m and B∈ℝd×rB ^d× r, i.e., ΔW=BA W=BA. Since r≪dr d, the requirements on both storage and computation are significantly reduced. Due to its decompositional nature, there is redundancy in the representation of ΔW W. Traditional optimization methods for LoRA are unable to exploit this redundancy, which consequently undermines the performance of the models. Instead, we reformulate LoRA fine-tuning as an optimization problem over the product of Stiefel manifolds and Euclidean spaces.111During the review of the initial version of this manuscript, several concurrent studies emerged (see, e.g., [7, 6, 42]). They adopt closely related manifold constraints to address redundancy in LoRA and use retraction-based algorithms. Therefore, we propose an algorithmic framework called Manifold-LoRA to accelerate the fine-tuning process and enhance model performance. Moreover, by exploiting projected gradients and incorporating a parameter-free penalty, the overhead that our method incurs is relatively negligible. Our contributions are as follows: • We prove the existence of an explicit penalty parameter by establishing a strong-convexity-like condition for the nonconvex penalty problem associated with the Stiefel manifold constraint. Building on the concept of proximal smoothness for the Stiefel manifold, we then derive convergence results for retraction-free algorithms with an explicit penalty parameter in both stochastic and deterministic settings. Notably, we show that the iterates converge exactly under a constant step size in the deterministic setting, thereby improving upon the prior convergence-to-neighborhood result in [1]. Furthermore, our explicit choice of the penalty parameter achieves a better iteration complexity for the constraint violation in the stochastic setting with a decaying step size than that reported in [2], since we use two-scale step sizes—decaying for the loss-gradient step but constant for the penalty-gradient step—whereas [2] uses a single-scale decaying step size for both. Moreover, our analysis framework—motivated by decentralized optimization[39, 19]—employs a linear decay of the constraint violation and the descent property of the loss function, which contrasts with the augmented Lagrangian-based approach used in [2]. • Building upon the established landing theory of retraction-free and penalty parameter-free method and the AdamW framework, we propose a new method, Manifold-LoRA, which employs a carefully designed step size strategy to accelerate the training process of fine-tuning. Compared with the conventional AdamW method, we use the penalized gradient instead of the usual gradient, and the computational overhead is negligible. Numerical experiments are conducted on a wide range of NLP tasks, demonstrating the efficiency of our algorithm. Specifically, compared to vanilla LoRA, our Manifold-LoRA with half the trainable parameters delivers fast convergence and competitive downstream performance. In particular, our method converges twice as fast as baseline methods on several typical datasets, including the SQuAD 2.0 dataset and the CoLA dataset. I-A Related Work Optimization over the Stiefel manifold Optimization over the Stiefel manifold has attracted lots of attention due to its broad applications. Through the use of retraction, known as the generalization of the exponential map, the Riemannian gradient descent is proposed [3, 11, 29], where all iterations lie in the manifold. When such retraction is computationally costly, the authors [24] develop a retraction-free algorithm based on the augmented Lagrangian method. More recently, by defining the constraint dissolving operator and adding a sufficiently large penalty term, the authors [55] convert the manifold constrained problem (1) into an unconstrained problem and then apply unconstrained optimization algorithms. A closely related paper to ours is [2], which first connects an augmented Lagrangian-based merit function with the landing field and then establishes the convergence by exploring the descent property of the merit function. Although the parameter associated with the penalty term could be arbitrarily chosen, an additional search is still needed when conducting landing algorithms. Inspired by the convergence of Oja’s flow, a retraction-free method is developed in [1] for the squared Stiefel manifold (i.e., d=rd=r), where the landing flow consists of the projected gradient and the gradient of the penalty function. All of these methods rely on an unknown penalty parameter to ensure the convergence. This motivates us to design penalty parameter-free algorithms, which could significantly reduce the need for tuning parameters in practical implementations. LoRA There are numerous variants of LoRA aiming to improve performance or reduce memory usage. AdaLoRA [58], a well-known successor, introduces the idea of adaptively adjusting the rank of different layers by incorporating an additional vector g to serve as the diagonal of a singular value matrix. This approach leverages a revised sensitivity-based importance measure to decide whether to disable the entries in the vector g and in the matrices A and B. A similar work, SoRA [20], adopts the same model architecture as AdaLoRA, but proposes a different way to update the vector g after training. This update rule is the proximal gradient of ℒ1L_1 loss, acting as a post-pruning method. Additionally, based on the idea that networks with random initialization contain subnetworks that are optimal[22], VeRA is proposed in [33] to reduce memory overhead. Although LoRA has gained significant popularity and various variants have been developed, the potential for efficient training through leveraging the manifold geometry to reduce redundancy has not been well-explored. I-B Notation For a matrix X∈ℝd×rX ^d× r, we use ‖X‖\|X\| to denote its Frobenius norm. For a squared matrix A∈ℝr×rA ^r× r, we define sym(A)=A+A⊤2 sym(A)= A+A 2 and use diag(A)∈ℝr diag(A) ^r to denote its diagonal part. For two matrices X,Y∈ℝd×rX,Y ^d× r, we use ⟨X,Y⟩:=∑i=1d∑j=1rXijYij X,Y := _i=1^d _j=1^rX_ijY_ij to denote their Euclidean inner product. For a differential function f:ℝd×r→ℝf:R^d× r , we use ∇f(X)∇ f(X) to denote its usual Euclidean gradient at X. We define USt(d,r)(18)=X∈ℝd×r∣dist(X,St(d,r))<18U_ St(d,r)( 18)=\X ^d× r dist(X, St(d,r))< 18\ and U¯St(d,r)(18)=X∈ℝd×r∣dist(X,St(d,r))≤18 U_ St(d,r)( 18)=\X ^d× r dist(X, St(d,r))≤ 18\ with dist(X,St(d,r)):=minY∈St(d,r)‖Y−X‖ dist(X, St(d,r)):= _Y∈ St(d,r)\|Y-X\|. Let 1 represent the all-ones vector. I Manifold Optimization for LoRA Fine-tuning In this section, we begin by reformulating LoRA fine-tuning as a manifold optimization problem, incorporating an additional constraint on the matrix B, which serves as the basis matrix in our reformulation. We further introduce retraction operators and review classical retraction-based methods. I-A Manifold Optimization Formulation of LoRA Fine-tuning One possible drawback in the current LoRA fine-tuning framework is that the low-rank decomposition ΔW W into product BABA is not unique. Specifically, for any invertible matrix C, it holds that BA=(BC)(C−1A)BA=(BC)(C^-1A). Note that BCBC shares the same column space with B. This suggests optimizing the subspace generated by B instead of B itself. Numerous studies in the field of low-rank optimization, e.g., [10, 18, 17], investigate the manifold geometry of the low-rank decomposition and develop efficient algorithms. However, such geometry has not been explored in the LoRA fine-tuning. To address such redundancy (i.e., the non-uniqueness of BABA representations), we regard B as the basis through the manifold constraint and A as the coordinate of ΔW W under B. Hence, the optimization problem can be formulated as minA∈ℝr×m,B∈ℝd×r _A ^r× m,\;B ^d× r ℒ(BA), (BA), (2) s.t. B∈St(d,r)orB∈Ob(d,r), B∈ St(d,r) ~or~B∈ Ob(d,r), where Ob(d,r):=B∈ℝd×r:diag(B⊤B)= Ob(d,r):=\B ^d× r: diag(B B)=1\ and ℒL represents the loss function. Compared to the Stiefel manifold St(d,r) St(d,r), the Oblique manifold Ob(d,r) Ob(d,r) necessitates that the matrix B has unit norms in its columns, without imposing requirements for orthogonality between the columns. Problem (2) is an optimization problem over the product of manifolds and Euclidean spaces. I-B Retraction-based Manifold Optimization Manifold optimization has attracted much attention in the past few decades, as evident in works such as [3, 29, 11]. For the Stiefel manifold St(d,r) St(d,r), its tangent space at X is denoted by TXSt(d,r):=ξ∈ℝd×r:X⊤ξ+ξ⊤X=0T_X St(d,r):=\ξ ^d× r:X ξ+ξ X=0\, which is defined as the subspace consisting of all tangent vectors. For a differentiable function f, the Riemannian gradient gradf(X)∈TXSt(d,r) gradf(X)∈ T_X St(d,r) is the unique tangent vector satisfying ⟨gradf(X),ξ⟩X=df(X)[ξ],∀ξ∈TXSt(d,r), gradf(X),ξ _X= df(X)[ξ],~~∀ξ∈ T_X St(d,r), where ⟨⋅,⋅⟩X ·,· _X is the Riemannian metric and df df denotes the differential of function f. Since St(d,r) St(d,r) is a submanifold embedded in ℝd×rR^d× r, by setting the Riemannian metric as the Euclidean metric, the Riemannian gradient of f at X is gradf(X)=∇f(X)−Xsym(X⊤∇f(X)), gradf(X)=∇ f(X)-X sym(X ∇ f(X)), which is the projection of ∇f(X)∇ f(X) onto the tangent space TXSt(d,r)T_X St(d,r). The normal space NXSt(d,r)N_X St(d,r) is defined as the orthogonal complement of TXSt(d,r)T_X St(d,r) in ℝd×rR^d× r. In the design of Riemannian algorithms, an essential concept is the so-called retraction operator. A retraction operator ℛR at X, denoted as ℛXR_X, is a mapping from TXSt(d,r)T_X St(d,r) to St(d,r) St(d,r) that satisfies the following two properties: • ℛX(0X)=XR_X(0_X)=X and 0X0_X is the zero element of TXSt(d,r)T_X St(d,r); • dtℛX(tξ)∣t=0=ξ d dtR_X(tξ) _t=0=ξ for any ξ∈TXSt(d,r)ξ∈ T_X St(d,r). It is well-known that the retraction operator is a generalization of the exponential map [3]. The iterative scheme of a Riemannian gradient descent method is usually given by Xk+1=ℛXk(tkgradf(Xk)),X_k+1=R_X_k(t_k gradf(X_k)), where tk>0t_k>0 is a step size. For the Stiefel manifold St(d,r) St(d,r), there are several choices for the retraction ℛR, such as the exponential map, the Cayley transform, the QR decomposition, and the polar decomposition, see [29] for details. Among them, the Cayley transformation proposed by [53] is popularly used. It can be expressed as, for any η∈TXSt(d,r)η∈ T_X St(d,r), ℛXCayley(−η)=X−U(I2r+12V⊤U)−1V⊤X,R_X Cayley(-η)=X-U (I_2r+ 12V U )^-1V X, where concatenated matrices U:=[(Id−12XX⊤)η,X]∈ℝd×(2r)U:=[(I_d- 12X )η,X] ^d×(2r) and V:=[X,−(Id−12XX⊤)η]∈ℝd×(2r)V:=[X,-(I_d- 12X )η] ^d×(2r). This needs to invert a (2r)(2r)-by-(2r)(2r) matrix and the total computational flops are 4dr2+403r34dr^2+ 403r^3 [31], which could be calculated fast for small r. I Retraction-free and Penalty Parameter-free Optimization In this section, we focus on the design of retraction-free and penalty parameter-free algorithms for solving problem (1). We will first present the retraction-free algorithm and then show how the penalty parameter can be explicitly determined by characterizing the landscape of the penalty function. I-A Proximal Smoothness The concept of proximal smoothness, as introduced by [15], refers to the characteristic of a closed set whereby the nearest-point projection becomes a singleton when the point is close enough to the set. This property facilitates algorithmic and theoretical advancements by endowing nonconvex sets with convex-like structures. Specifically, for any positive real number γ, we define the γ-tube around ℳM as Uℳ(γ):=X:dist(X,ℳ)<γU_M(γ):=\X: dist(X,M)<γ\. We say a closed set ℳM is γ-proximally smooth if the projection operator ℳ(X):=argminY∈ℳ‖Y−X‖2P_M(X):= *argmin_Y \|Y-X\|^2 is a singleton whenever X∈Uℳ(γ)X∈ U_M(γ). Indeed, as stated in [15, Corollary 4.6], a closed set ℳM is convex if and only if it is γ-proximally smooth for arbitrary γ∈(0,∞)γ∈(0,∞). The Stiefel manifold ℳ=St(d,r)M= St(d,r) of interest is 1-proximally smooth [5]. On the other hand, it is well known that for any closed convex set ℳ⊂ℝd×rM ^d× r, the projection operator ℳP_M is 1-Lipschitz continuous over ℝd×rR^d× r. Similarly, following the proof in [15, Theorem 4.8], we obtain the corresponding result for the Stiefel manifold: for any X,Y∈U¯St(d,r)(12)X,Y∈ U_ St(d,r)( 12), ‖St(d,r)(X)−St(d,r)(Y)‖≤2‖X−Y‖. \|P_ St(d,r)(X)-P_ St(d,r)(Y) \|≤ 2\|X-Y\|. The above properties ensure the Stiefel manifold locally behaves like a convex set, serving as a useful auxiliary result for our subsequent analysis. I-B Retraction-free Algorithms Inspired by retraction-free algorithms [55, 1, 2], we consider the following retraction-free gradient descent method for problem (1): Xk+1=Xk−αkTXkSt(d,r)(gk)−μXk(Xk⊤Xk−I),X_k+1=X_k- _kP_T_X_k St(d,r)(g_k)-μ X_k(X_k X_k-I), (3) where αk,μ>0 _k,μ>0 are step sizes, gkg_k is a stochastic estimate of ∇f(Xk)∇ f(X_k), e.g., mini-batch stochastic gradient, and the mapping TXkSt(d,r)(g):=g−Xksym(Xk⊤g).P_T_X_k St(d,r)(g):=g-X_k sym(X_k g). In contrast to retraction-based algorithms, XkX_k does not remain on St(d,r) St(d,r) at each iteration. Note that when Xk∈St(d,r)X_k∈ St(d,r), TXkSt(d,r)P_T_X_k St(d,r) becomes the projection operator onto the tangent space. Thus, for Xk∈St(d,r)X_k∈ St(d,r) and gk=∇f(Xk)g_k=∇ f(X_k), it holds TXkSt(d,r)(gk)=gradf(Xk)P_T_X_k St(d,r)(g_k)= gradf(X_k). Moreover, the term Xk(Xk⊤Xk−I)=∇φ(Xk)X_k(X_k X_k-I)=∇ (X_k) in (3) is exactly the gradient of the following quadratic penalty function φ(X):=14‖X⊤X−I‖2. (X):= 14\|X X-I\|^2. We present the detailed description of retraction-free algorithm in Algorithm 1. (a) SQuADv2.0 Train Loss (b) SQuADv2.0 Eval Exact Match (c) SQuADv2.0 Eval F1 Figure 1: The figures compare the training loss, evaluation exact match, and evaluation F1 metrics against epochs for the SQuADv2.0 dataset. It can be clearly seen that our proposed Manifold-LoRA method almost achieves a 2x speed-up in training epochs compared to the vanilla LoRA. Input: Initial point X0X_0, αk,μ>0 _k,μ>0, and k=0k=0. while Stopping conditions not met do Compute the stochastic gradient gkg_k of ∇f(Xk)∇ f(X_k). Compute Xk+1←Xk−αkTXkSt(d,r)(gk)−μXk(Xk⊤Xk−I).X_k+1← X_k- _kP_T_X_k St(d,r)(g_k)-μ X_k(X_k X_k-I). Update k←k+1k← k+1. end while Algorithm 1 Retraction-free algorithm for solving (1) As will be shown in our theorem, the negative penalty gradient −∇φ(Xk)-∇ (X_k) pulls the iterate Xk+1X_k+1 back to the manifold, while the use of the projected stochastic gradient TXkSt(d,r)(gk)P_T_X_k St(d,r)(g_k) is crucial for ensuring its asymptotic orthogonality with ∇φ(Xk)∇ (X_k), resulting in landing on the manifold and convergence to a stationary point simutaneously. This differs from the usual penalty method, which optimizes f(X)+μφ(X)f(X)+μ (X) using the update Xk+1=Xk−αkgk−μXk(Xk⊤Xk−I)X_k+1=X_k- _kg_k-μ X_k(X_k X_k-I), and requires μ→∞μ→∞ to guarantee the feasibility. A key distinction from existing works [55, 1, 2] is that our approach allows for a constant step size μ in the penalty term while ensuring convergence in both deterministic and stochastic settings. Notably, we set μ=1/3μ=1/3 by requiring X1X_1 is not far away from St(d,r) St(d,r). It is worth highlighting that similar explicit choices for constraint violation have been explored in decentralized optimization [39, 19], where a fixed step size—typically set to 1—is commonly employed for enforcing consensus constraints. The theoretical foundation supporting our approach relies on the restricted strong convexity of the penalty function φ and the proximal smoothness of the Stiefel manifold, which together facilitate convergence guarantees in Section IV. Compared with the popularly used Cayley transformation-based retraction-type algorithms, the computational cost therein is 4dr2+403r34dr^2+ 403r^3, which is more than twice the cost of our method at 2dr22dr^2 for any r. Moreover, retractions on the Stiefel manifold involve complex orthogonalization procedures, such as matrix inversion in the Cayley transformation, which are difficult to scale and parallelize. In contrast, the landing update (3) can be executed using scalable BLAS3 operations. I-C Manifold-LoRA The retraction-free method is well-suited to address (2), simultaneously minimizing the loss function ℒ(BA)L(BA) and constraint violation of B. To control the constraint violation, we use the quadratic penalties RSt(B):=‖B⊤B−I‖2R_St(B):=\|B B-I\|^2 and ROb(B):=‖diag(B⊤B)−‖2R_Ob(B):=\| diag(B B)-1\|^2 for the Stiefel manifold and Oblique manifold, respectively. By the retraction-free method, Algorithm 1, we use the projected gradient of the loss part instead of the Euclidean gradient. For simplicity, we write ∇Bℒ=∇Bℒ(BA) _BL= _BL(BA), and similarly ∇Aℒ _AL. For B∈St(d,r)B (d,r) or B∈Ob(d,r)B (d,r), the respective projected gradients are TBSt(d,r)(∇Bℒ)=∇Bℒ−Bsym(B⊤∇Bℒ)P_T_B St(d,r)( _BL)= _BL-B sym(B _BL) (4) and TBOb(d,r)(∇Bℒ)=∇Bℒ−Bddiag(B⊤∇Bℒ)P_T_B Ob(d,r)( _BL)= _BL-B ddiag(B _BL) (5) where ddiag(Z) ddiag(Z) denotes Z with all off-diagonal entries set to 0. Thus, the gradients of our retraction-free method for A and B are ∇Aℒ _AL and TBSt(d,r)(∇Bℒ)+μ∇RSt(B)(orTBOb(d,r)(∇Bℒ)+μ∇ROb(B))P_T_B St(d,r)( _BL)+μ∇ R_St(B)( ~or~P_T_B Ob(d,r)( _BL)+μ∇ R_Ob(B)). Note that B and A represent the basis and the coordinate of ΔW W, respectively. This results in different magnitudes and different Lipschitz constants of their gradient function. In fact, let X=BAX=BA. It follows ∇Aℒ(BA)=B⊤∇Xℒ(X),∇Bℒ(BA)=∇Xℒ(X)A⊤. _AL(BA)=B _XL(X), _BL(BA)= _XL(X)A . Then, it holds that for any A1,A2∈ℝr×m,B1,B2∈ℝd×rA_1,A_2 ^r× m,B_1,B_2 ^d× r, ‖∇Aℒ(BA1)−∇Aℒ(BA2)‖ \| _AL(BA_1)- _AL(BA_2)\| ≤‖B‖2Lg‖A1−A2‖, ≤\|B\|_2L_g\|A_1-A_2\|, ‖∇Bℒ(B1A)−∇Bℒ(B2A)‖ \| _BL(B_1A)- _BL(B_2A)\| ≤‖A‖2Lg‖B1−B2‖, ≤\|A\|_2L_g\|B_1-B_2\|, where LgL_g is the Lipschitz constant of ∇Xℒ(X) _XL(X) and ∥⋅∥2\|·\|_2 represents the matrix ℓ2 _2 norm (i.e., the largest singular value). Note that the step size should generally be proportional to the reciprocal of Lipschitz constant for gradient-type algorithms [40, 8]. Hence, we schedule the learning rates for the two matrices based on their respective ℓ2 _2 norms. Having prepared the above, we incorporate the SGD or AdamW optimizer [38] with our manifold-accelerated technique to enhance the LoRA fine-tuning, as presented in Algorithm 2, which can be seen as a generalization of Algorithm 1 to solve optimization problems over the product of Stiefel or Oblique manifold and Euclidean space. Note that the Oblique manifold Ob(d,r) Ob(d,r) is the product of r Stiefel manifolds St(d,1) St(d,1), which is the sphere in ℝdR^d. (a) Loss curves on CoLA dataset. (b) Loss curves on QQP dataset. (c) Loss curves on STSB dataset. Figure 2: The figures illustrate that both sphere constrained and Stiefel constrained manifold-LoRA achieve a faster convergence rate and attain a lower training loss within same optimization steps compared to LoRA method on three distinct datasets CoLA, QQP, STS-B. Input: Initial A0,B0A_0,B_0, choice update_type ∈SGD,AdamW∈\ SGD,\, AdamW\, αk,μ>0 _k,μ>0, β1=0.9 _1=0.9, β2=0.999 _2=0.999, ub≥lb>0 ub≥ lb>0, ϵ=10−8ε=10^-8, λ>0λ>0, m(C−1)=v(C−1)=0m(C_-1)=v(C_-1)=0, k=1k=1. while Stopping conditions not met do Compute the stochastic gradients g(Ak)g(A_k) and g(Bk)g(B_k) of ∇Aℒ(BkAk) _AL(B_kA_k) and ∇Bℒ(BkAk) _BL(B_kA_k), respectively. Let g^(Ak)←g(Ak) g(A_k)← g(A_k). Let g^(Bk)←TBkSt(d,r)(g(Bk)) g(B_k) _T_B_kSt(d,r) (g(B_k) ) as in (4), or g^(Bk)←TBkOb(d,r)(g(Bk)) g(B_k) _T_B_kOb(d,r) (g(B_k) ) as in (5). for C∈A,BC∈\A,B\ do /* Projected grad update for A, B */ if update_type == SGD then Ck+1←Ck−αkg^(Ck)C_k+1← C_k- _k\, g(C_k) end if if update_type == AdamW then m(Ck)←β1m(Ck−1)+(1−β1)g^(Ck)m(C_k)← _1\,m(C_k-1)+(1- _1)\, g(C_k) v(Ck)←β2v(Ck−1)+(1−β2)g^(Ck)⊙g^(Ck)v(C_k)← _2\,v(C_k-1)+(1- _2)\, g(C_k) g(C_k) m^(Ck)←m(Ck)1−β1k m(C_k)← m(C_k)1- _1^k v^(Ck)←v(Ck)1−β2k v(C_k)← v(C_k)1- _2^k α(Ck)←clip(‖Ck‖2,ub,lb)α(C_k) (\|C_k\|_2,ub,lb) Ck+1←Ck−α(Ck)m^(Ck)v^(Ck)+ϵ−λCkC_k+1← C_k-α(C_k)\, m(C_k) v(C_k)+ε-λ C_k end if /* Penalty grad update for B */ if C=BC=B then Ck+1←Ck−μ∇RSt(Ck)(or∇ROb(Ck))C_k+1← C_k-μ∇ R_St(C_k)( or~∇ R_Ob(C_k)) end if end for Update k←k+1k← k+1. end while Algorithm 2 Manifold-LoRA for solving (2) IV Convergence Analysis In this section, we analyze the convergence of our retraction-free gradient descent method (3). We first justify the choice of μ=13μ= 13 under an appropriate initialization. Subsequently, we establish convergence results for (3) in both deterministic and stochastic settings. IV-A Explicit Choice for the Penalty Parameter It is known that a large penalty parameter μ yields better feasibility [40, Chapter 17]. To make the iterative scheme (3) be penalty parameter-free, we need a careful investigation on the landscape of the following optimization problem: minX∈ℝd×rφ(X)=14‖X⊤X−I‖2. _X ^d× r\;\; (X)= 14 \|X X-I \|^2. (6) It can be easily verified that problem (6) is nonconvex and its optimal solution set is St(d,r) St(d,r). The key of obtaining an explicit formula of μ is to establish certain strong convexity-type inequality and show that the gradient descent method with step size μ has linear convergence. For any X∈ℝd×rX ^d× r, let us denote X¯:=St(d,r)(X) X:=P_ St(d,r)(X). Let X=USV⊤X=USV be the singular value decomposition with orthogonal matrices U∈ℝd×r,V∈ℝr×rU ^d× r,V ^r× r and diagonal matrix S∈ℝr×rS ^r× r, then X¯=UV⊤ X=UV . Building on these notations, we demonstrate that problem (6) satisfies the restricted secant inequality (RSI) [57], which serves as an alternative to the strong convexity in the linear convergence analysis of gradient-type methods. Lemma 1 (RSI). For any X∈ℝd×rX ^d× r with ‖X−X¯‖≤18\|X- X\|≤ 18, we have ⟨∇φ(X),X−X¯⟩≥‖X−X¯‖2. ∇ (X),X- X ≥\|X- X\|^2. With the given RSI, applying the gradient descent (GD) update to (6), i.e., Xk+1pen=Xkpen−μ∇φ(Xkpen),X pen_k+1=X pen_k-μ∇ (X pen_k), (7) yields the following linear convergence result. Lemma 2 (Linear convergence of GD for (6)). Let the sequence Xkpen\X_k pen\ be generated by (7) with μ=13μ= 13. Suppose that ‖X1pen−X1pen¯‖≤18\|X pen_1- X pen_1\|≤ 18, then we have ‖Xk+1pen−Xk+1pen¯‖2≤23‖Xk−Xkpen¯‖2.\|X_k+1 pen- X_k+1 pen\|^2≤ 23\|X_k- X_k pen\|^2. Note that the linear convergence of gradient descent follows from the established RSI. It would also be interesting to investigate Newton–Schulz-type updates for nearly orthogonal matrices and to exploit their superlinear local convergence [26, 34] in the design and analysis of retraction-free algorithms for problem (1). The proofs of Lemmas 1 and 2 are provided in Appendix A. IV-B Landing on the Stiefel Manifold Building on the established linear convergence of gradient descent for problem (6), we are now able to show that the iterates generated by (3) will land on the Stiefel manifold eventually, and the limiting point is a stationary point of (1), i.e., X∞∈St(d,r)X_∞∈ St(d,r), and gradf(X∞)=0 gradf(X_∞)=0. Assumption 1. Suppose the following smoothness and stochasticity conditions hold: • For each i, the component function fif_i is continuously differentiable, and its Euclidean gradient ∇fi∇ f_i is LfL_f-Lipschitz continuous over the convex hull of U¯St(d,r)(18) U_St(d,r) ( 18 ). • The stochastic gradient gkg_k is an unbiased estimator of ∇f(Xk)∇ f(X_k) with uniformly bounded variance, namely, for all k=1,2,…k=1,2,…, [gk]=∇f(Xk),[‖gk−∇f(Xk)‖2]≤σ2,E[\,g_k\,]\;=\;∇ f(X_k), [\|\,g_k-∇ f(X_k)\|^2 ]\;≤\;σ^2, where σ>0σ>0 is a constant. For simplicity, define ∇^fi(X)=TXSt(d,r)(∇fi(X)) ∇f_i(X)=P_T_X St(d,r)(∇ f_i(X)). Note that ∇^fi(X)=gradfi(X) ∇f_i(X)= gradf_i(X) whenever X∈St(d,r)X∈ St(d,r). We first have the following quadratic upper bound on fif_i and Lipschitz continuity of gradfi gradf_i. Lemma 3 (Quadratic upper bound). Suppose that Assumption 1 holds. There exists a constant L>0L>0 such that for any X,Y∈St(d,r)X,Y∈ St(d,r), and any i, the following quadratic upper bound holds: fi(Y)≤fi(X)+⟨gradfi(X),Y−X⟩+L2‖Y−X‖2.f_i(Y)≤ f_i(X)+ gradf_i(X),Y-X + L2\|Y-X\|^2. (8) In addition, there exists a constant L^>0 L>0 such that for any X∈St(d,r),Y∈U¯St(d,r)(18)X∈ St(d,r),Y∈ U_ St(d,r)( 18), and any i, ‖gradfi(X)−∇^fi(Y)‖≤L^‖X−Y‖.\| gradf_i(X)- ∇f_i(Y)\|≤ L\|X-Y\|. (9) (a) CoLA evaluation matthews correlation (b) QQP evaluation accuracy (c) STS-B evaluation pearson Figure 3: Performance on the validation sets across three datasets. The COLA dataset is evaluated using the matthews correlation metric, QQP is measured by accuracy, and STS-B is evaluated by Pearson correlation, all plotted against the number of epochs. By the linear convergence result in Lemma 2, we have the following decay on the feasibility error. Lemma 4 (Error bound of feasibility). Let Xk\X_k\ be the sequence generated by (3) with μ=13μ= 13. If ‖Xk−X¯k‖≤18\|X_k- X_k\|≤ 18, then ‖Xk+1−X¯k+1‖≤23‖Xk−X¯k‖+52αk‖gk‖.\|X_k+1- X_k+1\|≤ 23\|X_k- X_k\|+ 52 _k\|g_k\|. (10) The following one-step descent lemma on f is crucial in establishing the convergence. Detailed proofs of Lemmas 3, 4, and 5 are provided in Appendix A. Lemma 5 (One-step descent for f). Suppose that Assumption 1 holds. Let Xk\X_k\ be the sequence generated by Algorithm 1 with μ=13μ= 13. If ‖Xk−X¯k‖≤18\|X_k- X_k\|≤ 18, we have k[f(X¯k+1)]−f(X¯k) _k[f( X_k+1)]-f( X_k) (11) ≤ ≤ −(αk−(4L^2+4L+1)αk2)‖∇^f(Xk)‖2+ -( _k-(4 L^2+4L+1) _k^2)\| ∇f(X_k)\|^2+ 12(4D^f+9L^2+8L+3)‖Xk−X¯k‖2+ 12 (4 D_f+9 L^2+8L+3 )\|X_k- X_k\|^2+ 12‖Xk+1−X¯k+1‖2+7(4L^2+4L+1)αk2σ2, 12\|X_k+1- X_k+1\|^2+7(4 L^2+4L+1) _k^2σ^2, where D^f:=maxX∈U¯St(d,r)(18)‖∇f(X)‖ D_f:= _X∈ U_ St(d,r)( 18)\|∇ f(X)\| and we use k[⋅]E_k[·] to denote an expected value taken with respect to the distribution of the random variable in the estimation of gkg_k given XkX^k. From the above lemma, the one-step decrease on f is related to both the gradient norm of f and the feasibility error. Regarding convergence, we need both gradf(Xk) gradf(X_k) and ‖Xk⊤Xk−I‖\|X_k X_k-I\| converge to 0. The following theorem shows that the retraction-free and penalty parameter-free update (3) converges. Theorem 1 (Convergence of Algorithm 1). Suppose that Assumption 1 holds. Let Xk\X_k\ be the sequence generated by Algorithm 1 with μ=13μ= 13 and ‖X1−X¯1‖≤18.\|X_1- X_1\|\;≤\; 18. Denote the total expectation by [⋅]:=12⋯k[⋅].E[·]:=E_1\,E_2·sE_k[·]. Then the following statements hold: • (Deterministic case) If the full gradient is used (i.e. σ=0σ=0 in Assumption 1) and the step size is kept constant as αk≡α∈(0,12c1] _k≡α∈ (0, 12c_1 ], for some sufficiently large c1>0c_1>0, then mink≤K[‖∇^f(Xk)‖2+‖Xk⊤Xk−I‖2]=(1K). _k≤ K\;\;E [\| ∇f(X_k)\|^2+\|X_k X_k-I\|^2 ]=O ( 1K ). • (Stochastic case) If the step size decays as αk=α0k, _k= _0 k, with α0∈(0,12c1] _0∈ (0, 12c_1 ], for some sufficiently large c1>0c_1>0, then mink≤K[‖∇^f(Xk)‖2+‖Xk⊤Xk−I‖2]=(logK) _k≤ K\;\;E [\| ∇f(X_k)\|^2+\|X_k X_k-I\|^2 ]=O ( K K ) and mink≤K‖Xk⊤Xk−I‖2=(logK). _k≤ K\|X_k X_k-I\|^2=O( KK). The proof of Theorem 1 is presented in Appendix A. Remark 1. Compared with the landing algorithm [1], which targets only the squared Stiefel manifold and requires tuning both parameters α and μ, our approach handles general Stiefel manifolds and necessitates the tuning of only α, as established in Theorem 1. In addition, the landing algorithm in [1] converges only to a neighborhood whose size depends on the step size, as discussed in the paragraph following Proposition 10 of their paper. Moreover, our iteration complexity of (1/K)O(1/K) is on par with retraction-based algorithms [9]. Remark 2. Our penalty-parameter-free analysis establishes an improved convergence rate of (logK)O ( KK ) for constraint violation in the stochastic setting with decaying step sizes, improving upon the (logK)O ( K K ) rate reported in [2]. This improvement stems from our two-scale step-size scheme—decaying for the loss-gradient step while remaining constant for the penalty-gradient step—whereas [2] employs a single-scale decaying step size for both steps, despite allowing an arbitrary but finite penalty parameter. Furthermore, the proof of Theorem 1 relies on the linear decay of the constraint violation and the descent property of f, in contrast to the augmented Lagrangian-based framework of [2]. We also remark that our penalty-parameter-free algorithm design and analysis are motivated by insights from multi-agent decentralized optimization [39, 50, 45, 13, 19]: the stepsize for the consensus (constraint violation) term—which directly measures and penalizes the discrepancy across agents—is typically fixed to 11, while only the stepsizes for the local loss-gradient updates require tuning. In this literature, the consensus error (i.e., the constraint violation) is also known to converge faster than the loss gradient in stochastic settings with decaying step sizes. Related two-time-scale ideas have also been explored in minimax optimization; see, e.g., [37, 36]. Now, since Algorithm 2 with SGD-type update can be seen as a generalization of Algorithm 1 to solve problems over the products of the Stiefel manifold or the Oblique manifold and the Euclidean space, we immediately have the following convergence result. The proof of Corollary 1 is given in Appendix A. Corollary 1 (Convergence of Algorithm 2). Suppose that the gradients ∇Aℒ(BA) _AL(BA) and ∇Bℒ(BA) _BL(BA) are Lipschitz continuous, and that g^(Ak) g(A_k) and g^(Bk) g(B_k) are unbiased, bounded-variance estimators of ∇Aℒ(BkAk) _AL(B_kA_k) and ∇Bℒ(BkAk) _BL(B_kA_k), respectively. Let (Ak,Bk)\(A_k,B_k)\ be the sequence generated by Algorithm 2 with μ=13μ= 13 and ‖B0−B¯0‖≤1/8\|B_0- B_0\|≤ 1/8, using an SGD-type update. Then, if the step size is chosen as αk=α0k _k= _0 k with a small α0>0 _0>0, we have mink≤K _k≤ K [∥∇Aℒ(BkAk)∥2+∥∇^Bℒ(BkAk)∥2 [\| _AL(B_kA_k)\|^2+\| ∇_BL(B_kA_k)\|^2 +α0k∥Bk⊤Bk−I∥2]≤(logK). + _0 k\|B_k B_k-I\|^2 ] ( K K ). The proof of the above corollary is based on Theorem 1 and the geometry of the product manifolds. The original Adam method may fail to converge [49]; nevertheless, Euclidean Adam-type corrections can yield convergence guarantees for Algorithm 2 with Adam-type updates, e.g., AdaShift [59]; see Appendix B. V Experiments In this section, we present comprehensive experimental results to evaluate the performance of Manifold-LoRA, i.e., Algorithm 2 with AdamW update, across various tasks, including natural language understanding (NLU), question answering (QA), and natural language generation (NLG). We highlight the method’s advantages in terms of convergence speed, downstream performance, and memory efficiency. All experiments follow a consistent setup to ensure fair comparison. V-A Baselines and Implementation Details We compare Manifold-LoRA with several parameter-efficient fine-tuning (PEFT) baselines, including full fine-tuning, Adapter [27], BitFit [56], and LoRA [28]. Variants of the Adapter method are omitted due to similar performance trends. Note that, although retraction or projection onto the oblique manifold is inexpensive, we still adopt the retraction-free update to validate our theoretical predictions, in particular the effectiveness of the landing property. Our implementation is based on PyTorch [43], Huggingface Transformers [54], and OpenDelta [30]. For a fair comparison, we ensure that all methods (including Adapter and LoRA) have approximately the same number of trainable parameters by aligning the bottleneck dimensions (e.g., 16 or 32). LoRA updates are scaled by a fixed hyperparameter α (typically 16 or 32, as in [28]), and AdamW [38] is used as the optimizer with default exponential moving average parameters β1=0.9 _1=0.9, β2=0.999 _2=0.999. All experiments are conducted on NVIDIA A800 GPUs. 222The hyperparameters used for the GLUE benchmark, question-answering tasks, and E2E benchmark are reported in Tables V, IV, and I, respectively. To make a fair comparison, all hyperparameters such as batch size and learning rate scheduler, remain the same across experiments, except the additional parameters introduced by the Manifold-LoRA. V-B Natural Language Understanding We evaluate Manifold-LoRA using the DeBERTaV3-base model [25] on the GLUE benchmark [52], which contains nine subdatasets including MNLI, SST-2, CoLA, QQP, QNLI, RTE, MRPC, and STS-B. For all GLUE tasks, we employ a consistent training configuration: a warmup ratio of 0.06, linear learning rate scheduling, a maximum sequence length of 256, weight decay set to 0.1, and a batch size of 32. The LoRA modules are applied to the query and value projection matrices (WqW_q and WvW_v). Performance Comparison Table VI presents the GLUE results. Manifold-LoRA consistently outperforms LoRA and other baselines across most tasks. In particular, on the RTE and STS-B datasets, both sphere-constrained and Stiefel-constrained variants of Manifold-LoRA with rank r=8r=8 outperform LoRA with r=16r=16, indicating superior memory efficiency under equal memory budgets. Overall, the proposed algorithm achieves the best average performance across all methods, with the Oblique-constrained variant under rank r=16r=16 attaining the highest average score of 88.63%. Notably, even with a reduced rank of r=8r=8, both the Stiefel- and Oblique-constrained Manifold-LoRA variants outperform the LoRA baseline with r=16r=16, demonstrating superior parameter efficiency. Convergence speed To further assess the convergence speed of our proposed algorithm, we compare the training loss trajectories throughout the optimization process. As shown in Figure 2, Manifold-LoRA reaches the same training loss as the standard Adam optimizer in nearly half the number of epochs. In particular, on the CoLA dataset (Figure 2(a)), it exhibits almost 2× faster convergence. We also track the evolution of validation metrics during training. As illustrated in Figure 3, Manifold-LoRA consistently outperforms vanilla LoRA. Notably, on the STS-B dataset (Figure 3(c)), our method achieves a significantly larger performance margin. For the CoLA and QQP datasets, Manifold-LoRA shows slight improvements over LoRA, demonstrating steady performance gains across tasks. Stability across random seeds Results are averaged over five random seeds, with shaded areas indicating variance. As shown in Figure 3, Manifold-LoRA exhibits smaller variance compared to LoRA, confirming its stability. The focus of this work is to use manifold geometry to accelerate LoRA fine-tuning. A systematic study of how such geometric constraints affect generalization is beyond the scope of this paper and is left for future work. V-C Question Answering We further fine-tune DeBERTaV3-base on SQuAD v1.1 [48] and SQuADv2.0 [47] using Manifold-LoRA. The evaluation primarily focuses on F1 score and exact match accuracy to assess both the completeness and preciseness of predicted answers. For both SQuADv1.1 and SQuADv2.0, we adopt a consistent training setup: a warmup ratio of 0.06, a linear learning rate schedule, a weight decay of 0.1, and a batch size of 64. The learning rate is set to 3e-3, and training is conducted for 4 epochs. Following the LoRA framework, all low-rank modules are inserted into WqW_q, WkW_k, WvW_v, WoW_o, FC1FC_1, and FC2FC_2. Performance comparison Table VII summarizes experimental results when we fine-tune DeBERTaV3-base under different rank settings. For Adapter, we set the rank to 16 and 32, and for LoRA and our method, the rank is set to 8 and 16. It can be observed that Manifold-LoRA achieves superior performance with fewer trainable parameters. For instance, with the Stiefel constraint, Manifold-LoRA achieves an F1 score of 89.22 and an Exact Match score of 86.41 on the SQuADv2.0 dataset, significantly surpassing all other baselines. Training efficiency and manifold consistency We also evaluate training loss, validation Exact Match, and validation F1 score across training epochs on the SQuADv2.0 dataset, as shown in Figure 1. Across the three metrics, Manifold-LoRA exhibits a significantly faster convergence speed, nearly twice that of LoRA, demonstrating a faster loss reduction and a faster improvement in evaluation performance. Additionally, we observe that the models with Stiefel and Oblique constraints follow similar trends in both the loss and evaluation metrics, suggesting that both geometric constraints can achieve comparable effectiveness on question answering tasks. To further verify whether the learned low-rank matrix B effectively lies on the target manifold, we visualize its structure by plotting the heatmaps of B⊤B B in Figure 4. Specifically, we extract the checkpoint from the DeBERTa-base model after the second training epoch on the SQuADv2.0 dataset, and select several representative layers from the second and third transformer blocks for analysis. The resulting heatmaps reveal that B⊤B B closely adheres to the expected manifold geometry, validating the effectiveness of our manifold-constrained design. V-D Natural Language Generation Having demonstrated superior performance on natural language understanding and question answering tasks, we further extend our experiments to evaluate the effectiveness of our proposed method on a natural language generation task. Specifically, we conduct experiments on the E2E NLG Challenge dataset [41], using GPT-2 Medium and GPT-2 Large as backbone models. The E2E dataset comprises approximately 50K examples with 8 distinct semantic fields. It provides multiple reference outputs for each input table, with an average output length of 22.9 tokens. For both GPT-2 Medium and Large models on the E2E benchmark, we use a linear learning rate schedule with 500 warmup steps, a weight decay of 0.01, and no LoRA dropout. The models are trained for 5 epochs with a batch size of 8 and a learning rate of 2e-4. Performance comparison To ensure a fair comparison, we follow the same hyperparameter settings as in the original LoRA paper, except for the additional parameters introduced by Manifold-LoRA. The numerical results are summarized in Table VIII. It is evident that Manifold-LoRA achieves superior performance across all five metrics with limited trainable parameters. V-E Scaling Experiments Experimental Settings We largely follow the experimental protocol of prior work on LoRA-based fine-tuning with manifold-constrained optimization to ensure a fair and consistent comparison [42]. In particular, we adopt the same model architectures (LLaMA3.2-1B/3B and LLaMA3-8B [21]), downstream benchmarks (SQuAD, QuAC, GSM8K [16], and MATH), and evaluation metrics as in the reference study. For all experiments, LoRA is applied to the same set of layers with identical rank and scaling configurations, and models are fine-tuned under the same training budget. Reading Comprehension Table I reports results on SQuAD and QuAC when scaling our method from 1B to 8B models, where the method “Stiefel” is from [42]. Across all model sizes, Manifold-LoRA consistently outperforms other baselines on both datasets. Notably, the performance advantage on SQuAD and QuAC is maintained as the model size increases, indicating that our optimization approach scales well for extractive reading comprehension. On QuAC, which emphasizes semantic understanding and contextual coherence across conversational turns, the oblique manifold-constrained optimization demonstrates more consistent gains than Stiefel manifold-based alternatives. Mathematical Reasoning Results on GSM8K and MATH are summarized in Table I. Stiefel-based Manifold-LoRA yields consistent improvements over AdamW across all model scales, with particularly strong gains on the more challenging MATH benchmark. The improvements become more pronounced for larger models, especially LLaMA3-8B, indicating that Stiefel manifold-constrained optimization is effective at preserving and enhancing complex mathematical reasoning capabilities as model capacity increases. TABLE I: Reading comprehension results on SQuAD and QuAC (F1/EM). Model Method Opt. SQuAD QuAC LLaMA3.2-1B LoRA Stiefel 67.9/55.7 50.4 AdamW 64.1/51.5 45.9 Manifold-LoRA (Stiefel) 69.4/56.9 51.8 Manifold-LoRA (Sphere) 70.6/57.4 52.0 LLaMA3.2-3B LoRA Stiefel 80.3/72.1 61.8 AdamW 78.6/67.4 57.5 Manifold-LoRA (Stiefel) 81.7/72.8 62.3 Manifold-LoRA (Sphere) 80.0/71.9 61.5 LLaMA3-8B LoRA Stiefel 88.1/79.7 69.7 AdamW 84.3/74.6 65.8 Manifold-LoRA (Stiefel) 89.8/81.2 70.8 Manifold-LoRA (Sphere) 90.4/82.0 71.3 TABLE I: Mathematical reasoning accuracy on GSM8K and MATH. Model Method Opt. GSM8K MATH LLaMA3.2-1B LoRA Stiefel 35.4 26.5 AdamW 20.5 21.4 Manifold-LoRA (Stiefel) 37.2 27.9 Manifold-LoRA (Sphere) 34.9 26.3 LLaMA3.2-3B LoRA Stiefel 43.4 33.5 AdamW 29.1 27.7 Manifold-LoRA (Stiefel) 46.5 35.7 Manifold-LoRA (Sphere) 44.5 34.8 LLaMA3-8B LoRA Stiefel 58.8 22.5 AdamW 54.7 19.3 Manifold-LoRA (Stiefel) 60.5 23.8 Manifold-LoRA (Sphere) 57.6 21.9 TABLE I: Manifold-specific hyperparameters of Manifold-LoRA for the E2E benchmark. Method Hyperparameter GPT-2(M) GPT-2(L) Sphere μ 1 0.9 (r=4r=4) Lower 0.5 0.5 Upper 2 2 Stiefel μ 1 1.1 (r=4r=4) Lower 0.5 0.5 Upper 4 2 TABLE IV: Manifold-specific hyperparameters of Manifold-LoRA for question answering tasks. Method Hyperparameter SQuADv1.1 SQuADv2.0 Sphere (r=8r=8) μ 0.85 0.85 Lower 0.25 0.25 Upper 0.75 0.5 Sphere (r=16r=16) μ 0.9 0.85 Lower 0.25 0.25 Upper 0.5 0.5 Stiefel (r=8r=8) μ 0.85 0.85 Lower 0.25 0.25 Upper 0.5 0.5 Stiefel (r=16r=16) μ 0.9 0.85 Lower 0.25 0.25 Upper 0.5 0.5 TABLE V: Hyperparameter configurations of Manifold-LoRA for GLUE benchmark. Method Hyperparameter MNLI SST-2 CoLA QQP QNLI RTE MRPC STS-B Baseline Epochs 7 24 25 5 5 50 30 25 Learning Rate 5e-4 8e-4 5e-4 5e-4 1.2e-3 1.2e-3 1e-3 2.2e-3 Sphere (r=16r=16) μ 1 0.9 0.8 0.9 0.95 1.2 0.85 0.9 Lower 0.25 0.25 0.5 0.5 0.5 0.5 1 1 Upper 2 2 2 4 2 2 4 4 Sphere (r=8r=8) μ 0.95 0.95 1 0.9 1 0.9 0.85 1 Lower 2 0.5 1 0.5 0.5 0.25 2 1 Upper 8 2 8 2 2 0.5 4 8 Stiefel (r=16r=16) μ 0.8 0.85 0.95 0.9 0.95 1.2 0.8 1 Lower 2 0.5 2 0.5 0.5 0.5 1 1 Upper 8 1 8 4 1 2 4 16 Stiefel (r=8r=8) μ 0.8 0.95 0.95 0.9 0.85 0.9 1 1 Lower 2 0.5 2 0.5 0.5 0.25 1 1 Upper 8 2 8 2 2 1 4 16 TABLE VI: We present results using DeBERTaV3-base on the GLUE benchmark. For MNLI, we report the accuracy (combining matched and mismatched sets), with the left panel representing matched subset and the right panel representing mismatched subset. For CoLA, we report Matthew’s correlation, and for STS-B, we report Pearson correlation. For all other tasks, we report accuracy. All metrics are same as the original LoRA paper [28]. Higher values are better for all metrics. The best results are highlighted in bold. Method # Params MNLI SST-2 CoLA QQP QNLI RTE MRPC STS-B All Acc Acc Mcc Acc/ F1 Acc Acc Acc Corr Ave. Full FT 184.42M 90.4590.45/ 90.60 95.4895.48 68.1768.17 91.99/ 89.12 93.6093.60 79.2879.28 88.9388.93 90.9290.92 87.8587.85 Adapter 0.61M 90.1390.13/ 90.1690.16 94.8694.86 69.3769.37 91.3891.38/ 88.4688.46 93.5493.54 81.8781.87 89.1289.12 91.5291.52 88.0688.06 BitFit 0.06M 87.0887.08/ 86.3986.39 94.8894.88 69.1169.11 87.9687.96/ 84.3584.35 92.1992.19 76.5276.52 87.0687.06 90.9690.96 85.6585.65 LoRAr=8 0.30M 90.2090.20/ 90.0890.08 94.9394.93 68.1468.14 90.7890.78/ 87.6887.68 93.8593.85 80.1580.15 90.4090.40 90.2990.29 87.6087.60 LoRAr=16 0.59M 90.4490.44/ 90.1290.12 95.4195.41 68.1968.19 90.9290.92/ 87.7787.77 94.0094.00 80.5880.58 90.2090.20 90.3490.34 87.7487.74 Spherer=8 0.30M 90.3790.37/ 90.0990.09 95.4895.48 69.5569.55 91.2591.25/ 88.3488.34 94.0294.02 82.4482.44 91.5591.55 91.2691.26 88.4488.44 Spherer=16 0.59M 90.52/ 90.1990.19 95.6495.64 70.14 91.4691.46/ 88.6588.65 94.29 82.1682.16 91.67 91.59 88.63 Stiefelr=8 0.30M 90.2590.25/ 89.9989.99 95.4695.46 69.8569.85 91.4491.44/ 88.6088.60 94.0994.09 83.16 91.1891.18 91.2291.22 88.5288.52 Stiefelr=16 0.59M 90.2690.26/ 90.2890.28 95.76 68.9268.92 91.7191.71/ 89.0089.00 94.1094.10 82.1682.16 91.1091.10 91.5191.51 88.4888.48 TABLE VII: Results with DeBERTaV3-base on SQuAD v1.1 and SQuADv2.0. We report F1 and Exact Match (EM). The best results in each setting are shown in bold. Methods # Params Rank SQuADv1.1 SQuADv2.0 F1 Score Exact Match F1 Score Exact Match Full FT 184.42M – 92.8592.85 86.3086.30 87.5887.58 84.3084.30 Adapter 0.61M 16 93.4193.41 87.4687.46 88.2388.23 85.3085.30 Adapter 1.22M 32 93.5193.51 87.5387.53 88.3688.36 85.4285.42 Bitfit 0.07M – 88.7988.79 80.2680.26 87.1987.19 74.2174.21 LoRA 1.33M 8 93.8893.88 87.9087.90 88.5288.52 85.5685.56 LoRA 2.65M 16 93.7593.75 87.9487.94 88.8188.81 85.9085.90 Sphere 1.33M 8 94.25 88.5188.51 89.2089.20 86.3386.33 Sphere 2.65M 16 94.0394.03 88.3288.32 89.0389.03 86.1586.15 Stiefel 1.33M 8 94.2394.23 88.68 89.0989.09 86.3586.35 Stiefel 2.65M 16 94.0494.04 88.2588.25 89.22 86.41 Figure 4: The heat map of B⊤B B with the Stiefel manifold (the first and second rows) and the Oblique manifold (the third and fourth rows) at the end of training on SQuADv2.0 dataset. TABLE VIII: GPT-2 medium (M) and large (L) models evaluated on the E2E NLG Challenge. * denotes results from previously published works. Model Parameters BLEU NIST MET ROUGE-L CIDEr GPT-2 M (FT)* 354.92M 68.268.2 8.62 46.246.2 71.071.0 2.47 GPT-2 M (AdapterL)* 11.09M 68.968.9 8.71 46.146.1 71.371.3 2.47 GPT-2 M (AdapterH)* 11.09M 67.367.3 8.50 46.046.0 70.770.7 2.44 GPT-2 M (FTTop2)* 25.19M 68.168.1 8.59 46.046.0 70.870.8 2.41 GPT-2 M (PreLayer)* 0.35M 69.769.7 8.81 46.146.1 71.471.4 2.49 GPT-2 M (LoRA) 0.35M 68.968.9 8.69 46.546.5 71.571.5 2.51 GPT-2 M (Stiefel) 0.35M 70.170.1 8.82 46.8 71.7 2.53 GPT-2 M (Sphere) 0.35M 70.3 8.83 46.746.7 71.7 2.52 GPT-2 L (FT)* 774.03M 68.568.5 8.78 46.046.0 69.969.9 2.45 GPT-2 L (AdapterL)* 23.00M 68.968.9 8.70 46.146.1 71.371.3 2.45 GPT-2 L (PreLayer)* 0.77M 70.370.3 8.85 46.246.2 71.771.7 2.47 GPT-2 L (LoRA) 0.77M 70.170.1 8.82 46.746.7 72.072.0 2.53 GPT-2 L (Stiefel) 0.77M 70.470.4 8.86 46.8 72.172.1 2.53 GPT-2 L (Sphere) 0.77M 70.9 8.92 46.8 72.5 2.55 VI Conclusion Optimization over the Stiefel manifold has been widely used in machine learning tasks. In this work, we develop a retraction-free and penalty parameter-free gradient method, and prove that the generated iterates eventually land on the manifold and achieve the optimality simultaneously. Moreover, our convergence theory enables the use of a constant step size, improving on previous results that only ensured convergence to a neighborhood. We then apply this landing theory to avoid the possible redundancy of LoRA fine-tuning in LLMs. Specifically, we reformulate the LoRA fine-tuning as an optimization problem over the product of the Stiefel or Oblique manifold and Euclidean space, and propose a new algorithm, Manifold-LoRA, which incorporates a careful analysis of step sizes to enable fast training using the landing properties. Extensive experimental results demonstrate that our approach not only accelerates the training process but also yields significant performance improvements. Our study suggests several potential directions for future research. Although the established landing theory focuses on the Stiefel manifold, e.g., the generalized Stiefel manifold [51], extending this theory to general manifolds, is one potential direction. Additionally, evaluating the performance of Manifold-LoRA on LLMs with billions of parameters would be valuable. A systematic understanding of the generalization behavior induced by manifold-constrained adaptation is also an interesting direction for future research. Due to the heterogeneity of different layers, incorporating adaptive ranks for ΔW W across different layers is another possible direction. This may be achievable by adding sparsity regularization to the coordinate matrix A. Acknowledgement We thank the Associate Editor and the two reviewers for their constructive comments and suggestions, which have substantially improved the manuscript. References [1] P. Ablin and G. Peyré (2022) Fast and accurate optimization on the orthogonal manifold without retraction. In International Conference on Artificial Intelligence and Statistics, p. 5636–5657. Cited by: 1st item, §I-A, §I, §I-B, §I-B, Remark 1. [2] P. Ablin, S. Vary, B. Gao, and P. Absil (2024) Infeasible deterministic, stochastic, and variance-reduction algorithms for optimization under orthogonality constraints. Journal of Machine Learning Research 25 (389), p. 1–38. Cited by: 1st item, §I-A, §I, §I-B, §I-B, Remark 2. [3] P. Absil, R. Mahony, and R. Sepulchre (2008) Optimization algorithms on matrix manifolds. Princeton University Press. Cited by: §I-A, §I, §I-B, §I-B. [4] M. Arjovsky, A. Shah, and Y. Bengio (2016) Unitary evolution recurrent neural networks. In International conference on machine learning, p. 1120–1128. Cited by: §I. [5] M. Balashov and R. Kamalov (2021) The gradient projection method with armijo’s step size on manifolds. Computational Mathematics and Mathematical Physics 61, p. 1776–1786. Cited by: §I-A. [6] V. Bogachev, V. Aletov, A. Molozhavenko, D. Bobkov, V. Soboleva, A. Alanov, and M. Rakhuba (2025) LoRA meets riemannion: muon optimizer for parametrization-independent low-rank adapters. arXiv preprint arXiv:2507.12142. Cited by: footnote 1. [7] V. Bogachev, V. Aletov, A. Molozhavenko, D. Bobkov, V. Soboleva, A. Alanov, and M. Rakhuba (2025) Riemannlora: a unified riemannian framework for ambiguity-free lora optimization. arXiv e-prints, p. arXiv–2507. Cited by: footnote 1. [8] L. Bottou, F. E. Curtis, and J. Nocedal (2018) Optimization methods for large-scale machine learning. SIAM review 60 (2), p. 223–311. Cited by: §I-C. [9] N. Boumal, P. Absil, and C. Cartis (2019) Global rates of convergence for nonconvex optimization on manifolds. IMA Journal of Numerical Analysis 39 (1), p. 1–33. Cited by: Remark 1. [10] N. Boumal and P. Absil (2011) RTRMC: a Riemannian trust-region method for low-rank matrix completion. Advances in neural information processing systems 24. Cited by: §I-A. [11] N. Boumal (2023) An introduction to optimization on smooth manifolds. Cambridge University Press. Cited by: §I-A, §I, §I-B. [12] R. S. Chen, B. Lucier, Y. Singer, and V. Syrgkanis (2017) Robust optimization for non-convex objectives. Advances in Neural Information Processing Systems 30. Cited by: §I. [13] S. Chen, A. Garcia, M. Hong, and S. Shahrampour (2021) Decentralized Riemannian gradient descent on the Stiefel manifold. In International Conference on Machine Learning, p. 1594–1605. Cited by: §A-C, §I, Remark 2. [14] M. Cho and J. Lee (2017) Riemannian approach to batch normalization. Advances in Neural Information Processing Systems 30. Cited by: §I. [15] F. H. Clarke, R. J. Stern, and P. R. Wolenski (1995) Proximal smoothness and the lower-C2 property. Journal of Convex Analysis 2 (1-2), p. 117–144. Cited by: §I-A, §I-A. [16] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §V-E. [17] W. Dai, E. Kerman, and O. Milenkovic (2012) A geometric approach to low-rank matrix completion. IEEE Transactions on Information Theory 58 (1), p. 237–247. Cited by: §I-A. [18] W. Dai, O. Milenkovic, and E. Kerman (2011) Subspace evolution and transfer (SET) for low-rank matrix completion. IEEE Transactions on Signal Processing 59 (7), p. 3120–3132. Cited by: §I-A. [19] K. Deng and J. Hu (2023) Decentralized projected Riemannian gradient method for smooth optimization on compact submanifolds. arXiv preprint arXiv:2304.08241. Cited by: §A-C, 1st item, §I, §I-B, Remark 2. [20] N. Ding, X. Lv, Q. Wang, Y. Chen, B. Zhou, Z. Liu, and M. Sun (2023) Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311.11696. Cited by: §I-A. [21] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024) The llama 3 herd of models. arXiv e-prints, p. arXiv–2407. Cited by: §V-E. [22] J. Frankle and M. Carbin (2018) The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635. Cited by: §I-A. [23] B. Gao, G. Hu, Y. Kuang, and X. Liu (2022) An orthogonalization-free parallelizable framework for all-electron calculations in density functional theory. SIAM Journal on Scientific Computing 44 (3), p. B723–B745. Cited by: §I. [24] B. Gao, X. Liu, X. Chen, and Y. Yuan (2018) A new first-order algorithmic framework for optimization problems with orthogonality constraints. SIAM Journal on Optimization 28 (1), p. 302–332. Cited by: §I-A, §I. [25] P. He, J. Gao, and W. Chen (2021) Debertav3: improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543. Cited by: §V-B. [26] N. J. Higham (2008) Functions of matrices: theory and computation. SIAM. Cited by: §IV-A. [27] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019) Parameter-efficient transfer learning for NLP. In International conference on machine learning, p. 2790–2799. Cited by: §I, §V-A. [28] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021) Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: §I, §V-A, §V-A, TABLE VI, TABLE VI. [29] J. Hu, X. Liu, Z. Wen, and Y. Yuan (2020) A brief introduction to manifold optimization. Journal of the Operations Research Society of China 8, p. 199–248. Cited by: §I-A, §I, §I-B, §I-B. [30] S. Hu, N. Ding, W. Zhao, X. Lv, Z. Zhang, Z. Liu, and M. Sun (2023) OpenDelta: a Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models. arXiv preprint arXiv:2307.03084. Cited by: §V-A. [31] B. Jiang and Y. Dai (2015) A framework of constraint preserving update schemes for optimization on Stiefel manifold. Mathematical Programming 153 (2), p. 535–575. Cited by: §I-B. [32] L. Kong, Y. Wang, and M. Tao (2023) Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transport. In International Conference on Learning Representations, Cited by: §I. [33] D. J. Kopiczko, T. Blankevoort, and Y. M. Asano (2023) Vera: vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454. Cited by: §I-A. [34] S. Lakić (1998) On the computation of the matrix k-th root. ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik: Applied Mathematics and Mechanics 78 (3), p. 167–172. Cited by: §IV-A. [35] X. L. Li and P. Liang (2021) Prefix-tuning: optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. Cited by: §I. [36] T. Lin, C. Jin, and M. I. Jordan (2025) Two-timescale gradient descent ascent algorithms for nonconvex minimax optimization. Journal of Machine Learning Research 26 (11), p. 1–45. External Links: Link Cited by: Remark 2. [37] T. Lin, C. Jin, and M. Jordan (2020-13–18 Jul) On gradient descent ascent for nonconvex-concave minimax problems. In Proceedings of the 37th International Conference on Machine Learning, H. D. I and A. Singh (Eds.), Proceedings of Machine Learning Research, Vol. 119, p. 6083–6093. External Links: Link Cited by: Remark 2. [38] I. Loshchilov and F. Hutter (2017) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §I-C, §V-A. [39] A. Nedic and A. Ozdaglar (2009) Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control 54 (1), p. 48. Cited by: 1st item, §I, §I-B, Remark 2. [40] J. Nocedal and S. J. Wright (1999) Numerical optimization. Springer. Cited by: §I-C, §IV-A. [41] J. Novikova, O. Dušek, and V. Rieser (2017) The E2E dataset: new challenges for end-to-end generation. arXiv preprint arXiv:1706.09254. Cited by: §V-D. [42] J. Park, M. Kang, S. Lee, H. Lee, S. Kim, and J. Lee (2025-11) Riemannian optimization for LoRA on the stiefel manifold. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 20971–20985. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §V-E, §V-E, footnote 1. [43] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019) Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems 32. Cited by: §V-A. [44] C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang (2023) Is chatgpt a general-purpose natural language processing task solver?. arXiv preprint arXiv:2302.06476. Cited by: §I. [45] G. Qu and N. Li (2017) Harnessing smoothness to accelerate distributed optimization. IEEE Transactions on Control of Network Systems 5 (3), p. 1245–1260. Cited by: §I, Remark 2. [46] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2019) Language models are unsupervised multitask learners. OpenAI blog 1 (8), p. 9. Cited by: §I. [47] P. Rajpurkar, R. Jia, and P. Liang (2018) Know what you don’t know: unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822. Cited by: §V-C. [48] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang (2016) Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. Cited by: §V-C. [49] S. J. Reddi, S. Kale, and S. Kumar (2018) On the convergence of adam and beyond. In International Conference on Learning Representations, Cited by: Appendix B, §IV-B. [50] W. Shi, Q. Ling, G. Wu, and W. Yin (2015) EXTRA: an exact first-order algorithm for decentralized consensus optimization. SIAM Journal on Optimization 25 (2), p. 944–966. Cited by: §I, Remark 2. [51] S. Vary, P. Ablin, B. Gao, and P. Absil (2024) Optimization without retraction on the random generalized stiefel manifold. In International Conference on Machine Learning, p. 49226–49248. Cited by: §VI. [52] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman (2018) GLUE: a multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. Cited by: §V-B. [53] Z. Wen and W. Yin (2013) A feasible method for optimization with orthogonality constraints. Mathematical Programming 142 (1), p. 397–434. Cited by: §I, §I-B. [54] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush (2020-10) Transformers: state-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, p. 38–45. Cited by: §V-A. [55] N. Xiao, X. Liu, and K. Toh (2024) Dissolving constraints for Riemannian optimization. Mathematics of Operations Research 49 (1), p. 366–397. Cited by: §I-A, §I, §I-B, §I-B. [56] E. B. Zaken, S. Ravfogel, and Y. Goldberg (2021) Bitfit: simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199. Cited by: §I, §V-A. [57] H. Zhang and W. Yin (2013) Gradient methods for convex minimization: better rates under weaker conditions. arXiv preprint arXiv:1303.4645. Cited by: §IV-A. [58] Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao (2023) Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, Cited by: §I-A. [59] Z. Zhou, Q. Zhang, G. Lu, H. Wang, W. Zhang, and Y. Yu (2019) AdaShift: decorrelation and convergence of adaptive learning rate methods. In International Conference on Learning Representations, Cited by: Appendix B, §IV-B. Appendix A Proofs for the Retraction-Free Convergence Analysis This appendix provides the detailed arguments supporting the convergence results in Section IV. Throughout, X¯ X denotes a closest point to X on the Stiefel manifold. A-A Proof of Lemma 1 Proof. Let the singular-value decomposition of X be X=USV⊤,X=USV , and write X¯=UV⊤ X=UV . Then the distance from X to the Stiefel manifold is dist(X,St(d,r))=‖X−X¯‖=‖s−‖2,dist(X,St(d,r))\;=\;\|X- X\|\;=\;\|s-1\|_2, where s=diag(S)s=diag(S) is the vector of singular values of X. Under the assumption ‖X−X¯‖≤18\|X- X\|≤ 18, each singular value satisfies 78≤si≤98for all i. 78\;≤\;s_i\;≤\; 98 all i. We now compute the inner product between the gradient of φ(X)=14‖X⊤X−I‖F2 (X)= 14\|X X-I\|_F^2 and the deviation X−X¯X- X. First observe ∇φ(X)=X(X⊤X−I),∇ (X)=X (X X-I ), so ⟨∇φ(X),X−X¯⟩=⟨X(X⊤X−I),X−X¯⟩. ∇ (X),\,X- X = X(X X-I),\,X- X . Substitute X=USV⊤X=USV and X¯=UV⊤ X=UV , then ⟨∇φ(X),X−X¯⟩ ∇ (X),\,X- X =⟨USV⊤(VS2V⊤−I),USV⊤−UV⊤⟩ = USV (VS^2V -I ),\,USV -UV =⟨U(S3−S)V⊤,U(S−I)V⊤⟩=tr((S3−S)(S−I)). = U\,(S^3-S)\,V ,\;U\,(S-I)\,V =tr ((S^3-S)(S-I) ). Since (S3−S)(S−I)(S^3-S)(S-I) is diagonal, its trace is simply ∑i(si3−si)(si−1)=∑isi(si+1)(si−1)2. _i (s_i^3-s_i )(s_i-1)= _is_i(s_i+1)(s_i-1)^2. Because each si≥7/8s_i≥ 7/8, one checks that si(si+1)≥78⋅158=10564>32,s_i(s_i+1)\;≥\; 78· 158= 10564> 32, and hence ⟨∇φ(X),X−X¯⟩=∑isi(si+1)(si−1)2≥32∑i(si−1)2=32‖s−‖22=32‖X−X¯‖2. ∇ (X),\,X- X = _is_i(s_i+1)(s_i-1)^2\;≥\; 32 _i(s_i-1)^2= 32\|s-1\|_2^2= 32\|X- X\|^2. This completes the proof. ∎ A-B Proof of Lemma 2 Proof. Let the singular-value decomposition of the penalized iterate XkpenX_k pen be Xkpen=UkSkVk⊤,X_k pen=U_k\,S_k\,V_k , and write Xkpen¯=UkVk⊤ X_k pen=U_kV_k for its closest point on the Stiefel manifold. Since ∇φ(Xkpen)=Xkpen((Xkpen)⊤Xkpen−I)=Uk(Sk3−Sk)Vk⊤,∇ (X_k pen)=X_k pen ((X_k pen) X_k pen-I )=U_k\,(S_k^3-S_k)\,V_k , its squared Frobenius norm is ‖∇φ(Xkpen)‖F2=tr((Sk3−Sk)2)=∑i(si3−si)2=∑isi2(si+1)2(si−1)2.\|∇ (X_k pen)\|_F^2=tr ((S_k^3-S_k)^2 )= _i (s_i^3-s_i )^2= _is_i^2\,(s_i+1)^2\,(s_i-1)^2. Under the same small-deviation assumption ‖Xkpen−Xkpen¯‖≤18\|X_k pen- X_k pen\|≤ 18, each singular value sis_i lies in [78,98][ 78, 98], so si2(si+1)2≤(98)2(98+1)2< 6.s_i^2\,(s_i+1)^2\;≤\; ( 98 )^2 ( 98+1 )^2\;<\;6. Hence ‖∇φ(Xkpen)‖F2≤ 6∑i(si−1)2= 6‖Xkpen−Xkpen¯‖F2.\|∇ (X_k pen)\|_F^2\;≤\;6 _i(s_i-1)^2\;=\;6\,\|X_k pen- X_k pen\|_F^2. Next, we use the fact that projecting onto the Stiefel manifold cannot increase distance: ‖Xk+1pen−Xk+1pen¯‖2≤‖Xk+1pen−Xkpen¯‖2.\|X_k+1 pen- X_k+1 pen\|^2\;≤\; \|X_k+1 pen- X_k pen \|^2. By the gradient-step update Xk+1pen=Xkpen−13∇φ(Xkpen),X_k+1 pen=X_k pen- 13\,∇ (X_k pen), we expand ‖Xk+1pen−Xkpen¯‖2 \|X_k+1 pen- X_k pen \|^2 =‖Xkpen−13∇φ(Xkpen)−Xkpen¯‖2 = \|X_k pen- 13∇ (X_k pen)- X_k pen \|^2 =‖Xkpen−Xkpen¯‖2−23⟨Xkpen−Xkpen¯,∇φ(Xkpen)⟩+19‖∇φ(Xkpen)‖2. =\|X_k pen- X_k pen\|^2- 23 X_k pen- X_k pen,\,∇ (X_k pen) + 19\|∇ (X_k pen)\|^2. Now apply the two key inequalities we proved earlier: 1. From Lemma 1, ⟨∇φ(Xkpen),Xkpen−Xkpen¯⟩≥32‖Xkpen−Xkpen¯‖2 ∇ (X_k pen),\,X_k pen- X_k pen ≥ 32\|X_k pen- X_k pen\|^2. 2. From the singular-value bound, ‖∇φ(Xkpen)‖2≤6‖Xkpen−Xkpen¯‖2\|∇ (X_k pen)\|^2≤ 6\|X_k pen- X_k pen\|^2. Substituting gives ‖Xk+1pen−Xk+1pen¯‖2 \|X_k+1 pen- X_k+1 pen\|^2 ≤‖Xkpen−Xkpen¯‖2−23⋅32‖Xkpen−Xkpen¯‖2+19⋅6‖Xkpen−Xkpen¯‖2 ≤\|X_k pen- X_k pen\|^2- 23· 32\,\|X_k pen- X_k pen\|^2+ 19· 6\,\|X_k pen- X_k pen\|^2 =(1−1+69)‖Xkpen−Xkpen¯‖2=23‖Xkpen−Xkpen¯‖2. = (1-1+ 69 )\|X_k pen- X_k pen\|^2= 23\,\|X_k pen- X_k pen\|^2. This completes the proof. ∎ A-C Proof of Lemma 3 Proof. First inequality (8): Due to the Lipschitz continuity of f and the compactness of St(d,r) St(d,r), the inequality (8) directly follows from [13, Lemma 2.4] and [19, Lemma 4.2], where L:=Lf+DfL:=L_f+D_f with LfL_f being the Lipschitz constant of ∇fi(X)∇ f_i(X) over St(d,r) St(d,r) and Df:=maximaxX∈St(d,r)‖∇fi(X)‖D_f:= _i _X∈ St(d,r)\|∇ f_i(X)\|. Second inequality: We want to bound ‖gradf(X)−∇^f(Y)‖. \| gradf(X)- ∇f(Y) \|. Recall that ∇^f(X)=TXSt(d,r)(∇f(X)) ∇f(X)=P_T_XSt(d,r) (∇ f(X) ) and ∇^f(X)=gradf(X) ∇f(X)=gradf(X) whenever X∈St(d,r)X (d,r). We split the difference as ‖gradf(X)−∇^f(Y)‖ \| gradf(X)- ∇f(Y) \| ≤‖TXSt(d,r)(∇f(X))−TXSt(d,r)(∇f(Y))‖ ≤ \|P_T_X St(d,r)(∇ f(X))-P_T_X St(d,r)(∇ f(Y))\| +‖TXSt(d,r)(∇f(Y))−TYSt(d,r)(∇f(Y))‖. + \|P_T_X St(d,r)(∇ f(Y))-P_T_Y St(d,r)(∇ f(Y)) \|. 1. The first term is bounded by the contractive property of TXSt(d,r)P_T_X St(d,r) and the Lipschitz continuity of ∇f∇ f: ‖TXSt(d,r)(∇f(X))−TXSt(d,r)(∇f(Y))‖≤‖∇f(X)−∇f(Y)‖≤Lf‖X−Y‖. \|P_T_X St(d,r)(∇ f(X))-P_T_X St(d,r)(∇ f(Y)) \|\;≤\; \|∇ f(X)-∇ f(Y) \|\;≤\;L_f\,\|X-Y\|. 2. Recall that TXSt(d,r)(g):=g−Xsym(X⊤g)=g−12X(X⊤g+g⊤X)P_T_XSt(d,r)(g):=g-Xsym (X g )=g- 12X (X g+g X ). Thus, a direct calculation of second term shows ‖TXSt(d,r)(∇f(Y))−TYSt(d,r)(∇f(Y))‖ \|P_T_X St(d,r)(∇ f(Y))-P_T_Y St(d,r)(∇ f(Y)) \| = = 12‖X(X⊤∇f(Y)+∇f(Y)⊤X)−Y(Y⊤∇f(Y)+∇f(Y)⊤Y)‖ 12 \|X (X ∇ f(Y)+∇ f(Y) X )-Y (Y ∇ f(Y)+∇ f(Y) Y ) \| ≤ ≤ 12‖X((X−Y)⊤∇f(Y)+∇f(Y)⊤(X−Y))‖+12‖(X−Y)(Y⊤∇f(Y)+∇f(Y)⊤Y)‖ 12\|X((X-Y) ∇ f(Y)+∇ f(Y) (X-Y))\|+ 12\|(X-Y)(Y ∇ f(Y)+∇ f(Y) Y)\| ≤ ≤ 12(2D^f+3D^f)‖X−Y‖ 12(2 D_f+3 D_f)\|X-Y\| = = 52D^f‖X−Y‖, 52\, D_f\,\|X-Y\|, where D^f:=maxY∈U¯St(d,r)(18)‖∇f(Y)‖ D_f:= _Y∈ U_ St(d,r)( 18)\|∇ f(Y)\|, and the last inequality is from the fact that ‖Y‖2=σmax(Y)≤32\|Y\|_2= _ (Y)≤ 32. Putting both pieces together, ‖gradf(X)−∇^f(Y)‖≤(Lf+52D^f)‖X−Y‖.\| gradf(X)- ∇f(Y)\|\;≤\; (L_f+ 52\, D_f )\,\|X-Y\|. Setting L^=Lf+52D^f L=L_f+ 52\, D_f completes the proof. ∎ A-D Proof of Lemma 4 Proof. It follows from X¯k+1=argminX∈St(d,r)‖X−Xk+1‖2 X_k+1= *argmin_X∈ St(d,r)\|X-X_k+1\|^2 that ‖Xk+1−X¯k+1‖ \|X_k+1- X_k+1\| ≤‖Xk+1−X¯k‖ ≤\|X_k+1- X_k\| ≤‖Xk−αkTXkSt(d,r)(gk)−13∇φ(Xk)−X¯k‖ (by update (3), μ=13) ≤\|X_k- _kP_T_X_kSt(d,r)(g_k)- 13∇ (X_k)- X_k\| (by update eq:grad-it, $μ= 13$) ≤‖Xk−13∇φ(Xk)−X¯k‖+52αk‖gk‖, ≤\|X_k- 13∇ (X_k)- X_k\|+ 52 _k\|g_k\|, where we used the fact ‖TXkSt(d,r)(g)‖≤‖g‖+‖X‖22‖g‖≤52‖g‖.\|P_T_X_k St(d,r)(g)\|≤\|g\|+\|X\|_2^2\|g\|≤ 52\|g\|. (A.1) According to the proof of Lemma 2, we have ‖Xk−13∇φ(Xk)−X¯k‖2≤23‖Xk−X¯k‖2.\|X_k- 13∇ (X_k)- X_k\|^2≤ 23\|X_k- X_k\|^2. Hence, we conclude that ‖Xk+1−X¯k+1‖≤23‖Xk−X¯k‖+52αk‖gk‖.\|X_k+1- X_k+1\|≤ 23\|X_k- X_k\|+ 52 _k\|g_k\|. We complete the proof. ∎ A-E Proof of Lemma 5 Proof. First, let us prove the following equality ⟨∇^fi(X),∇ϕ(X)⟩=⟨∇fi(X),TXSt(d,r)(∇ϕ(X))⟩. ∇f_i(X),∇φ(X) = ∇ f_i(X),P_T_X St(d,r)(∇φ(X)) . In fact, using the definition of ⟨A,B⟩=tr(A⊤B) A,B = tr(A B), we have ⟨∇^fi(X),∇ϕ(X)⟩= ∇f_i(X),∇φ(X) = ⟨∇fi(X)−Xsym(X⊤∇fi(X)),∇ϕ(X)⟩ ∇ f_i(X)-X sym(X ∇ f_i(X)),∇φ(X) = = ⟨∇fi(X),∇ϕ(X)⟩−⟨Xsym(X⊤∇fi(X)),∇ϕ(X)⟩ ∇ f_i(X),∇φ(X) - X sym(X ∇ f_i(X)),∇φ(X) = = ⟨∇fi(X),∇ϕ(X)⟩−⟨sym(X⊤∇fi(X)),X⊤∇ϕ(X)⟩ ∇ f_i(X),∇φ(X) - sym(X ∇ f_i(X)),X ∇φ(X) = = ⟨∇fi(X),∇ϕ(X)⟩−⟨X⊤∇fi(X),sym(X⊤∇ϕ(X))⟩ ∇ f_i(X),∇φ(X) - X ∇ f_i(X), sym(X ∇φ(X)) = = ⟨∇fi(X),∇ϕ(X)⟩−⟨∇fi(X),Xsym(X⊤∇ϕ(X))⟩ ∇ f_i(X),∇φ(X) - ∇ f_i(X),X sym(X ∇φ(X)) = = ⟨∇fi(X),TXSt(d,r)(∇ϕ(X))⟩. ∇ f_i(X),P_T_X St(d,r)(∇φ(X)) . Then, it follows from (8) that f(X¯k+1)−f(X¯k)≤ f( X_k+1)-f( X_k)≤ ⟨gradf(X¯k),X¯k+1−X¯k⟩+L2‖X¯k+1−X¯k‖2 gradf( X_k), X_k+1- X_k + L2\| X_k+1- X_k\|^2 ≤ ≤ ⟨gradf(X¯k),X¯k+1−Xk+1+Xk−X¯k⟩+⟨gradf(X¯k),Xk+1−Xk⟩+2L‖Xk+1−Xk‖2 gradf( X_k), X_k+1-X_k+1+X_k- X_k + gradf( X_k),X_k+1-X_k +2L\|X_k+1-X_k\|^2 ≤ ≤ ⟨gradf(X¯k),X¯k+1−Xk+1⟩+⟨gradf(X¯k),Xk+1−Xk⟩+4L(αk2‖TXkSt(d,r)(gk)‖2+μ2‖∇φ(Xk)‖2) gradf( X_k), X_k+1-X_k+1 + gradf( X_k),X_k+1-X_k +4L( _k^2\|P_T_X_k St(d,r)(g_k)\|^2+μ^2\|∇ (X_k)\|^2) = = ⟨gradf(X¯k)−gradf(X¯k+1),X¯k+1−Xk+1⟩+⟨∇^f(Xk),Xk+1−Xk⟩ gradf( X_k)- gradf( X_k+1), X_k+1-X_k+1 + ∇f(X_k),X_k+1-X_k +⟨gradf(X¯k)−∇^f(Xk),Xk+1−Xk⟩+4L(αk2‖TXkSt(d,r)(gk)‖2+μ2‖∇φ(Xk)‖2) + gradf( X_k)- ∇f(X_k),X_k+1-X_k +4L( _k^2\|P_T_X_k St(d,r)(g_k)\|^2+μ^2\|∇ (X_k)\|^2) ≤ ≤ 2L^2‖Xk+1−Xk‖2+12‖Xk+1−X¯k+1‖2−αk⟨∇^f(Xk),TXkSt(d,r)(gk)⟩ 2 L^2\|X_k+1-X_k\|^2+ 12\|X_k+1- X_k+1\|^2- _k ∇f(X_k),P_T_X_k St(d,r)(g_k) −μ⟨∇^f(Xk),∇φ(Xk)⟩+12(L^2‖Xk−X¯k‖2+‖Xk+1−Xk‖2) -μ ∇f(X_k),∇ (X_k) + 12( L^2\|X_k- X_k\|^2+\|X_k+1-X_k\|^2) +4L(αk2‖TXkSt(d,r)(gk)‖2+μ2‖∇φ(Xk)‖2), +4L( _k^2\|P_T_X_k St(d,r)(g_k)\|^2+μ^2\|∇ (X_k)\|^2), where the second inequality is from the 2-Lipschitz continuity of St(d,r)P_ St(d,r) over U¯St(d,r)(18) U_ St(d,r)( 18), the last inequality is due to the facts that Xk−X¯k∈NX¯kSt(d,r)X_k- X_k∈ N_ X_k St(d,r) and ⟨A,B⟩≤12(‖A‖2+‖B‖2) A,B ≤ 12(\|A\|^2+\|B\|^2) for any A,B∈ℝd×rA,B ^d× r. Noticing that ‖TXkSt(d,r)(∇φ(Xk))‖ \|P_T_X_k St(d,r)(∇ (X_k))\| =‖Xk(Xk⊤Xk−I)2‖≤‖UkSkVk⊤(VkSk2Vk⊤−I)Vk‖ =\|X_k(X_k X_k-I)^2\|≤\|U_kS_kV_k (V_kS_k^2V_k -I)V_k\| ≤‖Sk(Sk+I)2‖‖Xk−X¯k‖2≤6‖Xk−X¯k‖2, ≤\|S_k(S_k+I)^2\|\|X_k- X_k\|^2≤ 6\|X_k- X_k\|^2, we have k[f(X¯k+1)]−f(X¯k)≤ _k[f( X_k+1)]-f( X_k)≤ −αk‖∇^f(Xk)‖2−μ⟨∇f(Xk),TXkSt(d,r)(∇φ(Xk))⟩+12k‖Xk+1−X¯k+1‖2 - _k\| ∇f(X_k)\|^2-μ ∇ f(X_k),P_T_X_k St(d,r)(∇ (X_k)) + 12E_k\|X_k+1- X_k+1\|^2 (A.2) +12‖Xk−X¯k‖2+(4L^2+4L+1)(αk2k[‖TXkSt(d,r)(gk)‖2]+μ2‖∇φ(Xk)‖2) + 12\|X_k- X_k\|^2+(4 L^2+4L+1)( _k^2E_k[\|P_T_X_k St(d,r)(g_k)\|^2]+μ^2\|∇ (X_k)\|^2) ≤ ≤ −(αk−(4L^2+4L+1)αk2)‖∇^f(Xk)‖2+12k‖Xk+1−X¯k+1‖2 -( _k-(4 L^2+4L+1) _k^2)\| ∇f(X_k)\|^2+ 12E_k\|X_k+1- X_k+1\|^2 +(6μD^f+12L^2+6(4L^2+4L+1)μ2)‖Xk−X¯k‖2+7(4L^2+4L+1)αk2σ2, +(6μ D_f+ 12 L^2+6(4 L^2+4L+1)μ^2)\|X_k- X_k\|^2+7(4 L^2+4L+1) _k^2σ^2, where we use [TXkSt(d,r)(gk)]=TXkSt(d,r)([gk])=∇^f(Xk)E[P_T_X_k St(d,r)(g_k)]=P_T_X_k St(d,r)(E[g_k])= ∇f(X_k) and [‖TXkSt(d,r)(gk)‖2]≤‖∇^f(Xk)‖2+7([‖gk−∇^f(Xk)‖2])=‖∇^f(Xk)‖2+7σ2E[\|P_T_X_k St(d,r)(g_k)\|^2]≤\| ∇f(X_k)\|^2+7(E[\|g_k- ∇f(X_k)\|^2])=\| ∇f(X_k)\|^2+7σ^2 from Assumption 1 and (A.1). Plugging μ=13μ= 13 into (A.2) gives (11). ∎ A-F Proof of Theorem 1 Proof. First, we show Xk∈U¯St(n,d)(18)X_k∈ U_ St(n,d)( 18) for any k≥0k≥ 0 if αk≤1120D^f _k≤ 1120 D_f. In fact, by proof of induction, we have from (10) that ‖Xk+1−X¯k+1‖≤23‖Xk−X¯k‖+5240D^f‖gk‖≤18.\|X_k+1- X_k+1\|≤ 23\|X_k- X_k\|+ 5240 D_f\|g_k\|≤ 18. Moreover, applying the standard sequence bound to (10) yields [∑k=1K‖Xk−X¯k‖2]≤60[∑k=1Kαk2‖TXkSt(d,r)(gk)‖2]+4≤60∑k=1K(αk2‖∇^f(Xk)‖2+7αk2σ2)+4 [ _k=1^K\|X_k- X_k\|^2 ]≤ 0E [ _k=1^K _k^2\|P_T_X_k St(d,r)(g_k)\|^2 ]+4≤ 0 _k=1^K ( _k^2\| ∇f(X_k)\|^2+7 _k^2σ^2 )+4 Then, summing (11) over k=1,…,Kk=1,…,K gives [f(X¯K+1)]−f(X¯1) [f( X_K+1)]-f( X_1) ≤ ≤ −(αk−(4L^2+4L+1)αk2)[∑k=1K‖∇^f(Xk)‖2] -( _k-(4 L^2+4L+1) _k^2)E [ _k=1^K\| ∇f(X_k)\|^2 ] +12(4D^f+9L^2+8L+4)[∑k=1K+1‖Xk−X¯k‖2] + 12 (4 D_f+9 L^2+8L+4 )E [ _k=1^K+1\|X_k- X_k\|^2 ] ≤ ≤ −(αk−(4L^2+4L+1)αk2+30(4D^f+9L^2+8L+4)αk2)[∑k=1K‖∇^f(Xk)‖2] -( _k-(4 L^2+4L+1) _k^2+0(4 D_f+9 L^2+8L+4) _k^2)E [ _k=1^K\| ∇f(X_k)\|^2 ] +12(4D^f+9L^2+8L+4)(60αK+12D^f2+420σ2∑k=1K+1αk2+4). + 12 (4 D_f+9 L^2+8L+4 )(0 _K+1^2 D_f^2+20σ^2 _k=1^K+1 _k^2+4). Define c1=275L^2+244L+120D^f+121c_1=275 L^2+244L+120 D_f+121 and c2(k)=(30αK+12D^f2+210σ2∑k=1K+1αk2+2)(4D^f+8L^2+8L+4)c_2(k)=(30 _K+1^2 D_f^2+210σ^2 _k=1^K+1 _k^2+2)(4 D_f+8 L^2+8L+4). Then, we have [∑k=1Kαk(1−c1αk)‖∇^f(Xk)‖2]≤f(X¯1)−fmin+c2(k),E [ _k=1^K _k(1-c_1 _k)\| ∇f(X_k)\|^2 ]≤ f( X_1)-f_ +c_2(k), (A.3) where fmin:=argminX∈St(d,r)f(X)f_ := *argmin_X∈ St(d,r)\;f(X). Therefore, we have the following iteration complexity results. • If taking constant step sizes αk≡α∈(0,12c1] _k≡α∈(0, 12c_1] (which also gives α≤1120D^fα≤ 1120 D_f), then 1K[∑k=1K‖∇^f(Xk)‖2]≤ 1KE [ _k=1^K\| ∇f(X_k)\|^2 ]≤ 2(f(X¯1)−fmin)+24c3αK+420ασ2, 2(f( X_1)-f_ )+24c_3α K+20ασ^2, 1K[∑k=1K‖Xk−X¯k‖2]≤ 1KE [ _k=1^K\|X_k- X_k\|^2 ]≤ 120α2(f(X¯1)−fmin)+1440α2c3+4αK+420α2(60α+1)σ2, 120α^2(f( X_1)-f_ )+1440α^2c_3+4α K+20α^2(0α+1)σ^2, where c3:=D^f+2L^2+2L+1c_3:= D_f+2 L^2+2L+1. If in addition the full gradient is used, i.e., gk=∇f(xk)g_k=∇ f(x_k), we have σ=0σ=0 and 1K[∑k=1K(‖∇^f(Xk)‖2+‖Xk−X¯k‖2)]=(1K). 1KE [ _k=1^K (\| ∇f(X_k)\|^2+\|X_k- X_k\|^2 ) ]=O ( 1K ). • If taking step sizes αk=α0k _k= _0 k with α0∈(0,12c1] _0∈(0, 12c_1], it follows from (10) and ‖∇^f(Xk)‖≤52D^f\| ∇f(X_k)\|≤ 52 D_f that mink≤K‖Xk−X¯k‖2≤1K[∑k=1K‖Xk−X¯k‖2]=(logK). _k≤ K\|X_k- X_k\|^2≤ 1KE [ _k=1^K\|X_k- X_k\|^2 ]=O ( KK ). Besides, summing (10) and (A.3) gives [∑k=1K(αk(1−c1αk)‖∇^f(Xk)‖2+‖Xk−X¯k‖2)]=(logK).E [ _k=1^K ( _k(1-c_1 _k)\| ∇f(X_k)\|^2+\|X_k- X_k\|^2 ) ]=O ( K ). Furthermore, dividing both sides of (A.3) by ∑k=1αk _k=1 _k leads to mink≤K[‖∇^f(Xk)‖2+‖Xk−X¯k‖2]=(logK). _k≤ K\;\;E [\| ∇f(X_k)\|^2+\|X_k- X_k\|^2 ]=O ( K K ). We complete the proof. ∎ A-G Proof of Corollary 1 Proof. To prove the corollary from Theorem 1, we concatenate (A,B)(A,B) into a single variable X, and verify that all assumptions and conclusions used in Theorem 1 remain valid. For simplicity, we assume B∈St(d,r)B∈ St(d,r). The oblique manifold Ob(d,r) Ob(d,r) can be seen as the product of r Stiefel manifolds St(d,1) St(d,1), and the results can be easily extended to the setting. The detailed verifications are as follows: • Lemma 1: Since the constraint set is now ℝd×r×St(d,r)R^d× r×St(d,r), we define X¯:=(A,B¯) X:=(A, B) and let φ(X)=14‖B⊤B−I‖2 (X)= 14\|B B-I\|^2. With this definition, the same conclusion as in the original lemma holds. • Lemma 2: This lemma follows naturally under the new definition of X¯ X, as it depends only on the conclusion of Lemma 1. • Lemma 3: A similar Lipschitz-type result holds by leveraging the Lipschitz continuity of ∇ℒ and the geometric properties of the Stiefel manifold associated with B. • Lemma 4: This is a direct consequence of Lemma 2 and does not require further modification. • Lemma 5: By using all the results above, we can obtain a descent inequality for ℒ(X¯k+1)−ℒ(X¯k)L( X_k+1)-L( X_k) in terms of the gradient norm and feasibility errors. Specifically, the descent is characterized by ‖∇^ℒ(Xk)‖,‖Bk−B¯k‖2,‖Bk+1−B¯k+1‖2,andσ2,\| ∇L(X_k)\|, \|B_k- B_k\|^2, \|B_k+1- B_k+1\|^2, σ^2, where ∇^ℒ(Xk)=[∇Aℒ(BkAk),∇^Bℒ(BkAk)]. ∇L(X_k)= [ _AL(B_kA_k),\, ∇_BL(B_kA_k) ]. Therefore, by the chosen step size αk _k, we can invoke Theorem 1 and complete the proof of the corollary. ∎ Appendix B Convergence of an Adam-style retraction-free update The convergence of Algorithm 2 with the AdamW update is difficult to guarantee directly, as even the Euclidean Adam method may fail to converge without suitable modifications [49]. Nevertheless, our retraction-free idea can be combined with Adam-type correction techniques from Euclidean optimization to obtain convergence guarantees. As an example, following the decorrelation idea in AdaShift [59], we use a delayed adaptive preconditioner: the preconditioner is fixed before the current stochastic gradient is drawn, which makes the alignment argument conditionally clean. We first introduce the Adam-style retraction-free update. Given a stochastic Euclidean gradient gkg_k, define its projected stochastic gradient by g~k:=PTXkSt(d,r)(gk) g_k:=P_T_X_kSt(d,r)(g_k). Let m0=0m_0=0, v0=0v_0=0, and v¯0=0 v_0=0. For k≥1k≥ 1, define mk m_k =β1,kmk−1+(1−β1,k)g~k, = _1,km_k-1+(1- _1,k) g_k, (B.1) vk v_k =β2vk−1+(1−β2)(g~k⊙g~k),v¯k=maxv¯k−1,vk, = _2v_k-1+(1- _2)( g_k g_k), v_k= \ v_k-1,v_k\, (B.2) where the maximum is taken componentwise. We identify matrices with their vectorizations when applying diagonal preconditioners, and define Hk−1:=Diag(1v¯k−1+ϵ),dk:=Hk−1mk.H_k-1:=Diag ( 1 v_k-1+ε ), d_k:=H_k-1m_k. The Adam-style retraction-free update is Xk+1=Xk−αkdk−13∇φ(Xk).X_k+1=X_k- _kd_k- 13∇ (X_k). (B.3) For the notation, let ℱkF_k denote the filtration generated by the history before drawing the stochastic gradient at iteration k, and let k[⋅]:=[⋅∣ℱk]E_k[·]:=E[· _k]. For a matrix X in a neighborhood of St(d,r)St(d,r), let X¯ X denote its projection onto St(d,r)St(d,r), and define δ(X):=‖X−X¯‖δ(X):=\|X- X\|. For the iterate XkX_k, we write X¯k X_k for its polar projection and set δk:=‖Xk−X¯k‖ _k:=\|X_k- X_k\|. We also use the notation in the main paper, ∇^f(X):=∇f(X)−Xsym(X⊤∇f(X)). ∇f(X):=∇ f(X)-Xsym(X ∇ f(X)). Condition B.1 (Stochasticity for the Adam-style update). For every k, conditioned on ℱkF_k, k[g~k]=∇^f(Xk),k‖g~k−∇^f(Xk)‖2≤σ2.E_k[ g_k]= ∇f(X_k), _k\| g_k- ∇f(X_k)\|^2≤σ^2. Moreover, there exist constants G>0G>0 and G∞>0G_∞>0 such that ‖g~k‖≤G\| g_k\|≤ G and ‖g~k‖∞≤G∞\| g_k\|_∞≤ G_∞ almost surely. Finally, 0≤β1,k≤β¯<10≤ _1,k≤ β<1 and 0≤β2<10≤ _2<1. Set h+:=1ϵ,h−:=1G∞+ϵ,q:=23.h_+:= 1ε, h_-:= 1G_∞+ε, q:= 23. Under Condition B.1, we have h−I⪯Hk⪯h+Ih_-I H_k h_+I for every k. Lemma B.1 (Boundedness of the Adam direction). Under Condition B.1, for every k≥1k≥ 1, ‖mk‖≤G\|m_k\|≤ G and ‖dk‖≤h+G\|d_k\|≤ h_+G. Moreover, k‖dk‖2≤2h+2(‖∇^f(Xk)‖2+σ2)+8h+2β1,k2G2.E_k\|d_k\|^2≤ 2h_+^2 (\| ∇f(X_k)\|^2+σ^2 )+8h_+^2 _1,k^2G^2. (B.4) Proof. The bound ‖mk‖≤G\|m_k\|≤ G follows by induction from (B.1), because mkm_k is a convex combination of mk−1m_k-1 and g~k g_k. Since Hk−1⪯h+IH_k-1 h_+I, we get ‖dk‖≤h+‖mk‖≤h+G\|d_k\|≤ h_+\|m_k\|≤ h_+G. For the second-moment bound, write mk=g~k+β1,k(mk−1−g~k)m_k= g_k+ _1,k(m_k-1- g_k). Then ‖mk‖2≤2‖g~k‖2+2β1,k2‖mk−1−g~k‖2≤2‖g~k‖2+8β1,k2G2.\|m_k\|^2≤ 2\| g_k\|^2+2 _1,k^2\|m_k-1- g_k\|^2≤ 2\| g_k\|^2+8 _1,k^2G^2. Since k[g~k]=∇^f(Xk)E_k[ g_k]= ∇f(X_k), k‖g~k‖2=‖∇^f(Xk)‖2+k‖g~k−∇^f(Xk)‖2≤‖∇^f(Xk)‖2+σ2.E_k\| g_k\|^2=\| ∇f(X_k)\|^2+E_k\| g_k- ∇f(X_k)\|^2≤\| ∇f(X_k)\|^2+σ^2. Using ‖dk‖≤h+‖mk‖\|d_k\|≤ h_+\|m_k\| proves (B.4). ∎ Lemma B.2 (Feasibility recursion). Suppose Xk∈U¯St(d,r)(1/8)X_k∈ U_St(d,r)(1/8). Then δk+1≤qδk+αk‖dk‖. _k+1≤ q _k+ _k\|d_k\|. (B.5) Consequently, if δ1≤1/8 _1≤ 1/8 and αk≤(1−q)/(8h+G) _k≤(1-q)/(8h_+G) for all k, then Xk∈U¯St(d,r)(1/8)X_k∈ U_St(d,r)(1/8) for all k. Moreover, for any K≥1K≥ 1, ∑k=1K[δk2]≤6δ12+60∑k=1K−1αk2‖dk‖2. _k=1^KE[ _k^2]≤ 6 _1^2+60 _k=1^K-1 _k^2E\|d_k\|^2. (B.6) In particular, ∑k=1K[δk2] _k=1^KE[ _k^2] ≤6δ12+120h+2∑k=1K−1αk2‖∇^f(Xk)‖2+120h+2σ2∑k=1K−1αk2 ≤ 6 _1^2+120h_+^2 _k=1^K-1 _k^2E\| ∇f(X_k)\|^2+120h_+^2σ^2 _k=1^K-1 _k^2 +480h+2G2∑k=1K−1αk2β1,k2. +480h_+^2G^2 _k=1^K-1 _k^2 _1,k^2. (B.7) Proof. By the definition of X¯k+1 X_k+1 and the update (B.3), δk+1 _k+1 =‖Xk+1−X¯k+1‖≤‖Xk+1−X¯k‖ =\|X_k+1- X_k+1\|≤\|X_k+1- X_k\| ≤‖Xk−13∇φ(Xk)−X¯k‖+αk‖dk‖≤qδk+αk‖dk‖, ≤ \|X_k- 13∇ (X_k)- X_k \|+ _k\|d_k\|≤ q _k+ _k\|d_k\|, where the last inequality follows from the proof of Lemma 2. Since ‖dk‖≤h+G\|d_k\|≤ h_+G, the step-size condition gives δk+1≤q/8+(1−q)/8=1/8 _k+1≤ q/8+(1-q)/8=1/8 whenever δk≤1/8 _k≤ 1/8. Thus the iterates remain in U¯St(d,r)(1/8) U_St(d,r)(1/8) by induction. Iterating (B.5) yields δk≤qk−1δ1+∑j=1k−1qk−1−jαj‖dj‖. _k≤ q^k-1 _1+ _j=1^k-1q^k-1-j _j\|d_j\|. Young’s convolution inequality gives ∑k=1Kδk2≤21−q2δ12+2(1−q)2∑k=1K−1αk2‖dk‖2. _k=1^K _k^2≤ 21-q^2 _1^2+ 2(1-q)^2 _k=1^K-1 _k^2\|d_k\|^2. Since 2/(1−q2)=62/(1-q^2)=6 and 2/(1−q)2<602/(1-q)^2<60, taking expectation proves (B.6). Combining this estimate with Lemma B.1 proves (B.7). ∎ Lemma B.3 (Adam-style alignment). Under Condition B.1, for every k≥1k≥ 1, k⟨∇^f(Xk),dk⟩≥a‖∇^f(Xk)‖2−Rk,Rk:=β1,k2h+2G22(1−β¯)h−,a:=(1−β¯)h−2.E_k ∇f(X_k),d_k ≥ a\| ∇f(X_k)\|^2-R_k, R_k:= _1,k^2h_+^2G^22(1- β)h_-, a:= (1- β)h_-2. (B.8) Proof. Since Hk−1H_k-1 and mk−1m_k-1 are ℱkF_k-measurable and k[g~k]=∇^f(Xk)E_k[ g_k]= ∇f(X_k), k⟨∇^f(Xk),dk⟩ _k ∇f(X_k),d_k =k⟨∇^f(Xk),Hk−1mk⟩ =E_k ∇f(X_k),H_k-1m_k =(1−β1,k)⟨∇^f(Xk),Hk−1∇^f(Xk)⟩+β1,k⟨∇^f(Xk),Hk−1mk−1⟩. =(1- _1,k) ∇f(X_k),H_k-1 ∇f(X_k) + _1,k ∇f(X_k),H_k-1m_k-1 . Using Hk−1⪰h−IH_k-1 h_-I, Hk−1⪯h+IH_k-1 h_+I, and ‖mk−1‖≤G\|m_k-1\|≤ G, we obtain k⟨∇^f(Xk),dk⟩≥(1−β1,k)h−‖∇^f(Xk)‖2−β1,kh+G‖∇^f(Xk)‖.E_k ∇f(X_k),d_k ≥(1- _1,k)h_-\| ∇f(X_k)\|^2- _1,kh_+G\| ∇f(X_k)\|. By Young’s inequality, β1,kh+G‖∇^f(Xk)‖≤(1−β1,k)h−2‖∇^f(Xk)‖2+β1,k2h+2G22(1−β1,k)h−. _1,kh_+G\| ∇f(X_k)\|≤ (1- _1,k)h_-2\| ∇f(X_k)\|^2+ _1,k^2h_+^2G^22(1- _1,k)h_-. Since β1,k≤β¯ _1,k≤ β, the claim follows. ∎ Lemma B.4 (One-step descent). Suppose Xk,Xk+1∈U¯St(d,r)(1/8)X_k,X_k+1∈ U_St(d,r)(1/8). Then k[f(X¯k+1)]−f(X¯k) _k[f( X_k+1)]-f( X_k) ≤−aαk‖∇^f(Xk)‖2+αkRk+12k[δk+12]+Cφδk2 ≤-a _k\| ∇f(X_k)\|^2+ _kR_k+ 12E_k[ _k+1^2]+C_ _k^2 +C0αk2k‖dk‖2. +C_0 _k^2E_k\|d_k\|^2. (B.9) where C0:=4maxL^2,D^f2+4L+1,Cφ:=2D^f+12L^2+23C0,C_0:=4 \ L^2, D_f^2\+4L+1, C_ :=2 D_f+ 12 L^2+ 23C_0, L and L L are the constants in Lemma 3, D^f:=maxX∈U¯St(d,r)(1/8)‖∇f(X)‖ D_f:= _X∈ U_St(d,r)(1/8)\|∇ f(X)\|, and a is defined in Lemma B.3. Proof. Let Δk:=Xk+1−Xk _k:=X_k+1-X_k, rk:=Xk−X¯kr_k:=X_k- X_k, rk+1:=Xk+1−X¯k+1r_k+1:=X_k+1- X_k+1, and ηk:=gradf(X¯k) _k:=gradf( X_k). Averaging the quadratic upper bound in Lemma 3 over the component functions gives f(X¯k+1)−f(X¯k)≤⟨ηk,X¯k+1−X¯k⟩+L2‖X¯k+1−X¯k‖2.f( X_k+1)-f( X_k)≤ _k, X_k+1- X_k + L2\| X_k+1- X_k\|^2. (B.10) Writing the compact singular value decomposition of XkX_k as UkSkVk⊤U_kS_kV_k , the polar projection is X¯k=UkVk⊤ X_k=U_kV_k , and rk=X¯kNkr_k= X_kN_k with the symmetric matrix Nk:=Vk(Sk−I)Vk⊤N_k:=V_k(S_k-I)V_k . Since ηk∈TX¯kSt(d,r) _k∈ T_ X_kSt(d,r), we have ⟨ηk,rk⟩=0 _k,r_k =0. Moreover, X¯k+1−X¯k=Δk+rk−rk+1 X_k+1- X_k= _k+r_k-r_k+1, and hence ⟨ηk,X¯k+1−X¯k⟩=⟨∇^f(Xk),Δk⟩+⟨ηk−∇^f(Xk),Δk⟩−⟨ηk,rk+1⟩. _k, X_k+1- X_k = ∇f(X_k), _k + _k- ∇f(X_k), _k - _k,r_k+1 . By Lemma 3, applied with X=X¯kX= X_k and Y=XkY=X_k and then averaged over the component functions, ⟨ηk−∇^f(Xk),Δk⟩≤12L^2δk2+12‖Δk‖2. _k- ∇f(X_k), _k ≤ 12 L^2 _k^2+ 12\| _k\|^2. Similarly, rk+1=X¯k+1Nk+1r_k+1= X_k+1N_k+1 for a symmetric matrix Nk+1N_k+1 with ‖Nk+1‖=δk+1\|N_k+1\|= _k+1. Since ⟨ηk,X¯kNk+1⟩=0 _k, X_kN_k+1 =0, we have |⟨ηk,rk+1⟩| | _k,r_k+1 | =|⟨ηk,(X¯k+1−X¯k)Nk+1⟩| = | _k,( X_k+1- X_k)N_k+1 | ≤‖ηk‖‖X¯k+1−X¯k‖δk+1. ≤\| _k\|\,\| X_k+1- X_k\|\, _k+1. Since Xk,Xk+1∈U¯St(d,r)(1/8)X_k,X_k+1∈ U_St(d,r)(1/8), their smallest singular values are at least 7/87/8. The standard perturbation bound for the polar factor therefore gives ‖X¯k+1−X¯k‖≤2‖Xk+1−Xk‖=2‖Δk‖\| X_k+1- X_k\|≤ 2\|X_k+1-X_k\|=2\| _k\|. Together with ‖ηk‖≤‖∇f(X¯k)‖≤D^f\| _k\|≤\|∇ f( X_k)\|≤ D_f, this gives |⟨ηk,rk+1⟩|≤2D^f‖Δk‖δk+1≤2D^f2‖Δk‖2+12δk+12. | _k,r_k+1 |≤ 2 D_f\| _k\| _k+1≤ 2 D_f^2\| _k\|^2+ 12 _k+1^2. The same projection Lipschitz estimate gives L2‖X¯k+1−X¯k‖2≤2L‖Δk‖2. L2\| X_k+1- X_k\|^2≤ 2L\| _k\|^2. Combining these bounds with (B.10) yields f(X¯k+1)−f(X¯k)≤⟨∇^f(Xk),Δk⟩+12δk+12+12L^2δk2+C02‖Δk‖2.f( X_k+1)-f( X_k)≤ ∇f(X_k), _k + 12 _k+1^2+ 12 L^2 _k^2+ C_02\| _k\|^2. (B.11) From (B.3), Δk=−αkdk−(1/3)∇φ(Xk) _k=- _kd_k-(1/3)∇ (X_k). Thus ⟨∇^f(Xk),Δk⟩=−αk⟨∇^f(Xk),dk⟩−13⟨∇^f(Xk),∇φ(Xk)⟩. ∇f(X_k), _k =- _k ∇f(X_k),d_k - 13 ∇f(X_k),∇ (X_k) . The linear map A↦PTXkSt(d,r)(A)A P_T_X_kSt(d,r)(A) is self-adjoint with respect to the Frobenius inner product. Therefore, ⟨∇^f(Xk),∇φ(Xk)⟩=⟨∇f(Xk),PTXkSt(d,r)(∇φ(Xk))⟩. ∇f(X_k),∇ (X_k) = ∇ f(X_k),P_T_X_kSt(d,r)(∇ (X_k)) . Let Ak=Xk⊤Xk−IA_k=X_k X_k-I. Since AkA_k is symmetric, PTXkSt(d,r)(∇φ(Xk))=XkAk−Xksym(Xk⊤XkAk)=−XkAk2.P_T_X_kSt(d,r)(∇ (X_k))=X_kA_k-X_ksym(X_k X_kA_k)=-X_kA_k^2. Writing Xk=UkSkVk⊤X_k=U_kS_kV_k , the proof of Lemma 1 gives 7/8≤si≤9/87/8≤ s_i≤ 9/8 for all singular values. Hence ‖PTXkSt(d,r)(∇φ(Xk))‖=‖UkSk(Sk2−I)2Vk⊤‖≤6δk2.\|P_T_X_kSt(d,r)(∇ (X_k))\|=\|U_kS_k(S_k^2-I)^2V_k \|≤ 6 _k^2. Together with the definition of D^f D_f, this gives −13⟨∇^f(Xk),∇φ(Xk)⟩≤13‖∇f(Xk)‖‖PTXkSt(d,r)(∇φ(Xk))‖≤2D^fδk2.- 13 ∇f(X_k),∇ (X_k) ≤ 13\|∇ f(X_k)\|\|P_T_X_kSt(d,r)(∇ (X_k))\|≤ 2 D_f _k^2. Furthermore, ‖Δk‖2≤2αk2‖dk‖2+29‖∇φ(Xk)‖2≤2αk2‖dk‖2+43δk2.\| _k\|^2≤ 2 _k^2\|d_k\|^2+ 29\|∇ (X_k)\|^2≤ 2 _k^2\|d_k\|^2+ 43 _k^2. Here the last inequality uses the bound ‖∇φ(Xk)‖2≤6δk2\|∇ (X_k)\|^2≤ 6 _k^2 from the proof of Lemma 2. Substituting these estimates into (B.11) gives f(X¯k+1)−f(X¯k)≤−αk⟨∇^f(Xk),dk⟩+12δk+12+Cφδk2+C0αk2‖dk‖2.f( X_k+1)-f( X_k)≤- _k ∇f(X_k),d_k + 12 _k+1^2+C_ _k^2+C_0 _k^2\|d_k\|^2. Taking conditional expectation and applying Lemma B.3 proves (B.9). ∎ Theorem B.1 (Convergence of the Adam-style retraction-free update). Suppose that Assumption 1 and Condition B.1 hold. Let Xk\X_k\ be generated by (B.3). Let fmin:=infX∈St(d,r)f(X)>−∞f_ := _X (d,r)f(X)>-∞. Assume δ1≤1/8 _1≤ 1/8 and αk≤min1−q8h+G,a2C⋆,C⋆:=h+2[2C0+120(Cφ+12)], _k≤ \ 1-q8h_+G, a2C_ \, C_ :=h_+^2 [2C_0+120 (C_ + 12 ) ], where a is defined in Lemma B.3, and C0C_0 and CφC_ are defined in Lemma B.4. Then all iterates stay in U¯St(d,r)(1/8) U_St(d,r)(1/8), and for every K≥1K≥ 1, ∑k=1Kαk‖∇^f(Xk)‖2 _k=1^K _kE\| ∇f(X_k)\|^2 ≤2a[f(X¯1)−fmin+6(Cφ+12)δ12+C⋆σ2∑k=1Kαk2 ≤ 2a [f( X_1)-f_ +6 (C_ + 12 ) _1^2+C_ σ^2 _k=1^K _k^2 +Cβ∑k=1Kαk2β1,k2+CR∑k=1Kαkβ1,k2], +C_β _k=1^K _k^2 _1,k^2+C_R _k=1^K _k _1,k^2 ], (B.12) where Cβ:=h+2G2[8C0+480(Cφ+12)],CR:=h+2G22(1−β¯)h−.C_β:=h_+^2G^2 [8C_0+480 (C_ + 12 ) ], C_R:= h_+^2G^22(1- β)h_-. Moreover, ∑k=1K[δk2] _k=1^KE[ _k^2] ≤6δ12+120h+2∑k=1K−1αk2‖∇^f(Xk)‖2+120h+2σ2∑k=1K−1αk2 ≤ 6 _1^2+120h_+^2 _k=1^K-1 _k^2E\| ∇f(X_k)\|^2+120h_+^2σ^2 _k=1^K-1 _k^2 +480h+2G2∑k=1K−1αk2β1,k2. +480h_+^2G^2 _k=1^K-1 _k^2 _1,k^2. (B.13) Proof. The step-size condition and Lemma B.2 imply that all iterates remain in U¯St(d,r)(1/8) U_St(d,r)(1/8). Hence Lemma B.4 applies for k=1,…,Kk=1,…,K. Summing (B.9) gives [f(X¯K+1)]−f(X¯1) [f( X_K+1)]-f( X_1) ≤−a∑k=1Kαk‖∇^f(Xk)‖2+∑k=1KαkRk ≤-a _k=1^K _kE\| ∇f(X_k)\|^2+ _k=1^K _kR_k +12∑k=1K[δk+12]+Cφ∑k=1K[δk2]+C0∑k=1Kαk2‖dk‖2. + 12 _k=1^KE[ _k+1^2]+C_ _k=1^KE[ _k^2]+C_0 _k=1^K _k^2E\|d_k\|^2. Since f(X¯K+1)≥fminf( X_K+1)≥ f_ , a∑k=1Kαk‖∇^f(Xk)‖2 a _k=1^K _kE\| ∇f(X_k)\|^2 ≤f(X¯1)−fmin+∑k=1KαkRk+(Cφ+12)∑k=1K+1[δk2] ≤ f( X_1)-f_ + _k=1^K _kR_k+ (C_ + 12 ) _k=1^K+1E[ _k^2] +C0∑k=1Kαk2‖dk‖2. +C_0 _k=1^K _k^2E\|d_k\|^2. (B.14) By Lemmas B.1 and B.2, ‖dk‖2≤2h+2‖∇^f(Xk)‖2+2h+2σ2+8h+2β1,k2G2,E\|d_k\|^2≤ 2h_+^2E\| ∇f(X_k)\|^2+2h_+^2σ^2+8h_+^2 _1,k^2G^2, and ∑k=1K+1[δk2] _k=1^K+1E[ _k^2] ≤6δ12+120h+2∑k=1Kαk2‖∇^f(Xk)‖2+120h+2σ2∑k=1Kαk2 ≤ 6 _1^2+120h_+^2 _k=1^K _k^2E\| ∇f(X_k)\|^2+120h_+^2σ^2 _k=1^K _k^2 +480h+2G2∑k=1Kαk2β1,k2. +480h_+^2G^2 _k=1^K _k^2 _1,k^2. Substituting these two estimates into (B.14) yields a∑k=1Kαk‖∇^f(Xk)‖2 a _k=1^K _kE\| ∇f(X_k)\|^2 ≤f(X¯1)−fmin+6(Cφ+12)δ12+C⋆∑k=1Kαk2‖∇^f(Xk)‖2 ≤ f( X_1)-f_ +6 (C_ + 12 ) _1^2+C_ _k=1^K _k^2E\| ∇f(X_k)\|^2 +C⋆σ2∑k=1Kαk2+Cβ∑k=1Kαk2β1,k2+∑k=1KαkRk. +C_ σ^2 _k=1^K _k^2+C_β _k=1^K _k^2 _1,k^2+ _k=1^K _kR_k. Since αk≤a/(2C⋆) _k≤ a/(2C_ ), we have C⋆αk2≤(a/2)αkC_ _k^2≤(a/2) _k. Moving this term to the left-hand side gives a2∑k=1Kαk‖∇^f(Xk)‖2≤f(X¯1)−fmin+6(Cφ+12)δ12+C⋆σ2∑k=1Kαk2+Cβ∑k=1Kαk2β1,k2+∑k=1KαkRk. a2 _k=1^K _kE\| ∇f(X_k)\|^2≤ f( X_1)-f_ +6 (C_ + 12 ) _1^2+C_ σ^2 _k=1^K _k^2+C_β _k=1^K _k^2 _1,k^2+ _k=1^K _kR_k. Finally, Rk=CRβ1,k2R_k=C_R _1,k^2, which proves (B.12). The feasibility estimate (B.13) is exactly (B.7). ∎ Corollary B.1 (Iteration complexity). Suppose the assumptions of Theorem B.1 hold. (i) If αk≡α _k≡α and α≤min1,1−q8h+G,a2C⋆,α≤ \1, 1-q8h_+G, a2C_ \, then 1K∑k=1K‖∇^f(Xk)‖2=O(1αK+ασ2+1K∑k=1Kβ1,k2). 1K _k=1^KE\| ∇f(X_k)\|^2=O ( 1α K+ασ^2+ 1K _k=1^K _1,k^2 ). If, in addition, β1,k=β1λk−1 _1,k= _1λ^k-1 with λ∈(0,1)λ∈(0,1), then 1K∑k=1K‖∇^f(Xk)‖2=O(1αK+ασ2), 1K _k=1^KE\| ∇f(X_k)\|^2=O ( 1α K+ασ^2 ), and 1K∑k=1K[δk2]=O(1K+α2σ2). 1K _k=1^KE[ _k^2]=O ( 1K+α^2σ^2 ). In particular, in the full-gradient case σ=0σ=0 and with geometrically decaying β1,k _1,k, 1K∑k=1K[‖∇^f(Xk)‖2+δk2]=O(1K). 1K _k=1^KE [\| ∇f(X_k)\|^2+ _k^2 ]=O ( 1K ). (i) If αk=α0/k _k= _0/ k with α0≤min1−q8h+G,a2C⋆, _0≤ \ 1-q8h_+G, a2C_ \, and if β1,k=β1λk−1 _1,k= _1λ^k-1 with λ∈(0,1)λ∈(0,1), then min1≤k≤K‖∇^f(Xk)‖2=O(logK),1K∑k=1K[δk2]=O(logK). _1≤ k≤ KE\| ∇f(X_k)\|^2=O ( K K ), 1K _k=1^KE[ _k^2]=O ( KK ). Consequently, min1≤k≤K[‖∇^f(Xk)‖2+δk2]=O(logK). _1≤ k≤ KE [\| ∇f(X_k)\|^2+ _k^2 ]=O ( K K ). Proof. For constant αk≡α _k≡α, dividing (B.12) by αKα K gives 1K∑k=1K‖∇^f(Xk)‖2=O(1αK+ασ2+αK∑k=1Kβ1,k2+1K∑k=1Kβ1,k2). 1K _k=1^KE\| ∇f(X_k)\|^2=O ( 1α K+ασ^2+ αK _k=1^K _1,k^2+ 1K _k=1^K _1,k^2 ). Since α≤1α≤ 1, this implies the stated stationarity estimate. If β1,k=β1λk−1 _1,k= _1λ^k-1, then ∑k=1Kβ1,k2=O(1) _k=1^K _1,k^2=O(1), and hence the averaged stationarity estimate reduces to 1K∑k=1K‖∇^f(Xk)‖2=O(1αK+ασ2). 1K _k=1^KE\| ∇f(X_k)\|^2=O ( 1α K+ασ^2 ). Substituting this bound into (B.13) gives 1K∑k=1K[δk2]=O(1K+α2⋅1K∑k=1K‖∇^f(Xk)‖2+α2σ2+α2K∑k=1Kβ1,k2). 1K _k=1^KE[ _k^2]=O ( 1K+α^2· 1K _k=1^KE\| ∇f(X_k)\|^2+α^2σ^2+ α^2K _k=1^K _1,k^2 ). Using α≤1α≤ 1 and the geometrically decaying momentum sequence, this yields 1K∑k=1K[δk2]=O(1K+α2σ2). 1K _k=1^KE[ _k^2]=O ( 1K+α^2σ^2 ). The full-gradient statement follows by setting σ=0σ=0. For αk=α0/k _k= _0/ k, we have ∑k=1Kαk=Ω(K) _k=1^K _k= ( K) and ∑k=1Kαk2=O(logK) _k=1^K _k^2=O( K). If β1,k _1,k decays geometrically, then ∑k=1Kαk2β1,k2=O(1),∑k=1Kαkβ1,k2=O(1). _k=1^K _k^2 _1,k^2=O(1), _k=1^K _k _1,k^2=O(1). Dividing (B.12) by ∑k=1Kαk _k=1^K _k gives min1≤k≤K‖∇^f(Xk)‖2≤∑k=1Kαk‖∇^f(Xk)‖2∑k=1Kαk=O(logK). _1≤ k≤ KE\| ∇f(X_k)\|^2≤ _k=1^K _kE\| ∇f(X_k)\|^2 _k=1^K _k=O ( K K ). Moreover, since αk2≤α0αk _k^2≤ _0 _k, substituting the same step sizes into (B.13) yields ∑k=1K[δk2]=O(logK), _k=1^KE[ _k^2]=O( K), and hence 1K∑k=1K[δk2]=O(logK). 1K _k=1^KE[ _k^2]=O ( KK ). Finally, since αk≤α0 _k≤ _0, ∑k=1Kαk[δk2]≤α0∑k=1K[δk2]=O(logK). _k=1^K _kE[ _k^2]≤ _0 _k=1^KE[ _k^2]=O( K). Combining this estimate with the weighted stationarity estimate and dividing by ∑k=1Kαk=Ω(K) _k=1^K _k= ( K) gives min1≤k≤K[‖∇^f(Xk)‖2+δk2]=O(logK). _1≤ k≤ KE [\| ∇f(X_k)\|^2+ _k^2 ]=O ( K K ). ∎ Remark B.1 (Bias correction and standard Adam notation). If bias-corrected moments are used while keeping the preconditioner delayed, for example by replacing mkm_k and v¯k−1 v_k-1 with their corresponding bias-corrected versions in the definition of dkd_k, then the same proof applies after replacing h+h_+, h−h_-, and G by enlarged constants. Thus bias correction affects only constants and not the convergence rates. The non-delayed version that uses the current adaptive preconditioner depending on g~k g_k is not covered by the conditional-independence argument above. Remark B.2 (Constant first-momentum parameter). If β1,k≡β1 _1,k≡ _1 is kept constant, then the same proof yields, for constant step size, 1K∑k=1K‖∇^f(Xk)‖2=O(1αK+ασ2+β12). 1K _k=1^KE\| ∇f(X_k)\|^2=O ( 1α K+ασ^2+ _1^2 ). Thus, under this proof strategy, a non-vanishing first-momentum parameter leads to convergence to a neighborhood. To obtain convergence to a stationary point, one may use a decaying momentum sequence, for example β1,k=β1λk−1 _1,k= _1λ^k-1 with λ∈(0,1)λ∈(0,1).