Paper deep dive
LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment
Haonan He, Xinyue Fan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/21/2026, 3:40:15 AM
Summary
The paper introduces LoRA-GA2, a fine-tuning algorithm for large language models that improves upon standard Low-Rank Adaptation (LoRA) by leveraging multi-step gradient information. Unlike previous methods that use single-step gradients or incur high memory costs, LoRA-GA2 uses a lightweight probe with the AdaLomo optimizer to accumulate gradients over a short trajectory. This multi-step gradient information is used for spectrum-aware rank allocation (combining sensitivity and effective rank) and SVD-based initialization of low-rank matrices. The method achieves superior performance on benchmarks like GLUE, GSM8K, and HumanEval while maintaining the memory efficiency of vanilla LoRA.
Entities (13)
Relation Signals (10)
LoRA-GA2 → achieveshighscoreon → GLUE
confidence 95% · LoRA-GA2 surpasses the leading baseline by an average of 0.66 points on the GLUE benchmark
LoRA-GA2 → achieveshighscoreon → HumanEval
confidence 95% · and 0.87 points on HumanEval, respectively.
LoRA-GA2 → achieveshighscoreon → GSM8K
confidence 95% · outperforms the strongest baseline by 1.03 points on GSM8K
LoRA-GA2 → outperforms → LoRA
confidence 95% · LoRA-GA2 consistently outperforms existing LoRA variants while preserving the efficiency advantages of vanilla LoRA.
LoRA-GA2 → usesgradienttype → Multi-step gradients
confidence 95% · LoRA-GA2... fully leverages multi-step gradient information.
LoRA-GA2 → uses → AdaLomo
confidence 92% · we employ AdaLomo (20), a memory-efficient optimizer... to serve as a lightweight probe for multi-step gradients.
LoRA-GA2 → usesmetricforrankallocation → Effective Rank
confidence 90% · but must be modulated by effective rank... This synergistic combination enables principled rank allocation
LoRA-GA2 → usesmetricforrankallocation → Sensitivity
confidence 90% · The key insight is that the required rank should grow with sensitivity... This synergistic combination enables principled rank allocation
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models, achieving competitive performance with reduced memory overhead. However, a persistent performance gap remains between LoRA and full fine-tuning. Recent studies have sought to narrow this gap by employing one-step gradient approximations of pretrained weights to align LoRA updates with the principal directions or intrinsic dimensionalities of full fine-tuning updates. Nevertheless, these approaches fail to capture the full dynamics of the gradients. In this paper, we propose LoRA-GA$^2$, an effective fine-tuning algorithm that fully leverages multi-step gradient information. Specifically, we introduce a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead. We further employ a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients. Extensive experimental results demonstrate that LoRA-GA$^2$ consistently outperforms existing LoRA variants while preserving the efficiency advantages of vanilla LoRA. For instance, LoRA-GA$^2$ surpasses the leading baseline by an average of 0.66 points on the GLUE benchmark, and outperforms the strongest baseline by 1.03 points on GSM8K and 0.87 points on HumanEval, respectively.
Tags
Links
- Source: https://arxiv.org/abs/2608.19800v1
- Canonical: https://arxiv.org/abs/2608.19800v1
Trouble viewing inline? Open PDF directly →
Full Text
70,513 characters extracted from source content.
Expand or collapse full text
LoRA-GA2: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment Haonan He Xinyue Fan Abstract Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models, achieving competitive performance with reduced memory overhead. However, a persistent performance gap remains between LoRA and full fine-tuning. Recent studies have sought to narrow this gap by employing one-step gradient approximations of pretrained weights to align LoRA updates with the principal directions or intrinsic dimensionalities of full fine-tuning updates. Nevertheless, these approaches fail to capture the full dynamics of the gradients. In this paper, we propose LoRA-GA2, an effective fine-tuning algorithm that fully leverages multi-step gradient information. Specifically, we introduce a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead. We further employ a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients. Extensive experimental results demonstrate that LoRA-GA2 consistently outperforms existing LoRA variants while preserving the efficiency advantages of vanilla LoRA. For instance, LoRA-GA2 surpasses the leading baseline by an average of 0.66 points on the GLUE benchmark, and outperforms the strongest baseline by 1.03 points on GSM8K and 0.87 points on HumanEval, respectively. Figure 1: Illustration of (a) LoRA; (b) One-step gradients based LoRA Variants; (c) Multi-step gradients based LoRA Variants; and (d) Ours (LoRA-GA2), which introduces an efficient probe for estimating multi-step gradients of pre-trained weights and utilizing the directions and intrinsic dimensionalities of the gradients to initialize low-rank weights. 1 Introduction The advent of open-source pre-trained large language models (LLMs), such as the Llama series (28; 7), has revolutionized natural language processing by enabling practitioners to adapt these powerful models to a wide range of downstream tasks through fine-tuning. However, full fine-tuning of LLMs entails substantial memory requirements, as modern optimizers like Adam (16) demand significant GPU memory to store 32-bit precision optimizer states for the entire model, often leading to out-of-memory issues under constrained resources. This memory bottleneck has spurred the development of Parameter-Efficient Fine-Tuning (PEFT) methods, among which Low-Rank Adaptation (LoRA) (13) has gained widespread adoption due to its strong empirical performance, zero inference latency, and straightforward implementation. LoRA is grounded in the hypothesis that weight updates during fine-tuning reside in low-rank subspaces. As shown in Figure 1(a), LoRA decomposes the weight update Δ∈ℝdout×din W ^d_out× d_in into low-rank matrices ∈ℝr×din A ^r× d_in and ∈ℝdout×r B ^d_out× r such that Δ=αr W= αr B A, where r≪min(dout,din)r (d_out,d_in) and α is a scaling factor. By freezing the pre-trained weights 0 W_0 and updating only A and B, LoRA significantly reduces the memory footprint of optimizer states. Despite its efficiency and superior performance relative to other PEFT methods, LoRA still underperforms full fine-tuning in many scenarios (1; 27). Over the past three years, a diverse array of LoRA variants has emerged to bridge the performance gap between LoRA and full fine-tuning. For instance, AdaLoRA (37) introduces an adaptive pruning strategy to eliminate less important ranks during training; LoRA+ (9) proposes a learning rate decoupling strategy to enhance training stability by mitigating the inherent imbalance between the down-projection and up-projection low-rank matrices. Among the various LoRA-based approaches, aligning low-rank updates with full fine-tuning updates via gradient information from pre-trained weights is considered a particularly promising direction for closing the gap, as LoRA adapters can be essentially interpreted as gradient compressors (8; 10). For example, LoRA-GA (30) presents a leading and easily implementable paradigm that computes accumulated one-step gradients prior to training and leverages the principal components to initialize the low-rank weights, effectively reducing the discrepancy between low-rank and full fine-tuning updates at the initial training step. LoRA-One (38) further substantiates the theoretical and empirical validity of this paradigm. However, existing gradient-guided methods suffer from critical limitations in both the scope of gradient information and the metrics used for rank allocation. First, single-step gradient approaches shown in the Figure 1(b) are inherently myopic; the gradient at the initial checkpoint fails to capture the complex optimization dynamics of the actual fine-tuning trajectory. Conversely, as shown in Figure 1(c), while multi-step alignment methods like LoRA-Pro (31) directly minimize the discrepancy between low-rank and full fine-tuning updates at each step, they do so at a severe cost. Modifying the low-rank gradients during training necessitates changes to the optimizer, drastically increases GPU memory consumption for optimizer states, prolongs training time, and suffers from poor compatibility with standard training pipelines. Second, current methods rely on one-sided metrics for rank allocation that fail to comprehensively evaluate the gradient’s structure. GoRA (10) allocates rank based solely on sensitivity-based importance scores, while RaLoRA (34) relies exclusively on intrinsic dimensionality derived from effective rank. Both metrics are suboptimal in isolation: a layer might be highly sensitive to the loss but have gradients concentrated in a single direction (requiring minimal rank capacity), or conversely, it might exhibit a high effective rank (widely dispersed gradients) but possess low overall sensitivity, making a high rank allocation wasteful. Relying on either metric alone leads to inefficient parameter distribution. This naturally raises the following question: How can we efficiently align low-rank updates with multi-step gradients’ directions, importances, and intrinsic dimensionalities without incurring unacceptable training overhead? To address this challenge, we employ AdaLomo (20), a memory-efficient optimizer that uses fused backward operations and in-training gradient accumulation, to serve as a lightweight probe for multi-step gradients. Unlike naive gradient averaging, we accumulate per-step gradients while updating weights along the training trajectory utilizing this lightweight optimizer, ensuring the accumulated gradient integrates information along the actual optimization path. After a few steps of accumulation, we restore the pre-trained weights and use this robust gradient for rank allocation and low-rank weight initialization. The subsequent training then proceeds seamlessly with common optimizers like Adam. This look-ahead strategy provides stable and representative gradient information that faithfully reflects the true fine-tuning dynamics, yet requires negligible time and zero permanent memory overhead. To this end, we propose LoRA-GA2 (Figure 1(d)) (Low-Rank Adaptation with Gradient Adaptive Alignment), a unified algorithm leveraging these multi-step accumulated gradients to simultaneously address rank allocation and initialization. Our approach introduces a novel gradient intrinsic dimensionality guided sensitivity-based importance score. The key insight is that the required rank should grow with sensitivity (more important layers need more capacity), but must be modulated by effective rank (concentrated gradients need less capacity than dispersed ones). This synergistic combination enables principled rank allocation that respects both the magnitude and the geometric structure of the gradients. For initialization, we design an SVD-based strategy that extracts principal directions from the accumulated multi-step gradients, providing a highly stable approximation of effective update directions without manipulating pre-trained weights. In summary, our core contributions are: • We systematically identify the informational deficiency of existing single-step gradient methods and the crippling computational overhead of continuous multi-step alignment methods. Furthermore, we reveal the theoretical blind spots of using solely sensitivity (e.g., GoRA) or effective rank (e.g., RaLoRA) for rank allocation. • We introduce a lightweight, multi-step gradient probing and initialization method, LoRA-GA2. By simulating the initial training trajectory via AdaLomo without permanent weight modifications, LoRA-GA2 extracts stable multi-step gradient information for rank allocation and initialization. This approach achieves superior empirical performance with a minimal time cost, no extra memory overhead, and seamless compatibility with modern distributed training frameworks. • We comprehensively evaluate LoRA-GA2 across diverse modalities, model architectures, and task complexities. Experimental results demonstrate that our method achieves the best average performance among the compared methods across natural language understanding (GLUE benchmark via T5), mathematical reasoning and code generation (GSM8K and HumanEval), and computer vision (image classification via CLIP), outperforming a wide array of recent initialization-optimized, convergence-optimized, and adaptive LoRA variants. 2 Related Work 2.1 LoRA Variants LoRA (13) has inspired numerous variants, which can be broadly categorized into three directions (21). Rank augmentation methods improve parameter utilization by dynamically reallocating ranks based on importance scores (37; 36; 10) or by stacking low-rank subspaces to increase effective rank (26; 18). Training dynamics optimization methods focus on stabilizing the learning process through improved scaling factors (15) or asymmetric learning rates (9). Initialization-based methods (22; 2; 30) leverage singular value decomposition or QR decomposition to initialize low-rank matrices, accelerating convergence and improving adaptation. 2.2 LoRA and Gradients of Pre-trained Weights Beyond this simple taxonomy, the connection between LoRA and gradient dynamics of pretrained weights has garnered increasing attention. FLoRA (8) interprets LoRA as a gradient compressor, demonstrating that low-rank matrices accumulate compressed gradient information during training. GaLore (39) projects gradients onto low-rank subspaces for memory-efficient training. LoRA-GA (30) initializes LoRA matrices using gradient singular features to minimize the difference between LoRA and full fine-tuning. GoRA (10) employs gradient sensitivity for adaptive rank allocation and pseudo-inverse initialization. However, these methods rely on single-step gradient estimates, which exhibit high variance and may not capture the persistent update directions essential for effective fine-tuning. 3 Method 3.1 LoRA as Gradient Compression LoRA can be analyzed as a structured gradient-compression mechanism. For a pretrained matrix 0 W_0 and a rank-r adapter, the effective update after T steps is constrained to the product space spanned by B A. Under the standard small-step approximation, or in the common setting where one low-rank factor is initialized to zero and the other determines the initial subspace, the accumulated LoRA update can be written as ΔT≈−η(αr)2∑t=1Tt0⊤0, W_T≈-η( αr)^2 _t=1^T G_t A_0 A_0, (1) where t=∂t/∂0 G_t=∂ L_t/∂ W_0 is the full-weight gradient. Thus, LoRA does not simply reduce parameter count; it projects the full gradient sequence through the adapter subspace. Vanilla random initialization makes this projection generic, so the approximation quality depends on whether 0⊤0 A_0 A_0 happens to preserve the dominant update directions. This view exposes two controllable error sources. First, the adapter subspace should align with the persistent gradient directions that appear during full fine-tuning process, not only with the directions of one-step gradients that computed without optimization steps. Second, the rank assigned to each layer should reflect how much gradient energy must be preserved in that layer. If l G^l has singular values σ1≥σ2≥⋯ _1≥ _2≥·s, then the best rank-r approximation preserves energy ∑i=1rσi2/∑jσj2 _i=1^r _i^2/ _j _j^2. A layer whose gradient spectrum is concentrated can be represented with a small rank, while a layer with dispersed singular values needs more capacity. LoRA-GA2 directly targets these two errors by estimating trajectory gradients and using their spectra for both allocation and initialization. 3.2 Trajectory-Aware Gradient Probe For a layer with frozen pretrained weight 0 W_0, LoRA computes =0+αr. h= W_0 x+ αr B A x. (2) The LoRA gradients are ∂/∂=αr⊤∂0∂ L/∂ A= αr B ∂ L∂ W_0 and ∂/∂=αr∂0⊤∂ L/∂ B= αr ∂ L∂ W_0 A , so adapter learning is controlled by how well the low-rank factors represent the full gradient subspace. Instead of estimating this subspace from a single batch, LoRA-GA2 collects avgl=1N∑i=1N∂i∂l|l=(i),l, G_avg^l= 1N _i=1^N . ∂ L_i∂ W^l |_ W^l= W^(i),l, (3) (i+1),l=(i),l−η(il). W^(i+1),l= W^(i),l- ( G_i^l). (4) where U is the AdaLomo update (20). Gradients are accumulated on CPU and the pretrained weights are restored after probing, thereby eliminating the parameter changes introduced during the probe. The accumulated gradients are retained only for rank allocation and adapter initialization. Figure 2: Framework of LoRA-GA2. A temporary multi-step probe collects trajectory gradients, restores the pretrained weights, and uses the accumulated signal for both rank allocation and SVD initialization before standard LoRA training. 3.3 Spectrum-Aware Rank Allocation Sensitivity measures task importance: Isensl=avg(|0l⊙avgl|).I_sens^l=avg(| W_0^l G_avg^l|). (5) This quantity approximates the first-order loss variation induced by perturbing the pretrained weights, and therefore identifies layers where downstream adaptation has large effect. However, sensitivity alone is not a rank requirement. Two layers can have the same gradient magnitude but very different singular-value spectra: if one gradient is nearly rank-one, a small adapter can capture most of its useful update; if another distributes energy across many directions, the same rank will lose much more information. Effective rank measures how many directions are needed to represent the accumulated gradient: erank(avgl)=exp(−∑ipilogpi),pi=σi∑jσj.erank( G_avg^l)= \! (- _ip_i p_i ), p_i= _i _j _j. (6) Effective rank alone is also insufficient because it ignores task importance. A layer may have a broad gradient spectrum but contribute little to the downstream loss, in which case assigning a large rank wastes the fixed LoRA budget. The two metrics therefore capture orthogonal properties: sensitivity measures how much the layer matters, while effective rank measures how many directions are needed to represent the useful update. We normalize both scores across target layers and define Sl=I¯sensl(I¯erankl)λ,S¯l=Sl∑kSk.S^l= I_sens^l( I_erank^l)^λ, S^l= S^l _kS^k. (7) Given reference rank rrefr_ref, the total budget is Ptotal=∑ldinl+doutlrrefP_total= _l d_in^l+d_out^l\,r_ref, and the allocated rank is rl=clip(⌊PtotalS¯ldinl+doutl⌋,rmin,rmax).r^l=clip ( P_total S^l d_in^l+d_out^l ,r_ ,r_ ). (8) The square-root shape factor prevents wide layers from consuming the entire budget. 3.4 SVD Initialization For each layer, we compute a truncated SVD of the negative accumulated gradient avgl=−avgl≈ll(l)⊤ D_avg^l=- G_avg^l≈ U^l ^l( V^l) and initialize the LoRA factors as 0l=ll/σ1lγ,0l=l(l)⊤/σ1lγ, B_0^l= U^l ^l/ _1^lγ, A_0^l= ^l( V^l) / _1^lγ, (9) where σ1l _1^l is the largest singular value of avgl D_avg^l. This places the initial adapter update along dominant descent directions. In the main experiments, we use N=64N=64, rref=8r_ref=8, and tune γ in a small stability range. Formal proofs and the approximation-error bound are provided in the supplement. Algorithm 1 LoRA-GA2: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment 1: Model f, weights 0ll=1L\ W_0^l\_l=1^L, steps N, rrefr_ref, γ, rmin,rmaxr_ ,r_ , λ. 2: Initialized 0l,0ll=1L\ A^l_0, B^l_0\_l=1^L. 3: Phase 1: Gradient Accumulation (AdaLomo) 4: Save pretrained weights 0l W_0^l to CPU, and set avgl← G_avg^l 0 for all l. 5: for i=1i=1 to N do 6: Sample batch, compute loss L, obtain il=FusedBackward(,l) G_i^l=FusedBackward( L, W^l) ⊳ AdaLomo fused kernel 7: Update l←l−η(il) W^l← W^l-η\,U( G_i^l) and accumulate avgl+=il/N G_avg^l += G_i^l/N ⊳ Adaptive LR & CPU accumulation 8: end for 9: Restore pretrained weights: l←CPUl W^l← W_CPU^l. 10: Phase 2: Importance Scores 11: Compute sensitivity Isensl=avg(|0l⊙avgl|)I_sens^l=avg(| W_0^l G_avg^l|) and effective rank Ierankl=erank(avgl)I_erank^l=erank( G_avg^l). 12: Normalize: S¯l=I¯sensl⋅(I¯erankl)λ∑kI¯sensk⋅(I¯erankk)λ S^l= I_sens^l·( I_erank^l)^λ _k I_sens^k·( I_erank^k)^λ ⊳ I¯ I denotes min-max normalization across layers 13: Phase 3: Rank Allocation 14: Total budget Ptotal←∑ldinl+doutl⋅rrefP_total← _l d_in^l+d_out^l· r_ref. 15: for l=1l=1 to L do 16: rl←clip(⌊Ptotal⋅S¯ldinl+doutl⌋,rmin,rmax)r^l \! ( P_total· S^l d_in^l+d_out^l ,\;r_ ,\;r_ ) 17: end for 18: Phase 4: SVD-based Initialization 19: for l=1l=1 to L do 20: (l,l,l)←SVDrl(avgl)( U^l, ^l, V^l) _r^l( G_avg^l) ⊳ Truncated SVD 21: 0l←ll/σ1lγ,0l←l(l)⊤/σ1lγ B_0^l← U^l ^l / _1^lγ, A_0^l← ^l( V^l) / _1^lγ ⊳ σ1l _1^l = largest singular value 22: end for 23: return 0l,0ll=1L\ A^l_0, B^l_0\_l=1^L 3.5 Why Multi-Step Alignment Helps The one-step estimator averages gradients only at 0 W_0, whereas LoRA training immediately moves through a sequence of nearby weights. For the quadratic loss ℒ()=12‖−⋆‖F2L( W)= 12\| W- W \|_F^2, a trajectory estimator has error bounded by a variance term plus a controllable drift term, σ2d/N+(η2N2)σ^2d/N+O(η^2N^2), while a static estimator retains the irreducible bias ‖0−⋆‖F2\| W_0- W \|_F^2. This stylized result explains why using many mini-batches at a frozen point is not equivalent to probing the early optimization path. The dual rank score is motivated by a complementary observation. Sensitivity alone can over-allocate rank to layers whose gradients are large but nearly one-dimensional; effective rank alone can over-allocate rank to layers whose gradients are dispersed but irrelevant to the downstream loss. Multiplying normalized sensitivity with normalized effective rank gives each layer capacity only when both conditions hold. The supplement provides the full compression-error and allocation derivations. 4 Experiments We evaluate LoRA-GA2 in three settings: T5-Base (25) on five tasks from GLUE (29), Llama-3.1-8B-Base (7) on GSM8K (6) and HumanEval (3) (trained on MetaMathQA-100K (35) and CodeFeedback-100K (32), respectively), and CLIP-ViT-B/16 (24) on seven image classification tasks (the classifier is constructed using text prompts such as “a photo of a class.”) (17; 5; 11; 12; 4; 33; 23). Unless otherwise specified, fixed-rank LoRA variants use rank 88, while adaptive-rank variants use a reference rank of 88 with a minimum rank of 44 and a maximum rank of 3232. We use the same optimizer settings as the corresponding baselines to ensure a fair comparison. All experiments are conducted on a single node equipped with four NVIDIA H200 GPUs. Each experiment is repeated using three different random seeds. 4.1 Experimental Settings For GLUE, we follow the settings of LoRA-GA (30) and GoRA (10): the Adam optimizer with a peak learning rate of 1×10−41× 10^-4, a batch size of 32, one training epoch, zero weight decay, cosine learning-rate decay with a warmup ratio of 0.03, a maximum sequence length of 128, and FP32 training. LoRA adapters are applied to all linear layers except the language head. For GSM8K (evaluated using accuracy) and HumanEval (evaluated using pass@1), we follow the setup of GoRA: AdamW with a peak learning rate of 5×10−55× 10^-5, a batch size of 64, one training epoch, zero weight decay, a maximum sequence length of 1024, BF16 model weights, and FP32 low-rank weights, with LoRA applied to the attention projection matrices. For CLIP-ViT-B/16, we follow RaLoRA (34) and fine-tune all linear layers using Adam with a peak learning rate of 1×10−41× 10^-4 and a batch size of 64. The number of training epochs for each image classification dataset is computed as max(1,round(1000Nbatch)) (1,round ( 1000N_batch ) ), where NbatchN_batch denotes the corresponding number of batches. Unless otherwise stated, we use a LoRA rank of 88, a LoRA scaling parameter α of 1616, and a LoRA dropout rate of 00 for LoRA and its variants. Table 1: Performance of fine-tuning T5-Base on 5 sub-tasks of the GLUE benchmark. Bold and underline indicate the highest and second-highest scores of low-rank methods with r=8r=8 or rref=8r^ref=8. Method MNLI SST-2 CoLA QNLI MRPC Average Full 86.33± 0.00 94.75± 0.21 80.70± 0.24 93.19± 0.22 84.56± 0.73 87.91 LoRA (13) 85.30± 0.04 94.04± 0.11 69.35± 0.05 93.19± 0.22 84.56± 0.73 85.29 Convergence Optimization Methods for LoRA rsLoRA (15) 85.73± 0.10 94.19± 0.23 72.32± 1.12 93.12± 0.09 52.86± 2.27 79.64 DoRA (19) 85.67± 0.09 94.04± 0.53 72.04± 0.94 93.04± 0.06 68.08± 0.51 82.57 LoRA+ (9) 85.81± 0.09 93.85± 0.24 77.53± 0.20 93.14± 0.03 74.43± 1.39 84.95 Initialization Optimization Methods for LoRA PiSSA (22) 85.75± 0.07 94.07± 0.06 74.27± 0.39 93.15± 0.14 76.31± 0.51 84.71 LoRA-GA (30) 85.70± 0.09 94.11± 0.18 80.57± 0.20 93.18± 0.06 85.29± 0.24 87.77 Adaptive Methods for LoRA AdaLoRA (37) 85.45± 0.11 93.69± 0.20 69.16± 0.24 91.66± 0.05 68.14± 0.28 81.62 RaLoRA (34) 85.76± 0.03 94.22± 0.29 78.11± 0.45 93.36± 0.14 84.74± 0.27 87.24 GoRA (10) 85.91± 0.22 94.68± 0.43 79.86± 0.35 93.27± 0.08 86.10± 0.20 87.96 LoRA-GA2 (Ours) 85.91± 0.01 94.72± 0.37 82.39± 0.24 93.19± 0.05 86.88± 0.14 88.62 Table 2: Performance of fine-tuning Llama3.1-8B-Base on GSM8K and HumanEval. Method GSM8K HumanEval Full 73.69±0.28± 0.28 51.63±1.27± 1.27 LoRA (13) 67.78±1.25± 1.25 43.09±0.35± 0.35 rsLoRA (15) 68.36±0.74± 0.74 45.78±2.80± 2.80 DoRA (19) 69.17±1.00± 1.00 43.70±1.54± 1.54 LoRA+ (9) 71.29±0.93± 0.93 44.51±2.11± 2.11 OLoRA (2) 68.54±0.42± 0.42 43.29±2.44± 2.44 PiSSA (22) 68.56±1.03± 1.03 44.10±1.54± 1.54 LoRA-GA (30) 71.39±0.90± 0.90 43.29±0.61± 0.61 AdaLoRA (37) 70.63±0.77± 0.77 41.46±3.66± 3.66 RaLoRA (34) 72.25± 0.59 48.78± 1.61 GoRA (10) 72.91± 0.76 48.98± 2.14 LoRA-GA2 (Ours) 73.94± 0.48 49.85± 0.33 Table 3: Performance of fine-tuning CLIP-ViT-B/16 on seven image classification tasks. Method Cars DTD EuroSAT GTSRB RESISC45 SUN397 SVHN Average Zero-shot 63.75 44.39 42.22 35.22 56.46 62.56 15.53 45.73 LoRA (13) 82.31± 0.08 76.97± 0.51 98.38± 0.20 97.10± 0.06 94.99± 0.11 77.19± 0.19 96.62± 0.06 89.08± 0.10 MELoRA (26) 82.65± 0.38 75.16± 0.59 98.64± 0.05 98.88± 0.05 95.78± 0.16 74.69± 0.22 96.95± 0.09 88.96± 0.15 MoRA (14) 84.61± 0.21 77.34± 0.14 98.65± 0.16 98.68± 0.18 96.33± 0.19 78.12± 0.06 97.17± 0.15 90.13± 0.16 AdaLoRA (37) 73.58± 0.09 73.79± 0.48 96.96± 0.12 58.87± 0.38 89.07± 0.60 72.00± 0.10 94.26± 0.13 79.79± 0.27 DoRA (19) 82.44± 0.26 76.86± 0.84 98.43± 0.17 97.25± 0.12 95.10± 0.16 77.30± 0.17 96.63± 0.04 89.14± 0.07 rsLoRA (15) 83.94± 0.22 77.64± 0.33 98.51± 0.17 98.69± 0.17 95.90± 0.20 77.96± 0.21 96.94± 0.06 89.94± 0.06 LoRA+ (9) 86.61± 0.36 73.33± 1.30 98.54± 0.14 98.99± 0.20 96.06± 0.38 76.80± 0.34 96.98± 0.08 89.62± 0.19 PiSSA (22) 83.36± 0.38 77.38± 0.57 98.54± 0.09 98.32± 0.09 95.92± 0.40 77.46± 0.13 97.00± 0.09 89.71± 0.25 OLoRA (2) 83.85± 0.13 78.60± 0.25 98.62± 0.03 98.49± 0.14 96.01± 0.28 77.30± 0.08 97.15± 0.14 90.00± 0.15 RaLoRA (34) 86.63± 0.30 77.75± 0.20 98.66± 0.27 98.98± 0.11 96.62± 0.28 77.86± 0.05 97.24± 0.11 90.53± 0.03 LoRA-GA2 (Ours) 87.82± 0.13 79.49± 0.11 98.83± 0.11 99.01± 0.08 96.48± 0.24 79.05± 0.14 97.42± 0.07 91.16± 0.05 4.2 Experimental Results Language understanding. On GLUE, LoRA-GA2 improves the average score to 88.62, outperforming GoRA by 0.66 points, LoRA-GA by 0.85 points, and full fine-tuning by 0.71 points under the reported setting. The largest gain appears on CoLA, where LoRA-GA2 reaches 82.39, improving over LoRA-GA by 1.82 points and over GoRA by 2.53 points. This task is sensitive to linguistic acceptability patterns, so the result is consistent with our claim that a short trajectory probe captures more stable early update directions than a one-step gradient. LoRA-GA2 also obtains the best MRPC score, suggesting that the same alignment helps on datasets with small scales. Reasoning and code. On Llama3.1-8B-Base, LoRA-GA2 improves over the strongest baseline GoRA by 1.03 points on GSM8K and 0.87 points on HumanEval. It also exceeds full fine-tuning on GSM8K by 0.25 points with reference rank 8, while HumanEval remains below full fine-tuning but substantially narrows the gap from LoRA’s 8.54-point deficit to 1.78 points. This pattern matches the motivation of the method: reasoning and code tasks expose the weakness of a randomly initialized low-rank subspace, and aligning the adapter with multi-step full-gradient directions helps recover part of the full fine-tuning trajectory without changing the formal LoRA training loop. As shown in Figure 3, LoRA-GA2 shows clear optimization and convergence advantages over LoRA for fine-tuning Llama3.1-8B-Base on MetamathQA, demonstrating the effectiveness of our method. Figure 3: Loss comparison of fine-tuning Llama3.1-8B-Base on MetamathQA using LoRA and LoRA-GA2 Vision transfer. On CLIP-ViT-B/16, LoRA-GA2 obtains a 91.16 average over seven image classification tasks, improving over RaLoRA by 0.63 points. It achieves the best result on six of seven datasets and is second best on RESISC45. The gains are not concentrated in a single dataset: Cars, DTD, SUN397, and SVHN all improve over the strongest baselines. This supports the claim that the combined sensitivity/effective-rank score is not tied to a single model family or modality. Since the CLIP experiments use the same reference-rank budget as the adaptive-rank baselines, the improvement is better interpreted as more effective allocation and initialization rather than a larger trainable parameter count. 5 Ablation and Efficiency As shown in Table 4, the main ablations isolate the two design choices: trajectory gradients and dual rank scoring. Multi-step gradients improve over a fixed-weight one-step variant on both GSM8K and HumanEval (73.94 vs. 72.13 on GSM8K; 49.85 vs. 49.65 on HumanEval). Removing SVD initialization causes the largest drop (68.92 on GSM8K), while replacing the dual score with sensitivity-only or effective-rank-only allocation also degrades performance. The supplement provides more ablation results on hyperparameters. The one-step comparison follows the same data budget as the trajectory probe: gradients are computed over 64 mini-batches, but the weights are kept fixed at the pretrained checkpoint. This controls for the number of gradient evaluations and isolates the effect of following the early optimization path. The 1.81-point GSM8K gap between one-step and multi-step variants indicates that the improvement is not merely due to averaging more minibatches; the location at which gradients are collected matters. HumanEval shows a smaller but consistent gain, suggesting that code generation is less sensitive to this particular trajectory signal but still benefits from more stable alignment. The component ablations separate two roles of the accumulated gradient. Without SVD initialization, the method loses the direct alignment between the initial adapter update and the dominant full-gradient directions, causing the largest GSM8K drop. Without adaptive rank allocation, the initialization remains useful but the fixed rank budget is not redistributed toward layers that are both important and spectrally complex. The score ablations further support the dual-signal design: sensitivity-only allocation ignores directional dispersion, while effective-rank-only allocation ignores task relevance. Their weaker performance explains why LoRA-GA2 combines both signals rather than treating either as a sufficient criterion. Table 4: Key ablations with standard deviations on Llama3.1-8B-Base. Variant GSM8K HumanEval One-step gradient 72.13±0.3472.13±0.34 49.65±1.0149.65±1.01 No SVD init. 68.92±0.1268.92±0.12 48.37±2.5048.37±2.50 No rank alloc. 72.40±1.3272.40±1.32 48.88±0.6348.88±0.63 Sensitivity only 72.98±0.6572.98±0.65 48.17±2.2848.17±2.28 Effective rank only 73.51±0.5173.51±0.51 48.78±1.0048.78±1.00 Full LoRA-GA2 73.94±0.4873.94 0.48 49.85±0.3349.85 0.33 The probing overhead is small relative to formal training. On Llama3.1-8B-Base, the multi-step gradient probe takes 5 minutes 55.83 seconds with 108,019 MB peak GPU memory, while the subsequent LoRA training takes 31 minutes 14.4 seconds with 137,695 MB peak GPU memory. Because all probe weights are restored before training, LoRA-GA2 does not change inference cost or checkpoint format. 6 Discussion Figure 4: Rank allocation results of fine-tuning Llama3.1-8B-Base-base on MetamathQA. The empirical pattern suggests that LoRA-GA2 is most useful when the initial gradient is not a sufficient proxy for the first phase of training. This is visible on GSM8K and CoLA, where the gains over LoRA-GA and GoRA are larger than on already-saturated classification tasks. The method also clarifies the roles of sensitivity and effective rank. Sensitivity answers whether a layer matters for the downstream loss; effective rank answers how many independent directions are required to represent its gradient. A layer with high sensitivity but low effective rank should receive enough rank to capture the dominant direction, but not a large budget. A layer with high effective rank but low sensitivity should also be restrained because its broad spectrum may not correspond to useful task adaptation. The multiplicative score enforces this interaction in a simple form. An example of rank allocation results of our method is shown in Figure 4. There are two limitations of our method. First, the probe introduces a small extra probing phase. Second, the current method uses a global exponent λ for all layers and tasks. A future version could adapt λ based on gradient-spectrum concentration or use a small validation set to choose it automatically. 7 Conclusion We presented LoRA-GA2, a LoRA variant with adaptive rank allocation and initialization based on multi-step gradient information. Specifically, we propose a lightweight probe for multi-step gradient approximation with no extra GPU memory cost and negligible extra computing time. Furthermore, we propose a spectrum-aware importance-based rank allocation and an SVD-based initialization for multi-step gradient alignment for improved alignment between low-rank adapters and directions, importances, and intrinsic dimensionalities of multi-step gradients. Extensive experiments demonstrate the effectiveness of our approach across diverse tasks and model architectures. References Biderman et al. (2024) D. Biderman, J. Portes, J. Jacobson, D. Baker, A. Ramezani-Moghadam, and S. Greene LoRA learns less and forgets less. arXiv preprint arXiv:2405.09673. Cited by: §1. Büyükakyüz (2024) K. Büyükakyüz OLoRA: orthonormal low-rank adaptation of large language models. arXiv preprint arXiv:2406.01775. Cited by: §2.1, Table 2, Table 3. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating large language models trained on code. External Links: 2107.03374, Link Cited by: §4. Cheng et al. (2017) G. Cheng, J. Han, and X. Lu Remote sensing image scene classification: benchmark and state of the art. Proceedings of the IEEE 105 (10), p. 1865–1883. Cited by: §4. Cimpoi et al. (2014) M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3606–3613. Cited by: §4. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. External Links: 2110.14168, Link Cited by: §A.1, §4. Dubey et al. (2024) A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §A.1, §1, §4. Hao et al. (2024) Y. Hao, Y. Liu, X. Li, Y. Dong, and L. Zhang Flora: low-rank adapters are secretly gradient compressors. arXiv preprint arXiv:2402.03293. Cited by: §1, §2.2. Hayou et al. (2024) S. Hayou, N. Ghosh, and B. Yu LoRA+: efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354. Cited by: §1, §2.1, Table 1, Table 2, Table 3. He et al. (2025) H. He et al. GoRA: gradient-driven adaptive low-rank adaptation for large language models. arXiv preprint arXiv:2502.12171. Cited by: §1, §1, §2.1, §2.2, §4.1, Table 1, Table 2. Helber et al. (2019) P. Helber, B. Bischke, A. Dengel, and D. Borth Eurosat: a novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12 (7), p. 2217–2226. Cited by: §4. Houben et al. (2013) S. Houben, J. Stallkamp, J. Salmen, M. Schlipsing, and C. Igel Detection of traffic signs in real-world images: the german traffic sign detection benchmark. In The 2013 international joint conference on neural networks (IJCNN), p. 1–8. Cited by: §4. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: §1, §2.1, Table 1, Table 2, Table 3. Jiang et al. (2024) T. Jiang, S. Huang, S. Luo, Z. Zhang, H. Huang, F. Wei, W. Deng, F. Sun, Q. Zhang, D. Wang, et al. Mora: high-rank updating for parameter-efficient fine-tuning. arXiv preprint arXiv:2405.12130. Cited by: Table 3. Kalajdzievski (2023) D. Kalajdzievski A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732. Cited by: §2.1, Table 1, Table 2, Table 3. Kingma and Ba (2014) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §1. Krause et al. (2013) J. Krause, M. Stark, J. Deng, and L. Fei-Fei 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, p. 554–561. Cited by: §4. Lialin et al. (2023) V. Lialin, N. Deshpande, and A. Rumshisky ReLoRA: high-rank training through low-rank updates. arXiv preprint arXiv:2307.05695. Cited by: §2.1. Liu et al. (2024) S. Liu, C. Wang, F. Liu, W. Chen, et al. DoRA: weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353. Cited by: Table 1, Table 2, Table 3. Lv et al. (2023) K. Lv et al. AdaLomo: low-memory optimization with adaptive learning rate. arXiv preprint arXiv:2310.10195. Cited by: §1, §3.2. Mao et al. (2024) Y. Mao et al. A survey on lora of large language models. arXiv preprint arXiv:2407.11046. Cited by: §2.1. Meng et al. (2024) F. Meng, Z. Wang, and C. Mu PiSSA: principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948. Cited by: §2.1, Table 1, Table 2, Table 3. Netzer et al. (2011) Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y. Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, Vol. 2011, p. 4. Cited by: §4. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. External Links: 2103.00020, Link Cited by: §4. Raffel et al. (2019) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR abs/1910.10683. External Links: Link, 1910.10683 Cited by: §4. Ren et al. (2024) R. Ren, Y. Wang, K. Liu, Z. Zhao, R. Zhang, H. Wei, et al. MELoRA: mini-ensemble low-rank adapters for parameter-efficient fine-tuning. arXiv preprint arXiv:2402.17295. Cited by: §2.1, Table 3. Shuttleworth et al. (2025) R. S. Shuttleworth, J. Andreas, A. Torralba, and P. Sharma LoRA vs full fine-tuning: an illusion of equivalence. External Links: Link Cited by: §1. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1. Wang et al. (2019) A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman GLUE: a multi-task benchmark and analysis platform for natural language understanding. External Links: 1804.07461, Link Cited by: §4. Wang et al. (2024) S. Wang, X. Li, Q. Ye, Y. Dong, and L. Zhang LoRA-ga: low-rank adaptation with gradient approximation. arXiv preprint arXiv:2407.05000. Cited by: §1, §2.1, §2.2, §4.1, Table 1, Table 2. Wang et al. (2025) Z. Wang, J. Liang, R. He, Z. Wang, and T. Tan LoRA-pro: are low-rank adapters properly optimized?. In The Thirteenth International Conference on Learning Representations, Cited by: §1. Weyssow et al. (2024) M. Weyssow, A. Kamanda, X. Zhou, and H. Sahraoui CodeUltraFeedback: an llm-as-a-judge dataset for aligning large language models to coding preferences. External Links: 2403.09032, Link Cited by: §4. Xiao et al. (2010) J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba Sun database: large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, p. 3485–3492. Cited by: §4. Ye et al. (2026) J. Ye, H. He, M. Li, F. Han, T. Chen, and P. Ye Gradient intrinsic dimensionality alignment: narrowing the gap between low-rank adaptation and full fine-tuning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §4.1, Table 1, Table 2, Table 3. Yu et al. (2024) L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu MetaMath: bootstrap your own mathematical questions for large language models. External Links: 2309.12284, Link Cited by: §A.1, §4. Zhang et al. (2023) L. Zhang et al. IncreLoRA: incremental parameter allocation strategy for parameter-efficient fine-tuning. arXiv preprint arXiv:2308.12043. Cited by: §2.1. Zhang et al. (2023) Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao AdaLoRA: adaptive budget allocation for parameter-efficient fine-tuning. In Proceedings of the International Conference on Learning Representations, Cited by: §1, §2.1, Table 1, Table 2, Table 3. Zhang et al. (2025) Y. Zhang, F. Liu, and Y. Chen Lora-one: one-step full gradient could suffice for fine-tuning large language models, provably and efficiently. arXiv preprint arXiv:2502.01235. Cited by: §1. Zhao et al. (2024) J. Zhao, Z. Zhang, B. Chen, Z. Wang, and A. Kyrillidis GaLore: memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507. Cited by: §2.2. Appendix A Implementation Details A.1 Hyperparameter Selection The key hyperparameters in LoRA-GA2 are as follows. (1) Gradient accumulation steps N: we use N=64N=64 for stable trajectory-gradient estimation. (2) Stability scaling γ: we increase γ over a power-of-two grid until the initialized adapter satisfies the initial-loss stability criterion described below. (3) Combined-score exponent λ: the default value 1.01.0 balances sensitivity and effective rank. (4) Rank bounds rminr_ and rmaxr_ : these are set relative to the reference rank rrefr_ref, typically as rmin=rref/2r_ =r_ref/2 and rmax=4rrefr_ =4r_ref. As shown in Figure 5, we additionally conduct an ablation study on N and γ by training Llama-3.1-8B-Base (7) on MetaMathQA-100K (35) and testing it on GSM8K (6) with a fixed random seed. Moderate values of N, such as 3232 and 6464, significantly improve performance. An excessively small N yields a high-variance estimate, whereas an excessively large N may include gradients from a probe trajectory that has drifted too far from the restored initialization; both effects can degrade downstream performance. Based on these observations, we use N=64N=64 in the main experiments. For γ, we evaluate candidates on a power-of-two grid. For each candidate, the pretrained weights are restored and the spectrally initialized LoRA branch is inserted. We accept the first candidate for which the resulting initial loss is lower than the corresponding starting loss of the restored model without a LoRA branch. This criterion directly prevents the nonzero initialization from producing an adverse initial-loss increase and requires only an initial-loss comparison for each candidate. In our search grid, small positive values up to 128128 can cause a sharp loss spike; increasing γ reduces the spectral magnitude of the initialization. Figure 5: Ablation of the hyperparameters N and γ. A.2 Probe Protocol and Accounting The temporary AdaLomo probe uses a fixed learning rate of 5×10−55× 10^-5, with no learning-rate schedule during its N steps. After the probe, all pretrained weights are restored before rank allocation, adapter initialization, and formal LoRA training. The formal-training data loader is restarted rather than advanced past the probe samples; consequently, mini-batches observed during probing may be used again during formal training. The reported probe time covers the complete preprocessing stage: the N forward/backward passes, AdaLomo updates, CPU gradient transfer and accumulation, restoration of the pretrained weights, sensitivity and effective-rank computation, rank allocation, truncated SVD, and construction of the initialized adapters. Thus, the reported timing is not restricted to gradient collection alone. A.3 Effective-Rank Computation Effective rank is computed from the full singular-value spectrum in FP32 on the device holding the accumulated gradient. Algorithm 2 gives implementation-equivalent pseudocode. We use ε=10−10 =10^-10. A non-matrix input, an empty spectrum, a numerically zero nuclear norm, or an SVD failure returns the conservative fallback value 11. Temporary tensors are released after the computation. Algorithm 2 Effective-rank computation 1: accumulated gradient G, ε=10−10 =10^-10 2: if G is not a two-dimensional matrix then 3: return 11 4: end if 5: cast G to FP32 on its current device 6: ←svdvals() σ ( G) ⊳ full spectrum 7: if the SVD fails or ||=0| σ|=0 then 8: return 11 9: end if 10: z←∑iσiz← _i _i 11: if z<εz< then 12: return 11 13: end if 14: pi←σi/zp_i← _i/z 15: q←exp(−∑ipilog(pi+ε))q← \! (- _ip_i (p_i+ ) ) 16: return max(1,q) (1,q) Sensitivity and effective rank are min–max normalized across target layers before they are combined. If all target layers have identical values and this normalization is degenerate, adaptive allocation is skipped and every target layer is assigned rrefr_ref. This fallback is therefore identical to the fixed-rank allocation of standard LoRA. A.4 Negative-Gradient Spectral Initialization Throughout the implementation, avgl G_avg^l denotes the accumulated loss gradient stored during the probe, while the spectral initialization is constructed from its descent direction. Explicitly, we define avgl=−avgl=−1N∑i=1N∂ℒi∂l|l=(i),l, D_avg^l=- G_avg^l=- 1N _i=1^N . _i∂ W^l |_ W^l= W^(i),l, (10) and compute its rank-rlr^l truncated SVD: avgl≈rlrl(rl)⊤. D_avg^l≈ U_r^l _r^l( V_r^l) . (11) Accordingly, SVD factors written compactly in terms of the accumulated gradient signal refer to the factors of avgl=−avgl D_avg^l=- G_avg^l. Equivalently, an implementation that decomposes avgl G_avg^l directly may absorb a minus sign into either one of the two LoRA factors. The LoRA factors are initialized as 0l=rlrlσ1lγ, B_0^l= U_r^l _r^l _1^lγ, (12) 0l=rl(rl)⊤σ1lγ, A_0^l= _r^l( V_r^l) _1^lγ, (13) where σ1l _1^l is the largest singular value of avgl D_avg^l. Consequently, 0l0l=(avgl)rlσ1lγ. B_0^l A_0^l= ( D_avg^l)_r^l _1^lγ. (14) Since avgl=−avgl D_avg^l=- G_avg^l, the initial effective LoRA perturbation is Δ0l=αrl0l0l=−αrlσ1lγ(avgl)rl. W_0^l= αr^l B_0^l A_0^l=- αr^l _1^lγ( G_avg^l)_r^l. (15) Thus, the nonzero adapter initialization is a controlled low-rank warm start aligned with the dominant accumulated descent directions. After gradient probing, we restore the original pretrained weights 0l W_0^l and keep them frozen during subsequent LoRA training. Since the adapter branch itself introduces a descent-aligned low-rank update, we do not apply an additional compensation or re-centering modification to 0l W_0^l; sufficient local descent conditions are given in Proposition 10. The adapter therefore begins training in the estimated task-relevant gradient subspace. The normalization by σ1lγ _1^lγ controls the magnitude of the initial perturbation: ‖0l0l‖2=1γ,‖αrl0l0l‖2=αrlγ. \| B_0^l A_0^l \|_2= 1γ, \| αr^l B_0^l A_0^l \|_2= αr^lγ. (16) Therefore, γ provides direct control over the spectral magnitude of the initialization. Using the negative gradient does not change layer-wise sensitivity, effective rank, or rank allocation. Indeed, avg(|0l⊙avgl|)=avg(|0l⊙avgl|),avg\! ( | W_0^l D_avg^l | )=avg\! ( | W_0^l G_avg^l | ), (17) and avgl D_avg^l and avgl G_avg^l have identical singular values. The sign convention affects only the orientation of the initialized adapter update. Appendix B Additional Theoretical Analysis Notation. For one adapted matrix, let ~t=0+ctt,c:=αr, W_t= W_0+c B_t A_t, c:= αr, (18) where t∈ℝr×din A_t ^r× d_in and t∈ℝdout×r B_t ^d_out× r. We write t:=∇~ℒt G_t:= _ WL_t for the full-weight gradient evaluated at ~t W_t. Unless stated otherwise, the factor optimizer is SGD with step size η. B.1 LoRA as Gradient Compression There are two complementary ways to derive the gradient-compression interpretation of LoRA. The first freezes one low-rank factor, t≡0 A_t≡ A_0, and trains only B. This setting yields an exact and transparent expression, but imposes a stronger assumption than standard LoRA. The second considers the usual coupled optimization of both A and B and shows that the same compressed-gradient expression remains the leading-order term over a short horizon. Proposition 1 (Exact compression with a frozen factor). Let ~t=0+ctt W_t= W_0+c B_t A_t, where c=α/rc=α/r. Suppose t≡0 A_t≡ A_0 and only B is trained by gradient descent. Then ~T−~0=−ηc2∑t=0T−1t0⊤0. W_T- W_0=-η c^2 _t=0^T-1 G_t A_0 A_0. (19) Thus, LoRA applies the fixed right-side compression operator c20⊤0c^2 A_0 A_0 to every full-weight gradient. Proof. Since ~t=0+ct0 W_t= W_0+c B_t A_0, the chain rule gives ∇ℒt=ct0⊤ _ BL_t=c G_t A_0 . Hence T−0=−ηc∑t=0T−1t0⊤. B_T- B_0=-η c _t=0^T-1 G_t A_0 . (20) Multiplying by c0c A_0 on the right proves equation 19. ∎ The preceding result is exact but assumes that the initial adapter subspace remains fixed. Standard LoRA updates both factors. The next proposition shows that, when 0= B_0=0 and the step is small, the frozen-subspace expression is the first nonvanishing term of the coupled dynamics. Proposition 2 (Coupled factors: leading-order dynamics). Let 0= B_0=0 and update t+1 A_t+1 =t−ηct⊤t, = A_t-η c B_t G_t, (21) t+1 B_t+1 =t−ηctt⊤. = B_t-η c G_t A_t . Assume, for 0≤t<T0≤ t<T, that ‖t‖2≤M,‖t‖F≤MF,‖0‖2≤a,ηcMT≤1.\| G_t\|_2≤ M, \| G_t\|_F≤ M_F, \| A_0\|_2≤ a, η cMT≤ 1. (22) Then ~T−~0=−ηc2∑t=0T−1t0⊤0+T, W_T- W_0=-η c^2 _t=0^T-1 G_t A_0 A_0+ R_T, (23) where, for an absolute constant C, ‖T‖F≤Cη3c4MFM2a2T3.\| R_T\|_F≤ Cη^3c^4M_FM^2a^2T^3. (24) Consequently, for fixed T, the frozen-subspace expression is the leading-order term as η→0η→ 0. Proof. Expanding one product update gives t+1t+1−tt= B_t+1 A_t+1- B_t A_t= −ηctt⊤t−ηctt⊤t -η c G_t A_t A_t-η c B_t B_t G_t (25) +η2c2tt⊤t⊤t. +η^2c^2 G_t A_t B_t G_t. Under equation 22, induction yields ‖t‖2≤2ηcMat,‖t−0‖2≤η2c2M2at(t−1),‖t‖2≤2a.\| B_t\|_2≤ 2η cMat, \| A_t- A_0\|_2≤η^2c^2M^2a\,t(t-1), \| A_t\|_2≤ 2a. (26) Insert these bounds into equation 25, sum over t, and multiply by the outer scale c. The difference between t⊤t A_t A_t and 0⊤0 A_0 A_0, the term containing tt⊤ B_t B_t , and the quadratic term are each bounded by a constant times η3c4MFM2a2T3η^3c^4M_FM^2a^2T^3. Absorbing constants into C proves the claim. ∎ Remark. The assumption 0= B_0=0 is used only to recover the one-sided expression emphasized in the main text. Equation equation 25 holds for arbitrary initial factors. In particular, when both factors are nonzero, ~1−~0= W_1- W_0= −ηc200⊤0−ηc200⊤0 -η c^2 G_0 A_0 A_0-η c^2 B_0 B_0 G_0 (27) +η2c300⊤0⊤0. +η^2c^3 G_0 A_0 B_0 G_0. Thus the nonzero spectral initialization used by LoRA-GA2 induces bidirectional compression to first order: 0 A_0 selects a right singular subspace and 0 B_0 selects a left singular subspace. B.2 Why a Task-Aligned Subspace Matters For a fixed gradient sequence tt=0T−1\ G_t\_t=0^T-1, let T:=∑t=0T−1t H_T:= _t=0^T-1 G_t. The corresponding full-SGD update is −ηT-η H_T, whereas the frozen-factor LoRA update is −ηT-η H_T P, with =c20⊤0 P=c^2 A_0 A_0. Their discrepancy is ηT(−).η H_T( I- P). (28) Proposition 3 (Optimal rank-r compression subspace). Let T=⊤ H_T= U V with singular values σ1≥⋯≥σp _1≥·s≥ _p. Among all rank-r orthogonal projectors P, min‖T(−)‖F2=∑i>rσi2. _ P\| H_T( I- P)\|_F^2= _i>r _i^2. (29) The minimum is attained by the projector onto the top-r right singular subspace of T H_T. Proof. For an orthogonal projector, ‖T(−)‖F2=∑iσi2‖(−)i‖22.\| H_T( I- P)\|_F^2= _i _i^2\|( I- P) v_i\|_2^2. (30) The minimum retains the r singular directions with the largest squared singular values, by the Eckart–Young–Mirsky theorem. ∎ Proposition 4 (Gaussian reconstruction error). Let ∈ℝr×d R ^r× d have independent entries Rij∼(0,1/r)R_ij (0,1/r). Then [⊤]=.E[ R R]= I. (31) However, for every fixed matrix G with d columns, ‖⊤−‖F2=d+1r‖F2.E\| G R R- G\|_F^2= d+1r\| G\|_F^2. (32) Thus Gaussian compression–decompression is unbiased but need not have small reconstruction error when r≪dr d. Proof. Equation equation 31 follows from [RkiRkj]=δij/rE[R_kiR_kj]= _ij/r. For a fixed vector x, rotational invariance allows =‖21 x=\| x\|_2 e_1. Writing Rki=Zki/rR_ki=Z_ki/ r gives ‖(⊤−)1‖22=2r+d−1r=d+1r.E\|( R R- I) e_1\|_2^2= 2r+ d-1r= d+1r. (33) Scaling by ‖22\| x\|_2^2 and summing over rows proves the claim. ∎ B.3 Why the Multi-Step Probe Helps The common estimation target below is the average population gradient along the early optimization trajectory. Under this target, the trajectory estimator directly samples the target and its mean-squared error contains only the sampling-variance term. The O(η2N2)O(η^2N^2) discrepancy discussed in the compact main-text account corresponds to the frozen-point bias of a static estimator that replaces the trajectory average by the gradient at the restored initialization; it is not an additional error term of the trajectory estimator under the common target used here. Proposition 5 (Trajectory versus static estimation). Vectorize the target weights and consider ℒ()=12‖−⋆‖22,0<η<1.L( w)= 12\| w- w \|_2^2, 0<η<1. (34) The population-gradient trajectory satisfies i=(1−η)i(0−⋆) g_i=(1-η)^i( w_0- w ). Define ¯N=1N∑i=0N−1i=aN(0−⋆),aN:=1−(1−η)NηN. g_N= 1N _i=0^N-1 g_i=a_N( w_0- w ), a_N:= 1-(1-η)^Nη N. (35) Assume noisy observations ~i=i+i g_i= g_i+ ξ_i are evaluated along this deterministic trajectory, where the i ξ_i are independent, zero mean, and satisfy ‖i‖22=dσ2E\| ξ_i\|_2^2=dσ^2. Then an N-sample trajectory estimator satisfies ‖^traj−¯N‖22=dσ2N.E\| g_traj- g_N\|_2^2= dσ^2N. (36) A static estimator using M samples at 0 w_0 satisfies ‖^static−¯N‖22=dσ2M+(1−aN)2‖0−⋆‖22.E\| g_static- g_N\|_2^2= dσ^2M+(1-a_N)^2\| w_0- w \|_2^2. (37) Thus additional static batches remove sampling variance but not the bias induced by evaluating every gradient at the initial point. Proof. The expression for aNa_N follows from a geometric series. The trajectory estimator is unbiased for ¯N g_N, and independence gives equation 36. The static estimator has bias (1−aN)(0−⋆)(1-a_N)( w_0- w ). The cross term between this deterministic bias and zero-mean noise vanishes, proving equation 37. ∎ Remark. For a short trajectory, 0≤1−aN≤η(N−1)2.0≤ 1-a_N≤ η(N-1)2. (38) Consequently, ‖∇ℒ(0)−¯N‖F2≤η2(N−1)24‖0−⋆‖F2.\| ( W_0)- G_N\|_F^2≤ η^2(N-1)^24\| W_0- W \|_F^2. (39) For fixed N and η, this term cannot be removed by increasing the number of mini-batches evaluated at 0 W_0. Thus static batch averaging and trajectory probing remain statistically distinct even with arbitrarily many static samples. This stylized result isolates the frozen-point bias; stochastic probe dynamics would introduce additional terms. B.4 Spectrum-Aware Rank Allocation Proposition 6 (Rank required for energy preservation). Let G have singular values σ1≥⋯≥σp≥0 _1≥·s≥ _p≥ 0. For ρ∈(0,1]ρ∈(0,1], define rρ:=minr:∑i=1rσi2≥ρ∑i=1pσi2.r_ρ:= \r: _i=1^r _i^2≥ρ _i=1^p _i^2 \. (40) There exists a rank-at-most-r matrix X satisfying ‖−‖F2≤(1−ρ)‖F2\| G- X\|_F^2≤(1-ρ)\| G\|_F^2 (41) if and only if r≥rρr≥ r_ρ. Proof. The Eckart–Young–Mirsky theorem gives minrank()≤r‖−‖F2=∑i>rσi2, _rank( X)≤ r\| G- X\|_F^2= _i>r _i^2, (42) which is equivalent to equation 40. ∎ Proposition 7 (Need for sensitivity and spectral complexity). Let Isens(,):=avg(|⊙|), I_sens( W, G):=avg(| W G|), (43) q():=exp(−∑ipilogpi), q( G):= \! (- _ip_i p_i ), (44) where pi=σi/∑jσjp_i= _i/ _j _j for a nonzero spectrum. Consistent with Algorithm 2, the implementation sets q()=1q( G)=1 when ∑jσj<ε _j _j< . Then: 1. q(a)=q()q(a G)=q( G) for a≠0a≠ 0, while Isens(,a)=|a|Isens(,)I_sens( W,a G)=|a|I_sens( W, G). 2. Sensitivity alone does not determine rank demand. For k≥2k≥ 2, let =k W= I_k, 1=diag(k,0,…,0) G_1=diag(k,0,…,0), and 2=k G_2= I_k. Both sensitivities equal 1/k1/k, but 1 G_1 has rank one, whereas preserving a fraction ρ of the energy of 2 G_2 requires rank at least ⌈ρk⌉ ρ k . Therefore a useful allocation score must depend on both task importance and spectral spread. Proof. Scaling all singular values by |a||a| leaves pip_i unchanged. For the second claim, both elementwise sensitivities equal the sum of diagonal magnitudes divided by k2k^2; the spectral claims follow from Proposition 6. ∎ Remark. The score Sl=I¯sensl(I¯erankl)λS^l= I_sens^l( I_erank^l)^λ (45) is a monotone combination of two necessary signals. Effective rank is only a scalar spectrum summary, so the product is not claimed to be the unique optimizer of the exact spectral tail. Proposition 8 (Proportional shape-adjusted allocation). Let sl>0s_l>0 be layer scores and κl>0 _l>0 per-rank surrogate costs. With bl:=κlrlb_l:= _lr_l, consider max∑l=1Lbl>0sllogbl,s.t. ∑l=1Lbl=P. _b_l>0 _l=1^Ls_l b_l, .t. _l=1^Lb_l=P. (46) Its unique solution is rl∗=Pκlsl∑ksk.r_l^*= P _l s_l _ks_k. (47) Taking sl=Sls_l=S^l and κl=dinl+doutl _l= d_in^l+d_out^l recovers the continuous, unclipped allocation used by LoRA-GA2. Proof. Strict concavity gives the stationarity condition sl/bl=μs_l/b_l=μ, so bl=sl/μb_l=s_l/μ. Enforcing ∑lbl=P _lb_l=P yields μ=(∑ksk)/Pμ=( _ks_k)/P and proves equation 47. ∎ Remark. The square-root shape factor defines a shape-adjusted capacity budget, not the exact number of LoRA parameters. The exact per-rank parameter cost is dinl+doutld_in^l+d_out^l. After integer rounding and clipping, any residual surrogate budget can be reassigned according to the largest marginal utility gain. If the layerwise normalization is degenerate, the implementation instead uses the fixed rl=rrefr_l=r_ref fallback described in Section A.3. B.5 Theory of Negative-Gradient Initialization The method computes the SVD of the negative trajectory gradient, making the nonzero adapter an intentional low-rank warm start rather than an arbitrary change to the restored model. Proposition 9 (Exact scale of the spectral warm start). Let −avg=⊤- G_avg= U V , with largest singular value σ1>0 _1>0, and write r:=rrr⊤ D_r:= U_r _r V_r . Initialize 0=rr1/2σ1γ,0=r1/2r⊤σ1γ. B_0= U_r _r^1/2 _1γ, A_0= _r^1/2 V_r _1γ. (48) Then c00=βr,β:=cσ1γ,c B_0 A_0=β D_r, β:= c _1γ, (49) and ‖c00‖2=cγ.\|c B_0 A_0\|_2= cγ. (50) Thus γ controls the spectral norm independently of the absolute gradient scale. If σ1=0 _1=0, the accumulated gradient vanishes and we use the zero adapter. Proof. Multiplying the factors gives 00=r/(σ1γ) B_0 A_0= D_r/( _1γ). Since the largest singular value of r D_r is σ1 _1, equation 50 follows. ∎ Proposition 10 (Descent guarantee without modifying 0 W_0). Let 0:=∇ℒ(0) G_0:= _ WL( W_0), and suppose ℒL is LsmL_sm-smooth on the line segment between 0 W_0 and 0+βr W_0+β D_r. Define a:=−⟨0,r⟩F‖r‖F2.a:=- G_0, D_r _F\| D_r\|_F^2. (51) If a>0a>0, then ~0=0+βr W_0= W_0+β D_r satisfies ℒ(~0)−ℒ(0)≤(−βa+Lsm2β2)‖r‖F2.L( W_0)-L( W_0)≤ (-β a+ L_sm2β^2 )\| D_r\|_F^2. (52) Hence the initialization is a strict descent step whenever 0<β<2aLsm.0<β< 2aL_sm. (53) No compensating change to the pretrained backbone is required: the additive LoRA branch implements the controlled low-rank warm start. Proof. Local smoothness gives ℒ(0+βr)≤ℒ(0)+β⟨0,r⟩F+Lsm2β2‖r‖F2.L( W_0+β D_r) ( W_0)+β G_0, D_r _F+ L_sm2β^2\| D_r\|_F^2. (54) Substituting equation 51 proves the bound, whose right-hand side is negative under equation 53. ∎ Corollary 11 (Exact-gradient case). If avg=0 G_avg= G_0, then r=−(0)r D_r=-( G_0)_r and a=1a=1. Therefore it is sufficient that 0<cσ1γ<2Lsm.0< c _1γ< 2L_sm. (55) Proof. Orthogonality of singular components gives −⟨0,r⟩F=‖(0)r‖F2=‖r‖F2- G_0, D_r _F=\|( G_0)_r\|_F^2=\| D_r\|_F^2. ∎ Corollary 12 (Trajectory-gradient alignment). Let :=0−avg E:= G_0- G_avg. Then a≥1−‖F‖r‖F.a≥ 1- \| E\|_F\| D_r\|_F. (56) In particular, if ‖F<‖r‖F\| E\|_F<\| D_r\|_F, then a>0a>0, and choosing γ large enough to satisfy equation 53 guarantees descent. Proof. Since r=−(avg)r D_r=-( G_avg)_r, −⟨0,r⟩F=‖r‖F2+⟨,(avg)r⟩F.- G_0, D_r _F=\| D_r\|_F^2+ E,( G_avg)_r _F. (57) Apply Cauchy–Schwarz and divide by ‖r‖F2\| D_r\|_F^2. ∎ Proposition 13 (Optimal local rank-r descent direction). Let 0 G_0 be the gradient at 0 W_0 and assume (0)r≠( G_0)_r 0. Among all perturbations with rank()≤rrank( )≤ r and ‖F≤τ\| \|_F≤τ, min⟨0,⟩F=−τ‖(0)r‖F, _ G_0, _F=-τ\|( G_0)_r\|_F, (58) attained by ∗=−τ(0)r‖(0)r‖F. ^*=-τ ( G_0)_r\|( G_0)_r\|_F. (59) Thus the negative-gradient singular subspace is locally optimal under rank and Frobenius-norm constraints. Proof. Von Neumann’s trace inequality and Cauchy–Schwarz give |⟨0,⟩F|≤∑i=1rσi(0)σi()≤‖(0)r‖F‖F.| G_0, _F|≤ _i=1^r _i( G_0) _i( )≤\|( G_0)_r\|_F\| \|_F. (60) The stated perturbation attains equality with negative sign. ∎ Interpretation. Early LoRA training compresses full gradients through its adapter subspace. A trajectory probe estimates directions encountered during early optimization rather than only at the restored point. The singular-value tail determines how much rank is needed in each layer, while sensitivity determines whether that layer matters to the task. Finally, the SVD of the negative trajectory gradient places the initialized adapter in a locally optimal low-rank descent subspace, and normalization by σ1γ _1γ makes the warm-start magnitude explicit and controllable.