Paper deep dive
FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment
Chuiyang Meng, Ming Tang, Vincent W. S. Wong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/21/2026, 12:31:00 AM
Summary
The paper introduces FLoRG, a federated fine-tuning framework for Large Language Models (LLMs) that addresses aggregation errors and decomposition drift inherent in standard Federated Low-Rank Adaptation (LoRA). FLoRG utilizes a single low-rank matrix and aggregates its Gram matrix to eliminate bilinear aggregation errors. It employs Procrustes alignment to minimize decomposition drift when recovering low-rank matrices from the aggregated Gram matrix, ensuring consistent updates across rounds. Theoretical analysis proves tighter convergence bounds, and experiments show FLoRG outperforms five state-of-the-art baselines with up to 2041x reduction in communication overhead.
Entities (11)
Relation Signals (11)
FLoRG → uses → Gram Matrix
confidence 95% · FLoRG ... employs a single low-rank matrix for fine-tuning and aggregates its Gram matrix
FLoRG → employs → Procrustes Alignment
confidence 94% · It also minimizes the decomposition drift by introducing a Procrustes alignment approach
Procrustes Alignment → mitigates → decomposition drift
confidence 94% · Procrustes alignment ... minimizes the decomposition drift
LoRA → suffersfrom → aggregation error
confidence 93% · the use of two separate low-rank matrices in LoRA for federated fine-tuning introduces ... aggregation error
LoRA → suffersfrom → decomposition drift
confidence 93% · it needs to decompose the aggregated matrix back into low-rank matrices ... can lead to decomposition drift
FLoRG → reduces → Communication Overhead
confidence 92% · FLoRG ... can reduce the communication overhead by up to 2041x
FLoRG → outperforms → FedEx-LoRA
confidence 90% · FLoRG outperforms five state-of-the-art baseline schemes ... including FedEx-LoRA
FLoRG → outperforms → FeDeRA
confidence 90% · FLoRG outperforms five state-of-the-art baseline schemes ... including FeDeRA
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Parameter-efficient fine-tuning techniques such as low-rank adaptation (LoRA) enable large language models (LLMs) to adapt to downstream tasks efficiently. Federated learning (FL) further facilitates this process by enabling collaborative fine-tuning across distributed clients without sharing private data. However, the use of two separate low-rank matrices in LoRA for federated fine-tuning introduces two types of challenges. First, aggregation error can arise from separately aggregating the two low-rank matrices. Second, even if the server aggregates the product of two low-rank matrices, it needs to decompose the aggregated matrix back into low-rank matrices. Since the decomposition is not unique, it can lead to decomposition drift. To tackle the aforementioned challenges, we propose federated low-rank Gram-matrix aggregation (FLoRG), a federated fine-tuning framework which employs a single low-rank matrix for fine-tuning and aggregates its Gram matrix (i.e., the matrix of inner products of its column vectors). FLoRG can eliminate the aggregation error and reduce the communication overhead. It also minimizes the decomposition drift by introducing a Procrustes alignment approach which aligns the decomposed matrix between consecutive fine-tuning rounds for consistent updates. We theoretically analyze the convergence of FLoRG and prove that adopting the Procrustes alignment results in a tighter convergence bound. Experimental results across multiple LLM fine-tuning benchmarks demonstrate that FLoRG outperforms five state-of-the-art baseline schemes by providing higher downstream task accuracy and can reduce the communication overhead by up to 2041$\times$.
Tags
Links
- Source: https://arxiv.org/abs/2602.17095v2
- Canonical: https://arxiv.org/abs/2602.17095v2
Trouble viewing inline? Open PDF directly →
Full Text
81,778 characters extracted from source content.
Expand or collapse full text
FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment Chuiyang Meng The University of British Columbia chuiyangmeng@ece.ubc.ca &Ming Tang Southern University of Science and Technology tangm3@sustech.edu.cn &Vincent W.S. Wong The University of British Columbia vincentw@ece.ubc.ca Corresponding author: Chuiyang Meng Abstract Parameter-efficient fine-tuning techniques such as low-rank adaptation (LoRA) enable large language models (LLMs) to adapt to downstream tasks efficiently. Federated learning (FL) further facilitates this process by enabling collaborative fine-tuning across distributed clients without sharing private data. However, the use of two separate low-rank matrices in LoRA for federated fine-tuning introduces two types of challenges. First, aggregation error can arise from separately aggregating the two low-rank matrices. Second, even if the server aggregates the product of two low-rank matrices, it needs to decompose the aggregated matrix back into low-rank matrices. Since the decomposition is not unique, it can lead to decomposition drift. To tackle the aforementioned challenges, we propose federated low-rank Gram-matrix aggregation (FLoRG), a federated fine-tuning framework which employs a single low-rank matrix for fine-tuning and aggregates its Gram matrix (i.e., the matrix of inner products of its column vectors). FLoRG can eliminate the aggregation error and reduce the communication overhead. It also minimizes the decomposition drift by introducing a Procrustes alignment approach which aligns the decomposed matrix between consecutive fine-tuning rounds for consistent updates. We theoretically analyze the convergence of FLoRG and prove that adopting the Procrustes alignment results in a tighter convergence bound. Experimental results across multiple LLM fine-tuning benchmarks demonstrate that FLoRG outperforms five state-of-the-art baseline schemes by providing higher downstream task accuracy and can reduce the communication overhead by up to 2041×. 1 Introduction Large language models (LLMs) (zhao2023survey; achiam2023gpt; touvron2023llama; grattafiori2024llama) have achieved state-of-the-art performance across a wide range of natural language processing tasks. However, their massive scale introduces critical challenges in terms of computation cost, memory consumption, and adaptability to downstream tasks. To address these concerns, low-rank adaptation (LoRA) (hu2022lora; hayou2024lora+; kopiczko2023vera; liu2024dora) has emerged as an effective approach. In particular, the LoRA module employs a fine-tuning matrix Δ with two low-rank matrices B and A into the pretrained model 0W^0 as =0+Δ=0+W=W^0+ =W^0+BA. Thus, it enables task-specific adaptation while only updating matrix Δ . This approach reduces both memory usage and computation overhead significantly when compared with full-model fine-tuning. However, fine-tuning still favors domain-specific data at scale. Such data is typically distributed across multiple clients and thus requires collaborative fine-tuning. To resolve this issue, federated learning (FL) (mcmahan2017communication; li2019convergence) provides a privacy-preserving framework for collaborative model training, where multiple clients fine-tune a shared global model without exposing their raw data. Combining LoRA with FL is therefore highly appealing: clients can collaboratively fine-tune a model by locally performing lightweight training via LoRA modules and uploading the low-rank matrix updates for global aggregation. The conventional works (10447454; fang2024automated; zhang2023fedpetuning; 10.1145/3637528.3671897) propose federated fine-tuning with LoRA, which enables each client n to transmit its low-rank matrices nB_n and nA_n to the central server. Afterwards, the central server aggregates nB_n and nA_n separately and then broadcasts the two aggregated matrices back to each client for performing fine-tuning in the subsequent rounds. In this case, the updated LoRA module after model aggregation can be expressed as (1N∑n=1Nn)(1N∑n=1Nn)( 1N _n=1^NB_n)( 1N _n=1^NA_n), where N is the number of clients. This approach introduces a challenge: The aggregation is fundamentally biased, because the true update should be 1N∑n=1N(nn) 1N _n=1^N(B_nA_n), which is different from (1N∑n=1Nn)(1N∑n=1Nn)( 1N _n=1^NB_n)( 1N _n=1^NA_n). This mismatch introduces a systematic aggregation error which affects the global model update in each fine-tuning round. As the number of rounds increases, the aggregation error is accumulated, which can degrade the fine-tuning performance. To alleviate the error induced by model aggregation, some works (yan2024federa; bai2024federated; yanfederated) calculate the product of nnB_nA_n and perform aggregation at the central server. Then, the central server performs matrix decomposition to recover two low-rank matrices for the next fine-tuning round. Although this approach can eliminate the error induced by separately aggregating matrices nB_n and nA_n, it introduces another non-trivial challenge: decomposition is generally not unique. In particular, the rank of matrices nB_n and nA_n module is typically much smaller than the dimension of the input or output of the parameter matrix. The aggregated matrix is typically rank-deficient and may also exhibit repeated eigenvalues. As a result, there may be multiple valid decompositions into two low-rank matrices. In addition, when the aggregated matrix has eigenvalue multiplicities, many valid decompositions exist. As a result, choosing different matrix decompositions fundamentally changes two low-rank matrices. It incurs a drift in the parameter subspace and changes the direction of the model update for the subsequent fine-tuning rounds. This drift will accumulate as the fine-tuning proceeds and degrade the fine-tuning performance. Furthermore, direct decomposition (e.g., eigendecomposition) may incur rank mismatch since the rank of the aggregated matrix may be different from the local low-rank matrices. Based on the aforementioned discussions, we focus on addressing the following question: Is there a federated fine-tuning approach which can eliminate the error induced by separate model aggregation while minimizing the drift induced by matrix decomposition? To address this question, we start by rethinking what to aggregate in federated fine-tuning. As LoRA involves matrix multiplication, either separate model aggregation or matrix decomposition is unavoidable. Therefore, one of our key insights is to reparameterize the LoRA module with a single low-rank Gram matrix. We aim to propose a federated fine-tuning framework which aggregates the corresponding Gram matrices to achieve an unbiased aggregation with a low communication overhead. Furthermore, another insight is to propose a Procrustes alignment approach to the decomposed matrix in order to stabilize the fine-tuning while preserving its Gram matrix, thereby mitigating the decomposition drift due to the non-uniqueness of matrix decomposition. Designing such a framework is challenging due to the following unexplored questions: (i) How can we design a low-rank parameterization which adapts to any pretrained matrix while supporting error-free aggregation? (i) How to optimize the Procrustes alignment matrix to minimize the decomposition drift? (i) How to characterize the overall convergence rate of the proposed framework under nonconvex losses, and disentangle the impact of Procrustes alignment on the convergence? In this work, we make the following contributions to address the aforementioned questions: • We propose FLoRG, a federated fine-tuning framework with a single low-rank matrix and Gram-matrix aggregation. By leveraging a shared semi-orthogonal basis, FLoRG adapts to parameter matrices with arbitrary dimensions. Each client only updates the single low-rank matrix, and the server aggregates the corresponding Gram matrix. FLoRG eliminates the aggregation error by turning the bilinear server-side aggregation into a linear operation. By transmitting a single matrix instead of two matrices, FLoRG is communication-efficient when compared with some of the existing federated LoRA schemes. • We propose a Procrustes alignment approach after matrix decomposition to preserve the aggregated Gram matrix while aligning the decomposed matrix across fine-tuning rounds to mitigate the decomposition drift. We formulate an optimization problem to minimize the decomposition drift via a Frobenius-norm objective. The closed-form optimal solution projects the decomposed matrix onto a target r-rank subspace without changing its Gram matrix. • We theoretically analyze the convergence rate of our proposed FLoRG in the nonconvex loss setting. In particular, incorporating our proposed Procrustes alignment mitigates the decomposition drift, thereby resulting in a tighter convergence bound. • We conduct extensive experiments on GLUE (wang2018glue) with MRPC, QQP, MNLI, QNLI, WNLI, and RTE datasets. We compare our proposed FLoRG with five state-of-the-art baseline schemes, including FedIT (10447454), FeDeRA (yan2024federa), FFA-LoRA (sun2024improving), FedSA-LoRA (guo2024selective), and FedEx-LoRA (singhal2024fedex). Results show that our proposed FLoRG achieves a higher testing accuracy than the baseline schemes under different settings and reduces the communication overhead by up to 2041×. 2 Related Works LoRA: Low-rank adaptation (LoRA) (hu2022lora) introduces two low-rank matrices to the pretrained model to perform parameter-efficient fine-tuning. Multiple LoRA variants have been proposed (zhang2023adalora; dettmers2023qlora; hayou2024lora+; kopiczko2023vera; liu2024dora; zhao2024galore; bensaid2025singlora). For example, in (zhang2023adalora), the authors proposed a LoRA framework to adaptively allocate the parameter budget among weight matrices based on the importance score. In (zhao2024galore), the authors projected the gradient matrix into a low-rank form to perform efficient fine-tuning. In (bensaid2025singlora), the authors reformulated the low-rank adaptation with a single matrix. While the aforementioned works have shown an improvement in the performance, they are primarily designed for centralized settings. Domain-specific data are often possessed by a number of distributed clients, which motivates the incorporation of FL. Federated Fine-tuning with LoRA: LoRA has been incorporated into FL to enable collaborative fine-tuning across distributed clients. The conventional federated fine-tuning works (10447454; fang2024automated; zhang2023fedpetuning; 10.1145/3637528.3671897; long2024dual; cho2024heterogeneous; byun-lee-2025-towards) directly aggregate two low-rank matrices separately to obtain the global model. Some works (babakniya2023slora; yan2024federa; bai2024federated; yanfederated) aggregate the LoRA modules (i.e., the products of two matrices) and then perform matrix decomposition to recover two low-rank matrices. In addition, the authors in (wang2024flora) proposed a stacking-based approach to aggregate the low-rank matrices. The authors in (sun2024improving) proposed to freeze matrix A and only update matrix B. The authors in (guo2024selective) proposed to let the clients locally update matrix B and only share matrix A for aggregation. 3 Methodology 3.1 FLoRG In this section, we propose a federated fine-tuning framework called FLoRG. FLoRG employs a single low-rank matrix and aggregates Gram matrices to eliminate the error induced by separate aggregation as in conventional LoRA. FLoRG performs Procrustes alignment to the decomposed matrix to minimize the decomposition drift. A schematic illustration is shown in Fig. 1. Figure 1: System model of our proposed FLoRG. We consider a central server and N clients. Let =1,2,…,TT=\1,2,…,T\ and =1,2,…,NN=\1,2,…,N\ denote the set of T fine-tuning rounds and the set of N clients, respectively. At the beginning of the first fine-tuning round, each client n∈n has the same pretrained weight matrix 0∈ℝdout×dinW^0 ^d_out× d_in. Note that 0W^0 is kept frozen and will not be updated during fine-tuning. We consider that each client n has a local dataset nD_n. Let n∼n ξ_n _n denote a mini-batch of training samples. Let Fn(t;n)F_n(W^t; ξ_n) denote the local loss function of client n on n ξ_n with model tW^t in the t-th fine-tuning round. We denote the expected loss of client n with model tW^t as fn(t)=n∼n[Fn(t;n)]f_n(W^t)=E_ ξ_n _n[F_n(W^t; ξ_n)]. Let ∇Fn(t;n)∈ℝdout×din∇ F_n(W^t; ξ_n) ^d_out× d_in denote the local stochastic gradient of client n with model tW^t in the t-th fine-tuning round. The learning procedure of FLoRG can be summarized in the following steps. (i) Local Update with Low-rank Gram Matrices: Different from LoRA, FLoRG uses a single matrix t∈ℝr×kA^t ^r× k for fine-tuning, where k=mindin,doutk= \d_in,d_out\ and r≪mindin,doutr \d_in,d_out\ denotes the rank of tA^t. Matrices ∈ℝdout×kL ^d_out× k and ∈ℝk×dinR ^k× d_in are initialized and shared across all clients. Both matrices L and R are semi-orthogonal, i.e., ⊺=kL L=I_k and ⊺=kRR =I_k, where kI_k denotes the k×k× k identity matrix. Note that matrices L and R remain unchanged during fine-tuning. The fine-tuning matrix in the t-th fine-tuning round is Δt=t=(t)⊺t,t∈, ^t=LQ^tR=L(A^t) A^tR, t , (1) where tQ^t denotes the square Gram parameter matrix in the t-th fine-tuning round. By leveraging L and R, FLoRG is compatible with parameter matrices of any dimensions. Thus, the full model can be expressed as t=0+ΔtW^t=W^0+ ^t. In the t-th fine-tuning round, the central server broadcasts tA^t to all clients. Client n∈n updates tA^t using its local dataset. Let ∇Fn(t;n)∈ℝr×k _AF_n(W^t; ξ_n) ^r× k denote the gradient of the low-rank matrix, which is given by ∇Fn(t;n)=t(nt+(nt)⊺),n∈,t∈, _AF_n(W^t; ξ_n)=A^t (H_n^t+(H_n^t) ), n ,t , (2) where nt=⊺∇Fn(t;n)⊺∈ℝk×kH_n^t=L ∇ F_n(W^t; ξ_n)R ^k× k. Client n performs stochastic gradient descent to update matrix tA^t. We define η as the learning rate. Let nt+12A_n^t+ 12 denote the locally updated low-rank matrix of client n in the t-th fine-tuning round, which is given by nt+12=t−η∇Fn(t;n),n∈,t∈. _n^t+ 12=A^t-η _AF_n(W^t; ξ_n), n ,t . (3) (i) Model Aggregation: When compared with the conventional federated LoRA schemes, in which clients must upload both locally-updated low-rank matrices, client n in FLoRG only needs to transmit nt+12A_n^t+ 12 to the central server, thereby reducing the per-round uplink communication overhead by more than half. Let nt+12=(nt+12)⊺nt+12Q_n^t+ 12= (A_n^t+ 12 ) A_n^t+ 12 denote the locally updated Gram matrix for client n in the t-th fine-tuning round. After receiving the locally updated parameter matrices from all clients, the central server performs model aggregation with respect to nt+12A_n^t+ 12 as t+1=1N∑n∈nt+12,t∈, ^t+1= 1N _n Q_n^t+ 12, t , (4) where t+1∈ℝk×kQ^t+1 ^k× k is the aggregated Gram matrix at the end of the t-th fine-tuning round. Due to the Gram matrix design, aggregating (nt+12)⊺nt+12 (A_n^t+ 12 ) A_n^t+ 12 is linear and preserves the positive semi-definite (PSD) property. Therefore, the central server can obtain the true aggregated matrix. This removes the bilinear inconsistency (i.e., aggregation error) induced by aggregating matrices nB_n and nA_n separately as in conventional federated LoRA schemes. Let r′,tr ,t denote the rank of matrix t+1Q^t+1, which satisfies rank(nt+12)≤r′,t=rank(t+1)≤(a)mink,Nr, (Q_n^t+ 12)≤ r ,t=rank(Q^t+1) (a)≤ \k,Nr \, (5) where inequality (a) follows from the subadditivity property of rank operator. (i) Decomposition with Procrustes Alignment: Since t+1Q^t+1 is a square PSD Gram matrix, the central server performs eigendecomposition to t+1Q^t+1 as follows: t+1=(t+1)⊺Λt+1t+1,t∈, ^t+1=(P^t+1) ^t+1P^t+1, t , (6) where t+1∈ℝr′,t×kP^t+1 ^r ,t× k. Λt+1∈ℝr′,t×r′,t ^t+1 ^r ,t× r ,t denotes the eigenvalue matrix of t+1Q^t+1. Thus, a canonical decomposition which satisfies (~t+1)⊺~t+1=t+1( A^t+1) A^t+1=Q^t+1 is ~t+1=(Λt+1)12t+1,t∈, A^t+1=( ^t+1) 12P^t+1, t , (7) where ~t+1 A^t+1 is a valid decomposition of t+1Q^t+1. However, directly applying ~t+1 A^t+1 for the subsequent fine-tuning round (i.e., t+1:=~t+1A^t+1:= A^t+1) may yield sub-optimal performance since the decomposition yields two challenges: non-unique decomposition and rank mismatch. First, the above expression of matrix ~t+1 A^t+1 provides a canonical representation of the decomposition of t+1Q^t+1, which is non-unique. This is because for any matrix with orthogonal columns ∈ℝr′,t×r′,tO ^r ,t× r ,t, we have (~t+1)⊺~t+1=t+1(O A^t+1) O A^t+1=Q^t+1. Furthermore, although the decomposition guarantees that (~t+1)⊺~t+1( A^t+1) A^t+1 preserves the aggregated Gram matrix, there exist many such decompositions due to the non-uniqueness of decompositions (e.g., Cholesky decomposition or singular value decomposition (SVD)) when the rank is deficient or the eigenvalues have multiplicities. However, this non-uniqueness affects the future fine-tuning. From the update rule in eqn. (2), each client performs local fine-tuning in the (t+1)(t+1)-th fine-tuning round using the gradient which depends explicitly on t+1A^t+1. As a result, different decompositions of t+1Q^t+1 may yield different t+1A^t+1 which result in divergent gradient paths. These paths may potentially lead to unstable fine-tuning performance across fine-tuning rounds. Second, as stated in inequality (5), the rank of the decomposed matrix may be different from that of the original one (i.e., r′,t≠r ,t≠ r). In such cases, we need to recover a matrix of the target rank r for consistency across fine-tuning rounds. The aforementioned challenges motivate reparameterizing matrix ~t+1 A^t+1 to stabilize the subsequent fine-tuning process while enforcing the target rank r. To this end, we propose Procrustes alignment to project ~t+1 A^t+1 onto the r-rank subspace. Let t∈ℝr×r′,tS^t ^r× r ,t denote this Procrustes alignment matrix in the t-th fine-tuning round. The matrix after projection is denoted as t~t+1S^t A^t+1. In particular, Procrustes alignment minimizes the Frobenius norm between the matrix after projection t~t+1S^t A^t+1 and tA^t, which minimizes the drift due to the non-uniqueness of matrix decomposition. We then formulate the following optimization problem in the t-th fine-tuning round: 1: _1: minimizet *minimize_S^t ‖t~t+1−t‖F2 \|S^t A^t+1-A^t \|_F^2 (8a) subjectto \>to (t)⊺t=r′,t. (S^t) S^t=I_r ,t. (8b) To solve problem 1P_1, we first convert the objective function (8a) into the following form: ‖t~t+1−t‖F2= \|S^t A^t+1-A^t \|_F^2= Tr((t)⊺t)+Tr((~t+1)⊺(t)⊺t~t+1)−2Tr(t(~t+1)⊺(t)⊺) \> ((A^t) A^t )+ (( A^t+1) (S^t) S^t A^t+1 )-2 (A^t( A^t+1) (S^t) ) = = ‖t‖F2+‖~t+1‖F2−2Tr(t(~t+1)⊺(t)⊺). \> \|A^t \|_F^2+ \| A^t+1 \|_F^2-2 (A^t( A^t+1) (S^t) ). (9) Since ‖t‖F2\|A^t\|_F^2 and ‖~t+1‖F2\| A^t+1\|_F^2 have been determined after matrix decomposition at the end of the t-th fine-tuning round, problem 1P_1 is equivalent to the following problem: 2: _2: maximizet *maximize_S^t Tr(t(~t+1)⊺(t)⊺) (A^t( A^t+1) (S^t) ) (10a) subjectto \>to constraint(8b). \>( c11). Then, we present the following theorem to obtain the optimal solution to problems 2P_2 and 1P_1, with the proof provided in Appendix A.1. Theorem 1. (Optimal Procrustes Alignment Matrix) We denote the SVD of matrix t(~t+1)⊺A^t( A^t+1) as t(~t+1)⊺=tt(t)⊺A^t( A^t+1) =U^t ^t(V^t) , where t∈ℝr×r′,tU^t ^r× r ,t and t∈ℝr′,t×r′,tV^t ^r ,t× r ,t have orthogonal columns. Let t=diag(σ1t,σ2t,…,σr′,t)∈ℝr′,t×r′,t ^t=diag( _1^t, _2^t,…, _r ,t^t) ^r ,t× r ,t denote the diagonal matrix of singular values of t(~t+1)⊺A^t( A^t+1) . The optimal solution t,⋆S^t, to problems 2P_2 and 1P_1 satisfies t,⋆=t(t)⊺. ^t, =U^t(V^t) . (11) The main benefits of our proposed Procrustes alignment are two-fold. First, it resolves the issue due to non-unique decomposition. In particular, the Procrustes alignment approach selects, among all valid decompositions of t+1Q^t+1, the one which is nearest to that in the last fine-tuning round (i.e., tA^t) in Frobenius norm. Thus, it stabilizes the gradients in the subsequent fine-tuning rounds. Second, it addresses the rank mismatch issue of the decomposed matrix by using a semi-orthogonal Procrustes alignment matrix to project ~t+1 A^t+1 with rank r′,tr ,t onto the target r-rank subspace. After the central server has determined matrix t,⋆S^t, , it calculates the low-rank matrix for the (t+1)(t+1)-th fine-tuning round as t+1=t,⋆~t+1A^t+1=S^t, A^t+1. Thus, the full model in the (t+1)(t+1)-th fine-tuning round is given by t+1=0+(t+1)⊺t+1. ^t+1=W^0+L(A^t+1) A^t+1R. (12) Then, the central server broadcasts the matrix t+1A^t+1 to all clients on the downlink. Note that the proposed FLoRG can effectively reduce the per-round communication overhead by more than half when compared with traditional federated LoRA schemes. For a model with L LoRA layers, the total per-round computation overhead of eigendecomposition and Procrustes alignment at the server is (Lk3)+(L(krr′,t+min(r,r′,t)2max(r,r′,t)))O(Lk^3)+O(L(krr ,t+ (r,r ,t)^2 (r,r ,t))). From this complexity analysis, the per-round server computation overhead scales linearly with the number of LoRA layers and is independent of the number of clients (beyond the standard Gram aggregation), indicating that FLoRG is scalable even with many layers and many clients. The workflow of our proposed FLoRG is presented in Appendix A.2. 3.2 Theoretical Analysis In this section, we analyze the convergence rate of our proposed FLoRG. We define the global loss function in the t-th fine-tuning round as f(t)=1N∑n∈fn(t)f(W^t)= 1N _n f_n(W^t), and its gradient as ∇f(t)=1N∑n∈∇fn(t)∈ℝdout×din∇ f(W^t)= 1N _n ∇ f_n(W^t) ^d_out× d_in. Without loss of generality, we consider nonconvex loss functions in our analysis. We first present the following assumptions which are widely used in the literature (e.g., (li2019convergence; wang2020tackling; guo2024selective)). Assumption 1. (L-Smoothness (li2019convergence; wang2020tackling)) The loss function of each client n is continuously differentiable and L-smooth. That is, for two arbitrary matrices tW^t and t+1W^t+1, we have fn(t+1)≤fn(t)+⟨∇fn(t),t+1−t⟩F+L2‖t+1−t‖F2,t∈,n∈.f_n(W^t+1)≤ f_n(W^t)+ ∇ f_n(W^t),W^t+1-W^t _F+ L2\|W^t+1-W^t\|_F^2,\>t ,\>n . Assumption 2. (Bounded Gradient (li2019convergence; wang2020tackling)) The local stochastic gradient of A is upper-bounded, i.e., n∼n[‖∇Fn(t;n)‖F2]≤ψ,t∈,n∈.E_ ξ_n _n [\| _AF_n(W^t; ξ_n)\|_F^2 ]≤ψ,\>t ,\>n . Assumption 3. (Bounded Parameter Space (guo2024selective)) The Frobenius norm of model parameter matrices tA^t and ~t A^t are upper-bounded by two positive constants C_A and C~ C_A, respectively, i.e., ‖t‖F≤C,‖~t‖F≤C~,t∈.\|A^t\|_F≤ C_A, \| A^t\|_F≤ C_A, t . In addition, we present two lemmas to facilitate our convergence analysis, with the proof provided in Appendices A.3 and A.4, respectively. Lemma 1. For illustration simplicity, we denote nt=∇Fn(t;n)∈ℝdout×dinG_n^t=∇ F_n(W^t; ξ_n) ^d_out× d_in. We denote t=1N∑n∈nt=1N∑n∈⊺nt⊺H^t= 1N _n H_n^t= 1N _n L G_n^tR . Let λmin() _ (X) denote the smallest positive eigenvalue of matrix X. For any matrices tA^t and tH^t, we have ⟨t,(t)⊺t(t+(t)⊺)+(t+(t)⊺)(t)⊺t⟩F≥4λmin((t)⊺t)‖t‖F2,t∈. \, ^t,\;(A^t) A^t (H^t+(H^t) )+ (H^t+(H^t) )(A^t) A^t _F≥ 4\, _ \! ((A^t) A^t )\; \|H^t \|_F^2,\>\>t . (13) Lemma 2. Let tS^t denote an arbitrary Procrustes alignment matrix in the t-th fine-tuning round. Let σmin() _ (X) denote the smallest positive singular value of matrix X. We define Δproct+1=‖t~t+1−t‖F2−‖t,⋆~t+1−t‖F2≥0 _proc^t+1= \|S^t A^t+1-A^t \|_F^2- \|S^t, A^t+1-A^t \|_F^2≥ 0. The difference of two Procrustes alignment matrices is bounded as follows: ‖t−t,⋆‖F2≤Δproct+1σmin(~t+1(t)⊺),t∈. \|S^t-S^t, \|_F^2≤ _proc^t+1 _ ( A^t+1(A^t) ), t . (14) Now, we present the convergence rate of our proposed FLoRG in the following theorem, with the proof provided in Appendix A.5. Theorem 2. (Convergence Rate of FLoRG) We denote the optimal model as ⋆W . Under Assumptions 1 −- 3 and Lemmas 1 −- 2, if the learning rate satisfies η<4mint∈λmin((t)⊺t)η<4 _t \ _ ((A^t) A^t )\, the convergence rate of our proposed FLoRG is bounded by 1T∑t∈‖∇f(t)‖F2≤ 1T _t \|∇ f(W^t) \|_F^2≤ f(1)−f(⋆)TΩ⏟Initial optimality gap+η2ψ22Ω+3Lη2ψ(η2ψ+2C2)2Ω⏟Residual bias term \> f(W^1)-f(W )T _Initial optimality gap+ η^2ψ^22 + 3Lη^2ψ (η^2ψ+2C_A^2 )2 _Residual bias term +ηψC~2∑t∈Δproct+1σmin(~t+1(t)⊺)2NTΩmint∈λmin((t)⊺t)⏟Procrustes alignment drift term, + ηψ C_A^2 _t _proc^t+1 _ ( A^t+1(A^t) )2NT _t \ _ ((A^t) A^t )\_Procrustes alignment drift term, (15) where Ω=2ηmint∈λmin((t)⊺t)−η22 =2η _t \ _ ((A^t) A^t )\- η^22. Based on the theoretical analysis, we can observe that the convergence rate of our proposed FLoRG depends on three terms. The first term is the initial optimality gap, which diminishes as the number of fine-tuning rounds T increases. The second term is the non-diminishing residual bias term. The third term is the Procrustes alignment drift term, which captures the impact of the Procrustes alignment on the convergence rate. When the Procrustes alignment is applied, Δproct+1 _proc^t+1 becomes zero. Hence, the third term becomes zero, under which we can achieve a tighter bound and improve the convergence rate. 4 Experimental Results 4.1 Experiment Setup Base Models, Datasets, and Baseline Schemes: We choose OPT-125M (zhang2022opt), RoBERTa-large (liu2019roberta), and Llama-3.2-3B (grattafiori2024llama) as three base models with three different scales. In particular, OPT-125M, RoBERTa-large, and Llama-3.2-3B have 125M, 355M, and 3B model parameters. We choose GLUE (wang2018glue) as a natural language understanding benchmark with MRPC, QQP, MNLI, QNLI, WNLI, and RTE datasets. In addition, we choose SQuAD (rajpurkar2016squad) v1.1 as a question-answering dataset. We compare the performance of our proposed FLoRG with the following baseline schemes: • FedIT (10447454): Client n∈n transmits the locally updated LoRA matrices nB_n and nA_n to the central server. The central server aggregates nB_n and nA_n separately. • FeDeRA (yan2024federa): Client n∈n transmits locally updated LoRA matrices nB_n and nA_n to the central server. The central server aggregates nnB_nA_n and performs SVD to obtain the updated matrices nB_n and nA_n. • FFA-LoRA (sun2024improving): Client n∈n freezes matrix nA_n and only updates matrix nB_n. The central server performs aggregation on matrix nB_n. • FedSA-LoRA (guo2024selective): Client n∈n locally updates matrices nB_n and nA_n but only shares matrix nA_n for aggregation. • FedEx-LoRA (singhal2024fedex): Client n∈n transmits locally updated LoRA matrices nB_n and nA_n to the central server. The central server aggregates nB_n and nA_n separately and determines a residual matrix to mitigate the aggregation error. Implementation Details: To show the learning performance of GLUE datasets, we show the testing accuracy. To characterize the communication overhead, we present the total number of parameters transmitted between all clients and the central server. For GLUE, to model the data heterogeneity across clients’ local datasets, we use the Dirichlet distribution Dir(ρ)Dir(ρ) to create non-independent and identically distributed (non-IID) data partitioning. In particular, ρ>0ρ>0 controls the degree of non-IIDness across clients’ local datasets. A lower value of ρ indicates a higher degree of data heterogeneity. To show the learning performance of the SQuAD v1.1 dataset, we show the exact match (EM) and F1 scores. For SQuAD v1.1, we consider the dataset to be independently and identically distributed (IID). In the ablation studies, we choose RoBERTa-large as the base model. Unless stated otherwise, we consider all clients participate in the fine-tuning and set η=5e−5η=5e-5, ρ=0.5ρ=0.5, N=20N=20, and r=4r=4. The details of the hyperparameter settings are presented in Appendix A.6. 4.2 Learning Performance In this section, we compare the testing accuracy of different schemes under different base models and datasets. Due to the space limit, we present the results under MNLI, QNLI, WNLI, and RTE datasets. The experiments are repeated under 2 random seeds, and we report the average testing accuracy. Results in Table 1 show that our proposed FLoRG outperforms the baseline schemes under those four datasets in most cases. In particular, on OPT-125M, FLoRG improves the testing accuracy over the strongest baseline by 1.52 on MNLI, 1.13 on WNLI, and 0.65 on RTE. On RoBERTa-large, the margins are 0.31 on MNLI, 0.45 on QNLI, 0.37 on WNLI, and 0.28 on RTE. On Llama-3.2-3B, FLoRG improves the testing accuracy over the strongest baseline by 0.41 on MNLI, 0.84 on WNLI, and 0.69 on RTE. Additional experimental results are presented in Appendix A.7. These results validate the superiority of FLoRG. Table 1: Comparison of the testing accuracy across different baseline schemes. Base Model Dataset FLoRG FedIT FeDeRA FFA-LoRA FedSA-LoRA FedEx-LoRA OPT-125M MNLI 87.35 79.42 81.15 83.54 84.61 85.83 QNLI 89.52 84.18 86.71 87.93 88.69 89.88 WNLI 65.28 58.45 59.34 62.61 62.83 64.15 RTE 68.92 61.08 64.51 66.02 67.39 68.27 RoBERTa-large MNLI 91.27 84.91 88.06 89.28 90.75 90.96 QNLI 92.58 87.49 90.07 90.96 91.54 92.13 WNLI 66.48 59.34 61.83 63.72 63.47 66.11 RTE 71.26 64.25 67.12 68.49 69.93 70.98 Llama-3.2-3B MNLI 93.15 87.24 89.83 91.05 92.38 92.74 QNLI 93.12 89.17 91.45 92.53 93.27 93.05 WNLI 68.73 61.52 64.19 65.97 66.81 67.89 RTE 73.84 67.08 69.75 71.33 72.56 73.15 4.3 Comparison of the Communication Overhead In this section, we compare the communication overhead incurred under different baseline schemes. We use the QNLI dataset to conduct the experiments. Results in Table 2 show that to achieve the target test accuracy, our proposed FLoRG uses a much lower total number of transmitted model parameters when compared with the baselines. This demonstrates that FLoRG can significantly reduce the communication overhead by up to 2041× when compared with the baseline schemes. Additional results of the computation overhead and memory usage are presented in Appendix A.7. Table 2: Comparison of the total number of transmitted parameters (upload and download) to achieve the target accuracy. Symbol “−-” means that the target accuracy cannot be achieved. Base Model Target Acc. FLoRG FedIT FeDeRA FFA-LoRA FedSA-LoRA FedEx-LoRA OPT-125M 80.00 8.2×8.2× 10^6 3.78×1073.78× 10^7 2.46×1072.46× 10^7 1.59×1071.59× 10^7 2.10×1072.10× 10^7 1.25×10101.25× 10^10 85.00 1.07×1.07× 10^7 −- 4.2×1074.2× 10^7 2.75×1072.75× 10^7 3.61×1073.61× 10^7 1.77×10101.77× 10^10 RoBERTa-large 80.00 8.7×8.7× 10^6 4.68×1074.68× 10^7 3.02×1073.02× 10^7 1.85×1071.85× 10^7 2.59×1072.59× 10^7 2.08×10102.08× 10^10 85.00 1.45×1.45× 10^7 8.12×1078.12× 10^7 5.17×1075.17× 10^7 3.35×1073.35× 10^7 4.42×1074.42× 10^7 2.96×10102.96× 10^10 4.4 Ablation Studies Impact of the Procrustes alignment In this subsection, we study the impact of our proposed Procrustes alignment on the learning performance. Results in Table 3 show that by applying Procrustes alignment, our proposed FLoRG yields a consistent improvement in terms of the testing accuracy. On OPT-125M, Procrustes alignment provides an improvement of 3.40 on MRPC, 2.86 on QQP, 6.27 on MNLI, 2.97 on QNLI, 5.60 on WNLI, and 4.45 on RTE, respectively. On RoBERTa-large, Procrustes alignment provides an improvement of 3.37 on MRPC, 2.84 on QQP, 2.46 on MNLI, 3.86 on QNLI, 4.34 on WNLI, and 4.31 on RTE, respectively, whereas FLoRG without Procrustes alignment can only achieve a comparable testing accuracy to FeDeRA, as shown in Table 1. It showcases the importance of our proposed Procrustes alignment to improve the fine-tuning performance. Table 3: Comparison of the testing accuracy of FLoRG with and without Procrustes alignment. Base Model FLoRG MRPC QQP MNLI QNLI WNLI RTE OPT-125M w/ Procrustes alignment 86.54 88.71 87.20 89.69 65.41 68.77 w/o Procrustes alignment 83.14 85.85 80.93 86.72 59.81 64.32 RoBERTa-large w/ Procrustes alignment 89.87 91.27 91.39 92.48 66.41 71.40 w/o Procrustes alignment 86.50 88.43 88.93 88.62 62.07 67.09 Impact of the Rank In this subsection, we vary r to demonstrate the impact of rank on the fine-tuning performance. In particular, we conduct experiments under r=2,4,8r=2,4,8, respectively. We present the results under the WNLI and RTE datasets. Results in Table 4 show that our proposed FLoRG outperforms the baseline schemes under different rank settings, demonstrating the robustness of our proposed FLoRG under various ranks. Additional experimental results can be found in Appendix A.7. Table 4: Comparison of the testing accuracy under different ranks. Rank Dataset FLoRG FedIT FeDeRA FFA-LoRA FedSA-LoRA FedEx-LoRA r=2r=2 WNLI 60.55 55.57 56.30 57.50 59.14 58.32 RTE 65.82 58.41 61.19 62.88 64.30 62.79 r=4r=4 WNLI 66.34 59.19 61.97 63.55 63.61 66.12 RTE 71.41 64.11 66.97 68.62 70.10 71.28 r=8r=8 WNLI 68.83 61.70 63.52 65.10 66.47 68.02 RTE 72.10 64.78 66.99 68.02 70.61 71.30 Robustness to the Data Heterogeneity In this subsection, we study the impact of the degree of data heterogeneity across clients’ local datasets on the fine-tuning performance. We present the results under the WNLI and RTE datasets. It can be observed in Table 5 that under different degrees of data heterogeneity, our proposed FLoRG outperforms the baseline schemes under more heterogeneous settings. In addition, as the degree of data heterogeneity increases (i.e., ρ decreases), the improvement over the baseline schemes also increases, showcasing the robustness and superiority of our proposed FLoRG under heterogeneous data settings. Additional experimental results are presented in Appendix A.7. Table 5: Comparison of the testing accuracy under different degrees of data heterogeneity. Non-IIDness Dataset FLoRG FedIT FeDeRA FFA-LoRA FedSA-LoRA FedEx-LoRA ρ=0.1ρ=0.1 WNLI 60.12 53.07 54.21 56.14 57.74 58.84 RTE 65.30 55.60 59.19 61.20 60.75 61.67 ρ=0.5ρ=0.5 WNLI 66.34 59.19 61.97 63.55 63.61 66.12 RTE 71.41 64.11 66.97 68.62 70.10 71.28 ρ=1ρ=1 WNLI 67.83 61.70 63.52 64.33 65.61 68.31 RTE 72.21 66.90 68.71 70.40 71.78 72.55 Matrix Initialization for Matrices L and R In this subsection, we show the impact of the initialization of matrices L and R on the learning performance. In particular, we compare our proposed semi-orthogonal initialization with Kaiming initialization (7410480) and SVD initialization (boutsidis2008svd). Results in Table 6 show that the semi-orthogonal approach outperforms the other two approaches in most cases, demonstrating the effectiveness of our proposed initialization approach for matrices L and R. Table 6: Comparison of the testing accuracy under different initialization schemes. Base Model Initialization MRPC QQP MNLI QNLI WNLI RTE OPT-125M Semi-orthogonal 86.54 88.71 87.20 89.69 65.41 68.77 Kaiming 84.35 88.90 85.23 87.73 62.29 69.31 SVD 86.41 87.69 83.19 88.74 64.37 67.69 RoBERTa-large Semi-orthogonal 89.87 91.27 91.39 92.48 66.41 71.40 Kaiming 87.68 92.34 89.11 91.57 64.19 70.32 SVD 88.70 90.37 91.49 91.45 65.08 71.40 Impact of the Client Availability In this subsection, we show the impact of the client availability on the learning performance. In particular, we conduct experiments under varying client participation ratios. Results in Table 7 show that under different ratios of client participation, our proposed FLoRG outperforms the baseline schemes in most cases, demonstrating its applicability. Table 7: Comparison of the testing accuracy under different client participation ratios. Client participation ratio Dataset FLoRG FedIT FeDeRA FFA-LoRA FedSA-LoRA FedEx-LoRA 0.20.2 WNLI 58.42 54.15 55.20 56.30 57.10 56.46 RTE 63.25 57.80 59.45 60.92 61.85 61.35 0.50.5 WNLI 64.50 58.10 60.35 61.80 62.40 63.58 RTE 69.15 62.50 65.20 66.75 68.20 68.61 11 WNLI 66.34 59.19 61.97 63.55 63.61 66.12 RTE 71.41 64.11 66.97 68.62 70.10 71.28 5 Conclusion In this work, we proposed a federated fine-tuning framework with low-rank Gram matrices called FLoRG. In particular, FLoRG features a single low-rank matrix instead of two low-rank matrices as in the conventional LoRA module. By transmitting this matrix and aggregating the corresponding Gram matrix, FLoRG eliminates the error induced by separately aggregating two matrices and significantly reduces the per-round communication overhead. Moreover, we proposed a Procrustes alignment approach to reparameterize the decomposed low-rank matrix after model aggregation. We theoretically analyzed the convergence rate of our proposed FLoRG framework and characterized the impact of our proposed Procrustes alignment on the convergence. Experimental results show that our proposed FLoRG framework achieves a higher testing accuracy and can reduce the communication overhead by up to 2041× when compared with five baseline schemes. References Appendix A Appendix A.1 Proof of Theorem 1 During the Procrustes alignment, we define t=(t)⊺tt∈ℝr′,t×r′,tZ^t=(U^t) S^tV^t ^r ,t× r ,t. Therefore, tS^t satisfies t=tt(t)⊺. ^t=U^tZ^t(V^t) . (16) Note that (t)⊺t=r′,t(S^t) S^t=I_r ,t. In addition, since matrices tU^t and tV^t have orthogonal columns, we have Tr(t(~t+1)⊺(t)⊺)= (A^t( A^t+1) (S^t) )= Tr(tt(t)⊺t(t)⊺(t)⊺) \> (U^t ^t(V^t) V^t(Z^t) (U^t) ) = = Tr(t(t)⊺) \> ( ^t(Z^t) ) = = ∑j=1r′,tσjtzj,jt. \> _j=1^r ,t _j^tz_j,j^t. (17) Since matrix tZ^t satisfies (t)⊺t=r′,t(Z^t) Z^t=I_r ,t, each element zj,jtz_j,j^t yields zj,jt≤1z_j,j^t≤ 1, we have Tr(t(~t+1)⊺(t)⊺)≤∑j=1r′,tσjt (A^t( A^t+1) (S^t) )≤ _j=1^r ,t _j^t, with the equality achieved at t=r′,tZ^t=I_r ,t. Therefore, when the maximum objective is achieved, matrix t,⋆S^t, satisfies t,⋆=tr′,t(t)⊺=t(t)⊺. ^t, =U^tI_r ,t(V^t) =U^t(V^t) . (18) This completes the proof of Theorem 1. A.2 Algorithm for FLoRG Algorithm 1 FLoRG 1: Input: Local fine-tuning datasets nD_n, n∈n ; learning rate η; pretrained model 0W^0; initialized low-rank matrix 1A^1. 2: The central server initializes L and R with orthogonal columns and broadcasts them to all clients. 3: 1:=0+(1)⊺1W^1:=W^0+L(A^1) A^1R. 4: For t∈t do 5: The central server broadcasts tA^t to all clients. 6: For client n∈n in parallel do 7: nt+12:=t−η∇Fn(t;n)A_n^t+ 12:=A^t-η _AF_n(W^t; ξ_n). 8: Transmits nt+12A_n^t+ 12 to the central server. 9: End for 10: t+1:=1N∑n∈(nt+12)⊺nt+12Q^t+1:= 1N _n (A_n^t+ 12 ) A_n^t+ 12. 11: Perform eigendecomposition to t+1Q^t+1 and obtain ~t+1:=(Λt+1)12t+1 A^t+1:=( ^t+1) 12P^t+1. 12: Perform Procrustes alignment to ~t+1 A^t+1 and obtain t,⋆:=t(t)⊺S^t, :=U^t(V^t) . 13: t+1:=t,⋆~t+1A^t+1:=S^t, A^t+1. 14: t+1:=0+(t+1)⊺t+1W^t+1:=W^0+L(A^t+1) A^t+1R. 15: End for 16: Output: Fine-tuned model T+1W^T+1. A.3 Proof of Lemma 1 ⟨t,(t)⊺t(t+(t)⊺)+(t+(t)⊺)(t)⊺t⟩F ^t,\;(A^t) A^t (H^t+(H^t) )+ (H^t+(H^t) )(A^t) A^t _F =(a) (a)= ⟨t+(t)⊺2,(t)⊺t(t+(t)⊺)+(t+(t)⊺)(t)⊺t⟩F \> H^t+(H^t) 2,\;(A^t) A^t (H^t+(H^t) )+ (H^t+(H^t) )(A^t) A^t _F = = 2⟨t+(t)⊺2,(t)⊺tt+(t)⊺2⟩F+2⟨t+(t)⊺2,t+(t)⊺2(t)⊺t⟩F \>2 H^t+(H^t) 2,\;(A^t) A^t H^t+(H^t) 2 _F+2 H^t+(H^t) 2,\; H^t+(H^t) 2(A^t) A^t _F ≥(b) (b)≥ 2λmin((t)⊺t)‖t+(t)⊺2‖F2+2λmin((t)⊺t)‖t+(t)⊺2‖F2 \>2\, _ \! ((A^t) A^t ) \| H^t+(H^t) 2 \|_F^\!2+2\, _ \! ((A^t) A^t ) \| H^t+(H^t) 2 \|_F^\!2 = = 4λmin((t)⊺t)‖t+(t)⊺2‖F2 \>4 _ \! ((A^t) A^t )\; \| H^t+(H^t) 2 \|_F^\!2 = = 4λmin((t)⊺t)‖t‖F2, \>4 _ \! ((A^t) A^t )\; \|H^t \|_F^\!2, (19) where equality (a) is obtained due to the fact that the second factor within the inner product is symmetric. Thus, the skew-symmetric part of tH^t is orthogonal. Inequality (b) results from (t)⊺t⪰λmin((t)⊺t)k(A^t) A^t _ ((A^t) A^t )I_k. This completes the proof of Lemma 1. A.4 Proof of Lemma 2 We expand ‖t~t+1−t‖F2 \|S^t A^t+1-A^t \|_F^2 as follows: ‖t~t+1−t‖F2=‖~t+1‖F2+‖t‖F2−2Tr(t~t+1(t)⊺). \|S^t A^t+1-A^t \|_F^2= \| A^t+1 \|_F^2+ \|A^t \|_F^2-2 (S^t A^t+1(A^t) ). (20) Similarly, we have ‖t,⋆~t+1−t‖F2=‖~t+1‖F2+‖t‖F2−2Tr(t,⋆~t+1(t)⊺). \|S^t, A^t+1-A^t \|_F^2= \| A^t+1 \|_F^2+ \|A^t \|_F^2-2 (S^t, A^t+1(A^t) ). (21) Therefore, Δproct+1 _proc^t+1 satisfies Δproct+1=2(Tr(t,⋆~t+1(t)⊺)−Tr(t~t+1(t)⊺)). _proc^t+1=2 ( (S^t, A^t+1(A^t) )- (S^t A^t+1(A^t) ) ). (22) Let t=~t+1(t)⊺=tttM^t= A^t+1(A^t) =U_M^t _M^tV_M^t, where matrices t∈ℝr′,t×r′,tU_M^t ^r ,t× r ,t and t∈ℝr′,t×rV_M^t ^r ,t× r have orthogonal columns, and t∈ℝr′,t×r′,t _M^t ^r ,t× r ,t. Let t=t(t)⊺(t)⊺P^t=U_M^t(S^t) (V_M^t) and t,⋆=t(t,⋆)⊺(t)⊺=r′,tP^t, =U_M^t(S^t, ) (V_M^t) =I_r ,t. Then, we have Tr(t~t+1(t)⊺)=Tr(tt)=∑j=1r′,tσjtpj,jt. (S^t A^t+1(A^t) )= (P^t _M^t )= _j=1^r ,t _j^tp_j,j^t. (23) Similarly, we have Tr(t,⋆~t+1(t)⊺)=∑j=1r′,tσjt. (S^t, A^t+1(A^t) )= _j=1^r ,t _j^t. (24) By combining eqns. (22), (23), and (24), we obtain Δproct+1=2∑j=1r′,tσjt(1−pj,jt). _proc^t+1=2 _j=1^r ,t _j^t(1-p_j,j^t). (25) Since matrices tU_M^t and tV_M^t have orthogonal columns, we have ‖t−t,⋆‖F2= \|S^t-S^t, \|_F^2= ‖t−r′,t‖F2 \> \|P^t-I_r ,t \|_F^2 = = Tr((t−r′,t)⊺(t−r′,t)) \> ((P^t-I_r ,t) (P^t-I_r ,t) ) = = 2r′,t−Tr(t)−Tr((t)⊺) \>2r ,t- (P^t)- ((P^t) ) = = 2∑j=1r′,t(1−pj,jt). \>2 _j=1^r ,t(1-p_j,j^t). (26) By combining eqns. (25) and (A.4), we have Δproct+1= _proc^t+1= 2∑j=1r′,tσjt(1−pj,jt) \>2 _j=1^r ,t _j^t(1-p_j,j^t) ≥(a) (a)≥ 2σmin(~t+1(t)⊺)∑j=1r′,t(1−pj,jt) \>2 _ ( A^t+1(A^t) ) _j=1^r ,t(1-p_j,j^t) = = σmin(~t+1(t)⊺)‖t−r′,t‖F2 \> _ ( A^t+1(A^t) ) \|P^t-I_r ,t \|_F^2 = = σmin(~t+1(t)⊺)‖t−t,⋆‖F2, \> _ ( A^t+1(A^t) ) \|S^t-S^t, \|_F^2, (27) where inequality (a) is obtained due to the fact that σmin(~t+1(t)⊺)≤σjt _ ( A^t+1(A^t) )≤ _j^t, 1≤j≤r′,t1≤ j≤ r ,t. By rearranging this inequality, we have ‖t−t,⋆‖F2≤Δproct+1σmin(~t+1(t)⊺). \|S^t-S^t, \|_F^2≤ _proc^t+1 _ ( A^t+1(A^t) ). (28) This completes the proof of Lemma 2. A.5 Proof of Theorem 2 Based on Assumption 1, we expand [f(t+1)]E[f(W^t+1)] as [f(t+1)]≤[f(t)]+[⟨∇f(t),t+1−t⟩F]⏟T1+L2[‖t+1−t‖F2]⏟T2. [f(W^t+1) ] [f(W^t) ]+ E [ ∇ f(W^t),W^t+1-W^t _F ]_T_1+ L2 E [ \|W^t+1-W^t \|_F^2 ]_T_2. (29) We define ¯t=1N∑n∈∇Fn(t;n)=t(t+(t)⊺) G_A^t= 1N _n _AF_n(W^t; ξ_n)=A^t (H^t+(H^t) ). t+1−tW^t+1-W^t satisfies t+1−t ^t+1-W^t = = 0+(t+1)⊺t+1−(0+(t)⊺t) \>W^0+L(A^t+1) A^t+1R- (W^0+L(A^t) A^tR ) = = (t+1)⊺t+1−(t)⊺t \>L(A^t+1) A^t+1R-L(A^t) A^tR = = 1N∑n∈((t+1)⊺t+1−(t)⊺t) \> 1N _n (L(A^t+1) A^t+1R-L(A^t) A^tR ) =(a) (a)= 1N∑n∈((nt+12)⊺nt+12−(t)⊺t) \>L 1N _n ((A_n^t+ 12) A_n^t+ 12-(A^t) A^t )R = = 1N∑n∈((t−η∇Fn(t;n))⊺(t−η∇Fn(t;n))−(t)⊺t) \>L 1N _n ( (A^t-η _AF_n(W^t; ξ_n) ) (A^t-η _AF_n(W^t; ξ_n) )-(A^t) A^t )R = = ((t)⊺t−η((t)⊺¯t+(¯t)⊺t)+η2(¯t)⊺¯t−(t)⊺t) \>L ((A^t) A^t-η ((A^t) G_A^t+( G_A^t) A^t )+η^2( G_A^t) G_A^t-(A^t) A^t )R = = (η2(¯t)⊺¯t−η((t)⊺¯t+(¯t)⊺t)), \>L (η^2( G_A^t) G_A^t-η ((A^t) G_A^t+( G_A^t) A^t ) )R, (30) where equality (a) is obtained due to the fact that 1N∑n∈(nt+12)⊺nt+12=t+1=(~t+1)⊺~t+1=(t,⋆~t+1)⊺t,⋆~t+1=(t+1)⊺t+1 1N _n (A_n^t+ 12) A_n^t+ 12=Q^t+1=( A^t+1) A^t+1=(S^t, A^t+1) S^t, A^t+1=(A^t+1) A^t+1. Let t,⋆A^t, denote the low-rank matrix obtained by applying the Procrustes alignment. We first bound T1T_1 as follows: T1 T_1 = = −η[⟨∇f(t),((t)⊺¯t+(¯t)⊺t)⟩F]+η2[⟨∇f(t),(¯t)⊺¯t⟩F] \>- [ ∇ f(W^t),L ((A^t) G_A^t+( G_A^t) A^t )R _F ]+η^2E [ ∇ f(W^t),L( G_A^t) G_A^tR _F ] = = η[⟨∇f(t),((t,⋆−t)⊺¯t+(¯t)⊺(t,⋆−t))⟩F]⏟T3 \> [ ∇ f(W^t),L ((A^t, -A^t) G_A^t+( G_A^t) (A^t, -A^t) )R _F ]_T_3 −η[⟨∇f(t),((t,⋆)⊺¯t+(¯t)⊺t)⟩F] - [ ∇ f(W^t),L ((A^t, ) G_A^t+( G_A^t) A^t )R _F ] +η2[⟨∇f(t),(¯t)⊺¯t⟩F] +η^2E [ ∇ f(W^t),L( G_A^t) G_A^tR _F ] =(a) (a)= T3−η[⟨t,((t)⊺¯t+(¯t)⊺t)⟩F]+η2[⟨t,(¯t)⊺¯t⟩F] \>T_3- [ ^t, ((A^t) G_A^t+( G_A^t) A^t ) _F ]+η^2E [ ^t,( G_A^t) G_A^t _F ] = = T3−η[⟨t,(t)⊺t(t+(t)⊺)+(t+(t)⊺)(t)⊺t⟩F] \>T_3- [ ^t,(A^t) A^t (H^t+(H^t) )+ (H^t+(H^t) )(A^t) A^t _F ] +η2[⟨t,(¯t)⊺¯t⟩F], +η^2E [ ^t,( G_A^t) G_A^t _F ], (31) where equality (a) is obtained due to the fact that for an arbitrary matrix X, we have ⟨∇f(t),⟩F=⟨t,⟩F ∇ f(W^t),LXR _F= ^t,X _F. T3T_3 can be bounded as follows: T3≤(a) T_3 (a)≤ βη2‖∇f(t)‖F2+η2β[‖((t,⋆−t)⊺¯t+(¯t)⊺(t,⋆−t))‖F2] \> βη2 \|∇ f(W^t) \|_F^2+ η2βE [ \|L ((A^t, -A^t) G_A^t+( G_A^t) (A^t, -A^t) )R \|_F^2 ] ≤(b) (b)≤ βη2‖∇f(t)‖F2+2ηβ[‖(t,⋆−t)⊺¯t‖F2] \> βη2 \|∇ f(W^t) \|_F^2+ 2ηβE [ \|(A^t, -A^t) G_A^t \|_F^2 ] ≤(c) (c)≤ βη2‖∇f(t)‖F2+2ηβ‖t,⋆−t‖F2[‖¯t‖F2] \> βη2 \|∇ f(W^t) \|_F^2+ 2ηβ \|A^t, -A^t \|_F^2E [ \| G_A^t \|_F^2 ] ≤(d) (d)≤ βη2‖∇f(t)‖F2+2ηβN2‖t,⋆−t‖F2∑n∈[‖∇Fn(t;n)‖F2] \> βη2 \|∇ f(W^t) \|_F^2+ 2ηβ N^2 \|A^t, -A^t \|_F^2 _n E [ \| _AF_n(W^t; ξ_n) \|_F^2 ] ≤(e) (e)≤ βη2‖∇f(t)‖F2+2ηψβN‖t,⋆−t‖F2 \> βη2 \|∇ f(W^t) \|_F^2+ 2ηψβ N \|A^t, -A^t \|_F^2 = = βη2‖∇f(t)‖F2+2ηψβN‖(t,⋆−t)~t‖F2 \> βη2 \|∇ f(W^t) \|_F^2+ 2ηψβ N \| (S^t, -S^t ) A^t \|_F^2 ≤(f) (f)≤ βη2‖∇f(t)‖F2+2ηψβN‖t,⋆−t‖F2‖~t‖F2 \> βη2 \|∇ f(W^t) \|_F^2+ 2ηψβ N \|S^t, -S^t \|_F^2 \| A^t \|_F^2 ≤(g) (g)≤ βη2‖∇f(t)‖F2+2ηψC~2Δproct+1βNσmin(~t+1(t)⊺), \> βη2 \|∇ f(W^t) \|_F^2+ 2ηψ C_A^2 _proc^t+1β N _ ( A^t+1(A^t) ), (32) where inequality (a) is obtained due to the fact that 2⟨,⟩F≤β‖F2+1β‖F22 ,Y _F≤β\|X\|_F^2+ 1β\|Y\|_F^2 holds for any β>0β>0. Inequalities (b) and (d) result from Jensen’s inequality. Inequalities (c) and (f) are obtained by using the fact ‖F≤‖F‖F\|AB\|_F≤\|A\|_F\|B\|_F. Inequality (e) results from Assumption 2. Inequality (g) results from Assumption 3 and Lemma 2. Then, we further bound the second term of T1T_1 as −η[⟨t,(t)⊺t(t+(t)⊺)+(t+(t)⊺)(t)⊺t⟩F] - [ ^t,(A^t) A^t (H^t+(H^t) )+ (H^t+(H^t) )(A^t) A^t _F ] ≤(a) (a)≤ −4ηλmin((t)⊺t)‖()⊺∇f(t)()⊺‖F2, \>-4η _ ((A^t) A^t ) \|(L) ∇ f(W^t)(R) \|_F^2, (33) where inequality (a) results from Lemma 1. Then, we bound η2[⟨t,(¯t)⊺¯t⟩F]η^2E [ ^t,( G_A^t) G_A^t _F ] as follows: η2[⟨t,(¯t)⊺¯t⟩F]≤(a) η^2E [ ^t,( G_A^t) G_A^t _F ] (a)≤ η2‖t‖F[‖(¯t)⊺¯t‖F] \>η^2 \|H^t \|_FE [ \|( G_A^t) G_A^t \|_F ] ≤(b) (b)≤ η2‖t‖F[‖¯t‖F2] \>η^2 \|H^t \|_FE [ \| G_A^t \|_F^2 ] ≤(c) (c)≤ η2‖t‖Fψ \>η^2 \|H^t \|_Fψ ≤(d) (d)≤ η22‖t‖F2+η22ψ2 \> η^22 \|H^t \|_F^2+ η^22ψ^2 ≤ ≤ η22‖()⊺∇f(t)()⊺‖F2+η2ψ22, \> η^22 \|(L) ∇ f(W^t)(R) \|_F^2+ η^2ψ^22, (34) where inequality (a) follows from Cauchy-Schwarz inequality. Inequality (b) results from the fact that ‖(¯t)⊺¯t‖F≤‖¯t‖F2\|( G_A^t) G_A^t\|_F≤\| G_A^t\|_F^2. Inequality (c) is obtained by using Assumption 2. Inequality (d) follows from Young’s inequality. By combining inequalities (A.5), (A.5), (A.5), and (A.5), we have T1≤2ηψC~2Δproct+1βNσmin(~t+1(t)⊺)+(−4ηλmin((t)⊺t)+η22+βη2)‖∇f(t)‖F2+η22ψ2. T_1≤ 2ηψ C_A^2 _proc^t+1β N _ ( A^t+1(A^t) )+ (-4η _ ((A^t) A^t )+ η^22+ βη2 ) \|∇ f(W^t) \|_F^2+ η^22ψ^2. (35) Now, we bound T2T_2. In particular, it satisfies T2= T_2= [‖(η2(¯t)⊺¯t−η((t)⊺¯t+(¯t)⊺t))‖F2] \>E [ \|L (η^2( G_A^t) G_A^t-η ((A^t) G_A^t+( G_A^t) A^t ) )R \|_F^2 ] ≤(a) (a)≤ 3η4[‖η2(¯t)⊺¯t‖F2]+6η2[‖η(t)⊺¯t‖F2] \>3η^4E [ \|Lη^2( G_A^t) G_A^tR \|_F^2 ]+6η^2E [ \|Lη(A^t) G_A^tR \|_F^2 ] ≤(b) (b)≤ 3η4[‖(¯t)⊺¯t‖F2]+6η2[‖(t)⊺¯t‖F2] \>3η^4E [ \|( G_A^t) G_A^t \|_F^2 ]+6η^2E [ \|(A^t) G_A^t \|_F^2 ] ≤(c) (c)≤ 3η4[‖¯t‖F4]+6η2[‖t‖F2][‖¯t‖F2] \>3η^4E [ \| G_A^t \|_F^4 ]+6η^2E [ \|A^t \|_F^2 ]E [ \| G_A^t \|_F^2 ] ≤(d) (d)≤ 3η2ψ(η2ψ+2C2). \>3η^2ψ (η^2ψ+2C_A^2 ). (36) where inequality (a) is obtained by using Jensen’s inequality. Inequality (b) is obtained since L and R have orthogonal columns. Inequality (c) results from the fact ‖F≤‖F‖F\|AB\|_F≤\|A\|_F\|B\|_F. Inequality (d) is obtained by using Assumption 2 and Assumption 3. Then, we combine inequalities (29), (35), and (A.5), we have f(t+1)≤ f(W^t+1)≤ f(t)+(η22+βη2−4ηλmin((t)⊺t))∥(∇f(t)∥F2 \>f(W^t)+ ( η^22+ βη2-4η _ ((A^t) A^t ) ) \|(∇ f(W^t) \|_F^2 +η2ψ22+3Lη2ψ(η2ψ+2C2)2+2ηψC~2Δproct+1βNσmin(~t+1(t)⊺). \>+ η^2ψ^22+ 3Lη^2ψ (η^2ψ+2C_A^2 )2+ 2ηψ C_A^2 _proc^t+1β N _ ( A^t+1(A^t) ). (37) We denote Ω=4ηmint∈λmin((t)⊺t)−η22−βη2 =4η _t \ _ ((A^t) A^t )\- η^22- βη2. We choose β=4mint∈λmin((t)⊺t)β=4 _t \ _ ((A^t) A^t )\. Then, we have Ω=2ηmint∈λmin((t)⊺t)−η22 =2η _t \ _ ((A^t) A^t )\- η^22. By rearranging inequality (A.5) and summing over all T fine-tuning rounds, we have Ω∑t∈‖∇f(t)‖F2≤ _t \|∇ f(W^t) \|_F^2≤ ∑t∈(f(t)−f(t+1))+T(η2ψ22+3Lη2ψ(η2ψ+2C2)2) \> _t (f(W^t)-f(W^t+1) )+T ( η^2ψ^22+ 3Lη^2ψ (η^2ψ+2C_A^2 )2 ) +∑t∈ηψC~2Δproct+12Nmint∈λmin((t)⊺t)σmin(~t+1(t)⊺) + _t ηψ C_A^2 _proc^t+12N _t \ _ ((A^t) A^t )\ _ ( A^t+1(A^t) ) ≤(a) (a)≤ f(1)−f(⋆)+T(η2ψ22+3Lη2ψ(η2ψ+2C2)2) \>f(W^1)-f(W )+T ( η^2ψ^22+ 3Lη^2ψ (η^2ψ+2C_A^2 )2 ) +ηψC~2∑t∈Δproct+1σmin(~t+1(t)⊺)2Nmint∈λmin((t)⊺t), + ηψ C_A^2 _t _proc^t+1 _ ( A^t+1(A^t) )2N _t \ _ ((A^t) A^t )\, (38) where inequality (a) is obtained due to the fact that f(⋆)≤f(T+1)f(W )≤ f(W^T+1). If η<4mint∈λmin((t)⊺t)η<4 _t \ _ ((A^t) A^t )\, we have Ω>0 >0. Then, we multiply 1TΩ 1T on both sides of inequality (A.5) and obtain 1T∑t∈‖∇f(t)‖F2≤ 1T _t \|∇ f(W^t) \|_F^2≤ f(1)−f(⋆)TΩ+η2ψ22Ω+3Lη2ψ(η2ψ+2C2)2Ω \> f(W^1)-f(W )T + η^2ψ^22 + 3Lη^2ψ (η^2ψ+2C_A^2 )2 +ηψC~2∑t∈Δproct+1σmin(~t+1(t)⊺)2NTΩmint∈λmin((t)⊺t). + ηψ C_A^2 _t _proc^t+1 _ ( A^t+1(A^t) )2NT _t \ _ ((A^t) A^t )\. (39) This completes the proof of Theorem 2. A.6 Hyperparameter Setting In our experiments, the batch size is set to be 4 by searching over a range of 2,4,8,16\2,4,8,16\. Each local training epoch is set to be 1. We select the optimal learning rate from the range 5e−4,1e−4,5e−5,1e−5\5e-4,1e-4,5e-5,1e-5\. The maximum sequence length is set to be 128, following the common practice in fine-tuning. We choose AdamW as the optimizer. We set the scaling factor of the LoRA module to be 16 for all algorithms. A.7 Additional Experimental Results Table 8: Comparison of the per-round client-side computation overhead (in FLOPs), per-round server-side computation overhead (in FLOPs), and memory usage (in MB). Symbol “−-” means that the overhead is negligible. Metrics FLoRG FedIT FeDeRA FFA-LoRA FedSA-LoRA FedEx-LoRA Client-side FLOPs 1.18×10121.18× 10^12 1.24×10121.24× 10^12 1.24×10121.24× 10^12 8.25×8.25× 10^11 1.24×10121.24× 10^12 1.18×10121.18× 10^12 Server-side FLOPs 8.50×10108.50× 10^10 “−-” 2.22×10112.22× 10^11 “−-” “−-” 2.09×1092.09× 10^9 Memory Usage 2815 2117 2117 2110 2117 3117 Computation overhead and memory usage: Table 8 compares the per-round computation overhead (client-side and server-side FLOPs) and the memory usage. On the client side, FLoRG incurs 1.18×10121.18× 10^12 FLOPs per round, which is comparable to FedIT/FeDeRA/FedSA-LoRA (1.24×10121.24× 10^12) and matches FedEx-LoRA (1.18×10121.18× 10^12), while FFA-LoRA achieves a lower client-side FLOPs (8.25×10118.25× 10^11) due to freezing one low-rank factor. On the server side, FLoRG introduces a non-negligible overhead (8.50×10108.50× 10^10 FLOPs) because it performs eigendecomposition and Procrustes alignment after Gram aggregation; nevertheless, its server-side overhead is substantially lower than FeDeRA (2.22×10112.22× 10^11 FLOPs), which relies on SVD-based decomposition. In terms of memory usage, FLoRG uses 2815 MB, which is higher than most baselines that only maintain two low-rank matrices (around 2110–2117 MB), but remains comparable in scale and is lower than FedEx-LoRA (3117 MB). Table 9: Comparison of the exact match (EM) and F1 scores on SQuAD v1.1. Base Model Metric FLoRG FedIT FeDeRA FFA-LoRA FedSA-LoRA FedEx-LoRA OPT-125M EM 68.52 62.31 64.78 63.19 67.14 66.85 F1 78.23 72.08 74.52 73.04 77.31 76.18 RoBERTa-large EM 86.34 81.53 83.12 81.69 84.47 86.02 F1 91.15 88.09 90.03 88.68 90.23 90.84 Llama-3.2-3B EM 89.83 84.21 86.58 85.37 87.12 88.25 F1 92.74 89.53 91.08 90.31 91.35 91.82 Question answering performance on SQuAD v1.1: Table 9 reports the exact match (EM) and F1 scores on SQuAD v1.1 under three base models. FLoRG achieves the best performance across all models and datasets. For OPT-125M, FLoRG improves EM/F1 to 68.52/78.23, outperforming the strongest baseline (e.g., FedSA-LoRA) by 1.38 EM and 0.92 F1. For RoBERTa-large, FLoRG reaches 86.34 EM and 91.15 F1, slightly exceeding the best competing method (FedEx-LoRA) by 0.32 EM and 0.31 F1. For Llama-3.2-3B, FLoRG obtains 89.83 EM and 92.74 F1, improving over the strongest baseline (FedEx-LoRA) by 1.58 EM and 0.92 F1. These results demonstrate that FLoRG generalizes beyond GLUE classification tasks and is also effective for extractive question and answering. Table 10: Additional comparison of the testing accuracy across different baseline schemes. Base Model Dataset FLoRG FedIT FeDeRA FFA-LoRA FedSA-LoRA OPT-125M MRPC 88.30 82.09 85.70 84.78 86.51 QQP 89.72 84.18 87.09 86.30 87.92 RoBERTa-large MRPC 90.80 85.16 88.30 87.50 89.21 QQP 91.52 87.89 89.30 88.51 90.01 Additional GLUE results: To complement Table 1 in the main content, Table 10 further reports the testing accuracy on MRPC and QQP datasets. FLoRG consistently outperforms the baselines on both OPT-125M and RoBERTa-large. In particular, on OPT-125M, FLoRG improves MRPC and QQP to 88.30 and 89.72, exceeding FedSA-LoRA by 1.79 and 1.80, respectively. On RoBERTa-large, FLoRG achieves 90.80 (MRPC) and 91.52 (QQP), exceeding FedSA-LoRA by 1.59 and 1.51, respectively. This reinforces the effectiveness of FLoRG across different GLUE tasks. Table 11: Additional comparison of the testing accuracy under different ranks. Rank Dataset FLoRG FedIT FeDeRA FFA-LoRA FedSA-LoRA r=2r=2 MRPC 85.91 79.66 83.17 82.32 83.00 QQP 87.62 81.88 84.90 84.09 85.65 MNLI 86.14 80.70 83.44 82.64 84.22 QNLI 89.70 86.67 87.22 86.40 88.10 r=4r=4 MRPC 90.80 85.16 88.30 87.50 89.21 QQP 91.52 87.89 89.30 88.51 90.01 MNLI 91.39 84.76 88.20 89.12 90.90 QNLI 92.44 87.63 89.91 90.82 91.69 r=8r=8 MRPC 91.70 85.55 87.12 88.20 89.91 QQP 91.80 86.39 89.26 88.45 90.10 MNLI 91.70 86.40 89.29 90.44 90.41 QNLI 92.50 87.77 90.20 90.41 91.13 Additional rank study: Table 11 provides more detailed results under different ranks (r∈2,4,8r∈\2,4,8\) on MRPC, QQP, MNLI, and QNLI (RoBERTa-large). FLoRG remains the top-performing method under all rank settings. Moreover, increasing r generally improves the testing accuracy for all methods, while the performance gap between FLoRG and the baselines persists, indicating that FLoRG is robust to rank choices and can better exploit higher-rank adaptation capacity when available. Table 12: Additional comparison of the testing accuracy under different degrees of data heterogeneity. Non-IIDness Dataset FLoRG FedIT FeDeRA FFA-LoRA FedSA-LoRA ρ=0.1ρ=0.1 MRPC 85.06 78.19 81.91 81.00 83.01 QQP 86.95 78.80 84.15 83.21 85.00 MNLI 81.35 75.44 78.42 77.57 79.36 QNLI 88.88 83.31 86.20 85.33 87.19 ρ=0.5ρ=0.5 MRPC 90.80 85.16 88.30 87.50 89.21 QQP 91.52 87.89 89.30 88.51 90.01 MNLI 91.39 84.76 88.20 89.12 90.90 QNLI 92.44 87.63 89.91 90.82 91.69 ρ=1ρ=1 MRPC 90.76 85.77 89.26 88.91 88.20 QQP 91.88 88.61 89.93 89.37 90.39 MNLI 91.75 86.47 90.05 91.70 91.33 QNLI 92.17 88.03 90.52 90.79 92.72 Robustness to data heterogeneity: Table 12 reports additional results under different degrees of non-IIDness controlled by ρ. Under highly heterogeneous settings (e.g., ρ=0.1ρ=0.1), FLoRG consistently outperforms the baseline schemes on MRPC/QQP/MNLI/QNLI. When the data distribution becomes less heterogeneous (ρ=0.5ρ=0.5), FLoRG still maintains consistent improvements over competing methods. In the near-IID case (ρ=1ρ=1), FLoRG remains competitive and achieves the best results on MRPC/QQP/MNLI, while FedSA-LoRA slightly surpasses FLoRG on QNLI. Overall, these results validate that FLoRG is robust across a wide range of client data heterogeneity levels, with particularly strong advantages under more heterogeneous federated settings. A.8 The Use of Large Language Models (LLMs) We used large language models (e.g., ChatGPT) for the general purpose of writing assistants to revise and polish the manuscript’s prose: improving grammar, wording, clarity, and fixing minor LaTeX/formatting issues. The models were not used for research ideation, designing algorithms/experiments, deriving proofs, selecting citations, or writing substantive technical content.